Elgato_dark/hyperdx
1
0
Fork 0
mirror of https://github.com/hyperdxio/hyperdx synced 2026-04-21 13:37:15 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 6
hyperdx/packages/otel-collector/builder-config.yaml
Warren Lee 0a4fb15df2
[HDX-4029] Add commonly-used core and contrib components to OTel Collector builder-config (#2121) 
## Summary

Update `packages/otel-collector/builder-config.yaml` to include commonly-used components from the upstream [opentelemetry-collector](https://github.com/open-telemetry/opentelemetry-collector) core and [opentelemetry-collector-contrib](https://github.com/open-telemetry/opentelemetry-collector-contrib) distributions. This gives users more flexibility in their custom OTel configs without pulling in the entire contrib distribution (which causes very long compile times).

Also adds Go module and build cache mounts to the OCB Docker build stage for faster rebuilds, and bumps CI timeouts for integration and smoke test jobs to account for the larger binary.

### Core extensions added (2)

- `memorylimiterextension` — memory-based limiting at the extension level
- `zpagesextension` — zPages debugging endpoints

### Contrib receivers added (4)

- `dockerstatsreceiver` — container metrics from Docker
- `filelogreceiver` — tail log files
- `k8sclusterreceiver` — Kubernetes cluster-level metrics
- `kubeletstatsreceiver` — node/pod/container metrics from kubelet

### Contrib processors added (12)

- `attributesprocessor` — insert/update/delete/hash attributes
- `cumulativetodeltaprocessor` — convert cumulative metrics to delta
- `filterprocessor` — drop unwanted telemetry
- `groupbyattrsprocessor` — reassign resource attributes
- `k8sattributesprocessor` — enrich telemetry with k8s metadata
- `logdedupprocessor` — deduplicate repeated log entries
- `metricstransformprocessor` — rename/aggregate/transform metrics
- `probabilisticsamplerprocessor` — percentage-based sampling
- `redactionprocessor` — mask/remove sensitive data
- `resourceprocessor` — modify resource attributes
- `spanprocessor` — rename spans, extract attributes
- `tailsamplingprocessor` — sample traces based on policies

### Contrib extensions added (1)

- `filestorage` — persistent file-based storage (used by clickhouse exporter sending queue in EE OpAMP controller)

### Other changes

- **Docker cache mounts**: Added `--mount=type=cache` for Go module and build caches in the OCB builder stage of both `docker/otel-collector/Dockerfile` and `docker/hyperdx/Dockerfile`
- **CI timeouts**: Bumped `integration` and `otel-smoke-test` jobs from 8 to 16 minutes in `.github/workflows/main.yml`

All existing HyperDX-specific components are preserved unchanged.

### How to test locally or on Vercel

1. Build the OTel Collector Docker image — verify OCB resolves all listed modules
2. Provide a custom OTel config that uses one of the newly-added components and verify it loads
3. Verify existing HyperDX OTel pipeline still functions

### References

- Linear Issue: https://linear.app/clickhouse/issue/HDX-4029
- Upstream core builder-config: https://github.com/open-telemetry/opentelemetry-collector/blob/main/cmd/otelcorecol/builder-config.yaml
- Upstream contrib builder-config: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/cmd/otelcontribcol/builder-config.yaml
2026-04-15 15:57:44 +00:00

173 lines
6.6 KiB
YAML
Raw Blame History

# OpenTelemetry Collector Builder (OCB) manifest for HyperDX.
# This replaces the pre-built otel/opentelemetry-collector-contrib image with a
# custom-built binary that includes only the components HyperDX needs, plus any
# custom receivers/processors we add in the future.
#
# Version placeholders are replaced at Docker build time via sed:
# __OTEL_COLLECTOR_VERSION__ -> contrib/core component version (e.g. 0.149.0)
# __OTEL_COLLECTOR_CORE_VERSION__ -> core confmap provider version (e.g. 1.55.0)
# Both values are defined in the root .env file.
#
# The upstream contrib manifest for reference:
# https://github.com/open-telemetry/opentelemetry-collector-releases/blob/main/distributions/otelcol-contrib/manifest.yaml
dist:
name: otelcol-hyperdx
description: HyperDX OpenTelemetry Collector
version: __OTEL_COLLECTOR_VERSION__
output_path: /build/output
build_tags: 'grpcnotrace'
receivers:
# Core
- gomod:
go.opentelemetry.io/collector/receiver/nopreceiver
v__OTEL_COLLECTOR_VERSION__
- gomod:
go.opentelemetry.io/collector/receiver/otlpreceiver
v__OTEL_COLLECTOR_VERSION__
# Contrib
- gomod:
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/dockerstatsreceiver
v__OTEL_COLLECTOR_VERSION__
- gomod:
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/filelogreceiver
v__OTEL_COLLECTOR_VERSION__
- gomod:
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/fluentforwardreceiver
v__OTEL_COLLECTOR_VERSION__
- gomod:
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/hostmetricsreceiver
v__OTEL_COLLECTOR_VERSION__
- gomod:
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/k8sclusterreceiver
v__OTEL_COLLECTOR_VERSION__
- gomod:
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/kubeletstatsreceiver
v__OTEL_COLLECTOR_VERSION__
- gomod:
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/prometheusreceiver
v__OTEL_COLLECTOR_VERSION__
processors:
# Core
- gomod:
go.opentelemetry.io/collector/processor/batchprocessor
v__OTEL_COLLECTOR_VERSION__
- gomod:
go.opentelemetry.io/collector/processor/memorylimiterprocessor
v__OTEL_COLLECTOR_VERSION__
# Contrib
- gomod:
github.com/open-telemetry/opentelemetry-collector-contrib/processor/attributesprocessor
v__OTEL_COLLECTOR_VERSION__
- gomod:
github.com/open-telemetry/opentelemetry-collector-contrib/processor/cumulativetodeltaprocessor
v__OTEL_COLLECTOR_VERSION__
- gomod:
github.com/open-telemetry/opentelemetry-collector-contrib/processor/filterprocessor
v__OTEL_COLLECTOR_VERSION__
- gomod:
github.com/open-telemetry/opentelemetry-collector-contrib/processor/groupbyattrsprocessor
v__OTEL_COLLECTOR_VERSION__
- gomod:
github.com/open-telemetry/opentelemetry-collector-contrib/processor/k8sattributesprocessor
v__OTEL_COLLECTOR_VERSION__
- gomod:
github.com/open-telemetry/opentelemetry-collector-contrib/processor/logdedupprocessor
v__OTEL_COLLECTOR_VERSION__
- gomod:
github.com/open-telemetry/opentelemetry-collector-contrib/processor/metricstransformprocessor
v__OTEL_COLLECTOR_VERSION__
- gomod:
github.com/open-telemetry/opentelemetry-collector-contrib/processor/probabilisticsamplerprocessor
v__OTEL_COLLECTOR_VERSION__
- gomod:
github.com/open-telemetry/opentelemetry-collector-contrib/processor/redactionprocessor
v__OTEL_COLLECTOR_VERSION__
- gomod:
github.com/open-telemetry/opentelemetry-collector-contrib/processor/resourcedetectionprocessor
v__OTEL_COLLECTOR_VERSION__
- gomod:
github.com/open-telemetry/opentelemetry-collector-contrib/processor/resourceprocessor
v__OTEL_COLLECTOR_VERSION__
- gomod:
github.com/open-telemetry/opentelemetry-collector-contrib/processor/spanprocessor
v__OTEL_COLLECTOR_VERSION__
- gomod:
github.com/open-telemetry/opentelemetry-collector-contrib/processor/tailsamplingprocessor
v__OTEL_COLLECTOR_VERSION__
- gomod:
github.com/open-telemetry/opentelemetry-collector-contrib/processor/transformprocessor
v__OTEL_COLLECTOR_VERSION__
exporters:
# Core
- gomod:
go.opentelemetry.io/collector/exporter/debugexporter
v__OTEL_COLLECTOR_VERSION__
- gomod:
go.opentelemetry.io/collector/exporter/nopexporter
v__OTEL_COLLECTOR_VERSION__
- gomod:
go.opentelemetry.io/collector/exporter/otlpexporter
v__OTEL_COLLECTOR_VERSION__
- gomod:
go.opentelemetry.io/collector/exporter/otlphttpexporter
v__OTEL_COLLECTOR_VERSION__
# Contrib
- gomod:
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/clickhouseexporter
v__OTEL_COLLECTOR_VERSION__
connectors:
# Core
- gomod:
go.opentelemetry.io/collector/connector/forwardconnector
v__OTEL_COLLECTOR_VERSION__
# Contrib
- gomod:
github.com/open-telemetry/opentelemetry-collector-contrib/connector/routingconnector
v__OTEL_COLLECTOR_VERSION__
extensions:
# Core
- gomod:
go.opentelemetry.io/collector/extension/memorylimiterextension
v__OTEL_COLLECTOR_VERSION__
- gomod:
go.opentelemetry.io/collector/extension/zpagesextension
v__OTEL_COLLECTOR_VERSION__
# Contrib
- gomod:
github.com/open-telemetry/opentelemetry-collector-contrib/extension/bearertokenauthextension
v__OTEL_COLLECTOR_VERSION__
- gomod:
github.com/open-telemetry/opentelemetry-collector-contrib/extension/storage/filestorage
v__OTEL_COLLECTOR_VERSION__
- gomod:
github.com/open-telemetry/opentelemetry-collector-contrib/extension/healthcheckextension
v__OTEL_COLLECTOR_VERSION__
- gomod:
github.com/open-telemetry/opentelemetry-collector-contrib/extension/opampextension
v__OTEL_COLLECTOR_VERSION__
- gomod:
github.com/open-telemetry/opentelemetry-collector-contrib/extension/pprofextension
v__OTEL_COLLECTOR_VERSION__
providers:
- gomod:
go.opentelemetry.io/collector/confmap/provider/envprovider
v__OTEL_COLLECTOR_CORE_VERSION__
- gomod:
go.opentelemetry.io/collector/confmap/provider/fileprovider
v__OTEL_COLLECTOR_CORE_VERSION__
- gomod:
go.opentelemetry.io/collector/confmap/provider/httpprovider
v__OTEL_COLLECTOR_CORE_VERSION__
- gomod:
go.opentelemetry.io/collector/confmap/provider/httpsprovider
v__OTEL_COLLECTOR_CORE_VERSION__
- gomod:
go.opentelemetry.io/collector/confmap/provider/yamlprovider
v__OTEL_COLLECTOR_CORE_VERSION__
Reference in a new issue View git blame Copy permalink