[HDX-4029] Add commonly-used core and contrib components to OTel Collector builder-config (#2121)

## Summary

Update `packages/otel-collector/builder-config.yaml` to include commonly-used components from the upstream [opentelemetry-collector](https://github.com/open-telemetry/opentelemetry-collector) core and [opentelemetry-collector-contrib](https://github.com/open-telemetry/opentelemetry-collector-contrib) distributions. This gives users more flexibility in their custom OTel configs without pulling in the entire contrib distribution (which causes very long compile times).

Also adds Go module and build cache mounts to the OCB Docker build stage for faster rebuilds, and bumps CI timeouts for integration and smoke test jobs to account for the larger binary.

### Core extensions added (2)

- `memorylimiterextension` — memory-based limiting at the extension level
- `zpagesextension` — zPages debugging endpoints

### Contrib receivers added (4)

- `dockerstatsreceiver` — container metrics from Docker
- `filelogreceiver` — tail log files
- `k8sclusterreceiver` — Kubernetes cluster-level metrics
- `kubeletstatsreceiver` — node/pod/container metrics from kubelet

### Contrib processors added (12)

- `attributesprocessor` — insert/update/delete/hash attributes
- `cumulativetodeltaprocessor` — convert cumulative metrics to delta
- `filterprocessor` — drop unwanted telemetry
- `groupbyattrsprocessor` — reassign resource attributes
- `k8sattributesprocessor` — enrich telemetry with k8s metadata
- `logdedupprocessor` — deduplicate repeated log entries
- `metricstransformprocessor` — rename/aggregate/transform metrics
- `probabilisticsamplerprocessor` — percentage-based sampling
- `redactionprocessor` — mask/remove sensitive data
- `resourceprocessor` — modify resource attributes
- `spanprocessor` — rename spans, extract attributes
- `tailsamplingprocessor` — sample traces based on policies

### Contrib extensions added (1)

- `filestorage` — persistent file-based storage (used by clickhouse exporter sending queue in EE OpAMP controller)

### Other changes

- **Docker cache mounts**: Added `--mount=type=cache` for Go module and build caches in the OCB builder stage of both `docker/otel-collector/Dockerfile` and `docker/hyperdx/Dockerfile`
- **CI timeouts**: Bumped `integration` and `otel-smoke-test` jobs from 8 to 16 minutes in `.github/workflows/main.yml`

All existing HyperDX-specific components are preserved unchanged.

### How to test locally or on Vercel

1. Build the OTel Collector Docker image — verify OCB resolves all listed modules
2. Provide a custom OTel config that uses one of the newly-added components and verify it loads
3. Verify existing HyperDX OTel pipeline still functions

### References

- Linear Issue: https://linear.app/clickhouse/issue/HDX-4029
- Upstream core builder-config: https://github.com/open-telemetry/opentelemetry-collector/blob/main/cmd/otelcorecol/builder-config.yaml
- Upstream contrib builder-config: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/cmd/otelcontribcol/builder-config.yaml

.changeset/otel-collector-add-core-extensions.md

@@ -0,0 +1,13 @@
+---
+'@hyperdx/otel-collector': minor
+---
+
+feat: Add missing core extensions, commonly-used contrib processors/receivers, and filestorage extension
+
+Add the two missing core extensions (memorylimiterextension, zpagesextension),
+12 commonly-used contrib processors (attributes, filter, resource, k8sattributes,
+tailsampling, probabilisticsampler, span, groupbyattrs, redaction, logdedup,
+metricstransform, cumulativetodelta), 4 commonly-used contrib receivers
+(filelog, dockerstats, k8scluster, kubeletstats), and the filestorage extension
+(used for persistent sending queue in the clickhouse exporter) to
+builder-config.yaml.

.github/workflows/main.yml

@@ -47,7 +47,7 @@ jobs:
       - name: Run unit tests
         run: make ci-unit
   integration:
-    timeout-minutes: 8
+    timeout-minutes: 16
     runs-on: ubuntu-24.04
     steps:
       - name: Checkout
@@ -93,7 +93,7 @@ jobs:
         working-directory: ./packages/otel-collector
         run: go test ./...
   otel-smoke-test:
-    timeout-minutes: 8
+    timeout-minutes: 16
     runs-on: ubuntu-24.04
     steps:
       - name: Checkout

docker/hyperdx/Dockerfile

@@ -29,7 +29,9 @@ ARG OTEL_COLLECTOR_CORE_VERSION
 COPY --from=ocb-bin /usr/local/bin/ocb /usr/local/bin/ocb
 WORKDIR /build
 COPY packages/otel-collector/builder-config.yaml .
-RUN sed -i "s/__OTEL_COLLECTOR_VERSION__/${OTEL_COLLECTOR_VERSION}/g; s/__OTEL_COLLECTOR_CORE_VERSION__/${OTEL_COLLECTOR_CORE_VERSION}/g" builder-config.yaml && \
+RUN --mount=type=cache,target=/go/pkg/mod,id=ocb-gomod \
+    --mount=type=cache,target=/root/.cache/go-build,id=ocb-gobuild \
+    sed -i "s/__OTEL_COLLECTOR_VERSION__/${OTEL_COLLECTOR_VERSION}/g; s/__OTEL_COLLECTOR_CORE_VERSION__/${OTEL_COLLECTOR_CORE_VERSION}/g" builder-config.yaml && \
     CGO_ENABLED=0 ocb --config=builder-config.yaml
 
 # Build the Go migration tool with full TLS support for ClickHouse

docker/otel-collector/Dockerfile

@@ -19,7 +19,9 @@ ARG OTEL_COLLECTOR_CORE_VERSION
 COPY --from=ocb-bin /usr/local/bin/ocb /usr/local/bin/ocb
 WORKDIR /build
 COPY packages/otel-collector/builder-config.yaml .
-RUN sed -i "s/__OTEL_COLLECTOR_VERSION__/${OTEL_COLLECTOR_VERSION}/g; s/__OTEL_COLLECTOR_CORE_VERSION__/${OTEL_COLLECTOR_CORE_VERSION}/g" builder-config.yaml && \
+RUN --mount=type=cache,target=/go/pkg/mod,id=ocb-gomod \
+    --mount=type=cache,target=/root/.cache/go-build,id=ocb-gobuild \
+    sed -i "s/__OTEL_COLLECTOR_VERSION__/${OTEL_COLLECTOR_VERSION}/g; s/__OTEL_COLLECTOR_CORE_VERSION__/${OTEL_COLLECTOR_CORE_VERSION}/g" builder-config.yaml && \
     CGO_ENABLED=0 ocb --config=builder-config.yaml
 
 # Build the Go migration tool with full TLS support for ClickHouse

packages/otel-collector/README.md

@@ -3,8 +3,8 @@
 Custom-built OpenTelemetry Collector for HyperDX, compiled via
 [OCB (OpenTelemetry Collector Builder)](https://github.com/open-telemetry/opentelemetry-collector/tree/main/cmd/builder).
 This replaces the pre-built `otel/opentelemetry-collector-contrib` image with a
-binary that includes only the components HyperDX needs, plus any custom
+binary that includes the components HyperDX needs plus commonly-used core and
-receivers/processors added in this package.
+contrib components, and any custom receivers/processors added in this package.
 
 ## Architecture
 
@@ -50,26 +50,44 @@ builds.
 
 ## Included components
 
-All components referenced in config files and the OpAMP controller:
+Components used by HyperDX internally are marked with their config references.
+Components marked "user configs" are included so users can reference them in
+custom OTel configurations without rebuilding the collector.
 
 ### Receivers
 
-| Component       | Module  | Used in                                           |
+| Component        | Module  | Used in                                           |
-| --------------- | ------- | ------------------------------------------------- |
+| ---------------- | ------- | ------------------------------------------------- |
-| `nop`           | core    | OpAMP controller                                  |
+| `nop`            | core    | OpAMP controller                                  |
-| `otlp`          | core    | standalone configs, OpAMP controller, smoke tests |
+| `otlp`           | core    | standalone configs, OpAMP controller, smoke tests |
-| `fluentforward` | contrib | standalone configs, OpAMP controller, smoke tests |
+| `dockerstats`    | contrib | user configs                                      |
-| `hostmetrics`   | contrib | custom.config.yaml                                |
+| `filelog`        | contrib | user configs                                      |
-| `prometheus`    | contrib | OpAMP controller, smoke tests                     |
+| `fluentforward`  | contrib | standalone configs, OpAMP controller, smoke tests |
+| `hostmetrics`    | contrib | custom.config.yaml                                |
+| `k8scluster`     | contrib | user configs                                      |
+| `kubeletstats`   | contrib | user configs                                      |
+| `prometheus`     | contrib | OpAMP controller, smoke tests                     |
 
 ### Processors
 
-| Component           | Module  | Used in                                           |
+| Component              | Module  | Used in                                           |
-| ------------------- | ------- | ------------------------------------------------- |
+| ---------------------- | ------- | ------------------------------------------------- |
-| `batch`             | core    | config.yaml, standalone configs, OpAMP controller |
+| `batch`                | core    | config.yaml, standalone configs, OpAMP controller |
-| `memory_limiter`    | core    | config.yaml, standalone configs, OpAMP controller |
+| `memory_limiter`       | core    | config.yaml, standalone configs, OpAMP controller |
-| `resourcedetection` | contrib | config.yaml                                       |
+| `attributes`           | contrib | user configs                                      |
-| `transform`         | contrib | config.yaml, standalone configs, OpAMP controller |
+| `cumulativetodelta`    | contrib | user configs                                      |
+| `filter`               | contrib | user configs                                      |
+| `groupbyattrs`         | contrib | user configs                                      |
+| `k8sattributes`        | contrib | user configs                                      |
+| `logdedup`             | contrib | user configs                                      |
+| `metricstransform`     | contrib | user configs                                      |
+| `probabilisticsampler` | contrib | user configs                                      |
+| `redaction`            | contrib | user configs                                      |
+| `resourcedetection`    | contrib | config.yaml                                       |
+| `resource`             | contrib | user configs                                      |
+| `span`                 | contrib | user configs                                      |
+| `tailsampling`         | contrib | user configs                                      |
+| `transform`            | contrib | config.yaml, standalone configs, OpAMP controller |
 
 ### Exporters
 
@@ -92,7 +110,10 @@ All components referenced in config files and the OpAMP controller:
 
 | Component         | Module  | Used in                                  |
 | ----------------- | ------- | ---------------------------------------- |
+| `memorylimiter`   | core    | user configs                             |
+| `zpages`          | core    | user configs                             |
 | `bearertokenauth` | contrib | standalone-auth config, OpAMP controller |
+| `file_storage`    | contrib | OpAMP controller (sending queue storage) |
 | `health_check`    | contrib | config.yaml, standalone-auth config      |
 | `opamp`           | contrib | used by OpAMP supervisor                 |
 | `pprof`           | contrib | included for debugging/profiling         |

packages/otel-collector/builder-config.yaml

@@ -27,12 +27,24 @@ receivers:
       go.opentelemetry.io/collector/receiver/otlpreceiver
       v__OTEL_COLLECTOR_VERSION__
   # Contrib
+  - gomod:
+      github.com/open-telemetry/opentelemetry-collector-contrib/receiver/dockerstatsreceiver
+      v__OTEL_COLLECTOR_VERSION__
+  - gomod:
+      github.com/open-telemetry/opentelemetry-collector-contrib/receiver/filelogreceiver
+      v__OTEL_COLLECTOR_VERSION__
   - gomod:
       github.com/open-telemetry/opentelemetry-collector-contrib/receiver/fluentforwardreceiver
       v__OTEL_COLLECTOR_VERSION__
   - gomod:
       github.com/open-telemetry/opentelemetry-collector-contrib/receiver/hostmetricsreceiver
       v__OTEL_COLLECTOR_VERSION__
+  - gomod:
+      github.com/open-telemetry/opentelemetry-collector-contrib/receiver/k8sclusterreceiver
+      v__OTEL_COLLECTOR_VERSION__
+  - gomod:
+      github.com/open-telemetry/opentelemetry-collector-contrib/receiver/kubeletstatsreceiver
+      v__OTEL_COLLECTOR_VERSION__
   - gomod:
       github.com/open-telemetry/opentelemetry-collector-contrib/receiver/prometheusreceiver
       v__OTEL_COLLECTOR_VERSION__
@@ -46,9 +58,45 @@ processors:
       go.opentelemetry.io/collector/processor/memorylimiterprocessor
       v__OTEL_COLLECTOR_VERSION__
   # Contrib
+  - gomod:
+      github.com/open-telemetry/opentelemetry-collector-contrib/processor/attributesprocessor
+      v__OTEL_COLLECTOR_VERSION__
+  - gomod:
+      github.com/open-telemetry/opentelemetry-collector-contrib/processor/cumulativetodeltaprocessor
+      v__OTEL_COLLECTOR_VERSION__
+  - gomod:
+      github.com/open-telemetry/opentelemetry-collector-contrib/processor/filterprocessor
+      v__OTEL_COLLECTOR_VERSION__
+  - gomod:
+      github.com/open-telemetry/opentelemetry-collector-contrib/processor/groupbyattrsprocessor
+      v__OTEL_COLLECTOR_VERSION__
+  - gomod:
+      github.com/open-telemetry/opentelemetry-collector-contrib/processor/k8sattributesprocessor
+      v__OTEL_COLLECTOR_VERSION__
+  - gomod:
+      github.com/open-telemetry/opentelemetry-collector-contrib/processor/logdedupprocessor
+      v__OTEL_COLLECTOR_VERSION__
+  - gomod:
+      github.com/open-telemetry/opentelemetry-collector-contrib/processor/metricstransformprocessor
+      v__OTEL_COLLECTOR_VERSION__
+  - gomod:
+      github.com/open-telemetry/opentelemetry-collector-contrib/processor/probabilisticsamplerprocessor
+      v__OTEL_COLLECTOR_VERSION__
+  - gomod:
+      github.com/open-telemetry/opentelemetry-collector-contrib/processor/redactionprocessor
+      v__OTEL_COLLECTOR_VERSION__
   - gomod:
       github.com/open-telemetry/opentelemetry-collector-contrib/processor/resourcedetectionprocessor
       v__OTEL_COLLECTOR_VERSION__
+  - gomod:
+      github.com/open-telemetry/opentelemetry-collector-contrib/processor/resourceprocessor
+      v__OTEL_COLLECTOR_VERSION__
+  - gomod:
+      github.com/open-telemetry/opentelemetry-collector-contrib/processor/spanprocessor
+      v__OTEL_COLLECTOR_VERSION__
+  - gomod:
+      github.com/open-telemetry/opentelemetry-collector-contrib/processor/tailsamplingprocessor
+      v__OTEL_COLLECTOR_VERSION__
   - gomod:
       github.com/open-telemetry/opentelemetry-collector-contrib/processor/transformprocessor
       v__OTEL_COLLECTOR_VERSION__
@@ -83,10 +131,20 @@ connectors:
       v__OTEL_COLLECTOR_VERSION__
 
 extensions:
+  # Core
+  - gomod:
+      go.opentelemetry.io/collector/extension/memorylimiterextension
+      v__OTEL_COLLECTOR_VERSION__
+  - gomod:
+      go.opentelemetry.io/collector/extension/zpagesextension
+      v__OTEL_COLLECTOR_VERSION__
   # Contrib
   - gomod:
       github.com/open-telemetry/opentelemetry-collector-contrib/extension/bearertokenauthextension
       v__OTEL_COLLECTOR_VERSION__
+  - gomod:
+      github.com/open-telemetry/opentelemetry-collector-contrib/extension/storage/filestorage
+      v__OTEL_COLLECTOR_VERSION__
   - gomod:
       github.com/open-telemetry/opentelemetry-collector-contrib/extension/healthcheckextension
       v__OTEL_COLLECTOR_VERSION__