Elgato_dark/hyperdx
1
0
Fork 0
mirror of https://github.com/hyperdxio/hyperdx synced 2026-04-21 13:37:15 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 6
hyperdx/smoke-tests/otel-collector
History
Warren Lee cb841457f2
[HDX-3994] Deprecate clickhouse.json feature gate in favor of per-exporter json config (#2119) 
## Summary

Deprecate the upstream-deprecated `--feature-gates=clickhouse.json` CLI flag in favor of the per-exporter `json: true` config option, as recommended by the OpenTelemetry ClickHouse exporter v0.149.0.

This introduces a new env var `HYPERDX_OTEL_EXPORTER_CLICKHOUSE_JSON_ENABLE` that controls JSON mode at the exporter config level. The old `OTEL_AGENT_FEATURE_GATE_ARG` env var remains backward-compatible — when it contains `clickhouse.json`, the entrypoint strips that gate, maps it to the new env var, and prints a deprecation warning. Other feature gates are preserved and passed through to the collector.

**Key changes:**
- **`docker/otel-collector/entrypoint.sh`** — Detects `clickhouse.json` in `OTEL_AGENT_FEATURE_GATE_ARG`, strips it, sets `HYPERDX_OTEL_EXPORTER_CLICKHOUSE_JSON_ENABLE=true`, and prints a deprecation warning. Remaining feature gates are still passed through to the collector in both standalone and supervisor modes.
- **`docker/otel-collector/config.standalone.yaml`** — Added `json: ${env:HYPERDX_OTEL_EXPORTER_CLICKHOUSE_JSON_ENABLE:-false}` to both ClickHouse exporter configs
- **`packages/api/src/opamp/controllers/opampController.ts`** — Added `json` field to the `CollectorConfig` type and both ClickHouse exporter configs for OpAMP-managed collectors
- **`docker/otel-collector/supervisor_docker.yaml.tmpl`** — Feature gate pass-through preserved for non-`clickhouse.json` gates (entrypoint strips the deprecated gate before supervisor template renders)
- **`smoke-tests/otel-collector/`** — Added a JSON-enabled otel-collector service and smoke tests verifying:
  - `ResourceAttributes` and `LogAttributes` columns in `otel_logs` are `JSON` type (not `Map`)
  - Log data with various attribute types (string, int, boolean) is inserted and queryable via JSON path access

### How to test locally or on Vercel

1. Run `yarn dev` to start the dev stack
2. Verify the `otel-collector-json` container starts without errors (the `clickhouse.json` feature gate is stripped, not passed to the collector)
3. Check container logs for the deprecation warning when `OTEL_AGENT_FEATURE_GATE_ARG` contains `clickhouse.json`
4. Verify the non-JSON `otel-collector` service continues to work normally (json defaults to false)
5. Run smoke tests: `cd smoke-tests/otel-collector && bats json-exporter.bats`

### References

- Linear Issue: https://linear.app/hyperdx/issue/HDX-3994
- Upstream deprecation: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/clickhouseexporter#experimental-json-support
2026-04-15 15:49:45 +00:00
..
data [HDX-3994] Deprecate clickhouse.json feature gate in favor of per-exporter json config (#2119) 2026-04-15 15:49:45 +00:00
test_helpers Add ingestion key check to otel collector via OpAMP (#825) 2025-05-23 01:41:35 +00:00
auto-parse-json.bats feat: normalize the text case of severity text (#834) 2025-05-22 04:42:00 +00:00
docker-compose.yaml [HDX-3994] Deprecate clickhouse.json feature gate in favor of per-exporter json config (#2119) 2026-04-15 15:49:45 +00:00
json-exporter.bats [HDX-3994] Deprecate clickhouse.json feature gate in favor of per-exporter json config (#2119) 2026-04-15 15:49:45 +00:00
normalize-severity.bats feat: normalize the text case of severity text (#834) 2025-05-22 04:42:00 +00:00
README.md Add ingestion key check to otel collector via OpAMP (#825) 2025-05-23 01:41:35 +00:00
receiver-config.yaml Add ingestion key check to otel collector via OpAMP (#825) 2025-05-23 01:41:35 +00:00
setup_suite.bash [HDX-3994] Deprecate clickhouse.json feature gate in favor of per-exporter json config (#2119) 2026-04-15 15:49:45 +00:00
severity-inference.bats fix(otel-collector): improve log level extraction with word boundaries in regex (#1747) 2026-02-18 22:16:07 +00:00

README.md

OpenTelemetry Collector Smoke Tests

This directory contains smoke tests for validating the OpenTelemetry Collector functionality in HyperDX.

Prerequisites

Before running the tests, ensure you have the following tools installed:

  • Bats - Bash Automated Testing System
  • Docker and Docker Compose
  • curl - Command line tool for transferring data
  • ClickHouse client - Command-line client for ClickHouse
    • Make sure to run clickhouse install to install the ClickHouse client after downloading ClickHouse

Running the Tests

To run all the tests:

cd smoke-tests/otel-collector
bats *.bats

To run a specific test file:

bats hdx-1453-auto-parse-json.bats

Test Structure

  • *.bats - Test files written in Bats
  • setup_suite.bash - Contains global setup_suite and teardown_suite functions that run once for the entire test suite
  • data/ - Test data used by the tests
  • test_helpers/ - Utility functions for the tests
  • docker-compose.yaml - Docker Compose configuration for the test environment

The test suite uses Bats' setup_suite and teardown_suite hooks to initialize the environment only once, regardless of how many test files are run. This optimizes test execution by:

  1. Validating the environment once
  2. Starting Docker containers once at the beginning of the test suite
  3. Cleaning up containers once at the end of the test suite

Debugging

If you need to debug the tests, you can set the SKIP_CLEANUP environment variable to prevent the Docker containers from being torn down after the tests complete:

SKIP_CLEANUP=1 bats *.bats

or

SKIP_CLEANUP=true bats *.bats

With SKIP_CLEANUP enabled, the test containers will remain running after the tests complete, allowing you to inspect logs, connect to the containers, and debug issues.

To manually clean up the containers after debugging:

docker compose down