- Introduce a new flag `HYPERDX_OTEL_EXPORTER_CREATE_LEGACY_SCHEMA` (default to false) to otel collector
- Custom ClickStack schemas should be enforced by default
- ClickHouse tables migration logs should be stored in `clickstack_db_version_xxx` tables
- The collector will run the migration at startup and retry if it fails to connect to the database (using exponential backoff).
- Fully backward compatible
Ref: HDX-3301
Today, users have to set up an OpAMP server to run with our clickstack OTel collector. Instead, we should allow users to disable OpAMP when they're using ClickHouse Cloud with the clickstack integration.
This can be determined by `OPAMP_SERVER_URL` not being defined by the user.
The end result is that a user can do
```
docker run \
-e CLICKHOUSE_ENDPOINT=${CLICKHOUSE_ENDPOINT} \
-e CLICKHOUSE_USER=default \
-e CLICKHOUSE_PASSWORD=${CLICKHOUSE_PASSWORD} \
-p 8080:8080 -p 4317:4317 -p 4318:4318 \
clickhouse/clickstack-otel-collector:latest
```
Ref: HDX-3300
Allows setting a custom setting prefix on a connection. When set in HyperDX and the ClickHouse settings, the HyperDX app will set a custom setting for each query. These are recorded in the query log and can be used to identify which user issues the query.
## Testing
The commit also updates the local dev ClickHouse instance to support a custom setting prefix of `hyeprdx`. After running `make dev-up`, you should be able to edit the connection and set the the prefix to `hyperdx`.
<img width="955" height="197" alt="Screenshot 2026-01-21 at 1 23 14 PM" src="https://github.com/user-attachments/assets/607fc945-d93f-4976-9862-3118b420c077" />
After saving, just allow the app to live tail a source like logs. If you connect to the ClickHouse database, you should then be able to run
```
SELECT query, Settings
FROM system.query_log
WHERE has(mapKeys(Settings), 'hyperdx_user')
FORMAT Vertical
```
and then see a bunch of queries with the user set to your logged in user.
```
Row 46:
───────
query: SELECT Timestamp, ServiceName, SeverityText, Body, TimestampTime FROM default.otel_logs WHERE (TimestampTime >= fromUnixTimestamp64Milli(_CAST(1769022372269, 'Int64'))) AND (TimestampTime <= fromUnixTimestamp64Milli(_CAST(1769023272269, 'Int64'))) ORDER BY (TimestampTime, Timestamp) DESC LIMIT _CAST(0, 'Int32'), _CAST(200, 'Int32') FORMAT JSONCompactEachRowWithNamesAndTypes
Settings: {'use_uncompressed_cache':'0','load_balancing':'in_order','log_queries':'1','max_memory_usage':'10000000000','cancel_http_readonly_queries_on_client_close':'1','parallel_replicas_for_cluster_engines':'0','date_time_output_format':'iso','hyperdx_user':'\'dan@hyperdx.io\''}
```
We removed the rotator script when we used the named pipe approach to the otel collector logging. There were some references left over that caused the docker build to fail.
There were two issues with the log rotation script:
1. Logs could be lost since copying and then truncating the file might not finish before logs arrive.
2. The otel collector application will keeps the file handle and offset cached. After truncating, it will write starting at the last offset leaving the unallocated garbage in the beginning of the file. This garbage uses space.
This commit moves the file instead of copying. That allows the collector to continue writing to the rolled file until a SIGHUP is sent. This causes a config refresh, which also opens a new log file. After, the rolled file and the new log file have correct sizes.
--
**ADDITIONAL NOTES**:
Claude's code review is not accurate here.
* The alpine image is based on busybox and fuser is a command implemented by busybox. This can be verified by just running the collector and watching the log rotate behavior.
* The mv command updates the name of the file in the file system but doesn't change the inode number. A process only uses the file path the first time the file is open to resolve it into a inode number. Moving the file changes the name but doesn't change the inode number so the process will continue to write to that file.
Closes HDX-2283
This adds an environment variable `INGESTION_API_KEY` that can be set by the user. This apiKey will be valid and accepted by the Otel collector. It is in addition to the autogenerated apiKey and will not show in the team settings apiKey section.
plus the fix to reduce bloat in opamp agent logs
Users should be able to mount the custom otel collector config file and add/overrider receivers, processors and exporters
For example:
```
receivers:
hostmetrics:
collection_interval: 5s
scrapers:
cpu:
load:
memory:
disk:
filesystem:
network:
# override the default processors
processors:
batch:
send_batch_size: 10000
timeout: 10s
memory_limiter:
limit_mib: 2000
service:
pipelines:
metrics/hostmetrics:
receivers: [hostmetrics]
# attach existing processors
processors: [memory_limiter, batch]
# attach existing exporters
exporters: [clickhouse]
```
This will add a new `hostmetrics` receiver + `metrics/hostmetrics` pipeline and update existing `batch` + `memory_limiter` processors
WARNING: This feature is still in beta, and future updates may change how it works, potentially affecting compatibility
Ref: HDX-1865
- Upgrades ClickHouse to 25.6, fixes breaking config change, needed for latest JSON type
- Upgrades OTel Collector to 0.129.1, fixes breaking config change, needed for latest JSON support in exporter
- Upgrades OTel OpAMP Supervisor to 0.128.0
- Fixes features to support JSON type columns in OTel in HyperDX (filtering, searching, graphing, opening rows, etc.)
Requires users to set `BETA_CH_OTEL_JSON_SCHEMA_ENABLED=true` in `ch-server` and `OTEL_AGENT_FEATURE_GATE_ARG='--feature-gates=clickhouse.json'` in `otel-collector` to enable JSON schema. Users must start a new ClickHouse DB or migrate their own table manually to enable as it is not schema compatible and migration is not automatic.
Closes HDX-1849, HDX-1969, HDX-1849, HDX-1966, HDX-1964
Co-authored-by: Tom Alexander <3245235+teeohhem@users.noreply.github.com>
Adds environment variable to allow for passthrough_logs to be enabled in the supervisor config
Test locally with:
Edit docker-compose.dev.yaml
Add `OTEL_SUPERVISOR_PASSTHROUGH_LOGS: 'true'` under the otel environment variables
```
make dev-up
```
Ref: HDX-1859
Ref: HDX-1801
startup logs are like
```
Send OpenTelemetry data via:
http/protobuf: OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318
gRPC: OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317
Exporting data to ClickHouse:
Endpoint: tcp://ch-server:9000?dial_timeout=10s
Database: default
Waiting for ClickHouse to be ready...
ClickHouse is ready!
Visit the HyperDX UI at http://localhost:8080
```
- Support `CLICKHOUSE_ENDPOINT` to switch aio clickhouse endpoint (Ref: HDX-1758)
- Support `HYPERDX_OTEL_EXPORTER_CLICKHOUSE_DATABASE ` (Ref: HDX-1786)
Support command like
```
docker run -e CLICKHOUSE_ENDPOINT=<CH-CLOUD-ENDPOINT> -e CLICKHOUSE_USER=default -e CLICKHOUSE_PASSWORD='BLABLA' -e HYPERDX_OTEL_EXPORTER_CLICKHOUSE_DATABASE =hyperdx -p 8080:8080 -p 4317:4317 -p 4318:4318 hyperdx/hyperdx-local:2-nightly
```
So users can export data to other services like clickhouse cloud
For users connecting to ClickHouse Cloud or a TLS endpoint, add the `secure=true` query parameter or use the HTTPS protocol. Providing the full URL via the `CLICKHOUSE_SERVER_ENDPOINT` in the exporter's endpoint field should resolve this issue
Ref: HDX-1743
1. Merge 'fullstack' and 'local' (auth + noauth) builds into a single Dockerfile
2. Introduce 'all-in-one-auth' and 'all-in-one-noauth' build stages
3. Lock `IS_LOCAL_APP_MODE` env var
4. Fix bug in ctrl-c exit with docker run
5. Enable alerts in local mode (no-auth)
6. Build `common-utils` on the fly (no longer needing pulling pkg from npm)
Ref: HDX-1709
Ref: HDX-1713
Ref: HDX-1254
Ref: HDX-1729
To match v2 product definition, we are going to release three images:
- hyperdx/hyperdx (--target=prod): app only without any other deps (clickhouse, otelcol, mongodb), used in default compose + helm deployment
- hyperdx/hyperdx-all-in-one (--target=all-in-one-auth): all-in-one build + auth
- hyperdx/hyperdx-local (--target=all-in-one-noauth): all-in-one build + no-auth
Production impacts:
- hyperdx/hyperdx: none
- hyperdx/hyperdx-all-in-one: new
- hyperdx/hyperdx-local: add server components (alerts, saved searches, dashboards)
Updates the OTEL pipeline to handle structured logs better. If the body content is an OTEL map, it will merge the body map into the log attributes map. If the body is a JSON object, it will parse the JSON string into an OTEL map, then merge the fields into the log attributes map.
Replacing the Body field doesn't work since the Clickhouse exporter schema defines Body as string, so any parsed out object ends up turning back into a string. At least as log resources, it's a lighter weight means of grouping and filtering in the UI.
Ref: HDX-1453
