hyperdx/docker/otel-collector/entrypoint.sh
Warren Lee cb841457f2
[HDX-3994] Deprecate clickhouse.json feature gate in favor of per-exporter json config (#2119)
## Summary

Deprecate the upstream-deprecated `--feature-gates=clickhouse.json` CLI flag in favor of the per-exporter `json: true` config option, as recommended by the OpenTelemetry ClickHouse exporter v0.149.0.

This introduces a new env var `HYPERDX_OTEL_EXPORTER_CLICKHOUSE_JSON_ENABLE` that controls JSON mode at the exporter config level. The old `OTEL_AGENT_FEATURE_GATE_ARG` env var remains backward-compatible — when it contains `clickhouse.json`, the entrypoint strips that gate, maps it to the new env var, and prints a deprecation warning. Other feature gates are preserved and passed through to the collector.

**Key changes:**
- **`docker/otel-collector/entrypoint.sh`** — Detects `clickhouse.json` in `OTEL_AGENT_FEATURE_GATE_ARG`, strips it, sets `HYPERDX_OTEL_EXPORTER_CLICKHOUSE_JSON_ENABLE=true`, and prints a deprecation warning. Remaining feature gates are still passed through to the collector in both standalone and supervisor modes.
- **`docker/otel-collector/config.standalone.yaml`** — Added `json: ${env:HYPERDX_OTEL_EXPORTER_CLICKHOUSE_JSON_ENABLE:-false}` to both ClickHouse exporter configs
- **`packages/api/src/opamp/controllers/opampController.ts`** — Added `json` field to the `CollectorConfig` type and both ClickHouse exporter configs for OpAMP-managed collectors
- **`docker/otel-collector/supervisor_docker.yaml.tmpl`** — Feature gate pass-through preserved for non-`clickhouse.json` gates (entrypoint strips the deprecated gate before supervisor template renders)
- **`smoke-tests/otel-collector/`** — Added a JSON-enabled otel-collector service and smoke tests verifying:
  - `ResourceAttributes` and `LogAttributes` columns in `otel_logs` are `JSON` type (not `Map`)
  - Log data with various attribute types (string, int, boolean) is inserted and queryable via JSON path access

### How to test locally or on Vercel

1. Run `yarn dev` to start the dev stack
2. Verify the `otel-collector-json` container starts without errors (the `clickhouse.json` feature gate is stripped, not passed to the collector)
3. Check container logs for the deprecation warning when `OTEL_AGENT_FEATURE_GATE_ARG` contains `clickhouse.json`
4. Verify the non-JSON `otel-collector` service continues to work normally (json defaults to false)
5. Run smoke tests: `cd smoke-tests/otel-collector && bats json-exporter.bats`

### References

- Linear Issue: https://linear.app/hyperdx/issue/HDX-3994
- Upstream deprecation: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/clickhouseexporter#experimental-json-support
2026-04-15 15:49:45 +00:00

102 lines
4.6 KiB
Bash

#!/bin/sh
set -e
# DEPRECATED: The clickhouse.json feature gate has been removed upstream.
# When OTEL_AGENT_FEATURE_GATE_ARG contains clickhouse.json, strip it and
# map it to HYPERDX_OTEL_EXPORTER_CLICKHOUSE_JSON_ENABLE instead. Other feature gates
# are preserved and passed through to the collector.
if echo "$OTEL_AGENT_FEATURE_GATE_ARG" | grep -q "clickhouse.json"; then
echo "WARNING: '--feature-gates=clickhouse.json' is deprecated and no longer supported by the collector."
echo "WARNING: Use HYPERDX_OTEL_EXPORTER_CLICKHOUSE_JSON_ENABLE=true instead. This flag will be removed in a future release."
export HYPERDX_OTEL_EXPORTER_CLICKHOUSE_JSON_ENABLE=true
# Strip clickhouse.json from the feature gates, keeping any other gates
REMAINING_GATES=$(echo "$OTEL_AGENT_FEATURE_GATE_ARG" | sed 's/--feature-gates=//' | tr ',' '\n' | grep -v 'clickhouse.json' | tr '\n' ',' | sed 's/,$//')
if [ -n "$REMAINING_GATES" ]; then
export OTEL_AGENT_FEATURE_GATE_ARG="--feature-gates=$REMAINING_GATES"
else
unset OTEL_AGENT_FEATURE_GATE_ARG
fi
fi
# Fall back to legacy schema when ClickHouse JSON exporter mode is enabled
if [ "$HYPERDX_OTEL_EXPORTER_CLICKHOUSE_JSON_ENABLE" = "true" ]; then
export HYPERDX_OTEL_EXPORTER_CREATE_LEGACY_SCHEMA=true
fi
# Run ClickHouse schema migrations if not using legacy schema creation
if [ "$HYPERDX_OTEL_EXPORTER_CREATE_LEGACY_SCHEMA" != "true" ]; then
# Run Go-based migrate tool with TLS support
# TLS configuration:
# - CLICKHOUSE_TLS_CA_FILE: CA certificate file
# - CLICKHOUSE_TLS_CERT_FILE: Client certificate file
# - CLICKHOUSE_TLS_KEY_FILE: Client private key file
# - CLICKHOUSE_TLS_SERVER_NAME_OVERRIDE: Server name for TLS verification
# - CLICKHOUSE_TLS_INSECURE_SKIP_VERIFY: Skip TLS verification (set to "true")
echo "🚀 Using Go-based migrate tool with TLS support 🔐"
migrate /etc/otel/schema/seed
fi
# Check if OPAMP_SERVER_URL is defined to determine mode
if [ -z "$OPAMP_SERVER_URL" ]; then
# Standalone mode - run collector directly without supervisor
echo "Running in standalone mode (OPAMP_SERVER_URL not set)"
# Build collector arguments with multiple config files
COLLECTOR_ARGS="--config /etc/otelcol-contrib/config.yaml --config /etc/otelcol-contrib/standalone-config.yaml"
# Add bearer token auth config if OTLP_AUTH_TOKEN is specified (only used in standalone mode)
if [ -n "$OTLP_AUTH_TOKEN" ]; then
echo "OTLP_AUTH_TOKEN is configured, enabling bearer token authentication"
COLLECTOR_ARGS="$COLLECTOR_ARGS --config /etc/otelcol-contrib/standalone-auth-config.yaml"
fi
# Add custom config file if specified
if [ -n "$CUSTOM_OTELCOL_CONFIG_FILE" ]; then
echo "Including custom config: $CUSTOM_OTELCOL_CONFIG_FILE"
COLLECTOR_ARGS="$COLLECTOR_ARGS --config $CUSTOM_OTELCOL_CONFIG_FILE"
fi
# Pass remaining feature gates to the collector in standalone mode
if [ -n "$OTEL_AGENT_FEATURE_GATE_ARG" ]; then
COLLECTOR_ARGS="$COLLECTOR_ARGS $OTEL_AGENT_FEATURE_GATE_ARG"
fi
# Execute collector directly
exec /otelcontribcol $COLLECTOR_ARGS
else
# Supervisor mode - run with OpAMP supervisor
echo "Running in supervisor mode (OPAMP_SERVER_URL: $OPAMP_SERVER_URL)"
if [ "$OTEL_SUPERVISOR_LOGS" = "true" ]; then
# Start log tailer process in background for agent.log
# Arguments: log_file_path [check_interval_seconds]
/log-tailer.sh /etc/otel/supervisor-data/agent.log 1 &
# Create a agent log file for the supervisor and collector child process. Normally
# this file would be created as a standard file but we just want a FIFO pipe that
# will pass data over to the tail process in the entrypoint script. This avoids
# the need to the supervisor to store and forward the logs in its memory while also
# eliminating the need for volume based storage.
if [ ! -e /etc/otel/supervisor-data/agent.log ]; then
mkfifo /etc/otel/supervisor-data/agent.log || echo "Failed to create FIFO" >&2
fi
fi
# Render the supervisor config template using gomplate
# Write to supervisor-data directory which has proper permissions for otel user
gomplate -f /etc/otel/supervisor.yaml.tmpl -o /etc/otel/supervisor-data/supervisor-runtime.yaml
# Log the configuration being used
if [ -n "$CUSTOM_OTELCOL_CONFIG_FILE" ]; then
echo "Using custom OTEL config file: $CUSTOM_OTELCOL_CONFIG_FILE"
else
echo "CUSTOM_OTELCOL_CONFIG_FILE not set, using default configuration"
fi
# Update the command arguments to use the rendered config file
set -- "$1" --config /etc/otel/supervisor-data/supervisor-runtime.yaml
# Execute the supervisor with all passed arguments
exec "$@"
fi