## Problem
High-throughput services can produce millions of spans per second. Storing every span is expensive, so we run the OpenTelemetry Collector's tail-sampling processor to keep only 1-in-N spans. Each kept span carries a `SampleRate` attribute recording N.
Once data is sampled, naive aggregations are wrong: count() returns N-x fewer events than actually occurred, sum()/avg() are biased, and percentiles shift. Dashboards show misleadingly low request counts, throughput, and error rates, making capacity planning and alerting unreliable.
### Why Materialized Views Cannot Solve This Alone
A materialized view that pre-aggregates sampled spans is a useful performance optimization for known dashboard queries, but it cannot replace a sampling-aware query engine.
**Fixed dimensions.** A materialized view pre-aggregates by a fixed set of GROUP BY keys (e.g. `ServiceName`, `SpanName`, `StatusCode`, `TimestampBucket`). Trace exploration requires slicing by arbitrary span attributes -- `http.target`, `k8s.pod.name`, custom business tags -- in combinations that cannot be predicted at view creation time. Grouping by a different dimension either requires going back to raw table or a separate materialized views for every possible dimension combination. If you try to work around the fixed-dimensions problem by adding high-cardinality span attributes to the GROUP BY, the materialized table approaches a 1:1 row ratio with the raw table. You end up doubling storage without meaningful compression.
**Fixed aggregation fields.** A typical MV only aggregates a single numeric column like `Duration`. Users want weighted aggregations over any numeric attribute: request body sizes, queue depths, retry counts, custom metrics attached to spans. Each new field requires adding more `AggregateFunction` columns and recreating the view.
**Industry precedent.** Platforms that rely solely on pre-aggregation (Datadog, Splunk, New Relic, Elastic) get accurate RED dashboards but cannot correct ad-hoc queries over sampled span data. Only query-engine weighting (Honeycomb) produces correct results for arbitrary ad-hoc queries, including weighted percentiles and heatmaps.
A better solution is making the query engine itself sampling-aware, so that all queries from dashboards, alerts, an ad-hoc searches, automatically weights by `SampleRate` regardless of which dimensions or fields the user picks. Materialized views remain a useful complement for accelerating known, fixed-dimension dashboard panels, but they are not a substitute for correct query-time weighting.
## Summary
TraceSourceSchema gets a new optional field `sampleRateExpression` - the ClickHouse expression that evaluates to the per-span sample rate (e.g. `SpanAttributes['SampleRate']`). When not configured, all queries are unchanged.
When set, the query builder rewrites SQL aggregations to weight each span by its sample rate:
aggFn | Before | After (sample-corrected) | Overhead
-------------- | ---------------------- | --------------------------------------------------- | --------
count | count() | sum(weight) | ~1x
count + cond | countIf(cond) | sumIf(weight, cond) | ~1x
avg | avg(col) | sum(col * weight) / sum(weight) | ~2x
sum | sum(col) | sum(col * weight) | ~1x
quantile(p) | quantile(p)(col) | quantileTDigestWeighted(p)(col, toUInt32(weight)) | ~1.5x
min/max | unchanged | unchanged | 1x
count_distinct | unchanged | unchanged (cannot correct) | 1x
**Types**:
- Add sampleRateExpression to TraceSourceSchema + Mongoose model
- Add sampleWeightExpression to ChartConfig schema
**Query builder:**
- sampleWeightExpression is wrapped as greatest(toUInt64OrZero(toString(expr)), 1) so
spans without a SampleRate attribute default to weight 1 (unsampled
data produces identical results to the original queries).
- Rewrite aggFnExpr in renderChartConfig.ts when sampleWeightExpression
is set, with safe default-to-1 wrapping
**Integration** (propagate sampleWeightExpression to all chart configs):
- ChartEditor/utils.ts, DBSearchPage, ServicesDashboardPage, sessions
- DBDashboardPage (raw SQL + builder branches)
- AlertPreviewChart
- SessionSubpanel
- ServiceDashboardEndpointPerformanceChart
- ServiceDashboardSlowestEventsTile (p95 query + events table)
- ServiceDashboardEndpointSidePanel (error rate + throughput)
- ServiceDashboardDbQuerySidePanel (total query time + throughput)
- External API v2 charts, AI controller, alerts (index + template)
**UI**:
- Add Sample Rate Expression field to trace source admin form
### Screenshots or video
| Before | After |
| :----- | :---- |
| | |
### How to test locally or on Vercel
1.
2.
3.
### References
- Linear Issue:
- Related PRs:
## Summary
This PR reduces flakiness in some E2E tests by
1. Updating Search filters to have test ids which are specific to both the column/key and the value, since identical values across columns caused strict mode failures
2. Updating the Saved Search tests to use more web first assertions
### Screenshots or video
### How to test locally or on Vercel
### References
- Linear Issue:
- Related PRs:
## Summary
When a table uses `Map(LowCardinality(String), String)` for columns like `LogAttributes` or `ResourceAttributes`, none of the Map sub-fields appear in the search page's facet/filter panel. This is because the filter logic extracts the Map **value** type (`String`) and checks it for `LowCardinality` to decide default visibility — so only `Map(LowCardinality(String), LowCardinality(String))` schemas worked.
The fix adds an `isMapSubField` check (`path.length > 1`) so that all Map/JSON sub-fields are always included in the default filter set, regardless of the Map value type. Regular top-level `String` columns (like `Body`) remain hidden by default.
### How to test locally or on Vercel
1. Configure a ClickHouse source with a table that uses `Map(LowCardinality(String), String)` for LogAttributes or ResourceAttributes (e.g. the default ClickHouse Cloud OTel exporter schema)
2. Navigate to the search page for that source
3. Verify that LogAttributes and ResourceAttributes sub-fields now appear in the facet panel by default, without needing to click "More filters"
### References
- Customer escalation
## Summary
- **Thumb styling**: Moved from `classNames` to inline `styles` to fix CSS specificity issue where Mantine's default `:where()` selectors override CSS module classes due to load order. Inline styles always win.
- **Mark styling**: Added `!important` to mark CSS module overrides (size, border-color, background-color) so theme tokens apply reliably over Mantine defaults.
- **Vertical centering**: Added `margin-top: 1px` to center 6px mark dots within the 8px track height.
- **Removed broken transform**: Removed the `translateX(1px) translateY(1px)` nudge that was misaligning marks and labels.
- **Token fix**: Corrected `--color-slider-dot-active` values in ClickStack dark/light modes.
Follows up on #1988.
### Before
<img width="1724" height="158" alt="image" src="https://github.com/user-attachments/assets/9983920d-d6dc-40ab-9f87-6f7370eb1209" />
### After
<img width="1732" height="174" alt="image" src="https://github.com/user-attachments/assets/a60370fb-c079-4408-804d-4d4c6e9451a4" />
## Test plan
- [ ] Verify slider thumb uses theme token colors (not Mantine white/default) in both ClickStack and HyperDX themes
- [ ] Verify slider mark dots are solid 6px circles with correct token colors
- [ ] Verify mark dots are vertically centered on the track
- [ ] Verify mark labels ("0", "10") are horizontally aligned with track endpoints
- [ ] Verify no visual regressions in both dark and light modes for both themes
## Summary
Resolve all remaining knip issues — removes unused exports/types, adds missing direct dependencies, deletes dead code, and updates knip config.
**Dependency fixes:**
- Root: swapped unused `eslint-config-next`/`eslint-plugin-react-hooks` for actually-imported `@eslint/js`, `typescript-eslint`, `tslib`
- App: added directly-used transitive deps (`@codemirror/*`, `react-resizable`, `postcss-simple-vars`, `rimraf`, `serve`, `@next/eslint-plugin-next`, `eslint-plugin-react`); removed unused `@storybook/react`
**Dead code removal:**
- Removed ~100 unused exports/types across api and app packages (removed `export` keyword where used locally, deleted entirely where not)
- Fixed duplicate `DBRowTableIconButton` default+named export; updated consumers to use named import
**knip.json updates:**
- Added `fixtures.ts` entry point and `opamp/**` ignore for api package
- Excluded `enumMembers` and `duplicates` issue types
- Enabled `ignoreExportsUsedInFile`
### How to test locally or on Vercel
1. `yarn install && yarn knip` — should produce zero output
2. `make ci-lint` — all packages pass
3. `make ci-unit` — all unit tests pass
## Summary
- Replace chunked `$in` aggregations with per-alert queries to leverage compound indexes for index-backed sorting in DocumentDB
- Eliminate N+1 query pattern from the alerts API endpoint by adding a concurrency-controlled batch function
Linear: https://linear.app/clickhouse/issue/HDX-3840/docdb-ro-instances-high-cpu
## Problem
Two aggregation queries on the `alerthistories` collection were causing sustained high CPU on DocDB read-only instances:
1. **`getPreviousAlertHistories`** (runs every minute via check-alerts cron) — used `$match: {alert: {$in: [50 ids]}}` which breaks index-backed sort optimization on the compound index `{alert: 1, group: 1, createdAt: -1}`. With a 7-day lookback, each chunk scanned ~500K documents and required an in-memory sort. Additionally, `$first: '$$ROOT'` prevented projection pushdown, forcing full document fetches.
2. **`getRecentAlertHistories`** (called from `GET /alerts` API) — fired one aggregation query per alert (N+1 pattern), multiplying load on every page view.
## Changes
### `getPreviousAlertHistories` (`packages/api/src/tasks/checkAlerts/index.ts`)
- Replaced chunked `$in` batches (50 IDs per chunk) with **per-alert queries** using `PQueue({ concurrency: 20 })`
- Each query matches on a single `alert` value, so the compound index `{alert: 1, group: 1, createdAt: -1}` delivers results already sorted — no in-memory sort needed
- Replaced `$first: '$$ROOT'` with `$first: '$createdAt'` / `$first: '$state'` to allow projection pushdown (DocumentDB can avoid full document fetches)
### `getRecentAlertHistories` (`packages/api/src/controllers/alertHistory.ts`)
- Added `getRecentAlertHistoriesBatch()` — runs per-alert queries with `PQueue({ concurrency: 20 })` to control parallelism
- Each per-alert query uses the `{alert: 1, createdAt: -1}` index for a single-range scan
### Alerts API route (`packages/api/src/routers/api/alerts.ts`)
- Replaced N individual `getRecentAlertHistories` calls with a single `getRecentAlertHistoriesBatch` call
## Why per-alert queries instead of `$in`?
The `$in` operator on the leading field of a compound index produces multiple index range scans. The merged results are **not** globally sorted, forcing DocumentDB to perform an expensive in-memory sort. With per-alert queries, each query walks a single contiguous range in the index — sort is free, and `$group` + `$first` can short-circuit immediately per group.
## Testing
- All 14 integration tests pass (`make dev-int FILE=alertHistory.test`)
- Updated the batching test in `checkAlerts.test.ts` to verify per-alert query behavior
- Added 5 new tests for `getRecentAlertHistoriesBatch` covering batch results, empty histories, per-alert limits, and ALERT state detection
## Summary
- Adds worktree-aware port isolation for E2E tests, mirroring the existing `dev-int` slot mechanism so multiple agents/developers can run E2E tests in parallel without port conflicts
- Fixes the navigation E2E test that was broken by Live Tail URL updates swallowing client-side navigation
- Adds `dev-e2e` Makefile target for running specific tests with `FILE=` and `GREP=` filters, plus `REPORT=1` to open the HTML report after tests finish
## Port Isolation
Each worktree gets a deterministic slot (0–99) computed from its directory name. All E2E service ports are offset by that slot in the **44000–50100** range, avoiding collisions with `dev` (4317–27017) and `dev-int` (14320–40098).
| Service | Base + slot | Variable |
|---|---|---|
| ClickHouse HTTP | 48123 + slot | `HDX_E2E_CH_PORT` |
| ClickHouse Native | 49000 + slot | `HDX_E2E_CH_NATIVE_PORT` |
| MongoDB | 49998 + slot | `HDX_E2E_MONGO_PORT` |
| API server | 49100 + slot | `HDX_E2E_API_PORT` |
| App (fullstack) | 48081 + slot | `HDX_E2E_APP_PORT` |
| App (local) | 48001 + slot | `HDX_E2E_APP_LOCAL_PORT` |
| OpAMP | 44320 + slot | `HDX_E2E_OPAMP_PORT` |
## New Make Targets
```bash
make dev-e2e FILE=navigation # Run specific test file
make dev-e2e FILE=navigation GREP="help menu" # Filter by test name
make dev-e2e GREP="should navigate" # Grep across all files
make dev-e2e FILE=navigation REPORT=1 # Open HTML report after run
make dev-e2e-clean # Remove test artifacts
```
## Linear
https://linear.app/hyperdx/issue/HDX-3796
## Summary
- Fix slow and jumpy vertical scrolling in Firefox on the timeline chart's onWheel handler
- Firefox reports small non-zero `deltaX` values during vertical trackpad scrolling (trackpad gestures are rarely perfectly vertical), while Chrome silently zeroes these out. The previous `if (deltaX !== 0)` check was calling `preventDefault()` on nearly every vertical scroll in Firefox, blocking native scroll behavior.
- Replace the `deltaX !== 0` check with `Math.abs(deltaX) > Math.abs(deltaY)` so horizontal panning only activates when the gesture is predominantly horizontal, and skip unnecessary `setOffset` calls and re-renders on vertical scrolls
### How to test locally or on Vercel
1. In Firefox, verify vertical scrolling on the timeline chart is smooth and not blocked
2. In Chrome, verify vertical scrolling still works as before
3. Verify horizontal trackpad panning still works in both browsers
4. Verify Ctrl/Cmd + scroll zoom still works in both browsers
## Summary
This PR introduces a new dashboards listing page, which lists the available dashboards. Each individual dashboard is no longer listed in the sidebar. The new listing page supports searching by name and filtering by tag. This PR is a continuation of @elizabetdev's #1805, with some changes, additional tests, and refactorings.
This page does client-side sort and filter. There is no server-side pagination, filtering, or sorting. That is left as a future improvement, should it become necessary.
### Screenshots or video
<img width="2556" height="794" alt="Screenshot 2026-03-24 at 7 45 54 AM" src="https://github.com/user-attachments/assets/e4c5dba0-6cdf-4f2a-a5f3-2e4e00979729" />
<img width="2553" height="842" alt="Screenshot 2026-03-24 at 7 45 43 AM" src="https://github.com/user-attachments/assets/fc0f5270-d6d3-47ff-be03-762abd82a7d1" />
<img width="2544" height="862" alt="Screenshot 2026-03-24 at 7 45 34 AM" src="https://github.com/user-attachments/assets/4b1957c3-0e6e-4910-ac66-830734604759" />
### How to test locally or on Vercel
The listing page can be tested in vercel preview.
### References
- Linear Issue: Closes HDX-3565
- Related PRs:
Bumps [flatted](https://github.com/WebReflection/flatted) from 3.3.3 to 3.4.2.
<details>
<summary>Commits</summary>
<ul>
<li><a href="3bf09091c3"><code>3bf0909</code></a> 3.4.2</li>
<li><a href="885ddcc33c"><code>885ddcc</code></a> fix CWE-1321</li>
<li><a href="0bdba705d1"><code>0bdba70</code></a> added flatted-view to the benchmark</li>
<li><a href="2a02dce7c6"><code>2a02dce</code></a> 3.4.1</li>
<li><a href="fba4e8f2e1"><code>fba4e8f</code></a> Merge pull request <a href="https://redirect.github.com/WebReflection/flatted/issues/89">#89</a> from WebReflection/python-fix</li>
<li><a href="5fe86485e6"><code>5fe8648</code></a> added "when in Rome" also a test for PHP</li>
<li><a href="53517adbef"><code>53517ad</code></a> some minor improvement</li>
<li><a href="b3e2a0c387"><code>b3e2a0c</code></a> Fixing recursion issue in Python too</li>
<li><a href="c4b46dbcbf"><code>c4b46db</code></a> Add SECURITY.md for security policy and reporting</li>
<li><a href="f86d071e0f"><code>f86d071</code></a> Create dependabot.yml for version updates</li>
<li>Additional commits viewable in <a href="https://github.com/WebReflection/flatted/compare/v3.3.3...v3.4.2">compare view</a></li>
</ul>
</details>
<br />
[](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/hyperdxio/hyperdx/network/alerts).
</details>
## Summary
Cleans up dead code and unused dependencies identified by [knip](https://knip.dev/). All removals were individually verified to have zero imports or references across the codebase. No functional changes.
**Deleted 9 unused files:**
- `packages/api/src/clickhouse/__tests__/clickhouse.V1_DEPRECATED_test.ts` — skipped deprecated test
- `packages/api/src/utils/email.ts` — empty stub functions, never imported
- `packages/api/src/utils/queue.ts` — unused utility class
- `packages/app/src/Checkbox.tsx` — replaced by Mantine Checkbox, never imported
- `packages/app/src/components/DBSearchPageFilters/index.ts` — dead barrel file (shadowed by sibling `.tsx`)
- `packages/app/src/components/Sources/index.ts` — dead barrel file (all consumers import submodules directly)
- `packages/app/src/components/WhereLanguageControlled.tsx` — unused component
- `packages/app/src/TabBarWithContent.tsx` — unused component
- `packages/app/src/vsc-dark-plus.ts` — unused Prism theme
**Removed 14 unused dependencies** from `packages/api`, `packages/app`, and `packages/common-utils` (e.g. `semver`, `react-query`, `react-sortable-hoc`, `@microsoft/fetch-event-source`, `store2`, `uuid`, etc.)
**Removed 17 unused devDependencies** across root, api, app, and common-utils (e.g. `@nx/workspace`, `@typescript-eslint/eslint-plugin`, `@typescript-eslint/parser`, `@types/semver`, `@types/react-table`, `rimraf`, `supertest`, `ts-node`, `tsc-alias`, `tsconfig-paths`, etc.)
**Replaced `react-papaparse` with `papaparse`** — code imports `papaparse` directly, not the React wrapper. Added `@types/papaparse` since the package doesn't bundle its own types.
**Cleaned up unused exports:**
- Trimmed barrel files (`AppNav/index.ts`, `SearchInput/index.ts`) to only re-export what's actually consumed
- Removed duplicate named exports where only the default export is used (`DBRowTableFieldWithPopover`, `DBRowTableRowButtons`)
- Un-exported interfaces and constants that are only used locally (`DBRowTableFieldWithPopoverProps`, `DBRowTableIconButtonProps`, `DBRowTableRowButtonsProps`, `BASE_URL`, `makeHandler`)
- Removed stale `supertest` from `allowModules` in `common-utils/eslint.config.mjs`
### How to test locally or on Vercel
1. `yarn install` — lockfile should resolve cleanly
2. `yarn workspace @hyperdx/common-utils ci:lint` — should pass (the only package where lint+tsc fully passes on this branch)
3. `npx knip` — should show reduced issue counts vs. main
### References
- Related PRs: #1973 (knip CI workflow)
## Summary
- **Link colors**: Override `--mantine-color-anchor` in ClickStack so links use blue in light mode and yellow in dark mode instead of Mantine's default primary color derivation.
- **Checkbox & Radio**: Use `vars` overrides to apply ClickStack accent color to active checkboxes and radios, with contrasting icon colors for readability in both themes.
- **Slider styling**: Replace inline `styles` with semantic tokens (`--color-slider-bar`, `--color-slider-thumb`, `--color-slider-dot`, etc.) and CSS modules for consistent 6px solid dot marks and token-based thumb/mark colors across both ClickStack and HyperDX themes.
- **Subtle Button variant**: Add `variant="subtle"` support to `Button` in both themes (transparent background, hover highlight, standard text color).
- **Docs**: Update `code_style.md` to document `variant="subtle"` as accepted for both `Button` and `ActionIcon`.
### Before
<img width="3896" height="1296" alt="image" src="https://github.com/user-attachments/assets/5a2f109a-88e3-46a1-8e38-95d51dfd5a6b" />
<img width="1806" height="2570" alt="image" src="https://github.com/user-attachments/assets/70cf6786-a487-477b-868f-7f2a18746053" />
### After
<img width="3596" height="1358" alt="image" src="https://github.com/user-attachments/assets/0ad3b885-e6b8-4edd-aade-97516740ed6b" />
<img width="1874" height="2684" alt="image" src="https://github.com/user-attachments/assets/fa00f2cc-49f8-4bd3-8379-3665b760bd4e" />
## Test plan
- [ ] Verify links are blue in ClickStack light mode and yellow in dark mode
- [ ] Verify checkboxes and radio buttons use the accent color when active in both themes
- [ ] Verify checkbox icon is dark in dark mode for contrast
- [ ] Verify slider marks are solid 6px dots, with correct colors in both modes
- [ ] Verify slider thumb uses theme-appropriate colors
- [ ] Verify `<Button variant="subtle">` renders correctly in both themes
- [ ] Verify no visual regressions in HyperDX theme slider styling
## Summary
This PR improves auto-complete in the following ways
1. Auto-complete suggestions will not appear after `AS`, since it is assumed that a user will not want to type an existing column or function name as a column alias
2. Accepting an auto-complete suggestion will replace characters after the cursor if they match the accepted suggestion. This is nice when, for example, I have typed `ResourceAttributes[]` and my cursor is before the `]` - accepting a suggestion will now replace the trailing `]` instead of leaving it be (in which case it would be duplicated after inserting the suggestion).
### Screenshots or video
https://github.com/user-attachments/assets/9577393c-6bfa-410b-b5ba-2ba6b00bc26b
### How to test locally or on Vercel
This can be tested in the preview environment.
### References
- Linear Issue: Closes HDX-2612
- Related PRs:
## Summary
This PR fixes a bug which caused searches with the defaultSelectExpression to be saved with an empty `select` value, resulting in no select value appearing when loading the saved search.
The fix is to save the default select expression when there is no select value in the `searchedConfig` value (from the URL params), which corresponds to the case in which the default select expression is shown on the (non-saved) search page.
### Screenshots or video
Before:
https://github.com/user-attachments/assets/2b578dab-2085-4e64-a79c-c2e6e8f085fa
After:
https://github.com/user-attachments/assets/bf713b86-a454-4fc8-9ed7-5370e2e665e0
### How to test locally or on Vercel
You can reproduce the behavior in the preview environment following the video demo above.
### References
- Linear Issue: Closes HDX-3298
- Related PRs:
## Summary
This PR fixes text index detection for indexes with quoted tokenizer arguments:
```
TYPE text(tokenizer = 'splitByNonAlpha')
```
### Screenshots or video
### How to test locally or on Vercel
The unit tests demonstrate the fix.
### References
- Linear Issue: Closes HDX-3812
- Related PRs:
## Summary
#1960 added support for OpenAI's chat completions api.
This change switches to using [OpenAI's new Responses API](https://developers.openai.com/api/docs/guides/migrate-to-responses) instead.
### How to test locally or on Vercel
### How to test locally
1. Set env vars:
`AI_PROVIDER=openai AI_API_KEY= AI_BASE_URL=<> AI_MODEL_NAME=<> AI_REQUEST_HEADERS={"X-Client-Id":"","X-Username":"", AI_ADDITIONAL_OPTIONS = {API_TYPE: "responses"}}`
3. Open Hyperdx's chart explorer and use the AI assistant chart builder
- e.g. "show me error count by service in the last hour"
4. Confirm the assistant returns a valid chart config.
### References
- Linear Issue:
- Related PRs:
## Summary
This PR improves dashboard filters
1. Dashboard filters can now have an associated WHERE condition which filters the rows from which filter values will be queried.
2. Multiple values can now be selected for a single dashboard filter
### Screenshots or video
Multiple values can now be selected for a single filters:
<img width="544" height="77" alt="Screenshot 2026-03-23 at 12 31 02 PM" src="https://github.com/user-attachments/assets/2390a2d7-8514-4eb8-ac3c-db102a5df99b" />
Filters now have an optional condition, which filters the values which show up in the dropdown:
<img width="451" height="476" alt="Screenshot 2026-03-23 at 12 30 44 PM" src="https://github.com/user-attachments/assets/eed7f69e-466e-42fd-93f1-c27bfbc06204" />
<img width="265" height="94" alt="Screenshot 2026-03-23 at 12 30 54 PM" src="https://github.com/user-attachments/assets/2ba46e33-a44a-45ea-a6bf-fb71f5373e46" />
This also applies to Preset Dashboard Filters
<img width="726" height="908" alt="Screenshot 2026-03-23 at 12 33 34 PM" src="https://github.com/user-attachments/assets/df648feb-32e2-4f5e-80e5-409e0443b38e" />
### How to test locally or on Vercel
This can be partially tested in the preview environment, but testing the following requires running locally
1. Preset dashboard filters
2. External API support
### References
- Linear Issue: Closes HDX-3631 Closes HDX-2987
- Related PRs:
## Summary
The cleanup logic only removed demo sources where connection === 'local', or the db != 'otel_v2'. Now removes all Demo/ClickPy sources by name before recreating them, and reuses an existing demo connection instead of creating duplicates.
### References
- Linear Issue: HDX-3800
## Summary
This update improves the Knip GitHub Action by adding detailed reporting of unused code issues. The changes include:
- Enhanced issue counting to include a breakdown of items per category.
- Added a detailed summary section in the PR comment that lists added and removed items for each category when there are changes.
- Improved error handling to ensure consistent return values even when parsing fails.
These enhancements provide clearer insights into unused code changes between branches, aiding in code maintenance and quality.
| Before | After |
| --- | --- |
| <img width="569" height="451" alt="Screenshot 2026-03-23 at 4 44 43 PM" src="https://github.com/user-attachments/assets/01c2fdcc-802f-4046-8ab3-f3e92ec46ae0" /> | <img width="981" height="706" alt="Screenshot 2026-03-23 at 4 42 47 PM" src="https://github.com/user-attachments/assets/8fefeb74-1606-4a26-b307-3aa5ff76265f" /> |
Note the additional details at bottom explaining the Function/Files causing diffs
Also, if no changes are detected, output is minimal:
<img width="1050" height="271" alt="Screenshot 2026-03-23 at 4 45 37 PM" src="https://github.com/user-attachments/assets/e075a5cd-4170-4999-a2c3-752447ac2f8a" />
## Summary
- Section expand/collapse is now tracked in URL query params (`collapsed`/`expanded`) instead of persisting to the DB on every chevron click
- The DB-stored `collapsed` field on `DashboardContainer` becomes the default fallback — what viewers see when opening a dashboard fresh (no URL state)
- Chevron click updates URL state only (per-viewer, shareable via link)
- "Collapse by Default" / "Expand by Default" menu action in the section header saves to the DB (via `setDashboard`), setting the default for all viewers
- `SectionHeader` now accepts separate `collapsed`/`defaultCollapsed` props and `onToggle`/`onToggleDefaultCollapsed` handlers
- Adds 7 unit tests for `SectionHeader`
Implements Drew's [review feedback on PR #1926](https://github.com/hyperdxio/hyperdx/pull/1926#discussion_r2966166505):
> IMO, expanding/collapsing should not be persisted to the dashboard UNLESS this option is used. [...] I think it would be nice to persist normal expand collapse states in the URL, and then fallback to the default state (saved in the DB based on this option here) if there is no URL state.
## Demo
Shows: expanded sections → chevron click collapses first section (URL updates to `?collapsed=...`) → menu shows "Collapse by Default" (DB action, separate from view state)
## Test plan
- [x] Open a dashboard with sections — collapse/expand via chevron click, verify URL updates (`?collapsed=...` / `?expanded=...`) without saving to DB
- [x] Copy the URL with collapse state and open in a new tab — verify sections reflect the URL state
- [x] Open the section menu and click "Collapse by Default" — verify this saves to DB (persists after page refresh without URL params)
- [x] Verify "Expand by Default" / "Collapse by Default" label reflects the DB default, not current view state
- [x] Run `yarn ci:unit --testPathPatterns='SectionHeader'` — all 7 tests pass
🤖 Generated with [Claude Code](https://claude.com/claude-code)
## Summary
Adds a column toggle button (+ / - icon) next to the "Show Distribution" button in each filter group header on the search page. Clicking the button adds or removes the filter's field from the `SELECT` statement, and the table reflects the change immediately.
### Changes
- **`FilterGroup`** (`DBSearchPageFilters.tsx`): Added `onColumnToggle` and `isColumnDisplayed` props. Renders an `ActionIcon` with `IconPlus` (add) or `IconMinus` (remove) between the distribution toggle and the pin field button.
- **`NestedFilterGroup`**: Passes the new column toggle props through to child `FilterGroup` components.
- **`DBSearchPage.tsx`**: Passes `toggleColumn` and `displayedColumns` to `DBSearchPageFilters`, reusing the existing `toggleColumn` callback that manages the `SELECT` form field.
### Screenshots or video
| Before | After |
| :----- | :---- |
| Only distribution and pin buttons in filter header | New +/- column button appears between distribution and pin buttons |
### How to test locally or on Vercel
1. Navigate to the Search page
2. Open the filter panel on the left side
3. Find any filter group and hover over the header area — a `+` icon should appear next to the distribution chart icon
4. Click the `+` icon — the field should be added to the `SELECT` input and appear as a column in the results table
5. Click the `-` icon (now shown since the column is displayed) — the field should be removed from `SELECT` and the column disappears
### References
- Linear Issue: HDX-3770
Linear Issue: [HDX-3770](https://linear.app/clickhouse/issue/HDX-3770/telstra-add-column-from-filter-panel-in-dbsearchpage)
<div><a href="https://cursor.com/agents/bc-11d702b5-a58e-485c-982f-61d990e45091"><picture><source media="(prefers-color-scheme: dark)" srcset="https://cursor.com/assets/images/open-in-web-dark.png"><source media="(prefers-color-scheme: light)" srcset="https://cursor.com/assets/images/open-in-web-light.png"><img alt="Open in Web" width="114" height="28" src="https://cursor.com/assets/images/open-in-web-dark.png"></picture></a> <a href="https://cursor.com/background-agent?bcId=bc-11d702b5-a58e-485c-982f-61d990e45091"><picture><source media="(prefers-color-scheme: dark)" srcset="https://cursor.com/assets/images/open-in-cursor-dark.png"><source media="(prefers-color-scheme: light)" srcset="https://cursor.com/assets/images/open-in-cursor-light.png"><img alt="Open in Cursor" width="131" height="28" src="https://cursor.com/assets/images/open-in-cursor-dark.png"></picture></a> </div>
Co-authored-by: Cursor Agent <199161495+cursoragent@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
## Summary
Adds [Knip](https://knip.dev) to the monorepo to detect unused files, dependencies, and exports. The goal is to reduce dead code over time and prevent new unused code from accumulating.
**What's included:**
- Root-level `knip.json` configured for all three workspaces (`packages/app`, `packages/api`, `packages/common-utils`)
- `yarn knip` and `yarn knip:ci` scripts for local and CI usage
- GitHub Action (`.github/workflows/knip.yml`) that runs on every PR to `main`, compares results against the base branch, and posts a summary comment showing any increase or decrease in unused code
- Removed the previous app-only `packages/app/knip.json` in favor of the monorepo-wide config
**How the CI workflow works:**
1. Runs Knip on the PR branch
2. Checks out `main` and runs Knip there
3. Compares issue counts per category and posts/updates a PR comment with a diff table
This is additive — Knip runs as an informational check and does not block PRs.
## Summary
- Replace the "Add New Tile" button and overflow "Add Section" menu item with a unified "Add" dropdown at the bottom of the dashboard
- Dropdown contains "New Tile" and "New Section" — creative actions only
- "Import Dashboard" stays in the overflow menu alongside Export, Delete, and other management actions
- Follows the Grafana/Datadog pattern of a single entry point for adding content, which scales to future container types (tabs, groups)
## Test plan
- [x] Click "Add" button at bottom of dashboard → dropdown opens upward with "New Tile" and "New Section"
- [x] Click "New Tile" → tile editor opens (same as before)
- [x] Click "New Section" → new section appears on dashboard
- [x] Overflow menu still contains "Import Dashboard" with contextual label
- [x] E2E tests pass (updated page object for two-click dropdown flow)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
## Summary
Improve the materialized column optimization by allowing it to be applied when `WITH` clauses are expression aliases (i.e., `isSubquery: false`). Previously, any `WITH` clause would disable this optimization. This change ensures that materialized columns are still considered for performance benefits when the `WITH` clause does not represent a subquery.
### Screenshots or video
| Before | After |
| :----- | :---- |
| | |
### How to test locally or on Vercel
1. Create a ClickHouse table with a materialized column, e.g.:
```sql
ALTER TABLE otel_logs ADD COLUMN awesome_attribute String MATERIALIZED LogAttributes['awesome_attribute']
```
2. Open the Explore view for logs (`/search`)
3. Add a filter for `awesome_attribute` (or `LogAttributes['awesome_attribute']`)
4. Inspect the POST body of `/clickhouse-proxy` requests in the network tab:
- Before fix: The histogram (time chart) query contains `LogAttributes['awesome_attribute']` (full map scan), while the search results query correctly uses `awesome_attribute`.
- After fix: Both the histogram and search results queries use `awesome_attribute` (the materialized column).
### References
- Linear Issue: #1957
- Related PRs:
## Summary
- **Replace QEMU-emulated multi-platform builds with native ARM64 runners** for both `release.yml` and `release-nightly.yml`, significantly speeding up CI build times
- Each architecture (amd64/arm64) now builds in parallel on native hardware, then a manifest-merge job combines them into a multi-arch Docker tag using `docker buildx imagetools create`
- Migrate from raw Makefile `docker buildx build` commands to `docker/build-push-action@v6` for better GHA integration
## Changes
### `.github/workflows/release.yml`
- Removed QEMU setup entirely
- Replaced single `release` matrix job with per-image build+publish job pairs:
- `build-otel-collector` / `publish-otel-collector` (runners: `ubuntu-latest` / `ubuntu-latest-arm64`)
- `build-app` / `publish-app` (runners: `Large-Runner-x64-32` / `Large-Runner-ARM64-32`)
- `build-local` / `publish-local` (runners: `Large-Runner-x64-32` / `Large-Runner-ARM64-32`)
- `build-all-in-one` / `publish-all-in-one` (runners: `Large-Runner-x64-32` / `Large-Runner-ARM64-32`)
- Added `check_version` job to centralize skip-if-exists logic (replaces per-image `docker manifest inspect` in Makefile)
- Removed `check_release_app_pushed` artifact upload/download — `publish-app` now outputs `app_was_pushed` directly
- Scoped GHA build cache per image+arch (e.g. `scope=app-amd64`) to avoid collisions
- All 4 images build in parallel (8 build jobs total), then 4 manifest-merge jobs, then downstream notifications
### `.github/workflows/release-nightly.yml`
- Same native runner pattern (no skip logic since nightly always rebuilds)
- 8 build + 4 publish jobs running in parallel
- Slack failure notification and OTel trace export now depend on publish jobs
### `Makefile`
- Removed `release-*` and `release-*-nightly` targets (lines 203-361) — build logic moved into workflow YAML
- Local `build-*` targets preserved for developer use
## Architecture
Follows the same pattern as `release-ee.yml` in the EE repo:
```
check_changesets → check_version
│
┌───────────────────┼───────────────────┬───────────────────┐
v v v v
build-app(x2) build-otel(x2) build-local(x2) build-aio(x2)
│ │ │ │
publish-app publish-otel publish-local publish-aio
│ │ │ │
└─────────┬─────────┴───────────────────┴───────────────────┘
v
notify_helm_charts / notify_clickhouse_clickstack
│
otel-cicd-action
```
## Notes
- `--squash` flag dropped — it's an experimental Docker feature incompatible with `build-push-action` in multi-platform mode. `sbom` and `provenance` are preserved via action params.
- Per-arch intermediate tags (e.g. `hyperdx/hyperdx:2.21.0-amd64`) remain visible on DockerHub — this is standard practice.
- Dual DockerHub namespace tagging (`hyperdx/*` + `clickhouse/clickstack-*`) preserved.
## Sample Run
https://github.com/hyperdxio/hyperdx/actions/runs/23362835749
## Summary
Closes [HDX-3784](https://linear.app/clickhouse/issue/HDX-3784/increase-discoverability-of-trace-sources)
- **Context-aware icons for all source kinds**: The data source dropdown now shows icons for every source type — `IconLogs` for logs, `IconConnection` for traces, `IconDeviceLaptop` for sessions, and `IconChartLine` for metrics. Falls back to `IconStack` when no source is selected.
- **Inline source actions**: "Create New Source" and "Edit Sources" actions are now part of the dropdown itself under an "Actions" group with a labeled separator, replacing the separate gear icon menu (`SourceEditMenu`).
- **Dependency update**: Updated `@tabler/icons-react` from v3.5.0 to v3.40.0 to get `IconConnection`.
- **Fix: source management regression when `HDX_LOCAL_DEFAULT_SOURCES` is set**: Before this PR, there were two ways to create/edit sources: (1) options inside the dropdown, which were hidden when `HDX_LOCAL_DEFAULT_SOURCES` is set, and (2) a gear icon button next to the dropdown, which was always visible. This PR removed the gear icon and kept only the dropdown options, but they were still configured to hide when `HDX_LOCAL_DEFAULT_SOURCES` is set — leaving users with no way to manage sources. Fixed by removing that guard so the dropdown options always appear.
<img width="1236" height="492" alt="image" src="https://github.com/user-attachments/assets/6999626b-685b-4037-a003-b09018cfbadf" />
<img width="426" height="240" alt="Screenshot 2026-03-20 at 17 49 30" src="https://github.com/user-attachments/assets/28aaef44-7574-4c54-b721-b2a3a79b3507" />
## Changes
- `packages/app/src/components/SourceSelect.tsx` -- Dynamic left icon based on selected source kind (all 4 kinds: log, trace, session, metric), `onEdit` prop, grouped action items with icons, `renderOption` for source kind and action item icons. Removed `hasLocalDefaultSources` guard so source management actions are always available.
- `packages/app/src/components/SelectControlled.tsx` -- Added `onEdit` callback support, fixed `selected` check to handle grouped data.
- `packages/app/src/DBSearchPage.tsx` -- Removed `SourceEditMenu` component, added `onEditSources` callback, wired `onEdit` to `SourceSelectControlled`.
- `packages/app/styles/SourceSelectControlled.module.scss` -- Group label separator styling with semantic `--color-border` token.
- `packages/app/package.json` -- Updated `@tabler/icons-react` to `^3.39.0`.
## Test plan
- [ ] Select a log source and verify `IconLogs` appears as the left icon
- [ ] Select a trace source and verify `IconConnection` appears as the left icon
- [ ] Select a session source and verify `IconDeviceLaptop` appears as the left icon
- [ ] Select a metric source and verify `IconChartLine` appears as the left icon
- [ ] Verify each source in the dropdown shows its corresponding kind icon
- [ ] Open the dropdown and verify "Create New Source" and "Edit Sources" appear under the "Actions" group with icons
- [ ] Click "Create New Source" and verify the modal opens
- [ ] Click "Edit Sources" and verify navigation to edit (local mode: modal, cloud mode: /team)
- [ ] Verify the gear icon menu is no longer present next to the select
- [ ] **With `NEXT_PUBLIC_HDX_LOCAL_DEFAULT_SOURCES` set**: verify "Create New Source" and "Edit Sources" still appear in the dropdown and work correctly
## Summary
- Override Mantine's `--mantine-color-text` CSS variable to use our semantic `--color-text` token in both themes (hyperdx and clickstack), for both dark and light modes.
- Ensures all Mantine components that rely on `--mantine-color-text` use the correct theme-aware text color.
- Overrides are placed in the `/* Mantine Overrides */` section at the end of each theme mixin, consistent with the existing `--mantine-color-body` override.
## Test plan
- [ ] Verify text color in Mantine components matches the theme's `--color-text` in dark mode (both themes)
- [ ] Verify text color in Mantine components matches the theme's `--color-text` in light mode (both themes)
Made with [Cursor](https://cursor.com)
## Summary
The node commands to start the API server and alert task are duplicated across 4+ files, each hardcoding the build output path and node require flags. When the build process changed (esbuild introduction/revert per HDX-2690), the downstream operator and helm chart broke because their entrypoint commands were stale.
This PR introduces `packages/api/bin/hyperdx`, a single shell script that is the **sole source of truth** for how to launch API and task processes. It resolves the build directory relative to its own location, applies the correct node flags (`-r @hyperdx/node-opentelemetry/build/src/tracing`), and exposes two subcommands:
- `hyperdx api` -- starts the API server
- `hyperdx task <name>` -- runs a named task (e.g., `check-alerts`)
All Dockerfiles and entry scripts now delegate to this script instead of inlining the node command. Future build changes only need updating in one place.
### How to test locally or on Vercel
1. **Build the standalone API image** and confirm the entrypoint works:
```bash
docker build . -f packages/api/Dockerfile -t hyperdx-api-test:latest --target prod
docker run -d --name hdx-api-test -p 18000:8000 hyperdx-api-test:latest
sleep 5
docker logs hdx-api-test 2>&1 | head -30
# Should show OpenTelemetry init + MongoStore error (expected without Mongo)
# No "file not found" or "permission denied" errors
docker stop hdx-api-test && docker rm hdx-api-test
```
2. **Build and run the all-in-one image** for a full integration test:
```bash
make build-local
docker run -d --name hdx-aio-test -p 18080:8080 -p 18000:8000 hyperdx/hyperdx-local:2.21.0
# Wait up to 90s for startup, then:
curl -sf http://localhost:18080/api/health # should return {"data":"OK",...}
curl -sf http://localhost:18000/health # should return {"data":"OK",...}
docker exec hdx-aio-test sh -c "ps aux"
# Confirm API, APP, and ALERT-TASK processes are running via the hyperdx script
docker stop hdx-aio-test && docker rm hdx-aio-test
```
3. **Build the prod image** to confirm the entry script changes are valid:
```bash
make build-app
```
**Testing performed:** All three Docker image targets were built and verified locally. The standalone API image started node via `hyperdx api` correctly (crashed on missing MongoDB as expected). The all-in-one image passed health checks on both `localhost:18080/api/health` and `localhost:18000/health`, with all three processes (API, APP, ALERT-TASK) confirmed running inside the container using the new entry point script.
### References
- Linear Issue: [HDX-2712](https://linear.app/clickhouse/issue/HDX-2712/use-a-single-entry-point-script-for-both-hyperdx-api-and-alert-job)
- Related PRs: HDX-2690 (root cause), HDX-2815 (downstream helm chart follow-up)
- **Follow-up needed:** Update helm chart cron job template and operator template in `ClickHouse/ClickStack-helm-charts` to use `./packages/api/bin/hyperdx task check-alerts`
Made with [Cursor](https://cursor.com)
## Summary
Adds the authoring experience for dashboard sections (create, rename, delete, manage tiles) and introduces a polymorphic `DashboardContainer` abstraction that future-proofs the schema for tabs and groups.
Builds on #1900 (core collapsible sections mechanics). Closes#1897.
### Schema: `DashboardSection` → `DashboardContainer`
- Renamed `DashboardSectionSchema` → `DashboardContainerSchema` with a new `type` field (`'section'` for now, extensible to `'group'` / `'tab'` later)
- `sectionId` → `containerId` on tiles
- `sections` → `containers` on dashboards
- Updated across all packages: common-utils types, API Mongoose model, app types, import/export utils
### Authoring UX
| Action | How |
|---|---|
| **Create section** | Dashboard `...` overflow menu → "Add Section" |
| **Rename section** | Click the title text directly (Kibana-style inline editing) |
| **Delete section** | Hover section header → `...` → Delete Section (tiles become ungrouped, not deleted) |
| **Collapse/expand** | Click section header chevron |
| **Toggle default state** | Hover header → `...` → Collapse/Expand by Default |
| **Add tile to section** | Hover section header → `+` button opens tile editor pre-assigned to that section |
| **Move tile to section** | Hover tile → grid icon → pick target section from dropdown |
| **Move tile out** | Same dropdown → "(Ungrouped)" |
### UX polish (informed by best practices research)
- **Click-to-rename** — click section title text to edit inline (no menu navigation needed)
- **Hover-only controls** — `...` menu and `+` button only appear on section header hover, keeping view mode clean
- **"Add Section" demoted** — moved from equal-sized button to dashboard overflow menu (section creation is less frequent than tile creation)
- **"Move to Section" reordered** — placed before delete button for discoverability, uses `IconLayoutList` instead of `IconFolders`
### What's NOT in this PR (follow-up work)
- **Drag tiles between sections** — needs `react-dnd` custom drag layer; data model already supports it (`containerId` update)
- **Reorder sections** — needs sortable list library; data model supports it (array order)
- **Tabs / Groups** — new container types; just add to the `type` enum and build UIs
## Test plan
- [x] 30 unit tests pass (16 existing schema/grouping + 14 new authoring operations)
- [x] All 110 dashboard tests pass unchanged
- [x] ESLint clean
- [x] No TypeScript errors in changed files
- [x] Backward compatible — dashboards without containers render exactly as before
🤖 Generated with [Claude Code](https://claude.com/claude-code)
== Motivation ==
Time histograms on the search page silently drop data past an hour/minute boundary when the source `timestampValueExpression` includes a `toStartOf*` expression for primary key optimization.
== Details ==
When `convertToTimeChartConfig` aligns the date range to granularity it sets `dateRangeEndInclusive: false`, which is correct for the raw timestamp column (end was rounded up, so `<` gives equivalent coverage). But `timeFilterExpr` applies that same `<` uniformly to every expression in a compound `timestampValueExpression`. With a range ending at `04:08`, this yields `toStartOfHour(ts) < toStartOfHour(04:08)` = `< 04:00` — excluding the entire `04:xx` hour.
The coarse filter exists only for index pruning; the raw column already enforces exact bounds. Making it wider by one interval is harmless, making it narrower drops real rows.
== Testing ==
- `yarn jest renderChartConfig.test.ts` — 54 passed, 29 snapshots passed
- Added cases for `toStartOfHour` with exclusive end, compound expr with exclusive end, and exclusive start
## Summary
This PR adds the Spectral linter for linting our OpenAPI spec, with rules preventing fields with missing examples or descriptions, which are often enforced in the Control Plane repo.
This PR also resolves lint errors that were already present.
### Screenshots or video
### How to test locally or on Vercel
Run `make ci-lint` to lint the openapi specs
### References
- Linear Issue: Closes HDX-3768
- Related PRs:
## Summary
Some Distributed tables refer to "local" tables which are not available on the local node. To read metadata (primary key, skip indexes) for such distributed tables, we can read from `cluster(<cluster with local tables>, system, <metadata table>)` instead of the local system tables.
### Screenshots or video
After adding the distributed table as a source, we can see that the order by optimization and the skip index detection are working as intended, indicating that the cluster() queries are working and fetching the required metadata:
<img width="2099" height="624" alt="Screenshot 2026-03-19 at 8 39 48 AM" src="https://github.com/user-attachments/assets/1f8384fb-8ae1-4549-9432-e4359ac72e02" />
### How to test locally or on Vercel
<details>
<summary>First, setup two clusters</summary>
- docker compose: https://pastila.clickhouse.com/?003644f9/c65444330e3601726c00b7cc9e095e71#7W62EjQox6MnTj0vCGL0AA==GCM
- config.xml: https://pastila.clickhouse.com/?002ee55c/d82248e8db633b3fbaf14cee2ee51b0e#royNZZ4snbBpZUd8xulw5w==GCM
- config-2.xml: https://pastila.clickhouse.com/?009f57b4/cf1d51fa36eee025f17beda4da6621fa#KBbHphEhcS+1m7mBqNfY4A==GCM
- config-3.xml: https://pastila.clickhouse.com/?003115c7/e984fc157de834095bedea86bc698dca#1rEmfXnq6H0tiT4qNgayNg==GCM
- keeper.xml: https://pastila.clickhouse.com/?005dc0a8/1599254d15dbac2868f04f5ab33125c2#R90W3HfA3J0yeTNf9hrDNQ==GCM
</details>
<details>
<summary>Then setup the local and distributed tables</summary>
```sql
CREATE TABLE default.otel_logs_toStartOf on cluster hdx_cluster_2
(
`Timestamp` DateTime64(9) CODEC(Delta(8), ZSTD(1)),
`TimestampTime` DateTime DEFAULT toDateTime(Timestamp),
`TraceId` String CODEC(ZSTD(1)),
`SpanId` String CODEC(ZSTD(1)),
`TraceFlags` UInt8,
`SeverityText` LowCardinality(String) CODEC(ZSTD(1)),
`SeverityNumber` UInt8,
`ServiceName` LowCardinality(String) CODEC(ZSTD(1)),
`Body` String CODEC(ZSTD(1)),
`ResourceSchemaUrl` LowCardinality(String) CODEC(ZSTD(1)),
`ResourceAttributes` Map(LowCardinality(String), String) CODEC(ZSTD(1)),
`ScopeSchemaUrl` LowCardinality(String) CODEC(ZSTD(1)),
`ScopeName` String CODEC(ZSTD(1)),
`ScopeVersion` LowCardinality(String) CODEC(ZSTD(1)),
`ScopeAttributes` Map(LowCardinality(String), String) CODEC(ZSTD(1)),
`LogAttributes` Map(LowCardinality(String), String) CODEC(ZSTD(1)),
INDEX idx_trace_id TraceId TYPE bloom_filter(0.001) GRANULARITY 1,
INDEX idx_res_attr_key mapKeys(ResourceAttributes) TYPE bloom_filter(0.01) GRANULARITY 1,
INDEX idx_res_attr_value mapValues(ResourceAttributes) TYPE bloom_filter(0.01) GRANULARITY 1,
INDEX idx_scope_attr_key mapKeys(ScopeAttributes) TYPE bloom_filter(0.01) GRANULARITY 1,
INDEX idx_scope_attr_value mapValues(ScopeAttributes) TYPE bloom_filter(0.01) GRANULARITY 1,
INDEX idx_log_attr_key mapKeys(LogAttributes) TYPE bloom_filter(0.01) GRANULARITY 1,
INDEX idx_log_attr_value mapValues(LogAttributes) TYPE bloom_filter(0.01) GRANULARITY 1,
INDEX idx_body Body TYPE tokenbf_v1(32768, 3, 0) GRANULARITY 8
)
ENGINE = MergeTree
PARTITION BY toDate(TimestampTime)
PRIMARY KEY (toStartOfMinute(TimestampTime), ServiceName, TimestampTime)
ORDER BY (toStartOfMinute(TimestampTime), ServiceName, TimestampTime, Timestamp)
TTL TimestampTime + toIntervalDay(30)
SETTINGS index_granularity = 8192, ttl_only_drop_parts = 1;
CREATE TABLE default.otel_logs_toStartOf_distributed on cluster hdx_cluster
(
`Timestamp` DateTime64(9) CODEC(Delta(8), ZSTD(1)),
`TimestampTime` DateTime DEFAULT toDateTime(Timestamp),
`TraceId` String CODEC(ZSTD(1)),
`SpanId` String CODEC(ZSTD(1)),
`TraceFlags` UInt8,
`SeverityText` LowCardinality(String) CODEC(ZSTD(1)),
`SeverityNumber` UInt8,
`ServiceName` LowCardinality(String) CODEC(ZSTD(1)),
`Body` String CODEC(ZSTD(1)),
`ResourceSchemaUrl` LowCardinality(String) CODEC(ZSTD(1)),
`ResourceAttributes` Map(LowCardinality(String), String) CODEC(ZSTD(1)),
`ScopeSchemaUrl` LowCardinality(String) CODEC(ZSTD(1)),
`ScopeName` String CODEC(ZSTD(1)),
`ScopeVersion` LowCardinality(String) CODEC(ZSTD(1)),
`ScopeAttributes` Map(LowCardinality(String), String) CODEC(ZSTD(1)),
`LogAttributes` Map(LowCardinality(String), String) CODEC(ZSTD(1))
)
ENGINE = Distributed('hdx_cluster_2', 'default', 'otel_logs_toStartOf', rand());
ALTER TABLE otel_logs_toStartOf ON CLUSTER hdx_cluster_2 ADD INDEX text_idx(Body)
TYPE text(tokenizer=splitByNonAlpha, preprocessor=lower(Body))
SETTINGS enable_full_text_index=1;
ALTER TABLE otel_logs_toStartOf ON CLUSTER hdx_cluster_2 MATERIALIZE INDEX text_idx;
```
</details>
<details>
<summary>To test text index detection, first enable full text indexes locally in your users.xml file</summary>
```xml
<clickhouse>
<profiles>
<default>
...
<enable_full_text_index>1</enable_full_text_index>
</default>
</profiles>
...
<clickhouse>
```
</details>
### References
- Linear Issue:
- Related PRs:
