Closes HDX-3311
# Summary
This PR adds E2E tests that verify that dashboards created and edited through the external API can be rendered correctly in the UI.
- The tests create a dashboard with all tile types through the external API, then assert that the tile edit form contains the correct values for all of the tile's properties.
- The tests verify that the dashboard can be saved from the UI, proving that the dashboard data conforms to the _internal_ dashboard API's schema, after being created by the external dashboards API
- The tests also validate that dashboard updates and deletes via the external API are reflected in the UI.
- Introduce a new flag `HYPERDX_OTEL_EXPORTER_CREATE_LEGACY_SCHEMA` (default to false) to otel collector
- Custom ClickStack schemas should be enforced by default
- ClickHouse tables migration logs should be stored in `clickstack_db_version_xxx` tables
- The collector will run the migration at startup and retry if it fails to connect to the database (using exponential backoff).
- Fully backward compatible
Ref: HDX-3301
Fixes HDX-3325
* Use enums where possible
* Add descriptions where missing
* Simplify Alert request and types
* Fix a disceprency with alert request and response fields: `team` vs `teamId`, `dashboard` vs `dashboardId`, `savedSearch` vs `savedSearchId`
Closes HDX-3303
# Summary
This PR reworks the external dashboard APIs in the following ways
1. Previously, there was no way to create a renderable dashboard using the create or update routes, because the APIs saved dashboard data in a format incompatible with v2 of HyperDX. This PR introduces a transformation layer to transform external API request and response formats into the v2 dashboard format prior to saving.
2. This PR also corrects and improves the openapi documentation associated with the dashboard endpoints.
This PR introduces a number of **breaking changes** in the dashboard API, which is acceptable because the APIs are already (a) effectively non-functional for writing dashboard data and (b) not aligned at all with the previously published spec when reading dashboard data.
1. The `dataSource` field has been removed from each series and replaced with a `sourceId` field, which specifies the sourceId of the source which should be queried. The `dataSource` argument was inflexible and could only specify `events`, `rrweb`, or `metrics` - these are leftover from V1, and are not very compatible with the design of HyperDX V2.
2. Dashboard responses previously would have contained `config` fields, in the format of a V2 dashboard config. Results are now transformed and align with the spec, which contains config data in a `series` array.
3. Additional validation has been added to prevent creation of some invalid dashboards
## Multi-Theme System
Adds infrastructure for supporting multiple brand themes (HyperDX & ClickStack) with the ability to switch between them in development mode.
### Preview
https://hyperdx-v2-oss-app-git-add-themes-hyperdx.vercel.app/
### How theme resolution works
Theme switching is **only available in dev/local mode** (`NODE_ENV=development` or `NEXT_PUBLIC_IS_LOCAL_MODE=true`).
**Resolution priority (highest to lowest):**
1. **localStorage** → `hdx-dev-theme` key _(set via explicit UI action)_
2. **Environment variable** → `NEXT_PUBLIC_THEME`
3. **Default** → `hyperdx`
**In production**, the theme is determined solely by `NEXT_PUBLIC_THEME` (defaults to `hyperdx`).
### Changes
#### Theme System
- Added `ThemeProvider` with context for theme-aware components
- Added theme configurations for HyperDX and ClickStack (logos, colors, favicons)
- Dynamic favicon switching based on active theme
#### Component Renaming
- `Icon` → `Logomark` (the icon/symbol only)
- `Logo` → `Wordmark` (icon + text branding)
- Each theme provides its own `Logomark` and `Wordmark` components
#### User Preferences (`useUserPreferences`)
> **Note:** The hook was **modified**, not deleted. It's still actively used across the codebase.
- Renamed `theme` property to `colorMode` to clarify it controls light/dark mode (not brand theme)
- Removed background overlay feature:
- Deleted `backgroundEnabled`, `backgroundUrl`, `backgroundBlur`, `backgroundOpacity`, `backgroundBlendMode` properties
- Deleted `useBackground` hook
- Removed background overlay UI from `UserPreferencesModal`
- Removed `.hdx-background-image` CSS rules
#### Security
- Added hex color validation for `theme-color` meta tag to prevent XSS
- Hydration-safe `DynamicFavicon` component
#### Type Safety & Documentation
- Added comprehensive documentation in `types.ts` explaining the distinction between:
- **Color Mode** (light/dark) — user-selectable, stored in `useUserPreferences`
- **Brand Theme** (hyperdx/clickstack) — deployment-configured, NOT user-selectable in production
- Clear type definitions for `ThemeName`, `ThemeConfig`, and `FaviconConfig`
#### Tests Added
- `theme/__tests__/index.test.ts` — Theme registry, `getTheme()`, localStorage helpers
- `theme/__tests__/ThemeProvider.test.tsx` — Context, hooks, hydration safety
- `components/__tests__/DynamicFavicon.test.tsx` — XSS sanitization, hex color validation
---
*This PR is a work in progress. Feedback welcome!*
You can use Cmd+F to search through the log search table. We've implemented a custom input that overrides the native input that allows you to search through virtual items. However, if you click on one of the rows, a drawer opens, and we don't want that search input to listen for keyboard events anymore at this point. This PR fixes this behaviour.
https://github.com/user-attachments/assets/cbcaea4d-77a8-4c37-a368-bffc44ba4d59
Fixes HDX-3302
Today, users have to set up an OpAMP server to run with our clickstack OTel collector. Instead, we should allow users to disable OpAMP when they're using ClickHouse Cloud with the clickstack integration.
This can be determined by `OPAMP_SERVER_URL` not being defined by the user.
The end result is that a user can do
```
docker run \
-e CLICKHOUSE_ENDPOINT=${CLICKHOUSE_ENDPOINT} \
-e CLICKHOUSE_USER=default \
-e CLICKHOUSE_PASSWORD=${CLICKHOUSE_PASSWORD} \
-p 8080:8080 -p 4317:4317 -p 4318:4318 \
clickhouse/clickstack-otel-collector:latest
```
Ref: HDX-3300
# Summary
This PR updates the external Alerts API specs so that the specs match the actual behavior of the API.
I've also added a validator to an API which was missing any validation.
For the clickstack api integration work, we need to make sure if there's a 400 bad request on hyperdx side, the error is nicely presented in a single message field, like it is for other errors like 401 and 500. This will make our clickstack openapi show better errors instead of showing "unknown error" because the clickstack proxy in control plane doesn't have the functionality to decipher and format zod errors nicely.
|Before|After|
|--|--|
|<img width="836" height="886" alt="CleanShot 2026-01-29 at 12 15 26@2x" src="https://github.com/user-attachments/assets/36e16371-1a1f-48de-88ac-e7c81ef238f0" />|<img width="1136" height="430" alt="CleanShot 2026-01-29 at 12 15 58@2x" src="https://github.com/user-attachments/assets/d3b70723-9049-4d32-8795-2ca4365ccf03" />|
This will now be similar to other error responses in our v2 external API:
<img width="938" height="676" alt="CleanShot 2026-01-29 at 12 16 33@2x" src="https://github.com/user-attachments/assets/50e9271e-62d8-44e6-b887-fae5dffc4f24" />
Fixes HDX-3309
Closes HDX-3148
As discussed in HDX-3148, we will print the error path in the notification for `SourceForm` validation when the error message is "Required".
Closes HDX-2655
# Summary
This PR fixes an incorrect lucene compilation that disregarded a `NOT` in front of a binary expression.
- eg. `NOT Body:Fetching AND NOT Body:Unsupported` was rendered as `(((Body ILIKE '%Fetching%') AND NOT (Body ILIKE '%Unsupported%')))`
This was due to a missing `ast.start` check, which was present in the `leftOnlyAst` branch, but not in the `binaryAst` branch.
Allows setting a custom setting prefix on a connection. When set in HyperDX and the ClickHouse settings, the HyperDX app will set a custom setting for each query. These are recorded in the query log and can be used to identify which user issues the query.
## Testing
The commit also updates the local dev ClickHouse instance to support a custom setting prefix of `hyeprdx`. After running `make dev-up`, you should be able to edit the connection and set the the prefix to `hyperdx`.
<img width="955" height="197" alt="Screenshot 2026-01-21 at 1 23 14 PM" src="https://github.com/user-attachments/assets/607fc945-d93f-4976-9862-3118b420c077" />
After saving, just allow the app to live tail a source like logs. If you connect to the ClickHouse database, you should then be able to run
```
SELECT query, Settings
FROM system.query_log
WHERE has(mapKeys(Settings), 'hyperdx_user')
FORMAT Vertical
```
and then see a bunch of queries with the user set to your logged in user.
```
Row 46:
───────
query: SELECT Timestamp, ServiceName, SeverityText, Body, TimestampTime FROM default.otel_logs WHERE (TimestampTime >= fromUnixTimestamp64Milli(_CAST(1769022372269, 'Int64'))) AND (TimestampTime <= fromUnixTimestamp64Milli(_CAST(1769023272269, 'Int64'))) ORDER BY (TimestampTime, Timestamp) DESC LIMIT _CAST(0, 'Int32'), _CAST(200, 'Int32') FORMAT JSONCompactEachRowWithNamesAndTypes
Settings: {'use_uncompressed_cache':'0','load_balancing':'in_order','log_queries':'1','max_memory_usage':'10000000000','cancel_http_readonly_queries_on_client_close':'1','parallel_replicas_for_cluster_engines':'0','date_time_output_format':'iso','hyperdx_user':'\'dan@hyperdx.io\''}
```
Closes HDX-3264
# Summary
This PR fixes a bug that caused dashboard tiles to incorrectly have no hover state after closing the edit modal, but while still hovering over the tile. This happened intermittently, but appeared to be caused by onMouseEnter not being fired reliably when closing the modal. onMouseOver appears to fire reliably, even when onMouseEnter does not. The difference between the events is that onMouseOver will fire more frequently - whenever the mouse enters a new child element. Since the event handler is idempotent, this is not an issue. I confirmed it does not result in extra re-renders.
Changed the button variant in the ConnectionForm component to reflect the test connection state, using 'danger' for invalid states and 'secondary' for others. This improves user feedback during connection testing
Closes HDX-3266
# Summary
This PR fixes an error in dashboards that occurred when using a Dashboard filter on a non-String column.
## Demo
Such filters work now:
<img width="1074" height="523" alt="Screenshot 2026-01-23 at 3 44 19 PM" src="https://github.com/user-attachments/assets/2e026590-f4cb-44a2-8677-37a80b65886b" />
Co-authored-by: Karl Power <85935352+karl-power@users.noreply.github.com>
Closes HDX-3236
# Summary
This PR fixes an error that occurs when a metricName/metricType is set for a dashboard tile configuration, despite the queried source not being a metric source.
1. Updates in DBEditTimeChartForm prevent us from saving configurations with metricName/metricType for non metric sources
2. Updates in DBDashboardPage ensure that metricName/metricType is ignored for any saved configurations for non-metric sources.
## Demo
A new tile would be saved with a metricName/Type incorrectly when
1. Create the tile
2. Select a metric source
3. Select a metric name
4. Switch back to a non-metric source
5. Save
And the Dashboard tile would then error:
<img width="1288" height="1012" alt="Screenshot 2026-01-23 at 2 39 38 PM" src="https://github.com/user-attachments/assets/4fa4b0bf-355e-47bb-a504-cd03e0dca2d0" />
Now, the configuration is not saved with metricName/Type, and the dashboard does not error for a saved configuration that has a metricName/Type:
<img width="769" height="423" alt="Screenshot 2026-01-23 at 2 43 04 PM" src="https://github.com/user-attachments/assets/92af36aa-dd46-47b8-ae59-d0e4bfcb28af" />
Closes HDX-3245
# Summary
This PR updates the Lucene to SQL compilation process to generate conditions using `hasAllTokens` when the target column has a text index defined.
`hasAllTokens` has a couple of limitations which are solved for:
1. The `needle` argument must be no more than 64 tokens, or `hasAllTokens` will error. To support search terms with more than 64 tokens, terms are first broken up into batches of 50 tokens, each batch is passed to a separate `hasAllTokens` call. When multiple `hasAllTokens` calls are used, we also use substring matching `lower(Body) LIKE '%term with many tokens...%'`.
2. `hasAllTokens` may only be used when `enable_full_text_index = 1`. The existence of a text index does not guarantee that `enable_full_text_index = 1`, since the text index could have been created with a query that explicitly specified `SETTINGS enable_full_text_index = 1`. We cannot set this option in every query HyperDX makes, because the setting was not available prior to v25.12. To solve for this, we check the value of `enable_full_text_index` in `system.settings`, and only use `hasAllTokens` if the setting exists and is enabled.
## Testing Setup
### Enable Full Text Index
First, make sure you're running at least ClickHouse 25.12.
Then, update the ClickHouse `users.xml`'s default profile with the following (or otherwise update your user's profile):
```xml
<clickhouse>
<profiles>
<default>
...
<enable_full_text_index>1</enable_full_text_index>
</default>
</profiles>
...
<clickhouse>
```
### Add a Full Text Index
```sql
ALTER TABLE otel_logs ADD INDEX text_idx(Body)
TYPE text(tokenizer=splitByNonAlpha, preprocessor=lower(Body))
SETTINGS enable_full_text_index=1;
ALTER TABLE otel_logs MATERIALIZE INDEX text_idx;
```
## Limitations
1. We currently only support the `splitByNonAlpha` tokenizer. If the text index is created with a different tokenizer, `hasAllTokens` will not be used. If needed, this limitation can be removed in the future by implementing `tokenizeTerm`, `termContainsSeparators`, and token batching logic specific to the other tokenizers.
2. This requires the latest (Beta) version of the full text index and related setting, available in ClickHouse v25.12.
