## Summary
This PR extends Raw SQL Charts to support [macros that are supported by the Grafana ClickHouse plugin](https://github.com/grafana/clickhouse-datasource?tab=readme-ov-file#macros).
Query Params and Macros are also now included as auto-complete suggestions in the SQL Editor.
### Screenshots or video
<img width="1434" height="1080" alt="Screenshot 2026-03-16 at 12 53 03 PM" src="https://github.com/user-attachments/assets/07f753e4-28f1-43f4-8add-f123dae0b12a" />
### How to test locally or on Vercel
This can be tested in Vercel preview - just reference the supported macros in a raw SQL chart.
### References
- Linear Issue: Closes HDX-3651
- Related PRs:
Closes HDX-3580
Closes HDX-3034
## Summary
This PR enhances the support for SQL-driven tables
1. The selected date range can now be referenced in the query with query params. The available query params are outlined in a new section above the SQL input.
2. The table no longer OOMs on large result sets (it is now truncated to the first 10k results), or crashes when selecting columns that are Map or JSON type
3. The table can now be sorted client-side for sql-driven tables
4. There is now placeholder / example SQL in the input
### Screenshots or video
https://github.com/user-attachments/assets/4f39fd0a-d33e-4f8c-9e91-84143d23e293
### How to test locally or on Vercel
This feature can be tested locally or on the preview environment without any special toggles.
### References
- Linear Issue: HDX-3580
- Related PRs:
## Summary
This PR is the first step towards raw SQL-driven charts.
- It introduces updated ChartConfig types, which are now unions of `BuilderChartConfig` (which is unchanged from the current `ChartConfig` types` and `RawSqlChartConfig` types which represent sql-driven charts.
- It adds _very basic_ support for SQL-driven tables in the Chart Explorer and Dashboard pages. This is currently behind a feature toggle and enabled only in preview environments and for local development.
The changes in most of the files in this PR are either type updates or the addition of type guards to handle the new ChartConfig union type.
The DBEditTimeChartForm has been updated significantly to (a) add the Raw SQL option to the table chart editor and (b) handle conversion from internal form state (which can now include properties from either branch of the ChartConfig union) to valid SavedChartConfigs (which may only include properties from one branch).
Significant changes are in:
- packages/app/src/components/ChartEditor/types.ts
- packages/app/src/components/ChartEditor/RawSqlChartEditor.tsx
- packages/app/src/components/ChartEditor/utils.ts
- packages/app/src/components/DBEditTimeChartForm.tsx
- packages/app/src/components/DBTableChart.tsx
- packages/app/src/components/SQLEditor.tsx
- packages/app/src/hooks/useOffsetPaginatedQuery.tsx
Future PRs will add templating to the Raw SQL driven charts for date range and granularity injection; support for other chart types driven by SQL; improved placeholder, validation, and error states; and improved support in the external API and import/export.
### Screenshots or video
https://github.com/user-attachments/assets/008579cc-ef3c-496e-9899-88bbb21eaa5e
### How to test locally or on Vercel
The SQL-driven table can be tested in the preview environment or locally.
### References
- Linear Issue: HDX-3580
- Related PRs:
Closes HDX-3154
This PR adds a feature that allows the user to add settings to a source. These settings are then added to the end of every query that is rendered through the `renderChartConfig` function, along with any other chart specific settings.
See: https://clickhouse.com/docs/sql-reference/statements/select#settings-in-select-query
Most of the work was to pass the `source` or `source.querySettings` value through the code to the `renderChartConfig` calls and to update the related tests. There are also some UI changes in the `SourceForm` components.
`SQLParser.Parser` from the `node-sql-parser` throws an error when it encounters a SETTINGS clause in a sql string, so a function was added to remove that clause from any sql that is passed to the parser. It assumes that the SETTINGS clause will always be at the end of the sql string, it removes any part of the string including and after the SETTINGS clause.
https://github.com/user-attachments/assets/7ac3b852-2c86-4431-88bc-106f982343bb
Closes HDX-3094
# Summary
This PR standardizes available granularities and inferred/auto granularities throughout the app
1. A duplicate convertDateRangeToGranularityString implementation was removed.
2. 10 minute granularity is no longer auto-inferred, because it (in combination with 15 minutes) breaks the property that all granularities are multiples of smaller granularities. Since MVs are only used when the chart granularity is a multiple of the MV granularity, we want to minimize the chance that a MV is 10 minutes and the chart is 15 minutes, or vice versa. To this end, MVs only support 15 minute granularity, and not 10 minute granularity (to align with alerts). By removing the 10 minute granularity from auto granularity inference, we decrease the chance of automatically choosing a granularity that can't be used with an MV.
3. The max buckets argument was standardized to a constant (DEFAULT_AUTO_GRANULARITY_MAX_BUCKETS) with value 60. It is now an optional argument, only passed when a non-default value is required.
Instead of using type assertions to narrow a type, it's better to rely on type guards, which help avoid potential runtime errors caused by unsafe type assertions.
Currently the rule is enforced in `common-utils` pkg
Dup of https://github.com/hyperdxio/hyperdx/pull/679
Adds support for histogram `count` aggregations. This partially resolves https://github.com/hyperdxio/hyperdx/issues/1441, which should probably be split into a new ticket to only address `sum`.
As part of this, I also moved the translation functionality for histograms to a new file `histogram.ts` to avoid contributing even more bloat to `renderChartConfig`. Happy to revert this and move that stuff back into the file if that's preferred.
I also noticed by doing this that there was actually a SQL error in the snapshots for the tests--the existing quantile test was missing a trailing `,` after the time bucket if no group was provided https://github.com/hyperdxio/hyperdx/blob/main/packages/common-utils/src/__tests__/__snapshots__/renderChartConfig.test.ts.snap#L194 so centralizing like this is probably desirable to keep things consistent.
I also personally use webstorm so I added that stuff to the gitignore.
Closes HDX-3082
# Summary
This PR back-ports support for materialized views from the EE repo. Note that this feature is in **Beta**, and is subject to significant changes.
This feature is intended to support:
1. Configuring AggregatingMergeTree (or SummingMergeTree) Materialized Views which are associated with a Source
2. Automatically selecting and querying an associated materialized view when a query supports it, in Chart Explorer, Custom Dashboards, the Services Dashboard, and the Search Page Histogram.
3. A UX for understanding what materialized views are available for a source, and whether (and why) it is or is not being used for a particular visualization.
## Note to Reviewer(s)
This is a large PR, but the code has largely already been reviewed.
- For net-new files, types, components, and utility functions, the code does not differ from the EE repo
- Changes to the various services dashboard pages do not differ from the EE repo
- Changes to `useOffsetPaginatedQuery`, `useChartConfig`, and `DBEditTimeChart` differ slightly due to unrelated (to MVs) drift between this repo and the EE repo, and due to the lack of feature toggles in this repo. **This is where slightly closer review would be most valuable.**
## Demo
<details>
<summary>Demo: MV Configuration</summary>
https://github.com/user-attachments/assets/fedf3bcf-892c-4b8d-a788-7e231e23bcc3
</details>
<details>
<summary>Demo: Chart Explorer</summary>
https://github.com/user-attachments/assets/fc8d1efa-7edc-42fc-98f0-75431cc056b8
</details>
<details>
<summary>Demo: Dashboards</summary>
https://github.com/user-attachments/assets/f3cb247e-711f-4d90-95b8-cf977e94f065
</details>
## Known Limitations
This feature is in Beta due to the following known limitations, which will be addressed in subsequent PRs:
1. Visualization start and end time, when not aligned with the granularity of MVs, will result in statistics based on the MV "time buckets" which fall inside the date range. This may not align exactly with the source table data which is in the selected date range.
2. Alerts do not make use of MVs, even if the associated visualization does. Due to (1), this means that alert values may not exactly match the values shown in the associated visualization.
## Differences in OSS vs EE Support
- In OSS, there is a beta label on the MV configurations section
- In EE there are feature toggles to enable MV support, in OSS the feature is enabled for all teams, but will only run for sources with MVs configured.
## Testing
To test, a couple of MVs can be created on the default `otel_traces` table, directly in ClickHouse:
<details>
<summary>Example MVs DDL</summary>
```sql
CREATE TABLE default.metrics_rollup_1m
(
`Timestamp` DateTime,
`ServiceName` LowCardinality(String),
`SpanKind` LowCardinality(String),
`StatusCode` LowCardinality(String),
`count` SimpleAggregateFunction(sum, UInt64),
`sum__Duration` SimpleAggregateFunction(sum, UInt64),
`avg__Duration` AggregateFunction(avg, UInt64),
`quantile__Duration` AggregateFunction(quantileTDigest(0.5), UInt64),
`min__Duration` SimpleAggregateFunction(min, UInt64),
`max__Duration` SimpleAggregateFunction(max, UInt64)
)
ENGINE = AggregatingMergeTree
PARTITION BY toDate(Timestamp)
ORDER BY (Timestamp, StatusCode, SpanKind, ServiceName);
CREATE MATERIALIZED VIEW default.metrics_rollup_1m_mv TO default.metrics_rollup_1m
(
`Timestamp` DateTime,
`ServiceName` LowCardinality(String),
`SpanKind` LowCardinality(String),
`version` LowCardinality(String),
`StatusCode` LowCardinality(String),
`count` UInt64,
`sum__Duration` Int64,
`avg__Duration` AggregateFunction(avg, UInt64),
`quantile__Duration` AggregateFunction(quantileTDigest(0.5), UInt64),
`min__Duration` SimpleAggregateFunction(min, UInt64),
`max__Duration` SimpleAggregateFunction(max, UInt64)
)
AS SELECT
toStartOfMinute(Timestamp) AS Timestamp,
ServiceName,
SpanKind,
StatusCode,
count() AS count,
sum(Duration) AS sum__Duration,
avgState(Duration) AS avg__Duration,
quantileTDigestState(0.5)(Duration) AS quantile__Duration,
minSimpleState(Duration) AS min__Duration,
maxSimpleState(Duration) AS max__Duration
FROM default.otel_traces
GROUP BY
Timestamp,
ServiceName,
SpanKind,
StatusCode;
```
```sql
CREATE TABLE default.span_kind_rollup_1m
(
`Timestamp` DateTime,
`ServiceName` LowCardinality(String),
`SpanKind` LowCardinality(String),
`histogram__Duration` AggregateFunction(histogram(20), UInt64)
)
ENGINE = AggregatingMergeTree
PARTITION BY toDate(Timestamp)
ORDER BY (Timestamp, ServiceName, SpanKind);
CREATE MATERIALIZED VIEW default.span_kind_rollup_1m_mv TO default.span_kind_rollup_1m
(
`Timestamp` DateTime,
`ServiceName` LowCardinality(String),
`SpanKind` LowCardinality(String),
`histogram__Duration` AggregateFunction(histogram(20), UInt64)
)
AS SELECT
toStartOfMinute(Timestamp) AS Timestamp,
ServiceName,
SpanKind,
histogramState(20)(Duration) AS histogram__Duration
FROM default.otel_traces
GROUP BY
Timestamp,
ServiceName,
SpanKind;
```
</details>
Then you'll need to configure the materialized views in your source settings:
<details>
<summary>Source Configuration (should auto-infer when MVs are selected)</summary>
<img width="949" height="1011" alt="Screenshot 2025-12-19 at 10 26 54 AM" src="https://github.com/user-attachments/assets/fc46a1b9-de8b-4b95-a8ef-ba5fee905685" />
</details>
Closes HDX-3033
# Summary
This PR fixes three bugs in the Services Dashboard
1. When using CTEs in chart configs, as we do on the HTTP and Databases tabs, there were frequent console errors as we tried to `DESCRIBE` the CTE names, to support the materialized columns optimization. With this PR, we no longer try to DESCRIBE CTEs, by skipping the materialized column optimization for configs without a `from.databaseName`.
2. Previously, the Request Throughput chart would reload whenever switching the Request Error Rate chart from `Overall` to `By Endpoint`. This was because the `displayType` in the Request Throughput chart was based on the toggle state, despite being unrelated. Now, the displayType of the Request Throughput chart is constant, eliminating the extra refetch.
3. Previously, when switching to the Services dashboard with a non-Trace Source ID in the URL params, the Services dashboard would initially be empty, then after toggling to a Trace Source, queries would briefly be issued against the non-Trace source (they would fail and/or be cancelled a moment later). Now, non-Trace sources are filtered out so that a Trace source is chosen as the default, and non-Trace sources are not queried.
4. Previously, we were spreading the entirety of `...source` into each config, which resulted in `metricTables` being in the config under particular circumstances (HDX-3035), which in turn caused errors from renderChartConfig. This has been fixed by `pick`ing only the fields we need from source.
Closes HDX-2960
# Summary
This PR makes two fixes to improve the charts on the Database tab of the Services dashboard.
1. Previously, the charts were not rendering a series per query, since ClickHouse returns type `Nullable(String)` for the statement expression (`coalesce(nullif(SpanAttributes['db.query.text'], ''), nullif(SpanAttributes['db.statement'], '')) AS "Statement"`). Our `convertCHDataTypeToJSType()` function failed to map this to a `String` type in JS, which resulted in the column not being inferred as a group column by `inferGroupColumns()` in `formatResponseForTimeChart()`.
2. Once we started rendering a series per query, the page immediately started OOM crashing on any serious volume of data because there are too many distinct groups/queries being returned. To fix this, the query now selects only the queries with the highest values in across any time bucket. The queries do the following:
1. First, apply filters and group by query and time bucket
2. Then, `groupArray` the values and time buckets for each query
3. Select the top 60 queries by max value across all time buckets
4. `arrayJoin(zip())` to transform the results back into the row-per-group-per-time-bucket format that `formatResponseForTimeChart` expects.
(This is the same approach recently applied to the `Request Error Rate by endpoint` chart on the HTTP tab).
## Before
<img width="1453" height="791" alt="Screenshot 2025-12-03 at 10 58 31 AM" src="https://github.com/user-attachments/assets/ffa697e4-25bb-4ac6-aed2-703cc3c547bf" />
## After
<img width="1451" height="825" alt="Screenshot 2025-12-03 at 10 57 40 AM" src="https://github.com/user-attachments/assets/42e46d2a-361e-490e-8976-18edeca39e0f" />
Closes HDX-2753
Closes HDX-2755
Closes HDX-2756
# Summary
This PR adds a search function for filtering spans and logs in the trace waterfall. Because the waterfall can consist of both spans and logs, from two different sources, there is one input per source. If the correlated log source is not available (ensuring there are no logs in the waterfall) then there is only one input.
The input persists in the query parameters so that the filtered side panel can be shared, and the parameters are cleared when the side panel closes.
Currently, only Lucene is supported for searching.
This PR also adds a couple of minor improvements to the waterfall
1. There is now a count of spans and errors
2. There is now a span status in the waterfall tooltip
## Demo
https://github.com/user-attachments/assets/fb623875-5811-4f7f-9f40-c0b34de1c541
Small improvements to heatmap logic:
1. Improve the logic around filtering the outliers. Previously it was hardcoded to Duration, now it will correctly use the `Value` from the user. If the value contains an aggregate function, it will also perform a CTE to properly calculate.
1. If the outliers query fails, we show the user the query error
2. We prioritize the outlier keys in the event deltas view over inliers (this was how it was before, now it only includes inliers if no outliers are found)
3. Ensure the autocomplete suggestions are displayed (there was a zindex issue)
moves them into a core folder, this allows us to easily track when core files are modified via path
no changeset because no version bump required
fixes HDX-2589
2025-10-30 15:16:33 +00:00
Renamed from packages/common-utils/src/renderChartConfig.ts (Browse further)
