Instead of using type assertions to narrow a type, it's better to rely on type guards, which help avoid potential runtime errors caused by unsafe type assertions.
Currently the rule is enforced in `common-utils` pkg
Dup of https://github.com/hyperdxio/hyperdx/pull/679
Adds support for histogram `count` aggregations. This partially resolves https://github.com/hyperdxio/hyperdx/issues/1441, which should probably be split into a new ticket to only address `sum`.
As part of this, I also moved the translation functionality for histograms to a new file `histogram.ts` to avoid contributing even more bloat to `renderChartConfig`. Happy to revert this and move that stuff back into the file if that's preferred.
I also noticed by doing this that there was actually a SQL error in the snapshots for the tests--the existing quantile test was missing a trailing `,` after the time bucket if no group was provided https://github.com/hyperdxio/hyperdx/blob/main/packages/common-utils/src/__tests__/__snapshots__/renderChartConfig.test.ts.snap#L194 so centralizing like this is probably desirable to keep things consistent.
I also personally use webstorm so I added that stuff to the gitignore.
Closes HDX-3082
# Summary
This PR back-ports support for materialized views from the EE repo. Note that this feature is in **Beta**, and is subject to significant changes.
This feature is intended to support:
1. Configuring AggregatingMergeTree (or SummingMergeTree) Materialized Views which are associated with a Source
2. Automatically selecting and querying an associated materialized view when a query supports it, in Chart Explorer, Custom Dashboards, the Services Dashboard, and the Search Page Histogram.
3. A UX for understanding what materialized views are available for a source, and whether (and why) it is or is not being used for a particular visualization.
## Note to Reviewer(s)
This is a large PR, but the code has largely already been reviewed.
- For net-new files, types, components, and utility functions, the code does not differ from the EE repo
- Changes to the various services dashboard pages do not differ from the EE repo
- Changes to `useOffsetPaginatedQuery`, `useChartConfig`, and `DBEditTimeChart` differ slightly due to unrelated (to MVs) drift between this repo and the EE repo, and due to the lack of feature toggles in this repo. **This is where slightly closer review would be most valuable.**
## Demo
<details>
<summary>Demo: MV Configuration</summary>
https://github.com/user-attachments/assets/fedf3bcf-892c-4b8d-a788-7e231e23bcc3
</details>
<details>
<summary>Demo: Chart Explorer</summary>
https://github.com/user-attachments/assets/fc8d1efa-7edc-42fc-98f0-75431cc056b8
</details>
<details>
<summary>Demo: Dashboards</summary>
https://github.com/user-attachments/assets/f3cb247e-711f-4d90-95b8-cf977e94f065
</details>
## Known Limitations
This feature is in Beta due to the following known limitations, which will be addressed in subsequent PRs:
1. Visualization start and end time, when not aligned with the granularity of MVs, will result in statistics based on the MV "time buckets" which fall inside the date range. This may not align exactly with the source table data which is in the selected date range.
2. Alerts do not make use of MVs, even if the associated visualization does. Due to (1), this means that alert values may not exactly match the values shown in the associated visualization.
## Differences in OSS vs EE Support
- In OSS, there is a beta label on the MV configurations section
- In EE there are feature toggles to enable MV support, in OSS the feature is enabled for all teams, but will only run for sources with MVs configured.
## Testing
To test, a couple of MVs can be created on the default `otel_traces` table, directly in ClickHouse:
<details>
<summary>Example MVs DDL</summary>
```sql
CREATE TABLE default.metrics_rollup_1m
(
`Timestamp` DateTime,
`ServiceName` LowCardinality(String),
`SpanKind` LowCardinality(String),
`StatusCode` LowCardinality(String),
`count` SimpleAggregateFunction(sum, UInt64),
`sum__Duration` SimpleAggregateFunction(sum, UInt64),
`avg__Duration` AggregateFunction(avg, UInt64),
`quantile__Duration` AggregateFunction(quantileTDigest(0.5), UInt64),
`min__Duration` SimpleAggregateFunction(min, UInt64),
`max__Duration` SimpleAggregateFunction(max, UInt64)
)
ENGINE = AggregatingMergeTree
PARTITION BY toDate(Timestamp)
ORDER BY (Timestamp, StatusCode, SpanKind, ServiceName);
CREATE MATERIALIZED VIEW default.metrics_rollup_1m_mv TO default.metrics_rollup_1m
(
`Timestamp` DateTime,
`ServiceName` LowCardinality(String),
`SpanKind` LowCardinality(String),
`version` LowCardinality(String),
`StatusCode` LowCardinality(String),
`count` UInt64,
`sum__Duration` Int64,
`avg__Duration` AggregateFunction(avg, UInt64),
`quantile__Duration` AggregateFunction(quantileTDigest(0.5), UInt64),
`min__Duration` SimpleAggregateFunction(min, UInt64),
`max__Duration` SimpleAggregateFunction(max, UInt64)
)
AS SELECT
toStartOfMinute(Timestamp) AS Timestamp,
ServiceName,
SpanKind,
StatusCode,
count() AS count,
sum(Duration) AS sum__Duration,
avgState(Duration) AS avg__Duration,
quantileTDigestState(0.5)(Duration) AS quantile__Duration,
minSimpleState(Duration) AS min__Duration,
maxSimpleState(Duration) AS max__Duration
FROM default.otel_traces
GROUP BY
Timestamp,
ServiceName,
SpanKind,
StatusCode;
```
```sql
CREATE TABLE default.span_kind_rollup_1m
(
`Timestamp` DateTime,
`ServiceName` LowCardinality(String),
`SpanKind` LowCardinality(String),
`histogram__Duration` AggregateFunction(histogram(20), UInt64)
)
ENGINE = AggregatingMergeTree
PARTITION BY toDate(Timestamp)
ORDER BY (Timestamp, ServiceName, SpanKind);
CREATE MATERIALIZED VIEW default.span_kind_rollup_1m_mv TO default.span_kind_rollup_1m
(
`Timestamp` DateTime,
`ServiceName` LowCardinality(String),
`SpanKind` LowCardinality(String),
`histogram__Duration` AggregateFunction(histogram(20), UInt64)
)
AS SELECT
toStartOfMinute(Timestamp) AS Timestamp,
ServiceName,
SpanKind,
histogramState(20)(Duration) AS histogram__Duration
FROM default.otel_traces
GROUP BY
Timestamp,
ServiceName,
SpanKind;
```
</details>
Then you'll need to configure the materialized views in your source settings:
<details>
<summary>Source Configuration (should auto-infer when MVs are selected)</summary>
<img width="949" height="1011" alt="Screenshot 2025-12-19 at 10 26 54 AM" src="https://github.com/user-attachments/assets/fc46a1b9-de8b-4b95-a8ef-ba5fee905685" />
</details>
Closes HDX-3033
# Summary
This PR fixes three bugs in the Services Dashboard
1. When using CTEs in chart configs, as we do on the HTTP and Databases tabs, there were frequent console errors as we tried to `DESCRIBE` the CTE names, to support the materialized columns optimization. With this PR, we no longer try to DESCRIBE CTEs, by skipping the materialized column optimization for configs without a `from.databaseName`.
2. Previously, the Request Throughput chart would reload whenever switching the Request Error Rate chart from `Overall` to `By Endpoint`. This was because the `displayType` in the Request Throughput chart was based on the toggle state, despite being unrelated. Now, the displayType of the Request Throughput chart is constant, eliminating the extra refetch.
3. Previously, when switching to the Services dashboard with a non-Trace Source ID in the URL params, the Services dashboard would initially be empty, then after toggling to a Trace Source, queries would briefly be issued against the non-Trace source (they would fail and/or be cancelled a moment later). Now, non-Trace sources are filtered out so that a Trace source is chosen as the default, and non-Trace sources are not queried.
4. Previously, we were spreading the entirety of `...source` into each config, which resulted in `metricTables` being in the config under particular circumstances (HDX-3035), which in turn caused errors from renderChartConfig. This has been fixed by `pick`ing only the fields we need from source.
Closes HDX-2960
# Summary
This PR makes two fixes to improve the charts on the Database tab of the Services dashboard.
1. Previously, the charts were not rendering a series per query, since ClickHouse returns type `Nullable(String)` for the statement expression (`coalesce(nullif(SpanAttributes['db.query.text'], ''), nullif(SpanAttributes['db.statement'], '')) AS "Statement"`). Our `convertCHDataTypeToJSType()` function failed to map this to a `String` type in JS, which resulted in the column not being inferred as a group column by `inferGroupColumns()` in `formatResponseForTimeChart()`.
2. Once we started rendering a series per query, the page immediately started OOM crashing on any serious volume of data because there are too many distinct groups/queries being returned. To fix this, the query now selects only the queries with the highest values in across any time bucket. The queries do the following:
1. First, apply filters and group by query and time bucket
2. Then, `groupArray` the values and time buckets for each query
3. Select the top 60 queries by max value across all time buckets
4. `arrayJoin(zip())` to transform the results back into the row-per-group-per-time-bucket format that `formatResponseForTimeChart` expects.
(This is the same approach recently applied to the `Request Error Rate by endpoint` chart on the HTTP tab).
## Before
<img width="1453" height="791" alt="Screenshot 2025-12-03 at 10 58 31 AM" src="https://github.com/user-attachments/assets/ffa697e4-25bb-4ac6-aed2-703cc3c547bf" />
## After
<img width="1451" height="825" alt="Screenshot 2025-12-03 at 10 57 40 AM" src="https://github.com/user-attachments/assets/42e46d2a-361e-490e-8976-18edeca39e0f" />
Closes HDX-2753
Closes HDX-2755
Closes HDX-2756
# Summary
This PR adds a search function for filtering spans and logs in the trace waterfall. Because the waterfall can consist of both spans and logs, from two different sources, there is one input per source. If the correlated log source is not available (ensuring there are no logs in the waterfall) then there is only one input.
The input persists in the query parameters so that the filtered side panel can be shared, and the parameters are cleared when the side panel closes.
Currently, only Lucene is supported for searching.
This PR also adds a couple of minor improvements to the waterfall
1. There is now a count of spans and errors
2. There is now a span status in the waterfall tooltip
## Demo
https://github.com/user-attachments/assets/fb623875-5811-4f7f-9f40-c0b34de1c541
Small improvements to heatmap logic:
1. Improve the logic around filtering the outliers. Previously it was hardcoded to Duration, now it will correctly use the `Value` from the user. If the value contains an aggregate function, it will also perform a CTE to properly calculate.
1. If the outliers query fails, we show the user the query error
2. We prioritize the outlier keys in the event deltas view over inliers (this was how it was before, now it only includes inliers if no outliers are found)
3. Ensure the autocomplete suggestions are displayed (there was a zindex issue)
moves them into a core folder, this allows us to easily track when core files are modified via path
no changeset because no version bump required
fixes HDX-2589
2025-10-30 15:16:33 +00:00
Renamed from packages/common-utils/src/renderChartConfig.ts (Browse further)
