Closes HDX-3067
Closes#1331Closes#1212Closes#1468
# Summary
This PR makes a number of improvements around the way we handle date ranges and granularities, in an effort to minimize discrepancies between aggregate values queried from original data and aggregate values queried from materialized views.
1. Date ranges for Line and Bar chart queries are now (by default) auto-aligned to the chart's granularity. **This is not limited to materialized view queries.** Since the chart granularity is a multiple of the MV granularity, this ensures that the date range is aligned to the MV granularity as well. This also address a number of related issues that point out 0-values or low-values in the first or last data points. This PR also includes an option to disable this behavior for charts in Chart Explorer or Dashboard Tiles.
2. All materialized view queries and all time chart queries are now end-exclusive, to avoid selecting the entirety of the next "time bucket" from the materialized view when the date range is aligned with the materialized view granularity
3. Materialized views are only used for a query with a granularity if the chart query granularity is a multiple of the MV granularity. Previously, we'd use the MV as long as the chart query granularity was at least as large as the MV granularity, but this could cause unequal distributions of data across time buckets. Nearly all available granularities are multiples of all smaller available granularities - so this should only impact queries with granularity 15 minutes with MVs with granularity 10 minutes. 10m granularity support is being removed in #1551
## Demo
<details>
<summary>Show Complete Intervals Option</summary>
https://github.com/user-attachments/assets/4b903adb-4edf-4481-93d6-2a0c42589a37
</details>
Closes HDX-3089
Closes HDX-3083
# Summary
This PR ensures that the Generated SQL and Materialized View Explanations are generated for the same config that is queried.
Previously, this was not the case because child components (eg. DBTableChart, DBTimeChart, DBNumberChart) would transform the config passed down by DBEditTimeChart or Dashboard.Tile. Now, the same transformations are applied at the parent component.
## Demo
<details>
<summary>Table Tile MV Indicator now correctly ignores granularity</summary>
Note that the time chart tiles cannot use MVs because the granularity is too small, but the table tile can use the MV:
<img width="1495" height="884" alt="Screenshot 2025-12-29 at 9 00 16 AM" src="https://github.com/user-attachments/assets/324d3511-6b86-4bd8-87b0-824b4586bb9b" />
</details>
<details>
<summary>Number and Table Chart generated SQL is correct</summary>
Number Chart has no granularity, group, or order by:
<img width="1493" height="830" alt="Screenshot 2025-12-29 at 9 14 08 AM" src="https://github.com/user-attachments/assets/34202de1-1d55-4675-9883-e15418f2bb2f" />
Table Chart has implicit group by, limit, and order by applied:
<img width="1495" height="837" alt="Screenshot 2025-12-29 at 9 14 00 AM" src="https://github.com/user-attachments/assets/2aa17106-527a-4e34-a76a-a6c1767a1e86" />
</details>
Closes HDX-3082
# Summary
This PR back-ports support for materialized views from the EE repo. Note that this feature is in **Beta**, and is subject to significant changes.
This feature is intended to support:
1. Configuring AggregatingMergeTree (or SummingMergeTree) Materialized Views which are associated with a Source
2. Automatically selecting and querying an associated materialized view when a query supports it, in Chart Explorer, Custom Dashboards, the Services Dashboard, and the Search Page Histogram.
3. A UX for understanding what materialized views are available for a source, and whether (and why) it is or is not being used for a particular visualization.
## Note to Reviewer(s)
This is a large PR, but the code has largely already been reviewed.
- For net-new files, types, components, and utility functions, the code does not differ from the EE repo
- Changes to the various services dashboard pages do not differ from the EE repo
- Changes to `useOffsetPaginatedQuery`, `useChartConfig`, and `DBEditTimeChart` differ slightly due to unrelated (to MVs) drift between this repo and the EE repo, and due to the lack of feature toggles in this repo. **This is where slightly closer review would be most valuable.**
## Demo
<details>
<summary>Demo: MV Configuration</summary>
https://github.com/user-attachments/assets/fedf3bcf-892c-4b8d-a788-7e231e23bcc3
</details>
<details>
<summary>Demo: Chart Explorer</summary>
https://github.com/user-attachments/assets/fc8d1efa-7edc-42fc-98f0-75431cc056b8
</details>
<details>
<summary>Demo: Dashboards</summary>
https://github.com/user-attachments/assets/f3cb247e-711f-4d90-95b8-cf977e94f065
</details>
## Known Limitations
This feature is in Beta due to the following known limitations, which will be addressed in subsequent PRs:
1. Visualization start and end time, when not aligned with the granularity of MVs, will result in statistics based on the MV "time buckets" which fall inside the date range. This may not align exactly with the source table data which is in the selected date range.
2. Alerts do not make use of MVs, even if the associated visualization does. Due to (1), this means that alert values may not exactly match the values shown in the associated visualization.
## Differences in OSS vs EE Support
- In OSS, there is a beta label on the MV configurations section
- In EE there are feature toggles to enable MV support, in OSS the feature is enabled for all teams, but will only run for sources with MVs configured.
## Testing
To test, a couple of MVs can be created on the default `otel_traces` table, directly in ClickHouse:
<details>
<summary>Example MVs DDL</summary>
```sql
CREATE TABLE default.metrics_rollup_1m
(
`Timestamp` DateTime,
`ServiceName` LowCardinality(String),
`SpanKind` LowCardinality(String),
`StatusCode` LowCardinality(String),
`count` SimpleAggregateFunction(sum, UInt64),
`sum__Duration` SimpleAggregateFunction(sum, UInt64),
`avg__Duration` AggregateFunction(avg, UInt64),
`quantile__Duration` AggregateFunction(quantileTDigest(0.5), UInt64),
`min__Duration` SimpleAggregateFunction(min, UInt64),
`max__Duration` SimpleAggregateFunction(max, UInt64)
)
ENGINE = AggregatingMergeTree
PARTITION BY toDate(Timestamp)
ORDER BY (Timestamp, StatusCode, SpanKind, ServiceName);
CREATE MATERIALIZED VIEW default.metrics_rollup_1m_mv TO default.metrics_rollup_1m
(
`Timestamp` DateTime,
`ServiceName` LowCardinality(String),
`SpanKind` LowCardinality(String),
`version` LowCardinality(String),
`StatusCode` LowCardinality(String),
`count` UInt64,
`sum__Duration` Int64,
`avg__Duration` AggregateFunction(avg, UInt64),
`quantile__Duration` AggregateFunction(quantileTDigest(0.5), UInt64),
`min__Duration` SimpleAggregateFunction(min, UInt64),
`max__Duration` SimpleAggregateFunction(max, UInt64)
)
AS SELECT
toStartOfMinute(Timestamp) AS Timestamp,
ServiceName,
SpanKind,
StatusCode,
count() AS count,
sum(Duration) AS sum__Duration,
avgState(Duration) AS avg__Duration,
quantileTDigestState(0.5)(Duration) AS quantile__Duration,
minSimpleState(Duration) AS min__Duration,
maxSimpleState(Duration) AS max__Duration
FROM default.otel_traces
GROUP BY
Timestamp,
ServiceName,
SpanKind,
StatusCode;
```
```sql
CREATE TABLE default.span_kind_rollup_1m
(
`Timestamp` DateTime,
`ServiceName` LowCardinality(String),
`SpanKind` LowCardinality(String),
`histogram__Duration` AggregateFunction(histogram(20), UInt64)
)
ENGINE = AggregatingMergeTree
PARTITION BY toDate(Timestamp)
ORDER BY (Timestamp, ServiceName, SpanKind);
CREATE MATERIALIZED VIEW default.span_kind_rollup_1m_mv TO default.span_kind_rollup_1m
(
`Timestamp` DateTime,
`ServiceName` LowCardinality(String),
`SpanKind` LowCardinality(String),
`histogram__Duration` AggregateFunction(histogram(20), UInt64)
)
AS SELECT
toStartOfMinute(Timestamp) AS Timestamp,
ServiceName,
SpanKind,
histogramState(20)(Duration) AS histogram__Duration
FROM default.otel_traces
GROUP BY
Timestamp,
ServiceName,
SpanKind;
```
</details>
Then you'll need to configure the materialized views in your source settings:
<details>
<summary>Source Configuration (should auto-infer when MVs are selected)</summary>
<img width="949" height="1011" alt="Screenshot 2025-12-19 at 10 26 54 AM" src="https://github.com/user-attachments/assets/fc46a1b9-de8b-4b95-a8ef-ba5fee905685" />
</details>
Improves the Chart Explorer page to only run the sample query when the accordion is open and visible. Also changed default to closed since it's below the fold.
@pulpdrew assigned to you as you originally observed this issue
Demo:
https://github.com/user-attachments/assets/6108323a-767f-4e9f-88cf-4b9e2de9def1
Fixes HDX-2895
This PR removes bootstrap-icons entirely from the app. It also adds an eslint plugin to detect uses and throw an error, this will help in the immediate short term with PRs in flight and merging downstream.
Fixes HDX-3050
Closes HDX-2845
# Summary
This PR adds support for specifying a persistent Order By in table charts. Previously, the user could sort by clicking a column, but this was not persisted to the saved chart config. Now, we show an input that allows the user to specify an ordering other than the default, and this order is persisted in the saved chart config.
## Demo
https://github.com/user-attachments/assets/960642fd-9749-4e54-9b84-ab82eb5af3d8
This PR adds the ability to click on tables or graphs across the app, and be able to deep link into the search page to view the correlated search events.
https://github.com/user-attachments/assets/e5c7c2f2-9b59-42de-bb73-2fa4cb5d09a2
Things to try:
1. **Attributable Functions** (min, max, average, p95, etc) can all reliably give us a y value back to enable filtering directly to the y-values value
2. **Non-attributable Functions** (count, sum, etc) can’t rely on the y value since it depends on the group by logic. As such, these will simply allow you to search by the service without passing the y value for deeper analysis.
3. **Multiple Group Bys** (ex `ServiceName, SpanName`)
4. **Where Clause** (SQL or Lucene)
5. Do a **general test** across the app to ensure everywhere that uses charting behaves the same or better than before.
Some things that don’t work:
1. **Aliases in the group by** - we are hitting limitations with `node-sql-parser` that causes the aliasMap to not generate correctly when using aggregate functions in most cases. We may need to either fix these issues or find an alternative solution.
2. **Expressions in Group By** - If you have expressions in the group by, the filtering will not work based on the current implementation, it does not factor in the result of the expression in the filter value.
3. **Filters are not reflected in the UI on Search** - When you deeplink, the filters in the sidebar do not reflect what is actually being queried. This is because the filter UI logic uses `IN` but the deeplinking uses `=` to achieve the best level of precision (and avoid false positives). As such, this is something that could be improved in the future.
Fixes HDX-2844, HDX-2922
Closes HDX-2777
# Summary
This PR adds a toggle that enables showing "previous period" data on line charts, overlayed with the current period data.
1. The "previous period" is a date range of the same length as the selected "current" date range, immediately prior to the current date range.
2. This feature is only enabled for line charts, bar charts are not enabled when this option is toggled on.
**This PR is organized into a number of commits which may be easier to review one at a time.**
## Followup work
- Improve layout of the DBEditTimeChartForm, pending design review
## Demo
https://github.com/user-attachments/assets/76b220da-810e-4280-8fb3-fa20a9919685
moves them into a core folder, this allows us to easily track when core files are modified via path
no changeset because no version bump required
fixes HDX-2589
Across the app, we are inconsistent with when we can open the sidebar and expand functionality. This is because the sidebar and logic was managed by the parent component.
Additionally, the expand logic was set to assume a certain structure that some places in the application could not support (ex clickhouse dashboard doesn't have a 'source').
As a result, I have created the `DBSqlRowTableWithSideBar` component which will manage a lot of the common use cases for us. This PR introduces that new component, and updates all references (that could be easily upgraded) to use the new component when applciable.
The result: a lot less duplicate code (see # of lines removed) and the ability to more easily maintain the components down the road.
This PR also fixes several bugs I found as I tested these flows, especially around sidebars opening subpanels.
Fixes: HDX-2341
Fixes: HDX-2296
When a user clicks the ">" icon, the row is expanded and live tail is paused.
<img width="1882" height="451" alt="image" src="https://github.com/user-attachments/assets/4b4f12e0-df80-49b0-b917-455b213b4f3a" />
The user is able to interact with with the inline view as well
<img width="1823" height="334" alt="image" src="https://github.com/user-attachments/assets/e0c1acfc-7523-470d-b385-343c6e84ad9f" />
The user can click the expand button in the top right to open the sidebar
<img width="258" height="180" alt="image" src="https://github.com/user-attachments/assets/40b40c59-29a8-44fc-a310-6b55121029b4" />
The tab last chosen by the user persists in local storage
Clicking on an item in the table results in the sidebar opening as it does today.
These changes also:
* Disable the chevron in places where the sidebar is not hooked up today
* Disables interactivity of Event Tags if it doesn't have the proper events passed into it as well.
This PR fixes: HDX-1641
The Linear issue only mentions:
> The Run button on the Markdown form doesn't do anything
However, in **Charts Explorer → Markdown**, the time picker also wasn’t working. I’ve removed it as well.
<img width="2681" height="2048" alt="image" src="https://github.com/user-attachments/assets/6f6b7fc5-544f-43e2-ab9d-05894f83e8d6" />
We now store `createdBy` in the `Alert` document when the alert
is created. This new field references the `User` document of the
currently logged in user. The property is optional to support
existing alerts in the database.
This information is also displayed in various places where alert
details are displayed. Older documents without the new field should
display nothing, resulting in the existing UX.
1. Created separate tsconfig files for tests and prod code, including the test files in the test config.
2. Fixed linting error in DBEditTimeChartForm found while running local linting.
3. Excluded coverage CSS from the style lint task.
4. Updated app Dockerfile with test configuration and ran locally to ensure it was working.
Ref: HDX-1502
React query's useQuery does a hash on the object you provide, including each data member. The previous had shallow copied form, which retained references to form elements, so select[0].aggCondition would change when you type in the search input, which in turn changes the hash of the queryKey in useQueriedChartConfig. Deep copying resolves the issue.
Ref: HDX-1392
<img width="1310" alt="Screenshot 2025-02-25 at 3 43 11 PM" src="https://github.com/user-attachments/assets/38c98bc2-2ff2-412c-b26d-4ed9952439f2" />
Co-authored-by: Mike Shi <2781687+MikeShi42@users.noreply.github.com>
Co-authored-by: Dan Hable <418679+dhable@users.noreply.github.com>
Co-authored-by: Tom Alexander <3245235+teeohhem@users.noreply.github.com>
