moves them into a core folder, this allows us to easily track when core files are modified via path
no changeset because no version bump required
fixes HDX-2589
Closes HDX-2576
Closes HDX-2491
# Summary
It is a common optimization to have a primary key like `toStartOfDay(Timestamp), ..., Timestamp`. This PR improves the experience when using such a primary key in the following ways:
1. HyperDX will now automatically filter on both `toStartOfDay(Timestamp)` and `Timestamp` in this case, instead of just `Timestamp`. This improves performance by better utilizing the primary index. Previously, this required a manual change to the source's Timestamp Column setting.
2. HyperDX now applies the same `toStartOfX` function to the right-hand-side of timestamp comparisons. So when filtering using an expression like `toStartOfDay(Timestamp)`, the generated SQL will have the condition `toStartOfDay(Timestamp) >= toStartOfDay(<selected start time>) AND toStartOfDay(Timestamp) <= toStartOfDay(<selected end time>)`. This resolves an issue where some data would be incorrectly filtered out when filtering on such timestamp expressions (such as time ranges less than 1 minute).
With this change, teams should no longer need to have multiple columns in their source timestamp column configuration. However, if they do, they will now have correct filtering.
## Testing
### Testing the fix
The part of this PR that fixes time filtering can be tested with the default logs table schema. Simply set the Timestamp Column source setting to `TimestampTime, toStartOfMinute(TimestampTime)`. Then, in the logs search, filter for a timespan < 1 minute.
<details>
<summary>Without the fix, you should see no logs, since they're incorrectly filtered out by the toStartOfMinute(TimestampTime) filter</summary>
https://github.com/user-attachments/assets/915d3922-55f8-4742-b686-5090cdecef60
</details>
<details>
<summary>With the fix, you should see logs in the selected time range</summary>
https://github.com/user-attachments/assets/f75648e4-3f48-47b0-949f-2409ce075a75
</details>
### Testing the optimization
The optimization part of this change is that when a table has a primary key like `toStartOfMinute(TimestampTime), ..., TimestampTime` and the Timestamp Column for the source is just `Timestamp`, the query will automatically filter by both `toStartOfMinute(TimestampTime)` and `TimestampTime`.
To test this, you'll need to create a table with such a primary key, then create a source based on that table. Optionally, you could copy data from the default `otel_logs` table into the new table (`INSERT INTO default.otel_logs_toStartOfMinute_Key SELECT * FROM default.otel_logs`).
<details>
<summary>DDL for log table with optimized key</summary>
```sql
CREATE TABLE default.otel_logs_toStartOfMinute_Key
(
`Timestamp` DateTime64(9) CODEC(Delta(8), ZSTD(1)),
`TimestampTime` DateTime DEFAULT toDateTime(Timestamp),
`TraceId` String CODEC(ZSTD(1)),
`SpanId` String CODEC(ZSTD(1)),
`TraceFlags` UInt8,
`SeverityText` LowCardinality(String) CODEC(ZSTD(1)),
`SeverityNumber` UInt8,
`ServiceName` LowCardinality(String) CODEC(ZSTD(1)),
`Body` String CODEC(ZSTD(1)),
`ResourceSchemaUrl` LowCardinality(String) CODEC(ZSTD(1)),
`ResourceAttributes` Map(LowCardinality(String), String) CODEC(ZSTD(1)),
`ScopeSchemaUrl` LowCardinality(String) CODEC(ZSTD(1)),
`ScopeName` String CODEC(ZSTD(1)),
`ScopeVersion` LowCardinality(String) CODEC(ZSTD(1)),
`ScopeAttributes` Map(LowCardinality(String), String) CODEC(ZSTD(1)),
`LogAttributes` Map(LowCardinality(String), String) CODEC(ZSTD(1)),
`__hdx_materialized_k8s.pod.name` String MATERIALIZED ResourceAttributes['k8s.pod.name'] CODEC(ZSTD(1)),
INDEX idx_trace_id TraceId TYPE bloom_filter(0.001) GRANULARITY 1,
INDEX idx_res_attr_key mapKeys(ResourceAttributes) TYPE bloom_filter(0.01) GRANULARITY 1,
INDEX idx_res_attr_value mapValues(ResourceAttributes) TYPE bloom_filter(0.01) GRANULARITY 1,
INDEX idx_scope_attr_key mapKeys(ScopeAttributes) TYPE bloom_filter(0.01) GRANULARITY 1,
INDEX idx_scope_attr_value mapValues(ScopeAttributes) TYPE bloom_filter(0.01) GRANULARITY 1,
INDEX idx_log_attr_key mapKeys(LogAttributes) TYPE bloom_filter(0.01) GRANULARITY 1,
INDEX idx_log_attr_value mapValues(LogAttributes) TYPE bloom_filter(0.01) GRANULARITY 1,
INDEX idx_body Body TYPE tokenbf_v1(32768, 3, 0) GRANULARITY 8,
INDEX idx_lower_body lower(Body) TYPE tokenbf_v1(32768, 3, 0) GRANULARITY 8
)
ENGINE = SharedMergeTree('/clickhouse/tables/{uuid}/{shard}', '{replica}')
PARTITION BY toDate(TimestampTime)
PRIMARY KEY (toStartOfMinute(TimestampTime), ServiceName, TimestampTime)
ORDER BY (toStartOfMinute(TimestampTime), ServiceName, TimestampTime, Timestamp)
TTL TimestampTime + toIntervalDay(90)
SETTINGS index_granularity = 8192, ttl_only_drop_parts = 1
```
</details>
Once you have that source, you can inspect the queries generated for that source. Whenever a date range filter is selected, the query should have a `WHERE` predicate that filters on both `TimestampTime` and `toStartOfMinute(TimestampTime)`, despite `toStartOfMinute(TimestampTime)` not being included in the Timestamp Column of the source's configuration.
# Summary
Closes HDX-2310
Closes HDX-2616
This PR implements chunking of chart queries to improve performance of charts on large data sets and long time ranges. Recent data is loaded first, then older data is loaded one-chunk-at-a-time until the full chart date range has been queried.
https://github.com/user-attachments/assets/83333041-9e41-438a-9763-d6f6c32a0576
## Performance Impacts
### Expectations
This change is intended to improve performance in a few ways:
1. Queries over long time ranges are now much less likely to time out, since the range is chunked into several smaller queries
2. Average memory usage should decrease, since the total result size and number of rows being read are smaller
3. _Perceived_ latency of queries over long date ranges is likely to decrease, because users will start seeing charts render (more recent) data as soon as the first chunk is queried, instead of after the entire date range has been queried. **However**, _total_ latency to display results for the entire date range is likely to increase, due to additional round-trip network latency being added for each additional chunk.
### Measured Results
Overall, the results match the expectations outlined above.
- Total latency changed between ~-4% and ~25%
- Average memory usage decreased by between 18% and 80%
<details>
<summary>Scenarios and data</summary>
In each of the following tests:
1. Queries were run 5 times before starting to measure, to ensure data is filesystem cached.
2. Queries were then run 3 times. The results shown are the median result from the 3 runs.
#### Scenario: Log Search Histogram in Staging V2, 2 Day Range, No Filter
| | Total Latency | Memory Usage (Avg) | Memory Usage (Max) | Chunk Count |
|---|---|---|---|---|
| Original | 5.36 | 409.23 MiB | 409.23 MiB | 1 |
| Chunked | 5.14 | 83.06 MiB | 232.69 MiB | 4 |
#### Scenario: Log Search Histogram in Staging V2, 14 Day Range, No Filter
| | Total Latency | Memory Usage (Avg) | Memory Usage (Max) | Chunk Count |
|---|---|---|---|---|
| Original | 26.56 | 383.63 MiB | 383.63 MiB | 1 |
| Chunked | 33.08 | 130.00 MiB | 241.21 MiB | 16 |
#### Scenario: Chart Explorer Line Chart with p90 and p99 trace durations, Staging V2 Traces, Filtering for "GET" spans, 7 Day range
| | Total Latency | Memory Usage (Avg) | Memory Usage (Max) | Chunk Count |
|---|---|---|---|---|
| Original | 2.79 | 346.12 MiB | 346.12 MiB | 1 |
| Chunked | 3.26 | 283.00 MiB | 401.38 MiB | 9 |
</details>
## Implementation Notes
<details>
<summary>When is chunking used?</summary>
Chunking is used when all of the following are true:
1. `granularity` and `timestampValueExpression` are defined in the config. This ensures that the query is already being bucketed. Without bucketing, chunking would break aggregation queries, since groups can span multiple chunks.
4. `dateRange` is defined in the config. Without a date range, we'd need an unbounded set of chunks or the start and end chunks would have to be unbounded at their start and end, respectively.
5. The config is not a metrics query. Metrics queries have complex logic which we want to avoid breaking with the initial delivery of this feature.
6. The consumer of `useQueriedChartConfig` does not pass the `disableQueryChunking: true` option. This option is provided to disable chunking when necessary.
</details>
<details>
<summary>How are time windows chosen?</summary>
1. First, generate the windows as they are generated for the existing search chunking feature (eg. 6 hours back, 6 hours back, 12 hours back, 24 hours back...)
4. Then, the start and end of each window is aligned to the start of a time bucket that depends on the "granularity" of the chart.
7. The first and last windows are shortened or extended so that the combined date range of all of the windows matches the start and end of the original config.
</details>
<details>
<summary>Which order are the chunks queried in?</summary>
Chunks are queried sequentially, most-recent first, due to the expectation that more recent data is typically more important to the user. Unlike with `useOffsetPaginatedSearch`, we are not paginating the data beyond the chunks, and all data is typically displayed together, so there is no need to support "ascending" order.
</details>
<details>
<summary>Does this improve client-side caching behavior?</summary>
One theoretical way in which query chunking could improve performance to enable client-side caching of individual chunks, which could then be re-used if the same query is run over a longer time range.
Unfortunately, using streamedQuery, react-query stores the entire time range as one item in the cache, so it does not re-use individual chunks or "pages" from another query.
We could accomplish this improvement by using useQueries instead of streamedQuery or useInfiniteQuery. In that case, we'd treat each chunk as its own query. This would require a number of changes:
1. Our query key would have to include the chunk's window duration
2. We'd need some hacky way of making the useQueries requests fire in sequence. This can be done using `enabled` but requires some additional state to figure out whether the previous query is done.
5. We'd need to emulate the return value of a useQuery using the useQueries result, or update consumers.
</details>
Now that the app has some complex queries that leverage CTEs, metrics for example, it's common for the logic in this optimization to throw an exception. When that happens, the query rendering logic continues without a problem but generates a noisy line in the console log. We can just remove this log message to clean up the debugging experience.
Ref: HDX-1763
* Utilizes renderChartConfig and CH client to query for chart data
* Implements API input schema
* Adds lots of tests
Testing Notes:
* To use swagger, go to localhost:8000/api/v2/docs
* Authorize using your access key found in localhost:8000/me
* Under the charts route, click "Try it out"
* Use example payload:
*
```
{
"startTime": <insert valid timestamp ms>,
"endTime": <insert valid timestamp ms>,
"granularity": "1h",
"series": [
{
"sourceId": "<insert valid sourceid>",
"aggFn": "count",
"where": "SeverityText:error",
"groupBy": []
}
]
}
```
It was easiest for me to go to the UI, create a new chart and grab the sourceid and start/end timestamps from the URL, plug it in and profit.
Note: It was apparent to me that we will need to provide at least GET support for sources, otherwise that ID is not easily obtained.
Ref: HDX-1651
Ref: HDX-1631
1. use temp centralized `queryChartConfig` to handle multi-series chart (metrics specifically)
2. move ratio computation logics (events chart) to the renderChartConfig
3. fix missing `seriesReturnType` prop in chartConfig in the checkAlert file
Pulls a set of test cases from the v1 code base that checks histogram metric queries against different quantile and queries for edge bounds as well.
Ref: HDX-1425
## feat: allow CTE definitions to be nested chart configs
In order to easily use a CTE for fixing large index issues with delta
trace events, this commit updates the type and `renderWith` function to
render a nested chart config.
Ref: HDX-1343
---
## fix: use CTE instead of listing all index parts in query
Instead of sending 2 queries to the DB and enumerating all of parts
and offsets in the query, this change uses a CTE to select the parts.
This reduces the size of the HTTP request, which fixes the URI too
long response.
Ref: HDX-1343
1. Eliminates a subquery select by pulling the handful of subquery fields up a level.
2. Removed `intDivOrZero` usage as this rounded fractional amounts to the nearest whole number, over/under stating the value.
3. Formatting of query now matches other queries.
Ref: HDX-1467
<img width="1310" alt="Screenshot 2025-02-25 at 3 43 11 PM" src="https://github.com/user-attachments/assets/38c98bc2-2ff2-412c-b26d-4ed9952439f2" />
Co-authored-by: Mike Shi <2781687+MikeShi42@users.noreply.github.com>
Co-authored-by: Dan Hable <418679+dhable@users.noreply.github.com>
Co-authored-by: Tom Alexander <3245235+teeohhem@users.noreply.github.com>
