Commit graph

948 commits

Author SHA1 Message Date
github-actions[bot]
f98193852c
Release HyperDX (#1289)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2025-10-30 11:13:50 +01:00
Brandon Pereira
de0b4fc755
Search Relative Time Queries (#1305)
Adds "Relative Time" switch to TimePicker component (if relative time is supported by parent). When enabled, searches will work similar to Live Tail but be relative to the option selected.

<img width="555" height="418" alt="Screenshot 2025-10-27 at 2 05 25 PM" src="https://github.com/user-attachments/assets/20d38011-d5d0-479f-a8ea-6b0be441ca87" />

Some notes:

1. When relative is enabled, I disabled very large time ranges to prioritize performance.
2. If you select "Last 15 mins" then reload, the Input will save "Live Tail" because these are the same option, this should be an edge case.
3. In the future, we might want to make "Relative Time" the default, but I didn't want to immediately do that. We could probably improve the UX further (cc @elizabetdev).
4. Moves a lot of the "Live Tail" logic out of various spots and centralizes it in a unified spot to support other values 

Fixes HDX-2653
2025-10-29 15:49:10 +00:00
Brandon Pereira
808413f5e8
Fix TimePicker Popovers (#1312)
Ensures Date Picker and Selects under TimePicker can be accessed. 

## Before

When trying to click a date in the TimePicker it would close the modal (due to click outside)

## After

Modal will remain open and interactive as expected

Fixes HDX-2662
2025-10-29 15:00:15 +00:00
Ruud Kamphuis
c6ad250f3d
Enable auto-provisioning for no-auth mode (#1297)
Co-authored-by: Aaron Knudtson <87577305+knudtty@users.noreply.github.com>
2025-10-29 09:42:39 -04:00
Drew Davis
7b6ed70c22
fix: Support custom Timestamp Columns in Surrounding Context panel (#1309)
Fixes HDX-2664

# Summary

This PR fixes an error in the surrounding context side panel that occurs when a source does not have a `Timestamp` column. To fix the error, the side panel will now reference the __hdx_timestamp alias queried by `useRowData`, which in turn is based on the Timestamp Column (or Displayed Timestamp Column) in the source's config.

## Testing

To reproduce the issue, create a source without a `Timestamp` column:

<details>
<summary>Source schema</summary>

```sql
CREATE TABLE default.otel_logs_other_ts
(
    `timestamp` DateTime64(9) CODEC(Delta(8), ZSTD(1)),
    `timestamp_time` DateTime DEFAULT toDateTime(timestamp),
    `TraceId` String CODEC(ZSTD(1)),
    `SpanId` String CODEC(ZSTD(1)),
    `TraceFlags` UInt8,
    `SeverityText` LowCardinality(String) CODEC(ZSTD(1)),
    `SeverityNumber` UInt8,
    `ServiceName` LowCardinality(String) CODEC(ZSTD(1)),
    `Body` String CODEC(ZSTD(1)),
    `ResourceSchemaUrl` LowCardinality(String) CODEC(ZSTD(1)),
    `ResourceAttributes` Map(LowCardinality(String), String) CODEC(ZSTD(1)),
    `ScopeSchemaUrl` LowCardinality(String) CODEC(ZSTD(1)),
    `ScopeName` String CODEC(ZSTD(1)),
    `ScopeVersion` LowCardinality(String) CODEC(ZSTD(1)),
    `ScopeAttributes` Map(LowCardinality(String), String) CODEC(ZSTD(1)),
    `LogAttributes` Map(LowCardinality(String), String) CODEC(ZSTD(1)),
    `__hdx_materialized_k8s.cluster.name` LowCardinality(String) MATERIALIZED ResourceAttributes['k8s.cluster.name'] CODEC(ZSTD(1)),
    `__hdx_materialized_k8s.container.name` LowCardinality(String) MATERIALIZED ResourceAttributes['k8s.container.name'] CODEC(ZSTD(1)),
    `__hdx_materialized_k8s.deployment.name` LowCardinality(String) MATERIALIZED ResourceAttributes['k8s.deployment.name'] CODEC(ZSTD(1)),
    `__hdx_materialized_k8s.namespace.name` LowCardinality(String) MATERIALIZED ResourceAttributes['k8s.namespace.name'] CODEC(ZSTD(1)),
    `__hdx_materialized_k8s.node.name` LowCardinality(String) MATERIALIZED ResourceAttributes['k8s.node.name'] CODEC(ZSTD(1)),
    `__hdx_materialized_k8s.pod.name` LowCardinality(String) MATERIALIZED ResourceAttributes['k8s.pod.name'] CODEC(ZSTD(1)),
    `__hdx_materialized_k8s.pod.uid` LowCardinality(String) MATERIALIZED ResourceAttributes['k8s.pod.uid'] CODEC(ZSTD(1)),
    `__hdx_materialized_deployment.environment.name` LowCardinality(String) MATERIALIZED ResourceAttributes['deployment.environment.name'] CODEC(ZSTD(1)),
    INDEX idx_trace_id TraceId TYPE bloom_filter(0.001) GRANULARITY 1,
    INDEX idx_res_attr_key mapKeys(ResourceAttributes) TYPE bloom_filter(0.01) GRANULARITY 1,
    INDEX idx_res_attr_value mapValues(ResourceAttributes) TYPE bloom_filter(0.01) GRANULARITY 1,
    INDEX idx_scope_attr_key mapKeys(ScopeAttributes) TYPE bloom_filter(0.01) GRANULARITY 1,
    INDEX idx_scope_attr_value mapValues(ScopeAttributes) TYPE bloom_filter(0.01) GRANULARITY 1,
    INDEX idx_log_attr_key mapKeys(LogAttributes) TYPE bloom_filter(0.01) GRANULARITY 1,
    INDEX idx_log_attr_value mapValues(LogAttributes) TYPE bloom_filter(0.01) GRANULARITY 1,
    INDEX idx_lower_body lower(Body) TYPE tokenbf_v1(32768, 3, 0) GRANULARITY 8
)
ENGINE = MergeTree
PARTITION BY toDate(timestamp_time)
PRIMARY KEY (ServiceName, timestamp_time)
ORDER BY (ServiceName, timestamp_time, timestamp)
TTL timestamp_time + toIntervalDay(30)
SETTINGS index_granularity = 8192, ttl_only_drop_parts = 1
```
</details>

Then try to open the surrounding context panel:

https://github.com/user-attachments/assets/d5f14e3c-83ce-40c0-b2c4-ef9fe8e1467e

With these changes, the error is fixed (as long as the source's timestamp column configuration is correct)

https://github.com/user-attachments/assets/605c50e5-9306-4d2e-a9b1-9afc3adca9b6
2025-10-28 17:58:22 +00:00
Drew Davis
3ee93ae918
feat: Show pinned filter values while filters are loading (#1308)
Closes HDX-2641

# Summary

With this change, HyperDX will now display pinned filter values as soon as the search page loads, without waiting for the filter values to be queried from ClickHouse. This enables users to quickly apply relevant filters, before the (sometimes very slow) filter values query completes.

## Demo

For this demo, I added an artificial delay to the filter query to simulate an environment where filter queries are slow

https://github.com/user-attachments/assets/6345cb91-7aba-4acc-a832-05efb3bf17d0
2025-10-28 16:47:27 +00:00
Drew Davis
d5a38c3e05
fix: Fix pattern sample query for sources with multi-column timestamp expressions (#1304)
Fixes HDX-2621

# Summary

This PR fixes a query error when opening a sample log line in the patterns table from a source with multiple timestamp columns. To fix the issue, we simply use the first of the timestamp columns. This is consistent with several other places in the app where we use just the first timestamp column - multiple timestamp columns is not fully supported.

## Testing

To reproduce the issue, set the Timestamp Column in source settings for the logs source to `TimestampTime, toStartOfMinute(TimestampTime)`. Then attempt to open a pattern sample:

https://github.com/user-attachments/assets/2464f97e-1423-437c-88f0-b45486feffcc

With these changes, the issue is fixed:

https://github.com/user-attachments/assets/54d8f0f2-532c-4eb4-a676-ab6a606ecac5
2025-10-28 16:19:46 +00:00
Drew Davis
15331acbee
feat: Auto-select correlated sources on k8s dashboard (#1302)
Closes HDX-2586

# Summary

This PR updates the K8s dashboard so that it auto-selects correlated log and metric sources.

Auto-selection of sources happens
1. During page load, if sources aren't specified in the URL params
2. When a new log source is selected, a correlated metric source is auto-selected. In this case, a notification is shown to the user to inform them that the metric source has been updated.

When a new metric source is selected, a correlated log source is not selected. This is to ensure the user has some way of selecting two non-correlated sources, if they truly want to. If the user does select a metric source which is not correlated with the selected log source, a warning notification will be shown to the user.

## Demo


https://github.com/user-attachments/assets/492121a1-0a51-4af9-a749-42771537678e
2025-10-28 12:17:53 +00:00
Brandon Pereira
757196f2e9
close modals when bluring (dates and search hints) (#1294)
When clicking outside the search hints or date modal, the modals close to prevent them appearing ontop of other modals when you click results

<img width="2168" height="1222" alt="Screenshot 2025-10-24 at 11 07 49 AM" src="https://github.com/user-attachments/assets/c930919a-7d91-420d-be46-1db5ca35c2de" />
<img width="1004" height="866" alt="Screenshot 2025-10-24 at 11 07 52 AM" src="https://github.com/user-attachments/assets/8969bc7d-2655-4a1d-8a34-1f301401edf8" />

Fixes HDX-2643
2025-10-27 18:55:39 +00:00
Dan Hable
431a9f01f3
refactor(tasks): create subdirectory for alerts code (#1281)
The alerts code and some of our private source tasks are becoming complex enough for multiple files. In order to keep the code clearer to debug and read, this commit moves the check alerts tasks into a sub-directory for just alert related code.
2025-10-27 18:16:50 +00:00
Aaron Knudtson
778092d34f
fix: long timeranges for task queries truncated (#1276)
Fixes HDX-2618

Related to https://github.com/ClickHouse/support-escalation/issues/6113

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2025-10-27 17:40:11 +00:00
Drew Davis
2162a69039
feat: Optimize and fix filtering on toStartOfX primary key expressions (#1265)
Closes HDX-2576
Closes HDX-2491

# Summary

It is a common optimization to have a primary key like `toStartOfDay(Timestamp), ..., Timestamp`. This PR improves the experience when using such a primary key in the following ways:

1. HyperDX will now automatically filter on both `toStartOfDay(Timestamp)` and `Timestamp` in this case, instead of just `Timestamp`. This improves performance by better utilizing the primary index. Previously, this required a manual change to the source's Timestamp Column setting.
2. HyperDX now applies the same `toStartOfX` function to the right-hand-side of timestamp comparisons. So when filtering using an expression like `toStartOfDay(Timestamp)`, the generated SQL will have the condition `toStartOfDay(Timestamp) >= toStartOfDay(<selected start time>) AND toStartOfDay(Timestamp) <= toStartOfDay(<selected end time>)`. This resolves an issue where some data would be incorrectly filtered out when filtering on such timestamp expressions (such as time ranges less than 1 minute).

With this change, teams should no longer need to have multiple columns in their source timestamp column configuration. However, if they do, they will now have correct filtering.

## Testing

### Testing the fix

The part of this PR that fixes time filtering can be tested with the default logs table schema. Simply set the Timestamp Column source setting to `TimestampTime, toStartOfMinute(TimestampTime)`. Then, in the logs search, filter for a timespan < 1 minute.

<details>
<summary>Without the fix, you should see no logs, since they're incorrectly filtered out by the toStartOfMinute(TimestampTime) filter</summary>

https://github.com/user-attachments/assets/915d3922-55f8-4742-b686-5090cdecef60
</details>

<details>
<summary>With the fix, you should see logs in the selected time range</summary>

https://github.com/user-attachments/assets/f75648e4-3f48-47b0-949f-2409ce075a75
</details>

### Testing the optimization

The optimization part of this change is that when a table has a primary key like `toStartOfMinute(TimestampTime), ..., TimestampTime` and the Timestamp Column for the source is just `Timestamp`, the query will automatically filter by both  `toStartOfMinute(TimestampTime)` and `TimestampTime`.

To test this, you'll need to create a table with such a primary key, then create a source based on that table. Optionally, you could copy data from the default `otel_logs` table into the new table (`INSERT INTO default.otel_logs_toStartOfMinute_Key SELECT * FROM default.otel_logs`).

<details>
<summary>DDL for log table with optimized key</summary>

```sql
CREATE TABLE default.otel_logs_toStartOfMinute_Key
(
    `Timestamp` DateTime64(9) CODEC(Delta(8), ZSTD(1)),
    `TimestampTime` DateTime DEFAULT toDateTime(Timestamp),
    `TraceId` String CODEC(ZSTD(1)),
    `SpanId` String CODEC(ZSTD(1)),
    `TraceFlags` UInt8,
    `SeverityText` LowCardinality(String) CODEC(ZSTD(1)),
    `SeverityNumber` UInt8,
    `ServiceName` LowCardinality(String) CODEC(ZSTD(1)),
    `Body` String CODEC(ZSTD(1)),
    `ResourceSchemaUrl` LowCardinality(String) CODEC(ZSTD(1)),
    `ResourceAttributes` Map(LowCardinality(String), String) CODEC(ZSTD(1)),
    `ScopeSchemaUrl` LowCardinality(String) CODEC(ZSTD(1)),
    `ScopeName` String CODEC(ZSTD(1)),
    `ScopeVersion` LowCardinality(String) CODEC(ZSTD(1)),
    `ScopeAttributes` Map(LowCardinality(String), String) CODEC(ZSTD(1)),
    `LogAttributes` Map(LowCardinality(String), String) CODEC(ZSTD(1)),
    `__hdx_materialized_k8s.pod.name` String MATERIALIZED ResourceAttributes['k8s.pod.name'] CODEC(ZSTD(1)),
    INDEX idx_trace_id TraceId TYPE bloom_filter(0.001) GRANULARITY 1,
    INDEX idx_res_attr_key mapKeys(ResourceAttributes) TYPE bloom_filter(0.01) GRANULARITY 1,
    INDEX idx_res_attr_value mapValues(ResourceAttributes) TYPE bloom_filter(0.01) GRANULARITY 1,
    INDEX idx_scope_attr_key mapKeys(ScopeAttributes) TYPE bloom_filter(0.01) GRANULARITY 1,
    INDEX idx_scope_attr_value mapValues(ScopeAttributes) TYPE bloom_filter(0.01) GRANULARITY 1,
    INDEX idx_log_attr_key mapKeys(LogAttributes) TYPE bloom_filter(0.01) GRANULARITY 1,
    INDEX idx_log_attr_value mapValues(LogAttributes) TYPE bloom_filter(0.01) GRANULARITY 1,
    INDEX idx_body Body TYPE tokenbf_v1(32768, 3, 0) GRANULARITY 8,
    INDEX idx_lower_body lower(Body) TYPE tokenbf_v1(32768, 3, 0) GRANULARITY 8
)
ENGINE = SharedMergeTree('/clickhouse/tables/{uuid}/{shard}', '{replica}')
PARTITION BY toDate(TimestampTime)
PRIMARY KEY (toStartOfMinute(TimestampTime), ServiceName, TimestampTime)
ORDER BY (toStartOfMinute(TimestampTime), ServiceName, TimestampTime, Timestamp)
TTL TimestampTime + toIntervalDay(90)
SETTINGS index_granularity = 8192, ttl_only_drop_parts = 1
```
</details>

Once you have that source, you can inspect the queries generated for that source. Whenever a date range filter is selected, the query should have a `WHERE` predicate that filters on both `TimestampTime` and `toStartOfMinute(TimestampTime)`, despite `toStartOfMinute(TimestampTime)` not being included in the Timestamp Column of the source's configuration.
2025-10-27 17:20:36 +00:00
Drew Davis
8190ee8f6a
perf: Improve getKeyValues query performance for JSON keys (#1284)
Closes HDX-2623

# Summary

This change improves the performance of `getKeyValues` when getting values of a JSON key. 

Generally, columns that are not referenced outside of a CTE will be pruned by the query planner. For JSON however, if the outer select references one field in a JSON column, then the inner select will read (it seems) the entire JSON object.

This PR also adds integration tests for `getKeyValues` to ensure that the function generates queries that work as expected in ClickHouse.

##  Performance impact (on single JSON Dashboard Filter)

- Original: 15.03s

<img width="584" height="71" alt="Screenshot 2025-10-21 at 3 28 07 PM" src="https://github.com/user-attachments/assets/184de198-cee1-4b1d-beed-ec4465d3e248" />

- Optimized: 0.443s

<img width="590" height="61" alt="Screenshot 2025-10-21 at 3 25 47 PM" src="https://github.com/user-attachments/assets/690d0ef0-15b8-47c5-9a7e-8b8f6b8f5e92" />
2025-10-27 16:46:46 +00:00
Aaron Knudtson
93edb6f84f
fix: memoize inputs to fix performance issues (#1291)
Fixes HDX-2644
Fixes #1288
2025-10-27 14:53:18 +00:00
Brandon Pereira
bb3539dd5d
Update Drawers to do a proper focus trap (#1290)
Improves user reported issues that clicking outside the modal would click elements unexpectedly, also improves accessibility (keyboard focus trap & esc to exit)

Fixes HDX-2642
2025-10-27 14:47:55 +00:00
Drew Davis
ff86d40006
perf: Implement query chunking for charts (#1233)
# Summary

Closes HDX-2310
Closes HDX-2616

This PR implements chunking of chart queries to improve performance of charts on large data sets and long time ranges. Recent data is loaded first, then older data is loaded one-chunk-at-a-time until the full chart date range has been queried.

https://github.com/user-attachments/assets/83333041-9e41-438a-9763-d6f6c32a0576

## Performance Impacts

### Expectations

This change is intended to improve performance in a few ways:

1. Queries over long time ranges are now much less likely to time out, since the range is chunked into several smaller queries
2. Average memory usage should decrease, since the total result size and number of rows being read are smaller
3. _Perceived_ latency of queries over long date ranges is likely to decrease, because users will start seeing charts render (more recent) data as soon as the first chunk is queried, instead of after the entire date range has been queried. **However**, _total_ latency to display results for the entire date range is likely to increase, due to additional round-trip network latency being added for each additional chunk.

### Measured Results

Overall, the results match the expectations outlined above.

- Total latency changed between ~-4% and ~25%
- Average memory usage decreased by between 18% and 80%

<details>
<summary>Scenarios and data</summary>

In each of the following tests:

1. Queries were run 5 times before starting to measure, to ensure data is filesystem cached.
2. Queries were then run 3 times. The results shown are the median result from the 3 runs.

#### Scenario: Log Search Histogram in Staging V2, 2 Day Range, No Filter

|   |  Total Latency | Memory Usage (Avg) | Memory Usage (Max)  |  Chunk Count |
|---|---|---|---|---|
|  Original |  5.36 |  409.23 MiB |  409.23 MiB | 1  |
|  Chunked |  5.14 | 83.06 MiB  | 232.69 MiB  |  4 |

#### Scenario: Log Search Histogram in Staging V2, 14 Day Range, No Filter

|   |  Total Latency | Memory Usage (Avg) | Memory Usage (Max)  |  Chunk Count |
|---|---|---|---|---|
|  Original |  26.56 |  383.63 MiB |  383.63 MiB | 1  |
|  Chunked |  33.08 | 130.00 MiB  | 241.21 MiB  |  16 |

#### Scenario: Chart Explorer Line Chart with p90 and p99 trace durations, Staging V2 Traces, Filtering for "GET" spans, 7 Day range

|   |  Total Latency | Memory Usage (Avg) | Memory Usage (Max)  |  Chunk Count |
|---|---|---|---|---|
|  Original |  2.79 |  346.12 MiB |  346.12 MiB | 1  |
|  Chunked |  3.26 | 283.00 MiB  | 401.38 MiB  |  9 |

</details>

## Implementation Notes

<details>
<summary>When is chunking used?</summary>
Chunking is used when all of the following are true:

1. `granularity` and `timestampValueExpression` are defined in the config. This ensures that the query is already being bucketed. Without bucketing, chunking would break aggregation queries, since groups can span multiple chunks.
4. `dateRange` is defined in the config. Without a date range, we'd need an unbounded set of chunks or the start and end chunks would have to be unbounded at their start and end, respectively.
5. The config is not a metrics query. Metrics queries have complex logic which we want to avoid breaking with the initial delivery of this feature.
6. The consumer of `useQueriedChartConfig` does not pass the `disableQueryChunking: true` option. This option is provided to disable chunking when necessary.
</details>

<details>
<summary>How are time windows chosen?</summary>

1. First, generate the windows as they are generated for the existing search chunking feature (eg. 6 hours back, 6 hours back, 12 hours back, 24 hours back...)
4. Then, the start and end of each window is aligned to the start of a time bucket that depends on the "granularity" of the chart.
7. The first and last windows are shortened or extended so that the combined date range of all of the windows matches the start and end of the original config.
</details>

<details>
<summary>Which order are the chunks queried in?</summary>

Chunks are queried sequentially, most-recent first, due to the expectation that more recent data is typically more important to the user. Unlike with `useOffsetPaginatedSearch`, we are not paginating the data beyond the chunks, and all data is typically displayed together, so there is no need to support "ascending" order.
</details>

<details>
<summary>Does this improve client-side caching behavior?</summary>

One theoretical way in which query chunking could improve performance to enable client-side caching of individual chunks, which could then be re-used if the same query is run over a longer time range.

Unfortunately, using streamedQuery, react-query stores the entire time range as one item in the cache, so it does not re-use individual chunks or "pages" from another query.

We could accomplish this improvement by using useQueries instead of streamedQuery or useInfiniteQuery. In that case, we'd treat each chunk as its own query. This would require a number of changes:

1. Our query key would have to include the chunk's window duration
2. We'd need some hacky way of making the useQueries requests fire in sequence. This can be done using `enabled` but requires some additional state to figure out whether the previous query is done.
5. We'd need to emulate the return value of a useQuery using the useQueries result, or update consumers.
</details>
2025-10-27 14:02:59 +00:00
Drew Davis
21614b94aa
feat: Include displayed timestamp in default order by (#1279)
Closes HDX-2593

# Summary 

This PR adds a source's `displayedTimestampValueExpression` (if one exists) to the default order by on the search page.

## Motivation

In schemas like our default otel_logs table, there are two timestamp columns: `TimestampTime DateTime` (1-second precision) and a `Timestamp DateTime64(9)` (nanosecond precision). `TimestampTime` is preferred for filtering because it is more granular and in the primary key. However, ordering by `TimestampTime` alone results in an arbitrary order of events within each second:

<img width="646" height="158" alt="Screenshot 2025-10-17 at 2 28 50 PM" src="https://github.com/user-attachments/assets/298a340f-387d-4fdf-9298-622388bb6962" />

## Details

The HyperDX source configuration form already supports configuring a 'Displayed Timestamp Column" for a log source. This  PR adds the same option for Trace sources. This field is inferred from the otel logs and traces schemas as `Timestamp`.

<img width="999" height="383" alt="Screenshot 2025-10-17 at 2 30 13 PM" src="https://github.com/user-attachments/assets/db1ed1eb-7ab1-4d6a-a702-b45b4d2274af" />

If the source has a displayed timestamp column configured, and if this column is different than the source's timestamp value expression, then this field will be added to the default order by which is generated for the search page. This results in a more precise ordering of the events in the logs table within each second:

<img width="950" height="233" alt="Screenshot 2025-10-17 at 2 33 16 PM" src="https://github.com/user-attachments/assets/1d8447c5-ce4c-40e5-bce6-f681fe881436" />
2025-10-27 13:44:48 +00:00
Brandon Pereira
ab7af41f7e
fix hydration error (#1296) 2025-10-25 18:01:43 +02:00
Warren
24b5477d62
feat: allow specifying webhook request headers (#1285)
Address https://github.com/hyperdxio/hyperdx/issues/1078

Ref: HDX-2225

<img width="842" height="688" alt="image" src="https://github.com/user-attachments/assets/66d8279c-d596-4d63-af05-14d74a1d7b54" />
2025-10-23 11:30:29 +00:00
github-actions[bot]
0325416235
Release HyperDX (#1224)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2025-10-22 16:13:54 +02:00
Mike Shi
5d50e0ad12
fix: dont show ai assistant if no anthropic api key exists (#1286)
<img width="1446" height="339" alt="image" src="https://github.com/user-attachments/assets/a0dbaff7-1238-4522-8d9b-56ce1501021a" />
2025-10-22 12:04:21 +00:00
Warren
131a1c1edb
revert: api esbuild (#1280)
This PR reverts https://github.com/hyperdxio/hyperdx/pull/937

Ref: HDX-2620
2025-10-21 09:27:47 +00:00
Nguyen Dat
b806116d73
feat: add subpath config (#1236)
Co-authored-by: Warren <5959690+wrn14897@users.noreply.github.com>
2025-10-17 14:43:58 -07:00
Tom Alexander
658728318b
fix: Preserve original select from time chart event selection (#1277)
FIxes: HDX-2606
2025-10-17 16:57:01 +00:00
Drew Davis
6262ced8ce
fix: Fix crash when navigating away from chart explorer search page (#1278) 2025-10-17 11:16:47 -04:00
Tom Alexander
065cabdb47
fix: Update "Copy Object" in line viewer to work with nested objects and arrays (#1274)
Fixes: HDX-2617
2025-10-16 16:55:22 +00:00
Drew Davis
4cfbd7579c
chore: Cleanup warnings in unit tests (#1273)
This PR fixes the following warnings from our unit tests, which were cluttering up the output:

- The regex warning is fixed by removing the /g global flag from element selectors
- The deprecation warning is fixed by the package upgrade

<img width="1156" height="184" alt="Screenshot 2025-10-16 at 9 06 57 AM" src="https://github.com/user-attachments/assets/37b6c408-e453-4fb8-baf9-323bbd79e53c" />
<img width="1445" height="191" alt="Screenshot 2025-10-16 at 9 06 50 AM" src="https://github.com/user-attachments/assets/71388182-5e4f-42c1-ae18-026c8d963e1f" />
2025-10-16 16:48:48 +00:00
Drew Davis
3c8f3b54c7
fix: Include connectionId in metadata cache key (#1272)
Closes HDX-2573

This PR adds connection ID to the metadata cache key, to ensure that metadata is not shared across tables with the same name in different connections.

For example, previously when switching between otel_logs tables in different connections, filter values would not reload:

https://github.com/user-attachments/assets/d3edb243-4564-438f-9fec-c400fdfbd5b3

Now they will:

https://github.com/user-attachments/assets/19b0f022-8e30-412e-8a06-3d9812794219
2025-10-16 15:39:49 +00:00
Tom Alexander
3d832ad787
chore: Update claude gh action to encourage shorter reviews (#1275) 2025-10-16 15:02:23 +00:00
Drew Davis
2d27fe27c1
fix: Support JSON keys in dashboard filters (#1271) 2025-10-16 08:14:57 -04:00
Drew Davis
2dc0079b08
feat: Sort dashboard filter values (#1270)
Closes HDX-2581

Dashboard filter options are now sorted alphabetically.
2025-10-15 19:13:31 +00:00
Warren
348a4044e8
migration: migrate to Pino (logger) (#1269)
Summary

  Migrates the HyperDX API and related services from Winston to Pino for standardized,
  faster, and more structured logging with improved OpenTelemetry integration.

Changes

  Core Migration

  - Replaced Winston with Pino across all logging infrastructure
  - Upgraded @hyperdx/node-opentelemetry from v0.8.2 to v0.9.0 to support Pino
  transport
  - Removed deprecated dependencies:
    - winston and express-winston
    - @opentelemetry/host-metrics and @opentelemetry/sdk-metrics (consolidated into
  newer OTel SDK)
  - Added new dependencies:
    - pino and pino-http for core logging
    - pino-pretty for development console output

  Logger Configuration (packages/api/src/utils/logger.ts)

  Production:
  - Outputs stringified JSON to stdout via pino/file transport
  - Maintains HyperDX transport integration when API key is configured
  - Includes full OpenTelemetry trace context (trace_id, span_id, trace_flags)
 
<img width="830" height="184" alt="Screenshot 2025-10-14 at 4 31 36 PM" src="https://github.com/user-attachments/assets/82e60919-5c4d-4688-a6f5-d54632aef749" />


  Development:
  - Uses pino-pretty for human-readable, colorized console output
  - Hides verbose fields from console: pid, hostname, trace_id, span_id, trace_flags
  - HTTP request/response objects excluded from logs via custom serializers

<img width="825" height="350" alt="image" src="https://github.com/user-attachments/assets/64b293d8-bc95-4715-931a-dbf73483d247" />

  HTTP Logging:
  - Replaced express-winston with pino-http
  - Custom log levels based on HTTP status codes (warn for 4xx, error for 5xx+)
  - Simplified log messages: HTTP {method} {url}

  Error Logging Updates

  Updated error logging patterns throughout the codebase to follow Pino's structured
  logging conventions:

  // Before (Winston)
  logger.error('Error message:', error);

  // After (Pino)
  logger.error({ err: error }, 'Error message');


Ref: HDX-2588

This PR should also address issue: https://github.com/hyperdxio/hyperdx/issues/1035
2025-10-15 17:15:46 +00:00
Warren
43e32aafc7
feat: revisit Otel metrics semantic convention migration logics (#1267)
Since users can still switch to the new metric name using feature gate

Follow up https://github.com/hyperdxio/hyperdx/pull/1248
2025-10-14 22:06:31 +00:00
Tom Alexander
79af32e576
chore: Update claude.md with more linting instrucitons (#1268) 2025-10-14 21:32:55 +00:00
Drew Davis
c428d98412
fix: Set team and connection attributes on span instead of trace (#1266)
Closes HDX-2506

`setTraceAttributes` is not working (see HDX-2599), so for now we will replace it with span attributes, to ensure that we have some way of correlating alert job logs to teams.

<img width="492" height="209" alt="Screenshot 2025-10-14 at 4 07 36 PM" src="https://github.com/user-attachments/assets/46f9391e-e4c9-4a12-bc6b-a37d53ecadf6" />
2025-10-14 20:29:57 +00:00
Brandon Pereira
3332d5eaf5
Event Deltas Heatmap Customization (#1246)
Introduces the ability to edit the y, count, and group by values on the search event deltas heatmap allowing for deeper analysis

https://github.com/user-attachments/assets/1f4d7ca2-86be-424a-84de-23eccaf02a68

Fixes HDX-2422
2025-10-14 17:59:34 +00:00
Drew Davis
2f49f9be37
fix: ignore max_rows_to_read for filter values distribution (#1259) 2025-10-10 13:05:07 -04:00
Brandon Pereira
05ca6cebcb
chore: attempt to make claude update last comment instead of a new one (#1255) 2025-10-10 11:56:13 -04:00
Drew Davis
eef3cb54d2
fix: Keep k8s dashboard params up to date with defaulted sources (#1258) 2025-10-10 11:42:59 -04:00
Brandon Pereira
ec2ea566af
Improve Support for Dynamic and JSON(<parameters>) Types (#1251)
1. Improves getJSONColumnNames to support JSON type with arguments
2. Improves useRowWhere to support arrays and JSON types

How I tested this:
1. Created a new table and setup various types of formats (`JSON, JSON(max_dynamic_paths = 3, test_key String), Dynamic, Array(Tuple(test_key String, test_value Dynamic)), Map(String, Dynamic)`)
2. Reproduced exact scenario from tickt and fixed (`getJSONColumnNames`)
4. Clicked around the app with various other types, observed that `Dynamic` fields could not be clicked if they had JSON or Array formats. Fixed that (`useRowWhere`)
5. Added tests for changes, ensured existing tests worked for `Dynamuc` useRowWhere by reproducing them in app 

Fixes HDX-2523
2025-10-10 14:57:57 +00:00
Elizabet Oliveira
eaff49293c
Add copy-to-clipboard buttons in RawLogTable for log line data and URLs (#1227)
Co-authored-by: Mike Shi <mike@hyperdx.io>
2025-10-10 15:50:34 +01:00
Aaron Knudtson
dbf16827a3
feat: add refresh to existing preset dashboards (#1249)
Closes HDX-2046

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2025-10-09 20:18:20 +00:00
Drew Davis
daffcf3594
feat: Add percentages to filter values (#1250)
# Summary

Closes HDX-1960

This PR adds a button to our search filters component which can be used to show the _approximate_ percentage of rows which have each filter value.

https://github.com/user-attachments/assets/2dba1b28-d2b9-4414-986c-0c515d252c89

Notes:
- The percentages are based on a sample of 100k rows. The sampling is done similarly to how EE version samples logs for patterns.
- We only fetch the most common 100 values in the sample. All other values are assumed to represent <1% of the data.
- The percentages represent the distribution within the dataset after it has been filtered by the selected filters and the where clause.
- This is a potentially expensive query, even with sampling, so the percentages are only queried if they're toggled on for a particular filter, and do not refresh in live mode. They do refresh if the search or date ranges changes (outside of live mode).
2025-10-09 19:26:39 +00:00
Drew Davis
13b191c8a0
feat: Allow selection of log and metric source on K8s dashboard (#1245)
Closes HDX-1887

This change allows the user to select which log and metric sources the k8s dashboard should show. Previously, the user could only select a connection, and the first log and metric source in that connection would be used.

<img width="1756" height="1121" alt="Screenshot 2025-10-07 at 2 50 34 PM" src="https://github.com/user-attachments/assets/f6e4f375-1f8d-486c-8940-4ee2ac38b94d" />
2025-10-09 19:15:21 +00:00
Tom Alexander
4949748f99
chore: Add Claude Code GitHub Workflow (#1252)
## 🤖 Installing Claude Code GitHub App

This PR adds a GitHub Actions workflow that enables Claude Code integration in our repository.

### What is Claude Code?

[Claude Code](https://claude.ai/code) is an AI coding agent that can help with:
- Bug fixes and improvements  
- Documentation updates
- Implementing new features
- Code reviews and suggestions
- Writing tests
- And more!

### How it works

Once this PR is merged, we'll be able to interact with Claude by mentioning @claude in a pull request or issue comment.
Once the workflow is triggered, Claude will analyze the comment and surrounding context, and execute on the request in a GitHub action.

### Important Notes

- **This workflow won't take effect until this PR is merged**
- **@claude mentions won't work until after the merge is complete**
- The workflow runs automatically whenever Claude is mentioned in PR or issue comments
- Claude gets access to the entire PR or issue context including files, diffs, and previous comments

### Security

- Our Anthropic API key is securely stored as a GitHub Actions secret
- Only users with write access to the repository can trigger the workflow
- All Claude runs are stored in the GitHub Actions run history
- Claude's default tools are limited to reading/writing files and interacting with our repo by creating comments, branches, and commits.
- We can add more allowed tools by adding them to the workflow file like:

```
allowed_tools: Bash(npm install),Bash(npm run build),Bash(npm run lint),Bash(npm run test)
```

There's more information in the [Claude Code action repo](https://github.com/anthropics/claude-code-action).

After merging this PR, let's try mentioning @claude in a comment on any PR to get started!
2025-10-09 14:20:48 +00:00
Brandon Pereira
0cf179fa68
small typo fix in type (#1242)
Noticed this when resolving a conflict, should be `Chart` not `Chat` :P 

Fixes HDX-2553
2025-10-09 14:09:04 +00:00
Warren
5efa2ffa0d
feat: handle k8s metrics semantic convention updates (#1248)
Handle OpenTelemetry semantic versions based on the ScopeVersion field (metrics)
Related to [changes](https://opentelemetry.io/blog/2025/kubeletstats-receiver-metrics-deprecation/)

Old (switched to v0.137.0)
<img width="818" height="317" alt="image" src="https://github.com/user-attachments/assets/ceea52c6-ad06-4295-afae-a44f21b2e962" />

New (be able to handle multiple versions)
<img width="568" height="329" alt="image" src="https://github.com/user-attachments/assets/d2e282b2-cfd7-490a-a64d-502881a360a2" />


Ref: HDX-2322, HDX-2562
2025-10-08 21:04:40 +00:00
Dan Hable
c90a93e6c3
fix(alerts): bump cron package to pick up stalled job fix (#1244)
Ref: https://github.com/kelektiv/node-cron/issues/962
2025-10-07 21:39:48 +00:00
Mike Shi
b8efb4924c
chart ai assistant (#1243) 2025-10-07 14:47:10 -04:00
Brandon Pereira
b34480411e
Log Results Sorting (#1232) 2025-10-07 09:35:42 -06:00