hyperdx/.changeset/add-cli-pattern-mining.md at 3c057720efb92496e68c2f2568544151b8fdeefc - Elgato_dark/hyperdx - Forgejo: Beyond coding. We Forge.

Elgato_dark/hyperdx

mirror of https://github.com/hyperdxio/hyperdx synced 2026-04-21 13:37:15 +00:00

Warren Lee 418f70c516

[HDX-3964] Add event pattern mining to CLI (Shift+P) (#2106 )

## Summary

Adds a pattern mining feature to the CLI, accessible via `Shift+P`. This mirrors the web app's Pattern Table functionality but runs entirely in TypeScript — no Pyodide/Python WASM needed.

**Linear:** https://linear.app/hyperdx/issue/HDX-3964

## What changed

### 1. Drain library in common-utils (`packages/common-utils/src/drain/`)

Ported the [browser-drain](https://github.com/DeploySentinel/browser-drain) TypeScript library into `@hyperdx/common-utils`. This is a pure TypeScript implementation of the Drain3 log template mining algorithm, including:

- `TemplateMiner` / `TemplateMinerConfig` — main API
- `Drain` — core algorithm with prefix tree and LRU cluster cache
- `LogMasker` — regex-based token masking (IPs, numbers, etc.)
- `LruCache` — custom LRU cache matching Python Drain3's eviction semantics
- 11 Jest tests ported from the original `node:test` suite

### 2. CLI pattern view (`packages/cli/src/components/EventViewer/`)

**Keybinding:** `Shift+P` toggles pattern view (pauses follow mode, restores on exit)

**Data flow (mirrors web app's `useGroupedPatterns`):**
- Issues `SELECT ... ORDER BY rand() LIMIT 100000` to randomly sample up to 100K events
- Issues parallel `SELECT count()` to get true total event count
- Feeds sampled log bodies through the TypeScript `TemplateMiner`
- Estimates pattern counts via `sampleMultiplier = totalCount / sampledRowCount`
- Computes time-bucketed trend data per pattern

**UI:**
- Pattern list with columns: Est. Count (with `~` prefix), Pattern
- `l`/`Enter` expands a pattern to show its sample events (full table columns)
- `h`/`Esc` returns to pattern list
- `j/k/G/g/Ctrl+D/Ctrl+U` navigation throughout
- Loading spinner while sampling query runs

**Alias fix:** Pattern and count queries compute `WITH` clauses from the source's `defaultTableSelectExpression` so Lucene searches using aliases (e.g. `level:error` where `level` is an alias for `SeverityText`) resolve correctly.

### New files
- `packages/common-utils/src/drain/` — 7 source files + barrel index
- `packages/common-utils/src/__tests__/drain.test.ts`
- `packages/cli/src/components/EventViewer/usePatternData.ts`
- `packages/cli/src/components/EventViewer/PatternView.tsx`
- `packages/cli/src/components/EventViewer/PatternSamplesView.tsx`

### Modified files
- `packages/cli/src/api/eventQuery.ts` — added `buildPatternSampleQuery`, `buildTotalCountQuery`, `buildAliasWithClauses`
- `packages/cli/src/components/EventViewer/EventViewer.tsx` — wired in pattern state + rendering
- `packages/cli/src/components/EventViewer/useKeybindings.ts` — added P, l, h keybindings + pattern/sample navigation
- `packages/cli/src/components/EventViewer/SubComponents.tsx` — added P to help screen

### Demo

https://github.com/user-attachments/assets/50a2edfc-8891-43ae-ab86-b96fca778c66

2026-04-14 18:03:56 +00:00

5 lines

110 B

Markdown

Raw Blame History

 ---
 "@hyperdx/cli": patch
 ---
 Add event pattern mining view (Shift+P) with sampled estimation and drill-down