## Summary
- `yarn app:dev:local` was broken because it didn't source `scripts/dev-env.sh`, so port variables (`HYPERDX_APP_PORT`, `HYPERDX_API_PORT`, etc.) referenced in `.env.development` were undefined
- This caused `dotenv-expand` to hit infinite recursion (stack overflow) when trying to interpolate the unset variables
- Now `app:dev:local` sources `dev-env.sh`, builds common-utils, and tees logs — matching the pattern used by `yarn dev`
## Summary
Fix ClickHouse query error when expanding log rows with Nullable(DateTime64) columns (and other Nullable types).
- The `convertCHDataTypeToJSType` function didn't generically unwrap `Nullable(...)` types, so `Nullable(DateTime64(...))` fell through to the default string comparison instead of using `parseDateTime64BestEffort()`
- Added general `Nullable(...)` recursive unwrapping (matching the existing `LowCardinality(...)` pattern)
- Hoisted null value handling above the type switch in `processRowToWhereClause` so all column types (Date, Array, Map, etc.) correctly emit `isNull()` for null values
### Screenshots or video
N/A — no UI changes.
### How to test locally or on Vercel
1. Set up a ClickHouse table with a `Nullable(DateTime64)` column and ingest some rows (including rows with null values in that column).
2. Open the log explorer and expand a row that has a `Nullable(DateTime64)` column.
3. Verify that clicking into the row no longer returns a 400 error.
4. Verify that clicking into a row where the `Nullable(DateTime64)` column is null correctly filters using `isNull()`.
### References
- Related PRs:
---
📍 Connect Copilot coding agent with [Jira](https://gh.io/cca-jira-docs), [Azure Boards](https://gh.io/cca-azure-boards-docs) or [Linear](https://gh.io/cca-linear-docs) to delegate work to Copilot in one click without leaving your project management tool.
## Summary
In response to the recent [axios supply chain attack](https://www.stepsecurity.io/blog/axios-compromised-on-npm-malicious-versions-drop-remote-access-trojan), we are tightening package management controls to reduce our exposure to malicious or compromised npm packages.
**Changes:**
- Updated `yarnPath` in `.yarnrc.yml` to point to Yarn 4.13.0
- Updated `packageManager` in `package.json` to reflect Yarn 4.13.0
- Removed old Yarn releases (4.5.1 and 1.22.18) from the `releases/` directory
- Added Yarn 4.13.0 to the `releases/` directory
- Set `npmMinimalAgeGate: 7` in `.yarnrc.yml` — Yarn will now block installation of any package version published less than 7 days ago, providing a buffer against freshly-injected malicious releases
### How to test locally or on Vercel
1. Pull this branch and run `yarn --version` — confirm it outputs `4.13.0`.
2. Run `yarn install` and verify it completes without errors.
3. Attempt to add a package version published within the last 7 days (e.g. a freshly released patch) and confirm Yarn rejects it with an age gate error.
4. Add a package version older than 7 days and confirm it installs successfully.
5. Confirm the old Yarn release files (`4.5.1`, `1.22.18`) are no longer present in `releases/`.
### References
- Blog post: [axios compromised on npm — malicious versions drop remote access trojan](https://www.stepsecurity.io/blog/axios-compromised-on-npm-malicious-versions-drop-remote-access-trojan)
## Summary
- Isolate dev, E2E, and integration test environments so multiple git worktrees can run all three simultaneously without port conflicts
- Each worktree gets a deterministic slot (0-99) with unique port ranges: dev (30100-31199), E2E (20320-21399), CI integration (14320-40098)
- Dev portal dashboard (http://localhost:9900) auto-discovers all running stacks, streams logs, and provides a History tab for past run logs
## Port Isolation
| Environment | Port Range | Project Name |
|---|---|---|
| Dev stack | 30100-31199 | `hdx-dev-<slot>` |
| E2E tests | 20320-21399 | `e2e-<slot>` |
| CI integration | 14320-40098 | `int-<slot>` |
All three can run simultaneously from the same worktree with zero port conflicts.
## Dev Portal Features
**Live tab:**
- Auto-discovers dev, E2E, and integration Docker containers + local services (API, App)
- Groups all environments for the same worktree into a single card
- SSE log streaming with ANSI color rendering, capped at 5000 lines
- Auto-starts in background from `make dev`, `make dev-e2e`, `make dev-int`
**History tab:**
- Logs archived to `~/.config/hyperdx/dev-slots/<slot>/history/` on exit (instead of deleted)
- Each archived run includes `meta.json` with worktree/branch metadata
- Grouped by worktree with collapsible cards, search by worktree/branch
- View any past log file in the same log panel, delete individual runs or clear all
- Custom dark-themed confirm modal (no native browser dialogs)
## What Changed
- **`scripts/dev-env.sh`** — Slot-based port assignments, portal auto-start, log archival on exit
- **`scripts/test-e2e.sh`** — E2E port range (20320-21399), log capture via `tee`, portal auto-start, log archival
- **`scripts/ensure-dev-portal.sh`** — Shared singleton portal launcher (works sourced or executed)
- **`scripts/dev-portal/server.js`** — Discovery for dev/E2E/CI containers, history API (list/read/delete), local service port probing
- **`scripts/dev-portal/index.html`** — Live/History tabs, worktree-grouped cards, search, collapse/expand, custom confirm modal, ANSI color log rendering
- **`docker-compose.dev.yml`** — Parameterized ports/volumes/project name with `hdx.dev.*` labels
- **`packages/app/tests/e2e/docker-compose.yml`** — Updated to new E2E port defaults
- **`Makefile`** — `dev-int`/`dev-e2e` targets with log capture + portal auto-start; `dev-portal-stop`; `dev-clean` stops everything + wipes slot data
- **`.env` files** — Ports use `${VAR:-default}` syntax across dev, E2E, and CI environments
- **`agent_docs/development.md`** — Full documentation for isolation, port tables, E2E/CI port ranges
## How to Use
```bash
# Start dev stack (auto-starts portal)
make dev
# Run E2E tests (auto-starts portal, separate ports)
make dev-e2e FILE=navigation
# Run integration tests (auto-starts portal, separate ports)
make dev-int FILE=alerts
# All three can run simultaneously from the same worktree
# Portal at http://localhost:9900 shows everything
# Stop portal
make dev-portal-stop
# Clean up everything (all stacks + portal + history)
make dev-clean
```
## Dev Portal
<img width="1692" height="944" alt="image" src="https://github.com/user-attachments/assets/6ed388a3-43bc-4552-aa8d-688077b79fb7" />
<img width="1689" height="935" alt="image" src="https://github.com/user-attachments/assets/8677a138-0a40-4746-93ed-3b355c8bd45e" />
## Test Plan
- [x] Run `make dev` — verify services start with slot-assigned ports
- [x] Run `make dev` in a second worktree — verify different ports, no conflicts
- [x] Run `make dev-e2e` and `make dev-int` simultaneously — no port conflicts
- [x] Open http://localhost:9900 — verify all stacks grouped by worktree
- [x] Click a service to view logs — verify ANSI colors render correctly
- [x] Stop a stack — verify logs archived to History tab with correct worktree
- [x] History tab — search, collapse/expand, view archived logs, delete
- [x] `make dev-clean` — stops everything, wipes slot data and history
## Summary
Resolve all remaining knip issues — removes unused exports/types, adds missing direct dependencies, deletes dead code, and updates knip config.
**Dependency fixes:**
- Root: swapped unused `eslint-config-next`/`eslint-plugin-react-hooks` for actually-imported `@eslint/js`, `typescript-eslint`, `tslib`
- App: added directly-used transitive deps (`@codemirror/*`, `react-resizable`, `postcss-simple-vars`, `rimraf`, `serve`, `@next/eslint-plugin-next`, `eslint-plugin-react`); removed unused `@storybook/react`
**Dead code removal:**
- Removed ~100 unused exports/types across api and app packages (removed `export` keyword where used locally, deleted entirely where not)
- Fixed duplicate `DBRowTableIconButton` default+named export; updated consumers to use named import
**knip.json updates:**
- Added `fixtures.ts` entry point and `opamp/**` ignore for api package
- Excluded `enumMembers` and `duplicates` issue types
- Enabled `ignoreExportsUsedInFile`
### How to test locally or on Vercel
1. `yarn install && yarn knip` — should produce zero output
2. `make ci-lint` — all packages pass
3. `make ci-unit` — all unit tests pass
## Summary
Cleans up dead code and unused dependencies identified by [knip](https://knip.dev/). All removals were individually verified to have zero imports or references across the codebase. No functional changes.
**Deleted 9 unused files:**
- `packages/api/src/clickhouse/__tests__/clickhouse.V1_DEPRECATED_test.ts` — skipped deprecated test
- `packages/api/src/utils/email.ts` — empty stub functions, never imported
- `packages/api/src/utils/queue.ts` — unused utility class
- `packages/app/src/Checkbox.tsx` — replaced by Mantine Checkbox, never imported
- `packages/app/src/components/DBSearchPageFilters/index.ts` — dead barrel file (shadowed by sibling `.tsx`)
- `packages/app/src/components/Sources/index.ts` — dead barrel file (all consumers import submodules directly)
- `packages/app/src/components/WhereLanguageControlled.tsx` — unused component
- `packages/app/src/TabBarWithContent.tsx` — unused component
- `packages/app/src/vsc-dark-plus.ts` — unused Prism theme
**Removed 14 unused dependencies** from `packages/api`, `packages/app`, and `packages/common-utils` (e.g. `semver`, `react-query`, `react-sortable-hoc`, `@microsoft/fetch-event-source`, `store2`, `uuid`, etc.)
**Removed 17 unused devDependencies** across root, api, app, and common-utils (e.g. `@nx/workspace`, `@typescript-eslint/eslint-plugin`, `@typescript-eslint/parser`, `@types/semver`, `@types/react-table`, `rimraf`, `supertest`, `ts-node`, `tsc-alias`, `tsconfig-paths`, etc.)
**Replaced `react-papaparse` with `papaparse`** — code imports `papaparse` directly, not the React wrapper. Added `@types/papaparse` since the package doesn't bundle its own types.
**Cleaned up unused exports:**
- Trimmed barrel files (`AppNav/index.ts`, `SearchInput/index.ts`) to only re-export what's actually consumed
- Removed duplicate named exports where only the default export is used (`DBRowTableFieldWithPopover`, `DBRowTableRowButtons`)
- Un-exported interfaces and constants that are only used locally (`DBRowTableFieldWithPopoverProps`, `DBRowTableIconButtonProps`, `DBRowTableRowButtonsProps`, `BASE_URL`, `makeHandler`)
- Removed stale `supertest` from `allowModules` in `common-utils/eslint.config.mjs`
### How to test locally or on Vercel
1. `yarn install` — lockfile should resolve cleanly
2. `yarn workspace @hyperdx/common-utils ci:lint` — should pass (the only package where lint+tsc fully passes on this branch)
3. `npx knip` — should show reduced issue counts vs. main
### References
- Related PRs: #1973 (knip CI workflow)
## Summary
Adds [Knip](https://knip.dev) to the monorepo to detect unused files, dependencies, and exports. The goal is to reduce dead code over time and prevent new unused code from accumulating.
**What's included:**
- Root-level `knip.json` configured for all three workspaces (`packages/app`, `packages/api`, `packages/common-utils`)
- `yarn knip` and `yarn knip:ci` scripts for local and CI usage
- GitHub Action (`.github/workflows/knip.yml`) that runs on every PR to `main`, compares results against the base branch, and posts a summary comment showing any increase or decrease in unused code
- Removed the previous app-only `packages/app/knip.json` in favor of the monorepo-wide config
**How the CI workflow works:**
1. Runs Knip on the PR branch
2. Checks out `main` and runs Knip there
3. Compares issue counts per category and posts/updates a PR comment with a diff table
This is additive — Knip runs as an informational check and does not block PRs.
## Summary
This PR adds the Spectral linter for linting our OpenAPI spec, with rules preventing fields with missing examples or descriptions, which are often enforced in the Control Plane repo.
This PR also resolves lint errors that were already present.
### Screenshots or video
### How to test locally or on Vercel
Run `make ci-lint` to lint the openapi specs
### References
- Linear Issue: Closes HDX-3768
- Related PRs:
## Summary
Addresses npm security vulnerabilities in transitive dependencies. Prefer direct dependency upgrades over broad resolutions where possible.
## Changes
**Direct upgrade:**
- **`@slack/webhook`**: `^6.1.0` → `^7.0.0` — v7 natively uses axios v1, eliminating the axios@0.21.4 SSRF/redirect vulnerabilities. Only breaking change in v7 is dropping Node <18 (we're on Node 22).
**Resolutions for transitive deps with no direct upgrade path:**
- **`fast-xml-parser`**: `^4.4.0` — fixes prototype pollution (High)
- **`systeminformation`**: `^5.24.0` — fixes command injection (High)
## Removed/Not Done
- `axios` resolution removed — covered by the `@slack/webhook` upgrade instead
- `tar` resolution removed — was a v6→v7 major jump on build-only tools (`cacache`, `node-gyp`); not present in the production image
- `glob` resolution removed — was breaking test coverage tooling (`test-exclude@6` depends on glob@^7)
## Related
Follow-up to #1731 which addressed base image vulnerabilities (Node, Go, ClickHouse).
Updates base images and patches vulnerable dependencies:
- Node.js 22.16.0 -> 22.22-alpine
- Go 1.25 -> 1.26-alpine
- Express 4.19.2 -> 4.22.1
- Cookie, send, serve-static, and other npm packages
- Fix ENV format warnings in Dockerfile
Reduces vulnerabilities from 178 to 168 (9C, 52H, 98M, 9L). Tested: all services start correctly, health checks pass.
Closes HDX-3124
# Summary
This PR makes the following changes
1. Date ranges for all MV queries are now aligned to the MV Granularity
2. Each chart type now has an indicator when the date range has been adjusted to align with either the MV Granularity or (in the case of Line/Bar charts) the Chart Granularity.
3. The useQueriedChartConfig, useRenderedSqlChartConfig, and useOffsetPaginatedQuery hooks have been updated to get the MV-optimized chart configuration from the useMVOptimizationExplanation, which allows us to share the `EXPLAIN ESTIMATE` query results between the MV Optimization Indicator (the lightning bolt icon on each chart) and the chart itself. This roughly halves the number of EXPLAIN ESTIMATE queries that are made.
## Demo
<img width="1628" height="1220" alt="Screenshot 2026-01-08 at 11 42 39 AM" src="https://github.com/user-attachments/assets/80a06e3a-bbfc-4193-b6b7-5e0056c588d3" />
<img width="1627" height="1131" alt="Screenshot 2026-01-08 at 11 40 54 AM" src="https://github.com/user-attachments/assets/69879e3d-3a83-4c4d-9604-0552a01c17d7" />
## Testing
To test locally with an MV, you can use the following DDL
<details>
<summary>DDL For an MV</summary>
```sql
CREATE TABLE default.metrics_rollup_1m
(
`Timestamp` DateTime,
`ServiceName` LowCardinality(String),
`SpanKind` LowCardinality(String),
`StatusCode` LowCardinality(String),
`count` SimpleAggregateFunction(sum, UInt64),
`sum__Duration` SimpleAggregateFunction(sum, UInt64),
`avg__Duration` AggregateFunction(avg, UInt64),
`quantile__Duration` AggregateFunction(quantileTDigest(0.5), UInt64),
`min__Duration` SimpleAggregateFunction(min, UInt64),
`max__Duration` SimpleAggregateFunction(max, UInt64)
)
ENGINE = AggregatingMergeTree
PARTITION BY toDate(Timestamp)
ORDER BY (Timestamp, StatusCode, SpanKind, ServiceName)
SETTINGS index_granularity = 8192;
CREATE MATERIALIZED VIEW default.metrics_rollup_1m_mv TO default.metrics_rollup_1m
(
`Timestamp` DateTime,
`ServiceName` LowCardinality(String),
`SpanKind` LowCardinality(String),
`version` LowCardinality(String),
`StatusCode` LowCardinality(String),
`count` UInt64,
`sum__Duration` Int64,
`avg__Duration` AggregateFunction(avg, UInt64),
`quantile__Duration` AggregateFunction(quantileTDigest(0.5), UInt64),
`min__Duration` SimpleAggregateFunction(min, UInt64),
`max__Duration` SimpleAggregateFunction(max, UInt64)
)
AS SELECT
toStartOfMinute(Timestamp) AS Timestamp,
ServiceName,
SpanKind,
StatusCode,
count() AS count,
sum(Duration) AS sum__Duration,
avgState(Duration) AS avg__Duration,
quantileTDigestState(0.5)(Duration) AS quantile__Duration,
minSimpleState(Duration) AS min__Duration,
maxSimpleState(Duration) AS max__Duration
FROM default.otel_traces
GROUP BY
Timestamp,
ServiceName,
SpanKind,
StatusCode;
```
</details>
- Improves common-utils build process so the server is ready immediately when started. Currently, when the server starts common-utils hasn't finished building, so it starts, crashes, then restarts correctly after build. Now it runs as expected the first try.
- Adds support for `.env.local` so you can easily provide secret keys without always passing it in via the CLI
- These features already exist downstream, but they seem necessary fro oss as well.
This brings the common-utils and the top level package.json versions into alignment with the app and api packages, e.g. 2.0.0.
Co-authored-by: Warren <5959690+wrn14897@users.noreply.github.com>
- add top level attributes to overview panel: HDX-1715
- add connection name to sources list
- introduce additional default aliases for otel (service, level, duration)
- fix bug with being unable to save source after deselecting correlated log source
- copy improvements
- `dev:down` npm command to tear down dev docker compose
This PR was opened by the [Changesets release](https://github.com/changesets/action) GitHub action. When you're ready to do a release, you can merge this and the packages will be published to npm automatically. If you're not ready to do a release yet, that's fine, whenever you add more changesets to v2, this PR will be updated.
⚠️⚠️⚠️⚠️⚠️⚠️
`v2` is currently in **pre mode** so this branch has prereleases rather than normal releases. If you want to exit prereleases, run `changeset pre exit` on `v2`.
⚠️⚠️⚠️⚠️⚠️⚠️
# Releases
## @hyperdx/common-utils@0.2.0-beta.4
### Minor Changes
- 79fe30f: Queries depending on numeric aggregates now use the type's default value (e.g. 0) instead of null when dealing with non-numeric data.
### Patch Changes
- cfdd523: feat: clickhouse queries are by default conducted through the clickhouse library via POST request. localMode still uses GET for CORS purposes
- 92a4800: feat: move rrweb event fetching to the client instead of an api route
- 7f0b397: feat: queryChartConfig method + events chart ratio
## @hyperdx/api@2.0.0-beta.15
### Minor Changes
- 79fe30f: Queries depending on numeric aggregates now use the type's default value (e.g. 0) instead of null when dealing with non-numeric data.
### Patch Changes
- 9a9581b: Adds external API for alerts and dashboards
- 293a2af: Adds openapidoc annotations for spec generation and swagger route for development
- 92a4800: feat: move rrweb event fetching to the client instead of an api route
- 7f0b397: feat: queryChartConfig method + events chart ratio
- b4b5f6b: style: remove unused routes/components + clickhouse utils (api)
- Updated dependencies [79fe30f]
- Updated dependencies [cfdd523]
- Updated dependencies [92a4800]
- Updated dependencies [7f0b397]
- @hyperdx/common-utils@0.2.0-beta.4
## @hyperdx/app@2.0.0-beta.15
### Patch Changes
- 7de8916: Removes trailing slash for connection urls
- cfdd523: feat: clickhouse queries are by default conducted through the clickhouse library via POST request. localMode still uses GET for CORS purposes
- 6dc6989: feat: Automatically use last used source when loading search page
- 92a4800: feat: move rrweb event fetching to the client instead of an api route
- 7f0b397: feat: queryChartConfig method + events chart ratio
- b4b5f6b: style: remove unused routes/components + clickhouse utils (api)
- Updated dependencies [79fe30f]
- Updated dependencies [cfdd523]
- Updated dependencies [92a4800]
- Updated dependencies [7f0b397]
- @hyperdx/common-utils@0.2.0-beta.4
Co-authored-by: Warren <5959690+wrn14897@users.noreply.github.com>
This PR was opened by the [Changesets release](https://github.com/changesets/action) GitHub action. When you're ready to do a release, you can merge this and the packages will be published to npm automatically. If you're not ready to do a release yet, that's fine, whenever you add more changesets to v2, this PR will be updated.
# Releases
## @hyperdx/common-utils@0.0.10
### Patch Changes
- fc4548f: feat: add alert schema + types
Co-authored-by: Warren <5959690+wrn14897@users.noreply.github.com>
