hyperdx/AGENTS.md

HyperDX Development Guide

What is HyperDX?

HyperDX is an observability platform that helps engineers search, visualize, and monitor logs, metrics, traces, and session replays. It's built on ClickHouse for blazing-fast queries and supports OpenTelemetry natively.

Core value: Unified observability with ClickHouse performance, schema-agnostic design, and correlation across all telemetry types in one place.

Architecture (WHAT)

This is a monorepo with three main packages:

• packages/app - Next.js frontend (TypeScript, Mantine UI, TanStack Query)
• packages/api - Express backend (Node.js 22+, MongoDB for metadata, ClickHouse for telemetry)
• packages/common-utils - Shared TypeScript utilities for query parsing and validation

Data flow: Apps → OpenTelemetry Collector → ClickHouse (telemetry data) / MongoDB (configuration/metadata)

Development Setup (HOW)

yarn setup          # Install dependencies
yarn dev            # Start full stack with worktree-isolated ports

The project uses Yarn 4.13.0 workspaces. Docker Compose manages ClickHouse, MongoDB, and the OTel Collector.

This repo is multi-agent friendly. yarn dev, make dev-int, and make dev-e2e all use slot-based port isolation so multiple worktrees can run dev servers, integration tests, and E2E tests simultaneously without conflicts. A dev portal at http://localhost:9900 auto-starts and shows all running stacks. See agent_docs/development.md for the full multi-worktree setup, port allocation tables, and available commands.

Working on the Codebase (HOW)

Before starting a task, read relevant documentation from the agent_docs/ directory:

• agent_docs/architecture.md - Detailed architecture patterns and data models
• agent_docs/tech_stack.md - Technology stack details and component patterns
• agent_docs/development.md - Development workflows, testing, and common tasks
• agent_docs/code_style.md - Code patterns and best practices (read only when actively coding)

After finishing all code edits, run yarn lint:fix to auto-fix formatting and lint issues across all packages. Pre-commit hooks handle this when committing, but if you finish edits without committing, run yarn lint:fix before stopping.

Key Principles

1. Multi-tenancy: All data is scoped to Team - ensure proper filtering
2. Type safety: Use TypeScript strictly; Zod schemas for validation
3. Existing patterns: Follow established patterns in the codebase - explore similar files before implementing
4. Component size: Keep files under 300 lines; break down large components
5. UI Components: Use custom Button/ActionIcon variants (primary, secondary, danger) - see agent_docs/code_style.md for required patterns
6. Testing: Tests live in __tests__/ directories; use Jest for unit/integration tests

Running Tests

Each package has different test commands available:

packages/app (unit tests only):

cd packages/app
yarn ci:unit           # Run unit tests
yarn dev:unit          # Watch mode for unit tests

packages/api (integration tests only):

make dev-int-build                  # Build dependencies (run once before tests)
make dev-int FILE=<TEST_FILE_NAME>  # Spins up Docker services and runs tests.
                                    # Ctrl-C to stop and wait for all services to tear down.

packages/common-utils (both unit and integration tests):

cd packages/common-utils
yarn ci:unit           # Run unit tests
yarn dev:unit          # Watch mode for unit tests
yarn ci:int            # Run integration tests
yarn dev:int           # Watch mode for integration tests

To run a specific test file or pattern:

yarn ci:unit <path/to/test.ts>                           # Run specific test file
yarn ci:unit --testNamePattern="test name pattern"       # Run tests matching pattern

Lint & type check across all packages:

make ci-lint        # Lint + TypeScript check across all packages
make ci-unit        # Unit tests across all packages

E2E tests (Playwright):

# First-time setup (install Chromium browser):
cd packages/app && yarn playwright install chromium

# Run all E2E tests:
make e2e

# Run a specific test file (dev mode: hot reload):
make dev-e2e FILE=navigation                    # Match files containing "navigation"
make dev-e2e FILE=navigation GREP="help menu"   # Also filter by test name
make dev-e2e GREP="should navigate"             # Filter by test name across all files
make dev-e2e FILE=navigation REPORT=1           # Open HTML report after run
make dev-e2e-clean                               # Remove test artifacts

After modifying shared E2E test utilities (page objects in tests/e2e/page-objects/, components in tests/e2e/components/, or app source files that add/change data-testid attributes), run the full E2E suite — not just new tests — to catch regressions in existing specs.

Reviewing or Writing E2E Tests

Before writing new E2E tests or reviewing a PR that adds them:

1. Map existing coverage first. Read every spec file under tests/e2e/features/ and build a list of what behavior each test exercises (not just its name — what it actually asserts).
2. Check each new test against that map. A test is redundant if its assertions are a subset of an existing test's, even when the existing test is part of a larger workflow. Do not add isolated tests for behavior already covered by comprehensive tests.
3. Present the overlap analysis before making changes. Show the full existing-vs-new matrix and get alignment, then write or modify code.
4. One spec file per feature area. Do not split tests into foo.spec.ts and foo-extended.spec.ts. If a file gets too long, split by sub-feature (e.g., dashboard.spec.ts for workflows, dashboard-interactions.spec.ts for isolated CRUD), not by "basic vs extended."

Important Context

• Authentication: Passport.js with team-based access control
• State management: Jotai (client), TanStack Query (server), URL params (filters)
• UI library: Mantine components are the standard (not custom UI)
• Database patterns: MongoDB for metadata with Mongoose, ClickHouse for telemetry queries

PR Hygiene for Agent-Generated Code

When using agentic tools to generate PRs, follow these practices to keep reviews efficient and accurate:

1. Scope PRs to a single logical change, even if the agent can produce more in one session. Smaller, focused PRs move through the review pipeline faster and are easier to classify accurately.

2. Write the PR description to explain intent (the "why"), not just what changed. Reviewers need to understand the goal to catch cases where the agent solved the wrong problem or made a plausible-but-wrong trade-off.

3. Name agent-generated branches with a claude/, agent/, or ai/ prefix (e.g., claude/add-rate-limiting). This allows the PR triage classifier to apply appropriate scrutiny and lets reviewers calibrate their attention.

4. Write or update tests alongside the implementation, not after. Configure your agent to produce tests before writing implementation code. See the Testing section below for the commands to use.

GitHub Action Workflow (when invoked via @claude)

When working on issues or PRs through the GitHub Action:

1. Before writing any code, post a comment outlining your implementation plan — which files you'll change, what approach you'll take, and any trade-offs or risks. Use gh issue comment for issues or gh pr comment for PRs.

2. After making any code changes, always run these in order and fix any failures before opening a PR:

   • make ci-lint — lint + TypeScript type check
   • make ci-unit — unit tests

3. Write a clear PR description explaining what changed and why.

Git Commits

When committing code, use the git author's default profile (name and email from git config). Do not add Co-Authored-By trailers.

Pre-commit hooks must pass before committing. Do not use --no-verify to skip hooks. If the pre-commit hook fails (e.g. due to husky not being set up in a worktree), run npx lint-staged manually before committing to ensure lint and formatting checks pass. Fix any issues before creating the commit.

Merge Conflict Resolution

1. Never blindly pick a side. Read both sides of every conflict to understand the intent of each change before choosing a resolution.

2. Refactor/move conflicts require extra verification. When one side refactored, moved, or extracted code (e.g., inline components to separate files), always diff the discarded side against the destination files before declaring the conflict resolved. Code can diverge after extraction — the other branch may have made fixes or additions that the extracting branch never picked up. A naive "keep ours" resolution silently drops those changes.

3. Verify the result compiles. After resolving, check for missing imports, broken references, or type errors introduced by the resolution — especially when discarding a side that added new dependencies or exports.

4. Ask for help when uncertain. If you are not 100% confident about which side to keep, or whether a change can be safely discarded, stop and ask for manual intervention rather than guessing. A wrong guess silently breaks things; asking is always cheaper than debugging later.

Cursor Cloud specific instructions

Docker requirement

Docker must be installed and running before starting the dev stack or running integration/E2E tests. The VM update script handles yarn install and yarn build:common-utils, but Docker daemon startup is a prerequisite that must already be available.

Starting the dev stack

yarn dev uses sh -c to source scripts/dev-env.sh, which contains bash-specific syntax (BASH_SOURCE). On systems where /bin/sh is dash (e.g. Ubuntu), this fails with "Bad substitution". Work around it by running with bash directly:

bash -c 'export PATH="/workspace/node_modules/.bin:$PATH" && source ./scripts/dev-env.sh && yarn build:common-utils && dotenvx run --convention=nextjs -- docker compose -p "$HDX_DEV_PROJECT" -f docker-compose.dev.yml up -d && yarn app:dev'

Port isolation assigns a slot based on the worktree directory name. In the default /workspace directory, the slot is 76, so services are at:

• App: http://localhost:30276
• API: http://localhost:30176
• ClickHouse: http://localhost:30576
• MongoDB: localhost:30476

Key commands reference

See AGENTS.md above and agent_docs/development.md for the full command reference. Quick summary:

• make ci-lint — lint + TypeScript type check
• make ci-unit — unit tests (all packages)
• make dev-int FILE=<name> — integration tests (spins up Docker services)
• make dev-e2e FILE=<name> — E2E tests (Playwright)

First-time registration

When the dev stack starts fresh (empty MongoDB), the app shows a registration page. Create any account to get started — no external auth provider is needed.

---

Need more details? Check the agent_docs/ directory or ask which documentation to read.