mirror of
https://github.com/hyperdxio/hyperdx
synced 2026-04-21 13:37:15 +00:00
6e8ddd3736
## Summary - Isolate dev, E2E, and integration test environments so multiple git worktrees can run all three simultaneously without port conflicts - Each worktree gets a deterministic slot (0-99) with unique port ranges: dev (30100-31199), E2E (20320-21399), CI integration (14320-40098) - Dev portal dashboard (http://localhost:9900) auto-discovers all running stacks, streams logs, and provides a History tab for past run logs ## Port Isolation | Environment | Port Range | Project Name | |---|---|---| | Dev stack | 30100-31199 | `hdx-dev-<slot>` | | E2E tests | 20320-21399 | `e2e-<slot>` | | CI integration | 14320-40098 | `int-<slot>` | All three can run simultaneously from the same worktree with zero port conflicts. ## Dev Portal Features **Live tab:** - Auto-discovers dev, E2E, and integration Docker containers + local services (API, App) - Groups all environments for the same worktree into a single card - SSE log streaming with ANSI color rendering, capped at 5000 lines - Auto-starts in background from `make dev`, `make dev-e2e`, `make dev-int` **History tab:** - Logs archived to `~/.config/hyperdx/dev-slots/<slot>/history/` on exit (instead of deleted) - Each archived run includes `meta.json` with worktree/branch metadata - Grouped by worktree with collapsible cards, search by worktree/branch - View any past log file in the same log panel, delete individual runs or clear all - Custom dark-themed confirm modal (no native browser dialogs) ## What Changed - **`scripts/dev-env.sh`** — Slot-based port assignments, portal auto-start, log archival on exit - **`scripts/test-e2e.sh`** — E2E port range (20320-21399), log capture via `tee`, portal auto-start, log archival - **`scripts/ensure-dev-portal.sh`** — Shared singleton portal launcher (works sourced or executed) - **`scripts/dev-portal/server.js`** — Discovery for dev/E2E/CI containers, history API (list/read/delete), local service port probing - **`scripts/dev-portal/index.html`** — Live/History tabs, worktree-grouped cards, search, collapse/expand, custom confirm modal, ANSI color log rendering - **`docker-compose.dev.yml`** — Parameterized ports/volumes/project name with `hdx.dev.*` labels - **`packages/app/tests/e2e/docker-compose.yml`** — Updated to new E2E port defaults - **`Makefile`** — `dev-int`/`dev-e2e` targets with log capture + portal auto-start; `dev-portal-stop`; `dev-clean` stops everything + wipes slot data - **`.env` files** — Ports use `${VAR:-default}` syntax across dev, E2E, and CI environments - **`agent_docs/development.md`** — Full documentation for isolation, port tables, E2E/CI port ranges ## How to Use ```bash # Start dev stack (auto-starts portal) make dev # Run E2E tests (auto-starts portal, separate ports) make dev-e2e FILE=navigation # Run integration tests (auto-starts portal, separate ports) make dev-int FILE=alerts # All three can run simultaneously from the same worktree # Portal at http://localhost:9900 shows everything # Stop portal make dev-portal-stop # Clean up everything (all stacks + portal + history) make dev-clean ``` ## Dev Portal <img width="1692" height="944" alt="image" src="https://github.com/user-attachments/assets/6ed388a3-43bc-4552-aa8d-688077b79fb7" /> <img width="1689" height="935" alt="image" src="https://github.com/user-attachments/assets/8677a138-0a40-4746-93ed-3b355c8bd45e" /> ## Test Plan - [x] Run `make dev` — verify services start with slot-assigned ports - [x] Run `make dev` in a second worktree — verify different ports, no conflicts - [x] Run `make dev-e2e` and `make dev-int` simultaneously — no port conflicts - [x] Open http://localhost:9900 — verify all stacks grouped by worktree - [x] Click a service to view logs — verify ANSI colors render correctly - [x] Stop a stack — verify logs archived to History tab with correct worktree - [x] History tab — search, collapse/expand, view archived logs, delete - [x] `make dev-clean` — stops everything, wipes slot data and history
148 lines
4.4 KiB
Markdown
148 lines
4.4 KiB
Markdown
# Contributing
|
|
|
|
## Architecture Overview
|
|
|
|
|
|
|
|
Service Descriptions:
|
|
|
|
- OpenTelemetry Collector (otel-collector): Receives OpenTelemetry data from
|
|
instrumented applications and forwards it to ClickHouse for storage. Includes
|
|
OpAMP supervisor that dynamically pulls configuration from HyperDX API.
|
|
- ClickHouse (ch-server): ClickHouse database, stores all telemetry.
|
|
- MongoDB (db): Stores user/saved search/alert/dashboard data.
|
|
- HyperDX API (api): Node.js API, executes ClickHouse queries on behalf of the
|
|
frontend and serves the frontend. serves the frontend. Can also run alert
|
|
checker.
|
|
- HyperDX UI (app): Next.js frontend, serves the UI.
|
|
|
|
## Development
|
|
|
|
Pre-requisites:
|
|
|
|
- Docker
|
|
- Node.js (`>=22`)
|
|
- Yarn (v4)
|
|
|
|
You can get started by deploying a complete development stack in dev mode.
|
|
|
|
```bash
|
|
yarn dev
|
|
```
|
|
|
|
This will start the Node.js API, Next.js frontend locally and the OpenTelemetry
|
|
collector and ClickHouse server in Docker.
|
|
|
|
Each worktree automatically gets unique ports so multiple developers (or agents)
|
|
can run `yarn dev` simultaneously without conflicts. A dev portal at
|
|
http://localhost:9900 auto-starts and shows all running stacks with their
|
|
assigned ports. Check the portal to find the URL for your instance.
|
|
|
|
To stop the stack:
|
|
|
|
```bash
|
|
yarn dev:down
|
|
```
|
|
|
|
To enable self-instrumentation and demo logs, you can set the `HYPERDX_API_KEY`
|
|
to your ingestion key (visit the Team settings page after creating your account).
|
|
|
|
To do this, create a `.env.local` file in the root of the project and add the
|
|
following:
|
|
|
|
```sh
|
|
HYPERDX_API_KEY=<YOUR_INGESTION_API_KEY_HERE>
|
|
```
|
|
|
|
Then restart the stack using `yarn dev`.
|
|
|
|
The core services are all hot-reloaded, so you can make changes to the code and
|
|
see them reflected in real-time.
|
|
|
|
### Volumes
|
|
|
|
The development stack mounts volumes locally for persisting storage under
|
|
`.volumes`. Each worktree gets its own volume directory (e.g.
|
|
`.volumes/ch_data_dev_89`). Clear the `.volumes` directory to reset ClickHouse
|
|
and MongoDB storage.
|
|
|
|
### Windows
|
|
|
|
If you are running WSL 2, Hot module reload on Nextjs (Frontend) does not work
|
|
out of the box on windows when run natively on docker. The fix here is to open
|
|
project directory in WSL and run the above docker compose commands directly in
|
|
WSL. Note that the project directory should not be under /mnt/c/ directory. You
|
|
can clone the git repo in /home/{username} for example.
|
|
|
|
To develop from WSL, follow instructions
|
|
[here](https://code.visualstudio.com/docs/remote/wsl).
|
|
|
|
## Testing
|
|
|
|
All test environments use slot-based port isolation, so they can run
|
|
simultaneously with the dev stack and across multiple worktrees.
|
|
|
|
### E2E Tests
|
|
|
|
E2E tests run against a full local stack (MongoDB + ClickHouse + API). Docker
|
|
must be running.
|
|
|
|
```bash
|
|
# Run all E2E tests
|
|
make e2e
|
|
|
|
# Run a specific spec file (dev mode: hot reload, containers kept running)
|
|
make dev-e2e FILE=search
|
|
|
|
# Run with grep pattern
|
|
make dev-e2e FILE=search GREP="filter"
|
|
|
|
# Run via script directly for more control
|
|
./scripts/test-e2e.sh --ui --last-failed
|
|
```
|
|
|
|
Tests live in `packages/app/tests/e2e/`. Page objects are in `page-objects/`,
|
|
shared components in `components/`.
|
|
|
|
### Integration Tests
|
|
|
|
```bash
|
|
# Build dependencies (run once before first test run)
|
|
make dev-int-build
|
|
|
|
# Run a specific test file
|
|
make dev-int FILE=checkAlerts
|
|
```
|
|
|
|
### Unit Tests
|
|
|
|
To run unit tests or update snapshots, you can go to the package you want (ex.
|
|
common-utils) to test and run:
|
|
|
|
```bash
|
|
yarn dev:unit
|
|
```
|
|
|
|
## AI-Assisted Development
|
|
|
|
The repo ships with configuration for AI coding assistants that enables interactive browser-based E2E test generation and debugging via the [Playwright MCP server](https://github.com/microsoft/playwright-mcp).
|
|
|
|
### Claude Code
|
|
|
|
The project includes agents and skills for test generation, healing, and planning under `.claude/`. These are loaded automatically when you open the project in Claude Code. No additional setup required.
|
|
|
|
### Cursor
|
|
|
|
A Playwright MCP server config is included at `.cursor/mcp.json`. To activate it:
|
|
|
|
1. Open **Cursor Settings → Tools & MCP**
|
|
2. The `playwright-test` server should appear automatically from the project config
|
|
3. Enable it
|
|
|
|
This gives Cursor's AI access to a live browser for test exploration and debugging.
|
|
|
|
## Additional support
|
|
|
|
If you need help getting started,
|
|
[join our Discord](https://discord.gg/FErRRKU78j) and we're more than happy to
|
|
get you set up!
|