## Summary
- Isolate dev, E2E, and integration test environments so multiple git worktrees can run all three simultaneously without port conflicts
- Each worktree gets a deterministic slot (0-99) with unique port ranges: dev (30100-31199), E2E (20320-21399), CI integration (14320-40098)
- Dev portal dashboard (http://localhost:9900) auto-discovers all running stacks, streams logs, and provides a History tab for past run logs
## Port Isolation
| Environment | Port Range | Project Name |
|---|---|---|
| Dev stack | 30100-31199 | `hdx-dev-<slot>` |
| E2E tests | 20320-21399 | `e2e-<slot>` |
| CI integration | 14320-40098 | `int-<slot>` |
All three can run simultaneously from the same worktree with zero port conflicts.
## Dev Portal Features
**Live tab:**
- Auto-discovers dev, E2E, and integration Docker containers + local services (API, App)
- Groups all environments for the same worktree into a single card
- SSE log streaming with ANSI color rendering, capped at 5000 lines
- Auto-starts in background from `make dev`, `make dev-e2e`, `make dev-int`
**History tab:**
- Logs archived to `~/.config/hyperdx/dev-slots/<slot>/history/` on exit (instead of deleted)
- Each archived run includes `meta.json` with worktree/branch metadata
- Grouped by worktree with collapsible cards, search by worktree/branch
- View any past log file in the same log panel, delete individual runs or clear all
- Custom dark-themed confirm modal (no native browser dialogs)
## What Changed
- **`scripts/dev-env.sh`** — Slot-based port assignments, portal auto-start, log archival on exit
- **`scripts/test-e2e.sh`** — E2E port range (20320-21399), log capture via `tee`, portal auto-start, log archival
- **`scripts/ensure-dev-portal.sh`** — Shared singleton portal launcher (works sourced or executed)
- **`scripts/dev-portal/server.js`** — Discovery for dev/E2E/CI containers, history API (list/read/delete), local service port probing
- **`scripts/dev-portal/index.html`** — Live/History tabs, worktree-grouped cards, search, collapse/expand, custom confirm modal, ANSI color log rendering
- **`docker-compose.dev.yml`** — Parameterized ports/volumes/project name with `hdx.dev.*` labels
- **`packages/app/tests/e2e/docker-compose.yml`** — Updated to new E2E port defaults
- **`Makefile`** — `dev-int`/`dev-e2e` targets with log capture + portal auto-start; `dev-portal-stop`; `dev-clean` stops everything + wipes slot data
- **`.env` files** — Ports use `${VAR:-default}` syntax across dev, E2E, and CI environments
- **`agent_docs/development.md`** — Full documentation for isolation, port tables, E2E/CI port ranges
## How to Use
```bash
# Start dev stack (auto-starts portal)
make dev
# Run E2E tests (auto-starts portal, separate ports)
make dev-e2e FILE=navigation
# Run integration tests (auto-starts portal, separate ports)
make dev-int FILE=alerts
# All three can run simultaneously from the same worktree
# Portal at http://localhost:9900 shows everything
# Stop portal
make dev-portal-stop
# Clean up everything (all stacks + portal + history)
make dev-clean
```
## Dev Portal
<img width="1692" height="944" alt="image" src="https://github.com/user-attachments/assets/6ed388a3-43bc-4552-aa8d-688077b79fb7" />
<img width="1689" height="935" alt="image" src="https://github.com/user-attachments/assets/8677a138-0a40-4746-93ed-3b355c8bd45e" />
## Test Plan
- [x] Run `make dev` — verify services start with slot-assigned ports
- [x] Run `make dev` in a second worktree — verify different ports, no conflicts
- [x] Run `make dev-e2e` and `make dev-int` simultaneously — no port conflicts
- [x] Open http://localhost:9900 — verify all stacks grouped by worktree
- [x] Click a service to view logs — verify ANSI colors render correctly
- [x] Stop a stack — verify logs archived to History tab with correct worktree
- [x] History tab — search, collapse/expand, view archived logs, delete
- [x] `make dev-clean` — stops everything, wipes slot data and history
## Summary
- Adds worktree-aware port isolation for E2E tests, mirroring the existing `dev-int` slot mechanism so multiple agents/developers can run E2E tests in parallel without port conflicts
- Fixes the navigation E2E test that was broken by Live Tail URL updates swallowing client-side navigation
- Adds `dev-e2e` Makefile target for running specific tests with `FILE=` and `GREP=` filters, plus `REPORT=1` to open the HTML report after tests finish
## Port Isolation
Each worktree gets a deterministic slot (0–99) computed from its directory name. All E2E service ports are offset by that slot in the **44000–50100** range, avoiding collisions with `dev` (4317–27017) and `dev-int` (14320–40098).
| Service | Base + slot | Variable |
|---|---|---|
| ClickHouse HTTP | 48123 + slot | `HDX_E2E_CH_PORT` |
| ClickHouse Native | 49000 + slot | `HDX_E2E_CH_NATIVE_PORT` |
| MongoDB | 49998 + slot | `HDX_E2E_MONGO_PORT` |
| API server | 49100 + slot | `HDX_E2E_API_PORT` |
| App (fullstack) | 48081 + slot | `HDX_E2E_APP_PORT` |
| App (local) | 48001 + slot | `HDX_E2E_APP_LOCAL_PORT` |
| OpAMP | 44320 + slot | `HDX_E2E_OPAMP_PORT` |
## New Make Targets
```bash
make dev-e2e FILE=navigation # Run specific test file
make dev-e2e FILE=navigation GREP="help menu" # Filter by test name
make dev-e2e GREP="should navigate" # Grep across all files
make dev-e2e FILE=navigation REPORT=1 # Open HTML report after run
make dev-e2e-clean # Remove test artifacts
```
## Linear
https://linear.app/hyperdx/issue/HDX-3796
## Summary
This PR backports the claude E2E skill and a few E2E script improvements to the open source repo from the EE repo.
### Screenshots or video
### How to test locally or on Vercel
### References
- Linear Issue: n/a
- Related PRs:
TLDR: This PR changes playwright full-stack tests to run against a local clickhouse instance (with seeded data) instead of relying on the clickhouse demo server, which can be unpredictable at times. This workflow allows us to fully control the data to make tests more predictable.
This PR:
* Adds local CH instance to the e2e dockerfile
* Adds a schema creation script
* Adds a data seeding script
* Updates playwright config
* Updates various tests to change hardcoded fields, metrics, or areas relying on play demo data
* Updates github workflow to use the dockerfile instead of separate services
* Runs against a local clickhouse instead of the demo server
Fixes: HDX-3193
Enables broader testing
Fixes: HDX-3069
To test:
- By default `make e2e` runs playwright tests with a docker compose for mongo
- To test the local-only mode, run `make e2e local=true`
- Since we manage play.hyperdx.io, I envision us running both commands on release
