Elgato_dark/hyperdx
1
0
Fork 0
mirror of https://github.com/hyperdxio/hyperdx synced 2026-04-21 21:37:41 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 6
hyperdx/agent_docs/development.md
Warren Lee b9c9682972
Enable parallel integration testing across multiple worktrees (#1917) 
## Summary

- Enable multiple agents/developers to run `make dev-int` simultaneously from different git worktrees without Docker port conflicts
- Compute a deterministic port offset (0-99) from the worktree directory name via `cksum`, giving each worktree its own isolated Docker Compose project and port range
- Switch `.env.test` files to use `${HDX_CI_*:-default}` variable expansion (powered by `dotenv-expand`) so test processes connect to the correct dynamic ports

## How it works

Each worktree gets a unique **slot** derived from its directory name. All service ports are offset by that slot:

| Service         | Base port | Example (slot 68) |
|-----------------|-----------|-------------------|
| ClickHouse HTTP | 18123     | 18191             |
| MongoDB         | 39999     | 40067             |
| API test server | 19000     | 19068             |
| OpAMP           | 14320     | 14388             |

Docker Compose project names are also unique (`int-<slot>`), isolating containers and networks.

Backward compatible — when no `HDX_CI_*` env vars are set, all ports fall back to their original defaults.

## Changes

- **Makefile**: Added `HDX_CI_SLOT` computation and dynamic project names/ports for all `dev-int` targets
- **docker-compose.ci.yml**: Ports use `${HDX_CI_*:-default}` env vars; removed unused OTel collector published port; removed hardcoded network name (auto-generated from project name)
- **packages/api/.env.test** / **packages/common-utils/.env.test**: Ports use `${HDX_CI_*:-default}` expansion syntax
- **packages/api/jest.config.js** / **packages/common-utils/jest.int.config.js**: Switched from `dotenv/config` to `dotenv-expand/config` to enable variable expansion
- **packages/api/package.json** / **packages/common-utils/package.json**: Added `dotenv-expand` devDependency
- **agent_docs/development.md**: Documented multi-agent worktree support

## Testing

Ran full Alert integration test suite (`make dev-int FILE=alerts`) — **6 test suites, 150 tests passed** on slot 68 with dynamic ports.
2026-03-16 19:42:08 +00:00

4.4 KiB
Raw Blame History

Development Workflows

Setup Commands

# Install dependencies and setup hooks
yarn setup

# Start full development stack (Docker + local services)
yarn dev

Key Development Scripts

  • yarn app:dev: Start API, frontend, alerts task, and common-utils in watch mode
  • yarn lint: Run linting across all packages
  • yarn dev:int: Run integration tests in watch mode
  • yarn dev:unit: Run unit tests in watch mode (per package)
  • yarn test:e2e: Run Playwright E2E tests (in packages/app)
  • yarn test:e2e:ci: Run Playwright E2E tests in CI Docker environment (in packages/app)

Environment Configuration

  • .env.development: Development environment variables
  • Docker Compose manages ClickHouse, MongoDB, OTel Collector
  • Hot reload enabled for all services in development

Testing Strategy

Testing Tools

  • Unit Tests: Jest with TypeScript support
  • Integration Tests: Jest with database fixtures
  • Frontend Testing: React Testing Library + Jest
  • E2E Testing: Playwright (frontend) and Custom smoke tests with BATS (ingestion)

Testing Patterns

  • TDD Approach: Write tests before implementation for new features
  • Test organization: Tests co-located with source files in __tests__/ directories
  • Mocking: MSW for API mocking in frontend tests
  • Database testing: Isolated test databases with fixtures

CI / Integration Testing

For integration testing:

# Build dependencies (run once before first test run)
make dev-int-build

# Run API integration tests (spins up Docker services, runs tests, tears down)
make dev-int FILE=<TEST_FILE_NAME>

# Run common-utils integration tests
make dev-int-common-utils FILE=<TEST_FILE_NAME>

Multi-agent / worktree support:

The make dev-int command automatically assigns unique Docker ports per worktree directory, so multiple agents can run integration tests in parallel without port conflicts.

  • A deterministic slot (0-99) is computed from the worktree directory name
  • Each slot gets its own Docker Compose project name and port range
  • Override the slot manually: make dev-int HDX_CI_SLOT=5 FILE=alerts
  • The slot and assigned ports are printed when dev-int starts

Port mapping (base + slot):

Service Default port (slot 0) Variable
ClickHouse HTTP 18123 HDX_CI_CH_PORT
MongoDB 39999 HDX_CI_MONGO_PORT
API test server 19000 HDX_CI_API_PORT
OpAMP 14320 HDX_CI_OPAMP_PORT

CI Testing Notes:

  • Uses separate Docker Compose configuration (docker-compose.ci.yml)
  • Isolated test environment with unique -p int-<slot> project name
  • Includes all necessary services (ClickHouse, MongoDB, OTel Collector)
  • Tests run against real database instances for accurate integration testing

Common Development Tasks

Adding New Features

  1. API First: Define API endpoints and data models
  2. Database Models: Create/update Mongoose schemas and ClickHouse queries
  3. Frontend Integration: Build UI components and integrate with API
  4. Testing: Add unit and integration tests
  5. Documentation: Update relevant docs

Debugging

  • Check browser and server console output for errors, warnings, or relevant logs
  • Add targeted logging to trace execution and variable states
  • For persistent issues, check fixes/ directory for documented solutions
  • Document complex fixes in fixes/ directory with descriptive filenames

Code Quality

Pre-commit Hooks

The project uses Husky + lint-staged to automatically run:

  • Prettier for formatting
  • ESLint for linting
  • API doc generation (for external API changes)

These run automatically on git commit for staged files.

Manual Linting (if needed)

If you need to manually lint:

# Per-package linting with auto-fix
cd packages/app && yarn run lint:fix
cd packages/api && yarn run lint:fix
cd packages/common-utils && yarn lint:fix

# Check all packages
yarn run lint

File Locations Quick Reference

  • Config: packages/api/src/config.ts, packages/app/next.config.mjs, docker-compose.dev.yml
  • Models: packages/api/src/models/
  • API Routes: packages/api/src/routers/
  • Controllers: packages/api/src/controllers/
  • Pages: packages/app/pages/
  • Components: packages/app/src/
  • Shared Utils: packages/common-utils/src/