## 🎯 Overview This PR introduces a comprehensive Docker Compose configuration for running a complete local DataHaven network, making it significantly easier for developers to spin up and test the entire stack locally. ## 🏗️ Architecture ```mermaid graph TB subgraph "DataHaven Network (Docker)" subgraph "Consensus Layer" Alice["🔷 Alice (Validator)<br/>:9944 RPC<br/>4 Keys: GRAN, BABE, IMON, BEEF"] Bob["🔷 Bob (Validator)<br/>:9945 RPC<br/>4 Keys: GRAN, BABE, IMON, BEEF"] Alice <-->|P2P/mDNS| Bob end subgraph "Storage Provider Layer" MSP["💾 MSP (Charlie)<br/>:9946 RPC<br/>1 GiB Storage<br/>1 Key: BCSV"] BSP01["💾 BSP01 (Dave)<br/>:9947 RPC<br/>1 GiB Storage<br/>1 Key: BCSV"] BSP02["💾 BSP02 (Eve)<br/>:9948 RPC<br/>1 GiB Storage<br/>1 Key: BCSV"] end subgraph "Monitoring Layer" Indexer["📊 Indexer<br/>:9949 RPC<br/>Full Mode<br/>No Keys"] Fisherman["🎣 Fisherman (Gustavo)<br/>:9950 RPC<br/>Storage Monitor<br/>1 Key: BCSV"] DB["🗄️ PostgreSQL<br/>:5432<br/>indexer/datahaven"] end Alice -.->|Syncs| MSP Bob -.->|Syncs| MSP Alice -.->|Syncs| BSP01 Bob -.->|Syncs| BSP02 Alice -.->|Syncs| Indexer Bob -.->|Syncs| Fisherman MSP -.->|Monitors| Fisherman BSP01 -.->|Monitors| Fisherman BSP02 -.->|Monitors| Fisherman Indexer -->|Writes| DB Fisherman -->|Writes| DB end style Alice fill:#4a90e2,stroke:#2e5c8a,color:#fff style Bob fill:#4a90e2,stroke:#2e5c8a,color:#fff style MSP fill:#50c878,stroke:#2d7a4a,color:#fff style BSP01 fill:#50c878,stroke:#2d7a4a,color:#fff style BSP02 fill:#50c878,stroke:#2d7a4a,color:#fff style Indexer fill:#f5a623,stroke:#b87818,color:#fff style Fisherman fill:#f5a623,stroke:#b87818,color:#fff style DB fill:#9b59b6,stroke:#6c3a82,color:#fff ``` **Legend:** - 🔷 Validators - Consensus and block production - 💾 Storage Providers - File storage and retrieval - 📊 Indexer - Full blockchain indexing - 🎣 Fisherman - Storage provider monitoring - 🗄️ PostgreSQL - Database for indexer/fisherman - `<-->` P2P Communication | `-.->` Network Sync | `-->` Database Connection ## ✨ What's Included ### Core Network (8 Services) - **2 Validator Nodes** (Alice & Bob) - Consensus and block production - **1 Main Storage Provider (MSP)** - Charlie with 1 GiB storage capacity - **2 Backup Storage Providers (BSPs)** - Dave and Eve with 1 GiB each - **1 StorageHub Indexer** - Full blockchain indexer with PostgreSQL - **1 Fisherman Node** - Gustavo monitoring storage provider behavior - **1 PostgreSQL Database** - Shared database for indexer and fisherman ### Key Features ✅ **Automated Key Injection** - All validator and storage provider keys automatically injected on startup ✅ **Health Checks** - Validators have health checks that verify RPC port is listening before dependent services start ✅ **Orchestrated Startup** - Services start in correct order with health-based dependencies ✅ **Persistent Storage** - Chain data, keystores, and database all persisted in Docker volumes ✅ **Unified Entrypoint Script** - Single script handles all node types (validator, MSP, BSP, fisherman) ✅ **Proper User Permissions** - Root for setup, switches to datahaven user for node execution ✅ **mDNS Peer Discovery** - Nodes automatically discover each other on the Docker network ✅ **Comprehensive Documentation** - Full setup guide, troubleshooting, and verification steps ### 🔄 Startup Sequence The network starts in a carefully orchestrated sequence to ensure stability: 1. **Alice (Validator)** starts first - Injects 4 keys (GRAN, BABE, IMON, BEEF) - Health check waits for RPC port 9944 to be listening - Uses `/proc/net/tcp` for minimal-dependency port checking 2. **Bob (Validator)** waits for Alice to be healthy - Ensures at least one validator is fully operational - Enables block production to start immediately 3. **Storage Providers & Monitoring** wait for both validators to be healthy - MSP, BSP01, BSP02 start after validators are ready - Indexer and Fisherman wait for validators before syncing - PostgreSQL starts independently with its own health check This dependency chain prevents race conditions and ensures reliable network formation. ## 🚀 Quick Start ```bash cd operator # Build the binary (development mode with fast blocks) ./scripts/docker-prepare.sh --fast # Start the entire network docker-compose up -d # View logs docker-compose logs -f # Check service health docker-compose ps # Stop the network docker-compose down -v ``` ## 📋 Port Mappings | Service | RPC/WebSocket | Prometheus | P2P | Database/API | |---------|---------------|------------|-----|--------------| | Alice | 9944 | 9615 | 30333 | - | | Bob | 9945 | 9616 | 30334 | - | | MSP | 9946 | 9617 | 30335 | - | | BSP01 | 9947 | 9618 | 30336 | - | | BSP02 | 9948 | 9619 | 30337 | - | | Indexer | 9949 | 9620 | 30338 | - | | Fisherman | 9950 | 9621 | 30339 | - | | PostgreSQL | - | - | - | 5432 | ## 🔑 Key Injection All cryptographic keys are automatically injected on startup: **Validators (Alice & Bob)** - 4 keys each: - GRANDPA (ed25519) - Finality - BABE (sr25519) - Block authoring - ImOnline (sr25519) - Heartbeat - BEEFY (ecdsa) - Bridge consensus **Storage Providers (MSP, BSPs, Fisherman)** - 1 key each: - BCSV (ecdsa) - Storage provider identity **Indexer** - No keys required (non-validating node) ## 🩺 Health Checks Validator nodes (Alice & Bob) implement health checks to ensure proper startup sequencing: **Health Check Method:** - Reads `/proc/net/tcp` directly to check if RPC port is listening - Zero dependencies - works in minimal debian:stable-slim containers - Converts port to hex and searches for LISTEN state (0A) **Configuration:** - Start period: 30s (allows node initialization) - Interval: 10s (check every 10 seconds) - Timeout: 5s (per health check) - Retries: 5 (must pass 5 consecutive checks) **Benefits:** - Prevents dependent services from starting before validators are ready - Eliminates race conditions during network formation - No additional packages required in Docker image ## 🍎 macOS Requirement **Important:** On Docker Desktop for macOS, you must use the **experimental DockerVMM** virtualization framework: 1. Open Docker Desktop settings 2. Go to "General" tab 3. Enable "Use experimental virtualization framework (DockerVMM)" 4. Restart Docker Desktop The default Apple Virtualization Framework causes networking issues with P2P connections. ## 📁 Files Changed - `operator/docker-compose.yml` - Main orchestration configuration - `operator/scripts/docker-entrypoint.sh` - Unified key injection script - `operator/scripts/docker-prepare.sh` - Binary build helper - `operator/scripts/docker-healthcheck.sh` - Health check script for validators - `operator/DOCKER-COMPOSE.md` - Comprehensive documentation ## 🔍 Testing The configuration has been tested on: - ✅ Docker Desktop for macOS (with DockerVMM) - ✅ Docker on Linux/Ubuntu All nodes successfully: - Inject required keys - Pass health checks - Discover peers via mDNS - Sync blocks and finalize - Connect to PostgreSQL database (indexer/fisherman) ## 📝 Notes - All settings are configured for **local development only** - Uses well-known test seed phrase (⚠️ never use in production!) - RPC exposed without authentication - Unsafe flags enabled for convenience --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|---|---|---|
| .github | ||
| contracts | ||
| deploy | ||
| docker | ||
| operator | ||
| resources | ||
| test | ||
| tools | ||
| .gitignore | ||
| .gitmodules | ||
| biome.json | ||
| CLAUDE.md | ||
| file_header.txt | ||
| LICENSE | ||
| README.md | ||
| taplo.toml | ||
DataHaven 🫎
An EVM-compatible Substrate blockchain secured by EigenLayer, bridging Ethereum and Substrate ecosystems through trustless cross-chain communication.
Overview
DataHaven is an EigenLayer Actively Validated Service (AVS) that combines:
- EVM Compatibility: Full Ethereum support via Frontier pallets for smart contracts and dApps
- EigenLayer Security: Validator set secured by Ethereum's economic security through restaking
- Cross-chain Bridge: Seamless asset and message transfers with Ethereum via Snowbridge
- Dynamic Validators: Operator registry managed on-chain through EigenLayer contracts
- Performance Rewards: Validator incentives distributed cross-chain from Ethereum
Architecture
DataHaven bridges two major blockchain ecosystems:
┌───────────────────────────────────────────────────────────────┐
│ Ethereum (L1) │
│ ┌────────────────────────────────────────────────────────┐ │
│ │ EigenLayer AVS Contracts │ │
│ │ • DataHavenServiceManager (operator lifecycle) │ │
│ │ • RewardsRegistry (performance tracking) │ │
│ │ • VetoableSlasher (misbehavior penalties) │ │
│ └────────────────────────────────────────────────────────┘ │
│ ↕ │
│ Snowbridge Protocol │
└───────────────────────────────────────────────────────────────┘
↕
┌───────────────────────────────────────────────────────────────┐
│ DataHaven (Substrate) │
│ ┌────────────────────────────────────────────────────────┐ │
│ │ Custom Pallets │ │
│ │ • External Validators (sync validator set) │ │
│ │ • Native Transfer (cross-chain tokens) │ │
│ │ • Rewards (distribute validator rewards) │ │
│ │ • Frontier (EVM compatibility) │ │
│ └────────────────────────────────────────────────────────┘ │
└───────────────────────────────────────────────────────────────┘
Repository Structure
datahaven/
├── contracts/ # EigenLayer AVS smart contracts
│ ├── src/ # Service Manager, Rewards Registry, Slasher
│ ├── script/ # Deployment scripts
│ └── test/ # Foundry test suites
├── operator/ # Substrate-based DataHaven node
│ ├── node/ # Node implementation & chain spec
│ ├── pallets/ # Custom pallets (validators, rewards, transfers)
│ └── runtime/ # Runtime configurations (mainnet/stagenet/testnet)
├── test/ # E2E testing framework
│ ├── suites/ # Integration test scenarios
│ ├── framework/ # Test utilities and helpers
│ └── launcher/ # Network deployment automation
├── deploy/ # Kubernetes deployment charts
│ ├── charts/ # Helm charts for nodes and relayers
│ └── environments/ # Environment-specific configurations
├── tools/ # GitHub automation and release scripts
└── .github/ # CI/CD workflows
Each directory contains its own README with detailed information. See:
- contracts/README.md - Smart contract development
- operator/README.md - Node building and runtime development
- test/README.md - E2E testing and network deployment
- deploy/README.md - Kubernetes deployment
- tools/README.md - Development tools
Quick Start
Prerequisites
- Kurtosis - Network orchestration
- Bun v1.3.2+ - TypeScript runtime
- Docker - Container management
- Foundry - Solidity toolkit
- Rust - For building the operator
- Helm - Kubernetes deployments (optional)
- Zig - For macOS cross-compilation (macOS only)
Launch Local Network
The fastest way to get started is with the interactive CLI:
cd test
bun i # Install dependencies
bun cli launch # Interactive launcher with prompts
This deploys a complete environment including:
- Ethereum network: 2x EL clients (reth), 2x CL clients (lodestar)
- Block explorers: Blockscout (optional), Dora consensus explorer
- DataHaven node: Single validator with fast block times
- AVS contracts: Deployed and configured on Ethereum
- Snowbridge relayers: Bidirectional message passing
For more options and detailed instructions, see the test README.
Run Tests
cd test
bun test:e2e # Run all integration tests
bun test:e2e:parallel # Run with limited concurrency
Development Workflows
Smart Contract Development:
cd contracts
forge build # Compile contracts
forge test # Run contract tests
Node Development:
cd operator
cargo build --release --features fast-runtime
cargo test
./scripts/run-benchmarks.sh
After Making Changes:
cd test
bun generate:wagmi # Regenerate contract bindings
bun generate:types # Regenerate runtime types
Key Features
EVM Compatibility
Full Ethereum Virtual Machine support via Frontier pallets:
- Deploy Solidity smart contracts
- Use existing Ethereum tooling (MetaMask, Hardhat, etc.)
- Compatible with ERC-20, ERC-721, and other standards
EigenLayer Integration
Validator security anchored to Ethereum:
- Operators register via
DataHavenServiceManagercontract - Economic security through ETH restaking
- Slashing protection with veto period via
VetoableSlasher - Performance-based rewards through
RewardsRegistry
Cross-chain Communication
Trustless bridging via Snowbridge:
- Native token transfers between Ethereum ↔ DataHaven
- Cross-chain message passing
- Finality proofs via BEEFY consensus
- Three specialized relayers (beacon, BEEFY, execution)
Dynamic Validator Set
Validator management synchronized with Ethereum:
- EigenLayer operator registry as source of truth
- On-chain validator set updates via External Validators pallet
- Automatic consensus participation changes
- Cross-chain coordination for validator lifecycle
Docker Images
Production images published to DockerHub.
Build optimizations:
- sccache - Rust compilation caching
- cargo-chef - Dependency layer caching
- BuildKit cache mounts - External cache restoration
Build locally:
cd test
bun build:docker:operator # Creates datahavenxyz/datahaven:local
Development Environment
VS Code Configuration
IDE configurations are excluded from version control for personalization, but these settings are recommended for optimal developer experience. Add to your .vscode/settings.json:
Rust Analyzer:
{
"rust-analyzer.linkedProjects": ["./operator/Cargo.toml"],
"rust-analyzer.cargo.allTargets": true,
"rust-analyzer.procMacro.enable": false,
"rust-analyzer.server.extraEnv": {
"CARGO_TARGET_DIR": "target/.rust-analyzer",
"SKIP_WASM_BUILD": 1
},
"rust-analyzer.diagnostics.disabled": ["unresolved-macro-call"],
"rust-analyzer.cargo.buildScripts.enable": false
}
Optimizations:
- Links
operator/directory as the primary Rust project - Disables proc macros and build scripts for faster analysis (Substrate macros are slow)
- Uses dedicated target directory to avoid conflicts
- Skips WASM builds during development
Solidity (Juan Blanco's extension):
{
"solidity.formatter": "forge",
"solidity.compileUsingRemoteVersion": "v0.8.28+commit.7893614a",
"[solidity]": {
"editor.defaultFormatter": "JuanBlanco.solidity"
}
}
Note: Solidity version must match foundry.toml
TypeScript (Biome):
{
"biome.lsp.bin": "test/node_modules/.bin/biome",
"[typescript]": {
"editor.defaultFormatter": "biomejs.biome",
"editor.codeActionsOnSave": {
"source.organizeImports.biome": "always"
}
}
}
CI/CD
Local CI Testing
Run GitHub Actions workflows locally using act:
# Run E2E workflow
act -W .github/workflows/e2e.yml -s GITHUB_TOKEN="$(gh auth token)"
# Run specific job
act -W .github/workflows/e2e.yml -j test-job-name
Automated Workflows
The repository includes GitHub Actions for:
- E2E Testing: Full integration tests on PR and main branch
- Contract Testing: Foundry test suites for smart contracts
- Rust Testing: Unit and integration tests for operator
- Docker Builds: Multi-platform image builds with caching
- Release Automation: Version tagging and changelog generation
See .github/workflows/ for workflow definitions.
Contributing
Development Cycle
- Make Changes: Edit contracts, runtime, or tests
- Run Tests: Component-specific tests (
forge test,cargo test) - Regenerate Types: Update bindings if contracts/runtime changed
- Integration Test: Run E2E tests to verify cross-component behavior
- Code Quality: Format and lint (
cargo fmt,forge fmt,bun fmt:fix)
Common Pitfalls
- Type mismatches: Regenerate with
bun generate:typesafter runtime changes - Contract changes not reflected: Run
bun generate:wagmiafter modifications - Kurtosis issues: Ensure Docker is running and Kurtosis engine is started
- Slow development: Use
--features fast-runtimefor shorter epochs/eras (block time stays 6s) - Network launch hangs: Check Blockscout - forge output can appear frozen
See CLAUDE.md for detailed development guidance.
License
GPL-3.0 - See LICENSE file for details