Elgato_dark/datahaven
1
0
Fork 0
mirror of https://github.com/datahaven-xyz/datahaven synced 2026-05-23 17:28:23 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
datahaven/operator/docker-compose.yml
Steve Degosserie a5522659bf
feat: ✨ Add Docker Compose setup for local DataHaven network (#314) 
## 🎯 Overview

This PR introduces a comprehensive Docker Compose configuration for
running a complete local DataHaven network, making it significantly
easier for developers to spin up and test the entire stack locally.

## 🏗️ Architecture

```mermaid
graph TB
    subgraph "DataHaven Network (Docker)"
        subgraph "Consensus Layer"
            Alice["🔷 Alice (Validator)<br/>:9944 RPC<br/>4 Keys: GRAN, BABE, IMON, BEEF"]
            Bob["🔷 Bob (Validator)<br/>:9945 RPC<br/>4 Keys: GRAN, BABE, IMON, BEEF"]
            Alice <-->|P2P/mDNS| Bob
        end

        subgraph "Storage Provider Layer"
            MSP["💾 MSP (Charlie)<br/>:9946 RPC<br/>1 GiB Storage<br/>1 Key: BCSV"]
            BSP01["💾 BSP01 (Dave)<br/>:9947 RPC<br/>1 GiB Storage<br/>1 Key: BCSV"]
            BSP02["💾 BSP02 (Eve)<br/>:9948 RPC<br/>1 GiB Storage<br/>1 Key: BCSV"]
        end

        subgraph "Monitoring Layer"
            Indexer["📊 Indexer<br/>:9949 RPC<br/>Full Mode<br/>No Keys"]
            Fisherman["🎣 Fisherman (Gustavo)<br/>:9950 RPC<br/>Storage Monitor<br/>1 Key: BCSV"]
            DB["🗄️ PostgreSQL<br/>:5432<br/>indexer/datahaven"]
        end

        Alice -.->|Syncs| MSP
        Bob -.->|Syncs| MSP
        Alice -.->|Syncs| BSP01
        Bob -.->|Syncs| BSP02
        Alice -.->|Syncs| Indexer
        Bob -.->|Syncs| Fisherman
        
        MSP -.->|Monitors| Fisherman
        BSP01 -.->|Monitors| Fisherman
        BSP02 -.->|Monitors| Fisherman
        
        Indexer -->|Writes| DB
        Fisherman -->|Writes| DB
    end

    style Alice fill:#4a90e2,stroke:#2e5c8a,color:#fff
    style Bob fill:#4a90e2,stroke:#2e5c8a,color:#fff
    style MSP fill:#50c878,stroke:#2d7a4a,color:#fff
    style BSP01 fill:#50c878,stroke:#2d7a4a,color:#fff
    style BSP02 fill:#50c878,stroke:#2d7a4a,color:#fff
    style Indexer fill:#f5a623,stroke:#b87818,color:#fff
    style Fisherman fill:#f5a623,stroke:#b87818,color:#fff
    style DB fill:#9b59b6,stroke:#6c3a82,color:#fff
```

**Legend:**
- 🔷 Validators - Consensus and block production
- 💾 Storage Providers - File storage and retrieval
- 📊 Indexer - Full blockchain indexing
- 🎣 Fisherman - Storage provider monitoring
- 🗄️ PostgreSQL - Database for indexer/fisherman
- `<-->` P2P Communication | `-.->` Network Sync | `-->` Database
Connection

## ✨ What's Included

### Core Network (8 Services)
- **2 Validator Nodes** (Alice & Bob) - Consensus and block production
- **1 Main Storage Provider (MSP)** - Charlie with 1 GiB storage
capacity
- **2 Backup Storage Providers (BSPs)** - Dave and Eve with 1 GiB each
- **1 StorageHub Indexer** - Full blockchain indexer with PostgreSQL
- **1 Fisherman Node** - Gustavo monitoring storage provider behavior
- **1 PostgreSQL Database** - Shared database for indexer and fisherman

### Key Features
✅ **Automated Key Injection** - All validator and storage provider keys
automatically injected on startup
✅ **Health Checks** - Validators have health checks that verify RPC port
is listening before dependent services start
✅ **Orchestrated Startup** - Services start in correct order with
health-based dependencies
✅ **Persistent Storage** - Chain data, keystores, and database all
persisted in Docker volumes
✅ **Unified Entrypoint Script** - Single script handles all node types
(validator, MSP, BSP, fisherman)
✅ **Proper User Permissions** - Root for setup, switches to datahaven
user for node execution
✅ **mDNS Peer Discovery** - Nodes automatically discover each other on
the Docker network
✅ **Comprehensive Documentation** - Full setup guide, troubleshooting,
and verification steps

### 🔄 Startup Sequence

The network starts in a carefully orchestrated sequence to ensure
stability:

1. **Alice (Validator)** starts first
   - Injects 4 keys (GRAN, BABE, IMON, BEEF)
   - Health check waits for RPC port 9944 to be listening
   - Uses `/proc/net/tcp` for minimal-dependency port checking

2. **Bob (Validator)** waits for Alice to be healthy
   - Ensures at least one validator is fully operational
   - Enables block production to start immediately

3. **Storage Providers & Monitoring** wait for both validators to be
healthy
   - MSP, BSP01, BSP02 start after validators are ready
   - Indexer and Fisherman wait for validators before syncing
   - PostgreSQL starts independently with its own health check

This dependency chain prevents race conditions and ensures reliable
network formation.

## 🚀 Quick Start

```bash
cd operator

# Build the binary (development mode with fast blocks)
./scripts/docker-prepare.sh --fast

# Start the entire network
docker-compose up -d

# View logs
docker-compose logs -f

# Check service health
docker-compose ps

# Stop the network
docker-compose down -v
```

## 📋 Port Mappings

| Service | RPC/WebSocket | Prometheus | P2P | Database/API |
|---------|---------------|------------|-----|--------------|
| Alice | 9944 | 9615 | 30333 | - |
| Bob | 9945 | 9616 | 30334 | - |
| MSP | 9946 | 9617 | 30335 | - |
| BSP01 | 9947 | 9618 | 30336 | - |
| BSP02 | 9948 | 9619 | 30337 | - |
| Indexer | 9949 | 9620 | 30338 | - |
| Fisherman | 9950 | 9621 | 30339 | - |
| PostgreSQL | - | - | - | 5432 |

## 🔑 Key Injection

All cryptographic keys are automatically injected on startup:

**Validators (Alice & Bob)** - 4 keys each:
- GRANDPA (ed25519) - Finality
- BABE (sr25519) - Block authoring
- ImOnline (sr25519) - Heartbeat
- BEEFY (ecdsa) - Bridge consensus

**Storage Providers (MSP, BSPs, Fisherman)** - 1 key each:
- BCSV (ecdsa) - Storage provider identity

**Indexer** - No keys required (non-validating node)

## 🩺 Health Checks

Validator nodes (Alice & Bob) implement health checks to ensure proper
startup sequencing:

**Health Check Method:**
- Reads `/proc/net/tcp` directly to check if RPC port is listening
- Zero dependencies - works in minimal debian:stable-slim containers
- Converts port to hex and searches for LISTEN state (0A)

**Configuration:**
- Start period: 30s (allows node initialization)
- Interval: 10s (check every 10 seconds)
- Timeout: 5s (per health check)
- Retries: 5 (must pass 5 consecutive checks)

**Benefits:**
- Prevents dependent services from starting before validators are ready
- Eliminates race conditions during network formation
- No additional packages required in Docker image

## 🍎 macOS Requirement

**Important:** On Docker Desktop for macOS, you must use the
**experimental DockerVMM** virtualization framework:

1. Open Docker Desktop settings
2. Go to "General" tab
3. Enable "Use experimental virtualization framework (DockerVMM)"
4. Restart Docker Desktop

The default Apple Virtualization Framework causes networking issues with
P2P connections.

## 📁 Files Changed

- `operator/docker-compose.yml` - Main orchestration configuration
- `operator/scripts/docker-entrypoint.sh` - Unified key injection script
- `operator/scripts/docker-prepare.sh` - Binary build helper
- `operator/scripts/docker-healthcheck.sh` - Health check script for
validators
- `operator/DOCKER-COMPOSE.md` - Comprehensive documentation

## 🔍 Testing

The configuration has been tested on:
- ✅ Docker Desktop for macOS (with DockerVMM)
- ✅ Docker on Linux/Ubuntu

All nodes successfully:
- Inject required keys
- Pass health checks
- Discover peers via mDNS
- Sync blocks and finalize
- Connect to PostgreSQL database (indexer/fisherman)

## 📝 Notes

- All settings are configured for **local development only**
- Uses well-known test seed phrase (⚠️ never use in production!)
- RPC exposed without authentication
- Unsafe flags enabled for convenience

---------

Co-authored-by: Claude <noreply@anthropic.com>
2025-11-22 14:07:46 +01:00

379 lines
No EOL
11 KiB
YAML
Raw Permalink Blame History

services:
alice:
build:
context: .
dockerfile: Dockerfile
image: datahavenxyz/datahaven:local
platform: linux/amd64
container_name: datahaven-alice
hostname: alice
user: "root" # Run as root to allow key injection, entrypoint switches to datahaven user
networks:
- datahaven-network
ports:
# Alice gets the standard port mappings
- "9944:9944" # WebSocket/RPC
- "9615:9615" # Prometheus metrics
- "30333:30333" # P2P networking
environment:
- NODE_NAME=alice
- NODE_TYPE=validator
- CHAIN=stagenet-local
- SEED=bottom drive obey lake curtain smoke basket hold race lonely fit walk
- RPC_PORT=9944
entrypoint: ["/scripts/docker-entrypoint.sh"]
command:
- --alice
- --chain=stagenet-local
- --unsafe-force-node-key-generation
- --base-path=/data
- --keystore-path=/data/keystore
- --validator
- --discover-local
- --no-prometheus
- --unsafe-rpc-external
- --rpc-cors=all
- --force-authoring
- --no-telemetry
- --enable-offchain-indexing=true
- --pool-type=fork-aware
volumes:
- ./scripts/docker-entrypoint.sh:/scripts/docker-entrypoint.sh:ro
- ./scripts/docker-healthcheck.sh:/scripts/docker-healthcheck.sh:ro
- alice-keystore:/data/keystore
healthcheck:
test: ["CMD-SHELL", "/scripts/docker-healthcheck.sh"]
interval: 10s
timeout: 5s
retries: 5
start_period: 30s
restart: unless-stopped
bob:
build:
context: .
dockerfile: Dockerfile
image: datahavenxyz/datahaven:local
platform: linux/amd64
container_name: datahaven-bob
hostname: bob
user: "root" # Run as root to allow key injection, entrypoint switches to datahaven user
networks:
- datahaven-network
ports:
# Bob gets different ports to avoid conflicts with Alice
- "9945:9944" # WebSocket/RPC
- "9616:9615" # Prometheus metrics
- "30334:30333" # P2P networking
environment:
- NODE_NAME=bob
- NODE_TYPE=validator
- CHAIN=stagenet-local
- SEED=bottom drive obey lake curtain smoke basket hold race lonely fit walk
- RPC_PORT=9944
entrypoint: ["/scripts/docker-entrypoint.sh"]
command:
- --bob
- --chain=stagenet-local
- --unsafe-force-node-key-generation
- --base-path=/data
- --keystore-path=/data/keystore
- --validator
- --discover-local
- --no-prometheus
- --unsafe-rpc-external
- --rpc-cors=all
- --no-telemetry
- --enable-offchain-indexing=true
- --pool-type=fork-aware
volumes:
- ./scripts/docker-entrypoint.sh:/scripts/docker-entrypoint.sh:ro
- ./scripts/docker-healthcheck.sh:/scripts/docker-healthcheck.sh:ro
- bob-keystore:/data/keystore
healthcheck:
test: ["CMD-SHELL", "/scripts/docker-healthcheck.sh"]
interval: 10s
timeout: 5s
retries: 5
start_period: 30s
restart: unless-stopped
depends_on:
alice:
condition: service_healthy
msp:
build:
context: .
dockerfile: Dockerfile
image: datahavenxyz/datahaven:local
platform: linux/amd64
container_name: datahaven-msp
hostname: msp
user: "root" # Run as root to allow key injection, entrypoint switches to datahaven user
networks:
- datahaven-network
ports:
# MSP gets different ports to avoid conflicts with validators
- "9946:9944" # WebSocket/RPC
- "9617:9615" # Prometheus metrics
- "30335:30333" # P2P networking
environment:
- NODE_NAME=charlie
- NODE_TYPE=msp
- SEED=bottom drive obey lake curtain smoke basket hold race lonely fit walk
entrypoint: ["/scripts/docker-entrypoint.sh"]
command:
- --name=msp
- --chain=stagenet-local
- --unsafe-force-node-key-generation
- --base-path=/data
- --keystore-path=/data/keystore
- --discover-local
- --unsafe-rpc-external
- --rpc-cors=all
- --no-prometheus
- --no-telemetry
- --enable-offchain-indexing=true
- --pool-type=fork-aware
- --provider
- --provider-type=msp
- --msp-charging-period=100
- --max-storage-capacity=1073741824
- --jump-capacity=104857600
- --msp-distribute-files
volumes:
- ./scripts/docker-entrypoint.sh:/scripts/docker-entrypoint.sh:ro
- msp-keystore:/data/keystore
restart: unless-stopped
depends_on:
alice:
condition: service_healthy
bob:
condition: service_healthy
bsp01:
build:
context: .
dockerfile: Dockerfile
image: datahavenxyz/datahaven:local
platform: linux/amd64
container_name: datahaven-bsp01
hostname: bsp01
user: "root" # Run as root to allow key injection, entrypoint switches to datahaven user
networks:
- datahaven-network
ports:
# BSP gets different ports to avoid conflicts with validators
- "9947:9944" # WebSocket/RPC
- "9618:9615" # Prometheus metrics
- "30336:30333" # P2P networking
environment:
- NODE_NAME=dave
- NODE_TYPE=bsp
- SEED=bottom drive obey lake curtain smoke basket hold race lonely fit walk
entrypoint: ["/scripts/docker-entrypoint.sh"]
command:
- --name=bsp01
- --chain=stagenet-local
- --unsafe-force-node-key-generation
- --base-path=/data
- --keystore-path=/data/keystore
- --discover-local
- --unsafe-rpc-external
- --rpc-cors=all
- --no-prometheus
- --no-telemetry
- --enable-offchain-indexing=true
- --pool-type=fork-aware
- --provider
- --provider-type=bsp
- --max-storage-capacity=1073741824
- --jump-capacity=104857600
volumes:
- ./scripts/docker-entrypoint.sh:/scripts/docker-entrypoint.sh:ro
- bsp01-keystore:/data/keystore
restart: unless-stopped
depends_on:
alice:
condition: service_healthy
bob:
condition: service_healthy
msp:
condition: service_started
bsp02:
build:
context: .
dockerfile: Dockerfile
image: datahavenxyz/datahaven:local
platform: linux/amd64
container_name: datahaven-bsp02
hostname: bsp02
user: "root" # Run as root to allow key injection, entrypoint switches to datahaven user
networks:
- datahaven-network
ports:
# BSP gets different ports to avoid conflicts with validators
- "9948:9944" # WebSocket/RPC
- "9619:9615" # Prometheus metrics
- "30337:30333" # P2P networking
environment:
- NODE_NAME=eve
- NODE_TYPE=bsp
- SEED=bottom drive obey lake curtain smoke basket hold race lonely fit walk
entrypoint: ["/scripts/docker-entrypoint.sh"]
command:
- --name=bsp02
- --chain=stagenet-local
- --unsafe-force-node-key-generation
- --base-path=/data
- --keystore-path=/data/keystore
- --discover-local
- --unsafe-rpc-external
- --rpc-cors=all
- --no-prometheus
- --no-telemetry
- --enable-offchain-indexing=true
- --pool-type=fork-aware
- --provider
- --provider-type=bsp
- --max-storage-capacity=1073741824
- --jump-capacity=104857600
volumes:
- ./scripts/docker-entrypoint.sh:/scripts/docker-entrypoint.sh:ro
- bsp02-keystore:/data/keystore
restart: unless-stopped
depends_on:
alice:
condition: service_healthy
bob:
condition: service_healthy
msp:
condition: service_started
postgres:
image: postgres:18-alpine
platform: linux/amd64
container_name: datahaven-postgres
hostname: postgres
networks:
- datahaven-network
ports:
- "5432:5432"
environment:
- POSTGRES_USER=indexer
- POSTGRES_PASSWORD=indexer
- POSTGRES_DB=datahaven
volumes:
- postgres-data:/var/lib/postgresql/data
restart: unless-stopped
healthcheck:
test: ["CMD-SHELL", "pg_isready -U indexer -d datahaven"]
interval: 10s
timeout: 5s
retries: 5
indexer:
build:
context: .
dockerfile: Dockerfile
image: datahavenxyz/datahaven:local
platform: linux/amd64
container_name: datahaven-indexer
hostname: indexer
networks:
- datahaven-network
ports:
- "9949:9944" # WebSocket/RPC
- "9620:9615" # Prometheus metrics
- "30338:30333" # P2P networking
environment:
- CHAIN=stagenet-local
command:
- --name=indexer
- --chain=stagenet-local
- --unsafe-force-node-key-generation
- --base-path=/data
- --discover-local
- --unsafe-rpc-external
- --rpc-cors=all
- --no-prometheus
- --no-telemetry
- --enable-offchain-indexing=true
- --pool-type=fork-aware
- --indexer
- --indexer-mode=full
- --indexer-database-url=postgresql://indexer:indexer@postgres:5432/datahaven
volumes:
- indexer-data:/data
restart: unless-stopped
depends_on:
postgres:
condition: service_healthy
alice:
condition: service_healthy
bob:
condition: service_healthy
fisherman:
build:
context: .
dockerfile: Dockerfile
image: datahavenxyz/datahaven:local
platform: linux/amd64
container_name: datahaven-fisherman
hostname: fisherman
user: "root" # Run as root to allow key injection, entrypoint switches to datahaven user
networks:
- datahaven-network
ports:
- "9950:9944" # WebSocket/RPC
- "9621:9615" # Prometheus metrics
- "30339:30333" # P2P networking
environment:
- NODE_NAME=gustavo
- NODE_TYPE=fisherman
- CHAIN=stagenet-local
- SEED=bottom drive obey lake curtain smoke basket hold race lonely fit walk
entrypoint: ["/scripts/docker-entrypoint.sh"]
command:
- --name=fisherman
- --chain=stagenet-local
- --unsafe-force-node-key-generation
- --base-path=/data
- --keystore-path=/data/keystore
- --discover-local
- --unsafe-rpc-external
- --rpc-cors=all
- --no-prometheus
- --no-telemetry
- --enable-offchain-indexing=true
- --pool-type=fork-aware
- --fisherman
- --fisherman-database-url=postgresql://indexer:indexer@postgres:5432/datahaven
volumes:
- ./scripts/docker-entrypoint.sh:/scripts/docker-entrypoint.sh:ro
- fisherman-keystore:/data/keystore
restart: unless-stopped
depends_on:
postgres:
condition: service_healthy
alice:
condition: service_healthy
bob:
condition: service_healthy
networks:
datahaven-network:
driver: bridge
name: datahaven-network
volumes:
alice-keystore:
bob-keystore:
msp-keystore:
bsp01-keystore:
bsp02-keystore:
postgres-data:
indexer-data:
fisherman-keystore:
Reference in a new issue View git blame Copy permalink