datahaven/deploy
Steve Degosserie 5988691a2f
feat: Add deployment charts for StorageHub MSP, BSP & Indexer nodes (Local & Stagenet envs) (#160)
## Summary

This PR adds comprehensive Kubernetes deployment infrastructure for
StorageHub components, enabling deployment of the full StorageHub
network stack (MSP, BSP, Indexer, and Fisherman nodes) alongside
DataHaven nodes in both local and stagenet environments.

### What's Added

**1. New Helm Chart: StorageHub MSP Backend API**
(`deploy/charts/backend/`)
- REST API service for StorageHub operations
- Connects to PostgreSQL database for indexed blockchain data
- Connects to RPC nodes for real-time blockchain queries
- Configurable via TOML configuration file
- Supports environment-specific overrides
- Includes comprehensive documentation

**2. StorageHub Node Deployment Charts**
(`deploy/charts/node/storagehub/`)
- **MSP Node** (`sh-mspnode`): Main Service Provider nodes with charging
capabilities
- **BSP Node** (`sh-bspnode`): Backup Service Provider nodes for
redundancy
- **Indexer Node** (`sh-idxnode`): Full indexing node with PostgreSQL
integration
- **Fisherman Node** (`sh-fisherman`): Network monitoring and
verification node

**3. Environment Configurations**
- **Local environment** (`deploy/environments/local/`): Development
setup with hostpath storage
- **Stagenet environment** (`deploy/environments/stagenet/`):
Production-like setup with AWS EBS
- PostgreSQL database configurations for Indexer and Fisherman nodes
- Proper service discovery and network configuration

**4. Enhanced CLI Tooling** (`test/cli/`)
- New `deploy storagehub` command for deploying StorageHub components
- Updated `launch storagehub` command for local testing
- Interactive deployment with environment selection
- Automatic database provisioning via Bitnami PostgreSQL charts

**5. Node Configuration Improvements**
- Fork-aware transaction pool for DH boot & validator nodes
- Unsafe RPC methods exposed on MSP nodes (for provider operations)
- JWT secret support for MSP Backend authentication
- ECDSA key scheme for StorageHub BCSV keys (DataHaven compatibility)

### Architecture

```
StorageHub Stack:
├── MSP Nodes (2 replicas) → Storage providers with charging
├── BSP Nodes (2 replicas) → Backup storage providers
├── Indexer Node → Database indexing + PostgreSQL
├── Fisherman Node → Monitoring + PostgreSQL (shared with Indexer)
└── MSP Backend API → REST API for StorageHub operations
```

### Testing

**Local Testing**:
```bash
cd test
bun cli launch storagehub  # Interactive launcher
# or
bun cli deploy storagehub  # Deploy via Helm
```

**Stagenet Deployment**:
```bash
cd deploy
helm install sh-mspnode ./charts/node \
  -f ./charts/node/storagehub/sh-mspnode.yaml \
  -f ./environments/stagenet/sh-mspnode.yaml \
  -n datahaven-stagenet
```

### Breaking Changes

None - This is purely additive infrastructure.

### Migration Notes

For existing deployments:
1. DataHaven nodes now use `--pool-type fork-aware` flag
2. Bootnode and validator node configs updated accordingly
3. No action required for existing DataHaven-only deployments
2025-10-21 23:18:50 +03:00
..
chainspecs feat: Add custom chainspec support to DataHaven CLI (#129) 2025-08-19 08:42:45 +02:00
charts feat: Add deployment charts for StorageHub MSP, BSP & Indexer nodes (Local & Stagenet envs) (#160) 2025-10-21 23:18:50 +03:00
environments feat: Add deployment charts for StorageHub MSP, BSP & Indexer nodes (Local & Stagenet envs) (#160) 2025-10-21 23:18:50 +03:00
README.md chore: ♻️ Update README files (#206) 2025-10-06 18:12:55 +02:00

DataHaven Deployment

This directory contains Helm charts and configurations for deploying DataHaven nodes and relayers to Kubernetes clusters across various environments (local development, staging, production).

Directory Structure

deploy/
├── charts/                    # Helm charts
│   ├── node/                 # Node chart
│   │   └── datahaven/       # DataHaven-specific node configurations
│   │       ├── dh-bootnode.yaml
│   │       └── dh-validator.yaml
│   └── relay/               # Relay chart
│       └── snowbridge/     # Snowbridge-specific relay configurations
│           ├── dh-beacon-relay.yaml    # Beacon chain relay
│           ├── dh-beefy-relay.yaml     # BEEFY consensus relay
│           └── dh-execution-relay.yaml # Execution layer relay
├── environments/             # Environment-specific configurations
│   ├── local/               # Local development environment
│   │   └── values.yaml
│   ├── stagenet/           # Staging environment
│       └── values.yaml
└── scripts/                  # Deployment scripts

Prerequisites

  • Kubernetes cluster
  • kubectl configured
  • Helm 3.x installed

Deployment

The recommended way to deploy is using the DataHaven CLI with the deploy command:

cd test && bun cli deploy --e <environment>

Example:

cd test && bun cli deploy --e local

Available environments:

  • local: Local development environment (minimal resources)
  • stagenet: Staging environment for pre-release testing

Environment Details

Local

  • Purpose: Local development and testing
  • Replicas: 1 (bootnode + validator)
  • Resources: Minimal (256Mi memory, 100m CPU)
  • Image: Local Docker builds (datahavenxyz/datahaven:local)
  • Storage: Small persistence (1-5Gi)
  • Network: Single-node network with fast block times

Stagenet

  • Purpose: Pre-production testing and staging
  • Replicas: 2+ validators
  • Resources: Medium (512Mi memory, 200m CPU)
  • Image: Stagenet tags from DockerHub
  • Storage: 20Gi+ persistent volumes
  • Network: Multi-validator network simulating production

Configuration Structure

The configuration is organized in layers, with later layers overriding earlier ones:

  1. Base Configurations (charts/node/datahaven/):

    • Base configurations for DataHaven nodes
    • Default values for bootnode and validator
  2. Environment-Specific Configurations (environments/<env>/values.yaml):

    • Environment-specific settings
    • Resource configurations
    • Image tags
    • Replica counts
    • Storage configurations

The deployment process:

  1. Loads base configurations from the respective chart directories
  2. Applies environment-specific overrides from environments/<env>/values.yaml
  3. Deploys the components with the merged configuration

Components

Nodes

  • Bootnode: Entry point for the network
  • Validator: Validates transactions and produces blocks

Relays (Snowbridge)

  • Beacon Relay: Relays Ethereum beacon chain finality to DataHaven
  • BEEFY Relay: Relays DataHaven BEEFY finality proofs to Ethereum
  • Execution Relay: Relays Ethereum execution layer messages to DataHaven
  • Solochain Relayers: Relays DataHaven chain operations to the DataHaven AVS

These relayers enable trustless bidirectional token and message passing between Ethereum and DataHaven.

Development Workflow

  1. Local Testing:

    cd test
    bun cli launch  # Starts local network without K8s
    
  2. K8s Deployment:

    cd test
    bun cli deploy --e local
    
  3. Building Local Images:

    cd test
    bun build:docker:operator  # Builds datahavenxyz/datahaven:local
    
  4. Updating Configurations:

    • Modify environments/<env>/values.yaml for environment-specific changes
    • Modify chart templates in charts/ for structural changes
    • Redeploy with bun cli deploy --e <env>

Troubleshooting

  • Pods not starting: Check logs with kubectl logs <pod-name>
  • Image pull failures: Verify Docker registry access and image tags
  • Persistent volume issues: Ensure storage class is available with kubectl get sc
  • Network connectivity: Check service endpoints with kubectl get svc

For more detailed deployment and testing workflows, see the test directory.