datahaven/test/docs/deployment.md

DataHaven Deployment Guide

This comprehensive guide covers prerequisites and deployment instructions for deploying DataHaven to a Kubernetes cluster from your machine.

Table of Contents

• Prerequisites
• Local Development Environment
• Production Deployment
• Troubleshooting
• Additional Resources

Prerequisites

System Requirements

• Operating System: macOS or Linux recommended
• Memory: Minimum 8GB RAM (16GB recommended)
• Storage: At least 20GB free disk space
• Network: Stable internet connection for downloading dependencies

Required Software

Core Dependencies

1. Docker & Docker Compose

   # macOS (using Homebrew)
   brew install --cask docker
   
   # Ubuntu/Debian
   sudo apt update
   sudo apt install docker.io docker-compose
   sudo usermod -aG docker $USER
   
   # Verify installation
   docker --version
   docker-compose --version
   

   Refer to https://docs.docker.com/engine/install/ for full installation instructions.

2. Kurtosis (for test networks)

   # macOS
   brew install kurtosis-tech/tap/kurtosis-cli
   
   # Linux
   curl -L https://github.com/kurtosis-tech/kurtosis-cli-release-artifacts/releases/latest/download/install-kurtosis.sh | bash
   
   # Verify installation
   kurtosis version
   

   Refer to https://docs.kurtosis.com/install/ for full installation instructions.

3. Bun (TypeScript runtime)

   # Homebrew 
   brew install oven-sh/bun/bun
   
   # macOS / Linux
   curl -fsSL https://bun.sh/install | bash
   
   # Verify installation
   bun --version
   

   Refer to https://bun.sh/docs/installation for full installation instructions.

Platform-Specific Requirements

macOS Users:

• Install Zig for cross-compilation:

  brew install zig
  

Linux Users:

• Consider disabling IPv6 if experiencing network issues with Kurtosis
• Ensure Docker networking is properly configured

Development Tools

1. AWS CLI (for cloud deployments)

   # macOS
   brew install awscli
   
   # Linux
   curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
   unzip awscliv2.zip
   sudo ./aws/install
   

   Refer to https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html for full installation instructions.

2. Helm (Kubernetes package manager)

   # macOS
   brew install helm
   
   # Linux
   curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash
   

3. k9s (Kubernetes CLI tool - optional)

   # macOS
   brew install k9s
   
   # Linux
   wget https://github.com/derailed/k9s/releases/latest/download/k9s_Linux_amd64.tar.gz
   tar -xzf k9s_Linux_amd64.tar.gz
   sudo mv k9s /usr/local/bin/
   

   Refer to https://k9scli.io/topics/install/ for full installation instructions.

Deployment prerequisites

Kubernetes Cluster Setup

1. Configure kubectl for AWS EKS

⚠️ WARNING
This is a permissioned step. You need to get credentials for configuring AWS.

# Install AWS CLI and configure credentials
aws configure --profile dh-poc

# Update kubeconfig for your EKS cluster
aws eks --region us-east-1 update-kubeconfig --name vira-dh-poc-cluster --profile dh-poc

2. Test your configuration

# Verify connection
kubectl cluster-info

# Verify nodes, you should see some nodes listed
kubectl get nodes

# Verify pods, you should see a long list of kubernetes pods
kubectl get pods -A

3. Configure Kurtosis for Cloud Deployment

Create/replace your Kurtosis config file with the following command.

cat > "$(kurtosis config path)" << 'EOF'
config-version: 2
should-send-metrics: true
kurtosis-clusters:
  docker:
    type: "docker"
  docker.k8s:
    type: "kubernetes"
    config:
      kubernetes-cluster-name: "docker-desktop"
      storage-class: "hostpath"
      enclave-size-in-megabytes: 10
  minikube:
    type: "kubernetes"
    config:
      kubernetes-cluster-name: "minikube"
      storage-class: "standard"
      enclave-size-in-megabytes: 10
  cloud:
    type: "kubernetes"
    config:
      kubernetes-cluster-name: "vira-dh-poc-cluster"
      storage-class: "gp2"
      enclave-size-in-megabytes: 10
EOF

This will add four kurtosis clusters. You won't need all of them at once, but is good to have them configured.

1. Docker containers

This is usually only for running kurtosis directly to docker containers, it doesn't need a specific config, and it's equivalent to what we use in the bun cli launch command.

  docker:
    type: "docker"

2. Docker Kubernetes

For macOS users or everyone that can run Docker Desktop, you can check this docs to enable Kubernetes natively on your Docker Desktop app: https://docs.docker.com/desktop/features/kubernetes/

  docker.k8s:
    type: "kubernetes"
    config:
      kubernetes-cluster-name: "docker-desktop"
      storage-class: "hostpath"
      enclave-size-in-megabytes: 10

2. Minikube

Great tool for running local Kubernetes clusters, you can check installation instructions here: https://minikube.sigs.k8s.io/docs/start/

  minikube:
    type: "kubernetes"
    config:
      kubernetes-cluster-name: "minikube"
      storage-class: "standard"
      enclave-size-in-megabytes: 10

4. Kubernetes

This is gonna be for a production deployment.

  cloud:
    type: "kubernetes"
    config:
      kubernetes-cluster-name: "vira-dh-poc-cluster"
      storage-class: "gp2"
      enclave-size-in-megabytes: 10

If yout don't want all of them, you can always check your Kurtosis config file and add the desired clusters under kurtosis-clusters:

kurtosis config path

And manually paste the contents:

config-version: 2
should-send-metrics: true
kurtosis-clusters:
  ...

4. Deployment

# Set your Docker kubernetes
kurtosis cluster set <pick-a-cluster-option> # For local use `kurtosis cluster set docker.k8s`

You can pick between the three options configure :

• docker.k8s -> For local deployment
• minikube -> To deploy with minikube
• cloud -> Use for cloud-hosted Kubernetes cluster

# In a separete terminal, run and keep the gateway running (we still need this to communicate from local machine to the local kubernetes cluster)
kurtosis gateway

5. Test a simple deployment (recommended)

Before going any further, it's highly recommended that you test your config by creating a simple test network and runing it. Below, the steps:

# Creates a test-network.yml file
echo -e "participants:\n  - el_type: geth\n    cl_type: prysm\n    vc_type: prysm\n    count: 2\n\nadditional_services:\n  - spamoor" > test-network.yml

# Run the test network and wait until it succeeds
kurtosis run --enclave local-eth-testnet github.com/ethpandaops/ethereum-package --args-file test-network.yml

You can also go for testing against the provided hello-world by Kurtosis.

~ kurtosis run --enclave test-k8s github.com/kurtosis-tech/awesome-kurtosis/hello-world

Deployment

Access to GitHub

⚠️ WARNING
This is a permissioned step. You need to get credentials for DockerHub.

1. Get DockerHub credentials

This is to be able to pull from the DockerHub private repo, and it's a temporary step until the repository is public.

2. Make sure you login into docker

# Complete the password interactively
docker login -u <username>

# Check you can access the datahaven image's manifest
docker manifest inspect datahavenxyz/datahaven:main

Remote deployment

3. Run the deploy command with the credentials

bun cli deploy --docker-username=<username> --docker-password=<pass> --docker-email=<email>

If everything went well, you will see something like:

[17:24:01.058] INFO (59757): ✅ Deploy function completed successfully in 28.4 minutes

Local deployment

bun cli deploy --docker-username=<username> --docker-password=<pass> --docker-email=<email> --e local

AVS owner & tx execution flags

When invoking bun cli deploy/bun cli contracts deploy in non-local environments you must:

• Provide the multisig address that should own the ServiceManager: --avs-owner-address 0x... (or set AVS_OWNER_ADDRESS). Local deployments can still fall back to the value in contracts/config/anvil.json.
• Decide whether the script should broadcast owner-only calls immediately:
  • Default (recommended for testnet/mainnet) is leaving tx execution disabled, which prints the ABI payloads you can hand off to a Safe.
  • To execute immediately (e.g. for local/dev or CI), pass --execute-owner-transactions or set TX_EXECUTION=true. If you do so, a signing key must be provided via --avs-owner-key / AVS_OWNER_PRIVATE_KEY.

Example (testnet Safe ownership, no immediate execution):

AVS_OWNER_ADDRESS=0x... bun cli contracts deploy --chain hoodi

Example (local dev, execute owner calls right away):

bun cli contracts deploy --chain anvil --avs-owner-key $LOCAL_OWNER_KEY --execute-owner-transactions

Access Kubernetes dashboard: k9s

# In a new terminal
k9s -n kt-datahaven-stagenet

Tip: type '?' to access to all key bindings to navigate the dashboard, press 'Enter' to access an object, and 'Esc' to go back.

You can also check https://k9scli.io/topics/commands/ for a list of available commands and bindings.

Troubleshooting

Using the right context

Ensure your Kubernetes context (shown by 'kubectl config current-context') matches the cluster Kurtosis is set to use (shown by 'kurtosis cluster get'). For Docker Desktop, use 'docker-desktop' context and 'docker.k8s' cluster. For Minikube, use 'minikube' context and 'minikube' cluster.

# List available contexts
kubectl config get-contexts

# If you want to use Docker Desktop's Kubernetes, switch context:
kubectl config use-context docker-desktop

# If you want to use Minikube, switch context:
kubectl config use-context minikube

# Verify your current context:
kubectl config current-context

# Make sure your Kurtosis cluster matches your Kubernetes context:
kurtosis cluster get

RBAC Permission Issues (Kubernetes clusters only)

You shouldn't, but If you get an error like "Failed to create cluster role with name 'kurtosis-logs-collector-*'" or "is attempting to grant RBAC permissions not currently held", you can use this to fix the RBAC permissions:

# Get the service account name from the error message and create a cluster role binding
kubectl create clusterrolebinding kurtosis-logs-collector --clusterrole=cluster-admin --serviceaccount=<namespace>:<serviceaccount>

# Example (replace with the actual service account from your error):
kubectl create clusterrolebinding kurtosis-logs-collector --clusterrole=cluster-admin --serviceaccount=kurtosis-engine-43c7ccedab104a1f86fa8839637141e2:kurtosis-engine-43c7ccedab104a1f86fa8839637141e2

Note: This gives the Kurtosis engine cluster-admin privileges, which is acceptable for local development but should be avoided in production environments.

Make sure storage-class matches your config

If you have a similar error to this :

▶ Deploying DataHaven Network
──────────────────────────────
[22:45:21.779] INFO (248627): ✅ Image datahavenxyz/datahaven:main found on Docker Hub
[22:45:21.780] INFO (248627): 🔍 Checking if Kubernetes namespace "kt-datahaven-local" exists...
[22:45:21.858] INFO (248627): ✅ Namespace "kt-datahaven-local" already exists
[22:45:21.858] INFO (248627): 🔐 Creating Docker Hub secret...
[22:45:21.927] INFO (248627): ✅ Docker Hub secret created successfully
[22:45:21.928] INFO (248627): 🚀 Deploying DataHaven bootnode with helm chart...
62 | 
63 |   // Deploy DataHaven bootnode and validators with helm chart.
64 |   logger.info("🚀 Deploying DataHaven bootnode with helm chart...");
65 |   const bootnodeTimeout = "10m"; // 10 minutes
66 |   logger.debug(
67 |     await $`helm upgrade --install dh-bootnode charts/node \
                    ^
ShellError: Failed with exit code 1
 exitCode: 1,
   stdout: "Release \"dh-bootnode\" does not exist. Installing it now.\n",
   stderr: "Error: context deadline exceeded\n",

      at new ShellError (13:16)
      at new ShellPromise (75:16)
      at BunShell (191:35)
      at <anonymous> (/home/lola/Workspace/datahavenxyz/datahaven/test/cli/handlers/deploy/datahaven.ts:67:11)

Bun v1.2.17 (Linux x64)
error: script "cli" exited with code 1

If say you're using a kurtosis cluster that has a storage-class different from "hostpath"when you run locally (i.e. "standard", for minikube), then you might get some errors when trying to execute the helm charts.

Look for this chunk in deploy/environments/local/values.yaml:

# Common node settings
node:
  chain: local
  chainData:
    storageClass: "hostpath"
    persistence:
      size: 10Gi
  ...

Minikube purge

To purge delete everything on minikube and restart :

minikube delete --all --purge

And try changing storageClass to whatever you have configured in the cluster. Good luck!

Help commands (for reference only)

⚠️ WARNING
No need to run these commands, they are just for reference and troubleshooting.

DataHaven

1. Access Validator Node

# Port forward to access Polkadot.js apps
kubectl port-forward svc/dh-validator-0 -n kt-datahaven-stagenet 9955:9955

# Access via Polkadot.js apps
# https://polkadot.js.org/apps/?rpc=ws%3A%2F%2F127.0.0.1%3A9955#/explorer

2. Run DataHaven charts

cd deployment/charts/node

# Deploy bootnode
helm upgrade --install dh-bootnode . \
  -f ./datahaven/dh-bootnode.yaml \
  -n kt-datahaven-stagenet

# Deploy validators
helm upgrade --install dh-validator . \
  -f ./datahaven/dh-validator.yaml \
  -n kt-datahaven-stagenet

Snowbridge Relayers Deployment

1. Create Required Secrets

# Create secrets for relayer private keys
kubectl create secret generic dh-beefy-relay-ethereum-key \
  --from-literal=pvk="<PRIVATE_KEY>" \
  -n kt-datahaven-stagenet

kubectl create secret generic dh-beacon-relay-substrate-key \
  --from-literal=pvk="<PRIVATE_KEY>" \
  -n kt-datahaven-stagenet

kubectl create secret generic dh-execution-relay-substrate-key \
  --from-literal=pvk="<PRIVATE_KEY>" \
  -n kt-datahaven-stagenet

2. Deploy Relayers

cd deployment/charts/snowbridge

# Deploy all relayers
helm upgrade --install dh-beacon-relay . \
  -f ./snowbridge/dh-beacon-relay.yaml \
  -n kt-datahaven-stagenet

helm upgrade --install dh-beefy-relay . \
  -f ./snowbridge/dh-beefy-relay.yaml \
  -n kt-datahaven-stagenet

helm upgrade --install dh-execution-relay . \
  -f ./snowbridge/dh-execution-relay.yaml \
  -n kt-datahaven-stagenet

Docker Registry Configuration

# Create Docker registry secret for private images
kubectl create secret docker-registry datahaven-dockerhub \
  --docker-server=https://index.docker.io/v1/ \
  --docker-username=<your-username> \
  --docker-password=<your-password> \
  --docker-email=<your-email> \
  -n kt-datahaven-stagenet

Cleanup and Maintenance

Remove Deployments

# Remove nodes
helm uninstall dh-bootnode -n kt-datahaven-stagenet
helm uninstall dh-validator -n kt-datahaven-stagenet

# Remove relayers
helm uninstall dh-beacon-relay -n kt-datahaven-stagenet
helm uninstall dh-beefy-relay -n kt-datahaven-stagenet
helm uninstall dh-execution-relay -n kt-datahaven-stagenet

# Clean up persistent volumes
kubectl delete pvc -l app.kubernetes.io/instance=dh-bootnode -n kt-datahaven-stagenet
kubectl delete pvc -l app.kubernetes.io/instance=dh-validator -n kt-datahaven-stagenet

# Delete secrets
kubectl delete secret <secret-name> -n kt-datahaven-stagenet