mirror of
https://github.com/datahaven-xyz/datahaven
synced 2026-05-24 09:50:01 +00:00
## Summary This PR adds comprehensive Kubernetes deployment infrastructure for StorageHub components, enabling deployment of the full StorageHub network stack (MSP, BSP, Indexer, and Fisherman nodes) alongside DataHaven nodes in both local and stagenet environments. ### What's Added **1. New Helm Chart: StorageHub MSP Backend API** (`deploy/charts/backend/`) - REST API service for StorageHub operations - Connects to PostgreSQL database for indexed blockchain data - Connects to RPC nodes for real-time blockchain queries - Configurable via TOML configuration file - Supports environment-specific overrides - Includes comprehensive documentation **2. StorageHub Node Deployment Charts** (`deploy/charts/node/storagehub/`) - **MSP Node** (`sh-mspnode`): Main Service Provider nodes with charging capabilities - **BSP Node** (`sh-bspnode`): Backup Service Provider nodes for redundancy - **Indexer Node** (`sh-idxnode`): Full indexing node with PostgreSQL integration - **Fisherman Node** (`sh-fisherman`): Network monitoring and verification node **3. Environment Configurations** - **Local environment** (`deploy/environments/local/`): Development setup with hostpath storage - **Stagenet environment** (`deploy/environments/stagenet/`): Production-like setup with AWS EBS - PostgreSQL database configurations for Indexer and Fisherman nodes - Proper service discovery and network configuration **4. Enhanced CLI Tooling** (`test/cli/`) - New `deploy storagehub` command for deploying StorageHub components - Updated `launch storagehub` command for local testing - Interactive deployment with environment selection - Automatic database provisioning via Bitnami PostgreSQL charts **5. Node Configuration Improvements** - Fork-aware transaction pool for DH boot & validator nodes - Unsafe RPC methods exposed on MSP nodes (for provider operations) - JWT secret support for MSP Backend authentication - ECDSA key scheme for StorageHub BCSV keys (DataHaven compatibility) ### Architecture ``` StorageHub Stack: ├── MSP Nodes (2 replicas) → Storage providers with charging ├── BSP Nodes (2 replicas) → Backup storage providers ├── Indexer Node → Database indexing + PostgreSQL ├── Fisherman Node → Monitoring + PostgreSQL (shared with Indexer) └── MSP Backend API → REST API for StorageHub operations ``` ### Testing **Local Testing**: ```bash cd test bun cli launch storagehub # Interactive launcher # or bun cli deploy storagehub # Deploy via Helm ``` **Stagenet Deployment**: ```bash cd deploy helm install sh-mspnode ./charts/node \ -f ./charts/node/storagehub/sh-mspnode.yaml \ -f ./environments/stagenet/sh-mspnode.yaml \ -n datahaven-stagenet ``` ### Breaking Changes None - This is purely additive infrastructure. ### Migration Notes For existing deployments: 1. DataHaven nodes now use `--pool-type fork-aware` flag 2. Bootnode and validator node configs updated accordingly 3. No action required for existing DataHaven-only deployments
270 lines
No EOL
6.4 KiB
Markdown
270 lines
No EOL
6.4 KiB
Markdown
# StorageHub MSP Backend Helm Chart
|
|
|
|
This Helm chart deploys the StorageHub MSP Backend API service that provides REST API access to the StorageHub network data.
|
|
|
|
## Overview
|
|
|
|
The StorageHub MSP Backend API:
|
|
- Connects to a StorageHub Indexer's database for indexed blockchain data
|
|
- Connects to a StorageHub MSP node for real-time blockchain queries
|
|
- Provides REST API endpoints for StorageHub operations
|
|
|
|
## Prerequisites
|
|
|
|
- Kubernetes 1.19+
|
|
- Helm 3.2.0+
|
|
- Running StorageHub Indexer database (PostgreSQL)
|
|
- Running StorageHub MSP node
|
|
|
|
## Installation
|
|
|
|
### Using base configuration
|
|
|
|
```bash
|
|
helm install sh-mspbackend ./charts/backend \
|
|
-f ./charts/backend/storagehub/sh-mspbackend.yaml
|
|
```
|
|
|
|
### For Local environment
|
|
|
|
```bash
|
|
helm install sh-mspbackend ./charts/backend \
|
|
-f ./charts/backend/storagehub/sh-mspbackend.yaml \
|
|
-f ./environments/local/sh-mspbackend.yaml \
|
|
-n kt-datahaven-local
|
|
```
|
|
|
|
### For Stagenet environment
|
|
|
|
```bash
|
|
helm install sh-mspbackend ./charts/backend \
|
|
-f ./charts/backend/storagehub/sh-mspbackend.yaml \
|
|
-f ./environments/stagenet/sh-mspbackend.yaml \
|
|
-n datahaven-stagenet
|
|
```
|
|
|
|
## Configuration
|
|
|
|
### Key Parameters
|
|
|
|
| Parameter | Description | Default |
|
|
|-----------|-------------|---------|
|
|
| `image.repository` | Container image repository | `moonsonglabs/storage-hub-msp-backend` |
|
|
| `image.tag` | Container image tag | `latest` |
|
|
| `replicaCount` | Number of replicas | `1` |
|
|
| `service.type` | Kubernetes service type | `ClusterIP` |
|
|
| `service.port` | Service port | `8080` |
|
|
| `service.targetPort` | Service target port | `80` |
|
|
| `backend.port` | Backend application port | `8080` |
|
|
| `backend.database.url` | PostgreSQL connection URL | `postgresql://storagehub:storagehub@sh-indexer-db:5432/storagehub` |
|
|
| `backend.rpc.endpoint` | WebSocket RPC endpoint | `ws://sh-idxnode:9944` |
|
|
| `backend.api.defaultPageSize` | Default page size for API results | `20` |
|
|
| `backend.api.maxPageSize` | Maximum page size for API results | `100` |
|
|
| `ingress.enabled` | Enable ingress | `false` |
|
|
|
|
### Configuration File
|
|
|
|
The backend uses a TOML configuration file passed via the `--config` CLI argument. This file is automatically generated from the Helm values and mounted as a ConfigMap at `/configs/config.toml`.
|
|
|
|
#### Basic Configuration:
|
|
```yaml
|
|
backend:
|
|
port: 8080
|
|
|
|
database:
|
|
url: postgresql://indexer:indexer@sh-idxnode-db-postgresql:5432/datahaven
|
|
|
|
rpc:
|
|
endpoint: ws://sh-mspnode-0:9955
|
|
|
|
api:
|
|
defaultPageSize: 20
|
|
maxPageSize: 100
|
|
|
|
auth:
|
|
jwtSecret: "your-secret-here"
|
|
|
|
args:
|
|
- "--config"
|
|
- "/configs/config.toml"
|
|
|
|
configMap:
|
|
enabled: true
|
|
```
|
|
|
|
#### Alternative: Building Database URL from Components
|
|
The chart can also construct the database URL from separate components:
|
|
```yaml
|
|
backend:
|
|
database:
|
|
host: sh-idxnode-db-postgresql
|
|
port: 5432
|
|
name: datahaven
|
|
user: indexer
|
|
password: production_password
|
|
```
|
|
|
|
**Note:** For production deployments, consider using Kubernetes Secrets or external secret management solutions for sensitive values like database passwords and JWT secrets.
|
|
|
|
### Environment Variables
|
|
|
|
Additional environment variables can be configured:
|
|
|
|
```yaml
|
|
backend:
|
|
env:
|
|
NODE_ENV: production
|
|
LOG_LEVEL: info
|
|
```
|
|
|
|
### Additional ConfigMap Data
|
|
|
|
You can add extra files to the ConfigMap:
|
|
|
|
```yaml
|
|
configMap:
|
|
enabled: true
|
|
data:
|
|
custom-config.yaml: |
|
|
# Your custom configuration here
|
|
key: value
|
|
```
|
|
|
|
### CLI Arguments
|
|
|
|
Additional CLI arguments can be specified to pass to the backend application:
|
|
|
|
```yaml
|
|
backend:
|
|
args:
|
|
- "--config"
|
|
- "/configs/config.toml"
|
|
- "--log-level"
|
|
- "debug"
|
|
```
|
|
|
|
### Using Environment Variables from ConfigMaps or Secrets
|
|
|
|
You can inject environment variables from existing ConfigMaps or Secrets:
|
|
|
|
```yaml
|
|
backend:
|
|
envFrom:
|
|
- configMapRef:
|
|
name: my-config
|
|
- secretRef:
|
|
name: my-secret
|
|
```
|
|
|
|
## Accessing the Service
|
|
|
|
### Local Environment
|
|
|
|
When deployed with `NodePort` service type:
|
|
```bash
|
|
# Access via NodePort (configured as 30300 in local environment)
|
|
curl http://localhost:30300/
|
|
|
|
# Or via ingress if enabled
|
|
curl http://sh-mspbackend.datahaven.local/
|
|
```
|
|
|
|
### Stagenet Environment
|
|
|
|
```bash
|
|
# Access via ingress
|
|
curl https://sh-mspbackend.datahaven-kt.xyz/
|
|
```
|
|
|
|
## Generated Configuration
|
|
|
|
The chart automatically generates a `config.toml` file with the following structure:
|
|
|
|
```toml
|
|
host = "0.0.0.0"
|
|
port = 8080
|
|
|
|
[api]
|
|
default_page_size = 20
|
|
max_page_size = 100
|
|
|
|
[storage_hub]
|
|
rpc_url = "ws://sh-mspnode-0:9955"
|
|
msp_callback_url = "http://sh-mspbackend:8080"
|
|
timeout_secs = 30
|
|
max_concurrent_requests = 100
|
|
verify_tls = true
|
|
mock_mode = false
|
|
|
|
[auth]
|
|
jwt_secret = "your-secret-here"
|
|
|
|
[database]
|
|
url = "postgresql://indexer:indexer@sh-idxnode-db-postgresql:5432/datahaven"
|
|
mock_mode = false
|
|
```
|
|
|
|
## Troubleshooting
|
|
|
|
### Check pod status
|
|
```bash
|
|
kubectl get pods -l app.kubernetes.io/name=sh-mspbackend -n <namespace>
|
|
```
|
|
|
|
### View logs
|
|
```bash
|
|
kubectl logs -l app.kubernetes.io/name=sh-mspbackend -n <namespace>
|
|
```
|
|
|
|
### Verify database connection
|
|
For local environment:
|
|
```bash
|
|
kubectl exec -it deployment/sh-mspbackend -n kt-datahaven-local -- nc -zv sh-idxnode-db-postgresql 5432
|
|
```
|
|
|
|
For stagenet environment:
|
|
```bash
|
|
kubectl exec -it deployment/sh-mspbackend -n datahaven-stagenet -- nc -zv sh-idxnode-db-postgresql 5432
|
|
```
|
|
|
|
### Verify RPC connection
|
|
For local environment:
|
|
```bash
|
|
kubectl exec -it deployment/sh-mspbackend -n kt-datahaven-local -- nc -zv sh-mspnode-0 9955
|
|
```
|
|
|
|
For stagenet environment:
|
|
```bash
|
|
kubectl exec -it deployment/sh-mspbackend -n datahaven-stagenet -- nc -zv sh-mspnode-0 9955
|
|
```
|
|
|
|
### View generated configuration
|
|
```bash
|
|
kubectl get configmap sh-mspbackend-config -n <namespace> -o yaml
|
|
```
|
|
|
|
## Uninstallation
|
|
|
|
```bash
|
|
# For local environment
|
|
helm uninstall sh-mspbackend -n kt-datahaven-local
|
|
|
|
# For stagenet environment
|
|
helm uninstall sh-mspbackend -n datahaven-stagenet
|
|
```
|
|
|
|
## Environment-Specific Examples
|
|
|
|
### Local Environment Values
|
|
See `environments/local/sh-mspbackend.yaml` for the complete local configuration, which includes:
|
|
- NodePort service on port 30300
|
|
- Debug logging
|
|
- Traefik ingress at `sh-mspbackend.datahaven.local`
|
|
- Minimal resource requests for development
|
|
|
|
### Stagenet Environment Values
|
|
See `environments/stagenet/sh-mspbackend.yaml` for the complete stagenet configuration, which includes:
|
|
- ClusterIP service with AWS NLB annotations
|
|
- Production logging levels
|
|
- ALB ingress with SSL at `sh-mspbackend.datahaven-kt.xyz`
|
|
- Production-level resource requests and limits |