mirror of
https://github.com/datahaven-xyz/datahaven
synced 2026-05-24 01:38:32 +00:00
feat: ✨ Add deployment charts for StorageHub MSP, BSP & Indexer nodes (Local & Stagenet envs) (#160)
## Summary This PR adds comprehensive Kubernetes deployment infrastructure for StorageHub components, enabling deployment of the full StorageHub network stack (MSP, BSP, Indexer, and Fisherman nodes) alongside DataHaven nodes in both local and stagenet environments. ### What's Added **1. New Helm Chart: StorageHub MSP Backend API** (`deploy/charts/backend/`) - REST API service for StorageHub operations - Connects to PostgreSQL database for indexed blockchain data - Connects to RPC nodes for real-time blockchain queries - Configurable via TOML configuration file - Supports environment-specific overrides - Includes comprehensive documentation **2. StorageHub Node Deployment Charts** (`deploy/charts/node/storagehub/`) - **MSP Node** (`sh-mspnode`): Main Service Provider nodes with charging capabilities - **BSP Node** (`sh-bspnode`): Backup Service Provider nodes for redundancy - **Indexer Node** (`sh-idxnode`): Full indexing node with PostgreSQL integration - **Fisherman Node** (`sh-fisherman`): Network monitoring and verification node **3. Environment Configurations** - **Local environment** (`deploy/environments/local/`): Development setup with hostpath storage - **Stagenet environment** (`deploy/environments/stagenet/`): Production-like setup with AWS EBS - PostgreSQL database configurations for Indexer and Fisherman nodes - Proper service discovery and network configuration **4. Enhanced CLI Tooling** (`test/cli/`) - New `deploy storagehub` command for deploying StorageHub components - Updated `launch storagehub` command for local testing - Interactive deployment with environment selection - Automatic database provisioning via Bitnami PostgreSQL charts **5. Node Configuration Improvements** - Fork-aware transaction pool for DH boot & validator nodes - Unsafe RPC methods exposed on MSP nodes (for provider operations) - JWT secret support for MSP Backend authentication - ECDSA key scheme for StorageHub BCSV keys (DataHaven compatibility) ### Architecture ``` StorageHub Stack: ├── MSP Nodes (2 replicas) → Storage providers with charging ├── BSP Nodes (2 replicas) → Backup storage providers ├── Indexer Node → Database indexing + PostgreSQL ├── Fisherman Node → Monitoring + PostgreSQL (shared with Indexer) └── MSP Backend API → REST API for StorageHub operations ``` ### Testing **Local Testing**: ```bash cd test bun cli launch storagehub # Interactive launcher # or bun cli deploy storagehub # Deploy via Helm ``` **Stagenet Deployment**: ```bash cd deploy helm install sh-mspnode ./charts/node \ -f ./charts/node/storagehub/sh-mspnode.yaml \ -f ./environments/stagenet/sh-mspnode.yaml \ -n datahaven-stagenet ``` ### Breaking Changes None - This is purely additive infrastructure. ### Migration Notes For existing deployments: 1. DataHaven nodes now use `--pool-type fork-aware` flag 2. Bootnode and validator node configs updated accordingly 3. No action required for existing DataHaven-only deployments
This commit is contained in:
Steve Degosserie
GitHub
parent
dc5105869f
commit
5988691a2f
38 changed files with 1775 additions and 8 deletions
9
deploy/charts/backend/Chart.yaml
Normal file
9
deploy/charts/backend/Chart.yaml
Normal file
|
|
@ -0,0 +1,9 @@
|
|||
apiVersion: v2
|
||||
name: sh-mspbackend
|
||||
description: A Helm chart for StorageHub MSP Backend API
|
||||
type: application
|
||||
version: 0.1.0
|
||||
appVersion: "1.0.0"
|
||||
maintainers:
|
||||
- name: StorageHub Team
|
||||
email: team@storagehub.io
|
||||
270
deploy/charts/backend/README.md
Normal file
270
deploy/charts/backend/README.md
Normal file
|
|
@ -0,0 +1,270 @@
|
|||
# StorageHub MSP Backend Helm Chart
|
||||
|
||||
This Helm chart deploys the StorageHub MSP Backend API service that provides REST API access to the StorageHub network data.
|
||||
|
||||
## Overview
|
||||
|
||||
The StorageHub MSP Backend API:
|
||||
- Connects to a StorageHub Indexer's database for indexed blockchain data
|
||||
- Connects to a StorageHub MSP node for real-time blockchain queries
|
||||
- Provides REST API endpoints for StorageHub operations
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Kubernetes 1.19+
|
||||
- Helm 3.2.0+
|
||||
- Running StorageHub Indexer database (PostgreSQL)
|
||||
- Running StorageHub MSP node
|
||||
|
||||
## Installation
|
||||
|
||||
### Using base configuration
|
||||
|
||||
```bash
|
||||
helm install sh-mspbackend ./charts/backend \
|
||||
-f ./charts/backend/storagehub/sh-mspbackend.yaml
|
||||
```
|
||||
|
||||
### For Local environment
|
||||
|
||||
```bash
|
||||
helm install sh-mspbackend ./charts/backend \
|
||||
-f ./charts/backend/storagehub/sh-mspbackend.yaml \
|
||||
-f ./environments/local/sh-mspbackend.yaml \
|
||||
-n kt-datahaven-local
|
||||
```
|
||||
|
||||
### For Stagenet environment
|
||||
|
||||
```bash
|
||||
helm install sh-mspbackend ./charts/backend \
|
||||
-f ./charts/backend/storagehub/sh-mspbackend.yaml \
|
||||
-f ./environments/stagenet/sh-mspbackend.yaml \
|
||||
-n datahaven-stagenet
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
### Key Parameters
|
||||
|
||||
| Parameter | Description | Default |
|
||||
|-----------|-------------|---------|
|
||||
| `image.repository` | Container image repository | `moonsonglabs/storage-hub-msp-backend` |
|
||||
| `image.tag` | Container image tag | `latest` |
|
||||
| `replicaCount` | Number of replicas | `1` |
|
||||
| `service.type` | Kubernetes service type | `ClusterIP` |
|
||||
| `service.port` | Service port | `8080` |
|
||||
| `service.targetPort` | Service target port | `80` |
|
||||
| `backend.port` | Backend application port | `8080` |
|
||||
| `backend.database.url` | PostgreSQL connection URL | `postgresql://storagehub:storagehub@sh-indexer-db:5432/storagehub` |
|
||||
| `backend.rpc.endpoint` | WebSocket RPC endpoint | `ws://sh-idxnode:9944` |
|
||||
| `backend.api.defaultPageSize` | Default page size for API results | `20` |
|
||||
| `backend.api.maxPageSize` | Maximum page size for API results | `100` |
|
||||
| `ingress.enabled` | Enable ingress | `false` |
|
||||
|
||||
### Configuration File
|
||||
|
||||
The backend uses a TOML configuration file passed via the `--config` CLI argument. This file is automatically generated from the Helm values and mounted as a ConfigMap at `/configs/config.toml`.
|
||||
|
||||
#### Basic Configuration:
|
||||
```yaml
|
||||
backend:
|
||||
port: 8080
|
||||
|
||||
database:
|
||||
url: postgresql://indexer:indexer@sh-idxnode-db-postgresql:5432/datahaven
|
||||
|
||||
rpc:
|
||||
endpoint: ws://sh-mspnode-0:9955
|
||||
|
||||
api:
|
||||
defaultPageSize: 20
|
||||
maxPageSize: 100
|
||||
|
||||
auth:
|
||||
jwtSecret: "your-secret-here"
|
||||
|
||||
args:
|
||||
- "--config"
|
||||
- "/configs/config.toml"
|
||||
|
||||
configMap:
|
||||
enabled: true
|
||||
```
|
||||
|
||||
#### Alternative: Building Database URL from Components
|
||||
The chart can also construct the database URL from separate components:
|
||||
```yaml
|
||||
backend:
|
||||
database:
|
||||
host: sh-idxnode-db-postgresql
|
||||
port: 5432
|
||||
name: datahaven
|
||||
user: indexer
|
||||
password: production_password
|
||||
```
|
||||
|
||||
**Note:** For production deployments, consider using Kubernetes Secrets or external secret management solutions for sensitive values like database passwords and JWT secrets.
|
||||
|
||||
### Environment Variables
|
||||
|
||||
Additional environment variables can be configured:
|
||||
|
||||
```yaml
|
||||
backend:
|
||||
env:
|
||||
NODE_ENV: production
|
||||
LOG_LEVEL: info
|
||||
```
|
||||
|
||||
### Additional ConfigMap Data
|
||||
|
||||
You can add extra files to the ConfigMap:
|
||||
|
||||
```yaml
|
||||
configMap:
|
||||
enabled: true
|
||||
data:
|
||||
custom-config.yaml: |
|
||||
# Your custom configuration here
|
||||
key: value
|
||||
```
|
||||
|
||||
### CLI Arguments
|
||||
|
||||
Additional CLI arguments can be specified to pass to the backend application:
|
||||
|
||||
```yaml
|
||||
backend:
|
||||
args:
|
||||
- "--config"
|
||||
- "/configs/config.toml"
|
||||
- "--log-level"
|
||||
- "debug"
|
||||
```
|
||||
|
||||
### Using Environment Variables from ConfigMaps or Secrets
|
||||
|
||||
You can inject environment variables from existing ConfigMaps or Secrets:
|
||||
|
||||
```yaml
|
||||
backend:
|
||||
envFrom:
|
||||
- configMapRef:
|
||||
name: my-config
|
||||
- secretRef:
|
||||
name: my-secret
|
||||
```
|
||||
|
||||
## Accessing the Service
|
||||
|
||||
### Local Environment
|
||||
|
||||
When deployed with `NodePort` service type:
|
||||
```bash
|
||||
# Access via NodePort (configured as 30300 in local environment)
|
||||
curl http://localhost:30300/
|
||||
|
||||
# Or via ingress if enabled
|
||||
curl http://sh-mspbackend.datahaven.local/
|
||||
```
|
||||
|
||||
### Stagenet Environment
|
||||
|
||||
```bash
|
||||
# Access via ingress
|
||||
curl https://sh-mspbackend.datahaven-kt.xyz/
|
||||
```
|
||||
|
||||
## Generated Configuration
|
||||
|
||||
The chart automatically generates a `config.toml` file with the following structure:
|
||||
|
||||
```toml
|
||||
host = "0.0.0.0"
|
||||
port = 8080
|
||||
|
||||
[api]
|
||||
default_page_size = 20
|
||||
max_page_size = 100
|
||||
|
||||
[storage_hub]
|
||||
rpc_url = "ws://sh-mspnode-0:9955"
|
||||
msp_callback_url = "http://sh-mspbackend:8080"
|
||||
timeout_secs = 30
|
||||
max_concurrent_requests = 100
|
||||
verify_tls = true
|
||||
mock_mode = false
|
||||
|
||||
[auth]
|
||||
jwt_secret = "your-secret-here"
|
||||
|
||||
[database]
|
||||
url = "postgresql://indexer:indexer@sh-idxnode-db-postgresql:5432/datahaven"
|
||||
mock_mode = false
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Check pod status
|
||||
```bash
|
||||
kubectl get pods -l app.kubernetes.io/name=sh-mspbackend -n <namespace>
|
||||
```
|
||||
|
||||
### View logs
|
||||
```bash
|
||||
kubectl logs -l app.kubernetes.io/name=sh-mspbackend -n <namespace>
|
||||
```
|
||||
|
||||
### Verify database connection
|
||||
For local environment:
|
||||
```bash
|
||||
kubectl exec -it deployment/sh-mspbackend -n kt-datahaven-local -- nc -zv sh-idxnode-db-postgresql 5432
|
||||
```
|
||||
|
||||
For stagenet environment:
|
||||
```bash
|
||||
kubectl exec -it deployment/sh-mspbackend -n datahaven-stagenet -- nc -zv sh-idxnode-db-postgresql 5432
|
||||
```
|
||||
|
||||
### Verify RPC connection
|
||||
For local environment:
|
||||
```bash
|
||||
kubectl exec -it deployment/sh-mspbackend -n kt-datahaven-local -- nc -zv sh-mspnode-0 9955
|
||||
```
|
||||
|
||||
For stagenet environment:
|
||||
```bash
|
||||
kubectl exec -it deployment/sh-mspbackend -n datahaven-stagenet -- nc -zv sh-mspnode-0 9955
|
||||
```
|
||||
|
||||
### View generated configuration
|
||||
```bash
|
||||
kubectl get configmap sh-mspbackend-config -n <namespace> -o yaml
|
||||
```
|
||||
|
||||
## Uninstallation
|
||||
|
||||
```bash
|
||||
# For local environment
|
||||
helm uninstall sh-mspbackend -n kt-datahaven-local
|
||||
|
||||
# For stagenet environment
|
||||
helm uninstall sh-mspbackend -n datahaven-stagenet
|
||||
```
|
||||
|
||||
## Environment-Specific Examples
|
||||
|
||||
### Local Environment Values
|
||||
See `environments/local/sh-mspbackend.yaml` for the complete local configuration, which includes:
|
||||
- NodePort service on port 30300
|
||||
- Debug logging
|
||||
- Traefik ingress at `sh-mspbackend.datahaven.local`
|
||||
- Minimal resource requests for development
|
||||
|
||||
### Stagenet Environment Values
|
||||
See `environments/stagenet/sh-mspbackend.yaml` for the complete stagenet configuration, which includes:
|
||||
- ClusterIP service with AWS NLB annotations
|
||||
- Production logging levels
|
||||
- ALB ingress with SSL at `sh-mspbackend.datahaven-kt.xyz`
|
||||
- Production-level resource requests and limits
|
||||
64
deploy/charts/backend/storagehub/sh-mspbackend.yaml
Normal file
64
deploy/charts/backend/storagehub/sh-mspbackend.yaml
Normal file
|
|
@ -0,0 +1,64 @@
|
|||
# StorageHub MSP Backend API base configuration
|
||||
# This file contains the base configuration for the StorageHub MSP Backend API
|
||||
|
||||
# Chart metadata
|
||||
fullnameOverride: sh-mspbackend
|
||||
|
||||
# Container image
|
||||
image:
|
||||
repository: moonsonglabs/storage-hub-msp-backend
|
||||
tag: latest
|
||||
pullPolicy: Always
|
||||
|
||||
# Service configuration
|
||||
service:
|
||||
type: ClusterIP
|
||||
port: 8080
|
||||
targetPort: 80
|
||||
|
||||
# Backend API configuration
|
||||
backend:
|
||||
port: 8080
|
||||
|
||||
# Database connection to StorageHub Indexer PostgreSQL
|
||||
database:
|
||||
url: postgresql://indexer:indexer@sh-idxnode-db-postgresql:5432/datahaven
|
||||
|
||||
# RPC connection to StorageHub Indexer node
|
||||
rpc:
|
||||
endpoint: ws://sh-mspnode-0:9955
|
||||
|
||||
# Authentication (set in environment-specific values)
|
||||
# auth:
|
||||
# jwtSecret: "set-in-environment-values"
|
||||
|
||||
# CLI arguments for the backend application
|
||||
args:
|
||||
- "--config"
|
||||
- "/configs/config.toml"
|
||||
|
||||
# Resource limits
|
||||
resources:
|
||||
requests:
|
||||
memory: "256Mi"
|
||||
cpu: "100m"
|
||||
limits:
|
||||
memory: "512Mi"
|
||||
cpu: "500m"
|
||||
|
||||
# Service account
|
||||
serviceAccount:
|
||||
create: true
|
||||
name: sh-mspbackend
|
||||
|
||||
# Security context
|
||||
securityContext:
|
||||
runAsNonRoot: true
|
||||
runAsUser: 1000
|
||||
capabilities:
|
||||
drop:
|
||||
- ALL
|
||||
|
||||
# Ingress configuration (disabled by default, enabled per environment)
|
||||
ingress:
|
||||
enabled: false
|
||||
60
deploy/charts/backend/templates/_helpers.tpl
Normal file
60
deploy/charts/backend/templates/_helpers.tpl
Normal file
|
|
@ -0,0 +1,60 @@
|
|||
{{/*
|
||||
Expand the name of the chart.
|
||||
*/}}
|
||||
{{- define "backend.name" -}}
|
||||
{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" }}
|
||||
{{- end }}
|
||||
|
||||
{{/*
|
||||
Create a default fully qualified app name.
|
||||
*/}}
|
||||
{{- define "backend.fullname" -}}
|
||||
{{- if .Values.fullnameOverride }}
|
||||
{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" }}
|
||||
{{- else }}
|
||||
{{- $name := default .Chart.Name .Values.nameOverride }}
|
||||
{{- if contains $name .Release.Name }}
|
||||
{{- .Release.Name | trunc 63 | trimSuffix "-" }}
|
||||
{{- else }}
|
||||
{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" }}
|
||||
{{- end }}
|
||||
{{- end }}
|
||||
{{- end }}
|
||||
|
||||
{{/*
|
||||
Create chart name and version as used by the chart label.
|
||||
*/}}
|
||||
{{- define "backend.chart" -}}
|
||||
{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" }}
|
||||
{{- end }}
|
||||
|
||||
{{/*
|
||||
Common labels
|
||||
*/}}
|
||||
{{- define "backend.labels" -}}
|
||||
helm.sh/chart: {{ include "backend.chart" . }}
|
||||
{{ include "backend.selectorLabels" . }}
|
||||
{{- if .Chart.AppVersion }}
|
||||
app.kubernetes.io/version: {{ .Chart.AppVersion | quote }}
|
||||
{{- end }}
|
||||
app.kubernetes.io/managed-by: {{ .Release.Service }}
|
||||
{{- end }}
|
||||
|
||||
{{/*
|
||||
Selector labels
|
||||
*/}}
|
||||
{{- define "backend.selectorLabels" -}}
|
||||
app.kubernetes.io/name: {{ include "backend.name" . }}
|
||||
app.kubernetes.io/instance: {{ .Release.Name }}
|
||||
{{- end }}
|
||||
|
||||
{{/*
|
||||
Create the name of the service account to use
|
||||
*/}}
|
||||
{{- define "backend.serviceAccountName" -}}
|
||||
{{- if .Values.serviceAccount.create }}
|
||||
{{- default (include "backend.fullname" .) .Values.serviceAccount.name }}
|
||||
{{- else }}
|
||||
{{- default "default" .Values.serviceAccount.name }}
|
||||
{{- end }}
|
||||
{{- end }}
|
||||
48
deploy/charts/backend/templates/configmap.yaml
Normal file
48
deploy/charts/backend/templates/configmap.yaml
Normal file
|
|
@ -0,0 +1,48 @@
|
|||
{{- if .Values.configMap.enabled -}}
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: {{ include "backend.fullname" . }}-config
|
||||
labels:
|
||||
{{- include "backend.labels" . | nindent 4 }}
|
||||
data:
|
||||
config.toml: |
|
||||
# StorageHub Backend Configuration
|
||||
host = "0.0.0.0"
|
||||
port = {{ .Values.backend.port }}
|
||||
|
||||
[api]
|
||||
default_page_size = {{ .Values.backend.api.defaultPageSize | default 20 }}
|
||||
max_page_size = {{ .Values.backend.api.maxPageSize | default 100 }}
|
||||
|
||||
[storage_hub]
|
||||
{{- if .Values.backend.rpc.endpoint }}
|
||||
rpc_url = {{ .Values.backend.rpc.endpoint | quote }}
|
||||
{{- end }}
|
||||
|
||||
msp_callback_url = "http://{{ include "backend.fullname" . }}:8080"
|
||||
|
||||
timeout_secs = 30
|
||||
max_concurrent_requests = 100
|
||||
verify_tls = true
|
||||
mock_mode = false
|
||||
|
||||
[auth]
|
||||
{{- if .Values.backend.auth.jwtSecret }}
|
||||
jwt_secret = {{ .Values.backend.auth.jwtSecret | quote }}
|
||||
{{- end }}
|
||||
|
||||
[database]
|
||||
{{- if .Values.backend.database.url }}
|
||||
url = {{ .Values.backend.database.url | quote }}
|
||||
{{- else if .Values.backend.database.host }}
|
||||
url = "postgresql://{{ .Values.backend.database.user | default "indexer" }}:{{ .Values.backend.database.password | default "password" }}@{{ .Values.backend.database.host }}:{{ .Values.backend.database.port | default 5432 }}/{{ .Values.backend.database.name | default "storagehub" }}"
|
||||
{{- else }}
|
||||
url = "postgresql://indexer:password@sh-indexer-db:5432/storagehub"
|
||||
{{- end }}
|
||||
|
||||
mock_mode = false
|
||||
{{- range $key, $value := .Values.configMap.data }}
|
||||
{{ $key }}: {{ $value | quote }}
|
||||
{{- end }}
|
||||
{{- end }}
|
||||
80
deploy/charts/backend/templates/deployment.yaml
Normal file
80
deploy/charts/backend/templates/deployment.yaml
Normal file
|
|
@ -0,0 +1,80 @@
|
|||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: {{ include "backend.fullname" . }}
|
||||
labels:
|
||||
{{- include "backend.labels" . | nindent 4 }}
|
||||
spec:
|
||||
replicas: {{ .Values.replicaCount }}
|
||||
selector:
|
||||
matchLabels:
|
||||
{{- include "backend.selectorLabels" . | nindent 6 }}
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
checksum/config: {{ include (print $.Template.BasePath "/configmap.yaml") . | sha256sum }}
|
||||
{{- with .Values.podAnnotations }}
|
||||
{{- toYaml . | nindent 8 }}
|
||||
{{- end }}
|
||||
labels:
|
||||
{{- include "backend.selectorLabels" . | nindent 8 }}
|
||||
spec:
|
||||
{{- with .Values.imagePullSecrets }}
|
||||
imagePullSecrets:
|
||||
{{- toYaml . | nindent 8 }}
|
||||
{{- end }}
|
||||
serviceAccountName: {{ include "backend.serviceAccountName" . }}
|
||||
securityContext:
|
||||
{{- toYaml .Values.podSecurityContext | nindent 8 }}
|
||||
containers:
|
||||
- name: {{ .Chart.Name }}
|
||||
securityContext:
|
||||
{{- toYaml .Values.securityContext | nindent 12 }}
|
||||
image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
|
||||
imagePullPolicy: {{ .Values.image.pullPolicy }}
|
||||
ports:
|
||||
- name: http
|
||||
containerPort: {{ .Values.backend.port }}
|
||||
protocol: TCP
|
||||
env:
|
||||
- name: PORT
|
||||
value: {{ .Values.backend.port | quote }}
|
||||
{{- range $key, $value := .Values.backend.env }}
|
||||
- name: {{ $key }}
|
||||
value: {{ $value | quote }}
|
||||
{{- end }}
|
||||
{{- with .Values.backend.envFrom }}
|
||||
envFrom:
|
||||
{{- toYaml . | nindent 12 }}
|
||||
{{- end }}
|
||||
{{- if .Values.backend.args }}
|
||||
args:
|
||||
{{- toYaml .Values.backend.args | nindent 12 }}
|
||||
{{- end }}
|
||||
resources:
|
||||
{{- toYaml .Values.resources | nindent 12 }}
|
||||
{{- if .Values.configMap.enabled }}
|
||||
volumeMounts:
|
||||
- name: config
|
||||
mountPath: "/configs/config.toml"
|
||||
subPath: "config.toml"
|
||||
readOnly: true
|
||||
{{- end }}
|
||||
{{- if .Values.configMap.enabled }}
|
||||
volumes:
|
||||
- name: config
|
||||
configMap:
|
||||
name: {{ include "backend.fullname" . }}-config
|
||||
{{- end }}
|
||||
{{- with .Values.nodeSelector }}
|
||||
nodeSelector:
|
||||
{{- toYaml . | nindent 8 }}
|
||||
{{- end }}
|
||||
{{- with .Values.affinity }}
|
||||
affinity:
|
||||
{{- toYaml . | nindent 8 }}
|
||||
{{- end }}
|
||||
{{- with .Values.tolerations }}
|
||||
tolerations:
|
||||
{{- toYaml . | nindent 8 }}
|
||||
{{- end }}
|
||||
61
deploy/charts/backend/templates/ingress.yaml
Normal file
61
deploy/charts/backend/templates/ingress.yaml
Normal file
|
|
@ -0,0 +1,61 @@
|
|||
{{- if .Values.ingress.enabled -}}
|
||||
{{- $fullName := include "backend.fullname" . -}}
|
||||
{{- $svcPort := .Values.service.port -}}
|
||||
{{- if and .Values.ingress.className (not (semverCompare ">=1.18-0" .Capabilities.KubeVersion.GitVersion)) }}
|
||||
{{- if not (hasKey .Values.ingress.annotations "kubernetes.io/ingress.class") }}
|
||||
{{- $_ := set .Values.ingress.annotations "kubernetes.io/ingress.class" .Values.ingress.className}}
|
||||
{{- end }}
|
||||
{{- end }}
|
||||
{{- if semverCompare ">=1.19-0" .Capabilities.KubeVersion.GitVersion -}}
|
||||
apiVersion: networking.k8s.io/v1
|
||||
{{- else if semverCompare ">=1.14-0" .Capabilities.KubeVersion.GitVersion -}}
|
||||
apiVersion: networking.k8s.io/v1beta1
|
||||
{{- else -}}
|
||||
apiVersion: extensions/v1beta1
|
||||
{{- end }}
|
||||
kind: Ingress
|
||||
metadata:
|
||||
name: {{ $fullName }}
|
||||
labels:
|
||||
{{- include "backend.labels" . | nindent 4 }}
|
||||
{{- with .Values.ingress.annotations }}
|
||||
annotations:
|
||||
{{- toYaml . | nindent 4 }}
|
||||
{{- end }}
|
||||
spec:
|
||||
{{- if and .Values.ingress.className (semverCompare ">=1.18-0" .Capabilities.KubeVersion.GitVersion) }}
|
||||
ingressClassName: {{ .Values.ingress.className }}
|
||||
{{- end }}
|
||||
{{- if .Values.ingress.tls }}
|
||||
tls:
|
||||
{{- range .Values.ingress.tls }}
|
||||
- hosts:
|
||||
{{- range .hosts }}
|
||||
- {{ . | quote }}
|
||||
{{- end }}
|
||||
secretName: {{ .secretName }}
|
||||
{{- end }}
|
||||
{{- end }}
|
||||
rules:
|
||||
{{- range .Values.ingress.hosts }}
|
||||
- host: {{ .host | quote }}
|
||||
http:
|
||||
paths:
|
||||
{{- range .paths }}
|
||||
- path: {{ .path }}
|
||||
{{- if and .pathType (semverCompare ">=1.18-0" $.Capabilities.KubeVersion.GitVersion) }}
|
||||
pathType: {{ .pathType }}
|
||||
{{- end }}
|
||||
backend:
|
||||
{{- if semverCompare ">=1.19-0" $.Capabilities.KubeVersion.GitVersion }}
|
||||
service:
|
||||
name: {{ $fullName }}
|
||||
port:
|
||||
number: {{ $svcPort }}
|
||||
{{- else }}
|
||||
serviceName: {{ $fullName }}
|
||||
servicePort: {{ $svcPort }}
|
||||
{{- end }}
|
||||
{{- end }}
|
||||
{{- end }}
|
||||
{{- end }}
|
||||
13
deploy/charts/backend/templates/secret.yaml
Normal file
13
deploy/charts/backend/templates/secret.yaml
Normal file
|
|
@ -0,0 +1,13 @@
|
|||
{{- if .Values.secrets.enabled -}}
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: {{ include "backend.fullname" . }}-custom
|
||||
labels:
|
||||
{{- include "backend.labels" . | nindent 4 }}
|
||||
type: Opaque
|
||||
data:
|
||||
{{- range $key, $value := .Values.secrets.data }}
|
||||
{{ $key }}: {{ $value | b64enc | quote }}
|
||||
{{- end }}
|
||||
{{- end }}
|
||||
22
deploy/charts/backend/templates/service.yaml
Normal file
22
deploy/charts/backend/templates/service.yaml
Normal file
|
|
@ -0,0 +1,22 @@
|
|||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: {{ include "backend.fullname" . }}
|
||||
labels:
|
||||
{{- include "backend.labels" . | nindent 4 }}
|
||||
{{- with .Values.service.annotations }}
|
||||
annotations:
|
||||
{{- toYaml . | nindent 4 }}
|
||||
{{- end }}
|
||||
spec:
|
||||
type: {{ .Values.service.type }}
|
||||
ports:
|
||||
- port: {{ .Values.service.port }}
|
||||
targetPort: http
|
||||
protocol: TCP
|
||||
name: http
|
||||
{{- if and (eq .Values.service.type "NodePort") .Values.service.nodePort }}
|
||||
nodePort: {{ .Values.service.nodePort }}
|
||||
{{- end }}
|
||||
selector:
|
||||
{{- include "backend.selectorLabels" . | nindent 4 }}
|
||||
12
deploy/charts/backend/templates/serviceaccount.yaml
Normal file
12
deploy/charts/backend/templates/serviceaccount.yaml
Normal file
|
|
@ -0,0 +1,12 @@
|
|||
{{- if .Values.serviceAccount.create -}}
|
||||
apiVersion: v1
|
||||
kind: ServiceAccount
|
||||
metadata:
|
||||
name: {{ include "backend.serviceAccountName" . }}
|
||||
labels:
|
||||
{{- include "backend.labels" . | nindent 4 }}
|
||||
{{- with .Values.serviceAccount.annotations }}
|
||||
annotations:
|
||||
{{- toYaml . | nindent 4 }}
|
||||
{{- end }}
|
||||
{{- end }}
|
||||
122
deploy/charts/backend/values.yaml
Normal file
122
deploy/charts/backend/values.yaml
Normal file
|
|
@ -0,0 +1,122 @@
|
|||
# Default values for backend.
|
||||
# This is a YAML-formatted file.
|
||||
# Declare variables to be passed into your templates.
|
||||
|
||||
replicaCount: 1
|
||||
|
||||
image:
|
||||
repository: storagehub/backend
|
||||
pullPolicy: IfNotPresent
|
||||
tag: "latest"
|
||||
|
||||
imagePullSecrets: []
|
||||
nameOverride: ""
|
||||
fullnameOverride: ""
|
||||
|
||||
serviceAccount:
|
||||
create: true
|
||||
annotations: {}
|
||||
name: ""
|
||||
|
||||
podAnnotations: {}
|
||||
|
||||
podSecurityContext: {}
|
||||
# fsGroup: 2000
|
||||
|
||||
securityContext: {}
|
||||
# capabilities:
|
||||
# drop:
|
||||
# - ALL
|
||||
# readOnlyRootFilesystem: true
|
||||
# runAsNonRoot: true
|
||||
# runAsUser: 1000
|
||||
|
||||
service:
|
||||
type: ClusterIP
|
||||
port: 3000
|
||||
targetPort: 3000
|
||||
annotations: {}
|
||||
|
||||
ingress:
|
||||
enabled: false
|
||||
className: ""
|
||||
annotations: {}
|
||||
hosts:
|
||||
- host: api.storagehub.local
|
||||
paths:
|
||||
- path: /
|
||||
pathType: Prefix
|
||||
tls: []
|
||||
|
||||
resources:
|
||||
limits:
|
||||
cpu: 500m
|
||||
memory: 512Mi
|
||||
requests:
|
||||
cpu: 250m
|
||||
memory: 256Mi
|
||||
|
||||
nodeSelector: {}
|
||||
|
||||
tolerations: []
|
||||
|
||||
affinity: {}
|
||||
|
||||
# Backend API Configuration
|
||||
backend:
|
||||
# Port the backend listens on
|
||||
port: 3000
|
||||
|
||||
# Database configuration
|
||||
database:
|
||||
# Full database connection URL
|
||||
url: "postgresql://storagehub:storagehub@sh-indexer-db:5432/storagehub"
|
||||
# Use existing secret for database URL
|
||||
existingSecret: ""
|
||||
# Key in the secret containing the DATABASE_URL
|
||||
existingSecretUrlKey: "database-url"
|
||||
|
||||
# RPC Node configuration
|
||||
rpc:
|
||||
endpoint: "ws://sh-idxnode:9944"
|
||||
# Alternative HTTP endpoint if needed
|
||||
httpEndpoint: "http://sh-idxnode:9933"
|
||||
|
||||
# API configuration
|
||||
api:
|
||||
defaultPageSize: 20
|
||||
maxPageSize: 100
|
||||
|
||||
# Authentication configuration
|
||||
auth:
|
||||
jwtSecret: ""
|
||||
|
||||
# Environment variables
|
||||
env:
|
||||
NODE_ENV: "production"
|
||||
LOG_LEVEL: "info"
|
||||
|
||||
# Additional environment variables from ConfigMap or Secret
|
||||
envFrom: []
|
||||
|
||||
# CLI arguments to pass to the backend application
|
||||
args: []
|
||||
# Example:
|
||||
# args:
|
||||
# - "--config"
|
||||
# - "/app/config.toml"
|
||||
# - "--log-level"
|
||||
# - "debug"
|
||||
|
||||
# ConfigMap for configuration files
|
||||
configMap:
|
||||
enabled: true
|
||||
# Additional configuration to merge into config.json
|
||||
extraConfig: {}
|
||||
# Additional files to add to ConfigMap
|
||||
data: {}
|
||||
|
||||
# Secrets for sensitive data
|
||||
secrets:
|
||||
enabled: false
|
||||
data: {}
|
||||
|
|
@ -26,6 +26,7 @@ node:
|
|||
- "--allow-private-ipv4"
|
||||
- "--discover-local"
|
||||
- "--network-backend libp2p"
|
||||
- "--pool-type fork-aware"
|
||||
|
||||
ingress:
|
||||
enabled: false
|
||||
|
|
|
|||
|
|
@ -43,6 +43,7 @@ node:
|
|||
persistGeneratedNodeKey: true
|
||||
flags:
|
||||
- "--network-backend libp2p"
|
||||
- "--pool-type fork-aware"
|
||||
# Note: Bootnode discovery will happen automatically via the chainspec downloaded from customChainspecUrl
|
||||
enableOffchainIndexing: true
|
||||
|
||||
|
|
|
|||
47
deploy/charts/node/storagehub/sh-bspnode.yaml
Normal file
47
deploy/charts/node/storagehub/sh-bspnode.yaml
Normal file
|
|
@ -0,0 +1,47 @@
|
|||
name: sh-bspnode
|
||||
description: Datahaven BSP node
|
||||
fullnameOverride: sh-bspnode
|
||||
|
||||
image:
|
||||
repository: datahavenxyz/datahaven
|
||||
tag: main
|
||||
pullPolicy: Always
|
||||
|
||||
imagePullSecrets:
|
||||
- name: datahaven-dockerhub
|
||||
|
||||
node:
|
||||
command: datahaven-node
|
||||
customChainspecUrl: http://dh-bootnode:8080/chainspec.json
|
||||
forceDownloadChainspec: true
|
||||
role: full
|
||||
replicas: 2
|
||||
chainData:
|
||||
pruning: 1000
|
||||
storageClass: "gp2"
|
||||
chainKeystore:
|
||||
storageClass: "gp2"
|
||||
keys:
|
||||
# This is Alice seed. To generate new seed run: docker run --rm datahavenxyz/datahaven:latest key generate
|
||||
- seed: "bottom drive obey lake curtain smoke basket hold race lonely fit walk"
|
||||
type: bcsv
|
||||
scheme: ecdsa
|
||||
# ${HOSTNAME##*-} will be evaluated as the pod index, pod-0: //Eve, pod-1: //Ferdie
|
||||
extraDerivation: '$([ "${HOSTNAME##*-}" = "0" ] && echo "//Eve" || echo "//Ferdie")'
|
||||
persistGeneratedNodeKey: true
|
||||
flags:
|
||||
- "--allow-private-ipv4"
|
||||
- "--discover-local"
|
||||
- "--network-backend libp2p"
|
||||
- "--provider"
|
||||
- "--provider-type bsp"
|
||||
- "--max-storage-capacity 10737418240" # 10 GiB
|
||||
- "--jump-capacity=1073741824" # 1 GiB
|
||||
|
||||
ingress:
|
||||
enabled: false
|
||||
perReplica: true
|
||||
wildcardDomain: datahaven.local
|
||||
# If enabled, this would generate:
|
||||
# - sh-bspnode-0.datahaven.local
|
||||
# - sh-bspnode-1.datahaven.local
|
||||
44
deploy/charts/node/storagehub/sh-fisherman.yaml
Normal file
44
deploy/charts/node/storagehub/sh-fisherman.yaml
Normal file
|
|
@ -0,0 +1,44 @@
|
|||
name: sh-fisherman
|
||||
description: Datahaven Fisherman node
|
||||
fullnameOverride: sh-fisherman
|
||||
|
||||
image:
|
||||
repository: datahavenxyz/datahaven
|
||||
tag: main
|
||||
pullPolicy: Always
|
||||
|
||||
imagePullSecrets:
|
||||
- name: datahaven-dockerhub
|
||||
|
||||
node:
|
||||
command: datahaven-node
|
||||
customChainspecUrl: http://dh-bootnode:8080/chainspec.json
|
||||
forceDownloadChainspec: true
|
||||
role: full
|
||||
replicas: 1
|
||||
chainData:
|
||||
pruning: 1000
|
||||
storageClass: "gp2"
|
||||
chainKeystore:
|
||||
storageClass: "gp2"
|
||||
keys:
|
||||
# This is Alice seed. To generate new seed run: docker run --rm datahavenxyz/datahaven:latest key generate
|
||||
- seed: "bottom drive obey lake curtain smoke basket hold race lonely fit walk"
|
||||
type: bcsv
|
||||
scheme: ecdsa
|
||||
# ${HOSTNAME##*-} will be evaluated as the pod index, pod-0: //Gustavo, pod-1: //Hermano
|
||||
extraDerivation: '$([ "${HOSTNAME##*-}" = "0" ] && echo "//Gustavo" || echo "//Hermano")'
|
||||
persistGeneratedNodeKey: true
|
||||
flags:
|
||||
- "--allow-private-ipv4"
|
||||
- "--discover-local"
|
||||
- "--network-backend libp2p"
|
||||
- "--fisherman"
|
||||
- "--fisherman-database-url postgresql://indexer:indexer@sh-idxnode-db-postgresql:5432/datahaven"
|
||||
|
||||
ingress:
|
||||
enabled: false
|
||||
perReplica: false
|
||||
wildcardDomain: datahaven.local
|
||||
# If enabled, this would generate:
|
||||
# - sh-fisherman-0.datahaven.local
|
||||
38
deploy/charts/node/storagehub/sh-idxnode.yaml
Normal file
38
deploy/charts/node/storagehub/sh-idxnode.yaml
Normal file
|
|
@ -0,0 +1,38 @@
|
|||
name: sh-idxnode
|
||||
description: Datahaven Indexer node
|
||||
fullnameOverride: sh-idxnode
|
||||
|
||||
image:
|
||||
repository: datahavenxyz/datahaven
|
||||
tag: main
|
||||
pullPolicy: Always
|
||||
|
||||
imagePullSecrets:
|
||||
- name: datahaven-dockerhub
|
||||
|
||||
node:
|
||||
command: datahaven-node
|
||||
customChainspecUrl: http://dh-bootnode:8080/chainspec.json
|
||||
forceDownloadChainspec: true
|
||||
role: full
|
||||
replicas: 1
|
||||
chainData:
|
||||
pruning: 1000
|
||||
storageClass: "gp2"
|
||||
chainKeystore:
|
||||
storageClass: "gp2"
|
||||
persistGeneratedNodeKey: true
|
||||
flags:
|
||||
- "--allow-private-ipv4"
|
||||
- "--discover-local"
|
||||
- "--network-backend libp2p"
|
||||
- "--indexer"
|
||||
- "--indexer-mode full"
|
||||
- "--indexer-database-url postgresql://indexer:indexer@sh-idxnode-db-postgresql:5432/datahaven"
|
||||
|
||||
ingress:
|
||||
enabled: false
|
||||
perReplica: false
|
||||
wildcardDomain: datahaven.local
|
||||
# If enabled, this would generate:
|
||||
# - sh-idxnode-0.datahaven.local
|
||||
49
deploy/charts/node/storagehub/sh-mspnode.yaml
Normal file
49
deploy/charts/node/storagehub/sh-mspnode.yaml
Normal file
|
|
@ -0,0 +1,49 @@
|
|||
name: sh-mspnode
|
||||
description: Datahaven MSP node
|
||||
fullnameOverride: sh-mspnode
|
||||
|
||||
image:
|
||||
repository: datahavenxyz/datahaven
|
||||
tag: main
|
||||
pullPolicy: Always
|
||||
|
||||
imagePullSecrets:
|
||||
- name: datahaven-dockerhub
|
||||
|
||||
node:
|
||||
command: datahaven-node
|
||||
customChainspecUrl: http://dh-bootnode:8080/chainspec.json
|
||||
forceDownloadChainspec: true
|
||||
role: full
|
||||
replicas: 2
|
||||
chainData:
|
||||
pruning: 1000
|
||||
storageClass: "gp2"
|
||||
chainKeystore:
|
||||
storageClass: "gp2"
|
||||
keys:
|
||||
# This is Alice seed. To generate new seed run: docker run --rm datahavenxyz/datahaven:latest key generate
|
||||
- seed: "bottom drive obey lake curtain smoke basket hold race lonely fit walk"
|
||||
type: bcsv
|
||||
scheme: ecdsa
|
||||
# ${HOSTNAME##*-} will be evaluated as the pod index, pod-0: //Charlie, pod-1: //Dave
|
||||
extraDerivation: '$([ "${HOSTNAME##*-}" = "0" ] && echo "//Charlie" || echo "//Dave")'
|
||||
persistGeneratedNodeKey: true
|
||||
allowUnsafeRpcMethods: true
|
||||
flags:
|
||||
- "--allow-private-ipv4"
|
||||
- "--discover-local"
|
||||
- "--network-backend libp2p"
|
||||
- "--provider"
|
||||
- "--provider-type msp"
|
||||
- "--msp-charging-period 100" # in blocks, 100 blocks = 10 minutes
|
||||
- "--max-storage-capacity 10737418240" # 10 GiB
|
||||
- "--jump-capacity=1073741824" # 1 GiB
|
||||
|
||||
ingress:
|
||||
enabled: false
|
||||
perReplica: true
|
||||
wildcardDomain: datahaven.local
|
||||
# If enabled, this would generate:
|
||||
# - sh-mspnode-0.datahaven.local
|
||||
# - sh-mspnode-1.datahaven.local
|
||||
|
|
@ -15,7 +15,7 @@ imagePullSecrets:
|
|||
ingress:
|
||||
enabled: false
|
||||
ingressClassName: traefik
|
||||
host: dh-bootnode-0.datahaven-kt.local
|
||||
host: dh-bootnode-0.datahaven.local
|
||||
|
||||
node:
|
||||
chain: local
|
||||
|
|
|
|||
44
deploy/environments/local/sh-bspnode.yaml
Normal file
44
deploy/environments/local/sh-bspnode.yaml
Normal file
|
|
@ -0,0 +1,44 @@
|
|||
# BSP-specific settings for Local environment
|
||||
|
||||
global:
|
||||
environment: local
|
||||
namespace: kt-datahaven-local
|
||||
|
||||
image:
|
||||
tag: local
|
||||
pullPolicy: IfNotPresent
|
||||
|
||||
imagePullSecrets:
|
||||
- name: datahaven-dockerhub
|
||||
|
||||
# Ingress disabled for local (can be enabled for testing)
|
||||
ingress:
|
||||
enabled: false
|
||||
ingressClassName: traefik
|
||||
perReplica: true
|
||||
wildcardDomain: datahaven.local
|
||||
# If enabled, this would generate:
|
||||
# - sh-bspnode-0.datahaven.local
|
||||
# - sh-bspnode-1.datahaven.local
|
||||
|
||||
node:
|
||||
chain: local
|
||||
customChainspecUrl:
|
||||
forceDownloadChainspec: false
|
||||
chainData:
|
||||
storageClass: "hostpath"
|
||||
persistence:
|
||||
size: 10Gi
|
||||
chainKeystore:
|
||||
storageClass: "hostpath"
|
||||
resources:
|
||||
requests:
|
||||
memory: "256Mi"
|
||||
cpu: "100m"
|
||||
limits:
|
||||
memory: "512Mi"
|
||||
cpu: "200m"
|
||||
perNodeServices:
|
||||
apiService:
|
||||
enabled: true
|
||||
type: NodePort
|
||||
44
deploy/environments/local/sh-fisherman.yaml
Normal file
44
deploy/environments/local/sh-fisherman.yaml
Normal file
|
|
@ -0,0 +1,44 @@
|
|||
# Fisherman-specific settings for Local environment
|
||||
|
||||
global:
|
||||
environment: local
|
||||
namespace: kt-datahaven-local
|
||||
|
||||
image:
|
||||
tag: local
|
||||
pullPolicy: IfNotPresent
|
||||
|
||||
imagePullSecrets:
|
||||
- name: datahaven-dockerhub
|
||||
|
||||
# Ingress disabled for local (can be enabled for testing)
|
||||
ingress:
|
||||
enabled: false
|
||||
ingressClassName: traefik
|
||||
perReplica: true
|
||||
wildcardDomain: datahaven.local
|
||||
# If enabled, this would generate:
|
||||
# - sh-fisherman-0.datahaven.local
|
||||
# - sh-fisherman-1.datahaven.local
|
||||
|
||||
node:
|
||||
chain: local
|
||||
customChainspecUrl:
|
||||
forceDownloadChainspec: false
|
||||
chainData:
|
||||
storageClass: "hostpath"
|
||||
persistence:
|
||||
size: 10Gi
|
||||
chainKeystore:
|
||||
storageClass: "hostpath"
|
||||
resources:
|
||||
requests:
|
||||
memory: "256Mi"
|
||||
cpu: "100m"
|
||||
limits:
|
||||
memory: "512Mi"
|
||||
cpu: "200m"
|
||||
perNodeServices:
|
||||
apiService:
|
||||
enabled: true
|
||||
type: NodePort
|
||||
21
deploy/environments/local/sh-idxnode-db.yaml
Normal file
21
deploy/environments/local/sh-idxnode-db.yaml
Normal file
|
|
@ -0,0 +1,21 @@
|
|||
# Install PostgreSQL into local k8S cluster
|
||||
# helm repo add bitnami https://charts.bitnami.com/bitnami
|
||||
# helm repo update
|
||||
# helm install sh-idxnode-db bitnami/postgresql -f ./deploy/environments/local/sh-idxnode-db.yaml -n datahaven-local
|
||||
|
||||
auth:
|
||||
username: indexer
|
||||
password: indexer
|
||||
database: datahaven
|
||||
postgresPassword: postgres
|
||||
primary:
|
||||
persistence:
|
||||
enabled: true
|
||||
size: 10Gi # requires a default StorageClass
|
||||
resources:
|
||||
requests: { cpu: "100m", memory: "256Mi" }
|
||||
limits: { cpu: "500m", memory: "512Mi" }
|
||||
volumePermissions:
|
||||
enabled: true # helps with FS perms on some storage classes
|
||||
metrics:
|
||||
enabled: true # optional: Prometheus exporter
|
||||
33
deploy/environments/local/sh-idxnode.yaml
Normal file
33
deploy/environments/local/sh-idxnode.yaml
Normal file
|
|
@ -0,0 +1,33 @@
|
|||
# Indexer-specific settings for Local environment
|
||||
|
||||
global:
|
||||
environment: local
|
||||
namespace: kt-datahaven-local
|
||||
|
||||
image:
|
||||
tag: local
|
||||
pullPolicy: IfNotPresent
|
||||
|
||||
imagePullSecrets:
|
||||
- name: datahaven-dockerhub
|
||||
|
||||
ingress:
|
||||
enabled: false
|
||||
|
||||
node:
|
||||
chain: local
|
||||
customChainspecUrl:
|
||||
forceDownloadChainspec: false
|
||||
chainData:
|
||||
storageClass: "hostpath"
|
||||
persistence:
|
||||
size: 10Gi
|
||||
chainKeystore:
|
||||
storageClass: "hostpath"
|
||||
resources:
|
||||
requests:
|
||||
memory: "256Mi"
|
||||
cpu: "100m"
|
||||
limits:
|
||||
memory: "512Mi"
|
||||
cpu: "200m"
|
||||
73
deploy/environments/local/sh-mspbackend.yaml
Normal file
73
deploy/environments/local/sh-mspbackend.yaml
Normal file
|
|
@ -0,0 +1,73 @@
|
|||
# StorageHub MSP Backend API configuration for Local environment
|
||||
|
||||
global:
|
||||
environment: local
|
||||
namespace: kt-datahaven-local
|
||||
|
||||
# Use local image
|
||||
image:
|
||||
repository: moonsonglabs/storage-hub-msp-backend
|
||||
tag: latest
|
||||
pullPolicy: Always
|
||||
|
||||
imagePullSecrets:
|
||||
- name: datahaven-dockerhub
|
||||
|
||||
# Single replica for local development
|
||||
replicaCount: 1
|
||||
|
||||
# Service configuration for local access
|
||||
service:
|
||||
type: NodePort
|
||||
port: 8080
|
||||
nodePort: 30300 # Fixed NodePort for local access
|
||||
|
||||
# Backend configuration for local environment
|
||||
backend:
|
||||
database:
|
||||
url: postgresql://indexer:indexer@sh-idxnode-db-postgresql:5432/datahaven
|
||||
|
||||
rpc:
|
||||
url: ws://sh-idxnode:9944
|
||||
|
||||
auth:
|
||||
jwtSecret: "local-development-secret"
|
||||
|
||||
env:
|
||||
LOG_LEVEL: debug
|
||||
|
||||
# CLI arguments for local development
|
||||
args:
|
||||
- "--config"
|
||||
- "/app/config/config.toml"
|
||||
|
||||
# Minimal resources for local development
|
||||
resources:
|
||||
requests:
|
||||
memory: "128Mi"
|
||||
cpu: "50m"
|
||||
limits:
|
||||
memory: "256Mi"
|
||||
cpu: "200m"
|
||||
|
||||
# Enable ingress for local development with Traefik
|
||||
ingress:
|
||||
enabled: true
|
||||
className: traefik
|
||||
annotations:
|
||||
traefik.ingress.kubernetes.io/router.entrypoints: web
|
||||
hosts:
|
||||
- host: sh-mspbackend.datahaven.local
|
||||
paths:
|
||||
- path: /
|
||||
pathType: Prefix
|
||||
|
||||
# ConfigMap for local environment
|
||||
configMap:
|
||||
enabled: true
|
||||
extraConfig:
|
||||
environment: "local"
|
||||
features:
|
||||
debug: true
|
||||
swagger: true
|
||||
metrics: true
|
||||
44
deploy/environments/local/sh-mspnode.yaml
Normal file
44
deploy/environments/local/sh-mspnode.yaml
Normal file
|
|
@ -0,0 +1,44 @@
|
|||
# MSP-specific settings for Local environment
|
||||
|
||||
global:
|
||||
environment: local
|
||||
namespace: kt-datahaven-local
|
||||
|
||||
image:
|
||||
tag: local
|
||||
pullPolicy: IfNotPresent
|
||||
|
||||
imagePullSecrets:
|
||||
- name: datahaven-dockerhub
|
||||
|
||||
# Ingress disabled for local (can be enabled for testing)
|
||||
ingress:
|
||||
enabled: false
|
||||
ingressClassName: traefik
|
||||
perReplica: true
|
||||
wildcardDomain: datahaven.local
|
||||
# If enabled, this would generate:
|
||||
# - sh-mspnode-0.datahaven.local
|
||||
# - sh-mspnode-1.datahaven.local
|
||||
|
||||
node:
|
||||
chain: local
|
||||
customChainspecUrl:
|
||||
forceDownloadChainspec: false
|
||||
chainData:
|
||||
storageClass: "hostpath"
|
||||
persistence:
|
||||
size: 10Gi
|
||||
chainKeystore:
|
||||
storageClass: "hostpath"
|
||||
resources:
|
||||
requests:
|
||||
memory: "256Mi"
|
||||
cpu: "100m"
|
||||
limits:
|
||||
memory: "512Mi"
|
||||
cpu: "200m"
|
||||
perNodeServices:
|
||||
apiService:
|
||||
enabled: true
|
||||
type: NodePort
|
||||
|
|
@ -5,7 +5,7 @@ global:
|
|||
namespace: kt-datahaven-stagenet
|
||||
|
||||
image:
|
||||
tag: main
|
||||
tag: latest
|
||||
pullPolicy: Always
|
||||
|
||||
imagePullSecrets:
|
||||
|
|
|
|||
|
|
@ -5,7 +5,7 @@ global:
|
|||
namespace: kt-datahaven-stagenet
|
||||
|
||||
image:
|
||||
tag: main
|
||||
tag: latest
|
||||
pullPolicy: Always
|
||||
|
||||
imagePullSecrets:
|
||||
|
|
|
|||
51
deploy/environments/stagenet/sh-bspnode.yaml
Normal file
51
deploy/environments/stagenet/sh-bspnode.yaml
Normal file
|
|
@ -0,0 +1,51 @@
|
|||
# BSP-specific settings for stagenet
|
||||
|
||||
global:
|
||||
environment: stagenet
|
||||
namespace: kt-datahaven-stagenet
|
||||
|
||||
image:
|
||||
tag: latest
|
||||
pullPolicy: Always
|
||||
|
||||
imagePullSecrets:
|
||||
- name: datahaven-dockerhub
|
||||
|
||||
ingress:
|
||||
enabled: true
|
||||
perReplica: true
|
||||
wildcardDomain: datahaven-kt.xyz
|
||||
# This will generate:
|
||||
# - sh-bspnode-0.datahaven-kt.xyz
|
||||
# - sh-bspnode-1.datahaven-kt.xyz
|
||||
# (based on replica count)
|
||||
annotations:
|
||||
kubernetes.io/ingress.class: alb
|
||||
alb.ingress.kubernetes.io/scheme: internet-facing
|
||||
alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS":443}]'
|
||||
alb.ingress.kubernetes.io/load-balancer-name: sh-bspnode
|
||||
alb.ingress.kubernetes.io/target-type: ip
|
||||
alb.ingress.kubernetes.io/ssl-policy: ELBSecurityPolicy-TLS-1-2-2017-01
|
||||
alb.ingress.kubernetes.io/load-balancer-attributes: idle_timeout.timeout_seconds=120
|
||||
alb.ingress.kubernetes.io/target-group-attributes: deregistration_delay.timeout_seconds=30
|
||||
alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:us-east-1:601766312530:certificate/61c2fc57-4ce4-4a80-90b8-2810c1221f09
|
||||
external-dns.alpha.kubernetes.io/hostname: sh-bspnode.datahaven-kt.xyz
|
||||
meta.helm.sh/release-name: sh-bspnode
|
||||
meta.helm.sh/release-namespace: kt-datahaven-stagenet
|
||||
|
||||
node:
|
||||
chain: stagenet-local
|
||||
chainData:
|
||||
persistence:
|
||||
size: 20Gi
|
||||
resources:
|
||||
requests:
|
||||
memory: "512Mi"
|
||||
cpu: "200m"
|
||||
limits:
|
||||
memory: "1Gi"
|
||||
cpu: "500m"
|
||||
perNodeServices:
|
||||
apiService:
|
||||
enabled: true
|
||||
type: NodePort
|
||||
51
deploy/environments/stagenet/sh-fisherman.yaml
Normal file
51
deploy/environments/stagenet/sh-fisherman.yaml
Normal file
|
|
@ -0,0 +1,51 @@
|
|||
# Fisherman-specific settings for stagenet
|
||||
|
||||
global:
|
||||
environment: stagenet
|
||||
namespace: kt-datahaven-stagenet
|
||||
|
||||
image:
|
||||
tag: latest
|
||||
pullPolicy: Always
|
||||
|
||||
imagePullSecrets:
|
||||
- name: datahaven-dockerhub
|
||||
|
||||
ingress:
|
||||
enabled: true
|
||||
perReplica: true
|
||||
wildcardDomain: datahaven-kt.xyz
|
||||
# This will generate:
|
||||
# - sh-fisherman-0.datahaven-kt.xyz
|
||||
# - sh-fisherman-1.datahaven-kt.xyz
|
||||
# (based on replica count)
|
||||
annotations:
|
||||
kubernetes.io/ingress.class: alb
|
||||
alb.ingress.kubernetes.io/scheme: internet-facing
|
||||
alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS":443}]'
|
||||
alb.ingress.kubernetes.io/load-balancer-name: sh-fisherman
|
||||
alb.ingress.kubernetes.io/target-type: ip
|
||||
alb.ingress.kubernetes.io/ssl-policy: ELBSecurityPolicy-TLS-1-2-2017-01
|
||||
alb.ingress.kubernetes.io/load-balancer-attributes: idle_timeout.timeout_seconds=120
|
||||
alb.ingress.kubernetes.io/target-group-attributes: deregistration_delay.timeout_seconds=30
|
||||
alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:us-east-1:601766312530:certificate/61c2fc57-4ce4-4a80-90b8-2810c1221f09
|
||||
external-dns.alpha.kubernetes.io/hostname: sh-fisherman.datahaven-kt.xyz
|
||||
meta.helm.sh/release-name: sh-fisherman
|
||||
meta.helm.sh/release-namespace: kt-datahaven-stagenet
|
||||
|
||||
node:
|
||||
chain: stagenet-local
|
||||
chainData:
|
||||
persistence:
|
||||
size: 20Gi
|
||||
resources:
|
||||
requests:
|
||||
memory: "512Mi"
|
||||
cpu: "200m"
|
||||
limits:
|
||||
memory: "1Gi"
|
||||
cpu: "500m"
|
||||
perNodeServices:
|
||||
apiService:
|
||||
enabled: true
|
||||
type: NodePort
|
||||
22
deploy/environments/stagenet/sh-idxnode-db.yaml
Normal file
22
deploy/environments/stagenet/sh-idxnode-db.yaml
Normal file
|
|
@ -0,0 +1,22 @@
|
|||
# Install PostgreSQL into local k8S cluster
|
||||
# helm repo add bitnami https://charts.bitnami.com/bitnami
|
||||
# helm repo update
|
||||
# helm install sh-idxnode-db bitnami/postgresql -f ./deploy/environments/local/sh-idxnode-db.yaml -n kt-datahaven-stagenet
|
||||
|
||||
auth:
|
||||
username: indexer
|
||||
password: indexer
|
||||
database: datahaven
|
||||
postgresPassword: postgres
|
||||
primary:
|
||||
persistence:
|
||||
enabled: true
|
||||
size: 10Gi # requires a default StorageClass
|
||||
storageClass: "gp2"
|
||||
resources:
|
||||
requests: { cpu: "100m", memory: "256Mi" }
|
||||
limits: { cpu: "500m", memory: "512Mi" }
|
||||
volumePermissions:
|
||||
enabled: true # helps with FS perms on some storage classes
|
||||
metrics:
|
||||
enabled: true # optional: Prometheus exporter
|
||||
28
deploy/environments/stagenet/sh-idxnode.yaml
Normal file
28
deploy/environments/stagenet/sh-idxnode.yaml
Normal file
|
|
@ -0,0 +1,28 @@
|
|||
# Indexer-specific settings for stagenet
|
||||
|
||||
global:
|
||||
environment: stagenet
|
||||
namespace: kt-datahaven-stagenet
|
||||
|
||||
image:
|
||||
tag: latest
|
||||
pullPolicy: Always
|
||||
|
||||
imagePullSecrets:
|
||||
- name: datahaven-dockerhub
|
||||
|
||||
ingress:
|
||||
enabled: false
|
||||
|
||||
node:
|
||||
chain: stagenet-local
|
||||
chainData:
|
||||
persistence:
|
||||
size: 20Gi
|
||||
resources:
|
||||
requests:
|
||||
memory: "512Mi"
|
||||
cpu: "200m"
|
||||
limits:
|
||||
memory: "1Gi"
|
||||
cpu: "500m"
|
||||
71
deploy/environments/stagenet/sh-mspbackend.yaml
Normal file
71
deploy/environments/stagenet/sh-mspbackend.yaml
Normal file
|
|
@ -0,0 +1,71 @@
|
|||
# StorageHub MSP Backend API configuration for Stagenet environment
|
||||
|
||||
global:
|
||||
environment: stagenet
|
||||
namespace: datahaven-stagenet
|
||||
|
||||
# Stagenet image configuration
|
||||
image:
|
||||
repository: moonsonglabs/storage-hub-msp-backend
|
||||
tag: latest
|
||||
pullPolicy: Always
|
||||
|
||||
imagePullSecrets:
|
||||
- name: datahaven-dockerhub
|
||||
|
||||
replicaCount: 1
|
||||
|
||||
# Service configuration
|
||||
service:
|
||||
type: ClusterIP
|
||||
port: 8080
|
||||
annotations:
|
||||
service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
|
||||
|
||||
# Backend configuration for stagenet
|
||||
backend:
|
||||
database:
|
||||
url: postgresql://indexer:indexer@sh-idxnode-db-postgresql:5432/datahaven
|
||||
|
||||
rpc:
|
||||
endpoint: ws://sh-mspnode-0:9955
|
||||
|
||||
auth:
|
||||
jwtSecret: "0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef"
|
||||
|
||||
# CLI arguments for production
|
||||
args:
|
||||
- "--config"
|
||||
- "/configs/config.toml"
|
||||
|
||||
# Production-ready resources
|
||||
resources:
|
||||
requests:
|
||||
memory: "512Mi"
|
||||
cpu: "250m"
|
||||
limits:
|
||||
memory: "1Gi"
|
||||
cpu: "1000m"
|
||||
|
||||
# Enable ingress with SSL
|
||||
ingress:
|
||||
enabled: true
|
||||
host: sh-mspbackend.datahaven-kt.xyz
|
||||
annotations:
|
||||
kubernetes.io/ingress.class: alb
|
||||
alb.ingress.kubernetes.io/scheme: internet-facing
|
||||
alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS":443}]'
|
||||
alb.ingress.kubernetes.io/load-balancer-name: sh-mspbackend
|
||||
alb.ingress.kubernetes.io/target-type: ip
|
||||
alb.ingress.kubernetes.io/ssl-policy: ELBSecurityPolicy-TLS-1-2-2017-01
|
||||
alb.ingress.kubernetes.io/load-balancer-attributes: idle_timeout.timeout_seconds=120
|
||||
alb.ingress.kubernetes.io/target-group-attributes: deregistration_delay.timeout_seconds=30
|
||||
alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:us-east-1:601766312530:certificate/61c2fc57-4ce4-4a80-90b8-2810c1221f09
|
||||
external-dns.alpha.kubernetes.io/hostname: sh-mspbackend.datahaven-kt.xyz
|
||||
meta.helm.sh/release-name: sh-mspbackend
|
||||
meta.helm.sh/release-namespace: kt-datahaven-stagenet
|
||||
hosts:
|
||||
- host: sh-mspbackend.datahaven-kt.xyz
|
||||
paths:
|
||||
- path: /
|
||||
pathType: Prefix
|
||||
51
deploy/environments/stagenet/sh-mspnode.yaml
Normal file
51
deploy/environments/stagenet/sh-mspnode.yaml
Normal file
|
|
@ -0,0 +1,51 @@
|
|||
# MSP-specific settings for stagenet
|
||||
|
||||
global:
|
||||
environment: stagenet
|
||||
namespace: kt-datahaven-stagenet
|
||||
|
||||
image:
|
||||
tag: latest
|
||||
pullPolicy: Always
|
||||
|
||||
imagePullSecrets:
|
||||
- name: datahaven-dockerhub
|
||||
|
||||
ingress:
|
||||
enabled: true
|
||||
perReplica: true
|
||||
wildcardDomain: datahaven-kt.xyz
|
||||
# This will generate:
|
||||
# - sh-mspnode-0.datahaven-kt.xyz
|
||||
# - sh-mspnode-1.datahaven-kt.xyz
|
||||
# (based on replica count)
|
||||
annotations:
|
||||
kubernetes.io/ingress.class: alb
|
||||
alb.ingress.kubernetes.io/scheme: internet-facing
|
||||
alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS":443}]'
|
||||
alb.ingress.kubernetes.io/load-balancer-name: sh-mspnode
|
||||
alb.ingress.kubernetes.io/target-type: ip
|
||||
alb.ingress.kubernetes.io/ssl-policy: ELBSecurityPolicy-TLS-1-2-2017-01
|
||||
alb.ingress.kubernetes.io/load-balancer-attributes: idle_timeout.timeout_seconds=120
|
||||
alb.ingress.kubernetes.io/target-group-attributes: deregistration_delay.timeout_seconds=30
|
||||
alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:us-east-1:601766312530:certificate/61c2fc57-4ce4-4a80-90b8-2810c1221f09
|
||||
external-dns.alpha.kubernetes.io/hostname: sh-mspnode.datahaven-kt.xyz
|
||||
meta.helm.sh/release-name: sh-mspnode
|
||||
meta.helm.sh/release-namespace: kt-datahaven-stagenet
|
||||
|
||||
node:
|
||||
chain: stagenet-local
|
||||
chainData:
|
||||
persistence:
|
||||
size: 20Gi
|
||||
resources:
|
||||
requests:
|
||||
memory: "512Mi"
|
||||
cpu: "200m"
|
||||
limits:
|
||||
memory: "1Gi"
|
||||
cpu: "500m"
|
||||
perNodeServices:
|
||||
apiService:
|
||||
enabled: true
|
||||
type: NodePort
|
||||
25
operator/Dockerfile.local
Normal file
25
operator/Dockerfile.local
Normal file
|
|
@ -0,0 +1,25 @@
|
|||
# Dockerfile.local - Fast local image build using pre-built binary
|
||||
# Usage: docker build -f Dockerfile.local -t datahaven:local .
|
||||
# Prerequisite: cargo build --release (or --release --features fast-runtime)
|
||||
|
||||
FROM docker.io/parity/base-bin:latest
|
||||
|
||||
# Copy the pre-built binary from local target directory
|
||||
COPY ./target/release/datahaven-node /usr/local/bin/datahaven-node
|
||||
|
||||
# Make sure binary is executable
|
||||
RUN chmod +x /usr/local/bin/datahaven-node
|
||||
|
||||
USER root
|
||||
RUN useradd -m -u 1001 -U -s /bin/sh -d /datahaven datahaven && \
|
||||
mkdir -p /data /datahaven/.local/share && \
|
||||
chown -R datahaven:datahaven /data && \
|
||||
ln -s /data /datahaven/.local/share/datahaven && \
|
||||
/usr/local/bin/datahaven-node --version
|
||||
|
||||
USER datahaven
|
||||
|
||||
EXPOSE 30333 9933 9944 9615
|
||||
VOLUME ["/data"]
|
||||
|
||||
ENTRYPOINT ["/usr/local/bin/datahaven-node"]
|
||||
|
|
@ -9,6 +9,7 @@ import { deployDataHavenSolochain } from "./datahaven";
|
|||
import { deployKurtosis } from "./kurtosis";
|
||||
import { setParametersFromCollection } from "./parameters";
|
||||
import { deployRelayers } from "./relayer";
|
||||
import { deployStorageHubComponents } from "./storagehub";
|
||||
import { performValidatorOperations } from "./validator";
|
||||
|
||||
// Non-optional properties determined by having default values
|
||||
|
|
@ -39,6 +40,7 @@ export interface DeployOptions {
|
|||
skipValidatorOperations: boolean;
|
||||
skipSetParameters: boolean;
|
||||
skipRelayers: boolean;
|
||||
skipStorageHub: boolean;
|
||||
}
|
||||
|
||||
const deployFunction = async (options: DeployOptions, launchedNetwork: LaunchedNetwork) => {
|
||||
|
|
@ -82,6 +84,8 @@ const deployFunction = async (options: DeployOptions, launchedNetwork: LaunchedN
|
|||
|
||||
await deployRelayers(options, launchedNetwork);
|
||||
|
||||
await deployStorageHubComponents(options, launchedNetwork);
|
||||
|
||||
// Cleaning up the port forwarding for the validator.
|
||||
await validatorPortForwardCleanup();
|
||||
|
||||
|
|
|
|||
216
test/cli/handlers/deploy/storagehub.ts
Normal file
216
test/cli/handlers/deploy/storagehub.ts
Normal file
|
|
@ -0,0 +1,216 @@
|
|||
import path from "node:path";
|
||||
import { $ } from "bun";
|
||||
import invariant from "tiny-invariant";
|
||||
import { logger, printDivider, printHeader } from "utils";
|
||||
import { waitFor } from "utils/waits";
|
||||
import { isNetworkReady } from "../../../launcher/datahaven";
|
||||
import type { LaunchedNetwork } from "../../../launcher/types/launchedNetwork";
|
||||
import { forwardPort } from "../common/kubernetes";
|
||||
import type { DeployOptions } from ".";
|
||||
|
||||
/**
|
||||
* Deploys StorageHub components (MSP, BSP, Indexer, Fisherman nodes and databases) in a Kubernetes namespace.
|
||||
*
|
||||
* @param options - Configuration options for launching the network.
|
||||
* @param launchedNetwork - An instance of LaunchedNetwork to track the network's state.
|
||||
* @returns A promise that resolves when all StorageHub components are deployed.
|
||||
*/
|
||||
export const deployStorageHubComponents = async (
|
||||
options: DeployOptions,
|
||||
launchedNetwork: LaunchedNetwork
|
||||
): Promise<void> => {
|
||||
if (options.skipStorageHub) {
|
||||
logger.info("🏳️ Skipping StorageHub components deployment");
|
||||
printDivider();
|
||||
return;
|
||||
}
|
||||
|
||||
printHeader("Deploying StorageHub Components");
|
||||
|
||||
invariant(options.datahavenImageTag, "❌ DataHaven image tag not defined");
|
||||
|
||||
if (!options.dockerUsername) {
|
||||
await checkTagExists(options.datahavenImageTag);
|
||||
}
|
||||
|
||||
// Deploy StorageHub Indexer database first (Indexer PostgreSQL database)
|
||||
await deployStorageHubDatabase(options, launchedNetwork);
|
||||
|
||||
// Deploy StorageHub nodes (MSP, BSP, Indexer, Fisherman)
|
||||
await deployStorageHubNodes(options, launchedNetwork);
|
||||
|
||||
// Deploy StorageHub MSP Backend API
|
||||
await deployStorageHubBackend(options, launchedNetwork);
|
||||
|
||||
await registerStorageHubNodes(launchedNetwork);
|
||||
|
||||
printDivider();
|
||||
};
|
||||
|
||||
/**
|
||||
* Deploys StorageHub PostgreSQL databases for Indexer and Fisherman nodes.
|
||||
*/
|
||||
const deployStorageHubDatabase = async (
|
||||
options: DeployOptions,
|
||||
launchedNetwork: LaunchedNetwork
|
||||
): Promise<void> => {
|
||||
logger.info("🗄️ Deploying StorageHub PostgreSQL database...");
|
||||
|
||||
const deployDatabase = async (name: string, component: string) => {
|
||||
const timeout = "3m";
|
||||
const args = [
|
||||
"upgrade",
|
||||
"--install",
|
||||
name,
|
||||
"oci://registry-1.docker.io/bitnamicharts/postgresql",
|
||||
"-f",
|
||||
`environments/${options.environment}/${component}-db.yaml`,
|
||||
"-n",
|
||||
launchedNetwork.kubeNamespace,
|
||||
"--wait",
|
||||
"--timeout",
|
||||
timeout
|
||||
];
|
||||
|
||||
logger.info(`📦 Deploying ${name} database...`);
|
||||
logger.debug(await $`helm ${args}`.cwd(path.join(process.cwd(), "../deploy")).text());
|
||||
logger.success(`${name} database deployed successfully`);
|
||||
};
|
||||
|
||||
// Deploy Indexer database
|
||||
await deployDatabase("sh-indexer-db", "sh-idxnode");
|
||||
};
|
||||
|
||||
/**
|
||||
* Deploys StorageHub nodes (MSP, BSP, Indexer, Fisherman).
|
||||
*/
|
||||
const deployStorageHubNodes = async (
|
||||
options: DeployOptions,
|
||||
launchedNetwork: LaunchedNetwork
|
||||
): Promise<void> => {
|
||||
logger.info("🚀 Deploying StorageHub nodes...");
|
||||
|
||||
const deployNode = async (name: string, component: string) => {
|
||||
const timeout = "5m";
|
||||
const args = [
|
||||
"upgrade",
|
||||
"--install",
|
||||
name,
|
||||
"charts/node",
|
||||
"-f",
|
||||
`charts/node/storagehub/${component}.yaml`,
|
||||
"-f",
|
||||
`environments/${options.environment}/${component}.yaml`,
|
||||
"-n",
|
||||
launchedNetwork.kubeNamespace,
|
||||
"--wait",
|
||||
"--timeout",
|
||||
timeout
|
||||
];
|
||||
|
||||
logger.info(`🏗️ Deploying ${name}...`);
|
||||
logger.debug(await $`helm ${args}`.cwd(path.join(process.cwd(), "../deploy")).text());
|
||||
logger.success(`${name} deployed successfully`);
|
||||
};
|
||||
|
||||
// Deploy StorageHub nodes in dependency order
|
||||
await deployNode("sh-mspnode", "sh-mspnode");
|
||||
await deployNode("sh-bspnode", "sh-bspnode");
|
||||
await deployNode("sh-idxnode", "sh-idxnode");
|
||||
await deployNode("sh-fisherman", "sh-fisherman");
|
||||
};
|
||||
|
||||
/**
|
||||
* Deploys StorageHub MSP Backend API.
|
||||
*/
|
||||
const deployStorageHubBackend = async (
|
||||
options: DeployOptions,
|
||||
launchedNetwork: LaunchedNetwork
|
||||
): Promise<void> => {
|
||||
logger.info("🚀 Deploying StorageHub MSP Backend API...");
|
||||
|
||||
const timeout = "3m";
|
||||
const args = [
|
||||
"upgrade",
|
||||
"--install",
|
||||
"sh-mspbackend",
|
||||
"charts/backend",
|
||||
"-f",
|
||||
"charts/backend/storagehub/sh-mspbackend.yaml",
|
||||
"-f",
|
||||
`environments/${options.environment}/sh-mspbackend.yaml`,
|
||||
"-n",
|
||||
launchedNetwork.kubeNamespace,
|
||||
"--wait",
|
||||
"--timeout",
|
||||
timeout
|
||||
];
|
||||
|
||||
logger.debug(await $`helm ${args}`.cwd(path.join(process.cwd(), "../deploy")).text());
|
||||
logger.success("StorageHub MSP Backend API deployed successfully");
|
||||
};
|
||||
|
||||
/**
|
||||
* Waits for StorageHub Indexer node to be ready and registers nodes in LaunchedNetwork.
|
||||
*/
|
||||
const registerStorageHubNodes = async (launchedNetwork: LaunchedNetwork): Promise<void> => {
|
||||
// Forward port from indexer node to localhost for health checks
|
||||
const indexerPort = 9944;
|
||||
const { cleanup: indexerPortForwardCleanup } = await forwardPort(
|
||||
"sh-idxnode-0",
|
||||
indexerPort,
|
||||
indexerPort + 100, // Use different local port to avoid conflicts
|
||||
launchedNetwork
|
||||
);
|
||||
|
||||
// Wait for the StorageHub Indexer to start
|
||||
logger.info("⌛️ Waiting for StorageHub Indexer to start...");
|
||||
const timeoutMs = 5000; // 5 second timeout
|
||||
const delayMs = 5000; // 5 second delay between iterations
|
||||
await waitFor({
|
||||
lambda: async () => {
|
||||
logger.info(`📡 Checking if StorageHub Indexer is ready (timeout: ${timeoutMs / 1000}s)...`);
|
||||
const isReady = await isNetworkReady(indexerPort + 100, timeoutMs);
|
||||
if (!isReady) {
|
||||
logger.info(
|
||||
`⌛️ StorageHub Indexer not ready, waiting ${delayMs / 1000}s to check again...`
|
||||
);
|
||||
}
|
||||
return isReady;
|
||||
},
|
||||
iterations: 12, // 12 iterations of 5 + 5 = 2 minutes
|
||||
delay: delayMs,
|
||||
errorMessage: "StorageHub Indexer not ready"
|
||||
});
|
||||
|
||||
logger.success("StorageHub Indexer is ready");
|
||||
|
||||
// Clean up the port forwarding
|
||||
await indexerPortForwardCleanup();
|
||||
|
||||
// Register StorageHub nodes in LaunchedNetwork
|
||||
launchedNetwork.addContainer("sh-mspnode-0", { ws: 9944 });
|
||||
launchedNetwork.addContainer("sh-bspnode-0", { ws: 9944 });
|
||||
launchedNetwork.addContainer("sh-idxnode-0", { ws: 9944 });
|
||||
launchedNetwork.addContainer("sh-fisherman-0", { ws: 9944 });
|
||||
|
||||
logger.info("📝 StorageHub nodes successfully registered in launchedNetwork.");
|
||||
};
|
||||
|
||||
/**
|
||||
* Checks if an image exists in Docker Hub.
|
||||
*
|
||||
* @param tag - The tag of the image to check.
|
||||
* @returns A promise that resolves when the image is found.
|
||||
*/
|
||||
const checkTagExists = async (tag: string) => {
|
||||
const cleanTag = tag.trim();
|
||||
logger.debug(`Checking if image ${cleanTag} is available on Docker Hub`);
|
||||
const result = await $`docker manifest inspect ${cleanTag}`.nothrow().quiet();
|
||||
invariant(
|
||||
result.exitCode === 0,
|
||||
`❌ Image ${tag} not found.\n Does this image exist?\n Are you logged and have access to the repository?`
|
||||
);
|
||||
|
||||
logger.success(`Image ${cleanTag} found on Docker Hub`);
|
||||
};
|
||||
|
|
@ -10,6 +10,7 @@ import { launchDataHavenSolochain } from "./datahaven";
|
|||
import { launchKurtosis } from "./kurtosis";
|
||||
import { setParametersFromCollection } from "./parameters";
|
||||
import { launchRelayers } from "./relayer";
|
||||
import { launchStorageHubComponents } from "./storagehub";
|
||||
import { performSummaryOperations } from "./summary";
|
||||
import { performValidatorOperations } from "./validator";
|
||||
|
||||
|
|
@ -44,6 +45,7 @@ export interface LaunchOptions {
|
|||
setParameters?: boolean;
|
||||
relayer?: boolean;
|
||||
relayerImageTag: string;
|
||||
storagehub?: boolean;
|
||||
cleanNetwork?: boolean;
|
||||
}
|
||||
|
||||
|
|
@ -94,6 +96,8 @@ const launchFunction = async (options: LaunchOptions, launchedNetwork: LaunchedN
|
|||
|
||||
await launchRelayers(options, launchedNetwork);
|
||||
|
||||
await launchStorageHubComponents(options, launchedNetwork);
|
||||
|
||||
await performSummaryOperations(options, launchedNetwork);
|
||||
const fullEnd = performance.now();
|
||||
const fullMinutes = ((fullEnd - timeStart) / (1000 * 60)).toFixed(1);
|
||||
|
|
@ -119,7 +123,8 @@ export const launchPreActionHook = (
|
|||
buildDatahaven,
|
||||
launchKurtosis,
|
||||
relayer,
|
||||
setParameters
|
||||
setParameters,
|
||||
storagehub
|
||||
} = thisCmd.opts();
|
||||
|
||||
// Check for conflicts with --all flag
|
||||
|
|
@ -132,10 +137,11 @@ export const launchPreActionHook = (
|
|||
fundValidators === false ||
|
||||
setupValidators === false ||
|
||||
setParameters === false ||
|
||||
relayer === false)
|
||||
relayer === false ||
|
||||
storagehub === false)
|
||||
) {
|
||||
thisCmd.error(
|
||||
"--all cannot be used with --no-datahaven, --no-build-datahaven, --no-launch-kurtosis, --no-deploy-contracts, --no-fund-validators, --no-setup-validators, --no-update-validator-set, --no-set-parameters, or --no-relayer"
|
||||
"--all cannot be used with --no-datahaven, --no-build-datahaven, --no-launch-kurtosis, --no-deploy-contracts, --no-fund-validators, --no-setup-validators, --no-update-validator-set, --no-set-parameters, --no-relayer, or --no-storagehub"
|
||||
);
|
||||
}
|
||||
|
||||
|
|
@ -149,6 +155,7 @@ export const launchPreActionHook = (
|
|||
thisCmd.setOptionValue("setupValidators", true);
|
||||
thisCmd.setOptionValue("setParameters", true);
|
||||
thisCmd.setOptionValue("relayer", true);
|
||||
thisCmd.setOptionValue("storagehub", true);
|
||||
thisCmd.setOptionValue("cleanNetwork", true);
|
||||
}
|
||||
|
||||
|
|
|
|||
36
test/cli/handlers/launch/storagehub.ts
Normal file
36
test/cli/handlers/launch/storagehub.ts
Normal file
|
|
@ -0,0 +1,36 @@
|
|||
import { logger, printHeader } from "utils";
|
||||
import type { LaunchedNetwork } from "../../../launcher/types/launchedNetwork";
|
||||
import { deployStorageHubComponents } from "../deploy/storagehub";
|
||||
import type { LaunchOptions } from ".";
|
||||
|
||||
/**
|
||||
* Launches StorageHub components by delegating to the deploy function.
|
||||
*
|
||||
* @param options - Launch options.
|
||||
* @param launchedNetwork - The launched network instance.
|
||||
* @returns A promise that resolves when StorageHub components are launched.
|
||||
*/
|
||||
export const launchStorageHubComponents = async (
|
||||
options: LaunchOptions,
|
||||
launchedNetwork: LaunchedNetwork
|
||||
): Promise<void> => {
|
||||
// Convert launch options to deploy options format
|
||||
const deployOptions = {
|
||||
environment: "local" as const, // Launch is typically used for local development
|
||||
skipStorageHub: !options.storagehub,
|
||||
datahavenImageTag: options.datahavenImageTag,
|
||||
dockerUsername: undefined,
|
||||
dockerPassword: undefined,
|
||||
dockerEmail: undefined
|
||||
};
|
||||
|
||||
printHeader("Launching StorageHub Components");
|
||||
logger.info(
|
||||
"🚀 Launching StorageHub components (MSP, BSP, Indexer, Fisherman nodes and databases)..."
|
||||
);
|
||||
|
||||
// Reuse the deploy StorageHub function
|
||||
await deployStorageHubComponents(deployOptions as any, launchedNetwork);
|
||||
|
||||
logger.success("StorageHub components launched successfully");
|
||||
};
|
||||
|
|
@ -49,7 +49,7 @@ program
|
|||
`🫎 DataHaven: Network Deployer CLI for deploying a full DataHaven network stack to a Kubernetes cluster
|
||||
It will deploy:
|
||||
- DataHaven solochain validators (all envs),
|
||||
- Storage providers (all envs) (TODO),
|
||||
- StorageHub components: MSP, BSP, Indexer, Fisherman nodes and databases (local & stagenet envs),
|
||||
- Kurtosis Ethereum private network (stagenet env),
|
||||
- Snowbridge Relayers (all envs)
|
||||
`
|
||||
|
|
@ -98,6 +98,11 @@ program
|
|||
.option("--skip-validator-operations", "Skip performing validator operations", false)
|
||||
.option("--skip-set-parameters", "Skip setting DataHaven runtime parameters", false)
|
||||
.option("--skip-relayers", "Skip deploying Snowbridge Relayers", false)
|
||||
.option(
|
||||
"--skip-storage-hub",
|
||||
"Skip deploying StorageHub components (MSP, BSP, Indexer, Fisherman, databases)",
|
||||
false
|
||||
)
|
||||
.hook("preAction", deployPreActionHook)
|
||||
.action(deploy);
|
||||
|
||||
|
|
@ -109,7 +114,7 @@ program
|
|||
`🫎 DataHaven: Network Launcher CLI for launching a full DataHaven network.
|
||||
Complete with:
|
||||
- Solo-chain validators,
|
||||
- Storage providers (TODO),
|
||||
- StorageHub components: MSP, BSP, Indexer, Fisherman nodes and databases,
|
||||
- Ethereum Private network,
|
||||
- Snowbridge Relayers
|
||||
`
|
||||
|
|
|
|||
Loading…
Reference in a new issue