Commit graph

9 commits

Author SHA1 Message Date
Steve Degosserie
5988691a2f
feat: Add deployment charts for StorageHub MSP, BSP & Indexer nodes (Local & Stagenet envs) (#160)
## Summary

This PR adds comprehensive Kubernetes deployment infrastructure for
StorageHub components, enabling deployment of the full StorageHub
network stack (MSP, BSP, Indexer, and Fisherman nodes) alongside
DataHaven nodes in both local and stagenet environments.

### What's Added

**1. New Helm Chart: StorageHub MSP Backend API**
(`deploy/charts/backend/`)
- REST API service for StorageHub operations
- Connects to PostgreSQL database for indexed blockchain data
- Connects to RPC nodes for real-time blockchain queries
- Configurable via TOML configuration file
- Supports environment-specific overrides
- Includes comprehensive documentation

**2. StorageHub Node Deployment Charts**
(`deploy/charts/node/storagehub/`)
- **MSP Node** (`sh-mspnode`): Main Service Provider nodes with charging
capabilities
- **BSP Node** (`sh-bspnode`): Backup Service Provider nodes for
redundancy
- **Indexer Node** (`sh-idxnode`): Full indexing node with PostgreSQL
integration
- **Fisherman Node** (`sh-fisherman`): Network monitoring and
verification node

**3. Environment Configurations**
- **Local environment** (`deploy/environments/local/`): Development
setup with hostpath storage
- **Stagenet environment** (`deploy/environments/stagenet/`):
Production-like setup with AWS EBS
- PostgreSQL database configurations for Indexer and Fisherman nodes
- Proper service discovery and network configuration

**4. Enhanced CLI Tooling** (`test/cli/`)
- New `deploy storagehub` command for deploying StorageHub components
- Updated `launch storagehub` command for local testing
- Interactive deployment with environment selection
- Automatic database provisioning via Bitnami PostgreSQL charts

**5. Node Configuration Improvements**
- Fork-aware transaction pool for DH boot & validator nodes
- Unsafe RPC methods exposed on MSP nodes (for provider operations)
- JWT secret support for MSP Backend authentication
- ECDSA key scheme for StorageHub BCSV keys (DataHaven compatibility)

### Architecture

```
StorageHub Stack:
├── MSP Nodes (2 replicas) → Storage providers with charging
├── BSP Nodes (2 replicas) → Backup storage providers
├── Indexer Node → Database indexing + PostgreSQL
├── Fisherman Node → Monitoring + PostgreSQL (shared with Indexer)
└── MSP Backend API → REST API for StorageHub operations
```

### Testing

**Local Testing**:
```bash
cd test
bun cli launch storagehub  # Interactive launcher
# or
bun cli deploy storagehub  # Deploy via Helm
```

**Stagenet Deployment**:
```bash
cd deploy
helm install sh-mspnode ./charts/node \
  -f ./charts/node/storagehub/sh-mspnode.yaml \
  -f ./environments/stagenet/sh-mspnode.yaml \
  -n datahaven-stagenet
```

### Breaking Changes

None - This is purely additive infrastructure.

### Migration Notes

For existing deployments:
1. DataHaven nodes now use `--pool-type fork-aware` flag
2. Bootnode and validator node configs updated accordingly
3. No action required for existing DataHaven-only deployments
2025-10-21 23:18:50 +03:00
Steve Degosserie
b2c1f3f250
Stagenet deployment fixes (#151)
Small fixes following the recent Stagenet deployment.
2025-09-08 21:03:19 +02:00
Gonza Montiel
5121ae002b
feat: Datahaven contracts deployment on public testnet (#123)
## Summary
This PR introduces support for deploying Datahaven contracts to
different chains (hoodi, holesky, mainnet), as well as a new cli command
to manage this deployment separately from the regular deployment, while
maintaining compatibility with it.

#### New CLI command
- **`bun cli contracts deploy`** - Deploy contracts to supported chains
(Hoodi, Holesky, Mainnet)
- **`bun cli contracts status`** - Check deployment configuration and
status
- **`bun cli contracts verify`** - Verify contracts on block explorers
- Commands need the chain parameter: `--chain <hoodi | holesky |
mainnet>`
- Right now only `hoodi` and `holesky` are supported

### Deployment

#### Hoodi & Holesky Network Support
- Added **DeployBase.s.sol** as common ground for
**DeployTestnet.s.sol** (also new) and **DeployLocal.s.sol** (existing).
- **Hoodi configuration** (`contracts/config/hoodi.json`) with deployed
EigenLayer contract addresses to reference.
- **Holesky configuration** (`contracts/config/hoodi.json`) with
deployed EigenLayer contract addresses to reference.

#### Contracts being deployed
- **DataHaven**: ServiceManager, VetoableSlasher, RewardsRegistry
- **Snowbridge**: BeefyClient, AgentExecutor, Gateway, RewardsAgent  
- **EigenLayer**: References existing deployed contracts (not
re-deployed)

#### Deployment files
When the deployment is done, a new file under `contracts/deployments` is
generated with the addresses of the deployed contracts, for each chain
(it will be overriden per chain if run multiple times). So we would have
one `anvil.json`, `hoodi.json`, `holesky.json`, etc, with the addresses
of the deployed contracts for reference and for later verification.

#### Todo
- [x] Test compatibility with existing `bun cli launch` and `bun cli
deploy` commands

#### For follow-up PRs
- Fix verification issue with `foundry verify-contracts` when specifying
the `chain` or `chain-id` parameter, needed for hoodi
(https://github.com/foundry-rs/foundry/issues/7466).
- Add `redeploy` feature to only override implementation contract and
leave the proxy address untouched

## Usage Examples
```bash
# Deploy to Hoodi network
bun cli contracts deploy --chain hoodi

# Check deployment status  
bun cli contracts status --chain hoodi

# Verify contracts on block explorer
bun cli contracts verify --chain hoodi
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Added deployment and configuration support for new networks "hoodi"
and "holesky", including new configuration and deployment files.
* Introduced a CLI tool for managing contract deployments, status
checks, and verification across supported chains.
* Added example environment configuration and comprehensive deployment
documentation.
* Enabled contract verification and status reporting via the CLI with
support for block explorer integration.

* **Improvements**
* Refactored deployment scripts for modularity, supporting both local
and testnet environments.
* Centralized and extended configuration loading to support additional
contract addresses and network parameters.
* Enhanced deployment utilities and typings to support multi-network
deployments.

* **Bug Fixes**
* Improved input validation and error handling in CLI commands and
deployment scripts.
* Added explicit handling for zero address in operator strategy
retrieval.

* **Chores**
* Updated documentation and configuration templates for easier
onboarding and deployment management.
* Improved logging and output formatting for deployment and verification
processes.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Steve Degosserie <723552+stiiifff@users.noreply.github.com>
Co-authored-by: Ahmad Kaouk <56095276+ahmadkaouk@users.noreply.github.com>
2025-08-21 10:02:31 +00:00
Steve Degosserie
9ce0a94979
feat: Add custom chainspec support to DataHaven CLI (#129)
## Summary

Adds `--chainspec` parameter to the DataHaven CLI deploy command for
using custom chainspec files across all environments.

## Usage

```bash
# Deploy with custom chainspec
bun cli deploy --environment testnet --chainspec /absolute/path/to/chainspec.json

# Normal deployment (unchanged)
bun cli deploy --environment testnet
```

## Changes

- **CLI**: Added `--chainspec <value>` parameter with absolute path
validation
- **Helm**: New ConfigMap template and init container for custom
chainspecs
- **Bootnode**: Conditionally uses custom chainspec or generates
dynamically
- **Distribution**: Bootnode serves chainspec via HTTP, validators
download from bootnode

---------

Co-authored-by: Claude <noreply@anthropic.com>
2025-08-19 08:42:45 +02:00
Steve Degosserie
1f0cd6de27
feat: Enhanced Helm deployment with ingress management and solochain support (#115)
This PR enhances the Helm-based deployment system with several key
improvements organized into the following areas:

## New Features

### Ingress Management
- **Ingress per replica**: Added `ingress-per-replica` chart template
that automatically generates an ingress for each node replica, exposing
individual pod instances
- **Traefik integration**: Added chart values to deploy Traefik as the
ingress controller for local K8s clusters, enabling proper ingress
testing (requires adding names to `/etc/hosts`)

### Solochain Support
- **New relay chart**: Added dedicated Helm chart for Solochain relay
deployment
- **CLI integration**: Updated DataHaven CLI to deploy both Execution
and Solochain relayers

## Configuration Improvements

### Environment Structure
- **Modular configs**: Refactored environmental configurations from
single `values.yaml` files into separate component-specific overrides
for better organization

### Node Configuration
- **Base config updates**: Improved base configurations for bootnode and
validator nodes
- **Network protocol**: Reverted from litep2p back to libp2p to resolve
node communication issues on Stagenet
- **Archive node routing**: In Stagenet, relayers now connect to the
bootnode (configured as archive node) instead of validator nodes

## Storage & Deployment

### Persistent Storage
- **Relayer database**: Added support for persistent volumes to store
relayer databases instead of using ephemeral storage

### Deployment Scripts
- **Documentation cleanup**: Removed obsolete test deploy.sh script and
updated the deployment README with clearer instructions

These changes provide a more robust, scalable, and maintainable
deployment system for DataHaven infrastructure.

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added support for deploying new relayer types: Solochain and Execution
relayers, with dedicated configuration and secret management.
* Introduced per-replica ingress configuration for node deployments,
allowing each replica to have its own ingress resource and hostname.
* Added persistent storage options for relay data, configurable via
storage path, class, and size.
* Added new deployment configuration files for local and stagenet
environments, including Traefik ingress controller setup.
* Introduced a new relay category, "Solochain Relayers," for standalone
chain operations and cross-chain communication.

* **Improvements**
* Updated deployment configurations to use container-specific
environment YAML files for more granular control.
* Enhanced relay and node configurations with new flags and backend
options, including dynamic peer ID generation and automatic bootnode
discovery.
* Updated relayer endpoints to consistently use the bootnode for
connections.
* Refined relay configuration files for improved structure, clarity, and
endpoint management.

* **Bug Fixes**
* Corrected deployment logic to reference the correct
environment-specific configuration files during Helm deployments.

* **Documentation**
* Simplified and updated deployment documentation to focus on CLI-based
deployment, removing outdated manual instructions and adding a concise
overview of components and relayer types.

* **Chores**
* Removed deprecated deployment scripts and outdated configuration files
to streamline the deployment process.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude <noreply@anthropic.com>
2025-08-11 14:30:01 +03:00
Gonza Montiel
10362d3361
fix: 🔌 CLI connection issues (#119)
### Problem
Introducing `--network` should make easy to container nodes to find each
other. But this change was made half-way for the relayers, and it was
using the external port to find the first datahaven node (usually
Alice). So:
- In cli launch, Alice node port mapping was left to random port `-p
9944` instead of `-p 9944:9944`.
- Relayers couldn't connect to DataHaven nodes because they were using
the external WS port (now random) instead of hitting the internal port
(which for a cli launch we actually fix it to 9944).

### Solution

- [x] **Fixed Docker port mapping**: Explicit `-p 9944:9944` for Alice
node under network `cli-launch`
- [x] **Enhanced container spec**: Added `internalPorts` tracking to
`LaunchedNetwork`
- [x] **Fixed relayer connections**: Use internal ports for container
communication
2025-07-21 15:02:25 +02:00
Facundo Farall
9b311e00ef
test: 🏗️ Setup e2e testing framework (#104)
## Implement E2E Testing Framework with Isolated Networks

### Summary
Refactors the existing E2E testing infrastructure to provide isolated
test environments with parallel execution support. Each test suite now
runs in its own network namespace, preventing resource conflicts.

### Key Changes
- **New Testing Framework** (`test/framework/`): Base classes for test
lifecycle management with automatic setup/teardown
- **Launcher Module** (`test/launcher/`): Extracted network
orchestration logic from CLI handlers for reusability
- **Parallel Execution**: Added `test-parallel.ts` script with
concurrency limits to prevent resource exhaustion
- **Test Isolation**: Each suite gets unique network IDs (format:
`suiteName-timestamp`) and Docker networks
- **Improved Test Organization**: Migrated tests to new framework,
deprecated old test structure

### Test Improvements
- Added 4 new test suites demonstrating framework usage. :
  - `contracts.test.ts` - Smart contract deployment/interaction
  - `datahaven-substrate.test.ts` - Substrate API operations  
  - `cross-chain.test.ts` - Snowbridge cross-chain messaging
  - `ethereum-basic.test.ts` - Ethereum network operations

> [!WARNING]
The test suites themselves are bad and shouldn't be consider examples of
good tests. They were AI generated just to test the concurrency of test
runners

### Documentation
- Added comprehensive framework overview (`E2E_FRAMEWORK_OVERVIEW.md`)
- Updated README with parallel testing commands
- Added test patterns and best practices

### Breaking Changes
- Old test suites moved to `e2e - DEPRECATED/` directory
- Test execution now requires extending `BaseTestSuite` class

### Testing
Run tests with: `bun test:e2e` or `bun test:e2e:parallel` (with
concurrency limits)

### TODO
- [ ] Implement good test examples.
- [ ] Implement useful test utils (like waiting for an event to show up
in DataHaven or Ethereum).
- [ ] Enforce tests with CI (currently cannot be done due to
intermittent error when sending a transaction with PAPI).

---------

Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: undercover-cactus <lola@moonsonglabs.com>
2025-07-16 18:51:07 +02:00
Steve Degosserie
c3e6f1258b
feat: Deployment improvements & environmental overrides (#103)
This PR contains improvements to the DataHaven deployment
infrastructure:
1. Directory restructure: Moved from `deployment/` to `deploy/` (more
common for K8s / Helm -based deployment configs).
2. Added **local environment** support: updated CLI to support deploying
to a local K8s cluster.
3. Manual deployment script: `deploy/scripts/deploy.sh` for manual
deployments.
4. Environment-specific configurations: Structured values files for each
environment.
5. Chart organization: Renamed bridges-common-relay to relay for
clarity.

---------

Co-authored-by: Gonza Montiel <gon.montiel@gmail.com>
Co-authored-by: Gonza Montiel <gonzamontiel@users.noreply.github.com>
2025-06-26 13:48:33 +02:00
Facundo Farall
d2bf185bcc
feat: 🚀 Add deploy command to CLI (#87)
### New Features
1. Add the `deploy` command to our CLI.
1. Conditionally deploys kurtosis eth network if we're in `stagenet`
environment.
    2. Deploys DH nodes.
3. Deploys contracts (all of them). In `mainnet` and `testnet` it
shouldn't deploy EL contracts, but for now that's not implemented.
4. Configures parameters, validators and relayers in the same way as
`launch`.
5. Currently, it only deploys `beefy` and `beacon` relayers, `execution`
and `solochain` relayers are pending for a subsequent PR.
2. Add `waitFor` utility function that receives a lambda.

### Refactors
1. Several common functionalities used both by the `launch` and `deploy`
command have been moved to the `cli/handlers/common` directory, from
where both commands use them. These include
    1. Checks for installed dependencies.
    2. Common constants.
    3. The `LaunchedNetwork` class has been moved to this directory.
    4. DataHaven nodes common functions.
    5. Kurtosis common functions.
    6. Relayer common functions.
7. Kubernetes functions (although only used by `deploy`, it seemed
fitting to have it here still).
8. Remove CLI questions and separator prints from `deploy-contracts.ts`
and `set-datahaven-parameters.ts` scripts. These things should be in the
`cli/launch` folder, which consumes the functions in these scripts.
9. Remove `setParametersFromCollection` from `utils` folder and put it
in `cli`.
10. Create base snowbridge relayer configs for `local` and `stagenet` as
two separate directories.

### Fixes
1. Sets the default time of the `deploy` command to 6s as Lodestar is
slower than Lighthouse.
2. In `runShellCommandWithLogger` only print `stderr` if the command
fails.

### Additional Minor Changes
1. K8s secret key names changed from `dh-beefy-relay-eth-key` to
`dh-beefy-relay-ethereum-key` and `dh-beacon-relay-sub-key` to
`dh-beacon-relay-substrate-key`, for simplicity in the deployment
script.
11. Update suggested configs for `.vscode` configs.

---------

Co-authored-by: Steve Degosserie <723552+stiiifff@users.noreply.github.com>
2025-06-12 10:24:03 +02:00