Commit graph

5 commits

Author SHA1 Message Date
Steve Degosserie
5988691a2f
feat: Add deployment charts for StorageHub MSP, BSP & Indexer nodes (Local & Stagenet envs) (#160)
## Summary

This PR adds comprehensive Kubernetes deployment infrastructure for
StorageHub components, enabling deployment of the full StorageHub
network stack (MSP, BSP, Indexer, and Fisherman nodes) alongside
DataHaven nodes in both local and stagenet environments.

### What's Added

**1. New Helm Chart: StorageHub MSP Backend API**
(`deploy/charts/backend/`)
- REST API service for StorageHub operations
- Connects to PostgreSQL database for indexed blockchain data
- Connects to RPC nodes for real-time blockchain queries
- Configurable via TOML configuration file
- Supports environment-specific overrides
- Includes comprehensive documentation

**2. StorageHub Node Deployment Charts**
(`deploy/charts/node/storagehub/`)
- **MSP Node** (`sh-mspnode`): Main Service Provider nodes with charging
capabilities
- **BSP Node** (`sh-bspnode`): Backup Service Provider nodes for
redundancy
- **Indexer Node** (`sh-idxnode`): Full indexing node with PostgreSQL
integration
- **Fisherman Node** (`sh-fisherman`): Network monitoring and
verification node

**3. Environment Configurations**
- **Local environment** (`deploy/environments/local/`): Development
setup with hostpath storage
- **Stagenet environment** (`deploy/environments/stagenet/`):
Production-like setup with AWS EBS
- PostgreSQL database configurations for Indexer and Fisherman nodes
- Proper service discovery and network configuration

**4. Enhanced CLI Tooling** (`test/cli/`)
- New `deploy storagehub` command for deploying StorageHub components
- Updated `launch storagehub` command for local testing
- Interactive deployment with environment selection
- Automatic database provisioning via Bitnami PostgreSQL charts

**5. Node Configuration Improvements**
- Fork-aware transaction pool for DH boot & validator nodes
- Unsafe RPC methods exposed on MSP nodes (for provider operations)
- JWT secret support for MSP Backend authentication
- ECDSA key scheme for StorageHub BCSV keys (DataHaven compatibility)

### Architecture

```
StorageHub Stack:
├── MSP Nodes (2 replicas) → Storage providers with charging
├── BSP Nodes (2 replicas) → Backup storage providers
├── Indexer Node → Database indexing + PostgreSQL
├── Fisherman Node → Monitoring + PostgreSQL (shared with Indexer)
└── MSP Backend API → REST API for StorageHub operations
```

### Testing

**Local Testing**:
```bash
cd test
bun cli launch storagehub  # Interactive launcher
# or
bun cli deploy storagehub  # Deploy via Helm
```

**Stagenet Deployment**:
```bash
cd deploy
helm install sh-mspnode ./charts/node \
  -f ./charts/node/storagehub/sh-mspnode.yaml \
  -f ./environments/stagenet/sh-mspnode.yaml \
  -n datahaven-stagenet
```

### Breaking Changes

None - This is purely additive infrastructure.

### Migration Notes

For existing deployments:
1. DataHaven nodes now use `--pool-type fork-aware` flag
2. Bootnode and validator node configs updated accordingly
3. No action required for existing DataHaven-only deployments
2025-10-21 23:18:50 +03:00
Steve Degosserie
9ce0a94979
feat: Add custom chainspec support to DataHaven CLI (#129)
## Summary

Adds `--chainspec` parameter to the DataHaven CLI deploy command for
using custom chainspec files across all environments.

## Usage

```bash
# Deploy with custom chainspec
bun cli deploy --environment testnet --chainspec /absolute/path/to/chainspec.json

# Normal deployment (unchanged)
bun cli deploy --environment testnet
```

## Changes

- **CLI**: Added `--chainspec <value>` parameter with absolute path
validation
- **Helm**: New ConfigMap template and init container for custom
chainspecs
- **Bootnode**: Conditionally uses custom chainspec or generates
dynamically
- **Distribution**: Bootnode serves chainspec via HTTP, validators
download from bootnode

---------

Co-authored-by: Claude <noreply@anthropic.com>
2025-08-19 08:42:45 +02:00
Facundo Farall
9b311e00ef
test: 🏗️ Setup e2e testing framework (#104)
## Implement E2E Testing Framework with Isolated Networks

### Summary
Refactors the existing E2E testing infrastructure to provide isolated
test environments with parallel execution support. Each test suite now
runs in its own network namespace, preventing resource conflicts.

### Key Changes
- **New Testing Framework** (`test/framework/`): Base classes for test
lifecycle management with automatic setup/teardown
- **Launcher Module** (`test/launcher/`): Extracted network
orchestration logic from CLI handlers for reusability
- **Parallel Execution**: Added `test-parallel.ts` script with
concurrency limits to prevent resource exhaustion
- **Test Isolation**: Each suite gets unique network IDs (format:
`suiteName-timestamp`) and Docker networks
- **Improved Test Organization**: Migrated tests to new framework,
deprecated old test structure

### Test Improvements
- Added 4 new test suites demonstrating framework usage. :
  - `contracts.test.ts` - Smart contract deployment/interaction
  - `datahaven-substrate.test.ts` - Substrate API operations  
  - `cross-chain.test.ts` - Snowbridge cross-chain messaging
  - `ethereum-basic.test.ts` - Ethereum network operations

> [!WARNING]
The test suites themselves are bad and shouldn't be consider examples of
good tests. They were AI generated just to test the concurrency of test
runners

### Documentation
- Added comprehensive framework overview (`E2E_FRAMEWORK_OVERVIEW.md`)
- Updated README with parallel testing commands
- Added test patterns and best practices

### Breaking Changes
- Old test suites moved to `e2e - DEPRECATED/` directory
- Test execution now requires extending `BaseTestSuite` class

### Testing
Run tests with: `bun test:e2e` or `bun test:e2e:parallel` (with
concurrency limits)

### TODO
- [ ] Implement good test examples.
- [ ] Implement useful test utils (like waiting for an event to show up
in DataHaven or Ethereum).
- [ ] Enforce tests with CI (currently cannot be done due to
intermittent error when sending a transaction with PAPI).

---------

Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: undercover-cactus <lola@moonsonglabs.com>
2025-07-16 18:51:07 +02:00
Steve Degosserie
c3e6f1258b
feat: Deployment improvements & environmental overrides (#103)
This PR contains improvements to the DataHaven deployment
infrastructure:
1. Directory restructure: Moved from `deployment/` to `deploy/` (more
common for K8s / Helm -based deployment configs).
2. Added **local environment** support: updated CLI to support deploying
to a local K8s cluster.
3. Manual deployment script: `deploy/scripts/deploy.sh` for manual
deployments.
4. Environment-specific configurations: Structured values files for each
environment.
5. Chart organization: Renamed bridges-common-relay to relay for
clarity.

---------

Co-authored-by: Gonza Montiel <gon.montiel@gmail.com>
Co-authored-by: Gonza Montiel <gonzamontiel@users.noreply.github.com>
2025-06-26 13:48:33 +02:00
Facundo Farall
d2bf185bcc
feat: 🚀 Add deploy command to CLI (#87)
### New Features
1. Add the `deploy` command to our CLI.
1. Conditionally deploys kurtosis eth network if we're in `stagenet`
environment.
    2. Deploys DH nodes.
3. Deploys contracts (all of them). In `mainnet` and `testnet` it
shouldn't deploy EL contracts, but for now that's not implemented.
4. Configures parameters, validators and relayers in the same way as
`launch`.
5. Currently, it only deploys `beefy` and `beacon` relayers, `execution`
and `solochain` relayers are pending for a subsequent PR.
2. Add `waitFor` utility function that receives a lambda.

### Refactors
1. Several common functionalities used both by the `launch` and `deploy`
command have been moved to the `cli/handlers/common` directory, from
where both commands use them. These include
    1. Checks for installed dependencies.
    2. Common constants.
    3. The `LaunchedNetwork` class has been moved to this directory.
    4. DataHaven nodes common functions.
    5. Kurtosis common functions.
    6. Relayer common functions.
7. Kubernetes functions (although only used by `deploy`, it seemed
fitting to have it here still).
8. Remove CLI questions and separator prints from `deploy-contracts.ts`
and `set-datahaven-parameters.ts` scripts. These things should be in the
`cli/launch` folder, which consumes the functions in these scripts.
9. Remove `setParametersFromCollection` from `utils` folder and put it
in `cli`.
10. Create base snowbridge relayer configs for `local` and `stagenet` as
two separate directories.

### Fixes
1. Sets the default time of the `deploy` command to 6s as Lodestar is
slower than Lighthouse.
2. In `runShellCommandWithLogger` only print `stderr` if the command
fails.

### Additional Minor Changes
1. K8s secret key names changed from `dh-beefy-relay-eth-key` to
`dh-beefy-relay-ethereum-key` and `dh-beacon-relay-sub-key` to
`dh-beacon-relay-substrate-key`, for simplicity in the deployment
script.
11. Update suggested configs for `.vscode` configs.

---------

Co-authored-by: Steve Degosserie <723552+stiiifff@users.noreply.github.com>
2025-06-12 10:24:03 +02:00