datahaven/test/cli/index.ts

286 lines
12 KiB
TypeScript
Raw Permalink Normal View History

#!/usr/bin/env bun
test: ⚡️ CI Refactor (#59) Eventually our CI will be required to run two private blockchains locally plus associated relayers. This PR is to prepare for this fate by improving run times and refactoring our existing CIs so they are a bit easier to reason about. ### Refactors - **_We now run ALL CIs on every PR!_** This is so that we decomplexify the logic around conditional builds and fetching built binaries from another source. This reduces the surface area of code we have to maintain at the cost of execution time - This penalty is ameliorated by a layered caching system. At best, it will be less than a minute to complete a build since everything will be cached. On GH runners this is about 6 minutes sadly. - We will no longer be at risk of important CIs being skipped erroneously which hide true failures. - Caching is a low-risk approach because at worst it has to build from scratch. A bad cache hit will never imply the wrong thing gets build since cargo is smart enough to just throw away any inappropriate build artefacts. - `setup-rust` action created so we have a unified way of setting up runner and unifying our approach to caching - Use a unique caching key for different activities and it will fallback to shared cache if no matches - we are using `mainnet` kurtosis config so that it works with relayer assumptions ### Additions - We can specify the ethereum block time via a new cli arg `--slot-time <seconds>` - We can specify arbitrary network_param args which get passed into the generated yaml - e.g. giving `bun cli --kurtosis-network-args="pet=cat food=fish" will add: ```yml network_params: # existing params... pet: cat food: fish ``` - We now have the ability to programmatically modify the yaml - This means we are back down to a single `minimal.yml` kurtosis config so we dont have to maintain changes between them - Flow is: `add new cli arg` -> `add if() block which mutates yaml` -> `profit` --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Facundo Farall <37149322+ffarall@users.noreply.github.com>
2025-05-06 20:20:02 +00:00
import { Command, InvalidArgumentError } from "@commander-js/extra-typings";
feat: 🚀 Add `deploy` command to CLI (#87) ### New Features 1. Add the `deploy` command to our CLI. 1. Conditionally deploys kurtosis eth network if we're in `stagenet` environment. 2. Deploys DH nodes. 3. Deploys contracts (all of them). In `mainnet` and `testnet` it shouldn't deploy EL contracts, but for now that's not implemented. 4. Configures parameters, validators and relayers in the same way as `launch`. 5. Currently, it only deploys `beefy` and `beacon` relayers, `execution` and `solochain` relayers are pending for a subsequent PR. 2. Add `waitFor` utility function that receives a lambda. ### Refactors 1. Several common functionalities used both by the `launch` and `deploy` command have been moved to the `cli/handlers/common` directory, from where both commands use them. These include 1. Checks for installed dependencies. 2. Common constants. 3. The `LaunchedNetwork` class has been moved to this directory. 4. DataHaven nodes common functions. 5. Kurtosis common functions. 6. Relayer common functions. 7. Kubernetes functions (although only used by `deploy`, it seemed fitting to have it here still). 8. Remove CLI questions and separator prints from `deploy-contracts.ts` and `set-datahaven-parameters.ts` scripts. These things should be in the `cli/launch` folder, which consumes the functions in these scripts. 9. Remove `setParametersFromCollection` from `utils` folder and put it in `cli`. 10. Create base snowbridge relayer configs for `local` and `stagenet` as two separate directories. ### Fixes 1. Sets the default time of the `deploy` command to 6s as Lodestar is slower than Lighthouse. 2. In `runShellCommandWithLogger` only print `stderr` if the command fails. ### Additional Minor Changes 1. K8s secret key names changed from `dh-beefy-relay-eth-key` to `dh-beefy-relay-ethereum-key` and `dh-beacon-relay-sub-key` to `dh-beacon-relay-substrate-key`, for simplicity in the deployment script. 11. Update suggested configs for `.vscode` configs. --------- Co-authored-by: Steve Degosserie <723552+stiiifff@users.noreply.github.com>
2025-06-12 08:24:03 +00:00
import type { DeployEnvironment } from "utils";
import {
feat: ✨ Datahaven contracts deployment on public testnet (#123) ## Summary This PR introduces support for deploying Datahaven contracts to different chains (hoodi, holesky, mainnet), as well as a new cli command to manage this deployment separately from the regular deployment, while maintaining compatibility with it. #### New CLI command - **`bun cli contracts deploy`** - Deploy contracts to supported chains (Hoodi, Holesky, Mainnet) - **`bun cli contracts status`** - Check deployment configuration and status - **`bun cli contracts verify`** - Verify contracts on block explorers - Commands need the chain parameter: `--chain <hoodi | holesky | mainnet>` - Right now only `hoodi` and `holesky` are supported ### Deployment #### Hoodi & Holesky Network Support - Added **DeployBase.s.sol** as common ground for **DeployTestnet.s.sol** (also new) and **DeployLocal.s.sol** (existing). - **Hoodi configuration** (`contracts/config/hoodi.json`) with deployed EigenLayer contract addresses to reference. - **Holesky configuration** (`contracts/config/hoodi.json`) with deployed EigenLayer contract addresses to reference. #### Contracts being deployed - **DataHaven**: ServiceManager, VetoableSlasher, RewardsRegistry - **Snowbridge**: BeefyClient, AgentExecutor, Gateway, RewardsAgent - **EigenLayer**: References existing deployed contracts (not re-deployed) #### Deployment files When the deployment is done, a new file under `contracts/deployments` is generated with the addresses of the deployed contracts, for each chain (it will be overriden per chain if run multiple times). So we would have one `anvil.json`, `hoodi.json`, `holesky.json`, etc, with the addresses of the deployed contracts for reference and for later verification. #### Todo - [x] Test compatibility with existing `bun cli launch` and `bun cli deploy` commands #### For follow-up PRs - Fix verification issue with `foundry verify-contracts` when specifying the `chain` or `chain-id` parameter, needed for hoodi (https://github.com/foundry-rs/foundry/issues/7466). - Add `redeploy` feature to only override implementation contract and leave the proxy address untouched ## Usage Examples ```bash # Deploy to Hoodi network bun cli contracts deploy --chain hoodi # Check deployment status bun cli contracts status --chain hoodi # Verify contracts on block explorer bun cli contracts verify --chain hoodi ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added deployment and configuration support for new networks "hoodi" and "holesky", including new configuration and deployment files. * Introduced a CLI tool for managing contract deployments, status checks, and verification across supported chains. * Added example environment configuration and comprehensive deployment documentation. * Enabled contract verification and status reporting via the CLI with support for block explorer integration. * **Improvements** * Refactored deployment scripts for modularity, supporting both local and testnet environments. * Centralized and extended configuration loading to support additional contract addresses and network parameters. * Enhanced deployment utilities and typings to support multi-network deployments. * **Bug Fixes** * Improved input validation and error handling in CLI commands and deployment scripts. * Added explicit handling for zero address in operator strategy retrieval. * **Chores** * Updated documentation and configuration templates for easier onboarding and deployment management. * Improved logging and output formatting for deployment and verification processes. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Steve Degosserie <723552+stiiifff@users.noreply.github.com> Co-authored-by: Ahmad Kaouk <56095276+ahmadkaouk@users.noreply.github.com>
2025-08-21 10:02:31 +00:00
contractsCheck,
contractsDeploy,
contractsPreActionHook,
contractsVerify,
feat: 🚀 Add `deploy` command to CLI (#87) ### New Features 1. Add the `deploy` command to our CLI. 1. Conditionally deploys kurtosis eth network if we're in `stagenet` environment. 2. Deploys DH nodes. 3. Deploys contracts (all of them). In `mainnet` and `testnet` it shouldn't deploy EL contracts, but for now that's not implemented. 4. Configures parameters, validators and relayers in the same way as `launch`. 5. Currently, it only deploys `beefy` and `beacon` relayers, `execution` and `solochain` relayers are pending for a subsequent PR. 2. Add `waitFor` utility function that receives a lambda. ### Refactors 1. Several common functionalities used both by the `launch` and `deploy` command have been moved to the `cli/handlers/common` directory, from where both commands use them. These include 1. Checks for installed dependencies. 2. Common constants. 3. The `LaunchedNetwork` class has been moved to this directory. 4. DataHaven nodes common functions. 5. Kurtosis common functions. 6. Relayer common functions. 7. Kubernetes functions (although only used by `deploy`, it seemed fitting to have it here still). 8. Remove CLI questions and separator prints from `deploy-contracts.ts` and `set-datahaven-parameters.ts` scripts. These things should be in the `cli/launch` folder, which consumes the functions in these scripts. 9. Remove `setParametersFromCollection` from `utils` folder and put it in `cli`. 10. Create base snowbridge relayer configs for `local` and `stagenet` as two separate directories. ### Fixes 1. Sets the default time of the `deploy` command to 6s as Lodestar is slower than Lighthouse. 2. In `runShellCommandWithLogger` only print `stderr` if the command fails. ### Additional Minor Changes 1. K8s secret key names changed from `dh-beefy-relay-eth-key` to `dh-beefy-relay-ethereum-key` and `dh-beacon-relay-sub-key` to `dh-beacon-relay-substrate-key`, for simplicity in the deployment script. 11. Update suggested configs for `.vscode` configs. --------- Co-authored-by: Steve Degosserie <723552+stiiifff@users.noreply.github.com>
2025-06-12 08:24:03 +00:00
deploy,
deployPreActionHook,
launch,
launchPreActionHook,
stop,
stopPreActionHook,
updateAVSMetadataURI
feat: 🚀 Add `deploy` command to CLI (#87) ### New Features 1. Add the `deploy` command to our CLI. 1. Conditionally deploys kurtosis eth network if we're in `stagenet` environment. 2. Deploys DH nodes. 3. Deploys contracts (all of them). In `mainnet` and `testnet` it shouldn't deploy EL contracts, but for now that's not implemented. 4. Configures parameters, validators and relayers in the same way as `launch`. 5. Currently, it only deploys `beefy` and `beacon` relayers, `execution` and `solochain` relayers are pending for a subsequent PR. 2. Add `waitFor` utility function that receives a lambda. ### Refactors 1. Several common functionalities used both by the `launch` and `deploy` command have been moved to the `cli/handlers/common` directory, from where both commands use them. These include 1. Checks for installed dependencies. 2. Common constants. 3. The `LaunchedNetwork` class has been moved to this directory. 4. DataHaven nodes common functions. 5. Kurtosis common functions. 6. Relayer common functions. 7. Kubernetes functions (although only used by `deploy`, it seemed fitting to have it here still). 8. Remove CLI questions and separator prints from `deploy-contracts.ts` and `set-datahaven-parameters.ts` scripts. These things should be in the `cli/launch` folder, which consumes the functions in these scripts. 9. Remove `setParametersFromCollection` from `utils` folder and put it in `cli`. 10. Create base snowbridge relayer configs for `local` and `stagenet` as two separate directories. ### Fixes 1. Sets the default time of the `deploy` command to 6s as Lodestar is slower than Lighthouse. 2. In `runShellCommandWithLogger` only print `stderr` if the command fails. ### Additional Minor Changes 1. K8s secret key names changed from `dh-beefy-relay-eth-key` to `dh-beefy-relay-ethereum-key` and `dh-beacon-relay-sub-key` to `dh-beacon-relay-substrate-key`, for simplicity in the deployment script. 11. Update suggested configs for `.vscode` configs. --------- Co-authored-by: Steve Degosserie <723552+stiiifff@users.noreply.github.com>
2025-06-12 08:24:03 +00:00
} from "./handlers";
test: ⚡️ CI Refactor (#59) Eventually our CI will be required to run two private blockchains locally plus associated relayers. This PR is to prepare for this fate by improving run times and refactoring our existing CIs so they are a bit easier to reason about. ### Refactors - **_We now run ALL CIs on every PR!_** This is so that we decomplexify the logic around conditional builds and fetching built binaries from another source. This reduces the surface area of code we have to maintain at the cost of execution time - This penalty is ameliorated by a layered caching system. At best, it will be less than a minute to complete a build since everything will be cached. On GH runners this is about 6 minutes sadly. - We will no longer be at risk of important CIs being skipped erroneously which hide true failures. - Caching is a low-risk approach because at worst it has to build from scratch. A bad cache hit will never imply the wrong thing gets build since cargo is smart enough to just throw away any inappropriate build artefacts. - `setup-rust` action created so we have a unified way of setting up runner and unifying our approach to caching - Use a unique caching key for different activities and it will fallback to shared cache if no matches - we are using `mainnet` kurtosis config so that it works with relayer assumptions ### Additions - We can specify the ethereum block time via a new cli arg `--slot-time <seconds>` - We can specify arbitrary network_param args which get passed into the generated yaml - e.g. giving `bun cli --kurtosis-network-args="pet=cat food=fish" will add: ```yml network_params: # existing params... pet: cat food: fish ``` - We now have the ability to programmatically modify the yaml - This means we are back down to a single `minimal.yml` kurtosis config so we dont have to maintain changes between them - Flow is: `add new cli arg` -> `add if() block which mutates yaml` -> `profit` --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Facundo Farall <37149322+ffarall@users.noreply.github.com>
2025-05-06 20:20:02 +00:00
// Function to parse integer
function parseIntValue(value: string): number {
const parsedValue = Number.parseInt(value, 10);
if (Number.isNaN(parsedValue)) {
throw new InvalidArgumentError("Not a number.");
}
return parsedValue;
}
feat: 🚀 Add `deploy` command to CLI (#87) ### New Features 1. Add the `deploy` command to our CLI. 1. Conditionally deploys kurtosis eth network if we're in `stagenet` environment. 2. Deploys DH nodes. 3. Deploys contracts (all of them). In `mainnet` and `testnet` it shouldn't deploy EL contracts, but for now that's not implemented. 4. Configures parameters, validators and relayers in the same way as `launch`. 5. Currently, it only deploys `beefy` and `beacon` relayers, `execution` and `solochain` relayers are pending for a subsequent PR. 2. Add `waitFor` utility function that receives a lambda. ### Refactors 1. Several common functionalities used both by the `launch` and `deploy` command have been moved to the `cli/handlers/common` directory, from where both commands use them. These include 1. Checks for installed dependencies. 2. Common constants. 3. The `LaunchedNetwork` class has been moved to this directory. 4. DataHaven nodes common functions. 5. Kurtosis common functions. 6. Relayer common functions. 7. Kubernetes functions (although only used by `deploy`, it seemed fitting to have it here still). 8. Remove CLI questions and separator prints from `deploy-contracts.ts` and `set-datahaven-parameters.ts` scripts. These things should be in the `cli/launch` folder, which consumes the functions in these scripts. 9. Remove `setParametersFromCollection` from `utils` folder and put it in `cli`. 10. Create base snowbridge relayer configs for `local` and `stagenet` as two separate directories. ### Fixes 1. Sets the default time of the `deploy` command to 6s as Lodestar is slower than Lighthouse. 2. In `runShellCommandWithLogger` only print `stderr` if the command fails. ### Additional Minor Changes 1. K8s secret key names changed from `dh-beefy-relay-eth-key` to `dh-beefy-relay-ethereum-key` and `dh-beacon-relay-sub-key` to `dh-beacon-relay-substrate-key`, for simplicity in the deployment script. 11. Update suggested configs for `.vscode` configs. --------- Co-authored-by: Steve Degosserie <723552+stiiifff@users.noreply.github.com>
2025-06-12 08:24:03 +00:00
// Function to parse and validate DeployEnvironment
function parseDeployEnvironment(value: string): DeployEnvironment {
if (value === "local" || value === "stagenet" || value === "testnet" || value === "mainnet") {
feat: 🚀 Add `deploy` command to CLI (#87) ### New Features 1. Add the `deploy` command to our CLI. 1. Conditionally deploys kurtosis eth network if we're in `stagenet` environment. 2. Deploys DH nodes. 3. Deploys contracts (all of them). In `mainnet` and `testnet` it shouldn't deploy EL contracts, but for now that's not implemented. 4. Configures parameters, validators and relayers in the same way as `launch`. 5. Currently, it only deploys `beefy` and `beacon` relayers, `execution` and `solochain` relayers are pending for a subsequent PR. 2. Add `waitFor` utility function that receives a lambda. ### Refactors 1. Several common functionalities used both by the `launch` and `deploy` command have been moved to the `cli/handlers/common` directory, from where both commands use them. These include 1. Checks for installed dependencies. 2. Common constants. 3. The `LaunchedNetwork` class has been moved to this directory. 4. DataHaven nodes common functions. 5. Kurtosis common functions. 6. Relayer common functions. 7. Kubernetes functions (although only used by `deploy`, it seemed fitting to have it here still). 8. Remove CLI questions and separator prints from `deploy-contracts.ts` and `set-datahaven-parameters.ts` scripts. These things should be in the `cli/launch` folder, which consumes the functions in these scripts. 9. Remove `setParametersFromCollection` from `utils` folder and put it in `cli`. 10. Create base snowbridge relayer configs for `local` and `stagenet` as two separate directories. ### Fixes 1. Sets the default time of the `deploy` command to 6s as Lodestar is slower than Lighthouse. 2. In `runShellCommandWithLogger` only print `stderr` if the command fails. ### Additional Minor Changes 1. K8s secret key names changed from `dh-beefy-relay-eth-key` to `dh-beefy-relay-ethereum-key` and `dh-beacon-relay-sub-key` to `dh-beacon-relay-substrate-key`, for simplicity in the deployment script. 11. Update suggested configs for `.vscode` configs. --------- Co-authored-by: Steve Degosserie <723552+stiiifff@users.noreply.github.com>
2025-06-12 08:24:03 +00:00
return value;
}
throw new InvalidArgumentError(
"Invalid environment. Must be one of 'local', 'stagenet', 'testnet', or 'mainnet'."
feat: 🚀 Add `deploy` command to CLI (#87) ### New Features 1. Add the `deploy` command to our CLI. 1. Conditionally deploys kurtosis eth network if we're in `stagenet` environment. 2. Deploys DH nodes. 3. Deploys contracts (all of them). In `mainnet` and `testnet` it shouldn't deploy EL contracts, but for now that's not implemented. 4. Configures parameters, validators and relayers in the same way as `launch`. 5. Currently, it only deploys `beefy` and `beacon` relayers, `execution` and `solochain` relayers are pending for a subsequent PR. 2. Add `waitFor` utility function that receives a lambda. ### Refactors 1. Several common functionalities used both by the `launch` and `deploy` command have been moved to the `cli/handlers/common` directory, from where both commands use them. These include 1. Checks for installed dependencies. 2. Common constants. 3. The `LaunchedNetwork` class has been moved to this directory. 4. DataHaven nodes common functions. 5. Kurtosis common functions. 6. Relayer common functions. 7. Kubernetes functions (although only used by `deploy`, it seemed fitting to have it here still). 8. Remove CLI questions and separator prints from `deploy-contracts.ts` and `set-datahaven-parameters.ts` scripts. These things should be in the `cli/launch` folder, which consumes the functions in these scripts. 9. Remove `setParametersFromCollection` from `utils` folder and put it in `cli`. 10. Create base snowbridge relayer configs for `local` and `stagenet` as two separate directories. ### Fixes 1. Sets the default time of the `deploy` command to 6s as Lodestar is slower than Lighthouse. 2. In `runShellCommandWithLogger` only print `stderr` if the command fails. ### Additional Minor Changes 1. K8s secret key names changed from `dh-beefy-relay-eth-key` to `dh-beefy-relay-ethereum-key` and `dh-beacon-relay-sub-key` to `dh-beacon-relay-substrate-key`, for simplicity in the deployment script. 11. Update suggested configs for `.vscode` configs. --------- Co-authored-by: Steve Degosserie <723552+stiiifff@users.noreply.github.com>
2025-06-12 08:24:03 +00:00
);
}
// ===== Program =====
const program = new Command()
.version("0.2.0")
.name("bun cli")
.summary("🫎 DataHaven CLI: Network Toolbox")
.usage("[options]");
feat: 🚀 Add `deploy` command to CLI (#87) ### New Features 1. Add the `deploy` command to our CLI. 1. Conditionally deploys kurtosis eth network if we're in `stagenet` environment. 2. Deploys DH nodes. 3. Deploys contracts (all of them). In `mainnet` and `testnet` it shouldn't deploy EL contracts, but for now that's not implemented. 4. Configures parameters, validators and relayers in the same way as `launch`. 5. Currently, it only deploys `beefy` and `beacon` relayers, `execution` and `solochain` relayers are pending for a subsequent PR. 2. Add `waitFor` utility function that receives a lambda. ### Refactors 1. Several common functionalities used both by the `launch` and `deploy` command have been moved to the `cli/handlers/common` directory, from where both commands use them. These include 1. Checks for installed dependencies. 2. Common constants. 3. The `LaunchedNetwork` class has been moved to this directory. 4. DataHaven nodes common functions. 5. Kurtosis common functions. 6. Relayer common functions. 7. Kubernetes functions (although only used by `deploy`, it seemed fitting to have it here still). 8. Remove CLI questions and separator prints from `deploy-contracts.ts` and `set-datahaven-parameters.ts` scripts. These things should be in the `cli/launch` folder, which consumes the functions in these scripts. 9. Remove `setParametersFromCollection` from `utils` folder and put it in `cli`. 10. Create base snowbridge relayer configs for `local` and `stagenet` as two separate directories. ### Fixes 1. Sets the default time of the `deploy` command to 6s as Lodestar is slower than Lighthouse. 2. In `runShellCommandWithLogger` only print `stderr` if the command fails. ### Additional Minor Changes 1. K8s secret key names changed from `dh-beefy-relay-eth-key` to `dh-beefy-relay-ethereum-key` and `dh-beacon-relay-sub-key` to `dh-beacon-relay-substrate-key`, for simplicity in the deployment script. 11. Update suggested configs for `.vscode` configs. --------- Co-authored-by: Steve Degosserie <723552+stiiifff@users.noreply.github.com>
2025-06-12 08:24:03 +00:00
// ===== Deploy ======
program
.command("deploy")
.addHelpText(
"before",
`🫎 DataHaven: Network Deployer CLI for deploying a full DataHaven network stack to a Kubernetes cluster
It will deploy:
- DataHaven solochain validators (all envs),
feat: ✨ Add deployment charts for StorageHub MSP, BSP & Indexer nodes (Local & Stagenet envs) (#160) ## Summary This PR adds comprehensive Kubernetes deployment infrastructure for StorageHub components, enabling deployment of the full StorageHub network stack (MSP, BSP, Indexer, and Fisherman nodes) alongside DataHaven nodes in both local and stagenet environments. ### What's Added **1. New Helm Chart: StorageHub MSP Backend API** (`deploy/charts/backend/`) - REST API service for StorageHub operations - Connects to PostgreSQL database for indexed blockchain data - Connects to RPC nodes for real-time blockchain queries - Configurable via TOML configuration file - Supports environment-specific overrides - Includes comprehensive documentation **2. StorageHub Node Deployment Charts** (`deploy/charts/node/storagehub/`) - **MSP Node** (`sh-mspnode`): Main Service Provider nodes with charging capabilities - **BSP Node** (`sh-bspnode`): Backup Service Provider nodes for redundancy - **Indexer Node** (`sh-idxnode`): Full indexing node with PostgreSQL integration - **Fisherman Node** (`sh-fisherman`): Network monitoring and verification node **3. Environment Configurations** - **Local environment** (`deploy/environments/local/`): Development setup with hostpath storage - **Stagenet environment** (`deploy/environments/stagenet/`): Production-like setup with AWS EBS - PostgreSQL database configurations for Indexer and Fisherman nodes - Proper service discovery and network configuration **4. Enhanced CLI Tooling** (`test/cli/`) - New `deploy storagehub` command for deploying StorageHub components - Updated `launch storagehub` command for local testing - Interactive deployment with environment selection - Automatic database provisioning via Bitnami PostgreSQL charts **5. Node Configuration Improvements** - Fork-aware transaction pool for DH boot & validator nodes - Unsafe RPC methods exposed on MSP nodes (for provider operations) - JWT secret support for MSP Backend authentication - ECDSA key scheme for StorageHub BCSV keys (DataHaven compatibility) ### Architecture ``` StorageHub Stack: ├── MSP Nodes (2 replicas) → Storage providers with charging ├── BSP Nodes (2 replicas) → Backup storage providers ├── Indexer Node → Database indexing + PostgreSQL ├── Fisherman Node → Monitoring + PostgreSQL (shared with Indexer) └── MSP Backend API → REST API for StorageHub operations ``` ### Testing **Local Testing**: ```bash cd test bun cli launch storagehub # Interactive launcher # or bun cli deploy storagehub # Deploy via Helm ``` **Stagenet Deployment**: ```bash cd deploy helm install sh-mspnode ./charts/node \ -f ./charts/node/storagehub/sh-mspnode.yaml \ -f ./environments/stagenet/sh-mspnode.yaml \ -n datahaven-stagenet ``` ### Breaking Changes None - This is purely additive infrastructure. ### Migration Notes For existing deployments: 1. DataHaven nodes now use `--pool-type fork-aware` flag 2. Bootnode and validator node configs updated accordingly 3. No action required for existing DataHaven-only deployments
2025-10-21 20:18:50 +00:00
- StorageHub components: MSP, BSP, Indexer, Fisherman nodes and databases (local & stagenet envs),
feat: 🚀 Add `deploy` command to CLI (#87) ### New Features 1. Add the `deploy` command to our CLI. 1. Conditionally deploys kurtosis eth network if we're in `stagenet` environment. 2. Deploys DH nodes. 3. Deploys contracts (all of them). In `mainnet` and `testnet` it shouldn't deploy EL contracts, but for now that's not implemented. 4. Configures parameters, validators and relayers in the same way as `launch`. 5. Currently, it only deploys `beefy` and `beacon` relayers, `execution` and `solochain` relayers are pending for a subsequent PR. 2. Add `waitFor` utility function that receives a lambda. ### Refactors 1. Several common functionalities used both by the `launch` and `deploy` command have been moved to the `cli/handlers/common` directory, from where both commands use them. These include 1. Checks for installed dependencies. 2. Common constants. 3. The `LaunchedNetwork` class has been moved to this directory. 4. DataHaven nodes common functions. 5. Kurtosis common functions. 6. Relayer common functions. 7. Kubernetes functions (although only used by `deploy`, it seemed fitting to have it here still). 8. Remove CLI questions and separator prints from `deploy-contracts.ts` and `set-datahaven-parameters.ts` scripts. These things should be in the `cli/launch` folder, which consumes the functions in these scripts. 9. Remove `setParametersFromCollection` from `utils` folder and put it in `cli`. 10. Create base snowbridge relayer configs for `local` and `stagenet` as two separate directories. ### Fixes 1. Sets the default time of the `deploy` command to 6s as Lodestar is slower than Lighthouse. 2. In `runShellCommandWithLogger` only print `stderr` if the command fails. ### Additional Minor Changes 1. K8s secret key names changed from `dh-beefy-relay-eth-key` to `dh-beefy-relay-ethereum-key` and `dh-beacon-relay-sub-key` to `dh-beacon-relay-substrate-key`, for simplicity in the deployment script. 11. Update suggested configs for `.vscode` configs. --------- Co-authored-by: Steve Degosserie <723552+stiiifff@users.noreply.github.com>
2025-06-12 08:24:03 +00:00
- Kurtosis Ethereum private network (stagenet env),
- Snowbridge Relayers (all envs)
`
)
.description("Deploy a full DataHaven network stack to a Kubernetes cluster")
.option("--e, --environment <value>", "Environment to deploy to", parseDeployEnvironment, "local")
feat: 🚀 Add `deploy` command to CLI (#87) ### New Features 1. Add the `deploy` command to our CLI. 1. Conditionally deploys kurtosis eth network if we're in `stagenet` environment. 2. Deploys DH nodes. 3. Deploys contracts (all of them). In `mainnet` and `testnet` it shouldn't deploy EL contracts, but for now that's not implemented. 4. Configures parameters, validators and relayers in the same way as `launch`. 5. Currently, it only deploys `beefy` and `beacon` relayers, `execution` and `solochain` relayers are pending for a subsequent PR. 2. Add `waitFor` utility function that receives a lambda. ### Refactors 1. Several common functionalities used both by the `launch` and `deploy` command have been moved to the `cli/handlers/common` directory, from where both commands use them. These include 1. Checks for installed dependencies. 2. Common constants. 3. The `LaunchedNetwork` class has been moved to this directory. 4. DataHaven nodes common functions. 5. Kurtosis common functions. 6. Relayer common functions. 7. Kubernetes functions (although only used by `deploy`, it seemed fitting to have it here still). 8. Remove CLI questions and separator prints from `deploy-contracts.ts` and `set-datahaven-parameters.ts` scripts. These things should be in the `cli/launch` folder, which consumes the functions in these scripts. 9. Remove `setParametersFromCollection` from `utils` folder and put it in `cli`. 10. Create base snowbridge relayer configs for `local` and `stagenet` as two separate directories. ### Fixes 1. Sets the default time of the `deploy` command to 6s as Lodestar is slower than Lighthouse. 2. In `runShellCommandWithLogger` only print `stderr` if the command fails. ### Additional Minor Changes 1. K8s secret key names changed from `dh-beefy-relay-eth-key` to `dh-beefy-relay-ethereum-key` and `dh-beacon-relay-sub-key` to `dh-beacon-relay-substrate-key`, for simplicity in the deployment script. 11. Update suggested configs for `.vscode` configs. --------- Co-authored-by: Steve Degosserie <723552+stiiifff@users.noreply.github.com>
2025-06-12 08:24:03 +00:00
.option(
"--k, --kube-namespace <value>",
"Kubernetes namespace to deploy to. In 'stagenet' this parameter is ignored and the Kurtosis namespace is used instead. Default will be `datahaven-<environment>`."
)
.option(
"--ke, --kurtosis-enclave-name <value>",
"Name of the Kurtosis enclave",
"datahaven-local"
feat: 🚀 Add `deploy` command to CLI (#87) ### New Features 1. Add the `deploy` command to our CLI. 1. Conditionally deploys kurtosis eth network if we're in `stagenet` environment. 2. Deploys DH nodes. 3. Deploys contracts (all of them). In `mainnet` and `testnet` it shouldn't deploy EL contracts, but for now that's not implemented. 4. Configures parameters, validators and relayers in the same way as `launch`. 5. Currently, it only deploys `beefy` and `beacon` relayers, `execution` and `solochain` relayers are pending for a subsequent PR. 2. Add `waitFor` utility function that receives a lambda. ### Refactors 1. Several common functionalities used both by the `launch` and `deploy` command have been moved to the `cli/handlers/common` directory, from where both commands use them. These include 1. Checks for installed dependencies. 2. Common constants. 3. The `LaunchedNetwork` class has been moved to this directory. 4. DataHaven nodes common functions. 5. Kurtosis common functions. 6. Relayer common functions. 7. Kubernetes functions (although only used by `deploy`, it seemed fitting to have it here still). 8. Remove CLI questions and separator prints from `deploy-contracts.ts` and `set-datahaven-parameters.ts` scripts. These things should be in the `cli/launch` folder, which consumes the functions in these scripts. 9. Remove `setParametersFromCollection` from `utils` folder and put it in `cli`. 10. Create base snowbridge relayer configs for `local` and `stagenet` as two separate directories. ### Fixes 1. Sets the default time of the `deploy` command to 6s as Lodestar is slower than Lighthouse. 2. In `runShellCommandWithLogger` only print `stderr` if the command fails. ### Additional Minor Changes 1. K8s secret key names changed from `dh-beefy-relay-eth-key` to `dh-beefy-relay-ethereum-key` and `dh-beacon-relay-sub-key` to `dh-beacon-relay-substrate-key`, for simplicity in the deployment script. 11. Update suggested configs for `.vscode` configs. --------- Co-authored-by: Steve Degosserie <723552+stiiifff@users.noreply.github.com>
2025-06-12 08:24:03 +00:00
)
.option("--st, --slot-time <number>", "Set slot time in seconds", parseIntValue, 12)
.option("--kn, --kurtosis-network-args <value>", "CustomKurtosis network args")
.option("--v, --verified", "Verify smart contracts with Blockscout")
.option("--b, --blockscout", "Enable Blockscout")
.option(
"--dit, --datahaven-image-tag <value>",
"Tag of the datahaven image to use",
"datahavenxyz/datahaven:main"
feat: 🚀 Add `deploy` command to CLI (#87) ### New Features 1. Add the `deploy` command to our CLI. 1. Conditionally deploys kurtosis eth network if we're in `stagenet` environment. 2. Deploys DH nodes. 3. Deploys contracts (all of them). In `mainnet` and `testnet` it shouldn't deploy EL contracts, but for now that's not implemented. 4. Configures parameters, validators and relayers in the same way as `launch`. 5. Currently, it only deploys `beefy` and `beacon` relayers, `execution` and `solochain` relayers are pending for a subsequent PR. 2. Add `waitFor` utility function that receives a lambda. ### Refactors 1. Several common functionalities used both by the `launch` and `deploy` command have been moved to the `cli/handlers/common` directory, from where both commands use them. These include 1. Checks for installed dependencies. 2. Common constants. 3. The `LaunchedNetwork` class has been moved to this directory. 4. DataHaven nodes common functions. 5. Kurtosis common functions. 6. Relayer common functions. 7. Kubernetes functions (although only used by `deploy`, it seemed fitting to have it here still). 8. Remove CLI questions and separator prints from `deploy-contracts.ts` and `set-datahaven-parameters.ts` scripts. These things should be in the `cli/launch` folder, which consumes the functions in these scripts. 9. Remove `setParametersFromCollection` from `utils` folder and put it in `cli`. 10. Create base snowbridge relayer configs for `local` and `stagenet` as two separate directories. ### Fixes 1. Sets the default time of the `deploy` command to 6s as Lodestar is slower than Lighthouse. 2. In `runShellCommandWithLogger` only print `stderr` if the command fails. ### Additional Minor Changes 1. K8s secret key names changed from `dh-beefy-relay-eth-key` to `dh-beefy-relay-ethereum-key` and `dh-beacon-relay-sub-key` to `dh-beacon-relay-substrate-key`, for simplicity in the deployment script. 11. Update suggested configs for `.vscode` configs. --------- Co-authored-by: Steve Degosserie <723552+stiiifff@users.noreply.github.com>
2025-06-12 08:24:03 +00:00
)
.option(
"--el-rpc-url <value>",
"URL of the Ethereum Execution Layer (EL) RPC endpoint to use. In local & stagenet environments (private networks), the Kurtosis Ethereum network will be used. In testnet and mainnet environments (public networks), this parameter is required."
feat: 🚀 Add `deploy` command to CLI (#87) ### New Features 1. Add the `deploy` command to our CLI. 1. Conditionally deploys kurtosis eth network if we're in `stagenet` environment. 2. Deploys DH nodes. 3. Deploys contracts (all of them). In `mainnet` and `testnet` it shouldn't deploy EL contracts, but for now that's not implemented. 4. Configures parameters, validators and relayers in the same way as `launch`. 5. Currently, it only deploys `beefy` and `beacon` relayers, `execution` and `solochain` relayers are pending for a subsequent PR. 2. Add `waitFor` utility function that receives a lambda. ### Refactors 1. Several common functionalities used both by the `launch` and `deploy` command have been moved to the `cli/handlers/common` directory, from where both commands use them. These include 1. Checks for installed dependencies. 2. Common constants. 3. The `LaunchedNetwork` class has been moved to this directory. 4. DataHaven nodes common functions. 5. Kurtosis common functions. 6. Relayer common functions. 7. Kubernetes functions (although only used by `deploy`, it seemed fitting to have it here still). 8. Remove CLI questions and separator prints from `deploy-contracts.ts` and `set-datahaven-parameters.ts` scripts. These things should be in the `cli/launch` folder, which consumes the functions in these scripts. 9. Remove `setParametersFromCollection` from `utils` folder and put it in `cli`. 10. Create base snowbridge relayer configs for `local` and `stagenet` as two separate directories. ### Fixes 1. Sets the default time of the `deploy` command to 6s as Lodestar is slower than Lighthouse. 2. In `runShellCommandWithLogger` only print `stderr` if the command fails. ### Additional Minor Changes 1. K8s secret key names changed from `dh-beefy-relay-eth-key` to `dh-beefy-relay-ethereum-key` and `dh-beacon-relay-sub-key` to `dh-beacon-relay-substrate-key`, for simplicity in the deployment script. 11. Update suggested configs for `.vscode` configs. --------- Co-authored-by: Steve Degosserie <723552+stiiifff@users.noreply.github.com>
2025-06-12 08:24:03 +00:00
)
.option(
"--cl-endpoint <value>",
"URL of the Ethereum Consensus Layer (CL) endpoint to use. In local & stagenet environments (private networks), the Kurtosis Ethereum network will be used. In testnet and mainnet environments (public networks), this parameter is required."
feat: 🚀 Add `deploy` command to CLI (#87) ### New Features 1. Add the `deploy` command to our CLI. 1. Conditionally deploys kurtosis eth network if we're in `stagenet` environment. 2. Deploys DH nodes. 3. Deploys contracts (all of them). In `mainnet` and `testnet` it shouldn't deploy EL contracts, but for now that's not implemented. 4. Configures parameters, validators and relayers in the same way as `launch`. 5. Currently, it only deploys `beefy` and `beacon` relayers, `execution` and `solochain` relayers are pending for a subsequent PR. 2. Add `waitFor` utility function that receives a lambda. ### Refactors 1. Several common functionalities used both by the `launch` and `deploy` command have been moved to the `cli/handlers/common` directory, from where both commands use them. These include 1. Checks for installed dependencies. 2. Common constants. 3. The `LaunchedNetwork` class has been moved to this directory. 4. DataHaven nodes common functions. 5. Kurtosis common functions. 6. Relayer common functions. 7. Kubernetes functions (although only used by `deploy`, it seemed fitting to have it here still). 8. Remove CLI questions and separator prints from `deploy-contracts.ts` and `set-datahaven-parameters.ts` scripts. These things should be in the `cli/launch` folder, which consumes the functions in these scripts. 9. Remove `setParametersFromCollection` from `utils` folder and put it in `cli`. 10. Create base snowbridge relayer configs for `local` and `stagenet` as two separate directories. ### Fixes 1. Sets the default time of the `deploy` command to 6s as Lodestar is slower than Lighthouse. 2. In `runShellCommandWithLogger` only print `stderr` if the command fails. ### Additional Minor Changes 1. K8s secret key names changed from `dh-beefy-relay-eth-key` to `dh-beefy-relay-ethereum-key` and `dh-beacon-relay-sub-key` to `dh-beacon-relay-substrate-key`, for simplicity in the deployment script. 11. Update suggested configs for `.vscode` configs. --------- Co-authored-by: Steve Degosserie <723552+stiiifff@users.noreply.github.com>
2025-06-12 08:24:03 +00:00
)
.option(
"--rit, --relayer-image-tag <value>",
"Tag of the relayer image to use",
"datahavenxyz/snowbridge-relay:latest"
feat: 🚀 Add `deploy` command to CLI (#87) ### New Features 1. Add the `deploy` command to our CLI. 1. Conditionally deploys kurtosis eth network if we're in `stagenet` environment. 2. Deploys DH nodes. 3. Deploys contracts (all of them). In `mainnet` and `testnet` it shouldn't deploy EL contracts, but for now that's not implemented. 4. Configures parameters, validators and relayers in the same way as `launch`. 5. Currently, it only deploys `beefy` and `beacon` relayers, `execution` and `solochain` relayers are pending for a subsequent PR. 2. Add `waitFor` utility function that receives a lambda. ### Refactors 1. Several common functionalities used both by the `launch` and `deploy` command have been moved to the `cli/handlers/common` directory, from where both commands use them. These include 1. Checks for installed dependencies. 2. Common constants. 3. The `LaunchedNetwork` class has been moved to this directory. 4. DataHaven nodes common functions. 5. Kurtosis common functions. 6. Relayer common functions. 7. Kubernetes functions (although only used by `deploy`, it seemed fitting to have it here still). 8. Remove CLI questions and separator prints from `deploy-contracts.ts` and `set-datahaven-parameters.ts` scripts. These things should be in the `cli/launch` folder, which consumes the functions in these scripts. 9. Remove `setParametersFromCollection` from `utils` folder and put it in `cli`. 10. Create base snowbridge relayer configs for `local` and `stagenet` as two separate directories. ### Fixes 1. Sets the default time of the `deploy` command to 6s as Lodestar is slower than Lighthouse. 2. In `runShellCommandWithLogger` only print `stderr` if the command fails. ### Additional Minor Changes 1. K8s secret key names changed from `dh-beefy-relay-eth-key` to `dh-beefy-relay-ethereum-key` and `dh-beacon-relay-sub-key` to `dh-beacon-relay-substrate-key`, for simplicity in the deployment script. 11. Update suggested configs for `.vscode` configs. --------- Co-authored-by: Steve Degosserie <723552+stiiifff@users.noreply.github.com>
2025-06-12 08:24:03 +00:00
)
.option("--docker-username <value>", "Docker Hub username")
.option("--docker-password <value>", "Docker Hub password")
.option("--docker-email <value>", "Docker Hub email")
.option("--chainspec <value>", "Absolute path to custom chainspec file")
feat: 🚀 Add `deploy` command to CLI (#87) ### New Features 1. Add the `deploy` command to our CLI. 1. Conditionally deploys kurtosis eth network if we're in `stagenet` environment. 2. Deploys DH nodes. 3. Deploys contracts (all of them). In `mainnet` and `testnet` it shouldn't deploy EL contracts, but for now that's not implemented. 4. Configures parameters, validators and relayers in the same way as `launch`. 5. Currently, it only deploys `beefy` and `beacon` relayers, `execution` and `solochain` relayers are pending for a subsequent PR. 2. Add `waitFor` utility function that receives a lambda. ### Refactors 1. Several common functionalities used both by the `launch` and `deploy` command have been moved to the `cli/handlers/common` directory, from where both commands use them. These include 1. Checks for installed dependencies. 2. Common constants. 3. The `LaunchedNetwork` class has been moved to this directory. 4. DataHaven nodes common functions. 5. Kurtosis common functions. 6. Relayer common functions. 7. Kubernetes functions (although only used by `deploy`, it seemed fitting to have it here still). 8. Remove CLI questions and separator prints from `deploy-contracts.ts` and `set-datahaven-parameters.ts` scripts. These things should be in the `cli/launch` folder, which consumes the functions in these scripts. 9. Remove `setParametersFromCollection` from `utils` folder and put it in `cli`. 10. Create base snowbridge relayer configs for `local` and `stagenet` as two separate directories. ### Fixes 1. Sets the default time of the `deploy` command to 6s as Lodestar is slower than Lighthouse. 2. In `runShellCommandWithLogger` only print `stderr` if the command fails. ### Additional Minor Changes 1. K8s secret key names changed from `dh-beefy-relay-eth-key` to `dh-beefy-relay-ethereum-key` and `dh-beacon-relay-sub-key` to `dh-beacon-relay-substrate-key`, for simplicity in the deployment script. 11. Update suggested configs for `.vscode` configs. --------- Co-authored-by: Steve Degosserie <723552+stiiifff@users.noreply.github.com>
2025-06-12 08:24:03 +00:00
.option("--skip-cleanup", "Skip cleaning up the network", false)
.option("--skip-kurtosis", "Skip deploying Kurtosis Ethereum private network", false)
.option("--skip-datahaven-solochain", "Skip deploying DataHaven solochain validators", false)
.option("--skip-contracts", "Skip deploying smart contracts", false)
.option("--skip-validator-operations", "Skip performing validator operations", false)
.option("--skip-set-parameters", "Skip setting DataHaven runtime parameters", false)
.option("--skip-relayers", "Skip deploying Snowbridge Relayers", false)
feat: ✨ Add deployment charts for StorageHub MSP, BSP & Indexer nodes (Local & Stagenet envs) (#160) ## Summary This PR adds comprehensive Kubernetes deployment infrastructure for StorageHub components, enabling deployment of the full StorageHub network stack (MSP, BSP, Indexer, and Fisherman nodes) alongside DataHaven nodes in both local and stagenet environments. ### What's Added **1. New Helm Chart: StorageHub MSP Backend API** (`deploy/charts/backend/`) - REST API service for StorageHub operations - Connects to PostgreSQL database for indexed blockchain data - Connects to RPC nodes for real-time blockchain queries - Configurable via TOML configuration file - Supports environment-specific overrides - Includes comprehensive documentation **2. StorageHub Node Deployment Charts** (`deploy/charts/node/storagehub/`) - **MSP Node** (`sh-mspnode`): Main Service Provider nodes with charging capabilities - **BSP Node** (`sh-bspnode`): Backup Service Provider nodes for redundancy - **Indexer Node** (`sh-idxnode`): Full indexing node with PostgreSQL integration - **Fisherman Node** (`sh-fisherman`): Network monitoring and verification node **3. Environment Configurations** - **Local environment** (`deploy/environments/local/`): Development setup with hostpath storage - **Stagenet environment** (`deploy/environments/stagenet/`): Production-like setup with AWS EBS - PostgreSQL database configurations for Indexer and Fisherman nodes - Proper service discovery and network configuration **4. Enhanced CLI Tooling** (`test/cli/`) - New `deploy storagehub` command for deploying StorageHub components - Updated `launch storagehub` command for local testing - Interactive deployment with environment selection - Automatic database provisioning via Bitnami PostgreSQL charts **5. Node Configuration Improvements** - Fork-aware transaction pool for DH boot & validator nodes - Unsafe RPC methods exposed on MSP nodes (for provider operations) - JWT secret support for MSP Backend authentication - ECDSA key scheme for StorageHub BCSV keys (DataHaven compatibility) ### Architecture ``` StorageHub Stack: ├── MSP Nodes (2 replicas) → Storage providers with charging ├── BSP Nodes (2 replicas) → Backup storage providers ├── Indexer Node → Database indexing + PostgreSQL ├── Fisherman Node → Monitoring + PostgreSQL (shared with Indexer) └── MSP Backend API → REST API for StorageHub operations ``` ### Testing **Local Testing**: ```bash cd test bun cli launch storagehub # Interactive launcher # or bun cli deploy storagehub # Deploy via Helm ``` **Stagenet Deployment**: ```bash cd deploy helm install sh-mspnode ./charts/node \ -f ./charts/node/storagehub/sh-mspnode.yaml \ -f ./environments/stagenet/sh-mspnode.yaml \ -n datahaven-stagenet ``` ### Breaking Changes None - This is purely additive infrastructure. ### Migration Notes For existing deployments: 1. DataHaven nodes now use `--pool-type fork-aware` flag 2. Bootnode and validator node configs updated accordingly 3. No action required for existing DataHaven-only deployments
2025-10-21 20:18:50 +00:00
.option(
"--skip-storage-hub",
"Skip deploying StorageHub components (MSP, BSP, Indexer, Fisherman, databases)",
false
)
feat: 🚀 Add `deploy` command to CLI (#87) ### New Features 1. Add the `deploy` command to our CLI. 1. Conditionally deploys kurtosis eth network if we're in `stagenet` environment. 2. Deploys DH nodes. 3. Deploys contracts (all of them). In `mainnet` and `testnet` it shouldn't deploy EL contracts, but for now that's not implemented. 4. Configures parameters, validators and relayers in the same way as `launch`. 5. Currently, it only deploys `beefy` and `beacon` relayers, `execution` and `solochain` relayers are pending for a subsequent PR. 2. Add `waitFor` utility function that receives a lambda. ### Refactors 1. Several common functionalities used both by the `launch` and `deploy` command have been moved to the `cli/handlers/common` directory, from where both commands use them. These include 1. Checks for installed dependencies. 2. Common constants. 3. The `LaunchedNetwork` class has been moved to this directory. 4. DataHaven nodes common functions. 5. Kurtosis common functions. 6. Relayer common functions. 7. Kubernetes functions (although only used by `deploy`, it seemed fitting to have it here still). 8. Remove CLI questions and separator prints from `deploy-contracts.ts` and `set-datahaven-parameters.ts` scripts. These things should be in the `cli/launch` folder, which consumes the functions in these scripts. 9. Remove `setParametersFromCollection` from `utils` folder and put it in `cli`. 10. Create base snowbridge relayer configs for `local` and `stagenet` as two separate directories. ### Fixes 1. Sets the default time of the `deploy` command to 6s as Lodestar is slower than Lighthouse. 2. In `runShellCommandWithLogger` only print `stderr` if the command fails. ### Additional Minor Changes 1. K8s secret key names changed from `dh-beefy-relay-eth-key` to `dh-beefy-relay-ethereum-key` and `dh-beacon-relay-sub-key` to `dh-beacon-relay-substrate-key`, for simplicity in the deployment script. 11. Update suggested configs for `.vscode` configs. --------- Co-authored-by: Steve Degosserie <723552+stiiifff@users.noreply.github.com>
2025-06-12 08:24:03 +00:00
.hook("preAction", deployPreActionHook)
.action(deploy);
// ===== Launch ======
program
.command("launch")
.addHelpText(
"before",
`🫎 DataHaven: Network Launcher CLI for launching a full DataHaven network.
Complete with:
- Solo-chain validators,
feat: ✨ Add deployment charts for StorageHub MSP, BSP & Indexer nodes (Local & Stagenet envs) (#160) ## Summary This PR adds comprehensive Kubernetes deployment infrastructure for StorageHub components, enabling deployment of the full StorageHub network stack (MSP, BSP, Indexer, and Fisherman nodes) alongside DataHaven nodes in both local and stagenet environments. ### What's Added **1. New Helm Chart: StorageHub MSP Backend API** (`deploy/charts/backend/`) - REST API service for StorageHub operations - Connects to PostgreSQL database for indexed blockchain data - Connects to RPC nodes for real-time blockchain queries - Configurable via TOML configuration file - Supports environment-specific overrides - Includes comprehensive documentation **2. StorageHub Node Deployment Charts** (`deploy/charts/node/storagehub/`) - **MSP Node** (`sh-mspnode`): Main Service Provider nodes with charging capabilities - **BSP Node** (`sh-bspnode`): Backup Service Provider nodes for redundancy - **Indexer Node** (`sh-idxnode`): Full indexing node with PostgreSQL integration - **Fisherman Node** (`sh-fisherman`): Network monitoring and verification node **3. Environment Configurations** - **Local environment** (`deploy/environments/local/`): Development setup with hostpath storage - **Stagenet environment** (`deploy/environments/stagenet/`): Production-like setup with AWS EBS - PostgreSQL database configurations for Indexer and Fisherman nodes - Proper service discovery and network configuration **4. Enhanced CLI Tooling** (`test/cli/`) - New `deploy storagehub` command for deploying StorageHub components - Updated `launch storagehub` command for local testing - Interactive deployment with environment selection - Automatic database provisioning via Bitnami PostgreSQL charts **5. Node Configuration Improvements** - Fork-aware transaction pool for DH boot & validator nodes - Unsafe RPC methods exposed on MSP nodes (for provider operations) - JWT secret support for MSP Backend authentication - ECDSA key scheme for StorageHub BCSV keys (DataHaven compatibility) ### Architecture ``` StorageHub Stack: ├── MSP Nodes (2 replicas) → Storage providers with charging ├── BSP Nodes (2 replicas) → Backup storage providers ├── Indexer Node → Database indexing + PostgreSQL ├── Fisherman Node → Monitoring + PostgreSQL (shared with Indexer) └── MSP Backend API → REST API for StorageHub operations ``` ### Testing **Local Testing**: ```bash cd test bun cli launch storagehub # Interactive launcher # or bun cli deploy storagehub # Deploy via Helm ``` **Stagenet Deployment**: ```bash cd deploy helm install sh-mspnode ./charts/node \ -f ./charts/node/storagehub/sh-mspnode.yaml \ -f ./environments/stagenet/sh-mspnode.yaml \ -n datahaven-stagenet ``` ### Breaking Changes None - This is purely additive infrastructure. ### Migration Notes For existing deployments: 1. DataHaven nodes now use `--pool-type fork-aware` flag 2. Bootnode and validator node configs updated accordingly 3. No action required for existing DataHaven-only deployments
2025-10-21 20:18:50 +00:00
- StorageHub components: MSP, BSP, Indexer, Fisherman nodes and databases,
feat: 🚀 Add `deploy` command to CLI (#87) ### New Features 1. Add the `deploy` command to our CLI. 1. Conditionally deploys kurtosis eth network if we're in `stagenet` environment. 2. Deploys DH nodes. 3. Deploys contracts (all of them). In `mainnet` and `testnet` it shouldn't deploy EL contracts, but for now that's not implemented. 4. Configures parameters, validators and relayers in the same way as `launch`. 5. Currently, it only deploys `beefy` and `beacon` relayers, `execution` and `solochain` relayers are pending for a subsequent PR. 2. Add `waitFor` utility function that receives a lambda. ### Refactors 1. Several common functionalities used both by the `launch` and `deploy` command have been moved to the `cli/handlers/common` directory, from where both commands use them. These include 1. Checks for installed dependencies. 2. Common constants. 3. The `LaunchedNetwork` class has been moved to this directory. 4. DataHaven nodes common functions. 5. Kurtosis common functions. 6. Relayer common functions. 7. Kubernetes functions (although only used by `deploy`, it seemed fitting to have it here still). 8. Remove CLI questions and separator prints from `deploy-contracts.ts` and `set-datahaven-parameters.ts` scripts. These things should be in the `cli/launch` folder, which consumes the functions in these scripts. 9. Remove `setParametersFromCollection` from `utils` folder and put it in `cli`. 10. Create base snowbridge relayer configs for `local` and `stagenet` as two separate directories. ### Fixes 1. Sets the default time of the `deploy` command to 6s as Lodestar is slower than Lighthouse. 2. In `runShellCommandWithLogger` only print `stderr` if the command fails. ### Additional Minor Changes 1. K8s secret key names changed from `dh-beefy-relay-eth-key` to `dh-beefy-relay-ethereum-key` and `dh-beacon-relay-sub-key` to `dh-beacon-relay-substrate-key`, for simplicity in the deployment script. 11. Update suggested configs for `.vscode` configs. --------- Co-authored-by: Steve Degosserie <723552+stiiifff@users.noreply.github.com>
2025-06-12 08:24:03 +00:00
- Ethereum Private network,
- Snowbridge Relayers
feat: 🚀 Add `deploy` command to CLI (#87) ### New Features 1. Add the `deploy` command to our CLI. 1. Conditionally deploys kurtosis eth network if we're in `stagenet` environment. 2. Deploys DH nodes. 3. Deploys contracts (all of them). In `mainnet` and `testnet` it shouldn't deploy EL contracts, but for now that's not implemented. 4. Configures parameters, validators and relayers in the same way as `launch`. 5. Currently, it only deploys `beefy` and `beacon` relayers, `execution` and `solochain` relayers are pending for a subsequent PR. 2. Add `waitFor` utility function that receives a lambda. ### Refactors 1. Several common functionalities used both by the `launch` and `deploy` command have been moved to the `cli/handlers/common` directory, from where both commands use them. These include 1. Checks for installed dependencies. 2. Common constants. 3. The `LaunchedNetwork` class has been moved to this directory. 4. DataHaven nodes common functions. 5. Kurtosis common functions. 6. Relayer common functions. 7. Kubernetes functions (although only used by `deploy`, it seemed fitting to have it here still). 8. Remove CLI questions and separator prints from `deploy-contracts.ts` and `set-datahaven-parameters.ts` scripts. These things should be in the `cli/launch` folder, which consumes the functions in these scripts. 9. Remove `setParametersFromCollection` from `utils` folder and put it in `cli`. 10. Create base snowbridge relayer configs for `local` and `stagenet` as two separate directories. ### Fixes 1. Sets the default time of the `deploy` command to 6s as Lodestar is slower than Lighthouse. 2. In `runShellCommandWithLogger` only print `stderr` if the command fails. ### Additional Minor Changes 1. K8s secret key names changed from `dh-beefy-relay-eth-key` to `dh-beefy-relay-ethereum-key` and `dh-beacon-relay-sub-key` to `dh-beacon-relay-substrate-key`, for simplicity in the deployment script. 11. Update suggested configs for `.vscode` configs. --------- Co-authored-by: Steve Degosserie <723552+stiiifff@users.noreply.github.com>
2025-06-12 08:24:03 +00:00
`
)
.description("Launch a full E2E DataHaven & Ethereum network and more")
.option("--A, --all", "Launch all components without prompting")
.option("--d, --datahaven", "(Re)Launch DataHaven network")
.option("--nd, --no-datahaven", "Skip launching DataHaven network")
.option("--bd, --build-datahaven", "Build DataHaven node local Docker image")
.option("--nbd, --no-build-datahaven", "Skip building DataHaven node local Docker image")
feat(CLI): ✨ Run beacon relay in CLI (#70) - **New Features** - Initialise Ethereum client pallet with a beacon chain checkpoint before starting relayers. - **Improvements** - Store Ethereum node RPC endpoints in `launchedNetwork` for later retrieval. - Standardised CLI options with explicit paired flags for enabling and disabling features, improving usability. - Increased slot frequency and number of validator keys per node in test network configurations. - Expanded and clarified test environment setup documentation and added a new CLI usage section in the main README. - **Bug Fixes** - Updated runtime fork version constants for testing environments, to match with Kurtosis'. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - Enhanced CLI with explicit enable/disable flags for network components and relayers. - Added initialization of the Ethereum Beacon Client pallet, ensuring the beacon chain is ready and submitting an initial checkpoint before starting relayers. - **Improvements** - Streamlined network setup by centralizing service endpoint registration and simplifying RPC URL handling. - Expanded and clarified CLI and test documentation with detailed setup instructions and option descriptions. - **Configuration Updates** - Updated network and beacon relay configurations for improved slot timing, validator key allocation, and sync committee period. - Adjusted Ethereum fork version constants to ensure compatibility. - **Bug Fixes** - Improved error handling and validation during network and relayer initialization. - **Documentation** - Added an "E2E CLI" section to the main README. - Enhanced test environment documentation with clearer steps and tips. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2025-05-15 21:56:36 +00:00
.option("--lk, --launch-kurtosis", "Launch Kurtosis Ethereum network with EL and CL clients")
.option("--nlk, --no-launch-kurtosis", "Skip launching Kurtosis Ethereum network")
.option("--dc, --deploy-contracts", "Deploy smart contracts")
.option("--ndc, --no-deploy-contracts", "Skip deploying smart contracts")
.option("--fv, --fund-validators", "Fund validators")
.option("--nfv, --no-fund-validators", "Skip funding validators")
.option("--sv, --setup-validators", "Setup validators")
.option("--nsv, --no-setup-validators", "Skip setup validators")
.option("--uv, --update-validator-set", "Update validator set")
.option("--nuv, --no-update-validator-set", "Skip update validator set")
feat: 🚀 Add `deploy` command to CLI (#87) ### New Features 1. Add the `deploy` command to our CLI. 1. Conditionally deploys kurtosis eth network if we're in `stagenet` environment. 2. Deploys DH nodes. 3. Deploys contracts (all of them). In `mainnet` and `testnet` it shouldn't deploy EL contracts, but for now that's not implemented. 4. Configures parameters, validators and relayers in the same way as `launch`. 5. Currently, it only deploys `beefy` and `beacon` relayers, `execution` and `solochain` relayers are pending for a subsequent PR. 2. Add `waitFor` utility function that receives a lambda. ### Refactors 1. Several common functionalities used both by the `launch` and `deploy` command have been moved to the `cli/handlers/common` directory, from where both commands use them. These include 1. Checks for installed dependencies. 2. Common constants. 3. The `LaunchedNetwork` class has been moved to this directory. 4. DataHaven nodes common functions. 5. Kurtosis common functions. 6. Relayer common functions. 7. Kubernetes functions (although only used by `deploy`, it seemed fitting to have it here still). 8. Remove CLI questions and separator prints from `deploy-contracts.ts` and `set-datahaven-parameters.ts` scripts. These things should be in the `cli/launch` folder, which consumes the functions in these scripts. 9. Remove `setParametersFromCollection` from `utils` folder and put it in `cli`. 10. Create base snowbridge relayer configs for `local` and `stagenet` as two separate directories. ### Fixes 1. Sets the default time of the `deploy` command to 6s as Lodestar is slower than Lighthouse. 2. In `runShellCommandWithLogger` only print `stderr` if the command fails. ### Additional Minor Changes 1. K8s secret key names changed from `dh-beefy-relay-eth-key` to `dh-beefy-relay-ethereum-key` and `dh-beacon-relay-sub-key` to `dh-beacon-relay-substrate-key`, for simplicity in the deployment script. 11. Update suggested configs for `.vscode` configs. --------- Co-authored-by: Steve Degosserie <723552+stiiifff@users.noreply.github.com>
2025-06-12 08:24:03 +00:00
.option("--sp, --set-parameters", "Set DataHaven runtime parameters")
.option("--nsp, --no-set-parameters", "Skip setting DataHaven runtime parameters")
feat(CLI): ✨ Run beacon relay in CLI (#70) - **New Features** - Initialise Ethereum client pallet with a beacon chain checkpoint before starting relayers. - **Improvements** - Store Ethereum node RPC endpoints in `launchedNetwork` for later retrieval. - Standardised CLI options with explicit paired flags for enabling and disabling features, improving usability. - Increased slot frequency and number of validator keys per node in test network configurations. - Expanded and clarified test environment setup documentation and added a new CLI usage section in the main README. - **Bug Fixes** - Updated runtime fork version constants for testing environments, to match with Kurtosis'. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - Enhanced CLI with explicit enable/disable flags for network components and relayers. - Added initialization of the Ethereum Beacon Client pallet, ensuring the beacon chain is ready and submitting an initial checkpoint before starting relayers. - **Improvements** - Streamlined network setup by centralizing service endpoint registration and simplifying RPC URL handling. - Expanded and clarified CLI and test documentation with detailed setup instructions and option descriptions. - **Configuration Updates** - Updated network and beacon relay configurations for improved slot timing, validator key allocation, and sync committee period. - Adjusted Ethereum fork version constants to ensure compatibility. - **Bug Fixes** - Improved error handling and validation during network and relayer initialization. - **Documentation** - Added an "E2E CLI" section to the main README. - Enhanced test environment documentation with clearer steps and tips. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2025-05-15 21:56:36 +00:00
.option("--r, --relayer", "Launch Snowbridge Relayers")
.option("--nr, --no-relayer", "Skip Snowbridge Relayers")
.option("--b, --blockscout", "Enable Blockscout")
test: ⚡️ CI Refactor (#59) Eventually our CI will be required to run two private blockchains locally plus associated relayers. This PR is to prepare for this fate by improving run times and refactoring our existing CIs so they are a bit easier to reason about. ### Refactors - **_We now run ALL CIs on every PR!_** This is so that we decomplexify the logic around conditional builds and fetching built binaries from another source. This reduces the surface area of code we have to maintain at the cost of execution time - This penalty is ameliorated by a layered caching system. At best, it will be less than a minute to complete a build since everything will be cached. On GH runners this is about 6 minutes sadly. - We will no longer be at risk of important CIs being skipped erroneously which hide true failures. - Caching is a low-risk approach because at worst it has to build from scratch. A bad cache hit will never imply the wrong thing gets build since cargo is smart enough to just throw away any inappropriate build artefacts. - `setup-rust` action created so we have a unified way of setting up runner and unifying our approach to caching - Use a unique caching key for different activities and it will fallback to shared cache if no matches - we are using `mainnet` kurtosis config so that it works with relayer assumptions ### Additions - We can specify the ethereum block time via a new cli arg `--slot-time <seconds>` - We can specify arbitrary network_param args which get passed into the generated yaml - e.g. giving `bun cli --kurtosis-network-args="pet=cat food=fish" will add: ```yml network_params: # existing params... pet: cat food: fish ``` - We now have the ability to programmatically modify the yaml - This means we are back down to a single `minimal.yml` kurtosis config so we dont have to maintain changes between them - Flow is: `add new cli arg` -> `add if() block which mutates yaml` -> `profit` --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Facundo Farall <37149322+ffarall@users.noreply.github.com>
2025-05-06 20:20:02 +00:00
.option("--slot-time <number>", "Set slot time in seconds", parseIntValue)
.option("--cn, --clean-network", "Always clean Kurtosis enclave and Docker containers")
.option("--ic, --inject-contracts", "Inject smart contracts from saved state diff")
.option(
"--datahaven-build-extra-args <value>",
"Extra args for DataHaven node Cargo build (the plain command is `cargo build --release` for linux, `cargo zigbuild --target x86_64-unknown-linux-gnu --release` for mac)",
"--features=fast-runtime"
)
.option(
"--e --kurtosis-enclave-name <value>",
"Name of the Kurtosis Enclave",
"datahaven-ethereum"
)
test: ⚡️ CI Refactor (#59) Eventually our CI will be required to run two private blockchains locally plus associated relayers. This PR is to prepare for this fate by improving run times and refactoring our existing CIs so they are a bit easier to reason about. ### Refactors - **_We now run ALL CIs on every PR!_** This is so that we decomplexify the logic around conditional builds and fetching built binaries from another source. This reduces the surface area of code we have to maintain at the cost of execution time - This penalty is ameliorated by a layered caching system. At best, it will be less than a minute to complete a build since everything will be cached. On GH runners this is about 6 minutes sadly. - We will no longer be at risk of important CIs being skipped erroneously which hide true failures. - Caching is a low-risk approach because at worst it has to build from scratch. A bad cache hit will never imply the wrong thing gets build since cargo is smart enough to just throw away any inappropriate build artefacts. - `setup-rust` action created so we have a unified way of setting up runner and unifying our approach to caching - Use a unique caching key for different activities and it will fallback to shared cache if no matches - we are using `mainnet` kurtosis config so that it works with relayer assumptions ### Additions - We can specify the ethereum block time via a new cli arg `--slot-time <seconds>` - We can specify arbitrary network_param args which get passed into the generated yaml - e.g. giving `bun cli --kurtosis-network-args="pet=cat food=fish" will add: ```yml network_params: # existing params... pet: cat food: fish ``` - We now have the ability to programmatically modify the yaml - This means we are back down to a single `minimal.yml` kurtosis config so we dont have to maintain changes between them - Flow is: `add new cli arg` -> `add if() block which mutates yaml` -> `profit` --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Facundo Farall <37149322+ffarall@users.noreply.github.com>
2025-05-06 20:20:02 +00:00
.option("--kurtosis-network-args <value>", "CustomKurtosis network args")
feat(CLI): ✨ Run beacon relay in CLI (#70) - **New Features** - Initialise Ethereum client pallet with a beacon chain checkpoint before starting relayers. - **Improvements** - Store Ethereum node RPC endpoints in `launchedNetwork` for later retrieval. - Standardised CLI options with explicit paired flags for enabling and disabling features, improving usability. - Increased slot frequency and number of validator keys per node in test network configurations. - Expanded and clarified test environment setup documentation and added a new CLI usage section in the main README. - **Bug Fixes** - Updated runtime fork version constants for testing environments, to match with Kurtosis'. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - Enhanced CLI with explicit enable/disable flags for network components and relayers. - Added initialization of the Ethereum Beacon Client pallet, ensuring the beacon chain is ready and submitting an initial checkpoint before starting relayers. - **Improvements** - Streamlined network setup by centralizing service endpoint registration and simplifying RPC URL handling. - Expanded and clarified CLI and test documentation with detailed setup instructions and option descriptions. - **Configuration Updates** - Updated network and beacon relay configurations for improved slot timing, validator key allocation, and sync committee period. - Adjusted Ethereum fork version constants to ensure compatibility. - **Bug Fixes** - Improved error handling and validation during network and relayer initialization. - **Documentation** - Added an "E2E CLI" section to the main README. - Enhanced test environment documentation with clearer steps and tips. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2025-05-15 21:56:36 +00:00
.option("--verified", "Verify smart contracts with Blockscout")
test: ⚙️ Parse & Generate Relayer Configs (#54) ## Human Written Description This PR adds the following to the E2E CLI: - Relayer config generation for: `beacon-relay` `beefy-relay` - The other two relayer types to be added later - Relayers don't actually work yet - By default turned off, this requires a binary to be present in: `<repo_root>/operator/target/release` dir - Datahaven network launching - DH network is using default `local` network chain spec - Launched with 5 nodes since our authority set is 6 large (and you need 2/3 + 1 of set size > [!NOTE] > Both the relayer and the DH node binaries are being run as local processes TEMPORARILY. This means that logging is done in a very rudimentary way (we pipe to a file whilst the CLI is running). > > This means that when the CLI finishes **the log files will no longer be written to**. > This is temporary since spawning binaries is a stop gap solution until docker images available. --- > [!IMPORTANT] > The following is AI generated slop describing this PR's changes: **Key Changes:** * **CLI Enhancements (`test/cli/index.ts`):** * Added options `--datahaven` and `--datahaven-bin-path` to enable launching local DataHaven nodes. * Added options `--relayer` and `--relayer-bin-path` to enable launching Snowbridge relayers (Beefy and Beacon). * Added negation flags (`--no-fund-validators`, `--no-setup-validators`, `--no-update-validator-set`) for more granular control over validator setup steps. * Added `--skip-cleaning` option to preserve Kurtosis state between runs. * Added a pre-action hook (`launchPreActionHook`) to validate flag combinations (e.g., `--verified` requires `--blockscout`). * **New CLI Handlers (`test/cli/handlers/launch/`):** * `datahaven.ts`: Logic for spawning DataHaven node processes using the specified binary. Manages ports and process cleanup. * `relayer.ts`: Logic for configuring and spawning Snowbridge relayer processes (Beefy and Beacon). Reads contract deployment addresses, updates relayer config templates, and uses specified private keys. Manages log files and process cleanup. * `summary.ts`: Generates and displays the table of running services (including dynamically launched DataHaven nodes) and their endpoints. * `validator.ts`: Extracted validator funding, setup, and set update logic into its own handler. * `index.ts`: Orchestrates the launch sequence based on CLI options, calling the appropriate handlers. Includes a `LaunchedNetwork` class to track spawned processes, file descriptors, and node ports for cleanup. * **Updated `package.json` Scripts:** * Added `start:e2e:minrelayer` script for a minimal setup including relayers and DataHaven nodes. * Modified `stop:e2e` to include `pkill datahaven` for proper cleanup. * Added `stop:e2e:quick` to only stop the Kurtosis enclave without full cleaning. * **Updated `launch-kurtosis.ts`:** Modified to use new Kurtosis utility functions and added a `skipCleaning` option. * **New Utility Functions:** * `test/utils/kurtosis.ts`: Functions to inspect Kurtosis services (`getServiceFromKurtosis`, `getPortFromKurtosis`, `getServicesFromKurtosis`). * `test/utils/parser.ts`: Zod schemas and parsing functions for Snowbridge relayer configurations. * **Constants & Minor Updates:** * Added `SUBSTRATE_FUNDED_ACCOUNTS` to `test/utils/constants.ts`. * Updated `tsconfig.json` include paths. * Refactored `test/utils/docker.ts` (though now largely superseded by Kurtosis utils). * Updated logging in `test/scripts/send-txn.ts`. **Reasoning:** This PR significantly expands the E2E testing capabilities by allowing developers to easily launch and integrate local DataHaven nodes and Snowbridge relayers into the test network, facilitating more comprehensive integration testing. The CLI refactoring makes managing these complex setups more robust and user-friendly.
2025-04-29 12:24:00 +00:00
.option(
"--dit, --datahaven-image-tag <value>",
test: 🐳 Add docker support for datahaven nodes (#71) > [!NOTE] > This is `Part 3` of the ongoing _Docker Series._ ## New Additions: - Launching Datahaven network will spin up containers, as opposed to native binaries - `stop:docker` script to kill all dh containers - `e2e` test suite for datahaven solochain network - Contains reference test file that uses papi for storage queries, submitting exts, runtime calls (good job on that facu and tobi) - Added new utils: - `waitForLog()` to wait for log lines in docker container logs - `createPapiConnectors()` helper for test cases to build and connect to dh network - `getPapiSigner()` helper to return a papi compatible signer using our prefunded accounts (alith by default) - `sendTxn()` helper to submit txn and wait for block inclusion, instead of finalization, which std library provides ## Changes: > [!CAUTION] > Launching native binaries for datahaven no longer supported. - Datahaven binary location cli option changed to `-i, --datahaven-image-tag` - To locally run this you'll need a datahaven docker image handy, you'll need to either: - Point to remote dockerhub e.g. `moonsonglabs/datahaven:main` (must be logged in and have permission) - Build this locally with `bun build:docker:operator` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - Added end-to-end tests for the Datahaven solochain, including runtime API queries, storage lookups, extrinsic submissions, and event listening. - Introduced CLI option to specify the Datahaven Docker image tag, with a default value. - Added CLI option to disable the Relayer. - Provided new scripts to stop Docker containers associated with Datahaven. - Added utility functions for Docker log monitoring and container startup checks. - Introduced utilities for interacting with the Datahaven Polkadot API. - **Improvements** - Switched Datahaven network launch from local binaries to Docker containers. - Enhanced cache accuracy in build workflows by including Rust source files in cache keys. - Improved build performance with TypeScript incremental build options. - Increased timeout for end-to-end tests for better reliability. - Updated CLI version to 0.2.0. - Modified Dockerfile build to enable the `fast-runtime` feature. - Extended network launch summary to include relayer and container details. - **Bug Fixes** - Fixed cleanup logic by tracking and preparing for forced removal of Docker containers after tests. - **Chores** - Updated workflow steps for Docker image handling and network checks. - Adjusted scripts and workflow logic for improved Docker and test management. - Removed top-level disk usage summaries from cleanup workflow for streamlined reporting. - Enhanced shell command utility to support asynchronous wait during execution. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Facundo Farall <37149322+ffarall@users.noreply.github.com>
2025-05-16 14:17:05 +00:00
"Tag of the datahaven image to use",
"datahavenxyz/datahaven:local"
test: ⚙️ Parse & Generate Relayer Configs (#54) ## Human Written Description This PR adds the following to the E2E CLI: - Relayer config generation for: `beacon-relay` `beefy-relay` - The other two relayer types to be added later - Relayers don't actually work yet - By default turned off, this requires a binary to be present in: `<repo_root>/operator/target/release` dir - Datahaven network launching - DH network is using default `local` network chain spec - Launched with 5 nodes since our authority set is 6 large (and you need 2/3 + 1 of set size > [!NOTE] > Both the relayer and the DH node binaries are being run as local processes TEMPORARILY. This means that logging is done in a very rudimentary way (we pipe to a file whilst the CLI is running). > > This means that when the CLI finishes **the log files will no longer be written to**. > This is temporary since spawning binaries is a stop gap solution until docker images available. --- > [!IMPORTANT] > The following is AI generated slop describing this PR's changes: **Key Changes:** * **CLI Enhancements (`test/cli/index.ts`):** * Added options `--datahaven` and `--datahaven-bin-path` to enable launching local DataHaven nodes. * Added options `--relayer` and `--relayer-bin-path` to enable launching Snowbridge relayers (Beefy and Beacon). * Added negation flags (`--no-fund-validators`, `--no-setup-validators`, `--no-update-validator-set`) for more granular control over validator setup steps. * Added `--skip-cleaning` option to preserve Kurtosis state between runs. * Added a pre-action hook (`launchPreActionHook`) to validate flag combinations (e.g., `--verified` requires `--blockscout`). * **New CLI Handlers (`test/cli/handlers/launch/`):** * `datahaven.ts`: Logic for spawning DataHaven node processes using the specified binary. Manages ports and process cleanup. * `relayer.ts`: Logic for configuring and spawning Snowbridge relayer processes (Beefy and Beacon). Reads contract deployment addresses, updates relayer config templates, and uses specified private keys. Manages log files and process cleanup. * `summary.ts`: Generates and displays the table of running services (including dynamically launched DataHaven nodes) and their endpoints. * `validator.ts`: Extracted validator funding, setup, and set update logic into its own handler. * `index.ts`: Orchestrates the launch sequence based on CLI options, calling the appropriate handlers. Includes a `LaunchedNetwork` class to track spawned processes, file descriptors, and node ports for cleanup. * **Updated `package.json` Scripts:** * Added `start:e2e:minrelayer` script for a minimal setup including relayers and DataHaven nodes. * Modified `stop:e2e` to include `pkill datahaven` for proper cleanup. * Added `stop:e2e:quick` to only stop the Kurtosis enclave without full cleaning. * **Updated `launch-kurtosis.ts`:** Modified to use new Kurtosis utility functions and added a `skipCleaning` option. * **New Utility Functions:** * `test/utils/kurtosis.ts`: Functions to inspect Kurtosis services (`getServiceFromKurtosis`, `getPortFromKurtosis`, `getServicesFromKurtosis`). * `test/utils/parser.ts`: Zod schemas and parsing functions for Snowbridge relayer configurations. * **Constants & Minor Updates:** * Added `SUBSTRATE_FUNDED_ACCOUNTS` to `test/utils/constants.ts`. * Updated `tsconfig.json` include paths. * Refactored `test/utils/docker.ts` (though now largely superseded by Kurtosis utils). * Updated logging in `test/scripts/send-txn.ts`. **Reasoning:** This PR significantly expands the E2E testing capabilities by allowing developers to easily launch and integrate local DataHaven nodes and Snowbridge relayers into the test network, facilitating more comprehensive integration testing. The CLI refactoring makes managing these complex setups more robust and user-friendly.
2025-04-29 12:24:00 +00:00
)
test: 🐳 Add Docker relay support to CLI (#74) ## Changes - Latest changes to have working relayer :tada: component - Changed spawning snowbridge relayers to docker containers - Small logging output changes - Refactoring to `LaunchedNetwork` class - new flag `--bd` `--build-datahaven` which will build a local docker container which is **much** quicker than the proper CI build (which uses a controlled build enviroment) - new bun script `start:e2e:local`, which is everything that `start:e2e:ci` has, but with building local docker container and log_level debug set --- <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit ## Summary by CodeRabbit - **New Features** - Added support for launching and managing relayer and DataHaven services using Docker containers and networks. - Introduced a CLI option to specify the relayer Docker image tag instead of a binary path. - **Improvements** - Enhanced log messages with clearer text and expressive emojis for better user feedback. - Improved summary display by removing relayer services from the output. - Updated build scripts to consistently enable the "fast-runtime" feature for cross-platform builds. - Refined validation and error reporting for checkpoint data parsing. - **Bug Fixes** - Improved Docker container cleanup and network management during service launch and teardown. - **Chores** - Updated and refactored npm scripts for Docker operations and end-to-end test cleanup. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Facundo Farall <37149322+ffarall@users.noreply.github.com>
2025-05-18 23:31:46 +00:00
.option(
"--rit, --relayer-image-tag <value>",
test: 🐳 Add Docker relay support to CLI (#74) ## Changes - Latest changes to have working relayer :tada: component - Changed spawning snowbridge relayers to docker containers - Small logging output changes - Refactoring to `LaunchedNetwork` class - new flag `--bd` `--build-datahaven` which will build a local docker container which is **much** quicker than the proper CI build (which uses a controlled build enviroment) - new bun script `start:e2e:local`, which is everything that `start:e2e:ci` has, but with building local docker container and log_level debug set --- <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit ## Summary by CodeRabbit - **New Features** - Added support for launching and managing relayer and DataHaven services using Docker containers and networks. - Introduced a CLI option to specify the relayer Docker image tag instead of a binary path. - **Improvements** - Enhanced log messages with clearer text and expressive emojis for better user feedback. - Improved summary display by removing relayer services from the output. - Updated build scripts to consistently enable the "fast-runtime" feature for cross-platform builds. - Refined validation and error reporting for checkpoint data parsing. - **Bug Fixes** - Improved Docker container cleanup and network management during service launch and teardown. - **Chores** - Updated and refactored npm scripts for Docker operations and end-to-end test cleanup. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Facundo Farall <37149322+ffarall@users.noreply.github.com>
2025-05-18 23:31:46 +00:00
"Tag of the relayer",
"datahavenxyz/snowbridge-relay:latest"
test: 🐳 Add Docker relay support to CLI (#74) ## Changes - Latest changes to have working relayer :tada: component - Changed spawning snowbridge relayers to docker containers - Small logging output changes - Refactoring to `LaunchedNetwork` class - new flag `--bd` `--build-datahaven` which will build a local docker container which is **much** quicker than the proper CI build (which uses a controlled build enviroment) - new bun script `start:e2e:local`, which is everything that `start:e2e:ci` has, but with building local docker container and log_level debug set --- <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit ## Summary by CodeRabbit - **New Features** - Added support for launching and managing relayer and DataHaven services using Docker containers and networks. - Introduced a CLI option to specify the relayer Docker image tag instead of a binary path. - **Improvements** - Enhanced log messages with clearer text and expressive emojis for better user feedback. - Improved summary display by removing relayer services from the output. - Updated build scripts to consistently enable the "fast-runtime" feature for cross-platform builds. - Refined validation and error reporting for checkpoint data parsing. - **Bug Fixes** - Improved Docker container cleanup and network management during service launch and teardown. - **Chores** - Updated and refactored npm scripts for Docker operations and end-to-end test cleanup. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Facundo Farall <37149322+ffarall@users.noreply.github.com>
2025-05-18 23:31:46 +00:00
)
.hook("preAction", launchPreActionHook)
.action(launch);
// ===== Stop ======
program
.command("stop")
.description("Stop any launched running network components")
.option("--A --all", "Stop all components associated with project")
.option("--d, --datahaven", "Stop DataHaven network")
.option("--nd, --no-datahaven", "Skip stopping DataHaven network")
.option("--e, --enclave", "Stop Ethereum Kurtosis enclave")
.option("--ne, --no-enclave", "Skip stopping Ethereum Kurtosis enclave")
.option("--kurtosis-engine", "Stop Kurtosis engine", false)
.option("--r, --relayer", "Stop Snowbridge Relayers")
.option("--nr, --no-relayer", "Skip stopping Snowbridge Relayers")
.hook("preAction", stopPreActionHook)
.action(stop);
feat: ✨ Datahaven contracts deployment on public testnet (#123) ## Summary This PR introduces support for deploying Datahaven contracts to different chains (hoodi, holesky, mainnet), as well as a new cli command to manage this deployment separately from the regular deployment, while maintaining compatibility with it. #### New CLI command - **`bun cli contracts deploy`** - Deploy contracts to supported chains (Hoodi, Holesky, Mainnet) - **`bun cli contracts status`** - Check deployment configuration and status - **`bun cli contracts verify`** - Verify contracts on block explorers - Commands need the chain parameter: `--chain <hoodi | holesky | mainnet>` - Right now only `hoodi` and `holesky` are supported ### Deployment #### Hoodi & Holesky Network Support - Added **DeployBase.s.sol** as common ground for **DeployTestnet.s.sol** (also new) and **DeployLocal.s.sol** (existing). - **Hoodi configuration** (`contracts/config/hoodi.json`) with deployed EigenLayer contract addresses to reference. - **Holesky configuration** (`contracts/config/hoodi.json`) with deployed EigenLayer contract addresses to reference. #### Contracts being deployed - **DataHaven**: ServiceManager, VetoableSlasher, RewardsRegistry - **Snowbridge**: BeefyClient, AgentExecutor, Gateway, RewardsAgent - **EigenLayer**: References existing deployed contracts (not re-deployed) #### Deployment files When the deployment is done, a new file under `contracts/deployments` is generated with the addresses of the deployed contracts, for each chain (it will be overriden per chain if run multiple times). So we would have one `anvil.json`, `hoodi.json`, `holesky.json`, etc, with the addresses of the deployed contracts for reference and for later verification. #### Todo - [x] Test compatibility with existing `bun cli launch` and `bun cli deploy` commands #### For follow-up PRs - Fix verification issue with `foundry verify-contracts` when specifying the `chain` or `chain-id` parameter, needed for hoodi (https://github.com/foundry-rs/foundry/issues/7466). - Add `redeploy` feature to only override implementation contract and leave the proxy address untouched ## Usage Examples ```bash # Deploy to Hoodi network bun cli contracts deploy --chain hoodi # Check deployment status bun cli contracts status --chain hoodi # Verify contracts on block explorer bun cli contracts verify --chain hoodi ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added deployment and configuration support for new networks "hoodi" and "holesky", including new configuration and deployment files. * Introduced a CLI tool for managing contract deployments, status checks, and verification across supported chains. * Added example environment configuration and comprehensive deployment documentation. * Enabled contract verification and status reporting via the CLI with support for block explorer integration. * **Improvements** * Refactored deployment scripts for modularity, supporting both local and testnet environments. * Centralized and extended configuration loading to support additional contract addresses and network parameters. * Enhanced deployment utilities and typings to support multi-network deployments. * **Bug Fixes** * Improved input validation and error handling in CLI commands and deployment scripts. * Added explicit handling for zero address in operator strategy retrieval. * **Chores** * Updated documentation and configuration templates for easier onboarding and deployment management. * Improved logging and output formatting for deployment and verification processes. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Steve Degosserie <723552+stiiifff@users.noreply.github.com> Co-authored-by: Ahmad Kaouk <56095276+ahmadkaouk@users.noreply.github.com>
2025-08-21 10:02:31 +00:00
// ===== Contracts ======
const contractsCommand = program
.command("contracts")
.addHelpText(
"before",
`🫎 DataHaven: Contracts Deployment CLI for deploying DataHaven AVS contracts to supported chains
Commands:
- status: Show deployment plan, configuration, and status (default)
- deploy: Deploy contracts to specified chain
- verify: Verify deployed contracts on block explorer
- update-metadata: Update the metadata URI of an existing AVS contract
feat: ✨ Datahaven contracts deployment on public testnet (#123) ## Summary This PR introduces support for deploying Datahaven contracts to different chains (hoodi, holesky, mainnet), as well as a new cli command to manage this deployment separately from the regular deployment, while maintaining compatibility with it. #### New CLI command - **`bun cli contracts deploy`** - Deploy contracts to supported chains (Hoodi, Holesky, Mainnet) - **`bun cli contracts status`** - Check deployment configuration and status - **`bun cli contracts verify`** - Verify contracts on block explorers - Commands need the chain parameter: `--chain <hoodi | holesky | mainnet>` - Right now only `hoodi` and `holesky` are supported ### Deployment #### Hoodi & Holesky Network Support - Added **DeployBase.s.sol** as common ground for **DeployTestnet.s.sol** (also new) and **DeployLocal.s.sol** (existing). - **Hoodi configuration** (`contracts/config/hoodi.json`) with deployed EigenLayer contract addresses to reference. - **Holesky configuration** (`contracts/config/hoodi.json`) with deployed EigenLayer contract addresses to reference. #### Contracts being deployed - **DataHaven**: ServiceManager, VetoableSlasher, RewardsRegistry - **Snowbridge**: BeefyClient, AgentExecutor, Gateway, RewardsAgent - **EigenLayer**: References existing deployed contracts (not re-deployed) #### Deployment files When the deployment is done, a new file under `contracts/deployments` is generated with the addresses of the deployed contracts, for each chain (it will be overriden per chain if run multiple times). So we would have one `anvil.json`, `hoodi.json`, `holesky.json`, etc, with the addresses of the deployed contracts for reference and for later verification. #### Todo - [x] Test compatibility with existing `bun cli launch` and `bun cli deploy` commands #### For follow-up PRs - Fix verification issue with `foundry verify-contracts` when specifying the `chain` or `chain-id` parameter, needed for hoodi (https://github.com/foundry-rs/foundry/issues/7466). - Add `redeploy` feature to only override implementation contract and leave the proxy address untouched ## Usage Examples ```bash # Deploy to Hoodi network bun cli contracts deploy --chain hoodi # Check deployment status bun cli contracts status --chain hoodi # Verify contracts on block explorer bun cli contracts verify --chain hoodi ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added deployment and configuration support for new networks "hoodi" and "holesky", including new configuration and deployment files. * Introduced a CLI tool for managing contract deployments, status checks, and verification across supported chains. * Added example environment configuration and comprehensive deployment documentation. * Enabled contract verification and status reporting via the CLI with support for block explorer integration. * **Improvements** * Refactored deployment scripts for modularity, supporting both local and testnet environments. * Centralized and extended configuration loading to support additional contract addresses and network parameters. * Enhanced deployment utilities and typings to support multi-network deployments. * **Bug Fixes** * Improved input validation and error handling in CLI commands and deployment scripts. * Added explicit handling for zero address in operator strategy retrieval. * **Chores** * Updated documentation and configuration templates for easier onboarding and deployment management. * Improved logging and output formatting for deployment and verification processes. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Steve Degosserie <723552+stiiifff@users.noreply.github.com> Co-authored-by: Ahmad Kaouk <56095276+ahmadkaouk@users.noreply.github.com>
2025-08-21 10:02:31 +00:00
Common options:
--chain: Target chain (required: hoodi, holesky, mainnet)
--rpc-url: Chain RPC URL (optional, defaults based on chain)
--private-key: Private key for deployment
--skip-verification: Skip contract verification
`
)
.description("Deploy and manage DataHaven AVS contracts on supported chains");
// Contracts Check (default)
contractsCommand
.command("status")
.description("Show deployment plan, configuration, and status")
.option("--chain <value>", "Target chain (hoodi, holesky, mainnet)")
.option("--rpc-url <value>", "Chain RPC URL (optional, defaults based on chain)")
.option("--private-key <value>", "Private key for deployment", process.env.PRIVATE_KEY || "")
.option("--skip-verification", "Skip contract verification", false)
.hook("preAction", contractsPreActionHook)
.action(contractsCheck);
// Contracts Deploy
contractsCommand
.command("deploy")
.description("Deploy DataHaven AVS contracts to specified chain")
.option("--chain <value>", "Target chain (hoodi, holesky, mainnet)")
.option("--rpc-url <value>", "Chain RPC URL (optional, defaults based on chain)")
.option("--private-key <value>", "Private key for deployment", process.env.PRIVATE_KEY || "")
.option("--skip-verification", "Skip contract verification", false)
.hook("preAction", contractsPreActionHook)
.action(contractsDeploy);
// Contracts Verify
contractsCommand
.command("verify")
.description("Verify deployed contracts on block explorer")
.option("--chain <value>", "Target chain (hoodi, holesky, mainnet)")
.option("--rpc-url <value>", "Chain RPC URL (optional, defaults based on chain)")
.option("--skip-verification", "Skip contract verification", false)
.hook("preAction", contractsPreActionHook)
.action(contractsVerify);
// Contracts Update Metadata
contractsCommand
.command("update-metadata")
.description("Update AVS metadata URI for the DataHaven Service Manager")
.option("--chain <value>", "Target chain (hoodi, holesky, mainnet)")
.option("--uri <value>", "New metadata URI (required)")
.option("--reset", "Use if you want to reset the metadata URI")
.option("--rpc-url <value>", "Chain RPC URL (optional, defaults based on chain)")
.action(async (options: any, command: any) => {
// Try to get chain from options or command
let chain = options.chain;
if (!chain && command.parent) {
chain = command.parent.getOptionValue("chain");
}
if (!chain) {
chain = command.getOptionValue("chain");
}
if (!options.uri && !options.reset) {
throw new Error("--uri parameter is required");
}
if (options.reset) {
options.uri = "";
}
if (!chain) {
throw new Error("--chain parameter is required");
}
await updateAVSMetadataURI(chain, options.uri);
});
feat: ✨ Datahaven contracts deployment on public testnet (#123) ## Summary This PR introduces support for deploying Datahaven contracts to different chains (hoodi, holesky, mainnet), as well as a new cli command to manage this deployment separately from the regular deployment, while maintaining compatibility with it. #### New CLI command - **`bun cli contracts deploy`** - Deploy contracts to supported chains (Hoodi, Holesky, Mainnet) - **`bun cli contracts status`** - Check deployment configuration and status - **`bun cli contracts verify`** - Verify contracts on block explorers - Commands need the chain parameter: `--chain <hoodi | holesky | mainnet>` - Right now only `hoodi` and `holesky` are supported ### Deployment #### Hoodi & Holesky Network Support - Added **DeployBase.s.sol** as common ground for **DeployTestnet.s.sol** (also new) and **DeployLocal.s.sol** (existing). - **Hoodi configuration** (`contracts/config/hoodi.json`) with deployed EigenLayer contract addresses to reference. - **Holesky configuration** (`contracts/config/hoodi.json`) with deployed EigenLayer contract addresses to reference. #### Contracts being deployed - **DataHaven**: ServiceManager, VetoableSlasher, RewardsRegistry - **Snowbridge**: BeefyClient, AgentExecutor, Gateway, RewardsAgent - **EigenLayer**: References existing deployed contracts (not re-deployed) #### Deployment files When the deployment is done, a new file under `contracts/deployments` is generated with the addresses of the deployed contracts, for each chain (it will be overriden per chain if run multiple times). So we would have one `anvil.json`, `hoodi.json`, `holesky.json`, etc, with the addresses of the deployed contracts for reference and for later verification. #### Todo - [x] Test compatibility with existing `bun cli launch` and `bun cli deploy` commands #### For follow-up PRs - Fix verification issue with `foundry verify-contracts` when specifying the `chain` or `chain-id` parameter, needed for hoodi (https://github.com/foundry-rs/foundry/issues/7466). - Add `redeploy` feature to only override implementation contract and leave the proxy address untouched ## Usage Examples ```bash # Deploy to Hoodi network bun cli contracts deploy --chain hoodi # Check deployment status bun cli contracts status --chain hoodi # Verify contracts on block explorer bun cli contracts verify --chain hoodi ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added deployment and configuration support for new networks "hoodi" and "holesky", including new configuration and deployment files. * Introduced a CLI tool for managing contract deployments, status checks, and verification across supported chains. * Added example environment configuration and comprehensive deployment documentation. * Enabled contract verification and status reporting via the CLI with support for block explorer integration. * **Improvements** * Refactored deployment scripts for modularity, supporting both local and testnet environments. * Centralized and extended configuration loading to support additional contract addresses and network parameters. * Enhanced deployment utilities and typings to support multi-network deployments. * **Bug Fixes** * Improved input validation and error handling in CLI commands and deployment scripts. * Added explicit handling for zero address in operator strategy retrieval. * **Chores** * Updated documentation and configuration templates for easier onboarding and deployment management. * Improved logging and output formatting for deployment and verification processes. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Steve Degosserie <723552+stiiifff@users.noreply.github.com> Co-authored-by: Ahmad Kaouk <56095276+ahmadkaouk@users.noreply.github.com>
2025-08-21 10:02:31 +00:00
// Default Contracts command (runs check)
contractsCommand
.description("Show deployment plan, configuration, and status")
.option("--chain <value>", "Target chain (hoodi, holesky, mainnet)")
.option("--rpc-url <value>", "Chain RPC URL (optional, defaults based on chain)")
.option("--private-key <value>", "Private key for deployment", process.env.PRIVATE_KEY || "")
.option("--skip-verification", "Skip contract verification", false)
.hook("preAction", contractsPreActionHook)
.action(contractsCheck);
// ===== Exec ======
// Disabled until need arises
// program
// .command("exec <action> [args]")
// .description("Execute a standalone function against an running running network");
program.parseAsync(Bun.argv);