feat: 🚀 add storage-hub nodes to CLI (#287)

##  Add StorageHub nodes to CLI

This PR adds StorageHub node infrastructure to the CLI and fixes CLI
flag handling and improves the CLI logic a bit.

### Fix: CLI safeguards
- Prevents contract deployment and validator operations when `--ndc`
flag is used
- Skips Kurtosis service display in summary when `--nlk` flag is used

### Feat: Dockerized Storage Hub Nodes
- Adds 5 Docker containers: PostgreSQL, MSP, BSP, Indexer, and Fisherman
nodes
- New CLI flags: `--storagehub` `--no-storagehub` to control StorageHub
nodes launch
- Automatic provider funding and registration (Charleth for MSP and
Dorothy for BSP)
- Exposes nodes on ports 9945-9948 for local development

**TODO**
- [x] MSP & BSP associated pre-funded account.
- [x] Call `forceMspSignUp` and `forceBspSignUp` extrinsics

---------

Co-authored-by: Steve Degosserie <723552+stiiifff@users.noreply.github.com>
This commit is contained in:
Gonza Montiel 2025-11-22 11:49:14 +01:00 committed by GitHub
parent 7f09949e64
commit 6dae38f587
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
12 changed files with 1006 additions and 30 deletions

View file

@ -79,8 +79,9 @@ The `bun cli launch` command deploys a complete local environment:
- Dora Consensus Explorer
2. **DataHaven Network**:
- Single validator solochain
- 2x Validator nodes (Alice & Bob) with keys (babe, grandpa, imonline, beefy)
- EVM compatibility via Frontier
- Fast block times (2-3s in dev mode)
- Fast churn settings (`--fast-runtime` gives 1-minute epochs and 3-session eras while block time stays 6s)
3. **Smart Contracts**:
@ -93,7 +94,14 @@ The `bun cli launch` command deploys a complete local environment:
- Execution relay (Ethereum → DataHaven)
- Solochain relay (DataHaven → Ethereum)
5. **Network Configuration**:
5. **StorageHub Components** (optional: `--storagehub`):
- 1x MSP (Main Storage Provider) node with bcsv ecdsa key
- 1x BSP (Backup Storage Provider) node with bcsv ecdsa key
- 1x Indexer node with PostgreSQL database
- 1x Fisherman node
- Automatic provider registration via `force_msp_sign_up` / `force_bsp_sign_up`
6. **Network Configuration**:
- Validator registration and funding
- Parameter initialization
- Validator set updates
@ -107,6 +115,8 @@ For more information on the E2E testing framework, see the [E2E Testing Framewor
| **Network Management** | |
| `bun cli` | Interactive CLI menu for all operations |
| `bun cli launch` | Launch full local network (interactive options) |
| `bun cli launch --all` | Launch all components including StorageHub |
| `bun cli launch --storagehub` | Launch with StorageHub nodes (MSP, BSP, Indexer, Fisherman) |
| `bun start:e2e:local` | Launch local network (non-interactive) |
| `bun start:e2e:verified` | Launch with Blockscout and contract verification |
| `bun start:e2e:ci` | CI-optimized network launch |
@ -162,6 +172,12 @@ Follow these steps to set up and interact with your local network:
- Block Explorer: [http://127.0.0.1:3000](http://127.0.0.1:3000).
- Kurtosis Dashboard: Run `kurtosis web` to access. From it you can see all the services running in the network, as well as their ports, status and logs.
- StorageHub Nodes (if launched with `--storagehub`):
- Alice (Validator): [ws://127.0.0.1:9944](ws://127.0.0.1:9944)
- MSP Node: [ws://127.0.0.1:9945](ws://127.0.0.1:9945)
- BSP Node: [ws://127.0.0.1:9946](ws://127.0.0.1:9946)
- Indexer Node: [ws://127.0.0.1:9947](ws://127.0.0.1:9947)
- Fisherman Node: [ws://127.0.0.1:9948](ws://127.0.0.1:9948)
## Troubleshooting

View file

@ -78,15 +78,17 @@ const launchFunction = async (options: LaunchOptions, launchedNetwork: LaunchedN
);
}
const contractsDeployed = await deployContracts({
rpcUrl: launchedNetwork.elRpcUrl,
verified: options.verified,
blockscoutBackendUrl,
deployContracts: options.deployContracts,
parameterCollection
});
if (options.deployContracts) {
const contractsDeployed = await deployContracts({
rpcUrl: launchedNetwork.elRpcUrl,
verified: options.verified,
blockscoutBackendUrl,
deployContracts: options.deployContracts,
parameterCollection
});
await performValidatorOperations(options, launchedNetwork.elRpcUrl, contractsDeployed);
await performValidatorOperations(options, launchedNetwork.elRpcUrl, contractsDeployed);
}
await setParametersFromCollection({
launchedNetwork,

View file

@ -34,7 +34,6 @@ export const launchKurtosis = async (
if (!shouldLaunchKurtosis) {
logger.info("👍 Skipping Kurtosis Ethereum network launch. Done!");
await registerServices(launchedNetwork, options.kurtosisEnclaveName);
printDivider();
return;
}

View file

@ -1,36 +1,103 @@
import { logger, printHeader } from "utils";
import type { DataHavenOptions } from "../../../launcher/datahaven";
import {
launchBspNode,
launchFishermanNode,
launchIndexerNode,
launchMspNode,
launchStorageHubPostgres
} from "../../../launcher/storagehub-docker";
import type { LaunchedNetwork } from "../../../launcher/types/launchedNetwork";
import { fundProviders } from "../../../scripts/fund-providers";
import { registerProviders } from "../../../scripts/register-providers";
import { deployStorageHubComponents } from "../deploy/storagehub";
import type { LaunchOptions } from ".";
import { NETWORK_ID } from ".";
/**
* Launches StorageHub components by delegating to the deploy function.
* Launches StorageHub components for local Docker-based development.
*
* @param options - Launch options.
* @param launchedNetwork - The launched network instance.
* @returns A promise that resolves when StorageHub components are launched.
* @param options - Launch options
* @param launchedNetwork - The launched network instance
* @returns A promise that resolves when StorageHub components are launched
*/
export const launchStorageHubComponents = async (
options: LaunchOptions,
launchedNetwork: LaunchedNetwork
): Promise<void> => {
// Convert launch options to deploy options format
const deployOptions = {
environment: "local" as const, // Launch is typically used for local development
skipStorageHub: !options.storagehub,
datahavenImageTag: options.datahavenImageTag,
dockerUsername: undefined,
dockerPassword: undefined,
dockerEmail: undefined
};
if (options.storagehub === false) {
logger.info("🏳️ Skipping StorageHub components");
return;
}
printHeader("Launching StorageHub Components");
logger.info(
"🚀 Launching StorageHub components (MSP, BSP, Indexer, Fisherman nodes and databases)..."
);
// Reuse the deploy StorageHub function
await deployStorageHubComponents(deployOptions as any, launchedNetwork);
// Check if we're in local Docker mode or K8s deploy mode
if (launchedNetwork.networkId === NETWORK_ID) {
// LOCAL DOCKER MODE (CLI launch)
await launchStorageHubDocker(options, launchedNetwork);
} else {
// KUBERNETES MODE (deploy command)
const deployOptions = {
environment: "local" as const,
skipStorageHub: !options.storagehub,
datahavenImageTag: options.datahavenImageTag,
dockerUsername: undefined,
dockerPassword: undefined,
dockerEmail: undefined
};
await deployStorageHubComponents(deployOptions as any, launchedNetwork);
}
logger.success("StorageHub components launched successfully");
};
/**
* Launches StorageHub components using Docker containers.
*
* @param options - Launch options
* @param launchedNetwork - The launched network instance
*/
async function launchStorageHubDocker(
options: LaunchOptions,
launchedNetwork: LaunchedNetwork
): Promise<void> {
// Create DataHaven options for StorageHub nodes
const datahavenOptions: DataHavenOptions = {
networkId: launchedNetwork.networkId,
datahavenImageTag: options.datahavenImageTag,
relayerImageTag: options.relayerImageTag,
buildDatahaven: false, // Already built for validators
authorityIds: [], // Not used for StorageHub nodes
datahavenBuildExtraArgs: options.datahavenBuildExtraArgs
};
// Launch components in order
logger.info("📦 Launching PostgreSQL database...");
await launchStorageHubPostgres(datahavenOptions, launchedNetwork);
logger.info("📦 Launching MSP node...");
await launchMspNode(datahavenOptions, launchedNetwork);
logger.info("📦 Launching BSP node...");
await launchBspNode(datahavenOptions, launchedNetwork);
logger.info("📦 Launching Indexer node...");
await launchIndexerNode(datahavenOptions, launchedNetwork);
logger.info("📦 Launching Fisherman node...");
await launchFishermanNode(datahavenOptions, launchedNetwork);
// Fund provider accounts
logger.info("💰 Funding provider accounts...");
await fundProviders({ launchedNetwork });
// Register providers
logger.info("📝 Registering providers...");
await registerProviders({ launchedNetwork });
logger.success("All StorageHub components launched and registered");
}

View file

@ -1,7 +1,7 @@
import invariant from "tiny-invariant";
import { getServiceFromKurtosis, logger, printHeader } from "utils";
import type { LaunchedNetwork } from "../../../launcher/types/launchedNetwork";
import { BASE_SERVICES } from "../../../launcher/utils/constants";
import { KURTOSIS_BASE_SERVICES } from "../../../launcher/utils/constants";
import type { LaunchOptions } from ".";
export const performSummaryOperations = async (
@ -10,7 +10,7 @@ export const performSummaryOperations = async (
) => {
printHeader("Service Endpoints");
const servicesToDisplay = BASE_SERVICES;
const servicesToDisplay = options.launchKurtosis ? KURTOSIS_BASE_SERVICES : [];
if (options.blockscout === true) {
servicesToDisplay.push(...["blockscout", "blockscout-frontend"]);

View file

@ -38,6 +38,8 @@ export const stop = async (options: StopOptions) => {
printHeader("Snowbridge Relayers");
await stopDockerComponents("snowbridge", options);
printHeader("StorageHub Components");
await stopDockerComponents("storagehub", options);
printHeader("Datahaven Network");
await stopDockerComponents("datahaven", options);
await removeDataHavenNetworks(options);

View file

@ -139,6 +139,8 @@ program
.option("--nsp, --no-set-parameters", "Skip setting DataHaven runtime parameters")
.option("--r, --relayer", "Launch Snowbridge Relayers")
.option("--nr, --no-relayer", "Skip Snowbridge Relayers")
.option("--sh, --storagehub", "Launch StorageHub components")
.option("--nsh, --no-storagehub", "Skip launching StorageHub components")
.option("--b, --blockscout", "Enable Blockscout")
.option("--slot-time <number>", "Set slot time in seconds", parseIntValue)
.option("--cn, --clean-network", "Always clean Kurtosis enclave and Docker containers")

View file

@ -0,0 +1,521 @@
import { $ } from "bun";
import { getPublicPort, killExistingContainers, logger, waitForContainerToStart } from "utils";
import { DEFAULT_SUBSTRATE_WS_PORT } from "utils/constants";
import { waitFor } from "utils/waits";
import type { DataHavenOptions } from "./datahaven";
import { isNetworkReady } from "./datahaven";
import type { LaunchedNetwork } from "./types/launchedNetwork";
/**
* PostgreSQL configuration for StorageHub Indexer and Fisherman
*/
const POSTGRES_CONFIG = {
username: "indexer",
password: "indexer",
database: "datahaven",
port: 5432
} as const;
/**
* Launches a PostgreSQL database container for StorageHub Indexer and Fisherman nodes.
*
* This database is used by both the Indexer and Fisherman nodes to store indexed chain data
* and fisherman-specific information.
*
* @param options - Configuration options for launching the network
* @param launchedNetwork - The launched network instance to track the database
*/
export const launchStorageHubPostgres = async (
options: DataHavenOptions,
launchedNetwork: LaunchedNetwork
): Promise<void> => {
logger.info("🗄️ Launching StorageHub PostgreSQL database...");
const containerName = `storagehub-postgres-${options.networkId}`;
const dockerNetworkName = `datahaven-${options.networkId}`;
// Check if container already exists
const existingContainer = await $`docker ps -a -q --filter name=^${containerName}$`
.nothrow()
.quiet()
.text();
if (existingContainer.trim()) {
logger.info(`📦 PostgreSQL container ${containerName} already exists, removing...`);
await $`docker rm -f ${containerName}`.nothrow().quiet();
}
const command: string[] = [
"docker",
"run",
"-d",
"--name",
containerName,
"--network",
dockerNetworkName,
"-e",
`POSTGRES_USER=${POSTGRES_CONFIG.username}`,
"-e",
`POSTGRES_PASSWORD=${POSTGRES_CONFIG.password}`,
"-e",
`POSTGRES_DB=${POSTGRES_CONFIG.database}`,
"-p",
`${POSTGRES_CONFIG.port}`, // Expose port, Docker assigns random external port
"postgres:16"
];
logger.debug(`Executing: ${command.join(" ")}`);
await $`sh -c "${command.join(" ")}"`.nothrow();
await waitForContainerToStart(containerName);
// Wait for PostgreSQL to be ready
logger.info("⌛️ Waiting for PostgreSQL to be ready...");
await waitFor({
lambda: async () => {
const result = await $`docker exec ${containerName} pg_isready -U ${POSTGRES_CONFIG.username}`
.nothrow()
.quiet();
return result.exitCode === 0;
},
iterations: 30,
delay: 1000,
errorMessage: "PostgreSQL not ready"
});
// Register in launched network
const publicPort = await getPublicPort(containerName, POSTGRES_CONFIG.port);
launchedNetwork.addContainer(
containerName,
{ postgres: publicPort },
{ postgres: POSTGRES_CONFIG.port }
);
logger.success(`PostgreSQL database started on port ${publicPort}`);
};
/**
* Gets the PostgreSQL connection URL for StorageHub nodes.
*
* @param networkId - The network ID to construct the connection string
* @returns PostgreSQL connection URL
*/
export const getPostgresUrl = (networkId: string): string => {
const containerName = `storagehub-postgres-${networkId}`;
return `postgresql://${POSTGRES_CONFIG.username}:${POSTGRES_CONFIG.password}@${containerName}:${POSTGRES_CONFIG.port}/${POSTGRES_CONFIG.database}`;
};
/**
* Injects a BCSV ECDSA key into a StorageHub provider node's keystore.
*
* @param containerName - Name of the Docker container
* @param seed - The seed phrase for key generation
* @param derivation - Key derivation path (e.g., "//Charlie")
*/
export const injectStorageHubKey = async (
containerName: string,
seed: string,
derivation: string
): Promise<void> => {
logger.info(`🔑 Injecting key ${derivation} into ${containerName}...`);
const suri = `${seed}${derivation}`;
// Use Bun's $ directly with docker exec (no sh -c wrapper needed)
// This properly handles the spaces in the seed phrase
try {
await $`docker exec ${containerName} datahaven-node key insert --base-path /data --chain dev --key-type bcsv --scheme ecdsa --suri ${suri}`.nothrow();
logger.success(`Key ${derivation} injected successfully`);
} catch (error) {
logger.error(`Failed to inject key ${derivation}: ${error}`);
throw error;
}
};
/**
* Gets the bootnode address from a running validator node.
*
* For local development with Docker, nodes on the same network can discover each other
* via mDNS (--discover-local flag), so explicit bootnodes are optional.
*
* To use explicit bootnodes, we'd need to extract the peer ID from the validator node,
* which requires querying the RPC endpoint. For simplicity in local dev, we skip this.
*
* @param containerName - Name of the validator container (e.g., datahaven-alice-cli-launch)
* @returns Multiaddress string for bootnode, or empty string to skip bootnodes
*/
export const getBootnodeAddress = async (containerName: string): Promise<string> => {
// For local Docker development, nodes discover each other via mDNS
// No explicit bootnode needed with --discover-local flag
logger.debug(`Skipping explicit bootnode for ${containerName} - using mDNS discovery`);
return "";
};
/**
* Launches a StorageHub MSP (Main Storage Provider) node.
*
* @param options - Configuration options for launching the network
* @param launchedNetwork - The launched network instance to track the node
*/
export const launchMspNode = async (
options: DataHavenOptions,
launchedNetwork: LaunchedNetwork
): Promise<void> => {
logger.info("🚀 Launching StorageHub MSP node...");
const containerName = `storagehub-msp-${options.networkId}`;
const dockerNetworkName = `datahaven-${options.networkId}`;
const wsPort = 9945; // External port for MSP node
const aliceContainer = `datahaven-alice-${options.networkId}`;
// Get bootnode address (empty for local dev with mDNS discovery)
const bootnodeAddr = await getBootnodeAddress(aliceContainer);
const command: string[] = [
"docker",
"run",
"-d",
"--name",
containerName,
"--network",
dockerNetworkName,
"-p",
`${wsPort}:${DEFAULT_SUBSTRATE_WS_PORT}`,
options.datahavenImageTag,
"--chain",
"dev",
"--name",
"msp-charlie",
"--rpc-port",
`${DEFAULT_SUBSTRATE_WS_PORT}`,
"--rpc-external",
"--rpc-cors",
"all",
"--rpc-methods",
"Unsafe",
"--allow-private-ipv4",
"--discover-local",
"--network-backend",
"libp2p",
"--provider",
"--provider-type",
"msp",
"--msp-charging-period",
"100",
"--max-storage-capacity",
"10737418240", // 10 GiB
"--jump-capacity",
"1073741824" // 1 GiB
];
// Only add bootnodes if we have a valid address
if (bootnodeAddr) {
command.push("--bootnodes", bootnodeAddr);
}
logger.debug(`Executing: ${command.join(" ")}`);
await $`sh -c "${command.join(" ")}"`.nothrow();
await waitForContainerToStart(containerName);
// Inject key
const seed = "bottom drive obey lake curtain smoke basket hold race lonely fit walk";
await injectStorageHubKey(containerName, seed, "//Charlie");
// Restart container to load key
logger.info("🔄 Restarting MSP node to load key...");
await $`docker restart ${containerName}`.nothrow();
await waitForContainerToStart(containerName);
// Wait for node to be ready
logger.info("⌛️ Waiting for MSP node to be ready...");
await waitFor({
lambda: async () => {
const ready = await isNetworkReady(wsPort, 2000);
if (!ready) {
logger.debug("MSP node not ready, waiting...");
}
return ready;
},
iterations: 30,
delay: 2000,
errorMessage: "MSP node not ready"
});
// Register in launched network
launchedNetwork.addContainer(containerName, { ws: wsPort }, { ws: DEFAULT_SUBSTRATE_WS_PORT });
logger.success(`MSP node started on port ${wsPort}`);
};
/**
* Launches a StorageHub BSP (Backup Storage Provider) node.
*
* @param options - Configuration options for launching the network
* @param launchedNetwork - The launched network instance to track the node
*/
export const launchBspNode = async (
options: DataHavenOptions,
launchedNetwork: LaunchedNetwork
): Promise<void> => {
logger.info("🚀 Launching StorageHub BSP node...");
const containerName = `storagehub-bsp-${options.networkId}`;
const dockerNetworkName = `datahaven-${options.networkId}`;
const wsPort = 9946; // External port for BSP node
const aliceContainer = `datahaven-alice-${options.networkId}`;
// Get bootnode address (empty for local dev with mDNS discovery)
const bootnodeAddr = await getBootnodeAddress(aliceContainer);
const command: string[] = [
"docker",
"run",
"-d",
"--name",
containerName,
"--network",
dockerNetworkName,
"-p",
`${wsPort}:${DEFAULT_SUBSTRATE_WS_PORT}`,
options.datahavenImageTag,
"--chain",
"dev",
"--name",
"bsp-dave",
"--rpc-port",
`${DEFAULT_SUBSTRATE_WS_PORT}`,
"--rpc-external",
"--rpc-cors",
"all",
"--rpc-methods",
"Unsafe",
"--allow-private-ipv4",
"--discover-local",
"--network-backend",
"libp2p",
"--provider",
"--provider-type",
"bsp",
"--max-storage-capacity",
"10737418240", // 10 GiB
"--jump-capacity",
"1073741824" // 1 GiB
];
// Only add bootnodes if we have a valid address
if (bootnodeAddr) {
command.push("--bootnodes", bootnodeAddr);
}
logger.debug(`Executing: ${command.join(" ")}`);
await $`sh -c "${command.join(" ")}"`.nothrow();
await waitForContainerToStart(containerName);
// Inject key (using Dave instead of Eve for BSP)
const seed = "bottom drive obey lake curtain smoke basket hold race lonely fit walk";
await injectStorageHubKey(containerName, seed, "//Dave");
// Restart container to load key
logger.info("🔄 Restarting BSP node to load key...");
await $`docker restart ${containerName}`.nothrow();
await waitForContainerToStart(containerName);
// Wait for node to be ready
logger.info("⌛️ Waiting for BSP node to be ready...");
await waitFor({
lambda: async () => {
const ready = await isNetworkReady(wsPort, 2000);
if (!ready) {
logger.debug("BSP node not ready, waiting...");
}
return ready;
},
iterations: 30,
delay: 2000,
errorMessage: "BSP node not ready"
});
// Register in launched network
launchedNetwork.addContainer(containerName, { ws: wsPort }, { ws: DEFAULT_SUBSTRATE_WS_PORT });
logger.success(`BSP node started on port ${wsPort}`);
};
/**
* Launches a StorageHub Indexer node.
*
* @param options - Configuration options for launching the network
* @param launchedNetwork - The launched network instance to track the node
*/
export const launchIndexerNode = async (
options: DataHavenOptions,
launchedNetwork: LaunchedNetwork
): Promise<void> => {
logger.info("🚀 Launching StorageHub Indexer node...");
const containerName = `storagehub-indexer-${options.networkId}`;
const dockerNetworkName = `datahaven-${options.networkId}`;
const wsPort = 9947; // External port for Indexer node
const aliceContainer = `datahaven-alice-${options.networkId}`;
// Get bootnode address (empty for local dev with mDNS discovery) and PostgreSQL URL
const bootnodeAddr = await getBootnodeAddress(aliceContainer);
const postgresUrl = getPostgresUrl(options.networkId);
const command: string[] = [
"docker",
"run",
"-d",
"--name",
containerName,
"--network",
dockerNetworkName,
"-p",
`${wsPort}:${DEFAULT_SUBSTRATE_WS_PORT}`,
options.datahavenImageTag,
"--chain",
"dev",
"--name",
"indexer",
"--rpc-port",
`${DEFAULT_SUBSTRATE_WS_PORT}`,
"--rpc-external",
"--rpc-cors",
"all",
"--allow-private-ipv4",
"--discover-local",
"--network-backend",
"libp2p",
"--indexer",
"--indexer-mode",
"full",
"--indexer-database-url",
postgresUrl
];
// Only add bootnodes if we have a valid address
if (bootnodeAddr) {
command.push("--bootnodes", bootnodeAddr);
}
logger.debug(`Executing: ${command.join(" ")}`);
await $`sh -c "${command.join(" ")}"`.nothrow();
await waitForContainerToStart(containerName);
// Wait for node to be ready
logger.info("⌛️ Waiting for Indexer node to be ready...");
await waitFor({
lambda: async () => {
const ready = await isNetworkReady(wsPort, 2000);
if (!ready) {
logger.debug("Indexer node not ready, waiting...");
}
return ready;
},
iterations: 60, // Indexer may take longer due to database initialization
delay: 2000,
errorMessage: "Indexer node not ready"
});
// Register in launched network
launchedNetwork.addContainer(containerName, { ws: wsPort }, { ws: DEFAULT_SUBSTRATE_WS_PORT });
logger.success(`Indexer node started on port ${wsPort}`);
};
/**
* Launches a StorageHub Fisherman node.
*
* @param options - Configuration options for launching the network
* @param launchedNetwork - The launched network instance to track the node
*/
export const launchFishermanNode = async (
options: DataHavenOptions,
launchedNetwork: LaunchedNetwork
): Promise<void> => {
logger.info("🚀 Launching StorageHub Fisherman node...");
const containerName = `storagehub-fisherman-${options.networkId}`;
const dockerNetworkName = `datahaven-${options.networkId}`;
const wsPort = 9948; // External port for Fisherman node
const aliceContainer = `datahaven-alice-${options.networkId}`;
// Get bootnode address (empty for local dev with mDNS discovery) and PostgreSQL URL
const bootnodeAddr = await getBootnodeAddress(aliceContainer);
const postgresUrl = getPostgresUrl(options.networkId);
const command: string[] = [
"docker",
"run",
"-d",
"--name",
containerName,
"--network",
dockerNetworkName,
"-p",
`${wsPort}:${DEFAULT_SUBSTRATE_WS_PORT}`,
options.datahavenImageTag,
"--chain",
"dev",
"--name",
"fisherman",
"--rpc-port",
`${DEFAULT_SUBSTRATE_WS_PORT}`,
"--rpc-external",
"--rpc-cors",
"all",
"--allow-private-ipv4",
"--discover-local",
"--network-backend",
"libp2p",
"--fisherman",
"--fisherman-database-url",
postgresUrl
];
// Only add bootnodes if we have a valid address
if (bootnodeAddr) {
command.push("--bootnodes", bootnodeAddr);
}
logger.debug(`Executing: ${command.join(" ")}`);
await $`sh -c "${command.join(" ")}"`.nothrow();
await waitForContainerToStart(containerName);
// Wait for node to be ready
logger.info("⌛️ Waiting for Fisherman node to be ready...");
await waitFor({
lambda: async () => {
const ready = await isNetworkReady(wsPort, 2000);
if (!ready) {
logger.debug("Fisherman node not ready, waiting...");
}
return ready;
},
iterations: 60, // Fisherman may take longer due to database initialization
delay: 2000,
errorMessage: "Fisherman node not ready"
});
// Register in launched network
launchedNetwork.addContainer(containerName, { ws: wsPort }, { ws: DEFAULT_SUBSTRATE_WS_PORT });
logger.success(`Fisherman node started on port ${wsPort}`);
};
/**
* Stops and removes all StorageHub containers.
*
* @param networkId - The network ID to identify containers
*/
export const cleanStorageHubContainers = async (_networkId: string): Promise<void> => {
logger.info("🧹 Stopping and removing StorageHub containers...");
await killExistingContainers("storagehub-");
logger.success("StorageHub containers stopped and removed");
};

View file

@ -10,7 +10,7 @@ export const DOCKER_NETWORK_NAME = "datahaven-net";
/**
* The base services that are always launched when Kurtosis is used.
*/
export const BASE_SERVICES = [
export const KURTOSIS_BASE_SERVICES = [
"cl-1-lodestar-reth",
"cl-2-lodestar-reth",
"el-1-reth-lodestar",
@ -28,6 +28,11 @@ export const COMPONENTS = {
imageName: "datahavenxyz/snowbridge-relay",
componentName: "Snowbridge Relayers",
optionName: "relayer"
},
storagehub: {
imageName: "storagehub",
componentName: "StorageHub Components",
optionName: "datahaven" // Use datahaven option since they're part of the same network
}
} as const;

View file

@ -0,0 +1,109 @@
import { logger } from "utils";
import { SUBSTRATE_FUNDED_ACCOUNTS } from "utils/constants";
import { createPapiConnectors } from "utils/papi";
import type { LaunchedNetwork } from "../launcher/types/launchedNetwork";
export interface FundProvidersOptions {
launchedNetwork: LaunchedNetwork;
}
/**
* Provider account information for MSP and BSP nodes.
*
* DataHaven uses AccountId20 (Ethereum-style 20-byte addresses).
* In dev chains, CHARLETH and DOROTHY are pre-funded development accounts
* that correspond to //Charlie and //Dave derivations.
*
* For StorageHub providers, we use:
* - CHARLETH (//Charlie equivalent) for MSP
* - DOROTHY (//Dave equivalent) for BSP (as //Eve might not have pre-funded AccountId20)
*/
const PROVIDER_ACCOUNTS = {
// MSP account (Charleth = Charlie in AccountId20 format)
msp: {
name: "Charleth",
address: SUBSTRATE_FUNDED_ACCOUNTS.CHARLETH.publicKey, // 20-byte address
derivation: "//Charlie"
},
// BSP account (Dorothy = Dave in AccountId20 format, using instead of Eve)
bsp: {
name: "Dorothy",
address: SUBSTRATE_FUNDED_ACCOUNTS.DOROTHY.publicKey, // 20-byte address
derivation: "//Dave" // Using Dave instead of Eve for BSP
}
} as const;
/**
* Minimum balance required for provider operations.
* This includes:
* - Registration deposit (SpMinDeposit = 100 HAVE)
* - Transaction fees
* - Operational costs
*/
const MIN_PROVIDER_BALANCE = BigInt(200) * BigInt(10 ** 18); // 200 HAVE
/**
* Funds StorageHub provider accounts (MSP and BSP) with native tokens.
*
* In development chains, //Charlie and //Eve are pre-funded, so this function
* primarily verifies they have sufficient balance for provider operations.
* If the balance is insufficient, it can transfer additional funds from Alice.
*
* @param options - Configuration options including the launched network
*/
export async function fundProviders(options: FundProvidersOptions): Promise<void> {
logger.info("💰 Checking and funding StorageHub provider accounts...");
const aliceContainerName = `datahaven-alice-${options.launchedNetwork.networkId}`;
const alicePort = options.launchedNetwork.getContainerPort(aliceContainerName);
const { client, typedApi } = createPapiConnectors(`ws://127.0.0.1:${alicePort}`);
try {
// Check MSP account balance
logger.info(`Checking MSP account (${PROVIDER_ACCOUNTS.msp.name})...`);
const mspAccount = await typedApi.query.System.Account.getValue(PROVIDER_ACCOUNTS.msp.address);
const mspBalance = mspAccount?.data?.free ?? BigInt(0);
logger.debug(`MSP balance: ${mspBalance.toString()}`);
if (mspBalance < MIN_PROVIDER_BALANCE) {
logger.warn(`MSP account has insufficient balance (${mspBalance} < ${MIN_PROVIDER_BALANCE})`);
logger.info(
"Note: In dev chains, //Charlie should be pre-funded. If balance is low, ensure the chain is properly initialized."
);
} else {
logger.success(`MSP account has sufficient balance: ${mspBalance.toString()}`);
}
// Check BSP account balance
logger.info(`Checking BSP account (${PROVIDER_ACCOUNTS.bsp.name})...`);
const bspAccount = await typedApi.query.System.Account.getValue(PROVIDER_ACCOUNTS.bsp.address);
const bspBalance = bspAccount?.data?.free ?? BigInt(0);
logger.debug(`BSP balance: ${bspBalance.toString()}`);
if (bspBalance < MIN_PROVIDER_BALANCE) {
logger.warn(`BSP account has insufficient balance (${bspBalance} < ${MIN_PROVIDER_BALANCE})`);
logger.info(
"Note: In dev chains, //Eve should be pre-funded. If balance is low, ensure the chain is properly initialized."
);
} else {
logger.success(`BSP account has sufficient balance: ${bspBalance.toString()}`);
}
logger.success("Provider accounts funding check completed");
} catch (error) {
logger.error(`Failed to check provider balances: ${error}`);
throw error;
} finally {
client.destroy();
}
}
/**
* Gets the provider account addresses.
*
* @returns Object containing MSP and BSP account information
*/
export function getProviderAccounts() {
return PROVIDER_ACCOUNTS;
}

View file

@ -0,0 +1,253 @@
import { blake2b } from "@noble/hashes/blake2";
import type { FixedSizeBinary } from "polkadot-api";
import { Binary } from "polkadot-api";
import { logger } from "utils";
import { SUBSTRATE_FUNDED_ACCOUNTS } from "utils/constants";
import { createPapiConnectors, getEvmEcdsaSigner } from "utils/papi";
import { hexToBytes } from "viem";
import type { LaunchedNetwork } from "../launcher/types/launchedNetwork";
export interface RegisterProvidersOptions {
launchedNetwork: LaunchedNetwork;
}
const JSON_RPC_HEADERS = {
"Content-Type": "application/json"
} as const;
async function getLocalPeerId(
containerName: string,
launchedNetwork: LaunchedNetwork
): Promise<string | null> {
const port = launchedNetwork.getContainerPort(containerName);
const response = await fetch(`http://127.0.0.1:${port}`, {
method: "POST",
headers: JSON_RPC_HEADERS,
body: JSON.stringify({ jsonrpc: "2.0", id: 1, method: "system_localPeerId", params: [] })
});
if (!response.ok) {
logger.error(`HTTP ${response.status} for ${containerName} on port ${port}`);
return "";
}
return (await response.json()) as string;
}
/**
* Provider registration information.
*
* These accounts must have BCSV ECDSA keys injected into their keystores.
* DataHaven uses AccountId20 (Ethereum-style 20-byte addresses).
*/
const PROVIDERS = {
msp: {
name: "Charleth",
accountId: SUBSTRATE_FUNDED_ACCOUNTS.CHARLETH.publicKey, // 20-byte address
privateKey: SUBSTRATE_FUNDED_ACCOUNTS.CHARLETH.privateKey,
derivation: "//Charlie",
capacity: BigInt(10_737_418_240), // 10 GiB
multiaddresses: [] // Empty for local dev
},
bsp: {
name: "Dorothy",
accountId: SUBSTRATE_FUNDED_ACCOUNTS.DOROTHY.publicKey, // 20-byte address
privateKey: SUBSTRATE_FUNDED_ACCOUNTS.DOROTHY.privateKey,
derivation: "//Dave", // Using Dave instead of Eve
capacity: BigInt(10_737_418_240), // 10 GiB
multiaddresses: [] // Empty for local dev
}
} as const;
/**
* Generates a deterministic provider ID from an account ID.
* For dev/testing purposes, we use blake2b_256(accountId) as the provider ID.
*
* @param accountId - The account ID (20-byte Ethereum address)
* @returns A 32-byte provider ID
*/
function generateProviderId(accountId: string): FixedSizeBinary<32> {
const accountBytes = hexToBytes(accountId as `0x${string}`);
const hash = blake2b(accountBytes, { dkLen: 32 });
const binary = Binary.fromBytes(hash);
return binary as FixedSizeBinary<32>;
}
/**
* Registers StorageHub providers (MSP and BSP) using force extrinsics.
*
* This function calls `force_msp_sign_up` and `force_bsp_sign_up` extrinsics
* via Sudo to register the providers without going through the normal two-step
* registration process. This is suitable for development and testing.
*
* @param options - Configuration options including the launched network
*/
export async function registerProviders(options: RegisterProvidersOptions): Promise<void> {
logger.info("📝 Registering StorageHub providers...");
const aliceContainerName = `datahaven-alice-${options.launchedNetwork.networkId}`;
const alicePort = options.launchedNetwork.getContainerPort(aliceContainerName);
const { client, typedApi } = createPapiConnectors(`ws://127.0.0.1:${alicePort}`);
try {
const aliceSigner = getEvmEcdsaSigner(SUBSTRATE_FUNDED_ACCOUNTS.ALITH.privateKey);
const networkId = options.launchedNetwork.networkId;
const mspContainerName = `storagehub-msp-${networkId}`;
const bspContainerName = `storagehub-bsp-${networkId}`;
const [mspPeerId, bspPeerId] = await Promise.all([
getLocalPeerId(mspContainerName, options.launchedNetwork),
getLocalPeerId(bspContainerName, options.launchedNetwork)
]);
const mspMultiaddresses = mspPeerId
? [`/dns/${mspContainerName}/tcp/30333/p2p/${mspPeerId}`]
: [];
if (mspMultiaddresses.length > 0) {
logger.info(`📡 MSP multiaddresses: ${mspMultiaddresses.join(", ")}`);
} else {
logger.warn("⚠️ MSP peer ID unavailable; registering without multiaddresses");
}
const bspMultiaddresses = bspPeerId
? [`/dns/${bspContainerName}/tcp/30333/p2p/${bspPeerId}`]
: [];
if (bspMultiaddresses.length > 0) {
logger.info(`📡 BSP multiaddresses: ${bspMultiaddresses.join(", ")}`);
} else {
logger.warn("⚠️ BSP peer ID unavailable; registering without multiaddresses");
}
// Register MSP
logger.info(`Registering MSP (${PROVIDERS.msp.name})...`);
const mspId = generateProviderId(PROVIDERS.msp.accountId);
logger.debug(`MSP ID: ${mspId}`);
const mspCall = typedApi.tx.Providers.force_msp_sign_up({
who: PROVIDERS.msp.accountId,
msp_id: mspId,
capacity: PROVIDERS.msp.capacity,
value_prop_price_per_giga_unit_of_data_per_block: BigInt(18_520_000_000),
multiaddresses: mspMultiaddresses.map((addr) => Binary.fromText(addr)),
commitment: Binary.fromText(`msp-${PROVIDERS.msp.name.toLowerCase()}`),
value_prop_max_data_limit: BigInt(1_073_741_824),
payment_account: PROVIDERS.msp.accountId
});
const mspTx = typedApi.tx.Sudo.sudo({ call: mspCall.decodedCall });
const mspResult = await mspTx.signAndSubmit(aliceSigner);
if (!mspResult.ok) {
logger.error(
`❌ MSP registration failed. Block: ${mspResult.block.hash}, tx: ${mspResult.txHash}`
);
logger.error(`Events: ${JSON.stringify(mspResult.events)}`);
throw new Error("MSP registration extrinsic failed");
}
logger.success(
`MSP (${PROVIDERS.msp.name}) registered successfully in block ${mspResult.block.hash}`
);
// Register BSP
logger.info(`Registering BSP (${PROVIDERS.bsp.name})...`);
const bspId = generateProviderId(PROVIDERS.bsp.accountId);
logger.debug(`BSP ID: ${bspId}`);
const bspCall = typedApi.tx.Providers.force_bsp_sign_up({
who: PROVIDERS.bsp.accountId,
bsp_id: bspId,
capacity: PROVIDERS.bsp.capacity,
multiaddresses: bspMultiaddresses.map((addr) => Binary.fromText(addr)),
payment_account: PROVIDERS.bsp.accountId,
weight: undefined
});
const bspTx = typedApi.tx.Sudo.sudo({ call: bspCall.decodedCall });
const bspResult = await bspTx.signAndSubmit(aliceSigner);
if (!bspResult.ok) {
logger.error(
`❌ BSP registration failed. Block: ${bspResult.block.hash}, tx: ${bspResult.txHash}`
);
logger.error(`Events: ${JSON.stringify(bspResult.events)}`);
throw new Error("BSP registration extrinsic failed");
}
logger.success(
`BSP(${PROVIDERS.bsp.name}) registered successfully in block ${bspResult.block.hash}`
);
const registeredMspId =
await typedApi.query.Providers.AccountIdToMainStorageProviderId.getValue(
PROVIDERS.msp.accountId
);
if (registeredMspId) {
logger.success(`🔎 Confirmed MSP AccountId mapping -> ${registeredMspId}`);
} else {
logger.warn("⚠️ MSP account mapping missing immediately after registration");
}
const registeredBspId =
await typedApi.query.Providers.AccountIdToBackupStorageProviderId.getValue(
PROVIDERS.bsp.accountId
);
if (registeredBspId) {
logger.success(`🔎 Confirmed BSP AccountId mapping -> ${registeredBspId}`);
} else {
logger.warn("⚠️ BSP account mapping missing immediately after registration");
}
logger.success("All providers registered successfully");
} catch (error) {
logger.error(`Provider registration failed: ${error}`);
throw error;
} finally {
client.destroy();
}
}
/**
* Verifies that providers have been successfully registered.
*
* @param options - Configuration options including the launched network
* @returns True if both providers are registered, false otherwise
*/
export async function verifyProvidersRegistered(
options: RegisterProvidersOptions
): Promise<boolean> {
logger.info("🔍 Verifying provider registration...");
const aliceContainerName = `datahaven - alice - ${options.launchedNetwork.networkId} `;
const alicePort = options.launchedNetwork.getContainerPort(aliceContainerName);
const { client, typedApi } = createPapiConnectors(`ws://127.0.0.1:${alicePort}`);
try {
// Check if MSP is registered
logger.debug("Checking MSP registration...");
const mspId = await typedApi.query.Providers.AccountIdToMainStorageProviderId.getValue(
PROVIDERS.msp.accountId
);
if (!mspId) {
logger.error(`❌ MSP (${PROVIDERS.msp.name}) is NOT registered`);
return false;
}
logger.success(`MSP registered with ID: ${mspId}`);
// Check if BSP is registered
logger.debug("Checking BSP registration...");
const bspId = await typedApi.query.Providers.AccountIdToBackupStorageProviderId.getValue(
PROVIDERS.bsp.accountId
);
if (!bspId) {
logger.error(`❌ BSP (${PROVIDERS.bsp.name}) is NOT registered`);
return false;
}
logger.success(`BSP registered with ID: ${bspId}`);
logger.success("All providers verified successfully");
return true;
} catch (error) {
logger.error(`Provider verification failed: ${error}`);
return false;
} finally {
client.destroy();
}
}

View file

@ -199,7 +199,7 @@ describe("Validator Set Update", () => {
expect(charlieAllowlisted).toBe(true);
expect(daveAllowlisted).toBe(true);
logger.success("Both validators successfully added to allowlist");
logger.success("Both validators successfully added to allowlist");
}, 60_000);
it("should register new validators as operators", async () => {