diff --git a/README.md b/README.md index 72ee20fc..fc846154 100644 --- a/README.md +++ b/README.md @@ -1,63 +1,190 @@ # DataHaven 🫎 -An EVM compatible Substrate chain, powered by StorageHub and secured by EigenLayer. +An EVM-compatible Substrate blockchain secured by EigenLayer, bridging Ethereum and Substrate ecosystems through trustless cross-chain communication. -## Repo Structure +## Overview + +DataHaven is an EigenLayer Actively Validated Service (AVS) that combines: + +- **EVM Compatibility**: Full Ethereum support via Frontier pallets for smart contracts and dApps +- **EigenLayer Security**: Validator set secured by Ethereum's economic security through restaking +- **Cross-chain Bridge**: Seamless asset and message transfers with Ethereum via Snowbridge +- **Dynamic Validators**: Operator registry managed on-chain through EigenLayer contracts +- **Performance Rewards**: Validator incentives distributed cross-chain from Ethereum + +## Architecture + +DataHaven bridges two major blockchain ecosystems: + +``` +┌───────────────────────────────────────────────────────────────┐ +│ Ethereum (L1) │ +│ ┌────────────────────────────────────────────────────────┐ │ +│ │ EigenLayer AVS Contracts │ │ +│ │ • DataHavenServiceManager (operator lifecycle) │ │ +│ │ • RewardsRegistry (performance tracking) │ │ +│ │ • VetoableSlasher (misbehavior penalties) │ │ +│ └────────────────────────────────────────────────────────┘ │ +│ ↕ │ +│ Snowbridge Protocol │ +└───────────────────────────────────────────────────────────────┘ + ↕ +┌───────────────────────────────────────────────────────────────┐ +│ DataHaven (Substrate) │ +│ ┌────────────────────────────────────────────────────────┐ │ +│ │ Custom Pallets │ │ +│ │ • External Validators (sync validator set) │ │ +│ │ • Native Transfer (cross-chain tokens) │ │ +│ │ • Rewards (distribute validator rewards) │ │ +│ │ • Frontier (EVM compatibility) │ │ +│ └────────────────────────────────────────────────────────┘ │ +└───────────────────────────────────────────────────────────────┘ +``` + +## Repository Structure + +``` +datahaven/ +├── contracts/ # EigenLayer AVS smart contracts +│ ├── src/ # Service Manager, Rewards Registry, Slasher +│ ├── script/ # Deployment scripts +│ └── test/ # Foundry test suites +├── operator/ # Substrate-based DataHaven node +│ ├── node/ # Node implementation & chain spec +│ ├── pallets/ # Custom pallets (validators, rewards, transfers) +│ └── runtime/ # Runtime configurations (mainnet/stagenet/testnet) +├── test/ # E2E testing framework +│ ├── suites/ # Integration test scenarios +│ ├── framework/ # Test utilities and helpers +│ └── launcher/ # Network deployment automation +├── deploy/ # Kubernetes deployment charts +│ ├── charts/ # Helm charts for nodes and relayers +│ └── environments/ # Environment-specific configurations +├── tools/ # GitHub automation and release scripts +└── .github/ # CI/CD workflows +``` + +Each directory contains its own README with detailed information. See: +- [contracts/README.md](contracts/README.md) - Smart contract development +- [operator/README.md](operator/README.md) - Node building and runtime development +- [test/README.md](test/README.md) - E2E testing and network deployment +- [deploy/README.md](deploy/README.md) - Kubernetes deployment +- [tools/README.md](tools/README.md) - Development tools + +## Quick Start + +### Prerequisites + +- [Kurtosis](https://docs.kurtosis.com/install) - Network orchestration +- [Bun](https://bun.sh/) v1.2+ - TypeScript runtime +- [Docker](https://www.docker.com/) - Container management +- [Foundry](https://getfoundry.sh/) - Solidity toolkit +- [Rust](https://www.rust-lang.org/tools/install) - For building the operator +- [Helm](https://helm.sh/) - Kubernetes deployments (optional) +- [Zig](https://ziglang.org/) - For macOS cross-compilation (macOS only) + +### Launch Local Network + +The fastest way to get started is with the interactive CLI: ```bash -datahaven/ -├── .github/ # GitHub Actions workflows. -├── contracts/ # Implementation of the DataHaven AVS (Autonomous Verifiable Service) smart contracts to interact with EigenLayer. -├── operator/ # DataHaven node based on Substrate. The "Operator" in EigenLayer terms. -├── test/ # Integration tests for the AVS and Operator. -├── resources/ # Miscellaneous resources for the DataHaven project. -└── README.md +cd test +bun i # Install dependencies +bun cli launch # Interactive launcher with prompts ``` -## E2E CLI +This deploys a complete environment including: +- **Ethereum network**: 2x EL clients (reth), 2x CL clients (lodestar) +- **Block explorers**: Blockscout (optional), Dora consensus explorer +- **DataHaven node**: Single validator with fast block times +- **AVS contracts**: Deployed and configured on Ethereum +- **Snowbridge relayers**: Bidirectional message passing -This repo comes with a CLI for launching a local DataHaven network, packaged with: +For more options and detailed instructions, see the [test README](./test/README.md). -1. A full Ethereum network with: - - 2 x Execution Layer clients (e.g., reth) - - 2 x Consensus Layer clients (e.g., lodestar) - - Blockscout Explorer services for EL (if enabled with --blockscout) - - Dora Explorer service for CL - - Contracts deployed and configured for the DataHaven network. -2. A DataHaven solochain. -3. Snowbridge relayers for cross-chain communication. +### Run Tests -To launch the network, follow the instructions in the [test README](./test/README.md). - -## Docker - -This repo publishes images to [DockerHub](https://hub.docker.com/r/datahavenxyz/datahaven). - -> [!TIP] -> -> If you cannot see this repo you must be added to the permission list for the private repo. - -To aid with speed it employs the following: - -- [sccache](https://github.com/mozilla/sccache/tree/main): De-facto caching tool to speed up rust builds. -- [cargo chef](https://lpalmieri.com/posts/fast-rust-docker-builds/): A method of caching building the dependencies as a docker layer to cut down compilation times. -- [buildx cache mounts](https://docs.docker.com/build/cache/optimize/#use-cache-mounts): Using buildx's new feature to mount an externally restored cache into a container. -- [cache dance](https://github.com/reproducible-containers/buildkit-cache-dance): Weird workaround (endorsed by docker themselves) to inject caches into containers and return the result back to the CI. - -To run a docker image locally (`datahavenxyz/datahaven:local`), from the `/test` folder run: - -```sh -bun build:docker:operator +```bash +cd test +bun test:e2e # Run all integration tests +bun test:e2e:parallel # Run with limited concurrency ``` -## Working with IDEs +### Development Workflows -### VS Code (and its forks) +**Smart Contract Development**: +```bash +cd contracts +forge build # Compile contracts +forge test # Run contract tests +``` -IDE configurations are ignored from this repo's version control, to allow for personalisation. However, there are a few key configurations that we suggest for a better experience. Here are the key suggested configurations to add to your `.vscode/settings.json` file: +**Node Development**: +```bash +cd operator +cargo build --release --features fast-runtime +cargo test +./scripts/run-benchmarks.sh +``` -#### Rust +**After Making Changes**: +```bash +cd test +bun generate:wagmi # Regenerate contract bindings +bun generate:types # Regenerate runtime types +``` +## Key Features + +### EVM Compatibility +Full Ethereum Virtual Machine support via Frontier pallets: +- Deploy Solidity smart contracts +- Use existing Ethereum tooling (MetaMask, Hardhat, etc.) +- Compatible with ERC-20, ERC-721, and other standards + +### EigenLayer Integration +Validator security anchored to Ethereum: +- Operators register via `DataHavenServiceManager` contract +- Economic security through ETH restaking +- Slashing protection with veto period via `VetoableSlasher` +- Performance-based rewards through `RewardsRegistry` + +### Cross-chain Communication +Trustless bridging via Snowbridge: +- Native token transfers between Ethereum ↔ DataHaven +- Cross-chain message passing +- Finality proofs via BEEFY consensus +- Three specialized relayers (beacon, BEEFY, execution) + +### Dynamic Validator Set +Validator management synchronized with Ethereum: +- EigenLayer operator registry as source of truth +- On-chain validator set updates via External Validators pallet +- Automatic consensus participation changes +- Cross-chain coordination for validator lifecycle + +## Docker Images + +Production images published to [DockerHub](https://hub.docker.com/r/datahavenxyz/datahaven). + +**Build optimizations**: +- [sccache](https://github.com/mozilla/sccache) - Rust compilation caching +- [cargo-chef](https://lpalmieri.com/posts/fast-rust-docker-builds/) - Dependency layer caching +- [BuildKit cache mounts](https://docs.docker.com/build/cache/optimize/#use-cache-mounts) - External cache restoration + +**Build locally**: +```bash +cd test +bun build:docker:operator # Creates datahavenxyz/datahaven:local +``` + +## Development Environment + +### VS Code Configuration + +IDE configurations are excluded from version control for personalization, but these settings are recommended for optimal developer experience. Add to your `.vscode/settings.json`: + +**Rust Analyzer**: ```json { "rust-analyzer.linkedProjects": ["./operator/Cargo.toml"], @@ -72,17 +199,13 @@ IDE configurations are ignored from this repo's version control, to allow for pe } ``` -These settings optimise Rust Analyzer for the DataHaven codebase: - -- Marks the `operator/` folder as a linked project for analysis. The root of this repo is a workspace, and this is the rust project that should be analysed by `rust-analyzer`. -- Disables proc macros and build scripts to improve performance. Otherwise, Substrate's proc macros will make iterative checks from `rust-analyzer` unbearably slow. -- Sets a dedicated target directory for Rust Analyzer to avoid conflicts with other build targets like `release` builds. -- Disables WASM builds during analysis for faster feedback. - -#### Solidity - -For [Juan Blanco's Solidity Extension](https://marketplace.visualstudio.com/items?itemName=JuanBlanco.solidity), add the following to your `.vscode/settings.json` file: +Optimizations: +- Links `operator/` directory as the primary Rust project +- Disables proc macros and build scripts for faster analysis (Substrate macros are slow) +- Uses dedicated target directory to avoid conflicts +- Skips WASM builds during development +**Solidity** ([Juan Blanco's extension](https://marketplace.visualstudio.com/items?itemName=JuanBlanco.solidity)): ```json { "solidity.formatter": "forge", @@ -93,16 +216,9 @@ For [Juan Blanco's Solidity Extension](https://marketplace.visualstudio.com/item } ``` -These settings configure Solidity support: - -- Uses Forge as the formatter for consistency with the project's tooling. -- Sets a specific Solidity version for compilation. This one should match the version used in [foundry.toml](./contracts/foundry.toml). -- Sets the Solidity extension as the default formatter. - -#### Typescript - -This repo uses [Biome](https://github.com/biomejs/biome) for TypeScript linting and formatting. To make the extension work nicely with this repo, add the following to your `.vscode/settings.json` file: +Note: Solidity version must match [foundry.toml](./contracts/foundry.toml) +**TypeScript** ([Biome](https://github.com/biomejs/biome)): ```json { "biome.lsp.bin": "test/node_modules/.bin/biome", @@ -115,38 +231,59 @@ This repo uses [Biome](https://github.com/biomejs/biome) for TypeScript linting } ``` -- Sets the Biome binary to the one in the `test/` folder. -- Sets Biome as the default formatter for TypeScript. -- Sets Biome to always organise imports on save. +## CI/CD -## CI +### Local CI Testing -Using the [act](https://github.com/nektos/act) binary, you can run GitHub Actions locally. - -For example, to run the entire `e2e` workflow: +Run GitHub Actions workflows locally using [act](https://github.com/nektos/act): ```bash +# Run E2E workflow act -W .github/workflows/e2e.yml -s GITHUB_TOKEN="$(gh auth token)" + +# Run specific job +act -W .github/workflows/e2e.yml -j test-job-name ``` -Which results in: +### Automated Workflows -```bash -INFO[0000] Using docker host 'unix:///var/run/docker.sock', and daemon socket 'unix:///var/run/docker.sock' -INFO[0000] Start server on http://192.168.1.97:34567 -[E2E - Kurtosis Deploy and Verify/kurtosis] ⭐ Run Set up job -[E2E - Kurtosis Deploy and Verify/kurtosis] 🚀 Start image=catthehacker/ubuntu:rust-24.04 -[E2E - Kurtosis Deploy and Verify/kurtosis] 🐳 docker pull image=catthehacker/ubuntu:rust-24.04 platform= username= forcePull=true -[E2E - Kurtosis Deploy and Verify/kurtosis] using DockerAuthConfig authentication for docker pull -[E2E - Kurtosis Deploy and Verify/kurtosis] 🐳 docker create image=catthehacker/ubuntu:rust-24.04 platform= entrypoint=["tail" "-f" "/dev/null"] cmd=[] network="host" -[E2E - Kurtosis Deploy and Verify/kurtosis] 🐳 docker run image=catthehacker/ubuntu:rust-24.04 platform= entrypoint=["tail" "-f" "/dev/null"] cmd=[] network="host" -[E2E - Kurtosis Deploy and Verify/kurtosis] 🐳 docker exec cmd=[node --no-warnings -e console.log(process.execPath)] user= workdir= -[E2E - Kurtosis Deploy and Verify/kurtosis] ✅ Success - Set up job -[E2E - Kurtosis Deploy and Verify/kurtosis] ☁ git clone 'https://github.com/oven-sh/setup-bun' # ref=v2 -... -[E2E - Kurtosis Deploy and Verify/kurtosis] ✅ Success - Post Install Foundry [212.864597ms] -[E2E - Kurtosis Deploy and Verify/kurtosis] ⭐ Run Complete job -[E2E - Kurtosis Deploy and Verify/kurtosis] Cleaning up container for job kurtosis -[E2E - Kurtosis Deploy and Verify/kurtosis] ✅ Success - Complete job -[E2E - Kurtosis Deploy and Verify/kurtosis] 🏁 Job succeeded -``` +The repository includes GitHub Actions for: +- **E2E Testing**: Full integration tests on PR and main branch +- **Contract Testing**: Foundry test suites for smart contracts +- **Rust Testing**: Unit and integration tests for operator +- **Docker Builds**: Multi-platform image builds with caching +- **Release Automation**: Version tagging and changelog generation + +See `.github/workflows/` for workflow definitions. + +## Contributing + +### Development Cycle + +1. **Make Changes**: Edit contracts, runtime, or tests +2. **Run Tests**: Component-specific tests (`forge test`, `cargo test`) +3. **Regenerate Types**: Update bindings if contracts/runtime changed +4. **Integration Test**: Run E2E tests to verify cross-component behavior +5. **Code Quality**: Format and lint (`cargo fmt`, `forge fmt`, `bun fmt:fix`) + +### Common Pitfalls + +- **Type mismatches**: Regenerate with `bun generate:types` after runtime changes +- **Contract changes not reflected**: Run `bun generate:wagmi` after modifications +- **Kurtosis issues**: Ensure Docker is running and Kurtosis engine is started +- **Slow development**: Use `--features fast-runtime` for faster block times +- **Network launch hangs**: Check Blockscout - forge output can appear frozen + +See [CLAUDE.md](./CLAUDE.md) for detailed development guidance. + +## License + +GPL-3.0 - See LICENSE file for details + +## Links + +- [EigenLayer Documentation](https://docs.eigenlayer.xyz/) +- [Substrate Documentation](https://docs.substrate.io/) +- [Snowbridge Documentation](https://docs.snowbridge.network/) +- [Foundry Book](https://book.getfoundry.sh/) +- [Polkadot-API Documentation](https://papi.how/) diff --git a/contracts/README.md b/contracts/README.md index ca651b3c..d3d03221 100644 --- a/contracts/README.md +++ b/contracts/README.md @@ -4,18 +4,31 @@ This directory contains the smart contracts for the DataHaven Actively Validated ## Overview -DataHaven is an AVS that provides secure and decentralised data storage services. The contracts in this repository implement the Service Manager, middleware, and associated utilities required for the DataHaven protocol. +DataHaven is an EVM-compatible Substrate blockchain secured by EigenLayer. These contracts implement the AVS Service Manager, middleware, and associated utilities that integrate with EigenLayer's operator registration, slashing, and rewards infrastructure. ## Project Structure -- `src/`: Smart contract source code - - `DataHavenServiceManager.sol`: Main service manager contract - - `interfaces/`: Contract interfaces - - `libraries/`: Utility libraries - - `middleware/`: Middleware contracts (similar to EigenLayer's [middleware contracts](https://github.com/Layr-Labs/eigenlayer-middleware)) -- `script/`: Deployment scripts -- `test/`: Test cases -- `foundry.toml`: Foundry configuration +``` +contracts/ +├── src/ # Smart contract source code +│ ├── DataHavenServiceManager.sol # Core AVS service manager +│ ├── RewardsRegistry.sol # Validator performance & rewards tracking +│ ├── VetoableSlasher.sol # Slashing with veto period +│ ├── interfaces/ # Contract interfaces +│ ├── libraries/ # Utility libraries +│ └── middleware/ # EigenLayer middleware integration +├── script/ # Deployment & setup scripts +│ └── deploy/ # Environment-specific deployment +├── test/ # Foundry test suites +└── foundry.toml # Foundry configuration +``` + +### Key Contracts + +- **DataHavenServiceManager**: Manages operator lifecycle, registration, and deregistration with EigenLayer +- **RewardsRegistry**: Tracks validator performance metrics and handles reward distribution via Snowbridge +- **VetoableSlasher**: Implements slashing mechanism with dispute resolution veto period +- **Middleware**: Integration layer with EigenLayer's core contracts (based on [eigenlayer-middleware](https://github.com/Layr-Labs/eigenlayer-middleware)) ## Prerequisites @@ -52,16 +65,22 @@ For maximum verbosity including stack traces: forge test -vvvv ``` -Run specific test suites: +Run specific test contracts: ```bash -forge test --match-contract RewardsRegistry --no-match-contract SnowbridgeIntegration +forge test --match-contract RewardsRegistry ``` -Run specific tests: +Run specific test functions: ```bash -forge test --match-test test_getRewardstest_newRewardsMessage --no-match-test test_newRewardsMessage_OnlyRewardsAgent +forge test --match-test test_newRewardsMessage +``` + +Exclude specific tests: + +```bash +forge test --no-match-test test_newRewardsMessage_OnlyRewardsAgent ``` ## Deployment @@ -102,3 +121,25 @@ The deployment configuration can be modified in: - `script/deploy/Config.sol`: Environment-specific configuration - `script/deploy/DeployParams.s.sol`: Deployment parameters + +## Code Generation + +After making changes to contracts, regenerate TypeScript bindings for the test framework: + +```bash +cd ../test +bun generate:wagmi +``` + +This generates type-safe contract interfaces used by the E2E test suite. + +## Integration with DataHaven + +These contracts integrate with the DataHaven Substrate node through: + +1. **Operator Registration**: Validators register on-chain via `DataHavenServiceManager` +2. **Performance Tracking**: Node submits validator metrics to `RewardsRegistry` +3. **Cross-chain Rewards**: Rewards distributed from Ethereum to DataHaven via Snowbridge +4. **Slashing**: Misbehavior triggers slashing through `VetoableSlasher` with veto period + +For full network integration testing, see the [test directory](../test/README.md). diff --git a/deploy/README.md b/deploy/README.md index 93eea2f1..8be4be4f 100644 --- a/deploy/README.md +++ b/deploy/README.md @@ -1,6 +1,6 @@ # DataHaven Deployment -This directory contains all the necessary files and configurations for deploying DataHaven to various environments. +This directory contains Helm charts and configurations for deploying DataHaven nodes and relayers to Kubernetes clusters across various environments (local development, staging, production). ## Directory Structure @@ -50,16 +50,20 @@ Available environments: ## Environment Details ### Local -- Single replica -- Minimal resources (256Mi memory, 100m CPU) -- Local image tags -- Small persistence size +- **Purpose**: Local development and testing +- **Replicas**: 1 (bootnode + validator) +- **Resources**: Minimal (256Mi memory, 100m CPU) +- **Image**: Local Docker builds (`datahavenxyz/datahaven:local`) +- **Storage**: Small persistence (1-5Gi) +- **Network**: Single-node network with fast block times ### Stagenet -- 2 replicas -- Medium resources (512Mi memory, 200m CPU) -- Stagenet image tags -- 20Gi persistence size +- **Purpose**: Pre-production testing and staging +- **Replicas**: 2+ validators +- **Resources**: Medium (512Mi memory, 200m CPU) +- **Image**: Stagenet tags from DockerHub +- **Storage**: 20Gi+ persistent volumes +- **Network**: Multi-validator network simulating production ## Configuration Structure @@ -87,9 +91,44 @@ The deployment process: - **Bootnode**: Entry point for the network - **Validator**: Validates transactions and produces blocks -### Relays -- **Snowbridge Relays**: Handle cross-chain communication with Ethereum - - **Beacon Relay**: Relays Ethereum beacon chain data - - **BEEFY Relay**: Relays BEEFY consensus data for finality - - **Execution Relay**: Relays Ethereum execution layer data -- **Solochain Relayers**: Handle standalone chain operations and cross-chain communication +### Relays (Snowbridge) +- **Beacon Relay**: Relays Ethereum beacon chain finality to DataHaven +- **BEEFY Relay**: Relays DataHaven BEEFY finality proofs to Ethereum +- **Execution Relay**: Relays Ethereum execution layer messages to DataHaven +- **Solochain Relayers**: Relays DataHaven chain operations to the DataHaven AVS + +These relayers enable trustless bidirectional token and message passing between Ethereum and DataHaven. + +## Development Workflow + +1. **Local Testing**: + ```bash + cd test + bun cli launch # Starts local network without K8s + ``` + +2. **K8s Deployment**: + ```bash + cd test + bun cli deploy --e local + ``` + +3. **Building Local Images**: + ```bash + cd test + bun build:docker:operator # Builds datahavenxyz/datahaven:local + ``` + +4. **Updating Configurations**: + - Modify `environments//values.yaml` for environment-specific changes + - Modify chart templates in `charts/` for structural changes + - Redeploy with `bun cli deploy --e ` + +## Troubleshooting + +- **Pods not starting**: Check logs with `kubectl logs ` +- **Image pull failures**: Verify Docker registry access and image tags +- **Persistent volume issues**: Ensure storage class is available with `kubectl get sc` +- **Network connectivity**: Check service endpoints with `kubectl get svc` + +For more detailed deployment and testing workflows, see the [test directory](../test/README.md). diff --git a/operator/README.md b/operator/README.md index 086df50f..319b0309 100644 --- a/operator/README.md +++ b/operator/README.md @@ -1,6 +1,88 @@ -# DataHaven 🫎 +# DataHaven Operator (Substrate Node) 🫎 -Based on [polkadot-sdk-solochain-template](https://github.com/paritytech/polkadot-sdk-solochain-template) +The DataHaven operator is a Substrate-based blockchain node that serves as an EigenLayer AVS operator. It combines Substrate's modular framework with EVM compatibility (via Frontier) and cross-chain capabilities (via Snowbridge). + +## Overview + +Built on the [polkadot-sdk-solochain-template](https://github.com/paritytech/polkadot-sdk-solochain-template), this node implements: + +- **EVM Compatibility**: Full Ethereum compatibility via Frontier pallets +- **EigenLayer Integration**: Operator registration and management via AVS contracts +- **External Validators**: Dynamic validator set controlled by EigenLayer registry +- **Cross-chain Communication**: Token and message passing via Snowbridge +- **Rewards System**: Performance-based validator rewards from Ethereum + +## Project Structure + +``` +operator/ +├── node/ # Node implementation +│ ├── src/ +│ │ ├── chain_spec.rs # Chain specification & genesis config +│ │ ├── cli.rs # CLI interface +│ │ ├── command.rs # Command handlers +│ │ ├── rpc.rs # RPC configuration +│ │ └── service.rs # Node service setup +├── pallets/ # Custom pallets +│ ├── external-validators/ # EigenLayer validator set management +│ ├── native-transfer/ # Cross-chain token transfers +│ └── rewards/ # Validator rewards distribution +├── runtime/ # Runtime configurations +│ ├── mainnet/ # Mainnet runtime +│ ├── stagenet/ # Stagenet runtime +│ └── testnet/ # Testnet runtime (with fast-runtime feature) +└── scripts/ # Utility scripts + └── run-benchmarks.sh # Runtime benchmarking automation +``` + +## Prerequisites + +- [Rust](https://www.rust-lang.org/tools/install) (latest stable) +- [Substrate dependencies](https://docs.substrate.io/install/) +- [Zig](https://ziglang.org/) (macOS only, for cross-compilation) + +## Building + +### Development Build (Fast Runtime) + +For local development with faster block times: + +```bash +cargo build --release --features fast-runtime +``` + +This enables 3-second block times instead of the production 12-second blocks. + +### Production Build + +For production or stagenet deployments: + +```bash +cargo build --release +``` + +### Running Tests + +```bash +# Run all tests +cargo test + +# Run tests for specific pallet +cargo test -p pallet-external-validators + +# Run with output +cargo test -- --nocapture +``` + +### Code Quality + +```bash +# Format code +cargo fmt + +# Lint with clippy +cargo clippy --all-targets --all-features +``` ## Benchmarking @@ -8,13 +90,12 @@ DataHaven uses runtime benchmarking to generate accurate weight calculations for ### Requirements -Make sure you have the lastest rust version - -- `frame-omni-bencher` - Install with: `cargo install frame-omni-bencher --profile=production` (or `cargo install --git https://github.com/paritytech/polkadot-sdk frame-omni-bencher --profile=production --locked`) +- Latest Rust stable version +- `frame-omni-bencher`: Install with `cargo install frame-omni-bencher --profile=production` ### Running Benchmarks -Execute the benchmarking script from the project root: +Execute from the operator directory: ```bash # Benchmark all pallets for testnet runtime (default) @@ -28,23 +109,87 @@ Execute the benchmarking script from the project root: ``` The script will: -1. Automatically discover all available pallets -2. Build the runtime WASM with `runtime-benchmarks` feature +1. Discover all available pallets +2. Build runtime WASM with `runtime-benchmarks` feature 3. Generate weight files in `runtime/{runtime}/src/weights/` -4. Provide a summary of successful and failed benchmarks - -### Script Parameters +4. Provide summary of results +**Parameters**: - `runtime`: Runtime to benchmark (testnet, stagenet, mainnet). Default: testnet -- `steps`: Number of steps for benchmarking. Default: 50 +- `steps`: Number of steps. Default: 50 - `repeat`: Number of repetitions. Default: 20 -## Zombienet testing +## Zombienet Testing -First, install [zombienet](https://github.com/paritytech/zombienet). +[Zombienet](https://github.com/paritytech/zombienet) provides local multi-validator network testing. -To spawn a local solo chain with four validators and BABE finality, run: +### Setup + +1. Install Zombienet: + ```bash + # Download binary from releases + # Or install via npm + npm install -g @zombienet/cli + ``` + +2. Spawn local network with four validators: + ```bash + zombienet -p native spawn test/config/zombie-datahaven-local.toml + ``` + +This launches a local solochain with BABE consensus for testing validator coordination. + +## Docker Image + +Build local Docker image for testing: ```bash -zombienet -p native spawn test/config/zombie-datahaven-local.toml +cd ../test +bun build:docker:operator ``` + +This creates `datahavenxyz/datahaven:local` using optimized caching: +- [sccache](https://github.com/mozilla/sccache): Rust build caching +- [cargo-chef](https://lpalmieri.com/posts/fast-rust-docker-builds/): Dependency layer caching +- BuildKit cache mounts: External cache restoration + +## Type Generation + +After runtime changes, regenerate Polkadot-API TypeScript types: + +```bash +cd ../test +bun generate:types # Production runtime +bun generate:types:fast # Fast runtime (development) +``` + +## Integration Testing + +For full network integration tests with Ethereum, Snowbridge, and contracts: + +```bash +cd ../test +bun cli launch # Interactive launcher +bun test:e2e # Run E2E test suite +``` + +See the [test directory](../test/README.md) for comprehensive testing documentation. + +## Custom Pallets + +### External Validators +Manages the dynamic validator set based on EigenLayer operator registry. Syncs validator changes from Ethereum to the Substrate consensus layer. + +**Location**: `pallets/external-validators/` + +### Native Transfer +Handles cross-chain token transfers between Ethereum and DataHaven via Snowbridge messaging. + +**Location**: `pallets/native-transfer/` + +### Rewards +Distributes performance-based rewards to validators, processing reward messages from the Ethereum `RewardsRegistry` contract. + +**Location**: `pallets/rewards/` + +Each pallet includes its own tests and benchmarks. See pallet-specific README files for details. diff --git a/test/README.md b/test/README.md index ae428f28..a26493ab 100644 --- a/test/README.md +++ b/test/README.md @@ -1,6 +1,8 @@ # DataHaven E2E Testing -Quick start guide for running DataHaven end-to-end tests. For comprehensive documentation, see [E2E Testing Guide](./docs/E2E_TESTING_GUIDE.md). +End-to-end testing framework for DataHaven, providing automated network deployment, contract interaction, and cross-chain scenario testing. This directory contains all tools needed to launch a complete local DataHaven network with Ethereum, Snowbridge relayers, and run comprehensive integration tests. + +For comprehensive documentation, see [E2E Testing Guide](./docs/E2E_TESTING_GUIDE.md). ## Pre-requisites @@ -51,20 +53,68 @@ bun test:e2e:parallel # Run a specific test suite bun test suites/some-test.test.ts - ``` +## What Gets Launched + +The `bun cli launch` command deploys a complete local environment: + +1. **Ethereum Network** (via Kurtosis): + - 2x Execution Layer clients (reth) + - 2x Consensus Layer clients (lodestar) + - Blockscout Explorer (optional: `--blockscout`) + - Dora Consensus Explorer + +2. **DataHaven Network**: + - Single validator solochain + - EVM compatibility via Frontier + - Fast block times (3s with `--fast-runtime`) + +3. **Smart Contracts**: + - EigenLayer AVS contracts deployed to Ethereum + - Optional Blockscout verification (`--verified`) + +4. **Snowbridge Relayers**: + - Beacon relay (Ethereum → DataHaven) + - BEEFY relay (DataHaven → Ethereum) + - Execution relay (Ethereum → DataHaven) + - Solochain relay (DataHaven → Ethereum) + +5. **Network Configuration**: + - Validator registration and funding + - Parameter initialization + - Validator set updates + For more information on the E2E testing framework, see the [E2E Testing Framework Overview](./docs/E2E_FRAMEWORK_OVERVIEW.md). -## Other Common Commands +## Common Commands -| Command | Description | -| ------------------------- | ----------------------------------------------------------------------------------------------------------- | -| `bun cli stop` | Stop all local DataHaven networks (interactive, will ask for confirmation on each component of the network) | -| `bun cli deploy` | Deploy the DataHaven network to a remote Kubernetes cluster | -| `bun generate:wagmi` | Generate contract TypeScript bindings for the contracts in the `contracts` directory | -| `bun generate:types` | Generate Polkadot API types | -| `bun generate:types:fast` | Generate Polkadot API types with the `--fast-runtime` feature enabled | +| Command | Description | +| ------------------------- | -------------------------------------------------------------------------------------------------- | +| **Network Management** | | +| `bun cli` | Interactive CLI menu for all operations | +| `bun cli launch` | Launch full local network (interactive options) | +| `bun start:e2e:local` | Launch local network (non-interactive) | +| `bun start:e2e:verified` | Launch with Blockscout and contract verification | +| `bun start:e2e:ci` | CI-optimized network launch | +| `bun cli stop` | Stop all services (interactive) | +| `bun stop:dh` | Stop DataHaven only | +| `bun stop:sb` | Stop Snowbridge relayers only | +| `bun stop:eth` | Stop Ethereum network only | +| **Testing** | | +| `bun test:e2e` | Run all E2E test suites | +| `bun test:e2e:parallel` | Run tests with limited concurrency | +| `bun test ` | Run specific test file | +| **Code Generation** | | +| `bun generate:wagmi` | Generate TypeScript contract bindings (after contract changes) | +| `bun generate:types` | Generate Polkadot-API types from runtime | +| `bun generate:types:fast` | Generate types with fast-runtime feature | +| **Code Quality** | | +| `bun fmt:fix` | Fix TypeScript formatting with Biome | +| `bun typecheck` | TypeScript type checking | +| **Deployment** | | +| `bun cli deploy` | Deploy to Kubernetes cluster (interactive) | +| `bun build:docker:operator` | Build local Docker image (`datahavenxyz/datahaven:local`) | ## Local Network Deployment @@ -178,8 +228,56 @@ This script will: > > The script uses the `--release` flag by default, meaning it uses the WASM binary from `./operator/target/release`. If you need to use a different build target, you may need to adjust the script or run the steps manually. +## Project Structure + +``` +test/ +├── suites/ # E2E test suites +│ ├── contracts.test.ts # Contract deployment & configuration +│ ├── cross-chain.test.ts # Cross-chain message passing +│ ├── datahaven-substrate.test.ts # Block production & finality +│ ├── ethereum-basic.test.ts # Ethereum network validation +│ ├── native-token-transfer.test.ts # Cross-chain token transfers +│ ├── rewards-message.test.ts # Validator rewards distribution +│ └── validator-set-update.test.ts # Dynamic validator set updates +├── framework/ # Test utilities & helpers +│ ├── connectors.ts # Network connectors +│ ├── manager.ts # Test environment manager +│ ├── suite.ts # Test suite utilities +│ └── index.ts # Framework exports +├── launcher/ # Network deployment tools +│ ├── kurtosis/ # Ethereum network launcher +│ ├── snowbridge/ # Relayer management +│ └── datahaven/ # DataHaven node management +├── generated/ # Generated types +│ ├── wagmi/ # Contract bindings +│ └── polkadot-api/ # Runtime types +└── docs/ # Testing documentation + ├── E2E_TESTING_GUIDE.md + └── E2E_FRAMEWORK_OVERVIEW.md +``` + +## Test Suites + +- **contracts.test.ts**: Contract deployment and configuration validation +- **cross-chain.test.ts**: Cross-chain message passing between Ethereum and DataHaven +- **datahaven-substrate.test.ts**: Block production, finalization, and consensus +- **ethereum-basic.test.ts**: Ethereum network health and basic functionality +- **native-token-transfer.test.ts**: Cross-chain token transfers via Snowbridge +- **rewards-message.test.ts**: Validator reward distribution from Ethereum to DataHaven +- **validator-set-update.test.ts**: Dynamic validator registration/deregistration via EigenLayer + +Run individual suites: +```bash +bun test suites/rewards-message.test.ts +bun test suites/native-token-transfer.test.ts +bun test suites/validator-set-update.test.ts +``` + ## Further Information -- [Kurtosis](https://docs.kurtosis.com/): Used for launching a full Ethereum network -- [Zombienet](https://paritytech.github.io/zombienet/): Used for launching a Polkadot-SDK based network -- [Bun](https://bun.sh/): TypeScript runtime and ecosystem tooling +- [Kurtosis](https://docs.kurtosis.com/): Ethereum network orchestration +- [Zombienet](https://paritytech.github.io/zombienet/): Polkadot-SDK network testing +- [Bun](https://bun.sh/): TypeScript runtime and tooling +- [Foundry](https://book.getfoundry.sh/): Solidity development framework +- [Polkadot-API](https://papi.how/): Type-safe Substrate interactions diff --git a/tools/README.md b/tools/README.md new file mode 100644 index 00000000..9735b232 --- /dev/null +++ b/tools/README.md @@ -0,0 +1,86 @@ +# DataHaven Development Tools + +Utility scripts for GitHub automation and release management. + +## Overview + +This directory contains Node.js/TypeScript tools for automating release workflows and generating standardized release notes for both runtime and client releases. + +## Structure + +``` +tools/ +├── github/ # GitHub automation utilities +│ ├── github-utils.ts # Common GitHub API utilities +│ ├── generate-release-body.ts # Client release notes generator +│ └── generate-runtime-body.ts # Runtime release notes generator +├── package.json # Dependencies and scripts +└── tsconfig.json # TypeScript configuration +``` + +## Prerequisites + +- Node.js (version specified in `.nvmrc`) +- npm (for dependency management) + +## Setup + +```bash +cd tools +npm install +``` + +## Available Scripts + +### Generate Client Release Notes + +Creates formatted release notes for client (node binary) releases: + +```bash +npm run print-client-release-issue +``` + +This script: +- Generates changelog from Git history +- Formats release notes for GitHub +- Includes version information and breaking changes +- Outputs markdown suitable for GitHub releases + +### Generate Runtime Release Notes + +Creates formatted release notes for runtime (WASM) releases: + +```bash +npm run print-runtime-release-issue +``` + +This script: +- Generates runtime-specific changelog +- Highlights runtime version bumps +- Includes migration information +- Formats for GitHub release pages + +## Usage in CI/CD + +These tools are typically invoked by GitHub Actions workflows during the release process. They can also be run manually for testing or preparing draft releases. + +## GitHub Integration + +The tools use the [Octokit](https://github.com/octokit/octokit.js) library to interact with the GitHub API. Authentication is typically handled via `GITHUB_TOKEN` environment variable in CI/CD contexts. + +## Development + +The tools are written in TypeScript and use: +- `@polkadot/api`: For parsing runtime metadata +- `octokit`: For GitHub API interactions +- `yargs`: For CLI argument parsing +- `ts-node`: For direct TypeScript execution + +## Customization + +To modify release note formatting or add new automation: + +1. Edit the respective generator in `github/` +2. Update `package.json` scripts if adding new commands +3. Test locally with `npm run ` +4. Update CI workflows in `.github/workflows/` if needed