Elgato_dark/datahaven
1
0
Fork 0
mirror of https://github.com/datahaven-xyz/datahaven synced 2026-05-24 01:38:32 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

chore: ♻️ Update README files (#206)

Browse source 
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Ahmad Kaouk <56095276+ahmadkaouk@users.noreply.github.com>
This commit is contained in:
Steve Degosserie 2025-10-06 18:12:55 +02:00 committed by GitHub
parent 0110a94978
commit b4f697f954
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
6 changed files with 693 additions and 147 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

317
README.md
View file

@ -1,63 +1,190 @@
# DataHaven 🫎
An EVM compatible Substrate chain, powered by StorageHub and secured by EigenLayer.
An EVM-compatible Substrate blockchain secured by EigenLayer, bridging Ethereum and Substrate ecosystems through trustless cross-chain communication.
## Repo Structure
## Overview
DataHaven is an EigenLayer Actively Validated Service (AVS) that combines:
- **EVM Compatibility**: Full Ethereum support via Frontier pallets for smart contracts and dApps
- **EigenLayer Security**: Validator set secured by Ethereum's economic security through restaking
- **Cross-chain Bridge**: Seamless asset and message transfers with Ethereum via Snowbridge
- **Dynamic Validators**: Operator registry managed on-chain through EigenLayer contracts
- **Performance Rewards**: Validator incentives distributed cross-chain from Ethereum
## Architecture
DataHaven bridges two major blockchain ecosystems:
```
┌───────────────────────────────────────────────────────────────┐
│ Ethereum (L1) │
│ ┌────────────────────────────────────────────────────────┐ │
│ │ EigenLayer AVS Contracts │ │
│ │ • DataHavenServiceManager (operator lifecycle) │ │
│ │ • RewardsRegistry (performance tracking) │ │
│ │ • VetoableSlasher (misbehavior penalties) │ │
│ └────────────────────────────────────────────────────────┘ │
│ ↕ │
│ Snowbridge Protocol │
└───────────────────────────────────────────────────────────────┘
↕
┌───────────────────────────────────────────────────────────────┐
│ DataHaven (Substrate) │
│ ┌────────────────────────────────────────────────────────┐ │
│ │ Custom Pallets │ │
│ │ • External Validators (sync validator set) │ │
│ │ • Native Transfer (cross-chain tokens) │ │
│ │ • Rewards (distribute validator rewards) │ │
│ │ • Frontier (EVM compatibility) │ │
│ └────────────────────────────────────────────────────────┘ │
└───────────────────────────────────────────────────────────────┘
```
## Repository Structure
```
datahaven/
├── contracts/ # EigenLayer AVS smart contracts
│ ├── src/ # Service Manager, Rewards Registry, Slasher
│ ├── script/ # Deployment scripts
│ └── test/ # Foundry test suites
├── operator/ # Substrate-based DataHaven node
│ ├── node/ # Node implementation & chain spec
│ ├── pallets/ # Custom pallets (validators, rewards, transfers)
│ └── runtime/ # Runtime configurations (mainnet/stagenet/testnet)
├── test/ # E2E testing framework
│ ├── suites/ # Integration test scenarios
│ ├── framework/ # Test utilities and helpers
│ └── launcher/ # Network deployment automation
├── deploy/ # Kubernetes deployment charts
│ ├── charts/ # Helm charts for nodes and relayers
│ └── environments/ # Environment-specific configurations
├── tools/ # GitHub automation and release scripts
└── .github/ # CI/CD workflows
```
Each directory contains its own README with detailed information. See:
- [contracts/README.md](contracts/README.md) - Smart contract development
- [operator/README.md](operator/README.md) - Node building and runtime development
- [test/README.md](test/README.md) - E2E testing and network deployment
- [deploy/README.md](deploy/README.md) - Kubernetes deployment
- [tools/README.md](tools/README.md) - Development tools
## Quick Start
### Prerequisites
- [Kurtosis](https://docs.kurtosis.com/install) - Network orchestration
- [Bun](https://bun.sh/) v1.2+ - TypeScript runtime
- [Docker](https://www.docker.com/) - Container management
- [Foundry](https://getfoundry.sh/) - Solidity toolkit
- [Rust](https://www.rust-lang.org/tools/install) - For building the operator
- [Helm](https://helm.sh/) - Kubernetes deployments (optional)
- [Zig](https://ziglang.org/) - For macOS cross-compilation (macOS only)
### Launch Local Network
The fastest way to get started is with the interactive CLI:
```bash
datahaven/
├── .github/ # GitHub Actions workflows.
├── contracts/ # Implementation of the DataHaven AVS (Autonomous Verifiable Service) smart contracts to interact with EigenLayer.
├── operator/ # DataHaven node based on Substrate. The "Operator" in EigenLayer terms.
├── test/ # Integration tests for the AVS and Operator.
├── resources/ # Miscellaneous resources for the DataHaven project.
└── README.md
cd test
bun i # Install dependencies
bun cli launch # Interactive launcher with prompts
```
## E2E CLI
This deploys a complete environment including:
- **Ethereum network**: 2x EL clients (reth), 2x CL clients (lodestar)
- **Block explorers**: Blockscout (optional), Dora consensus explorer
- **DataHaven node**: Single validator with fast block times
- **AVS contracts**: Deployed and configured on Ethereum
- **Snowbridge relayers**: Bidirectional message passing
This repo comes with a CLI for launching a local DataHaven network, packaged with:
For more options and detailed instructions, see the [test README](./test/README.md).
1. A full Ethereum network with:
- 2 x Execution Layer clients (e.g., reth)
- 2 x Consensus Layer clients (e.g., lodestar)
- Blockscout Explorer services for EL (if enabled with --blockscout)
- Dora Explorer service for CL
- Contracts deployed and configured for the DataHaven network.
2. A DataHaven solochain.
3. Snowbridge relayers for cross-chain communication.
### Run Tests
To launch the network, follow the instructions in the [test README](./test/README.md).
## Docker
This repo publishes images to [DockerHub](https://hub.docker.com/r/datahavenxyz/datahaven).
> [!TIP]
>
> If you cannot see this repo you must be added to the permission list for the private repo.
To aid with speed it employs the following:
- [sccache](https://github.com/mozilla/sccache/tree/main): De-facto caching tool to speed up rust builds.
- [cargo chef](https://lpalmieri.com/posts/fast-rust-docker-builds/): A method of caching building the dependencies as a docker layer to cut down compilation times.
- [buildx cache mounts](https://docs.docker.com/build/cache/optimize/#use-cache-mounts): Using buildx's new feature to mount an externally restored cache into a container.
- [cache dance](https://github.com/reproducible-containers/buildkit-cache-dance): Weird workaround (endorsed by docker themselves) to inject caches into containers and return the result back to the CI.
To run a docker image locally (`datahavenxyz/datahaven:local`), from the `/test` folder run:
```sh
bun build:docker:operator
```bash
cd test
bun test:e2e # Run all integration tests
bun test:e2e:parallel # Run with limited concurrency
```
## Working with IDEs
### Development Workflows
### VS Code (and its forks)
**Smart Contract Development**:
```bash
cd contracts
forge build # Compile contracts
forge test # Run contract tests
```
IDE configurations are ignored from this repo's version control, to allow for personalisation. However, there are a few key configurations that we suggest for a better experience. Here are the key suggested configurations to add to your `.vscode/settings.json` file:
**Node Development**:
```bash
cd operator
cargo build --release --features fast-runtime
cargo test
./scripts/run-benchmarks.sh
```
#### Rust
**After Making Changes**:
```bash
cd test
bun generate:wagmi # Regenerate contract bindings
bun generate:types # Regenerate runtime types
```
## Key Features
### EVM Compatibility
Full Ethereum Virtual Machine support via Frontier pallets:
- Deploy Solidity smart contracts
- Use existing Ethereum tooling (MetaMask, Hardhat, etc.)
- Compatible with ERC-20, ERC-721, and other standards
### EigenLayer Integration
Validator security anchored to Ethereum:
- Operators register via `DataHavenServiceManager` contract
- Economic security through ETH restaking
- Slashing protection with veto period via `VetoableSlasher`
- Performance-based rewards through `RewardsRegistry`
### Cross-chain Communication
Trustless bridging via Snowbridge:
- Native token transfers between Ethereum ↔ DataHaven
- Cross-chain message passing
- Finality proofs via BEEFY consensus
- Three specialized relayers (beacon, BEEFY, execution)
### Dynamic Validator Set
Validator management synchronized with Ethereum:
- EigenLayer operator registry as source of truth
- On-chain validator set updates via External Validators pallet
- Automatic consensus participation changes
- Cross-chain coordination for validator lifecycle
## Docker Images
Production images published to [DockerHub](https://hub.docker.com/r/datahavenxyz/datahaven).
**Build optimizations**:
- [sccache](https://github.com/mozilla/sccache) - Rust compilation caching
- [cargo-chef](https://lpalmieri.com/posts/fast-rust-docker-builds/) - Dependency layer caching
- [BuildKit cache mounts](https://docs.docker.com/build/cache/optimize/#use-cache-mounts) - External cache restoration
**Build locally**:
```bash
cd test
bun build:docker:operator # Creates datahavenxyz/datahaven:local
```
## Development Environment
### VS Code Configuration
IDE configurations are excluded from version control for personalization, but these settings are recommended for optimal developer experience. Add to your `.vscode/settings.json`:
**Rust Analyzer**:
```json
{
"rust-analyzer.linkedProjects": ["./operator/Cargo.toml"],
@ -72,17 +199,13 @@ IDE configurations are ignored from this repo's version control, to allow for pe
}
```
These settings optimise Rust Analyzer for the DataHaven codebase:
- Marks the `operator/` folder as a linked project for analysis. The root of this repo is a workspace, and this is the rust project that should be analysed by `rust-analyzer`.
- Disables proc macros and build scripts to improve performance. Otherwise, Substrate's proc macros will make iterative checks from `rust-analyzer` unbearably slow.
- Sets a dedicated target directory for Rust Analyzer to avoid conflicts with other build targets like `release` builds.
- Disables WASM builds during analysis for faster feedback.
#### Solidity
For [Juan Blanco's Solidity Extension](https://marketplace.visualstudio.com/items?itemName=JuanBlanco.solidity), add the following to your `.vscode/settings.json` file:
Optimizations:
- Links `operator/` directory as the primary Rust project
- Disables proc macros and build scripts for faster analysis (Substrate macros are slow)
- Uses dedicated target directory to avoid conflicts
- Skips WASM builds during development
**Solidity** ([Juan Blanco's extension](https://marketplace.visualstudio.com/items?itemName=JuanBlanco.solidity)):
```json
{
"solidity.formatter": "forge",
@ -93,16 +216,9 @@ For [Juan Blanco's Solidity Extension](https://marketplace.visualstudio.com/item
}
```
These settings configure Solidity support:
- Uses Forge as the formatter for consistency with the project's tooling.
- Sets a specific Solidity version for compilation. This one should match the version used in [foundry.toml](./contracts/foundry.toml).
- Sets the Solidity extension as the default formatter.
#### Typescript
This repo uses [Biome](https://github.com/biomejs/biome) for TypeScript linting and formatting. To make the extension work nicely with this repo, add the following to your `.vscode/settings.json` file:
Note: Solidity version must match [foundry.toml](./contracts/foundry.toml)
**TypeScript** ([Biome](https://github.com/biomejs/biome)):
```json
{
"biome.lsp.bin": "test/node_modules/.bin/biome",
@ -115,38 +231,59 @@ This repo uses [Biome](https://github.com/biomejs/biome) for TypeScript linting
}
```
- Sets the Biome binary to the one in the `test/` folder.
- Sets Biome as the default formatter for TypeScript.
- Sets Biome to always organise imports on save.
## CI/CD
## CI
### Local CI Testing
Using the [act](https://github.com/nektos/act) binary, you can run GitHub Actions locally.
For example, to run the entire `e2e` workflow:
Run GitHub Actions workflows locally using [act](https://github.com/nektos/act):
```bash
# Run E2E workflow
act -W .github/workflows/e2e.yml -s GITHUB_TOKEN="$(gh auth token)"
# Run specific job
act -W .github/workflows/e2e.yml -j test-job-name
```
Which results in:
### Automated Workflows
```bash
INFO[0000] Using docker host 'unix:///var/run/docker.sock', and daemon socket 'unix:///var/run/docker.sock'
INFO[0000] Start server on http://192.168.1.97:34567
[E2E - Kurtosis Deploy and Verify/kurtosis] ⭐ Run Set up job
[E2E - Kurtosis Deploy and Verify/kurtosis] 🚀 Start image=catthehacker/ubuntu:rust-24.04
[E2E - Kurtosis Deploy and Verify/kurtosis] 🐳 docker pull image=catthehacker/ubuntu:rust-24.04 platform= username= forcePull=true
[E2E - Kurtosis Deploy and Verify/kurtosis] using DockerAuthConfig authentication for docker pull
[E2E - Kurtosis Deploy and Verify/kurtosis] 🐳 docker create image=catthehacker/ubuntu:rust-24.04 platform= entrypoint=["tail" "-f" "/dev/null"] cmd=[] network="host"
[E2E - Kurtosis Deploy and Verify/kurtosis] 🐳 docker run image=catthehacker/ubuntu:rust-24.04 platform= entrypoint=["tail" "-f" "/dev/null"] cmd=[] network="host"
[E2E - Kurtosis Deploy and Verify/kurtosis] 🐳 docker exec cmd=[node --no-warnings -e console.log(process.execPath)] user= workdir=
[E2E - Kurtosis Deploy and Verify/kurtosis] ✅ Success - Set up job
[E2E - Kurtosis Deploy and Verify/kurtosis] ☁ git clone 'https://github.com/oven-sh/setup-bun' # ref=v2
...
[E2E - Kurtosis Deploy and Verify/kurtosis] ✅ Success - Post Install Foundry [212.864597ms]
[E2E - Kurtosis Deploy and Verify/kurtosis] ⭐ Run Complete job
[E2E - Kurtosis Deploy and Verify/kurtosis] Cleaning up container for job kurtosis
[E2E - Kurtosis Deploy and Verify/kurtosis] ✅ Success - Complete job
[E2E - Kurtosis Deploy and Verify/kurtosis] 🏁 Job succeeded
```
The repository includes GitHub Actions for:
- **E2E Testing**: Full integration tests on PR and main branch
- **Contract Testing**: Foundry test suites for smart contracts
- **Rust Testing**: Unit and integration tests for operator
- **Docker Builds**: Multi-platform image builds with caching
- **Release Automation**: Version tagging and changelog generation
See `.github/workflows/` for workflow definitions.
## Contributing
### Development Cycle
1. **Make Changes**: Edit contracts, runtime, or tests
2. **Run Tests**: Component-specific tests (`forge test`, `cargo test`)
3. **Regenerate Types**: Update bindings if contracts/runtime changed
4. **Integration Test**: Run E2E tests to verify cross-component behavior
5. **Code Quality**: Format and lint (`cargo fmt`, `forge fmt`, `bun fmt:fix`)
### Common Pitfalls
- **Type mismatches**: Regenerate with `bun generate:types` after runtime changes
- **Contract changes not reflected**: Run `bun generate:wagmi` after modifications
- **Kurtosis issues**: Ensure Docker is running and Kurtosis engine is started
- **Slow development**: Use `--features fast-runtime` for faster block times
- **Network launch hangs**: Check Blockscout - forge output can appear frozen
See [CLAUDE.md](./CLAUDE.md) for detailed development guidance.
## License
GPL-3.0 - See LICENSE file for details
## Links
- [EigenLayer Documentation](https://docs.eigenlayer.xyz/)
- [Substrate Documentation](https://docs.substrate.io/)
- [Snowbridge Documentation](https://docs.snowbridge.network/)
- [Foundry Book](https://book.getfoundry.sh/)
- [Polkadot-API Documentation](https://papi.how/)

67
contracts/README.md
View file

@ -4,18 +4,31 @@ This directory contains the smart contracts for the DataHaven Actively Validated
## Overview
DataHaven is an AVS that provides secure and decentralised data storage services. The contracts in this repository implement the Service Manager, middleware, and associated utilities required for the DataHaven protocol.
DataHaven is an EVM-compatible Substrate blockchain secured by EigenLayer. These contracts implement the AVS Service Manager, middleware, and associated utilities that integrate with EigenLayer's operator registration, slashing, and rewards infrastructure.
## Project Structure
- `src/`: Smart contract source code
- `DataHavenServiceManager.sol`: Main service manager contract
- `interfaces/`: Contract interfaces
- `libraries/`: Utility libraries
- `middleware/`: Middleware contracts (similar to EigenLayer's [middleware contracts](https://github.com/Layr-Labs/eigenlayer-middleware))
- `script/`: Deployment scripts
- `test/`: Test cases
- `foundry.toml`: Foundry configuration
```
contracts/
├── src/ # Smart contract source code
│ ├── DataHavenServiceManager.sol # Core AVS service manager
│ ├── RewardsRegistry.sol # Validator performance & rewards tracking
│ ├── VetoableSlasher.sol # Slashing with veto period
│ ├── interfaces/ # Contract interfaces
│ ├── libraries/ # Utility libraries
│ └── middleware/ # EigenLayer middleware integration
├── script/ # Deployment & setup scripts
│ └── deploy/ # Environment-specific deployment
├── test/ # Foundry test suites
└── foundry.toml # Foundry configuration
```
### Key Contracts
- **DataHavenServiceManager**: Manages operator lifecycle, registration, and deregistration with EigenLayer
- **RewardsRegistry**: Tracks validator performance metrics and handles reward distribution via Snowbridge
- **VetoableSlasher**: Implements slashing mechanism with dispute resolution veto period
- **Middleware**: Integration layer with EigenLayer's core contracts (based on [eigenlayer-middleware](https://github.com/Layr-Labs/eigenlayer-middleware))
## Prerequisites
@ -52,16 +65,22 @@ For maximum verbosity including stack traces:
forge test -vvvv
```
Run specific test suites:
Run specific test contracts:
```bash
forge test --match-contract RewardsRegistry --no-match-contract SnowbridgeIntegration
forge test --match-contract RewardsRegistry
```
Run specific tests:
Run specific test functions:
```bash
forge test --match-test test_getRewardstest_newRewardsMessage --no-match-test test_newRewardsMessage_OnlyRewardsAgent
forge test --match-test test_newRewardsMessage
```
Exclude specific tests:
```bash
forge test --no-match-test test_newRewardsMessage_OnlyRewardsAgent
```
## Deployment
@ -102,3 +121,25 @@ The deployment configuration can be modified in:
- `script/deploy/Config.sol`: Environment-specific configuration
- `script/deploy/DeployParams.s.sol`: Deployment parameters
## Code Generation
After making changes to contracts, regenerate TypeScript bindings for the test framework:
```bash
cd ../test
bun generate:wagmi
```
This generates type-safe contract interfaces used by the E2E test suite.
## Integration with DataHaven
These contracts integrate with the DataHaven Substrate node through:
1. **Operator Registration**: Validators register on-chain via `DataHavenServiceManager`
2. **Performance Tracking**: Node submits validator metrics to `RewardsRegistry`
3. **Cross-chain Rewards**: Rewards distributed from Ethereum to DataHaven via Snowbridge
4. **Slashing**: Misbehavior triggers slashing through `VetoableSlasher` with veto period
For full network integration testing, see the [test directory](../test/README.md).

69
deploy/README.md
View file

@ -1,6 +1,6 @@
# DataHaven Deployment
This directory contains all the necessary files and configurations for deploying DataHaven to various environments.
This directory contains Helm charts and configurations for deploying DataHaven nodes and relayers to Kubernetes clusters across various environments (local development, staging, production).
## Directory Structure
@ -50,16 +50,20 @@ Available environments:
## Environment Details
### Local
- Single replica
- Minimal resources (256Mi memory, 100m CPU)
- Local image tags
- Small persistence size
- **Purpose**: Local development and testing
- **Replicas**: 1 (bootnode + validator)
- **Resources**: Minimal (256Mi memory, 100m CPU)
- **Image**: Local Docker builds (`datahavenxyz/datahaven:local`)
- **Storage**: Small persistence (1-5Gi)
- **Network**: Single-node network with fast block times
### Stagenet
- 2 replicas
- Medium resources (512Mi memory, 200m CPU)
- Stagenet image tags
- 20Gi persistence size
- **Purpose**: Pre-production testing and staging
- **Replicas**: 2+ validators
- **Resources**: Medium (512Mi memory, 200m CPU)
- **Image**: Stagenet tags from DockerHub
- **Storage**: 20Gi+ persistent volumes
- **Network**: Multi-validator network simulating production
## Configuration Structure
@ -87,9 +91,44 @@ The deployment process:
- **Bootnode**: Entry point for the network
- **Validator**: Validates transactions and produces blocks
### Relays
- **Snowbridge Relays**: Handle cross-chain communication with Ethereum
- **Beacon Relay**: Relays Ethereum beacon chain data
- **BEEFY Relay**: Relays BEEFY consensus data for finality
- **Execution Relay**: Relays Ethereum execution layer data
- **Solochain Relayers**: Handle standalone chain operations and cross-chain communication
### Relays (Snowbridge)
- **Beacon Relay**: Relays Ethereum beacon chain finality to DataHaven
- **BEEFY Relay**: Relays DataHaven BEEFY finality proofs to Ethereum
- **Execution Relay**: Relays Ethereum execution layer messages to DataHaven
- **Solochain Relayers**: Relays DataHaven chain operations to the DataHaven AVS
These relayers enable trustless bidirectional token and message passing between Ethereum and DataHaven.
## Development Workflow
1. **Local Testing**:
```bash
cd test
bun cli launch # Starts local network without K8s
```
2. **K8s Deployment**:
```bash
cd test
bun cli deploy --e local
```
3. **Building Local Images**:
```bash
cd test
bun build:docker:operator # Builds datahavenxyz/datahaven:local
```
4. **Updating Configurations**:
- Modify `environments/<env>/values.yaml` for environment-specific changes
- Modify chart templates in `charts/` for structural changes
- Redeploy with `bun cli deploy --e <env>`
## Troubleshooting
- **Pods not starting**: Check logs with `kubectl logs <pod-name>`
- **Image pull failures**: Verify Docker registry access and image tags
- **Persistent volume issues**: Ensure storage class is available with `kubectl get sc`
- **Network connectivity**: Check service endpoints with `kubectl get svc`
For more detailed deployment and testing workflows, see the [test directory](../test/README.md).

177
operator/README.md
View file

@ -1,6 +1,88 @@
# DataHaven 🫎
# DataHaven Operator (Substrate Node) 🫎
Based on [polkadot-sdk-solochain-template](https://github.com/paritytech/polkadot-sdk-solochain-template)
The DataHaven operator is a Substrate-based blockchain node that serves as an EigenLayer AVS operator. It combines Substrate's modular framework with EVM compatibility (via Frontier) and cross-chain capabilities (via Snowbridge).
## Overview
Built on the [polkadot-sdk-solochain-template](https://github.com/paritytech/polkadot-sdk-solochain-template), this node implements:
- **EVM Compatibility**: Full Ethereum compatibility via Frontier pallets
- **EigenLayer Integration**: Operator registration and management via AVS contracts
- **External Validators**: Dynamic validator set controlled by EigenLayer registry
- **Cross-chain Communication**: Token and message passing via Snowbridge
- **Rewards System**: Performance-based validator rewards from Ethereum
## Project Structure
```
operator/
├── node/ # Node implementation
│ ├── src/
│ │ ├── chain_spec.rs # Chain specification & genesis config
│ │ ├── cli.rs # CLI interface
│ │ ├── command.rs # Command handlers
│ │ ├── rpc.rs # RPC configuration
│ │ └── service.rs # Node service setup
├── pallets/ # Custom pallets
│ ├── external-validators/ # EigenLayer validator set management
│ ├── native-transfer/ # Cross-chain token transfers
│ └── rewards/ # Validator rewards distribution
├── runtime/ # Runtime configurations
│ ├── mainnet/ # Mainnet runtime
│ ├── stagenet/ # Stagenet runtime
│ └── testnet/ # Testnet runtime (with fast-runtime feature)
└── scripts/ # Utility scripts
└── run-benchmarks.sh # Runtime benchmarking automation
```
## Prerequisites
- [Rust](https://www.rust-lang.org/tools/install) (latest stable)
- [Substrate dependencies](https://docs.substrate.io/install/)
- [Zig](https://ziglang.org/) (macOS only, for cross-compilation)
## Building
### Development Build (Fast Runtime)
For local development with faster block times:
```bash
cargo build --release --features fast-runtime
```
This enables 3-second block times instead of the production 12-second blocks.
### Production Build
For production or stagenet deployments:
```bash
cargo build --release
```
### Running Tests
```bash
# Run all tests
cargo test
# Run tests for specific pallet
cargo test -p pallet-external-validators
# Run with output
cargo test -- --nocapture
```
### Code Quality
```bash
# Format code
cargo fmt
# Lint with clippy
cargo clippy --all-targets --all-features
```
## Benchmarking
@ -8,13 +90,12 @@ DataHaven uses runtime benchmarking to generate accurate weight calculations for
### Requirements
Make sure you have the lastest rust version
- `frame-omni-bencher` - Install with: `cargo install frame-omni-bencher --profile=production` (or `cargo install --git https://github.com/paritytech/polkadot-sdk frame-omni-bencher --profile=production --locked`)
- Latest Rust stable version
- `frame-omni-bencher`: Install with `cargo install frame-omni-bencher --profile=production`
### Running Benchmarks
Execute the benchmarking script from the project root:
Execute from the operator directory:
```bash
# Benchmark all pallets for testnet runtime (default)
@ -28,23 +109,87 @@ Execute the benchmarking script from the project root:
```
The script will:
1. Automatically discover all available pallets
2. Build the runtime WASM with `runtime-benchmarks` feature
1. Discover all available pallets
2. Build runtime WASM with `runtime-benchmarks` feature
3. Generate weight files in `runtime/{runtime}/src/weights/`
4. Provide a summary of successful and failed benchmarks
### Script Parameters
4. Provide summary of results
**Parameters**:
- `runtime`: Runtime to benchmark (testnet, stagenet, mainnet). Default: testnet
- `steps`: Number of steps for benchmarking. Default: 50
- `steps`: Number of steps. Default: 50
- `repeat`: Number of repetitions. Default: 20
## Zombienet testing
## Zombienet Testing
First, install [zombienet](https://github.com/paritytech/zombienet).
[Zombienet](https://github.com/paritytech/zombienet) provides local multi-validator network testing.
To spawn a local solo chain with four validators and BABE finality, run:
### Setup
1. Install Zombienet:
```bash
# Download binary from releases
# Or install via npm
npm install -g @zombienet/cli
```
2. Spawn local network with four validators:
```bash
zombienet -p native spawn test/config/zombie-datahaven-local.toml
```
This launches a local solochain with BABE consensus for testing validator coordination.
## Docker Image
Build local Docker image for testing:
```bash
zombienet -p native spawn test/config/zombie-datahaven-local.toml
cd ../test
bun build:docker:operator
```
This creates `datahavenxyz/datahaven:local` using optimized caching:
- [sccache](https://github.com/mozilla/sccache): Rust build caching
- [cargo-chef](https://lpalmieri.com/posts/fast-rust-docker-builds/): Dependency layer caching
- BuildKit cache mounts: External cache restoration
## Type Generation
After runtime changes, regenerate Polkadot-API TypeScript types:
```bash
cd ../test
bun generate:types # Production runtime
bun generate:types:fast # Fast runtime (development)
```
## Integration Testing
For full network integration tests with Ethereum, Snowbridge, and contracts:
```bash
cd ../test
bun cli launch # Interactive launcher
bun test:e2e # Run E2E test suite
```
See the [test directory](../test/README.md) for comprehensive testing documentation.
## Custom Pallets
### External Validators
Manages the dynamic validator set based on EigenLayer operator registry. Syncs validator changes from Ethereum to the Substrate consensus layer.
**Location**: `pallets/external-validators/`
### Native Transfer
Handles cross-chain token transfers between Ethereum and DataHaven via Snowbridge messaging.
**Location**: `pallets/native-transfer/`
### Rewards
Distributes performance-based rewards to validators, processing reward messages from the Ethereum `RewardsRegistry` contract.
**Location**: `pallets/rewards/`
Each pallet includes its own tests and benchmarks. See pallet-specific README files for details.

124
test/README.md
View file

@ -1,6 +1,8 @@
# DataHaven E2E Testing
Quick start guide for running DataHaven end-to-end tests. For comprehensive documentation, see [E2E Testing Guide](./docs/E2E_TESTING_GUIDE.md).
End-to-end testing framework for DataHaven, providing automated network deployment, contract interaction, and cross-chain scenario testing. This directory contains all tools needed to launch a complete local DataHaven network with Ethereum, Snowbridge relayers, and run comprehensive integration tests.
For comprehensive documentation, see [E2E Testing Guide](./docs/E2E_TESTING_GUIDE.md).
## Pre-requisites
@ -51,20 +53,68 @@ bun test:e2e:parallel
# Run a specific test suite
bun test suites/some-test.test.ts
```
## What Gets Launched
The `bun cli launch` command deploys a complete local environment:
1. **Ethereum Network** (via Kurtosis):
- 2x Execution Layer clients (reth)
- 2x Consensus Layer clients (lodestar)
- Blockscout Explorer (optional: `--blockscout`)
- Dora Consensus Explorer
2. **DataHaven Network**:
- Single validator solochain
- EVM compatibility via Frontier
- Fast block times (3s with `--fast-runtime`)
3. **Smart Contracts**:
- EigenLayer AVS contracts deployed to Ethereum
- Optional Blockscout verification (`--verified`)
4. **Snowbridge Relayers**:
- Beacon relay (Ethereum → DataHaven)
- BEEFY relay (DataHaven → Ethereum)
- Execution relay (Ethereum → DataHaven)
- Solochain relay (DataHaven → Ethereum)
5. **Network Configuration**:
- Validator registration and funding
- Parameter initialization
- Validator set updates
For more information on the E2E testing framework, see the [E2E Testing Framework Overview](./docs/E2E_FRAMEWORK_OVERVIEW.md).
## Other Common Commands
## Common Commands
| Command | Description |
| ------------------------- | ----------------------------------------------------------------------------------------------------------- |
| `bun cli stop` | Stop all local DataHaven networks (interactive, will ask for confirmation on each component of the network) |
| `bun cli deploy` | Deploy the DataHaven network to a remote Kubernetes cluster |
| `bun generate:wagmi` | Generate contract TypeScript bindings for the contracts in the `contracts` directory |
| `bun generate:types` | Generate Polkadot API types |
| `bun generate:types:fast` | Generate Polkadot API types with the `--fast-runtime` feature enabled |
| Command | Description |
| ------------------------- | -------------------------------------------------------------------------------------------------- |
| **Network Management** | |
| `bun cli` | Interactive CLI menu for all operations |
| `bun cli launch` | Launch full local network (interactive options) |
| `bun start:e2e:local` | Launch local network (non-interactive) |
| `bun start:e2e:verified` | Launch with Blockscout and contract verification |
| `bun start:e2e:ci` | CI-optimized network launch |
| `bun cli stop` | Stop all services (interactive) |
| `bun stop:dh` | Stop DataHaven only |
| `bun stop:sb` | Stop Snowbridge relayers only |
| `bun stop:eth` | Stop Ethereum network only |
| **Testing** | |
| `bun test:e2e` | Run all E2E test suites |
| `bun test:e2e:parallel` | Run tests with limited concurrency |
| `bun test <file>` | Run specific test file |
| **Code Generation** | |
| `bun generate:wagmi` | Generate TypeScript contract bindings (after contract changes) |
| `bun generate:types` | Generate Polkadot-API types from runtime |
| `bun generate:types:fast` | Generate types with fast-runtime feature |
| **Code Quality** | |
| `bun fmt:fix` | Fix TypeScript formatting with Biome |
| `bun typecheck` | TypeScript type checking |
| **Deployment** | |
| `bun cli deploy` | Deploy to Kubernetes cluster (interactive) |
| `bun build:docker:operator` | Build local Docker image (`datahavenxyz/datahaven:local`) |
## Local Network Deployment
@ -178,8 +228,56 @@ This script will:
>
> The script uses the `--release` flag by default, meaning it uses the WASM binary from `./operator/target/release`. If you need to use a different build target, you may need to adjust the script or run the steps manually.
## Project Structure
```
test/
├── suites/ # E2E test suites
│ ├── contracts.test.ts # Contract deployment & configuration
│ ├── cross-chain.test.ts # Cross-chain message passing
│ ├── datahaven-substrate.test.ts # Block production & finality
│ ├── ethereum-basic.test.ts # Ethereum network validation
│ ├── native-token-transfer.test.ts # Cross-chain token transfers
│ ├── rewards-message.test.ts # Validator rewards distribution
│ └── validator-set-update.test.ts # Dynamic validator set updates
├── framework/ # Test utilities & helpers
│ ├── connectors.ts # Network connectors
│ ├── manager.ts # Test environment manager
│ ├── suite.ts # Test suite utilities
│ └── index.ts # Framework exports
├── launcher/ # Network deployment tools
│ ├── kurtosis/ # Ethereum network launcher
│ ├── snowbridge/ # Relayer management
│ └── datahaven/ # DataHaven node management
├── generated/ # Generated types
│ ├── wagmi/ # Contract bindings
│ └── polkadot-api/ # Runtime types
└── docs/ # Testing documentation
├── E2E_TESTING_GUIDE.md
└── E2E_FRAMEWORK_OVERVIEW.md
```
## Test Suites
- **contracts.test.ts**: Contract deployment and configuration validation
- **cross-chain.test.ts**: Cross-chain message passing between Ethereum and DataHaven
- **datahaven-substrate.test.ts**: Block production, finalization, and consensus
- **ethereum-basic.test.ts**: Ethereum network health and basic functionality
- **native-token-transfer.test.ts**: Cross-chain token transfers via Snowbridge
- **rewards-message.test.ts**: Validator reward distribution from Ethereum to DataHaven
- **validator-set-update.test.ts**: Dynamic validator registration/deregistration via EigenLayer
Run individual suites:
```bash
bun test suites/rewards-message.test.ts
bun test suites/native-token-transfer.test.ts
bun test suites/validator-set-update.test.ts
```
## Further Information
- [Kurtosis](https://docs.kurtosis.com/): Used for launching a full Ethereum network
- [Zombienet](https://paritytech.github.io/zombienet/): Used for launching a Polkadot-SDK based network
- [Bun](https://bun.sh/): TypeScript runtime and ecosystem tooling
- [Kurtosis](https://docs.kurtosis.com/): Ethereum network orchestration
- [Zombienet](https://paritytech.github.io/zombienet/): Polkadot-SDK network testing
- [Bun](https://bun.sh/): TypeScript runtime and tooling
- [Foundry](https://book.getfoundry.sh/): Solidity development framework
- [Polkadot-API](https://papi.how/): Type-safe Substrate interactions

86
tools/README.md Normal file
View file

@ -0,0 +1,86 @@
# DataHaven Development Tools
Utility scripts for GitHub automation and release management.
## Overview
This directory contains Node.js/TypeScript tools for automating release workflows and generating standardized release notes for both runtime and client releases.
## Structure
```
tools/
├── github/ # GitHub automation utilities
│ ├── github-utils.ts # Common GitHub API utilities
│ ├── generate-release-body.ts # Client release notes generator
│ └── generate-runtime-body.ts # Runtime release notes generator
├── package.json # Dependencies and scripts
└── tsconfig.json # TypeScript configuration
```
## Prerequisites
- Node.js (version specified in `.nvmrc`)
- npm (for dependency management)
## Setup
```bash
cd tools
npm install
```
## Available Scripts
### Generate Client Release Notes
Creates formatted release notes for client (node binary) releases:
```bash
npm run print-client-release-issue
```
This script:
- Generates changelog from Git history
- Formats release notes for GitHub
- Includes version information and breaking changes
- Outputs markdown suitable for GitHub releases
### Generate Runtime Release Notes
Creates formatted release notes for runtime (WASM) releases:
```bash
npm run print-runtime-release-issue
```
This script:
- Generates runtime-specific changelog
- Highlights runtime version bumps
- Includes migration information
- Formats for GitHub release pages
## Usage in CI/CD
These tools are typically invoked by GitHub Actions workflows during the release process. They can also be run manually for testing or preparing draft releases.
## GitHub Integration
The tools use the [Octokit](https://github.com/octokit/octokit.js) library to interact with the GitHub API. Authentication is typically handled via `GITHUB_TOKEN` environment variable in CI/CD contexts.
## Development
The tools are written in TypeScript and use:
- `@polkadot/api`: For parsing runtime metadata
- `octokit`: For GitHub API interactions
- `yargs`: For CLI argument parsing
- `ts-node`: For direct TypeScript execution
## Customization
To modify release note formatting or add new automation:
1. Edit the respective generator in `github/`
2. Update `package.json` scripts if adding new commands
3. Test locally with `npm run <script-name>`
4. Update CI workflows in `.github/workflows/` if needed