This PR add the storagehub pallets to the `testnet` and `mainnet`
runtime. The storage hub pallets are only build with the runtime if the
`storage-hub` feature is activated.
It is also add minor fixes to the `stagenet` runtime (fix wrong path
dependencies, index collision, etc...).
---------
Co-authored-by: Steve Degosserie <723552+stiiifff@users.noreply.github.com>
## Summary
This PR resolves all CI failures following the migration to the new
DataHaven Github & Docker Hub organizations, and correctly leverage
self-hosted GitHub runners (`DH-runners` group) by eliminating sudo
dependencies.
## Key Changes
### 🔧 **Eliminated sudo requirements across all workflows**
- **Setup Environment**: Installed mold linker and system dependencies
in userspace without sudo
- **Tool Installation**: Replaced apt/system package installations with
direct binary downloads:
- Kurtosis: Direct binary download from GitHub releases (v1.10.3)
- Taplo: Direct binary installation for Cargo.toml formatting
- cargo-nextest: Using `cargo install` instead of GitHub action
(v0.9.100)
- **Runner Cleanup**: Skipped cleanup-runner action entirely on
self-hosted runners (bare-metal manages disk space externally)
### 🏷️ **Workflow optimizations**
- **Group-based targeting**: All heavy workloads (Rust builds, E2E
tests) now run on `DH-runners` runners
- **Dependency management**: Used `install-deps: false` flag instead of
hardcoded runner detection
Co-authored-by: Claude <noreply@anthropic.com>
### Summary
- **Add** `test/suites/native-token-transfer.test.ts` focused on the
HAVE native token lifecycle via Snowbridge v2.
- **Validate** registration, DataHaven → Ethereum mints, Ethereum →
DataHaven unlocks, event emission, and 1:1 backing invariant.
### Tests added
- should register DataHaven native token on Ethereum
- should transfer tokens from DataHaven to Ethereum
- should maintain 1:1 backing ratio
- should emit transfer events
- should transfer tokens from Ethereum to DataHaven (Snowbridge v2)
### What the suite covers
- **Registration**: Sudo-registers the native token; confirms
`ForeignTokenRegistered` on the Gateway; verifies ERC-20 metadata
(`HAVE`/`wHAVE`, 18 decimals).
- **DataHaven → Ethereum**: Executes `transfer_to_ethereum`; asserts
Substrate events (`TokensLocked`, `TokensTransferredToEthereum`);
observes Ethereum `Transfer` mint (from zero address); validates sender
balance delta, sovereign account increase, and ERC-20 recipient credit.
- **Backing invariant**: Ensures sovereign account balance ≥ ERC-20
total supply.
- **Event emission**: Confirms key Substrate events without polling
delays.
- **Ethereum → DataHaven**: Approves and calls `Gateway.sendToken`; if
unsupported locally, the test skips; otherwise asserts burn on Ethereum
and unlock on DataHaven with corresponding balance deltas.
---------
Co-authored-by: Steve Degosserie <723552+stiiifff@users.noreply.github.com>
## Summary
This PR introduces support for deploying Datahaven contracts to
different chains (hoodi, holesky, mainnet), as well as a new cli command
to manage this deployment separately from the regular deployment, while
maintaining compatibility with it.
#### New CLI command
- **`bun cli contracts deploy`** - Deploy contracts to supported chains
(Hoodi, Holesky, Mainnet)
- **`bun cli contracts status`** - Check deployment configuration and
status
- **`bun cli contracts verify`** - Verify contracts on block explorers
- Commands need the chain parameter: `--chain <hoodi | holesky |
mainnet>`
- Right now only `hoodi` and `holesky` are supported
### Deployment
#### Hoodi & Holesky Network Support
- Added **DeployBase.s.sol** as common ground for
**DeployTestnet.s.sol** (also new) and **DeployLocal.s.sol** (existing).
- **Hoodi configuration** (`contracts/config/hoodi.json`) with deployed
EigenLayer contract addresses to reference.
- **Holesky configuration** (`contracts/config/hoodi.json`) with
deployed EigenLayer contract addresses to reference.
#### Contracts being deployed
- **DataHaven**: ServiceManager, VetoableSlasher, RewardsRegistry
- **Snowbridge**: BeefyClient, AgentExecutor, Gateway, RewardsAgent
- **EigenLayer**: References existing deployed contracts (not
re-deployed)
#### Deployment files
When the deployment is done, a new file under `contracts/deployments` is
generated with the addresses of the deployed contracts, for each chain
(it will be overriden per chain if run multiple times). So we would have
one `anvil.json`, `hoodi.json`, `holesky.json`, etc, with the addresses
of the deployed contracts for reference and for later verification.
#### Todo
- [x] Test compatibility with existing `bun cli launch` and `bun cli
deploy` commands
#### For follow-up PRs
- Fix verification issue with `foundry verify-contracts` when specifying
the `chain` or `chain-id` parameter, needed for hoodi
(https://github.com/foundry-rs/foundry/issues/7466).
- Add `redeploy` feature to only override implementation contract and
leave the proxy address untouched
## Usage Examples
```bash
# Deploy to Hoodi network
bun cli contracts deploy --chain hoodi
# Check deployment status
bun cli contracts status --chain hoodi
# Verify contracts on block explorer
bun cli contracts verify --chain hoodi
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added deployment and configuration support for new networks "hoodi"
and "holesky", including new configuration and deployment files.
* Introduced a CLI tool for managing contract deployments, status
checks, and verification across supported chains.
* Added example environment configuration and comprehensive deployment
documentation.
* Enabled contract verification and status reporting via the CLI with
support for block explorer integration.
* **Improvements**
* Refactored deployment scripts for modularity, supporting both local
and testnet environments.
* Centralized and extended configuration loading to support additional
contract addresses and network parameters.
* Enhanced deployment utilities and typings to support multi-network
deployments.
* **Bug Fixes**
* Improved input validation and error handling in CLI commands and
deployment scripts.
* Added explicit handling for zero address in operator strategy
retrieval.
* **Chores**
* Updated documentation and configuration templates for easier
onboarding and deployment management.
* Improved logging and output formatting for deployment and verification
processes.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Steve Degosserie <723552+stiiifff@users.noreply.github.com>
Co-authored-by: Ahmad Kaouk <56095276+ahmadkaouk@users.noreply.github.com>
## Summary
This PR integrates the Substrate Proxy pallet into DataHaven runtimes
(testnet, stagenet, mainnet) with comprehensive test coverage. The proxy
pallet enables account delegation functionality, allowing accounts to
authorize other accounts to execute calls on their behalf with
configurable permissions.
## Changes
### Proxy Pallet Integration
- **Added proxy pallet** to all three DataHaven runtimes (testnet,
stagenet, mainnet)
- **Configured custom ProxyType enum** with DataHaven-specific proxy
types:
- `Any` - Unrestricted proxy access
- `NonTransfer` - All calls except balance transfers
- `Governance` - Governance and utility calls only
- `Staking` - Staking operations (placeholder)
- `CancelProxy` - Proxy announcement cancellation
- `Balances` - Balance transfer operations only
- `IdentityJudgement` - Identity judgement operations
- `SudoOnly` - Privileged sudo operations only
### Runtime Configuration
- **Integrated pallet-proxy** into runtime construction macros
- **Configured proxy deposits** (base + per-proxy fees)
- **Set proxy limits** (maximum proxies per account)
- **Implemented InstanceFilter** for call filtering per proxy type
- **Added proxy pallet to runtime APIs** and metadata
### Comprehensive Test Suite
- `operator/runtime/testnet/tests/proxy.rs` - 24 comprehensive proxy
tests
- `operator/runtime/stagenet/tests/proxy.rs` - 24 comprehensive proxy
tests
- `operator/runtime/mainnet/tests/proxy.rs` - 24 comprehensive proxy
tests
- Updated `lib.rs` files in all three runtimes to include proxy test
modules
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- New Features
- Enable account proxies across mainnet, stagenet, and testnet with
configurable types, delays, and per-type call permissions.
- Support anonymous (pure) proxies, proxy announcements with delays, and
deposit/limit parameters for proxy management.
- Tests
- Add comprehensive integration tests covering proxy lifecycle,
filtering, pure proxies, announcements, batching, chaining, multisig,
identity, and sudo paths.
- Test builder now supports optional sudo setup.
- Chores
- Add benchmarking, weights, and try-runtime support for proxies.
- Update internal package metadata version.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Ahmad Kaouk <ahmadkaouk.93@gmail.com>
Co-authored-by: Ahmad Kaouk <56095276+ahmadkaouk@users.noreply.github.com>
This PR enhances the Helm-based deployment system with several key
improvements organized into the following areas:
## New Features
### Ingress Management
- **Ingress per replica**: Added `ingress-per-replica` chart template
that automatically generates an ingress for each node replica, exposing
individual pod instances
- **Traefik integration**: Added chart values to deploy Traefik as the
ingress controller for local K8s clusters, enabling proper ingress
testing (requires adding names to `/etc/hosts`)
### Solochain Support
- **New relay chart**: Added dedicated Helm chart for Solochain relay
deployment
- **CLI integration**: Updated DataHaven CLI to deploy both Execution
and Solochain relayers
## Configuration Improvements
### Environment Structure
- **Modular configs**: Refactored environmental configurations from
single `values.yaml` files into separate component-specific overrides
for better organization
### Node Configuration
- **Base config updates**: Improved base configurations for bootnode and
validator nodes
- **Network protocol**: Reverted from litep2p back to libp2p to resolve
node communication issues on Stagenet
- **Archive node routing**: In Stagenet, relayers now connect to the
bootnode (configured as archive node) instead of validator nodes
## Storage & Deployment
### Persistent Storage
- **Relayer database**: Added support for persistent volumes to store
relayer databases instead of using ephemeral storage
### Deployment Scripts
- **Documentation cleanup**: Removed obsolete test deploy.sh script and
updated the deployment README with clearer instructions
These changes provide a more robust, scalable, and maintainable
deployment system for DataHaven infrastructure.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added support for deploying new relayer types: Solochain and Execution
relayers, with dedicated configuration and secret management.
* Introduced per-replica ingress configuration for node deployments,
allowing each replica to have its own ingress resource and hostname.
* Added persistent storage options for relay data, configurable via
storage path, class, and size.
* Added new deployment configuration files for local and stagenet
environments, including Traefik ingress controller setup.
* Introduced a new relay category, "Solochain Relayers," for standalone
chain operations and cross-chain communication.
* **Improvements**
* Updated deployment configurations to use container-specific
environment YAML files for more granular control.
* Enhanced relay and node configurations with new flags and backend
options, including dynamic peer ID generation and automatic bootnode
discovery.
* Updated relayer endpoints to consistently use the bootnode for
connections.
* Refined relay configuration files for improved structure, clarity, and
endpoint management.
* **Bug Fixes**
* Corrected deployment logic to reference the correct
environment-specific configuration files during Helm deployments.
* **Documentation**
* Simplified and updated deployment documentation to focus on CLI-based
deployment, removing outdated manual instructions and adding a concise
overview of components and relayer types.
* **Chores**
* Removed deprecated deployment scripts and outdated configuration files
to streamline the deployment process.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Claude <noreply@anthropic.com>
This pull request updates the identity pallet configuration to ensure
that slashed funds are now sent to the treasury, rather than being
discarded. This change is a step towards proper fund management within
the runtime.
Identity pallet configuration update:
* Changed the `Slashed` associated type implementation in the
`pallet_identity::Config` for `Runtime` to use `Treasury`, so that
slashed funds are now sent to the treasury instead of being ignored.
* Add `run_benchmarks.sh` script to run runtime benchmarks
* Sets up benchmark configs and directory structure to store weights
(`operator/runtime/<RUNTIME>/weights`)
* (naive) fixes to some benchmarks:
* `pallet_datahaven_native_transfer`:
* use a mock for `NativeTokenId`
* look at the balance difference of the treasury instead of the total
(this makes the benchmark agnostic to genesis setup)
* `snowbridge_pallet_system` / `snowbridge_pallet_system_v2` use native
token xcm location vs relay chain one. Add missing benchmark methods and
update fixture with valid data.
* `snowbridge_pallet_ethereum_client`: update fixtures with valid data
* `snowbrige_pallet_inbound_queue_v2`: set EthereumGatewayAddress when
initializing storage on benchmark and use a mock message processor ( as
fixture has `CreateAsset` payload which is not supported in the
`EigenLayerMessageProcessor`)
* `snowbridge_pallet_outbound_queue_v2`: add missing
`submit_delivery_receipt` benchmark which required a dedicated fixture
(all copied from the upstream pallet)
* `pallet_treasury`: Use an `ExistentialDeposit` of `1` on benchmark,
else payout fails.
* `pallet_transaction_payment`: Use a custom `WeightToFee` that makes
the Fee small, else account in benchmark cannot pay for fees (It is
funded a multiplier of `ExistentialDeposit` and is expected for that to
be enough, but it's not in our particular setup).
* comment out `pallet_identity` and `pallet_im_online` due to
incompatibilities (to be addressed later)
* Basic benchmark run to set `WeightInfo` from `weights` in configs
(real run should be done later using target hardware)
---------
Co-authored-by: Ahmad Kaouk <ahmadkaouk.93@gmail.com>
Co-authored-by: Tobi Demeco <50408393+TDemeco@users.noreply.github.com>
Co-authored-by: undercover-cactus <lola@moonsonglabs.com>
Co-authored-by: TDemeco <tdemeco@itba.edu.ar>
## Summary
This PR introduces comprehensive event waiting utilities for both
DataHaven (Substrate) and Ethereum chains, providing a unified
interface for handling blockchain events in E2E tests.
## What's New
- **Event Utilities** (`test/utils/events.ts`): New utilities for
waiting on blockchain events
- `waitForDataHavenEvent`: Type-safe event waiting for Substrate chain
events
- `waitForEthereumEvent`: Event waiting for Ethereum contract events
- Graceful timeout handling (returns null instead of throwing)
- Support for event filtering, callbacks, and custom timeouts
- **Documentation** (`test/docs/event-utilities-guide.md`):
Comprehensive guide covering usage examples for both DataHaven and
Ethereum.
## Test Plan
- [ ] New event utilities work as expected
- [ ] Event filtering works correctly for both chains
- [ ] Timeout handling behaves as documented
- [ ] Parallel event waiting with `Promise.all()` works
---------
Co-authored-by: Claude <noreply@anthropic.com>
We remove `calculateMerkleRootUnsorted` and `buildMerkleProofUnsorted`
and instead introduce a new parameter `sorting`. This parameter
`sorting` will indicate if we need to sort the pair before hashing the
new node.
## Summary
This PR add some error example for troubleshooting and an instruction to
purge minikube (can be useful to restart fresh).
It also replaces the command `kurtosis cluster set docker.k8s` by
`kurtosis cluster set <pick-a-cluster-option>` to force the reader to
actually read the different options and not just copy/paste the command
in the terminal. The options here matters to complete the deployment.
### Problem
Introducing `--network` should make easy to container nodes to find each
other. But this change was made half-way for the relayers, and it was
using the external port to find the first datahaven node (usually
Alice). So:
- In cli launch, Alice node port mapping was left to random port `-p
9944` instead of `-p 9944:9944`.
- Relayers couldn't connect to DataHaven nodes because they were using
the external WS port (now random) instead of hitting the internal port
(which for a cli launch we actually fix it to 9944).
### Solution
- [x] **Fixed Docker port mapping**: Explicit `-p 9944:9944` for Alice
node under network `cli-launch`
- [x] **Enhanced container spec**: Added `internalPorts` tracking to
`LaunchedNetwork`
- [x] **Fixed relayer connections**: Use internal ports for container
communication
## Implement E2E Testing Framework with Isolated Networks
### Summary
Refactors the existing E2E testing infrastructure to provide isolated
test environments with parallel execution support. Each test suite now
runs in its own network namespace, preventing resource conflicts.
### Key Changes
- **New Testing Framework** (`test/framework/`): Base classes for test
lifecycle management with automatic setup/teardown
- **Launcher Module** (`test/launcher/`): Extracted network
orchestration logic from CLI handlers for reusability
- **Parallel Execution**: Added `test-parallel.ts` script with
concurrency limits to prevent resource exhaustion
- **Test Isolation**: Each suite gets unique network IDs (format:
`suiteName-timestamp`) and Docker networks
- **Improved Test Organization**: Migrated tests to new framework,
deprecated old test structure
### Test Improvements
- Added 4 new test suites demonstrating framework usage. :
- `contracts.test.ts` - Smart contract deployment/interaction
- `datahaven-substrate.test.ts` - Substrate API operations
- `cross-chain.test.ts` - Snowbridge cross-chain messaging
- `ethereum-basic.test.ts` - Ethereum network operations
> [!WARNING]
The test suites themselves are bad and shouldn't be consider examples of
good tests. They were AI generated just to test the concurrency of test
runners
### Documentation
- Added comprehensive framework overview (`E2E_FRAMEWORK_OVERVIEW.md`)
- Updated README with parallel testing commands
- Added test patterns and best practices
### Breaking Changes
- Old test suites moved to `e2e - DEPRECATED/` directory
- Test execution now requires extending `BaseTestSuite` class
### Testing
Run tests with: `bun test:e2e` or `bun test:e2e:parallel` (with
concurrency limits)
### TODO
- [ ] Implement good test examples.
- [ ] Implement useful test utils (like waiting for an event to show up
in DataHaven or Ethereum).
- [ ] Enforce tests with CI (currently cannot be done due to
intermittent error when sending a transaction with PAPI).
---------
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: undercover-cactus <lola@moonsonglabs.com>
## Summary of changes
- We decided to remove the topics and nonce from the massage encoding
since we don't use them (original commit:
ee2a3f2fd4).
- Besides, we already have a nonce at the Snowbridge message level
f4ab5c2b2e/operator/primitives/snowbridge/inbound-queue/src/v2/message.rs (L105)
- I had to recreate the static test for _encoding_ (happens in
[DataHavenSnowbridgeMessages.sol](d12d40634f/contracts/src/libraries/DataHavenSnowbridgeMessages.sol)
) / _decoding_ (happens in
[operator/primitives/bridge/src/lib.rs)](f9f9cc65fe/operator/primitives/bridge/src/lib.rs).
Now it matches the current structure. The idea is that now we can test
that we don't break the decoding in followup refactoring.
- Fixes a problem with EigenLayer validator addresses. In all our
contracts we were using `bytes32` to refer to a Solochain validator
address. But on our Substrate change we actually expect AccountId20, so
only 20 bytes. This was causing the decoding to fail.
- I opted for the minimal change that would be to take the right-most 20
bytes to send that to our chain. But we might want aswell to limit our
EigenLayer contracts to be only 20 bytes long. @ahmadkaouk showcase this
[here](92a34c273c)
- Adds a bash script to run the static test. The test will compile the
contracts, run the encoding test, compile the operator, and run the
decoding test. This saves a huge amount of time since we don't need to
run the full e2e setup. The way of running it is the following:
```bash
cd operator/test/scripts
./test_message_encoding.sh
```
- As a consequence of this PR, the execution relayer now works properly.
EDIT:
> [!IMPORTANT]
**We decided to use 20-byte addresses in our contracts**. So what is
stated above is not valid anymore.
The change implies that the mapping from Ethereum addresses to bytes32
addresses now it's a mapping as follows:
dd3ba99ac0/contracts/src/DataHavenServiceManager.sol (L51-L52)
I've updated helper functions, tests, etc to be compliant with this
change. The execution relayer and beefy relayer look stable now.
---------
Co-authored-by: Ahmad Kaouk <ahmadkaouk.93@gmail.com>
Co-authored-by: Ahmad Kaouk <56095276+ahmadkaouk@users.noreply.github.com>
~~Improve CI runs by adding a caching action. Between each run crates
should be cached to speed up building datahaven binary.~~
## Summary
In the end the real issue was not that we were missing a caching action
but that we were caching too much. We went way over the 10GB limit
imposed by github (we were using 60GB of cache). So the most recent
cache were deleted right away or not cache at all.
The over use of the cache was happening because we were caching twice
sccache folder. Once with the `mozilla-actions/sccache-action` and an
other time with `Swatinem/rust-cache`.
The solution was to keep caching sccache on github with
`mozilla-actions/sccache-action`.
In this PR we also exchange the `Swatinem/rust-cache` action to use the
more standard `actions/cache@v4`. It would avoid in the future unwanted
breaking changes and security issues.
---------
Co-authored-by: Ahmad Kaouk <56095276+ahmadkaouk@users.noreply.github.com>
# Description
This PR implements a comprehensive overhaul of the `RewardsRegistry`
contract to maintain complete history of reward merkle roots while
providing index-based claim tracking for operators. The new architecture
enables operators to claim rewards from any historical merkle root
instead of only the latest one. To do so, it:
- Adds the `merkleRootHistory` storage array to the contract, in which
we keep all rewards roots that ever came from the DataHaven side.
- Adds the `operatorClaimedByIndex` storage map to the contract, in
which we keep track, for each validator and root index, if it has
claimed it or not.
- This works even for new validators, since theoretically with this
system you could argue they could claim older roots that they were not a
part of which would be catastrophic, but they could never draft a
correct proof for those to claim them.
- Keeps some of the interface from before the overhaul, to have quick
access to the latest rewards merkle root through `getLatestMerkleRoot()`
and to claim rewards for it with `claimRewards()`. This is because the
expected behaviour is for validators to claim their rewards every era.
- Adds a way to batch claim rewards with `claimRewardsBatch()`. This
function allows a validator to claim rewards for multiple root indices
in one call by providing multiple proofs, useful if the validator has
fallen behind claims and has to catch up, although special care will
have to be taken by it to avoid reaching the gas limit of a transaction.
## Storage Efficiency Analysis
One might think this solution is not as storage-efficient as other
solutions that we can think of (I even had two other alternatives in
mind as well), but a simple back-of-the-envelope calculation gives us
peace of mind that the impact of this solution on the overal state size
of the chain is negligible:
### Assumptions (Worst Case Scenario):
- 1,000 validators (actual estimate for DataHaven: ~50/100 validators)
- 6-hour eras (most-likely scenario, following what Polkadot does:
~24-hour eras)
- Which means 4 merkle root updates per day
### Annual Storage Requirements:
- Merkle Root History: **46,720 bytes/year**
- 4 roots/day × 32 bytes/root × 365 days/year = 46,720 bytes/year
- Operator Claim Tracking: **~1.46 MB/year**
- 1,000 operators × 1 boolean/(operator * root index) × 1 byte/boolean ×
4 root indices/day × 365 days/year = 1,460,000 bytes/year
- **Total: ~1.5 MB/year**
This represents negligible storage overhead compared to the significant
operational benefits gained.
## TODO
Since we want to allow the operators/validators to only have to interact
with the AVS contract (that's why the `claimRewards` functions have the
`onlyAVS` modifier), we still have to:
- [x] Add the required functions to the AVS to allow operators to claim
their rewards.
- [x] Adds comprehensive unit tests for them.
---------
Co-authored-by: Steve Degosserie <723552+stiiifff@users.noreply.github.com>
Co-authored-by: Ahmad Kaouk <56095276+ahmadkaouk@users.noreply.github.com>
Co-authored-by: Ahmad Kaouk <ahmadkaouk.93@gmail.com>
## Overview
This PR implements the **return path** for DataHaven's native token
bridge, enabling tokens that were previously transferred from DataHaven
to Ethereum to flow back to DataHaven.
**Context**: DataHaven native tokens can be transferred to Ethereum
where they exist as wrapped ERC20 tokens. This PR completes the
bidirectional bridge by implementing the mechanism to transfer these
tokens back from Ethereum to DataHaven, effectively "unwrapping" them
and restoring them to their native form.
## Key Changes
- **Added `NativeTokenTransferMessageProcessor`** , a new message
processor specifically designed to handle native token return transfers
from Ethereum, and extended `InboundQueueV2` processor tuple to include
native token transfer handling alongside existing EigenLayer message
processing
## How It Works
### Detailed Flow: Ethereum → DataHaven
#### 1. **Ethereum Initiation**
- **User Transaction**: User calls `v2_sendMessage` on Snowbridge
Gateway contract:
```solidity
v2_sendMessage(
assets: [{
kind: ForeignToken,
assetId: <registered_native_token_id>, // Must match pre-registered
token ID
amount: <transfer_amount>
}],
claimer: <datahaven_recipient_as_h160>, // 20-byte recipient address
(SCALE-encoded)
message: [], // Empty for native transfers
// ... standard Snowbridge fees
)
```
- **Token Burning**: Gateway burns the wrapped ERC20 tokens on Ethereum
(removing them from circulation)
- **Message Encoding**: Gateway encodes transfer details into
standardized Snowbridge message format
- **Event Emission**: `MessageAccepted` event logged on Ethereum for
relayers to pick up
#### 2. **Cross-Chain Delivery Infrastructure**
- **Relayer Network**: Snowbridge relayers monitor Ethereum events and
submit messages to DataHaven
- **Consensus Verification**: DataHaven's `EthereumBeaconClient` pallet
verifies Ethereum beacon chain consensus
- **Cryptographic Proofs**: Messages include Merkle proofs and execution
receipts for tamper-proof verification
- **Direct Processing**: Relayers submit messages via
`InboundQueueV2::submit()` which processes them immediately
#### 3. **DataHaven Message Processing**
**Immediate Processing Flow:**
When a relayer calls `InboundQueueV2::submit()`, the message is
processed immediately using a tuple-based processor system:
```rust
type MessageProcessor = (
EigenLayerMessageProcessor<Runtime>, // Handles validator operations
NativeTokenTransferMessageProcessor<Runtime>, // Handles native token returns
);
```
**Processing Steps:**
1. **Message Verification**: Cryptographic verification and nonce
checking (prevents replay attacks)
2. **Processor Selection**: Evaluation of processors (First processor
returning `true` handles the message):
- `EigenLayerMessageProcessor::can_process_message()` - returns `false`
for token transfers
- `NativeTokenTransferMessageProcessor::can_process_message()` -
validates:
- Native token is registered via `SnowbridgeSystemV2::register_token()`
(`T::NativeTokenId::get().is_some()`)
- All assets are `ForeignTokenERC20` with matching native token ID
- Assets array is not empty
**Token Unlocking Process:**
1. **Recipient Extraction**: Decodes H160 address from `claimer` field →
converts to DataHaven AccountId (must contain valid SCALE-encoded H160
address)
2. **Sovereign Account Transfer**: Moves tokens from Ethereum sovereign
account to recipient (requires sufficient balance from previous outbound
transfers)
3. **Event Emission**: `TokensUnlocked` event confirms successful
transfer completion
---------
Co-authored-by: Tobi Demeco <50408393+TDemeco@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
This PR solves an issue where the solochain relayer was fatally crashing
because the DataHaven chain initialized with an empty
`RewardsRegistryAddress` parameter. This caused all initial reward
merkle root update messages to target the zero address before the
parameter could be properly set by the launch command of the CLI. When
the relayer tried to call the update function on the zero address (which
isn't a valid contract), the transactions reverted and crashed the
relayer.
The band-aid fix implemented is to set the actual `RewardsRegistry`
contract address as the default value for the `RewardsRegistryAddress`
parameter (since we know it as it's consistent between our CLI runs).
This ensures initial messages have a valid target contract address from
startup, preventing the fatal crashes.
Again, this is a temporary fix until we implement a more robust solution
for parameter initialization, since it won't hold up if any changes
happen to our contracts.
---------
Co-authored-by: Gonza Montiel <gon.montiel@gmail.com>
In this PR we add StorageHub pallets to the runtime. It will only be
added if the `storage-hub` feature is activated.
---------
Co-authored-by: Gonza Montiel <gonzamontiel@users.noreply.github.com>
- Added a parameter `--all` to `bun cli launch` (now coherent with `bun
cli stop`)
- Equivalent to run `bun cli launch --d --bd --lk --dc --fv --sv --uv
--sp --r --cn`
- Removed `bun start:all` command
- Added a kurtosis cluster type check
- The problem was that we now support deploying and launching the CLI
tool with different types of clusters. If you're deploying, your
kurtosis cluster most likely needs to point to a `kubernetes` type
cluster, if you're running `bun cli launch` you need to use native
docker containers. This PR adds a check and warns the user to point to
the right config.
- *Disclaimer*: we don't know the cluster name of the user so we can't
force it to be anything 🫠
This PR contains improvements to the DataHaven deployment
infrastructure:
1. Directory restructure: Moved from `deployment/` to `deploy/` (more
common for K8s / Helm -based deployment configs).
2. Added **local environment** support: updated CLI to support deploying
to a local K8s cluster.
3. Manual deployment script: `deploy/scripts/deploy.sh` for manual
deployments.
4. Environment-specific configurations: Structured values files for each
environment.
5. Chart organization: Renamed bridges-common-relay to relay for
clarity.
---------
Co-authored-by: Gonza Montiel <gon.montiel@gmail.com>
Co-authored-by: Gonza Montiel <gonzamontiel@users.noreply.github.com>
This PR integrates the Substrate FRAME Treasury pallet across all three
DataHaven runtime environments (`testnet`, `mainnet`, `stagenet`) with a
custom fee allocation mechanism and comprehensive test coverage.
### Key Changes
#### Treasury Pallet Integration:
- Added Treasury pallet to all three runtimes (testnet, mainnet,
stagenet) with 20% fee allocation and 80% burn mechanism.
- Implemented dynamic fee proportion control via
`FeesTreasuryProportion` runtime parameter.
- Integrated treasury with custom fee handlers:
`DealWithEthereumBaseFees`, `DealWithEthereumPriorityFees`, and
`DealWithSubstrateFeesAndTip`.
#### Comprehensive Testing Infrastructure:
- Created robust test architecture with session management and validator
setup across all runtimes
- Added block author management utilities for treasury testing requiring
pallet_authorship integration
- Implemented 15 total tests (5 tests × 3 runtimes) covering:
- EVM transaction fee allocation with/without priority fees
- Substrate fee and tip handling validation
- Treasury spending functionality via sudo
- Complete fee flow verification with actual network base fee
calculations
#### Technnical Implementation
Fee Allocation Logic:
- Base fees: 20% to treasury, 80% burned
- Priority fees: 100% to block author
- Substrate tips: 100% to block author
The implementation is based on
https://github.com/moonbeam-foundation/moonbeam/pull/3120 in
[Moonbeam](https://github.com/moonbeam-foundation/moonbeam), and
assisted by Claude Code.
Add CI check for Polkadot-API metadata freshness
This PR adds a new CI workflow that ensures the Polkadot-API metadata
file (`test/.papi/metadata/datahaven.scale`)
is kept up-to-date when runtime changes are made.
Changes:
- Added task-check-metadata.yml workflow that:
- Reuses the WASM artifact from the build-operator job (no duplicate
compilation)
- Runs `bun x papi add` to regenerate metadata
- Fails if the metadata file has uncommitted changes
- Integrated the check into `CI.yml` as a second-tier job alongside
`docker-build`
Why:
- Prevents TypeScript type definitions from becoming out of sync with
the runtime
- Reminds developers to run `bun generate:types:fast` when making
runtime changes
- Ensures consistent type safety across the codebase
The check provides clear error messages with instructions when metadata
is outdated.
This PR improves the CLI to get from the deployments the
`RewardsRegistryAddress` (address of the RewardsRegistry contract
deployed), `RewardsAgentOrigin` (origin used for the agent in charge of
updating the rewards merkle root in the RewardsRegistry contract) and
`RewardsUpdateSelector` (function selector of the function that the
agent must execute to do the aforementioned update) and then set these
values in the `parameters` pallet of the runtime.
After these changes the rewards merkle root is being updated on the
Ethereum side. 🎉
This PR adds a quick check to see if the image to use for the relayer is
a locally-built one (by checking that it ends with `:local`) and avoids
pulling from Docker Hub if so. This fixes the issue in which we were
unable to run the CLI with a local relayer image since it was always
trying to pull from the corresponding repository which of course didn't
exist for local images.
### New Features
1. Add the `deploy` command to our CLI.
1. Conditionally deploys kurtosis eth network if we're in `stagenet`
environment.
2. Deploys DH nodes.
3. Deploys contracts (all of them). In `mainnet` and `testnet` it
shouldn't deploy EL contracts, but for now that's not implemented.
4. Configures parameters, validators and relayers in the same way as
`launch`.
5. Currently, it only deploys `beefy` and `beacon` relayers, `execution`
and `solochain` relayers are pending for a subsequent PR.
2. Add `waitFor` utility function that receives a lambda.
### Refactors
1. Several common functionalities used both by the `launch` and `deploy`
command have been moved to the `cli/handlers/common` directory, from
where both commands use them. These include
1. Checks for installed dependencies.
2. Common constants.
3. The `LaunchedNetwork` class has been moved to this directory.
4. DataHaven nodes common functions.
5. Kurtosis common functions.
6. Relayer common functions.
7. Kubernetes functions (although only used by `deploy`, it seemed
fitting to have it here still).
8. Remove CLI questions and separator prints from `deploy-contracts.ts`
and `set-datahaven-parameters.ts` scripts. These things should be in the
`cli/launch` folder, which consumes the functions in these scripts.
9. Remove `setParametersFromCollection` from `utils` folder and put it
in `cli`.
10. Create base snowbridge relayer configs for `local` and `stagenet` as
two separate directories.
### Fixes
1. Sets the default time of the `deploy` command to 6s as Lodestar is
slower than Lighthouse.
2. In `runShellCommandWithLogger` only print `stderr` if the command
fails.
### Additional Minor Changes
1. K8s secret key names changed from `dh-beefy-relay-eth-key` to
`dh-beefy-relay-ethereum-key` and `dh-beacon-relay-sub-key` to
`dh-beacon-relay-substrate-key`, for simplicity in the deployment
script.
11. Update suggested configs for `.vscode` configs.
---------
Co-authored-by: Steve Degosserie <723552+stiiifff@users.noreply.github.com>
### Summary
- Implement native token transfers from DataHaven to Ethereum using
Snowbridge infrastructure
- Add comprehensive datahaven-native-transfer pallet with lock-and-mint
mechanism for cross-chain token representation
### Key Features
**New Pallet: datahaven-native-transfer**
- Cross-chain transfers: Transfer DataHaven native tokens to Ethereum
addresses via `transfer_to_ethereum extrinsic`
- Token locking mechanism: Secure token storage in deterministic
Ethereum sovereign account during transfers
- Fee management: Required fees for all transfers to compensate relayers
and prevent spam
- Emergency controls: Pause/unpause functionality
- Token registration: Integration with Snowbridge's token registration
for native token identification
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Introduced the DataHaven Native Transfer pallet enabling secure
cross-chain transfers of DataHaven native tokens to and from Ethereum
via Snowbridge.
- Added token locking on DataHaven, minting on Ethereum, and mandatory
fee collection to cover gas costs and incentivize relayers.
- Implemented pause and unpause controls for emergency management of
token transfers.
- **Configuration**
- Integrated the new pallet into mainnet, stagenet, and testnet runtimes
with updated network, account, and treasury settings.
- Added Ethereum sovereign account constants and native token ID support
for optimized cross-chain operations.
- **Documentation**
- Added comprehensive README and inline documentation detailing pallet
features, extrinsics, events, errors, and security considerations.
- **Tests**
- Added extensive unit and integration tests covering token transfers,
native token registration, fee handling, pause functionality, and event
validation across all supported networks.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Steve Degosserie <723552+stiiifff@users.noreply.github.com>
This PR fixes a critical issue in which the messages enqueued in the
`MessageQueue` pallet weren't being correctly commited by the
`SnowbridgeOutboundQueueV2` pallet because of a misconfiguration in our
runtimes:
- Before this PR, the `MessageQueue` pallet had a lower index than the
`SnowbridgeOutboundQueueV2` pallet. This means that its `on_initialize`
hook was executed before the `on_initialize` hook of the latter.
- The `MessageQueue` pallets processes messages in its `on_initialize`
hook. In our case, this processing meant calling the
`do_process_message` function of the `SnowbridgeOutboundQueueV2` pallet,
which pushes the message into both its `Messages` and `MessageLeaves`
storage for further processing afterwards, sets a pending order for it
for the relayer and increments the nonce for the next message.
- Then, the `SnowbridgeOutboundQueueV2`'s `on_initialize` hook gets
called, and it completely clears both the `Messages` and `MessageLeaves`
storage to start fresh for the new block. This means it would clear up
the previously processed messages.
- Finally, in the `on_finalize` hook, the messages processed in this
block would get commited from the `MessageLeaves` storage, but since it
would always be empty nothing got commited, which means the Solochain
relayer never found anything to relay and messages weren't getting to
Ethereum.
This PR renames all usage of `moonsonglabs/snowbridge-relayer` to
`moonsonglabs/snowbridge-relay` to adapt to the refactor done in [PR
#18](https://github.com/Moonsong-Labs/snowbridge/pull/18) of our
Snowbridge fork. The aforementioned PR should be merged before this one,
although caution is not really needed since the CI would fail otherwise.
Lighthouse consensus layer client has a bug when building the next sync
committee merkle proof, that is preventing the Snowbridge relayer from
updating the next sync committee in the Ethereum Beacon Client pallet,
on the DataHaven Substrate chain. See issue
[here](https://github.com/sigp/lighthouse/issues/7552).
As a consequence, we're moving to the lodestar implementation.
## This PR includes:
- Running the execution relayer on the CLI
- Modifying the Payload generation in the `DataHavenServiceManager.sol`
- Modified the `EigenLayerMessageProcessor` to work with the
ValidatorSet update message, but for this change we are loosing the
generic message type (it was the only way to make it work so far).
- Adds a `--no-wait` argument to the cli launch and stop commands to
bootstrap faster.
### Testing the Snowbridge message encoding / decoding
- Added`MessageEncoding.t.sol` is documented and explains how to
generate bin data to use in the rust test.
- Added a Rust unit test to `EigenLayerMessageProcessor` to compare the
message encoding/decoding taking the bytes from a file (previously
generated with some mock data).
Specifically, we want that:
3cbca0db6d/contracts/src/libraries/DataHavenSnowbridgeMessages.sol (L78-L85)
Generates the right bytes encoding for
0e2c9cd518/operator/primitives/bridge/src/lib.rs (L51)
If the test passes, it's very likely that the CLI will also pass, if
not, then we might wanna check something else is missing.
### Breaking change ⚠️
For compatibility reasons with Snowbridge contracts (they call specific
extrinsics of specific pallets), I had to rename:
- `InboundQueueV2` -> `EthereumInboundQueueV2`
- `OutboundQueueV2` -> `EthereumOutboundQueueV2`
## For follow up PRs:
- Add an automated way of generating the Solidity bytes fo testing, so
we don't need to maintain the MessageEncoding.t.sol and generate the
binary data manually.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Added support for the "execution" relayer type in relay configuration,
parsing, and CLI launch utilities.
- Introduced a Solidity test contract for encoding and logging validator
set messages.
- Added a comprehensive "start:all" script to streamline launching and
setup processes.
- **Enhancements**
- Improved message encoding for validator set updates, aligning with new
struct field names and message formats.
- Updated relay configuration schema and validation to support execution
relayers and OFAC settings.
- Increased beacon datastore capacity and adjusted relay scheduling
parameters in configuration files.
- **Refactor**
- Renamed runtime type aliases for inbound/outbound queues to more
descriptive names across mainnet, stagenet, and testnet.
- Centralized and streamlined validator set update logic in CLI
utilities.
- Centralized message decoding logic and improved visibility of message
fields in Rust components.
- **Bug Fixes**
- Improved error handling and decoding logic for message processing in
Rust components.
- **Tests**
- Added Rust and Solidity tests for message encoding and processing
validation.
- **Chores**
- Updated dependencies and feature flags in Rust project configuration.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Facundo Farall <37149322+ffarall@users.noreply.github.com>
This PR:
- Adds launching the new `solochain` relayer in the relayers' script,
with its config files and schemas.
- Updates the relayer launching to use switch-case logic since we are
now going to be running 4 relayers, making it cleaner.
- Minor CLI fixes (adding additional args to build command for Linux,
improving naming, deleting idle containers instead of only active ones).
- Deletes unused files `substrate-relay.json` (now
`solochain-relay.json`), `gen-snowbridge-cfgs.ts` and
`snowbridge-relayer.ts` (now Snowbridge binary and docker image
generation can be done exclusively from our [Snowbridge
repo](https://github.com/Moonsong-Labs/snowbridge))
- Updates the `UniversalLocation` of our stagenet runtime to be under
the global consensus for XCM instead of actually being the global
consensus. This makes it so we can actually use the Snowbridge System
pallets to queue up messages. For mainnet we are going to want to have
our own Network ID instead of using Polkadot's.
> [!WARNING]
> ~~All in all these changes allows us to run the solochain relayer, but
it won't work without the refactor that's being worked on in
https://github.com/Moonsong-Labs/snowbridge/pull/14. I'd advise waiting
for that PR to be merged before merging this one.~~ MERGED ✅
## Changes
- New CI file for making Docker Prod images
- Changed E2E tests use an image built from a local dockerfile
- Some cargo build options to make it quicker
- Fix the cache hit rate
- added `tsgo` preview to the project 😎
- Can be invoked with `bun tsgo` to typecheck
- Install in IDE
[VSCode](https://code.visualstudio.com/docs/configure/extensions/extension-marketplace)
& [Zed](https://github.com/zed-extensions/tsgo) for super-fast inline
typechecking (as you type basically)
## Context
This PR attempts to make the frankly unacceptable CI times better. This
achieves that aim by making a crappy image for day-to-day usage and let
the prod issue take ages since that will be infrequently used. The
reason why the original design didn't work for us is because: 1) we are
using the free GH runners 2) when we goto baremetal runners we'll lose
our rapid caching abilities which make using docker cheap.
Also, we add `tsgo` support to improve devex. The improvement is
astounding.
```sh
hyperfine -n tsc "bun tsc --incremental false --extendedDiagnostics" -n tsgo "bun tsgo --incremental false --extendedDiagnostics"
Benchmark 1: tsc
Time (mean ± σ): 5.500 s ± 0.221 s [User: 8.939 s, System: 0.400 s]
Range (min … max): 5.196 s … 5.845 s 10 runs
Benchmark 2: tsgo
Time (mean ± σ): 99.1 ms ± 8.4 ms [User: 392.8 ms, System: 54.1 ms]
Range (min … max): 88.3 ms … 116.0 ms 29 runs
Summary
tsgo ran
55.48 ± 5.22 times faster than tsc
```
This PRs extracts [this
commit](74c0c0be0a)
from @TDemeco's PR to add a way to include parameters as part of the
CLI.
### Key changes
- CLI tool to set DataHaven runtime parameters via JSON configuration
- Supports both interactive prompts and command-line flags
- Type-safe parameter parsing and validation
- Already adds the parameters for the `EthereumGatewayAddress`, that
otherwise we would need to add manually to the node using an explorer.
---------
Co-authored-by: TDemeco <tdemeco@itba.edu.ar>
## Changes
- New option: `--kurtosis-enclave-name` to allow you to specify a new
ethereum network with a different enclave name. Neccesary step for
setting up local testing with multiple enclaves running at once
- Refactor: all single dash options (e.g. `-i`) have been renamed to
have double dashes `--` for consistency others
- sad times: CLI now must be invoked `bun cli launch` since we have
multiple fns now
- Package Update
- New Function: `stop` which allows you to kill all components (eth, dh,
relayers) or only a single part
- Gonza's mac target build fix
- Misc fixes to ci like caching and rate limits
## Additional Comment
The CLI needs multiple commands and this PR adds the first new one
`stop`.
This syntax is faily extensible so @ffarall 's k8 command can follow
suite.
Originally we were going to have an `exec` command to expose internal
fns to be callable by command line, i.e. generate beacon checkpoint -
however the need for it has since been superceded by the deploy to k8
command that's going to be added. I've left the code in place though so
when another usecase comes up we can just plug that in.
---------
Co-authored-by: Facundo Farall <37149322+ffarall@users.noreply.github.com>
This PR introduces and integrates the new
`pallet-external-validators-rewards` into the operator runtimes. This
pallet is responsible for managing and distributing rewards to external
validators. The pallet was originally introduced by Tanssi. It was
heavily modified to abstract the message building and sending with
Snowbridge.
### Key Changes Related to `pallet-external-validators-rewards`:
* **Pallet and Runtime API Addition:**
* The `pallet-external-validators-rewards` has been added to `mainnet`,
`stagenet`, and `testnet` runtimes
* **Runtime Configuration
(`operator/runtime/{mainnet,stagenet,testnet}/src/configs/mod.rs`):**
* **`RewardsRegistryAddress` Parameter:** A new dynamic runtime
parameter `RewardsRegistryAddress` (type `H160`) has been added to
`runtime_params.rs`. This will hold the Ethereum address of the Rewards
Registry contract.
* **Author Rewards with `RewardsPoints`:**
* A new struct `RewardsPoints` implements
`pallet_authorship::EventHandler`.
* Its `note_author` function rewards block authors (if they are not
whitelisted validators) by calling
`ExternalValidatorsRewards::reward_by_ids`.
* The `EventHandler` in `pallet_authorship::Config` is updated to
`(RewardsPoints, ImOnline)`.
* **Integration with `pallet-external-validators`:**
* The `OnEraStart` and `OnEraEnd` hooks in
`pallet_external_validators::Config` are now set to
`ExternalValidatorsRewards`, allowing the rewards pallet to react to era
changes.
* **`RewardsSendAdapter` for Ethereum Communication:**
* A new struct `RewardsSendAdapter` implements
`pallet_external_validators_rewards::types::SendMessage`.
* `build()`: Constructs an `OutboundMessage` to call the
`updateRewardsMerkleRoot(bytes32)` function on the
`RewardsRegistryAddress` on Ethereum. The calldata includes the selector
(hashed from `UpdateRewardsMerkleRootSignature`) and the
`rewards_merkle_root`.
* `validate()` and `deliver()`: Utilize `OutboundQueueV2` for message
validation and delivery.
* A constant `UpdateRewardsMerkleRootSignature` (
`b"updateRewardsMerkleRoot(bytes32)"`) is defined for this purpose.
* **Runtime API
(`operator/runtime/{mainnet,stagenet,testnet}/src/lib.rs`):**
* The runtime implements
`pallet_external_validators_rewards_runtime_api::ExternalValidatorsRewardsApi`.
* This exposes two functions:
* `generate_rewards_merkle_proof(account_id: AccountId, era_index:
EraIndex) -> Option<MerkleProof>`
* `verify_rewards_merkle_proof(merkle_proof: MerkleProof) -> bool`
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Introduced a rewards system for external validators with era-based
tracking, Merkle proof generation, and verification.
- Added runtime APIs to generate and verify validator rewards Merkle
proofs.
- Enabled rewards distribution via outbound messaging to an external
registry contract.
- Configured runtime parameters for the rewards registry address.
- Added fixed author reward points and integrated rewards logic into
block authorship events.
- Provided a mock runtime environment and benchmarking support for the
rewards pallet.
- **Tests**
- Added comprehensive tests and benchmarks for validator rewards
distribution and Merkle proof functionality.
- **Chores**
- Integrated new pallets and runtime APIs into mainnet, stagenet, and
testnet configurations.
- Updated dependencies and feature groups to include new rewards-related
pallets.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Co-authored-by: Facundo Farall <37149322+ffarall@users.noreply.github.com>
