## Overview
Implements deterministic weighted-stake-based validator selection in
`DataHavenServiceManager`, building on the era-targeting submitter model
from PR #433. Previously, `buildNewValidatorSetMessage()` forwarded all
registered operators in arbitrary membership order with no stake-based
ranking, meaning high-stake operators could be displaced by lower-stake
ones when downstream caps applied. This PR fixes that by computing a
weighted stake score per operator and selecting the top-32 candidates
before bridging the set to DataHaven.
Spec: `specs/validator-set-selection/validator-set-selection.md`
## Contract Changes (`DataHavenServiceManager.sol`)
**New state:**
- `MAX_ACTIVE_VALIDATORS = 32` — cap on the outbound validator set
- `mapping(IStrategy => uint96) public strategiesAndMultipliers` —
per-strategy weight used in the selection formula
**Updated `buildNewValidatorSetMessage()`:**
1. Fetches allocated stake for all operators × strategies from
`AllocationManager`
2. Computes `weightedStake(op) = Σ(allocatedStake[op][j] ×
multiplier[j])` across all strategies
3. Filters operators with no solochain address mapping or zero weighted
stake
4. Runs a partial selection sort to pick the top `min(candidateCount,
32)` by descending weighted stake; ties broken by lower operator address
(deterministic)
5. Reverts with `EmptyValidatorSet()` if no eligible candidates remain
**Admin API changes:**
- `addStrategiesToValidatorsSupportedStrategies()` signature changed
from `IStrategy[]` to `IRewardsCoordinatorTypes.StrategyAndMultiplier[]`
— strategy and multiplier are stored atomically in one call, eliminating
the risk of a strategy being registered without a multiplier
- New `setStrategiesAndMultipliers(StrategyAndMultiplier[])` — updates
multiplier weights for existing strategies without touching the
EigenLayer strategy set
- New `getStrategiesAndMultipliers()` — returns all strategies with
their current multipliers
- `removeStrategiesFromValidatorsSupportedStrategies()` now cleans up
multiplier entries on removal
**New error / event:**
- `EmptyValidatorSet()` — reverts when no eligible candidates exist
- `StrategiesAndMultipliersSet(StrategyAndMultiplier[])` — emitted on
add or update of multipliers
## Tests (`ValidatorSetSelection.t.sol`)
New 552-line Foundry test suite covering all cases from the spec:
| Case |
|------|
| `addStrategies` stores multiplier atomically |
| `removeStrategies` deletes multiplier |
| `setStrategiesAndMultipliers` updates without touching the strategy
set |
| `getStrategiesAndMultipliers` returns correct pairs |
| Weighted stake computed correctly across multiple strategies |
| Operators with zero weighted stake are excluded |
| Unset multiplier treated as 0 |
| Top-32 selection when candidate count > 32 |
| All candidates included when count < 32 |
| Tie-breaking by lower operator address |
| `EmptyValidatorSet` revert when no eligible operators |
## Deploy Scripts
- **`DeployBase.s.sol`**: Sets a default multiplier of `1` for all
configured validator strategies after AVS registration via
`setStrategiesAndMultipliers`
- **New `AllocateOperatorStake.s.sol`**: Forge script that allocates
full magnitude (`1e18`) to the validator operator set for a given
operator. Must be run at least one block after `SignUpValidator` to
respect EigenLayer's allocation configuration delay.
## E2E Framework
- **`validators.ts` — `registerOperator()`**: Extended to deposit tokens
into each deployed strategy and allocate full magnitude to the DataHaven
operator set after registration. Previously operators registered without
staking, producing zero weighted stake and getting filtered out by the
new selection logic.
- **`setup-validators.ts`**: Added a stake allocation pass after the
registration loop, invoking `AllocateOperatorStake.s.sol` per validator.
- **`validator-set-update.test.ts`**: Added debug logging for
transaction receipts and the `OutboundMessageAccepted` /
`ExternalValidatorsSet` events.
- **`generated.ts`**: Regenerated contract bindings to include new
functions, events, and the `EmptyValidatorSet` error.
## ⚠️ Breaking Changes ⚠️
- `addStrategiesToValidatorsSupportedStrategies(IStrategy[])` →
`addStrategiesToValidatorsSupportedStrategies(StrategyAndMultiplier[])`:
callers must supply multipliers alongside strategies.
- Operators with zero weighted stake are no longer included in the
bridged validator set.
## Rollout Notes
1. PR #433 (era-targeting + submitter role) must be deployed first
2. Deploy this `ServiceManager` upgrade
3. Confirm `strategiesAndMultipliers` is set for all active strategies
(default multiplier `1` applied automatically by `DeployBase`)
4. Deploy the runtime cap-enforcement changes (spec section 10.2)
5. Submitter daemon requires no changes — continues submitting
`targetEra = ActiveEra + 1`
## Era-targeted validator set submission with dedicated submitter role
> **Note:** This PR includes a detailed specification at
[`specs/validator-set-submission/validator-set-submission.md`](https://github.com/datahaven-xyz/datahaven/blob/feat/validator-set-submitter/specs/validator-set-submission/validator-set-submission.md)
that covers the design rationale, submission lifecycle, era-targeting
rules, and failure modes. Reading the spec first will make the contract,
pallet, and daemon changes easier to follow.
### Summary
- Introduce a dedicated `validatorSetSubmitter` role on
`DataHavenServiceManager`, separating validator set submission authority
from the contract owner
- Replace the unscoped `sendNewValidatorSet` with
`sendNewValidatorSetForEra`, which encodes a `targetEra` into the
Snowbridge message payload
- Add server-side era validation in the `external-validators` pallet to
reject stale, duplicate, or out-of-range submissions
- Add a long-running TypeScript daemon that watches session changes and
automatically submits each era's validator set at the right time
### Contract changes (`contracts/`)
- **New `validatorSetSubmitter` storage slot** — set during `initialize`
and rotatable via `setValidatorSetSubmitter` (owner-only). The storage
gap is decremented accordingly.
- **`sendNewValidatorSet` → `sendNewValidatorSetForEra`** — accepts a
`uint64 targetEra` parameter and is restricted to
`onlyValidatorSetSubmitter` instead of `onlyOwner`.
- **`buildNewValidatorSetMessageForEra`** — the
`NewValidatorSetPayload.externalIndex` is now caller-supplied instead of
hardcoded to `0`.
- **New events** — `ValidatorSetSubmitterUpdated`,
`ValidatorSetMessageSubmitted`.
- **New error** — `OnlyValidatorSetSubmitter`.
- **New test suite** — `ValidatorSetSubmitter.t.sol` covering submitter
set/rotate, access control, era encoding, and legacy function removal.
### Pallet changes (`operator/`)
- **`validate_target_era`** in `external-validators` — enforces
`activeEra < targetEra <= activeEra + 1` and `targetEra > ExternalIndex`
(dedup guard).
- **New errors** — `TargetEraTooOld`, `TargetEraTooNew`,
`DuplicateOrStaleTargetEra`.
- **Tests** — five new test cases for era boundary conditions (next-era
acceptance, old-era rejection, too-new rejection, duplicate rejection,
genesis behavior). Existing `era_hooks_with_external_index` test updated
to use valid target eras.
- **Runtime test fixes** — `external_index: 0` → `1` in
mainnet/stagenet/testnet EigenLayer message processor tests to satisfy
the new validation.
### Validator set submitter daemon
(`test/tools/validator-set-submitter/`)
- Event-driven service that subscribes to finalized
`Session.CurrentIndex` via Polkadot-API `watchValue`.
- Submits once per era during the last session, targeting `ActiveEra +
1`.
- Tracks submitted eras to avoid duplicates; skips if `ExternalIndex`
already covers the target.
- Startup self-checks: Ethereum connectivity, DataHaven connectivity,
on-chain submitter authorization.
- Supports `--dry-run` mode and YAML configuration.
- Graceful shutdown on `SIGINT`/`SIGTERM`.
### Test & tooling updates
- **E2E test** (`validator-set-update.test.ts`) — calls
`sendNewValidatorSetForEra` with a computed `targetEra` and filters the
substrate event by `external_index`.
- **`update-validator-set.ts` script** — accepts `--target-era` flag;
defaults to era 1 for fresh networks.
- **CLI launch** — wires validator set update as an interactive step
after relayer launch.
- **`package.json`** — new `submitter` and `submitter:dry-run` scripts.
- Regenerated contract bindings, PAPI metadata, state-diff, and storage
layout snapshots.
### Test plan
- [x] `forge test` — passes, including new `ValidatorSetSubmitter.t.sol`
- [x] `cargo test` — passes, including new era-validation tests in
`external-validators`
- [x] `bun test:e2e` — validator-set-update suite passes with
era-targeted flow
- [x] Manual: run submitter daemon against local network (`bun
submitter`), verify it submits once per era at the correct session
## ⚠️ Breaking Changes ⚠️
- **`sendNewValidatorSet` removed** — replaced by
`sendNewValidatorSetForEra(uint64 targetEra, ...)`. Callers must now
supply a `targetEra` parameter.
- **Access control changed** — validator set submission is now
restricted to the `validatorSetSubmitter` role instead of the contract
`owner`. The submitter address is set during `initialize` and rotatable
via `setValidatorSetSubmitter` (owner-only).
- **`external-validators` pallet now validates `targetEra`** — messages
with a stale, duplicate, or out-of-range `external_index` are rejected
on-chain. Existing integrations sending `external_index: 0` will fail
validation.
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
## Summary
Slashing and rewards submissions were submitted through the bridge with
their **solochain address** , while EigenLayer expects the **ethereum
operator address**, the addresses were not being translated, so the
protocol was broken.
This PR adds a **reverse mapping** (Solochain address → Eth address) and
uses it in both the slashing and rewards paths so that:
- `slashValidatorsOperator` accepts requests where `operator` is a
Solochain address and translates it to the Eth operator before calling
EigenLayer.
- `submitRewards` translates each `operatorRewards[].operator` from
Solochain to Eth before calling the RewardsCoordinator.
- Unknown or unmapped solochain addresses cause a revert
(`UnknownSolochainAddress`) instead of silently failing.
## What's changed
### DataHavenServiceManager
- **Reverse mapping**: `mapping(address => address) public
validatorSolochainAddressToEthAddress` (Solochain → Eth), with `__GAP`
reduced by one slot for upgradeable layout.
- **Helper**: `_ethOperatorFromSolochain(address)` – returns Eth
operator for a Solochain address, reverts with
`UnknownSolochainAddress()` if unmapped.
- **Registration / lifecycle**:
- `registerOperator`: populates both forward and reverse mappings;
enforces uniqueness (one Solochain per operator) and clears old reverse
entry when an operator re-registers with a new Solochain.
- `deregisterOperator`: clears both forward and reverse entries.
- `updateSolochainAddressForValidator`: updates both mappings, enforces
uniqueness and clears the previous Solochain's reverse entry.
- **Slashing**: `slashValidatorsOperator` uses
`_ethOperatorFromSolochain(slashings[i].operator)` so requests keyed by
Solochain address are translated before calling EigenLayer.
- **Rewards**: `submitRewards` builds a translated copy of the
submission with each `operatorRewards[].operator` set via
`_ethOperatorFromSolochain(...)`; unmapped addresses revert.
### IDataHavenServiceManager
- New getter: `validatorSolochainAddressToEthAddress(address solochain)
external view returns (address)`.
- New errors: `UnknownSolochainAddress()`,
`SolochainAddressAlreadyAssigned()`.
### Storage and fixtures
- Storage snapshot updated for the new state variable.
- `DataHavenServiceManagerBadLayout.sol` updated (reverse mapping + gap)
for layout negative tests.
- Storage layout test extended to assert the reverse mapping is
preserved across proxy upgrade.
### Tests
- **Slashing.t.sol**: Slashing with Solochain address (translation and
emit of Eth operator); negative test for unmapped Solochain reverting
with `UnknownSolochainAddress()`.
- **RewardsSubmitter.t.sol**: Rewards submission with Solochain
addresses (translation to Eth in RewardsCoordinator calldata); negative
test for unmapped Solochain.
- **StorageLayout.t.sol**: Reverse mapping preserved after upgrade.
- **OperatorAddressMappings.t.sol** (new): Uniqueness (Solochain already
assigned to another operator), update/deregister clearing reverse
mapping, and getter behaviour.
## Testing
- **Unit tests**: `forge test` from `contracts/` (all existing and new
tests pass).
- **Storage**:
- `./scripts/check-storage-layout.sh`
- `./scripts/check-storage-layout-negative.sh`
- **Coverage**: Slashing path (Solochain → Eth translation + revert),
rewards path (translation + revert), registration/update/deregister
(reverse mapping and uniqueness), and storage layout upgrade
preservation.
---------
Co-authored-by: Ahmad Kaouk <56095276+ahmadkaouk@users.noreply.github.com>
Co-authored-by: Ahmad Kaouk <ahmadkaouk.93@gmail.com>
## Summary
We are now launching the MSP backend when starting stpragehub services.
In this PR, we also fix the MSP and BSP node configuration and register
it with the correct keys.
## What changed
* Added a launch Backend MSP function that is called when launching
storage hub services
* Fix the wrong genesis error message in storagehub node by removing the
`--chain dev` flags (so it can be launch of the same network as our
local datahaven nodes).
* Use the correct keys to register MSP and BSP. We were injecting
different keys that the one we used for MSP and BSP registration leading
to the MSP and BSP node to never fully register as storage providers.
---------
Co-authored-by: Ahmad Kaouk <56095276+ahmadkaouk@users.noreply.github.com>
## Summary
- Add multi-environment deployment support (stagenet, testnet, mainnet)
to CLI and contracts
- Configure stagenet and testnet runtimes with correct genesis hashes
and Snowbridge Agent IDs
- Add CLI commands for BEEFY checkpoint updates and rewards origin
computation
- Add ETH validator strategies (native beacon chain ETH + LSTs) to all
config files
## Changes
### Runtime Configuration
**Stagenet Runtime:**
- Set `StagenetGenesisHash` to DataHaven stagenet genesis hash
- Configure `RewardsAgentOrigin` with computed Snowbridge Agent ID
- Add tests verifying rewards account derivation and agent ID
computation
**Testnet Runtime:**
- Set `TestnetGenesisHash` to DataHaven testnet genesis hash
- Configure `RewardsAgentOrigin` with computed Snowbridge Agent ID
- Add tests verifying rewards account derivation and agent ID
computation
The Rewards Agent ID is computed following Snowbridge's location
description pattern:
```
blake2_256(SCALE_ENCODE("GlobalConsensus", ByGenesis(genesis), "AccountKey20", rewards_account))
```
### CLI Enhancements
- All contracts subcommands (`status`, `deploy`, `verify`,
`update-metadata`) now accept `--environment` option
- Config and deployment files use environment-prefixed naming (e.g.,
`stagenet-hoodi.json`, `testnet-hoodi.json`)
- New `update-beefy-checkpoint` command that:
- Connects to a live DataHaven chain via WebSocket RPC
- Fetches all BEEFY data at the same finalized block for consistency
- Uses parallel queries with `Promise.all` for better performance
- Computes authority hashes (keccak256 of Ethereum addresses derived
from BEEFY public keys)
- Uses Snowbridge's quorum formula `n - floor((n-1)/3)` for strictly >
2/3 majority
- New `update-rewards-origin` command that computes the Snowbridge Agent
ID for the rewards pallet
- Centralized validation via `contractsPreActionHook` for all contract
commands
- Environment validation against allowlist (`stagenet`, `testnet`,
`mainnet`)
### Contract Changes
- Network validation uses explicit allowlist instead of suffix matching
- Added `initialValidatorSetId` and `nextValidatorSetId` fields to
`SnowbridgeConfig` struct
- `DeployBase.s.sol` now uses config values for validator set IDs
instead of hardcoded 0/1
- `DeployParams.s.sol` loads validator set IDs from config with
backwards compatibility
### Validator Strategies
Added ETH-equivalent strategies to allow validators to stake using
native ETH or LSTs:
**All Networks:**
- `0xbeaC0eeEeeeeEEeEeEEEEeeEEeEeeeEeeEEBEaC0` - Native beacon chain ETH
(virtual strategy)
**Hoodi Testnet:**
- `0xf8a1a66130d614c7360e868576d5e59203475fe0` - stETH
- `0x24579aD4fe83aC53546E5c2D3dF5F85D6383420d` - WETH
**Ethereum Mainnet:**
- `0x93c4b944D05dfe6df7645A86cd2206016c51564D` - stETH
- `0x1BeE69b7dFFfA4E2d53C2a2Df135C388AD25dCD2` - rETH
- `0x54945180dB7943c0ed0FEE7EdaB2Bd24620256bc` - cbETH
### Config Files
- `stagenet-hoodi.json` - Hoodi testnet with stagenet EigenLayer
addresses
- `testnet-hoodi.json` - Hoodi testnet with testnet EigenLayer addresses
- `mainnet-ethereum.json` - Ethereum mainnet with mainnet EigenLayer
addresses
- Removed `hoodi.json` (replaced by environment-prefixed files)
## Usage
```bash
# Deploy to stagenet on Hoodi
bun cli contracts deploy --chain hoodi --environment stagenet
# Update BEEFY checkpoint from live chain
bun cli contracts update-beefy-checkpoint \
--chain hoodi \
--environment stagenet \
--rpc-url wss://services.datahaven-dev.network/stagenet
# Compute rewards origin for a chain
bun cli contracts update-rewards-origin \
--chain hoodi \
--environment stagenet \
--rpc-url wss://services.datahaven-dev.network/stagenet
# Check deployment status
bun cli contracts status --chain hoodi --environment stagenet
```
---------
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
## Summary
Reorganizes the test directory structure for better clarity and
maintainability:
- **Rename `test/datahaven/` → `test/moonwall/`**: Clearly identifies
these as Moonwall single-node tests
- **Move `test/framework/` → `test/e2e/framework/`**: Groups e2e test
utilities under a dedicated folder
- **Move `test/suites/` → `test/e2e/suites/`**: Groups e2e test suites
with the framework
- **Add `test/e2e/framework/validators.ts`**: Extracts validator test
helpers from utils into the e2e framework
- **Update documentation**: README.md and E2E_FRAMEWORK_OVERVIEW.md
reflect the new structure
### New Directory Structure
```
test/
├── e2e/
│ ├── suites/ # E2E test suites (Kurtosis-based)
│ └── framework/ # E2E test utilities & helpers
├── moonwall/ # Moonwall single-node tests
├── launcher/ # Network deployment tools
└── ...
```
---------
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
## Summary
This PR improve the generating state workflow. It will also check for
outdated state-diff.json and add a practical script to easily generate a
new one.
The way we generate state has also been changed to make it work with
macOS M1 system. We don't run the tool in the container anymore but
instead directly on the machine.
## What changes
* A check-generated-state.js script was added to quickly look for
outdated test
* The check was added in the CI
* A generate-contracts.ts script was added to easily generate the new
state with the new instructions to run on MacOS
---------
Co-authored-by: Gonza Montiel <gon.montiel@gmail.com>
Co-authored-by: Ahmad Kaouk <56095276+ahmadkaouk@users.noreply.github.com>
Co-authored-by: Steve Degosserie <723552+stiiifff@users.noreply.github.com>
Co-authored-by: Gonza Montiel <gonzamontiel@users.noreply.github.com>
## Summary
This PR remove the middlewares contracts from eigen layer. Instead we
are planning to use the eigne layer contract directly. It also removes
the tests related to the middleware slasher code and the mock contract
used in it.
## Motivation
When slashing an operator in the Dathaven we are going through the
substrate slashing pallet already implemented. It already allow to
configure a slashing window and/or to cancel a slashing. In the future
it will also be compatible with a government pallet. This part of code
is therefore redundant. For the same reason we remove the tests because
we are not using the slashing middleware contracts.
## What changed
* Remove the slasher middleware files
* Remove the tests related to the middleware slasher file
This PR significantly refactors and improves the end-to-end testing
framework and infrastructure. The primary focus was on simplifying the
test suites, improving reliability through better resource management,
and hardening the relayer infrastructure.
All E2E tests are now passing on the CI and demonstrate consistent
reliability when run locally.
### Key Changes
#### 1. E2E Test Suite Refactor & Cleanup
* **Simplified Test Logic**: Heavily refactored the core test suites
(`native-token-transfer.test.ts`, `rewards-message.test.ts`, and
`validator-set-update.test.ts`). The new implementation is much cleaner,
utilizing shared helpers to reduce boilerplate.
* **Utility Consolidation**: Removed redundant utility files
(`storage.ts`, `rewards-helpers.ts`) and simplified `events.ts`. Event
waiting now uses `rxjs` for Substrate and native `viem` watchers for
Ethereum, which is more robust and easier to maintain.
* **Better Connector Management**: Unified the creation and cleanup of
test clients in `ConnectorFactory`. It now handles the lifecycle of
WebSocket connections more gracefully, including clearing the
`socketClientCache` to prevent reconnection noise during teardown.
#### 2. Infrastructure & Stability
* **Relayer Relaunch Policy**: Added a restart policy for Snowbridge
relayer containers. They are now configured with `--restart
on-failure:5`, ensuring that relayers automatically relaunch if they
crash during the sensitive initialization phase.
* **WebSocket Integration**:
* Updated the `ConnectorFactory` to prefer **WebSockets** for the
Ethereum public client, which is essential for efficient, event-heavy
E2E testing.
* Enhanced `launchKurtosisNetwork` to correctly identify and register
the Execution Layer's WebSocket endpoint from Kurtosis.
* **Disabled Contract Injection**: This PR temporarily disables the
automatic injection of contracts into the genesis state by default.
* *Reason*: I encountered issues generating a valid `state-diff.json`
for the latest contract versions. Even after applying several
workarounds, the injected state remained unstable. As a result, I've
reverted to manual contract deployment during the launch sequence for
better reliability for now.
#### 3. Documentation & Maintenance
* Removed obsolete documentation (`event-utilities-guide.md`) that no
longer reflects the simplified event-handling API.
* Cleaned up `test/launcher/validators.ts` and moved logic into more
appropriate helpers.
---------
Co-authored-by: Steve Degosserie <723552+stiiifff@users.noreply.github.com>
## Summary
Optimizes the DataHaven parameter configuration during E2E test
infrastructure setup by batching multiple extrinsics into a single
transaction, reducing setup time.
## Problem
Setting runtime parameters required 5 separate
`Parameters.set_parameter` calls, each waiting for block finality. This
created unnecessary delays during infrastructure setup since each call
blocked sequentially.
## Solution
- **Batch parameter updates:** Combine all `Parameters.set_parameter
`calls into a single `Utility.batch_all` transaction wrapped in
`Sudo.sudo`.
- **~5× faster parameter setup:** Only wait for finality once instead of
5 separate times
- **Code simplification:** Refactored parameter handling code, removing
~190 lines of unnecessary abstractions and complexity
### Context
The function `v2_sendMessage()` on Snowbridge Gateway contract is
**permissionless** (I'm shocked this is the design choice). Any
EOA/contract on Ethereum can build a message and send it through our DH
bridge. While we don't change our Snowbridge fork, then this will
continue to be the case.
### Problem
We use `v2_sendMessage()` to send **permissioned** operations to our
chain. For instance: update our validator set message (coming next,
_slashing-related_ messages). So we do need to restrict the processing
of the incoming messages on the Substrate side.
### Fix
- I've added a check to `EigenLayerMessageProcessor` that enforces
`message.origin` to be only a configured `AuthorisedOrigin`.
- I've added an `AuthorisedOrigin` to
`pallet_external_validators::Config`
- I've configured the `AuthorisedOrigin` to be
`DatahavenServiceManagerAddress` in all three runtimes
### Stages
- [x] Implementation
- [x] Runtime integration tests
- [x] Collect `DatahavenServiceManagerAddress` parameter for e2e tests
to work
Fixes https://github.com/datahaven-xyz/sr-datahaven/issues/12
---------
Co-authored-by: Steve Degosserie <723552+stiiifff@users.noreply.github.com>
# Enable AVS owner workflow
Until now, the deployer of the contracts and the owner of the deployed
contracts where the same account. Even if we allowed a different owner
to be specified, we were using the same. For this reason, a private key
was required, so after the deployment we could execute owned
transactions needed for the CLI.
In this PR we:
- Add a mechanism to the CLI to specify a different owner account other
than the deployer via `--avs-owner-address`
- Add CLI flags `--avs-owner-key` and`--execute-owner-transactions` so
account ownership vs. immediate execution is explicit and deferred. If
both previous parameters are provided, the CLI will execute the
transactions using the private key provided.
- Allow DataHaven AVS deploy scripts to toggle owner-call execution via
an env flag `TX_EXECUTION`
- Add documentation on how the new parameters work in `test/README.md`
and `test/docs/deployment.md`.
---------
Co-authored-by: Steve Degosserie <723552+stiiifff@users.noreply.github.com>
Co-authored-by: Ahmad Kaouk <56095276+ahmadkaouk@users.noreply.github.com>
## Summary
This PR fixes several issues with the CLI deploy-contracts command to
properly support local Anvil deployments and improves the overall
contract deployment workflow.
### Key fixes:
- Add support for anvil chain in the CLI deploy contracts command
- Rename PRIVATE_KEY to DEPLOYER_PRIVATE_KEY for consistency and clarity
across the deployment flow
- Fix EigenLayer contract status display for local/anvil chains by
reading addresses from the deployments file instead of config
- Fix runShellCommandWithLogger to properly throw errors on command
failure
- Correct totalSteps in DeployTestnet.s.sol from 2 to 4
### Housekeeping:
- Update .gitignore to ignore the entire broadcast/ folder
(autogenerated Foundry artifacts)
- Streamline contracts/README.md with clearer structure and deployment
instructions
Apparently, EigenLayer started including mock files inside
`test/mocks/*`, that weren't in our remappings. That's why the e2e
contract building was failing, both for e2e tests and CLI launch, while
trying to solve `src/test/mocks/AllocationManagerMock.sol` in our own
contracts source. I added the necessary remappings and the tests run
now.
## Add StorageHub nodes to CLI
This PR adds StorageHub node infrastructure to the CLI and fixes CLI
flag handling and improves the CLI logic a bit.
### Fix: CLI safeguards
- Prevents contract deployment and validator operations when `--ndc`
flag is used
- Skips Kurtosis service display in summary when `--nlk` flag is used
### Feat: Dockerized Storage Hub Nodes
- Adds 5 Docker containers: PostgreSQL, MSP, BSP, Indexer, and Fisherman
nodes
- New CLI flags: `--storagehub` `--no-storagehub` to control StorageHub
nodes launch
- Automatic provider funding and registration (Charleth for MSP and
Dorothy for BSP)
- Exposes nodes on ports 9945-9948 for local development
**TODO**
- [x] MSP & BSP associated pre-funded account.
- [x] Call `forceMspSignUp` and `forceBspSignUp` extrinsics
---------
Co-authored-by: Steve Degosserie <723552+stiiifff@users.noreply.github.com>
In this PR, we introduce a way to save Ethereum state into a file. This
saved state can then be injected into Ethereum to speed up e2e initial
test setup.
This is a rewrite of the now closed PR
https://github.com/datahaven-xyz/datahaven/pull/90 .
It uses a an external tool written in rust to save state from the
Ethereum running container : https://github.com/undercover-cactus/Chaos
---------
Co-authored-by: Gonza Montiel <gonzamontiel@users.noreply.github.com>
Co-authored-by: Steve Degosserie <723552+stiiifff@users.noreply.github.com>
This PR restructures and port the block validation test suite from
Moonbeam and add the necessary infrastructure for contract-based
testing.
### Test Suites Added
**Block Validation Suite 1** (`test-block-1.ts`) - *Refactored from
original `test-block.ts`*
- T01: Validates block number increments correctly after manual block
creation
- T02: Checks block timestamps are valid and within expected bounds
- T03: Verifies complete block structure including gasLimit, difficulty,
receiptsRoot, transactionsRoot, logsBloom, and other Ethereum-compatible
fields
- T04: Confirms blocks are retrievable by hash
- T05: Confirms blocks are retrievable by number
**Block Validation Suite 2** (`test-block-2.ts`) - *New*
- T01: Verifies block number persistence across test cases
- T02: Validates parent-child block hash linkage in the chain
**Block Gas Limits Suite** (`test-block-gas.ts`) - *New*
- T01-T06: Tests all three transaction types (legacy, eip1559, eip2930)
can deploy contracts at max extrinsic gas limit, and correctly reject
transactions exceeding that limit
- T07: Deploys `BlockVariables` contract and verifies runtime gas limit
is accessible from within contract execution
**Block Genesis Suite** (`test-block-genesis.ts`) - *New*
- T01: Validates genesis block (block 0) contains correct
Ethereum-compatible structure and empty transaction/uncle lists
- T02: Confirms genesis block is retrievable by hash
### Infrastructure Additions
**Contract Deployment Helpers**
- `fetchCompiledContract`: Loads compiled Solidity artifacts with ABI
and bytecode from JSON output
- `deployCompiledContract`: Handles contract deployment with support for
legacy, EIP-1559, and EIP-2930 transaction types, including gas
parameter configuration
**Solidity Test Fixtures**
- `BlockVariables.sol`: Exposes `block.gaslimit`, `block.chainid`,
`block.number` via view functions for runtime validation
- `Fibonacci.sol`: Provides `fib2(n)` pure function for computational
gas testing
- `MultiplyBy7.sol`: Minimal pure function contract for basic deployment
testing
**Enhanced Runtime Constants**
- Added `EXTRINSIC_GAS_LIMIT` (52M), `BLOCK_WEIGHT_LIMIT` (2T),
`MAX_POV_SIZE` (10MB) for all three environments
- Constants derived from and documented against Rust runtime
configuration in `operator/runtime/*/src/lib.rs`
### Dependencies
- Added `solc@0.8.30` and supporting packages (`yargs`,
`command-exists`, `js-sha3`, `memorystream`) for local Solidity
compilation
## Add E2E validator-set update flow
- feat: `test/utils/validators.ts` for on-demand validator
orchestration.
- feat: `test/suites/validator-set-update.test.ts` covering allowlist →
register → update.
- some minor launcher updates: avoid docker cache, add `--platform` when
building datahaven image, avoid sending validator-set update on launch.
- Helpers: ABI shortcut in `test/utils/contracts.ts`; config tweaks in
`test/configs/validator-set.json`.
- Minor cleanup/formatting across `test/launcher/*`,
`test/scripts/setup-validators.ts`, and related tests.
- added `keepAlive` flag to `BaseTestSuite`, in order to avoid tearing
down the network while debugging. Defaults, obviously, to false.
- added a `failOnTomeout` option on to waitForDataHavenEvents() so the
test fails of the timeout is reached and no event was captured.
### Coverage
- The test simulates an scenario in which we have two active authorities
(alice and bob), which are running, and registered as operators, which
is the normal state after the chain launches. Then:
- It launches two more nodes (charlie and dave)
- It add the nodes to allowlist and register them as operators
- It sends the validator set update message
- Checks that the validator update message was propagated through the
gateway and arrived the external-validators pallet
- Checks that the chain continues producing blocks
### Notes
The last test case has a timeout of 10 minutes. This is to respect
propagation times of the message through the relayers. We are testing
that the external validators pallet actually updated the validator set.
Locally, I could expect 5~6 minutes, I just wanted to be on the safe
side. CI is passing showing that this was enough indeed.
---------
Co-authored-by: Steve Degosserie <723552+stiiifff@users.noreply.github.com>
Co-authored-by: Ahmad Kaouk <56095276+ahmadkaouk@users.noreply.github.com>
## Summary
This PR introduces support for deploying Datahaven contracts to
different chains (hoodi, holesky, mainnet), as well as a new cli command
to manage this deployment separately from the regular deployment, while
maintaining compatibility with it.
#### New CLI command
- **`bun cli contracts deploy`** - Deploy contracts to supported chains
(Hoodi, Holesky, Mainnet)
- **`bun cli contracts status`** - Check deployment configuration and
status
- **`bun cli contracts verify`** - Verify contracts on block explorers
- Commands need the chain parameter: `--chain <hoodi | holesky |
mainnet>`
- Right now only `hoodi` and `holesky` are supported
### Deployment
#### Hoodi & Holesky Network Support
- Added **DeployBase.s.sol** as common ground for
**DeployTestnet.s.sol** (also new) and **DeployLocal.s.sol** (existing).
- **Hoodi configuration** (`contracts/config/hoodi.json`) with deployed
EigenLayer contract addresses to reference.
- **Holesky configuration** (`contracts/config/hoodi.json`) with
deployed EigenLayer contract addresses to reference.
#### Contracts being deployed
- **DataHaven**: ServiceManager, VetoableSlasher, RewardsRegistry
- **Snowbridge**: BeefyClient, AgentExecutor, Gateway, RewardsAgent
- **EigenLayer**: References existing deployed contracts (not
re-deployed)
#### Deployment files
When the deployment is done, a new file under `contracts/deployments` is
generated with the addresses of the deployed contracts, for each chain
(it will be overriden per chain if run multiple times). So we would have
one `anvil.json`, `hoodi.json`, `holesky.json`, etc, with the addresses
of the deployed contracts for reference and for later verification.
#### Todo
- [x] Test compatibility with existing `bun cli launch` and `bun cli
deploy` commands
#### For follow-up PRs
- Fix verification issue with `foundry verify-contracts` when specifying
the `chain` or `chain-id` parameter, needed for hoodi
(https://github.com/foundry-rs/foundry/issues/7466).
- Add `redeploy` feature to only override implementation contract and
leave the proxy address untouched
## Usage Examples
```bash
# Deploy to Hoodi network
bun cli contracts deploy --chain hoodi
# Check deployment status
bun cli contracts status --chain hoodi
# Verify contracts on block explorer
bun cli contracts verify --chain hoodi
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added deployment and configuration support for new networks "hoodi"
and "holesky", including new configuration and deployment files.
* Introduced a CLI tool for managing contract deployments, status
checks, and verification across supported chains.
* Added example environment configuration and comprehensive deployment
documentation.
* Enabled contract verification and status reporting via the CLI with
support for block explorer integration.
* **Improvements**
* Refactored deployment scripts for modularity, supporting both local
and testnet environments.
* Centralized and extended configuration loading to support additional
contract addresses and network parameters.
* Enhanced deployment utilities and typings to support multi-network
deployments.
* **Bug Fixes**
* Improved input validation and error handling in CLI commands and
deployment scripts.
* Added explicit handling for zero address in operator strategy
retrieval.
* **Chores**
* Updated documentation and configuration templates for easier
onboarding and deployment management.
* Improved logging and output formatting for deployment and verification
processes.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Steve Degosserie <723552+stiiifff@users.noreply.github.com>
Co-authored-by: Ahmad Kaouk <56095276+ahmadkaouk@users.noreply.github.com>
## Implement E2E Testing Framework with Isolated Networks
### Summary
Refactors the existing E2E testing infrastructure to provide isolated
test environments with parallel execution support. Each test suite now
runs in its own network namespace, preventing resource conflicts.
### Key Changes
- **New Testing Framework** (`test/framework/`): Base classes for test
lifecycle management with automatic setup/teardown
- **Launcher Module** (`test/launcher/`): Extracted network
orchestration logic from CLI handlers for reusability
- **Parallel Execution**: Added `test-parallel.ts` script with
concurrency limits to prevent resource exhaustion
- **Test Isolation**: Each suite gets unique network IDs (format:
`suiteName-timestamp`) and Docker networks
- **Improved Test Organization**: Migrated tests to new framework,
deprecated old test structure
### Test Improvements
- Added 4 new test suites demonstrating framework usage. :
- `contracts.test.ts` - Smart contract deployment/interaction
- `datahaven-substrate.test.ts` - Substrate API operations
- `cross-chain.test.ts` - Snowbridge cross-chain messaging
- `ethereum-basic.test.ts` - Ethereum network operations
> [!WARNING]
The test suites themselves are bad and shouldn't be consider examples of
good tests. They were AI generated just to test the concurrency of test
runners
### Documentation
- Added comprehensive framework overview (`E2E_FRAMEWORK_OVERVIEW.md`)
- Updated README with parallel testing commands
- Added test patterns and best practices
### Breaking Changes
- Old test suites moved to `e2e - DEPRECATED/` directory
- Test execution now requires extending `BaseTestSuite` class
### Testing
Run tests with: `bun test:e2e` or `bun test:e2e:parallel` (with
concurrency limits)
### TODO
- [ ] Implement good test examples.
- [ ] Implement useful test utils (like waiting for an event to show up
in DataHaven or Ethereum).
- [ ] Enforce tests with CI (currently cannot be done due to
intermittent error when sending a transaction with PAPI).
---------
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: undercover-cactus <lola@moonsonglabs.com>
This PR improves the CLI to get from the deployments the
`RewardsRegistryAddress` (address of the RewardsRegistry contract
deployed), `RewardsAgentOrigin` (origin used for the agent in charge of
updating the rewards merkle root in the RewardsRegistry contract) and
`RewardsUpdateSelector` (function selector of the function that the
agent must execute to do the aforementioned update) and then set these
values in the `parameters` pallet of the runtime.
After these changes the rewards merkle root is being updated on the
Ethereum side. 🎉
### New Features
1. Add the `deploy` command to our CLI.
1. Conditionally deploys kurtosis eth network if we're in `stagenet`
environment.
2. Deploys DH nodes.
3. Deploys contracts (all of them). In `mainnet` and `testnet` it
shouldn't deploy EL contracts, but for now that's not implemented.
4. Configures parameters, validators and relayers in the same way as
`launch`.
5. Currently, it only deploys `beefy` and `beacon` relayers, `execution`
and `solochain` relayers are pending for a subsequent PR.
2. Add `waitFor` utility function that receives a lambda.
### Refactors
1. Several common functionalities used both by the `launch` and `deploy`
command have been moved to the `cli/handlers/common` directory, from
where both commands use them. These include
1. Checks for installed dependencies.
2. Common constants.
3. The `LaunchedNetwork` class has been moved to this directory.
4. DataHaven nodes common functions.
5. Kurtosis common functions.
6. Relayer common functions.
7. Kubernetes functions (although only used by `deploy`, it seemed
fitting to have it here still).
8. Remove CLI questions and separator prints from `deploy-contracts.ts`
and `set-datahaven-parameters.ts` scripts. These things should be in the
`cli/launch` folder, which consumes the functions in these scripts.
9. Remove `setParametersFromCollection` from `utils` folder and put it
in `cli`.
10. Create base snowbridge relayer configs for `local` and `stagenet` as
two separate directories.
### Fixes
1. Sets the default time of the `deploy` command to 6s as Lodestar is
slower than Lighthouse.
2. In `runShellCommandWithLogger` only print `stderr` if the command
fails.
### Additional Minor Changes
1. K8s secret key names changed from `dh-beefy-relay-eth-key` to
`dh-beefy-relay-ethereum-key` and `dh-beacon-relay-sub-key` to
`dh-beacon-relay-substrate-key`, for simplicity in the deployment
script.
11. Update suggested configs for `.vscode` configs.
---------
Co-authored-by: Steve Degosserie <723552+stiiifff@users.noreply.github.com>
This PR:
- Adds launching the new `solochain` relayer in the relayers' script,
with its config files and schemas.
- Updates the relayer launching to use switch-case logic since we are
now going to be running 4 relayers, making it cleaner.
- Minor CLI fixes (adding additional args to build command for Linux,
improving naming, deleting idle containers instead of only active ones).
- Deletes unused files `substrate-relay.json` (now
`solochain-relay.json`), `gen-snowbridge-cfgs.ts` and
`snowbridge-relayer.ts` (now Snowbridge binary and docker image
generation can be done exclusively from our [Snowbridge
repo](https://github.com/Moonsong-Labs/snowbridge))
- Updates the `UniversalLocation` of our stagenet runtime to be under
the global consensus for XCM instead of actually being the global
consensus. This makes it so we can actually use the Snowbridge System
pallets to queue up messages. For mainnet we are going to want to have
our own Network ID instead of using Polkadot's.
> [!WARNING]
> ~~All in all these changes allows us to run the solochain relayer, but
it won't work without the refactor that's being worked on in
https://github.com/Moonsong-Labs/snowbridge/pull/14. I'd advise waiting
for that PR to be merged before merging this one.~~ MERGED ✅
## Changes
- New CI file for making Docker Prod images
- Changed E2E tests use an image built from a local dockerfile
- Some cargo build options to make it quicker
- Fix the cache hit rate
- added `tsgo` preview to the project 😎
- Can be invoked with `bun tsgo` to typecheck
- Install in IDE
[VSCode](https://code.visualstudio.com/docs/configure/extensions/extension-marketplace)
& [Zed](https://github.com/zed-extensions/tsgo) for super-fast inline
typechecking (as you type basically)
## Context
This PR attempts to make the frankly unacceptable CI times better. This
achieves that aim by making a crappy image for day-to-day usage and let
the prod issue take ages since that will be infrequently used. The
reason why the original design didn't work for us is because: 1) we are
using the free GH runners 2) when we goto baremetal runners we'll lose
our rapid caching abilities which make using docker cheap.
Also, we add `tsgo` support to improve devex. The improvement is
astounding.
```sh
hyperfine -n tsc "bun tsc --incremental false --extendedDiagnostics" -n tsgo "bun tsgo --incremental false --extendedDiagnostics"
Benchmark 1: tsc
Time (mean ± σ): 5.500 s ± 0.221 s [User: 8.939 s, System: 0.400 s]
Range (min … max): 5.196 s … 5.845 s 10 runs
Benchmark 2: tsgo
Time (mean ± σ): 99.1 ms ± 8.4 ms [User: 392.8 ms, System: 54.1 ms]
Range (min … max): 88.3 ms … 116.0 ms 29 runs
Summary
tsgo ran
55.48 ± 5.22 times faster than tsc
```
This PRs extracts [this
commit](74c0c0be0a)
from @TDemeco's PR to add a way to include parameters as part of the
CLI.
### Key changes
- CLI tool to set DataHaven runtime parameters via JSON configuration
- Supports both interactive prompts and command-line flags
- Type-safe parameter parsing and validation
- Already adds the parameters for the `EthereumGatewayAddress`, that
otherwise we would need to add manually to the node using an explorer.
---------
Co-authored-by: TDemeco <tdemeco@itba.edu.ar>
## Changes
- New option: `--kurtosis-enclave-name` to allow you to specify a new
ethereum network with a different enclave name. Neccesary step for
setting up local testing with multiple enclaves running at once
- Refactor: all single dash options (e.g. `-i`) have been renamed to
have double dashes `--` for consistency others
- sad times: CLI now must be invoked `bun cli launch` since we have
multiple fns now
- Package Update
- New Function: `stop` which allows you to kill all components (eth, dh,
relayers) or only a single part
- Gonza's mac target build fix
- Misc fixes to ci like caching and rate limits
## Additional Comment
The CLI needs multiple commands and this PR adds the first new one
`stop`.
This syntax is faily extensible so @ffarall 's k8 command can follow
suite.
Originally we were going to have an `exec` command to expose internal
fns to be callable by command line, i.e. generate beacon checkpoint -
however the need for it has since been superceded by the deploy to k8
command that's going to be added. I've left the code in place though so
when another usecase comes up we can just plug that in.
---------
Co-authored-by: Facundo Farall <37149322+ffarall@users.noreply.github.com>
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Documentation**
- Added detailed IDE configuration recommendations for Rust, Solidity,
and TypeScript in the README to enhance developer experience.
- **Chores**
- Updated Biome configuration files and package dependencies to the
latest schema and version.
- Refined code formatting, linting, and import organization settings for
consistency across the project.
- **Refactor**
- Reordered import statements in multiple files for improved
readability.
- Simplified function signatures and ensured proper async handling in
utility scripts.
- **Bug Fixes**
- Ensured proper completion of asynchronous operations in shell utility
functions.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Gonza Montiel <gonzamontiel@users.noreply.github.com>
## Changes
- Latest changes to have working relayer 🎉 component
- Changed spawning snowbridge relayers to docker containers
- Small logging output changes
- Refactoring to `LaunchedNetwork` class
- new flag `--bd` `--build-datahaven` which will build a local docker
container which is **much** quicker than the proper CI build (which uses
a controlled build enviroment)
- new bun script `start:e2e:local`, which is everything that
`start:e2e:ci` has, but with building local docker container and
log_level debug set
---
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Summary by CodeRabbit
- **New Features**
- Added support for launching and managing relayer and DataHaven
services using Docker containers and networks.
- Introduced a CLI option to specify the relayer Docker image tag
instead of a binary path.
- **Improvements**
- Enhanced log messages with clearer text and expressive emojis for
better user feedback.
- Improved summary display by removing relayer services from the output.
- Updated build scripts to consistently enable the "fast-runtime"
feature for cross-platform builds.
- Refined validation and error reporting for checkpoint data parsing.
- **Bug Fixes**
- Improved Docker container cleanup and network management during
service launch and teardown.
- **Chores**
- Updated and refactored npm scripts for Docker operations and
end-to-end test cleanup.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Facundo Farall <37149322+ffarall@users.noreply.github.com>
In this PR:
1. Add new `datahaven-node-local.dockerfile` for building a local image
with the locally built DataHaven Node binary. This severely improves
iteration speed on running `bun cli` if there are changes in the DH
node. Previously, it relied on the published dockerfile, which builds
the Cargo project inside of it, taking +20m even with no changes.
2. Building this local dockerfile is integrated to the CLI, which now
also asks if the user wants to rebuild the local docker image of the DH
node.
3. A new script `cargo-crossbuild` is added, to be able to build the
DataHaven node Cargo project both from Mac and Linux, with the target
being `x86_64-unknown-linux-gnu`. For building from Mac, it uses `cargo
zigbuild`, so `zig` is now a dependency. Building for this target is
needed because the docker image is an Ubuntu image, so it will need to
run a linux binary.
4. Added `zig` as dependency in docs.
5. CI still uses the docker image built by the CI itself, which builds
the Cargo project inside of it. The CI can take advantage of caching for
this.
> [!NOTE]
> This is `Part 3` of the ongoing _Docker Series._
## New Additions:
- Launching Datahaven network will spin up containers, as opposed to
native binaries
- `stop:docker` script to kill all dh containers
- `e2e` test suite for datahaven solochain network
- Contains reference test file that uses papi for storage queries,
submitting exts, runtime calls (good job on that facu and tobi)
- Added new utils:
- `waitForLog()` to wait for log lines in docker container logs
- `createPapiConnectors()` helper for test cases to build and connect to
dh network
- `getPapiSigner()` helper to return a papi compatible signer using our
prefunded accounts (alith by default)
- `sendTxn()` helper to submit txn and wait for block inclusion, instead
of finalization, which std library provides
## Changes:
> [!CAUTION]
> Launching native binaries for datahaven no longer supported.
- Datahaven binary location cli option changed to `-i,
--datahaven-image-tag`
- To locally run this you'll need a datahaven docker image handy, you'll
need to either:
- Point to remote dockerhub e.g. `moonsonglabs/datahaven:main` (must be
logged in and have permission)
- Build this locally with `bun build:docker:operator`
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Added end-to-end tests for the Datahaven solochain, including runtime
API queries, storage lookups, extrinsic submissions, and event
listening.
- Introduced CLI option to specify the Datahaven Docker image tag, with
a default value.
- Added CLI option to disable the Relayer.
- Provided new scripts to stop Docker containers associated with
Datahaven.
- Added utility functions for Docker log monitoring and container
startup checks.
- Introduced utilities for interacting with the Datahaven Polkadot API.
- **Improvements**
- Switched Datahaven network launch from local binaries to Docker
containers.
- Enhanced cache accuracy in build workflows by including Rust source
files in cache keys.
- Improved build performance with TypeScript incremental build options.
- Increased timeout for end-to-end tests for better reliability.
- Updated CLI version to 0.2.0.
- Modified Dockerfile build to enable the `fast-runtime` feature.
- Extended network launch summary to include relayer and container
details.
- **Bug Fixes**
- Fixed cleanup logic by tracking and preparing for forced removal of
Docker containers after tests.
- **Chores**
- Updated workflow steps for Docker image handling and network checks.
- Adjusted scripts and workflow logic for improved Docker and test
management.
- Removed top-level disk usage summaries from cleanup workflow for
streamlined reporting.
- Enhanced shell command utility to support asynchronous wait during
execution.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Facundo Farall <37149322+ffarall@users.noreply.github.com>
In this PR:
1. Add [Polkadot API](https://papi.how/) support to `test/` directory
(i.e. the e2e CLI and testing framework) to be able to interact with the
Substrate chain.
1. This allows typed interactions with transactions, constants, storage,
runtime APIs, and non-typed interaction with RPC methods.
2. Types are autogenerated when running `bun i`, from the
`datahaven.scale` file that is part of this repo's version control.
2. Add new utilities file to `papi` related functionalities. For the
time being, generating a new signer from a private key.
3. ~Add a new step to the CLI that sends a transaction to the DataHaven
network. _*THIS SHOULD BE REMOVED SOON, IT'S JUST FOR TESTING
PURPOSES*_~
1. Both steps that send test transactions have been removed from the
CLI, for convenience and being error prone. Their scripts remain usable
for testing purposes if needed.
4. Removes the `apis.rs` files from the runtime definitions. Having them
in a separate file meant that the runtime was not including the Runtime
APIs in the metadata blob, preventing `papi` from creating types from
it. This change can be reapplied after upgrading to `polkadot-sdk`
`stable-2503`. More info
[here](https://github.com/paritytech/polkadot-sdk/issues/6659).
5. Add script to re-generate types, and corresponding docs.
6. Makes logger synchronous to avoid prints happening before logs.
This PR:
1. Generally improves the logging of the testing CLI, making the logs
more concise and easier to follow, with clearer sections and
separations.
2. Launches DataHaven solochain nodes at the beginning not the end.
3. Prompts the user if they want to launch DataHaven nodes and
Snowbridge Relayers.
---------
Co-authored-by: Tim B <79199034+timbrinded@users.noreply.github.com>
Eventually our CI will be required to run two private blockchains
locally plus associated relayers.
This PR is to prepare for this fate by improving run times and
refactoring our existing CIs so they are a bit easier to reason about.
### Refactors
- **_We now run ALL CIs on every PR!_** This is so that we decomplexify
the logic around conditional builds and fetching built binaries from
another source. This reduces the surface area of code we have to
maintain at the cost of execution time
- This penalty is ameliorated by a layered caching system. At best, it
will be less than a minute to complete a build since everything will be
cached. On GH runners this is about 6 minutes sadly.
- We will no longer be at risk of important CIs being skipped
erroneously which hide true failures.
- Caching is a low-risk approach because at worst it has to build from
scratch. A bad cache hit will never imply the wrong thing gets build
since cargo is smart enough to just throw away any inappropriate build
artefacts.
- `setup-rust` action created so we have a unified way of setting up
runner and unifying our approach to caching
- Use a unique caching key for different activities and it will fallback
to shared cache if no matches
- we are using `mainnet` kurtosis config so that it works with relayer
assumptions
### Additions
- We can specify the ethereum block time via a new cli arg `--slot-time
<seconds>`
- We can specify arbitrary network_param args which get passed into the
generated yaml
- e.g. giving `bun cli --kurtosis-network-args="pet=cat food=fish" will
add:
```yml
network_params:
# existing params...
pet: cat
food: fish
```
- We now have the ability to programmatically modify the yaml
- This means we are back down to a single `minimal.yml` kurtosis config
so we dont have to maintain changes between them
- Flow is: `add new cli arg` -> `add if() block which mutates yaml` ->
`profit`
---------
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Facundo Farall <37149322+ffarall@users.noreply.github.com>
## Summary
This PR adds statically typed bindings for contracts. This allows you to
write E2E tests with full completions in TS.
## Additions
- `ts-build.yml` New CI, this will make sure that if there's changes
made to the contracts that the contract-bindings are up to date.
- `package.json` script changes
- `start:e2e:ci` - Designed to be run with all options specified since
CIs are famously bad with iteractive CLI prompts
- `test:e2e` - added timeout
- `generate:wagmi` - This generates the smart contract bindings for our
tests
- New Function Helpers:
- `generateRandomAccount()` Returns a viem account type object for a
random account. Useful for tests where you want idempotency on a long
lived network since the state is probabilistically fresh
- `getContractInstance()` Returns a viem contract instance that allows
you to read/write to the deployed contract. You should get full type
inference here for the methods available and parameters required.
### Example
```ts
it("avs() can be read from contract instance", async () => {
const value = await instance.read.avs();
expect(isAddress(value), "AVS getter should return an address").toBeTrue();
});
```
---------
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Facundo Farall <37149322+ffarall@users.noreply.github.com>
## Human Written Description
This PR adds the following to the E2E CLI:
- Relayer config generation for: `beacon-relay` `beefy-relay`
- The other two relayer types to be added later
- Relayers don't actually work yet
- By default turned off, this requires a binary to be present in:
`<repo_root>/operator/target/release` dir
- Datahaven network launching
- DH network is using default `local` network chain spec
- Launched with 5 nodes since our authority set is 6 large (and you need
2/3 + 1 of set size
> [!NOTE]
> Both the relayer and the DH node binaries are being run as local
processes TEMPORARILY. This means that logging is done in a very
rudimentary way (we pipe to a file whilst the CLI is running).
>
> This means that when the CLI finishes **the log files will no longer
be written to**.
> This is temporary since spawning binaries is a stop gap solution until
docker images available.
---
> [!IMPORTANT]
> The following is AI generated slop describing this PR's changes:
**Key Changes:**
* **CLI Enhancements (`test/cli/index.ts`):**
* Added options `--datahaven` and `--datahaven-bin-path` to enable
launching local DataHaven nodes.
* Added options `--relayer` and `--relayer-bin-path` to enable launching
Snowbridge relayers (Beefy and Beacon).
* Added negation flags (`--no-fund-validators`, `--no-setup-validators`,
`--no-update-validator-set`) for more granular control over validator
setup steps.
* Added `--skip-cleaning` option to preserve Kurtosis state between
runs.
* Added a pre-action hook (`launchPreActionHook`) to validate flag
combinations (e.g., `--verified` requires `--blockscout`).
* **New CLI Handlers (`test/cli/handlers/launch/`):**
* `datahaven.ts`: Logic for spawning DataHaven node processes using the
specified binary. Manages ports and process cleanup.
* `relayer.ts`: Logic for configuring and spawning Snowbridge relayer
processes (Beefy and Beacon). Reads contract deployment addresses,
updates relayer config templates, and uses specified private keys.
Manages log files and process cleanup.
* `summary.ts`: Generates and displays the table of running services
(including dynamically launched DataHaven nodes) and their endpoints.
* `validator.ts`: Extracted validator funding, setup, and set update
logic into its own handler.
* `index.ts`: Orchestrates the launch sequence based on CLI options,
calling the appropriate handlers. Includes a `LaunchedNetwork` class to
track spawned processes, file descriptors, and node ports for cleanup.
* **Updated `package.json` Scripts:**
* Added `start:e2e:minrelayer` script for a minimal setup including
relayers and DataHaven nodes.
* Modified `stop:e2e` to include `pkill datahaven` for proper cleanup.
* Added `stop:e2e:quick` to only stop the Kurtosis enclave without full
cleaning.
* **Updated `launch-kurtosis.ts`:** Modified to use new Kurtosis utility
functions and added a `skipCleaning` option.
* **New Utility Functions:**
* `test/utils/kurtosis.ts`: Functions to inspect Kurtosis services
(`getServiceFromKurtosis`, `getPortFromKurtosis`,
`getServicesFromKurtosis`).
* `test/utils/parser.ts`: Zod schemas and parsing functions for
Snowbridge relayer configurations.
* **Constants & Minor Updates:**
* Added `SUBSTRATE_FUNDED_ACCOUNTS` to `test/utils/constants.ts`.
* Updated `tsconfig.json` include paths.
* Refactored `test/utils/docker.ts` (though now largely superseded by
Kurtosis utils).
* Updated logging in `test/scripts/send-txn.ts`.
**Reasoning:**
This PR significantly expands the E2E testing capabilities by allowing
developers to easily launch and integrate local DataHaven nodes and
Snowbridge relayers into the test network, facilitating more
comprehensive integration testing. The CLI refactoring makes managing
these complex setups more robust and user-friendly.
This PR replaces our E2E script 😢 with Commander.
This allows us to extend the CLI with new options, in a fully typesafe
way without having to rely on manual definitions and nested logic.
This also opens the path to adding new commands too if we want to do
additional functionality through here. This may include, additional
scripting, running tests, uploading artefacts etc.
This PR adds the `setup-validators` Typescript script that, given an
already started up network, sets up a new validator set and sends it
through Snowbridge's Gateway to the solochain. To accomplish that
purpose, this PR:
- Modifies the `DeployLocal` script to save in the `anvil.json` file not
only the deployed strategies' addresses but also the owner of each
strategy's underlying token. This owner is used as the source of funds
to transfer tokens to other validators so they can register under that
strategy.
- Adds an `OPERATOR_SOLOCHAIN_ADDRESS` to the `Accounts` utility script
contract. This address is the one used as the Solochain address when
registering a new Operator.
- Updates the `SignUpOperator` (which I believe is now deprecated since
we have multiple Operator Sets) and `SignUpOperatorBase` scripts to
adapt to both aforementioned changes.
- Updates the `ELScriptStorage` script to save the new extra information
of each deployed strategy (the creator of the underlying token) in
storage.
- Adds a `validator-set.json` file which contains the validators that
should be registered in the AVS and sent to the Solochain network
through the Snowbridge Gateway when starting any integration test. This
is currently hardcoded but could be generated in any other way, giving
us flexibility for testing.
- Adds both a Markdown file and a Excalidraw diagram showcasing both how
the setup of integration tests work and possible integration tests that
will be added in a future PR. This list is not exahustive as there are
many more scenarios we will want to test using integration tests.
- Updates the `e2e-cli.ts` script to execute the validator setup when
bootstrapping the network used for integration testing.
---------
Co-authored-by: Facundo Farall <37149322+ffarall@users.noreply.github.com>
Due to our constrained CI resources available to us, this PR disables
`blockscout` from running in the CI.
This PR makes it so `minimal` runs don't use blockscout, and changes the
CI to use this instead of the verified network.
WARNING: This PR changes the kurtosis package to use the one from
upstream, not our fork, as it was not working at the moment. This should
be changed when fixed.
In this PR:
1. Turn `launch-kurtosis` script into a CLI, which parses parameters and
can interactively run multiple steps.
2. Separate steps of such CLI into their own scripts.
1. New script created `launch-kurtosis`, which detects if an enclave is
running and prompts to relaunch it if it is.
2. New script created `deploy-contracts` to deploy all contracts. It can
optionally verify them as well.
3. Each step can be interactively opted in/out, or choose whether to run
it via CLI params.
4. The CLI offers a help command as well.
5. Cleanup logs of CLI. Logs of internal commands ran can be printed
with LOG_LEVEL=debug. In case of error, they are always printed out.
This PR adds the basic framework for E2E and the greater typescript
testing framework for the repo.
### Contents
- CI workflow for running E2E tests on push to main, PRs and manual
- Test suite for E2E
- Currently only read-only tests
- Using `bun:test` for time being but we can always change in future
- We are using the logging library `pino` so we can help with debugging
- set environment variable `LOG_LEVEL` to get extra information on
helpers
- Local `utils` namespace for all our test suites for easy importing
- Has helpers which interact with both the docker driver and also
blockscout backend
- Allows us to fetch deployed smart contracts by their name or address
and interact with them
---------
Co-authored-by: Facundo Farall <37149322+ffarall@users.noreply.github.com>
This PR adds a script one-liner to spin up an ethereum network, with
attached explorers and deploys our `/contracts` folder into there.
---------
Co-authored-by: Facundo Farall <37149322+ffarall@users.noreply.github.com>
