mirror of
https://github.com/datahaven-xyz/datahaven
synced 2026-05-24 09:50:01 +00:00
9 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Ahmad Kaouk
|
401f646286
|
feat: automated validator set submission with era targeting (#433)
## Era-targeted validator set submission with dedicated submitter role > **Note:** This PR includes a detailed specification at [`specs/validator-set-submission/validator-set-submission.md`](https://github.com/datahaven-xyz/datahaven/blob/feat/validator-set-submitter/specs/validator-set-submission/validator-set-submission.md) that covers the design rationale, submission lifecycle, era-targeting rules, and failure modes. Reading the spec first will make the contract, pallet, and daemon changes easier to follow. ### Summary - Introduce a dedicated `validatorSetSubmitter` role on `DataHavenServiceManager`, separating validator set submission authority from the contract owner - Replace the unscoped `sendNewValidatorSet` with `sendNewValidatorSetForEra`, which encodes a `targetEra` into the Snowbridge message payload - Add server-side era validation in the `external-validators` pallet to reject stale, duplicate, or out-of-range submissions - Add a long-running TypeScript daemon that watches session changes and automatically submits each era's validator set at the right time ### Contract changes (`contracts/`) - **New `validatorSetSubmitter` storage slot** — set during `initialize` and rotatable via `setValidatorSetSubmitter` (owner-only). The storage gap is decremented accordingly. - **`sendNewValidatorSet` → `sendNewValidatorSetForEra`** — accepts a `uint64 targetEra` parameter and is restricted to `onlyValidatorSetSubmitter` instead of `onlyOwner`. - **`buildNewValidatorSetMessageForEra`** — the `NewValidatorSetPayload.externalIndex` is now caller-supplied instead of hardcoded to `0`. - **New events** — `ValidatorSetSubmitterUpdated`, `ValidatorSetMessageSubmitted`. - **New error** — `OnlyValidatorSetSubmitter`. - **New test suite** — `ValidatorSetSubmitter.t.sol` covering submitter set/rotate, access control, era encoding, and legacy function removal. ### Pallet changes (`operator/`) - **`validate_target_era`** in `external-validators` — enforces `activeEra < targetEra <= activeEra + 1` and `targetEra > ExternalIndex` (dedup guard). - **New errors** — `TargetEraTooOld`, `TargetEraTooNew`, `DuplicateOrStaleTargetEra`. - **Tests** — five new test cases for era boundary conditions (next-era acceptance, old-era rejection, too-new rejection, duplicate rejection, genesis behavior). Existing `era_hooks_with_external_index` test updated to use valid target eras. - **Runtime test fixes** — `external_index: 0` → `1` in mainnet/stagenet/testnet EigenLayer message processor tests to satisfy the new validation. ### Validator set submitter daemon (`test/tools/validator-set-submitter/`) - Event-driven service that subscribes to finalized `Session.CurrentIndex` via Polkadot-API `watchValue`. - Submits once per era during the last session, targeting `ActiveEra + 1`. - Tracks submitted eras to avoid duplicates; skips if `ExternalIndex` already covers the target. - Startup self-checks: Ethereum connectivity, DataHaven connectivity, on-chain submitter authorization. - Supports `--dry-run` mode and YAML configuration. - Graceful shutdown on `SIGINT`/`SIGTERM`. ### Test & tooling updates - **E2E test** (`validator-set-update.test.ts`) — calls `sendNewValidatorSetForEra` with a computed `targetEra` and filters the substrate event by `external_index`. - **`update-validator-set.ts` script** — accepts `--target-era` flag; defaults to era 1 for fresh networks. - **CLI launch** — wires validator set update as an interactive step after relayer launch. - **`package.json`** — new `submitter` and `submitter:dry-run` scripts. - Regenerated contract bindings, PAPI metadata, state-diff, and storage layout snapshots. ### Test plan - [x] `forge test` — passes, including new `ValidatorSetSubmitter.t.sol` - [x] `cargo test` — passes, including new era-validation tests in `external-validators` - [x] `bun test:e2e` — validator-set-update suite passes with era-targeted flow - [x] Manual: run submitter daemon against local network (`bun submitter`), verify it submits once per era at the correct session ## ⚠️ Breaking Changes ⚠️ - **`sendNewValidatorSet` removed** — replaced by `sendNewValidatorSetForEra(uint64 targetEra, ...)`. Callers must now supply a `targetEra` parameter. - **Access control changed** — validator set submission is now restricted to the `validatorSetSubmitter` role instead of the contract `owner`. The submitter address is set during `initialize` and rotatable via `setValidatorSetSubmitter` (owner-only). - **`external-validators` pallet now validates `targetEra`** — messages with a stale, duplicate, or out-of-range `external_index` are rejected on-chain. Existing integrations sending `external_index: 0` will fail validation. --------- Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
Gonza Montiel
|
dc0f0673e2
|
test: Update validator set e2e test (#126)
## Add E2E validator-set update flow - feat: `test/utils/validators.ts` for on-demand validator orchestration. - feat: `test/suites/validator-set-update.test.ts` covering allowlist → register → update. - some minor launcher updates: avoid docker cache, add `--platform` when building datahaven image, avoid sending validator-set update on launch. - Helpers: ABI shortcut in `test/utils/contracts.ts`; config tweaks in `test/configs/validator-set.json`. - Minor cleanup/formatting across `test/launcher/*`, `test/scripts/setup-validators.ts`, and related tests. - added `keepAlive` flag to `BaseTestSuite`, in order to avoid tearing down the network while debugging. Defaults, obviously, to false. - added a `failOnTomeout` option on to waitForDataHavenEvents() so the test fails of the timeout is reached and no event was captured. ### Coverage - The test simulates an scenario in which we have two active authorities (alice and bob), which are running, and registered as operators, which is the normal state after the chain launches. Then: - It launches two more nodes (charlie and dave) - It add the nodes to allowlist and register them as operators - It sends the validator set update message - Checks that the validator update message was propagated through the gateway and arrived the external-validators pallet - Checks that the chain continues producing blocks ### Notes The last test case has a timeout of 10 minutes. This is to respect propagation times of the message through the relayers. We are testing that the external validators pallet actually updated the validator set. Locally, I could expect 5~6 minutes, I just wanted to be on the safe side. CI is passing showing that this was enough indeed. --------- Co-authored-by: Steve Degosserie <723552+stiiifff@users.noreply.github.com> Co-authored-by: Ahmad Kaouk <56095276+ahmadkaouk@users.noreply.github.com> |
||
|
Facundo Farall
|
9b311e00ef
|
test: 🏗️ Setup e2e testing framework (#104)
## Implement E2E Testing Framework with Isolated Networks ### Summary Refactors the existing E2E testing infrastructure to provide isolated test environments with parallel execution support. Each test suite now runs in its own network namespace, preventing resource conflicts. ### Key Changes - **New Testing Framework** (`test/framework/`): Base classes for test lifecycle management with automatic setup/teardown - **Launcher Module** (`test/launcher/`): Extracted network orchestration logic from CLI handlers for reusability - **Parallel Execution**: Added `test-parallel.ts` script with concurrency limits to prevent resource exhaustion - **Test Isolation**: Each suite gets unique network IDs (format: `suiteName-timestamp`) and Docker networks - **Improved Test Organization**: Migrated tests to new framework, deprecated old test structure ### Test Improvements - Added 4 new test suites demonstrating framework usage. : - `contracts.test.ts` - Smart contract deployment/interaction - `datahaven-substrate.test.ts` - Substrate API operations - `cross-chain.test.ts` - Snowbridge cross-chain messaging - `ethereum-basic.test.ts` - Ethereum network operations > [!WARNING] The test suites themselves are bad and shouldn't be consider examples of good tests. They were AI generated just to test the concurrency of test runners ### Documentation - Added comprehensive framework overview (`E2E_FRAMEWORK_OVERVIEW.md`) - Updated README with parallel testing commands - Added test patterns and best practices ### Breaking Changes - Old test suites moved to `e2e - DEPRECATED/` directory - Test execution now requires extending `BaseTestSuite` class ### Testing Run tests with: `bun test:e2e` or `bun test:e2e:parallel` (with concurrency limits) ### TODO - [ ] Implement good test examples. - [ ] Implement useful test utils (like waiting for an event to show up in DataHaven or Ethereum). - [ ] Enforce tests with CI (currently cannot be done due to intermittent error when sending a transaction with PAPI). --------- Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: undercover-cactus <lola@moonsonglabs.com> |
||
|
Gonza Montiel
|
f07afda0b0
|
feat: 🏗️ run execution relayer (#73)
## This PR includes: - Running the execution relayer on the CLI - Modifying the Payload generation in the `DataHavenServiceManager.sol` - Modified the `EigenLayerMessageProcessor` to work with the ValidatorSet update message, but for this change we are loosing the generic message type (it was the only way to make it work so far). - Adds a `--no-wait` argument to the cli launch and stop commands to bootstrap faster. ### Testing the Snowbridge message encoding / decoding - Added`MessageEncoding.t.sol` is documented and explains how to generate bin data to use in the rust test. - Added a Rust unit test to `EigenLayerMessageProcessor` to compare the message encoding/decoding taking the bytes from a file (previously generated with some mock data). Specifically, we want that: |
||
|
Tim B
|
6ac58c532f
|
test: ✨ New CLI functions (#84)
## Changes - New option: `--kurtosis-enclave-name` to allow you to specify a new ethereum network with a different enclave name. Neccesary step for setting up local testing with multiple enclaves running at once - Refactor: all single dash options (e.g. `-i`) have been renamed to have double dashes `--` for consistency others - sad times: CLI now must be invoked `bun cli launch` since we have multiple fns now - Package Update - New Function: `stop` which allows you to kill all components (eth, dh, relayers) or only a single part - Gonza's mac target build fix - Misc fixes to ci like caching and rate limits ## Additional Comment The CLI needs multiple commands and this PR adds the first new one `stop`. This syntax is faily extensible so @ffarall 's k8 command can follow suite. Originally we were going to have an `exec` command to expose internal fns to be callable by command line, i.e. generate beacon checkpoint - however the need for it has since been superceded by the deploy to k8 command that's going to be added. I've left the code in place though so when another usecase comes up we can just plug that in. --------- Co-authored-by: Facundo Farall <37149322+ffarall@users.noreply.github.com> |
||
|
Tim B
|
431d1f7181
|
test: 🐳 Add Docker relay support to CLI (#74)
## Changes
- Latest changes to have working relayer 🎉 component
- Changed spawning snowbridge relayers to docker containers
- Small logging output changes
- Refactoring to `LaunchedNetwork` class
- new flag `--bd` `--build-datahaven` which will build a local docker
container which is **much** quicker than the proper CI build (which uses
a controlled build enviroment)
- new bun script `start:e2e:local`, which is everything that
`start:e2e:ci` has, but with building local docker container and
log_level debug set
---
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Summary by CodeRabbit
- **New Features**
- Added support for launching and managing relayer and DataHaven
services using Docker containers and networks.
- Introduced a CLI option to specify the relayer Docker image tag
instead of a binary path.
- **Improvements**
- Enhanced log messages with clearer text and expressive emojis for
better user feedback.
- Improved summary display by removing relayer services from the output.
- Updated build scripts to consistently enable the "fast-runtime"
feature for cross-platform builds.
- Refined validation and error reporting for checkpoint data parsing.
- **Bug Fixes**
- Improved Docker container cleanup and network management during
service launch and teardown.
- **Chores**
- Updated and refactored npm scripts for Docker operations and
end-to-end test cleanup.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Facundo Farall <37149322+ffarall@users.noreply.github.com>
|
||
|
Tobi Demeco
|
625718c6d2
|
fix: 🩹 update to allow BEEFY relayer to run correctly (#67)
This PR:
- Adds a check to make sure the BEEFY RPC endpoint is ready before
spinning up the BEEFY relayer, otherwise it would just fail and crash.
- Adds the `--enable-offchain-indexing=true` flag to the Datahaven nodes
run when starting up the E2E infra. This is needed because otherwise
nodes can't be queried by the relayer to generate the required proofs
since they would not store the MMR leafs/nodes/root, so the relayer
would just crash.
- Updates the way we were generating the merkle root of the validator
set.
- The BEEFY pallet (and as such, the relayer) generate the validator set
merkle tree by getting the validator list and treating it as an already
ordered set of merkle leafs, hashing each pair in succession without
caring what each leaf value is.
- Meanwhile, the OpenZeppelin crypto library (and as such, the
EigenLayer contracts) also gets the merkle leafs list but hashes each
pair of leafs in value order.
- This means that, for example, if the list of leafs is: ["0x124",
"0x123"]
- BEEFY would do `hash("0x124123")` to get the value of the parent node.
- OZ and EigenLayer would do `hash("0x123124")` to get the value of the
parent node.
- This created a mismatch between what the BEEFY relayer was expecting
and what was actually calculated in our script. A way to obtain a merkle
tree using the BEEFY way was added to solve this.
- Updates the authority set to not be a hardcoded array of keys, now the
BEEFY keys are obtained by directly querying the runtime before
deploying the BEEFY contracts.
- Renames a few files from `flamingo` to `datahaven`.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Added support for calculating and generating Merkle roots and proofs
without sorting, enabling new use cases for validator set management.
- Introduced dynamic fetching and configuration of Datahaven validator
authorities during network launch.
- Added readiness check for the BEEFY protocol before relayer startup,
improving reliability of relayer operations.
- **Bug Fixes**
- Fixed indentation issues in configuration files for improved
readability and consistency.
- **Chores**
- Updated validator lists and addresses in configuration files.
- Enhanced e2e test scripts and added new commands for relayer and
minimal test scenarios.
- Added new dependency to test package.
- Updated version strings in package metadata.
- **Refactor**
- Improved logging and configuration handling during network and relayer
launches.
- Simplified import statements and removed unused code.
- **Style**
- Reformatted configuration and TypeScript files for better readability.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Tim B <79199034+timbrinded@users.noreply.github.com>
Co-authored-by: Facundo Farall <37149322+ffarall@users.noreply.github.com>
|
||
|
Facundo Farall
|
e161accac2
|
fix: 🧑💻 Fix and improve bun cli logging and functionalities (#60)
This PR: 1. Generally improves the logging of the testing CLI, making the logs more concise and easier to follow, with clearer sections and separations. 2. Launches DataHaven solochain nodes at the beginning not the end. 3. Prompts the user if they want to launch DataHaven nodes and Snowbridge Relayers. --------- Co-authored-by: Tim B <79199034+timbrinded@users.noreply.github.com> |
||
|
Tim B
|
95171a5e10
|
test: ⚙️ Parse & Generate Relayer Configs (#54)
## Human Written Description
This PR adds the following to the E2E CLI:
- Relayer config generation for: `beacon-relay` `beefy-relay`
- The other two relayer types to be added later
- Relayers don't actually work yet
- By default turned off, this requires a binary to be present in:
`<repo_root>/operator/target/release` dir
- Datahaven network launching
- DH network is using default `local` network chain spec
- Launched with 5 nodes since our authority set is 6 large (and you need
2/3 + 1 of set size
> [!NOTE]
> Both the relayer and the DH node binaries are being run as local
processes TEMPORARILY. This means that logging is done in a very
rudimentary way (we pipe to a file whilst the CLI is running).
>
> This means that when the CLI finishes **the log files will no longer
be written to**.
> This is temporary since spawning binaries is a stop gap solution until
docker images available.
---
> [!IMPORTANT]
> The following is AI generated slop describing this PR's changes:
**Key Changes:**
* **CLI Enhancements (`test/cli/index.ts`):**
* Added options `--datahaven` and `--datahaven-bin-path` to enable
launching local DataHaven nodes.
* Added options `--relayer` and `--relayer-bin-path` to enable launching
Snowbridge relayers (Beefy and Beacon).
* Added negation flags (`--no-fund-validators`, `--no-setup-validators`,
`--no-update-validator-set`) for more granular control over validator
setup steps.
* Added `--skip-cleaning` option to preserve Kurtosis state between
runs.
* Added a pre-action hook (`launchPreActionHook`) to validate flag
combinations (e.g., `--verified` requires `--blockscout`).
* **New CLI Handlers (`test/cli/handlers/launch/`):**
* `datahaven.ts`: Logic for spawning DataHaven node processes using the
specified binary. Manages ports and process cleanup.
* `relayer.ts`: Logic for configuring and spawning Snowbridge relayer
processes (Beefy and Beacon). Reads contract deployment addresses,
updates relayer config templates, and uses specified private keys.
Manages log files and process cleanup.
* `summary.ts`: Generates and displays the table of running services
(including dynamically launched DataHaven nodes) and their endpoints.
* `validator.ts`: Extracted validator funding, setup, and set update
logic into its own handler.
* `index.ts`: Orchestrates the launch sequence based on CLI options,
calling the appropriate handlers. Includes a `LaunchedNetwork` class to
track spawned processes, file descriptors, and node ports for cleanup.
* **Updated `package.json` Scripts:**
* Added `start:e2e:minrelayer` script for a minimal setup including
relayers and DataHaven nodes.
* Modified `stop:e2e` to include `pkill datahaven` for proper cleanup.
* Added `stop:e2e:quick` to only stop the Kurtosis enclave without full
cleaning.
* **Updated `launch-kurtosis.ts`:** Modified to use new Kurtosis utility
functions and added a `skipCleaning` option.
* **New Utility Functions:**
* `test/utils/kurtosis.ts`: Functions to inspect Kurtosis services
(`getServiceFromKurtosis`, `getPortFromKurtosis`,
`getServicesFromKurtosis`).
* `test/utils/parser.ts`: Zod schemas and parsing functions for
Snowbridge relayer configurations.
* **Constants & Minor Updates:**
* Added `SUBSTRATE_FUNDED_ACCOUNTS` to `test/utils/constants.ts`.
* Updated `tsconfig.json` include paths.
* Refactored `test/utils/docker.ts` (though now largely superseded by
Kurtosis utils).
* Updated logging in `test/scripts/send-txn.ts`.
**Reasoning:**
This PR significantly expands the E2E testing capabilities by allowing
developers to easily launch and integrate local DataHaven nodes and
Snowbridge relayers into the test network, facilitating more
comprehensive integration testing. The CLI refactoring makes managing
these complex setups more robust and user-friendly.
|