Add support for multiple runtimes: `stagenet`, `testnet` & `mainnet`.
- Moved common types to `datahaven-runtime-common` crate.
- Made the node's command & service code generic over different
runtimes. Each runtime has a `dev` & `local` genesis config preset.
- More types / constants can be moved to the `datahaven-runtime-common`
crate ... this will be done in subsequent PRs.
---------
Co-authored-by: Gonza Montiel <gon.montiel@gmail.com>
Co-authored-by: Gonza Montiel <gonzamontiel@users.noreply.github.com>
Co-authored-by: Facundo Farall <37149322+ffarall@users.noreply.github.com>
This PR:
1. Generally improves the logging of the testing CLI, making the logs
more concise and easier to follow, with clearer sections and
separations.
2. Launches DataHaven solochain nodes at the beginning not the end.
3. Prompts the user if they want to launch DataHaven nodes and
Snowbridge Relayers.
---------
Co-authored-by: Tim B <79199034+timbrinded@users.noreply.github.com>
Eventually our CI will be required to run two private blockchains
locally plus associated relayers.
This PR is to prepare for this fate by improving run times and
refactoring our existing CIs so they are a bit easier to reason about.
### Refactors
- **_We now run ALL CIs on every PR!_** This is so that we decomplexify
the logic around conditional builds and fetching built binaries from
another source. This reduces the surface area of code we have to
maintain at the cost of execution time
- This penalty is ameliorated by a layered caching system. At best, it
will be less than a minute to complete a build since everything will be
cached. On GH runners this is about 6 minutes sadly.
- We will no longer be at risk of important CIs being skipped
erroneously which hide true failures.
- Caching is a low-risk approach because at worst it has to build from
scratch. A bad cache hit will never imply the wrong thing gets build
since cargo is smart enough to just throw away any inappropriate build
artefacts.
- `setup-rust` action created so we have a unified way of setting up
runner and unifying our approach to caching
- Use a unique caching key for different activities and it will fallback
to shared cache if no matches
- we are using `mainnet` kurtosis config so that it works with relayer
assumptions
### Additions
- We can specify the ethereum block time via a new cli arg `--slot-time
<seconds>`
- We can specify arbitrary network_param args which get passed into the
generated yaml
- e.g. giving `bun cli --kurtosis-network-args="pet=cat food=fish" will
add:
```yml
network_params:
# existing params...
pet: cat
food: fish
```
- We now have the ability to programmatically modify the yaml
- This means we are back down to a single `minimal.yml` kurtosis config
so we dont have to maintain changes between them
- Flow is: `add new cli arg` -> `add if() block which mutates yaml` ->
`profit`
---------
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Facundo Farall <37149322+ffarall@users.noreply.github.com>
This PR introduces a new script in `./operator/scripts` that allows us
to sort cargo dependencies alphabetically, it is based in my
[gist](https://gist.github.com/gonzamontiel/b4594c62685175f99760442ad2e2dd98).
I modified it so we can run it both in `--fix` and `--check` modes.
To sort a single cargo file, you can do:
```
./scripts/sort-cargo-deps.sh /path/to/Cargo.toml
```
Btw, make sure you are in the operator folder and you have exec
permissions:
```
cd operator
chmod +x ./scripts/sort-cargo-deps.sh
```
But what I recommend is that you format every Cargo.toml just in case,
like this:
```
find . -name "Cargo.toml" -exec ./scripts/sort-cargo-deps.sh {} \;
```
The CI will run
```
find . -name "Cargo.toml" -exec ./scripts/sort-cargo-deps.sh {} check \;
```
---------
Co-authored-by: Facundo Farall <37149322+ffarall@users.noreply.github.com>
This PR implements functionality to store outbound message commitments
in the BEEFY MMR leaf extra field for cross-chain verification. A new
`pallet-outbound-commitment-store` has been introduced to facilitate
this process.
## Changes
- Added a new `pallet-outbound-commitment-store` pallet to capture and
store outbound message commitments
- Implemented the `CommitmentHandler` to receive commitments from the
outbound queue and store them in the new pallet
- Updated the `LeafExtraDataProvider` to include these commitments in
the BEEFY MMR leaf extra field
## Implementation Details
The process works as follows:
1. The outbound queue generates a commitment hash for messages
2. The commitment is stored in the `pallet-outbound-commitment-store`
via the `CommitmentHandler`
3. The `LeafExtraDataProvider` retrieves the latest commitment and
includes it in the BEEFY MMR leaf extra field
4. This commitment can then be verified by other chains using the BEEFY
light client
The new pallet provides the necessary functions to store and retrieve
these commitments.
## Summary
This PR adds statically typed bindings for contracts. This allows you to
write E2E tests with full completions in TS.
## Additions
- `ts-build.yml` New CI, this will make sure that if there's changes
made to the contracts that the contract-bindings are up to date.
- `package.json` script changes
- `start:e2e:ci` - Designed to be run with all options specified since
CIs are famously bad with iteractive CLI prompts
- `test:e2e` - added timeout
- `generate:wagmi` - This generates the smart contract bindings for our
tests
- New Function Helpers:
- `generateRandomAccount()` Returns a viem account type object for a
random account. Useful for tests where you want idempotency on a long
lived network since the state is probabilistically fresh
- `getContractInstance()` Returns a viem contract instance that allows
you to read/write to the deployed contract. You should get full type
inference here for the methods available and parameters required.
### Example
```ts
it("avs() can be read from contract instance", async () => {
const value = await instance.read.avs();
expect(isAddress(value), "AVS getter should return an address").toBeTrue();
});
```
---------
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Facundo Farall <37149322+ffarall@users.noreply.github.com>
This PR introduces the Snowbridge `system-v2` pallet and associated
runtime components
**Key Changes:**
* **Added `system-v2` Pallet:** Integrated the
`snowbridge-pallet-system-v2` pallet, providing functionalities for the
Ethereum side of the bridge.
* **Runtime API Integration:**
* Implemented the `ControlV2Api` trait in the runtime
(`operator/runtime/src/apis.rs`) to allow looking up the `AgentId`
associated with a `VersionedLocation`.
* **System V1 Compatibility:** Added the `system-v1` pallet
(`snowbridge-pallet-system`) and related configuration/code references
in various locations.
**Important:** This `system-v1` is included *solely* because the
`system-v2` pallet requires it for compilation and compatibility. It is
**not functionally used** in this runtime.
---------
Co-authored-by: Facundo Farall <37149322+ffarall@users.noreply.github.com>
This PR adds Ethereum RPC API support to the node, enabling the
interaction with the node using standard Ethereum tools.
1. Integration of Frontier's Ethereum RPC modules (eth, net, web3,
txpool)
2. Added RPC configurations necessary for Ethereum compatibility
3. Implemented a BABE consensus data provider for handling pending
blocks
4. Added required dependencies and configurations in the node's service
5. Created necessary filter pools, block data caches, and notification
systems
6. Configured the RPC system to handle Ethereum API calls
7. Changed the transaction pool implementation to be compatible with the
fork-aware transaction pool
(https://github.com/paritytech/polkadot-sdk/pull/4639)
EDIT:
The new `transaction_pool` in `polkadot-stable2412` that comes with a
`fork-aware` feature, allowing different views of the transaction pool,
is not compatible with the current state in Frontier.
In Frontier the only supported pool is the `BasicPool`, that maintains a
similar behaviour to the one in `polkadot-stable2409`. I used this pool
directly.
---------
Co-authored-by: Ahmad Kaouk <56095276+ahmadkaouk@users.noreply.github.com>
Co-authored-by: Facundo Farall <37149322+ffarall@users.noreply.github.com>
## Human Written Description
This PR adds the following to the E2E CLI:
- Relayer config generation for: `beacon-relay` `beefy-relay`
- The other two relayer types to be added later
- Relayers don't actually work yet
- By default turned off, this requires a binary to be present in:
`<repo_root>/operator/target/release` dir
- Datahaven network launching
- DH network is using default `local` network chain spec
- Launched with 5 nodes since our authority set is 6 large (and you need
2/3 + 1 of set size
> [!NOTE]
> Both the relayer and the DH node binaries are being run as local
processes TEMPORARILY. This means that logging is done in a very
rudimentary way (we pipe to a file whilst the CLI is running).
>
> This means that when the CLI finishes **the log files will no longer
be written to**.
> This is temporary since spawning binaries is a stop gap solution until
docker images available.
---
> [!IMPORTANT]
> The following is AI generated slop describing this PR's changes:
**Key Changes:**
* **CLI Enhancements (`test/cli/index.ts`):**
* Added options `--datahaven` and `--datahaven-bin-path` to enable
launching local DataHaven nodes.
* Added options `--relayer` and `--relayer-bin-path` to enable launching
Snowbridge relayers (Beefy and Beacon).
* Added negation flags (`--no-fund-validators`, `--no-setup-validators`,
`--no-update-validator-set`) for more granular control over validator
setup steps.
* Added `--skip-cleaning` option to preserve Kurtosis state between
runs.
* Added a pre-action hook (`launchPreActionHook`) to validate flag
combinations (e.g., `--verified` requires `--blockscout`).
* **New CLI Handlers (`test/cli/handlers/launch/`):**
* `datahaven.ts`: Logic for spawning DataHaven node processes using the
specified binary. Manages ports and process cleanup.
* `relayer.ts`: Logic for configuring and spawning Snowbridge relayer
processes (Beefy and Beacon). Reads contract deployment addresses,
updates relayer config templates, and uses specified private keys.
Manages log files and process cleanup.
* `summary.ts`: Generates and displays the table of running services
(including dynamically launched DataHaven nodes) and their endpoints.
* `validator.ts`: Extracted validator funding, setup, and set update
logic into its own handler.
* `index.ts`: Orchestrates the launch sequence based on CLI options,
calling the appropriate handlers. Includes a `LaunchedNetwork` class to
track spawned processes, file descriptors, and node ports for cleanup.
* **Updated `package.json` Scripts:**
* Added `start:e2e:minrelayer` script for a minimal setup including
relayers and DataHaven nodes.
* Modified `stop:e2e` to include `pkill datahaven` for proper cleanup.
* Added `stop:e2e:quick` to only stop the Kurtosis enclave without full
cleaning.
* **Updated `launch-kurtosis.ts`:** Modified to use new Kurtosis utility
functions and added a `skipCleaning` option.
* **New Utility Functions:**
* `test/utils/kurtosis.ts`: Functions to inspect Kurtosis services
(`getServiceFromKurtosis`, `getPortFromKurtosis`,
`getServicesFromKurtosis`).
* `test/utils/parser.ts`: Zod schemas and parsing functions for
Snowbridge relayer configurations.
* **Constants & Minor Updates:**
* Added `SUBSTRATE_FUNDED_ACCOUNTS` to `test/utils/constants.ts`.
* Updated `tsconfig.json` include paths.
* Refactored `test/utils/docker.ts` (though now largely superseded by
Kurtosis utils).
* Updated logging in `test/scripts/send-txn.ts`.
**Reasoning:**
This PR significantly expands the E2E testing capabilities by allowing
developers to easily launch and integrate local DataHaven nodes and
Snowbridge relayers into the test network, facilitating more
comprehensive integration testing. The CLI refactoring makes managing
these complex setups more robust and user-friendly.
This pull request integrates the snowbridge-pallet-outbound-queue-v2 and
its dependencies into the DataHaven runtime. This pallet is responsible
for handling the submission, processing, and commitment of outbound
messages destined for the Ethereum network via the Snowbridge bridge
(v2).
### Key Changes
#### Added New Pallets
- **snowbridge-pallet-outbound-queue-v2**: The core pallet for managing
the outbound message lifecycle.
- **pallet-message-queue**: Introduced to handle the queuing and
processing of messages.
#### Added Crates to Datahaven codebase
- snowbridge-merkle-tree
- bridge-hub-common
- snowbridge-outbound-queue-v2-runtime-api
---------
Co-authored-by: Facundo Farall <37149322+ffarall@users.noreply.github.com>
This PR replaces our E2E script 😢 with Commander.
This allows us to extend the CLI with new options, in a fully typesafe
way without having to rely on manual definitions and nested logic.
This also opens the path to adding new commands too if we want to do
additional functionality through here. This may include, additional
scripting, running tests, uploading artefacts etc.
This PR adds the [pallet
parameters](https://docs.rs/pallet-parameters/0.11.0/pallet_parameters/index.html)
to the DataHaven runtime.
This allows config parameters for pallets to be dynamically modified,
needed, for instance, to set `EthereumfGatewayAddress` after the
deployment of that smart contract on Ethereum. StorageHub also uses this
pallet extensively.
Additionally, refactors the pallet configuration files to have them more
visually organised.
Currently, it is not possible to sign a transaction when submitting an
extrinsic with Polkadot.js Apps. This is due to the fact that
Polkadot.js does not consider DataHaven chain as an Ethereum chain
(exclusively using AccountId20 accounts).
This PR renames the runtime to `datahaven-stagenet` and this should make
Polkadot.js only use Ethereum AccountId20 style accounts once this
[PR](https://github.com/polkadot-js/apps/pull/11501) is merged.
Note: another 'trick' to get it working locally, meanwhile, is to name
the runtime any of the Moonbeam runtimes (e.g. `moonbase`).
This PR adds the `setup-validators` Typescript script that, given an
already started up network, sets up a new validator set and sends it
through Snowbridge's Gateway to the solochain. To accomplish that
purpose, this PR:
- Modifies the `DeployLocal` script to save in the `anvil.json` file not
only the deployed strategies' addresses but also the owner of each
strategy's underlying token. This owner is used as the source of funds
to transfer tokens to other validators so they can register under that
strategy.
- Adds an `OPERATOR_SOLOCHAIN_ADDRESS` to the `Accounts` utility script
contract. This address is the one used as the Solochain address when
registering a new Operator.
- Updates the `SignUpOperator` (which I believe is now deprecated since
we have multiple Operator Sets) and `SignUpOperatorBase` scripts to
adapt to both aforementioned changes.
- Updates the `ELScriptStorage` script to save the new extra information
of each deployed strategy (the creator of the underlying token) in
storage.
- Adds a `validator-set.json` file which contains the validators that
should be registered in the AVS and sent to the Solochain network
through the Snowbridge Gateway when starting any integration test. This
is currently hardcoded but could be generated in any other way, giving
us flexibility for testing.
- Adds both a Markdown file and a Excalidraw diagram showcasing both how
the setup of integration tests work and possible integration tests that
will be added in a future PR. This list is not exahustive as there are
many more scenarios we will want to test using integration tests.
- Updates the `e2e-cli.ts` script to execute the validator setup when
bootstrapping the network used for integration testing.
---------
Co-authored-by: Facundo Farall <37149322+ffarall@users.noreply.github.com>
This PR adds the Snowbridge Inbound Queue V2 pallet to the runtime. It
also defines bridge primitives, including the message structure for
EigenLayer, and introduces the EigenLayerMessageProcessor to handle
messages coming from EigenLayer.
---------
Co-authored-by: Facundo Farall <37149322+ffarall@users.noreply.github.com>
In this PR:
1. Implement application-specific functionalities in the
`DataHavenServiceManager` contract:
1. Registering of 3 operator sets: Validators, BSPs and MSPs.
2. Allowlisted sign up of operators.
3. Integration with Snowbridge to send message of new validator set.
2. Basic testing of the above functionalities.
3. Tests now use less mocked contracts (especially from EigenLayer).
4. Refactor of `SignUpOperator` script, which now supports the three
kinds of Operator sets.
Due to our constrained CI resources available to us, this PR disables
`blockscout` from running in the CI.
This PR makes it so `minimal` runs don't use blockscout, and changes the
CI to use this instead of the verified network.
WARNING: This PR changes the kurtosis package to use the one from
upstream, not our fork, as it was not working at the moment. This should
be changed when fixed.
In this PR:
1. Turn `launch-kurtosis` script into a CLI, which parses parameters and
can interactively run multiple steps.
2. Separate steps of such CLI into their own scripts.
1. New script created `launch-kurtosis`, which detects if an enclave is
running and prompts to relaunch it if it is.
2. New script created `deploy-contracts` to deploy all contracts. It can
optionally verify them as well.
3. Each step can be interactively opted in/out, or choose whether to run
it via CLI params.
4. The CLI offers a help command as well.
5. Cleanup logs of CLI. Logs of internal commands ran can be printed
with LOG_LEVEL=debug. In case of error, they are always printed out.
This PR adds the basic framework for E2E and the greater typescript
testing framework for the repo.
### Contents
- CI workflow for running E2E tests on push to main, PRs and manual
- Test suite for E2E
- Currently only read-only tests
- Using `bun:test` for time being but we can always change in future
- We are using the logging library `pino` so we can help with debugging
- set environment variable `LOG_LEVEL` to get extra information on
helpers
- Local `utils` namespace for all our test suites for easy importing
- Has helpers which interact with both the docker driver and also
blockscout backend
- Allows us to fetch deployed smart contracts by their name or address
and interact with them
---------
Co-authored-by: Facundo Farall <37149322+ffarall@users.noreply.github.com>
Based on https://github.com/paritytech/polkadot-sdk/pull/5868.
Note: this (somehow) breaks the ability to run a local network with the
zombienet config. It seems that the config generated by Zombienet that
automatically set the balance of validators is leading to an issue
(because of the address type).
---------
Co-authored-by: Facundo Farall <37149322+ffarall@users.noreply.github.com>
This PR adds a script one-liner to spin up an ethereum network, with
attached explorers and deploys our `/contracts` folder into there.
---------
Co-authored-by: Facundo Farall <37149322+ffarall@users.noreply.github.com>
I noticed `cargo clippy` was failing in CI, I fixed it. I still think we
should tweak the lints.
For example, clippy complains at things like `1 * HOURS ` because it's
redundant to multiply by 1, buy maybe we want it explicitly that way, so
it's more evident it can be changed. In this PR I don't tweak the rule
but try to make clippy happy.
In this PR:
1. Implement deployment script for all contracts involved. This includes
1. EigenLayer core contracts.
2. Snowbridge contracts.
3. Our custom contracts to interact with both protocols.
4. Update README to document functionalities of `contracts` directory.
Future work:
1. Deployment of EigenLayer contracts should be conditional, as we
wouldn't deploy them in testnet or mainnet, but we would do it in an
internal stagenet.
5. Cleaning up unused smart contracts.
This PR:
1. Adds some missing functionalities to connect the Snowbridge contracts
to the RewardsRegistry contract, through an Agent.
2. Adds a test suite for this integration, with a happy and unhappy
path. The former being with a valid and honest update of rewards, which
then a validator uses to claim them. The latter with a malicious message
that tries to update the rewards root, but it's not allowed.
This PR adds an initial implementation for a rewards registry, which
will be the contract in charge of allowing DataHaven validators to claim
the rewards they earned for being validators in the previous epoch. The
logic behind it is as follows:
- Whenever an epoch finishes, the corresponding BEEFY block gets relayed
to Ethereum through Snowbridge. This BEEFY block contains, in its
`extra` field, the merkle root of the tree that contains as leafs all
the message commitments of the messages of corresponding block, one of
which is the rewards distribution message.
- The rewards distribution message commitment is the root of the merkle
tree where each leaf is a tuple of the operator ID and the obtained era
points in the finished epoch. In this case, the operator ID is the
corresponding validator's Ethereum address.
- When the rewards distribution message is received, Snowbridge
validates it using the aforementioned BEEFY block and then dispatches
it. The dispatch invokes the `callContract` function of the
`RewardsAgent` agent, with the corresponding parameters so that this
agent calls the `updateRewardsMerkleRoot` function of the
`RewardsRegistry` contract with the new rewards distribution message
commitment.
- After this root is updated, any validator/operator can submit a proof
that it is in a leaf of the merkle tree that produced that root, which
means it has pending rewards to claim, through the
`ServiceManagerBase`'s `claimOperatorRewards` function.
- Each operator set of the AVS can have an assigned `RewardsRegistry`
contract. Operator sets that do not have an assigned `RewardsRegistry`
contract won't be able to received rewards.
This PR also adds two separate unit-test suites: one for the added
functionality to the `ServiceManagerBase` contract and one specific to
the new `RewardsRegistry` contract.
> [!CAUTION]
The `RewardsAgent` agent is the only one allowed to update the rewards'
merkle root, which means if a malicious user could get access to it it
could set the pending rewards to be claimed to an arbitrary tree that
benefits it. Extreme caution must be taken in the Substrate side so only
validated messages are sent to the Ethereum side, as to not allow any
users to impersonate being this agent.
### TODO:
Ideally, we would use the `RewardsCoordinator` contract from the
EigenLayer core to distribute the rewards, but currently that adds a
huge overhead for Operators since they'd have to wait for EigenLayer's
SideCar to snapshot state and update the distribution root (which
happens once a day), generate a proof that they belong to the tree of
that distribution root, store it while waiting for the `activationDelay`
(currently a week) to pass, and just then they would be able to claim
their earned rewards.
---------
Co-authored-by: Facundo Farall <37149322+ffarall@users.noreply.github.com>
Current implementation was incorrect as `Babe::find_author` returns an
index, not the author's account.
Deferred the logic of looking up the author's account based on the index
to the `pallet_session::FindAccountFromAuthorIndex` function, and use
the existing conversion from `AccountId20` to `H160`.
This PR remove the unneeded action that install rustfmt as it is now
already available. It remove a step in the CI.
The action removed is `dtolnay/rust-toolchain@stable`
---------
Co-authored-by: Facundo Farall <37149322+ffarall@users.noreply.github.com>
This PR adds a new job at the end of the `DataHave Operator Rust Tests`
(`rust-tests.yml`) that checks:
- If the tests were skipped (no changes in the `operator/` directory),
it succeeds.
- If the tests run and succeeded, it succeeds.
- If the tests run and failed, it fails.
This is useful to set this job as required in our merging ruleset, and
account for cases where the tests are skipped.
This PR:
- Sets up the slasher infrastructure with the base functionality
required (in `ISlasher.sol`, `SlasherBase.sol` and
`SlasherBaseStorage.sol`) and adds the tests for it (in
`SlasherBase.t.sol`).
- Adds an implementation of a more complex slasher (in
`IVetoableSlasher.sol` and `VetoableSlasher.sol`) and tests for it (in
`VetoableSlasher.t.sol`).
- Modifies the `ServiceManagerBase` contract to use the new
`VetoableSlasher` contract to manage slashing.
- Updates mocks and tests to reflect the newly added functionality.
---------
Co-authored-by: Facundo Farall <37149322+ffarall@users.noreply.github.com>
