## Polkadot upgrade 2503
In this PR, we upgrade form Polkadot SDK 2412 to Polkadot SDK 2503. In
order to upgrade the SDK we need to upgrade some dependencies :
StorageHub and frontier simultaneously.
## What changes
### Trivial changes
* https://github.com/paritytech/polkadot-sdk/pull/7634 -> The new trait
is required in all the pallets using scale encoding.
* https://github.com/paritytech/polkadot-sdk/pull/7043 -> Remove
deprecated `sp-std` and replace with `alloc` or `core`.
* https://github.com/paritytech/polkadot-sdk/pull/6140 -> Accurate
weight reclaim with frame_system::WeightReclaim
### Breaking changes
* https://github.com/paritytech/polkadot-sdk/pull/2072 -> There is a
change in `pallet-referenda`. Now, the tracks are retrieved as a list of
`Track`s. Also, the names of the tracks might have some trailing null
values (`\0`). This means display representation of the tracks' names
must be sanitized.
* https://github.com/paritytech/polkadot-sdk/pull/5723 -> adds the
ability for these pallets to specify their source of the block number.
This is useful when these pallets are migrated from the relay chain to a
parachain and vice versa. (Not entirely sure it is breaking as it is
being marked as backward compatible).
* https://github.com/paritytech/polkadot-sdk/pull/6338 -> Update
Referenda to Support Block Number Provider
### Notable changes
* https://github.com/paritytech/polkadot-sdk/pull/5703 -> Not changes
required in the codebase but impact fastSync mode. Should improve
testing.
* https://github.com/paritytech/polkadot-sdk/pull/5842 -> Removes
`libp2p` types in authority-discovery, and replace them with network
backend agnostic types from `sc-network-types`. The `sc-network`
interface is therefore updated accordingly.
## What changes in Frontier
* https://github.com/polkadot-evm/frontier/pull/1693 -> Add support for
EIP 7702 which has been enable with Pectra. This EIP add a new field
`AuthorizationList` in Ethereum transaction.
Changes example :
```rust
#[test]
fn validate_transaction_fails_on_filtered_call() {
...
pallet_evm::Call::<Runtime>::call {
source: H160::default(),
target: H160::default(),
input: Vec::new(),
value: sp_core::U256::zero(),
gas_limit: 21000,
max_fee_per_gas: sp_core::U256::zero(),
max_priority_fee_per_gas: Some(sp_core::U256::zero()),
nonce: None,
access_list: Vec::new(),
authorization_list: Vec::new(),
}
.into(),
```
## ⚠️ Breaking Changes ⚠️
* Upgrade to Stirage hub v0.5.1
* Support for Ethereum latest upgrade (txs now have the
`authoriation_list` for support EIP 7702)
---------
Co-authored-by: Steve Degosserie <723552+stiiifff@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
This is to better align the behaviour of the node with StorageHub's
configurations, otherwise a MSP could pass `--trusted-msps` and use that
functionality only intended for BSPs
Update StorageHub dependencies from v0.4.1 to v0.4.2.
- Bump all `storage-hub` crate references to v0.4.2
- Wire new `--trusted-msps` CLI parameter and `with_file_transfer` API
changes
- Wire new `--trusted-file-transfer-batch-size-bytes` config option
- Update Rust toolchain from 1.88.0 to 1.90.0
- Apply cargo fmt fixes for Rust 1.90
See [StorageHub v0.4.2 release
notes](https://github.com/Moonsong-Labs/storage-hub/releases/tag/v0.4.2)
for full details.
## ⚠️ Breaking Changes ⚠️
New `--trusted-msps` parameter will have to be added to the launch
options of BSPs, and new `--trusted-file-transfer-batch-size-bytes` has
sane default so adding it to the MSPs configuration is not required.
---------
Co-authored-by: Steve Degosserie <723552+stiiifff@users.noreply.github.com>
## Summary
- Wire the Frontier `EthPubSub` module into the node's RPC layer,
enabling WebSocket-based `eth_subscribe`/`eth_unsubscribe` support for
`newHeads`, `logs`, and `newPendingTransactions`
- Use Ethereum-style hex (`0x`-prefixed) subscription IDs via
`EthereumSubIdProvider` for client compatibility
- Add Moonwall test suites (adapted from Moonbeam) covering block header
subscriptions, log filtering (by address, topics, wildcards, conditional
parameters), and pending transaction notifications
## Changes
### `operator/node/src/rpc.rs`
- Import and merge `EthPubSub` / `EthPubSubApiServer` into the RPC
module
- Accept `subscription_task_executor` and `pubsub_notification_sinks`
parameters in `create_full()`
- Remove stale commented-out boilerplate
### `operator/node/src/service.rs`
- Clone `pubsub_notification_sinks` and forward it (along with
`subscription_executor`) into the RPC factory closure
- Set `config.rpc.id_provider` to `EthereumSubIdProvider` for
Ethereum-compatible subscription IDs
### `test/moonwall/suites/dev/stagenet/subscription/`
- `test-subscription.ts` — `newHeads`: subscription ID format, block
header field validation
- `test-subscription-logs.ts` — `logs`: basic log notification on
contract deployment
- `test-subscription-logs2.ts` — `logs`: filtering by single/multiple
addresses, topics, wildcards, conditional and combined parameters (8
cases)
- `test-subscription-pending.ts` — `newPendingTransactions`: pending tx
hash notification
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Ahmad Kaouk <56095276+ahmadkaouk@users.noreply.github.com>
## Summary
We are now launching the MSP backend when starting stpragehub services.
In this PR, we also fix the MSP and BSP node configuration and register
it with the correct keys.
## What changed
* Added a launch Backend MSP function that is called when launching
storage hub services
* Fix the wrong genesis error message in storagehub node by removing the
`--chain dev` flags (so it can be launch of the same network as our
local datahaven nodes).
* Use the correct keys to register MSP and BSP. We were injecting
different keys that the one we used for MSP and BSP registration leading
to the MSP and BSP node to never fully register as storage providers.
---------
Co-authored-by: Ahmad Kaouk <56095276+ahmadkaouk@users.noreply.github.com>
## Summary
- Add the `mmr-gadget` to the DataHaven client for proper MMR leaf
indexing in offchain storage
- Gate the gadget on `offchain_worker.indexing_enabled` to avoid running
when indexing is disabled
- Enable efficient MMR proof queries by block number via the MMR RPC
## Problem
The DataHaven client was missing the `mmr-gadget`, which prevented MMR
leaves from being correctly indexed in the offchain database. Without
it:
- MMR proofs could only be queried by block hash, not block number
- Light clients and bridge relayers could not efficiently verify
finality
- The `mmr_generateProof` RPC had degraded functionality
## Changes
| File | Change |
|------|--------|
| `operator/Cargo.toml` | Add workspace deps for `mmr-gadget`,
`sp-mmr-primitives` |
| `operator/node/Cargo.toml` | Add node deps for `mmr-gadget`,
`sp-mmr-primitives` |
| `operator/node/src/service.rs` | Add import and spawn `MmrGadget`
after BEEFY gadget |
## Test plan
- [x] Build passes: `cd operator && cargo build --release --features
fast-runtime`
- [x] Run node with debug logging: `--log mmr-gadget=debug`
- [x] Verify `mmr-gadget` task starts in logs
- [x] Test MMR RPC by block number works:
```bash
curl -H "Content-Type: application/json" \
-d '{"id":1,"jsonrpc":"2.0","method":"mmr_generateProof","params":[[1],
null, null]}' \
http://localhost:9944
```
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Upgrades StorageHub dependencies from v0.3.3 to v0.3.5. This requires a
client upgrade.
## ⚠️ Breaking Changes ⚠️
Fisherman CLI options have been added to support specifying filtering
and ordering strategies for pending file deletions with reasonable
defaults:
- `--fisherman-filtering`: The filtering strategy [**`none` (default)**,
`ttl`]
- `--fisherman-ordering`: The ordering strategy [**`chronological`
(default)**, `randomized`]
- `--fisherman-ttl-threshold-seconds`: TTL for a file to be ignored for
deletion in seconds
MSP and BSP CLI options have been added to support specifying a specific
batch response and confirm size for MSP and BSP nodes with reasonable
defaults.
- `--bsp-confirm-file-batch-size`: How many storage requests to respond
to (confirming) in a single extrinsic call **(default: 20)**
- `--msp-respond-storage-batch-size`: How many storage requests to
respond to (accepting or rejecting) in a single extrinsic call
**(default: 20)**
This PR upgrades the StorageHub version to
[v0.3.1](https://github.com/Moonsong-Labs/storage-hub/releases/tag/v0.3.1).
The changes applied are the ones suggested in the corresponding release
notes, which in short are:
- Adding the `get_number_of_active_users_of_provider` runtime API to the
`PaymentStreams` pallet runtime APIs.
- Supporting `--max-open-forests` CLI param (has defaults).
- Supporting Prometheus telemetry.
IMPORTANT: This upgrade requires a Runtime upgrade as well as a Client
upgrade.
---------
Co-authored-by: Steve Degosserie <723552+stiiifff@users.noreply.github.com>
Upgrade to StorageHub version 0.3.0. This is a minor release, including
breaking changes.
## ⚠️ Breaking Changes ⚠️
The changes applied in this PR are according to the suggested changes in
StorageHub's [v0.3.0
release](https://github.com/Moonsong-Labs/storage-hub/releases/tag/v0.3.0)
---------
Co-authored-by: Steve Degosserie <723552+stiiifff@users.noreply.github.com>
## Summary
Re-add the static build feature option to bundle postgres dependency
into the binary. It simplify the installation because now to run the
node the operator doesn't need to have postgres dependencies installed
on its system.
## What changed ?
* Added a `static` feature that can be activated to add the extra
dependencies during the build.
* A task that run every time a dependency has been modified so we can
make sure the build with the feature is still working correctly. (we are
assuming simple change in the code won't have an impact on it because
postgres is being used in diesel which is not a direct dependecy to
datahaven).
## Summary
- Feature-gate `frame-benchmarking-cli` behind `runtime-benchmarks`
feature, making it an optional dependency
- Remove unused `cumulus-client-service` workspace dependency
- Remove unused `storage-hub-runtime` workspace dependency
- Add `#[cfg(feature = "runtime-benchmarks")]` guards to
benchmark-related code
## Motivation
The `frame-benchmarking-cli` crate pulls in
`cumulus-client-parachain-inherent` and other cumulus dependencies
transitively. Since DataHaven is a solochain (not a parachain), these
dependencies are unnecessary for regular builds.
By making the benchmarking CLI optional and only compiling it when the
`runtime-benchmarks` feature is enabled, we reduce:
- Compile time for regular development builds
- Final binary size (when not benchmarking)
- Dependency tree complexity
Co-authored-by: Claude <noreply@anthropic.com>
## ⚠️ Breaking Changes ⚠️
Upgrades to SH version 0.2.0. Breaking changes for this version are
outlined in the corresponding
[release](https://github.com/Moonsong-Labs/storage-hub/releases/tag/v0.2.0).
Particularly, in this PR, the following breaking changes are implemented
for DH node operators:
### Breaking CLI changes vs `main`
- **Fisherman vs provider role**
- `--fisherman` now has `conflicts_with = "provider"`
(`FishermanConfigurations::fisherman`).
- Any existing scripts that started a node with both `--provider` and
`--fisherman` will now fail clap validation.
- **Removed / replaced fisherman tuning flags**
- The following flags no longer exist and will cause errors if still
used:
- `--fisherman-incomplete-sync-max` (field
`fisherman_incomplete_sync_max`)
- `--fisherman-incomplete-sync-page-size` (field
`fisherman_incomplete_sync_page_size`)
- `--fisherman-sync-mode-min-blocks-behind` (field
`fisherman_sync_mode_min_blocks_behind`)
- They are replaced by:
- `--fisherman-batch-interval-seconds`
(`fisherman_batch_interval_seconds`, default `60`)
- `--fisherman-batch-deletion-limit` (`fisherman_batch_deletion_limit`,
default `1000`)
- **MSP DB wiring no longer piggybacks on the indexer DB**
- Previously, enabling the indexer (`IndexerConfigurations`) also wired
its DB pool into the MSP move‑bucket path via
`with_indexer_db_pool(maybe_indexer_db_pool)`.
- Now, MSP DB access is **only** configured if you pass the new
`--msp-database-url` provider flag; the indexer’s `--indexer` /
`--indexer-database-url` no longer implicitly provide DB access to MSP
logic. This will change behaviour for MSP nodes that relied on just the
indexer flags.
### New / additive CLI flags (non‑breaking but behaviourally relevant)
- **Provider flags**
- `--pending-db-url` (`pending_db_url`, env `SH_PENDING_DB_URL`) for
persisting pending extrinsics.
- `--internal-buffer-size` (`internal_buffer_size`, default `1024`) for
DB chunk batching during file transfer.
- **Reordered but unchanged**
- `--msp-distribute-files` still exists (bool flag), just moved within
`ProviderConfigurations`; name and type are unchanged, but now also
explicitly toggles `enable_msp_distribute_files` only when
`provider_type == msp`.
## Summary
- Build the node with Frontier’s rpc-binary-search-estimate feature so
eth_estimateGas runs the same iterative search as Moonbeam.
- Instead of returning the gas spent by a single max-allowance dry run,
the RPC now repeatedly replays the transaction while shrinking the gas
cap until it finds the smallest limit that still succeeds.
Update the 3 DataHaven environments' chain IDs & native token ticker as
follows:
* **Mainnet**
* **Chain ID**: 55930
* **Ticker**: HAVE
* **TestNet**
* **Chain ID**: 55931
* **Ticker**: MOCK
* **Stagenet**
* **Chain ID**: 55932
* **Ticker**: STAGE
The PR includes a storage migration for the Stagenet & Testnet
environments, that are already live, to update the EVM Chain ID stored
in the `pallet-evm-chain-id` pallet.
Note: the token symbol will only be updated with the genesis config
presets or newly generated chain specs. For already live networks, the
existing chain spec must be updated (i.e. the tokenSymbol property
changed) and used by all nodes in the network. This change in the chain
spec will not alter the chain genesis so it safe to do (in the very
early stages of the chain obviously).
---------
Co-authored-by: Claude <noreply@anthropic.com>
Upgrade to SH release
[v0.1.1](https://github.com/Moonsong-Labs/storage-hub/releases/tag/v0.1.1)
---------
Co-authored-by: Ahmad Kaouk <56095276+ahmadkaouk@users.noreply.github.com>
Co-authored-by: Ahmad Kaouk <ahmadkaouk.93@gmail.com>
Co-authored-by: Steve Degosserie <723552+stiiifff@users.noreply.github.com>
## Summary
- Fixes srtool build failure with `UnknownOpcode(252)` error during
metadata hash generation
- Removes hardcoded `WASM_BUILD_STD=0` to allow srtool to auto-detect
correct settings based on Rust version
- Adds `metadata-hash` feature to node Cargo.toml for CheckMetadataHash
extension
## Root Cause
The `WASM_BUILD_STD=0` setting forced srtool to use pre-built standard
library crates that were compiled with incompatible WASM features.
During metadata hash generation, the runtime builder encountered opcode
252 (likely from bulk memory operations) which the deserializer couldn't
recognize, causing the build to fail with:
```
thread 'main' panicked at metadata_hash.rs:73:10:
`Metadata::metadata_at_version` should exist.: RuntimeConstruction(Other("cannot deserialize module: UnknownOpcode(252)"))
```
## Changes
1. **operator/scripts/build-runtime-srtool.sh**: Removed hardcoded `-e
WASM_BUILD_STD=0` line
- Allows srtool to determine appropriate setting based on Rust version
- For Rust < 1.84: defaults to WASM_BUILD_STD=1 (enabled)
- For Rust >= 1.84: defaults to WASM_BUILD_STD=0 (disabled)
2. **operator/node/Cargo.toml**: Added `metadata-hash` feature
propagation
- Enables metadata-hash feature for all runtime variants (stagenet,
mainnet, testnet)
- Required for CheckMetadataHash extension support
## Test Plan
- [x] Successfully built stagenet runtime with srtool 1.88.0
- [x] Build completed in ~14 minutes without metadata hash errors
- [x] Verified WASM artifacts generated correctly
## Testing Command
```bash
GH_WORKFLOW_MATRIX_CHAIN=stagenet \
RUNTIME_BUILD_OPTS="--features=on-chain-release-build" \
RUNTIME_BUILD_PROFILE="production" \
GH_WORKFLOW_MATRIX_SRTOOL_IMAGE="paritytech/srtool" \
GH_WORKFLOW_MATRIX_SRTOOL_IMAGE_TAG="1.88.0" \
WASM_BUILD_STD=1 \
./operator/scripts/build-runtime-srtool.sh
```
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude <noreply@anthropic.com>
### Description
This PR introduces the **Moonwall** end-to-end (E2E) testing framework.
The primary motivation for this is to enable the porting of existing
Mobeam tests into the `DataHaven` repository.
### Key Changes
* **Node Manual Sealing:**
* Introduced a `--sealing=manual` flag for the `datahaven-node`. When
enabled, blocks are only produced on demand via an RPC call. This is the
core mechanism that allows for deterministic tests.
* **Moonwall Framework Integration:**
* Added `@moonwall/cli` and `@moonwall/util` dependencies to the
`test/package.json`.
* A new `test/moonwall.config.json` file configures the test
environment, defining how Moonwall should launch the `datahaven-node`
with the manual sealing flag.
* Added a `moonwall:test` script to `package.json` for running the
tests.
* **CI Workflow:**
* A new reusable workflow, `.github/workflows/task-moonwall-tests.yml`,
has been created to handle the setup, execution, and reporting of
Moonwall tests.
* The main `CI.yml` now includes a `moonwall-tests` job that runs after
the `build-operator` job, ensuring it always tests the correct,
freshly-built binary.
* **Example Test Suite:**
* A new test suite, `test/datahaven/suites/dev/test-block.ts`, had been
copied from moonbeam.
### How to Run Locally
1. Navigate to the `test` directory.
2. Install dependencies: `bun install`
3. Run the tests: `bun run moonwall:test`
---------
Co-authored-by: undercover-cactus <lola@moonsonglabs.com>
In this PR we update StorageHub to its latest tage `v0.0.4-alpha`. It
includes so minor changes on the FilesystemAPI.
Its also force to specify the `maintenance_mode` options for fisherman.
For now we are not allowing this mode so we just skip entirely.
---------
Co-authored-by: Steve Degosserie <723552+stiiifff@users.noreply.github.com>
Added missing CLI option for the indexer configuration for Storage Hub.
Indexer is needed for MSP user.
---------
Co-authored-by: Steve Degosserie <723552+stiiifff@users.noreply.github.com>
## Summary
- Upgraded Rust toolchain from 1.87 to 1.88 for latest compiler
improvements
- Updated Docker build environment with latest paritytech base image and
build tools
- Fixed node branding and support URL to reflect DataHaven project
## Changes
### Rust Toolchain Update (`rust-toolchain.toml`)
- Upgraded from Rust 1.87 to 1.88
### Docker Build Environment (`Dockerfile`)
- Updated base image from `paritytech/ci-unified:bullseye-1.85.0` to
`bullseye-1.88.0`
- Upgraded mold linker from v2.39.0 to v2.40.4 for faster builds
- Added protoc v21.12 for protobuf compilation support
- Added libpq-dev dependency for PostgreSQL integration
- Updated cargo-chef from 0.1.71 to 0.1.72
---------
Co-authored-by: Claude <noreply@anthropic.com>
This PR updates the project dependencies as follows:
- **Polkadot SDK**: from the `stable2412` (moving) branch to the
`polkadot-stable2412-6` (fixed) tag.
- **StorageHub**: to revision
`f8281283b6003a3009a32431ed0f3cd628561d6b`, which also depends on
Polkadot SDK `polkadot-stable2412-6`.
- **Frontier**: revision `75329a2df49e2cc7981485392c31160929d1bd48n`
which, likewise, depends on Polkadot SDK `polkadot-stable2412-6`.
In this PR we add StorageHub pallets to the runtime. It will only be
added if the `storage-hub` feature is activated.
---------
Co-authored-by: Gonza Montiel <gonzamontiel@users.noreply.github.com>
In this PR:
1. Add new `datahaven-node-local.dockerfile` for building a local image
with the locally built DataHaven Node binary. This severely improves
iteration speed on running `bun cli` if there are changes in the DH
node. Previously, it relied on the published dockerfile, which builds
the Cargo project inside of it, taking +20m even with no changes.
2. Building this local dockerfile is integrated to the CLI, which now
also asks if the user wants to rebuild the local docker image of the DH
node.
3. A new script `cargo-crossbuild` is added, to be able to build the
DataHaven node Cargo project both from Mac and Linux, with the target
being `x86_64-unknown-linux-gnu`. For building from Mac, it uses `cargo
zigbuild`, so `zig` is now a dependency. Building for this target is
needed because the docker image is an Ubuntu image, so it will need to
run a linux binary.
4. Added `zig` as dependency in docs.
5. CI still uses the docker image built by the CI itself, which builds
the Cargo project inside of it. The CI can take advantage of caching for
this.
In this PR:
1. Add [Polkadot API](https://papi.how/) support to `test/` directory
(i.e. the e2e CLI and testing framework) to be able to interact with the
Substrate chain.
1. This allows typed interactions with transactions, constants, storage,
runtime APIs, and non-typed interaction with RPC methods.
2. Types are autogenerated when running `bun i`, from the
`datahaven.scale` file that is part of this repo's version control.
2. Add new utilities file to `papi` related functionalities. For the
time being, generating a new signer from a private key.
3. ~Add a new step to the CLI that sends a transaction to the DataHaven
network. _*THIS SHOULD BE REMOVED SOON, IT'S JUST FOR TESTING
PURPOSES*_~
1. Both steps that send test transactions have been removed from the
CLI, for convenience and being error prone. Their scripts remain usable
for testing purposes if needed.
4. Removes the `apis.rs` files from the runtime definitions. Having them
in a separate file meant that the runtime was not including the Runtime
APIs in the metadata blob, preventing `papi` from creating types from
it. This change can be reapplied after upgrading to `polkadot-sdk`
`stable-2503`. More info
[here](https://github.com/paritytech/polkadot-sdk/issues/6659).
5. Add script to re-generate types, and corresponding docs.
6. Makes logger synchronous to avoid prints happening before logs.
Add support for multiple runtimes: `stagenet`, `testnet` & `mainnet`.
- Moved common types to `datahaven-runtime-common` crate.
- Made the node's command & service code generic over different
runtimes. Each runtime has a `dev` & `local` genesis config preset.
- More types / constants can be moved to the `datahaven-runtime-common`
crate ... this will be done in subsequent PRs.
---------
Co-authored-by: Gonza Montiel <gon.montiel@gmail.com>
Co-authored-by: Gonza Montiel <gonzamontiel@users.noreply.github.com>
Co-authored-by: Facundo Farall <37149322+ffarall@users.noreply.github.com>
This PR adds Ethereum RPC API support to the node, enabling the
interaction with the node using standard Ethereum tools.
1. Integration of Frontier's Ethereum RPC modules (eth, net, web3,
txpool)
2. Added RPC configurations necessary for Ethereum compatibility
3. Implemented a BABE consensus data provider for handling pending
blocks
4. Added required dependencies and configurations in the node's service
5. Created necessary filter pools, block data caches, and notification
systems
6. Configured the RPC system to handle Ethereum API calls
7. Changed the transaction pool implementation to be compatible with the
fork-aware transaction pool
(https://github.com/paritytech/polkadot-sdk/pull/4639)
EDIT:
The new `transaction_pool` in `polkadot-stable2412` that comes with a
`fork-aware` feature, allowing different views of the transaction pool,
is not compatible with the current state in Frontier.
In Frontier the only supported pool is the `BasicPool`, that maintains a
similar behaviour to the one in `polkadot-stable2409`. I used this pool
directly.
---------
Co-authored-by: Ahmad Kaouk <56095276+ahmadkaouk@users.noreply.github.com>
Co-authored-by: Facundo Farall <37149322+ffarall@users.noreply.github.com>
This PR adds the Snowbridge Inbound Queue V2 pallet to the runtime. It
also defines bridge primitives, including the message structure for
EigenLayer, and introduces the EigenLayerMessageProcessor to handle
messages coming from EigenLayer.
---------
Co-authored-by: Facundo Farall <37149322+ffarall@users.noreply.github.com>
