In this PR we have a fix for the linter. We also allow for the non-null
assertion operator `!` to be used in typescript.
The typescript being used to setup tests and some convenient scripts we
should allow some the convenient feature of typescript.
Co-authored-by: Ahmad Kaouk <56095276+ahmadkaouk@users.noreply.github.com>
### Description
This PR introduces the **Moonwall** end-to-end (E2E) testing framework.
The primary motivation for this is to enable the porting of existing
Mobeam tests into the `DataHaven` repository.
### Key Changes
* **Node Manual Sealing:**
* Introduced a `--sealing=manual` flag for the `datahaven-node`. When
enabled, blocks are only produced on demand via an RPC call. This is the
core mechanism that allows for deterministic tests.
* **Moonwall Framework Integration:**
* Added `@moonwall/cli` and `@moonwall/util` dependencies to the
`test/package.json`.
* A new `test/moonwall.config.json` file configures the test
environment, defining how Moonwall should launch the `datahaven-node`
with the manual sealing flag.
* Added a `moonwall:test` script to `package.json` for running the
tests.
* **CI Workflow:**
* A new reusable workflow, `.github/workflows/task-moonwall-tests.yml`,
has been created to handle the setup, execution, and reporting of
Moonwall tests.
* The main `CI.yml` now includes a `moonwall-tests` job that runs after
the `build-operator` job, ensuring it always tests the correct,
freshly-built binary.
* **Example Test Suite:**
* A new test suite, `test/datahaven/suites/dev/test-block.ts`, had been
copied from moonbeam.
### How to Run Locally
1. Navigate to the `test` directory.
2. Install dependencies: `bun install`
3. Run the tests: `bun run moonwall:test`
---------
Co-authored-by: undercover-cactus <lola@moonsonglabs.com>
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Documentation**
- Added detailed IDE configuration recommendations for Rust, Solidity,
and TypeScript in the README to enhance developer experience.
- **Chores**
- Updated Biome configuration files and package dependencies to the
latest schema and version.
- Refined code formatting, linting, and import organization settings for
consistency across the project.
- **Refactor**
- Reordered import statements in multiple files for improved
readability.
- Simplified function signatures and ensured proper async handling in
utility scripts.
- **Bug Fixes**
- Ensured proper completion of asynchronous operations in shell utility
functions.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Gonza Montiel <gonzamontiel@users.noreply.github.com>
In this PR:
1. Add new `datahaven-node-local.dockerfile` for building a local image
with the locally built DataHaven Node binary. This severely improves
iteration speed on running `bun cli` if there are changes in the DH
node. Previously, it relied on the published dockerfile, which builds
the Cargo project inside of it, taking +20m even with no changes.
2. Building this local dockerfile is integrated to the CLI, which now
also asks if the user wants to rebuild the local docker image of the DH
node.
3. A new script `cargo-crossbuild` is added, to be able to build the
DataHaven node Cargo project both from Mac and Linux, with the target
being `x86_64-unknown-linux-gnu`. For building from Mac, it uses `cargo
zigbuild`, so `zig` is now a dependency. Building for this target is
needed because the docker image is an Ubuntu image, so it will need to
run a linux binary.
4. Added `zig` as dependency in docs.
5. CI still uses the docker image built by the CI itself, which builds
the Cargo project inside of it. The CI can take advantage of caching for
this.
> [!NOTE]
> This is `Part 3` of the ongoing _Docker Series._
## New Additions:
- Launching Datahaven network will spin up containers, as opposed to
native binaries
- `stop:docker` script to kill all dh containers
- `e2e` test suite for datahaven solochain network
- Contains reference test file that uses papi for storage queries,
submitting exts, runtime calls (good job on that facu and tobi)
- Added new utils:
- `waitForLog()` to wait for log lines in docker container logs
- `createPapiConnectors()` helper for test cases to build and connect to
dh network
- `getPapiSigner()` helper to return a papi compatible signer using our
prefunded accounts (alith by default)
- `sendTxn()` helper to submit txn and wait for block inclusion, instead
of finalization, which std library provides
## Changes:
> [!CAUTION]
> Launching native binaries for datahaven no longer supported.
- Datahaven binary location cli option changed to `-i,
--datahaven-image-tag`
- To locally run this you'll need a datahaven docker image handy, you'll
need to either:
- Point to remote dockerhub e.g. `moonsonglabs/datahaven:main` (must be
logged in and have permission)
- Build this locally with `bun build:docker:operator`
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Added end-to-end tests for the Datahaven solochain, including runtime
API queries, storage lookups, extrinsic submissions, and event
listening.
- Introduced CLI option to specify the Datahaven Docker image tag, with
a default value.
- Added CLI option to disable the Relayer.
- Provided new scripts to stop Docker containers associated with
Datahaven.
- Added utility functions for Docker log monitoring and container
startup checks.
- Introduced utilities for interacting with the Datahaven Polkadot API.
- **Improvements**
- Switched Datahaven network launch from local binaries to Docker
containers.
- Enhanced cache accuracy in build workflows by including Rust source
files in cache keys.
- Improved build performance with TypeScript incremental build options.
- Increased timeout for end-to-end tests for better reliability.
- Updated CLI version to 0.2.0.
- Modified Dockerfile build to enable the `fast-runtime` feature.
- Extended network launch summary to include relayer and container
details.
- **Bug Fixes**
- Fixed cleanup logic by tracking and preparing for forced removal of
Docker containers after tests.
- **Chores**
- Updated workflow steps for Docker image handling and network checks.
- Adjusted scripts and workflow logic for improved Docker and test
management.
- Removed top-level disk usage summaries from cleanup workflow for
streamlined reporting.
- Enhanced shell command utility to support asynchronous wait during
execution.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Facundo Farall <37149322+ffarall@users.noreply.github.com>
In this PR:
1. Add [Polkadot API](https://papi.how/) support to `test/` directory
(i.e. the e2e CLI and testing framework) to be able to interact with the
Substrate chain.
1. This allows typed interactions with transactions, constants, storage,
runtime APIs, and non-typed interaction with RPC methods.
2. Types are autogenerated when running `bun i`, from the
`datahaven.scale` file that is part of this repo's version control.
2. Add new utilities file to `papi` related functionalities. For the
time being, generating a new signer from a private key.
3. ~Add a new step to the CLI that sends a transaction to the DataHaven
network. _*THIS SHOULD BE REMOVED SOON, IT'S JUST FOR TESTING
PURPOSES*_~
1. Both steps that send test transactions have been removed from the
CLI, for convenience and being error prone. Their scripts remain usable
for testing purposes if needed.
4. Removes the `apis.rs` files from the runtime definitions. Having them
in a separate file meant that the runtime was not including the Runtime
APIs in the metadata blob, preventing `papi` from creating types from
it. This change can be reapplied after upgrading to `polkadot-sdk`
`stable-2503`. More info
[here](https://github.com/paritytech/polkadot-sdk/issues/6659).
5. Add script to re-generate types, and corresponding docs.
6. Makes logger synchronous to avoid prints happening before logs.
## Human Written Description
This PR adds the following to the E2E CLI:
- Relayer config generation for: `beacon-relay` `beefy-relay`
- The other two relayer types to be added later
- Relayers don't actually work yet
- By default turned off, this requires a binary to be present in:
`<repo_root>/operator/target/release` dir
- Datahaven network launching
- DH network is using default `local` network chain spec
- Launched with 5 nodes since our authority set is 6 large (and you need
2/3 + 1 of set size
> [!NOTE]
> Both the relayer and the DH node binaries are being run as local
processes TEMPORARILY. This means that logging is done in a very
rudimentary way (we pipe to a file whilst the CLI is running).
>
> This means that when the CLI finishes **the log files will no longer
be written to**.
> This is temporary since spawning binaries is a stop gap solution until
docker images available.
---
> [!IMPORTANT]
> The following is AI generated slop describing this PR's changes:
**Key Changes:**
* **CLI Enhancements (`test/cli/index.ts`):**
* Added options `--datahaven` and `--datahaven-bin-path` to enable
launching local DataHaven nodes.
* Added options `--relayer` and `--relayer-bin-path` to enable launching
Snowbridge relayers (Beefy and Beacon).
* Added negation flags (`--no-fund-validators`, `--no-setup-validators`,
`--no-update-validator-set`) for more granular control over validator
setup steps.
* Added `--skip-cleaning` option to preserve Kurtosis state between
runs.
* Added a pre-action hook (`launchPreActionHook`) to validate flag
combinations (e.g., `--verified` requires `--blockscout`).
* **New CLI Handlers (`test/cli/handlers/launch/`):**
* `datahaven.ts`: Logic for spawning DataHaven node processes using the
specified binary. Manages ports and process cleanup.
* `relayer.ts`: Logic for configuring and spawning Snowbridge relayer
processes (Beefy and Beacon). Reads contract deployment addresses,
updates relayer config templates, and uses specified private keys.
Manages log files and process cleanup.
* `summary.ts`: Generates and displays the table of running services
(including dynamically launched DataHaven nodes) and their endpoints.
* `validator.ts`: Extracted validator funding, setup, and set update
logic into its own handler.
* `index.ts`: Orchestrates the launch sequence based on CLI options,
calling the appropriate handlers. Includes a `LaunchedNetwork` class to
track spawned processes, file descriptors, and node ports for cleanup.
* **Updated `package.json` Scripts:**
* Added `start:e2e:minrelayer` script for a minimal setup including
relayers and DataHaven nodes.
* Modified `stop:e2e` to include `pkill datahaven` for proper cleanup.
* Added `stop:e2e:quick` to only stop the Kurtosis enclave without full
cleaning.
* **Updated `launch-kurtosis.ts`:** Modified to use new Kurtosis utility
functions and added a `skipCleaning` option.
* **New Utility Functions:**
* `test/utils/kurtosis.ts`: Functions to inspect Kurtosis services
(`getServiceFromKurtosis`, `getPortFromKurtosis`,
`getServicesFromKurtosis`).
* `test/utils/parser.ts`: Zod schemas and parsing functions for
Snowbridge relayer configurations.
* **Constants & Minor Updates:**
* Added `SUBSTRATE_FUNDED_ACCOUNTS` to `test/utils/constants.ts`.
* Updated `tsconfig.json` include paths.
* Refactored `test/utils/docker.ts` (though now largely superseded by
Kurtosis utils).
* Updated logging in `test/scripts/send-txn.ts`.
**Reasoning:**
This PR significantly expands the E2E testing capabilities by allowing
developers to easily launch and integrate local DataHaven nodes and
Snowbridge relayers into the test network, facilitating more
comprehensive integration testing. The CLI refactoring makes managing
these complex setups more robust and user-friendly.
This PR adds a script one-liner to spin up an ethereum network, with
attached explorers and deploys our `/contracts` folder into there.
---------
Co-authored-by: Facundo Farall <37149322+ffarall@users.noreply.github.com>
