feat: ✨ Add Polkadot API support to CLI and e2e testing infra (#68)
In this PR:
1. Add [Polkadot API](https://papi.how/) support to `test/` directory
(i.e. the e2e CLI and testing framework) to be able to interact with the
Substrate chain.
1. This allows typed interactions with transactions, constants, storage,
runtime APIs, and non-typed interaction with RPC methods.
2. Types are autogenerated when running `bun i`, from the
`datahaven.scale` file that is part of this repo's version control.
2. Add new utilities file to `papi` related functionalities. For the
time being, generating a new signer from a private key.
3. ~Add a new step to the CLI that sends a transaction to the DataHaven
network. _*THIS SHOULD BE REMOVED SOON, IT'S JUST FOR TESTING
PURPOSES*_~
1. Both steps that send test transactions have been removed from the
CLI, for convenience and being error prone. Their scripts remain usable
for testing purposes if needed.
4. Removes the `apis.rs` files from the runtime definitions. Having them
in a separate file meant that the runtime was not including the Runtime
APIs in the metadata blob, preventing `papi` from creating types from
it. This change can be reapplied after upgrading to `polkadot-sdk`
`stable-2503`. More info
[here](https://github.com/paritytech/polkadot-sdk/issues/6659).
5. Add script to re-generate types, and corresponding docs.
6. Makes logger synchronous to avoid prints happening before logs.
2025-05-13 03:03:21 +00:00
|
|
|
import { secp256k1 } from "@noble/curves/secp256k1";
|
|
|
|
|
import { keccak_256 } from "@noble/hashes/sha3";
|
test: 🐳 Add docker support for datahaven nodes (#71)
> [!NOTE]
> This is `Part 3` of the ongoing _Docker Series._
## New Additions:
- Launching Datahaven network will spin up containers, as opposed to
native binaries
- `stop:docker` script to kill all dh containers
- `e2e` test suite for datahaven solochain network
- Contains reference test file that uses papi for storage queries,
submitting exts, runtime calls (good job on that facu and tobi)
- Added new utils:
- `waitForLog()` to wait for log lines in docker container logs
- `createPapiConnectors()` helper for test cases to build and connect to
dh network
- `getPapiSigner()` helper to return a papi compatible signer using our
prefunded accounts (alith by default)
- `sendTxn()` helper to submit txn and wait for block inclusion, instead
of finalization, which std library provides
## Changes:
> [!CAUTION]
> Launching native binaries for datahaven no longer supported.
- Datahaven binary location cli option changed to `-i,
--datahaven-image-tag`
- To locally run this you'll need a datahaven docker image handy, you'll
need to either:
- Point to remote dockerhub e.g. `moonsonglabs/datahaven:main` (must be
logged in and have permission)
- Build this locally with `bun build:docker:operator`
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Added end-to-end tests for the Datahaven solochain, including runtime
API queries, storage lookups, extrinsic submissions, and event
listening.
- Introduced CLI option to specify the Datahaven Docker image tag, with
a default value.
- Added CLI option to disable the Relayer.
- Provided new scripts to stop Docker containers associated with
Datahaven.
- Added utility functions for Docker log monitoring and container
startup checks.
- Introduced utilities for interacting with the Datahaven Polkadot API.
- **Improvements**
- Switched Datahaven network launch from local binaries to Docker
containers.
- Enhanced cache accuracy in build workflows by including Rust source
files in cache keys.
- Improved build performance with TypeScript incremental build options.
- Increased timeout for end-to-end tests for better reliability.
- Updated CLI version to 0.2.0.
- Modified Dockerfile build to enable the `fast-runtime` feature.
- Extended network launch summary to include relayer and container
details.
- **Bug Fixes**
- Fixed cleanup logic by tracking and preparing for forced removal of
Docker containers after tests.
- **Chores**
- Updated workflow steps for Docker image handling and network checks.
- Adjusted scripts and workflow logic for improved Docker and test
management.
- Removed top-level disk usage summaries from cleanup workflow for
streamlined reporting.
- Enhanced shell command utility to support asynchronous wait during
execution.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Facundo Farall <37149322+ffarall@users.noreply.github.com>
2025-05-16 14:17:05 +00:00
|
|
|
import { datahaven } from "@polkadot-api/descriptors";
|
2025-05-19 22:28:43 +00:00
|
|
|
import { createClient, type PolkadotClient, type TypedApi } from "polkadot-api";
|
test: 🐳 Add docker support for datahaven nodes (#71)
> [!NOTE]
> This is `Part 3` of the ongoing _Docker Series._
## New Additions:
- Launching Datahaven network will spin up containers, as opposed to
native binaries
- `stop:docker` script to kill all dh containers
- `e2e` test suite for datahaven solochain network
- Contains reference test file that uses papi for storage queries,
submitting exts, runtime calls (good job on that facu and tobi)
- Added new utils:
- `waitForLog()` to wait for log lines in docker container logs
- `createPapiConnectors()` helper for test cases to build and connect to
dh network
- `getPapiSigner()` helper to return a papi compatible signer using our
prefunded accounts (alith by default)
- `sendTxn()` helper to submit txn and wait for block inclusion, instead
of finalization, which std library provides
## Changes:
> [!CAUTION]
> Launching native binaries for datahaven no longer supported.
- Datahaven binary location cli option changed to `-i,
--datahaven-image-tag`
- To locally run this you'll need a datahaven docker image handy, you'll
need to either:
- Point to remote dockerhub e.g. `moonsonglabs/datahaven:main` (must be
logged in and have permission)
- Build this locally with `bun build:docker:operator`
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Added end-to-end tests for the Datahaven solochain, including runtime
API queries, storage lookups, extrinsic submissions, and event
listening.
- Introduced CLI option to specify the Datahaven Docker image tag, with
a default value.
- Added CLI option to disable the Relayer.
- Provided new scripts to stop Docker containers associated with
Datahaven.
- Added utility functions for Docker log monitoring and container
startup checks.
- Introduced utilities for interacting with the Datahaven Polkadot API.
- **Improvements**
- Switched Datahaven network launch from local binaries to Docker
containers.
- Enhanced cache accuracy in build workflows by including Rust source
files in cache keys.
- Improved build performance with TypeScript incremental build options.
- Increased timeout for end-to-end tests for better reliability.
- Updated CLI version to 0.2.0.
- Modified Dockerfile build to enable the `fast-runtime` feature.
- Extended network launch summary to include relayer and container
details.
- **Bug Fixes**
- Fixed cleanup logic by tracking and preparing for forced removal of
Docker containers after tests.
- **Chores**
- Updated workflow steps for Docker image handling and network checks.
- Adjusted scripts and workflow logic for improved Docker and test
management.
- Removed top-level disk usage summaries from cleanup workflow for
streamlined reporting.
- Enhanced shell command utility to support asynchronous wait during
execution.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Facundo Farall <37149322+ffarall@users.noreply.github.com>
2025-05-16 14:17:05 +00:00
|
|
|
import { withPolkadotSdkCompat } from "polkadot-api/polkadot-sdk-compat";
|
2025-05-19 22:28:43 +00:00
|
|
|
import { getPolkadotSigner, type PolkadotSigner } from "polkadot-api/signer";
|
test: 🐳 Add docker support for datahaven nodes (#71)
> [!NOTE]
> This is `Part 3` of the ongoing _Docker Series._
## New Additions:
- Launching Datahaven network will spin up containers, as opposed to
native binaries
- `stop:docker` script to kill all dh containers
- `e2e` test suite for datahaven solochain network
- Contains reference test file that uses papi for storage queries,
submitting exts, runtime calls (good job on that facu and tobi)
- Added new utils:
- `waitForLog()` to wait for log lines in docker container logs
- `createPapiConnectors()` helper for test cases to build and connect to
dh network
- `getPapiSigner()` helper to return a papi compatible signer using our
prefunded accounts (alith by default)
- `sendTxn()` helper to submit txn and wait for block inclusion, instead
of finalization, which std library provides
## Changes:
> [!CAUTION]
> Launching native binaries for datahaven no longer supported.
- Datahaven binary location cli option changed to `-i,
--datahaven-image-tag`
- To locally run this you'll need a datahaven docker image handy, you'll
need to either:
- Point to remote dockerhub e.g. `moonsonglabs/datahaven:main` (must be
logged in and have permission)
- Build this locally with `bun build:docker:operator`
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Added end-to-end tests for the Datahaven solochain, including runtime
API queries, storage lookups, extrinsic submissions, and event
listening.
- Introduced CLI option to specify the Datahaven Docker image tag, with
a default value.
- Added CLI option to disable the Relayer.
- Provided new scripts to stop Docker containers associated with
Datahaven.
- Added utility functions for Docker log monitoring and container
startup checks.
- Introduced utilities for interacting with the Datahaven Polkadot API.
- **Improvements**
- Switched Datahaven network launch from local binaries to Docker
containers.
- Enhanced cache accuracy in build workflows by including Rust source
files in cache keys.
- Improved build performance with TypeScript incremental build options.
- Increased timeout for end-to-end tests for better reliability.
- Updated CLI version to 0.2.0.
- Modified Dockerfile build to enable the `fast-runtime` feature.
- Extended network launch summary to include relayer and container
details.
- **Bug Fixes**
- Fixed cleanup logic by tracking and preparing for forced removal of
Docker containers after tests.
- **Chores**
- Updated workflow steps for Docker image handling and network checks.
- Adjusted scripts and workflow logic for improved Docker and test
management.
- Removed top-level disk usage summaries from cleanup workflow for
streamlined reporting.
- Enhanced shell command utility to support asynchronous wait during
execution.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Facundo Farall <37149322+ffarall@users.noreply.github.com>
2025-05-16 14:17:05 +00:00
|
|
|
import { getWsProvider } from "polkadot-api/ws-provider/web";
|
|
|
|
|
import { SUBSTRATE_FUNDED_ACCOUNTS } from "./constants";
|
|
|
|
|
import type { Prettify } from "./types";
|
feat: ✨ Add Polkadot API support to CLI and e2e testing infra (#68)
In this PR:
1. Add [Polkadot API](https://papi.how/) support to `test/` directory
(i.e. the e2e CLI and testing framework) to be able to interact with the
Substrate chain.
1. This allows typed interactions with transactions, constants, storage,
runtime APIs, and non-typed interaction with RPC methods.
2. Types are autogenerated when running `bun i`, from the
`datahaven.scale` file that is part of this repo's version control.
2. Add new utilities file to `papi` related functionalities. For the
time being, generating a new signer from a private key.
3. ~Add a new step to the CLI that sends a transaction to the DataHaven
network. _*THIS SHOULD BE REMOVED SOON, IT'S JUST FOR TESTING
PURPOSES*_~
1. Both steps that send test transactions have been removed from the
CLI, for convenience and being error prone. Their scripts remain usable
for testing purposes if needed.
4. Removes the `apis.rs` files from the runtime definitions. Having them
in a separate file meant that the runtime was not including the Runtime
APIs in the metadata blob, preventing `papi` from creating types from
it. This change can be reapplied after upgrading to `polkadot-sdk`
`stable-2503`. More info
[here](https://github.com/paritytech/polkadot-sdk/issues/6659).
5. Add script to re-generate types, and corresponding docs.
6. Makes logger synchronous to avoid prints happening before logs.
2025-05-13 03:03:21 +00:00
|
|
|
|
|
|
|
|
// A signer for EVM like chains that use AccountId20 as their public address
|
|
|
|
|
export const getEvmEcdsaSigner = (privateKey: string): PolkadotSigner => {
|
|
|
|
|
const privateKeyBytes = Buffer.from(
|
|
|
|
|
privateKey.startsWith("0x") ? privateKey.slice(2) : privateKey,
|
|
|
|
|
"hex"
|
|
|
|
|
);
|
|
|
|
|
const publicAddress = keccak_256(secp256k1.getPublicKey(privateKeyBytes, false).slice(1)).slice(
|
|
|
|
|
-20
|
|
|
|
|
);
|
|
|
|
|
|
|
|
|
|
return getPolkadotSigner(publicAddress, "Ecdsa", (input) =>
|
|
|
|
|
signEcdsa(keccak_256, input, privateKeyBytes)
|
|
|
|
|
);
|
|
|
|
|
};
|
|
|
|
|
|
|
|
|
|
export const signEcdsa = (
|
|
|
|
|
hasher: (input: Uint8Array) => Uint8Array,
|
|
|
|
|
value: Uint8Array,
|
|
|
|
|
priv: Uint8Array
|
|
|
|
|
) => {
|
|
|
|
|
const signature = secp256k1.sign(hasher(value), priv);
|
|
|
|
|
const signedBytes = signature.toCompactRawBytes();
|
|
|
|
|
|
|
|
|
|
const result = new Uint8Array(signedBytes.length + 1);
|
|
|
|
|
result.set(signedBytes);
|
|
|
|
|
result[signedBytes.length] = signature.recovery;
|
|
|
|
|
|
|
|
|
|
return result;
|
|
|
|
|
};
|
test: 🐳 Add docker support for datahaven nodes (#71)
> [!NOTE]
> This is `Part 3` of the ongoing _Docker Series._
## New Additions:
- Launching Datahaven network will spin up containers, as opposed to
native binaries
- `stop:docker` script to kill all dh containers
- `e2e` test suite for datahaven solochain network
- Contains reference test file that uses papi for storage queries,
submitting exts, runtime calls (good job on that facu and tobi)
- Added new utils:
- `waitForLog()` to wait for log lines in docker container logs
- `createPapiConnectors()` helper for test cases to build and connect to
dh network
- `getPapiSigner()` helper to return a papi compatible signer using our
prefunded accounts (alith by default)
- `sendTxn()` helper to submit txn and wait for block inclusion, instead
of finalization, which std library provides
## Changes:
> [!CAUTION]
> Launching native binaries for datahaven no longer supported.
- Datahaven binary location cli option changed to `-i,
--datahaven-image-tag`
- To locally run this you'll need a datahaven docker image handy, you'll
need to either:
- Point to remote dockerhub e.g. `moonsonglabs/datahaven:main` (must be
logged in and have permission)
- Build this locally with `bun build:docker:operator`
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Added end-to-end tests for the Datahaven solochain, including runtime
API queries, storage lookups, extrinsic submissions, and event
listening.
- Introduced CLI option to specify the Datahaven Docker image tag, with
a default value.
- Added CLI option to disable the Relayer.
- Provided new scripts to stop Docker containers associated with
Datahaven.
- Added utility functions for Docker log monitoring and container
startup checks.
- Introduced utilities for interacting with the Datahaven Polkadot API.
- **Improvements**
- Switched Datahaven network launch from local binaries to Docker
containers.
- Enhanced cache accuracy in build workflows by including Rust source
files in cache keys.
- Improved build performance with TypeScript incremental build options.
- Increased timeout for end-to-end tests for better reliability.
- Updated CLI version to 0.2.0.
- Modified Dockerfile build to enable the `fast-runtime` feature.
- Extended network launch summary to include relayer and container
details.
- **Bug Fixes**
- Fixed cleanup logic by tracking and preparing for forced removal of
Docker containers after tests.
- **Chores**
- Updated workflow steps for Docker image handling and network checks.
- Adjusted scripts and workflow logic for improved Docker and test
management.
- Removed top-level disk usage summaries from cleanup workflow for
streamlined reporting.
- Enhanced shell command utility to support asynchronous wait during
execution.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Facundo Farall <37149322+ffarall@users.noreply.github.com>
2025-05-16 14:17:05 +00:00
|
|
|
|
|
|
|
|
export const createPapiConnectors = (
|
|
|
|
|
wsUrl?: string
|
|
|
|
|
): { client: PolkadotClient; typedApi: DataHavenApi } => {
|
|
|
|
|
const url = wsUrl ?? "ws://127.0.0.1:9944";
|
|
|
|
|
const client = createClient(withPolkadotSdkCompat(getWsProvider(url)));
|
|
|
|
|
return { client, typedApi: client.getTypedApi(datahaven) };
|
|
|
|
|
};
|
|
|
|
|
|
|
|
|
|
export const getPapiSigner = (person: keyof typeof SUBSTRATE_FUNDED_ACCOUNTS = "ALITH") =>
|
|
|
|
|
getEvmEcdsaSigner(SUBSTRATE_FUNDED_ACCOUNTS[person].privateKey);
|
|
|
|
|
|
|
|
|
|
export type DataHavenApi = Prettify<TypedApi<typeof datahaven>>;
|