In this PR, we remove the caching of the sccache folder because it is
too big (~3GB) and fill our cache too fast.
What to expect ?
* It will make the build a bit slower but it is fine because it only
build on `main`. We are preparing another PR that will speed up the
build of the prod image. Also we are not sure the cache is actually
being used (`gha` cache is in beta).
* Will free some space for caching and stop deleting our cache which
make other jobs work faster.
Co-authored-by: Ahmad Kaouk <56095276+ahmadkaouk@users.noreply.github.com>
Co-authored-by: Gonza Montiel <gonzamontiel@users.noreply.github.com>
## Summary
- Upgraded Rust toolchain from 1.87 to 1.88 for latest compiler
improvements
- Updated Docker build environment with latest paritytech base image and
build tools
- Fixed node branding and support URL to reflect DataHaven project
## Changes
### Rust Toolchain Update (`rust-toolchain.toml`)
- Upgraded from Rust 1.87 to 1.88
### Docker Build Environment (`Dockerfile`)
- Updated base image from `paritytech/ci-unified:bullseye-1.85.0` to
`bullseye-1.88.0`
- Upgraded mold linker from v2.39.0 to v2.40.4 for faster builds
- Added protoc v21.12 for protobuf compilation support
- Added libpq-dev dependency for PostgreSQL integration
- Updated cargo-chef from 0.1.71 to 0.1.72
---------
Co-authored-by: Claude <noreply@anthropic.com>
## Changes
- New CI file for making Docker Prod images
- Changed E2E tests use an image built from a local dockerfile
- Some cargo build options to make it quicker
- Fix the cache hit rate
- added `tsgo` preview to the project 😎
- Can be invoked with `bun tsgo` to typecheck
- Install in IDE
[VSCode](https://code.visualstudio.com/docs/configure/extensions/extension-marketplace)
& [Zed](https://github.com/zed-extensions/tsgo) for super-fast inline
typechecking (as you type basically)
## Context
This PR attempts to make the frankly unacceptable CI times better. This
achieves that aim by making a crappy image for day-to-day usage and let
the prod issue take ages since that will be infrequently used. The
reason why the original design didn't work for us is because: 1) we are
using the free GH runners 2) when we goto baremetal runners we'll lose
our rapid caching abilities which make using docker cheap.
Also, we add `tsgo` support to improve devex. The improvement is
astounding.
```sh
hyperfine -n tsc "bun tsc --incremental false --extendedDiagnostics" -n tsgo "bun tsgo --incremental false --extendedDiagnostics"
Benchmark 1: tsc
Time (mean ± σ): 5.500 s ± 0.221 s [User: 8.939 s, System: 0.400 s]
Range (min … max): 5.196 s … 5.845 s 10 runs
Benchmark 2: tsgo
Time (mean ± σ): 99.1 ms ± 8.4 ms [User: 392.8 ms, System: 54.1 ms]
Range (min … max): 88.3 ms … 116.0 ms 29 runs
Summary
tsgo ran
55.48 ± 5.22 times faster than tsc
```
> [!NOTE]
> This is `Part 3` of the ongoing _Docker Series._
## New Additions:
- Launching Datahaven network will spin up containers, as opposed to
native binaries
- `stop:docker` script to kill all dh containers
- `e2e` test suite for datahaven solochain network
- Contains reference test file that uses papi for storage queries,
submitting exts, runtime calls (good job on that facu and tobi)
- Added new utils:
- `waitForLog()` to wait for log lines in docker container logs
- `createPapiConnectors()` helper for test cases to build and connect to
dh network
- `getPapiSigner()` helper to return a papi compatible signer using our
prefunded accounts (alith by default)
- `sendTxn()` helper to submit txn and wait for block inclusion, instead
of finalization, which std library provides
## Changes:
> [!CAUTION]
> Launching native binaries for datahaven no longer supported.
- Datahaven binary location cli option changed to `-i,
--datahaven-image-tag`
- To locally run this you'll need a datahaven docker image handy, you'll
need to either:
- Point to remote dockerhub e.g. `moonsonglabs/datahaven:main` (must be
logged in and have permission)
- Build this locally with `bun build:docker:operator`
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Added end-to-end tests for the Datahaven solochain, including runtime
API queries, storage lookups, extrinsic submissions, and event
listening.
- Introduced CLI option to specify the Datahaven Docker image tag, with
a default value.
- Added CLI option to disable the Relayer.
- Provided new scripts to stop Docker containers associated with
Datahaven.
- Added utility functions for Docker log monitoring and container
startup checks.
- Introduced utilities for interacting with the Datahaven Polkadot API.
- **Improvements**
- Switched Datahaven network launch from local binaries to Docker
containers.
- Enhanced cache accuracy in build workflows by including Rust source
files in cache keys.
- Improved build performance with TypeScript incremental build options.
- Increased timeout for end-to-end tests for better reliability.
- Updated CLI version to 0.2.0.
- Modified Dockerfile build to enable the `fast-runtime` feature.
- Extended network launch summary to include relayer and container
details.
- **Bug Fixes**
- Fixed cleanup logic by tracking and preparing for forced removal of
Docker containers after tests.
- **Chores**
- Updated workflow steps for Docker image handling and network checks.
- Adjusted scripts and workflow logic for improved Docker and test
management.
- Removed top-level disk usage summaries from cleanup workflow for
streamlined reporting.
- Enhanced shell command utility to support asynchronous wait during
execution.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Facundo Farall <37149322+ffarall@users.noreply.github.com>
## Summary
This PR attempts to improve caching, and thus speeds, for Docker image
generation.
This is to dramatically reduce the times of building images by using:
- cargo chef
- cache mounts
- sccache
- cache dance
## Context
As a result this means thats changes that Do not Affect the code, should
(in theory) not trigger a new build to be run.
Changes that do change the rust code should also in theory be shorter as
the dependencies are unlikely to have changed and so that too can be
reused. In fact some part of the process should always be able to be
re-used unless we do something like drastic like change rust-toolchain,
but even then should only be a one time thing to regenerate that part of
the cache.
---------
Co-authored-by: Facundo Farall <37149322+ffarall@users.noreply.github.com>
Adds the `Substrate` node and runtime, as well as configuration and test
files, from https://github.com/Moonsong-Labs/flamingo to the `operator`
folder in DataHaven
