mirror of
https://github.com/datahaven-xyz/datahaven
synced 2026-05-23 17:28:23 +00:00
e04023ef11
## Summary
Improve CI performance through better caching and simplified workflows:
- Add a new `warm-sccache` job that runs before all Rust CI jobs to
pre-populate the sccache cache
- Cache locally installed tools (`~/.local/bin`, `~/.local/lib`,
`~/.local/include`) with a version-based hash key
- Simplify Rust tests by removing matrix partitioning and switching from
cargo-nextest to cargo test
## Problem
**Poor sccache hit rates:**
- `rust-lint`, `unit-tests`, and `build-operator` ran in parallel with
cold caches
- Each job compiled dependencies independently
- Cache was only saved at job completion (too late for parallel jobs to
benefit)
**Redundant tool downloads:**
- Mold, LLVM/Clang, protoc, and libpq (~500MB+) were downloaded fresh on
each job
- No caching of locally installed tools
**Overcomplicated test setup:**
- 2-partition matrix for tests added complexity without significant
benefit
- cargo-nextest required installation step (~30s overhead)
- Separate result-checker job wasn't necessary
## Solution
### 1. sccache warm-up job (`task-warm-sccache.yml`)
- Runs first (Tier 0) before all Rust jobs
- Compiles with release mode + all features (`fast-runtime`,
`try-runtime`, `runtime-benchmarks`)
- Compiles with debug mode to cover test builds
- Uses `SKIP_WASM_BUILD=1` to minimize warm-up time
### 2. Local tools caching (`setup-env/action.yml`)
- Define tool versions as env vars (`MOLD_VERSION`, `LLVM_VERSION`,
`PROTOC_VERSION`, `LIBPQ_VERSION`)
- Generate SHA256 hash from versions for cache key
- Cache `~/.local/bin`, `~/.local/lib`, `~/.local/include` (not all of
`~/.local` to avoid container storage)
- Set up PATH and env vars immediately after cache restore
### 3. Simplified Rust tests (`task-rust-tests.yml`)
- Remove 2-partition matrix strategy
- Replace cargo-nextest with `cargo test --locked`
- Remove separate tests-result-checker job
## CI Flow
```
┌─ build-operator (warm sccache + cached tools)
│
CI Start → warm-sccache ─┼─ rust-lint (warm sccache + cached tools)
│
└─ unit-tests (warm sccache + cached tools)
```
## Test plan
- [x] CI workflow runs successfully
- [x] Warm-sccache job completes and shows cache stats
- [x] Local tools cache restores correctly (no permission errors)
- [x] Downstream Rust jobs show improved cache hit rates
- [x] Rust tests pass with simplified single-job setup
---------
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Co-authored-by: Ahmad Kaouk <56095276+ahmadkaouk@users.noreply.github.com>
Co-authored-by: undercover-cactus <lola@moonsonglabs.com>
56 lines
1.7 KiB
YAML
56 lines
1.7 KiB
YAML
# Warm sccache: Pre-populate the sccache cache for downstream Rust jobs
|
|
#
|
|
# This job runs first and compiles the project to warm up the shared
|
|
# sccache cache, ensuring better cache hit rates for parallel Rust jobs.
|
|
|
|
name: Warm sccache
|
|
|
|
on:
|
|
workflow_call:
|
|
|
|
# Explicit minimal permissions
|
|
permissions:
|
|
contents: read
|
|
|
|
jobs:
|
|
warm-cache:
|
|
name: Warm sccache cache
|
|
runs-on:
|
|
group: DH-runners
|
|
env:
|
|
RUSTC_WRAPPER: "sccache"
|
|
CARGO_INCREMENTAL: "0"
|
|
CARGO_TERM_COLOR: always
|
|
SCCACHE_GHA_ENABLED: "true"
|
|
defaults:
|
|
run:
|
|
working-directory: ./operator
|
|
steps:
|
|
- uses: actions/checkout@v4
|
|
with:
|
|
submodules: recursive
|
|
fetch-depth: 1
|
|
|
|
- uses: ./.github/workflows/actions/setup-env
|
|
with:
|
|
cache-key: WARM
|
|
install-deps: false
|
|
|
|
- name: Set build flags
|
|
run: echo "RUSTFLAGS=${{ env.RUSTFLAGS }} -C linker=clang -C link-arg=-fuse-ld=mold" >> $GITHUB_ENV
|
|
|
|
# Compile with release mode and superset of features to warm the cache
|
|
# This covers: build-operator (fast-runtime) and rust-lint (try-runtime, runtime-benchmarks)
|
|
- name: Warm cache - release build with all features
|
|
run: |
|
|
echo "Building release with combined features to warm sccache..."
|
|
SKIP_WASM_BUILD=1 cargo check --release --locked --features fast-runtime,try-runtime,runtime-benchmarks
|
|
|
|
# Also warm the debug build cache for tests
|
|
- name: Warm cache - debug build for tests
|
|
run: |
|
|
echo "Building debug mode to warm sccache for tests..."
|
|
cargo check --locked
|
|
|
|
- name: Show sccache stats
|
|
run: sccache --show-stats
|