datahaven/.github/workflows/task-docker.yml
Tim B 1997c298a1
refactor: 🐳 Improve docker caching (again) (#86)
## Changes

- New CI file for making Docker Prod images
- Changed E2E tests use an image built from a local dockerfile
- Some cargo build options to make it quicker
- Fix the cache hit rate
- added `tsgo` preview to the project 😎
  - Can be invoked with `bun tsgo` to typecheck
- Install in IDE
[VSCode](https://code.visualstudio.com/docs/configure/extensions/extension-marketplace)
& [Zed](https://github.com/zed-extensions/tsgo) for super-fast inline
typechecking (as you type basically)

## Context

This PR attempts to make the frankly unacceptable CI times better. This
achieves that aim by making a crappy image for day-to-day usage and let
the prod issue take ages since that will be infrequently used. The
reason why the original design didn't work for us is because: 1) we are
using the free GH runners 2) when we goto baremetal runners we'll lose
our rapid caching abilities which make using docker cheap.

Also, we add `tsgo` support to improve devex. The improvement is
astounding.

```sh
hyperfine -n tsc "bun tsc --incremental false --extendedDiagnostics" -n tsgo "bun tsgo --incremental false --extendedDiagnostics"
Benchmark 1: tsc
  Time (mean ± σ):      5.500 s ±  0.221 s    [User: 8.939 s, System: 0.400 s]
  Range (min … max):    5.196 s …  5.845 s    10 runs
 
Benchmark 2: tsgo
  Time (mean ± σ):      99.1 ms ±   8.4 ms    [User: 392.8 ms, System: 54.1 ms]
  Range (min … max):    88.3 ms … 116.0 ms    29 runs
 
Summary
  tsgo ran
   55.48 ± 5.22 times faster than tsc
```
2025-05-27 16:14:15 +00:00

136 lines
No EOL
4.4 KiB
YAML

name: Docker Build & Publish
on:
workflow_dispatch:
inputs:
binary-hash:
description: "The hash of the operator binary"
required: false
type: string
workflow_call:
inputs:
binary-hash:
description: "The hash of the operator binary"
required: true
type: string
outputs:
image-tag:
description: "The tag of the docker image"
value: ${{ jobs.build-test-push.outputs.image-tag }}
concurrency:
group: docker-build-${{ github.ref }}
cancel-in-progress: true
jobs:
build-test-push:
runs-on: ubuntu-latest
outputs:
image-tag: ${{ steps.last_tag_extractor.outputs.last_tag_value }}
steps:
- name: Checkout repository
uses: actions/checkout@v4
- name: Download binary artifact
uses: actions/download-artifact@v4
with:
name: datahaven-node-${{ inputs.binary-hash }}
path: ./operator/target/x86_64-unknown-linux-gnu/release/
- name: Prepare binary
run: |
chmod +x ./operator/target/x86_64-unknown-linux-gnu/release/datahaven-node
ls -la ./operator/target/x86_64-unknown-linux-gnu/release/
- name: Docker meta
id: meta
uses: docker/metadata-action@v5
with:
images: moonsonglabs/datahaven
flavor: |
latest=auto
tags: |
type=sha,format=short,prefix=sha-
type=ref,event=tag
type=ref,event=branch
type=ref,event=pr
- name: Extract last tag for job output
id: last_tag_extractor
run: |
echo "last_tag_value=$(echo '${{ steps.meta.outputs.json }}' | jq -r '.tags[-1]')" >> $GITHUB_OUTPUT
- name: Log Docker Metadata
run: |
echo "Generated tags: ${{ steps.meta.outputs.tags }}"
echo "Generated labels: ${{ steps.meta.outputs.labels }}"
echo "Generated JSON: ${{ steps.meta.outputs.json }}"
- uses: docker/setup-qemu-action@v3
- uses: docker/setup-buildx-action@v3
with:
driver-opts: |
image=moby/buildkit:master
network=host
buildkitd-flags: |
--allow-insecure-entitlement network.host
--allow-insecure-entitlement security.insecure
- uses: docker/login-action@v3
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
- name: Build and push Docker image
id: build
uses: docker/build-push-action@v5
with:
context: .
file: ./test/docker/datahaven-node-local.dockerfile
push: true
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
platforms: linux/amd64
cache-from: type=gha,scope=datahaven-local-build
cache-to: type=gha,mode=max,scope=datahaven-local-build
provenance: mode=max
sbom: true
- name: Log build cache statistics
run: |
echo "Build cache statistics:"
docker buildx du --verbose
# --- Smoke tests ---
- name: Pull and test node --help
run: |
docker pull ${{ steps.last_tag_extractor.outputs.last_tag_value }}
docker run --rm ${{ steps.last_tag_extractor.outputs.last_tag_value }} --help
- name: Integration test (dev chain starts)
run: |
docker run --rm -d -p 9944:9944 --name local-dh-node \
${{ steps.last_tag_extractor.outputs.last_tag_value }} --dev --unsafe-rpc-external
- name: Wait for node to be healthy and test
run: |
echo "Waiting for node to start..."
for i in {1..30}; do # Retry for 30 * 5s = 150 seconds
if curl --fail --location 'http://127.0.0.1:9944' \
--header 'Content-Type: application/json' \
--data '{"jsonrpc":"2.0","id":1,"method":"system_chain","params":[]}' ; then
echo "Node is healthy!"
docker logs local-dh-node --tail 100
exit 0
fi
echo "Attempt $i: Node not ready yet, sleeping 5s..."
sleep 5
done
echo "Node failed to start or respond in time."
docker logs local-dh-node --tail 100
exit 1
- name: Cleanup integration test container
if: always()
run: docker rm -f local-dh-node