mirror of
https://github.com/datahaven-xyz/datahaven
synced 2026-05-24 01:38:32 +00:00
9a5404de82
## Overview This PR consolidates and optimizes the Docker build system, reducing redundancy and improving CI/CD performance. The changes eliminate duplicate Dockerfiles, introduce a flexible build template, and optimize release builds to reuse CI artifacts. ## Changes Summary ### 🐳 Docker Images Restructured **Before:** 5 Dockerfiles with significant overlap **After:** 4 focused images + 1 utility #### Final Structure: 1. **`operator/Dockerfile`** ✨ Updated - **Standard operator image** for CI and release builds - Minimal node image (accepts pre-built binaries) - GHCR: `ghcr.io/datahaven-xyz/datahaven/datahaven` (CI) - DockerHub: `datahavenxyz/datahaven` (releases) 2. **`docker/datahaven-build.Dockerfile`** (moved from `operator/Dockerfile`) - Full source-to-binary build for manual releases - DockerHub: `datahavenxyz/datahaven:{label}` - Supports custom RUSTFLAGS and fast-runtime feature - Only used for manual workflow_dispatch builds 3. **`docker/datahaven-production.Dockerfile`** (kept) - Binary builder for CPU-specific releases - Used by build-prod-binary workflow template - Supports custom target-cpu flags 4. **`docker/datahaven-dev.Dockerfile`** ✨ NEW (local dev only) - **FOR LOCAL DEVELOPMENT/TROUBLESHOOTING ONLY** - Includes debug tools: gdb, strace, vim, sudo - Extra dependencies: librocksdb-dev, curl - RUST_BACKTRACE enabled by default - **DO NOT USE for CI or production builds** 5. **`test/docker/crossbuild-mac-libpq.dockerfile`** (kept) - Utility for macOS → Linux cross-compilation #### Removed (Redundant): - ❌ `docker/datahaven.Dockerfile` → replaced by operator/Dockerfile - ❌ `test/docker/datahaven-node-local.dockerfile` → replaced by datahaven-dev.Dockerfile --- ### 🔄 Workflow Improvements #### Enhanced `publish-docker` Template - Supports both GHCR and DockerHub registries - Flexible inputs: dockerfile, context, build-args, cache scope - Auto-generates OCI-compliant labels - Reduces code duplication (~70 lines → ~15 per workflow) #### Refactored CI Pipeline - **`docker-build-ci`**: Builds `operator/Dockerfile` → GHCR for CI/E2E testing - **`docker-build-release`**: Builds `operator/Dockerfile` → DockerHub (main branch only) - Both CI and release workflows now use the same minimal operator image - Release builds **reuse CI binaries** instead of rebuilding from source #### Optimized Release Workflow The `task-docker-release` workflow now has dual modes: **Mode 1: `workflow_call` (CI - main pushes)** - ✅ Reuses binary from CI's build-operator task - ✅ Uses lightweight `operator/Dockerfile` - ✅ Tags: `latest`, `sha-{short}` - ⚡ **Fast**: ~5 minutes (vs ~30 min previously) **Mode 2: `workflow_dispatch` (Manual)** - ✅ Full source build with `datahaven-build.Dockerfile` - ✅ Custom branch and label selection - ✅ Optional fast-runtime feature - ✅ Tags: `PROD-{label}` or user-defined --- ### 🔧 Additional Optimizations - Copy libpq5 from builder stage instead of reinstalling (smaller, faster) - Remove redundant protobuf-compiler package (use protoc v21.12 directly) - Standardize user UID to 1000 across all runtime images - Consistent OCI labeling and metadata --------- Co-authored-by: Claude <noreply@anthropic.com>
82 lines
2.6 KiB
YAML
82 lines
2.6 KiB
YAML
# Build Operator: CI for building the operator binary
|
|
|
|
name: DataHaven Operator Binary Build
|
|
|
|
on:
|
|
workflow_dispatch:
|
|
workflow_call:
|
|
outputs:
|
|
binary-hash:
|
|
description: "The hash of the operator binary"
|
|
value: ${{ jobs.build-node.outputs.binary-hash }}
|
|
|
|
jobs:
|
|
build-node:
|
|
outputs:
|
|
binary-hash: ${{ steps.hash-binary.outputs.datahaven-node-hash }}
|
|
name: Build operator binary
|
|
runs-on:
|
|
group: DH-runners
|
|
env:
|
|
RUSTC_WRAPPER: "sccache"
|
|
CARGO_INCREMENTAL: "0"
|
|
CARGO_TERM_COLOR: always
|
|
SCCACHE_GHA_ENABLED: "true"
|
|
defaults:
|
|
run:
|
|
working-directory: ./operator
|
|
steps:
|
|
- uses: actions/checkout@v4
|
|
with:
|
|
submodules: recursive
|
|
fetch-depth: 1
|
|
|
|
# Re-enable if you get GH issues with the runner being full
|
|
# - uses: ./.github/workflows/actions/cleanup-runner
|
|
|
|
- uses: ./.github/workflows/actions/setup-env
|
|
with:
|
|
cache-key: BUILD-RELEASE
|
|
install-deps: false
|
|
|
|
- name: Set build flags
|
|
run: echo "RUSTFLAGS=${{ env.RUSTFLAGS }} -C linker=clang -C link-arg=-fuse-ld=mold" >> $GITHUB_ENV
|
|
|
|
- name: Build node binary
|
|
run: |
|
|
cargo build --release --locked --features fast-runtime
|
|
|
|
- name: Prepare binary
|
|
run: |
|
|
mkdir -p ./target/ci
|
|
mkdir -p ../build
|
|
cp ./target/release/datahaven-node ./target/ci/datahaven-node
|
|
cp ./target/release/datahaven-node ../build/datahaven-node
|
|
|
|
- name: Hash binary
|
|
id: hash-binary
|
|
run: |
|
|
TIMESTAMP=$(date +%s)
|
|
BINARY_PATH=./target/ci/datahaven-node
|
|
HASH=$(echo "$TIMESTAMP" | cat - $BINARY_PATH | sha256sum | awk '{ print $1 }')
|
|
echo "datahaven-node-hash=$HASH" >> $GITHUB_OUTPUT
|
|
echo "Hash of the datahaven-node is: $HASH (with timestamp: $TIMESTAMP)"
|
|
|
|
- name: Upload binary to workflow
|
|
uses: actions/upload-artifact@v4
|
|
with:
|
|
name: datahaven-node-${{ steps.hash-binary.outputs.datahaven-node-hash }}
|
|
path: build/datahaven-node
|
|
retention-days: 1
|
|
|
|
- name: Upload WASM to workflow
|
|
uses: actions/upload-artifact@v4
|
|
with:
|
|
name: datahaven-wasm-${{ steps.hash-binary.outputs.datahaven-node-hash }}
|
|
path: operator/target/release/wbuild/datahaven-stagenet-runtime/datahaven_stagenet_runtime.wasm
|
|
retention-days: 1
|
|
|
|
- name: Build Stats
|
|
run: |
|
|
sccache --show-stats
|
|
echo "Binary size: $(du -h ./target/ci/datahaven-node | cut -f1)"
|