mirror of
https://github.com/NVIDIA-NeMo/DataDesigner
synced 2026-05-24 09:48:29 +00:00
2564834a47
* fix: cache notebook builds to avoid failures from flaky upstream models The build-notebooks CI executes all tutorial notebooks on every run. When an upstream model (e.g. black-forest-labs/flux.2-pro) is down, the entire docs build fails even if no notebooks changed. Add per-notebook caching based on source file SHA-256 hashes. Unchanged notebooks are served from cache, and only modified ones are re-executed. On the first CI run (empty cache), the workflow seeds the cache from the last successful build artifact. Also add a minimal test script (test_flux_image_gen.py) to reproduce the flux.2-pro health check failure locally. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: address review comments on notebook caching - Don't write .sha256 during seeding so changed notebooks are detected - Rename TMPDIR to SEED_TMPDIR to avoid shadowing the POSIX env var - Use portable sha256 helper (sha256sum with shasum fallback) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: only seed cache when truly empty, restore hash writing Skip artifact seeding when a partial cache was restored (it already has correct per-file hashes). Only seed + write current hashes when the cache dir is completely empty (true bootstrapping). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: restrict artifact seed lookup to main branch Prevents seeding from feature branch runs that may have different notebook sources. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: add actions:read permission for artifact seeding The seed step uses gh run list and gh run download which require actions:read. Without it, these calls silently fail and the cold-start cache bootstrapping never executes. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: only use notebook cache when called from build-docs Scheduled Monday runs and manual workflow_dispatch should execute all notebooks to catch regressions (e.g. library changes that break a notebook). Caching is only used via workflow_call (from build-docs) where the goal is fast, resilient doc deployment. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: use jq // empty to avoid "null" string on empty run list Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: add use_cache input flag to notebook and docs workflows Replace event_name-based cache logic with an explicit use_cache boolean input. Defaults: - build-notebooks: workflow_call=true, dispatch=false, schedule=false - build-docs: dispatch=true (toggleable), release=false This gives full control over caching from the GitHub Actions UI. --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
63 lines
2 KiB
Bash
Executable file
63 lines
2 KiB
Bash
Executable file
#!/usr/bin/env bash
|
|
# SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
|
|
# SPDX-License-Identifier: Apache-2.0
|
|
|
|
# Build notebooks with per-file caching. Only re-executes notebooks whose
|
|
# source .py file changed since the last cached build.
|
|
#
|
|
# Usage:
|
|
# ./docs/scripts/build_notebooks_cached.sh [CACHE_DIR]
|
|
#
|
|
# CACHE_DIR defaults to .notebook-cache
|
|
|
|
set -euo pipefail
|
|
|
|
compute_sha256() {
|
|
if command -v sha256sum >/dev/null 2>&1; then
|
|
sha256sum "$1" | cut -d' ' -f1
|
|
else
|
|
shasum -a 256 "$1" | cut -d' ' -f1
|
|
fi
|
|
}
|
|
|
|
REPO_ROOT="$(cd "$(dirname "$0")/../.." && pwd)"
|
|
SOURCE_DIR="$REPO_ROOT/docs/notebook_source"
|
|
OUTPUT_DIR="$REPO_ROOT/docs/notebooks"
|
|
CACHE_DIR="${1:-$REPO_ROOT/.notebook-cache}"
|
|
|
|
mkdir -p "$OUTPUT_DIR" "$CACHE_DIR"
|
|
|
|
# Copy static files
|
|
cp "$SOURCE_DIR/_README.md" "$OUTPUT_DIR/README.md"
|
|
cp "$SOURCE_DIR/_pyproject.toml" "$OUTPUT_DIR/pyproject.toml"
|
|
|
|
needs_cleanup=false
|
|
|
|
for src in "$SOURCE_DIR"/*.py; do
|
|
name="$(basename "$src" .py)"
|
|
hash="$(compute_sha256 "$src")"
|
|
cached_hash_file="$CACHE_DIR/${name}.sha256"
|
|
cached_notebook="$CACHE_DIR/${name}.ipynb"
|
|
|
|
if [ -f "$cached_hash_file" ] && [ -f "$cached_notebook" ] && [ "$(cat "$cached_hash_file")" = "$hash" ]; then
|
|
echo " ✅ $name.ipynb — cached (unchanged)"
|
|
cp "$cached_notebook" "$OUTPUT_DIR/${name}.ipynb"
|
|
else
|
|
echo " 🔄 $name.ipynb — executing..."
|
|
uv run --all-packages --group notebooks --group docs jupytext --to ipynb --execute "$src"
|
|
mv "$SOURCE_DIR/${name}.ipynb" "$OUTPUT_DIR/${name}.ipynb"
|
|
needs_cleanup=true
|
|
|
|
# Update cache
|
|
cp "$OUTPUT_DIR/${name}.ipynb" "$cached_notebook"
|
|
echo "$hash" > "$cached_hash_file"
|
|
fi
|
|
done
|
|
|
|
if [ "$needs_cleanup" = true ]; then
|
|
# Clean up artifacts from executed notebooks
|
|
[ -d "$SOURCE_DIR/artifacts" ] && rm -rf "$SOURCE_DIR/artifacts"
|
|
find "$SOURCE_DIR" -name '*.csv' -delete 2>/dev/null || true
|
|
fi
|
|
|
|
echo "✅ Notebooks ready in $OUTPUT_DIR"
|