mirror of
https://github.com/NVIDIA-NeMo/DataDesigner
synced 2026-05-24 09:48:29 +00:00
7b5854ca36
* docs: migrate documentation from MkDocs to Fern
Adds a Fern Docs build under fern/ alongside the existing mkdocs site.
Production target docs.nvidia.com/nemo/datadesigner with floating-latest
pointer (latest.yml symlink) at v0.5.8. Migrated all concept, recipe, plugin,
dev-note, and tutorial pages to MDX with NVIDIA theme and custom components
(Authors, MetricsTable, TrajectoryViewer, NotebookViewer, BadgeLinks).
Tutorial notebooks now render via NotebookViewer with captured outputs (text,
DataFrames, inline images) - new make targets generate-fern-notebooks and
generate-fern-notebooks-with-outputs drive the .py -> executed .ipynb -> Fern
JSON+TS pipeline, pinning docs to Python 3.13 to dodge pyarrow wheel issues
on 3.14. Python API reference is configured via Fern libraries: pointing at
data-designer-config; output is gitignored and regenerated locally with
'fern docs md generate'.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: Lawrence Lane <llane@nvidia.com>
* docs: add datadesigner-docs agent skill
Captures the patterns established in the Fern migration so agents (and humans)
can maintain fern/ confidently. Modeled after NVIDIA-NeMo/Gym's
nemo-gym-docs SKILL.md, adapted for our floating-latest versioning,
notebook-with-outputs pipeline, dev-notes kit components, and the MDX gotchas
hit during migration (pymdown attr_list, --8<-- snippet syntax, frontmatter
authors-as-JSX-scope-variable, etc.). Routes triggers like "edit docs", "add
doc page", "regenerate notebooks", "update dev note", "add API reference" to
this skill.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: Lawrence Lane <llane@nvidia.com>
* docs: address PR review for Fern migration
- Delete stale fern/versions/_nav_order.yml (references non-existent
./versions/latest/pages/ — paths were never updated when latest/ was
renamed to v0.5.8/, no consumer found in docs.yml or v0.5.8.yml).
- Remove unused custom components: Tag.tsx, CustomCard.tsx, Include.tsx
(had its own untested markdown parser), ExpandableCode.tsx (broken in
Fern SSR runtime). Drop expandable-code.css from docs.yml. Authors,
BadgeLinks, MetricsTable, NotebookViewer, TrajectoryViewer remain
(each has at least one call site).
- BadgeLinks: remove DEFAULT_BADGES with placeholder URLs; make `badges`
prop required so we can never accidentally ship 'your-org/your-repo'.
- NotebookViewer: document the XSS trust boundary on output cells of
format: "html". Outputs flow .py source → jupytext --execute → committed
*.ts (review boundary). Add an inline comment at the dangerouslySetInnerHTML
call site pointing back to the trust-model section.
- README: add Windows caveat on the latest.yml symlink — Windows users need
core.symlinks=true before clone or Fern will reject the version config.
- Makefile: tighten generate-fern-notebooks source probe from `ls .../*.ipynb`
(which can return success on non-file errors) to `[ -f docs/notebooks/1-the-basics.ipynb ]`,
matching the reviewer's suggestion.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: Lawrence Lane <llane@nvidia.com>
* docs: address @aschilling-nv review on fern/docs.yml
Three suggestions from the Fern review, all matching Curator's docs.yml
conventions:
- instances[0].url: drop the https:// protocol prefix to match Curator's
shape (e.g. nemo-curator.docs.buildwithfern.com/nemo/curator).
- logo.href: was '/'; now points at /nemo/datadesigner/getting-started/welcome
(the actual landing page) so clicking the logo lands on real content
instead of the bare basepath.
- experimental.basepath-aware: true — opts into Fern's basepath-aware
routing so internal links don't double-prefix the /nemo/datadesigner
segment.
- redirects: also fix /nemo/datadesigner/index.html → getting-started/welcome
(was bouncing to /latest, which is just the version slug); add
/getting-started → /getting-started/welcome to mirror Curator's
/home → /home/welcome convention.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: Lawrence Lane <llane@nvidia.com>
* docs: put dev notes overview timestamps on separate lines
Signed-off-by: Kirit93 <kthadaka@nvidia.com>
Made-with: Cursor
* docs: redesign dev-notes index with BlogCard component
Replaces the generic <CardGroup>/<Card> grid (same green icon × 10, date
glued to bottom of description) with a purpose-built BlogCard for the
dev-notes landing page.
Each card now has:
- Hero image (16:9, lazy-loaded, click-to-zoom via Fern's rmiz wrapper)
- ALL-CAPS date eyebrow as proper subtitle styling
- Title, 3-line clamped description
- Author byline at the bottom: avatar stack (overlapping) + first author
name + "+N", pulling from the existing devnotes/.authors.yml registry
- Hover: NVIDIA-green border + subtle lift
Posts without a hero image fall back to a deterministic hash-based
gradient placeholder + monogram (DJB2 hash of href → HSL hue, with the
muddy-yellow band 40–90° remapped). Same post always gets the same look.
Notes:
- Image prop is React.ReactNode (not string) — pass <img> JSX from MDX
so Fern's link rewriter can resolve the src to /_local/... in dev and
/nemo/datadesigner/assets/... in prod. Raw string props bypass the
rewriter and 404 in dev.
- Card href runs through a small withBasepath() helper since the <a>
also bypasses Fern's link rewriter.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: Lawrence Lane <llane@nvidia.com>
* docs: flush blog-card hero images to the top of the card
Fern's prose stylesheet applies a top margin to <img> tags, and the
click-to-zoom wrapper Fern injects around each image (<span data-rmiz>)
inherits that margin too. Result: a ~1rem gap between the card's top
edge and the hero image.
Reset margin/padding on the rmiz wrapper spans + the img itself inside
.blog-card__media so the image renders flush against the top border.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: Lawrence Lane <llane@nvidia.com>
* docs: stop blog-card hero from opening Fern's click-to-zoom modal
When an <img> appears in MDX, Fern auto-wraps it with a click-to-zoom
shell (<span data-rmiz>...). On the dev-notes index that shell intercepts
clicks meant for the card's <a> wrapper, so clicking a hero opens a
lightbox AND tries to navigate.
Set pointer-events: none on the rmiz spans + img inside .blog-card__media
so clicks bubble straight to the parent <a> and the card behaves as a
single, predictable link target. Hover still works because pointer-events
on children doesn't block :hover on the ancestor <a>.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: Lawrence Lane <llane@nvidia.com>
* docs: render notebook markdown at build time with markdown-it-py
Replaces NotebookViewer's hand-rolled JS markdown parser (the one with
the ^@BR^@ sentinel the reviewer flagged as fragile) with build-time
rendering in the converter.
ipynb-to-fern-json.py now uses markdown-it-py (CommonMark + tables +
strikethrough + raw HTML) to render each markdown cell's source into
source_html, mirroring how code cells already store Pygments-highlighted
source_html. NotebookViewer's markdown branch becomes a single
dangerouslySetInnerHTML on the pre-rendered HTML, with a plain-escape
fallback for old snapshots.
Removes the dead JS helpers (renderMarkdown, isSafeUrl, UL_CLASS,
OL_CLASS) — ~60 lines of brittle regex-based markdown parsing.
Fixes broken rendering of:
- Blockquotes (showed literal > characters before)
- Nested content inside blockquotes (e.g. blockquote with bullet list)
- Fenced code blocks
- Tables
- Multi-paragraph list items
Includes regenerated fern/components/notebooks/*.{json,ts} for all 6
tutorials.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: Lawrence Lane <llane@nvidia.com>
* docs: rewrite recipes index + replace octicons download links with Fern Info callouts
The recipes/cards.mdx page was still in MkDocs Material format:
- <div class="grid cards" markdown> wrapper (no-op in MDX)
- :material-snake:, :material-database:, :material-tools:, etc. (rendered
as literal text — Fern uses Font Awesome, not Material icons)
- !!! tip Prerequisite (mkdocs admonition syntax)
- [:material-book-open-page-variant: View Recipe] / [Download Code
:octicons-download-24:] links with embedded icon shortcodes
Rewrite using Fern's native components: <CardGroup cols={2}> with <Card
title icon href> grouped by category (Code Generation, QA and Chat,
Trace Ingestion, MCP and Tool Use, Plugin Development). Each card has
one primary action (the recipe page); download lives on the recipe page
itself.
Replace the trailing "Download Code :octicons-download-24:" link on
every recipe page (and 2 dev notes) with a <Info title="Download Recipe">
callout pointing at the GitHub blob URL — matching PR #215's
convention. 12 occurrences across 12 files.
Also fixes 6 recipe pages whose frontmatter title was "Untitled"
(unfilled placeholder from auto-migration): text_to_python, basic_mcp,
pdf_qa, multi_turn_chat, product_info_qa, agent_rollout_distillation.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: Lawrence Lane <llane@nvidia.com>
* docs(fern): mirror main's content updates into v0.5.8 MDX pages
Forward-port the doc changes that landed in main since this branch was
cut, translating MkDocs admonition syntax to Fern components. Three
product changes drove the updates:
PR #594 — deprecate implicit default-provider routing:
- concepts/models/configure-model-settings-with-the-cli.mdx: deprecate
"Change default provider" workflow + inline mark on `data-designer
config list` output
- concepts/models/custom-model-settings.mdx: warning that `provider=`
is now required on every ModelConfig
- concepts/models/default-model-settings.mdx: warning that the
registry-level default-provider concept is deprecated
- concepts/models/model-providers.mdx: same warning at the top of the
ModelProvider overview
- concepts/models/inference-parameters.mdx: add explicit `provider=
"openai"` to the dalle ModelConfig example
PR #592 — async engine becomes the default:
- concepts/architecture-and-performance.mdx: rewrite Execution Model
intro to mention both engines, qualify "How It Works" as sync-engine
semantics, update Concurrency Formula and Throttle notes from "Sync
engine caveat" to "Engine paths", and add a full new "## Async
Engine" section (per-model timeouts, run outcomes / Early Shutdown,
opt-out via DATA_DESIGNER_ASYNC_ENGINE=0). Add `provider="nvidia"`
to the my-model example.
- concepts/custom_columns.mdx: note that sync `cell_by_cell`
generators dispatch concurrently under the async engine; mock with
`MagicMock(spec=ModelFacade)` so async methods are auto-detected.
- concepts/processors.mdx: warning that the async engine enforces
row-count invariance in process_before/after_batch.
- devnotes/posts/async-all-the-way-down.mdx: append an "Update" callout
noting the engine is now default, with a link to the Architecture
page anchor.
All `!!! warning|note|tip "Title"` admonitions converted to Fern
<Warning|Note|Tip title="..."> components. Internal links to mkdocs
relative paths (`../../concepts/foo.md#anchor`) rewritten to canonical
Fern URLs (`/concepts/foo#anchor`).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: Lawrence Lane <llane@nvidia.com>
* docs(fern): address @andreatgretel review comments
Four issues from Andre's review pass:
1. /devnotes 404 (index.mdx:23) — section slug is /dev-notes, page slug
is /dev-notes/overview. Fix the link in the landing page so visitors
actually reach the dev notes index.
2. TrajectoryViewer.tsx final-answer body shown as literal markdown
(line 66) — the renderer uses dangerouslySetInnerHTML but
example-marcia.ts shipped raw markdown (**bold**, \n\n breaks). Visible
on the deep-research devnote where the trajectory is defaultOpen.
Pre-render body to HTML in the fixture (matches the original hand-coded
format pre-migration); document the convention in the ToolCall.body
doc comment so future fixtures don't regress.
3. Tutorials 5/6 (image generation/editing) ship with 0 captured outputs
because Flux runs through OpenRouter and OPENROUTER_API_KEY isn't set
at build time. Cannot regenerate without the key, so add a <Note> at
the top of each wrapper page pointing readers at the Colab link to
execute the cells live and see the generated images. Maintainers with
the key in their environment should re-run
`make generate-fern-notebooks-with-outputs` before merge to capture
the snapshots.
4. Legacy nvidia-nemo.github.io/DataDesigner/* URLs in MDX prose (8
occurrences across 5 files) rewritten to canonical Fern paths so
visitors don't get sent back to the legacy GitHub Pages site once
docs.nvidia.com/nemo/datadesigner becomes the production URL:
- The single deep link in data-designer-got-skills.mdx →
/concepts/models/default-model-settings
- All other "documentation home" links (CONTRIBUTING ×2,
async-all-the-way-down ×2, owning-the-model-stack, design-principles
×2) → /getting-started/welcome (the canonical landing slug, matches
logo.href in docs.yml)
Notebook .py source URLs are tracked separately as part of the
notebook-regen work.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: Lawrence Lane <llane@nvidia.com>
* docs(fern): regenerate notebook snapshots with Flux outputs captured
Re-ran make generate-fern-notebooks-with-outputs with NVIDIA_API_KEY +
OPENROUTER_API_KEY set, now that we have a NVIDIA key with permission
on nemotron-3-nano-30b-a3b. All 6 tutorials regenerated; the two image
tutorials (5 and 6) which had been shipping with 0 outputs now have
captured Flux generations:
1 the-basics: 12/15 outputs
2 structured-outputs: 13/17 outputs
3 seeding-with-a-dataset: 10/13 outputs
4 providing-images: 13/17 outputs (1 image)
5 generating-images: 8/10 outputs (2 images) ← was 0/12
6 image-to-image-editing: 9/12 outputs (10 images) ← was 0/14
The two `<Note title="Run in Colab to see ...">` workarounds I added on
the 5/6 wrapper pages are no longer needed — outputs render inline now.
NotebookViewer's own "Run in Google Colab" banner is still rendered
from the wrapper's `colabUrl` prop, so the live-execute path stays one
click away.
Bumps the diff size noticeably (notebook 6 .ts is ~22MB of base64-
encoded PNGs from 10 edited images), but that's intentional — these
images are the proof points for what the Flux/MCP image-context
tutorials actually produce.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: Lawrence Lane <llane@nvidia.com>
* docs(fern): unbreak SSR — shrink notebook image outputs + fix BlogCard React import
Two server-side render bugs surfaced when running `fern docs md generate &&
fern docs dev` (the static-preview path):
1. The 22 MB notebook 6 .ts module (full-resolution Flux PNGs from 10 edited
images) tripped Fern's SSR module-evaluation step. Once that module
failed to evaluate, the shared component bundle failed to load on every
page, replacing each MDX body with `<span data-intent="error">Something
went wrong!</span>` while the layout chrome continued to render.
Fix in fern/scripts/ipynb-to-fern-json.py: after extracting an
image/png output, pass it through Pillow to (a) downscale so the
longest edge is at most 800 px, (b) re-encode as JPEG q=82 progressive
(Flux outputs are photographic — JPEG compresses 5–10× better than PNG
for this content). NotebookViewer's CellOutput interface gains a
`mime` field so the data URL uses the actual encoded MIME type. Result:
notebook 6: 22 MB → 4.6 MB
notebook 5: 3.8 MB → 1.8 MB
notebook 4: 514 KB → 116 KB
(notebooks 1–3 unaffected — no image outputs)
2. fern/components/BlogCard.tsx referenced `React.ReactNode` twice without
importing React. Other components in the kit use `import type
{ ReactNode } from "react"`; BlogCard was the outlier. Aligned the
import style — even though this didn't end up being the trigger, leaving
the dangling reference would have eventually caused a strict-mode SSR
regression.
Sweep test against http://localhost:3000/nemo/datadesigner/* — landing,
concepts, tutorials (including 5/6 image notebooks), dev notes, recipes,
and code-reference topic pages all render with their content; no error
spans.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: Lawrence Lane <llane@nvidia.com>
* docs(fern): add MkDocs-shape redirects for legacy URLs
The legacy site at https://nvidia-nemo.github.io/DataDesigner/ used
MkDocs-Material conventions (mkdocstrings + blog plugin + mkdocs-jupyter
+ directory URLs). Several path segments and page slugs differ from
Fern's slugified-title routing — search-engine indexed links and
copy-pasted bookmarks land on 404 without redirects.
Adds 30+ specific redirect rules covering every renamed surface:
- Tutorials: /notebooks/<filename>/ -> /tutorials/<title-slug>
(page-title slugs differ from .ipynb filenames; one rule per notebook
plus a README -> overview alias).
- Recipes: /recipes/<snake_subsection>/<snake_page>/ ->
/recipes/<kebab-subsection>/<kebab-page>. Per-page rules for each of
the 10 recipes (page titles diverged from .py filenames — e.g.
basic_mcp -> basic-mcp-tool-use, search_agent -> nemotron-super-search-agent),
followed by subsection :rest* fallbacks.
- Concepts: /concepts/mcp/* -> /concepts/tool-use-mcp/* (subsection
rename, with & dropped, not -and-). Per-page rules for safety-and-limits
-> safety-limits and configure-mcp-cli -> cli-configuration where
page titles diverged from filenames.
- Code Reference: /code_reference/<module>/ ->
/code-reference/topic-overviews/<module>. Per-page rules for the six
underscored modules (column_configs, config_builder, run_config,
sampler_params, validator_params, data_designer_config) since Fern's
page-slug rule kebabs underscores.
- Plugins: filesystem_seed_reader -> file-system-seed-reader-plugins
(Fern inserts hyphens between CamelCase words). example -> example-plugin,
available -> available-plugin-list (page-title slugs).
- Dev Notes: blog plugin's /devnotes/posts/<slug>/ -> /dev-notes/<slug>.
Per-page rules for text-to-sql -> text-to-sql-for-nemotron-super and
rqa -> rqa-dataset (post titles diverged from filenames).
- /devnotes -> /dev-notes/overview (section landing).
MkDocs's directory-URL trailing-slash convention is handled natively by
Fern's runtime (both /foo and /foo/ return the same page), so no
explicit slash-strip rule is needed.
Smoke-tested all 34 legacy URLs against http://localhost:3000 — every
one resolves to a 200 page on the new structure.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: Lawrence Lane <llane@nvidia.com>
---------
Signed-off-by: Lawrence Lane <llane@nvidia.com>
Signed-off-by: Kirit93 <kthadaka@nvidia.com>
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Co-authored-by: Kirit93 <kthadaka@nvidia.com>
Co-authored-by: Andre Manoel <165937436+andreatgretel@users.noreply.github.com>
|
||
|---|---|---|
| .. | ||
| 1-the-basics.ipynb | ||
| 2-structured-outputs-and-jinja-expressions.ipynb | ||
| 3-seeding-with-a-dataset.ipynb | ||
| 4-providing-images-as-context.ipynb | ||
| 5-generating-images.ipynb | ||
| 6-editing-images-with-image-context.ipynb | ||