1 commit

Author	SHA1	Message	Date
Lawrence Lane	7b5854ca36	docs: migrate documentation from MkDocs to Fern (#581) ... * docs: migrate documentation from MkDocs to Fern Adds a Fern Docs build under fern/ alongside the existing mkdocs site. Production target docs.nvidia.com/nemo/datadesigner with floating-latest pointer (latest.yml symlink) at v0.5.8. Migrated all concept, recipe, plugin, dev-note, and tutorial pages to MDX with NVIDIA theme and custom components (Authors, MetricsTable, TrajectoryViewer, NotebookViewer, BadgeLinks). Tutorial notebooks now render via NotebookViewer with captured outputs (text, DataFrames, inline images) - new make targets generate-fern-notebooks and generate-fern-notebooks-with-outputs drive the .py -> executed .ipynb -> Fern JSON+TS pipeline, pinning docs to Python 3.13 to dodge pyarrow wheel issues on 3.14. Python API reference is configured via Fern libraries: pointing at data-designer-config; output is gitignored and regenerated locally with 'fern docs md generate'. Co-Authored-By: Claude Opus 4.7 <[email protected]> Signed-off-by: Lawrence Lane <[email protected]> * docs: add datadesigner-docs agent skill Captures the patterns established in the Fern migration so agents (and humans) can maintain fern/ confidently. Modeled after NVIDIA-NeMo/Gym's nemo-gym-docs SKILL.md, adapted for our floating-latest versioning, notebook-with-outputs pipeline, dev-notes kit components, and the MDX gotchas hit during migration (pymdown attr_list, --8<-- snippet syntax, frontmatter authors-as-JSX-scope-variable, etc.). Routes triggers like "edit docs", "add doc page", "regenerate notebooks", "update dev note", "add API reference" to this skill. Co-Authored-By: Claude Opus 4.7 <[email protected]> Signed-off-by: Lawrence Lane <[email protected]> * docs: address PR review for Fern migration - Delete stale fern/versions/_nav_order.yml (references non-existent ./versions/latest/pages/ — paths were never updated when latest/ was renamed to v0.5.8/, no consumer found in docs.yml or v0.5.8.yml). - Remove unused custom components: Tag.tsx, CustomCard.tsx, Include.tsx (had its own untested markdown parser), ExpandableCode.tsx (broken in Fern SSR runtime). Drop expandable-code.css from docs.yml. Authors, BadgeLinks, MetricsTable, NotebookViewer, TrajectoryViewer remain (each has at least one call site). - BadgeLinks: remove DEFAULT_BADGES with placeholder URLs; make `badges` prop required so we can never accidentally ship 'your-org/your-repo'. - NotebookViewer: document the XSS trust boundary on output cells of format: "html". Outputs flow .py source → jupytext --execute → committed *.ts (review boundary). Add an inline comment at the dangerouslySetInnerHTML call site pointing back to the trust-model section. - README: add Windows caveat on the latest.yml symlink — Windows users need core.symlinks=true before clone or Fern will reject the version config. - Makefile: tighten generate-fern-notebooks source probe from `ls .../*.ipynb` (which can return success on non-file errors) to `[ -f docs/notebooks/1-the-basics.ipynb ]`, matching the reviewer's suggestion. Co-Authored-By: Claude Opus 4.7 <[email protected]> Signed-off-by: Lawrence Lane <[email protected]> * docs: address @aschilling-nv review on fern/docs.yml Three suggestions from the Fern review, all matching Curator's docs.yml conventions: - instances[0].url: drop the https:// protocol prefix to match Curator's shape (e.g. nemo-curator.docs.buildwithfern.com/nemo/curator). - logo.href: was '/'; now points at /nemo/datadesigner/getting-started/welcome (the actual landing page) so clicking the logo lands on real content instead of the bare basepath. - experimental.basepath-aware: true — opts into Fern's basepath-aware routing so internal links don't double-prefix the /nemo/datadesigner segment. - redirects: also fix /nemo/datadesigner/index.html → getting-started/welcome (was bouncing to /latest, which is just the version slug); add /getting-started → /getting-started/welcome to mirror Curator's /home → /home/welcome convention. Co-Authored-By: Claude Opus 4.7 <[email protected]> Signed-off-by: Lawrence Lane <[email protected]> * docs: put dev notes overview timestamps on separate lines Signed-off-by: Kirit93 <[email protected]> Made-with: Cursor * docs: redesign dev-notes index with BlogCard component Replaces the generic <CardGroup>/<Card> grid (same green icon × 10, date glued to bottom of description) with a purpose-built BlogCard for the dev-notes landing page. Each card now has: - Hero image (16:9, lazy-loaded, click-to-zoom via Fern's rmiz wrapper) - ALL-CAPS date eyebrow as proper subtitle styling - Title, 3-line clamped description - Author byline at the bottom: avatar stack (overlapping) + first author name + "+N", pulling from the existing devnotes/.authors.yml registry - Hover: NVIDIA-green border + subtle lift Posts without a hero image fall back to a deterministic hash-based gradient placeholder + monogram (DJB2 hash of href → HSL hue, with the muddy-yellow band 40–90° remapped). Same post always gets the same look. Notes: - Image prop is React.ReactNode (not string) — pass <img> JSX from MDX so Fern's link rewriter can resolve the src to /_local/... in dev and /nemo/datadesigner/assets/... in prod. Raw string props bypass the rewriter and 404 in dev. - Card href runs through a small withBasepath() helper since the <a> also bypasses Fern's link rewriter. Co-Authored-By: Claude Opus 4.7 <[email protected]> Signed-off-by: Lawrence Lane <[email protected]> * docs: flush blog-card hero images to the top of the card Fern's prose stylesheet applies a top margin to <img> tags, and the click-to-zoom wrapper Fern injects around each image (<span data-rmiz>) inherits that margin too. Result: a ~1rem gap between the card's top edge and the hero image. Reset margin/padding on the rmiz wrapper spans + the img itself inside .blog-card__media so the image renders flush against the top border. Co-Authored-By: Claude Opus 4.7 <[email protected]> Signed-off-by: Lawrence Lane <[email protected]> * docs: stop blog-card hero from opening Fern's click-to-zoom modal When an <img> appears in MDX, Fern auto-wraps it with a click-to-zoom shell (<span data-rmiz>...). On the dev-notes index that shell intercepts clicks meant for the card's <a> wrapper, so clicking a hero opens a lightbox AND tries to navigate. Set pointer-events: none on the rmiz spans + img inside .blog-card__media so clicks bubble straight to the parent <a> and the card behaves as a single, predictable link target. Hover still works because pointer-events on children doesn't block :hover on the ancestor <a>. Co-Authored-By: Claude Opus 4.7 <[email protected]> Signed-off-by: Lawrence Lane <[email protected]> * docs: render notebook markdown at build time with markdown-it-py Replaces NotebookViewer's hand-rolled JS markdown parser (the one with the ^@BR^@ sentinel the reviewer flagged as fragile) with build-time rendering in the converter. ipynb-to-fern-json.py now uses markdown-it-py (CommonMark + tables + strikethrough + raw HTML) to render each markdown cell's source into source_html, mirroring how code cells already store Pygments-highlighted source_html. NotebookViewer's markdown branch becomes a single dangerouslySetInnerHTML on the pre-rendered HTML, with a plain-escape fallback for old snapshots. Removes the dead JS helpers (renderMarkdown, isSafeUrl, UL_CLASS, OL_CLASS) — ~60 lines of brittle regex-based markdown parsing. Fixes broken rendering of: - Blockquotes (showed literal > characters before) - Nested content inside blockquotes (e.g. blockquote with bullet list) - Fenced code blocks - Tables - Multi-paragraph list items Includes regenerated fern/components/notebooks/*.{json,ts} for all 6 tutorials. Co-Authored-By: Claude Opus 4.7 <[email protected]> Signed-off-by: Lawrence Lane <[email protected]> * docs: rewrite recipes index + replace octicons download links with Fern Info callouts The recipes/cards.mdx page was still in MkDocs Material format: - <div class="grid cards" markdown> wrapper (no-op in MDX) - :material-snake:, :material-database:, :material-tools:, etc. (rendered as literal text — Fern uses Font Awesome, not Material icons) - !!! tip Prerequisite (mkdocs admonition syntax) - [:material-book-open-page-variant: View Recipe] / [Download Code :octicons-download-24:] links with embedded icon shortcodes Rewrite using Fern's native components: <CardGroup cols={2}> with <Card title icon href> grouped by category (Code Generation, QA and Chat, Trace Ingestion, MCP and Tool Use, Plugin Development). Each card has one primary action (the recipe page); download lives on the recipe page itself. Replace the trailing "Download Code :octicons-download-24:" link on every recipe page (and 2 dev notes) with a <Info title="Download Recipe"> callout pointing at the GitHub blob URL — matching PR #215's convention. 12 occurrences across 12 files. Also fixes 6 recipe pages whose frontmatter title was "Untitled" (unfilled placeholder from auto-migration): text_to_python, basic_mcp, pdf_qa, multi_turn_chat, product_info_qa, agent_rollout_distillation. Co-Authored-By: Claude Opus 4.7 <[email protected]> Signed-off-by: Lawrence Lane <[email protected]> * docs(fern): mirror main's content updates into v0.5.8 MDX pages Forward-port the doc changes that landed in main since this branch was cut, translating MkDocs admonition syntax to Fern components. Three product changes drove the updates: PR #594 — deprecate implicit default-provider routing: - concepts/models/configure-model-settings-with-the-cli.mdx: deprecate "Change default provider" workflow + inline mark on `data-designer config list` output - concepts/models/custom-model-settings.mdx: warning that `provider=` is now required on every ModelConfig - concepts/models/default-model-settings.mdx: warning that the registry-level default-provider concept is deprecated - concepts/models/model-providers.mdx: same warning at the top of the ModelProvider overview - concepts/models/inference-parameters.mdx: add explicit `provider= "openai"` to the dalle ModelConfig example PR #592 — async engine becomes the default: - concepts/architecture-and-performance.mdx: rewrite Execution Model intro to mention both engines, qualify "How It Works" as sync-engine semantics, update Concurrency Formula and Throttle notes from "Sync engine caveat" to "Engine paths", and add a full new "## Async Engine" section (per-model timeouts, run outcomes / Early Shutdown, opt-out via DATA_DESIGNER_ASYNC_ENGINE=0). Add `provider="nvidia"` to the my-model example. - concepts/custom_columns.mdx: note that sync `cell_by_cell` generators dispatch concurrently under the async engine; mock with `MagicMock(spec=ModelFacade)` so async methods are auto-detected. - concepts/processors.mdx: warning that the async engine enforces row-count invariance in process_before/after_batch. - devnotes/posts/async-all-the-way-down.mdx: append an "Update" callout noting the engine is now default, with a link to the Architecture page anchor. All `!!! warning|note|tip "Title"` admonitions converted to Fern <Warning|Note|Tip title="..."> components. Internal links to mkdocs relative paths (`../../concepts/foo.md#anchor`) rewritten to canonical Fern URLs (`/concepts/foo#anchor`). Co-Authored-By: Claude Opus 4.7 <[email protected]> Signed-off-by: Lawrence Lane <[email protected]> * docs(fern): address @andreatgretel review comments Four issues from Andre's review pass: 1. /devnotes 404 (index.mdx:23) — section slug is /dev-notes, page slug is /dev-notes/overview. Fix the link in the landing page so visitors actually reach the dev notes index. 2. TrajectoryViewer.tsx final-answer body shown as literal markdown (line 66) — the renderer uses dangerouslySetInnerHTML but example-marcia.ts shipped raw markdown (**bold**, \n\n breaks). Visible on the deep-research devnote where the trajectory is defaultOpen. Pre-render body to HTML in the fixture (matches the original hand-coded format pre-migration); document the convention in the ToolCall.body doc comment so future fixtures don't regress. 3. Tutorials 5/6 (image generation/editing) ship with 0 captured outputs because Flux runs through OpenRouter and OPENROUTER_API_KEY isn't set at build time. Cannot regenerate without the key, so add a <Note> at the top of each wrapper page pointing readers at the Colab link to execute the cells live and see the generated images. Maintainers with the key in their environment should re-run `make generate-fern-notebooks-with-outputs` before merge to capture the snapshots. 4. Legacy nvidia-nemo.github.io/DataDesigner/* URLs in MDX prose (8 occurrences across 5 files) rewritten to canonical Fern paths so visitors don't get sent back to the legacy GitHub Pages site once docs.nvidia.com/nemo/datadesigner becomes the production URL: - The single deep link in data-designer-got-skills.mdx → /concepts/models/default-model-settings - All other "documentation home" links (CONTRIBUTING ×2, async-all-the-way-down ×2, owning-the-model-stack, design-principles ×2) → /getting-started/welcome (the canonical landing slug, matches logo.href in docs.yml) Notebook .py source URLs are tracked separately as part of the notebook-regen work. Co-Authored-By: Claude Opus 4.7 <[email protected]> Signed-off-by: Lawrence Lane <[email protected]> * docs(fern): regenerate notebook snapshots with Flux outputs captured Re-ran make generate-fern-notebooks-with-outputs with NVIDIA_API_KEY + OPENROUTER_API_KEY set, now that we have a NVIDIA key with permission on nemotron-3-nano-30b-a3b. All 6 tutorials regenerated; the two image tutorials (5 and 6) which had been shipping with 0 outputs now have captured Flux generations: 1 the-basics: 12/15 outputs 2 structured-outputs: 13/17 outputs 3 seeding-with-a-dataset: 10/13 outputs 4 providing-images: 13/17 outputs (1 image) 5 generating-images: 8/10 outputs (2 images) ← was 0/12 6 image-to-image-editing: 9/12 outputs (10 images) ← was 0/14 The two `<Note title="Run in Colab to see ...">` workarounds I added on the 5/6 wrapper pages are no longer needed — outputs render inline now. NotebookViewer's own "Run in Google Colab" banner is still rendered from the wrapper's `colabUrl` prop, so the live-execute path stays one click away. Bumps the diff size noticeably (notebook 6 .ts is ~22MB of base64- encoded PNGs from 10 edited images), but that's intentional — these images are the proof points for what the Flux/MCP image-context tutorials actually produce. Co-Authored-By: Claude Opus 4.7 <[email protected]> Signed-off-by: Lawrence Lane <[email protected]> * docs(fern): unbreak SSR — shrink notebook image outputs + fix BlogCard React import Two server-side render bugs surfaced when running `fern docs md generate && fern docs dev` (the static-preview path): 1. The 22 MB notebook 6 .ts module (full-resolution Flux PNGs from 10 edited images) tripped Fern's SSR module-evaluation step. Once that module failed to evaluate, the shared component bundle failed to load on every page, replacing each MDX body with `<span data-intent="error">Something went wrong!</span>` while the layout chrome continued to render. Fix in fern/scripts/ipynb-to-fern-json.py: after extracting an image/png output, pass it through Pillow to (a) downscale so the longest edge is at most 800 px, (b) re-encode as JPEG q=82 progressive (Flux outputs are photographic — JPEG compresses 5–10× better than PNG for this content). NotebookViewer's CellOutput interface gains a `mime` field so the data URL uses the actual encoded MIME type. Result: notebook 6: 22 MB → 4.6 MB notebook 5: 3.8 MB → 1.8 MB notebook 4: 514 KB → 116 KB (notebooks 1–3 unaffected — no image outputs) 2. fern/components/BlogCard.tsx referenced `React.ReactNode` twice without importing React. Other components in the kit use `import type { ReactNode } from "react"`; BlogCard was the outlier. Aligned the import style — even though this didn't end up being the trigger, leaving the dangling reference would have eventually caused a strict-mode SSR regression. Sweep test against http://localhost:3000/nemo/datadesigner/* — landing, concepts, tutorials (including 5/6 image notebooks), dev notes, recipes, and code-reference topic pages all render with their content; no error spans. Co-Authored-By: Claude Opus 4.7 <[email protected]> Signed-off-by: Lawrence Lane <[email protected]> * docs(fern): add MkDocs-shape redirects for legacy URLs The legacy site at https://nvidia-nemo.github.io/DataDesigner/ used MkDocs-Material conventions (mkdocstrings + blog plugin + mkdocs-jupyter + directory URLs). Several path segments and page slugs differ from Fern's slugified-title routing — search-engine indexed links and copy-pasted bookmarks land on 404 without redirects. Adds 30+ specific redirect rules covering every renamed surface: - Tutorials: /notebooks/<filename>/ -> /tutorials/<title-slug> (page-title slugs differ from .ipynb filenames; one rule per notebook plus a README -> overview alias). - Recipes: /recipes/<snake_subsection>/<snake_page>/ -> /recipes/<kebab-subsection>/<kebab-page>. Per-page rules for each of the 10 recipes (page titles diverged from .py filenames — e.g. basic_mcp -> basic-mcp-tool-use, search_agent -> nemotron-super-search-agent), followed by subsection :rest* fallbacks. - Concepts: /concepts/mcp/* -> /concepts/tool-use-mcp/* (subsection rename, with & dropped, not -and-). Per-page rules for safety-and-limits -> safety-limits and configure-mcp-cli -> cli-configuration where page titles diverged from filenames. - Code Reference: /code_reference/<module>/ -> /code-reference/topic-overviews/<module>. Per-page rules for the six underscored modules (column_configs, config_builder, run_config, sampler_params, validator_params, data_designer_config) since Fern's page-slug rule kebabs underscores. - Plugins: filesystem_seed_reader -> file-system-seed-reader-plugins (Fern inserts hyphens between CamelCase words). example -> example-plugin, available -> available-plugin-list (page-title slugs). - Dev Notes: blog plugin's /devnotes/posts/<slug>/ -> /dev-notes/<slug>. Per-page rules for text-to-sql -> text-to-sql-for-nemotron-super and rqa -> rqa-dataset (post titles diverged from filenames). - /devnotes -> /dev-notes/overview (section landing). MkDocs's directory-URL trailing-slash convention is handled natively by Fern's runtime (both /foo and /foo/ return the same page), so no explicit slash-strip rule is needed. Smoke-tested all 34 legacy URLs against http://localhost:3000 — every one resolves to a 200 page on the new structure. Co-Authored-By: Claude Opus 4.7 <[email protected]> Signed-off-by: Lawrence Lane <[email protected]> --------- Signed-off-by: Lawrence Lane <[email protected]> Signed-off-by: Kirit93 <[email protected]> Co-authored-by: Claude Opus 4.7 <[email protected]> Co-authored-by: Kirit93 <[email protected]> Co-authored-by: Andre Manoel <[email protected]>	2026-05-07 14:12:58 -03:00