138 commits

Author	SHA1	Message	Date
Andre Manoel	b6de38d894	docs: remove docs code reference (#674)	2026-05-21 18:29:18 -04:00
Eric W. Tramel	c0a4dcbb85	feat: implement async scheduling admission control (#661)	2026-05-20 20:58:05 -04:00
Steve Han	abb4a242df	docs: add retriever SDG toolkit dev note (#666) ... * docs: add retriever SDG toolkit dev note Signed-off-by: Steve Han <sthan@nvidia.com> * docs: restyle retriever SDG pipeline diagram Signed-off-by: Steve Han <sthan@nvidia.com> * docs: fix retriever SDG pipeline flow order Signed-off-by: Steve Han <sthan@nvidia.com> * docs: address retriever SDG dev note review Signed-off-by: Steve Han <sthan@nvidia.com> * docs: clarify retriever SDG wording Signed-off-by: Steve Han <sthan@nvidia.com> --------- Signed-off-by: Steve Han <sthan@nvidia.com>	2026-05-19 16:19:03 -04:00
Andre Manoel	6055290136	feat: add workflow chaining (#636) ... * feat: add workflow chaining * test: tidy workflow chaining coverage * fix: harden workflow chaining concurrency * docs: update workflow chaining plan * feat: add workflow stage postprocessors * feat: expose workflow stage outputs * fix: align workflow selected output export * fix: address workflow chaining review issues * fix: align workflow parquet export selection Signed-off-by: Andre Manoel <amanoel@nvidia.com> * fix: preserve generated columns in drop validation Signed-off-by: Andre Manoel <amanoel@nvidia.com> * fix: clarify workflow output processors Signed-off-by: Andre Manoel <amanoel@nvidia.com> * docs: add workflow chaining page * docs: align workflow chaining warning * fix: address workflow review nits --------- Signed-off-by: Andre Manoel <amanoel@nvidia.com>	2026-05-18 20:15:47 -03:00
Eric W. Tramel	a4085c441a	feat: add AIMD startup ramp (#638)	2026-05-13 16:25:03 -04:00
Johnny Greco	ef761b824b	docs: add "Have It Your Way" plugin dev note (#608) ... * docs: add plugins dev note * docs: mention custom columns Signed-off-by: Johnny Greco <jogreco@nvidia.com> * docs: update plugins dev note * docs: refine plugins dev note * docs: link v0.6.0 release --------- Signed-off-by: Johnny Greco <jogreco@nvidia.com>	2026-05-13 15:01:37 -04:00
Johnny Greco	d14c9b3ccc	feat(cli): add plugin catalog core (#618) ... * feat(cli): add plugin catalog services Add typed catalog and tap models, persistent tap storage, cached catalog loading, compatibility evaluation, install plan generation, and runtime plugin discovery helpers. Refs #617 * feat(cli): add plugins command group Wire list, search, info, install, installed, and tap management commands through the existing command-controller CLI pattern. Refs #617 * test(cli): cover plugin catalog workflows Add regression coverage for tap caching, catalog compatibility, installer command generation, local path resolution, and Typer command delegation. Refs #617 * fix(cli): align plugin taps with schema v2 Validate tap catalogs against the schema v2 contract used by NVIDIA-NeMo/DataDesignerPlugins#36, including source union fields, docs URLs, package paths, compatibility metadata, and unique runtime plugin names. Derive Git install targets as package-qualified PEP 508 direct references so git tap entries install the package described by the catalog source metadata. Refs #617 * fix(cli): address plugin review feedback - Invalidate import caches before post-install entry point verification - Make tap aliases case-insensitive and cache catalogs by alias plus URL - Prefer compatible catalog entries before falling back to forced installs - Clarify unused --tap behavior and list installed entry points without imports - Add direct controller coverage and update CLI plugin documentation Refs #617 * fix(cli): gate incompatible plugin installs Fetch install targets before compatibility filtering so the controller owns the final --force decision and the incompatible install guard stays reachable. Refs #617 * style(cli): format plugin catalog files Apply ruff formatting to the plugin command and tap repository tests so CI format checks pass on the PR merge commit. Refs #617 * fix(cli): reject duplicate plugin entry names Key catalog duplicate detection by entry_point.name so distinct catalog entries cannot register the same runtime plugin name. Refs #617 * fix(cli): preserve GitHub tree tap paths * fix(cli): verify plugin entry point names * align plugin CLI with catalog schema - adopt catalog terminology for plugin source aliases - parse package-first plugin catalog metadata from the plugin repo - install package requirements with optional catalog indexes * tidy plugin catalog workflow docs * align plugin catalog CLI with package contract * add plugin package uninstall workflow * test plugin package command targets * document plugin package aliases * address plugin catalog review feedback * prefer runtime plugin lookup matches * rename plugins command to plugin * show plugin package descriptions * rename plugin catalogs command * add protected plugin package installs * document plugin package install modes * avoid building project during plugin installs * harden plugin package installs * tighten plugin catalog contracts * fix no-args help exit code * make plugin docs links robust * document plugin CLI catalog workflows * clarify plugin entry point verification * simplify plugin CLI docs * narrow plugin search fields * hide plugin catalog cache ttl * remove plugin catalog trust flag * improve plugin CLI recovery UX * polish plugin catalog table display * stabilize plugin catalog table test * tighten plugin catalog edge cases * harden plugin catalog verification - Escape catalog-provided Rich markup before rendering CLI output - Reject runtime plugin names that collide after enum-key normalization - Load installed runtime entry points in a subprocess before reporting success * simplify plugin entry point verification Load matching entry points directly after install instead of spawning a separate Python process. This keeps the check package-scoped while still catching broken entry-point targets and non-Plugin objects. * require newer uv for plugin plans Use uv >= 0.10.0 as the single supported uv requirement for plugin package commands. Auto mode now falls back to a pip plan with an upgrade warning when uv is unavailable or too old, while explicit uv selection remains strict. * verify pip fallback availability * polish plugin CLI status markers * clarify plugin compatibility labels * simplify plugin info install details * address plugin CLI review nits * support versioned plugin package installs * share plugin install metadata rendering * show installed plugin packages * harden versioned plugin installs - Preserve catalog requirement constraints for versioned installs - Remove stale install-plan metadata fields - Expand parser, uv, controller, and local-catalog dry-run coverage * harden plugin help tests * show plugin package versions Add package version metadata support for plugin catalogs and resolve current versions from exact requirements or simple indexes when catalog entries omit them. Update plugin list/info/install metadata to show the plugin package version and Data Designer compatibility requirement while removing the separate Data Designer version line. * format plugin catalog tests * harden plugin package metadata checks * harden plugin CLI test coverage * add plugin discovery docs (#642) Signed-off-by: Johnny Greco <jogreco@nvidia.com> --------- Signed-off-by: Johnny Greco <jogreco@nvidia.com>	2026-05-13 12:26:58 -04:00
Andre Manoel	46dc8b232a	docs: prepare Fern docs workflow (#622) ... * docs: prepare fern generated artifacts * docs: update fern migration artifacts * docs: leave colab notebooks unchanged * docs: add VLM recipe cards to Fern * docs: trim Dev Notes sidebar * docs: collapse older Dev Notes in sidebar * docs: add Fern publishing workflows * docs: gate Fern publishing on check * docs: restrict hosted previews for fork PRs * docs: clean Fern preview URL * docs: cancel stale preview runs * docs: clarify devnotes notebook reuse * docs: clean older versions route * docs: document Fern versioning conventions * docs: add Fern release version guard * docs: harden Fern release tag handling * ci: let docs preview continue after fern failure * ci: split docs preview deploy * docs: clarify fern make commands * ci: harden fern deploy workflows * docs: render preview notebooks without outputs * ci: keep docs preview deploy inline * docs: align notebook code highlighting * docs: show notebook snippet scrollbars * docs: isolate fern preview check failures * ci: align fern release docs behavior	2026-05-12 18:18:26 -03:00
Nabin Mulepati	4b93f5b245	feat: let column configs declare all model aliases for the startup health check (#626) ... * feat(engine): let column configs declare all model aliases for the startup health check Plugin column configs that depend on more than one model alias (generator + judge, critic, etc.) previously could not opt their secondary aliases into the standard startup health check, and configs without a `model_alias` field crashed the collection loop with AttributeError. Add `SingleColumnConfig.get_model_aliases()` as the single override hook the builder uses to enumerate aliases. The default returns the column's primary `model_alias` (if any), so built-in LLM, embedding, and image columns work unchanged. `CustomColumnConfig` overrides it to surface decorator-declared aliases, replacing the special-case `isinstance` branch in the builder. Plugin configs with multiple model fields override it to opt every endpoint into the health check. Fixes #606 Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix(config): forward empty model_alias to startup health check SingleColumnConfig.get_model_aliases() used `if alias` to filter, which also dropped empty-string aliases. Empty model_alias values are accepted by the config model and previously reached run_health_check, where they failed fast with "No model config with alias '' found!". Treating them as "no model endpoints" silently delayed that error to first generation. Use `alias is not None` so only a truly missing attribute skips the health check, and add a regression test that exercises an empty-string model_alias on a built-in config. Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> --------- Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com>	2026-05-11 11:33:50 -06:00
Andre Manoel	16db8d61fa	fix(config): update OpenRouter vision model id (#630) ... * fix(config): update OpenRouter vision model id * fix(ci): harden provider health checks * fix(config): use nano omni for OpenRouter vision * docs: warn about hosted provider data handling * fix(config): align OpenRouter vision params	2026-05-11 13:49:17 -03:00
Lawrence Lane	7b5854ca36	docs: migrate documentation from MkDocs to Fern (#581) ... * docs: migrate documentation from MkDocs to Fern Adds a Fern Docs build under fern/ alongside the existing mkdocs site. Production target docs.nvidia.com/nemo/datadesigner with floating-latest pointer (latest.yml symlink) at v0.5.8. Migrated all concept, recipe, plugin, dev-note, and tutorial pages to MDX with NVIDIA theme and custom components (Authors, MetricsTable, TrajectoryViewer, NotebookViewer, BadgeLinks). Tutorial notebooks now render via NotebookViewer with captured outputs (text, DataFrames, inline images) - new make targets generate-fern-notebooks and generate-fern-notebooks-with-outputs drive the .py -> executed .ipynb -> Fern JSON+TS pipeline, pinning docs to Python 3.13 to dodge pyarrow wheel issues on 3.14. Python API reference is configured via Fern libraries: pointing at data-designer-config; output is gitignored and regenerated locally with 'fern docs md generate'. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> Signed-off-by: Lawrence Lane <llane@nvidia.com> * docs: add datadesigner-docs agent skill Captures the patterns established in the Fern migration so agents (and humans) can maintain fern/ confidently. Modeled after NVIDIA-NeMo/Gym's nemo-gym-docs SKILL.md, adapted for our floating-latest versioning, notebook-with-outputs pipeline, dev-notes kit components, and the MDX gotchas hit during migration (pymdown attr_list, --8<-- snippet syntax, frontmatter authors-as-JSX-scope-variable, etc.). Routes triggers like "edit docs", "add doc page", "regenerate notebooks", "update dev note", "add API reference" to this skill. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> Signed-off-by: Lawrence Lane <llane@nvidia.com> * docs: address PR review for Fern migration - Delete stale fern/versions/_nav_order.yml (references non-existent ./versions/latest/pages/ — paths were never updated when latest/ was renamed to v0.5.8/, no consumer found in docs.yml or v0.5.8.yml). - Remove unused custom components: Tag.tsx, CustomCard.tsx, Include.tsx (had its own untested markdown parser), ExpandableCode.tsx (broken in Fern SSR runtime). Drop expandable-code.css from docs.yml. Authors, BadgeLinks, MetricsTable, NotebookViewer, TrajectoryViewer remain (each has at least one call site). - BadgeLinks: remove DEFAULT_BADGES with placeholder URLs; make `badges` prop required so we can never accidentally ship 'your-org/your-repo'. - NotebookViewer: document the XSS trust boundary on output cells of format: "html". Outputs flow .py source → jupytext --execute → committed *.ts (review boundary). Add an inline comment at the dangerouslySetInnerHTML call site pointing back to the trust-model section. - README: add Windows caveat on the latest.yml symlink — Windows users need core.symlinks=true before clone or Fern will reject the version config. - Makefile: tighten generate-fern-notebooks source probe from `ls .../*.ipynb` (which can return success on non-file errors) to `[ -f docs/notebooks/1-the-basics.ipynb ]`, matching the reviewer's suggestion. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> Signed-off-by: Lawrence Lane <llane@nvidia.com> * docs: address @aschilling-nv review on fern/docs.yml Three suggestions from the Fern review, all matching Curator's docs.yml conventions: - instances[0].url: drop the https:// protocol prefix to match Curator's shape (e.g. nemo-curator.docs.buildwithfern.com/nemo/curator). - logo.href: was '/'; now points at /nemo/datadesigner/getting-started/welcome (the actual landing page) so clicking the logo lands on real content instead of the bare basepath. - experimental.basepath-aware: true — opts into Fern's basepath-aware routing so internal links don't double-prefix the /nemo/datadesigner segment. - redirects: also fix /nemo/datadesigner/index.html → getting-started/welcome (was bouncing to /latest, which is just the version slug); add /getting-started → /getting-started/welcome to mirror Curator's /home → /home/welcome convention. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> Signed-off-by: Lawrence Lane <llane@nvidia.com> * docs: put dev notes overview timestamps on separate lines Signed-off-by: Kirit93 <kthadaka@nvidia.com> Made-with: Cursor * docs: redesign dev-notes index with BlogCard component Replaces the generic <CardGroup>/<Card> grid (same green icon × 10, date glued to bottom of description) with a purpose-built BlogCard for the dev-notes landing page. Each card now has: - Hero image (16:9, lazy-loaded, click-to-zoom via Fern's rmiz wrapper) - ALL-CAPS date eyebrow as proper subtitle styling - Title, 3-line clamped description - Author byline at the bottom: avatar stack (overlapping) + first author name + "+N", pulling from the existing devnotes/.authors.yml registry - Hover: NVIDIA-green border + subtle lift Posts without a hero image fall back to a deterministic hash-based gradient placeholder + monogram (DJB2 hash of href → HSL hue, with the muddy-yellow band 40–90° remapped). Same post always gets the same look. Notes: - Image prop is React.ReactNode (not string) — pass <img> JSX from MDX so Fern's link rewriter can resolve the src to /_local/... in dev and /nemo/datadesigner/assets/... in prod. Raw string props bypass the rewriter and 404 in dev. - Card href runs through a small withBasepath() helper since the <a> also bypasses Fern's link rewriter. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> Signed-off-by: Lawrence Lane <llane@nvidia.com> * docs: flush blog-card hero images to the top of the card Fern's prose stylesheet applies a top margin to <img> tags, and the click-to-zoom wrapper Fern injects around each image (<span data-rmiz>) inherits that margin too. Result: a ~1rem gap between the card's top edge and the hero image. Reset margin/padding on the rmiz wrapper spans + the img itself inside .blog-card__media so the image renders flush against the top border. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> Signed-off-by: Lawrence Lane <llane@nvidia.com> * docs: stop blog-card hero from opening Fern's click-to-zoom modal When an <img> appears in MDX, Fern auto-wraps it with a click-to-zoom shell (<span data-rmiz>...). On the dev-notes index that shell intercepts clicks meant for the card's <a> wrapper, so clicking a hero opens a lightbox AND tries to navigate. Set pointer-events: none on the rmiz spans + img inside .blog-card__media so clicks bubble straight to the parent <a> and the card behaves as a single, predictable link target. Hover still works because pointer-events on children doesn't block :hover on the ancestor <a>. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> Signed-off-by: Lawrence Lane <llane@nvidia.com> * docs: render notebook markdown at build time with markdown-it-py Replaces NotebookViewer's hand-rolled JS markdown parser (the one with the ^@BR^@ sentinel the reviewer flagged as fragile) with build-time rendering in the converter. ipynb-to-fern-json.py now uses markdown-it-py (CommonMark + tables + strikethrough + raw HTML) to render each markdown cell's source into source_html, mirroring how code cells already store Pygments-highlighted source_html. NotebookViewer's markdown branch becomes a single dangerouslySetInnerHTML on the pre-rendered HTML, with a plain-escape fallback for old snapshots. Removes the dead JS helpers (renderMarkdown, isSafeUrl, UL_CLASS, OL_CLASS) — ~60 lines of brittle regex-based markdown parsing. Fixes broken rendering of: - Blockquotes (showed literal > characters before) - Nested content inside blockquotes (e.g. blockquote with bullet list) - Fenced code blocks - Tables - Multi-paragraph list items Includes regenerated fern/components/notebooks/*.{json,ts} for all 6 tutorials. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> Signed-off-by: Lawrence Lane <llane@nvidia.com> * docs: rewrite recipes index + replace octicons download links with Fern Info callouts The recipes/cards.mdx page was still in MkDocs Material format: - <div class="grid cards" markdown> wrapper (no-op in MDX) - :material-snake:, :material-database:, :material-tools:, etc. (rendered as literal text — Fern uses Font Awesome, not Material icons) - !!! tip Prerequisite (mkdocs admonition syntax) - [:material-book-open-page-variant: View Recipe] / [Download Code :octicons-download-24:] links with embedded icon shortcodes Rewrite using Fern's native components: <CardGroup cols={2}> with <Card title icon href> grouped by category (Code Generation, QA and Chat, Trace Ingestion, MCP and Tool Use, Plugin Development). Each card has one primary action (the recipe page); download lives on the recipe page itself. Replace the trailing "Download Code :octicons-download-24:" link on every recipe page (and 2 dev notes) with a <Info title="Download Recipe"> callout pointing at the GitHub blob URL — matching PR #215's convention. 12 occurrences across 12 files. Also fixes 6 recipe pages whose frontmatter title was "Untitled" (unfilled placeholder from auto-migration): text_to_python, basic_mcp, pdf_qa, multi_turn_chat, product_info_qa, agent_rollout_distillation. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> Signed-off-by: Lawrence Lane <llane@nvidia.com> * docs(fern): mirror main's content updates into v0.5.8 MDX pages Forward-port the doc changes that landed in main since this branch was cut, translating MkDocs admonition syntax to Fern components. Three product changes drove the updates: PR #594 — deprecate implicit default-provider routing: - concepts/models/configure-model-settings-with-the-cli.mdx: deprecate "Change default provider" workflow + inline mark on `data-designer config list` output - concepts/models/custom-model-settings.mdx: warning that `provider=` is now required on every ModelConfig - concepts/models/default-model-settings.mdx: warning that the registry-level default-provider concept is deprecated - concepts/models/model-providers.mdx: same warning at the top of the ModelProvider overview - concepts/models/inference-parameters.mdx: add explicit `provider= "openai"` to the dalle ModelConfig example PR #592 — async engine becomes the default: - concepts/architecture-and-performance.mdx: rewrite Execution Model intro to mention both engines, qualify "How It Works" as sync-engine semantics, update Concurrency Formula and Throttle notes from "Sync engine caveat" to "Engine paths", and add a full new "## Async Engine" section (per-model timeouts, run outcomes / Early Shutdown, opt-out via DATA_DESIGNER_ASYNC_ENGINE=0). Add `provider="nvidia"` to the my-model example. - concepts/custom_columns.mdx: note that sync `cell_by_cell` generators dispatch concurrently under the async engine; mock with `MagicMock(spec=ModelFacade)` so async methods are auto-detected. - concepts/processors.mdx: warning that the async engine enforces row-count invariance in process_before/after_batch. - devnotes/posts/async-all-the-way-down.mdx: append an "Update" callout noting the engine is now default, with a link to the Architecture page anchor. All `!!! warning|note|tip "Title"` admonitions converted to Fern <Warning|Note|Tip title="..."> components. Internal links to mkdocs relative paths (`../../concepts/foo.md#anchor`) rewritten to canonical Fern URLs (`/concepts/foo#anchor`). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> Signed-off-by: Lawrence Lane <llane@nvidia.com> * docs(fern): address @andreatgretel review comments Four issues from Andre's review pass: 1. /devnotes 404 (index.mdx:23) — section slug is /dev-notes, page slug is /dev-notes/overview. Fix the link in the landing page so visitors actually reach the dev notes index. 2. TrajectoryViewer.tsx final-answer body shown as literal markdown (line 66) — the renderer uses dangerouslySetInnerHTML but example-marcia.ts shipped raw markdown (**bold**, \n\n breaks). Visible on the deep-research devnote where the trajectory is defaultOpen. Pre-render body to HTML in the fixture (matches the original hand-coded format pre-migration); document the convention in the ToolCall.body doc comment so future fixtures don't regress. 3. Tutorials 5/6 (image generation/editing) ship with 0 captured outputs because Flux runs through OpenRouter and OPENROUTER_API_KEY isn't set at build time. Cannot regenerate without the key, so add a <Note> at the top of each wrapper page pointing readers at the Colab link to execute the cells live and see the generated images. Maintainers with the key in their environment should re-run `make generate-fern-notebooks-with-outputs` before merge to capture the snapshots. 4. Legacy nvidia-nemo.github.io/DataDesigner/* URLs in MDX prose (8 occurrences across 5 files) rewritten to canonical Fern paths so visitors don't get sent back to the legacy GitHub Pages site once docs.nvidia.com/nemo/datadesigner becomes the production URL: - The single deep link in data-designer-got-skills.mdx → /concepts/models/default-model-settings - All other "documentation home" links (CONTRIBUTING ×2, async-all-the-way-down ×2, owning-the-model-stack, design-principles ×2) → /getting-started/welcome (the canonical landing slug, matches logo.href in docs.yml) Notebook .py source URLs are tracked separately as part of the notebook-regen work. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> Signed-off-by: Lawrence Lane <llane@nvidia.com> * docs(fern): regenerate notebook snapshots with Flux outputs captured Re-ran make generate-fern-notebooks-with-outputs with NVIDIA_API_KEY + OPENROUTER_API_KEY set, now that we have a NVIDIA key with permission on nemotron-3-nano-30b-a3b. All 6 tutorials regenerated; the two image tutorials (5 and 6) which had been shipping with 0 outputs now have captured Flux generations: 1 the-basics: 12/15 outputs 2 structured-outputs: 13/17 outputs 3 seeding-with-a-dataset: 10/13 outputs 4 providing-images: 13/17 outputs (1 image) 5 generating-images: 8/10 outputs (2 images) ← was 0/12 6 image-to-image-editing: 9/12 outputs (10 images) ← was 0/14 The two `<Note title="Run in Colab to see ...">` workarounds I added on the 5/6 wrapper pages are no longer needed — outputs render inline now. NotebookViewer's own "Run in Google Colab" banner is still rendered from the wrapper's `colabUrl` prop, so the live-execute path stays one click away. Bumps the diff size noticeably (notebook 6 .ts is ~22MB of base64- encoded PNGs from 10 edited images), but that's intentional — these images are the proof points for what the Flux/MCP image-context tutorials actually produce. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> Signed-off-by: Lawrence Lane <llane@nvidia.com> * docs(fern): unbreak SSR — shrink notebook image outputs + fix BlogCard React import Two server-side render bugs surfaced when running `fern docs md generate && fern docs dev` (the static-preview path): 1. The 22 MB notebook 6 .ts module (full-resolution Flux PNGs from 10 edited images) tripped Fern's SSR module-evaluation step. Once that module failed to evaluate, the shared component bundle failed to load on every page, replacing each MDX body with `<span data-intent="error">Something went wrong!</span>` while the layout chrome continued to render. Fix in fern/scripts/ipynb-to-fern-json.py: after extracting an image/png output, pass it through Pillow to (a) downscale so the longest edge is at most 800 px, (b) re-encode as JPEG q=82 progressive (Flux outputs are photographic — JPEG compresses 5–10× better than PNG for this content). NotebookViewer's CellOutput interface gains a `mime` field so the data URL uses the actual encoded MIME type. Result: notebook 6: 22 MB → 4.6 MB notebook 5: 3.8 MB → 1.8 MB notebook 4: 514 KB → 116 KB (notebooks 1–3 unaffected — no image outputs) 2. fern/components/BlogCard.tsx referenced `React.ReactNode` twice without importing React. Other components in the kit use `import type { ReactNode } from "react"`; BlogCard was the outlier. Aligned the import style — even though this didn't end up being the trigger, leaving the dangling reference would have eventually caused a strict-mode SSR regression. Sweep test against http://localhost:3000/nemo/datadesigner/* — landing, concepts, tutorials (including 5/6 image notebooks), dev notes, recipes, and code-reference topic pages all render with their content; no error spans. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> Signed-off-by: Lawrence Lane <llane@nvidia.com> * docs(fern): add MkDocs-shape redirects for legacy URLs The legacy site at https://nvidia-nemo.github.io/DataDesigner/ used MkDocs-Material conventions (mkdocstrings + blog plugin + mkdocs-jupyter + directory URLs). Several path segments and page slugs differ from Fern's slugified-title routing — search-engine indexed links and copy-pasted bookmarks land on 404 without redirects. Adds 30+ specific redirect rules covering every renamed surface: - Tutorials: /notebooks/<filename>/ -> /tutorials/<title-slug> (page-title slugs differ from .ipynb filenames; one rule per notebook plus a README -> overview alias). - Recipes: /recipes/<snake_subsection>/<snake_page>/ -> /recipes/<kebab-subsection>/<kebab-page>. Per-page rules for each of the 10 recipes (page titles diverged from .py filenames — e.g. basic_mcp -> basic-mcp-tool-use, search_agent -> nemotron-super-search-agent), followed by subsection :rest* fallbacks. - Concepts: /concepts/mcp/* -> /concepts/tool-use-mcp/* (subsection rename, with & dropped, not -and-). Per-page rules for safety-and-limits -> safety-limits and configure-mcp-cli -> cli-configuration where page titles diverged from filenames. - Code Reference: /code_reference/<module>/ -> /code-reference/topic-overviews/<module>. Per-page rules for the six underscored modules (column_configs, config_builder, run_config, sampler_params, validator_params, data_designer_config) since Fern's page-slug rule kebabs underscores. - Plugins: filesystem_seed_reader -> file-system-seed-reader-plugins (Fern inserts hyphens between CamelCase words). example -> example-plugin, available -> available-plugin-list (page-title slugs). - Dev Notes: blog plugin's /devnotes/posts/<slug>/ -> /dev-notes/<slug>. Per-page rules for text-to-sql -> text-to-sql-for-nemotron-super and rqa -> rqa-dataset (post titles diverged from filenames). - /devnotes -> /dev-notes/overview (section landing). MkDocs's directory-URL trailing-slash convention is handled natively by Fern's runtime (both /foo and /foo/ return the same page), so no explicit slash-strip rule is needed. Smoke-tested all 34 legacy URLs against http://localhost:3000 — every one resolves to a 200 page on the new structure. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> Signed-off-by: Lawrence Lane <llane@nvidia.com> --------- Signed-off-by: Lawrence Lane <llane@nvidia.com> Signed-off-by: Kirit93 <kthadaka@nvidia.com> Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> Co-authored-by: Kirit93 <kthadaka@nvidia.com> Co-authored-by: Andre Manoel <165937436+andreatgretel@users.noreply.github.com>	2026-05-07 14:12:58 -03:00
Johnny Greco	8b8d748446	docs: graduate plugins out of experimental mode (#603) ... * chore: add __init__.py to engine namespace subpackages Griffe (used by mkdocstrings) skips directories without __init__.py when resolving module paths, which prevented the new plugins code reference from rendering SeedReader, FileSystemSeedReader, and Processor. Adding empty __init__.py files in engine/resources/, engine/processing/, and engine/processing/processors/ aligns with the convention already used in engine/mcp/, engine/models/, etc. * docs: flesh out docstrings on plugin extension-point classes Plugin authors now see meaningful descriptions for every field and method on the bases rendered in the plugins code reference: - Plugin and PluginType: class docstrings + Attributes tables for fields and enum members; fix typo in config_qualified_name field description. - SingleColumnConfig: document allow_resize. - ProcessorConfig: document processor_type discriminator. - SeedSource: document seed_type discriminator. - FileSystemSeedSource: add class docstring + Attributes table for path / file_pattern / recursive. - ColumnGeneratorFullColumn and ColumnGeneratorCellByCell: add class docstrings explaining when to use each base, plus method docstrings on the abstract generate() implementations. * docs: graduate plugins out of experimental mode Restructures plugin documentation around the now-stable extension points (column generator, seed reader, processor) and treats plugins as a first-class story for customizing Data Designer. - Add code_reference/plugins.md: single-stop reference for the Plugin object and the config + implementation base classes used by all three plugin types. - Add code_reference/generators.md: column generator implementation base classes, separated from column configs. - Surface SingleColumnConfig in code_reference/column_configs.md. - Add plugins/implement.md ("Build Your Own"): per-type implementation instructions across column generators, seed readers, and processors. - Add plugins/processor.md: complete processor plugin package example. - Rewrite plugins/overview.md: open with why plugins exist, drop the internal-helpers note (PluginRegistry / PluginManager), and focus the guide on what plugin builders need. - Refresh plugins/available.md (Catalog) and plugins/filesystem_seed_reader.md to match the new structure. - Delete plugins/example.md (replaced by per-type guides). - Reorder Code Reference nav alphabetically and add the new pages. - Minor link / wording fixes in concepts/processors.md and concepts/deployment-options.md. * docs: simplify plugin docs structure Replace the overview's how-to walkthrough and the per-type plugin guides with a single Build Your Own page that covers all three plugin types side-by-side. Add a dedicated Using Models in Plugins guide and a seed_readers code reference, and trim the overview down to what the plugin types are, how to use one, and how discovery works. - Rename plugins/implement.md to plugins/build_your_own.md. - Delete plugins/filesystem_seed_reader.md and plugins/processor.md (their content is now in build_your_own.md and the per-type code references). - Add plugins/models.md for model-backed column generator authoring. - Add code_reference/seed_readers.md for seed reader implementation base classes. - Rewrite plugins/overview.md: shorter intro, type bullets link to the relevant code reference, drop the multi-step "How do you create plugins" walkthrough in favor of a single Build a Plugin pointer, tighten Discovery troubleshooting. - Refresh plugins/available.md (Available Plugins): point to the DataDesignerPlugins catalog and explain how to request a community listing. - Update cross-page links in concepts/processors.md, concepts/seed-datasets.md, recipes/plugin_development/markdown_seed_reader.md, code_reference/plugins.md, and code_reference/generators.md to match the new structure. - Update mkdocs.yml nav: rename to Build Your Own, add Using Models, add seed_readers code reference. * docs: scroll wide tables horizontally instead of wrapping Code-heavy reference tables (plugin bases, column generators, etc.) were wrapping aggressively on narrow viewports, breaking long identifiers across multiple lines. Switch the table container to horizontal overflow and prevent code cells from wrapping so identifiers stay readable. * docs: address PR #603 review feedback - Add an Implementation base section to code_reference/processors.md rendering the engine-side Processor class. This justifies the engine/processing/__init__.py files added earlier and gives processor plugin authors an auto-rendered API reference, matching the pattern used by code_reference/generators.md and seed_readers.md. - build_your_own.md: replace the placeholder "x" emoji on the IndexMultiplier example with the actual multiplication sign. - build_your_own.md: drop the manual `re.compile + apply(lambda)` pattern in the regex-filter processor in favor of the idiomatic `Series.str.contains(..., regex=True)`. - build_your_own.md: add a kernel-restart caveat after the editable install instructions — PluginRegistry caches discovery on first import, so notebooks need a fresh kernel to pick up freshly installed plugins. - build_your_own.md: state explicitly what `assert_valid_plugin` checks (config base + plugin-type-appropriate impl base). - code_reference/plugins.md: link out to the processors code reference alongside generators and seed_readers. * docs: split code reference by package * docs: add interface code reference * docs: add code reference overviews * docs: refine code reference pages * docs: improve code reference tables * docs: correct reference docstrings * docs: embed plugin catalog table * docs: note plugin discovery restart caveat * docs: explain generator base class choice * docs: mention async cell generator examples * docs: clarify plugin model usage * docs: clarify plugin model aliases * docs: address plugin review feedback * docs: update available plugins page	2026-05-06 18:12:44 -04:00
Nabin Mulepati	f73da1975c	feat(models): deprecate implicit default provider routing (#594) ... * feat(models): deprecate implicit default provider routing Emit DeprecationWarning whenever the legacy "implicit default provider" path is exercised: `ModelConfig.provider=None`, the registry-level `ModelProviderRegistry.default`, the YAML `default:` key in `~/.data-designer/model_providers.yaml`, and the CLI's "Change default provider" workflow. `resolve_model_provider_registry` skips passing `default=` in the single-provider case so the common construction path stays quiet. Multi-provider registries still pass `default` (per `check_implicit_default`) and warn accordingly. Update docs, the package README, and test fixtures to specify `provider=` explicitly on every `ModelConfig`. New tests cover each warning entry point and pin the post-deprecation happy paths. Refs #589 Made-with: Cursor * fix(models): address PR #594 review feedback Greptile P1: ProviderRepository.load emitted its DeprecationWarning inside a `try/except Exception` block. Under `filterwarnings("error", DeprecationWarning)` the warn would raise, the except would swallow it, and `load()` would silently return None (losing the registry). Move the warn outside the catch-all so the strict-warning path no longer drops valid configs. Greptile P2 / johnnygreco: `_warn_on_implicit_provider` and `_warn_on_explicit_default` use `stacklevel=2`, which lands inside pydantic v2's validator dispatch rather than at the user's `ModelConfig(...)` / `ModelProviderRegistry(...)` call. That broke both attribution (the source line was unhelpful) and Python's once-per-location dedup (every call collapsed to the same pydantic-internal key, suppressing all but the first warning). Introduce `data_designer.config.utils.warning_helpers.warn_at_caller`, which walks past the helper, validator, and any pydantic frames to find the user's call site and emits via `warnings.warn_explicit` with the user frame's `__warningregistry__`. Keeps attribution accurate and dedup keyed on the user's (filename, lineno). johnnygreco: align the `provider_repository.py` warning copy with the sibling site in `default_model_settings.py` ("specify provider= explicitly on each ModelConfig instead") so both YAML-default warning sites give the same migration instruction. The previous wording pointed users at "ModelConfig entries" inside `model_providers.yaml`, where ModelConfig entries don't actually live. johnnygreco: dedup the cascade in `DataDesigner.__init__`. With `model_providers=None` and a YAML `default:`, the user previously saw two DeprecationWarnings for the same root cause — `get_default_provider_name()` warns about the YAML key, then `resolve_model_provider_registry(...)` re-warns from `_warn_on_explicit_default`. Suppress the registry-level duplicate in the YAML-fallback branch via `warnings.catch_warnings()` so users see exactly one warning per user action. johnnygreco: tighten `_warn_on_explicit_default` to fire only when `default is not None`. Passing `default=None` explicitly is semantically equivalent to omitting it (caller is opting *out* of a registry-level default), and shouldn't trigger the deprecation nudge. johnnygreco: add a `model_validate({...})` regression test for `ModelConfig` so the deserialization path (legacy on-disk configs) is pinned alongside the construction path. Tests: - Update `test_load_exists` and `test_save` to omit `default=` so the roundtrip stops exercising the deprecated YAML-default path unguarded (Greptile note). - Wrap `test_resolve_model_provider_registry_with_explicit_default`, `test_get_provider`, and `test_init_user_supplied_providers_preserve_first_wins_over_yaml_default` in `pytest.warns` so the suite stays green under `-W error::DeprecationWarning` (Greptile note). - Add `test_explicit_default_none_does_not_emit_deprecation_warning` to pin the tightened predicate. - Add `test_init_yaml_default_emits_single_deprecation_warning` to pin the cascade-dedup behavior. Refs #589 Made-with: Cursor * fix(models): make deprecation warnings visible under default filters andreatgretel (PR #594): the YAML-default warning in `get_default_provider_name` and the registry-default warning emitted from inside DataDesigner helpers were attributing to data_designer library frames, not user code. Python's default filter chain includes `ignore::DeprecationWarning`, so library-attributed entries are silenced — meaning a normal `DataDesigner()` call with a YAML `default:` set showed nothing, and `resolve_model_provider_registry` warnings were similarly invisible. Two related changes: 1. `warn_at_caller`: extend the default skip-list from `("pydantic",)` to `("pydantic", "pydantic_core", "data_designer")` so the walk escapes both pydantic's validator-dispatch frames and data_designer helper frames before attributing. Also tighten the prefix predicate to exact-or-dotted-prefix matching (`name == p or name.startswith(p + ".")`) so e.g. `pydantic_helpers` is not falsely matched as part of `pydantic` (johnnygreco nit). Allow callers to pass a custom `skip_prefixes` for flexibility. Drop the "skip frame 0+1 unconditionally" guard now that prefix matching covers it. 2. `get_default_provider_name`: switch from `warnings.warn(stacklevel=2)` to `warn_at_caller`. The previous stacklevel pointed into `default_model_settings.py`, which is a library file → silenced under default filters. Verified the fix empirically with `python -W default`: warning is now attributed to the user's call site and rendered. johnnygreco (PR #594): add the missing `test_explicit_default_none_does_not_emit_deprecation_warning` regression for the `self.default is not None` predicate landed in the prior round. Tests: - New `test_warning_helpers.py` pins prefix-matching precision (rejects `pydantic_helpers` / `data_designer_other`), default skip-list contents, attribution past skip-prefix frames, and per-call-site dedup behavior. - `test_get_default_provider_name_warning_attributes_to_user_frame` pins andreatgretel's repro for the YAML-default site. - `test_explicit_default_warning_attributes_to_user_frame` pins the multi-frame case: construction goes through `resolve_model_provider_registry`, so the walk has to escape both pydantic and data_designer before landing on the test file. - `test_explicit_default_none_does_not_emit_deprecation_warning` pins johnnygreco's predicate-tightening regression. 3,124 tests pass (540 config + 1,923 engine + 653 interface; +10 net from this round). Refs #589 Made-with: Cursor * fix(models): apply warn_at_caller to remaining deprecation sites greptile-apps (PR #594, r3189904028): `ProviderRepository.load`'s YAML-default `DeprecationWarning` was using `warnings.warn(stacklevel=2)`, which attributes to whichever data_designer frame called `load()` — controllers, services, list/reset commands, agent introspection. Every real call path lands on `data_designer.cli.*`, which falls under Python's default `ignore::DeprecationWarning` filter and is silenced. Audit found two more sites with the same problem: - `DatasetBuilder._resolve_async_compatibility` (`allow_resize` / issue #552) — was using `stacklevel=4` to walk past `_resolve_async_compatibility -> build/build_preview -> interface -> user`. Brittle: any added frame (decorator, async wrapping, the `try/except DeprecationWarning: raise` boundary) shifts attribution silently. The existing test passed only because it used `simplefilter("always") + record=True`, which records warnings regardless of attribution. - `ProviderController._handle_change_default` — was using `stacklevel=2`, which lands on the menu dispatcher in the same controller module. `print_warning` already shows the message visually, but programmatic observers (`pytest.warns`, `filterwarnings("error", ...)`) saw a library-attributed entry that default filters silenced. All three migrated to `warn_at_caller` (the helper from 247fa30) so attribution lands on the user's call site regardless of internal chain shape. `data_designer` is already in `DEFAULT_INTERNAL_PREFIXES`, so the walk escapes the entire library in one pass. Added attribution regression tests at each site asserting `warning.filename == __file__`. A future regression to `warnings.warn(stacklevel=N)` now fails CI instead of silently silencing the user-facing nudge: - `test_load_with_yaml_default_attributes_warning_to_caller` (test_provider_repository.py) - `test_resolve_async_compatibility` extended with the same assertion - `test_handle_change_default_emits_deprecation_warning` rewritten from `pytest.warns(...)` to a `catch_warnings(record=True)` block that filters for the message and asserts `filename == __file__` (`pytest.warns` does not check attribution, so the rewrite is required to actually catch the regression). 3,125 tests pass (548 config + 1,923 engine + 654 interface). Refs #589	2026-05-05 13:39:12 -06:00
Andre Manoel	61cdeefb17	feat: make async engine the default execution path (#592) ... * feat: make async engine the default execution path The async engine has been hardening as opt-in for several releases. Make it the default and address the prerequisites flagged for the flip. Default flip - DATA_DESIGNER_ASYNC_ENGINE defaults to "1" at both consumption sites - Set DATA_DESIGNER_ASYNC_ENGINE=0 for one transitional release to opt out - allow_resize=True still falls back to sync with a DeprecationWarning Python 3.10 support - Replace asyncio.TaskGroup (3.11+) in async_concurrency.py with gather-with-explicit-cancel; semantics preserved because _run_task already swallows its own exceptions and uses _shutdown_event for sibling cancellation - Remove the sys.version_info < (3, 11) runtime guard - Remove the matching pytest skipif so the executor tests run on 3.10 too Derived timeouts (replaces two hardcoded 300s constants) - ThrottleManager.acquire_sync/async default to timeout=None (no deadline) instead of DEFAULT_ACQUIRE_TIMEOUT=300; HTTP request timeout already bounds actual work, queue waits scale with provider speed and AIMD - _AsyncBridgedModelFacade derives the sync->async bridge timeout from the model's inference_parameters.timeout and the call's max_correction_steps; one knob (per-model timeout) drives both deadlines, no new config surface - Add ModelFacade.request_timeout property so the bridge can read the effective timeout the client is configured with Root-cause surfacing - AsyncTaskScheduler captures the first non-retryable error and exposes it via first_non_retryable_error - Interface threads it through DataDesignerGenerationError when 0 records are produced without early-shutdown, so deterministic failures (e.g. bad seed sources) surface their original message instead of a wrapped FileNotFoundError on the parquet path Tests - New: throttle no-deadline default behavior (sync+async), parametrized derived bridge timeout, restored async_concurrency tests on 3.10 - Updated: test_dataset_builder.py uses an autouse fixture to pin its Mock-based tests to the sync engine they cover; existing bridge tests set facade.request_timeout for the new derivation Docs - Replace the stale LiteLLM security notice in README with a short async-default heads-up and link to the migration guide - Add docs/migration-async-default.md covering per-model timeouts, custom-column thread safety, mocking model calls, run outcomes, and the opt-out - Append a short Update section to the async-all-the-way-down dev note * test: extract _compute_bridge_timeout helper for direct testing The parametrized bridge-timeout test was patching ``concurrent.futures.Future.result`` to capture the timeout the bridge passed in. That reaches into stdlib internals (DEVELOPMENT.md "Mock at boundaries: Keep mocking shallow") and the ``ids=`` argument on the parametrize was missing. Extracts the formula into a module-level ``_compute_bridge_timeout`` helper. The test now calls the helper directly with no mocking, and the parametrize gets readable ids. Behavior is unchanged. * test(e2e): align demo plugins with async engine contracts The e2e demo plugins exercise plugin discovery and full DD lifecycle. Two of them were written against sync-engine semantics that the async engine restricts: - DemoColumnGeneratorImpl was a ColumnGeneratorFullColumn with no required_columns. The async engine routes ``no-upstream`` columns through the from-scratch path, which passes an empty DataFrame to generators that aren't FromScratchColumnGenerator subclasses. The generator then produces 0 rows and the scheduler raises ``update_batch received 0 values``. Switching the plugin to FromScratchColumnGenerator with generate_from_scratch(num_records) matches what the plugin actually does (produces a constant column without input) and works on both engines. - RegexFilterProcessor implemented process_before_batch with row-count changes. The async engine enforces row-count invariance in pre- and post-batch processor stages by design. Moving the filter to process_after_generation preserves the plugin's purpose (regex-based row filtering) at a stage that supports row-count changes on both engines. Test assertions check the final dataset, so the stage shift is transparent. Both changes are demo-plugin updates only; no production code change. * fix: address Codex review findings on async-default flip Three bugs and two test-quality concerns surfaced by an independent review of the prior commits. Each was real and worth fixing in the flip PR. Bug fixes - Sync-fallback path was creating async-only model clients. The default flip meant ``client_concurrency_mode = ASYNC`` for every default run, but the ``allow_resize=True`` path falls back to the sync engine — sync ``model.generate()`` calls then hit ``SyncClientUnavailableError``. The resolution decision now lives at the DataDesigner interface level via ``_resolve_client_concurrency_mode``: it considers both the env var and the config (allow_resize forces sync clients) and is passed explicitly to ``create_resource_provider``. Direct callers of the factory still get the env-var default. - Sync→async bridge timeout ignored the per-call ``timeout=`` override. A custom column calling ``model.generate(timeout=600)`` against a slow endpoint was being cancelled at the model-config default, not 600s. The bridge now prefers ``kwargs.get("timeout")`` over ``facade.request_timeout``. - Bridge timeout formula missed ``max_conversation_restarts``. One logical generation can do ``(1 + max_conversation_restarts) × (1 + max_correction_steps)`` HTTP requests; the formula now multiplies both, matching the worst-case attempt budget. Engine routing fix (also surfaced by failing e2e plugin tests) - ``_run_from_scratch`` else-branch passed an empty DataFrame to non-FromScratch generators classified as seeds (no upstream columns), so ``ColumnGeneratorFullColumn`` with no required_columns produced 0 rows for an ``rg_size``-row buffer. Now passes an ``rg_size``-row snapshot of the row-group buffer, mirroring the sync engine's FULL_COLUMN contract. - The earlier ``DemoColumnGeneratorImpl`` workaround (rewrite as ``FromScratchColumnGenerator``) is reverted; the engine fix subsumes it. The processor-plugin fix (``process_after_generation`` for the regex filter) stays — pre-batch row-count change is intentionally rejected by the async engine. Test improvements - Throttle no-deadline tests are parametrized over ``(timeout=0.0, raises)`` and ``(timeout=None, waits)``, pinning that ``None`` is genuinely distinct from any finite default. Sync and async counterparts mirror. - New regression tests for ``first_non_retryable_error`` surfacing covering both load-raises and load-returns-empty paths, asserting the original exception is chained via ``__cause__`` and that the typed ``DataDesignerEarlyShutdownError`` doesn't fire in this branch. - New parametrized regression test for ``_resolve_client_concurrency_mode`` covering all four (env × allow_resize) combinations. - New parametrized test for the per-call ``timeout=`` override flowing into the bridge timeout calculation. - Bridge formula tests extended with ``max_conversation_restarts`` cases. * test: trim redundant parametrize cases in async-default tests Three parametrize cases were duplicating coverage already provided by existing standalone tests: - ``test_acquire_*_timeout_branches`` parametrized over ``(0.0, raises)`` and ``(None, waits)``. The ``raises`` half duplicates ``test_acquire_*_raises_timeout_when_at_capacity``. Replaced with two focused ``..._default_no_deadline_waits_for_release`` tests covering only the no-deadline branch. - ``test_resolve_client_concurrency_mode_matches_engine_choice`` had four cases. The ``async-off + allow-resize`` case asserts ``SYNC`` because the env var alone forces it; the allow_resize input is moot. Dropped. - ``test_async_bridge_honors_per_call_timeout`` had three cases. The "override below floor" case cross-products the per-call override flow with the floor-clamping behavior already covered by ``test_compute_bridge_timeout``. Dropped. Net: -25 lines of test code with no loss of essential coverage. * docs: fold migration page into existing concept docs The standalone ``Migrating to the async default`` page didn't fit the existing docs style — present tense, behavior over comparisons, content in the natural concept home. Folding it in: - ``architecture-and-performance.md`` gets a new ``Async Engine`` section covering per-model timeouts, run outcomes (partial completion + ``DataDesignerEarlyShutdownError``), and the transitional opt-out. Three stale ``async engine is landing soon`` callouts updated to reflect the flip. - ``custom_columns.md`` gets two short notes: a thread-safety callout near Generation Strategies, and a mocking-with-spec note in Development Testing. - ``async-all-the-way-down.md`` Update section now points at the new arch-and-perf section. - README heads-up links to the same anchor. - ``migration-async-default.md`` removed; mkdocs.yml entry dropped. * docs: frame Execution Model as sync-engine specifics Small targeted edits to make the user-facing concept docs consistent with the post-flip state. No restructuring. - ``architecture-and-performance.md``: the ``Execution Model`` callout now opens with two engines, links to the new ``Async Engine`` section, and frames the existing column-at-a-time description as sync-engine semantics. The ``Step 2: Process columns sequentially`` paragraph notes the async engine relaxes this. The ``Key Concepts`` table differentiates per-engine for ``Batching`` and ``Sequential columns``; ``Parallel cells`` is the same on both. - ``processors.md``: added a warning callout about the async engine's row-count invariance in pre- and post-batch stages, with the guidance to use ``process_after_generation()`` for row-filtering or expansion. * fix: address review nits from PR #592 (Nabin) Four targeted fixes from the review. Worth-addressing (warning): - ``test_acquire_async_default_no_deadline_waits_for_release`` was spawning the release task without holding a strong reference. The loop's weak-ref bookkeeping could GC it before the inner ``await`` observes the release, producing a CI flake. Hold the task and ``await`` it in ``finally``. Take-it-or-leave-it (applied): - Root-cause error surfacing now includes the exception type name: ``f"🛑 {type(root_cause).__name__}: {root_cause}"`` so users see ``ValueError: ...`` instead of just the message string. The ``__cause__`` chain is preserved either way. - Drop the defensive ``getattr(c, "allow_resize", False)`` in ``_resolve_client_concurrency_mode`` — every member of ``ColumnConfigT`` inherits ``allow_resize: bool = False`` from ``SingleColumnConfig``. - One-line comment near the root-cause surfacing branch noting that ``actual_num_records == 0`` is async-only (sync runs leave it at ``-1``), so the branch is async-only by construction. Not addressed in this PR (filing as follow-ups): - ``SYNC_BRIDGE_TIMEOUT = 300`` still hardcoded in ``column_generators/generators/base.py:_run_coroutine_sync``. That bridge has no model-facade context to derive a timeout from, so the fix is a structural refactor outside this PR's scope. - First-error capture loses subsequent-error context. The "first wins" heuristic is documented; richer aggregation is a follow-up. * fix: drop SYNC_BRIDGE_TIMEOUT in _run_coroutine_sync This was the third hardcoded 300s timeout (Nabin flagged it on PR #592). The path is the generic sync→async bridge in ``ColumnGenerator.generate()``: when a subclass overrides only ``agenerate()``, the sync entry point runs the coroutine in a background thread. Same philosophy we applied to the throttle queue wait elsewhere in the PR: a defensive deadline on top of work that's already bounded by the HTTP timeout doesn't add safety, it just produces spurious failures on slow self-hosted endpoints. Drop the constant, the timeout exception handling, and the ``timed_out`` bookkeeping. ``pool.shutdown(wait=True)`` becomes the simple cleanup. Tests in ``test_async_generators.py`` exercise the happy path only and don't depend on the timeout firing. * Revert "fix: drop SYNC_BRIDGE_TIMEOUT in _run_coroutine_sync" This reverts commit 7a0b77d44c. * docs+feat: deprecate the sync-engine opt-out path Nabin asked whether the docs should adopt explicit "deprecation" language on the opt-out path. Doing both: - Doc: ``architecture-and-performance.md``'s ``Opting out`` section now uses an ``!!! warning "Deprecated"`` admonition that names the env var as a deprecated escape hatch and notes the run-time warning. - Code: ``DataDesigner._resolve_client_concurrency_mode`` emits a ``DeprecationWarning`` when ``DATA_DESIGNER_ASYNC_ENGINE=0`` is detected. Same precedent as the existing ``allow_resize=True`` warning. Auto-fallback via ``allow_resize`` does not double-warn here; the builder layer emits its own warning later. - Test: parametrized regression now asserts ``pytest.warns(DeprecationWarning)`` on the opt-out branch and treats any warning on the async-on branches as a failure (``simplefilter("error")`` inside the ``catch_warnings`` block). * fix: emit logger.warning alongside DeprecationWarning on env-var opt-out Parity fix from Nabin's re-review of PR #592. The ``allow_resize=True`` auto-fallback path in ``_resolve_async_compatibility`` emits both a ``logger.warning("⚠️ ...")`` and a ``DeprecationWarning``. The new ``DATA_DESIGNER_ASYNC_ENGINE=0`` opt-out path was only emitting the ``DeprecationWarning``, leaving users who run with default warning filters silenced and inconsistent with the established precedent. Match the pattern: same message body, both signals, same stacklevel. * docs: breadcrumb explaining why SYNC_BRIDGE_TIMEOUT survives PR #592 Nabin's re-review pointed out that ``base.py`` is the lone place where the 300s pattern survives, while ``custom.py`` and ``throttle_manager.py`` both retired theirs. Without a comment, a future reader (or a lint sweep) could mistake this for an oversight and "consistency-fix" it the wrong way. Add a short note at the constant naming the two retired siblings, the reason this one stayed (no ``ModelFacade`` context to derive from), and the fact that it's tracked for a structural follow-up.	2026-05-04 16:22:13 -03:00
Kirit Thadaka	c119f8f580	docs: Updated telemetry disclosure (#595) ... * Updated telemetry details * Updated chart to YTD	2026-05-01 10:56:59 -07:00
Nabin Mulepati	93ae8757f2	feat: update default nvidia-vision model to nemotron-3-nano-omni (#583) ... * update nvidia-vision * update temp	2026-04-28 13:35:54 -06:00
Nabin Mulepati	e92c76b134	add links back to announcement pages (#580)	2026-04-28 10:20:07 -06:00
Nabin Mulepati	7c5a7221e0	docs: add VLM long-document understanding dev note and recipes (#579) ... * Add resources for long-document-understanding-dev-note * added links	2026-04-28 09:59:03 -06:00
Johnny Greco	a65903eb1a	chore: add ko_KR locale to nemotron personas datasets (#572) ... * chore: add ko_KR locale to nemotron personas datasets Register Korean (ko_KR, 2.66 GB) as an available managed persona dataset locale, update related CLI/repository tests, and document the new locale and its NGC download command. * update person fields * update fr_FR size * docs: reconcile personas field tables with installed parquet schemas Remove stale per-locale fields that no longer exist in any managed parquet (commune, departement, prefecture), drop district from the India-specific section since it's already listed in Core Fields, rename digital_skills → digital_skill to match the actual ja_JP column, and add sections for ko_KR, en_SG, and the en_US/en_SG shared ethnic_background. Corrects the religion-family membership to include en_SG. * test: add missing fr_FR assertion in test_run_personas_with_all_flag The test asserts all 9 locales were downloaded but only enumerates 8 in its per-locale checks — fr_FR has been missing since before the ko_KR addition. Align the enumeration with the count. * docs: add ko_KR to locale parameter list	2026-04-24 17:19:02 -04:00
Kirit Thadaka	4c6823c586	fix: Updated default reasoning model for nvidia (#568) ... * Updated default reasoning model for nvidia * Updated inference params for super * Add reasoning_effort to Nemotron Super params, update stale docs - Add extra_body.reasoning_effort=medium to NEMOTRON_3_SUPER_120B_A12B_INFERENCE_PARAMS (mirrors GPT-5 config) - Update README telemetry example and model-configs.md to use nvidia/nemotron-3-super-120b-a12b instead of openai/gpt-oss-20b - Broaden inference-parameters.md reasoning effort tip to cover Nemotron Super * Remove build-time README accidentally tracked --------- Co-authored-by: Andre Manoel <amanoel@nvidia.com> Co-authored-by: Andre Manoel <165937436+andreatgretel@users.noreply.github.com>	2026-04-22 13:50:26 -03:00
Eric W. Tramel	8be4ff787f	feat: add RunConfig jinja rendering engine (#557)	2026-04-17 15:06:27 -04:00
Nabin Mulepati	cebfb0e967	docs: Added starter dev notes on push to hugging face hub (#355) ... * Added starter dev notes on push to huggingface hub * fix: move excerpt marker to intro and remove redundant markers Move the single <\!-- more --> to after the intro paragraph for a shorter blog teaser and remove the 6 redundant markers throughout the post. * Update docs/devnotes/posts/push-datasets-to-hugging-face-hub.md Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * docs: add HF ecosystem context to push-to-hub dev notes (#474) * docs: add HF ecosystem context to push-to-hub dev notes Add section on what datasets get on the Hub (Dataset Viewer, streaming, Viewer API), link to Hub search for DataDesigner datasets, and note that private datasets can be flipped to public. * Update docs/devnotes/posts/push-datasets-to-hugging-face-hub.md Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * fix: remove doubled library: prefix in Hub search URL --------- Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * Update date * fix date for text-to-sql * update hero images" * updates --------- Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> Co-authored-by: Daniel van Strien <davanstrien@users.noreply.github.com>	2026-04-16 11:29:33 -06:00
Nabin Mulepati	a9af365e8e	feat: add skip.when conditional column generation (#502) ... * plan: add skip_when for conditional column generation (#479) Adds implementation plan for a `skip_when` field on `SingleColumnConfig` that enables conditional column generation. When the Jinja2 expression evaluates truthy, the cell is set to None and the generator is skipped. Skips auto-propagate through the DAG to downstream columns. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * plan: remove HopChain example from skip_when plan Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * plan: replace HopChain example with generic product review example Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * plan: add open questions on skip sentinel value and row filtering Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * plan: major revision — SkipConfig model, sync engine support, decouple propagation - Introduce SkipConfig(when, value) as nested model on SingleColumnConfig - Move propagate_skip to SingleColumnConfig as independent field, fixing bug where columns with no SkipConfig couldn't participate in propagation - Add full sync engine implementation (Steps 4a-4d) covering both _fan_out_with_threads and _run_full_column_generator dispatch paths - Add serialization boundary stripping for both DatasetBatchManager (sync) and RowGroupBufferManager (async) - Simplify architecture diagrams for readability - Update all references, design decisions, verification plan Made-with: Cursor * updates * plan: document get_required_columns for skip propagation - Explain why propagation must not use get_upstream_columns() once skip.when adds DAG edges; add _required_columns and get_required_columns() to the execution graph plan - Point async _run_cell at get_required_columns for parity with sync - Clarify DropSkippedRowsProcessorConfig vs stripping __skipped__ for DataFrames; tighten resolved-questions wording - Extend DAG/graph verification with gating_col regression case Refs #479 Made-with: Cursor * plan: centralize __skipped__ handling in skip_provenance - Document new skip_provenance.py (key constant, read/write/strip API) - Point sync builder, async scheduler, and batch buffers at shared helpers - Strip metadata before every DataFrame from buffer dicts, including FULL_COLUMN active subsets - Split §3 into skip_evaluator vs skip_provenance; extend verification Refs #479 Made-with: Cursor * plan: align doc title with SkipConfig / skip.when Drop legacy skip_when naming in headings and #362 cross-reference. Refs #479 Made-with: Cursor * plan: address review — delimiter validation, centralized error handling, caller-owns-deserialization - SkipConfig._validate_when_syntax now checks find_undeclared_variables is non-empty, rejecting expressions without {{ }} delimiters that would silently skip every row - evaluate_skip_when centralizes try/except so both sync and async engines get identical fail-safe behavior on eval errors - evaluate_skip_when takes a single pre-deserialized record; caller runs deserialize_json_values once and passes to both skip eval and generator (no double deserialization, no redundant parameter) - Update _should_skip_cell, async _run_cell, Files Modified table, and verification section accordingly Refs #479 Made-with: Cursor * plan: add get_side_effect_columns accessor to execution graph spec Document _side_effects_by_producer inverse map and get_side_effect_columns() accessor on ExecutionGraph, needed by _write_skip_to_record / apply_skip_to_record to clear __trace, __reasoning_content, etc. on skip. Added to both Step 2b metadata section and Files Modified table. The __skipped__ leak into active_df (greptile's other P1) was already fixed in 70463789 via strip_skip_metadata_from_records. Refs #479 Made-with: Cursor * add skip.when conditional column generation Introduce SkipConfig on SingleColumnConfig to gate column generation with a Jinja2 expression. Columns can be skipped by expression or by upstream propagation (propagate_skip flag). - SkipConfig: Pydantic model with config-time syntax/delimiter/variable validation and cached column extraction from the Jinja2 AST - skip_evaluator: runtime expression evaluation via NativeSandboxedEnvironment with fail-safe error handling (skip on expected failures) - skip_provenance: centralized __skipped__ record tracking shared by sync builder, async scheduler, and buffer managers - DAG/ExecutionGraph: skip.columns wired as dependency edges in both topological sort and static execution graph - Validation: validate_skip_references checks reference existence, sampler/seed scope, and allow_resize conflicts - Sync builder: cell-by-cell and full-column skip with merge-back - Async scheduler: cell and batch skip with live-buffer provenance Made-with: Cursor * fix review findings for skip.when implementation - Add skip evaluation to _fan_out_with_async (was missing, causing skipped rows to still be sent to the LLM) - Preserve __skipped__ provenance on non-skipped records after full-column generation so multi-hop propagation works - Use single live-buffer reference in _run_batch skip loop for consistency with _run_cell - Move Template import to TYPE_CHECKING and reorder import blocks - Replace O(n²) sum() with itertools.chain in dag.py - Add set_required_columns/set_propagate_skip/set_skip_config setters to ExecutionGraph for symmetry with existing API Made-with: Cursor * add conditional generation with skip recipe and refactor skip helpers Add a new recipe demonstrating skip.when patterns (expression gate, propagation, opt-out) with a customer support ticket pipeline. Also extract _should_skip_record in async_scheduler, remove the redundant propagate_skip param from should_skip_by_propagation, and pass a precomputed all_side_effects set through the DAG sort. Made-with: Cursor * updates * fixes * remove recipe > inject conditional gen into existing tutorial * regen colab notebooks * fix: handle missing execution graph in _column_can_skip Return False when the graph has not been initialized instead of raising, since skip logic cannot apply before generators are set up. Made-with: Cursor * parametrize some tests * public before private * slight refactor for readability * parametrize some tests * minor fixes * reanme internla skip tracker key name * clarify intent in comment * when skipped _run_cell should return skipped value even though the consumer doesn't currenlty care about it * remove inline import * minor refactor for clarity * fix: preserve skip metadata across replace_buffer and exclude allow_resize from skip branch Two bugs in the sequential engine's _run_full_column_generator: 1. replace_buffer(df.to_dict()) erased __internal_skipped_columns in three code paths (MultiColumnConfig, non-skip-aware, has_skipped=False fallthrough), breaking propagate_skip for downstream columns when an independent FULL_COLUMN generator ran between skip-setting and propagating columns. 2. _column_can_skip returned True for allow_resize=True columns via propagation, causing the skip-aware merge path to raise on the 1:1 row-count check for 1:N generators. - Add restore_skip_metadata helper to skip_tracker.py - Guard _column_can_skip against allow_resize=True columns - Refactor _run_full_column_generator into three focused methods - Remove dead allow_resize / _log_resize_if_changed from skip path - Remove redundant _require_graph() calls in skip helpers - Add single_column_config_by_name cached property - Add integration tests for both bugs and unit tests for the helper Made-with: Cursor * address review comments on skip.when PR (#502) - Extract shared skip decision logic (_should_skip_cell / _should_skip_record) into should_skip_column_for_record() in skip_evaluator.py so both sync and async engines call the same function (andreatgretel review comment) - Extend SkipConfig self-reference validation to cover side-effect columns (e.g. review__trace on the review column) — previously only checked self.name, now checks self.name | self.side_effect_columns - Add async engine integration tests for skip paths: cell-by-cell with propagation and full-column batch skip (exercises _run_cell / _run_batch) - Fix test_allow_resize_column_not_blocked_by_upstream_skip to use default propagate_skip=True so it actually exercises the allow_resize guard - Move get_skipped_column_names from skip_tracker to skip_evaluator (sole production consumer) Made-with: Cursor * address cr feedback * Fix issue with full column generating messing up order of skipped rows * add skip conditional generation edge case tests - test_skip_evaluator: parametrized should_skip_column_for_record covering propagation, expression gates, short-circuiting, and disabled propagation - test_execution_graph: skip metadata accessors (get_skip_config, should_propagate_skip, get_required_columns, get_side_effect_columns, resolve_side_effect, skip.when DAG edges) - test_dataset_builder: chained transitive propagation (4 levels), two independent skip gates, custom skip.value, row count preservation Made-with: Cursor * fix: make expression jinja validator private Rename assert_expression_valid_jinja to _assert_expression_valid_jinja to match the private naming convention used by other model validators. Made-with: Cursor --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-04-15 09:31:50 -06:00
Andre Manoel	1a237d95d0	fix: text-to-sql devnote date, images, and publish-devnotes nav (#546) ... - Update post date from 2026-03-11 to 2026-04-14 so it appears as the newest post on the devnotes page. - Replace raw <img> tags with markdown image syntax so mkdocs rewrites relative paths correctly for the blog plugin's slug-based URLs. - Overlay mkdocs.yml from HEAD in publish-devnotes workflow so new nav entries are included in devnotes-only rebuilds.	2026-04-14 15:48:23 -03:00
dhruvnathawani	1448f9cbda	docs: add text-to-sql dev note (#349) ... * docs: add text-to-sql devnote * add diagram, update content * correct inconsistencies * docs: address PR #349 feedback and add BIRD benchmark results PR feedback fixes: - Fix Window Functions contradiction: Key Takeaway #1 now uses "Geospatial SQL" (Advanced) instead of "Window Functions" (Intermediate) - Fix score-0 truthiness bug: use `is not none` instead of truthy check in Jinja2 expression columns (inline example + production pipeline) - Soften Code Sandbox language: "A natural next step would be..." instead of "We are actively implementing..." - Cut Gretel reference per mvansegbroeck: replaced with NVIDIA/Nemotron team description - Replace Qwen model references with Nemotron per mvansegbroeck: MODEL_NAME, ASCII diagram labels, Pipeline Overview prose - Rename sdg_qwen_235b.py -> sdg_ndd_text2sql.py per mvansegbroeck - Fix Try It Yourself: use MODEL_ALIAS = "nvidia-text" with default provider pattern (matches structured-outputs dev note), remove unused explicit ModelConfig - Remove placeholder dataset link (#), add "Dataset: Internal" note New content: - Add BIRD Benchmark Results section with bar chart (JPG), data table, BIRD caveat paragraph, and Jocelyn Huang acknowledgement (Nemotron Super EX: 26.77% -> 41.80%, +15 pts, beats GPT-OSS-120B) - Replace "Looking Ahead: Code Sandbox" with broader "Next Steps": Code Sandbox, RL on BIRD via NeMo Gym, schema representation, Spider 2.0 - Add Project Summary table at end of post * docs: address second round of PR #349 feedback - Fix "EHR Systems" -> "Electronic Health Records" in Key Takeaway #1 to match the exact taxonomy string in the code example (greptile) - Add admonition clarifying code snippets are illustrative, not runnable, with link to Enterprise Text-to-SQL Recipe (nabinchha) - Add context before score extraction snippet referencing the five LLMJudgeColumnConfig columns and linking to full recipe (nabinchha) - Add companion file note and recipe link to production pipeline details block for prompts.py, rubrics.py, text2sql_seed.json (nabinchha) * docs: address round 2 PR #349 feedback, replace production block with recipe - Fix "EHR Systems" -> "Electronic Health Records" in Key Takeaway #1 to match the exact taxonomy string in the code example (greptile) - Add admonition clarifying inline code snippets are illustrative, with link to runnable Enterprise Text-to-SQL Recipe (nabinchha) - Add context before score extraction snippet referencing the five LLMJudgeColumnConfig columns and linking to full recipe (nabinchha) - Replace production pipeline <details> block (230 lines with phantom imports from prompts.py, rubrics.py, text2sql_seed.json) with snippet include of enterprise_text_to_sql.py recipe — self-contained and runnable, consistent with other merged dev notes (nabinchha) * docs: polish Try It Yourself and Summary sections - Wrap minimal inline example in collapsible <details> dropdown - Rename "A Team Effort" section to "Summary" - Remove redundant Scale/Dialects/Dataset line * docs: add missing sql_dialect sampler to Step 1 code snippet The Step 3/4 prompt templates reference {{ sql_dialect }} but the Step 1 seeding code never defined it, leaving an unresolved Jinja2 variable for readers following along. Add the sql_dialect sampler with a comment explaining the pipeline runs once per dialect. * fix ascii diagram * docs: fix BIRD score framing and MySQL dialect wording - Remove specific "60-70%" BIRD claim from intro to avoid contradiction with the 41.80%/38.25% direct-generation results shown later (those higher figures come from specialized systems with schema linking) - Reword MySQL "forbids" to "prompts exclude" -- REGEXP_REPLACE and CONVERT_TZ are valid MySQL functions; the pipeline excluded them for portability, not because the dialect forbids them * docs: move text-to-sql images to assets/ convention and update refs * docs: address text-to-sql devnote review comments - Add devnote to mkdocs nav after Async All the Way Down - Swap Recursive CTEs to Advanced, CASE Expressions to Intermediate (matches recipe) - Fix score extraction truthy check to use 'is not none' (preserves score-0 values) - Drop REPLACE() vs regexp_replace from dialect takeaway (REPLACE is cross-dialect) - Tighten prose: remove 'The key insight:', use actual BIRD number, trim X-not-Y - Fix knowledge dependency count: 8 -> 9 concepts (3x3 in recipe) --------- Signed-off-by: Yev Meyer <ymeyer@nvidia.com> Co-authored-by: Yev Meyer <ymeyer@nvidia.com>	2026-04-14 11:10:14 -07:00
Johnny Greco	fdd5ebb5ef	feat: add Pi Coding Agent rollout seed source (#513) (#514) ... Add support for ingesting Pi Coding Agent session artifacts as an agent rollout seed source. Pi sessions are tree-structured JSONL files; the handler resolves the active conversation path by walking from the last entry back to the root via id/parentId links. Key points: - Tree-structured sessions with automatic active-path resolution - Entry-level types: model_change, compaction, branch_summary, custom_message, thinking_level_change - Message roles: user, assistant (inline ToolCall/ThinkingContent/ TextContent blocks), toolResult, bashExecution (synthesized as tool-call pairs), custom, compactionSummary, branchSummary - Extract shared normalize_message_content to utils.py (was duplicated in Hermes handler)	2026-04-09 12:07:47 -04:00
Andre Manoel	0e90ea644b	docs: add async engine dev note (#490) ... * fix: address review feedback on async engine dev note - Fix wall-clock claim: 41% -> 22% to match benchmark table - Fix dual-model speedup rounding: 1.7x -> 1.6x (10.0/6.1 = 1.64) - Fix run_config API: use dd.set_run_config() instead of passing to create() * docs: add async engine dev note Add "Async All the Way Down" dev note covering the async task-queue scheduler built across PRs #356, #378, #404, #429, #456. Includes benchmark results, architecture diagrams, and DAG shape illustrations. * feat: add docs preview workflow for PRs Build MkDocs site on PRs that touch docs and deploy to Cloudflare Pages. Each PR gets a browseable preview URL posted as a comment. Notebook tutorials use placeholder stubs since they require API keys to execute. Requires CLOUDFLARE_API_TOKEN and CLOUDFLARE_ACCOUNT_ID repo secrets. * fix: update speedup chart alt text from 1.7x to 1.6x * docs: improve timeline figure context and labeling Add DAG subtitle to sync-vs-async timeline figure and bridge the surrounding text to explain which workload shape is being shown. * edits+additions to async-all-the-way-down dev notes * clarify two semaphore dance * remove dead link * replace hero image * docs: update scale figures with nginx-accurate data and adjust sizing Regenerate scale-model-timeline and scale-boxplot from nginx access logs (column_progress.csv, sync/summary.json) instead of buffered execution logs. Optimize both PNGs to palette mode. Adjust figure widths and update model timeline commentary. * add link from owning-the-model-stack to async-dev-node * docs: address review feedback on async blog post - Tighten intro to a concise abstract, move pipeline narrative into "The Bottleneck Was Structural" section - Remove multi-column generators / seed readers paragraph (TMI) - Clarify sync engine ran columns sequentially within each batch --------- Co-authored-by: Nabin Mulepati <nmulepati@nvidia.com>	2026-04-08 15:51:04 -03:00
Eric W. Tramel	5f04e5d593	docs: add agent rollout ingestion docs entry point (#499)	2026-04-08 10:13:12 -04:00
Eric W. Tramel	7891dd53cb	feat: add Hermes Agent rollout support (#500)	2026-04-07 12:39:49 -04:00
Nabin Mulepati	f78c4e0cf7	Fix repeated header/footer on native-model-client-hero image (#492)	2026-04-06 09:45:27 -06:00
Eric W. Tramel	58870bb83f	feat: add ATIF rollout ingestion (#495)	2026-04-06 11:06:14 -04:00
Nabin Mulepati	7087add042	docs: update architecture-and-performance.md to reflect AIMD changes (#467) ... * docs: update architecture-and-performance.md to reflect AIMD concurrency control (#466) Account for the AIMD throttle manager throughout the architecture doc: concurrency formula, max_parallel_requests guidance, new ThrottleConfig section, common problems table, and tuning workflow. Add sync-engine caveats noting AIMD is fully active on the async path. Made-with: Cursor * updates * docs: address PR #467 review feedback on AIMD doc - Clarify that only the first 429 in a burst reduces the concurrency limit; subsequent in-flight cascade 429s hold it steady - Soften sync engine caveats: AIMD engages as a fallback when transport retries exhaust, not "no effect" - Note salvage queue difference between async and sync paths when recommending aggressive max_parallel_requests values Made-with: Cursor	2026-03-31 16:12:17 -06:00
Nabin Mulepati	7a05a83157	docs: center diagram images in native model client dev note (#483) ... Wrap non-hero images in text-align:center divs so they display centered on wide monitors instead of left-aligned. Made-with: Cursor	2026-03-31 16:05:04 -06:00
Nabin Mulepati	a1eb244321	docs: add native model client dev note (#465) ... * add images * re-ran slopguard * update dev notes * address greptile comments * update example model name * add info on throttlemanager * address pr feedback * Add link to model aliases * address pr feedback * update key resources * update key resources * crop image for better fit * Fix max_parallel_requests * refine concluding paragraph	2026-03-31 15:45:56 -06:00
Johnny Greco	0d10bf8dc6	feat: add fr_FR locale to nemotron personas datasets (#468) ... * feat: add fr_FR locale to nemotron personas datasets Register the France locale (fr_FR, 2.71 GB) in NEMOTRON_PERSONAS_DATASET_SIZES and add 7 France-specific PII fields: first_name_heritage, name_heritage, is_first_gen_immigrant, household_type, monthly_income_eur, commune, departement. * fix: update download controller and service tests for fr_FR locale Update hardcoded locale counts from 7 to 8 and add fr_FR assertions in download controller and download service tests. * fix: generate CLI locale help text dynamically from constants The --locale help text was hardcoded and already stale (missing en_SG, pt_BR, fr_FR). Build it from LOCALES_WITH_MANAGED_DATASETS so it stays in sync automatically. * refactor: add LOCALES_WITH_MANAGED_DATASETS_STR constant Centralise the comma-joined locale list so it is defined once in constants and reused in the CLI help text, PersonSamplerParams field description, and locale validation error message.	2026-03-31 17:28:03 -04:00
Eric W. Tramel	116184b5e6	docs: consolidated seed reader documentation for filesystem and agent rollout sources (#481) ... Add comprehensive documentation for DirectorySeedSource, FileContentsSeedSource, and AgentRolloutSeedSource to the seed datasets concept page. Add FileSystemSeedReader plugin authoring guide and Markdown section seed reader recipe. Supersedes #425 and #452. Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-03-31 13:31:42 -04:00
Johnny Greco	e4bc401d93	docs: update dev note with TL;DR tips and install instructions (#461) ... * docs: add TL;DR tips to dev note and update install instructions Add a tip block with four key lessons for building agent skills to the "Data Designer Got Skills" dev note. Remove the Claude Code marketplace install option from both the blog and README, keeping only the skills.sh method. Update skill mode descriptions and clarify Claude Code testing scope. * docs: add sentence about CLI delivering curated context * docs: fix stray asterisk in README install instructions * docs: remove claude-plugin marketplace directory	2026-03-25 12:14:03 -04:00
Johnny Greco	c25a708a4b	fix: Claude Code marketplace plugin structure and install docs (#458)	2026-03-25 08:22:54 -04:00
Johnny Greco	0a7b9e0d6d	docs: Data Designer Got Skills dev note (#457) ... * docs: add skeleton for "Data Designer Got Skills" dev note * create assets folder and add blog directory name * docs: add Claude Code plugin marketplace configuration Register the repo as a Claude Code plugin marketplace so users can install the data-designer skill via `/plugin marketplace add`. * docs: write first draft of "Data Designer Got Skills" dev note Full prose for all sections: intro with hero benchmark figure, agents as first-class users, baseline trace walkthrough, CLI and skill design, benchmark results (228 sessions), getting started with marketplace and npx install paths, and what's next. * docs: add error breakdown table and minor refinements * docs: add sdg and data-designer keywords to plugin metadata * docs: refine CLI framing, reduce em dashes, slop guard pass * docs: fix grammar in dev note (serial comma, double-which clause) * update hero image * docs: swap hero image, move benchmark figure, minor wording tweaks * docs: add narrative lead-in to skill trace summary * docs: refine quality bullet, streamline getting started modes * remove old image * slope-guard tweaks	2026-03-24 21:03:00 -04:00
Nabin Mulepati	1356408c6e	feat: remove litellm dependency and bridge path (#455) ... * adjust plan * feat: remove litellm dependency and bridge path (PR-7) - Delete litellm_bridge.py adapter, litellm_overrides.py, and their tests - Remove LiteLLM fallback branch and DATA_DESIGNER_MODEL_BACKEND env var from clients/factory.py; unknown provider_type now raises ValueError - Remove apply_litellm_patches() call from models/factory.py - Remove LiteLLM exception match arms and DownstreamLLMExceptionMessageParser from models/errors.py; port context window detail extraction to _extract_context_window_detail for native ProviderError path - Remove litellm from lazy_heavy_imports.py and pyproject.toml runtime deps - Remove flatten_extra_body parameter from TransportKwargs.from_request - Clean up LiteLLM references in docstrings, comments, and AGENTS.md - Add full ProviderErrorKind test coverage to test_model_errors.py - Update benchmark script to patch OpenAICompatibleClient instead of CustomRouter Made-with: Cursor * fix: forward tools to _fake_response in benchmark patch The old CustomRouter patch forwarded **kwargs (including tools) to _fake_response, but the new OpenAICompatibleClient patch only passed model and messages — silently disabling tool-call simulation in benchmark scenarios that exercise allow_tools. Made-with: Cursor * fix: address PR-7 review feedback - Return ChatCompletionResponse from benchmark fakes instead of FakeResponse to match the native client contract (facade expects .message, not .choices[0].message) - Add ids= to parametrize block in test_model_errors.py for readability - Remove unnecessary try/except from _extract_context_window_detail; the `if marker in` guard is sufficient - Make context window marker match case-insensitive - Replace stale httpx.AsyncClient callout in async_concurrency.py docstring with generic "async-stateful resources" Made-with: Cursor	2026-03-24 11:41:00 -06:00
Kirit Thadaka	7abdde99cb	docs: Updated telemetry (#451) ... * Updated telemetry * Updated Badge	2026-03-23 08:30:37 -07:00
Eric W. Tramel	a0fb04ee07	feat: agent rollout trace ingestion (#399)	2026-03-20 09:17:35 -04:00
mvansegbroeck	02744d152d	docs: add Open in Colab badges to tutorial notebooks (#391) ... * Add Open in Colab badges to tutorial notebooks * docs: inject Colab badge via generate_colab_notebooks.py Move the "Open in Colab" badge injection into the notebook generation script so badges survive regeneration. The badge is prepended as the first markdown cell in process_notebook. * autogenerate notebooks --------- Co-authored-by: Andre Manoel <amanoel@nvidia.com> Co-authored-by: Johnny Greco <jogreco@nvidia.com>	2026-03-13 17:47:10 -04:00
Nabin Mulepati	313f347363	chore: simplify tutorial 4 image dataset and use default model config (#403) ... * chore: simplify tutorial 4 image dataset and use default model config Switch from the large ColPali dataset (52 GB) to rokmr/pets (~23 MB) for faster downloads in the vision tutorial. Use the default nvidia-vision model alias instead of a custom ModelConfig block. * regen colab notebooks	2026-03-13 12:26:41 -06:00
dhruvnathawani	0e868c9dfc	fix: correct broken dev note links in recipe pages (#407) ... Recipe pages are 3 levels deep (recipes/category/recipe/), so links to dev notes need ../../../devnotes/ not ../../devnotes/. The extra level was missing, causing links to resolve to recipes/devnotes/ instead of devnotes/.	2026-03-12 18:52:35 -07:00
dhruvnathawani	eac63a1136	docs: search agent dev note (#350)	2026-03-12 11:43:39 -07:00
dhruvnathawani	7de879acfa	docs: add Enterprise Text-to-SQL and Search Agent recipes (#395) ... feat: add Nemotron Super Text-to-SQL and Search Agent recipes Add two new recipes derived from the Nemotron Super post-training pipelines: Nemotron Super Text-to-SQL: - Five-stage pipeline: seeding, prompt generation, schema with distractors, dialect-specific SQL, validation + quality scoring - 14 conditional samplers (10 industries, 50 topics, complexity-gated task types, data quality concepts, knowledge dependencies, 100 style combos) - Dialect-specific prompts for SQLite, MySQL, and PostgreSQL - 5 LLM judges (prompt, SQL, context, data quality, knowledge) with 15 scoring dimensions and flat score extraction columns - Per-dialect syntax validation via CodeValidatorParams Nemotron Super Search Agent: - Four-stage pipeline: Wikidata KG seed paths, two-stage riddle generation (draft + BrowseComp-style obfuscation), Tavily web search trajectories via MCP, structured JSON formatting - Tavily hosted MCP endpoint (streamable_http) -- no local server or extra dependencies beyond data-designer - Full tool-call trace capture (with_trace=ALL_MESSAGES) for SFT data - Built-in demo seeds (3 Wikidata paths) for quick testing Both recipes include ASCII pipeline diagrams, Nemotron Super context in docstrings, dev note links in the markdown pages, and follow existing recipe conventions (PEP 723 metadata, --model-alias/--num-records/ --artifact-path CLI args).	2026-03-11 11:19:58 -07:00
Andre Manoel	2564834a47	fix: cache notebook builds to avoid flaky upstream model failures (#370) ... * fix: cache notebook builds to avoid failures from flaky upstream models The build-notebooks CI executes all tutorial notebooks on every run. When an upstream model (e.g. black-forest-labs/flux.2-pro) is down, the entire docs build fails even if no notebooks changed. Add per-notebook caching based on source file SHA-256 hashes. Unchanged notebooks are served from cache, and only modified ones are re-executed. On the first CI run (empty cache), the workflow seeds the cache from the last successful build artifact. Also add a minimal test script (test_flux_image_gen.py) to reproduce the flux.2-pro health check failure locally. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: address review comments on notebook caching - Don't write .sha256 during seeding so changed notebooks are detected - Rename TMPDIR to SEED_TMPDIR to avoid shadowing the POSIX env var - Use portable sha256 helper (sha256sum with shasum fallback) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: only seed cache when truly empty, restore hash writing Skip artifact seeding when a partial cache was restored (it already has correct per-file hashes). Only seed + write current hashes when the cache dir is completely empty (true bootstrapping). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: restrict artifact seed lookup to main branch Prevents seeding from feature branch runs that may have different notebook sources. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: add actions:read permission for artifact seeding The seed step uses gh run list and gh run download which require actions:read. Without it, these calls silently fail and the cold-start cache bootstrapping never executes. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: only use notebook cache when called from build-docs Scheduled Monday runs and manual workflow_dispatch should execute all notebooks to catch regressions (e.g. library changes that break a notebook). Caching is only used via workflow_call (from build-docs) where the goal is fast, resilient doc deployment. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: use jq // empty to avoid "null" string on empty run list Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: add use_cache input flag to notebook and docs workflows Replace event_name-based cache logic with an explicit use_cache boolean input. Defaults: - build-notebooks: workflow_call=true, dispatch=false, schedule=false - build-docs: dispatch=true (toggleable), release=false This gives full control over caching from the GitHub Actions UI. --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-03-05 12:30:14 -03:00
Johnny Greco	be91adc2da	docs: rename structured outputs dev note for Nemotron (#368) ... Align the dev note path/nav with the Nemotron-specific title and add a blog excerpt marker for cleaner post previews.	2026-03-04 15:30:02 -05:00
Nabin Mulepati	e4857f62fa	feat: add Streamable HTTP transport support for remote MCP providers (#358) ... * feat: add Streamable HTTP transport support for remote MCP providers (#357) Add `streamable_http` as a supported transport type for `MCPProvider`, enabling connections to MCP servers that use the Streamable HTTP protocol (e.g. Tavily remote endpoints). Previously only SSE transport was supported, causing silent 5-minute timeouts when connecting to incompatible endpoints. - Expand `MCPProvider.provider_type` to `Literal["sse", "streamable_http"]` (default remains `"sse"` for backwards compatibility) - Route `streamable_http` providers through `streamablehttp_client` from the MCP SDK in `MCPIOService._get_or_create_session()` - Handle variable-length context manager results from MCP transport clients - Add `DataDesigner.list_mcp_tool_names()` for discovering available tools - Update CLI form builder and controller to support the new transport option - Add tests for streamable_http config, session creation, and form builder Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * updates * simplify import * address greptile comments --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-04 08:11:54 -07:00