* Add Open in Colab badges to tutorial notebooks
* docs: inject Colab badge via generate_colab_notebooks.py
Move the "Open in Colab" badge injection into the notebook generation
script so badges survive regeneration. The badge is prepended as the
first markdown cell in process_notebook.
* autogenerate notebooks
---------
Co-authored-by: Andre Manoel <amanoel@nvidia.com>
Co-authored-by: Johnny Greco <jogreco@nvidia.com>
* chore: simplify tutorial 4 image dataset and use default model config
Switch from the large ColPali dataset (52 GB) to rokmr/pets (~23 MB)
for faster downloads in the vision tutorial. Use the default
nvidia-vision model alias instead of a custom ModelConfig block.
* regen colab notebooks
Recipe pages are 3 levels deep (recipes/category/recipe/), so links
to dev notes need ../../../devnotes/ not ../../devnotes/. The extra
level was missing, causing links to resolve to recipes/devnotes/
instead of devnotes/.
feat: add Nemotron Super Text-to-SQL and Search Agent recipes
Add two new recipes derived from the Nemotron Super post-training pipelines:
Nemotron Super Text-to-SQL:
- Five-stage pipeline: seeding, prompt generation, schema with distractors,
dialect-specific SQL, validation + quality scoring
- 14 conditional samplers (10 industries, 50 topics, complexity-gated task
types, data quality concepts, knowledge dependencies, 100 style combos)
- Dialect-specific prompts for SQLite, MySQL, and PostgreSQL
- 5 LLM judges (prompt, SQL, context, data quality, knowledge) with 15
scoring dimensions and flat score extraction columns
- Per-dialect syntax validation via CodeValidatorParams
Nemotron Super Search Agent:
- Four-stage pipeline: Wikidata KG seed paths, two-stage riddle generation
(draft + BrowseComp-style obfuscation), Tavily web search trajectories
via MCP, structured JSON formatting
- Tavily hosted MCP endpoint (streamable_http) -- no local server or extra
dependencies beyond data-designer
- Full tool-call trace capture (with_trace=ALL_MESSAGES) for SFT data
- Built-in demo seeds (3 Wikidata paths) for quick testing
Both recipes include ASCII pipeline diagrams, Nemotron Super context in
docstrings, dev note links in the markdown pages, and follow existing
recipe conventions (PEP 723 metadata, --model-alias/--num-records/
--artifact-path CLI args).
* fix: cache notebook builds to avoid failures from flaky upstream models
The build-notebooks CI executes all tutorial notebooks on every run.
When an upstream model (e.g. black-forest-labs/flux.2-pro) is down, the
entire docs build fails even if no notebooks changed.
Add per-notebook caching based on source file SHA-256 hashes. Unchanged
notebooks are served from cache, and only modified ones are re-executed.
On the first CI run (empty cache), the workflow seeds the cache from the
last successful build artifact.
Also add a minimal test script (test_flux_image_gen.py) to reproduce the
flux.2-pro health check failure locally.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: address review comments on notebook caching
- Don't write .sha256 during seeding so changed notebooks are detected
- Rename TMPDIR to SEED_TMPDIR to avoid shadowing the POSIX env var
- Use portable sha256 helper (sha256sum with shasum fallback)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: only seed cache when truly empty, restore hash writing
Skip artifact seeding when a partial cache was restored (it already has
correct per-file hashes). Only seed + write current hashes when the
cache dir is completely empty (true bootstrapping).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: restrict artifact seed lookup to main branch
Prevents seeding from feature branch runs that may have different
notebook sources.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: add actions:read permission for artifact seeding
The seed step uses gh run list and gh run download which require
actions:read. Without it, these calls silently fail and the cold-start
cache bootstrapping never executes.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: only use notebook cache when called from build-docs
Scheduled Monday runs and manual workflow_dispatch should execute all
notebooks to catch regressions (e.g. library changes that break a
notebook). Caching is only used via workflow_call (from build-docs)
where the goal is fast, resilient doc deployment.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: use jq // empty to avoid "null" string on empty run list
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: add use_cache input flag to notebook and docs workflows
Replace event_name-based cache logic with an explicit use_cache boolean
input. Defaults:
- build-notebooks: workflow_call=true, dispatch=false, schedule=false
- build-docs: dispatch=true (toggleable), release=false
This gives full control over caching from the GitHub Actions UI.
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: add Streamable HTTP transport support for remote MCP providers (#357)
Add `streamable_http` as a supported transport type for `MCPProvider`,
enabling connections to MCP servers that use the Streamable HTTP protocol
(e.g. Tavily remote endpoints). Previously only SSE transport was supported,
causing silent 5-minute timeouts when connecting to incompatible endpoints.
- Expand `MCPProvider.provider_type` to `Literal["sse", "streamable_http"]`
(default remains `"sse"` for backwards compatibility)
- Route `streamable_http` providers through `streamablehttp_client` from
the MCP SDK in `MCPIOService._get_or_create_session()`
- Handle variable-length context manager results from MCP transport clients
- Add `DataDesigner.list_mcp_tool_names()` for discovering available tools
- Update CLI form builder and controller to support the new transport option
- Add tests for streamable_http config, session creation, and form builder
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* updates
* simplify import
* address greptile comments
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* feat: add processor plugin support
Add PluginType.PROCESSOR to the plugin system, enabling third-party
processor plugins via entry points. Includes a demo plugin package
with RegexFilterProcessor (process_before_batch) and
SemanticDedupProcessor (process_after_generation).
- Add PluginType.PROCESSOR with processor_type discriminator
- Create processor_types.py for ProcessorConfigT with plugin injection
- Register plugin processors in engine ProcessorRegistry
- Use RLock in PluginRegistry to prevent deadlocks during discovery
- Add demo package: data-designer-demo-processors
- Update processor and plugin documentation
* test: add processor plugin registration test
Verify that processor plugins from PluginRegistry are picked up
by create_default_processor_registry and registered correctly.
* test: simplify processor plugin registration test
* move ProcessorConfig to base and convert demo to e2e test
- Move ProcessorConfig from processors.py to config.base to guard
against circular deps (alongside SingleColumnConfig)
- Delete demo/ directory with regex_filter and semantic_dedup plugins
- Add regex_filter as an e2e processor plugin test in tests_e2e/
* move plan to plans/299/
* fix: repair notebook CI by replacing dead vision model and adding missing API key
- Replace `meta/llama-4-scout-17b-16e-instruct` (no longer serving on
build.nvidia.com) with `nvidia/nemotron-nano-12b-v2-vl` (project default)
in tutorial notebook 4
- Add `OPENROUTER_API_KEY` to the `build-notebooks` workflow so notebooks
5 and 6 (which use OpenRouter for image generation) can authenticate
- Regenerate colab notebooks to reflect the model change
* fix: handle pyarrow list types in notebook 6 display_image
When image columns are loaded from parquet with pyarrow backend,
list values are pyarrow ListScalars, not Python lists. The
isinstance(x, list) check fails, causing the whole ListScalar to be
treated as a single path string (producing filenames ending in
`png')]`). Use isinstance(x, str) instead to correctly handle any
iterable type.
* feat: add allow_resize for 1:N and N:1 generation patterns
Adds support for generators that produce a different number of records
than the input (expansion or retraction). This addresses GitHub issue #265.
Changes:
- Add `allow_resize` parameter to `update_records()` in DatasetBatchManager
- Add `allow_resize` field to CustomColumnConfig
- Add validation requiring FULL_COLUMN strategy when allow_resize=True
- Track and report actual_num_records in metadata (may differ from target)
- Add logging when batch size changes
- Add example_allow_resize.py demonstrating the feature
- Add comprehensive tests
* docs: add allow_resize to custom columns documentation
* refactor: consolidate buffer API and elevate allow_resize to base config
- Merge update_records and replace_buffer into a single replace_buffer
method with allow_resize parameter on DatasetBatchManager
- Move allow_resize field from CustomColumnConfig to SingleColumnConfig
so plugins inherit it without needing a mixin
- Align example and logging with final CustomColumn API
- Parametrize resize tests and extract shared stub in test_columns
* test: add chained resize and multi-batch integration tests
- Add expand->retract->expand chaining test (single batch)
- Add multi-batch resize test verifying combined parquet output
- Update example to chain expand/retract/expand with preview+build
- Use 💥/✂️ emojis for resize logging (expand/retract)
* extend allow_resize to cell-by-cell (return dict or list[dict])
- Config: allow allow_resize with CELL_BY_CELL; relax validator
- Custom generator: accept dict | list[dict] when cell_by_cell + allow_resize;
validate per row via _validate_cell_output
- Builder: collect results by index when cell allow_resize, flatten and
replace_buffer; add _log_resize_if_changed and _column_display_name
- Docs: ALL_CAPS for strategies, simplify allow_resize table text
- Tests: parametrized preview and multibatch; factories with n param;
_RESIZE_SPECS with inline factory calls; ids ordered like specs
* reorder allow_resize specs and add edge-case tests
- Rename specs: full_x3, cell_x2, cell_plus_full_chain; add cell_filter_odd,
cell_drop_all to _RESIZE_SPECS
- Stubs before specs: _resize_full_keep_first, _resize_cell_expand,
_resize_cell_filter_odd, _resize_cell_drop_all; drop cell factories
- Remove FULL/CELL constants; use GenerationStrategy.* in _RESIZE_SPECS
- Preview/multibatch parametrize: _preview and _multibatch ids; two full_x3
multibatch cases (5_2, 4_2) first
- Handle all-batches-skipped in multibatch test (empty df when path missing)
- test_custom: add test_cell_by_cell_allow_resize_return_list_single (1:1 via list)
* tidy allow_resize: drop validator, shared stub, explicit flag
- Remove validate_allow_resize_requires_full_column from CustomColumnConfig
- Rename StubColumnConfigWithoutEmoji to StubColumnConfig in test_columns
- Pass allow_resize=False in _write_processed_batch replace_buffer call
* fix: add missing f prefix to error message in custom.py
* docs(plugins): add section on setting allow_resize=True for resize plugins
* fix: address PR review comments on allow_resize
- Replace getattr with direct attribute access where config is always
SingleColumnConfig (custom.py, cell-by-cell path in builder)
- Keep getattr in _run_full_column_generator which also handles
multi-column configs without allow_resize
- Restructure allow_resize validation branching in CustomColumnGenerator
- Fix error message wording: "key" -> "column"
* fix: remove duplicate tool_alias log, fix test docstring
- Remove tool_alias log from _setup_fan_out (callers already log it)
- Fix docstring: CELL_BY_CELL -> FULL_COLUMN in resize test factory
* fix: avoid duplicate undeclared-column warning in _validate_output
Inline the strip instead of delegating to _validate_cell_output,
which would log the same warning a second time.
* fix: use lazy.pd instead of pd for runtime pandas usage in tests
The pd import is under TYPE_CHECKING, so runtime calls need lazy.pd.
* docs: restructure plugin docs with multi-file layout and seed reader type
- Update plugin overview to document both column generator and seed
reader plugin types
- Restructure example plugin to use separate config.py, impl.py, and
plugin.py files instead of a single-file approach
- Add sections for plugin validation and multiple plugins per package
- Document required config class methods (get_column_emoji,
required_columns, side_effect_columns)
* docs: clarify benefits of multi-file plugin structure
Expand explanation to mention circular dependency prevention
as a key reason for separating config, impl, and plugin modules.
* docs: fix import ordering in plugin example
* import spacing
* better example column name
* add a bit to the comment
* Updated plugin docs
* update plugin overview call-to-action wording
---------
Co-authored-by: Kirit93 <kthadaka@nvidia.com>
* add tool usage statistics tracking
- Add ToolUsageStats class with metrics for tool calls, turns, and
statistical aggregates (mean/stddev per generation)
- Extend ModelUsageStats to include tool_usage tracking
- Update ModelFacade.generate() to track total tool calls and turns
- Update tests with tool_call_count method and new assertions
* silence noisy mcp logs
* log message updates
* add tools enabled info message
* exclude empty tool_usage from usage stats output
* add tool usage summary logging after column generation
- Track tool usage snapshots before/after column processing
- Log mean tool calls per generation for columns with tools enabled
- Add get_tool_usage_snapshot/get_tool_usage_delta methods to ModelRegistry
- Remove unused extra_info parameter from progress_tracker.log_start()
- Add comprehensive tests for ToolUsageStats
* pretty format model usage logs
* reuse stubs and fixtures
* add merge method to ToolUsageStats for accurate stats aggregation
The previous implementation used extend() to combine tool usage stats,
but extend() is designed for single generation data. This caused
incorrect stddev calculations when merging stats from multiple sources.
- Add ToolUsageStats.merge() that properly combines sum-of-squares
- Update ModelUsageStats.extend() to use merge() for tool usage
- Add tests verifying stddev accuracy after merging
* fix tool usage stats missing generations_with_tools count
When tracking tool usage after generation, the ToolUsageStats was
created without setting generations_with_tools, causing the usage
summary to report zeros for calls/gen and turns/gen metrics.
* fix tool usage delta objects returning incorrect stddev values
- Simplify facade API to use tool_usage.extend() directly
- Return NaN for stddev when sum of squares wasn't tracked
- Add docstring to get_tool_usage_delta explaining NaN behavior
- Add comprehensive tests for stddev variance calculation
* fix tool usage delta stddev by including sum of squares in deltas
Convert sum_of_squares_turns and sum_of_squares_calls from private
attributes to public fields, enabling them to be included in delta
calculations. This allows get_tool_usage_delta to return objects that
compute accurate stddev values instead of NaN.
* fix test to use get_tool_usage_snapshot for accurate stddev tracking
The test was manually constructing a ToolUsageStats snapshot without
sum_of_squares fields, causing stddev to be NaN. Now uses the proper
snapshot method that includes all fields needed for delta calculations.
* use nvidia-reasoning by default
* mean -> average in log message
* refactor log indentation to use centralized LOG_INDENT constant
- Add LOG_INDENT constant to logging.py for consistent indentation
- Replace hardcoded " |-- " strings across all log statements
- Add tool alias and MCP provider info to pre-generation logs
- Improve model usage log format for better consistency
- Update tests to match new log formats
* simplify usage stats dict access in model registry
Remove defensive .get() calls and unnecessary type casts since
the usage statistics dictionary structure is now guaranteed.
* walrus baby
* simplify tool usage tracking and reduce log verbosity
- Remove mean/stddev calculations from ToolUsageStats in favor of simple
counts and generation ratios
- Add total_generations field to track all tool-enabled generations
- Simplify registry log output to show generations ratio (with_tools/total)
- Remove per-column tool usage snapshot/delta logging from column builder
- Track tool usage for all tool-enabled generations, not just those with calls
* format inference parameters as multi-line log output
- Add get_formatted_params() method to BaseInferenceParams
- Add LOG_DOUBLE_INDENT constant for nested indentation
- Update log_pre_generation() to display each parameter on its own line
* update tests to use LOG_INDENT constants
Align test assertions with the centralized log indentation
constants introduced in the logging module refactor.
* two-space consistency
* Add RQA dataset blog post and improve blog navigation
- Add new blog post about RQA (Reasoning Question-Answer) dataset
- Add excerpt separator for blog index blurbs
- Configure left nav to show individual blog posts
- Add navigation.indexes feature for better section handling
- Update authors.yml with new contributors
* Update avatar.
* Update Eric avatar.
* Fix formatting.
* Fix formatting.
---------
Co-authored-by: Dane Corneil <dane.corneil@gretel.ai>
Co-authored-by: Eric W. Tramel <eric.tramel@gmail.com>
* first attempt
* iterating a bit
* some improvements + multiturn example
* adapting to new monorepo structure
* refining
* fixed test
* fixing license headers
* adding docs
* adding test for failed generation
* allowing strategy to be picked
* renaming argument
* lint
* remove recommendation
* renaming for consistency
* addressing comments pt1
* addressing comments pt2
* addressing comments pt3
* adding a mock for development
* addressing greptile comments
* revamping
* docs: streamline custom columns documentation
* docs: simplify CustomColumnConfig docstring
Remove verbose code example and detailed function signatures from
docstring to match the pattern of other config classes in the file.
* test: clean up custom column tests
- Remove tests for private _custom_column_metadata attribute
- Combine redundant generator creation tests
- Reuse stub_resource_provider and stub_model_facade fixtures
* test: consolidate custom column tests
Reduce from 26 to 11 tests while maintaining coverage:
- Combine redundant config/decorator/creation tests
- Use parametrized tests for error conditions
- Remove duplicate validation tests for full_column strategy
- Simplify section headers
* refactor: deduplicate CustomColumnGenerator logic
Merge cell-by-cell and full-column code paths:
- _generate_cell_by_cell + _generate_full_column -> _generate
- _validate_output_columns + _validate_output_columns_df -> _validate_output
* chore: merge example files into single notebook-style example.py
Combine example.py, example_multiturn.py, and example_benchmark_strategies.py
into a single file with #%% cell markers for Jupyter/VS Code notebook mode.
* addressing greptile comments
* refactor: reuse generate_text in generate_text_batch
* refactor: replace CustomColumnContext with models dict
- Remove CustomColumnContext class; users now receive models dict directly
- Add DataDesigner.get_models() for experimentation outside pipeline
- Make parser optional in ModelFacade.generate() (defaults to identity)
- Validate parameter names: row/df, generator_params, models
- Update examples, tests, and docs for new API
* fix: address PR review comments from Nabin and greptile
- Make decorator metadata public (custom_column_metadata)
- Simplify get_generation_strategy() to directly return config value
- Use !r formatting in error messages
- Use lazy imports pattern for pandas (TYPE_CHECKING + lazy_heavy_imports)
- Remove redundant error logging before re-raise
- Validate max 3 positional parameters
- Use GenerationStrategy enum in example instead of string
* fix: replace lambda with module-level identity function in facade
Use pickleable _identity function instead of lambda x: x for the
default parser argument, ensuring compatibility with multiprocessing.
* fix: restore inherited attributes in LLM column docstrings
Restores the "Inherited Attributes" sections that were unintentionally
removed from LLMCodeColumnConfig, LLMStructuredColumnConfig, and
LLMJudgeColumnConfig docstrings.
* docs: clarify model_aliases is required for LLM access
Updated documentation and docstrings to clarify that model_aliases
populates the models dict (not just health checks).
* fix: address PR review comments from nabinchha
- clarify model_aliases requirement in docs
- add note about model alias validation during health check
- combine two loops into one in _run_model_health_check_if_needed
- add signature validation at decoration time
- enforce decorated functions in CustomColumnConfig validator
- simplify generator to only validate strategy-specific first param
* fix: address remaining PR review comments
- remove example.py (development artifact)
- fix get_models return type to dict[str, ModelFacade]
* test: update tests for decoration-time validation
- expect ValidationError instead of InvalidConfigError for non-callable
- split param validation test into decoration-time and runtime tests
* create top-level base file
* add note
* update license header
* move exportable config and move base to config module
* update references in docs
* do not include single column config in init
* add inverse import order e2e test
- Convert notebook 3 from string-based columns to class specs (dd.SamplerColumnConfig, etc.)
- Fix grammar: "is the main object is responsible" → "is the main object responsible"
- Remove stray "A" at end of URL in notebook 2
- Remove empty markdown cell in notebook 4
- Add missing data_designer.validate() call in notebook 4
- Regenerate colab notebooks from source
* docs: add deployment, performance tuning guides and streamline getting started
- Add deployment-options.md: Library vs. Microservice decision guide
- Add inference-architecture.md: Separation of concerns with LLM servers
- Add performance-tuning.md: Concurrency and batching optimization guide
- Streamline index.md: Merge installation, add quick example, simplify
- Remove quick-start.md: Content merged into welcome page
- Remove installation.md: Content merged into welcome page
- Update model docs: Add concurrency control sections and cross-references
- Update mkdocs.yml: Add new Architecture section to navigation
* docs: add tasteful emojis to new documentation pages
* docs: consolidate redundant concurrency and troubleshooting content
- Remove duplicate max_parallel_requests tables from model-configs.md and inference-parameters.md
- Remove duplicate Concurrency Control section from model-configs.md
- Simplify Concurrency Control in inference-parameters.md to link to performance-tuning.md
- Remove Troubleshooting section from inference-architecture.md (covered in performance-tuning.md)
- performance-tuning.md is now the authoritative source for tuning guidance
* Simplified doc additions
* Switched default model to nemotron 3 nano
* Addressed feedback
* Added first blog draft
Add support for five high-priority programming languages to Data Designer's
code generation capabilities:
- **Bash**: Universal DevOps and automation scripting
- **C, C++, C#**: Systems programming and enterprise development
- **COBOL**: Legacy mainframe systems and modernization
These languages address critical enterprise use cases including legacy code
maintenance, systems programming, and infrastructure automation.
Changes:
- Add new CodeLang enum values for bash, c, cpp, csharp, cobol
- Update code_lang_to_syntax_lexer() with Pygments lexer mappings
- Update documentation to reflect new supported languages
- Update tests to account for 21 total supported languages (up from 16)
Co-authored-by: Johnny Greco <jogreco@nvidia.com>
Add support for capturing full conversation traces during LLM generation,
enabling debugging and fine-tuning dataset creation.
Changes:
- Add `with_trace` field to LLMTextColumnConfig for per-column trace control
- Add `debug_override_save_all_column_traces` to RunConfig for global trace
- Introduce ChatMessage dataclass for structured message representation
- Update ModelFacade.generate() to return full message trace
- Rename trace column postfix from `__reasoning_trace` to `__trace`
- Add comprehensive traces documentation
Traces capture system/user/assistant messages in order, enabling visibility
into the full generation conversation including correction retries.
* removing required resources
* fix tests
* add get required resources method to base column generator
* move classification functions to engine; remove required resources
* drop single from subclass names
* update model config logging
* fix unit test
* typo
* update type hint
* move tests
