* fix(engine): actionable error when a Jinja field is missing/None/empty
Empty-render and missing-attribute failures used to surface as the
generic "User provided prompt generation template is invalid." either
because `sanitize_user_exceptions` stripped the detail or because
Jinja's raw `UndefinedError` leaked through. Both now raise a new
`EmptyTemplateRenderError` carrying a row-level diagnostic that names
the offending chain and includes copy-pasteable Jinja conditional and
SkipConfig fix patterns.
Closes#629.
* fix(engine): address PR review feedback on EmptyTemplateRenderError
Addresses the open review comments on #633:
1. (Greptile P1) Gate expression in the suggested remediation template
was one accessor too deep when the root variable was entirely absent
from the record, causing the suggested fix to itself raise
UndefinedError. Fall back to gating on the root name alone when
sample_name is not in record.
2. (andreatgretel) The AST walker reported loop-local names as missing
culprits (e.g. ``person`` in ``{% for person in people %}...{% endfor %}``).
Filter extracted chains through ``meta.find_undeclared_variables`` to
defer to Jinja's canonical scope tracking.
3. (andreatgretel follow-up) Empty collections used as loop iterables
(``items=[]``) fell through to the no-culprit fallback. Add a new
``_CULPRIT_EMPTY_COLLECTION`` classification so they're surfaced.
4. Minor: add ``from exception`` to ``safe_render``'s UndefinedError
re-raise for traceback consistency with the native engine path, and
add a note on the load-bearing exception ordering in
``sanitize_user_exceptions``.
* feat: support nested field access in schema transform templates
Enable {{ result.quality.score }} style dot notation in schema
transform Jinja2 templates, where result is a deserialized JSON column.
Previously, _json_escape_record flattened all dict values to escaped
JSON strings before Jinja2 saw them. This made the rendered output
valid JSON but prevented nested access since Jinja2 only saw strings.
The fix introduces TemplateValue, a wrapper that defers the choice
between "drill into nested dict" and "render as escaped string" to
template evaluation time. Jinja2 resolves dot notation via __getattr__
(returning a new TemplateValue for the nested value), and converts to
string via __str__ (delegating to a caller-provided str_fn). This is
necessary because plain dicts render as Python repr ({'key': 'val'})
which is invalid JSON - we need to control __str__ to produce properly
escaped JSON, and that requires a wrapper object.
Other Jinja2 consumers (prompt templates, expression columns) don't
need this - Jinja2 natively supports dot access on plain dicts via
getattr-to-getitem fallback, and plain str() is fine for text output.
Schema transform is unique because its output must be valid JSON.
* Address PR review comments
- Fix boolean serialization: add bool check before str in _escape_value_for_json
to produce JSON 'true'/'false' instead of Python 'True'/'False'
- Add class-level _record_str_fn annotation to WithJinja2UserTemplateRendering
- Rename skip_record_sanitization to _skip_record_sanitization (underscore prefix)
to signal internal-only usage, and document it in safe_render docstring
- Add defensive error handling in TemplateValue.__getitem__ and __iter__
- Promote test input data to parametrize column, removing brittle string scan
* Address second round of PR review comments
- Add __eq__ and __hash__ to TemplateValue so Jinja2 equality
conditionals (e.g. {% if result.label == "excellent" %}) work
- Add inline comment explaining deliberate double-encode in
_escape_value_for_json for dict/list values
- Default _record_str_fn to None at class level so accessing it
before prepare_jinja2_template_renderer doesn't mask the real error
* refactor: replace TemplateValue with Jinja2 finalize hook
Use Jinja2's built-in finalize hook for value-to-string conversion
and a getattr override for dict-key-priority lookup, eliminating the
custom TemplateValue wrapper class entirely.
* docs: add missing param docs to prepare_jinja2_template_renderer
* fix: handle discriminated unions in oneOf pruning validator
The pruning validator modifies instances in-place during oneOf
validation. When trying a wrong variant, it strips properties needed
by the correct variant, causing all variants to fail.
Add a discriminator-aware oneOf validator that reads the discriminator
mapping to select the correct variant directly, skipping the
try-all-variants loop that causes the corruption.
Fixes#375
* test: add regression test for non-discriminated oneOf fallback
* feat: add processor plugin support
Add PluginType.PROCESSOR to the plugin system, enabling third-party
processor plugins via entry points. Includes a demo plugin package
with RegexFilterProcessor (process_before_batch) and
SemanticDedupProcessor (process_after_generation).
- Add PluginType.PROCESSOR with processor_type discriminator
- Create processor_types.py for ProcessorConfigT with plugin injection
- Register plugin processors in engine ProcessorRegistry
- Use RLock in PluginRegistry to prevent deadlocks during discovery
- Add demo package: data-designer-demo-processors
- Update processor and plugin documentation
* test: add processor plugin registration test
Verify that processor plugins from PluginRegistry are picked up
by create_default_processor_registry and registered correctly.
* test: simplify processor plugin registration test
* move ProcessorConfig to base and convert demo to e2e test
- Move ProcessorConfig from processors.py to config.base to guard
against circular deps (alongside SingleColumnConfig)
- Delete demo/ directory with regex_filter and semantic_dedup plugins
- Add regex_filter as an e2e processor plugin test in tests_e2e/
* move plan to plans/299/
* fix: make DropColumnsProcessorConfig idempotent and support reasoning columns
- add_processor now uses upsert semantics: re-adding a processor with the
same name replaces the old one and reverts its drop=True side-effects,
making notebook cells safely re-runnable.
- validate_drop_columns_processor now includes side-effect columns
(reasoning_content, trace) so reasoning columns can be dropped.
Fixes#332
* test: reduce duplication in drop-columns tests
- Use parametrize for reasoning column validation cases
- Extract _add_sampler helper to avoid repeated SamplerColumnConfig setup
- Move validate_drop_columns_processor import to top of file
* feat: support glob patterns in DropColumnsProcessorConfig column_names
Patterns like "*__reasoning_content" or "col_*" are now expanded against
available columns at validation time and at runtime. Validation emits a
warning when a glob pattern matches no columns.
* fix: preserve drop flag when column is referenced by other processors
When removing a DropColumnsProcessor, only revert drop=True on columns
that are not also dropped by another processor.
* fix: deduplicate resolved column names from overlapping glob patterns
Prevents duplicates when column_names contains both a literal and a
matching glob (e.g. ["col_a", "col_*"]), which would cause a KeyError
at runtime when dropping the same column twice.
* fix: restrict glob detection to * only
Avoids false positives from column names containing [ or ? characters.
* refactor: simplify resolve helpers and flatten test class
- Use dict-as-ordered-set pattern instead of seen + list for dedup
- Flatten TestAddProcessorIdempotent into top-level test functions
* test: use fixture and parametrize for add_processor tests
* fix: remove redundant quotes around repr in validation message
* perf: defer heavy imports to improve CLI startup time
Move expensive imports (engine, models, controllers) out of the module-level import path so that data-designer --help and other non-generation commands no longer pay the full startup cost.
Key changes:
- Defer controller imports to inside command functions
- Remove eager re-export chains from CLI package __init__ files
- Move default-settings bootstrap into load_config_builder() and DataDesigner.__init__() instead of running at import time
- Add lazy __getattr__ exports in interface/__init__.py
- Replace module-level tokenizer init with cached lazy getter
- Fix ModelProvider import to use config layer instead of engine
- Update test mock paths to match new import locations
Reduces CLI import-time from ~1.67s to ~0.46s.
* perf: defer pandas/numpy in io_helpers and add config_list benchmark
- Replace eager `from lazy_heavy_imports import pd, np` in io_helpers
with module-level __getattr__ (for backwards-compatible external
access / test mocks) and function-level imports in the 3 functions
that actually use them (read_parquet_dataset, smart_load_dataframe,
_convert_to_serializable). Importing io_helpers no longer triggers
pandas/numpy loading.
- Defer heavy imports in list and reset CLI commands into function
bodies to avoid loading repositories, Rich, and prompt_toolkit at
module import time.
- Add `config_list` (data-designer config list) measurement to the
CLI startup benchmark with isolated cold measurement in a separate
venv and a --skip-config-list-check flag.
- Update test mock paths to match new import locations.
* Refine lazy import usage and TYPE_CHECKING cleanup
* Run license header updater on PR-touched files
* fix: update sqlfluff mock target for lazy imports in test_sql
* perf: cache globals() in lazy __getattr__ to avoid repeated lookups
Add globals() caching and explanatory comment to all three lazy
__getattr__ implementations (lazy_heavy_imports, config/__init__,
interface/__init__) so subsequent attribute accesses bypass __getattr__.
* perf: lazy CLI command loading and deferred heavy import evaluations
- Add LazyTyperGroup to defer command module loading until invocation, allowing module-level imports in all CLI command files
- Split DataFrameSeedSource into seed_source_dataframe.py to isolate pandas dependency from other seed source classes
- Move TypeVar/TypeAlias definitions (DataT, NumpyArray1dT, RadomStateT, EngineT) to TYPE_CHECKING blocks with runtime fallbacks
- Wrap module-level constants in lru_cache (phone_number parquet data, jsonschema validator) to defer I/O and heavy imports to first use
- Update test mock targets to patch at usage-site for module-level imports
* refactor: use direct pandas import in seed_source_dataframe
Drop lazy-loading for pandas in DataFrameSeedSource; use direct import
for simplicity.
* update lazy import pattern
* update tests to use lazy import namespace
Switch test modules to import data_designer.lazy_heavy_imports as lazy
and reference heavy libraries through that namespace. This keeps heavy
imports deferred during module import and aligns tests with the new
lazy-import usage pattern.
* tighten import perf test thresholds
Document recent baseline timings and lower the allowed average
import time and timeout so regressions are detected sooner.
* document pandas import requirement
Clarify that Pydantic needs DataFrame resolved at module load and
that keeping the direct import preserves IDE typing support.
* increase timeout time
* use lazy pandas imports in visualization tests
- replace direct pandas usage with lazy.pd in visualization tests to avoid eager imports
- add TYPE_CHECKING pandas import and keep CLI controller imports sorted
* fix lazy pandas runtime usage and preview mocks
Switch sample-record handling to lazy pandas types so runtime paths no longer
depend on TYPE_CHECKING imports. Align preview controller tests to patch the
module-local DataDesigner symbol, preventing real engine invocation in save
results scenarios.
Fixes GitHub issue #227 where SchemaTransformProcessor fails with
JSONDecodeError when LLM-generated content contains quotes, backslashes,
newlines, or other special characters that break JSON parsing.
The fix properly escapes all string values before template rendering
using json.dumps to handle all JSON-special characters.
