* chore: rename ColumnWiseDatasetBuilder, wire async preview, unify row-group lifecycle
- Rename ColumnWiseDatasetBuilder to DatasetBuilder and
column_wise_builder.py to dataset_builder.py, update all references
- Extract _prepare_async_run() factory shared by build and preview paths
- Add _build_async_preview() for async preview with no disk checkpoints
- Replace on_row_group_complete/on_checkpoint_complete with single
on_finalize_row_group callback; caller handles checkpointing
- Add free_row_group() on RowGroupBufferManager for discard-without-write
- Free fully-dropped row groups instead of finalizing them
- Add consolidated AsyncProgressReporter for async generation logging
Closes#437, closes#442, closes#444
* feat: add consolidated async progress reporting with row group context
- Add AsyncProgressReporter: groups per-column progress into a single
log block emitted at configurable intervals (default 5s)
- Add quiet mode to ProgressTracker to suppress per-tracker logging
when used with the consolidated reporter
- Add ContextVar-based row group tagging (RG1, RG2, ...) for log
messages emitted inside async tasks (samplers, expressions, seeds)
- Add progress_interval to RunConfig for user-configurable reporting
- Remove log_start_async from ProgressTracker (superseded by reporter)
Closes#443
* fix async preview and progress reporting
* feat: add opt-in sticky ANSI progress bars for generation
Add RunConfig.progress_bar setting that replaces periodic log-line
progress with sticky terminal bars that stay at the bottom while
logs scroll above. Pure ANSI escape codes, no new dependencies.
Disabled by default - existing log-based output unchanged.
* docs: add missing progress_interval docstring in RunConfig
* fix: update progress bar on every completion in async path
Skip the time gate when the progress bar is active so the bar
redraws on every record instead of every progress_interval seconds.
* fix: resolve row-group semaphore deadlock when all tasks are deferred
When all tasks for admitted row groups fail with transient errors,
the row-group semaphore never releases, blocking admission of new
row groups. Fix by salvaging stalled row groups inline - retrying
deferred tasks immediately so row groups can checkpoint and free
their semaphore slots.
Also updates row group log format to (x/X) with leading zeros.
* fix: eagerly salvage stalled row groups to avoid wasting semaphore slots
Run inline salvage after every checkpoint pass instead of only when
globally stalled. Row groups with 0 in-flight and only deferred tasks
are salvaged immediately, freeing their semaphore slot for new work.
* fix: address review findings from greptile and codex
- Use `with` statement for progress bar context (safe __exit__ on error)
- Check bar.is_active instead of bar is not None (non-TTY fallback)
- Record failures (not skips) for tasks that exhaust salvage retries
- Record skipped tasks when pre-batch filtering drops rows
* fix: stable progress bar width and accurate failure counts
- Pre-compute fixed stats width at bar creation to prevent bar
resizing when failed count appears
- Cap displayed completed at total to avoid >100% on retries
- Exclude already-failed columns from skip recording to prevent
double-counting in progress reporter
* fix: address Nabin's review - exclude_columns, dead code, docstring
- Add exclude_columns={task.column} on non-retryable batch/from_scratch
drop path to prevent double-counting (same pattern as cell path)
- Simplify salvage drop to per-task exclude (is_dropped guard handles
multi-column case)
- Remove dead _in_flight_for_rg method
- Fix context.py docstring to match actual (x/X) format
* fix: _drain_frontier exits before dispatching ready salvage tasks
After salvage discards a cell task from dispatched (making it
available in the frontier), _drain_frontier broke immediately
because nothing was in-flight yet. The task and its downstream
were never re-dispatched, leaving the row group incomplete.
Fix: only break when both ready and in-flight are empty.
* fix: salvage edge cases found by code review
- _drain_frontier: only break when both ready and in-flight are empty
- _salvage_rounds: re-mark sibling columns as dispatched after
from_scratch retry to prevent duplicate dispatch
- _salvage_stalled_row_groups: separate exhausted tasks from new
drain failures to avoid treating non-stalled tasks as permanent
- _checkpoint_completed_row_groups: clean up deferred tasks for
checkpointed row groups
- Early shutdown: salvage stalled row groups before exiting
* fix: skip record_failure for already-dropped rows in salvage
When multiple columns fail for the same row, the first drop records
a skip for the other column. Without this guard, record_failure
fires again for the second column, double-counting it.
* adjust plan
* feat: remove litellm dependency and bridge path (PR-7)
- Delete litellm_bridge.py adapter, litellm_overrides.py, and their tests
- Remove LiteLLM fallback branch and DATA_DESIGNER_MODEL_BACKEND env var
from clients/factory.py; unknown provider_type now raises ValueError
- Remove apply_litellm_patches() call from models/factory.py
- Remove LiteLLM exception match arms and DownstreamLLMExceptionMessageParser
from models/errors.py; port context window detail extraction to
_extract_context_window_detail for native ProviderError path
- Remove litellm from lazy_heavy_imports.py and pyproject.toml runtime deps
- Remove flatten_extra_body parameter from TransportKwargs.from_request
- Clean up LiteLLM references in docstrings, comments, and AGENTS.md
- Add full ProviderErrorKind test coverage to test_model_errors.py
- Update benchmark script to patch OpenAICompatibleClient instead of
CustomRouter
Made-with: Cursor
* fix: forward tools to _fake_response in benchmark patch
The old CustomRouter patch forwarded **kwargs (including tools) to
_fake_response, but the new OpenAICompatibleClient patch only passed
model and messages — silently disabling tool-call simulation in
benchmark scenarios that exercise allow_tools.
Made-with: Cursor
* fix: address PR-7 review feedback
- Return ChatCompletionResponse from benchmark fakes instead of
FakeResponse to match the native client contract (facade expects
.message, not .choices[0].message)
- Add ids= to parametrize block in test_model_errors.py for readability
- Remove unnecessary try/except from _extract_context_window_detail;
the `if marker in` guard is sufficient
- Make context window marker match case-insensitive
- Replace stale httpx.AsyncClient callout in async_concurrency.py
docstring with generic "async-stateful resources"
Made-with: Cursor
* perf: defer heavy imports to improve CLI startup time
Move expensive imports (engine, models, controllers) out of the module-level import path so that data-designer --help and other non-generation commands no longer pay the full startup cost.
Key changes:
- Defer controller imports to inside command functions
- Remove eager re-export chains from CLI package __init__ files
- Move default-settings bootstrap into load_config_builder() and DataDesigner.__init__() instead of running at import time
- Add lazy __getattr__ exports in interface/__init__.py
- Replace module-level tokenizer init with cached lazy getter
- Fix ModelProvider import to use config layer instead of engine
- Update test mock paths to match new import locations
Reduces CLI import-time from ~1.67s to ~0.46s.
* perf: defer pandas/numpy in io_helpers and add config_list benchmark
- Replace eager `from lazy_heavy_imports import pd, np` in io_helpers
with module-level __getattr__ (for backwards-compatible external
access / test mocks) and function-level imports in the 3 functions
that actually use them (read_parquet_dataset, smart_load_dataframe,
_convert_to_serializable). Importing io_helpers no longer triggers
pandas/numpy loading.
- Defer heavy imports in list and reset CLI commands into function
bodies to avoid loading repositories, Rich, and prompt_toolkit at
module import time.
- Add `config_list` (data-designer config list) measurement to the
CLI startup benchmark with isolated cold measurement in a separate
venv and a --skip-config-list-check flag.
- Update test mock paths to match new import locations.
* Refine lazy import usage and TYPE_CHECKING cleanup
* Run license header updater on PR-touched files
* fix: update sqlfluff mock target for lazy imports in test_sql
* perf: cache globals() in lazy __getattr__ to avoid repeated lookups
Add globals() caching and explanatory comment to all three lazy
__getattr__ implementations (lazy_heavy_imports, config/__init__,
interface/__init__) so subsequent attribute accesses bypass __getattr__.
* perf: lazy CLI command loading and deferred heavy import evaluations
- Add LazyTyperGroup to defer command module loading until invocation, allowing module-level imports in all CLI command files
- Split DataFrameSeedSource into seed_source_dataframe.py to isolate pandas dependency from other seed source classes
- Move TypeVar/TypeAlias definitions (DataT, NumpyArray1dT, RadomStateT, EngineT) to TYPE_CHECKING blocks with runtime fallbacks
- Wrap module-level constants in lru_cache (phone_number parquet data, jsonschema validator) to defer I/O and heavy imports to first use
- Update test mock targets to patch at usage-site for module-level imports
* refactor: use direct pandas import in seed_source_dataframe
Drop lazy-loading for pandas in DataFrameSeedSource; use direct import
for simplicity.
* update lazy import pattern
* update tests to use lazy import namespace
Switch test modules to import data_designer.lazy_heavy_imports as lazy
and reference heavy libraries through that namespace. This keeps heavy
imports deferred during module import and aligns tests with the new
lazy-import usage pattern.
* tighten import perf test thresholds
Document recent baseline timings and lower the allowed average
import time and timeout so regressions are detected sooner.
* document pandas import requirement
Clarify that Pydantic needs DataFrame resolved at module load and
that keeping the direct import preserves IDE typing support.
* increase timeout time
* use lazy pandas imports in visualization tests
- replace direct pandas usage with lazy.pd in visualization tests to avoid eager imports
- add TYPE_CHECKING pandas import and keep CLI controller imports sorted
* fix lazy pandas runtime usage and preview mocks
Switch sample-record handling to lazy pandas types so runtime paths no longer
depend on TYPE_CHECKING imports. Align preview controller tests to patch the
module-local DataDesigner symbol, preventing real engine invocation in save
results scenarios.
