mirror of
https://github.com/NVIDIA-NeMo/DataDesigner
synced 2026-05-24 09:48:29 +00:00
c64b0a5707
* fix: processor artifacts type annotation, discovery, and loading - Fix PreviewResults.processor_artifacts type from dict[str, list[str] | str] to dict[str, list[dict]], matching the actual data produced by df.to_dict(orient="records") - Add ArtifactStorage.load_processor_dataset() to centralize loading logic, handling both directory-based (batched) and single-file (preview) layouts - Add ArtifactStorage.list_processor_names() to discover processor outputs from disk rather than iterating config, fixing a bug where processors that don't write artifacts (e.g. DropColumnsProcessor) would crash the library's preview - Simplify DatasetCreationResults.load_processor_dataset() to delegate to ArtifactStorage * fix: deduplicate list_processor_names when both dir and file exist * refactor: extract standalone functions for processor discovery and loading Extract list_processor_names() and load_processor_dataset() as module-level functions that accept a Path, so consumers can use them without constructing an ArtifactStorage. The ArtifactStorage methods now delegate to these. * refactor: move standalone functions to io_helpers in config package Move list_processor_names() and load_processor_dataset() to data_designer.config.utils.io_helpers so they're usable by any consumer that depends on the lightweight config package, without needing the engine's ArtifactStorage. The standalone functions raise FileNotFoundError; the ArtifactStorage methods wrap this as ArtifactStorageError. * chore: trim docstrings and deduplicate tests * fix: align get_processor_file_paths with list_processor_names Reuse list_processor_names so both methods discover directories and single parquet files consistently. |
||
|---|---|---|
| .. | ||
| __init__.py | ||
| test_artifact_storage.py | ||
| test_media_storage.py | ||