mirror of
https://github.com/NVIDIA-NeMo/DataDesigner
synced 2026-05-24 09:48:29 +00:00
0a7b9e0d6d
* docs: add skeleton for "Data Designer Got Skills" dev note * create assets folder and add blog directory name * docs: add Claude Code plugin marketplace configuration Register the repo as a Claude Code plugin marketplace so users can install the data-designer skill via `/plugin marketplace add`. * docs: write first draft of "Data Designer Got Skills" dev note Full prose for all sections: intro with hero benchmark figure, agents as first-class users, baseline trace walkthrough, CLI and skill design, benchmark results (228 sessions), getting started with marketplace and npx install paths, and what's next. * docs: add error breakdown table and minor refinements * docs: add sdg and data-designer keywords to plugin metadata * docs: refine CLI framing, reduce em dashes, slop guard pass * docs: fix grammar in dev note (serial comma, double-which clause) * update hero image * docs: swap hero image, move benchmark figure, minor wording tweaks * docs: add narrative lead-in to skill trace summary * docs: refine quality bullet, streamline getting started modes * remove old image * slope-guard tweaks
757 lines
46 KiB
HTML
757 lines
46 KiB
HTML
<!-- ── 2. Typical Baseline Session ── -->
|
||
<details class="session-trace" open>
|
||
<summary>
|
||
<span class="st-arrow">▶</span>
|
||
<span class="st-badge baseline">Baseline</span>
|
||
<span class="st-name">Typical Baseline Session</span>
|
||
<span class="st-stats">
|
||
<span class="st-stat">35 tools</span>
|
||
<span class="st-stat err">1 errors</span>
|
||
<span class="st-stat">159s</span>
|
||
</span>
|
||
</summary>
|
||
<div class="st-body">
|
||
<div class="st-step">
|
||
<span class="st-tool msg">Agent</span>
|
||
<span class="st-desc"><em>Let me start by exploring the Data Designer package and checking available configurations.</em></span>
|
||
</div>
|
||
<div class="st-step">
|
||
<span class="st-num">1</span>
|
||
<span class="st-tool bash">Bash</span>
|
||
<span class="st-desc">
|
||
<code>.venv/bin/python -c "import data_designer.config as dd, os; print(os.path.dirname(dd.__file__))"</code><div>.venv/lib/python3.13/site-packages/data_designer/config</div>
|
||
</span>
|
||
</div>
|
||
<div class="st-step">
|
||
<span class="st-num">2</span>
|
||
<span class="st-tool bash">Bash</span>
|
||
<span class="st-desc">
|
||
<code>.venv/bin/data-designer config list</code><details class="st-expand"><summary><span class="st-arrow">▶</span> output (5,284 chars)</summary><div> ─────────────────────── Data Designer Configurations ───────────────────────
|
||
|
||
💡 Configuration directory: ~/.data-designer
|
||
|
||
Model Providers
|
||
┏━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━┓
|
||
┃ Name ┃ Endpoint ┃ Type ┃ API Key ┃ Default ┃
|
||
┡━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━┩
|
||
│ nvidia │ https://integrate.a… │ openai │ NVIDIA_API_KEY │ ✓ │
|
||
│ openai │ https://inference-a… │ openai │ NVIDIA_INFERENCE_API… │ │
|
||
│ openrouter │ https://openrouter.… │ openai │ OPENROUTER_API_KEY │ │
|
||
└────────────┴──────────────────────┴────────┴───────────────────────┴─────────┘
|
||
|
||
Model Configurations
|
||
┏━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┓
|
||
┃ ┃ ┃ ┃ Inference ┃
|
||
┃ Alias ┃ Model ┃ Provider ┃ Parameters ┃
|
||
┡━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━┩
|
||
│ nvidia-text │ nvidia/nemotron-3-… │ nvidia │ generation_type=c… │
|
||
│ │ │ │ max_parallel_requ… │
|
||
│ │ │ │ temperature=1.00, │
|
||
│ │ │ │ top_p=1.00 │
|
||
│ nvidia-reasoning │ openai/gpt-oss-20b │ nvidia │ generation_type=c… │
|
||
│ │ │ │ max_parallel_requ… │
|
||
│ │ │ │ temperature=0.35, │
|
||
│ │ │ │ top_p=0.95 │
|
||
│ nvidia-vision │ nvidia/nemotron-na… │ nvidia │ generation_type=c… │
|
||
│ │ │ │ max_parallel_requ… │
|
||
│ │ │ │ temperature=0.85, │
|
||
│ │ │ │ top_p=0.95 │
|
||
│ nvidia-embedding │ nvidia/llama-3.2-n… │ nvidia │ generation_type=e… │
|
||
│ │ │ │ max_parallel_requ… │
|
||
│ │ │ │ extra_body={'inpu… │
|
||
│ │ │ │ 'query'}, │
|
||
│ │ │ │ encoding_format=f… │
|
||
│ openai-text │ us/azure/openai/gp… │ openai │ generation_type=c… │
|
||
│ │ │ │ max_parallel_requ… │
|
||
│ │ │ │ temperature=0.85, │
|
||
│ │ │ │ top_p=0.95 │
|
||
│ openai-reasoning │ openai/openai/gpt-… │ openai │ generation_type=c… │
|
||
│ │ │ │ max_parallel_requ… │
|
||
│ │ │ │ extra_body={'reas… │
|
||
│ │ │ │ 'medium'} │
|
||
│ openai-vision │ openai/openai/gpt-… │ openai │ generation_type=c… │
|
||
│ │ │ │ max_parallel_requ… │
|
||
│ │ │ │ extra_body={'reas… │
|
||
│ │ │ │ 'medium'} │
|
||
│ openai-embedding │ text-embedding-3-l… │ openai │ generation_type=e… │
|
||
│ │ │ │ max_parallel_requ… │
|
||
│ │ │ │ encoding_format=f… │
|
||
│ openrouter-text │ nvidia/nemotron-3-… │ openrouter │ generation_type=c… │
|
||
│ │ │ │ max_parallel_requ… │
|
||
│ │ │ │ temperature=1.00, │
|
||
│ │ │ │ top_p=1.00 │
|
||
│ openrouter-reasoning │ openai/gpt-oss-20b │ openrouter │ generation_type=c… │
|
||
│ │ │ │ max_parallel_requ… │
|
||
│ │ │ │ temperature=0.35, │
|
||
│ │ │ │ top_p=0.95 │
|
||
│ openrouter-vision │ nvidia/nemotron-na… │ openrouter │ generation_type=c… │
|
||
│ │ │ │ max_parallel_requ… │
|
||
│ │ │ │ temperature=0.85, │
|
||
│ │ │ │ top_p=0.95 │
|
||
│ openrouter-embedding │ openai/text-embedd… │ openrouter │ generation_type=e… │
|
||
│ │ │ │ max_parallel_requ… │
|
||
│ │ │ │ encoding_format=f… │
|
||
└──────────────────────┴─────────────────────┴────────────┴────────────────────┘
|
||
|
||
⚠️ MCP providers have not been configured. Run 'data-designer config mcp' to
|
||
configure them.
|
||
|
||
⚠️ Tool configs have not been configured. Run 'data-designer config tools'
|
||
to configure them.</div></details>
|
||
</span>
|
||
</div>
|
||
<div class="st-step">
|
||
<span class="st-tool msg">Agent</span>
|
||
<span class="st-desc"><em>Now let me explore the Data Designer package structure and understand the API.</em></span>
|
||
</div>
|
||
<div class="st-step err">
|
||
<span class="st-num">3</span>
|
||
<span class="st-tool agent">Agent</span>
|
||
<span class="st-desc">
|
||
<code>Explore</code> — Explore Data Designer API<details class="st-expand"><summary><span class="st-arrow">▶</span> prompt (974 chars)</summary><div>Explore the Data Designer package thoroughly. The package is located at:
|
||
.venv/lib/python3.13/site-packages/data_designer/
|
||
|
||
I need to understand:
|
||
1. How to create a DataDesignerConfig or ConfigBuilder — look for the main config builder API
|
||
2. What column types are available (especially for LLM generation, code validation, sampling/categorical, LLM-as-judge)
|
||
3. How columns are defined and added to a config
|
||
4. How to specify model aliases for columns
|
||
5. Any example configs or tests that show how to build a config
|
||
6. How the `data-designer validate` CLI command works (what it expects)
|
||
7. The `load_config_builder` function pattern — what should it return?
|
||
|
||
Be very thorough — check __init__.py files, column types, config module, CLI module, and any examples directory. Look at the actual class signatures and constructors.</div></details><details class="st-expand"><summary><span class="st-arrow">▶</span> subagent work (25 tool calls, <strong style="color:#ef4444">1 error</strong>)</summary><div style="white-space:normal"><div style="display:flex;gap:.5rem;align-items:baseline;padding:.15rem 0;border-left:2px solid transparent;padding-left:.4rem"><span style="min-width:1.2rem;text-align:right;opacity:.4;font-size:.7rem">1</span><span class="st-tool bash" style="font-size:.62rem;min-width:2.2rem;padding:.02rem .25rem">Bash</span><span style="font-size:.72rem"><code>find .venv/lib/python3.13/site-packages/data_designer</code></span></div>
|
||
<div style="display:flex;gap:.5rem;align-items:baseline;padding:.15rem 0;border-left:2px solid transparent;padding-left:.4rem"><span style="min-width:1.2rem;text-align:right;opacity:.4;font-size:.7rem">2</span><span class="st-tool bash" style="font-size:.62rem;min-width:2.2rem;padding:.02rem .25rem">Bash</span><span style="font-size:.72rem"><code>find .venv/lib/python3.13/site-packages/data_designer</code></span></div>
|
||
<div style="display:flex;gap:.5rem;align-items:baseline;padding:.15rem 0;border-left:2px solid #ef4444;padding-left:.4rem"><span style="min-width:1.2rem;text-align:right;opacity:.4;font-size:.7rem">3</span><span class="st-tool read" style="font-size:.62rem;min-width:2.2rem;padding:.02rem .25rem">Read</span><span style="font-size:.72rem"><code>data_designer/__init__.py</code> — <strong style="color:#ef4444">error</strong></span></div>
|
||
<div style="display:flex;gap:.5rem;align-items:baseline;padding:.15rem 0;border-left:2px solid transparent;padding-left:.4rem"><span style="min-width:1.2rem;text-align:right;opacity:.4;font-size:.7rem">4</span><span class="st-tool read" style="font-size:.62rem;min-width:2.2rem;padding:.02rem .25rem">Read</span><span style="font-size:.72rem"><code>data_designer/config/__init__.py</code></span></div>
|
||
<div style="display:flex;gap:.5rem;align-items:baseline;padding:.15rem 0;border-left:2px solid transparent;padding-left:.4rem"><span style="min-width:1.2rem;text-align:right;opacity:.4;font-size:.7rem">5</span><span class="st-tool read" style="font-size:.62rem;min-width:2.2rem;padding:.02rem .25rem">Read</span><span style="font-size:.72rem"><code>data_designer/config/config_builder.py</code></span></div>
|
||
<div style="display:flex;gap:.5rem;align-items:baseline;padding:.15rem 0;border-left:2px solid transparent;padding-left:.4rem"><span style="min-width:1.2rem;text-align:right;opacity:.4;font-size:.7rem">6</span><span class="st-tool read" style="font-size:.62rem;min-width:2.2rem;padding:.02rem .25rem">Read</span><span style="font-size:.72rem"><code>data_designer/config/column_types.py</code></span></div>
|
||
<div style="display:flex;gap:.5rem;align-items:baseline;padding:.15rem 0;border-left:2px solid transparent;padding-left:.4rem"><span style="min-width:1.2rem;text-align:right;opacity:.4;font-size:.7rem">7</span><span class="st-tool bash" style="font-size:.62rem;min-width:2.2rem;padding:.02rem .25rem">Bash</span><span style="font-size:.72rem"><code>find .venv/lib/python3.13/site-packages/data_designer</code></span></div>
|
||
<div style="display:flex;gap:.5rem;align-items:baseline;padding:.15rem 0;border-left:2px solid transparent;padding-left:.4rem"><span style="min-width:1.2rem;text-align:right;opacity:.4;font-size:.7rem">8</span><span class="st-tool read" style="font-size:.62rem;min-width:2.2rem;padding:.02rem .25rem">Read</span><span style="font-size:.72rem"><code>data_designer/config/column_configs.py</code> <span style="opacity:.5">(limit=300)</span></span></div>
|
||
<div style="display:flex;gap:.5rem;align-items:baseline;padding:.15rem 0;border-left:2px solid transparent;padding-left:.4rem"><span style="min-width:1.2rem;text-align:right;opacity:.4;font-size:.7rem">9</span><span class="st-tool read" style="font-size:.62rem;min-width:2.2rem;padding:.02rem .25rem">Read</span><span style="font-size:.72rem"><code>data_designer/config/column_configs.py</code> <span style="opacity:.5">(offset=300, limit=300)</span></span></div>
|
||
<div style="display:flex;gap:.5rem;align-items:baseline;padding:.15rem 0;border-left:2px solid transparent;padding-left:.4rem"><span style="min-width:1.2rem;text-align:right;opacity:.4;font-size:.7rem">10</span><span class="st-tool bash" style="font-size:.62rem;min-width:2.2rem;padding:.02rem .25rem">Bash</span><span style="font-size:.72rem"><code>find .venv/lib/python3.13/site-packages/data_designer</code></span></div>
|
||
<div style="display:flex;gap:.5rem;align-items:baseline;padding:.15rem 0;border-left:2px solid transparent;padding-left:.4rem"><span style="min-width:1.2rem;text-align:right;opacity:.4;font-size:.7rem">11</span><span class="st-tool bash" style="font-size:.62rem;min-width:2.2rem;padding:.02rem .25rem">Bash</span><span style="font-size:.72rem"><code>find .venv/lib/python3.13/site-packages/data_designer</code></span></div>
|
||
<div style="display:flex;gap:.5rem;align-items:baseline;padding:.15rem 0;border-left:2px solid transparent;padding-left:.4rem"><span style="min-width:1.2rem;text-align:right;opacity:.4;font-size:.7rem">12</span><span class="st-tool read" style="font-size:.62rem;min-width:2.2rem;padding:.02rem .25rem">Read</span><span style="font-size:.72rem"><code>data_designer/cli/commands/validate.py</code></span></div>
|
||
<div style="display:flex;gap:.5rem;align-items:baseline;padding:.15rem 0;border-left:2px solid transparent;padding-left:.4rem"><span style="min-width:1.2rem;text-align:right;opacity:.4;font-size:.7rem">13</span><span class="st-tool read" style="font-size:.62rem;min-width:2.2rem;padding:.02rem .25rem">Read</span><span style="font-size:.72rem"><code>data_designer/cli/utils/config_loader.py</code></span></div>
|
||
<div style="display:flex;gap:.5rem;align-items:baseline;padding:.15rem 0;border-left:2px solid transparent;padding-left:.4rem"><span style="min-width:1.2rem;text-align:right;opacity:.4;font-size:.7rem">14</span><span class="st-tool read" style="font-size:.62rem;min-width:2.2rem;padding:.02rem .25rem">Read</span><span style="font-size:.72rem"><code>data_designer/cli/controllers/generation_controller.py</code> <span style="opacity:.5">(limit=200)</span></span></div>
|
||
<div style="display:flex;gap:.5rem;align-items:baseline;padding:.15rem 0;border-left:2px solid transparent;padding-left:.4rem"><span style="min-width:1.2rem;text-align:right;opacity:.4;font-size:.7rem">15</span><span class="st-tool read" style="font-size:.62rem;min-width:2.2rem;padding:.02rem .25rem">Read</span><span style="font-size:.72rem"><code>data_designer/config/models.py</code> <span style="opacity:.5">(limit=250)</span></span></div>
|
||
<div style="display:flex;gap:.5rem;align-items:baseline;padding:.15rem 0;border-left:2px solid transparent;padding-left:.4rem"><span style="min-width:1.2rem;text-align:right;opacity:.4;font-size:.7rem">16</span><span class="st-tool read" style="font-size:.62rem;min-width:2.2rem;padding:.02rem .25rem">Read</span><span style="font-size:.72rem"><code>data_designer/config/models.py</code> <span style="opacity:.5">(offset=250, limit=250)</span></span></div>
|
||
<div style="display:flex;gap:.5rem;align-items:baseline;padding:.15rem 0;border-left:2px solid transparent;padding-left:.4rem"><span style="min-width:1.2rem;text-align:right;opacity:.4;font-size:.7rem">17</span><span class="st-tool read" style="font-size:.62rem;min-width:2.2rem;padding:.02rem .25rem">Read</span><span style="font-size:.72rem"><code>data_designer/config/models.py</code> <span style="opacity:.5">(offset=500, limit=200)</span></span></div>
|
||
<div style="display:flex;gap:.5rem;align-items:baseline;padding:.15rem 0;border-left:2px solid transparent;padding-left:.4rem"><span style="min-width:1.2rem;text-align:right;opacity:.4;font-size:.7rem">18</span><span class="st-tool read" style="font-size:.62rem;min-width:2.2rem;padding:.02rem .25rem">Read</span><span style="font-size:.72rem"><code>data_designer/config/sampler_params.py</code> <span style="opacity:.5">(limit=200)</span></span></div>
|
||
<div style="display:flex;gap:.5rem;align-items:baseline;padding:.15rem 0;border-left:2px solid transparent;padding-left:.4rem"><span style="min-width:1.2rem;text-align:right;opacity:.4;font-size:.7rem">19</span><span class="st-tool bash" style="font-size:.62rem;min-width:2.2rem;padding:.02rem .25rem">Bash</span><span style="font-size:.72rem"><code>find .venv/lib/python3.13/site-packages/data_designer</code></span></div>
|
||
<div style="display:flex;gap:.5rem;align-items:baseline;padding:.15rem 0;border-left:2px solid transparent;padding-left:.4rem"><span style="min-width:1.2rem;text-align:right;opacity:.4;font-size:.7rem">20</span><span class="st-tool read" style="font-size:.62rem;min-width:2.2rem;padding:.02rem .25rem">Read</span><span style="font-size:.72rem"><code>data_designer/config/data_designer_config.py</code> <span style="opacity:.5">(limit=200)</span></span></div>
|
||
<div style="display:flex;gap:.5rem;align-items:baseline;padding:.15rem 0;border-left:2px solid transparent;padding-left:.4rem"><span style="min-width:1.2rem;text-align:right;opacity:.4;font-size:.7rem">21</span><span class="st-tool bash" style="font-size:.62rem;min-width:2.2rem;padding:.02rem .25rem">Bash</span><span style="font-size:.72rem"><code>find .venv/lib/python3.13/site-packages/data_designer</code></span></div>
|
||
<div style="display:flex;gap:.5rem;align-items:baseline;padding:.15rem 0;border-left:2px solid transparent;padding-left:.4rem"><span style="min-width:1.2rem;text-align:right;opacity:.4;font-size:.7rem">22</span><span class="st-tool read" style="font-size:.62rem;min-width:2.2rem;padding:.02rem .25rem">Read</span><span style="font-size:.72rem"><code>data_designer/config/validator_params.py</code> <span style="opacity:.5">(limit=200)</span></span></div>
|
||
<div style="display:flex;gap:.5rem;align-items:baseline;padding:.15rem 0;border-left:2px solid transparent;padding-left:.4rem"><span style="min-width:1.2rem;text-align:right;opacity:.4;font-size:.7rem">23</span><span class="st-tool read" style="font-size:.62rem;min-width:2.2rem;padding:.02rem .25rem">Read</span><span style="font-size:.72rem"><code>data_designer/interface/data_designer.py</code> <span style="opacity:.5">(limit=150)</span></span></div>
|
||
<div style="display:flex;gap:.5rem;align-items:baseline;padding:.15rem 0;border-left:2px solid transparent;padding-left:.4rem"><span style="min-width:1.2rem;text-align:right;opacity:.4;font-size:.7rem">24</span><span class="st-tool read" style="font-size:.62rem;min-width:2.2rem;padding:.02rem .25rem">Read</span><span style="font-size:.72rem"><code>data_designer/config/utils/code_lang.py</code></span></div>
|
||
<div style="display:flex;gap:.5rem;align-items:baseline;padding:.15rem 0;border-left:2px solid transparent;padding-left:.4rem"><span style="min-width:1.2rem;text-align:right;opacity:.4;font-size:.7rem">25</span><span class="st-tool read" style="font-size:.62rem;min-width:2.2rem;padding:.02rem .25rem">Read</span><span style="font-size:.72rem"><code>data_designer/config/sampler_params.py</code> <span style="opacity:.5">(offset=200, limit=300)</span></span></div></div></details><details class="st-expand"><summary><span class="st-arrow">▶</span> subagent report (12,076 chars)</summary><div>Now I have comprehensive information about the Data Designer package. Let me compile a thorough report.
|
||
|
||
## Data Designer Package Exploration Report
|
||
|
||
I've conducted a thorough exploration of the Data Designer package located at `.venv/lib/python3.13/site-packages/data_designer/`. Here's my comprehensive understanding:
|
||
|
||
### 1. Creating a DataDesignerConfig or ConfigBuilder
|
||
|
||
**Main API:** The primary builder class is `DataDesignerConfigBuilder` located at:
|
||
- `.venv/lib/python3.13/site-packages/data_designer/config/config_builder.py`
|
||
|
||
**Constructor:**
|
||
```python
|
||
DataDesignerConfigBuilder(
|
||
model_configs: list[ModelConfig] | str | Path | None = None,
|
||
tool_configs: list[ToolConfig] | None = None
|
||
)
|
||
```
|
||
|
||
**Key Methods:**
|
||
- `__init__()`: Initialize with optional model configs (defaults to local models) and tool configs
|
||
- `add_column()`: Add columns to the config (most commonly used)
|
||
- `add_model_config()`: Add model configurations
|
||
- `add_tool_config()`: Add MCP tool configurations
|
||
- `with_seed_dataset()`: Attach a seed dataset source
|
||
- `build()`: Returns a `DataDesignerConfig` object
|
||
- `get_builder_config()`: Returns a `BuilderConfig` wrapping the config
|
||
- `write_config()`: Save to YAML/JSON file
|
||
- `from_config()`: Create builder from existing config file or dict
|
||
|
||
**Alternative Creation:**
|
||
```python
|
||
# From existing config file/dict
|
||
builder = DataDesignerConfigBuilder.from_config(config_path_or_dict)
|
||
```
|
||
|
||
### 2. Available Column Types
|
||
|
||
**Location:** `.venv/lib/python3.13/site-packages/data_designer/config/column_types.py`
|
||
|
||
**Column Types (enum: DataDesignerColumnType):**
|
||
|
||
1. **LLM Generation Columns:**
|
||
- `llm-text` → `LLMTextColumnConfig`: Free-form text generation with Jinja2 templates
|
||
- `llm-code` → `LLMCodeColumnConfig`: Code generation with language specification (Python, SQL, JS, etc.)
|
||
- `llm-structured` → `LLMStructuredColumnConfig`: Structured JSON output using JSON schema or Pydantic models
|
||
- `llm-judge` → `LLMJudgeColumnConfig`: Multi-dimensional evaluation/scoring of other columns
|
||
|
||
2. **Sampling/Categorical Columns:**
|
||
- `sampler` → `SamplerColumnConfig`: Numerical and categorical sampling (13+ sampler types available)
|
||
|
||
3. **Code Validation:**
|
||
- `validation` → `ValidationColumnConfig`: Run validation code (Python/SQL) or call validation functions
|
||
|
||
4. **Other Column Types:**
|
||
- `seed-dataset` → `SeedDatasetColumnConfig`: Columns sourced from seed data (auto-created)
|
||
- `expression` → `ExpressionColumnConfig`: Jinja2-based derived/computed columns
|
||
- `embedding` → `EmbeddingColumnConfig`: Generate embeddings for text
|
||
- `image` → `ImageColumnConfig`: Generate images from prompts
|
||
- `custom` → `CustomColumnConfig`: User-defined generation logic with decorators
|
||
|
||
### 3. How Columns Are Defined and Added
|
||
|
||
**Two approaches:**
|
||
|
||
**Approach 1: Direct column objects**
|
||
```python
|
||
from data_designer.config import LLMTextColumnConfig, SamplerColumnConfig
|
||
|
||
text_col = LLMTextColumnConfig(
|
||
name="review",
|
||
prompt="Write a product review about {{ product_name }}",
|
||
model_alias="claude-3.5-sonnet"
|
||
)
|
||
builder.add_column(text_col)
|
||
|
||
# Sampler column
|
||
sampler_col = SamplerColumnConfig(
|
||
name="age",
|
||
sampler_type="uniform",
|
||
params={"low": 18, "high": 65}
|
||
)
|
||
builder.add_column(sampler_col)
|
||
```
|
||
|
||
**Approach 2: Via kwargs (using add_column)**
|
||
```python
|
||
builder.add_column(
|
||
name="review",
|
||
column_type="llm-text",
|
||
prompt="Write a product review about {{ product_name }}",
|
||
model_alias="claude-3.5-sonnet"
|
||
)
|
||
```
|
||
|
||
**Column Properties:**
|
||
- Each column config extends `SingleColumnConfig` with a `column_type` discriminator field
|
||
- Columns support:
|
||
- `required_columns`: List of columns this column depends on
|
||
- `side_effect_columns`: Additional columns created as side effects (e.g., traces, reasoning_content)
|
||
- Jinja2 template validation for prompts
|
||
- Type conversion and formatting options
|
||
|
||
### 4. Model Aliases for Columns
|
||
|
||
**Location:** `.venv/lib/python3.13/site-packages/data_designer/config/models.py`
|
||
|
||
**Model Configuration:**
|
||
```python
|
||
from data_designer.config import ModelConfig, ChatCompletionInferenceParams
|
||
|
||
model_config = ModelConfig(
|
||
alias="my-claude", # User-defined alias referenced in columns
|
||
model="claude-3.5-sonnet",
|
||
inference_parameters=ChatCompletionInferenceParams(
|
||
temperature=0.7,
|
||
top_p=0.95,
|
||
max_tokens=2000
|
||
),
|
||
provider=None, # Optional custom provider
|
||
skip_health_check=False
|
||
)
|
||
|
||
builder.add_model_config(model_config)
|
||
```
|
||
|
||
**Inference Parameter Types:**
|
||
- `ChatCompletionInferenceParams`: For LLM generation (temperature, top_p, max_tokens, dynamic distributions)
|
||
- `EmbeddingInferenceParams`: For embeddings (encoding_format, dimensions)
|
||
- `ImageInferenceParams`: For image generation (extra_body for model-specific options)
|
||
|
||
**Model Properties:**
|
||
- `alias`: User-defined reference used in columns (e.g., `model_alias="my-claude"`)
|
||
- `model`: Model identifier (e.g., "claude-3.5-sonnet")
|
||
- `inference_parameters`: Supports fixed values or distributions for sampling
|
||
- `generation_type`: Auto-determined from inference_parameters (chat-completion, embedding, or image)
|
||
|
||
### 5. Available Sampler Types
|
||
|
||
**Location:** `.venv/lib/python3.13/site-packages/data_designer/config/sampler_params.py`
|
||
|
||
**All Sampler Types (SamplerType enum):**
|
||
|
||
1. **Statistical Distributions:**
|
||
- `uniform`: Continuous uniform distribution
|
||
- `gaussian`: Normal/Gaussian distribution
|
||
- `binomial`: Binomial distribution (success count)
|
||
- `bernoulli`: Binary (0/1) outcome
|
||
- `bernoulli_mixture`: Mixed zero-inflation with continuous distribution
|
||
- `poisson`: Poisson count distribution
|
||
- `scipy`: Any scipy.stats distribution
|
||
|
||
2. **Categorical/Discrete:**
|
||
- `category`: Discrete categories with optional weights
|
||
- `subcategory`: Hierarchical sampling based on parent category
|
||
|
||
3. **Temporal:**
|
||
- `datetime`: Uniform datetime sampling within range
|
||
- `timedelta`: Time offset relative to reference column
|
||
|
||
4. **Special:**
|
||
- `uuid`: UUID generation (with optional prefix/formatting)
|
||
- `person`: Synthetic person data with demographic filtering (from managed datasets or Faker)
|
||
- `person_from_faker`: Pure Faker-based person generation
|
||
|
||
### 6. `data-designer validate` CLI Command
|
||
|
||
**Location:** `.venv/lib/python3.13/site-packages/data_designer/cli/commands/validate.py`
|
||
|
||
**Usage:**
|
||
```bash
|
||
data-designer validate <config_source>
|
||
```
|
||
|
||
**What it validates:**
|
||
- Checks configuration is well-formed
|
||
- Resolves all references (models, columns, seed datasets, etc.)
|
||
- Validates model aliases are defined
|
||
- Validates column dependencies and required columns
|
||
- Checks constraint validity
|
||
- Validates processor configurations
|
||
|
||
**Accepted Config Sources:**
|
||
1. **YAML/JSON files**: Local paths or HTTP(S) URLs
|
||
2. **Python modules**: `.py` files that define a `load_config_builder()` function
|
||
|
||
**Controller Implementation:**
|
||
```python
|
||
# In GenerationController.run_validate()
|
||
data_designer = DataDesigner()
|
||
data_designer.validate(config_builder)
|
||
```
|
||
|
||
Exits with code 0 on success, code 1 on validation failure.
|
||
|
||
### 7. `load_config_builder` Function Pattern
|
||
|
||
**Location:** `.venv/lib/python3.13/site-packages/data_designer/cli/utils/config_loader.py`
|
||
|
||
**Function Signature Required:**
|
||
```python
|
||
def load_config_builder() -> DataDesignerConfigBuilder:
|
||
"""Must return a DataDesignerConfigBuilder instance."""
|
||
...
|
||
```
|
||
|
||
**What it should return:**
|
||
- A `DataDesignerConfigBuilder` instance (not BuilderConfig, not DataDesignerConfig)
|
||
- The builder is used by the CLI to access the full configuration
|
||
|
||
**Validation Requirements:**
|
||
1. The `.py` file **must** define `load_config_builder()` function
|
||
2. Function must be callable and return exactly a `DataDesignerConfigBuilder`
|
||
3. The CLI will call this function and validate the returned builder
|
||
4. The parent directory of the `.py` file is temporarily added to sys.path to support local imports
|
||
|
||
**Example Pattern:**
|
||
```python
|
||
# my_config.py
|
||
from data_designer.config import DataDesignerConfigBuilder
|
||
|
||
def load_config_builder() -> DataDesignerConfigBuilder:
|
||
builder = DataDesignerConfigBuilder()
|
||
|
||
# Add model
|
||
builder.add_model_config(...)
|
||
|
||
# Add columns
|
||
builder.add_column(...)
|
||
|
||
return builder
|
||
|
||
# CLI usage:
|
||
# data-designer validate my_config.py
|
||
```
|
||
|
||
### 8. LLM-as-Judge Column Configuration
|
||
|
||
**Location:** `.venv/lib/python3.13/site-packages/data_designer/config/column_configs.py`
|
||
|
||
**Class:** `LLMJudgeColumnConfig` (extends `LLMTextColumnConfig`)
|
||
|
||
**Configuration Example:**
|
||
```python
|
||
from data_designer.config import LLMJudgeColumnConfig, Score
|
||
|
||
judge_col = LLMJudgeColumnConfig(
|
||
name="quality_assessment",
|
||
prompt="Evaluate the quality of: {{ generated_text }}",
|
||
model_alias="claude-3.5-sonnet",
|
||
system_prompt="You are a quality evaluator...",
|
||
scores=[
|
||
Score(
|
||
name="Relevance",
|
||
description="How relevant is the content to the topic?",
|
||
options={
|
||
1: "Not relevant",
|
||
2: "Somewhat relevant",
|
||
3: "Highly relevant"
|
||
}
|
||
),
|
||
Score(
|
||
name="Fluency",
|
||
description="How fluent and natural is the writing?",
|
||
options={
|
||
"poor": "Difficult to understand",
|
||
"good": "Clear and readable",
|
||
"excellent": "Excellent quality"
|
||
}
|
||
)
|
||
],
|
||
with_trace=TraceType.ALL_MESSAGES,
|
||
extract_reasoning_content=False
|
||
)
|
||
|
||
builder.add_column(judge_col)
|
||
```
|
||
|
||
**Key Features:**
|
||
- Inherits all LLMTextColumnConfig capabilities (prompts, traces, reasoning extraction, tool calling)
|
||
- `scores`: List of `Score` objects defining evaluation dimensions
|
||
- Each score has: name, description, options (dict of value → description)
|
||
- Can reference other columns in prompts via Jinja2 templating
|
||
- Supports system prompts and tool aliases for complex evaluations
|
||
|
||
### Key File Locations Summary
|
||
|
||
| Component | File Path |
|
||
|-----------|-----------|
|
||
| Config Builder | `.../data_designer/config/config_builder.py` |
|
||
| Column Types | `.../data_designer/config/column_types.py` |
|
||
| Column Configs | `.../data_designer/config/column_configs.py` |
|
||
| Model Config | `.../data_designer/config/models.py` |
|
||
| Sampler Params | `.../data_designer/config/sampler_params.py` |
|
||
| Validator Params | `.../data_designer/config/validator_params.py` |
|
||
| Code Languages | `.../data_designer/config/utils/code_lang.py` |
|
||
| Validate CLI | `.../data_designer/cli/commands/validate.py` |
|
||
| Config Loader | `.../data_designer/cli/utils/config_loader.py` |
|
||
| Main Interface | `.../data_designer/interface/data_designer.py` |
|
||
| Data Config | `.../data_designer/config/data_designer_config.py` |
|
||
agentId: af6f4be5db40ad2af (use SendMessage with to: 'af6f4be5db40ad2af' to continue this agent)
|
||
<usage>total_tokens: 87439
|
||
tool_uses: 25
|
||
duration_ms: 75309</usage></div></details>
|
||
</span>
|
||
</div>
|
||
<div class="st-step">
|
||
<span class="st-tool msg">Agent</span>
|
||
<span class="st-desc"><em>Let me check a few specific details about the column configs and validation column.</em></span>
|
||
</div>
|
||
<div class="st-step">
|
||
<span class="st-num">4</span>
|
||
<span class="st-tool read">Read</span>
|
||
<span class="st-desc">
|
||
<code>data_designer/config/column_configs.py</code>
|
||
</span>
|
||
</div>
|
||
<div class="st-step">
|
||
<span class="st-tool msg">Agent</span>
|
||
<span class="st-desc"><em>Now let me check the validator params and the config builder's `add_column` method, plus the `__init__` exports.</em></span>
|
||
</div>
|
||
<div class="st-step">
|
||
<span class="st-num">5</span>
|
||
<span class="st-tool read">Read</span>
|
||
<span class="st-desc">
|
||
<code>data_designer/config/validator_params.py</code>
|
||
</span>
|
||
</div>
|
||
<div class="st-step">
|
||
<span class="st-num">6</span>
|
||
<span class="st-tool read">Read</span>
|
||
<span class="st-desc">
|
||
<code>data_designer/config/__init__.py</code>
|
||
</span>
|
||
</div>
|
||
<div class="st-step">
|
||
<span class="st-tool msg">Agent</span>
|
||
<span class="st-desc"><em>Let me check the sampler params for category type and the config builder's `add_column` signature.</em></span>
|
||
</div>
|
||
<div class="st-step">
|
||
<span class="st-num">7</span>
|
||
<span class="st-tool grep">Grep</span>
|
||
<span class="st-desc">
|
||
<code>class CategorySamplerParams</code><details class="st-expand"><summary><span class="st-arrow">▶</span> results (1,072 chars)</summary><div>45:class CategorySamplerParams(ConfigBase):
|
||
46- """Parameters for categorical sampling with optional probability weighting.
|
||
47-
|
||
48- Samples values from a discrete set of categories. When weights are provided, values are
|
||
49- sampled according to their assigned probabilities. Without weights, uniform sampling is used.
|
||
50-
|
||
51- Attributes:
|
||
52- values: List of possible categorical values to sample from. Can contain strings, integers,
|
||
53- or floats. Must contain at least one value.
|
||
54- weights: Optional unnormalized probability weights for each value. If provided, must be
|
||
55- the same length as `values`. Weights are automatically normalized to sum to 1.0.
|
||
56- Larger weights result in higher sampling probability for the corresponding value.
|
||
57- """
|
||
58-
|
||
59- values: list[str | int | float] = Field(
|
||
60- ...,
|
||
61- min_length=1,
|
||
62- description="List of possible categorical values that can be sampled from.",
|
||
63- )
|
||
64- weights: list[float] | None = Field(
|
||
65- default=None,</div></details>
|
||
</span>
|
||
</div>
|
||
<div class="st-step">
|
||
<span class="st-num">8</span>
|
||
<span class="st-tool grep">Grep</span>
|
||
<span class="st-desc">
|
||
<code>def add_column</code><details class="st-expand"><summary><span class="st-arrow">▶</span> results (1,590 chars)</summary><div>267: def add_column(
|
||
268- self,
|
||
269- column_config: ColumnConfigT | None = None,
|
||
270- *,
|
||
271- name: str | None = None,
|
||
272- column_type: DataDesignerColumnType | None = None,
|
||
273- **kwargs,
|
||
274- ) -> Self:
|
||
275- """Add a Data Designer column configuration to the current Data Designer configuration.
|
||
276-
|
||
277- If no column config object is provided, you must provide the `name`, `column_type`, and any
|
||
278- additional keyword arguments that are required by the column config constructor.
|
||
279-
|
||
280- Args:
|
||
281- column_config: Data Designer column config object to add.
|
||
282- name: Name of the column to add. This is only used if `column_config` is not provided.
|
||
283- column_type: Column type to add. This is only used if `column_config` is not provided.
|
||
284- **kwargs: Additional keyword arguments to pass to the column constructor.
|
||
285-
|
||
286- Returns:
|
||
287- The current Data Designer config builder instance.
|
||
288-
|
||
289- Raises:
|
||
290- BuilderConfigurationError: If the column name collides with an existing seed dataset column.
|
||
291- """
|
||
292- if column_config is None:
|
||
293- if name is None or column_type is None:
|
||
294- raise BuilderConfigurationError(
|
||
295- "🛑 You must provide either a 'column_config' object or 'name' *and* 'column_type' "
|
||
296- f"with additional keyword arguments. You provided {column_config=}, {name=}, and {column_type=}."
|
||
297- )</div></details>
|
||
</span>
|
||
</div>
|
||
<div class="st-step">
|
||
<span class="st-tool msg">Agent</span>
|
||
<span class="st-desc"><em>Now I have everything I need. Let me write the config file.</em></span>
|
||
</div>
|
||
<div class="st-step">
|
||
<span class="st-num">9</span>
|
||
<span class="st-tool write">Write</span>
|
||
<span class="st-desc">
|
||
<code>config.py</code><details class="st-expand"><summary><span class="st-arrow">▶</span> written content (6,872 chars)</summary><div>from data_designer.config import (
|
||
CategorySamplerParams,
|
||
CodeLang,
|
||
CodeValidatorParams,
|
||
DataDesignerConfigBuilder,
|
||
LLMCodeColumnConfig,
|
||
LLMJudgeColumnConfig,
|
||
LLMTextColumnConfig,
|
||
SamplerColumnConfig,
|
||
Score,
|
||
ValidationColumnConfig,
|
||
ValidatorType,
|
||
)
|
||
|
||
|
||
def load_config_builder() -> DataDesignerConfigBuilder:
|
||
builder = DataDesignerConfigBuilder()
|
||
|
||
# --- Difficulty level ---
|
||
builder.add_column(
|
||
SamplerColumnConfig(
|
||
name="difficulty",
|
||
sampler_type="category",
|
||
params=CategorySamplerParams(
|
||
values=["beginner", "intermediate", "advanced"],
|
||
weights=[0.35, 0.40, 0.25],
|
||
),
|
||
)
|
||
)
|
||
|
||
# --- Subtopic ---
|
||
builder.add_column(
|
||
SamplerColumnConfig(
|
||
name="subtopic",
|
||
sampler_type="category",
|
||
params=CategorySamplerParams(
|
||
values=[
|
||
"data cleaning",
|
||
"exploratory analysis",
|
||
"aggregation and groupby operations",
|
||
"visualization with matplotlib/seaborn",
|
||
"statistical testing",
|
||
"feature engineering",
|
||
"working with messy or missing data",
|
||
],
|
||
),
|
||
)
|
||
)
|
||
|
||
# --- Natural language instruction ---
|
||
builder.add_column(
|
||
LLMTextColumnConfig(
|
||
name="instruction",
|
||
model_alias="nvidia-text",
|
||
prompt=(
|
||
"Write a clear, specific natural-language instruction for a {{ difficulty }}-level "
|
||
"Python data science task in the area of {{ subtopic }}.\n\n"
|
||
"The instruction should describe a concrete problem that a data scientist might encounter "
|
||
"and should be solvable in a single self-contained Python script. "
|
||
"Do NOT include any code in your response — only the task description.\n\n"
|
||
"Vary the complexity to match the difficulty level:\n"
|
||
"- beginner: simple operations using pandas, basic plots, straightforward cleaning\n"
|
||
"- intermediate: multi-step workflows, combining libraries, moderate data manipulation\n"
|
||
"- advanced: complex pipelines, statistical modeling, performance optimization, "
|
||
"advanced visualization techniques\n\n"
|
||
"Return only the instruction text, nothing else."
|
||
),
|
||
)
|
||
)
|
||
|
||
# --- Python solution ---
|
||
builder.add_column(
|
||
LLMCodeColumnConfig(
|
||
name="solution",
|
||
model_alias="nvidia-reasoning",
|
||
code_lang=CodeLang.PYTHON,
|
||
prompt=(
|
||
"Write a complete, self-contained Python solution for the following "
|
||
"{{ difficulty }}-level data science task:\n\n"
|
||
"{{ instruction }}\n\n"
|
||
"Requirements:\n"
|
||
"- The code must be syntactically valid and runnable as-is.\n"
|
||
"- Include all necessary imports.\n"
|
||
"- Generate or define any sample data needed (do not load from external files).\n"
|
||
"- Use appropriate libraries (pandas, numpy, matplotlib, seaborn, scipy, "
|
||
"scikit-learn, etc.) as needed.\n"
|
||
"- Add brief inline comments explaining key steps.\n"
|
||
"- Follow Python best practices and PEP 8 style.\n"
|
||
),
|
||
)
|
||
)
|
||
|
||
# --- Code validation ---
|
||
builder.add_column(
|
||
ValidationColumnConfig(
|
||
name="code_validation",
|
||
target_columns=["solution"],
|
||
validator_type=ValidatorType.CODE,
|
||
validator_params=CodeValidatorParams(code_lang=CodeLang.PYTHON),
|
||
)
|
||
)
|
||
|
||
# --- LLM-as-judge evaluation ---
|
||
builder.add_column(
|
||
LLMJudgeColumnConfig(
|
||
name="quality_score",
|
||
model_alias="nvidia-reasoning",
|
||
prompt=(
|
||
"Evaluate the following Python data science solution against its instruction.\n\n"
|
||
"**Difficulty:** {{ difficulty }}\n"
|
||
"**Subtopic:** {{ subtopic }}\n\n"
|
||
"**Instruction:**\n{{ instruction }}\n\n"
|
||
"**Solution:**\n```python\n{{ solution }}\n```\n\n"
|
||
"Score the solution on each dimension below. Be rigorous but fair."
|
||
),
|
||
scores=[
|
||
Score(
|
||
name="correctness",
|
||
description=(
|
||
"Does the code correctly implement what the instruction asks for? "
|
||
"Consider logical correctness, proper use of libraries, and whether "
|
||
"the output would match expectations."
|
||
),
|
||
options={
|
||
1: "Completely wrong or does not address the instruction",
|
||
2: "Major errors that would produce incorrect results",
|
||
3: "Partially correct but has notable issues",
|
||
4: "Mostly correct with minor issues",
|
||
5: "Fully correct and complete implementation",
|
||
},
|
||
),
|
||
Score(
|
||
name="code_quality",
|
||
description=(
|
||
"Assess the code style, readability, and Pythonic practices. "
|
||
"Consider variable naming, structure, comments, and adherence "
|
||
"to best practices."
|
||
),
|
||
options={
|
||
1: "Very poor quality, unreadable or badly structured",
|
||
2: "Below average, multiple style issues",
|
||
3: "Acceptable but could be improved",
|
||
4: "Good quality, clean and well-structured",
|
||
5: "Excellent quality, exemplary Python code",
|
||
},
|
||
),
|
||
Score(
|
||
name="instruction_clarity",
|
||
description=(
|
||
"How clear and well-defined is the original instruction? "
|
||
"Consider whether the task is unambiguous, specific enough to implement, "
|
||
"and appropriately scoped for the stated difficulty level."
|
||
),
|
||
options={
|
||
1: "Vague or incomprehensible instruction",
|
||
2: "Unclear, missing important details",
|
||
3: "Adequate but could be more specific",
|
||
4: "Clear and well-defined",
|
||
5: "Excellent clarity, perfectly scoped for difficulty level",
|
||
},
|
||
),
|
||
],
|
||
)
|
||
)
|
||
|
||
return builder
|
||
</div></details><div>File created successfully at: ./config.py</div>
|
||
</span>
|
||
</div>
|
||
<div class="st-step">
|
||
<span class="st-tool msg">Agent</span>
|
||
<span class="st-desc"><em>Now let me validate it.</em></span>
|
||
</div>
|
||
<div class="st-step ok">
|
||
<span class="st-num">10</span>
|
||
<span class="st-tool bash">Bash</span>
|
||
<span class="st-desc">
|
||
<code>.venv/bin/data-designer validate config.py</code><div> ────────────────────────── Data Designer Validate ──────────────────────────
|
||
|
||
Config: config.py
|
||
|
||
[21:25:04] [INFO] ✅ Validation passed
|
||
✅ Configuration is valid</div>
|
||
</span>
|
||
</div>
|
||
<div class="st-step">
|
||
<span class="st-tool msg">Agent</span>
|
||
<span class="st-desc">
|
||
<details class="st-expand"><summary><span class="st-arrow">▶</span> message (999 chars)</summary><div>The configuration validates successfully. Here's what the pipeline does:
|
||
|
||
1. **`difficulty`** — Sampler column that randomly picks from beginner (35%), intermediate (40%), advanced (25%)
|
||
2. **`subtopic`** — Sampler column that uniformly samples from 7 data science subtopics (data cleaning, exploratory analysis, aggregation/groupby, visualization, statistical testing, feature engineering, messy/missing data)
|
||
3. **`instruction`** — LLM text column (`nvidia-text`) that generates a natural language task description conditioned on the difficulty and subtopic
|
||
4. **`solution`** — LLM code column (`nvidia-reasoning`) that generates a complete, self-contained Python solution for the instruction
|
||
5. **`code_validation`** — Validation column that checks the generated Python code for syntactic correctness
|
||
6. **`quality_score`** — LLM judge column (`nvidia-reasoning`) that scores each (instruction, solution) pair on three dimensions (correctness, code quality, instruction clarity) using a 1–5 scale</div></details>
|
||
</span>
|
||
</div>
|
||
</div>
|
||
</details>
|