mirror of https://github.com/NVIDIA-NeMo/DataDesigner synced 2026-05-24 09:48:29 +00:00

feat(cli): add plugin catalog core (#618 )

* feat(cli): add plugin catalog services

Add typed catalog and tap models, persistent tap storage, cached
catalog loading, compatibility evaluation, install plan generation,
and runtime plugin discovery helpers.

Refs #617

* feat(cli): add plugins command group

Wire list, search, info, install, installed, and tap management
commands through the existing command-controller CLI pattern.

Refs #617

* test(cli): cover plugin catalog workflows

Add regression coverage for tap caching, catalog compatibility,
installer command generation, local path resolution, and Typer command
delegation.

Refs #617

* fix(cli): align plugin taps with schema v2

Validate tap catalogs against the schema v2 contract used by
NVIDIA-NeMo/DataDesignerPlugins#36, including source union fields,
docs URLs, package paths, compatibility metadata, and unique runtime
plugin names.

Derive Git install targets as package-qualified PEP 508 direct
references so git tap entries install the package described by the
catalog source metadata.

Refs #617

* fix(cli): address plugin review feedback

- Invalidate import caches before post-install entry point verification
- Make tap aliases case-insensitive and cache catalogs by alias plus URL
- Prefer compatible catalog entries before falling back to forced installs
- Clarify unused --tap behavior and list installed entry points without imports
- Add direct controller coverage and update CLI plugin documentation

Refs #617

* fix(cli): gate incompatible plugin installs

Fetch install targets before compatibility filtering so the controller
owns the final --force decision and the incompatible install guard stays
reachable.

Refs #617

* style(cli): format plugin catalog files

Apply ruff formatting to the plugin command and tap repository tests so
CI format checks pass on the PR merge commit.

Refs #617

* fix(cli): reject duplicate plugin entry names

Key catalog duplicate detection by entry_point.name so distinct catalog
entries cannot register the same runtime plugin name.

Refs #617

* fix(cli): preserve GitHub tree tap paths

* fix(cli): verify plugin entry point names

* align plugin CLI with catalog schema

- adopt catalog terminology for plugin source aliases
- parse package-first plugin catalog metadata from the plugin repo
- install package requirements with optional catalog indexes

* tidy plugin catalog workflow docs

* align plugin catalog CLI with package contract

* add plugin package uninstall workflow

* test plugin package command targets

* document plugin package aliases

* address plugin catalog review feedback

* prefer runtime plugin lookup matches

* rename plugins command to plugin

* show plugin package descriptions

* rename plugin catalogs command

* add protected plugin package installs

* document plugin package install modes

* avoid building project during plugin installs

* harden plugin package installs

* tighten plugin catalog contracts

* fix no-args help exit code

* make plugin docs links robust

* document plugin CLI catalog workflows

* clarify plugin entry point verification

* simplify plugin CLI docs

* narrow plugin search fields

* hide plugin catalog cache ttl

* remove plugin catalog trust flag

* improve plugin CLI recovery UX

* polish plugin catalog table display

* stabilize plugin catalog table test

* tighten plugin catalog edge cases

* harden plugin catalog verification

- Escape catalog-provided Rich markup before rendering CLI output
- Reject runtime plugin names that collide after enum-key normalization
- Load installed runtime entry points in a subprocess before reporting success

* simplify plugin entry point verification

Load matching entry points directly after install instead of spawning a
separate Python process. This keeps the check package-scoped while still
catching broken entry-point targets and non-Plugin objects.

* require newer uv for plugin plans

Use uv >= 0.10.0 as the single supported uv requirement for
plugin package commands. Auto mode now falls back to a pip plan with
an upgrade warning when uv is unavailable or too old, while explicit
uv selection remains strict.

* verify pip fallback availability

* polish plugin CLI status markers

* clarify plugin compatibility labels

* simplify plugin info install details

* address plugin CLI review nits

* support versioned plugin package installs

* share plugin install metadata rendering

* show installed plugin packages

* harden versioned plugin installs

- Preserve catalog requirement constraints for versioned installs
- Remove stale install-plan metadata fields
- Expand parser, uv, controller, and local-catalog dry-run coverage

* harden plugin help tests

* show plugin package versions

Add package version metadata support for plugin catalogs and resolve current versions from exact requirements or simple indexes when catalog entries omit them.

Update plugin list/info/install metadata to show the plugin package version and Data Designer compatibility requirement while removing the separate Data Designer version line.

* format plugin catalog tests

* harden plugin package metadata checks

* harden plugin CLI test coverage

* add plugin discovery docs (#642)

Signed-off-by: Johnny Greco <jogreco@nvidia.com>

---------

Signed-off-by: Johnny Greco <jogreco@nvidia.com>

2026-05-13 12:26:58 -04:00

2.8 KiB

Raw Blame History

Data Designer Plugins

Plugins let you add new object types to Data Designer without modifying the core library. Once installed, plugins behave like native Data Designer objects: they use the same declarative config patterns, builder APIs, discovery flow, and runtime execution paths as the built-in objects.

Supported plugin types

Data Designer supports three plugin types:

Column generator plugins: Custom column generators you pass to the config builder's add_column method.
Seed reader plugins: Custom seed readers that load data from new sources, such as databases, cloud storage, or custom file formats.
Processor plugins: Custom processor implementations configured by processor config objects that transform data before batches, after batches, or after generation completes. Pass them to the config builder's add_processor method.

Use an Installed Plugin

Plugin packages register their Plugin objects through Python package entry points. Data Designer discovers installed plugin entry points automatically, so no extra registration code is required. Once a plugin package is installed, use its new object types in your Data Designer workflow.

If you install a plugin after data_designer has already been imported, restart the Python process so plugin discovery can rebuild from the new entry points.

Build a Plugin

For implementation instructions across all plugin types, see the Build Your Own section.

Find Plugins

Use the Data Designer CLI to discover and install published plugin packages from catalogs. See Discover Plugins for the catalog workflow, first-party plugin documentation, and source links.

Discovery troubleshooting

If a plugin is installed but not available, check these items first:

The entry point group must be exactly data_designer.plugins.
Check the value of the DISABLE_DATA_DESIGNER_PLUGINS environment variable. If it is set to true, entry point discovery is disabled.
The plugin discriminator default must be a string. Use column_type, seed_type, or processor_type, depending on the plugin type.
Avoid duplicate plugin names. Discovery stores plugins by plugin.name, which comes from the discriminator default.
For plugin packages under development, call assert_valid_plugin on the plugin object to catch common structural issues at import time.

2.8 KiB Raw Blame History