OpenMetadata/bootstrap/sql/migrations/native/1.11.1/postgres/postDataMigrationSQLScript.sql

Ignoring revisions in .git-blame-ignore-revs. Click here to bypass and see the normal blame view.

14 lines
348 B
MySQL
Raw Permalink Normal View History

Allow multiple classifications in `TagProcessor` (#24545) * Oh boy, factory-boy Created a bunch of `factory-boy` factories that help creating mock test data easily * Update `try_bind` docker utility to ease debugging * Resolve conflicts between `Classification` tags * Refactor `TagClassifier` into another entity This is so: 1. We're not tied to the `ColumnClassifier` interface that forced returning `Mapping[T, float]` (unnecessary since we're returning `List[ScoredTag]` 2. The tag analyzer uses the same `recognizer_factories` registry we used for `PIIProcessor` 3. Create a separate service that abstracts using `TagScorer` and `TagAnalyzer` to return `TagScore`s (makes testing upstream code easier) * Interface to retrieve available `Tag`s and `Classification`s * Refactor `TagProcessor` to support multi-classification - Depends `ClassificationManagerInterface` to retrieve `Tag`s and `Classification`s - Uses a callable dependency to score tags for a column - Accepts a classification filter parameter - Leverages `ConflictResolver` to resolve conflicts between tags of the same `Classification` * Add an integration test for the `TagProcessor` * Ensure `PII` classification is configured with migrations # Conflicts: # bootstrap/sql/migrations/native/1.11.1/mysql/postDataMigrationSQLScript.sql # bootstrap/sql/migrations/native/1.11.1/postgres/postDataMigrationSQLScript.sql * Move `FakeClassificationManager` to `_openmetadata_testutils` This is because importing from `tests` breaks in the CI when running pytests from the root of the repo * Fix broken mutually exclusive classifications This is because the implementation did not take into account previous tags when resolving conflicts. This caused that running the classifier twice for a classification, with a mutually exclusive configuration, would end up breaking the exclusivity
2025-12-10 15:26:12 +00:00
UPDATE
classification
SET
json = json::jsonb || json_build_object(
'autoClassificationConfig', json_build_object(
'enabled', true,
'conflictResolution', 'highest_priority',
'minimumConfidence', 0.6,
'requireExplicitMatch', true
)
)::jsonb
WHERE
json->>'name' = 'PII';