mirror of
https://github.com/open-metadata/OpenMetadata
synced 2026-05-24 09:39:11 +00:00
c89bc2adac
* Oh boy, factory-boy Created a bunch of `factory-boy` factories that help creating mock test data easily * Update `try_bind` docker utility to ease debugging * Resolve conflicts between `Classification` tags * Refactor `TagClassifier` into another entity This is so: 1. We're not tied to the `ColumnClassifier` interface that forced returning `Mapping[T, float]` (unnecessary since we're returning `List[ScoredTag]` 2. The tag analyzer uses the same `recognizer_factories` registry we used for `PIIProcessor` 3. Create a separate service that abstracts using `TagScorer` and `TagAnalyzer` to return `TagScore`s (makes testing upstream code easier) * Interface to retrieve available `Tag`s and `Classification`s * Refactor `TagProcessor` to support multi-classification - Depends `ClassificationManagerInterface` to retrieve `Tag`s and `Classification`s - Uses a callable dependency to score tags for a column - Accepts a classification filter parameter - Leverages `ConflictResolver` to resolve conflicts between tags of the same `Classification` * Add an integration test for the `TagProcessor` * Ensure `PII` classification is configured with migrations # Conflicts: # bootstrap/sql/migrations/native/1.11.1/mysql/postDataMigrationSQLScript.sql # bootstrap/sql/migrations/native/1.11.1/postgres/postDataMigrationSQLScript.sql * Move `FakeClassificationManager` to `_openmetadata_testutils` This is because importing from `tests` breaks in the CI when running pytests from the root of the repo * Fix broken mutually exclusive classifications This is because the implementation did not take into account previous tags when resolving conflicts. This caused that running the classifier twice for a classification, with a mutually exclusive configuration, would end up breaking the exclusivity
25 lines
684 B
SQL
25 lines
684 B
SQL
UPDATE test_definition
|
|
SET json = jsonb_set(
|
|
json::jsonb,
|
|
'{supportedServices}',
|
|
'["Snowflake", "BigQuery", "Athena", "Redshift", "Postgres", "MySQL", "Mssql", "Oracle", "Trino", "SapHana"]'::jsonb
|
|
)
|
|
WHERE name = 'tableDiff'
|
|
AND (
|
|
json->'supportedServices' IS NULL
|
|
OR json->'supportedServices' = '[]'::jsonb
|
|
);
|
|
|
|
UPDATE
|
|
classification
|
|
SET
|
|
json = json::jsonb || json_build_object(
|
|
'autoClassificationConfig', json_build_object(
|
|
'enabled', true,
|
|
'conflictResolution', 'highest_priority',
|
|
'minimumConfidence', 0.6,
|
|
'requireExplicitMatch', true
|
|
)
|
|
)::jsonb
|
|
WHERE
|
|
json->>'name' = 'PII';
|