mirror of
https://github.com/open-metadata/OpenMetadata
synced 2026-05-24 09:39:11 +00:00
547 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Sriharsha Chintalapani
|
5620121e50
|
SearchIndex: tunable index settings + per-stage latency metrics (#27865)
Some checks are pending
Integration Tests - MySQL + Elasticsearch / Detect Changes (push) Waiting to run
Integration Tests - MySQL + Elasticsearch / integration-tests-mysql-elasticsearch (push) Blocked by required conditions
Integration Tests - PostgreSQL + OpenSearch / Detect Changes (push) Waiting to run
Integration Tests - PostgreSQL + OpenSearch / integration-tests-postgres-opensearch (push) Blocked by required conditions
Java Checkstyle / java-checkstyle (push) Waiting to run
Maven Collate Tests / maven-collate-ci (push) Waiting to run
OpenMetadata Service Unit Tests / Detect Changes (push) Waiting to run
OpenMetadata Service Unit Tests / openmetadata-service-unit-tests (mysql) (push) Blocked by required conditions
OpenMetadata Service Unit Tests / openmetadata-service-unit-tests (postgresql) (push) Blocked by required conditions
OpenMetadata Service Unit Tests / k8s_operator-unit-tests (push) Blocked by required conditions
OpenMetadata Service Unit Tests / openmetadata-service-unit-tests-status (push) Blocked by required conditions
Publish Package to Maven Central Repository / publish-maven-packages (push) Waiting to run
* SearchIndex: configurable index settings + per-stage latency metrics Adds two diagnostic and operational improvements to the distributed search indexing pipeline so operators can both tune cluster behavior per installation and pinpoint where reindex latency is being spent. Configurable index settings (per-installation, no code changes needed) - New SearchIndexing app config fields: liveIndexSettings (post-promote), bulkIndexSettings (during reindex), and per-entity overrides. - DefaultRecreateHandler applies bulk overrides on staged-index creation (e.g. refresh=-1, replicas=0, async translog) and reverts to live values before alias swap. Optional force-merge before swap. - Safety revert ensures the promoted index never inherits a disabled refresh interval, even if the admin only configured bulk overrides. - Live UX is preserved: refresh defaults to 1s so users and agents that read-after-write see near-real-time results. - New IndexManagementClient methods (updateIndexSettings, forceMerge) with implementations for OpenSearch and Elasticsearch. Per-stage latency metrics (consumer-vs-producer attribution) - StageStatsTracker accumulates per-stage wall-clock time alongside existing counters; added timing-only addStageTime() so per-record callbacks and per-batch wall-clock don't double-count. - DB migration 1.13.0 adds readerTimeMs / processTimeMs / sinkTimeMs / vectorTimeMs columns to search_index_server_stats. Existing rows get DEFAULT 0; aggregation queries SUM the new columns. - Reader timing wraps PartitionWorker.readEntitiesKeyset (DB latency). Process timing wraps the doc-build join in OpenSearch and Elasticsearch bulk sinks (CPU/serialization). Sink timing wraps client.indices().bulk (pure search-cluster latency), attributed per participating tracker. - DistributedJobStatsAggregator surfaces totalTimeMs on each StepStats so the UI can compute avg latency = totalTimeMs / successRecords and throughput = successRecords / (totalTimeMs / 1000) on every WebSocket push without server-side derivation. - New per-server aggregation query (getStatsByServer) for distributed visibility, fed into SearchIndexJob.ServerStats with timing fields. UI: each of the four stage cards (Reader / Process / Sink / Vector) shows "Latency: X ms · Y r/s" when timing is available; per-entity table gains Sink avg + Sink throughput columns. Docs panel updated. New SearchIndexing config section added with sane defaults that preserve current behavior. Tests: 6 new StageStatsTracker timing tests, new aggregator test that asserts StepStats.totalTimeMs is populated at job and per-entity level. All existing tests updated for new arg shapes; 60 unit tests pass. The pattern operators see: Reader avg climbing means DB-side issue (cache/autovacuum); Sink avg climbing means OS-side issue (segments/ back-pressure); only one entity's row climbing identifies the offender. |
||
|
Sriharsha Chintalapani
|
b118a87df2
|
Add text_pattern_ops index on entity-table fqnHash for Postgres listings (#27868)
* Add text_pattern_ops index on entity-table fqnHash for Postgres listings Service-filtered listings (`?service=` / `?database=` / `?databaseSchema=` / `?parent=` / `?apiCollection=` / `?spreadsheet=` / `?testSuite=`) compile to `<table>.fqnHash LIKE 'prefix%'` via ListFilter.getFqnPrefixCondition. The unique B-tree on `fqnHash` uses default `text_ops` opclass and the column inherits the database default collation (`en_US.UTF-8` on managed Postgres / RDS), neither of which lets the planner satisfy LIKE prefix from the index. Cold count(*) and the page query both fall back to a parallel seq scan over the JSONB heap — measured at ~3s on a ~580k-row storage_container_entity even after VACUUM/ANALYZE tuning and an RDS upsize. The unfiltered listing (`?limit=15`) clears the same dataset in ~215ms because it uses `idx_storage_container_entity_deleted_name_id` from 1.8.2, which the LIKE predicate cannot. Append a `text_pattern_ops` partial index on `fqnHash` for every entity table that hits getFqnPrefixCondition (24 tables: chart_entity through worksheet_entity). The `text_pattern_ops` opclass supports LIKE prefix regardless of column collation, switching the cold count(*) plan from parallel seq scan to bitmap index scan. MySQL is unaffected: every entity-table `fqnHash` column already ships with `CHARACTER SET ascii COLLATE ascii_bin`, a binary collation that lets the existing unique B-tree answer LIKE prefix predicates directly. The MySQL counterpart gets a documentation-only comment explaining the asymmetry so the next migration audit doesn't have to re-derive it. |
||
|
Sriharsha Chintalapani
|
ecc4b17579
|
Redis caching for container ancestors and children-page (#27858)
* Cache resolved ancestor chains in Redis
The /containers/name/{fqn}/ancestors endpoint runs on every detail-page
render to populate breadcrumbs. The resolution itself is one indexed
findReferencesByFqns call (already slim) plus FQN string walking, but the
DB round-trip and JSON deserialization are repeated for every navigation.
Bundle this behind Redis with the same shape as CachedReadBundle.
Cache key: om:anc:container:{fqnHash} → JSON List<EntityReference>, TTL =
entityTtlSeconds (default 5 min).
Invalidation:
- Writer drops its own key on update/delete (EntityRepository.invalidateCache)
- Cross-instance: the existing CacheInvalidationPubSub handler now also
drops the ancestors key for the published FQN.
- Renames are self-healing: the new FQN is a different key, the old key
TTL-expires.
- Display-name drift on a remote ancestor is bounded by TTL — acceptable
since breadcrumb metadata is cosmetic.
The cache is wired into ContainerRepository.getAncestors only — generalising
to other hierarchical entity types is straightforward when more /ancestors
endpoints land.
|
||
|
Ram Narayan Balaji
|
368fae160b
|
Revert "Feature #18173: Version API Improvements" (#26307) (#27837)
Some checks are pending
Integration Tests - MySQL + Elasticsearch / Detect Changes (push) Waiting to run
Integration Tests - MySQL + Elasticsearch / integration-tests-mysql-elasticsearch (push) Blocked by required conditions
Integration Tests - PostgreSQL + OpenSearch / Detect Changes (push) Waiting to run
Integration Tests - PostgreSQL + OpenSearch / integration-tests-postgres-opensearch (push) Blocked by required conditions
Java Checkstyle / java-checkstyle (push) Waiting to run
Maven Collate Tests / maven-collate-ci (push) Waiting to run
OpenMetadata Service Unit Tests / Detect Changes (push) Waiting to run
OpenMetadata Service Unit Tests / openmetadata-service-unit-tests (mysql) (push) Blocked by required conditions
OpenMetadata Service Unit Tests / openmetadata-service-unit-tests (postgresql) (push) Blocked by required conditions
OpenMetadata Service Unit Tests / k8s_operator-unit-tests (push) Blocked by required conditions
OpenMetadata Service Unit Tests / openmetadata-service-unit-tests-status (push) Blocked by required conditions
Publish Package to Maven Central Repository / publish-maven-packages (push) Waiting to run
* Revert "Feature #18173: Version API Improvements, Last x versions order by desc, versions from specific timeline, versions for specific metadata changes, sdk support and UI integration (#26307)"
This reverts commit
|
||
|
sonika-shah
|
52548550e8
|
fix migration: update legacy relatedTerms in glossaryTerm version history after the glossary term realtion changes (#27770)
* fix: strip stale relatedTerms from glossary term version snapshots Extends PR #26586. That fix cleaned glossary_term_entity but not the version snapshots in entity_extension, so GET /versions/{v} still 500s on any pre-1.13 term whose relatedTerms had legacy shape: UnrecognizedPropertyException: Unrecognized field "id" (class TermRelation, has only "term" and "relationType") Predicate matches only legacy snapshots — first item has bare `id` (EntityReference) instead of `term` (TermRelation). Skips correctly- shaped snapshots written on 1.13+. Stripping is safe: relatedTerms is loaded from entity_relationship at read time post-#25886. * v1130: transform legacy relatedTerms in version snapshots instead of stripping Replace the SQL UPDATE that stripped relatedTerms from entity_extension version snapshots with a Java migration that wraps each legacy EntityReference[] item as TermRelation[] (term + relationType="relatedTo"). Version reads deserialize entity_extension JSON directly without rehydrating from entity_relationship, so a strip would lose history per version. The transform preserves it. Designed for tables with millions of rows: keyset paginated by PK (id, extension), batched updates, idempotent on re-run. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(mysql): remove leftover entity_extension strip in v1130 post-migration The previous edit added the comment pointer above the legacy UPDATE entity_extension SET json = JSON_REMOVE(... '$.relatedTerms') block without removing it. On MySQL that SQL would have stripped relatedTerms from version snapshots BEFORE the Java transform runs, defeating the migration and losing related-term history. Postgres was already correct. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Eugenio
|
88c44502ae
|
feat: Add auto-classification support for storage service containers (#26495)
Some checks failed
Java Checkstyle / java-checkstyle (push) Waiting to run
Maven Collate Tests / maven-collate-ci (push) Waiting to run
OpenMetadata Service Unit Tests / Detect Changes (push) Waiting to run
OpenMetadata Service Unit Tests / openmetadata-service-unit-tests (mysql) (push) Blocked by required conditions
OpenMetadata Service Unit Tests / openmetadata-service-unit-tests (postgresql) (push) Blocked by required conditions
OpenMetadata Service Unit Tests / k8s_operator-unit-tests (push) Blocked by required conditions
OpenMetadata Service Unit Tests / openmetadata-service-unit-tests-status (push) Blocked by required conditions
Integration Tests - MySQL + Elasticsearch / Detect Changes (push) Has been cancelled
Integration Tests - PostgreSQL + OpenSearch / Detect Changes (push) Has been cancelled
Publish Package to Maven Central Repository / publish-maven-packages (push) Has been cancelled
Integration Tests - PostgreSQL + OpenSearch / integration-tests-postgres-opensearch (push) Has been cancelled
Integration Tests - MySQL + Elasticsearch / integration-tests-mysql-elasticsearch (push) Has been cancelled
* Add schema support for container auto-classification Extend container entity schema to support sample data storage, enabling PII detection and classification workflows on storage service containers. Changes: - Add sampleData field to container.json for storing sample data - Create storageServiceAutoClassificationPipeline.json schema defining configuration for storage service auto-classification pipelines - Update workflow.json to include StorageServiceAutoClassificationPipeline as a supported pipeline type This provides the schema foundation for running auto-classification workflows on S3, GCS, and other storage service containers. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Add backend support for container sample data and classification Implement Java backend functionality to handle sample data ingestion, storage, and PII masking for container entities. Changes: - ContainerRepository: Add sample data retrieval and storage operations - EntityRepository: Extend sample data support to container entities - ContainerResource: Add REST endpoint for container sample data ingestion - PIIMasker: Extend PII masking to support container entities This enables the backend to process and store sample data from storage service containers and apply PII masking rules during data retrieval. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Extend classifiable entity types to include containers Add Container to the ClassifiableEntityType union, enabling PII detection and auto-classification workflows to process storage service containers alongside database tables. Changes: - Update ClassifiableEntityType from Table-only to Union[Table, Container] - Import Container entity type - Update module docstring to reflect current support This type extension allows the PII processor to handle both database tables and storage containers uniformly. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Add container sample data ingestion to OpenMetadata API Implement container-specific API mixin for sample data operations and integrate it into the main OpenMetadata client. Changes: - Add OMetaContainerMixin with ingest_container_sample_data method - Handle binary data encoding (base64) and serialization errors - Register mixin in OpenMetadata class hierarchy - Mirror table sample data ingestion patterns for consistency This provides the Python API layer for ingesting sample data from storage service containers into OpenMetadata. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Implement storage service samplers for S3 and GCS Add sampler implementations for storage services to extract sample data from structured containers (Parquet, CSV) for auto-classification. Changes: - Create base StorageSamplerInterface for storage service sampling - Implement S3Sampler for AWS S3 containers with structured file support - Implement GCSSampler for Google Cloud Storage containers - Support column extraction and data sampling for structured formats - Handle dataModel-based column definitions from containers Storage samplers read container metadata, fetch file contents, and generate sample datasets for downstream PII detection. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Update PII processor to support container entities Extend the base PII processor to handle both Table and Container entities with unified column extraction logic. Changes: - Add _get_entity_columns helper to extract columns from Table or Container - Handle Container entities with optional dataModel.columns structure - Improve column matching with safe fallback for missing columns - Use generic entity reference in error reporting - Add early return when entity has no columns to process This enables PII detection to run on storage containers the same way it processes database tables. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Add storage service support to sampler processor Extend the sampler processor to handle both database and storage service entities with appropriate sampler class selection. Changes: - Detect service type from source config (Database vs Storage) - Import StorageServiceAutoClassificationPipeline - Handle both Table and Container entity types in _run method - Add column validation for Container entities (via dataModel.columns) - Create storage-specific sampler interfaces for S3 and GCS - Update sampler_interface to support Container entities - Improve error messages with entity type context The processor now dynamically selects database or storage samplers based on the pipeline configuration type. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Add storage fetcher strategy for container classification Implement fetcher strategy pattern for storage services to retrieve containers for auto-classification workflows. Changes: - Add StorageFetcherStrategy to handle storage service entity fetching - Update EntityFetcher to select appropriate strategy based on service type - Support both DatabaseService and StorageService in strategy selection - Import StorageService type for service detection - Improve error messages with specific service type information The fetcher now dynamically creates database or storage-specific strategies to retrieve entities based on pipeline configuration. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Register auto-classification pipeline in storage service specs Add AutoClassification pipeline support to S3 and GCS storage service specifications, enabling UI and workflow registration. Changes: - Add AutoClassification to S3ServiceSpec supported pipelines - Add AutoClassification to GCSServiceSpec supported pipelines - Import StorageServiceAutoClassificationPipeline in both specs This registers the auto-classification workflow type for storage services in the ingestion framework's service registry. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Add container support to metadata sink and patch operations Extend metadata sink and patch mixin to handle container entities, enabling sample data ingestion and tag updates for containers. Changes: - Add Container to MetadataRestSink entity type handling - Implement container sample data ingestion in sink._run - Add Container to PatchMixin tag operations - Import Container entity type in both modules This completes the metadata ingestion pipeline by allowing the sink to persist sample data and classification tags for container entities. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Update classification workflow for storage service support Extend the auto-classification workflow to handle both database and storage service pipelines with unified step orchestration. Changes: - Import StorageServiceAutoClassificationPipeline - Add type checking for both Database and Storage pipeline configs - Remove unnecessary cast, use direct type checks - Add validation warning for unsupported config types - Preserve enableAutoClassification flag behavior for both types The workflow now supports running PII detection and classification on both database tables and storage containers based on config type. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Add unit tests for container classification components Add test coverage for container-specific fetcher and sampler components. Changes: - Add test_container_fetcher.py for StorageFetcherStrategy tests - Add test_container_sampler_processor.py for container sampler tests Tests validate: - Storage service fetcher strategy selection and instantiation - Container sampler processor initialization and execution - Proper handling of Container entities vs Table entities 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Reorganize integration tests by entity type Restructure auto-classification integration tests into separate directories for databases and containers to improve organization. Changes: - Move database classification tests to databases/ subdirectory - Move conftest.py, init.sql, and test_tag_processor.py into databases/ - Container tests already organized in containers/ subdirectory - Remove old flat test structure This organization makes it clearer which tests target database entities vs storage container entities in classification workflows. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Properly retrieve sample data * Update generated TypeScript types * Apply Gitar bot * Fix tests * feat: Add supportsProfiler to storage connection schemas Add supportsProfiler field to storage connection schemas (S3, GCS, ADLS, Custom Storage) to enable auto-classification pipeline support for storage services. This aligns with the backend changes in PR #26495 that added container auto-classification functionality. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * feat: Add UI support for storage service auto-classification - Update IngestionWorkflowUtils to route storage services to storage-specific auto-classification schema - Modify getSupportedPipelineTypes to filter pipeline types based on service category (storage services only show AutoClassification, not Profiler) - Update AddIngestionButton to pass serviceCategory parameter - Add unit test to verify storage services only get AutoClassification option This enables users to configure and run auto-classification agents on storage services (S3, GCS, ADLS) for PII detection on containers. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * fix: Add BucketArn field to S3BucketResponse model AWS S3 API now returns a BucketArn field in list_buckets() responses. Add this optional field to prevent Pydantic extra_forbidden validation errors. Error: BucketArn Extra inputs are not permitted [type=extra_forbidden] 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * fix: Add Container permissions to AutoClassificationBotPolicy Add Container entity permissions to AutoClassificationBotPolicy to allow the autoClassification-bot to apply tags and sample data to storage containers. Previously, the bot only had permissions for Table entities, causing permission denied errors when running auto-classification on storage services. Changes: - Add Container rule with EditAll and ViewAll operations to policy seed data - Create migrations for MySQL and PostgreSQL to update existing installations Error fixed: Principal: CatalogPrincipal{name='autoclassification-bot'} operations [EditTags] not allowed 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Update generated TypeScript types * fix: Add fallback for storage service type detection in sampler Add fallback logic to detect storage services by source type name when the pipeline config type check fails. This handles cases where the Airflow environment might not have the updated schema/package with StorageServiceAutoClassificationPipeline. Changes: - Add fallback detection for s3, gcs, azuredatalake, customstorage - Add debug logging for service type detection - Preserve primary instanceof check for proper type detection This fixes the "No module named 'metadata.ingestion.source.database.gcs'" error when running storage auto-classification pipelines. * Guide to support new entities in classification agent * docs: Update auto-classification guide with debugging learnings Add critical troubleshooting information discovered during container classification debugging: 1. storeSampleData defaults to false - Sample data NOT ingested unless explicitly enabled - Document why this is by design (avoid large datasets) - Add troubleshooting steps to verify flag is set 2. Service type detection fallback pattern - Explain why fallback is needed (Airflow package caching) - Show complete implementation with source type lists - Add debug logging pattern 3. Troubleshooting section - Sample data not appearing: check storeSampleData, database, logs - Module import errors: service type detection issues - PII tags not applied: config and data issues 4. Common pitfalls additions - Emphasize storeSampleData default value - Service type detection in cached environments These updates reflect real debugging scenarios and will help future developers avoid the same issues. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Apply gitar bot suggestions * Fix suggestions, linting, and SonarCloud issues * More gitar bot suggestions * Fix compile error * Fix linting * Fix broken tests * Fix unorganized import * Improve config parsing This is so that we rightly discover polymorphic properties of `source` when the config does not provide enough fields for Pydantic to correctly discriminate between models (e.g: confusing database source config with storage source config) * Gitar bot comment * Fix s3 source test * Apply comments from reviews * Extract cantidate column logic in samplers * Fix tests * Fix container customization test --------- Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> |
||
|
Sriharsha Chintalapani
|
51ecf4502f
|
Task redesign (#25894)
* Task Redesign: Add Task entity & tests * Task Redesign: Add Task entity & tests * Task Redesign: Add Permissions checks for Task APIs * Task UI changed to the new APIs * Migrate UI and APIs to new tasks system inlcuding suggestions * Add Suggestions integration * Activity Feed Refactor * ActivityFeed -> ActivityStream publisher * Activity Feed redesign * Activity Feed redesign, adding tests * Incident Manager update * Migrate Incidents to new tasks * Migrate Incidents to new tasks * Update generated TypeScript types * Update generated TypeScript types * feat(tasks): add domain-aware task cutover and workflow v2 migration * test(tasks): cover domain filters and task feed visibility flows * Address comments * Fix workflow tests to use new Task entity API and fix UserApprovalTaskV2 candidate transformation Migrated 9 WorkflowDefinitionResourceIT tests from legacy Feed/Thread API to the new Task entity API (UserApprovalTaskV2 creates Task entities, not Thread entities). Fixed a bug in UserApprovalTaskV2 where candidates were passed as raw EntityReferences instead of being transformed into users/teams FQN arrays for SetApprovalAssigneesImpl. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Fix tests * refactor: stabilize task entity workflows * refactor: finish task entity cutover and activity migration * refactor: migrate legacy thread feed during cutover * refactor: split legacy thread rename and archive migrations * Merge main; fix tests * Update generated TypeScript types * feat: advance task redesign through phase 2 * Merge main; fix tests * Update generated TypeScript types * Fix failing tests * Update generated TypeScript types * fininsh phase 6 of the design, configurable task forms * Update generated TypeScript types * Update generated TypeScript types * Fix linting * Address gitar comments * Address gitar comments * Fix build * Address giar comments * fix build * Add task custom forms * Fix tests * Address tests * Apply UI lint autofixes * Fix tess * Fix linter * Fix task patching * Fix tests * Fix playwright tests * fix java checkstyle * Add python sdk support for tasks, annoucements * Fix playwright tests * Fix playwright tests * Fix playwright tests * Fix python tests * Fix python tests * Fix linting workflows * fix pycheck * fix pycheck * Fix tests * Fix build * Address deviations from main and fix tests * Fix integration tests * Fix integration tests * Fix integration tests * Update generated TypeScript types * Fix Playwright tests * Fix Playwright tests * feat(incident): wire incident manager to task-first architecture (#27369) * feat(incident): wire incident manager to task-first architecture Connect the incident manager to the task redesign so it works end-to-end: resolve data persistence, backward transitions, reopen from resolved, and incident discovery via TCRS. * Update generated TypeScript types * refactor: single-query incident task lookup with parameterized statuses Replace two sequential queries (Open, InProgress) in getOrCreateIncident with one findByAboutAndTypeAndStatuses query using @BindList for status IN (...). --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> * Fix Playwright tests * Update generated TypeScript types * Fix linter * Fix tests * Fix tests * Fix checkstyle * Fix tests * Fix checkstyle * Update FeedResourceIT.java * Update TableRepository.java * fix tests * Update ActivityFeedProvider.tsx * fix tests * fix tests * Address Task comments * Fix unit test * Fix the feed summary panel showing on landing page * Fix comment functionality * Fix pytests * Fix failing playwright tests * Fix test flakiness * Fix ui-checkstyle * Fix advanced search spec failure * Fix playwright tests Co-authored-by: Copilot <copilot@github.com> * Fix checkstyle * Fix the flaky tests Co-authored-by: Copilot <copilot@github.com> * fix checkstyle * Reduce the workflow polling * Update generated TypeScript types * skip failing tests Co-authored-by: Copilot <copilot@github.com> * Fix ui-checkstyle --------- Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Aniket Katkar <aniketkatkar97@gmail.com> Co-authored-by: IceS2 <pablo.takara@getcollate.io> Co-authored-by: karanh37 <karanh37@gmail.com> Co-authored-by: Karan Hotchandani <33024356+karanh37@users.noreply.github.com> Co-authored-by: Copilot <copilot@github.com> |
||
|
Sriharsha Chintalapani
|
e4d3e423e1
|
Feature #18173: Version API Improvements, Last x versions order by desc, versions from specific timeline, versions for specific metadata changes, sdk support and UI integration (#26307)
* Feature #18173: Improve Version API, through paginatio, get x latest versions, specifict time, specific metadata changes * Feature #18173: Version API Improvements, Last x versions order by desc, versions from specific timeline, versions for specific metadata changes, sdk support and UI integration * Update generated TypeScript types * address comments * fix py check * Address comments * Address comments * Fix tests * Fix tests * Fix tests * Better way to lookup versions * Fix pytests * Fix tests * Address comments * chore(migrations): move version API schema additions from 1.13.0 to 1.12.7 Moves the PR's new entity_extension columns (versionNum, changedFieldKeys), indexes, and backfill scripts from the 1.13.0 migration directory into a new 1.12.7 directory. Keeps 1.13.0 identical to upstream main; only this PR's additions land in 1.12.7. Also updates MigrationSqlStatementHashTest to exercise the relocated files. * fix(versions): address CI failures and review feedback - testAPI.test.ts: update getTestCaseVersionList mock expectation to include the new params argument (APIClient.get is called with { params } since the function now supports limit/offset/fieldChanged). - PaginatedVersionHistory.spec.ts: replace banned networkidle waits and waitForSelector with web-first assertion on version-button visibility (satisfies playwright/no-networkidle and playwright/no-wait-for-selector). - EntityVersionTimeLine.tsx: implement infinite scroll via IntersectionObserver on a sentinel element at the bottom of the version list. Hooks up the onLoadMore/hasMore/isLoadingMore props that were in the interface but previously unused. - EntityVersionPage.component.tsx: fix stale-closure bugs in fetchMoreVersions (gitar-bot review). Use versionListRef for currentOffset and isLoadingMoreRef to gate concurrent invocations so IntersectionObserver double-firing does not cause duplicate appends. - EntityResource.java: accept offset > 0 with default limit when no fieldChanged is provided, so pagination params are no longer silently ignored (Copilot review). - datamodel_generation.py: raise explicit errors if generated files or expected replacement targets are missing, instead of silently succeeding when the generator output drifts (Copilot review). * fix(checkstyle): format Java, ESLint/Prettier on UI, relax datamodel_generation strict check - Java: spotless:apply on EntityResource.java (line-break formatting). - Python: relax datamodel_generation.py DIRECT_IMPORT_FIXES check — replacement targets are alternative forms the generator may or may not emit. Only require the final marker ('from .paging import Paging') is present after replacements; the prior strict per-target check broke 'make generate'. - UI lint: organize-imports, ESLint --fix, Prettier on all version-related files touched by the PR (resolves lint-src + lint-playwright CI checks). - EntityVersionTimeLine: guard IntersectionObserver effect with isLoadingMore so the observer is torn down while a fetch is in flight (Copilot review). - EntityVersionTimeline.test.tsx: add unit tests covering sentinel rendering conditions (hasMore, onLoadMore) and the isLoadingMore observer-guard (Copilot review). * fix(ui-checkstyle): prettier+eslint on EntityVersionTimeline.test.tsx Collapse import line and reorder JSX props (callbacks last) per repo lint rules. Reruns ui-checkstyle-changed caught these in the new test file from the previous commit. * test(playwright): address @aniketkatkar97 review on PaginatedVersionHistory spec - Add waitUntil: 'domcontentloaded' to every page.goto() call. - Wait for loaders (waitForAllLoadersToDisappear) before asserting the version-button to avoid racing the initial entity render. - Replace the manual { timeout: 15_000 } on versionSelectors.nth(1) with an explicit waitForResponse on the second paginated /versions call (offset > 0). This deterministically synchronises on the infinite-scroll fetch instead of a wall-clock timeout. * fix: address Copilot review — one-shot observer + local SQL splitter 1. EntityVersionTimeLine.tsx: call observer.unobserve(entry.target) as soon as the sentinel first intersects so onLoadMore fires only once per attached observer. The effect reattaches a fresh observer after isLoadingMore flips back to false, so subsequent pages still load — we just no longer rely on the parent's in-flight ref as the sole stopgap against repeated fires for the same page. 2. MigrationSqlStatementHashTest.java: replace Flyway's non-public org.flywaydb.core.internal.* parser classes with a small, local SQL statement splitter. Handles line (--) and block comments, single-, double-, and backtick-quoted strings, backslash escapes, and doubled- quote escapes. Removes a brittle dependency on Flyway internals that could break on upgrades. Tested: - mvn test -pl openmetadata-service -Dtest=MigrationSqlStatementHashTest → 2 tests pass. - yarn test EntityVersionTimeline.test.tsx → 8/8 tests pass. --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: sonika-shah <sonika-shah@users.noreply.github.com> Co-authored-by: mohitdeuex <mohit.y@deuexsolutions.com> Co-authored-by: sonika-shah <sonikashah94@gmail.com> Co-authored-by: Mohit Yadav <105265192+mohityadav766@users.noreply.github.com> |
||
|
Teddy
|
47c88d49ce
|
ISSUE #3031 - Dynamic Sampling Config (#27184)
* feat: move flat sampling to sampling config + dynamic sampling option * feat: move flat sampling on the backend to sample profile conifg object * feat: fix circular import * feat: align UI with new profiler config * feat: fix json schema * feat: align python imports with new schema path * feat: update migration to look at extension * feat: remove enable * feat: remove enable * feat: added titles to sample config * feat: generated ts classes * feat: addressed comments * feat: change sample config instantiation to match new structure * feat: removed backward compatible fields * feat: ran java linting * UI fixes, tests and locale changes * fix failing test * fix ui check style * fix failing profiler test * feat: fix ci failures * feat: generated ts classes * feat: fix ci failure * fix: failing ci * address comments * fix failing test * fix: ci failure --------- Co-authored-by: Harshit Shah <dinkushah169@gmail.com> |
||
|
Mohit Yadav
|
c2e6d907dd
|
fix(lineage): service nodes appearing in entity lineage view and empty By Service view (#27258)
* fix(lineage): prevent pipeline annotation inheritance in service/domain/dataProduct lineage and add pipeline service edges
Bug #1: Service nodes (e.g., DatabaseService, MessagingService) were incorrectly appearing in
entity-level lineage views. Root cause: getOrCreateLineageDetails() in addServiceLineage(),
addDomainLineage(), and addDataProductsLineage() was copying the pipeline annotation from
entity-level LineageDetails to service/domain/dataProduct-level LineageDetails. This caused
service entities to have upstreamLineage.pipeline.fqnHash set in their Elasticsearch documents,
making them match the PIPELINE_AS_EDGE_KEY query during BFS traversal and incorrectly appear
alongside actual data assets. Fix: add .withPipeline(null) on each service/domain/dataProduct
LineageDetails object to strip the pipeline annotation before persisting.
Bug #2: "By Service" view was empty when viewing lineage for pipeline entities that were stored
as edge annotators (Case B: table → topic with pipeline=flink_pipeline in LineageDetails) rather
than as actual nodes (Case A). Root cause: addServiceLineage() only created database_service →
kafka_service edges but no edges involving flink_pipeline_service. Fix: add addPipelineServiceEdges()
called from addServiceLineage() that creates fromService → pipelineService and pipelineService →
toService edges when a pipeline annotation exists in the entity-level lineage details.
Also add unit tests covering both fixes to prevent regression.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(lineage): add migration to remove pipeline annotation from service/domain/dataProduct lineage edges
The previous fix (
|
||
|
sonika-shah
|
077982c348
|
Move ontology/glossary relation migration from 1.14.0 back to 1.13.0 (#27431)
* Move ontology/glossary relation migration from 1.14.0 back to 1.13.0 Ontology feature will ship in 1.13.0, not 1.14.0. Move the glossary term relation migrations (relationType backfill, settings insert, stale relatedTerms strip, conceptMappings backfill) back to the 1.13.0 postDataMigrationSQLScript for both MySQL and PostgreSQL. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Restore empty 1.14.0 SQL migration files for Java migration framework The V114 MigrationUtil.java package requires the 1.14.0 migration directory to exist with SQL files for the migration to be picked up. Keep them as empty files (matching convention of other versions with no post-data SQL). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Add schemaChanges.sql and comment all 1.14.0 SQL migration files Add both schemaChanges.sql and postDataMigrationSQLScript.sql for mysql and postgres with a comment explaining the directory is required for the V114 Java migrations to be picked up by the migration framework. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Fix missing trailing newline in postgres postDataMigrationSQLScript Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * address feedback --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: Karan Hotchandani <33024356+karanh37@users.noreply.github.com> |
||
|
Ram Narayan Balaji
|
35ede8fe5f
|
fix(migration): revert webhook authType back to secretKey in v1126 and remove broken v1125 migration (#27427)
* fix(migration): add v1126 reverse migration to revert webhook authType back to secretKey * fix(migration): remove migrateWebhookSecretKeyToAuthType from v1125 migration * fix(test): remove migrateWebhookSecretKeyToAuthType references from v1125 migration tests * fix(migration): address copilot review comments on v1126 migration * fix(migration): case-insensitive bearer check and verify JSON content in v1126 tests * fix(migration): remove unused constants from v1125 and add postgres path + SQL verification to v1126 tests |
||
|
Sriharsha Chintalapani
|
bb0daa180e
|
RDF, cleanup relations and remove unnecessary bindings, add distributed mode for RDF reindex (#26902)
* RDF, cleanup relations and remove unnecessary bindings, add distributed mode for RDF reindex * Update generated TypeScript types * Address comments from copilot * Update generated TypeScript types * fix test issues * Fix minor UI bugs * Add the missing filters * Fix RDF export API error * Add export functionality * Fix ui-checkstyle * Fix java checkstyle * Fix unit tests * Fix and increase the coverage for KnowledgeGraph.spec.ts * Fix tests * Remove rdf as default in playwright and local docker * fix ui-checkstyle * Address comments * Potential fix for pull request finding 'CodeQL / Artifact poisoning' Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> * Address copilot comments * Address copilot comments * FIx tests * FIx docker * Update openmetadata-service/src/main/java/org/openmetadata/service/apps/bundles/rdf/distributed/DistributedRdfIndexCoordinator.java Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Address copilot review comments: license headers, JSON escaping, type safety, border-color, stop semantics Agent-Logs-Url: https://github.com/open-metadata/OpenMetadata/sessions/c026e52e-162b-4c9a-9874-43791d4aaac1 Co-authored-by: harshach <38649+harshach@users.noreply.github.com> * Show error toast for unsupported export format in KnowledgeGraph Agent-Logs-Url: https://github.com/open-metadata/OpenMetadata/sessions/c026e52e-162b-4c9a-9874-43791d4aaac1 Co-authored-by: harshach <38649+harshach@users.noreply.github.com> * Fix docker * Fix docker for playwright * Fix docker for playwright * Fix tests * Fix tests * Fix docker * Fix docker * Fix glossary and pagination spec flakiness * update the missing translations * Fix docker * Fix docker * Fix integration test * Fix fuseki not starting * Fixed the run local docker script * worked on comments * Fix flakiness in knowledge graph tests * Fix checkstyle --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Aniket Katkar <aniketkatkar97@gmail.com> Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: harshach <38649+harshach@users.noreply.github.com> |
||
|
Rajdeep Singh
|
5e1416447f
|
fix(sampler): Respect randomizedSample flag at 100% percentage sampling (#26966)
* fix(sampler): respect randomizedSample flag at 100% percentage sampling When profileSample is 100% with PERCENTAGE type, the sampler short-circuits and returns the raw dataset without any randomization, even when randomizedSample is True (the default). Split the combined condition so: - No profileSample set -> return raw dataset (no sampling configured) - 100% PERCENTAGE + randomizedSample=False -> return raw dataset (optimization) - 100% PERCENTAGE + randomizedSample=True -> go through normal sampling path which applies RandomNumFn/df.sample for proper row shuffling Fixes #21304 * Address review: use 'is False' for Optional[bool] and add unit tests - Fix randomizedSample check from 'not' to 'is False' in both SQASampler and DatalakeSampler to correctly handle None (Optional[bool] default=True) - Add unit tests verifying 100%% PERCENTAGE behavior for randomizedSample values True, False, and None * Add ORDER BY on random column in fetch_sample_data for true randomization The get_dataset() fix ensures 100% PERCENTAGE + randomizedSample routes through get_sample_query() which produces a CTE with a random column. Now fetch_sample_data() detects that column and applies ORDER BY before LIMIT, so each call returns a different subset of rows. Also add real-DB integration tests using SQLite for the 100% PERCENTAGE edge case (True, False, None). * Address review: remove stale comment, unused import, add return assertions * Apply suggestion from @Copilot Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Address review: move ORDER BY to get_sample_query, clean up fetch_sample_data - Move ORDER BY rnd.c.random into get_sample_query() PERCENTAGE branch, gated on randomizedSample is not False (mirrors ABSOLUTE branch pattern) - Revert fetch_sample_data() to original: remove ds_columns variable, random_column detection, and ORDER BY logic (ordering now handled in CTE) - Remove duplicate assertions in DatalakeSampler100Pct tests * Address review: None defaults to False for randomizedSample Per TeddyCr's feedback, randomization is computationally heavy and should not be the default. Changed from 'is False'/'is not False' to truthiness checks so None (unset) behaves the same as False. Only explicit randomizedSample=True triggers ORDER BY and skips the 100% fast path. This is consistent with the ABSOLUTE branch which already uses truthiness checks. * Fix integration test: None should skip sample_query (matches truthiness semantics) * fix(tests): update BigQuery view sampling expected queries with ORDER BY BigQuery views fall through to SQASampler.get_sample_query() which now adds ORDER BY rnd.random when randomizedSample is enabled. Update the expected SQL strings in test_sampling_for_views and test_sampling_view_with_partition to match. * refactor: use explicit is False for randomizedSample checks Address review comments: SampleConfig.randomizedSample defaults to True, so only an explicit False should disable randomization. Using is False / is not False instead of truthiness ensures None follows the model default (enabled) rather than being incorrectly treated as disabled. * ci: re-trigger checks after SIGSEGV flake * refactor: only explicit True randomizes, add non-determinism tests * test: increase non-determinism iterations to reduce flakiness * chore: added randomize as false * fix: align randomizedSample defaults with schema (false) * fix: remove ORDER BY from BigQuery test expectations BigQuery sampling tests create SampleConfig without setting randomizedSample, which now defaults to False. Since ORDER BY is only added when randomizedSample is True, the expected query strings should not include ORDER BY. Also fix inaccurate docstring in test_sample.py. * test: increase non-determinism test iterations to reduce flakiness Increase fetch_sample_data loop from 10 to 20 iterations to further reduce the theoretical probability of a false failure in the randomized ordering test. --------- Co-authored-by: Teddy <teddy.crepineau@gmail.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> |
||
|
sonika-shah
|
733921f510
|
Fix: align glossary term relation type colors with design system (#27142)
* Fix: align glossary term relation type colors with design system System-defined relation types (relatedTo, synonym, antonym, etc.) were initialized with old Ant Design palette colors (#1890ff, #722ed1, …) while the frontend RELATION_META constants had been updated to the new design system colors (#1570ef, #b42318, …). Because renderColorBadge used record.color (from the backend) unconditionally, the stale Ant Design colors were always displayed instead of the intended ones. - Frontend: renderColorBadge now treats RELATION_META as authoritative for system-defined types so the correct design-system color is always shown, regardless of what color value is stored in the backend. - Backend (SettingsCache.java): default colors updated for new installs. - DB migration (2.0.0): postDataMigrationSQLScript added for MySQL and PostgreSQL to update colors in existing deployments without touching user-added custom relation types. - Tests: unit tests for renderColorBadge color-resolution logic; integration test asserting all ten system-defined types return the expected hex values from the API. Fixes #openmetadata/OpenMetadata * Remove dev-only MySQL 2.0.0 migration script * Remove dev-only PostgreSQL 2.0.0 migration script * Fix: align glossary term relation settings colors and remove duplicate 1.13.0 migration; Remove glossary term relation migrations mistakenly re-added in 1.13.0 and update relation type colors in the 1.14.0 migration INSERT to use design system tokens instead of old Ant Design colors. * fix lint * add more test * address feedback * fix prettier formatting in test file Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * remove GlossaryTermRelationSettings test file from branch Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
||
|
Mohit Yadav
|
7693a5b04b
|
Update indexing schedule (#27204)
* Update schedule to weekly * Migration |
||
|
Suman Maharana
|
a06b7e74cc
|
Chore: Remove iceberg standalone connector (#26365)
* Chore: Remove iceberg standalone connector * add migration scripts * Update generated TypeScript types * py_format * address comments * Addressed changes * add tests * migrate to custom database * fix tests * fix tests * fix migrations * hard delete exising ingestion pipelines for iceberg * Update generated TypeScript types * Delete openmetadata-ui/src/main/resources/ui/src/generated/entity/services/ingestionPipelines/ingestionPipeline.ts * Delete openmetadata-ui/src/main/resources/ui/src/generated/entity/automations/workflow.ts * Delete openmetadata-ui/src/main/resources/ui/src/generated/api/automations/createWorkflow.ts * Delete openmetadata-ui/src/main/resources/ui/src/generated/api/services/ingestionPipelines/createIngestionPipeline.ts * Delete openmetadata-ui/src/main/resources/ui/src/generated/api/services/createDatabaseService.ts * Delete openmetadata-ui/src/main/resources/ui/src/generated/entity/automations/testServiceConnection.ts * Update generated TypeScript types * Update bootstrap/sql/migrations/native/1.13.0/mysql/postDataMigrationSQLScript.sql Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> |
||
|
Sriharsha Chintalapani
|
ed58077197
|
MCP services (#23623) | ||
|
Ram Narayan Balaji
|
b9d8c08b5b
|
Refactor(certification): store asset certification in tag_usage table (#26448)
* refactor(certification): store asset certification in tag_usage table Previously, asset certification was stored as a JSON blob directly on the entity row. This created a split system where the tag FQN lived in the entity JSON while tag metadata (name, description, style) had to be re-fetched from the tag table on every read. It also meant certification was invisible to the tag_usage propagation pipeline, so renaming a certification tag's FQN left stale data on certified entities. Certification is now stored in tag_usage alongside all other tags, using the metadata column to carry expiryDate (added to TagLabelMetadata schema). The entity's certification field remains the input/output surface, but tag_usage is now the source of truth. Key changes: Storage & retrieval - applyCertification() writes the certification tag into tag_usage on store - deleteCertificationTag() removes it from tag_usage on clear/replace - getCertification() reads from tag_usage filtered by the configured certification classification instead of parsing entity JSON - getTags() now strips certification-classification tags so they are surfaced exclusively through getCertification() Performance improvements - batchFetchCertification() rewritten to a single batch query on tag_usage by FQN hash instead of performing N individual tag lookups Tag update handling - handleTagEntityUpdate() reads the allowed classification from settings (no longer hardcoded) - correctly computes oldFQN on name change so Elasticsearch documents are found and updated using the correct key DAO & schema changes - deleteTagsByPrefixAndTarget() added to CollectionDAO for targeted certification tag removal - TagLabel mappers hardened against unknown metadata fields Migrations - v1123 migrations backfill existing entity JSON certifications into tag_usage so no data is lost during upgrade Tests - TagResourceIT updated to assert getCertification() instead of getTags(), since certification tags are intentionally excluded from the tags list * Update generated TypeScript types * chore: apply changes Co-authored-by: yan-3005 <yan-3005@users.noreply.github.com> * fix(certification): prevent updateTags() from clobbering cert tags written by updateCertification() * fix(certification): compute tagFQNHash per-segment in Java during migration and make applyCertification idempotent * Update generated TypeScript types * Fix: SQL-filtered cert batch fetch, remove double-delete, schema strict mode, ordinal bounds check, migration logging * Update generated TypeScript types * Fix Migration * Fix Migration * fix(certification): address Copilot review feedback on PR #26448 - Use exact field name comparison (FIELD_NAME.equals) instead of contains() in SearchRepository to avoid incorrect FQN-rename branch triggers when displayName changes - Log previously swallowed exception in getCertificationClassificationFromSettings() to improve observability of certification search propagation failures - Fix v1124 migration by building selectedIds inside the insert loop and skipping rows with null tagFQN, preventing UPDATE from removing certifications without corresponding tag_usage entries (avoids silent data loss) - Update integration test to rename tag name (not displayName) so it correctly validates the FQN-change regression from #26432 and asserts propagation to entity certification field and search index * fix(migration): fix v1124 certification migration correctness issues - Fix wrong version string in error messages: both mysql and postgres Migration.java logged "v1123" instead of "v1124" - Fix potential infinite loop: null-tagFQN rows were excluded from the INSERT but still counted in the return value (rows.size()), so when a full batch of 500 rows all had null tagFQN the loop never terminated. Fix by filtering null tagFQN at SQL level (WHERE tagFQN IS NOT NULL) and returning selectedIds.size() so the loop count reflects rows that were actually migrated Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(certification): fix missing tables in migration and optimize getCertification query - Add 6 missing entity tables to v1124 certification migration: file_entity, directory_entity, spreadsheet_entity, worksheet_entity, llm_model_entity, ai_application_entity — all define the certification field in their JSON schema; omitting them caused silent data loss on upgrade (certification stripped from JSON but never written to tag_usage) - Replace getCertification() full-tag-fetch with getCertTagsInternalBatch() so single-entity reads issue a targeted WHERE tagFQN LIKE query instead of fetching all tags and filtering in Java (consistent with the bulk path) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(certification): preserve appliedDate in migration and avoid appliedAt reset on unchanged cert - v1124 migration now extracts certification.appliedDate from entity JSON and inserts it as tag_usage.appliedAt, preserving the original certification timestamp instead of defaulting to migration time - applyCertification() now checks whether the existing certification tag matches the incoming one before doing delete+reinsert; if unchanged it returns early, preventing appliedAt from being reset on every entity write Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(certification): also compare expiryDate in applyCertification idempotency check The previous fix skipped delete+reinsert when tagFQN was unchanged, but this incorrectly swallowed expiryDate updates — re-certifying with the same tag but a new validity period would return early and never write the new expiryDate to tag_usage. Adding Objects.equals(expiryDate) to the guard ensures metadata-only changes are still persisted. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * test(certification): replace fixed sleeps with Awaitility polling in rename test Fixed sleeps are flaky under CI load and always waste time when indexing is faster. Replace both TimeUnit.SECONDS.sleep(2) calls and all subsequent search/entity assertions with Awaitility.await().untilAsserted() blocks (30s timeout, 1s poll interval) so the test waits exactly as long as needed. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(migration): include exception in certification migration warning log Pass the exception object to LOG.warn so the stack trace is available for diagnosing production migration failures. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * perf: cache getCertificationClassification() via SettingsCache Replace direct SystemRepository DB call with SettingsCache.getSettingOrDefault() (Guava LoadingCache, 3-min TTL) to eliminate repeated DB hits on every certification-related call in EntityRepository. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * skip the test * Added new column for certification and tier * nit * Add test for tier and certification * fix unit test * Fix Unit tests * Move Migrations to 1.12.5 and unit tests * Fix NPE, batch certification writes, and improve test coverage - Guard against null tagLabel in applyCertification to prevent NPE on malformed input - Replace per-entity applyCertification loop in storeRelationshipsInternal with applyCertificationBatch, reducing 3N DB calls to 2 (one batch DELETE + one batch INSERT via existing applyTagsBatchMultiTarget) - Add deleteTagsByPrefixAndTargets to TagUsageDAO as the batch variant of deleteTagsByPrefixAndTarget - Add tests for applyCertificationBatch paths, getTags cert filtering, and TagLabelWithFQNHash.toTagLabel to meet 90% new-code coverage threshold Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Add coverage tests for RowMappers, batchFetchCertification, and toTagLabel fallbacks - Add TagLabelMapper and TagLabelWithFQNHashMapper tests using mock ResultSet to cover the new metadata-parsing code paths in CollectionDAO - Add toTagLabel fallback tests for out-of-bounds enum ordinals covering the defensive conversion logic in TagLabelWithFQNHash - Add storeRelationshipsInternal single-entity overload test covering line 2322 - Add fetchAndSetFields tests to cover batchFetchCertification happy path and exception fallback path Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * resolved the linting issue * nit * fix lint issue --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Gitar <noreply@gitar.ai> Co-authored-by: yan-3005 <yan-3005@users.noreply.github.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Co-authored-by: Anujkumar Yadav <anujf0510@gmail.com> |
||
|
Ram Narayan Balaji
|
10cf2f9ea0
|
Move ontology/glossary relation migration from 1.13.0 to 1.14.0 (#26755)
The glossary term relation migration (relationType backfill, default glossaryTermRelationSettings insert, relatedTerms cleanup, conceptMappings backfill) was accidentally placed in the 1.13.0 migration scripts. This commit moves it to the correct 1.14.0 slot, restoring 1.13.0 to its original content (computeMetrics profiler pipeline cleanup only). Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> |
||
|
Ram Narayan Balaji
|
ee4f9316c1
|
Move Migration to 1.12.4 from 1.12.3 (#26629) | ||
|
sonika-shah
|
aff1343643
|
fix: strip stale relatedTerms from glossary_term_entity JSON to fix 500 on listAfter (#26586)
* fix: strip stale relatedTerms from glossary_term_entity JSON to fix 500 on listAfter Pre-1.13.0, relatedTerms was stored as EntityReference[] directly in the glossary_term_entity JSON column. PR #25886 changed relatedTerms to TermRelation[] and moved storage to entity_relationship table, but missed adding a migration to clean up the old EntityReference data still present in existing rows. When listAfter() deserializes the entity JSON, Jackson fails with: UnrecognizedPropertyException: Unrecognized field "id" (class TermRelation) The existing migration already backfilled entity_relationship rows with relationType="relatedTo", so stripping relatedTerms from entity JSON is safe — the data is already in entity_relationship and will be loaded from there. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> * fix: strip stale relatedTerms from glossary_term_entity JSON to fix 500 on listAfter Pre-1.13.0, relatedTerms was stored as EntityReference[] directly in the glossary_term_entity JSON column. PR #25886 changed relatedTerms to TermRelation[] and moved storage to entity_relationship table, but missed adding a migration to clean up the old EntityReference data still present in existing rows. When listAfter() deserializes the entity JSON, Jackson fails with: UnrecognizedPropertyException: Unrecognized field "id" (class TermRelation) The existing migration already backfilled entity_relationship rows with relationType="relatedTo", so stripping relatedTerms from entity JSON is safe — the data is already in entity_relationship and will be loaded from there. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: Ram Narayan Balaji <81347100+yan-3005@users.noreply.github.com> |
||
|
Vishnu Jain
|
6e93754a2f
|
Mcp oauth (#25391)
* Add OAuth MCP
* Implement internal OAuth flow for MCP with database
persistence
This commit implements a redirect-free OAuth flow for the OpenMetadata MCP
server that uses stored connector OAuth credentials internally, eliminating
the need for external browser redirects.
Key Features:
- Internal OAuth authorization using stored connector credentials
- Database persistence of OAuth tokens (survives container restarts)
- Automatic token refresh when expired
- PKCE support for authorization code flow
- OAuth discovery metadata endpoint (RFC 8414)
How It Works:
1. Admin performs one-time OAuth setup via /api/v1/mcp/oauth/setup
2. OAuth credentials (access token, refresh token) stored encrypted in database
3. MCP clients connect without browser - server uses stored credentials internally
4. Expired tokens automatically refreshed and re-persisted to database
Tested With:
- Snowflake OAuth (session:role:PUBLIC scope)
- Container restart verification (credentials persist)
- Automatic token refresh verification
* feat: Add MCP OAuth database persistence with repositories and DAOs
- Implement OAuthClientRepository, OAuthTokenRepository, OAuthAuthorizationCodeRepository
- Add DAO methods in CollectionDAO for OAuth entities
- Create database migration scripts for OAuth tables (oauth_client, oauth_access_token, oauth_refresh_token, oauth_authorization_code)
- Add Fernet encryption for tokens and client secrets
- Implement SHA-256 hashing for token lookups
- Add OAuth connector plugin system (Snowflake, Databricks)
- Add scope authorization and validation
- Update ConnectorOAuthProvider to use database persistence
- Add comprehensive tests for OAuth provider
* Add MySQL migration for MCP OAuth tables (v1.12.1)
- Create oauth_client, oauth_authorization_code, oauth_access_token, oauth_refresh_token tables
- Convert Postgres schema to MySQL syntax
- Add indexes for performance optimization
- Tables manually applied in this session, migration framework integration needed
* feat: Complete MCP OAuth implementation with critical fixes and MCP Inspector support
1. **Scope Validation Fix**
- Set validScopes to null in McpServer to skip validation for connector-based OAuth
- Modified RegistrationHandler to skip validation if validScopes is empty
- Fixes: Client registration error "Invalid scope: api://apiId/.default"
2. **Metadata Endpoint URLs**
- Fixed all OAuth discovery endpoints to include /mcp prefix
- Updated OAuthHttpStatelessServerTransportProvider endpoint construction
- Ensures proper OAuth metadata discovery
3. **Token Exchange Security**
- Added client_id validation during token exchange
- Added redirect_uri validation to prevent security vulnerabilities
- Load authorization code from database for validation
- Prevents authorization code interception attacks
4. **Time Unit Consistency**
- Fixed deleteExpired methods to use seconds instead of milliseconds
- Updated OAuthTokenRepository and OAuthAuthorizationCodeRepository
- Enables proper cleanup of expired tokens and codes
5. **Authorization Code Loading**
- Fixed loadAuthorizationCode to load all fields from database
- Populates AuthorizationCode object with clientId, redirectUri, codeChallenge
- Resolves: NullPointerException during token validation
6. **Connector Name Parameter Support**
- Added connectorName field to AuthorizationParams
- Extract connector_name from HTTP request in AuthorizationHandler
- Priority: connector_name parameter > state (if not random hash) > default
7. **Default Connector Fallback**
- Detect random hash in state parameter (64 hex chars for CSRF)
- Default to test-snowflake-mcp connector for MCP Inspector testing
- Enables MCP Inspector to work without manual URL modification
8. **MySQL Migration**
- Added MySQL schema changes for OAuth tables
- Matches PostgreSQL schema structure
- Tables: oauth_clients, oauth_authorization_codes, oauth_access_tokens, oauth_refresh_tokens
9. **Documentation Cleanup**
- Removed 12+ redundant and outdated documentation files
- Created single comprehensive MCP_OAUTH_IMPLEMENTATION.md
- Added .shell-fix-note for shell script compatibility guidance
10. **Test Script Organization**
- Organized test scripts into scripts/mcp-oauth-tests/
- Added test-default-connector.sh for testing with MCP Inspector
- Preserved all OAuth flow testing scripts
- McpServer.java - Disabled scope validation for connector OAuth
- RegistrationHandler.java - Skip empty validScopes
- AuthorizationHandler.java - Extract connector_name parameter
- AuthorizationParams.java - Added connectorName field
- ConnectorOAuthProvider.java - Default connector logic, loadAuthorizationCode fix
- OAuthHttpStatelessServerTransportProvider.java - Fixed endpoints, added validations
- OAuthTokenRepository.java - Fixed time unit to seconds
- OAuthAuthorizationCodeRepository.java - Fixed time unit to seconds
- CollectionDAO.java - OAuth DAO registration
- DatabaseServiceRepository.java - Database service queries
- OAuthRecords.java - Database record types
- Deleted: 15+ outdated documentation files
- Deleted: Unused auth provider (OpenMetadataAuthProvider.java)
- Deleted: Unused OAuth callback servlet
- Added: Single comprehensive documentation file
✅ OAuth flow working end-to-end
✅ Client registration, authorization, token exchange successful
✅ Database persistence for all OAuth entities
✅ MCP Inspector compatibility with default connector
✅ Snowflake OAuth credentials configured for testing
⚠️ MCP Inspector SSE connection error (under investigation)
- OAuth authentication completes successfully
- Issue is with MCP protocol SSE connection, not OAuth
Run MCP Inspector:
```bash
npx @modelcontextprotocol/inspector http://localhost:8585/mcp
```
Test with default connector:
```bash
./test-default-connector.sh
```
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
* fix: Add CORS preflight support and security fixes for MCP OAuth
## CORS Fix
Allow OPTIONS requests without authentication in McpAuthFilter to support
CORS preflight checks from web-based MCP clients.
This enables proper CORS flow:
1. Browser sends OPTIONS preflight
2. Server responds with CORS headers (200 OK)
3. Browser sends actual POST request with Authorization header
4. Server authenticates and processes request
Without this fix, OPTIONS requests were blocked with 401, preventing
web clients from connecting to MCP endpoints.
## Security Fixes
### Critical Security Issues Fixed:
1. **Sensitive Token Logging** (95% severity)
- Sanitize OAuth request parameters before logging
- Remove client_secret, code, code_verifier, refresh_token, access_token from logs
- Prevents credential leakage in log files
2. **Token Expiry Integer Overflow** (100% severity)
- Changed all expiry timestamps from int/Integer to long/Long
- Fixes 2038 problem (32-bit timestamp overflow)
- Updated: AccessToken, RefreshToken, AuthorizationCode, ConnectorOAuthProvider, OAuthTokenRepository
3. **Hardcoded Default Connector** (80% severity)
- Made default connector configurable via MCP_DEFAULT_CONNECTOR env var
- Defaults to null in production (requires explicit connector_name)
- Prevents unauthorized access to test credentials in production
4. **Missing Null Checks** (85% severity)
- Added validation for token refresh response fields
- Validates access_token and expires_in exist before use
- Added bounds checking for expires_in (max 1 year)
5. **Missing Input Validation** (75% severity)
- Added connector name format validation
- Only allows: a-z, A-Z, 0-9, _, - characters
- Prevents path traversal and injection attacks
## Documentation
- Moved MCP docs to organized structure: openmetadata-mcp/docs/
- Created openmetadata-mcp/README.md with foundation documentation
- Moved implementation guide and testing guide to docs/ directory
## Cleanup
- Removed development test scripts (scripts/mcp-oauth-tests/)
- Removed .shell-fix-note and test-default-connector.sh
- Kept only clean final test script: test-mcp-with-token.sh
Changes:
- openmetadata-mcp/src/main/java/org/openmetadata/mcp/McpAuthFilter.java: OPTIONS CORS support
- openmetadata-mcp/src/main/java/org/openmetadata/mcp/server/transport/OAuthHttpStatelessServerTransportProvider.java: Sanitized logging
- openmetadata-mcp/src/main/java/org/openmetadata/mcp/server/auth/provider/ConnectorOAuthProvider.java: Multiple security fixes
- openmetadata-mcp/src/main/java/org/openmetadata/mcp/McpServer.java: Configurable default connector
- openmetadata-mcp/src/main/java/org/openmetadata/mcp/auth/*.java: Long timestamps
- openmetadata-mcp/src/main/java/org/openmetadata/mcp/server/auth/repository/OAuthTokenRepository.java: Long timestamps
Testing:
- OAuth flow: ✅ Working with any OAuth-enabled connector
- MCP protocol: ✅ Working via HTTP POST with JWT
- Default connector: Configurable via MCP_DEFAULT_CONNECTOR env var
- General solution: Works with ANY connector with OAuth credentials
Test command:
export MCP_DEFAULT_CONNECTOR=test-snowflake-mcp # For testing only
./test-mcp-with-token.sh
* feat: MCP OAuth security hardening and production readiness
Implemented security improvements and production configuration for MCP OAuth:
- Added constant-time secret comparison to prevent timing attacks
- Implemented token logging sanitization to protect sensitive credentials
- Fixed timestamp overflow (Integer → Long) to prevent 2038 issues
- Added input validation for connector names
- Implemented HttpClient resource cleanup (AutoCloseable)
- Added token refresh response validation with null checks
- Replaced hardcoded base URL with dynamic SystemRepository configuration
- Fixed MCP Inspector compatibility (removed unimplemented logging capability)
- Added example credential files and test setup documentation
- Removed commented code and unused files for cleaner codebase
Security TODOs documented for future work:
- Race condition in authorization code exchange (requires DB schema changes)
- Rate limiting for OAuth endpoints (requires new infrastructure)
Testing:
- All changes tested with Snowflake OAuth connector
- MCP Inspector connection verified working
- Code formatted with spotless
Breaking Changes: None
* fix: Address security vulnerabilities from code review bots
Implemented fixes based on automated code review bot findings:
**Critical:**
- SSRF prevention: Added URL validation in OAuthSetupHandler to block private IPs and validate schemes
- ThreadLocal leak: Added try-finally cleanup in doGet() to prevent auth context leakage
**High:**
- Removed hardcoded JWT tokens and client secrets (replaced with dynamic UUIDs)
- Added warning logs for missing connector names to improve auditability
Security impact: Prevents internal network access, credential exposure, and auth state leakage.
Testing: All changes formatted with spotless and validated.
* fix: Optimize SSRF prevention per code review bot recommendations
Improved SSRF mitigation based on detailed bot feedback:
**Optimization:**
- Refactored validateTokenEndpoint() → validateAndResolveTokenEndpoint()
- Returns validated URI object to avoid double parsing
- Integrates endpoint resolution and validation in single method
- Reuses URI throughout method to prevent inconsistencies
**Implementation Details:**
- Validates URL scheme, host, and IP ranges
- Blocks private IPs (10.x, 192.168.x, 172.16-31.x)
- Blocks link-local addresses (169.254.x)
- Validates before HTTP request and credential storage
**Benefits:**
- More efficient (single URI parse instead of two)
- Safer (validated URI reused consistently)
- Cleaner code (DRY principle)
Based on GitHub Copilot autofix suggestion for SSRF vulnerability.
* fix(mcp-oauth): Critical security fixes per code review bots
- SSRF: Add DNS resolution and validate all resolved IPs for token endpoints
- Race condition: Atomic authorization code exchange prevents replay attacks
- Refresh token: Fix expiry check using ofEpochSecond instead of ofEpochMilli
- Remove unrelated ingestion yaml files from PR
Addresses: CodeQL, Copilot Autofix, Gitar bot feedback
* fix(mcp-oauth): Address bot feedback - security and code quality
- Remove shell scripts with hardcoded JWT tokens from PR (added to .gitignore)
- Fix admin fallback: Use ingestion-bot instead of admin for security
- Fix connector name validation: Fail refresh if connector name missing
- Add TODO comments for hardcoded localhost URIs (requires MCPConfiguration wiring)
Addresses bot feedback on security concerns and configuration flexibility
* fix: SSRF - reconstruct URI from validated components
* fix: CodeQL suppression, Y2038 bug, test provider safeguards
* MCP OAuth: implement CORS development mode detection and token cleanup scheduler
- Add development mode detection for CORS origins based on baseUrl
- Development: allow localhost origins with warning
- Production: empty allowedOrigins (same-origin only) with warning
- Implement OAuth token cleanup scheduler with Quartz
- OAuthTokenCleanupJob: deletes expired tokens and auth codes
- OAuthTokenCleanupScheduler: runs cleanup hourly
- Prevents unbounded token table growth
* fix: SSRF with allowlist and rate limiting
Use allowlist for OAuth endpoints, add rate limiting (10/5 req/min)
* fix: SSRF, OAuth security, and MySQL schema bugs
- SSRF: Remove user-provided tokenEndpoint, always infer from connector config using allowlist
- Schema: Fix MySQL table names (plural), authorization codes schema, add missing tables
- OAuth: Restore session redirect URI and re-enable nonce validation
* fix: Duplicate clientId variable and missing user_name column in Postgres migration
* security: Remove sensitive OAuth tokens and authorization codes from log statements
* security: Remove sensitive client metadata from registration logs
* chore: Remove connector OAuth infrastructure for user SSO implementation
* feat: Add MCP user SSO OAuth MVP implementation
- Updated database schema (MySQL + PostgreSQL) to use user_name instead of connector_name
- Removed connector OAuth infrastructure (plugins, ConnectorOAuthProvider)
- Created UserSSOOAuthProvider MVP skeleton with TODO markers
- Added comprehensive IMPLEMENTATION_TODO.md tracking all remaining work
- Added QUICK_START.md guide for setup instructions
- Added Claude Desktop configuration example
- Maintained backward compatibility with PAT authentication
See openmetadata-mcp/docs/IMPLEMENTATION_TODO.md for complete implementation checklist
* feat: Complete MCP OAuth SSO flow with database-backed state persistence
This commit implements a robust OAuth SSO flow for MCP server integration
that survives cross-domain redirects during SSO authentication (Google, etc).
Key changes:
- Add mcp_pending_auth_requests table for database-backed state storage
- Add McpPendingAuthRequestRepository for managing pending auth requests
- Add SSOCallbackServlet to handle SSO provider callbacks
- Add handleDirectIdTokenFlow for already-authenticated users (pac4j token flow)
- Add HtmlTemplates for secure error pages with XSS protection
- Add Claude Desktop OAuth bridge script for stdio transport integration
- Fix OIDC_CREDENTIAL_PROFILE constant shadowing issue
- Fix Postgres schema references to non-existent connector_name column
- Restore pac4j session attributes (State, Nonce, CodeVerifier) correctly
The solution stores OAuth state in the database instead of HTTP sessions,
which fail across cross-domain redirects due to SameSite cookie policy.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix: Critical OAuth security fixes - thread safety, URL encoding, JWT validation, PKCE validation
* fix: Complete ThreadLocal migration for currentRequest.getSession()
* feat: Add development bypass for PKCE validation to enable local testing
* feat: Add OAuth support with ID token validation, refresh tokens, and security fixes
- Add JWKS-based ID token signature validation
- Implement refresh token generation and exchange with rotation
- Add redirect URI validation to prevent open redirect attacks
- Fix clock skew logic and time unit consistency
- Add comprehensive test coverage (15 tests)
* fix: Critical OAuth security fixes - client validation, redirect URI validation, error handling, Fernet decryption
- Add client ID validation in token exchange (prevents authorization code theft)
- Add redirect URI validation in token exchange (RFC 6749 Section 4.1.3)
- Fix time unit inconsistency in OAuthAuthorizationCodeRepository
- Improve error handling to distinguish replay attacks from expired codes
- Add user status validation in refresh token exchange
- Fix session regeneration to prevent session fixation attacks
- Add username/email validation in SSO callback handlers
- Improve Fernet decryption error handling for key rotation scenarios
All tests passing (15/15)
* fix: Clean up pom.xml - fix malformed dependency and remove duplicate dropwizard-jersey
* javacheck style fix
* fix: Addressing issues raised by Gitar code review
* fix: Merge McpAuthFilter changes - add impersonation support while preserving OAuth endpoints
* docs: Add comprehensive README for MCP OAuth implementation
* feat: Add MCP OAuth dynamic client registration
* feat: Add OAuth token revocation endpoint (RFC 7009)
* fix: OAuth basic auth flow - auto-redirect with code and optional scope enforcement
* feat: Match MCP auth page design to OpenMetadata signin UI
* fix: Support separate callback URLs for MCP OAuth and web login flows
* feat: Add OAuth scope enforcement, domain validation and session handling for MCP
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
* feat: Improve MCP OAuth login UI and add TODO for success page
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
* fix: MCP OAuth cleanup - security fixes, remove redundant scope system, improve error handling
- Fix timing attacks in CSRF and PKCE validation using MessageDigest.isEqual()
- Remove redundant @RequireScope system (OpenMetadata Authorizer handles permissions)
- Make OAuth scopes provider-aware (Google/Okta/Azure)
- Add baseUrl config to MCPConfiguration for cluster deployments
- Delete duplicate RootOAuthEndpointsResource (handled by OAuthWellKnownFilter)
- Fix silent failures: propagate errors instead of returning null/200
- Downgrade excessive logging to DEBUG level
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Update generated TypeScript types
* fix: Move OAuth migrations from 1.12.1 to 1.12.0
- Consolidate OAuth schema tables into 1.12.0 migration
- Add Snowflake backward compatibility migration to 1.12.0
- Remove empty 1.12.1 migration folder
- Update README with security enhancements and permission model
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix: critical OAuth security and reliability issues
Fix ThreadLocal leak, atomic token rotation, PKCE validation, fail-closed error handling, and password sanitization
* fix: URL encode authorization code
* fix: MCP OAuth stateless transport compatibility and SSO initialization reliability
* feat: Add MCP configuration to database settings system
- Create mcpConfiguration.json schema for MCP-specific settings
- Add MCP_CONFIGURATION to SettingsType enum
- Add MCP configuration bootstrap logic to SettingsCache
- Extend SecurityConfigurationManager with MCP config support
- Add mcpConfiguration field to OpenMetadataApplicationConfig
- Update MCPConfiguration.java with timeout settings and comments
* feat: Complete McpServer dynamic configuration resolution
- Add getBaseUrlFromConfig() to read from SecurityConfigurationManager with fallback
- Add getAllowedOriginsFromConfig() for database-backed CORS configuration
- Remove hardcoded baseUrl and CORS origins initialization
- Remove System.setProperty for HTTP timeouts (will be handled per-request)
- Fix SSO handler to use dynamic resolution via getInstance()
- Fix NoSuchAlgorithmException import in UserSSOOAuthProvider
- All configuration now comes from database via SecurityConfigurationManager
* Update generated TypeScript types
* feat: Add database-backed MCP configuration with dynamic reload
- Add GET/PUT /api/v1/system/mcp/config API endpoints for MCP configuration management
- Refactor SSOCallbackServlet to read claims/domains/validators dynamically from SecurityConfigurationManager
- Add configuration reload support to OAuthHttpStatelessServerTransportProvider (volatile allowedOrigins, updateAllowedOrigins method)
- Implement ConfigurationChangeListener pattern in SecurityConfigurationManager for component notification
- Add HTTP timeout configuration (connectTimeout/readTimeout) to AuthenticationCodeFlowHandler from MCP config
- All configuration stored in open_metadata_settings table with SecurityConfigurationManager as single source of truth
* fix: Add volatile config fields, CopyOnWriteArrayList, null checks, and correct HTTP timeout properties
* Remove hardcoded OAuth credentials and unrelated Snowflake migration
* Fix HTTP timeout system properties and session regeneration null check
* Implement cluster polling, DB-first loading, listener pattern, and fix race conditions
* added unit tests
* removed connector OAuth code
* updated readme
* fix: MCP OAuth cleanup — security fixes, migration move, and code quality
- Move OAuth SQL migrations from 1.12.0 to 1.12.1 (release target)
- Fix XSS in auth error page (no longer reflects exception messages into HTML)
- Fix CSRF bypass in state validation (throw instead of return-after-write)
- Fix token expiration check in BearerAuthenticator (millis vs seconds mismatch)
- Require S256 code_challenge_method explicitly (reject null/plain)
- Fix GetLineageTool: use VIEW_BASIC auth, add input validation, use singleton LineageRepository
- Rename SESSION_GOOGLE_CALLBACK_URL to SESSION_SSO_CALLBACK_URL (provider-agnostic)
- Remove 10-second config polling from SecurityConfigurationManager (use SettingsCache TTL)
- Remove unnecessary synchronized on volatile field getters
- Downgrade verbose LOG.info calls to LOG.debug (session state, admin principals, tokens)
- Fix FQN imports in AuthenticationCodeFlowHandler (MCPConfiguration, Role)
- URL-encode redirect parameters (id_token, email, name)
- Remove invalid "default": null from defaultOAuthRole JSON schema
- Add error logging in AuthorizationHandler.exceptionally() block
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* add TODOs for unfixed security review findings
* fixed critical review issues: added client_secret validation, registration rate limiting, session regeneration bug, exact path matching, dead code removal
* fixed auth filter 500→401 for invalid tokens, exact path matching in transport provider
* added revocation client auth, redirect URI scheme validation, ID token validation in SSO flow, rate limiter race fix, downgraded PII logging to DEBUG
* fix MCP config loading to use getSettingOrDefault, cache IdTokenValidator
* google sso login working here
* add basic auth login flow for MCP OAuth, fix web UI redirect_uri_mismatch
* revert cosmetic UI formatting changes accidentally introduced in merge
* fix CodeQL info exposure and GitarBot security findings: redirect_uri validation, pac4j race condition
* harden MCP OAuth: fix error handling, remove dead code, prevent info leaks
* remove dead code and harden MCP OAuth: delete 5 unused files, inline metadata handlers, add PKCE validation, fix error handling
* fix GitarBot findings: restrict HTTP redirects to loopback, add token rate limiting, restore GET 405, deny-all CORS fallback, reduce JWK cache TTL
* fix Azure SSO: always register callback servlet, use baseUrl for token exchange, show success page
* security hardening: early user check, ID token audience validation, token rotation, shorter JWT TTL
* LDAP support, allow native app redirect schemes, tolerate unknown registration fields
* fix open redirect in MCP callback detection, check auth code expiry before consumption, warn on fallback baseUrl
* null safety for PKCE, grant_type, and refresh_token params in token endpoint
* fix RevocationHandler test exception type mismatch
* add registration metadata length validation, fix loopback host check
* fix MCP OAuth SSO callback for Okta: use registered redirect_uri, fix pac4j session attribute names, forward /callback to /mcp/callback
* fix missing return in MCP callback error path, skip SSO registration for basic/ldap, improve comment
* MCP OAuth security hardening: bcrypt secrets, atomic CAS rotation, XFF rate limiting, review fixes
* fix XFF rate-limit bypass: validate IP format, cap map size to prevent heap exhaustion
* move MCP OAuth migrations from 1.12.2 to 1.12.3, remove unused oauth_audit_log table, simplify
* fix client_secret_basic removal, MySQL index idempotency, token auto-delete on decrypt failure
* Update generated TypeScript types
* Update generated TypeScript types
* fix impersonation compatibility after McpAuthFilter deletion
* hash authorization codes with SHA-256 before storing in DB
---------
Co-authored-by: mohitdeuex <mohit.y@deuexsolutions.com>
Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
|
||
|
Vishnu Jain
|
0e8de77dd0
|
Mcp impersonation (#26488)
* fix MCP bot impersonation and app registration * add MCP audit log impersonation and change event publishing * add unit tests for MCP audit log and impersonation context * fix getMcpBotName startup race and remove unused WEBSOCKET_HANDLER * Fix: enforce limits in CreateTestCaseTool like other create tools * Fix: add migration for McpApplicationBot impersonation * Move allowBotImpersonation to app definition schema instead of hardcoding * Update generated TypeScript types * Fix McpAuthFilter error handling --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> |
||
|
Sriharsha Chintalapani
|
a4998bc1c7
|
Continuous indexing to handle failures (#26111)
* Add Continuous Indexing
* Add continuous Search indexing
* Update to 1.12.3
* Make search index retry queue reliable with stale recovery, health checks, and silent failure coverage
- Add entityType, retryCount, claimedAt columns to search_index_retry_queue table
- Implement stale IN_PROGRESS recovery (10min threshold, 60s sweep interval)
- Replace static isClientAvailable flag with cached ping health check (5s TTL)
- Narrow catch blocks in resolveById/resolveByFqn to EntityNotFoundException
- Use entityType hint for O(1) entity resolution instead of scanning all types
- Switch from status-string-based retry to retryCount-based (< 3 retries → PENDING, ≥ 3 → FAILED)
- Batch cascade reindex at 200 entities instead of accumulating up to 5000
- Add retry queue enqueue in catch blocks of createTimeSeriesEntity, updateTimeSeriesEntity,
deleteTimeSeriesEntityById, bulkIndexPipelineExecutions, reindexAcrossIndices, and
TestSuiteRepository.postCreate
- Re-throw exceptions from indexTableColumns/deleteTableColumns to parent catch blocks
- Add Micrometer counters for enqueued, processed (success/failure), and stale recovered
* Add missing lineage call site and Add test
* Review comments
* Add resilience to search index retry worker: client availability checks, backoff, and error classification
- Add exponential backoff when search client is unreachable so the
worker does not burn retries during cluster outages (5s → 10s → … → 60s cap)
- Classify errors using HTTP status codes from ES/OS exceptions:
4xx (except 429) are non-retryable and skip straight to FAILED;
429, 5xx, and IOException are retryable
- Preserve first bulk failure detail in RuntimeException so error
classification works for the bulk indexing path
- Reorganize SearchIndexRetryWorker into clearly separated sections
(lifecycle, main loop, record processing, entity resolution,
reindexing, resilience, suspension, utilities)
- Add isRetryableStatusCode utility to SearchIndexRetryQueue
- Add integration tests: status code classification, retry exhaustion
to FAILED, recovery from PENDING_RETRY_1, error detail preservation
* Address review comments
* Revert fqn size
* Spotless
* Address volatile review comments
* Fix Failing Test
* update review comments
---------
Co-authored-by: mohitdeuex <mohit.y@deuexsolutions.com>
Co-authored-by: Mohit Yadav <105265192+mohityadav766@users.noreply.github.com>
|
||
|
Sriharsha Chintalapani
|
6d99ba2dc0
|
Glossary relations (#25886)
* Glossary Term Relations * Add GlossaryTerm Relations * Add GlossaryTerm Relations, Add custom relations, onotolgoy explorer * Add Translations * Update generated TypeScript types * Address comments * Address comments * Address comments * Update generated TypeScript types * Update yarn.lock after merging cytoscape dependencies from glossary_relations * fix zoom in and out functionality and added missing translate keys * fix test * Remove unwanted changes * nit * nit * nit * Remove conflict test * nit * fix test * Add test for ontology explorer * New yarn lock and 2.0.0 schema changes missed during merge conflicts * Revamped glossary term relation settings * Refactor code * Addressed comments * nit * Update generated TypeScript types * Java Checkstyle and Yarn lock * Update generated TypeScript types * fix unit test * Remove 2.0.0 migration folders placed at wrong loc * Merge main * fix navigation to relation graph in glossary * fix ontology explorer spec * Added filter support in the data mode * Fix glossary term relation CI failures ### Canonical Relation Storage (GlossaryTermRepository) * Introduced `computeCanonicalRelationType()` to normalize relation direction using UUID ordering (lower UUID is always treated as "from") * Prevents duplicate and inconsistent relation rows when created from either side * Updated `setTermRelations()` and `addRelation()` to store canonical relation types * Fixed `setFields()` read logic: * Invert relation type for `fromRecords` (entity is the TO side) * Keep `toRecords` unchanged * Updated `deleteBidirectionalRelatedTo()` to match canonical storage format * Added `RequestEntityCache.invalidate()` after relation mutations to ensure consistency ### Lazy RDF Resource Initialization * Added `RdfRepository.getInstanceOrNull()` for null-safe access without throwing * Refactored `RdfResource` constructor to avoid eager `RdfRepository.getInstance()` call * Enabled resource registration even when Fuseki is not initialized * Introduced lazy getters: * `getRdfRepository()` * `getSemanticSearchEngine()` * Updated all endpoints to guard with null checks before `isEnabled()` * Return `503 Service Unavailable` when RDF is not ready ### Graceful Test Degradation (Fuseki-dependent tests) * Added `TestSuiteBootstrap.isFusekiEnabled()` to detect Fuseki availability * `GlossaryOntologyExportIT`: * Falls back to Testcontainers-based local Fuseki when bootstrap Fuseki is unavailable * `GlossaryTermRelationIT`: * Skipped via `assumeTrue` when Fuseki is unavailable * `MetricResourceIT`: * Skips RDF-specific tests when Fuseki is unavailable * fix package conflicts * nit * Fix merge conflicts, Python test, RDF reliability, and VectorDocBuilder tests - Fix Python test_patch_glossary_term_related_terms to use TermRelation instead of EntityReferenceList (schema changed relatedTerms type) - Rewrite VectorDocBuilder tests for current buildEmbeddingFields API - Improve JenaFusekiStorage retry logic to retry on all HTTP errors - Increase Fuseki tmpfs size to prevent disk space exhaustion in tests Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix pycheck * Address all 8 PR review findings 1. Add authorization check on getTermRelationGraph endpoint 2. Add null guard on getBaseUri() to prevent NPE 3. Add React key prop on RelatedTermTagButton in map renders 4. Mark RdfResource lazy-init fields as volatile for thread safety 5. Replace exception messages with generic errors in API responses 6. Unify DEFAULT_RELATION_TYPES between CSV and repository (10 types) 7. Add jitter backoff to deadlock retry in CollectionDAO 8. Replace N+1 queries in prefetchGraphTerms with batch fetch Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Fix Fuseki tmpfs exhaustion and GlossaryTermRelationIT double init - Remove tmpfs size limit on Fuseki container to prevent disk exhaustion - Guard RdfUpdater.initialize() in GlossaryTermRelationIT to skip if already initialized by bootstrap Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Fix duplicate edges, null term NPE, and silent exception in graph builder - Deduplicate edges in buildGraph() using edgesSeen set - Skip TermRelation entries with null term references to prevent NPE - Add warning log when glossary term relation settings fail to load Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Fix cardinality count after canonical swap and double-checked locking - getRelationCount now matches inverse relation type for fromRecords where the term is the target, fixing cardinality bypass after bidirectional UUID canonicalization - Use double-checked locking in RdfResource.getSemanticSearchEngine() to prevent duplicate instance creation under concurrency Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: anuj-kumary <anujf0510@gmail.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Co-authored-by: Ram Narayan Balaji <ramnarayanb3005@gmail.com> Co-authored-by: Ram Narayan Balaji <81347100+yan-3005@users.noreply.github.com> |
||
|
Sriharsha Chintalapani
|
12b364313c
|
Fix Metrics collection; reduce no.of metrics; improve slow request lo… (#25751)
* Fix Metrics collection; reduce no.of metrics; improve slow request logging
* Move sync calls to search & rdf to async
* Improve slow request tracking
* Improve slow request tracking
* Add clear breakdown in slow request
* Batch TestCaseRepository calls
* Batch API calls
* Initial Implementation of ReadEngine
* Improvements with ReadEngine/WriteEngine
* Improvements with ReadEngine/WriteEngine
* Improvements with ReadEngine/WriteEngine
* Improve by removing unnecessary ser/de
* Additional improvements with PatchFieldsPlanner
* Further performance improvements
* Further performance improvements
* Address comments
* Merge from main
* Address comments
* Address comments
* Address latest feedback - 2/21
* fix merge conflict
* Address Slow Request review
* Address the comments
* Address comments; Fix tests
* Fixes to the failing tests
* Fix bugs in tests
* Fix checkstyle
* Address playwright tests
* Fix tests
* Fix bugs
* Fix tests
* address comments
* Fix issues from playwright
* Fix playwright tests
* Fix tests for playwright
* Address comments
* Fix glossary test
* fix checkstyle
* Fix playwright issues
* Fix playwright issues - incrementalChagneDesc
* Restore ApprovalTaskWorkflow in GlossaryTerm and TestCase repositories
The slow_request branch accidentally removed entity-specific ApprovalTaskWorkflow
overrides, causing the generic parent to use checkUpdatedByTaskAssignee instead of
checkUpdatedByReviewer. This broke Glossary approval and TestCase approval Playwright tests.
- GlossaryTermRepository: restore ApprovalTaskWorkflow with checkUpdatedByReviewer
- TestCaseRepository: restore ApprovalTaskWorkflow, preDelete guard, updateReviewers
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Fix base ApprovalTaskWorkflow to use reviewer check instead of task assignee
The centralized ApprovalTaskWorkflow in EntityRepository was using
checkUpdatedByTaskAssignee instead of checkUpdatedByReviewer, breaking
approval workflows for all entity types. Added verifyReviewer() as a
top-level static method on EntityRepository and restored missing
updateReviewers() and preDelete IN_REVIEW guards in DataContract,
DataProduct, Metric, and Tag repositories. Removed now-redundant
entity-specific ApprovalTaskWorkflow overrides from GlossaryTerm and
TestCase repositories.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Fix regression introduced in backend tests; make the playwright tests stable
* Stabilize the playwright tests
* Stabilize the playwright tests
* Improve playwright tests
* Improve playwright tests
* Fix team playwrights
* Fix merge from main
* Fix playwrigt tests
* Fix playwright tests
* Batch domain/data product asset counts into single ES aggregation queries
Replace N individual ES count queries with single aggregation query per
entity type. Domain counts roll up child counts to parent domains.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Improve Playwright test reliability and expand CI shards
Add polling waits for async ES indexing, fix lineage edge selectors,
use API-based setup for domain/data product widget tests, and expand
CI from 6 to 8 shards with dedicated graph/landing projects.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Playwright: Improve test reliability with response checks and guards
- Add API response status checks in create() for Domain, DataProduct,
Glossary, TableClass, and UserClass — silent API failures now throw
immediately with status code and response body
- Add guards in selectDataProduct() and addAssetsToDataProduct() for
undefined name/fqn — clear error messages instead of cryptic
"locator.fill: value: expected string, got undefined"
- Fix GlossaryPermissions double navigation — remove redundant
redirectToHomePage + sidebarClick before glossary.visitEntityPage()
- Increase OnlineUsers timeout from 5s to 15s for CI resource pressure
- Increase Tour badge timeout from 10s to 20s
- Fix visitGlossaryPage: wait for loader before clicking menuitem
- Remove chromium testIgnore for graph/landing/stateful test files
(these must run in chromium project for 6-shard CI workflow)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Playwright: Remove all networkidle waits and improve CI reliability
- Remove ~780 networkidle waits across 144 test/utility files — these
hang or resolve prematurely under CI load causing false negatives
- Add polling.ts with waitForSearchIndexed and waitForPageLoaded helpers
- Convert checkAssetsCount and search functions to expect.poll() for
async ES indexing tolerance
- Increase expect timeout to 15s for CI environments
- Split CI into 8 shards with dedicated projects (stateful/graph/landing)
to reduce thread contention
- Fix GITHUB_STEP_SUMMARY size overflow (base64 screenshots → table)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Playwright: Fix genuine test failures from networkidle removal
- GlossaryPagination: Fix waitForResponse race conditions - register
listener BEFORE the triggering action, add **/ URL prefix
- LanguageOverride: Fix selector from getByText('EN') to
getByText('English - EN') matching actual dropdown text
- NestedColumnsExpandCollapse: Fix URL glob pattern, use dispatchEvent
to avoid inner Link navigation, add waitForResponse for filtered search
- lineage.ts: Revert dragConnection hover approach that broke React
Flow connection mode, keep direct dispatchEvent
- customizeLandingPage.ts: Remove waitForURL that hangs after page.goto
- Teams.spec.ts: Add isJoinable: false for private team creation
- UserDetails.spec.ts: Revert Escape/clickOutside save flow that
dismissed edit mode before saving roles
- Users.spec.ts: Revert Data Consumer permissions test to original
simple approach using fixtures
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Playwright: Relax OnlineUsers activity time assertion
The "Online now" exact match fails under CI load because the activity
timestamp may show as "X seconds ago" or "X minutes ago" by the time
the page renders. Changed to accept any recent activity format.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Playwright: Fix 4 genuine test failures from CI run
1. saveCustomizeLayoutPage: Use response predicate matching both
POST (create) and PUT (update) patterns instead of glob that
only matched updates. Fixes 180s timeout in drag-and-drop test
when layout doesn't exist yet (fullyParallel=true).
2. GlossaryMiscOperations: Add test.slow(true) — test does 9
sequential page navigations that exceed the 60s timeout.
3. DomainDataProductsWidgets "Assign Widgets": Add test.slow(true)
— calls addAndVerifyWidget twice, each with multiple navigations.
4. DomainFilterQueryFilter: Add waitForAllLoadersToDisappear before
clicking domain-dropdown after search operations that trigger
page re-renders.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Playwright: Fix AutoPilot test — reload page after API status poll
The AutoPilot status banner never appeared because:
1. checkAutoPilotStatus polls the workflow API directly via apiContext
(outside the browser), not through page network requests
2. The UI uses WebSocket for live updates, but the socket connection
is only established when the page loads with status=RUNNING
3. Since the page loaded before the workflow started, the socket was
never connected, so the UI never received the completion event
Fix: reload the page after checkAutoPilotStatus confirms the workflow
finished, so the UI renders with the current state. Also increase the
banner visibility timeout to 30s for CI environments.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Playwright: Fix flaky tests — entity collisions, missing cleanup, expect timeout
- Replace Date.now() with uuid() for entity names in CustomProperties tests
to prevent collisions when parallel workers execute within the same millisecond
- Fix FollowingWidget: move shared adminUser create/delete to top-level
base.beforeAll/afterAll to prevent duplicate user creation across 11
parallel test.describe blocks
- Add missing afterAll cleanup to OnlineUsers, Metric, CustomPropertyAdvanceSearch,
and CustomProperties tests to prevent entity/user leaks between runs
- Replace hardcoded metric name in MetricSearch with uuid-based name
- Add global expect timeout of 15s (up from 5s default) for CI resilience
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Fix Playwright CI: include UI in build-once Maven build
The build-once optimization (#26423) used -DonlyBackend -pl !openmetadata-ui
which produces a tar.gz without the compiled React app. The Docker container
starts but cannot serve the login page, causing auth.setup.ts to timeout
on all 6 shards waiting for input[id="email"] to appear.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Fix CodeQL security warnings
- Replace Math.random() with crypto.randomUUID() for test data generation
- Escape backslash characters in CSS selectors for glossary FQN values
- Use page.getByTestId() instead of raw CSS selectors in entity utils
- Increase RSA key size from 512 to 2048 bits in JwtFilterTest
- Skip archive entries containing '..' in JsonUtils.getResourcesFromJarFile
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Playwright: Fix user cleanup to prevent 'Email Already Exists' failures
- Glossary.spec.ts: Fix typo user3.create→delete in afterAll, add missing adminUser.delete
- Teams.spec.ts: Add afterAll cleanup hooks for 3 nested describe blocks that were missing them (EditUser, DataConsumer, Owner)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Playwright: Add afterAll cleanup hooks and fix test reliability
- InputOutputPorts.spec.ts: Add afterAll for domain/tables/topics/dashboards
- Users.spec.ts: Add top-level afterAll for all shared entities
- Entity.spec.ts: Add afterAll for shared + per-entity-type cleanup
- Pagination.spec.ts: Add afterAll for 13 describe blocks (services, DBs, etc.)
- DataProductRename.spec.ts: Add afterAll cleanup
- TestCaseIncidentPermissions.spec.ts: Add afterAll for users/roles/policies/table
- ImpactAnalysis.spec.ts: Add afterAll for all 7 entity types
- NestedColumnsExpandCollapse.spec.ts: Add afterAll for 4 describe blocks
- DataProductPermissions.spec.ts: Add afterAll cleanup
- ServiceEntityPermissions.spec.ts: Add afterAll for testUser + per-entity
- ServiceForm.spec.ts: Add afterAll for adminUser
- domain.ts: Replace waitForTimeout(2000) with proper loader/tab waits
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Trigger Playwright CI
* Playwright: Fix 2 failures and 26 flaky tests with proper waits
Fix remaining 2 genuine failures:
- DomainDataProductsWidgets: add test.slow(true) for ES indexing lag
- Users.spec.ts: add test.slow(true) and loader waits for owner search
Fix 26 flaky tests by addressing 5 root cause patterns:
- Response listener after trigger: MetricCustomUnitFlow, DomainUIInteractions
- Missing loader wait after navigation: 16 tests across CustomizeDetailPage,
DataProductPersonaCustomization, DataContracts, ExploreTree, and others
- Element not rendered after API response: EntityVersionPages, ODCSImportExport
- DOM not settled after loader: Domains nested rename
- Permission cache propagation: GlossaryPermissions
Shared utility improvements:
- waitForPatchResponse uses entity-specific URL pattern
- openColumnDetailPanel accepts entityEndpoint param with API response wait
- Entity.spec.ts uses dynamic entity.endpoint instead of hardcoded tables
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Playwright: Fix addOwner retry to wait for search API response
The owner search retry loop was refilling the search input but not
waiting for the API response before checking item visibility. This
caused the poll to repeatedly check stale/empty results.
Fix: await search response and loader detach in each retry iteration.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Playwright: Fix owner listitem selector — remove exact match
The owner selection list items include avatar initials (e.g., "G") in their
accessible name, making exact: true fail since the accessible name is
"G UserName" not just "UserName". Switching to substring matching fixes
the Users.spec.ts persistent failure.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Playwright: Fix 10 remaining flaky tests with proper waits
- ColumnLevelTests: loader wait after visiting test case panel
- DataQualityPermissions: loader wait after visiting test suite page
- IncidentManagerDateFilter: loader wait after page reload
- InputOutputPorts: wait for warning alert before asserting
- Lineage: replace 5 hardcoded waitForTimeout(500) with loader waits
- CustomizeDetailPage: dialog close waits, fix missing await on expect
- DataProductPersonaCustomization: loader wait + modal visibility check
- GlossaryPermissions: increase permission propagation wait, loader wait
- GlossaryHierarchy: loader waits after modal close and glossary select
- ExploreTree: loader waits after API response before UI interaction
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Fix CodeQL security alerts: incomplete escaping and Zip Slip
1. entity.ts: Use JSON.stringify().slice(1,-1) for proper escaping of
both backslashes and double quotes in filter values, replacing the
incomplete .replace(/"/g, '\\"') approach.
2. JsonUtils.java: Strengthen Zip Slip protection by normalizing paths
via Paths.get().normalize() and rejecting entries starting with "/"
or resolving to parent traversal after normalization.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Fix tests
* Fix tests
* Fix recordChange field name mismatches and CodeQL alert
- ServiceEntityRepository: recordChange("ingestionAgent") → "ingestionRunner"
to match the JSON property name. The shouldCompare() gate in PATCH flow
was silently dropping ingestionRunner changes because the field name
didn't match patchedFields.
- DataContractRepository: compareAndUpdate("status") → "entityStatus"
to match the JSON property name, same root cause.
- JsonUtils: Simplify Zip Slip check to string-based validation to
satisfy CodeQL taint analysis.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Remove serial mode from Users.spec.ts to prevent cascade failures
A single flaky test failure was causing ~19 tests across 5 unrelated
describe blocks to be skipped. Matches main branch behavior (parallel).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Playwright: Fix flaky tests — missing awaits, hardcoded waits, silent catches
- DataProductPersonaCustomization: add missing await on expect() calls
- TestCaseIncidentPermissions: poll for incident creation instead of one-shot query
- TestCaseResultPermissions: add loader wait after Data Quality tab click
- GlossaryPermissions: replace waitForTimeout(3000) with toPass() retry
- BulkImport: remove 4 unnecessary waitForTimeout calls
- importUtils/testCases: replace waitForTimeout(500) with grid visibility assert
- GlossaryAssets: add loader wait, remove silent .catch(() => false) pattern
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Fix CodeQL Zip Slip alert with Path.normalize() sanitization
CodeQL doesn't recognize String.contains("..") as proper Zip Slip
mitigation. Use Path.normalize() + isAbsolute/startsWith checks which
CodeQL's taint analysis model understands.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Fix Playwright flaky tests: modal visibility, toast race, query card assertion
- DataProductPersonaCustomization: wait for dialog close before clicking add-widget-button
- entity.ts restoreEntity: dismiss stale toast before restore to avoid race condition
- QueryEntity: replace page.$$() with auto-retrying expect().toBeVisible()
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Fix flaky TableResourceIT by preventing parallel multi-domain rule mutation
Both test_multipleDomainInheritance (TableResourceIT) and
test_csvImportEntityRuleValidation (DatabaseServiceResourceIT) toggle
the global "Multiple Domains are not allowed" rule. When running
concurrently, one overwrites the other's setting causing spurious
failures. Add @ResourceLock("MULTI_DOMAIN_RULE") to serialize only
these two tests while keeping all others concurrent.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Mohit Yadav <105265192+mohityadav766@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
|
||
|
Pere Miquel Brull
|
62c12a133d
|
Fix 1.13.0 preview→enabled migration for event subscriptions (#26473)
* Fix preview→enabled migration for event_subscription_entity and QRTZ tables The 1.13.0 migration renamed `preview` to `enabled` in `apps_marketplace` and `installed_apps`, but missed the `event_subscription_entity` table. The ReverseMetadata app stores the full App entity as an escaped JSON string inside `event_subscription_entity.json -> config -> app`. Since it's a string value (not a nested JSON object), standard JSON path operations can't reach the `"preview"` field — string replacement is needed instead. Also truncates QRTZ tables to clear stale Quartz job data that may contain old App JSON. Both schedulers re-create their jobs from the database on startup. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Use DELETE instead of TRUNCATE for QRTZ cleanup to respect FK constraints TRUNCATE fails on tables referenced by foreign keys in MySQL (and without CASCADE in PostgreSQL). Switch to DELETE FROM with correct FK ordering (children before parents) and add missing child tables (QRTZ_SIMPROP_TRIGGERS, QRTZ_BLOB_TRIGGERS). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> |
||
|
Teddy
|
8270e01415
|
ISSUE-3030 - Profiler autotune threading (#26385)
* FIX - Redshift converter (#26229)
(cherry picked from commit
|
||
|
Ram Narayan Balaji
|
bb6a99b953
|
Feat# Include Fields Filter in EventBased Workflows and CheckChangeDescription Node (#26230)
* Include Fields in EventBased Workflows - Initial Commit
* Update generated TypeScript types
* Fix Include fields to be a map of arrays, Introduce checkChangeDescriptionTask as a separate node
* Update generated TypeScript types
* Extract common code into field value extractor
* chore: apply changes
Co-authored-by: yan-3005 <yan-3005@users.noreply.github.com>
* java checkstyle
* Fix Compilation errors
* Fix NPE bug
* Test fixes and improvements
* chore: apply changes
Co-authored-by: yan-3005 <yan-3005@users.noreply.github.com>
* Schema Changes for include fields and check change description
* Update generated TypeScript types
* Fixed 4 valid code review issues: migration idempotency bug (preventing false failures on re-runs), empty pattern string vulnerability (preventing unintended filter bypasses),
removed unused dead code method, and corrected Javadoc inconsistency from {} to [] notation.
---------
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Gitar <noreply@gitar.ai>
Co-authored-by: yan-3005 <yan-3005@users.noreply.github.com>
Co-authored-by: Anujkumar Yadav <anujf0510@gmail.com>
|
||
|
Trang Nguyen [INT-DE]
|
de2e703fdd
|
Fixes #26225: Add index and FORCE INDEX for listLastTestCaseResultsForTestSuite (MySQL) (#26235)
* ISSUE-26225: add index idx_entity_timestamp_desc for data_quality_data_time_series * ISSUE-26225: add index idx_entity_timestamp_desc for data_quality_data_time_series * Update bootstrap/sql/migrations/native/1.12.2/mysql/schemaChanges.sql * ISSUE-26225: fix the suggestion --------- Co-authored-by: Teddy <teddy.crepineau@gmail.com> Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com> |
||
|
Pere Miquel Brull
|
f890e004ce
|
Move preview-to-enabled migrations from 1.11.13 to 1.13.0 (#26281)
The migrations renaming the 'preview' property to 'enabled' in apps were incorrectly placed under 1.11.13. Move them to 1.13.0 where they belong, since this change targets the next major release. Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> |
||
|
Teddy
|
40bf82f604
|
Minor move 20 migrations (#26236)
* FIX - Redshift converter (#26229)
(cherry picked from commit
|
||
|
Pere Miquel Brull
|
c7f911e43a
|
Rename app 'preview' property to 'enabled' (#26170)
* Rename app 'preview' property to 'enabled' with inverted semantics The 'preview' property was confusing: preview=false meant the app CAN be used. Replace with 'enabled' where enabled=true means usable, which is much more intuitive. Changes across the full stack: - JSON schemas: preview (default false) → enabled (default true) - Java backend: isPreview/raisePreviewMessage → isEnabled/raiseNotEnabledMessage - TypeScript types: preview → enabled - Frontend component: isPreviewApp → isAppDisabled (checks enabled===false) - SQL migrations for 1.11.12: rename + invert boolean in apps_marketplace and installed_apps tables (MySQL and PostgreSQL) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Update generated TypeScript types * format * improve deletion process for disabled apps * improve deletion process for disabled apps * improve deletion process for disabled apps * improve deletion process for disabled apps * format * fix tests * migration * migration --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> |
||
|
Sid
|
12d85f310f
|
fix glossary status frontend filtering logic to move to backend (#25428)
* fix glossary status * add glossaryTerm spec * fix: improve ListFilter implementation in list filtering logic Co-authored-by: siddhant1 <siddhant1@users.noreply.github.com> * reset main backend * reset backend * fix be * rever * spottless * Fix GlossrayTerm search api endpoint * status enum validation * fix spec * Replace quotes, validate enum * bind param queries * Move migrations to 1.12.0 * fix api docs * optimize performance of fallback , refactoring * fix ListFilter * GlossaryTermService.java cleanup * address gitar-bot feedback * add entityStatus param in list api * add entityStatus param in list api * Send entityStatus param with both search and list glossary term APIs - Pass entityStatus to searchGlossaryTermsPaginated and getFirstLevelGlossaryTermsPaginated when a specific status filter is active (not 'all') - Keep 'All' option in status dropdown with default selection of Approved, Draft, InReview - Show appropriate empty state message when status filter returns no results Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * update list API path (ListFilter.getEntityStatusCondition) to validate against the enum, in case if an invalid value like "Bogus" is passed * fix playwright * Fix rejected glossary term staying visible in listing Remove rejected terms from visible list when status filter excludes them, and fix reused waitForResponse promise in Playwright test. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * add initian load * Fix Expand All ignoring active status filter and add E2E tests Pass entityStatus parameter in fetchExpadedTree so Expand All respects the active status filter. Add E2E test suite to verify the behavior. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Rewrite Glossary Expand All E2E tests to follow Playwright handbook patterns Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Fix flaky GlossaryPagination test by scoping locators to glossary table Scoped unscoped `tbody .ant-table-row` locators to `glossary-terms-table` testid, and replaced unreliable row count assertion in empty state test with visibility checks on `no-data-placeholder`. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Siddhant <siddhant@MacBook-Pro.local> Co-authored-by: Gitar <noreply@gitar.ai> Co-authored-by: siddhant1 <siddhant1@users.noreply.github.com> Co-authored-by: Ram Narayan Balaji <ramnarayanb3005@gmail.com> Co-authored-by: Ram Narayan Balaji <81347100+yan-3005@users.noreply.github.com> Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com> Co-authored-by: sonika-shah <58761340+sonika-shah@users.noreply.github.com> Co-authored-by: Siddhant <siddhant@MacBook-Pro-3.local> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: Siddhant <siddhant@MacBook-Pro-4.local> |
||
|
Teddy
|
a456a194a9
|
ISSUE #3027 - Better Default (#26158)
* feat(metric default): move profiler and dq to obs folder * feat(metric default): validate metric registry and schema * feat(metric default): map metric type name to enum name * feat(metric default): updated default metrics in profiler * feat(metric default): migration to remove computeMetrics setting * feat(metric default): fix CI failures * feat(metric default): fix CI failures * fix ci failures * fix ci failures * fix typo in psql migration query * fix psql migration query * fix ci failure * fix: CI failures |
||
|
Mayur Singal
|
31e2e59a00
|
Fix #26178: Add support for IAM auth for redshift (#26179)
* Fix #26178: Add support for IAM auth for redshift * Missing files for the implementation * Update generated TypeScript types * adderess guitar comments * address comments * fix python tests * fix redshift playright * fix checkstyle --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> |
||
|
Himanshu Khairajani
|
cf0fa0a519
|
Openlineage: Added Kinesis Support #24752 (#26050)
* Openlineage Kinesis Support * Update generated TypeScript types * marking field as required * test-connection name improvement * pagination improvement * test-connection name improvement * Update generated TypeScript types * nested broker-config migration file * newline added to yaml * Migration to 1.11.2 * Migration to 1.11.12* * fix: add throttle mechanism to kinesis get_records loop Co-authored-by: Khairajani <Khairajani@users.noreply.github.com> * fix: prevent timeout reset on sequential shard polling Co-authored-by: Khairajani <Khairajani@users.noreply.github.com> * Kinesis test-case * Kinesis test-case * setting lineageInformation object model and not raw dict --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Gitar <noreply@gitar.ai> Co-authored-by: Khairajani <Khairajani@users.noreply.github.com> |
||
|
Sriharsha Chintalapani
|
7465810fdd
|
Audit Log performance improvements (#26023)
* Audit Log performance improvements * Audit Log performance improvements * Address comments * removed fixme from audit log tests --------- Co-authored-by: Rohit0301 <rj03012002@gmail.com> Co-authored-by: Rohit Jain <60229265+Rohit0301@users.noreply.github.com> |
||
|
Mohit Yadav
|
82b9d34806
|
Optimize indexing Processing to EsDoc (#26079)
* Optimize Reads with Keyset * Optimize Search Index Processing stage * Fix KeySet Cursor * revert keyset for time series * Fix Review Comments * Move to 1.12.2 * Fix Review Comment * Remove IF NOT EXISTS from mysql and update common mthod |
||
|
harshsoni2024
|
4d017d3f32
|
Fix-20713: Add support for metadata ingestion using local file in REST connector (#26036) | ||
|
Pere Miquel Brull
|
a1e3a49dae
|
MINOR - Allow app definition to pass the impersonation rules for bots (#25909)
* MINOR - Streamline bot impersonation from apps * MINOR - Streamline bot impersonation from apps * MINOR - Streamline bot impersonation from apps * MINOR - Streamline bot impersonation from apps * Update generated TypeScript types * policy flag * policy flag * policy flag * policy flag * fix feedback --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> |
||
|
Ram Narayan Balaji
|
91239164f5
|
delete workflow instance entries if status is null in migration (#25867) | ||
|
Ram Narayan Balaji
|
f418203338
|
Fix: Resolve v1.12.0 migration failure due to NULL workflow status (#25834)
* Fix: Resolve v1.12.0 migration failure due to NULL workflow status ## Root Cause Analysis - Migration failed when modifying entityLink column in workflow_instance_time_series - MySQL's ALTER TABLE MODIFY COLUMN re-validates ALL generated columns for ALL rows - Found 184+ workflow instances created between Dec 2024 - Jan 2025 with NULL status - These were created with pre-v1.7.0 code that didn't set status field in JSON - v1.7.0 added status column as GENERATED NOT NULL but old instances had NULL values - v1.12.0 migration triggered constraint validation, causing "Column 'status' cannot be null" ## Solution - Add UPDATE statements before ALTER TABLE in v1.12.0 migration - Set status='FINISHED' for workflows with endedAt (completed) - Set status='FAILED' for workflows without endedAt (incomplete) - Use two separate queries for better performance vs CASE statements - Handle both workflow_instance_time_series and workflow_instance_state_time_series * failed to FAILURE status |
||
|
Sriharsha Chintalapani
|
b244798f22
|
Add bulk apis for pipeline status (#25731)
* Add bulk apis for pipeline status * Update generated TypeScript types * Fix gitar comments * Update generated TypeScript types * Fix pycheck * Address comments * Fix databricks test * Move schema changes to 1.11.9 --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: harshsoni2024 <harshsoni2024@gmail.com> |
||
|
Sriharsha Chintalapani
|
6f577656c1
|
Fix integration tests (#25753)
* Fix - disk space in github workflows * Fix - disk space in github workflows * Fix - disk space in github workflows * Fix running tests with bulk apis * Fix running tests with bulk apis * Address comments; make awaitability for tests * Address comments |
||
|
sonika-shah
|
30a4d32720
|
Fix entity version history of dataProducts after removing inputPorts/ field (#25702) | ||
|
Aleksei Sviridkin
|
b2ac6f70d9
|
Fixes #24546: Add sobjectNames field for multi-object selection in Salesforce connector (#24547)
* feat(salesforce): add sobjectNames field for multi-object selection Add support for specifying multiple Salesforce objects to ingest instead of just one or all. The new `sobjectNames` array field allows users to select specific objects (e.g., Contact, Account, Lead) without having to ingest all objects and filter them. Priority order: 1. sobjectNames (array) - if specified, use only these 2. sobjectName (string) - if specified and sobjectNames empty 3. All objects from describe() - if neither specified tableFilterPattern applies in all cases as a final filter. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> * refactor: removed sobjectName field and added a migration for 1.11.8 to migrate sobjectName values to sobjectNames * fix: sobjectNames priority comment * refactor: sobjectNames changes in ts files * fix: yaml structure in test_salesforce * fix: test_salesforce.py - metadata as OpenMetadata object * fix: added new line in sql migrations * fix: sql migration serviceType --------- Signed-off-by: Aleksei Sviridkin <f@lex.la> Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: Keshav Mohta <keshavmohta09@gmail.com> Co-authored-by: Keshav Mohta <68001229+keshavmohta09@users.noreply.github.com> Co-authored-by: Mohit Yadav <105265192+mohityadav766@users.noreply.github.com> |
||
|
Ajith Prasad
|
f1fe02daff
|
Moved AI Application and LLM Model entities migrations to 1.12.0 (#25659) | ||
|
Himanshu Khairajani
|
e86a0201ab
|
Fix #25645: MySQL timestamp precision for tag_usage.appliedAt (#25643)
* Fix MySQL timestamp precision for tag_usage.appliedAt MySQL's TIMESTAMP type defaults to second precision, while PostgreSQL returns microsecond precision. This causes _normalize_datetime_strings in the Python ingestion client to produce spurious appliedAt diffs in JSON patches, which then fail with "Failed to convert JsonValue to target class" during deserialization in JsonUtils.applyPatch(). Upgrade appliedAt to TIMESTAMP(6) to match PostgreSQL behavior and eliminate the spurious patch diffs. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * Add 1.11.8 migration for MySQL appliedAt timestamp precision Backport the TIMESTAMP(6) fix to the 1.11.x release line so existing deployments on 1.11.x pick up the fix without requiring a 1.12.0 upgrade. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com> |