mirror of
https://github.com/open-metadata/OpenMetadata
synced 2026-05-24 09:39:11 +00:00
538 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Mohit Yadav
|
c2e6d907dd
|
fix(lineage): service nodes appearing in entity lineage view and empty By Service view (#27258)
* fix(lineage): prevent pipeline annotation inheritance in service/domain/dataProduct lineage and add pipeline service edges
Bug #1: Service nodes (e.g., DatabaseService, MessagingService) were incorrectly appearing in
entity-level lineage views. Root cause: getOrCreateLineageDetails() in addServiceLineage(),
addDomainLineage(), and addDataProductsLineage() was copying the pipeline annotation from
entity-level LineageDetails to service/domain/dataProduct-level LineageDetails. This caused
service entities to have upstreamLineage.pipeline.fqnHash set in their Elasticsearch documents,
making them match the PIPELINE_AS_EDGE_KEY query during BFS traversal and incorrectly appear
alongside actual data assets. Fix: add .withPipeline(null) on each service/domain/dataProduct
LineageDetails object to strip the pipeline annotation before persisting.
Bug #2: "By Service" view was empty when viewing lineage for pipeline entities that were stored
as edge annotators (Case B: table → topic with pipeline=flink_pipeline in LineageDetails) rather
than as actual nodes (Case A). Root cause: addServiceLineage() only created database_service →
kafka_service edges but no edges involving flink_pipeline_service. Fix: add addPipelineServiceEdges()
called from addServiceLineage() that creates fromService → pipelineService and pipelineService →
toService edges when a pipeline annotation exists in the entity-level lineage details.
Also add unit tests covering both fixes to prevent regression.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(lineage): add migration to remove pipeline annotation from service/domain/dataProduct lineage edges
The previous fix (
|
||
|
sonika-shah
|
077982c348
|
Move ontology/glossary relation migration from 1.14.0 back to 1.13.0 (#27431)
* Move ontology/glossary relation migration from 1.14.0 back to 1.13.0 Ontology feature will ship in 1.13.0, not 1.14.0. Move the glossary term relation migrations (relationType backfill, settings insert, stale relatedTerms strip, conceptMappings backfill) back to the 1.13.0 postDataMigrationSQLScript for both MySQL and PostgreSQL. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Restore empty 1.14.0 SQL migration files for Java migration framework The V114 MigrationUtil.java package requires the 1.14.0 migration directory to exist with SQL files for the migration to be picked up. Keep them as empty files (matching convention of other versions with no post-data SQL). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Add schemaChanges.sql and comment all 1.14.0 SQL migration files Add both schemaChanges.sql and postDataMigrationSQLScript.sql for mysql and postgres with a comment explaining the directory is required for the V114 Java migrations to be picked up by the migration framework. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Fix missing trailing newline in postgres postDataMigrationSQLScript Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * address feedback --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: Karan Hotchandani <33024356+karanh37@users.noreply.github.com> |
||
|
Ram Narayan Balaji
|
35ede8fe5f
|
fix(migration): revert webhook authType back to secretKey in v1126 and remove broken v1125 migration (#27427)
* fix(migration): add v1126 reverse migration to revert webhook authType back to secretKey * fix(migration): remove migrateWebhookSecretKeyToAuthType from v1125 migration * fix(test): remove migrateWebhookSecretKeyToAuthType references from v1125 migration tests * fix(migration): address copilot review comments on v1126 migration * fix(migration): case-insensitive bearer check and verify JSON content in v1126 tests * fix(migration): remove unused constants from v1125 and add postgres path + SQL verification to v1126 tests |
||
|
Sriharsha Chintalapani
|
bb0daa180e
|
RDF, cleanup relations and remove unnecessary bindings, add distributed mode for RDF reindex (#26902)
* RDF, cleanup relations and remove unnecessary bindings, add distributed mode for RDF reindex * Update generated TypeScript types * Address comments from copilot * Update generated TypeScript types * fix test issues * Fix minor UI bugs * Add the missing filters * Fix RDF export API error * Add export functionality * Fix ui-checkstyle * Fix java checkstyle * Fix unit tests * Fix and increase the coverage for KnowledgeGraph.spec.ts * Fix tests * Remove rdf as default in playwright and local docker * fix ui-checkstyle * Address comments * Potential fix for pull request finding 'CodeQL / Artifact poisoning' Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> * Address copilot comments * Address copilot comments * FIx tests * FIx docker * Update openmetadata-service/src/main/java/org/openmetadata/service/apps/bundles/rdf/distributed/DistributedRdfIndexCoordinator.java Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Address copilot review comments: license headers, JSON escaping, type safety, border-color, stop semantics Agent-Logs-Url: https://github.com/open-metadata/OpenMetadata/sessions/c026e52e-162b-4c9a-9874-43791d4aaac1 Co-authored-by: harshach <38649+harshach@users.noreply.github.com> * Show error toast for unsupported export format in KnowledgeGraph Agent-Logs-Url: https://github.com/open-metadata/OpenMetadata/sessions/c026e52e-162b-4c9a-9874-43791d4aaac1 Co-authored-by: harshach <38649+harshach@users.noreply.github.com> * Fix docker * Fix docker for playwright * Fix docker for playwright * Fix tests * Fix tests * Fix docker * Fix docker * Fix glossary and pagination spec flakiness * update the missing translations * Fix docker * Fix docker * Fix integration test * Fix fuseki not starting * Fixed the run local docker script * worked on comments * Fix flakiness in knowledge graph tests * Fix checkstyle --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Aniket Katkar <aniketkatkar97@gmail.com> Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: harshach <38649+harshach@users.noreply.github.com> |
||
|
Rajdeep Singh
|
5e1416447f
|
fix(sampler): Respect randomizedSample flag at 100% percentage sampling (#26966)
* fix(sampler): respect randomizedSample flag at 100% percentage sampling When profileSample is 100% with PERCENTAGE type, the sampler short-circuits and returns the raw dataset without any randomization, even when randomizedSample is True (the default). Split the combined condition so: - No profileSample set -> return raw dataset (no sampling configured) - 100% PERCENTAGE + randomizedSample=False -> return raw dataset (optimization) - 100% PERCENTAGE + randomizedSample=True -> go through normal sampling path which applies RandomNumFn/df.sample for proper row shuffling Fixes #21304 * Address review: use 'is False' for Optional[bool] and add unit tests - Fix randomizedSample check from 'not' to 'is False' in both SQASampler and DatalakeSampler to correctly handle None (Optional[bool] default=True) - Add unit tests verifying 100%% PERCENTAGE behavior for randomizedSample values True, False, and None * Add ORDER BY on random column in fetch_sample_data for true randomization The get_dataset() fix ensures 100% PERCENTAGE + randomizedSample routes through get_sample_query() which produces a CTE with a random column. Now fetch_sample_data() detects that column and applies ORDER BY before LIMIT, so each call returns a different subset of rows. Also add real-DB integration tests using SQLite for the 100% PERCENTAGE edge case (True, False, None). * Address review: remove stale comment, unused import, add return assertions * Apply suggestion from @Copilot Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Address review: move ORDER BY to get_sample_query, clean up fetch_sample_data - Move ORDER BY rnd.c.random into get_sample_query() PERCENTAGE branch, gated on randomizedSample is not False (mirrors ABSOLUTE branch pattern) - Revert fetch_sample_data() to original: remove ds_columns variable, random_column detection, and ORDER BY logic (ordering now handled in CTE) - Remove duplicate assertions in DatalakeSampler100Pct tests * Address review: None defaults to False for randomizedSample Per TeddyCr's feedback, randomization is computationally heavy and should not be the default. Changed from 'is False'/'is not False' to truthiness checks so None (unset) behaves the same as False. Only explicit randomizedSample=True triggers ORDER BY and skips the 100% fast path. This is consistent with the ABSOLUTE branch which already uses truthiness checks. * Fix integration test: None should skip sample_query (matches truthiness semantics) * fix(tests): update BigQuery view sampling expected queries with ORDER BY BigQuery views fall through to SQASampler.get_sample_query() which now adds ORDER BY rnd.random when randomizedSample is enabled. Update the expected SQL strings in test_sampling_for_views and test_sampling_view_with_partition to match. * refactor: use explicit is False for randomizedSample checks Address review comments: SampleConfig.randomizedSample defaults to True, so only an explicit False should disable randomization. Using is False / is not False instead of truthiness ensures None follows the model default (enabled) rather than being incorrectly treated as disabled. * ci: re-trigger checks after SIGSEGV flake * refactor: only explicit True randomizes, add non-determinism tests * test: increase non-determinism iterations to reduce flakiness * chore: added randomize as false * fix: align randomizedSample defaults with schema (false) * fix: remove ORDER BY from BigQuery test expectations BigQuery sampling tests create SampleConfig without setting randomizedSample, which now defaults to False. Since ORDER BY is only added when randomizedSample is True, the expected query strings should not include ORDER BY. Also fix inaccurate docstring in test_sample.py. * test: increase non-determinism test iterations to reduce flakiness Increase fetch_sample_data loop from 10 to 20 iterations to further reduce the theoretical probability of a false failure in the randomized ordering test. --------- Co-authored-by: Teddy <teddy.crepineau@gmail.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> |
||
|
sonika-shah
|
733921f510
|
Fix: align glossary term relation type colors with design system (#27142)
* Fix: align glossary term relation type colors with design system System-defined relation types (relatedTo, synonym, antonym, etc.) were initialized with old Ant Design palette colors (#1890ff, #722ed1, …) while the frontend RELATION_META constants had been updated to the new design system colors (#1570ef, #b42318, …). Because renderColorBadge used record.color (from the backend) unconditionally, the stale Ant Design colors were always displayed instead of the intended ones. - Frontend: renderColorBadge now treats RELATION_META as authoritative for system-defined types so the correct design-system color is always shown, regardless of what color value is stored in the backend. - Backend (SettingsCache.java): default colors updated for new installs. - DB migration (2.0.0): postDataMigrationSQLScript added for MySQL and PostgreSQL to update colors in existing deployments without touching user-added custom relation types. - Tests: unit tests for renderColorBadge color-resolution logic; integration test asserting all ten system-defined types return the expected hex values from the API. Fixes #openmetadata/OpenMetadata * Remove dev-only MySQL 2.0.0 migration script * Remove dev-only PostgreSQL 2.0.0 migration script * Fix: align glossary term relation settings colors and remove duplicate 1.13.0 migration; Remove glossary term relation migrations mistakenly re-added in 1.13.0 and update relation type colors in the 1.14.0 migration INSERT to use design system tokens instead of old Ant Design colors. * fix lint * add more test * address feedback * fix prettier formatting in test file Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * remove GlossaryTermRelationSettings test file from branch Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
||
|
Mohit Yadav
|
7693a5b04b
|
Update indexing schedule (#27204)
* Update schedule to weekly * Migration |
||
|
Suman Maharana
|
a06b7e74cc
|
Chore: Remove iceberg standalone connector (#26365)
* Chore: Remove iceberg standalone connector * add migration scripts * Update generated TypeScript types * py_format * address comments * Addressed changes * add tests * migrate to custom database * fix tests * fix tests * fix migrations * hard delete exising ingestion pipelines for iceberg * Update generated TypeScript types * Delete openmetadata-ui/src/main/resources/ui/src/generated/entity/services/ingestionPipelines/ingestionPipeline.ts * Delete openmetadata-ui/src/main/resources/ui/src/generated/entity/automations/workflow.ts * Delete openmetadata-ui/src/main/resources/ui/src/generated/api/automations/createWorkflow.ts * Delete openmetadata-ui/src/main/resources/ui/src/generated/api/services/ingestionPipelines/createIngestionPipeline.ts * Delete openmetadata-ui/src/main/resources/ui/src/generated/api/services/createDatabaseService.ts * Delete openmetadata-ui/src/main/resources/ui/src/generated/entity/automations/testServiceConnection.ts * Update generated TypeScript types * Update bootstrap/sql/migrations/native/1.13.0/mysql/postDataMigrationSQLScript.sql Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> |
||
|
Sriharsha Chintalapani
|
ed58077197
|
MCP services (#23623) | ||
|
Ram Narayan Balaji
|
b9d8c08b5b
|
Refactor(certification): store asset certification in tag_usage table (#26448)
* refactor(certification): store asset certification in tag_usage table Previously, asset certification was stored as a JSON blob directly on the entity row. This created a split system where the tag FQN lived in the entity JSON while tag metadata (name, description, style) had to be re-fetched from the tag table on every read. It also meant certification was invisible to the tag_usage propagation pipeline, so renaming a certification tag's FQN left stale data on certified entities. Certification is now stored in tag_usage alongside all other tags, using the metadata column to carry expiryDate (added to TagLabelMetadata schema). The entity's certification field remains the input/output surface, but tag_usage is now the source of truth. Key changes: Storage & retrieval - applyCertification() writes the certification tag into tag_usage on store - deleteCertificationTag() removes it from tag_usage on clear/replace - getCertification() reads from tag_usage filtered by the configured certification classification instead of parsing entity JSON - getTags() now strips certification-classification tags so they are surfaced exclusively through getCertification() Performance improvements - batchFetchCertification() rewritten to a single batch query on tag_usage by FQN hash instead of performing N individual tag lookups Tag update handling - handleTagEntityUpdate() reads the allowed classification from settings (no longer hardcoded) - correctly computes oldFQN on name change so Elasticsearch documents are found and updated using the correct key DAO & schema changes - deleteTagsByPrefixAndTarget() added to CollectionDAO for targeted certification tag removal - TagLabel mappers hardened against unknown metadata fields Migrations - v1123 migrations backfill existing entity JSON certifications into tag_usage so no data is lost during upgrade Tests - TagResourceIT updated to assert getCertification() instead of getTags(), since certification tags are intentionally excluded from the tags list * Update generated TypeScript types * chore: apply changes Co-authored-by: yan-3005 <yan-3005@users.noreply.github.com> * fix(certification): prevent updateTags() from clobbering cert tags written by updateCertification() * fix(certification): compute tagFQNHash per-segment in Java during migration and make applyCertification idempotent * Update generated TypeScript types * Fix: SQL-filtered cert batch fetch, remove double-delete, schema strict mode, ordinal bounds check, migration logging * Update generated TypeScript types * Fix Migration * Fix Migration * fix(certification): address Copilot review feedback on PR #26448 - Use exact field name comparison (FIELD_NAME.equals) instead of contains() in SearchRepository to avoid incorrect FQN-rename branch triggers when displayName changes - Log previously swallowed exception in getCertificationClassificationFromSettings() to improve observability of certification search propagation failures - Fix v1124 migration by building selectedIds inside the insert loop and skipping rows with null tagFQN, preventing UPDATE from removing certifications without corresponding tag_usage entries (avoids silent data loss) - Update integration test to rename tag name (not displayName) so it correctly validates the FQN-change regression from #26432 and asserts propagation to entity certification field and search index * fix(migration): fix v1124 certification migration correctness issues - Fix wrong version string in error messages: both mysql and postgres Migration.java logged "v1123" instead of "v1124" - Fix potential infinite loop: null-tagFQN rows were excluded from the INSERT but still counted in the return value (rows.size()), so when a full batch of 500 rows all had null tagFQN the loop never terminated. Fix by filtering null tagFQN at SQL level (WHERE tagFQN IS NOT NULL) and returning selectedIds.size() so the loop count reflects rows that were actually migrated Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(certification): fix missing tables in migration and optimize getCertification query - Add 6 missing entity tables to v1124 certification migration: file_entity, directory_entity, spreadsheet_entity, worksheet_entity, llm_model_entity, ai_application_entity — all define the certification field in their JSON schema; omitting them caused silent data loss on upgrade (certification stripped from JSON but never written to tag_usage) - Replace getCertification() full-tag-fetch with getCertTagsInternalBatch() so single-entity reads issue a targeted WHERE tagFQN LIKE query instead of fetching all tags and filtering in Java (consistent with the bulk path) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(certification): preserve appliedDate in migration and avoid appliedAt reset on unchanged cert - v1124 migration now extracts certification.appliedDate from entity JSON and inserts it as tag_usage.appliedAt, preserving the original certification timestamp instead of defaulting to migration time - applyCertification() now checks whether the existing certification tag matches the incoming one before doing delete+reinsert; if unchanged it returns early, preventing appliedAt from being reset on every entity write Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(certification): also compare expiryDate in applyCertification idempotency check The previous fix skipped delete+reinsert when tagFQN was unchanged, but this incorrectly swallowed expiryDate updates — re-certifying with the same tag but a new validity period would return early and never write the new expiryDate to tag_usage. Adding Objects.equals(expiryDate) to the guard ensures metadata-only changes are still persisted. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * test(certification): replace fixed sleeps with Awaitility polling in rename test Fixed sleeps are flaky under CI load and always waste time when indexing is faster. Replace both TimeUnit.SECONDS.sleep(2) calls and all subsequent search/entity assertions with Awaitility.await().untilAsserted() blocks (30s timeout, 1s poll interval) so the test waits exactly as long as needed. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(migration): include exception in certification migration warning log Pass the exception object to LOG.warn so the stack trace is available for diagnosing production migration failures. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * perf: cache getCertificationClassification() via SettingsCache Replace direct SystemRepository DB call with SettingsCache.getSettingOrDefault() (Guava LoadingCache, 3-min TTL) to eliminate repeated DB hits on every certification-related call in EntityRepository. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * skip the test * Added new column for certification and tier * nit * Add test for tier and certification * fix unit test * Fix Unit tests * Move Migrations to 1.12.5 and unit tests * Fix NPE, batch certification writes, and improve test coverage - Guard against null tagLabel in applyCertification to prevent NPE on malformed input - Replace per-entity applyCertification loop in storeRelationshipsInternal with applyCertificationBatch, reducing 3N DB calls to 2 (one batch DELETE + one batch INSERT via existing applyTagsBatchMultiTarget) - Add deleteTagsByPrefixAndTargets to TagUsageDAO as the batch variant of deleteTagsByPrefixAndTarget - Add tests for applyCertificationBatch paths, getTags cert filtering, and TagLabelWithFQNHash.toTagLabel to meet 90% new-code coverage threshold Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Add coverage tests for RowMappers, batchFetchCertification, and toTagLabel fallbacks - Add TagLabelMapper and TagLabelWithFQNHashMapper tests using mock ResultSet to cover the new metadata-parsing code paths in CollectionDAO - Add toTagLabel fallback tests for out-of-bounds enum ordinals covering the defensive conversion logic in TagLabelWithFQNHash - Add storeRelationshipsInternal single-entity overload test covering line 2322 - Add fetchAndSetFields tests to cover batchFetchCertification happy path and exception fallback path Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * resolved the linting issue * nit * fix lint issue --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Gitar <noreply@gitar.ai> Co-authored-by: yan-3005 <yan-3005@users.noreply.github.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Co-authored-by: Anujkumar Yadav <anujf0510@gmail.com> |
||
|
Ram Narayan Balaji
|
10cf2f9ea0
|
Move ontology/glossary relation migration from 1.13.0 to 1.14.0 (#26755)
The glossary term relation migration (relationType backfill, default glossaryTermRelationSettings insert, relatedTerms cleanup, conceptMappings backfill) was accidentally placed in the 1.13.0 migration scripts. This commit moves it to the correct 1.14.0 slot, restoring 1.13.0 to its original content (computeMetrics profiler pipeline cleanup only). Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> |
||
|
Ram Narayan Balaji
|
ee4f9316c1
|
Move Migration to 1.12.4 from 1.12.3 (#26629) | ||
|
sonika-shah
|
aff1343643
|
fix: strip stale relatedTerms from glossary_term_entity JSON to fix 500 on listAfter (#26586)
* fix: strip stale relatedTerms from glossary_term_entity JSON to fix 500 on listAfter Pre-1.13.0, relatedTerms was stored as EntityReference[] directly in the glossary_term_entity JSON column. PR #25886 changed relatedTerms to TermRelation[] and moved storage to entity_relationship table, but missed adding a migration to clean up the old EntityReference data still present in existing rows. When listAfter() deserializes the entity JSON, Jackson fails with: UnrecognizedPropertyException: Unrecognized field "id" (class TermRelation) The existing migration already backfilled entity_relationship rows with relationType="relatedTo", so stripping relatedTerms from entity JSON is safe — the data is already in entity_relationship and will be loaded from there. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> * fix: strip stale relatedTerms from glossary_term_entity JSON to fix 500 on listAfter Pre-1.13.0, relatedTerms was stored as EntityReference[] directly in the glossary_term_entity JSON column. PR #25886 changed relatedTerms to TermRelation[] and moved storage to entity_relationship table, but missed adding a migration to clean up the old EntityReference data still present in existing rows. When listAfter() deserializes the entity JSON, Jackson fails with: UnrecognizedPropertyException: Unrecognized field "id" (class TermRelation) The existing migration already backfilled entity_relationship rows with relationType="relatedTo", so stripping relatedTerms from entity JSON is safe — the data is already in entity_relationship and will be loaded from there. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: Ram Narayan Balaji <81347100+yan-3005@users.noreply.github.com> |
||
|
Vishnu Jain
|
6e93754a2f
|
Mcp oauth (#25391)
* Add OAuth MCP
* Implement internal OAuth flow for MCP with database
persistence
This commit implements a redirect-free OAuth flow for the OpenMetadata MCP
server that uses stored connector OAuth credentials internally, eliminating
the need for external browser redirects.
Key Features:
- Internal OAuth authorization using stored connector credentials
- Database persistence of OAuth tokens (survives container restarts)
- Automatic token refresh when expired
- PKCE support for authorization code flow
- OAuth discovery metadata endpoint (RFC 8414)
How It Works:
1. Admin performs one-time OAuth setup via /api/v1/mcp/oauth/setup
2. OAuth credentials (access token, refresh token) stored encrypted in database
3. MCP clients connect without browser - server uses stored credentials internally
4. Expired tokens automatically refreshed and re-persisted to database
Tested With:
- Snowflake OAuth (session:role:PUBLIC scope)
- Container restart verification (credentials persist)
- Automatic token refresh verification
* feat: Add MCP OAuth database persistence with repositories and DAOs
- Implement OAuthClientRepository, OAuthTokenRepository, OAuthAuthorizationCodeRepository
- Add DAO methods in CollectionDAO for OAuth entities
- Create database migration scripts for OAuth tables (oauth_client, oauth_access_token, oauth_refresh_token, oauth_authorization_code)
- Add Fernet encryption for tokens and client secrets
- Implement SHA-256 hashing for token lookups
- Add OAuth connector plugin system (Snowflake, Databricks)
- Add scope authorization and validation
- Update ConnectorOAuthProvider to use database persistence
- Add comprehensive tests for OAuth provider
* Add MySQL migration for MCP OAuth tables (v1.12.1)
- Create oauth_client, oauth_authorization_code, oauth_access_token, oauth_refresh_token tables
- Convert Postgres schema to MySQL syntax
- Add indexes for performance optimization
- Tables manually applied in this session, migration framework integration needed
* feat: Complete MCP OAuth implementation with critical fixes and MCP Inspector support
1. **Scope Validation Fix**
- Set validScopes to null in McpServer to skip validation for connector-based OAuth
- Modified RegistrationHandler to skip validation if validScopes is empty
- Fixes: Client registration error "Invalid scope: api://apiId/.default"
2. **Metadata Endpoint URLs**
- Fixed all OAuth discovery endpoints to include /mcp prefix
- Updated OAuthHttpStatelessServerTransportProvider endpoint construction
- Ensures proper OAuth metadata discovery
3. **Token Exchange Security**
- Added client_id validation during token exchange
- Added redirect_uri validation to prevent security vulnerabilities
- Load authorization code from database for validation
- Prevents authorization code interception attacks
4. **Time Unit Consistency**
- Fixed deleteExpired methods to use seconds instead of milliseconds
- Updated OAuthTokenRepository and OAuthAuthorizationCodeRepository
- Enables proper cleanup of expired tokens and codes
5. **Authorization Code Loading**
- Fixed loadAuthorizationCode to load all fields from database
- Populates AuthorizationCode object with clientId, redirectUri, codeChallenge
- Resolves: NullPointerException during token validation
6. **Connector Name Parameter Support**
- Added connectorName field to AuthorizationParams
- Extract connector_name from HTTP request in AuthorizationHandler
- Priority: connector_name parameter > state (if not random hash) > default
7. **Default Connector Fallback**
- Detect random hash in state parameter (64 hex chars for CSRF)
- Default to test-snowflake-mcp connector for MCP Inspector testing
- Enables MCP Inspector to work without manual URL modification
8. **MySQL Migration**
- Added MySQL schema changes for OAuth tables
- Matches PostgreSQL schema structure
- Tables: oauth_clients, oauth_authorization_codes, oauth_access_tokens, oauth_refresh_tokens
9. **Documentation Cleanup**
- Removed 12+ redundant and outdated documentation files
- Created single comprehensive MCP_OAUTH_IMPLEMENTATION.md
- Added .shell-fix-note for shell script compatibility guidance
10. **Test Script Organization**
- Organized test scripts into scripts/mcp-oauth-tests/
- Added test-default-connector.sh for testing with MCP Inspector
- Preserved all OAuth flow testing scripts
- McpServer.java - Disabled scope validation for connector OAuth
- RegistrationHandler.java - Skip empty validScopes
- AuthorizationHandler.java - Extract connector_name parameter
- AuthorizationParams.java - Added connectorName field
- ConnectorOAuthProvider.java - Default connector logic, loadAuthorizationCode fix
- OAuthHttpStatelessServerTransportProvider.java - Fixed endpoints, added validations
- OAuthTokenRepository.java - Fixed time unit to seconds
- OAuthAuthorizationCodeRepository.java - Fixed time unit to seconds
- CollectionDAO.java - OAuth DAO registration
- DatabaseServiceRepository.java - Database service queries
- OAuthRecords.java - Database record types
- Deleted: 15+ outdated documentation files
- Deleted: Unused auth provider (OpenMetadataAuthProvider.java)
- Deleted: Unused OAuth callback servlet
- Added: Single comprehensive documentation file
✅ OAuth flow working end-to-end
✅ Client registration, authorization, token exchange successful
✅ Database persistence for all OAuth entities
✅ MCP Inspector compatibility with default connector
✅ Snowflake OAuth credentials configured for testing
⚠️ MCP Inspector SSE connection error (under investigation)
- OAuth authentication completes successfully
- Issue is with MCP protocol SSE connection, not OAuth
Run MCP Inspector:
```bash
npx @modelcontextprotocol/inspector http://localhost:8585/mcp
```
Test with default connector:
```bash
./test-default-connector.sh
```
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
* fix: Add CORS preflight support and security fixes for MCP OAuth
## CORS Fix
Allow OPTIONS requests without authentication in McpAuthFilter to support
CORS preflight checks from web-based MCP clients.
This enables proper CORS flow:
1. Browser sends OPTIONS preflight
2. Server responds with CORS headers (200 OK)
3. Browser sends actual POST request with Authorization header
4. Server authenticates and processes request
Without this fix, OPTIONS requests were blocked with 401, preventing
web clients from connecting to MCP endpoints.
## Security Fixes
### Critical Security Issues Fixed:
1. **Sensitive Token Logging** (95% severity)
- Sanitize OAuth request parameters before logging
- Remove client_secret, code, code_verifier, refresh_token, access_token from logs
- Prevents credential leakage in log files
2. **Token Expiry Integer Overflow** (100% severity)
- Changed all expiry timestamps from int/Integer to long/Long
- Fixes 2038 problem (32-bit timestamp overflow)
- Updated: AccessToken, RefreshToken, AuthorizationCode, ConnectorOAuthProvider, OAuthTokenRepository
3. **Hardcoded Default Connector** (80% severity)
- Made default connector configurable via MCP_DEFAULT_CONNECTOR env var
- Defaults to null in production (requires explicit connector_name)
- Prevents unauthorized access to test credentials in production
4. **Missing Null Checks** (85% severity)
- Added validation for token refresh response fields
- Validates access_token and expires_in exist before use
- Added bounds checking for expires_in (max 1 year)
5. **Missing Input Validation** (75% severity)
- Added connector name format validation
- Only allows: a-z, A-Z, 0-9, _, - characters
- Prevents path traversal and injection attacks
## Documentation
- Moved MCP docs to organized structure: openmetadata-mcp/docs/
- Created openmetadata-mcp/README.md with foundation documentation
- Moved implementation guide and testing guide to docs/ directory
## Cleanup
- Removed development test scripts (scripts/mcp-oauth-tests/)
- Removed .shell-fix-note and test-default-connector.sh
- Kept only clean final test script: test-mcp-with-token.sh
Changes:
- openmetadata-mcp/src/main/java/org/openmetadata/mcp/McpAuthFilter.java: OPTIONS CORS support
- openmetadata-mcp/src/main/java/org/openmetadata/mcp/server/transport/OAuthHttpStatelessServerTransportProvider.java: Sanitized logging
- openmetadata-mcp/src/main/java/org/openmetadata/mcp/server/auth/provider/ConnectorOAuthProvider.java: Multiple security fixes
- openmetadata-mcp/src/main/java/org/openmetadata/mcp/McpServer.java: Configurable default connector
- openmetadata-mcp/src/main/java/org/openmetadata/mcp/auth/*.java: Long timestamps
- openmetadata-mcp/src/main/java/org/openmetadata/mcp/server/auth/repository/OAuthTokenRepository.java: Long timestamps
Testing:
- OAuth flow: ✅ Working with any OAuth-enabled connector
- MCP protocol: ✅ Working via HTTP POST with JWT
- Default connector: Configurable via MCP_DEFAULT_CONNECTOR env var
- General solution: Works with ANY connector with OAuth credentials
Test command:
export MCP_DEFAULT_CONNECTOR=test-snowflake-mcp # For testing only
./test-mcp-with-token.sh
* feat: MCP OAuth security hardening and production readiness
Implemented security improvements and production configuration for MCP OAuth:
- Added constant-time secret comparison to prevent timing attacks
- Implemented token logging sanitization to protect sensitive credentials
- Fixed timestamp overflow (Integer → Long) to prevent 2038 issues
- Added input validation for connector names
- Implemented HttpClient resource cleanup (AutoCloseable)
- Added token refresh response validation with null checks
- Replaced hardcoded base URL with dynamic SystemRepository configuration
- Fixed MCP Inspector compatibility (removed unimplemented logging capability)
- Added example credential files and test setup documentation
- Removed commented code and unused files for cleaner codebase
Security TODOs documented for future work:
- Race condition in authorization code exchange (requires DB schema changes)
- Rate limiting for OAuth endpoints (requires new infrastructure)
Testing:
- All changes tested with Snowflake OAuth connector
- MCP Inspector connection verified working
- Code formatted with spotless
Breaking Changes: None
* fix: Address security vulnerabilities from code review bots
Implemented fixes based on automated code review bot findings:
**Critical:**
- SSRF prevention: Added URL validation in OAuthSetupHandler to block private IPs and validate schemes
- ThreadLocal leak: Added try-finally cleanup in doGet() to prevent auth context leakage
**High:**
- Removed hardcoded JWT tokens and client secrets (replaced with dynamic UUIDs)
- Added warning logs for missing connector names to improve auditability
Security impact: Prevents internal network access, credential exposure, and auth state leakage.
Testing: All changes formatted with spotless and validated.
* fix: Optimize SSRF prevention per code review bot recommendations
Improved SSRF mitigation based on detailed bot feedback:
**Optimization:**
- Refactored validateTokenEndpoint() → validateAndResolveTokenEndpoint()
- Returns validated URI object to avoid double parsing
- Integrates endpoint resolution and validation in single method
- Reuses URI throughout method to prevent inconsistencies
**Implementation Details:**
- Validates URL scheme, host, and IP ranges
- Blocks private IPs (10.x, 192.168.x, 172.16-31.x)
- Blocks link-local addresses (169.254.x)
- Validates before HTTP request and credential storage
**Benefits:**
- More efficient (single URI parse instead of two)
- Safer (validated URI reused consistently)
- Cleaner code (DRY principle)
Based on GitHub Copilot autofix suggestion for SSRF vulnerability.
* fix(mcp-oauth): Critical security fixes per code review bots
- SSRF: Add DNS resolution and validate all resolved IPs for token endpoints
- Race condition: Atomic authorization code exchange prevents replay attacks
- Refresh token: Fix expiry check using ofEpochSecond instead of ofEpochMilli
- Remove unrelated ingestion yaml files from PR
Addresses: CodeQL, Copilot Autofix, Gitar bot feedback
* fix(mcp-oauth): Address bot feedback - security and code quality
- Remove shell scripts with hardcoded JWT tokens from PR (added to .gitignore)
- Fix admin fallback: Use ingestion-bot instead of admin for security
- Fix connector name validation: Fail refresh if connector name missing
- Add TODO comments for hardcoded localhost URIs (requires MCPConfiguration wiring)
Addresses bot feedback on security concerns and configuration flexibility
* fix: SSRF - reconstruct URI from validated components
* fix: CodeQL suppression, Y2038 bug, test provider safeguards
* MCP OAuth: implement CORS development mode detection and token cleanup scheduler
- Add development mode detection for CORS origins based on baseUrl
- Development: allow localhost origins with warning
- Production: empty allowedOrigins (same-origin only) with warning
- Implement OAuth token cleanup scheduler with Quartz
- OAuthTokenCleanupJob: deletes expired tokens and auth codes
- OAuthTokenCleanupScheduler: runs cleanup hourly
- Prevents unbounded token table growth
* fix: SSRF with allowlist and rate limiting
Use allowlist for OAuth endpoints, add rate limiting (10/5 req/min)
* fix: SSRF, OAuth security, and MySQL schema bugs
- SSRF: Remove user-provided tokenEndpoint, always infer from connector config using allowlist
- Schema: Fix MySQL table names (plural), authorization codes schema, add missing tables
- OAuth: Restore session redirect URI and re-enable nonce validation
* fix: Duplicate clientId variable and missing user_name column in Postgres migration
* security: Remove sensitive OAuth tokens and authorization codes from log statements
* security: Remove sensitive client metadata from registration logs
* chore: Remove connector OAuth infrastructure for user SSO implementation
* feat: Add MCP user SSO OAuth MVP implementation
- Updated database schema (MySQL + PostgreSQL) to use user_name instead of connector_name
- Removed connector OAuth infrastructure (plugins, ConnectorOAuthProvider)
- Created UserSSOOAuthProvider MVP skeleton with TODO markers
- Added comprehensive IMPLEMENTATION_TODO.md tracking all remaining work
- Added QUICK_START.md guide for setup instructions
- Added Claude Desktop configuration example
- Maintained backward compatibility with PAT authentication
See openmetadata-mcp/docs/IMPLEMENTATION_TODO.md for complete implementation checklist
* feat: Complete MCP OAuth SSO flow with database-backed state persistence
This commit implements a robust OAuth SSO flow for MCP server integration
that survives cross-domain redirects during SSO authentication (Google, etc).
Key changes:
- Add mcp_pending_auth_requests table for database-backed state storage
- Add McpPendingAuthRequestRepository for managing pending auth requests
- Add SSOCallbackServlet to handle SSO provider callbacks
- Add handleDirectIdTokenFlow for already-authenticated users (pac4j token flow)
- Add HtmlTemplates for secure error pages with XSS protection
- Add Claude Desktop OAuth bridge script for stdio transport integration
- Fix OIDC_CREDENTIAL_PROFILE constant shadowing issue
- Fix Postgres schema references to non-existent connector_name column
- Restore pac4j session attributes (State, Nonce, CodeVerifier) correctly
The solution stores OAuth state in the database instead of HTTP sessions,
which fail across cross-domain redirects due to SameSite cookie policy.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix: Critical OAuth security fixes - thread safety, URL encoding, JWT validation, PKCE validation
* fix: Complete ThreadLocal migration for currentRequest.getSession()
* feat: Add development bypass for PKCE validation to enable local testing
* feat: Add OAuth support with ID token validation, refresh tokens, and security fixes
- Add JWKS-based ID token signature validation
- Implement refresh token generation and exchange with rotation
- Add redirect URI validation to prevent open redirect attacks
- Fix clock skew logic and time unit consistency
- Add comprehensive test coverage (15 tests)
* fix: Critical OAuth security fixes - client validation, redirect URI validation, error handling, Fernet decryption
- Add client ID validation in token exchange (prevents authorization code theft)
- Add redirect URI validation in token exchange (RFC 6749 Section 4.1.3)
- Fix time unit inconsistency in OAuthAuthorizationCodeRepository
- Improve error handling to distinguish replay attacks from expired codes
- Add user status validation in refresh token exchange
- Fix session regeneration to prevent session fixation attacks
- Add username/email validation in SSO callback handlers
- Improve Fernet decryption error handling for key rotation scenarios
All tests passing (15/15)
* fix: Clean up pom.xml - fix malformed dependency and remove duplicate dropwizard-jersey
* javacheck style fix
* fix: Addressing issues raised by Gitar code review
* fix: Merge McpAuthFilter changes - add impersonation support while preserving OAuth endpoints
* docs: Add comprehensive README for MCP OAuth implementation
* feat: Add MCP OAuth dynamic client registration
* feat: Add OAuth token revocation endpoint (RFC 7009)
* fix: OAuth basic auth flow - auto-redirect with code and optional scope enforcement
* feat: Match MCP auth page design to OpenMetadata signin UI
* fix: Support separate callback URLs for MCP OAuth and web login flows
* feat: Add OAuth scope enforcement, domain validation and session handling for MCP
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
* feat: Improve MCP OAuth login UI and add TODO for success page
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
* fix: MCP OAuth cleanup - security fixes, remove redundant scope system, improve error handling
- Fix timing attacks in CSRF and PKCE validation using MessageDigest.isEqual()
- Remove redundant @RequireScope system (OpenMetadata Authorizer handles permissions)
- Make OAuth scopes provider-aware (Google/Okta/Azure)
- Add baseUrl config to MCPConfiguration for cluster deployments
- Delete duplicate RootOAuthEndpointsResource (handled by OAuthWellKnownFilter)
- Fix silent failures: propagate errors instead of returning null/200
- Downgrade excessive logging to DEBUG level
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Update generated TypeScript types
* fix: Move OAuth migrations from 1.12.1 to 1.12.0
- Consolidate OAuth schema tables into 1.12.0 migration
- Add Snowflake backward compatibility migration to 1.12.0
- Remove empty 1.12.1 migration folder
- Update README with security enhancements and permission model
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix: critical OAuth security and reliability issues
Fix ThreadLocal leak, atomic token rotation, PKCE validation, fail-closed error handling, and password sanitization
* fix: URL encode authorization code
* fix: MCP OAuth stateless transport compatibility and SSO initialization reliability
* feat: Add MCP configuration to database settings system
- Create mcpConfiguration.json schema for MCP-specific settings
- Add MCP_CONFIGURATION to SettingsType enum
- Add MCP configuration bootstrap logic to SettingsCache
- Extend SecurityConfigurationManager with MCP config support
- Add mcpConfiguration field to OpenMetadataApplicationConfig
- Update MCPConfiguration.java with timeout settings and comments
* feat: Complete McpServer dynamic configuration resolution
- Add getBaseUrlFromConfig() to read from SecurityConfigurationManager with fallback
- Add getAllowedOriginsFromConfig() for database-backed CORS configuration
- Remove hardcoded baseUrl and CORS origins initialization
- Remove System.setProperty for HTTP timeouts (will be handled per-request)
- Fix SSO handler to use dynamic resolution via getInstance()
- Fix NoSuchAlgorithmException import in UserSSOOAuthProvider
- All configuration now comes from database via SecurityConfigurationManager
* Update generated TypeScript types
* feat: Add database-backed MCP configuration with dynamic reload
- Add GET/PUT /api/v1/system/mcp/config API endpoints for MCP configuration management
- Refactor SSOCallbackServlet to read claims/domains/validators dynamically from SecurityConfigurationManager
- Add configuration reload support to OAuthHttpStatelessServerTransportProvider (volatile allowedOrigins, updateAllowedOrigins method)
- Implement ConfigurationChangeListener pattern in SecurityConfigurationManager for component notification
- Add HTTP timeout configuration (connectTimeout/readTimeout) to AuthenticationCodeFlowHandler from MCP config
- All configuration stored in open_metadata_settings table with SecurityConfigurationManager as single source of truth
* fix: Add volatile config fields, CopyOnWriteArrayList, null checks, and correct HTTP timeout properties
* Remove hardcoded OAuth credentials and unrelated Snowflake migration
* Fix HTTP timeout system properties and session regeneration null check
* Implement cluster polling, DB-first loading, listener pattern, and fix race conditions
* added unit tests
* removed connector OAuth code
* updated readme
* fix: MCP OAuth cleanup — security fixes, migration move, and code quality
- Move OAuth SQL migrations from 1.12.0 to 1.12.1 (release target)
- Fix XSS in auth error page (no longer reflects exception messages into HTML)
- Fix CSRF bypass in state validation (throw instead of return-after-write)
- Fix token expiration check in BearerAuthenticator (millis vs seconds mismatch)
- Require S256 code_challenge_method explicitly (reject null/plain)
- Fix GetLineageTool: use VIEW_BASIC auth, add input validation, use singleton LineageRepository
- Rename SESSION_GOOGLE_CALLBACK_URL to SESSION_SSO_CALLBACK_URL (provider-agnostic)
- Remove 10-second config polling from SecurityConfigurationManager (use SettingsCache TTL)
- Remove unnecessary synchronized on volatile field getters
- Downgrade verbose LOG.info calls to LOG.debug (session state, admin principals, tokens)
- Fix FQN imports in AuthenticationCodeFlowHandler (MCPConfiguration, Role)
- URL-encode redirect parameters (id_token, email, name)
- Remove invalid "default": null from defaultOAuthRole JSON schema
- Add error logging in AuthorizationHandler.exceptionally() block
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* add TODOs for unfixed security review findings
* fixed critical review issues: added client_secret validation, registration rate limiting, session regeneration bug, exact path matching, dead code removal
* fixed auth filter 500→401 for invalid tokens, exact path matching in transport provider
* added revocation client auth, redirect URI scheme validation, ID token validation in SSO flow, rate limiter race fix, downgraded PII logging to DEBUG
* fix MCP config loading to use getSettingOrDefault, cache IdTokenValidator
* google sso login working here
* add basic auth login flow for MCP OAuth, fix web UI redirect_uri_mismatch
* revert cosmetic UI formatting changes accidentally introduced in merge
* fix CodeQL info exposure and GitarBot security findings: redirect_uri validation, pac4j race condition
* harden MCP OAuth: fix error handling, remove dead code, prevent info leaks
* remove dead code and harden MCP OAuth: delete 5 unused files, inline metadata handlers, add PKCE validation, fix error handling
* fix GitarBot findings: restrict HTTP redirects to loopback, add token rate limiting, restore GET 405, deny-all CORS fallback, reduce JWK cache TTL
* fix Azure SSO: always register callback servlet, use baseUrl for token exchange, show success page
* security hardening: early user check, ID token audience validation, token rotation, shorter JWT TTL
* LDAP support, allow native app redirect schemes, tolerate unknown registration fields
* fix open redirect in MCP callback detection, check auth code expiry before consumption, warn on fallback baseUrl
* null safety for PKCE, grant_type, and refresh_token params in token endpoint
* fix RevocationHandler test exception type mismatch
* add registration metadata length validation, fix loopback host check
* fix MCP OAuth SSO callback for Okta: use registered redirect_uri, fix pac4j session attribute names, forward /callback to /mcp/callback
* fix missing return in MCP callback error path, skip SSO registration for basic/ldap, improve comment
* MCP OAuth security hardening: bcrypt secrets, atomic CAS rotation, XFF rate limiting, review fixes
* fix XFF rate-limit bypass: validate IP format, cap map size to prevent heap exhaustion
* move MCP OAuth migrations from 1.12.2 to 1.12.3, remove unused oauth_audit_log table, simplify
* fix client_secret_basic removal, MySQL index idempotency, token auto-delete on decrypt failure
* Update generated TypeScript types
* Update generated TypeScript types
* fix impersonation compatibility after McpAuthFilter deletion
* hash authorization codes with SHA-256 before storing in DB
---------
Co-authored-by: mohitdeuex <mohit.y@deuexsolutions.com>
Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
|
||
|
Vishnu Jain
|
0e8de77dd0
|
Mcp impersonation (#26488)
* fix MCP bot impersonation and app registration * add MCP audit log impersonation and change event publishing * add unit tests for MCP audit log and impersonation context * fix getMcpBotName startup race and remove unused WEBSOCKET_HANDLER * Fix: enforce limits in CreateTestCaseTool like other create tools * Fix: add migration for McpApplicationBot impersonation * Move allowBotImpersonation to app definition schema instead of hardcoding * Update generated TypeScript types * Fix McpAuthFilter error handling --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> |
||
|
Sriharsha Chintalapani
|
a4998bc1c7
|
Continuous indexing to handle failures (#26111)
* Add Continuous Indexing
* Add continuous Search indexing
* Update to 1.12.3
* Make search index retry queue reliable with stale recovery, health checks, and silent failure coverage
- Add entityType, retryCount, claimedAt columns to search_index_retry_queue table
- Implement stale IN_PROGRESS recovery (10min threshold, 60s sweep interval)
- Replace static isClientAvailable flag with cached ping health check (5s TTL)
- Narrow catch blocks in resolveById/resolveByFqn to EntityNotFoundException
- Use entityType hint for O(1) entity resolution instead of scanning all types
- Switch from status-string-based retry to retryCount-based (< 3 retries → PENDING, ≥ 3 → FAILED)
- Batch cascade reindex at 200 entities instead of accumulating up to 5000
- Add retry queue enqueue in catch blocks of createTimeSeriesEntity, updateTimeSeriesEntity,
deleteTimeSeriesEntityById, bulkIndexPipelineExecutions, reindexAcrossIndices, and
TestSuiteRepository.postCreate
- Re-throw exceptions from indexTableColumns/deleteTableColumns to parent catch blocks
- Add Micrometer counters for enqueued, processed (success/failure), and stale recovered
* Add missing lineage call site and Add test
* Review comments
* Add resilience to search index retry worker: client availability checks, backoff, and error classification
- Add exponential backoff when search client is unreachable so the
worker does not burn retries during cluster outages (5s → 10s → … → 60s cap)
- Classify errors using HTTP status codes from ES/OS exceptions:
4xx (except 429) are non-retryable and skip straight to FAILED;
429, 5xx, and IOException are retryable
- Preserve first bulk failure detail in RuntimeException so error
classification works for the bulk indexing path
- Reorganize SearchIndexRetryWorker into clearly separated sections
(lifecycle, main loop, record processing, entity resolution,
reindexing, resilience, suspension, utilities)
- Add isRetryableStatusCode utility to SearchIndexRetryQueue
- Add integration tests: status code classification, retry exhaustion
to FAILED, recovery from PENDING_RETRY_1, error detail preservation
* Address review comments
* Revert fqn size
* Spotless
* Address volatile review comments
* Fix Failing Test
* update review comments
---------
Co-authored-by: mohitdeuex <mohit.y@deuexsolutions.com>
Co-authored-by: Mohit Yadav <105265192+mohityadav766@users.noreply.github.com>
|
||
|
Sriharsha Chintalapani
|
6d99ba2dc0
|
Glossary relations (#25886)
* Glossary Term Relations * Add GlossaryTerm Relations * Add GlossaryTerm Relations, Add custom relations, onotolgoy explorer * Add Translations * Update generated TypeScript types * Address comments * Address comments * Address comments * Update generated TypeScript types * Update yarn.lock after merging cytoscape dependencies from glossary_relations * fix zoom in and out functionality and added missing translate keys * fix test * Remove unwanted changes * nit * nit * nit * Remove conflict test * nit * fix test * Add test for ontology explorer * New yarn lock and 2.0.0 schema changes missed during merge conflicts * Revamped glossary term relation settings * Refactor code * Addressed comments * nit * Update generated TypeScript types * Java Checkstyle and Yarn lock * Update generated TypeScript types * fix unit test * Remove 2.0.0 migration folders placed at wrong loc * Merge main * fix navigation to relation graph in glossary * fix ontology explorer spec * Added filter support in the data mode * Fix glossary term relation CI failures ### Canonical Relation Storage (GlossaryTermRepository) * Introduced `computeCanonicalRelationType()` to normalize relation direction using UUID ordering (lower UUID is always treated as "from") * Prevents duplicate and inconsistent relation rows when created from either side * Updated `setTermRelations()` and `addRelation()` to store canonical relation types * Fixed `setFields()` read logic: * Invert relation type for `fromRecords` (entity is the TO side) * Keep `toRecords` unchanged * Updated `deleteBidirectionalRelatedTo()` to match canonical storage format * Added `RequestEntityCache.invalidate()` after relation mutations to ensure consistency ### Lazy RDF Resource Initialization * Added `RdfRepository.getInstanceOrNull()` for null-safe access without throwing * Refactored `RdfResource` constructor to avoid eager `RdfRepository.getInstance()` call * Enabled resource registration even when Fuseki is not initialized * Introduced lazy getters: * `getRdfRepository()` * `getSemanticSearchEngine()` * Updated all endpoints to guard with null checks before `isEnabled()` * Return `503 Service Unavailable` when RDF is not ready ### Graceful Test Degradation (Fuseki-dependent tests) * Added `TestSuiteBootstrap.isFusekiEnabled()` to detect Fuseki availability * `GlossaryOntologyExportIT`: * Falls back to Testcontainers-based local Fuseki when bootstrap Fuseki is unavailable * `GlossaryTermRelationIT`: * Skipped via `assumeTrue` when Fuseki is unavailable * `MetricResourceIT`: * Skips RDF-specific tests when Fuseki is unavailable * fix package conflicts * nit * Fix merge conflicts, Python test, RDF reliability, and VectorDocBuilder tests - Fix Python test_patch_glossary_term_related_terms to use TermRelation instead of EntityReferenceList (schema changed relatedTerms type) - Rewrite VectorDocBuilder tests for current buildEmbeddingFields API - Improve JenaFusekiStorage retry logic to retry on all HTTP errors - Increase Fuseki tmpfs size to prevent disk space exhaustion in tests Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix pycheck * Address all 8 PR review findings 1. Add authorization check on getTermRelationGraph endpoint 2. Add null guard on getBaseUri() to prevent NPE 3. Add React key prop on RelatedTermTagButton in map renders 4. Mark RdfResource lazy-init fields as volatile for thread safety 5. Replace exception messages with generic errors in API responses 6. Unify DEFAULT_RELATION_TYPES between CSV and repository (10 types) 7. Add jitter backoff to deadlock retry in CollectionDAO 8. Replace N+1 queries in prefetchGraphTerms with batch fetch Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Fix Fuseki tmpfs exhaustion and GlossaryTermRelationIT double init - Remove tmpfs size limit on Fuseki container to prevent disk exhaustion - Guard RdfUpdater.initialize() in GlossaryTermRelationIT to skip if already initialized by bootstrap Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Fix duplicate edges, null term NPE, and silent exception in graph builder - Deduplicate edges in buildGraph() using edgesSeen set - Skip TermRelation entries with null term references to prevent NPE - Add warning log when glossary term relation settings fail to load Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Fix cardinality count after canonical swap and double-checked locking - getRelationCount now matches inverse relation type for fromRecords where the term is the target, fixing cardinality bypass after bidirectional UUID canonicalization - Use double-checked locking in RdfResource.getSemanticSearchEngine() to prevent duplicate instance creation under concurrency Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: anuj-kumary <anujf0510@gmail.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Co-authored-by: Ram Narayan Balaji <ramnarayanb3005@gmail.com> Co-authored-by: Ram Narayan Balaji <81347100+yan-3005@users.noreply.github.com> |
||
|
Sriharsha Chintalapani
|
12b364313c
|
Fix Metrics collection; reduce no.of metrics; improve slow request lo… (#25751)
* Fix Metrics collection; reduce no.of metrics; improve slow request logging
* Move sync calls to search & rdf to async
* Improve slow request tracking
* Improve slow request tracking
* Add clear breakdown in slow request
* Batch TestCaseRepository calls
* Batch API calls
* Initial Implementation of ReadEngine
* Improvements with ReadEngine/WriteEngine
* Improvements with ReadEngine/WriteEngine
* Improvements with ReadEngine/WriteEngine
* Improve by removing unnecessary ser/de
* Additional improvements with PatchFieldsPlanner
* Further performance improvements
* Further performance improvements
* Address comments
* Merge from main
* Address comments
* Address comments
* Address latest feedback - 2/21
* fix merge conflict
* Address Slow Request review
* Address the comments
* Address comments; Fix tests
* Fixes to the failing tests
* Fix bugs in tests
* Fix checkstyle
* Address playwright tests
* Fix tests
* Fix bugs
* Fix tests
* address comments
* Fix issues from playwright
* Fix playwright tests
* Fix tests for playwright
* Address comments
* Fix glossary test
* fix checkstyle
* Fix playwright issues
* Fix playwright issues - incrementalChagneDesc
* Restore ApprovalTaskWorkflow in GlossaryTerm and TestCase repositories
The slow_request branch accidentally removed entity-specific ApprovalTaskWorkflow
overrides, causing the generic parent to use checkUpdatedByTaskAssignee instead of
checkUpdatedByReviewer. This broke Glossary approval and TestCase approval Playwright tests.
- GlossaryTermRepository: restore ApprovalTaskWorkflow with checkUpdatedByReviewer
- TestCaseRepository: restore ApprovalTaskWorkflow, preDelete guard, updateReviewers
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Fix base ApprovalTaskWorkflow to use reviewer check instead of task assignee
The centralized ApprovalTaskWorkflow in EntityRepository was using
checkUpdatedByTaskAssignee instead of checkUpdatedByReviewer, breaking
approval workflows for all entity types. Added verifyReviewer() as a
top-level static method on EntityRepository and restored missing
updateReviewers() and preDelete IN_REVIEW guards in DataContract,
DataProduct, Metric, and Tag repositories. Removed now-redundant
entity-specific ApprovalTaskWorkflow overrides from GlossaryTerm and
TestCase repositories.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Fix regression introduced in backend tests; make the playwright tests stable
* Stabilize the playwright tests
* Stabilize the playwright tests
* Improve playwright tests
* Improve playwright tests
* Fix team playwrights
* Fix merge from main
* Fix playwrigt tests
* Fix playwright tests
* Batch domain/data product asset counts into single ES aggregation queries
Replace N individual ES count queries with single aggregation query per
entity type. Domain counts roll up child counts to parent domains.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Improve Playwright test reliability and expand CI shards
Add polling waits for async ES indexing, fix lineage edge selectors,
use API-based setup for domain/data product widget tests, and expand
CI from 6 to 8 shards with dedicated graph/landing projects.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Playwright: Improve test reliability with response checks and guards
- Add API response status checks in create() for Domain, DataProduct,
Glossary, TableClass, and UserClass — silent API failures now throw
immediately with status code and response body
- Add guards in selectDataProduct() and addAssetsToDataProduct() for
undefined name/fqn — clear error messages instead of cryptic
"locator.fill: value: expected string, got undefined"
- Fix GlossaryPermissions double navigation — remove redundant
redirectToHomePage + sidebarClick before glossary.visitEntityPage()
- Increase OnlineUsers timeout from 5s to 15s for CI resource pressure
- Increase Tour badge timeout from 10s to 20s
- Fix visitGlossaryPage: wait for loader before clicking menuitem
- Remove chromium testIgnore for graph/landing/stateful test files
(these must run in chromium project for 6-shard CI workflow)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Playwright: Remove all networkidle waits and improve CI reliability
- Remove ~780 networkidle waits across 144 test/utility files — these
hang or resolve prematurely under CI load causing false negatives
- Add polling.ts with waitForSearchIndexed and waitForPageLoaded helpers
- Convert checkAssetsCount and search functions to expect.poll() for
async ES indexing tolerance
- Increase expect timeout to 15s for CI environments
- Split CI into 8 shards with dedicated projects (stateful/graph/landing)
to reduce thread contention
- Fix GITHUB_STEP_SUMMARY size overflow (base64 screenshots → table)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Playwright: Fix genuine test failures from networkidle removal
- GlossaryPagination: Fix waitForResponse race conditions - register
listener BEFORE the triggering action, add **/ URL prefix
- LanguageOverride: Fix selector from getByText('EN') to
getByText('English - EN') matching actual dropdown text
- NestedColumnsExpandCollapse: Fix URL glob pattern, use dispatchEvent
to avoid inner Link navigation, add waitForResponse for filtered search
- lineage.ts: Revert dragConnection hover approach that broke React
Flow connection mode, keep direct dispatchEvent
- customizeLandingPage.ts: Remove waitForURL that hangs after page.goto
- Teams.spec.ts: Add isJoinable: false for private team creation
- UserDetails.spec.ts: Revert Escape/clickOutside save flow that
dismissed edit mode before saving roles
- Users.spec.ts: Revert Data Consumer permissions test to original
simple approach using fixtures
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Playwright: Relax OnlineUsers activity time assertion
The "Online now" exact match fails under CI load because the activity
timestamp may show as "X seconds ago" or "X minutes ago" by the time
the page renders. Changed to accept any recent activity format.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Playwright: Fix 4 genuine test failures from CI run
1. saveCustomizeLayoutPage: Use response predicate matching both
POST (create) and PUT (update) patterns instead of glob that
only matched updates. Fixes 180s timeout in drag-and-drop test
when layout doesn't exist yet (fullyParallel=true).
2. GlossaryMiscOperations: Add test.slow(true) — test does 9
sequential page navigations that exceed the 60s timeout.
3. DomainDataProductsWidgets "Assign Widgets": Add test.slow(true)
— calls addAndVerifyWidget twice, each with multiple navigations.
4. DomainFilterQueryFilter: Add waitForAllLoadersToDisappear before
clicking domain-dropdown after search operations that trigger
page re-renders.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Playwright: Fix AutoPilot test — reload page after API status poll
The AutoPilot status banner never appeared because:
1. checkAutoPilotStatus polls the workflow API directly via apiContext
(outside the browser), not through page network requests
2. The UI uses WebSocket for live updates, but the socket connection
is only established when the page loads with status=RUNNING
3. Since the page loaded before the workflow started, the socket was
never connected, so the UI never received the completion event
Fix: reload the page after checkAutoPilotStatus confirms the workflow
finished, so the UI renders with the current state. Also increase the
banner visibility timeout to 30s for CI environments.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Playwright: Fix flaky tests — entity collisions, missing cleanup, expect timeout
- Replace Date.now() with uuid() for entity names in CustomProperties tests
to prevent collisions when parallel workers execute within the same millisecond
- Fix FollowingWidget: move shared adminUser create/delete to top-level
base.beforeAll/afterAll to prevent duplicate user creation across 11
parallel test.describe blocks
- Add missing afterAll cleanup to OnlineUsers, Metric, CustomPropertyAdvanceSearch,
and CustomProperties tests to prevent entity/user leaks between runs
- Replace hardcoded metric name in MetricSearch with uuid-based name
- Add global expect timeout of 15s (up from 5s default) for CI resilience
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Fix Playwright CI: include UI in build-once Maven build
The build-once optimization (#26423) used -DonlyBackend -pl !openmetadata-ui
which produces a tar.gz without the compiled React app. The Docker container
starts but cannot serve the login page, causing auth.setup.ts to timeout
on all 6 shards waiting for input[id="email"] to appear.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Fix CodeQL security warnings
- Replace Math.random() with crypto.randomUUID() for test data generation
- Escape backslash characters in CSS selectors for glossary FQN values
- Use page.getByTestId() instead of raw CSS selectors in entity utils
- Increase RSA key size from 512 to 2048 bits in JwtFilterTest
- Skip archive entries containing '..' in JsonUtils.getResourcesFromJarFile
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Playwright: Fix user cleanup to prevent 'Email Already Exists' failures
- Glossary.spec.ts: Fix typo user3.create→delete in afterAll, add missing adminUser.delete
- Teams.spec.ts: Add afterAll cleanup hooks for 3 nested describe blocks that were missing them (EditUser, DataConsumer, Owner)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Playwright: Add afterAll cleanup hooks and fix test reliability
- InputOutputPorts.spec.ts: Add afterAll for domain/tables/topics/dashboards
- Users.spec.ts: Add top-level afterAll for all shared entities
- Entity.spec.ts: Add afterAll for shared + per-entity-type cleanup
- Pagination.spec.ts: Add afterAll for 13 describe blocks (services, DBs, etc.)
- DataProductRename.spec.ts: Add afterAll cleanup
- TestCaseIncidentPermissions.spec.ts: Add afterAll for users/roles/policies/table
- ImpactAnalysis.spec.ts: Add afterAll for all 7 entity types
- NestedColumnsExpandCollapse.spec.ts: Add afterAll for 4 describe blocks
- DataProductPermissions.spec.ts: Add afterAll cleanup
- ServiceEntityPermissions.spec.ts: Add afterAll for testUser + per-entity
- ServiceForm.spec.ts: Add afterAll for adminUser
- domain.ts: Replace waitForTimeout(2000) with proper loader/tab waits
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Trigger Playwright CI
* Playwright: Fix 2 failures and 26 flaky tests with proper waits
Fix remaining 2 genuine failures:
- DomainDataProductsWidgets: add test.slow(true) for ES indexing lag
- Users.spec.ts: add test.slow(true) and loader waits for owner search
Fix 26 flaky tests by addressing 5 root cause patterns:
- Response listener after trigger: MetricCustomUnitFlow, DomainUIInteractions
- Missing loader wait after navigation: 16 tests across CustomizeDetailPage,
DataProductPersonaCustomization, DataContracts, ExploreTree, and others
- Element not rendered after API response: EntityVersionPages, ODCSImportExport
- DOM not settled after loader: Domains nested rename
- Permission cache propagation: GlossaryPermissions
Shared utility improvements:
- waitForPatchResponse uses entity-specific URL pattern
- openColumnDetailPanel accepts entityEndpoint param with API response wait
- Entity.spec.ts uses dynamic entity.endpoint instead of hardcoded tables
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Playwright: Fix addOwner retry to wait for search API response
The owner search retry loop was refilling the search input but not
waiting for the API response before checking item visibility. This
caused the poll to repeatedly check stale/empty results.
Fix: await search response and loader detach in each retry iteration.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Playwright: Fix owner listitem selector — remove exact match
The owner selection list items include avatar initials (e.g., "G") in their
accessible name, making exact: true fail since the accessible name is
"G UserName" not just "UserName". Switching to substring matching fixes
the Users.spec.ts persistent failure.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Playwright: Fix 10 remaining flaky tests with proper waits
- ColumnLevelTests: loader wait after visiting test case panel
- DataQualityPermissions: loader wait after visiting test suite page
- IncidentManagerDateFilter: loader wait after page reload
- InputOutputPorts: wait for warning alert before asserting
- Lineage: replace 5 hardcoded waitForTimeout(500) with loader waits
- CustomizeDetailPage: dialog close waits, fix missing await on expect
- DataProductPersonaCustomization: loader wait + modal visibility check
- GlossaryPermissions: increase permission propagation wait, loader wait
- GlossaryHierarchy: loader waits after modal close and glossary select
- ExploreTree: loader waits after API response before UI interaction
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Fix CodeQL security alerts: incomplete escaping and Zip Slip
1. entity.ts: Use JSON.stringify().slice(1,-1) for proper escaping of
both backslashes and double quotes in filter values, replacing the
incomplete .replace(/"/g, '\\"') approach.
2. JsonUtils.java: Strengthen Zip Slip protection by normalizing paths
via Paths.get().normalize() and rejecting entries starting with "/"
or resolving to parent traversal after normalization.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Fix tests
* Fix tests
* Fix recordChange field name mismatches and CodeQL alert
- ServiceEntityRepository: recordChange("ingestionAgent") → "ingestionRunner"
to match the JSON property name. The shouldCompare() gate in PATCH flow
was silently dropping ingestionRunner changes because the field name
didn't match patchedFields.
- DataContractRepository: compareAndUpdate("status") → "entityStatus"
to match the JSON property name, same root cause.
- JsonUtils: Simplify Zip Slip check to string-based validation to
satisfy CodeQL taint analysis.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Remove serial mode from Users.spec.ts to prevent cascade failures
A single flaky test failure was causing ~19 tests across 5 unrelated
describe blocks to be skipped. Matches main branch behavior (parallel).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Playwright: Fix flaky tests — missing awaits, hardcoded waits, silent catches
- DataProductPersonaCustomization: add missing await on expect() calls
- TestCaseIncidentPermissions: poll for incident creation instead of one-shot query
- TestCaseResultPermissions: add loader wait after Data Quality tab click
- GlossaryPermissions: replace waitForTimeout(3000) with toPass() retry
- BulkImport: remove 4 unnecessary waitForTimeout calls
- importUtils/testCases: replace waitForTimeout(500) with grid visibility assert
- GlossaryAssets: add loader wait, remove silent .catch(() => false) pattern
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Fix CodeQL Zip Slip alert with Path.normalize() sanitization
CodeQL doesn't recognize String.contains("..") as proper Zip Slip
mitigation. Use Path.normalize() + isAbsolute/startsWith checks which
CodeQL's taint analysis model understands.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Fix Playwright flaky tests: modal visibility, toast race, query card assertion
- DataProductPersonaCustomization: wait for dialog close before clicking add-widget-button
- entity.ts restoreEntity: dismiss stale toast before restore to avoid race condition
- QueryEntity: replace page.$$() with auto-retrying expect().toBeVisible()
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Fix flaky TableResourceIT by preventing parallel multi-domain rule mutation
Both test_multipleDomainInheritance (TableResourceIT) and
test_csvImportEntityRuleValidation (DatabaseServiceResourceIT) toggle
the global "Multiple Domains are not allowed" rule. When running
concurrently, one overwrites the other's setting causing spurious
failures. Add @ResourceLock("MULTI_DOMAIN_RULE") to serialize only
these two tests while keeping all others concurrent.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Mohit Yadav <105265192+mohityadav766@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
|
||
|
Pere Miquel Brull
|
62c12a133d
|
Fix 1.13.0 preview→enabled migration for event subscriptions (#26473)
* Fix preview→enabled migration for event_subscription_entity and QRTZ tables The 1.13.0 migration renamed `preview` to `enabled` in `apps_marketplace` and `installed_apps`, but missed the `event_subscription_entity` table. The ReverseMetadata app stores the full App entity as an escaped JSON string inside `event_subscription_entity.json -> config -> app`. Since it's a string value (not a nested JSON object), standard JSON path operations can't reach the `"preview"` field — string replacement is needed instead. Also truncates QRTZ tables to clear stale Quartz job data that may contain old App JSON. Both schedulers re-create their jobs from the database on startup. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Use DELETE instead of TRUNCATE for QRTZ cleanup to respect FK constraints TRUNCATE fails on tables referenced by foreign keys in MySQL (and without CASCADE in PostgreSQL). Switch to DELETE FROM with correct FK ordering (children before parents) and add missing child tables (QRTZ_SIMPROP_TRIGGERS, QRTZ_BLOB_TRIGGERS). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> |
||
|
Teddy
|
8270e01415
|
ISSUE-3030 - Profiler autotune threading (#26385)
* FIX - Redshift converter (#26229)
(cherry picked from commit
|
||
|
Ram Narayan Balaji
|
bb6a99b953
|
Feat# Include Fields Filter in EventBased Workflows and CheckChangeDescription Node (#26230)
* Include Fields in EventBased Workflows - Initial Commit
* Update generated TypeScript types
* Fix Include fields to be a map of arrays, Introduce checkChangeDescriptionTask as a separate node
* Update generated TypeScript types
* Extract common code into field value extractor
* chore: apply changes
Co-authored-by: yan-3005 <yan-3005@users.noreply.github.com>
* java checkstyle
* Fix Compilation errors
* Fix NPE bug
* Test fixes and improvements
* chore: apply changes
Co-authored-by: yan-3005 <yan-3005@users.noreply.github.com>
* Schema Changes for include fields and check change description
* Update generated TypeScript types
* Fixed 4 valid code review issues: migration idempotency bug (preventing false failures on re-runs), empty pattern string vulnerability (preventing unintended filter bypasses),
removed unused dead code method, and corrected Javadoc inconsistency from {} to [] notation.
---------
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Gitar <noreply@gitar.ai>
Co-authored-by: yan-3005 <yan-3005@users.noreply.github.com>
Co-authored-by: Anujkumar Yadav <anujf0510@gmail.com>
|
||
|
Trang Nguyen [INT-DE]
|
de2e703fdd
|
Fixes #26225: Add index and FORCE INDEX for listLastTestCaseResultsForTestSuite (MySQL) (#26235)
* ISSUE-26225: add index idx_entity_timestamp_desc for data_quality_data_time_series * ISSUE-26225: add index idx_entity_timestamp_desc for data_quality_data_time_series * Update bootstrap/sql/migrations/native/1.12.2/mysql/schemaChanges.sql * ISSUE-26225: fix the suggestion --------- Co-authored-by: Teddy <teddy.crepineau@gmail.com> Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com> |
||
|
Pere Miquel Brull
|
f890e004ce
|
Move preview-to-enabled migrations from 1.11.13 to 1.13.0 (#26281)
The migrations renaming the 'preview' property to 'enabled' in apps were incorrectly placed under 1.11.13. Move them to 1.13.0 where they belong, since this change targets the next major release. Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> |
||
|
Teddy
|
40bf82f604
|
Minor move 20 migrations (#26236)
* FIX - Redshift converter (#26229)
(cherry picked from commit
|
||
|
Pere Miquel Brull
|
c7f911e43a
|
Rename app 'preview' property to 'enabled' (#26170)
* Rename app 'preview' property to 'enabled' with inverted semantics The 'preview' property was confusing: preview=false meant the app CAN be used. Replace with 'enabled' where enabled=true means usable, which is much more intuitive. Changes across the full stack: - JSON schemas: preview (default false) → enabled (default true) - Java backend: isPreview/raisePreviewMessage → isEnabled/raiseNotEnabledMessage - TypeScript types: preview → enabled - Frontend component: isPreviewApp → isAppDisabled (checks enabled===false) - SQL migrations for 1.11.12: rename + invert boolean in apps_marketplace and installed_apps tables (MySQL and PostgreSQL) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Update generated TypeScript types * format * improve deletion process for disabled apps * improve deletion process for disabled apps * improve deletion process for disabled apps * improve deletion process for disabled apps * format * fix tests * migration * migration --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> |
||
|
Sid
|
12d85f310f
|
fix glossary status frontend filtering logic to move to backend (#25428)
* fix glossary status * add glossaryTerm spec * fix: improve ListFilter implementation in list filtering logic Co-authored-by: siddhant1 <siddhant1@users.noreply.github.com> * reset main backend * reset backend * fix be * rever * spottless * Fix GlossrayTerm search api endpoint * status enum validation * fix spec * Replace quotes, validate enum * bind param queries * Move migrations to 1.12.0 * fix api docs * optimize performance of fallback , refactoring * fix ListFilter * GlossaryTermService.java cleanup * address gitar-bot feedback * add entityStatus param in list api * add entityStatus param in list api * Send entityStatus param with both search and list glossary term APIs - Pass entityStatus to searchGlossaryTermsPaginated and getFirstLevelGlossaryTermsPaginated when a specific status filter is active (not 'all') - Keep 'All' option in status dropdown with default selection of Approved, Draft, InReview - Show appropriate empty state message when status filter returns no results Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * update list API path (ListFilter.getEntityStatusCondition) to validate against the enum, in case if an invalid value like "Bogus" is passed * fix playwright * Fix rejected glossary term staying visible in listing Remove rejected terms from visible list when status filter excludes them, and fix reused waitForResponse promise in Playwright test. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * add initian load * Fix Expand All ignoring active status filter and add E2E tests Pass entityStatus parameter in fetchExpadedTree so Expand All respects the active status filter. Add E2E test suite to verify the behavior. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Rewrite Glossary Expand All E2E tests to follow Playwright handbook patterns Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Fix flaky GlossaryPagination test by scoping locators to glossary table Scoped unscoped `tbody .ant-table-row` locators to `glossary-terms-table` testid, and replaced unreliable row count assertion in empty state test with visibility checks on `no-data-placeholder`. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Siddhant <siddhant@MacBook-Pro.local> Co-authored-by: Gitar <noreply@gitar.ai> Co-authored-by: siddhant1 <siddhant1@users.noreply.github.com> Co-authored-by: Ram Narayan Balaji <ramnarayanb3005@gmail.com> Co-authored-by: Ram Narayan Balaji <81347100+yan-3005@users.noreply.github.com> Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com> Co-authored-by: sonika-shah <58761340+sonika-shah@users.noreply.github.com> Co-authored-by: Siddhant <siddhant@MacBook-Pro-3.local> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: Siddhant <siddhant@MacBook-Pro-4.local> |
||
|
Teddy
|
a456a194a9
|
ISSUE #3027 - Better Default (#26158)
* feat(metric default): move profiler and dq to obs folder * feat(metric default): validate metric registry and schema * feat(metric default): map metric type name to enum name * feat(metric default): updated default metrics in profiler * feat(metric default): migration to remove computeMetrics setting * feat(metric default): fix CI failures * feat(metric default): fix CI failures * fix ci failures * fix ci failures * fix typo in psql migration query * fix psql migration query * fix ci failure * fix: CI failures |
||
|
Mayur Singal
|
31e2e59a00
|
Fix #26178: Add support for IAM auth for redshift (#26179)
* Fix #26178: Add support for IAM auth for redshift * Missing files for the implementation * Update generated TypeScript types * adderess guitar comments * address comments * fix python tests * fix redshift playright * fix checkstyle --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> |
||
|
Himanshu Khairajani
|
cf0fa0a519
|
Openlineage: Added Kinesis Support #24752 (#26050)
* Openlineage Kinesis Support * Update generated TypeScript types * marking field as required * test-connection name improvement * pagination improvement * test-connection name improvement * Update generated TypeScript types * nested broker-config migration file * newline added to yaml * Migration to 1.11.2 * Migration to 1.11.12* * fix: add throttle mechanism to kinesis get_records loop Co-authored-by: Khairajani <Khairajani@users.noreply.github.com> * fix: prevent timeout reset on sequential shard polling Co-authored-by: Khairajani <Khairajani@users.noreply.github.com> * Kinesis test-case * Kinesis test-case * setting lineageInformation object model and not raw dict --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Gitar <noreply@gitar.ai> Co-authored-by: Khairajani <Khairajani@users.noreply.github.com> |
||
|
Sriharsha Chintalapani
|
7465810fdd
|
Audit Log performance improvements (#26023)
* Audit Log performance improvements * Audit Log performance improvements * Address comments * removed fixme from audit log tests --------- Co-authored-by: Rohit0301 <rj03012002@gmail.com> Co-authored-by: Rohit Jain <60229265+Rohit0301@users.noreply.github.com> |
||
|
Mohit Yadav
|
82b9d34806
|
Optimize indexing Processing to EsDoc (#26079)
* Optimize Reads with Keyset * Optimize Search Index Processing stage * Fix KeySet Cursor * revert keyset for time series * Fix Review Comments * Move to 1.12.2 * Fix Review Comment * Remove IF NOT EXISTS from mysql and update common mthod |
||
|
harshsoni2024
|
4d017d3f32
|
Fix-20713: Add support for metadata ingestion using local file in REST connector (#26036) | ||
|
Pere Miquel Brull
|
a1e3a49dae
|
MINOR - Allow app definition to pass the impersonation rules for bots (#25909)
* MINOR - Streamline bot impersonation from apps * MINOR - Streamline bot impersonation from apps * MINOR - Streamline bot impersonation from apps * MINOR - Streamline bot impersonation from apps * Update generated TypeScript types * policy flag * policy flag * policy flag * policy flag * fix feedback --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> |
||
|
Ram Narayan Balaji
|
91239164f5
|
delete workflow instance entries if status is null in migration (#25867) | ||
|
Ram Narayan Balaji
|
f418203338
|
Fix: Resolve v1.12.0 migration failure due to NULL workflow status (#25834)
* Fix: Resolve v1.12.0 migration failure due to NULL workflow status ## Root Cause Analysis - Migration failed when modifying entityLink column in workflow_instance_time_series - MySQL's ALTER TABLE MODIFY COLUMN re-validates ALL generated columns for ALL rows - Found 184+ workflow instances created between Dec 2024 - Jan 2025 with NULL status - These were created with pre-v1.7.0 code that didn't set status field in JSON - v1.7.0 added status column as GENERATED NOT NULL but old instances had NULL values - v1.12.0 migration triggered constraint validation, causing "Column 'status' cannot be null" ## Solution - Add UPDATE statements before ALTER TABLE in v1.12.0 migration - Set status='FINISHED' for workflows with endedAt (completed) - Set status='FAILED' for workflows without endedAt (incomplete) - Use two separate queries for better performance vs CASE statements - Handle both workflow_instance_time_series and workflow_instance_state_time_series * failed to FAILURE status |
||
|
Sriharsha Chintalapani
|
b244798f22
|
Add bulk apis for pipeline status (#25731)
* Add bulk apis for pipeline status * Update generated TypeScript types * Fix gitar comments * Update generated TypeScript types * Fix pycheck * Address comments * Fix databricks test * Move schema changes to 1.11.9 --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: harshsoni2024 <harshsoni2024@gmail.com> |
||
|
Sriharsha Chintalapani
|
6f577656c1
|
Fix integration tests (#25753)
* Fix - disk space in github workflows * Fix - disk space in github workflows * Fix - disk space in github workflows * Fix running tests with bulk apis * Fix running tests with bulk apis * Address comments; make awaitability for tests * Address comments |
||
|
sonika-shah
|
30a4d32720
|
Fix entity version history of dataProducts after removing inputPorts/ field (#25702) | ||
|
Aleksei Sviridkin
|
b2ac6f70d9
|
Fixes #24546: Add sobjectNames field for multi-object selection in Salesforce connector (#24547)
* feat(salesforce): add sobjectNames field for multi-object selection Add support for specifying multiple Salesforce objects to ingest instead of just one or all. The new `sobjectNames` array field allows users to select specific objects (e.g., Contact, Account, Lead) without having to ingest all objects and filter them. Priority order: 1. sobjectNames (array) - if specified, use only these 2. sobjectName (string) - if specified and sobjectNames empty 3. All objects from describe() - if neither specified tableFilterPattern applies in all cases as a final filter. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> * refactor: removed sobjectName field and added a migration for 1.11.8 to migrate sobjectName values to sobjectNames * fix: sobjectNames priority comment * refactor: sobjectNames changes in ts files * fix: yaml structure in test_salesforce * fix: test_salesforce.py - metadata as OpenMetadata object * fix: added new line in sql migrations * fix: sql migration serviceType --------- Signed-off-by: Aleksei Sviridkin <f@lex.la> Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: Keshav Mohta <keshavmohta09@gmail.com> Co-authored-by: Keshav Mohta <68001229+keshavmohta09@users.noreply.github.com> Co-authored-by: Mohit Yadav <105265192+mohityadav766@users.noreply.github.com> |
||
|
Ajith Prasad
|
f1fe02daff
|
Moved AI Application and LLM Model entities migrations to 1.12.0 (#25659) | ||
|
Himanshu Khairajani
|
e86a0201ab
|
Fix #25645: MySQL timestamp precision for tag_usage.appliedAt (#25643)
* Fix MySQL timestamp precision for tag_usage.appliedAt MySQL's TIMESTAMP type defaults to second precision, while PostgreSQL returns microsecond precision. This causes _normalize_datetime_strings in the Python ingestion client to produce spurious appliedAt diffs in JSON patches, which then fail with "Failed to convert JsonValue to target class" during deserialization in JsonUtils.applyPatch(). Upgrade appliedAt to TIMESTAMP(6) to match PostgreSQL behavior and eliminate the spurious patch diffs. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * Add 1.11.8 migration for MySQL appliedAt timestamp precision Backport the TIMESTAMP(6) fix to the 1.11.x release line so existing deployments on 1.11.x pick up the fix without requiring a 1.12.0 upgrade. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com> |
||
|
sonika-shah
|
cec1829645
|
Fix DataProduct inputPorts/outputPorts orphaned fields migration issue after migration from 1.10.x to 1.12.x (#25634)
* Fix DataProduct inputPorts/outputPorts orphaned fields migration issue after migration from 1.10.x to 1.12.x * escape ? as ?? for JDBI |
||
|
mohitdeuex
|
fcc0c1d944 | Drop constraint from postgres | ||
|
Mohit Yadav
|
21750aaa90
|
Feature/search indexing issues (#25594)
* Add design doc for search indexing stats redesign Covers: - Simplified 4-stage pipeline model (Reader, Process, Sink, Vector) - Per-entity index promotion instead of batch promotion - Alias management from indexMapping.json - Payload-aware vector bulk processor Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * Add Support for Per Entity Index Promotion * Add UI Bit * Add Lang * Add AppLog View Test coverage * Add Bathced Vector index querying * Add Improvements for Vector to be async and also stats to be better handled * Use Virtual Thread * Use Virtual Thread * Fix Tests * Make reading stats easier * Fixed Stats to be accurate * Fix Stats getting null * Fix partition worker stats * Fix Reader Stats - final * Update generated TypeScript types * Make updates in 1.12.0 * Revert "Use Virtual Thread" This reverts commit |
||
|
Mohit Yadav
|
0129f274ed
|
ReApply changes Fix Stats Issue and Add Tests (#25521)
* Fix Issue and Add Tests * Update generated TypeScript types * Fix CI jest failure --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> |
||
|
Teddy
|
9e77872972
|
ISSUE #25482 - rule library validator implementation (#25497)
* feat(rule library): expend safe token * feat(rule library): added validator class to testDefinition * chore * feat(rule library): implement validator logic * feat(rule library): fix runtime errors * feat(rule library): implement table level rule library * feat(rule library): implement integration test for rule library * feat(rule library): ran python linting * feat(rule library): fix wrong import * feat(rule library): added logic to catch template error * feat(rule library): fix test to handle new validator class behavior * feat(rule library): fix test to handle new validator class behavior |
||
|
Sriharsha Chintalapani
|
b09f4828c4
|
Learning Resources (#25005)
* Add Learning Resources with-in product * Translations * Add Learning Resources in-line with-in product * Add Learning Resources in-line with-in product * Potential fix for code scanning alert no. 1844: Incomplete URL substring sanitization Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> * Update generated TypeScript types * Update the design * Update the design * Add leanring resources * Update generated TypeScript types * Add learning resources * Update generated TypeScript types * Address comments * Address comments * fixed build issue * fix java checkstyle * fixed initital bugs * fixed less file name * resolve conflict * fixed failing unit test * Address update issues, add more playwright tests * Address update issues, add more playwright tests * fixed code quality and updated all the missed pages with leanrning icon * fixed invalid translation * Added icon for rules library * fixed unit tests * replaced string with constants * addressed comments * resolved backend merge conflict * removed plural label * fixed header actions position * fixed git-r comment * added fixme to a test * fixed label * fixed flaky test * Update generated TypeScript types * removed playwright config file * hide column view * playwright fixes --------- Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Dhruv Parmar <83108871+dhruvjsx@users.noreply.github.com> Co-authored-by: sonika-shah <58761340+sonika-shah@users.noreply.github.com> |
||
|
Eugenio
|
ce007263ef
|
Improve TagLabel with rich metadata (#25472)
* Ensure columns are retrieved in the right order This is because since introducing ordering for `getTableColumnsByFQN`, the patches created in `removeTagFromEntity` were open to pointing to different columns if the default order didn't match how they were persisted in db * Allow exception list to be updated on all feedback * Apply gitar comments * Add `metadata` to `tag_usage` table * Update JSON schema object to include `TagLabel.metadata` * Apply feedback to selected recognizer * Add backend integration tests * Update `ingestion` to return `TagLabel.metadata.recognizer` * Update generated TypeScript types * Update generated TypeScript types * Send recognizer result metadata in feedback approval task (#25485) * Send `TagLabelRecognizerMetadata` in `TaskDetails` This is so we can show an explanation behind the classification in the feedback approval card * Update typescript types * Run Spotless * Ensure `applyTagsBatchInternal` works equally for pg and mysql * Tag metadata fixes * Fix CI test --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Rohit Jain <60229265+Rohit0301@users.noreply.github.com> Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com> |
||
|
mohitdeuex
|
c006bdb2b0 |
Revert "Fix stats and Improve Search with Insights (#25495)"
This reverts commit
|
||
|
Mohit Yadav
|
19725a7130
|
Fix stats and Improve Search with Insights (#25495)
* Fix Stats * Add Warning logs and reindex failure analysis * Add Search Insights in Preferences * Add Label * Fix Full Error not available * Add check for reindex run |