* fix(lineage): prevent pipeline annotation inheritance in service/domain/dataProduct lineage and add pipeline service edges
Bug #1: Service nodes (e.g., DatabaseService, MessagingService) were incorrectly appearing in
entity-level lineage views. Root cause: getOrCreateLineageDetails() in addServiceLineage(),
addDomainLineage(), and addDataProductsLineage() was copying the pipeline annotation from
entity-level LineageDetails to service/domain/dataProduct-level LineageDetails. This caused
service entities to have upstreamLineage.pipeline.fqnHash set in their Elasticsearch documents,
making them match the PIPELINE_AS_EDGE_KEY query during BFS traversal and incorrectly appear
alongside actual data assets. Fix: add .withPipeline(null) on each service/domain/dataProduct
LineageDetails object to strip the pipeline annotation before persisting.
Bug #2: "By Service" view was empty when viewing lineage for pipeline entities that were stored
as edge annotators (Case B: table → topic with pipeline=flink_pipeline in LineageDetails) rather
than as actual nodes (Case A). Root cause: addServiceLineage() only created database_service →
kafka_service edges but no edges involving flink_pipeline_service. Fix: add addPipelineServiceEdges()
called from addServiceLineage() that creates fromService → pipelineService and pipelineService →
toService edges when a pipeline annotation exists in the entity-level lineage details.
Also add unit tests covering both fixes to prevent regression.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(lineage): add migration to remove pipeline annotation from service/domain/dataProduct lineage edges
The previous fix (e6df7a6c62) prevented new lineage from inheriting pipeline annotations on
service/domain/dataProduct-level edges. However, existing data in the entity_relationship table
already has pipeline set on those edges from before the fix, and Elasticsearch reindex reads from
the DB — so reindex alone does not fix stale data.
This migration removes the pipeline field from all service-to-service, domain-to-domain, and
dataProduct-to-dataProduct lineage edges (relation=13/UPSTREAM) in entity_relationship.
After upgrading and running this migration, operators should trigger an Elasticsearch/OpenSearch
reindex so that the corrected DB records are reflected in the search index, which is what the
lineage graph BFS traversal reads from.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(lineage): move pipeline annotation migration from 1.12.0 to 1.13.0
Moves the data migration that removes the pipeline field from
service/domain/dataProduct lineage edges in entity_relationship to the
1.13.0 migration scripts, which is the correct target version.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(lineage): move pipeline annotation migration from 1.13.0 to new 1.12.6
Creates a new 1.12.6 migration with the data fix that removes the pipeline
field from service/domain/dataProduct lineage edges in entity_relationship,
and removes it from 1.13.0 where it was previously placed.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(lineage): add v1126 Java migration to create pipeline service edges for existing data
For installations upgrading to 1.12.6 with existing lineage data, service edges
fromService→pipelineService and pipelineService→toService were never created
(only added by the code fix for new lineage going forward). This migration
reads service-level lineage edges that have a pipeline annotation, resolves
the pipeline entity's service, and inserts the two missing service edges into
entity_relationship (DB only). After the SQL migration strips pipeline from
service edges and a reindex runs, the "By Service" lineage view for pipeline
services correctly shows their upstream/downstream service connections.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(lineage): fix v1126 migration to read entity-level edges for pipeline service creation
The original migration read service-level edges (databaseService→messagingService)
looking for pipeline annotations, but those had already been cleaned by the SQL
migration before the Java migration could run in subsequent server restarts.
Fix: read data-asset-level edges (table→topic etc.) which retain their pipeline
annotation permanently. For each such edge, resolve fromEntity.service,
toEntity.service, and pipeline.service, then create the two missing
pipelineService edges in entity_relationship.
Verified: after running the migration manually via direct SQL + OpenSearch update,
the By Service view for lineage_test_flink_svc correctly shows 3 nodes with
upstream (db_svc→flink_svc) and downstream (flink_svc→kafka_svc) edges.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(lineage): clean up pipeline service edges when entity lineage is deleted
When entity-level lineage (table→topic) is deleted, cleanUpExtendedLineage
only cleaned up fromService→toService (db_svc→kafka_svc) but left the new
pipeline service edges (db_svc→flink_svc, flink_svc→kafka_svc) as orphans
in both entity_relationship and OpenSearch.
Fix:
- Pass lineageDetails (which contains the pipeline reference) into
cleanUpExtendedLineage from both deleteLineage and deleteLineageByFQN
- Add cleanUpPipelineServiceEdges that mirrors addPipelineServiceEdges:
uses getPipelineService(lineageDetails) to resolve the pipelineService,
then calls processExtendedLineageCleanup for fromService→pipelineService
and pipelineService→toService edges (decrement assetEdges or delete+remove
from search if count reaches zero)
- Also fix deleteLineageByFQN which was missing cleanUpExtendedLineage call
entirely (pre-existing gap for service edge cleanup)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* test(lineage): add unit tests for pipeline annotation stripping and pipeline service edge creation
- Add 4 new unit tests to LineageRepositoryTest covering:
- Bug #1 (2 tests): service-level edges do not inherit pipeline annotation
from entity lineage, both for new and existing edges
- Bug #2 (2 tests): addPipelineServiceEdges creates fromService→pipelineService
and pipelineService→toService edges when pipeline annotator is present,
and skips them when no pipeline is set
- Fix MySQL migration: add metadataService to entity type list (was in Java
migration's SERVICE_ENTITY_TYPES but missing from SQL) and replace
JSON_EXTRACT IS NOT NULL with JSON_CONTAINS_PATH to correctly handle both
present and explicit-null pipeline fields
- Fix PostgreSQL migration: add metadataService to entity type list
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* test(lineage): add integration tests for pipeline-as-annotator lineage scenario
Tests Bug #1 (service nodes absent from entity-level lineage) and Bug #2
(pipeline service connected in service-level lineage) using a table → topic
edge annotated with a pipeline entity reference.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* test(e2e): add Playwright tests for pipeline-as-annotator lineage scenario
Tests Bug #1 (service nodes absent from entity-level lineage) and Bug #2
(pipeline service appears in service-level lineage) using API interception
and direct request assertions via page.request.get().
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* style: apply spotless formatting to LineageRepositoryTest
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* style: apply prettier formatting to LineagePipelineAnnotator spec
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(lineage): guard against null LineageDetails in getPipelineService
When the json column in entity_relationship is NULL, JsonUtils.readValue
returns null. getPipelineService now short-circuits on a null argument
instead of throwing NullPointerException via entityLineageDetails.getPipeline().
Fixes NPE in deleteLineageByFQN and deleteLineage cleanup paths.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(e2e): use authenticated apiContext for service lineage assertions
page.request.get() sends browser cookies but OpenMetadata authenticates
via JWT in localStorage, so those calls were unauthenticated (non-2xx).
Replace with getToken + getAuthContext pattern used elsewhere.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(migration): add driveService to 1.12.6 pipeline annotation cleanup
Directory, File, Spreadsheet, and Worksheet entities map to driveService,
so service-level lineage edges between driveService instances could also
have incorrectly inherited the pipeline annotation. Include driveService
in the 1.12.6 cleanup migration for both MySQL and PostgreSQL.
Also drops the stray trailing-newline changes from the 1.12.0 migration
files — those edits were unnecessary.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* new line remove
* fix(migration): add DRIVE_SERVICE to v1126 SERVICE_ENTITY_TYPES set
driveService-to-driveService edges must be skipped during the pipeline
service edge migration scan, same as all other service-level edges.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(migration): resolve merge conflict in v1126 MigrationUtil
The rebase left MigrationUtil with duplicate imports and a missing closing
brace on insertEdgeIfMissing. Merged both method sets cleanly and ran
spotless.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
* MINOR - Streamline bot impersonation from apps
* MINOR - Streamline bot impersonation from apps
* MINOR - Streamline bot impersonation from apps
* MINOR - Streamline bot impersonation from apps
* Update generated TypeScript types
* policy flag
* policy flag
* policy flag
* policy flag
* fix feedback
---------
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
* Fix: Resolve v1.12.0 migration failure due to NULL workflow status
## Root Cause Analysis
- Migration failed when modifying entityLink column in workflow_instance_time_series
- MySQL's ALTER TABLE MODIFY COLUMN re-validates ALL generated columns for ALL rows
- Found 184+ workflow instances created between Dec 2024 - Jan 2025 with NULL status
- These were created with pre-v1.7.0 code that didn't set status field in JSON
- v1.7.0 added status column as GENERATED NOT NULL but old instances had NULL values
- v1.12.0 migration triggered constraint validation, causing "Column 'status' cannot be null"
## Solution
- Add UPDATE statements before ALTER TABLE in v1.12.0 migration
- Set status='FINISHED' for workflows with endedAt (completed)
- Set status='FAILED' for workflows without endedAt (incomplete)
- Use two separate queries for better performance vs CASE statements
- Handle both workflow_instance_time_series and workflow_instance_state_time_series
* failed to FAILURE status
* Fix - disk space in github workflows
* Fix - disk space in github workflows
* Fix - disk space in github workflows
* Fix running tests with bulk apis
* Fix running tests with bulk apis
* Address comments; make awaitability for tests
* Address comments
* Add design doc for search indexing stats redesign
Covers:
- Simplified 4-stage pipeline model (Reader, Process, Sink, Vector)
- Per-entity index promotion instead of batch promotion
- Alias management from indexMapping.json
- Payload-aware vector bulk processor
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Add Support for Per Entity Index Promotion
* Add UI Bit
* Add Lang
* Add AppLog View Test coverage
* Add Bathced Vector index querying
* Add Improvements for Vector to be async and also stats to be better handled
* Use Virtual Thread
* Use Virtual Thread
* Fix Tests
* Make reading stats easier
* Fixed Stats to be accurate
* Fix Stats getting null
* Fix partition worker stats
* Fix Reader Stats - final
* Update generated TypeScript types
* Make updates in 1.12.0
* Revert "Use Virtual Thread"
This reverts commit 4eb23374d1.
* Revert "Use Virtual Thread"
This reverts commit efe8d03b5d.
* Reapply "Use Virtual Thread"
This reverts commit d59cde18b2.
* Reapply "Use Virtual Thread"
This reverts commit 769e5710c3.
* Fix Final Update on stat
* - Add atomic alias swap
- remove unnecessary migration
* Fix Sonar test jest
* Fix Final Update on stat
---------
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
* Ensure columns are retrieved in the right order
This is because since introducing ordering for `getTableColumnsByFQN`, the patches created in `removeTagFromEntity` were open to pointing to different columns if the default order didn't match how they were persisted in db
* Allow exception list to be updated on all feedback
* Apply gitar comments
* Add `metadata` to `tag_usage` table
* Update JSON schema object to include `TagLabel.metadata`
* Apply feedback to selected recognizer
* Add backend integration tests
* Update `ingestion` to return `TagLabel.metadata.recognizer`
* Update generated TypeScript types
* Update generated TypeScript types
* Send recognizer result metadata in feedback approval task (#25485)
* Send `TagLabelRecognizerMetadata` in `TaskDetails`
This is so we can show an explanation behind the classification in the feedback approval card
* Update typescript types
* Run Spotless
* Ensure `applyTagsBatchInternal` works equally for pg and mysql
* Tag metadata fixes
* Fix CI test
---------
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Rohit Jain <60229265+Rohit0301@users.noreply.github.com>
Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
* Fix Stats
* Add Warning logs and reindex failure analysis
* Add Search Insights in Preferences
* Add Label
* Fix Full Error not available
* Add check for reindex run
* FIX#24374 - Data Contract at Data Product level
* Update generated TypeScript types
* FIX#24374 - Data Contract at Data Product level
* fix DP page
* fix: preserve termsOfUse object format in filtered contract
The termsOfUse field was being converted to a string during filtering,
but the form components expect it to be an object with {content: string}.
This was causing test failures where form elements were not visible.
- Keep termsOfUse as object format when not inherited
- Convert old string format to new object format for consistency
- Fixes 21 test failures in DataContracts.spec.ts and DataContractInheritance.spec.ts
* fix: address code review findings - state sync and immutability
Frontend changes:
- Add useEffect to sync formValues with filteredContract changes
- Ensures edit form updates when contract prop changes
Backend changes:
- Create deep copy at start of mergeContracts() to avoid mutating input
- Prevents side effects if contract object is reused elsewhere
Co-authored-by: pmbrull <pmbrull@users.noreply.github.com>
* Addressing feedback
Co-authored-by: pmbrull <pmbrull@users.noreply.github.com>
* fix tests
* fix inherited contract delete and status
* fix inherited contract delete and status
* fix inherited contract execution in app
* fix test
* fix: resolve playwright postgresql ci test failure
Co-authored-by: pmbrull <pmbrull@users.noreply.github.com>
* ci: fix yaml validation and checkstyle failures
Co-authored-by: pmbrull <pmbrull@users.noreply.github.com>
* fix: correct JSON/YAML validation errors
Co-authored-by: pmbrull <pmbrull@users.noreply.github.com>
* fix: resolve maven-collate and ui-coverage test failures
Co-authored-by: pmbrull <pmbrull@users.noreply.github.com>
* gitar feedback
* fix ci
* fix ci
* fix ci
* fix ci
* include .claude
* validate
* fix playwright
* playwright
* fix playwright
---------
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Gitar <gitar@collate.io>
Co-authored-by: Gitar <noreply@gitar.ai>
Co-authored-by: pmbrull <pmbrull@users.noreply.github.com>
Co-authored-by: Karan Hotchandani <33024356+karanh37@users.noreply.github.com>
Co-authored-by: karanh37 <karanh37@gmail.com>
* feat: added repository logic to list all versions (including latest) for a specific entity type
* feat: added list all versions for all the entity resources
* feat: moved endpoint to EntityResource
* feat: renamed endpoint to /history and methods to EntityHistory
* feat: ran java linting
* feat: remove v1 implementation left over code
* feat: fix failing tests
* feat: ran klinting
* feat: fix psql query
* feat: address PR comments
* feat: ran klinting
* feat: increase cache duration
* feat: address query edge cases
* fix(dq): psql migration for row insert test parameters
* fix(dq): use name and add trailing new line
* Fix description formatting in postDataMigrationSQLScript.sql
* Apply suggestion from @Copilot
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Add DQ Rules Library
* Add DQ Rules Library - Add Tests and enable testDefinitions through APIs to list
* Add DQ Rules Library - Add Tests and enable testDefinitions through APIs to list
* Add DQ Rules Library - Add Tests and enable testDefinitions through APIs to list
* Update generated TypeScript types
* Add DQ Rules Library - Add Tests and enable testDefinitions through APIs to list
* Add DQ Rules Library - Add Tests and enable testDefinitions through APIs to list
* Update generated TypeScript types
* Refactor tests to use toStrictEqual for string comparisons and improve consistency
- Updated various test files to replace `toBe` with `toStrictEqual` for string assertions in ImportStatus, SummaryCard, TabsLabel, and others.
- Enhanced regex tests to ensure accurate validation of entity names and tags.
- Added new translations for test platform warnings in en-us.json.
- Improved utility tests for alerts, authentication, CSV handling, and task messages to use `toEqual` for better clarity.
* Refactor TestDefinitionForm and TestDefinitionList components to use updated API methods and improve SQL expression handling
* Enhance TestDefinitionList component with permission checks for edit and delete actions, and update tests to reflect changes in permission handling
* Remove debug log from handleSubmit in TestDefinitionForm component
* Add permission loading state and enhance permission handling in TestDefinitionList component
* Update generated TypeScript types
* Update generated TypeScript types
* Update generated TypeScript types
* fix build failure
* Revert "Update generated TypeScript types"
This reverts commit 67b062216f.
* Enhance TestDefinitionForm and TestDefinitionList components with improved UI and pagination handling
* fix: update RulesLibrary tests and enhance TestDefinitionForm styling
* fix: Enhance TestDefinitionForm with error handling and improved UX
* fix: Update test definition handling and improve rendering in TestDefinitionList
* fix: Refactor TestDefinitionPermissions tests for improved permission checks and API context handling
* fix: Update system test definition retrieval to use findLast for improved accuracy
* feat: Add end-to-end tests for Rules Library and Test Definition Permissions
* fix: Update edit button visibility check to use beDisabled for better clarity
* fix: Refactor response handling in TestDefinitionPermissions tests for improved reliability
* move migrations execution order
* fix: remove existing columns
* style: remove migration extra line break
* chore: fix migration
---------
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Shailesh Parmar <shailesh.parmar.webdev@gmail.com>
Co-authored-by: TeddyCr <teddy.crepineau@gmail.com>
