* Fix: Pass secrets provider to the ingestion pipelines
For the noop providers where we were using the DB provider we introduced
a regression by the end of 2023 due to stop passing the provider
forward. This broke the `aws`, `aws-ssm`, and `azure-kv` providers.
This commit passes the provider forward so the ingestion pipeline knows
how to act instead of treating everything as the DB provider.
* Fix tests
* Fix k8s client by passing the needed env vars to ingestion
---------
Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
* use promote entity reindex in distributed
* Add Logs and finalize remaining entities
* fix nullpointer
* Fix Canonical Index Deletion
* Fix Test
* Optimize Reindexing
* Fix Default Recreate Handler
* Fix entity version history of dataProducts after removing inputPorts/ field (#25702)
* Fix: Use Downward API for pipelineRunId in standard K8s CronJobs (#25640)
When using the K8s pipeline client with useOMJobOperator=false, scheduled
pipelines (CronJobs) were failing because pipelineRunId was set to the
literal string "{{ omjob.uid }}" instead of a valid UUID.
This template syntax is designed for the OMJob operator to resolve at
runtime, but standard K8s CronJobs have no templating engine.
The fix uses the Kubernetes Downward API to inject the pod's own UID
(metadata.uid) as the pipelineRunId. Pod UIDs are valid UUIDv4 values
and unique per execution, making them suitable replacements.
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
* Rename deny list recognizer (#25698)
* Rename deny list recognizer
* Update generated TypeScript types
---------
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
* Fix/testsuite cross suite result consistency (#25587)
* refactor: replace testCaseResultSummary with lastResultTimestamp in ES index
* feat: add lean batch SQL queries for test suite result summaries
* fix: recompute related test suite results on pipeline completion
* test: update UI sort field and tests for lastResultTimestamp
* Revert MUITagSuggestion changes from PR #25588 (#25710)
* Initial plan
* Revert MUITagSuggestion changes from PR #25588
Co-authored-by: karanh37 <33024356+karanh37@users.noreply.github.com>
* Revert test file changes from PR #25588
Co-authored-by: karanh37 <33024356+karanh37@users.noreply.github.com>
* fix test
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: karanh37 <33024356+karanh37@users.noreply.github.com>
Co-authored-by: karanh37 <karanh37@gmail.com>
* Fix Elasticsearch/OpenSearch field explosion with custom properties (#25627)
* Flatten Custom Properties
* Fix UI Side
* Fix Reindex matching old index conflict with alias
* Hacky wacky
* Review comments
* Add tests for custom properties to check field don't exceed
* Change to nested
* Align tests
* Fix Column and DataModel and subassets customProperties
* Fix extension updates
* Fix Description Updates
* Temp fix from other PR
* Fix Test failures
* Fix Review comments
* Fix Tests for playwright
* Import export improvements (#25542)
* Batched Import
* Batched Import
* Batched Import
* Optimised Internal methods for adding relationships for import, fixed circular dependency, generated changeEvents (#25582)
* Fix tag clearing and circular dependency detection in batch CSV imports
- **Tag clearing fix**: Add deleteTagsByTarget before applying new tags in batch imports to match single entity import behavior, ensuring empty CSV fields properly clear existing
tags
- **Circular dependency detection fix**: Pre-track entities in dryRunCreatedEntities before parent resolution to enable proper circular reference validation during CSV team
imports
- Resolves test failures in TeamResourceIT.test_importCsv_circularDependency_trueRun and tag-related import issues
- Maintains batch import performance while restoring pre-batch-import validation contracts
* improve storeRelationshipsInternal internal methods - make them truly batched operations
* - Add storeEntities override to all repositories (57 repos)
- Add batch lock check to HierarchicalLockManager
- Add batch cache write to EntityRepository
- Fix createManyEntitiesForImport with batched operations
- Fix updateManyEntitiesForImport with batched operations
- Add change event creation in flushPendingEntityOperations
---------
Co-authored-by: sonika-shah <58761340+sonika-shah@users.noreply.github.com>
* have default implementation of storeEntities
* add batch relationship clearing while updating during import
* Fix column import failing for tables in same batch
When importing CSV with batching, column rows failed to find their
parent table because the table was queued but not yet persisted to DB.
Added pendingEntityFQNs set to track entities in current batch. When
processing columns, check if table is in this set - if so, add columns
directly to the queued table object instead of creating a patch context.
This ensures columns are persisted with the table in a single operation.
* fix UserResourceTest, TestCaseResourceIT
* Fix Circular Dependency Validation for GlossaryTerms
* Batch Insert ChangeEvents and Process Async
* fix: defer CSV result reporting until batch operations complete to ensure accuracy
* fix: ensure custom properties persist during batch CSV imports
* Fix rows processed count
* Fix: Moving of glossaryTerms to correctly detect update operation during import
* Increment version history
* fix TeamResourceTest
* Fix: Mutual Exclusivity Tag Validation during dry run, Deferring of failures
* Fix: Table Constraints Preservation on table import
* revert validateCsvString logic, needed for csv validation
* revert test changes
* Use dependency resolution to flush pending operations to find the parent
* fix search groping based on entityType in updateEntitiesBulk with tests
* Fetch original from db as cache takes some time to reload
* Test Cases
* Remove logs, fix TableResourceIT, Fix ChangeEvent Race Condition
---------
Co-authored-by: Ram Narayan Balaji <81347100+yan-3005@users.noreply.github.com>
Co-authored-by: sonika-shah <58761340+sonika-shah@users.noreply.github.com>
Co-authored-by: Ram Narayan Balaji <ramnarayanb3005@gmail.com>
Co-authored-by: Karan Hotchandani <33024356+karanh37@users.noreply.github.com>
Co-authored-by: karanh37 <karanh37@gmail.com>
* fix: databricks view definition (#25700)
* Column Edit - Address feedback (#25669)
* Address Test feedback for columns
* Add Translations
* Fix count issue
* nit
* nit
* fix pagination control
* fix e2e test
---------
Co-authored-by: anuj-kumary <anujf0510@gmail.com>
* chore(ui): fix pagination tests (#25715)
* update table selectors
* modify selector
* feat(ui): Learning Resources page improvements and form updates (#25681)
* feat(ui): Learning Resources page improvements and form updates
* Improved footer component
* fix color code combination
* fix view more issue in card view
* Addressed card related feedback
* fix test
* fix playwright test
* nit
* nit
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix pagination space between layout (#25728)
* Fix#25615 - Use system tables instead of API for lineage in UC (#25697)
* WIP - chore(ci): update playwright workflow to run ingestion with dataAsset… (#24753)
* chore(ci): update playwright workflow to run ingestion with dataAssetRules
* limit setup to required steps
* rearrnage spec files
* update condition
* update workflow
* update runners
* update console
* update config
* update arrangements
* fix typo
* rearrange specs
* remove unwanted import
* fix tests
* fix tests issues
* fix comment
* fix failing tests
* fix advance search
* fix test arrangements
* fix failing tests
* feat(ui): Remove resources count badge and add icon in left panel (#25735)
* Remove resources count badge and add icon in left panel
* nit
* Remove important from CSS
* Flaky(UI): Glossary import export (#25693)
* fixed glossary import export flakyness
* removed timeout
* Flaky(UI) : Bulk import (#25653)
* fixed flaky bulk import
* fixed flaky bulk import
* formatted code
* removed unncessary loader await
* removed timeout
* Flaky(UI): Customize Detail Page spec (#25650)
* fixed customize detail page
* fixed button not clicking issue
* removed unncessary timeout
* added test id
* MINOR: use stats tables for MySQL and PSQL profiler (#25724)
* feat(system): use stats tables for mysl and psql profiler
* fix: skip tests if fail
* fix(trino): shadowing of http_scheme argument (#25726)
* fix(trino): shadowing of http_scheme argument
* fix: import chain
* fix: remove mock trino library
* [Search] Upgrade Clients (#25719)
* Upgrade Clients
* Update clients in docker files
* Fix Tests
* Fix integration test
* Fix Review Comments
* Fix More review comments :-
1. ElasticSearchClient.java - Added keep-alive timeout configuration
2. OpenSearchClient.java - Added keep-alive timeout configuration
3. OpenMetadataOperations.java - Added logging for caught exception
4. SigV4Hc5RequestSigningInterceptor.java - Now throws exception instead of silently returning
* Fix More review comments :-
1. ElasticSearchClient.java - Added keep-alive timeout configuration
2. OpenSearchClient.java - Added keep-alive timeout configuration
3. OpenMetadataOperations.java - Added logging for caught exception
4. SigV4Hc5RequestSigningInterceptor.java - Now throws exception instead of silently returning
Co-authored-by: mohityadav766 <mohityadav766@users.noreply.github.com>
* upgrade to 9.3.0 vs 3.4.0 server since earlier had bug
* fix version in pom
* Fix Review Comments
* FIX IAM OpenSearch FIx
---------
Co-authored-by: Gitar <noreply@gitar.ai>
Co-authored-by: mohityadav766 <mohityadav766@users.noreply.github.com>
* Fix Failing Test (#25745)
* Optimize Reindexing
* Fix Issue in Partition Worker
* Fix Entity Stats writing too much to db
---------
Co-authored-by: sonika-shah <58761340+sonika-shah@users.noreply.github.com>
Co-authored-by: IceS2 <pablo.takara@getcollate.io>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Co-authored-by: Eugenio <eugenio.donaque@getcollate.io>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Adrià Manero <adria.estivill@getcollate.io>
Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com>
Co-authored-by: karanh37 <33024356+karanh37@users.noreply.github.com>
Co-authored-by: karanh37 <karanh37@gmail.com>
Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
Co-authored-by: Ram Narayan Balaji <81347100+yan-3005@users.noreply.github.com>
Co-authored-by: Ram Narayan Balaji <ramnarayanb3005@gmail.com>
Co-authored-by: Keshav Mohta <68001229+keshavmohta09@users.noreply.github.com>
Co-authored-by: anuj-kumary <anujf0510@gmail.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>
Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
Co-authored-by: Dhruv Parmar <83108871+dhruvjsx@users.noreply.github.com>
Co-authored-by: Teddy <teddy.crepineau@gmail.com>
Co-authored-by: Gitar <noreply@gitar.ai>
Co-authored-by: mohityadav766 <mohityadav766@users.noreply.github.com>
* Update workflow to avoid extra DB hits
* Updates on `openmetadata-integration-tests
Recover lost profiles configuration
Add cleanup steps and opensearch configuration needed in test suite bootstrap
Make tag recognizer tests retryable
* Use updated images
* Initial plan
* Add UPDATED_BY field to common entity fields for all entities
- Move UPDATED_BY from glossary-specific fields to common fields
- Update eventBasedEntityTrigger maxItems from 1 to 2 to allow both relatedEntity and updatedBy
- This enables UPDATED_BY field in workflow check conditions for all entities including test suites
Co-authored-by: karanh37 <33024356+karanh37@users.noreply.github.com>
* Revert eventBasedEntityTrigger.json maxItems change back to 1
Co-authored-by: karanh37 <33024356+karanh37@users.noreply.github.com>
* glossary term
* rt
* multi
* fix
* fix
* fixer
* fix
* fix
* fixer
* fix
* entity status
* fix realted terms
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: karanh37 <33024356+karanh37@users.noreply.github.com>
Co-authored-by: Sid <30566406+siddhant1@users.noreply.github.com>
Co-authored-by: Siddhant <siddhant@MacBook-Pro.local>
* Fix - disk space in github workflows
* Fix - disk space in github workflows
* Fix - disk space in github workflows
* Fix running tests with bulk apis
* Fix running tests with bulk apis
* Address comments; make awaitability for tests
* Address comments
* Fix the table nested columns not immediately updating after changes in the UI
* Move the nested column updates spec to flow folder
* Fix the nested columns not updating for dataModels and SearchIndexes and added test coverage for all entities
* Fix unit test
* Fix the failing tests for the display name edit
* fix: PATCH not saving column tags for File entities
* Fix the failing tests
* Address comments
---------
Co-authored-by: Adrià Manero <adria.estivill@getcollate.io>
Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
* Add 20% threashold on bulk api connections and semaphores to control it
* Address comments
* Add bulk apis to use bulkWrite/bulkUpdate methods to avoid using too many db connections
* Add batch updates and remove semaphores
* Fix test failures; address comments
* Fix test failures
* Fix test failures
* Fix test failures
* Add comment section for bulk API support in DatabaseSchemaResourceIT
* Add CsvImportResult import to multiple test classes
---------
Co-authored-by: Ayush Shah <ayush@getcollate.io>
* Batched Import
* Batched Import
* Batched Import
* Optimised Internal methods for adding relationships for import, fixed circular dependency, generated changeEvents (#25582)
* Fix tag clearing and circular dependency detection in batch CSV imports
- **Tag clearing fix**: Add deleteTagsByTarget before applying new tags in batch imports to match single entity import behavior, ensuring empty CSV fields properly clear existing
tags
- **Circular dependency detection fix**: Pre-track entities in dryRunCreatedEntities before parent resolution to enable proper circular reference validation during CSV team
imports
- Resolves test failures in TeamResourceIT.test_importCsv_circularDependency_trueRun and tag-related import issues
- Maintains batch import performance while restoring pre-batch-import validation contracts
* improve storeRelationshipsInternal internal methods - make them truly batched operations
* - Add storeEntities override to all repositories (57 repos)
- Add batch lock check to HierarchicalLockManager
- Add batch cache write to EntityRepository
- Fix createManyEntitiesForImport with batched operations
- Fix updateManyEntitiesForImport with batched operations
- Add change event creation in flushPendingEntityOperations
---------
Co-authored-by: sonika-shah <58761340+sonika-shah@users.noreply.github.com>
* have default implementation of storeEntities
* add batch relationship clearing while updating during import
* Fix column import failing for tables in same batch
When importing CSV with batching, column rows failed to find their
parent table because the table was queued but not yet persisted to DB.
Added pendingEntityFQNs set to track entities in current batch. When
processing columns, check if table is in this set - if so, add columns
directly to the queued table object instead of creating a patch context.
This ensures columns are persisted with the table in a single operation.
* fix UserResourceTest, TestCaseResourceIT
* Fix Circular Dependency Validation for GlossaryTerms
* Batch Insert ChangeEvents and Process Async
* fix: defer CSV result reporting until batch operations complete to ensure accuracy
* fix: ensure custom properties persist during batch CSV imports
* Fix rows processed count
* Fix: Moving of glossaryTerms to correctly detect update operation during import
* Increment version history
* fix TeamResourceTest
* Fix: Mutual Exclusivity Tag Validation during dry run, Deferring of failures
* Fix: Table Constraints Preservation on table import
* revert validateCsvString logic, needed for csv validation
* revert test changes
* Use dependency resolution to flush pending operations to find the parent
* fix search groping based on entityType in updateEntitiesBulk with tests
* Fetch original from db as cache takes some time to reload
* Test Cases
* Remove logs, fix TableResourceIT, Fix ChangeEvent Race Condition
---------
Co-authored-by: Ram Narayan Balaji <81347100+yan-3005@users.noreply.github.com>
Co-authored-by: sonika-shah <58761340+sonika-shah@users.noreply.github.com>
Co-authored-by: Ram Narayan Balaji <ramnarayanb3005@gmail.com>
Co-authored-by: Karan Hotchandani <33024356+karanh37@users.noreply.github.com>
Co-authored-by: karanh37 <karanh37@gmail.com>
* refactor: replace testCaseResultSummary with lastResultTimestamp in ES index
* feat: add lean batch SQL queries for test suite result summaries
* fix: recompute related test suite results on pipeline completion
* test: update UI sort field and tests for lastResultTimestamp
When using the K8s pipeline client with useOMJobOperator=false, scheduled
pipelines (CronJobs) were failing because pipelineRunId was set to the
literal string "{{ omjob.uid }}" instead of a valid UUID.
This template syntax is designed for the OMJob operator to resolve at
runtime, but standard K8s CronJobs have no templating engine.
The fix uses the Kubernetes Downward API to inject the pod's own UID
(metadata.uid) as the pipelineRunId. Pod UIDs are valid UUIDv4 values
and unique per execution, making them suitable replacements.
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
