* Add Continuous Indexing
* Add continuous Search indexing
* Update to 1.12.3
* Make search index retry queue reliable with stale recovery, health checks, and silent failure coverage
- Add entityType, retryCount, claimedAt columns to search_index_retry_queue table
- Implement stale IN_PROGRESS recovery (10min threshold, 60s sweep interval)
- Replace static isClientAvailable flag with cached ping health check (5s TTL)
- Narrow catch blocks in resolveById/resolveByFqn to EntityNotFoundException
- Use entityType hint for O(1) entity resolution instead of scanning all types
- Switch from status-string-based retry to retryCount-based (< 3 retries → PENDING, ≥ 3 → FAILED)
- Batch cascade reindex at 200 entities instead of accumulating up to 5000
- Add retry queue enqueue in catch blocks of createTimeSeriesEntity, updateTimeSeriesEntity,
deleteTimeSeriesEntityById, bulkIndexPipelineExecutions, reindexAcrossIndices, and
TestSuiteRepository.postCreate
- Re-throw exceptions from indexTableColumns/deleteTableColumns to parent catch blocks
- Add Micrometer counters for enqueued, processed (success/failure), and stale recovered
* Add missing lineage call site and Add test
* Review comments
* Add resilience to search index retry worker: client availability checks, backoff, and error classification
- Add exponential backoff when search client is unreachable so the
worker does not burn retries during cluster outages (5s → 10s → … → 60s cap)
- Classify errors using HTTP status codes from ES/OS exceptions:
4xx (except 429) are non-retryable and skip straight to FAILED;
429, 5xx, and IOException are retryable
- Preserve first bulk failure detail in RuntimeException so error
classification works for the bulk indexing path
- Reorganize SearchIndexRetryWorker into clearly separated sections
(lifecycle, main loop, record processing, entity resolution,
reindexing, resilience, suspension, utilities)
- Add isRetryableStatusCode utility to SearchIndexRetryQueue
- Add integration tests: status code classification, retry exhaustion
to FAILED, recovery from PENDING_RETRY_1, error detail preservation
* Address review comments
* Revert fqn size
* Spotless
* Address volatile review comments
* Fix Failing Test
* update review comments
---------
Co-authored-by: mohitdeuex <mohit.y@deuexsolutions.com>
Co-authored-by: Mohit Yadav <105265192+mohityadav766@users.noreply.github.com>
* Glossary Term Relations
* Add GlossaryTerm Relations
* Add GlossaryTerm Relations, Add custom relations, onotolgoy explorer
* Add Translations
* Update generated TypeScript types
* Address comments
* Address comments
* Address comments
* Update generated TypeScript types
* Update yarn.lock after merging cytoscape dependencies from glossary_relations
* fix zoom in and out functionality and added missing translate keys
* fix test
* Remove unwanted changes
* nit
* nit
* nit
* Remove conflict test
* nit
* fix test
* Add test for ontology explorer
* New yarn lock and 2.0.0 schema changes missed during merge conflicts
* Revamped glossary term relation settings
* Refactor code
* Addressed comments
* nit
* Update generated TypeScript types
* Java Checkstyle and Yarn lock
* Update generated TypeScript types
* fix unit test
* Remove 2.0.0 migration folders placed at wrong loc
* Merge main
* fix navigation to relation graph in glossary
* fix ontology explorer spec
* Added filter support in the data mode
* Fix glossary term relation CI failures
### Canonical Relation Storage (GlossaryTermRepository)
* Introduced `computeCanonicalRelationType()` to normalize relation direction
using UUID ordering (lower UUID is always treated as "from")
* Prevents duplicate and inconsistent relation rows when created from either side
* Updated `setTermRelations()` and `addRelation()` to store canonical relation types
* Fixed `setFields()` read logic:
* Invert relation type for `fromRecords` (entity is the TO side)
* Keep `toRecords` unchanged
* Updated `deleteBidirectionalRelatedTo()` to match canonical storage format
* Added `RequestEntityCache.invalidate()` after relation mutations to ensure consistency
### Lazy RDF Resource Initialization
* Added `RdfRepository.getInstanceOrNull()` for null-safe access without throwing
* Refactored `RdfResource` constructor to avoid eager `RdfRepository.getInstance()` call
* Enabled resource registration even when Fuseki is not initialized
* Introduced lazy getters:
* `getRdfRepository()`
* `getSemanticSearchEngine()`
* Updated all endpoints to guard with null checks before `isEnabled()`
* Return `503 Service Unavailable` when RDF is not ready
### Graceful Test Degradation (Fuseki-dependent tests)
* Added `TestSuiteBootstrap.isFusekiEnabled()` to detect Fuseki availability
* `GlossaryOntologyExportIT`:
* Falls back to Testcontainers-based local Fuseki when bootstrap Fuseki is unavailable
* `GlossaryTermRelationIT`:
* Skipped via `assumeTrue` when Fuseki is unavailable
* `MetricResourceIT`:
* Skips RDF-specific tests when Fuseki is unavailable
* fix package conflicts
* nit
* Fix merge conflicts, Python test, RDF reliability, and VectorDocBuilder tests
- Fix Python test_patch_glossary_term_related_terms to use TermRelation
instead of EntityReferenceList (schema changed relatedTerms type)
- Rewrite VectorDocBuilder tests for current buildEmbeddingFields API
- Improve JenaFusekiStorage retry logic to retry on all HTTP errors
- Increase Fuseki tmpfs size to prevent disk space exhaustion in tests
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix pycheck
* Address all 8 PR review findings
1. Add authorization check on getTermRelationGraph endpoint
2. Add null guard on getBaseUri() to prevent NPE
3. Add React key prop on RelatedTermTagButton in map renders
4. Mark RdfResource lazy-init fields as volatile for thread safety
5. Replace exception messages with generic errors in API responses
6. Unify DEFAULT_RELATION_TYPES between CSV and repository (10 types)
7. Add jitter backoff to deadlock retry in CollectionDAO
8. Replace N+1 queries in prefetchGraphTerms with batch fetch
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* Fix Fuseki tmpfs exhaustion and GlossaryTermRelationIT double init
- Remove tmpfs size limit on Fuseki container to prevent disk exhaustion
- Guard RdfUpdater.initialize() in GlossaryTermRelationIT to skip if
already initialized by bootstrap
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* Fix duplicate edges, null term NPE, and silent exception in graph builder
- Deduplicate edges in buildGraph() using edgesSeen set
- Skip TermRelation entries with null term references to prevent NPE
- Add warning log when glossary term relation settings fail to load
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* Fix cardinality count after canonical swap and double-checked locking
- getRelationCount now matches inverse relation type for fromRecords
where the term is the target, fixing cardinality bypass after
bidirectional UUID canonicalization
- Use double-checked locking in RdfResource.getSemanticSearchEngine()
to prevent duplicate instance creation under concurrency
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: anuj-kumary <anujf0510@gmail.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Ram Narayan Balaji <ramnarayanb3005@gmail.com>
Co-authored-by: Ram Narayan Balaji <81347100+yan-3005@users.noreply.github.com>
* Fix Metrics collection; reduce no.of metrics; improve slow request logging
* Move sync calls to search & rdf to async
* Improve slow request tracking
* Improve slow request tracking
* Add clear breakdown in slow request
* Batch TestCaseRepository calls
* Batch API calls
* Initial Implementation of ReadEngine
* Improvements with ReadEngine/WriteEngine
* Improvements with ReadEngine/WriteEngine
* Improvements with ReadEngine/WriteEngine
* Improve by removing unnecessary ser/de
* Additional improvements with PatchFieldsPlanner
* Further performance improvements
* Further performance improvements
* Address comments
* Merge from main
* Address comments
* Address comments
* Address latest feedback - 2/21
* fix merge conflict
* Address Slow Request review
* Address the comments
* Address comments; Fix tests
* Fixes to the failing tests
* Fix bugs in tests
* Fix checkstyle
* Address playwright tests
* Fix tests
* Fix bugs
* Fix tests
* address comments
* Fix issues from playwright
* Fix playwright tests
* Fix tests for playwright
* Address comments
* Fix glossary test
* fix checkstyle
* Fix playwright issues
* Fix playwright issues - incrementalChagneDesc
* Restore ApprovalTaskWorkflow in GlossaryTerm and TestCase repositories
The slow_request branch accidentally removed entity-specific ApprovalTaskWorkflow
overrides, causing the generic parent to use checkUpdatedByTaskAssignee instead of
checkUpdatedByReviewer. This broke Glossary approval and TestCase approval Playwright tests.
- GlossaryTermRepository: restore ApprovalTaskWorkflow with checkUpdatedByReviewer
- TestCaseRepository: restore ApprovalTaskWorkflow, preDelete guard, updateReviewers
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Fix base ApprovalTaskWorkflow to use reviewer check instead of task assignee
The centralized ApprovalTaskWorkflow in EntityRepository was using
checkUpdatedByTaskAssignee instead of checkUpdatedByReviewer, breaking
approval workflows for all entity types. Added verifyReviewer() as a
top-level static method on EntityRepository and restored missing
updateReviewers() and preDelete IN_REVIEW guards in DataContract,
DataProduct, Metric, and Tag repositories. Removed now-redundant
entity-specific ApprovalTaskWorkflow overrides from GlossaryTerm and
TestCase repositories.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Fix regression introduced in backend tests; make the playwright tests stable
* Stabilize the playwright tests
* Stabilize the playwright tests
* Improve playwright tests
* Improve playwright tests
* Fix team playwrights
* Fix merge from main
* Fix playwrigt tests
* Fix playwright tests
* Batch domain/data product asset counts into single ES aggregation queries
Replace N individual ES count queries with single aggregation query per
entity type. Domain counts roll up child counts to parent domains.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Improve Playwright test reliability and expand CI shards
Add polling waits for async ES indexing, fix lineage edge selectors,
use API-based setup for domain/data product widget tests, and expand
CI from 6 to 8 shards with dedicated graph/landing projects.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Playwright: Improve test reliability with response checks and guards
- Add API response status checks in create() for Domain, DataProduct,
Glossary, TableClass, and UserClass — silent API failures now throw
immediately with status code and response body
- Add guards in selectDataProduct() and addAssetsToDataProduct() for
undefined name/fqn — clear error messages instead of cryptic
"locator.fill: value: expected string, got undefined"
- Fix GlossaryPermissions double navigation — remove redundant
redirectToHomePage + sidebarClick before glossary.visitEntityPage()
- Increase OnlineUsers timeout from 5s to 15s for CI resource pressure
- Increase Tour badge timeout from 10s to 20s
- Fix visitGlossaryPage: wait for loader before clicking menuitem
- Remove chromium testIgnore for graph/landing/stateful test files
(these must run in chromium project for 6-shard CI workflow)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Playwright: Remove all networkidle waits and improve CI reliability
- Remove ~780 networkidle waits across 144 test/utility files — these
hang or resolve prematurely under CI load causing false negatives
- Add polling.ts with waitForSearchIndexed and waitForPageLoaded helpers
- Convert checkAssetsCount and search functions to expect.poll() for
async ES indexing tolerance
- Increase expect timeout to 15s for CI environments
- Split CI into 8 shards with dedicated projects (stateful/graph/landing)
to reduce thread contention
- Fix GITHUB_STEP_SUMMARY size overflow (base64 screenshots → table)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Playwright: Fix genuine test failures from networkidle removal
- GlossaryPagination: Fix waitForResponse race conditions - register
listener BEFORE the triggering action, add **/ URL prefix
- LanguageOverride: Fix selector from getByText('EN') to
getByText('English - EN') matching actual dropdown text
- NestedColumnsExpandCollapse: Fix URL glob pattern, use dispatchEvent
to avoid inner Link navigation, add waitForResponse for filtered search
- lineage.ts: Revert dragConnection hover approach that broke React
Flow connection mode, keep direct dispatchEvent
- customizeLandingPage.ts: Remove waitForURL that hangs after page.goto
- Teams.spec.ts: Add isJoinable: false for private team creation
- UserDetails.spec.ts: Revert Escape/clickOutside save flow that
dismissed edit mode before saving roles
- Users.spec.ts: Revert Data Consumer permissions test to original
simple approach using fixtures
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Playwright: Relax OnlineUsers activity time assertion
The "Online now" exact match fails under CI load because the activity
timestamp may show as "X seconds ago" or "X minutes ago" by the time
the page renders. Changed to accept any recent activity format.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Playwright: Fix 4 genuine test failures from CI run
1. saveCustomizeLayoutPage: Use response predicate matching both
POST (create) and PUT (update) patterns instead of glob that
only matched updates. Fixes 180s timeout in drag-and-drop test
when layout doesn't exist yet (fullyParallel=true).
2. GlossaryMiscOperations: Add test.slow(true) — test does 9
sequential page navigations that exceed the 60s timeout.
3. DomainDataProductsWidgets "Assign Widgets": Add test.slow(true)
— calls addAndVerifyWidget twice, each with multiple navigations.
4. DomainFilterQueryFilter: Add waitForAllLoadersToDisappear before
clicking domain-dropdown after search operations that trigger
page re-renders.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Playwright: Fix AutoPilot test — reload page after API status poll
The AutoPilot status banner never appeared because:
1. checkAutoPilotStatus polls the workflow API directly via apiContext
(outside the browser), not through page network requests
2. The UI uses WebSocket for live updates, but the socket connection
is only established when the page loads with status=RUNNING
3. Since the page loaded before the workflow started, the socket was
never connected, so the UI never received the completion event
Fix: reload the page after checkAutoPilotStatus confirms the workflow
finished, so the UI renders with the current state. Also increase the
banner visibility timeout to 30s for CI environments.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Playwright: Fix flaky tests — entity collisions, missing cleanup, expect timeout
- Replace Date.now() with uuid() for entity names in CustomProperties tests
to prevent collisions when parallel workers execute within the same millisecond
- Fix FollowingWidget: move shared adminUser create/delete to top-level
base.beforeAll/afterAll to prevent duplicate user creation across 11
parallel test.describe blocks
- Add missing afterAll cleanup to OnlineUsers, Metric, CustomPropertyAdvanceSearch,
and CustomProperties tests to prevent entity/user leaks between runs
- Replace hardcoded metric name in MetricSearch with uuid-based name
- Add global expect timeout of 15s (up from 5s default) for CI resilience
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Fix Playwright CI: include UI in build-once Maven build
The build-once optimization (#26423) used -DonlyBackend -pl !openmetadata-ui
which produces a tar.gz without the compiled React app. The Docker container
starts but cannot serve the login page, causing auth.setup.ts to timeout
on all 6 shards waiting for input[id="email"] to appear.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Fix CodeQL security warnings
- Replace Math.random() with crypto.randomUUID() for test data generation
- Escape backslash characters in CSS selectors for glossary FQN values
- Use page.getByTestId() instead of raw CSS selectors in entity utils
- Increase RSA key size from 512 to 2048 bits in JwtFilterTest
- Skip archive entries containing '..' in JsonUtils.getResourcesFromJarFile
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Playwright: Fix user cleanup to prevent 'Email Already Exists' failures
- Glossary.spec.ts: Fix typo user3.create→delete in afterAll, add missing adminUser.delete
- Teams.spec.ts: Add afterAll cleanup hooks for 3 nested describe blocks that were missing them (EditUser, DataConsumer, Owner)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Playwright: Add afterAll cleanup hooks and fix test reliability
- InputOutputPorts.spec.ts: Add afterAll for domain/tables/topics/dashboards
- Users.spec.ts: Add top-level afterAll for all shared entities
- Entity.spec.ts: Add afterAll for shared + per-entity-type cleanup
- Pagination.spec.ts: Add afterAll for 13 describe blocks (services, DBs, etc.)
- DataProductRename.spec.ts: Add afterAll cleanup
- TestCaseIncidentPermissions.spec.ts: Add afterAll for users/roles/policies/table
- ImpactAnalysis.spec.ts: Add afterAll for all 7 entity types
- NestedColumnsExpandCollapse.spec.ts: Add afterAll for 4 describe blocks
- DataProductPermissions.spec.ts: Add afterAll cleanup
- ServiceEntityPermissions.spec.ts: Add afterAll for testUser + per-entity
- ServiceForm.spec.ts: Add afterAll for adminUser
- domain.ts: Replace waitForTimeout(2000) with proper loader/tab waits
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Trigger Playwright CI
* Playwright: Fix 2 failures and 26 flaky tests with proper waits
Fix remaining 2 genuine failures:
- DomainDataProductsWidgets: add test.slow(true) for ES indexing lag
- Users.spec.ts: add test.slow(true) and loader waits for owner search
Fix 26 flaky tests by addressing 5 root cause patterns:
- Response listener after trigger: MetricCustomUnitFlow, DomainUIInteractions
- Missing loader wait after navigation: 16 tests across CustomizeDetailPage,
DataProductPersonaCustomization, DataContracts, ExploreTree, and others
- Element not rendered after API response: EntityVersionPages, ODCSImportExport
- DOM not settled after loader: Domains nested rename
- Permission cache propagation: GlossaryPermissions
Shared utility improvements:
- waitForPatchResponse uses entity-specific URL pattern
- openColumnDetailPanel accepts entityEndpoint param with API response wait
- Entity.spec.ts uses dynamic entity.endpoint instead of hardcoded tables
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Playwright: Fix addOwner retry to wait for search API response
The owner search retry loop was refilling the search input but not
waiting for the API response before checking item visibility. This
caused the poll to repeatedly check stale/empty results.
Fix: await search response and loader detach in each retry iteration.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Playwright: Fix owner listitem selector — remove exact match
The owner selection list items include avatar initials (e.g., "G") in their
accessible name, making exact: true fail since the accessible name is
"G UserName" not just "UserName". Switching to substring matching fixes
the Users.spec.ts persistent failure.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Playwright: Fix 10 remaining flaky tests with proper waits
- ColumnLevelTests: loader wait after visiting test case panel
- DataQualityPermissions: loader wait after visiting test suite page
- IncidentManagerDateFilter: loader wait after page reload
- InputOutputPorts: wait for warning alert before asserting
- Lineage: replace 5 hardcoded waitForTimeout(500) with loader waits
- CustomizeDetailPage: dialog close waits, fix missing await on expect
- DataProductPersonaCustomization: loader wait + modal visibility check
- GlossaryPermissions: increase permission propagation wait, loader wait
- GlossaryHierarchy: loader waits after modal close and glossary select
- ExploreTree: loader waits after API response before UI interaction
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Fix CodeQL security alerts: incomplete escaping and Zip Slip
1. entity.ts: Use JSON.stringify().slice(1,-1) for proper escaping of
both backslashes and double quotes in filter values, replacing the
incomplete .replace(/"/g, '\\"') approach.
2. JsonUtils.java: Strengthen Zip Slip protection by normalizing paths
via Paths.get().normalize() and rejecting entries starting with "/"
or resolving to parent traversal after normalization.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Fix tests
* Fix tests
* Fix recordChange field name mismatches and CodeQL alert
- ServiceEntityRepository: recordChange("ingestionAgent") → "ingestionRunner"
to match the JSON property name. The shouldCompare() gate in PATCH flow
was silently dropping ingestionRunner changes because the field name
didn't match patchedFields.
- DataContractRepository: compareAndUpdate("status") → "entityStatus"
to match the JSON property name, same root cause.
- JsonUtils: Simplify Zip Slip check to string-based validation to
satisfy CodeQL taint analysis.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Remove serial mode from Users.spec.ts to prevent cascade failures
A single flaky test failure was causing ~19 tests across 5 unrelated
describe blocks to be skipped. Matches main branch behavior (parallel).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Playwright: Fix flaky tests — missing awaits, hardcoded waits, silent catches
- DataProductPersonaCustomization: add missing await on expect() calls
- TestCaseIncidentPermissions: poll for incident creation instead of one-shot query
- TestCaseResultPermissions: add loader wait after Data Quality tab click
- GlossaryPermissions: replace waitForTimeout(3000) with toPass() retry
- BulkImport: remove 4 unnecessary waitForTimeout calls
- importUtils/testCases: replace waitForTimeout(500) with grid visibility assert
- GlossaryAssets: add loader wait, remove silent .catch(() => false) pattern
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Fix CodeQL Zip Slip alert with Path.normalize() sanitization
CodeQL doesn't recognize String.contains("..") as proper Zip Slip
mitigation. Use Path.normalize() + isAbsolute/startsWith checks which
CodeQL's taint analysis model understands.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Fix Playwright flaky tests: modal visibility, toast race, query card assertion
- DataProductPersonaCustomization: wait for dialog close before clicking add-widget-button
- entity.ts restoreEntity: dismiss stale toast before restore to avoid race condition
- QueryEntity: replace page.$$() with auto-retrying expect().toBeVisible()
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Fix flaky TableResourceIT by preventing parallel multi-domain rule mutation
Both test_multipleDomainInheritance (TableResourceIT) and
test_csvImportEntityRuleValidation (DatabaseServiceResourceIT) toggle
the global "Multiple Domains are not allowed" rule. When running
concurrently, one overwrites the other's setting causing spurious
failures. Add @ResourceLock("MULTI_DOMAIN_RULE") to serialize only
these two tests while keeping all others concurrent.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Mohit Yadav <105265192+mohityadav766@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* Fix preview→enabled migration for event_subscription_entity and QRTZ tables
The 1.13.0 migration renamed `preview` to `enabled` in `apps_marketplace`
and `installed_apps`, but missed the `event_subscription_entity` table.
The ReverseMetadata app stores the full App entity as an escaped JSON
string inside `event_subscription_entity.json -> config -> app`. Since
it's a string value (not a nested JSON object), standard JSON path
operations can't reach the `"preview"` field — string replacement is
needed instead.
Also truncates QRTZ tables to clear stale Quartz job data that may
contain old App JSON. Both schedulers re-create their jobs from the
database on startup.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Use DELETE instead of TRUNCATE for QRTZ cleanup to respect FK constraints
TRUNCATE fails on tables referenced by foreign keys in MySQL (and
without CASCADE in PostgreSQL). Switch to DELETE FROM with correct
FK ordering (children before parents) and add missing child tables
(QRTZ_SIMPROP_TRIGGERS, QRTZ_BLOB_TRIGGERS).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
The migrations renaming the 'preview' property to 'enabled' in apps
were incorrectly placed under 1.11.13. Move them to 1.13.0 where they
belong, since this change targets the next major release.
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* Rename app 'preview' property to 'enabled' with inverted semantics
The 'preview' property was confusing: preview=false meant the app CAN
be used. Replace with 'enabled' where enabled=true means usable, which
is much more intuitive.
Changes across the full stack:
- JSON schemas: preview (default false) → enabled (default true)
- Java backend: isPreview/raisePreviewMessage → isEnabled/raiseNotEnabledMessage
- TypeScript types: preview → enabled
- Frontend component: isPreviewApp → isAppDisabled (checks enabled===false)
- SQL migrations for 1.11.12: rename + invert boolean in apps_marketplace
and installed_apps tables (MySQL and PostgreSQL)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Update generated TypeScript types
* format
* improve deletion process for disabled apps
* improve deletion process for disabled apps
* improve deletion process for disabled apps
* improve deletion process for disabled apps
* format
* fix tests
* migration
* migration
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
* fix glossary status
* add glossaryTerm spec
* fix: improve ListFilter implementation in list filtering logic
Co-authored-by: siddhant1 <siddhant1@users.noreply.github.com>
* reset main backend
* reset backend
* fix be
* rever
* spottless
* Fix GlossrayTerm search api endpoint
* status enum validation
* fix spec
* Replace quotes, validate enum
* bind param queries
* Move migrations to 1.12.0
* fix api docs
* optimize performance of fallback , refactoring
* fix ListFilter
* GlossaryTermService.java cleanup
* address gitar-bot feedback
* add entityStatus param in list api
* add entityStatus param in list api
* Send entityStatus param with both search and list glossary term APIs
- Pass entityStatus to searchGlossaryTermsPaginated and
getFirstLevelGlossaryTermsPaginated when a specific status filter
is active (not 'all')
- Keep 'All' option in status dropdown with default selection of
Approved, Draft, InReview
- Show appropriate empty state message when status filter returns
no results
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* update list API path (ListFilter.getEntityStatusCondition) to validate against the enum, in case if an invalid value like "Bogus" is passed
* fix playwright
* Fix rejected glossary term staying visible in listing
Remove rejected terms from visible list when status filter excludes
them, and fix reused waitForResponse promise in Playwright test.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* add initian load
* Fix Expand All ignoring active status filter and add E2E tests
Pass entityStatus parameter in fetchExpadedTree so Expand All respects
the active status filter. Add E2E test suite to verify the behavior.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Rewrite Glossary Expand All E2E tests to follow Playwright handbook patterns
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Fix flaky GlossaryPagination test by scoping locators to glossary table
Scoped unscoped `tbody .ant-table-row` locators to `glossary-terms-table`
testid, and replaced unreliable row count assertion in empty state test
with visibility checks on `no-data-placeholder`.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Siddhant <siddhant@MacBook-Pro.local>
Co-authored-by: Gitar <noreply@gitar.ai>
Co-authored-by: siddhant1 <siddhant1@users.noreply.github.com>
Co-authored-by: Ram Narayan Balaji <ramnarayanb3005@gmail.com>
Co-authored-by: Ram Narayan Balaji <81347100+yan-3005@users.noreply.github.com>
Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
Co-authored-by: sonika-shah <58761340+sonika-shah@users.noreply.github.com>
Co-authored-by: Siddhant <siddhant@MacBook-Pro-3.local>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Siddhant <siddhant@MacBook-Pro-4.local>
* Optimize Reads with Keyset
* Optimize Search Index Processing stage
* Fix KeySet Cursor
* revert keyset for time series
* Fix Review Comments
* Move to 1.12.2
* Fix Review Comment
* Remove IF NOT EXISTS from mysql and update common mthod
* MINOR - Streamline bot impersonation from apps
* MINOR - Streamline bot impersonation from apps
* MINOR - Streamline bot impersonation from apps
* MINOR - Streamline bot impersonation from apps
* Update generated TypeScript types
* policy flag
* policy flag
* policy flag
* policy flag
* fix feedback
---------
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
* Fix: Resolve v1.12.0 migration failure due to NULL workflow status
## Root Cause Analysis
- Migration failed when modifying entityLink column in workflow_instance_time_series
- MySQL's ALTER TABLE MODIFY COLUMN re-validates ALL generated columns for ALL rows
- Found 184+ workflow instances created between Dec 2024 - Jan 2025 with NULL status
- These were created with pre-v1.7.0 code that didn't set status field in JSON
- v1.7.0 added status column as GENERATED NOT NULL but old instances had NULL values
- v1.12.0 migration triggered constraint validation, causing "Column 'status' cannot be null"
## Solution
- Add UPDATE statements before ALTER TABLE in v1.12.0 migration
- Set status='FINISHED' for workflows with endedAt (completed)
- Set status='FAILED' for workflows without endedAt (incomplete)
- Use two separate queries for better performance vs CASE statements
- Handle both workflow_instance_time_series and workflow_instance_state_time_series
* failed to FAILURE status
* Fix - disk space in github workflows
* Fix - disk space in github workflows
* Fix - disk space in github workflows
* Fix running tests with bulk apis
* Fix running tests with bulk apis
* Address comments; make awaitability for tests
* Address comments
* feat(salesforce): add sobjectNames field for multi-object selection
Add support for specifying multiple Salesforce objects to ingest
instead of just one or all. The new `sobjectNames` array field
allows users to select specific objects (e.g., Contact, Account,
Lead) without having to ingest all objects and filter them.
Priority order:
1. sobjectNames (array) - if specified, use only these
2. sobjectName (string) - if specified and sobjectNames empty
3. All objects from describe() - if neither specified
tableFilterPattern applies in all cases as a final filter.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
* refactor: removed sobjectName field and added a migration for 1.11.8 to migrate sobjectName values to sobjectNames
* fix: sobjectNames priority comment
* refactor: sobjectNames changes in ts files
* fix: yaml structure in test_salesforce
* fix: test_salesforce.py - metadata as OpenMetadata object
* fix: added new line in sql migrations
* fix: sql migration serviceType
---------
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Keshav Mohta <keshavmohta09@gmail.com>
Co-authored-by: Keshav Mohta <68001229+keshavmohta09@users.noreply.github.com>
Co-authored-by: Mohit Yadav <105265192+mohityadav766@users.noreply.github.com>
* Fix MySQL timestamp precision for tag_usage.appliedAt
MySQL's TIMESTAMP type defaults to second precision, while PostgreSQL
returns microsecond precision. This causes _normalize_datetime_strings
in the Python ingestion client to produce spurious appliedAt diffs in
JSON patches, which then fail with "Failed to convert JsonValue to
target class" during deserialization in JsonUtils.applyPatch().
Upgrade appliedAt to TIMESTAMP(6) to match PostgreSQL behavior and
eliminate the spurious patch diffs.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Add 1.11.8 migration for MySQL appliedAt timestamp precision
Backport the TIMESTAMP(6) fix to the 1.11.x release line so existing
deployments on 1.11.x pick up the fix without requiring a 1.12.0 upgrade.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
* Add design doc for search indexing stats redesign
Covers:
- Simplified 4-stage pipeline model (Reader, Process, Sink, Vector)
- Per-entity index promotion instead of batch promotion
- Alias management from indexMapping.json
- Payload-aware vector bulk processor
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Add Support for Per Entity Index Promotion
* Add UI Bit
* Add Lang
* Add AppLog View Test coverage
* Add Bathced Vector index querying
* Add Improvements for Vector to be async and also stats to be better handled
* Use Virtual Thread
* Use Virtual Thread
* Fix Tests
* Make reading stats easier
* Fixed Stats to be accurate
* Fix Stats getting null
* Fix partition worker stats
* Fix Reader Stats - final
* Update generated TypeScript types
* Make updates in 1.12.0
* Revert "Use Virtual Thread"
This reverts commit 4eb23374d1.
* Revert "Use Virtual Thread"
This reverts commit efe8d03b5d.
* Reapply "Use Virtual Thread"
This reverts commit d59cde18b2.
* Reapply "Use Virtual Thread"
This reverts commit 769e5710c3.
* Fix Final Update on stat
* - Add atomic alias swap
- remove unnecessary migration
* Fix Sonar test jest
* Fix Final Update on stat
---------
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
* Ensure columns are retrieved in the right order
This is because since introducing ordering for `getTableColumnsByFQN`, the patches created in `removeTagFromEntity` were open to pointing to different columns if the default order didn't match how they were persisted in db
* Allow exception list to be updated on all feedback
* Apply gitar comments
* Add `metadata` to `tag_usage` table
* Update JSON schema object to include `TagLabel.metadata`
* Apply feedback to selected recognizer
* Add backend integration tests
* Update `ingestion` to return `TagLabel.metadata.recognizer`
* Update generated TypeScript types
* Update generated TypeScript types
* Send recognizer result metadata in feedback approval task (#25485)
* Send `TagLabelRecognizerMetadata` in `TaskDetails`
This is so we can show an explanation behind the classification in the feedback approval card
* Update typescript types
* Run Spotless
* Ensure `applyTagsBatchInternal` works equally for pg and mysql
* Tag metadata fixes
* Fix CI test
---------
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Rohit Jain <60229265+Rohit0301@users.noreply.github.com>
Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
* Fix Stats
* Add Warning logs and reindex failure analysis
* Add Search Insights in Preferences
* Add Label
* Fix Full Error not available
* Add check for reindex run
* FIX#24374 - Data Contract at Data Product level
* Update generated TypeScript types
* FIX#24374 - Data Contract at Data Product level
* fix DP page
* fix: preserve termsOfUse object format in filtered contract
The termsOfUse field was being converted to a string during filtering,
but the form components expect it to be an object with {content: string}.
This was causing test failures where form elements were not visible.
- Keep termsOfUse as object format when not inherited
- Convert old string format to new object format for consistency
- Fixes 21 test failures in DataContracts.spec.ts and DataContractInheritance.spec.ts
* fix: address code review findings - state sync and immutability
Frontend changes:
- Add useEffect to sync formValues with filteredContract changes
- Ensures edit form updates when contract prop changes
Backend changes:
- Create deep copy at start of mergeContracts() to avoid mutating input
- Prevents side effects if contract object is reused elsewhere
Co-authored-by: pmbrull <pmbrull@users.noreply.github.com>
* Addressing feedback
Co-authored-by: pmbrull <pmbrull@users.noreply.github.com>
* fix tests
* fix inherited contract delete and status
* fix inherited contract delete and status
* fix inherited contract execution in app
* fix test
* fix: resolve playwright postgresql ci test failure
Co-authored-by: pmbrull <pmbrull@users.noreply.github.com>
* ci: fix yaml validation and checkstyle failures
Co-authored-by: pmbrull <pmbrull@users.noreply.github.com>
* fix: correct JSON/YAML validation errors
Co-authored-by: pmbrull <pmbrull@users.noreply.github.com>
* fix: resolve maven-collate and ui-coverage test failures
Co-authored-by: pmbrull <pmbrull@users.noreply.github.com>
* gitar feedback
* fix ci
* fix ci
* fix ci
* fix ci
* include .claude
* validate
* fix playwright
* playwright
* fix playwright
---------
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Gitar <gitar@collate.io>
Co-authored-by: Gitar <noreply@gitar.ai>
Co-authored-by: pmbrull <pmbrull@users.noreply.github.com>
Co-authored-by: Karan Hotchandani <33024356+karanh37@users.noreply.github.com>
Co-authored-by: karanh37 <karanh37@gmail.com>
* feat: added repository logic to list all versions (including latest) for a specific entity type
* feat: added list all versions for all the entity resources
* feat: moved endpoint to EntityResource
* feat: renamed endpoint to /history and methods to EntityHistory
* feat: ran java linting
* feat: remove v1 implementation left over code
* feat: fix failing tests
* feat: ran klinting
* feat: fix psql query
* feat: address PR comments
* feat: ran klinting
* feat: increase cache duration
* feat: address query edge cases
* fix(dq): psql migration for row insert test parameters
* fix(dq): use name and add trailing new line
* Fix description formatting in postDataMigrationSQLScript.sql
* Apply suggestion from @Copilot
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Add DQ Rules Library
* Add DQ Rules Library - Add Tests and enable testDefinitions through APIs to list
* Add DQ Rules Library - Add Tests and enable testDefinitions through APIs to list
* Add DQ Rules Library - Add Tests and enable testDefinitions through APIs to list
* Update generated TypeScript types
* Add DQ Rules Library - Add Tests and enable testDefinitions through APIs to list
* Add DQ Rules Library - Add Tests and enable testDefinitions through APIs to list
* Update generated TypeScript types
* Refactor tests to use toStrictEqual for string comparisons and improve consistency
- Updated various test files to replace `toBe` with `toStrictEqual` for string assertions in ImportStatus, SummaryCard, TabsLabel, and others.
- Enhanced regex tests to ensure accurate validation of entity names and tags.
- Added new translations for test platform warnings in en-us.json.
- Improved utility tests for alerts, authentication, CSV handling, and task messages to use `toEqual` for better clarity.
* Refactor TestDefinitionForm and TestDefinitionList components to use updated API methods and improve SQL expression handling
* Enhance TestDefinitionList component with permission checks for edit and delete actions, and update tests to reflect changes in permission handling
* Remove debug log from handleSubmit in TestDefinitionForm component
* Add permission loading state and enhance permission handling in TestDefinitionList component
* Update generated TypeScript types
* Update generated TypeScript types
* Update generated TypeScript types
* fix build failure
* Revert "Update generated TypeScript types"
This reverts commit 67b062216f.
* Enhance TestDefinitionForm and TestDefinitionList components with improved UI and pagination handling
* fix: update RulesLibrary tests and enhance TestDefinitionForm styling
* fix: Enhance TestDefinitionForm with error handling and improved UX
* fix: Update test definition handling and improve rendering in TestDefinitionList
* fix: Refactor TestDefinitionPermissions tests for improved permission checks and API context handling
* fix: Update system test definition retrieval to use findLast for improved accuracy
* feat: Add end-to-end tests for Rules Library and Test Definition Permissions
* fix: Update edit button visibility check to use beDisabled for better clarity
* fix: Refactor response handling in TestDefinitionPermissions tests for improved reliability
* move migrations execution order
* fix: remove existing columns
* style: remove migration extra line break
* chore: fix migration
---------
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Shailesh Parmar <shailesh.parmar.webdev@gmail.com>
Co-authored-by: TeddyCr <teddy.crepineau@gmail.com>
* Added `appliedAt` field to `TagLabel`s
This is to track insertions to `tag_usage` with timestamps
* Capture and format recognition explanations in `TagAnalyzer`
This creates a function to build an explanation to why something was scored as it was.
# Conflicts:
# ingestion/src/metadata/pii/algorithms/presidio_utils.py
* Refactor `TagProcessor`
* Capture results for the old-style `PIIProcessor`
* Move strings to constants
* Add `TagLabel.appliedBy` field
This change also patches the user's name into the tags declared in `JsonPatch` objects to fill it up
* Update typescript types
* Fix python tests
* Fix java tests
* Simplify setting tag's `appliedBy` using `EntityUpdater.updatingUser`
* Remove unnecessary f-string
* Moar fixes
* Move migrations to 1.11.5
