mirror of
https://github.com/open-metadata/OpenMetadata
synced 2026-05-24 09:39:11 +00:00
3 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Sriharsha Chintalapani
|
12b364313c
|
Fix Metrics collection; reduce no.of metrics; improve slow request lo… (#25751)
* Fix Metrics collection; reduce no.of metrics; improve slow request logging
* Move sync calls to search & rdf to async
* Improve slow request tracking
* Improve slow request tracking
* Add clear breakdown in slow request
* Batch TestCaseRepository calls
* Batch API calls
* Initial Implementation of ReadEngine
* Improvements with ReadEngine/WriteEngine
* Improvements with ReadEngine/WriteEngine
* Improvements with ReadEngine/WriteEngine
* Improve by removing unnecessary ser/de
* Additional improvements with PatchFieldsPlanner
* Further performance improvements
* Further performance improvements
* Address comments
* Merge from main
* Address comments
* Address comments
* Address latest feedback - 2/21
* fix merge conflict
* Address Slow Request review
* Address the comments
* Address comments; Fix tests
* Fixes to the failing tests
* Fix bugs in tests
* Fix checkstyle
* Address playwright tests
* Fix tests
* Fix bugs
* Fix tests
* address comments
* Fix issues from playwright
* Fix playwright tests
* Fix tests for playwright
* Address comments
* Fix glossary test
* fix checkstyle
* Fix playwright issues
* Fix playwright issues - incrementalChagneDesc
* Restore ApprovalTaskWorkflow in GlossaryTerm and TestCase repositories
The slow_request branch accidentally removed entity-specific ApprovalTaskWorkflow
overrides, causing the generic parent to use checkUpdatedByTaskAssignee instead of
checkUpdatedByReviewer. This broke Glossary approval and TestCase approval Playwright tests.
- GlossaryTermRepository: restore ApprovalTaskWorkflow with checkUpdatedByReviewer
- TestCaseRepository: restore ApprovalTaskWorkflow, preDelete guard, updateReviewers
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Fix base ApprovalTaskWorkflow to use reviewer check instead of task assignee
The centralized ApprovalTaskWorkflow in EntityRepository was using
checkUpdatedByTaskAssignee instead of checkUpdatedByReviewer, breaking
approval workflows for all entity types. Added verifyReviewer() as a
top-level static method on EntityRepository and restored missing
updateReviewers() and preDelete IN_REVIEW guards in DataContract,
DataProduct, Metric, and Tag repositories. Removed now-redundant
entity-specific ApprovalTaskWorkflow overrides from GlossaryTerm and
TestCase repositories.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Fix regression introduced in backend tests; make the playwright tests stable
* Stabilize the playwright tests
* Stabilize the playwright tests
* Improve playwright tests
* Improve playwright tests
* Fix team playwrights
* Fix merge from main
* Fix playwrigt tests
* Fix playwright tests
* Batch domain/data product asset counts into single ES aggregation queries
Replace N individual ES count queries with single aggregation query per
entity type. Domain counts roll up child counts to parent domains.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Improve Playwright test reliability and expand CI shards
Add polling waits for async ES indexing, fix lineage edge selectors,
use API-based setup for domain/data product widget tests, and expand
CI from 6 to 8 shards with dedicated graph/landing projects.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Playwright: Improve test reliability with response checks and guards
- Add API response status checks in create() for Domain, DataProduct,
Glossary, TableClass, and UserClass — silent API failures now throw
immediately with status code and response body
- Add guards in selectDataProduct() and addAssetsToDataProduct() for
undefined name/fqn — clear error messages instead of cryptic
"locator.fill: value: expected string, got undefined"
- Fix GlossaryPermissions double navigation — remove redundant
redirectToHomePage + sidebarClick before glossary.visitEntityPage()
- Increase OnlineUsers timeout from 5s to 15s for CI resource pressure
- Increase Tour badge timeout from 10s to 20s
- Fix visitGlossaryPage: wait for loader before clicking menuitem
- Remove chromium testIgnore for graph/landing/stateful test files
(these must run in chromium project for 6-shard CI workflow)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Playwright: Remove all networkidle waits and improve CI reliability
- Remove ~780 networkidle waits across 144 test/utility files — these
hang or resolve prematurely under CI load causing false negatives
- Add polling.ts with waitForSearchIndexed and waitForPageLoaded helpers
- Convert checkAssetsCount and search functions to expect.poll() for
async ES indexing tolerance
- Increase expect timeout to 15s for CI environments
- Split CI into 8 shards with dedicated projects (stateful/graph/landing)
to reduce thread contention
- Fix GITHUB_STEP_SUMMARY size overflow (base64 screenshots → table)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Playwright: Fix genuine test failures from networkidle removal
- GlossaryPagination: Fix waitForResponse race conditions - register
listener BEFORE the triggering action, add **/ URL prefix
- LanguageOverride: Fix selector from getByText('EN') to
getByText('English - EN') matching actual dropdown text
- NestedColumnsExpandCollapse: Fix URL glob pattern, use dispatchEvent
to avoid inner Link navigation, add waitForResponse for filtered search
- lineage.ts: Revert dragConnection hover approach that broke React
Flow connection mode, keep direct dispatchEvent
- customizeLandingPage.ts: Remove waitForURL that hangs after page.goto
- Teams.spec.ts: Add isJoinable: false for private team creation
- UserDetails.spec.ts: Revert Escape/clickOutside save flow that
dismissed edit mode before saving roles
- Users.spec.ts: Revert Data Consumer permissions test to original
simple approach using fixtures
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Playwright: Relax OnlineUsers activity time assertion
The "Online now" exact match fails under CI load because the activity
timestamp may show as "X seconds ago" or "X minutes ago" by the time
the page renders. Changed to accept any recent activity format.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Playwright: Fix 4 genuine test failures from CI run
1. saveCustomizeLayoutPage: Use response predicate matching both
POST (create) and PUT (update) patterns instead of glob that
only matched updates. Fixes 180s timeout in drag-and-drop test
when layout doesn't exist yet (fullyParallel=true).
2. GlossaryMiscOperations: Add test.slow(true) — test does 9
sequential page navigations that exceed the 60s timeout.
3. DomainDataProductsWidgets "Assign Widgets": Add test.slow(true)
— calls addAndVerifyWidget twice, each with multiple navigations.
4. DomainFilterQueryFilter: Add waitForAllLoadersToDisappear before
clicking domain-dropdown after search operations that trigger
page re-renders.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Playwright: Fix AutoPilot test — reload page after API status poll
The AutoPilot status banner never appeared because:
1. checkAutoPilotStatus polls the workflow API directly via apiContext
(outside the browser), not through page network requests
2. The UI uses WebSocket for live updates, but the socket connection
is only established when the page loads with status=RUNNING
3. Since the page loaded before the workflow started, the socket was
never connected, so the UI never received the completion event
Fix: reload the page after checkAutoPilotStatus confirms the workflow
finished, so the UI renders with the current state. Also increase the
banner visibility timeout to 30s for CI environments.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Playwright: Fix flaky tests — entity collisions, missing cleanup, expect timeout
- Replace Date.now() with uuid() for entity names in CustomProperties tests
to prevent collisions when parallel workers execute within the same millisecond
- Fix FollowingWidget: move shared adminUser create/delete to top-level
base.beforeAll/afterAll to prevent duplicate user creation across 11
parallel test.describe blocks
- Add missing afterAll cleanup to OnlineUsers, Metric, CustomPropertyAdvanceSearch,
and CustomProperties tests to prevent entity/user leaks between runs
- Replace hardcoded metric name in MetricSearch with uuid-based name
- Add global expect timeout of 15s (up from 5s default) for CI resilience
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Fix Playwright CI: include UI in build-once Maven build
The build-once optimization (#26423) used -DonlyBackend -pl !openmetadata-ui
which produces a tar.gz without the compiled React app. The Docker container
starts but cannot serve the login page, causing auth.setup.ts to timeout
on all 6 shards waiting for input[id="email"] to appear.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Fix CodeQL security warnings
- Replace Math.random() with crypto.randomUUID() for test data generation
- Escape backslash characters in CSS selectors for glossary FQN values
- Use page.getByTestId() instead of raw CSS selectors in entity utils
- Increase RSA key size from 512 to 2048 bits in JwtFilterTest
- Skip archive entries containing '..' in JsonUtils.getResourcesFromJarFile
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Playwright: Fix user cleanup to prevent 'Email Already Exists' failures
- Glossary.spec.ts: Fix typo user3.create→delete in afterAll, add missing adminUser.delete
- Teams.spec.ts: Add afterAll cleanup hooks for 3 nested describe blocks that were missing them (EditUser, DataConsumer, Owner)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Playwright: Add afterAll cleanup hooks and fix test reliability
- InputOutputPorts.spec.ts: Add afterAll for domain/tables/topics/dashboards
- Users.spec.ts: Add top-level afterAll for all shared entities
- Entity.spec.ts: Add afterAll for shared + per-entity-type cleanup
- Pagination.spec.ts: Add afterAll for 13 describe blocks (services, DBs, etc.)
- DataProductRename.spec.ts: Add afterAll cleanup
- TestCaseIncidentPermissions.spec.ts: Add afterAll for users/roles/policies/table
- ImpactAnalysis.spec.ts: Add afterAll for all 7 entity types
- NestedColumnsExpandCollapse.spec.ts: Add afterAll for 4 describe blocks
- DataProductPermissions.spec.ts: Add afterAll cleanup
- ServiceEntityPermissions.spec.ts: Add afterAll for testUser + per-entity
- ServiceForm.spec.ts: Add afterAll for adminUser
- domain.ts: Replace waitForTimeout(2000) with proper loader/tab waits
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Trigger Playwright CI
* Playwright: Fix 2 failures and 26 flaky tests with proper waits
Fix remaining 2 genuine failures:
- DomainDataProductsWidgets: add test.slow(true) for ES indexing lag
- Users.spec.ts: add test.slow(true) and loader waits for owner search
Fix 26 flaky tests by addressing 5 root cause patterns:
- Response listener after trigger: MetricCustomUnitFlow, DomainUIInteractions
- Missing loader wait after navigation: 16 tests across CustomizeDetailPage,
DataProductPersonaCustomization, DataContracts, ExploreTree, and others
- Element not rendered after API response: EntityVersionPages, ODCSImportExport
- DOM not settled after loader: Domains nested rename
- Permission cache propagation: GlossaryPermissions
Shared utility improvements:
- waitForPatchResponse uses entity-specific URL pattern
- openColumnDetailPanel accepts entityEndpoint param with API response wait
- Entity.spec.ts uses dynamic entity.endpoint instead of hardcoded tables
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Playwright: Fix addOwner retry to wait for search API response
The owner search retry loop was refilling the search input but not
waiting for the API response before checking item visibility. This
caused the poll to repeatedly check stale/empty results.
Fix: await search response and loader detach in each retry iteration.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Playwright: Fix owner listitem selector — remove exact match
The owner selection list items include avatar initials (e.g., "G") in their
accessible name, making exact: true fail since the accessible name is
"G UserName" not just "UserName". Switching to substring matching fixes
the Users.spec.ts persistent failure.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Playwright: Fix 10 remaining flaky tests with proper waits
- ColumnLevelTests: loader wait after visiting test case panel
- DataQualityPermissions: loader wait after visiting test suite page
- IncidentManagerDateFilter: loader wait after page reload
- InputOutputPorts: wait for warning alert before asserting
- Lineage: replace 5 hardcoded waitForTimeout(500) with loader waits
- CustomizeDetailPage: dialog close waits, fix missing await on expect
- DataProductPersonaCustomization: loader wait + modal visibility check
- GlossaryPermissions: increase permission propagation wait, loader wait
- GlossaryHierarchy: loader waits after modal close and glossary select
- ExploreTree: loader waits after API response before UI interaction
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Fix CodeQL security alerts: incomplete escaping and Zip Slip
1. entity.ts: Use JSON.stringify().slice(1,-1) for proper escaping of
both backslashes and double quotes in filter values, replacing the
incomplete .replace(/"/g, '\\"') approach.
2. JsonUtils.java: Strengthen Zip Slip protection by normalizing paths
via Paths.get().normalize() and rejecting entries starting with "/"
or resolving to parent traversal after normalization.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Fix tests
* Fix tests
* Fix recordChange field name mismatches and CodeQL alert
- ServiceEntityRepository: recordChange("ingestionAgent") → "ingestionRunner"
to match the JSON property name. The shouldCompare() gate in PATCH flow
was silently dropping ingestionRunner changes because the field name
didn't match patchedFields.
- DataContractRepository: compareAndUpdate("status") → "entityStatus"
to match the JSON property name, same root cause.
- JsonUtils: Simplify Zip Slip check to string-based validation to
satisfy CodeQL taint analysis.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Remove serial mode from Users.spec.ts to prevent cascade failures
A single flaky test failure was causing ~19 tests across 5 unrelated
describe blocks to be skipped. Matches main branch behavior (parallel).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Playwright: Fix flaky tests — missing awaits, hardcoded waits, silent catches
- DataProductPersonaCustomization: add missing await on expect() calls
- TestCaseIncidentPermissions: poll for incident creation instead of one-shot query
- TestCaseResultPermissions: add loader wait after Data Quality tab click
- GlossaryPermissions: replace waitForTimeout(3000) with toPass() retry
- BulkImport: remove 4 unnecessary waitForTimeout calls
- importUtils/testCases: replace waitForTimeout(500) with grid visibility assert
- GlossaryAssets: add loader wait, remove silent .catch(() => false) pattern
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Fix CodeQL Zip Slip alert with Path.normalize() sanitization
CodeQL doesn't recognize String.contains("..") as proper Zip Slip
mitigation. Use Path.normalize() + isAbsolute/startsWith checks which
CodeQL's taint analysis model understands.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Fix Playwright flaky tests: modal visibility, toast race, query card assertion
- DataProductPersonaCustomization: wait for dialog close before clicking add-widget-button
- entity.ts restoreEntity: dismiss stale toast before restore to avoid race condition
- QueryEntity: replace page.$$() with auto-retrying expect().toBeVisible()
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Fix flaky TableResourceIT by preventing parallel multi-domain rule mutation
Both test_multipleDomainInheritance (TableResourceIT) and
test_csvImportEntityRuleValidation (DatabaseServiceResourceIT) toggle
the global "Multiple Domains are not allowed" rule. When running
concurrently, one overwrites the other's setting causing spurious
failures. Add @ResourceLock("MULTI_DOMAIN_RULE") to serialize only
these two tests while keeping all others concurrent.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Mohit Yadav <105265192+mohityadav766@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
|
||
|
Pere Miquel Brull
|
62c12a133d
|
Fix 1.13.0 preview→enabled migration for event subscriptions (#26473)
* Fix preview→enabled migration for event_subscription_entity and QRTZ tables The 1.13.0 migration renamed `preview` to `enabled` in `apps_marketplace` and `installed_apps`, but missed the `event_subscription_entity` table. The ReverseMetadata app stores the full App entity as an escaped JSON string inside `event_subscription_entity.json -> config -> app`. Since it's a string value (not a nested JSON object), standard JSON path operations can't reach the `"preview"` field — string replacement is needed instead. Also truncates QRTZ tables to clear stale Quartz job data that may contain old App JSON. Both schedulers re-create their jobs from the database on startup. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Use DELETE instead of TRUNCATE for QRTZ cleanup to respect FK constraints TRUNCATE fails on tables referenced by foreign keys in MySQL (and without CASCADE in PostgreSQL). Switch to DELETE FROM with correct FK ordering (children before parents) and add missing child tables (QRTZ_SIMPROP_TRIGGERS, QRTZ_BLOB_TRIGGERS). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> |
||
|
Pere Miquel Brull
|
f890e004ce
|
Move preview-to-enabled migrations from 1.11.13 to 1.13.0 (#26281)
The migrations renaming the 'preview' property to 'enabled' in apps were incorrectly placed under 1.11.13. Move them to 1.13.0 where they belong, since this change targets the next major release. Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> |
Renamed from bootstrap/sql/migrations/native/1.11.13/postgres/schemaChanges.sql (Browse further)