mirror of
https://github.com/open-metadata/OpenMetadata
synced 2026-05-24 09:39:11 +00:00
761 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
mohitdeuex
|
e90a48fadd | Update name | ||
|
mohitdeuex
|
4f78c6cf05 |
test(review): address pmbrull nits on PR #28008
- nightly workflow: reformat the Topology comment block (drop the column-aligned space padding that read as "weird spaces"). - nightly workflow: hoist the stress cohort sizes (simpleReindex tables/topics/dashboards/pipelines, searchAvailable tables) into workflow_dispatch inputs with the current values as defaults, so they're tunable from the Actions UI per run. - remove openmetadata-integration-tests/REINDEX_TEST_PLAN.md — a planning/tracking doc that doesn't belong in the repo. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
mohitdeuex
|
b933eee132 |
test(review): address PR #28008 bot review comments
Real bugs: - UiTestServer: external mode (OM_URL+OM_ADMIN_TOKEN) now honours the operator token instead of minting a local one the external server won't trust; no TokenRefresher for the static external token. - UiSession.uiUrl(): strip the /api REST base before appending UI paths instead of relying on URI.resolve (fragile for relative paths / trailing-slash bases → /api/<route> 404s). - CpuSampler.percentile(): index off (length-1); floor(p*length) returned the max for small n, overstating p95. - OidcEnvBuilder: keep OM's own JWKS in AUTHENTICATION_PUBLIC_KEYS alongside the mock IdP's — SSO mode still validates OM-minted internal/bot tokens. - DataQualityDashboardPage.tryClickDimensionCard: stop swallowing click/navigation failures as "card absent"; only true absence skips. - UiSessionExtension: don't save a trace for TestAbortedException (a skipped assumption is not a failure). Robustness / cleanup: - GoogleSsoBootstrapUIIT: build expected authority from MockOidcServer.NETWORK_ALIAS/PORT instead of a hardcoded :1080. - EntityLoaderSmokeUIIT: log load duration instead of asserting a wall-clock bound (flaky on shared runners). - ReindexHelpers.stopAppAndWait: drop unused stopRequestedAt. - nightly workflow: dedupe apt package list. - Javadoc fixes (UiSessionExtension AuthStrategy ref, IncidentManager seed count 18 -> 20). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Mohit Yadav
|
23ccb2dcff
|
Merge branch 'main' into java-playwrights | ||
|
IceS2
|
14f880636a
|
ci(airflow-apis-tests): migrate Sonar step to sonarqube-scan-action@v7 with retry + add workflow_dispatch (#28292)
* ci(airflow-apis-tests): retry Sonar PR scan on JRE-provisioning flake Mirror the py-tests pattern: migrate from the deprecated sonarsource/sonarcloud-github-action@master to SonarSource/sonarqube-scan-action@v7, mark the PR scan continue-on-error, and add a sleep+retry step so a transient 'Failed to query JRE metadata' from Sonar's JRE-provisioning endpoint no longer fails the job on first attempt. Hoist the shared sonar args into a workflow-level SONAR_OPTS env. * ci(airflow-apis-tests): allow workflow_dispatch + run Sonar step on it Add workflow_dispatch trigger so the Sonar retry path can be exercised from the Actions UI without opening a PR, and extend the Sonar PR step (plus its wait+retry siblings) to run on the dispatch event. * ci(airflow-apis-tests): scope Sonar steps to pull_request_target only Drop workflow_dispatch from the Sonar PR/retry step conditions so manual runs don't fire the scanner with empty -Dsonar.pullrequest.* flags (would create a branch entry in SonarCloud, per gitar-bot review). Dispatch trigger stays for re-running the build/test surface; Sonar will only fire on a real PR where the pull-request context exists. |
||
|
mohitdeuex
|
e1d6734acb | Update webhook | ||
|
mohitdeuex
|
3895cd1f70 |
Speed up nightly Playwright workflow + fix flaky reindex assertions
Workflow:
- Split into a build-image job that bakes openmetadata-server:jpw-snapshot
via docker/build-push-action with a shared GHA layer cache, then exports
the loaded image as a workflow artifact. Both matrix entries
(elasticsearch + opensearch) download + `docker load` the same image
and set OM_TEST_IMAGE so ContainerizedServer skips its own `docker build`.
Result: one image build per workflow run (was 2x duplicated dist build
+ per-launch in-test builds) and stable cross-matrix correctness — the
binary both shards run against comes from the exact same source SHA.
- workflow_dispatch input `includeSsoBootstrap` toggles whether the
@Tag("sso-bootstrap") SSO bootstrap UIITs run; default off because they
spin up their own ContainerizedServer (second OM lifecycle) for an env
wiring check that doesn't change between most runs.
- Slack notification migrated to slackapi/slack-github-action@v2 with the
incoming-webhook payload shape it now requires, and guarded behind
`env.SLACK_WEBHOOK_URL != ''` so a missing secret no-ops instead of
failing the post-step.
- Publish Test Report step set fail_on_test_failures=false — mvn verify
already gates the job conclusion, and a flake in the report action
shouldn't cascade into the Slack step.
Test fixes:
- SearchAvailableDuringReindexUIIT: baseline probe now asserts
`>= seeded.countOf(TABLE)` instead of strict equality. The OM container
is shared across the suite so the index can legitimately have residual
entities from earlier tests; assertEventualConsistency already checks
that none of *our* baseline entities go missing across the recreate.
- SimpleReindexTriggerUIIT: assertExploreCount now polls via Awaitility
with a 2-minute budget, re-opening the Explore page on every tick.
Playwright's `hasText` polled only the DOM, which wedges against a
stale aggregation cache; re-issuing the search aggregation on each
retry lets ES catch up after the alias swap.
Tagging:
- @Tag("sso-bootstrap") on GoogleSsoBootstrapUIIT + MockIdpSmokeUIIT, and
the `ui-it` profile now reads `ui.it.excludedGroups` (default
`sso-bootstrap`) so default `mvn verify -P ui-it` skips them. Pass
`-Dui.it.excludedGroups=` to include them.
|
||
|
mohitdeuex
|
db481fdeac |
Address remaining PR review comments
Bugs
- IndexFieldExplosionIT: SCHEMA_ALIAS was `databaseSchema_search_index`; the
canonical indexMapping.json name is `database_schema_search_index`.
- ExplorePage:
- `tabTestId(GLOSSARY_TERMS)` produced `glossaries-tab`, but the UI builds
the testid from the i18n label (`Glossary Terms` → `glossary terms-tab`).
- `Tab.DASHBOARD_DATA_MODELS` path was `dashboardDataModels`; the Explore
route segment is singular (`dashboardDataModel`).
- Javadoc {@link} now points at the correct `openWithSearch` overload.
- UiSessionExtension: split video lifecycle so the `Video` handles are
snapshotted before `context.close()` (pages() is empty after close) but
`video.path()` is resolved AFTER close (Playwright finalises the file on
close — calling .path() earlier blocks/fails).
- GoogleSsoSignInUIIT: removed the empty alternative from the
`(my-data|explore|)` regex; it matched almost any post-auth URL and
weakened the assertion.
- MockOidcServer: still requires a single fixed port (token `iss` claim has
to match across container/host/browser), but the port is now overridable
via `-Dom.mockOidc.port=NNNN` and a fast pre-flight `ServerSocket` probe
fails clearly when the chosen port is busy. GoogleSsoSignInUIIT now reads
the port from `MockOidcServer.PORT` instead of hard-coding 1080.
Test hygiene
- SearchAvailableDuringReindexUIIT: replaced `Thread.sleep` polling with
Awaitility (`.atMost(REINDEX_TIMEOUT).pollInterval(PROBE_INTERVAL)`),
giving the loop a real deadline and removing the antipattern.
- ClipboardHelper: replaced the fixed `waitForTimeout(300)` with bounded
paste-retries until the hidden textarea has a non-empty value; textarea
cleanup moved to a `finally` block.
- SimpleReindexTriggerUIIT / SearchAvailableDuringReindexUIIT: defaults are
now PR-friendly (200/100/100/100 tables/topics/dashboards/pipelines and
500 tables respectively) overridable via system properties; the nightly
workflow sets the historical 5k stress numbers.
Quality
- DistributedAutoTuneReindexUIIT.distributedAutoTuneConfig now returns
`Map.of(...)` instead of a mutable `HashMap`.
- SearchQueryHelper.SearchProbe defensively copies `ids` / `uniqueIds` to
immutable collections in the canonical constructor.
- EntityLoader: every parameter and local that doesn't change is now
`final`.
- AuthAssumptions: `toLowerCase` calls now pin `Locale.ROOT` to stay stable
under Turkish / other surprising locales.
Docs
- PageObject javadoc: list of rules updated to reflect actual contract
(Page Objects may expose `Locator`-returning accessors, `rawPage()` is a
documented escape hatch).
- UI_TEST_CONVENTIONS.md: layering diagram now lists the real packages
(`playwright.scenarios`, `playwright.ui.pages`, `it.auth`, `it.server`).
Rule about Locator/Page softened to match the real contract. Headed-debug
recipe points at `:openmetadata-integration-tests` (the
`:openmetadata-java-playwright` module was removed). Stale references to
MIGRATION_TRACKING.md and SearchAfterReindexUIIT replaced with
REINDEX_TEST_PLAN.md and SimpleReindexTriggerUIIT.
- REINDEX_TEST_PLAN.md: helpers table now flagged as a planning shape with
an explicit list of what's shipped today vs. what's still aspirational.
|
||
|
mohitdeuex
|
c6682464df | Remove playwright-or | ||
|
mohitdeuex
|
b015277df3 |
Address PR review comments: antlr CLI, URL encoding, catch-block split
- Install antlr4 CLI + native build deps in java-playwright PR and nightly workflows (yarn install of openmetadata-ui runs the .g4 → JS codegen, which fails with `antlr4: not found` otherwise). - SearchClient: split combined IOException|InterruptedException catch so only InterruptedException re-sets the interrupt flag; an IOException shouldn't make unrelated higher-level code think the thread was interrupted. - SearchQueryHelper.probeIndex: URL-encode `query` and `indexAlias` before splicing into the query string. - OidcBackend.acquireToken: URL-encode DEFAULT_USER (contains `@`) and DEFAULT_PASSWORD in the password-grant form body. - openmetadata-integration-tests/pom.xml: mark Playwright dependency as test-scoped. |
||
|
mohitdeuex
|
0f766ac2dc |
Fix Java Playwright CI: build local image before runMigrations
Two coupled fixes for the ContainerFetchException seen in https://github.com/open-metadata/OpenMetadata/actions/runs/26087848414: 1. ContainerizedServer.launch() now materialises the openmetadata-server image at the very top via a new ensureServerImageAvailable() helper. Previously runMigrations() ran first and tried to start a container using the jpw-snapshot tag — testcontainers then attempted a registry pull, fell over with ContainerFetchException, and the whole run failed before newServer()/buildLocalImageContainer() had a chance to build anything. The image build is now done once, before any container needs the tag. Honors OM_TEST_IMAGE override (skips local build). 2. Nightly workflow gets an explicit "Build openmetadata-dist tarball" step. The previous `mvn install -pl :openmetadata-integration-tests -am` doesn't transitively build openmetadata-dist (not a test dependency), so openmetadata-*.tar.gz was never produced — meaning ensureServerImageAvailable() would still fail in CI at locateDistTarball(). Added before the test run, after deps build. |
||
|
Mohit Yadav
|
fb954a9141
|
ci: add Java Playwright UIIT workflow (dispatch-only) (#28251)
Lands java-playwright-nightly.yml on main so the workflow becomes dispatchable. workflow_dispatch only registers when the workflow file exists on the default branch; once merged, the suite can be run on demand against any branch ref. Tracks EPIC #3731. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
mohitdeuex
|
ddfc275b6c |
ci: disable nightly UIIT cron, keep workflow_dispatch only
Run the UI integration suite on demand while it stabilises; re-add the schedule trigger once it is green on main. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
mohitdeuex
|
609269db07 | Workflow changes | ||
|
Mohit Yadav
|
6463edbb5a
|
Merge branch 'main' into java-playwrights | ||
|
Harsh Vador
|
286a26f81f
|
ci(security-scan): post Snyk summary to Slack + fail on high/critical (#28200)
Some checks are pending
Integration Tests - MySQL + Elasticsearch / integration-tests-mysql-elasticsearch (push) Blocked by required conditions
Integration Tests - MySQL + Elasticsearch / Detect Changes (push) Waiting to run
Integration Tests - PostgreSQL + Elasticsearch + Redis / Detect Changes (push) Waiting to run
Integration Tests - PostgreSQL + Elasticsearch + Redis / integration-tests-postgres-elasticsearch-redis (push) Blocked by required conditions
Integration Tests - PostgreSQL + OpenSearch / Detect Changes (push) Waiting to run
Integration Tests - PostgreSQL + OpenSearch / integration-tests-postgres-opensearch (push) Blocked by required conditions
Java Checkstyle / java-checkstyle (push) Waiting to run
Maven Collate Tests / maven-collate-ci (push) Waiting to run
OpenMetadata Service Unit Tests / Detect Changes (push) Waiting to run
OpenMetadata Service Unit Tests / openmetadata-service-unit-tests (push) Blocked by required conditions
OpenMetadata Service Unit Tests / k8s_operator-unit-tests (push) Blocked by required conditions
OpenMetadata Service Unit Tests / openmetadata-service-unit-tests-status (push) Blocked by required conditions
Publish Package to Maven Central Repository / publish-maven-packages (push) Waiting to run
* ci(security-scan): post Snyk summary to Slack + fail on high/critical * fix slack post channel * mention repo name * address gitar |
||
|
Harsh Vador
|
d5bc00d1da
|
ci(security-scan): readable Snyk job summary + consolidated Slack alert (#28170)
* generate snyk summary * address gitar * address gitar * generate summary * remove duplicate notification |
||
|
Sriharsha Chintalapani
|
5696286b27
|
Address Transitive vulnerabilities (#28169)
* Address transitive vulnerabilities * Address transitive vulnerabilities * fix(deps): resolve pyOpenSSL/cryptography conflict and align constraint pins CI dependency resolution failed because pyOpenSSL~=24.1.0 caps cryptography at <43, conflicting with the cryptography>=44.0.1 bump. Widens pyOpenSSL to >=24.3.0 (first version compatible with cryptography 44.x) and aligns the airflow constraint file pins for cryptography and GitPython with the upstream setup.py bumps so pip install -c can resolve. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Harsh Vador
|
bb5c64658e
|
ci: consolidate security scan Slack notifications into single combined alert (#28135)
Some checks are pending
Integration Tests - MySQL + Elasticsearch / Detect Changes (push) Waiting to run
Integration Tests - MySQL + Elasticsearch / integration-tests-mysql-elasticsearch (push) Blocked by required conditions
Integration Tests - PostgreSQL + Elasticsearch + Redis / Detect Changes (push) Waiting to run
Integration Tests - PostgreSQL + Elasticsearch + Redis / integration-tests-postgres-elasticsearch-redis (push) Blocked by required conditions
Integration Tests - PostgreSQL + OpenSearch / Detect Changes (push) Waiting to run
Integration Tests - PostgreSQL + OpenSearch / integration-tests-postgres-opensearch (push) Blocked by required conditions
Java Checkstyle / java-checkstyle (push) Waiting to run
Maven Collate Tests / maven-collate-ci (push) Waiting to run
OpenMetadata Service Unit Tests / Detect Changes (push) Waiting to run
OpenMetadata Service Unit Tests / openmetadata-service-unit-tests (push) Blocked by required conditions
OpenMetadata Service Unit Tests / k8s_operator-unit-tests (push) Blocked by required conditions
OpenMetadata Service Unit Tests / openmetadata-service-unit-tests-status (push) Blocked by required conditions
Publish Package to Maven Central Repository / publish-maven-packages (push) Waiting to run
* ci: consolidate security scan Slack notifications into single combined alert * address gitar * add env |
||
|
Sriharsha Chintalapani
|
64f49c1747
|
Cache improvements: lineage + search layers, observability, CI gate (#28012)
* cache: lineage cache, per-type metrics, invalidation registry, search-cache Add Redis-backed lineage response cache and search response cache, both gated by the existing CACHE_PROVIDER toggle and falling through to direct computation when the cache is unavailable. The cache remains optional — verified end-to-end by toggling CACHE_PROVIDER=none on a live stack and confirming all paths continue to work (just without the L2 hit). Coverage: - CachedLineage wraps LineageRepository.getLineage with hybrid TTL + direct invalidation (60s default). Direct edits invalidate the affected root cache entries; transitive changes fall through to TTL. - CachedSearchLayer wraps /api/v1/search/query with auth-aware caching (cache key includes principal so users with different ACLs don't share results). 30s default TTL. Observability: - /api/v1/system/cache/stats response now includes a metrics block with hits/misses/hitRatio/evictions/errors/writes plus read/write latency Timers, and a byType breakdown so coverage gaps are visible per entity-type and per cache-layer. Correctness: - New Invalidatable interface + CacheBundle registry + invalidateEntity helper so future cache layers plug in by implementing one method instead of editing multiple mutation paths. - Edge mutations in LineageRepository.addLineage/deleteLineage invalidate both endpoints; entity mutations in EntityRepository.postUpdate / postDelete / restoreEntity invalidate the lineage rooted at the entity. - Pub/sub handler in CacheBundle iterates registered Invalidatables so remote-pod evictions flow to all layers automatically. Tooling: - docker-compose.cache-off.yml overlay flips CACHE_PROVIDER=none for local A/B testing without tearing down DB/ES volumes. - CachedSearchLayerIT exercises hit-on-second-call, distinct-query misses, distinct-page-size misses, and byType shape via the metrics endpoint. Each test gracefully no-ops when the cluster runs cache-off. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * cache: phase 2 ops + correctness — single-flight, slow-read, negative cache, admin endpoints Builds on the phase 1 commit ( |
||
|
Harshit Shah
|
77a85bffde
|
[CI] Add on-demand Playwright search-nightly workflow (#27908)
* test(ci): add on-demand playwright search-nightly workflow Create a manual Playwright search-nightly workflow with the same bootstrap, reporting, Slack notification, and cleanup structure as the SSO nightly job. Add a dedicated search-nightly Playwright project and a basic nightly search smoke spec without using issue-closing keywords for #3792. * address comments * revert changes * minor updates |
||
|
Mohit Yadav
|
8ac53bfecc
|
Merge branch 'main' into java-playwrights | ||
|
mohitdeuex
|
74bda64340 |
Merge openmetadata-java-playwright into openmetadata-integration-tests
Folds the UI integration test module into the canonical integration-tests
module under a `ui-it` Maven profile. One test home, one classpath, no
more test-jar reinstall dance or cross-module IntelliJ classpath quirks.
Why: most of the value the UI test module shipped was reusable backend
infra (factories, search helpers, server harness) that worked fine
without a browser. Keeping it in a separate module forced multiple
unnecessary boundaries — test-jar publication, IntelliJ test-classes-
jar-tests phantom paths, src/test placement for AuthBackend code that
should have been in src/main, "where does this test go?" friction.
Layout in integration-tests:
org/openmetadata/it/auth/ JwtAuthProvider + AuthBackend / OidcBackend
/ AuthSession / TokenRefresher / ...
org/openmetadata/it/server/ ContainerizedServer / ServerHandle /
ExternalServer / sso/ profile records
org/openmetadata/it/search/ ReindexHelpers / SearchClient /
SearchAssertions / SearchQueryHelper
org/openmetadata/it/ui/ SessionBrowser / UiSession /
UiSessionExtension / TraceRecorder /
ClipboardHelper / pages/
org/openmetadata/it/scenarios/ *UIIT.java tests
org/openmetadata/it/util/ SdkClients + UiTestServer / OssTestServer
org/openmetadata/it/factories/ existing + EntityLoader
Build:
- integration-tests pom gains com.microsoft.playwright:playwright
(test scope). Other testcontainers / jwt deps already there.
- test-jar publish-test-harness includes pattern expanded to ship
server/, search/, ui/ packages alongside auth/, util/, factories/,
bootstrap/. Downstream consumers (collate) inherit the full UI
test harness, not just backend factories.
- New `ui-it` profile runs `**/*UIIT.java` with skip.embedded.bootstrap
=true, PW_VIDEO=true, per-method parallel @ 0.5 factor. Mirrors the
failsafe execution from the old playwright module.
- Existing parallel-tests executions across all profiles gain a
`**/*UIIT.java` exclude so embedded-mode IT runs don't pick up UI
tests they can't run.
Module removal:
- openmetadata-java-playwright/ deleted.
- parent pom <modules> entry removed.
- .github/workflows/java-playwright-nightly.yml updated to build and
test `openmetadata-integration-tests -P ui-it` instead.
Docs:
- MIGRATION_TRACKING.md and CONVENTIONS.md from the old module are
UI_MIGRATION_TRACKING.md / UI_TEST_CONVENTIONS.md at the
integration-tests root.
No test code semantics changed — pure reorganization. The 4-5 backend-
flavored *UIIT.java tests we identified as misplaced (running against
SDK with vestigial UI checks) still live under scenarios/ for now; a
follow-up will rename them to *IT.java and have them target the
embedded TestSuiteBootstrap directly to drop their ~3-minute Docker boot
overhead.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Sriharsha Chintalapani
|
d3bbbefe37
|
fix(rdf): dedupe lineage edges, surface Fuseki failures, port distributed-mode improvements (#27999)
* fix(rdf): dedupe lineage edges and broaden PROV-O coverage
The RDF Knowledge Graph endpoint was emitting two edges per lineage
relationship — once as `om:UPSTREAM` (forward) and once as
`prov:wasDerivedFrom` (reverse) — because the parser preserved each
predicate's native subject/object orientation instead of canonicalizing
both into a single `(upstream, downstream)` edge.
Also extend PROV-O coverage so external SPARQL clients can use the W3C
Provenance vocabulary directly:
- `prov:Entity` / `prov:Activity` / `prov:Agent` class typing on
datasets / pipelines / users
- `prov:wasAttributedTo` mirror of `om:owners`
- `prov:generated` (inverse of existing `wasGeneratedBy`) and `prov:used`
on lineageDetails so the Entity → Activity → Entity chain is complete
- `prov:hadPlan` + `prov:Plan` for SQL transformation recipes
- `prov:startedAtTime` / `prov:endedAtTime` on Activity instances
- `prov:wasAssociatedWith` Activity → Agent linking
- `prov:invalidatedAtTime` on soft-deleted entities
Other RDF cleanups in the same area:
- LineageDetails URIs are now deterministic (driven by from/to ids
instead of a timestamp), so re-indexing collapses duplicate Activity
resources via the existing DELETE+INSERT idempotency
- Skip emitting the redundant `om:owners` JSON-string literal — the
mapped path already produces clean `om:hasOwner <agent>` triples
- Skip empty `[]` array literals in the unmapped path
- Propagate failures from `RdfRepository.{addRelationship,
addLineageWithDetails, bulkAddRelationships,
bulkAddGlossaryTermRelations}` instead of silently swallowing them,
so downstream callers can surface the failure
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* fix(rdf-index-app): surface Fuseki failures in app run record
Per-entity and per-batch failures from the RDF index app used to be
logged via SLF4J only — they never made it into the AppRunRecord, so
the UI/run history showed "completed" even when every entity had
silently failed to write to Fuseki.
- `RdfBatchProcessor.processEntities` now captures the last error per
entity, returns it in `BatchProcessingResult.lastError`, and
accumulates relationship-processing failures into the same result.
- Relationship and lineage processing methods (`processBatchRelationships`,
`processLineageRelationship`, `processGlossaryTermRelations`) return
structured results with failure counts and last-error messages instead
of `void`, so failures are visible to the partition worker.
- `RdfIndexApp` records the failure on `jobData` for both the
distributed and non-distributed code paths, so users see a real
error message in the run history (e.g.
"Failed to write entity X to Fuseki: ConnectException").
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* perf(rdf-index-app): port distributed-mode improvements from SearchIndex
The RDF distributed-indexing fork was lagging behind several SearchIndex
improvements that addressed concrete reliability and throughput issues.
Port them across:
Core perf / reliability
- Precomputed partition start cursors: coordinator walks each entity
once via keyset pagination at job init and caches the boundary cursor
per (jobId, entityType, rangeStart). Workers consult the cache before
falling back to the OFFSET-based path. Eliminates the previous O(N²)
per-partition cursor lookup.
- `cancelInFlightPartitions` + `requestStop` + `checkAndUpdateJobCompletion`
on the coordinator. Stop now cancels both PENDING and PROCESSING
partitions in a single SQL update and immediately drives the job
status from STOPPING → STOPPED, so the UI status no longer hangs
while workers drain.
- Selective field hydration: `RdfPartitionWorker.readEntitiesKeyset`
uses `ReindexingUtil.getSearchIndexFields(entityType)` instead of
`List.of("*")`, avoiding expensive fetchers (e.g. fetchAndSetOwns)
per batch.
- Partition heartbeat thread: virtual thread refreshes
`lastUpdateAt` every 30s for partitions actively being processed by
this server, so the stale reclaimer no longer interrupts active work.
- `MAX_IN_FLIGHT_PARTITIONS_PER_SERVER = 5` backpressure: claim path
rejects when the server already holds 5 PROCESSING partitions, giving
fair distribution across pods. Verified the existing claim DAO uses
`FOR UPDATE SKIP LOCKED` for both MySQL and Postgres.
- Gate WebSocket stat broadcasts during the STOPPING phase so the
Quartz-scheduler-driven STOPPED status push isn't overwritten.
Multi-server scaffolding (single-pod is unaffected)
- `RdfPollingJobNotifier`: DB-polling discovery for other server pods
to find an in-flight RDF reindex they can join.
- `RdfEntityCompletionTracker`: per-entity-type partition tracking with
callback firing once all partitions for an entity complete, foundation
for early per-entity index promotion.
Tests: precomputed-cursor cache lookup, in-flight backpressure,
cancelInFlight delegation, completion tracker callback semantics,
notifier start/stop.
DAO additions on `rdf_index_partition`:
- `cancelInFlightPartitions(jobId, now)` — covers both PENDING and
PROCESSING in one statement
- `countInFlightPartitionsForServer(jobId, serverId)` — backpressure
- `countPartitionsByStatus(jobId, status)` — used by completion check
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* fix(ui-apps): hide misleading data on synthetic 'CurrentConfig' row
When an app has no run history, AppRunsHistory fabricated a synthetic
placeholder row that looked like a real run — `runType: "CurrentConfig"`,
a fake `Run At` timestamp pulled from `appData.updatedAt`, an
ever-growing `Duration` (`now − updatedAt`), and an active `Stop` button
that targeted nothing.
Render `--` for `Run At`, `Run Type`, and `Duration` on synthetic rows,
and hide the `Stop` button so users no longer see "Run now → 19-minute
Running with Stop button" when the actual job never registered. Real
app runs are unaffected — they still display `runType` from the
backend (OnDemandJob, Hourly, Daily, Custom, etc.).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* fix(rdf): address PR review findings
Four issues raised in PR #27999 review:
- **Cursor format consistency in walkAndRecord** (bug):
The defensive branch produced cursors via a custom `{name, id}` map
while the regular path used `repo.getCursorValue()`. For entities
with quoted names these encodings diverge — a quoted-name entity
could land in the cache with a cursor incompatible with what the
worker fetches via keyset pagination. Track the last seen entity
reference and run it through `repo.getCursorValue()` in both paths.
`encodeBoundaryCursor` is removed.
- **Adaptive scheduling in RdfPollingJobNotifier** (perf):
The previous implementation woke the scheduler thread every 1s and
short-circuited inside the poll method when idle. Reschedule the
task at the appropriate interval (1s active / 30s idle) when
`setParticipating` flips, so the thread genuinely sleeps when idle.
- **Cursor cache cleanup on startup recovery** (edge case):
`partitionStartCursors` was only evicted by `refreshAggregatedJob`
/ `checkAndUpdateJobCompletion`. If a coordinator crashed mid-job
and never reached either, the cache entry leaked until process
restart. Add `evictStaleCursorCacheEntries()` invoked by
`performStartupRecovery` that drops entries for jobs that no longer
exist in the DB or are already terminal.
- **Consolidate describeError helpers** (quality):
`describeError`, `describeBulkError`, and `describeLineageError` in
`RdfBatchProcessor` all walked the cause chain and formatted a
prefixed message with the same logic. Reduced to a single
`describeError(prefix, error)` plus a thin `describeEntityError`
adapter for the per-entity call site.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* fix(rdf-index-app): avoid double workerExecutor.shutdownNow() in stop()
stop() called workerExecutor.shutdownNow() inline AND through
cleanupLocalExecution -> shutdownWorkerExecutor, which broke the
DistributedRdfIndexExecutorTest.stopAndCoordinatorCleanupOnlyTearDownLocalExecutionOnce
verify(workerExecutor, times(1)).shutdownNow() expectation. Drop the
inline call — cleanupLocalExecution is the single owner of the
shutdown path.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* ci: drop redundant DB matrix from openmetadata-service unit tests
The {mysql, postgresql} strategy matrix on openmetadata-service unit
tests doubled CI cost without adding signal: both jobs ran the same
surefire suite. The `-Pmysql` / `-Ppostgresql` profiles are defined
only in `openmetadata-sdk/pom.xml` (lines 190-206), set a single
`test.database` property, and that property is consumed exclusively by
the failsafe plugin (integration tests `*IT.java` / `*IntegrationTest.java`),
which only runs under `-Pintegration-tests` — not enabled here.
`openmetadata-service` itself has zero tests that read `test.database`
or use `MySQLContainer`/`PostgreSQLContainer` (verified by grep). The
only testcontainer-based DB code in the repo lives in
`openmetadata-integration-tests`, a different module that this workflow
doesn't build.
Run the unit suite once. The `openmetadata-service-unit-tests-status`
required-check aggregator is unaffected (it depends on the renamed job
which still has the same name).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* fix(rdf): address Copilot PR review findings
Six correctness issues raised on PR #27999:
- **Lineage-details DELETE was too broad** (RdfRepository): the cleanup
step deleted *all* `<fromUri> om:hasLineageDetails ?d` triples,
so reindexing one (fromId, toId) edge wiped lineage-details links
for every other downstream of the same source entity. Pin the
delete to the specific `<fromUri> om:hasLineageDetails <detailsUri>`
triple. Same with prov:generated cleanup — anchor it to the
specific detailsUri instead of any details resource.
- **Predicate not flipped during canonicalization** (RdfRepository):
`parseEntityGraphEdgesFromResults` swapped subject/object for
reverse-direction predicates (`prov:wasDerivedFrom`,
`prov:wasInfluencedBy`) but kept the original predicate URI on the
resulting EdgeInfo. Exported graphs could carry semantically
invalid triples like `<upstream> prov:wasDerivedFrom <downstream>`.
Add `forwardEquivalentPredicate` to substitute the OM-native
forward predicate when the direction flips.
- **`dct:modified` was an invalid xsd:dateTime** (RdfPropertyMapper):
`entity.getUpdatedAt().toString()` returns the epoch-millis Long as
a string, but the literal was tagged `xsd:dateTime`. Convert via
`Instant.ofEpochMilli(...).toString()` so the lexical form matches
the type — same fix already in place for prov:invalidatedAtTime.
- **Unmapped EntityReference arrays were dropped entirely**
(RdfPropertyMapper): the previous fix to skip noisy JSON-string
literals also dropped fields like `domains`, `reviewers`, `voters`
for entity contexts that don't have a JSON-LD mapping for them —
the unmapped path was the only path emitting them, so nothing
landed in RDF. Expand each array element through
`addEntityReference` so the data still produces proper
`om:<fieldName> <ref>` triples; mapped-path duplicates are
collapsed by Jena's Model dedupe.
- **Partition failure detection missed reader errors**
(DistributedRdfIndexExecutor): the EntityCompletionTracker was fed
`result.errorMessage() != null`, but `RdfPartitionWorker` can
increment `failedCount` from `readerErrors` without ever setting
`lastError`. Use `result.failedCount() > 0` so partitions whose
failures came from `ResultList.getErrors()` are also marked as
failed when promoting an entity.
- **`COMPLETED_WITH_ERRORS` was hidden when failedRecords == 0**
(RdfIndexApp): the coordinator marks a job COMPLETED_WITH_ERRORS
whenever any partition is FAILED or CANCELLED, including for
user-initiated stops where no record-level failures accrued. The
monitor's `completedWithErrors` gate required `failedRecords > 0`,
so those terminal states never hit `jobData.setFailure(...)` and
the run record showed success. Drop the failedRecords precondition
and tailor the fallback message based on whether there are
record-level failures or partition-level only.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* fix(rdf): separate relationship failures + type lineage as prov:Activity
Two more PR review findings on #27999:
- **Relationship failures inflated failedRecords stat**: `processEntities`
was folding relationship/lineage edge failures into `failedCount`,
which becomes `failedRecords` in the index stats. Records there mean
entities, computed from entity counts in `totalRecords`. Counting
per-edge relationship failures could push `failedRecords` above
`processedRecords`/`totalRecords` and produce nonsensical
per-entity stats.
Track them separately: add `relationshipFailureCount` to
`BatchProcessingResult` and `PartitionResult`. `failedCount` now stays
entity-level. The completion tracker is fed the broader
`result.hasAnyFailure()` so partitions where relationship triples
failed don't get prematurely promoted as success even though their
entity writes succeeded.
- **`detailsResource` wasn't typed as prov:Activity**: the resource
carries Activity-shaped predicates (prov:startedAtTime,
prov:endedAtTime, prov:used, prov:hadPlan, prov:wasGeneratedBy,
prov:wasAssociatedWith) but only the OM-specific
`om:LineageDetails` rdf:type. Add an explicit
`rdf:type prov:Activity` so PROV-O reasoners and federated SPARQL
clients recognize it as an Activity without having to learn the
OM type.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* fix(rdf): label lineage edges relative to focal node
The Knowledge Graph view was labeling every edge with relation
type "upstream" as "Upstream" regardless of direction relative to the
focal node. For a focal node F, the raw stored relation `(F, X, upstream)`
means "F is upstream of X" — i.e. X is *downstream* of F. The previous
output labeled both `F → X` and `X → F` edges as "Upstream", which made
bidirectional lineage look like a duplicated relation.
Re-orient the label in `convertEdgesToGraphData` based on whether the
focal is the edge's source or target:
- focal → X → "Downstream"
- X → focal → "Upstream"
- non-focal-touching edges keep the raw relation label.
Reported on a sample-data table with a circular lineage cycle
(`dim_customer ↔ fact_orders`) where both directions showed "Upstream".
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* fix(rdf): close remaining Copilot review gaps
Three findings from PR #27999's third review pass — all about failure
signals being silently dropped between layers:
- **`RdfIndexApp.processTask` ignored relationship failures**: only
`result.failedCount() > 0` was treated as a failure, so partitions
whose Fuseki relationship/lineage writes failed (incrementing
`relationshipFailureCount` but not `failedCount`) never wrote
`jobData.failure`. Switch to `result.hasAnyFailure()` and report the
combined count.
- **`checkAndUpdateJobCompletion` ignored partition `lastError`**: a
partition can finish COMPLETED with `lastError` set when a relationship
bulk write was caught and recorded but didn't bump `failedRecords` or
flip the partition to FAILED. The job would then go to COMPLETED even
though there were real failures. Treat the presence of any
`rdf_index_partition.lastError` as an error signal — promote to
COMPLETED_WITH_ERRORS and aggregate sample errors into the job's
errorMessage if it was blank.
- **`forwardEquivalentPredicate` mapped to a non-existent
`om:DOWNSTREAM` URI**: OpenMetadata only stores lineage with
`om:UPSTREAM` (forward) and `prov:wasDerivedFrom` (reverse PROV-O
pair); there is no `om:DOWNSTREAM` predicate written anywhere — the
downstream view is derived by reading the same UPSTREAM edge from the
other side. Map both `prov:wasDerivedFrom` and `prov:wasInfluencedBy`
to `om:UPSTREAM` (both are reverse-direction causation predicates: in
`B wasDerivedFrom A` / `B wasInfluencedBy A` the source is A and
effect is B, so the canonical forward predicate is the same).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* Fix RDF tag mapper
* Fix all the comments
Cherry-picked from #27562 (without bin/ autogenerated noise).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* Align RdfPropertyMapper tests with refactor and isolate ontology export IT
RdfPropertyMapperTest still referenced the removed addVotes helper and
expected addStructuredProperty to dispatch votes — both gone after votes
was added to IGNORED_PROPERTIES. Update the assertions accordingly.
GlossaryOntologyExportIT timed out on the full suite because it flips a
global RDF singleton in @BeforeAll and each test blocks a server thread on
synchronous Fuseki writes. SAME_THREAD only serialized methods within the
class — concurrent classes still raced for server threads. Adding @Isolated
matches the pattern already used by RdfResourceIT for the same reason.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(rdf): align addCertification typing + relationType after predicate flip
Two findings on PR #27999 from the post-cherry-pick review pass:
- **`addCertification` mis-typed glossary-source certifications and
skipped skos:Concept**: it always emitted `om:Tag` regardless of
source, even though `resolveTagResource` returns a glossaryTerm URI
when the certification points at a glossary term. It also didn't add
`skos:Concept` (or the `createTypeResource("tag")` `skos:Concept` for
classification tags), so SPARQL queries filtering certification
targets by `a skos:Concept` missed them while `addTagLabel`-emitted
tags were findable. Mirror `addTagLabel`: branch on source
(`Glossary` vs `Classification`), emit the right primary type plus
`skos:Concept` (glossary) or `om:Tag` (classification), and include
`om:tagSource`.
- **`relationType` left stale after predicate flip**: when
`parseEntityGraphEdgesFromResults` flipped subject/object for a
reverse-direction predicate and rewrote `canonicalPredicate` to
`om:UPSTREAM`, it kept the original `relationType` derived from the
reverse predicate. So `prov:wasInfluencedBy` produced an EdgeInfo
with `relationType=downstream` + `predicate=om:UPSTREAM` —
internally inconsistent, and the mismatched `edgeKey` prevented
dedup against an existing UPSTREAM edge with the same endpoints.
Re-derive `relationType` from the canonical predicate after the
flip.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* fix(rdf): close 2 review findings + add parser-helper unit tests
Two outstanding Copilot findings on PR #27999 plus targeted unit
coverage for the helpers that drive lineage canonicalization.
Findings:
- **`colLineageUri` collision risk** (RdfRepository): the deterministic
key replaced non-alphanumerics in `toColumn` with `_`, so distinct
column names (e.g. `a-b` vs `a_b`) collapsed onto the same URI, which
would lose / overwrite column-lineage resources during reindex.
Append the loop index as a tiebreaker so distinct columns keep
distinct URIs.
- **`createTypeResource` missing dprod prefix** (RdfPropertyMapper):
the `getNamespace` switch didn't recognize `dprod`, so
`RdfUtils.getRdfType("dataProduct")` (returns `dprod:DataProduct`)
produced an invalid `dprod:DataProduct` URI on the wire. Added the
`DPROD_NS = https://ekgf.github.io/dprod/` constant and a `dprod`
case in the switch.
Coverage:
- New `RdfParserHelpersTest` exercises the canonicalization helpers
via reflection: `isReverseDirectionPredicate` (recognizes
PROV-O causation predicates, ignores forward predicates),
`forwardEquivalentPredicate` (both `wasDerivedFrom` and
`wasInfluencedBy` collapse to `om:UPSTREAM` so dedup works),
`relativeRelationLabel` (focal-relative Upstream/Downstream
flipping with all the boundary cases — non-focal edges,
non-lineage relations, null focal).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* fix(rdf): merge array contexts before per-field resolution
The third (low-confidence "suppressed") finding on review 4256830399
turned out to be a real duplication: when a field is mapped in one
context map of an array context but absent from another, the previous
processArrayContext ran processContextMappings once per map. The pass
where the field IS mapped emits the proper `om:hasOwner <ref>` triples
(plus `prov:wasAttributedTo`); the pass where the field is absent
falls through to processUnmappedField and emits an additional
`om:owners <ref>` triple. Net: two predicates for the same logical
relationship.
Verified on the live Fuseki: 113 `om:hasOwner` triples vs 112
`om:owners` triples — one set per pass.
Fix: flatten all context maps in the array into a single merged map
once, then iterate entity fields exactly once against that combined
view (later contexts win on key conflicts, matching JSON-LD context
merge semantics). Each field is resolved against the union of
mappings, so the unmapped fallback only fires for fields truly absent
from every context. Net effect: `prov:wasAttributedTo` count is
unchanged, `om:hasOwner` is unchanged, and the redundant `om:owners`
triples disappear.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* fix(rdf): close 2 review findings on coordinator finalization race
Two findings from PR #27999 review 4259628860:
- **`checkAndUpdateJobCompletion` early-returned before lastError check
could promote**: `refreshAggregatedJob` already marks the job COMPLETED
when partitions all finish without `failedRecords`/`failedPartitions`,
so `checkAndUpdateJobCompletion`'s subsequent `if (job.isTerminal())`
short-circuit silently dropped the lastError signal. Move the
partition-lastError check INTO `refreshAggregatedJob` so both code
paths produce consistent terminal status — a partition that finished
COMPLETED but carries a non-null lastError now correctly promotes the
job to COMPLETED_WITH_ERRORS regardless of which finalizer wins the
race.
- **`completePartition` / `failPartition` overwrote CANCELLED state**:
the unconditional partition row update lost a concurrent Stop's
CANCELLED status if a worker finished its batch after the Stop
request landed but before noticing it. Add a status-guarded
`updateIfProcessing` DAO method (UPDATE ... WHERE id = :id AND
status = 'PROCESSING') and have both completion paths use it; if 0
rows update, log and skip the side effects (no server-stat increment,
no refreshAggregatedJob call) so the authoritative CANCELLED status
stays. Mirrors the pattern SearchIndex's coordinator uses for the
same race.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
|
||
|
Harsh Vador
|
86e1d88386
|
security: Include branch name in security scan Slack alerts and fail only on high vulnerabilities (#27977)
* Add branch context to security scan Slack alerts and upload CSV findings summary * change failing severity from medium to high & address gitar * fix csv formatting * revert flattening changes |
||
|
Sriharsha Chintalapani
|
b837ade95a
|
docs(github): require issue link, design, tests, UI recording in PR template (#27891)
Expands `.github/pull_request_template.md` to require a linked issue, a high-level design (for large PRs), a structured Tests section (use cases, unit + coverage %, backend/ingestion integration tests, Playwright, manual steps), and a UI screen recording for any UI change. Adds a `/pr-checklist` skill that walks the template, gathers evidence, and drafts the PR body before opening via `gh pr create`. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
mohitdeuex
|
fefa998b0a |
Add MockOidcServer testcontainer for SSO test infrastructure (S0 spike)
First step of the SSO-flow testing initiative. Wraps navikt/mock-oauth2-server as a testcontainer wired into the OM Docker network under alias om-mock-idp on port 1080. The same URL — http://om-mock-idp:1080/<issuer> — is used by: - the OM container (via Docker network alias) - the Playwright browser on the host (via /etc/hosts loopback entry) - the iss claim in tokens issued by the mock IdP so token validation, browser redirects, and OIDC discovery all line up against one source of truth — required for the public/id_token flow where the browser receives the token directly and iss is derived from the URL it hit. Setup cost: one /etc/hosts line (127.0.0.1 om-mock-idp), added once per machine. CI workflow does it automatically. MockOidcServer.launch() throws with a clear remediation message if the entry is missing. MockIdpSmokeUIIT validates the network premise end-to-end: starts the container standalone and confirms discovery + JWKS endpoints respond from the host JVM with the expected om-mock-idp issuer URL. Next (S1): SsoProfile sealed interface, ContainerizedServer.launch(profile) overload, and the first Google SSO end-to-end test. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
mohitdeuex
|
a0e501ba11 |
Add openmetadata-java-playwright scenario test module
Phase 1 of EPIC #3731: Java-driven E2E scenarios for reindex + UI tests. Reuses TestSuiteBootstrap as a test-jar dependency. Three execution modes: - embedded (in-JVM, fast, backend-only) - containerized (Testcontainers + prod server image, UI capable) - external (connect to a running stack) Includes 3 backend reindex scenarios (full / incremental / orphan cleanup), 1 Playwright UI scenario (search-after-reindex in Explore), and 2 CI workflows (PR path-filter + nightly cron). Satisfies #3767 and #3792. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Ariel Schulz
|
297c01cea7
|
Fix (#27660): Re-enable Exasol cli-e2e-tests after fixing issues (#27661)
* Re-enable Exasol cli-e2e-tests after fixing issues * Revert accidental changes from branch switch * Adapt exasol.yml for tests * Add get_table_comment setup and re-enable test_vanilla_ingestion * Add type hints to maintain signature * SQLA-E does not include get_all_table_comments and will come later, so ignore for now * Add return type too |
||
|
Mayur Singal
|
60a2e6546e
|
Migrate Databricks from sqlalchemy-databricks to databricks-sqlalchemy (#26896)
Some checks are pending
Integration Tests - MySQL + Elasticsearch / Detect Changes (push) Waiting to run
Integration Tests - MySQL + Elasticsearch / integration-tests-mysql-elasticsearch (push) Blocked by required conditions
Integration Tests - PostgreSQL + OpenSearch / Detect Changes (push) Waiting to run
Maven Collate Tests / maven-collate-ci (push) Waiting to run
Integration Tests - PostgreSQL + OpenSearch / integration-tests-postgres-opensearch (push) Blocked by required conditions
Java Checkstyle / java-checkstyle (push) Waiting to run
OpenMetadata Service Unit Tests / Detect Changes (push) Waiting to run
OpenMetadata Service Unit Tests / openmetadata-service-unit-tests (mysql) (push) Blocked by required conditions
OpenMetadata Service Unit Tests / openmetadata-service-unit-tests (postgresql) (push) Blocked by required conditions
OpenMetadata Service Unit Tests / k8s_operator-unit-tests (push) Blocked by required conditions
OpenMetadata Service Unit Tests / openmetadata-service-unit-tests-status (push) Blocked by required conditions
Publish Package to Maven Central Repository / publish-maven-packages (push) Waiting to run
* Update Databricks Dependency to databricks-sqlalchemy * Update generated TypeScript types * address comments and pyformat * pyformat * fix log filtering * address comments * fix static unit tests * fix rule for static type * pyformat * update baseline * revert basepyright changes --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Aniket Katkar <aniketkatkar97@gmail.com> |
||
|
Sid
|
ca2d0122db
|
test(playwright): add nightly SAML session renewal coverage (#27619)
* test(playwright): add nightly SAML session renewal spec Covers OM's JWT refresh behavior for SAML sessions end-to-end against the local Keycloak fixture: silent refresh after expiry, concurrent 401s queuing behind a single refresh call, and forced re-login when the server-side SAML HttpSession is gone. Reuses the snapshot/restore mechanism and keycloak-azure-saml provider helper introduced in #27164; shortens samlConfiguration.security.token Validity to 10s so the suite observes multiple expiry cycles in <60s. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Update openmetadata-ui/src/main/resources/ui/playwright/utils/sessionRenewal.ts Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * test(playwright): drop expiry wait from refresh-on-reload SSO specs The reactive 401 refresh path races with the AuthProvider useEffect that wires tokenService.renewToken from authenticatorRef — if the 401 from /users/loggedInUser lands before that effect commits the populated ref, refreshToken() returns null and the user is logged out instead of refreshed. With tokenValidity=10s (< EXPIRY_THRESHOLD_MILLES=60s), the UI's proactive timer in startTokenExpiryTimer fires immediately on every mount, so /auth/refresh is exercised on each reload regardless of expiry state. Assertions on token rotation and session continuity still cover "silent refresh works end-to-end". The SAML-session-gone case still waits for expiry — it needs to. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test(playwright): trigger refresh via SPA nav in SSO renewal specs page.reload() remounts React and re-races the axios interceptor setup in AuthProvider — the useEffect that wires authenticatorRef.renewIdToken onto TokenService has a ref-typed dependency that doesn't reliably re-run, so the first 401 after reload sometimes finds renewToken=null and the interceptor silently logs the user out instead of refreshing. Click the Explore sidebar link instead. The click triggers authenticated API calls while staying inside the already-mounted React tree, so the interceptor always reaches the wired TokenService. Spec now passes 10/10 locally. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Siddhant <siddhant@MacBook-Pro-621.local> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> |
||
|
Chirag Madlani
|
d095413ed1
|
fix(ci): nightly workflow running stale project getting failed [skip-ci] (#27849)
Some checks are pending
Integration Tests - MySQL + Elasticsearch / Detect Changes (push) Waiting to run
Integration Tests - MySQL + Elasticsearch / integration-tests-mysql-elasticsearch (push) Blocked by required conditions
Integration Tests - PostgreSQL + OpenSearch / Detect Changes (push) Waiting to run
Integration Tests - PostgreSQL + OpenSearch / integration-tests-postgres-opensearch (push) Blocked by required conditions
Java Checkstyle / java-checkstyle (push) Waiting to run
Maven Collate Tests / maven-collate-ci (push) Waiting to run
OpenMetadata Service Unit Tests / Detect Changes (push) Waiting to run
OpenMetadata Service Unit Tests / openmetadata-service-unit-tests (mysql) (push) Blocked by required conditions
OpenMetadata Service Unit Tests / openmetadata-service-unit-tests (postgresql) (push) Blocked by required conditions
OpenMetadata Service Unit Tests / k8s_operator-unit-tests (push) Blocked by required conditions
OpenMetadata Service Unit Tests / openmetadata-service-unit-tests-status (push) Blocked by required conditions
Publish Package to Maven Central Repository / publish-maven-packages (push) Waiting to run
|
||
|
miriann-uu
|
7b01731754
|
GEN-5164: Add cherry pick matrix (#27674) | ||
|
Teddy
|
11e5ac95d4
|
chore: update sqlalchemy to 1.0.0 (#27776)
Some checks are pending
Integration Tests - MySQL + Elasticsearch / Detect Changes (push) Waiting to run
Integration Tests - MySQL + Elasticsearch / integration-tests-mysql-elasticsearch (push) Blocked by required conditions
Integration Tests - PostgreSQL + OpenSearch / Detect Changes (push) Waiting to run
Integration Tests - PostgreSQL + OpenSearch / integration-tests-postgres-opensearch (push) Blocked by required conditions
Java Checkstyle / java-checkstyle (push) Waiting to run
Maven Collate Tests / maven-collate-ci (push) Waiting to run
OpenMetadata Service Unit Tests / Detect Changes (push) Waiting to run
OpenMetadata Service Unit Tests / openmetadata-service-unit-tests (mysql) (push) Blocked by required conditions
OpenMetadata Service Unit Tests / openmetadata-service-unit-tests (postgresql) (push) Blocked by required conditions
OpenMetadata Service Unit Tests / k8s_operator-unit-tests (push) Blocked by required conditions
OpenMetadata Service Unit Tests / openmetadata-service-unit-tests-status (push) Blocked by required conditions
Publish Package to Maven Central Repository / publish-maven-packages (push) Waiting to run
|
||
|
IceS2
|
e9c87c6adb
|
chore(ingestion): drop pylint, expand ruff (#27774)
* chore(ingestion): drop pylint, expand ruff to Stage 2c
Replace pylint with a coherent ruff-only stack (Stage 2c of the modernize
roadmap). Pylint is dropped from dev deps and CI workflows; ruff selected
ruleset expanded to ~22 families covering style, bug catchers, hygiene,
and the pylint port (PLE/PLC/PLW/PLR with the noisy "too-many-X"
complexity caps + magic-value disabled).
What's selected (with rationale in pyproject.toml):
E, W, F, I, N — style + correctness baseline + naming
UP — pyupgrade (py>=3.10 modernizations)
B, C4, C90, RET, SIM, TRY — bug catchers
PIE, ICN, T20, TC, TID, PTH, PERF — hygiene
PLE, PLC, PLW, PLR — pylint port (PLR complexity caps ignored)
RUF — ruff-native (incl. RUF100 unused-noqa)
What's removed:
- .pylintrc (root) — duplicate of the ingestion pylint config
- [tool.pylint.*] block in ingestion/pyproject.toml (~140 lines)
- ingestion/plugins/{print_checker,import_checker}.py + tests + README
(replaced by built-in T20 + TID251 banned-api respectively)
- pylint dep from ingestion/setup.py and openmetadata-airflow-apis/pyproject.toml
- `make lint` Makefile target + the pylint invocation in py_format_check
- dead pylint TODO comment + ignored test entry in noxfile.py
Cwd-stable config: ruff is invoked both from the repo root (pre-commit,
CI) and from ingestion/ (`make py_format_check`). The `src`,
`extend-exclude`, and per-file-ignores entries are listed twice — once
relative to ingestion/ and once with the `ingestion/` prefix — so
first-party isort detection and exclusions match in both invocations.
Grandfathering: ran `ruff check --add-noqa` once + format-stable
iteration. ~12,130 noqa directives across ~1,400 files. Cleanup is
deferred to follow-up PRs that drop noqas one rule at a time.
Documentation sweep: replaced `make lint` references in CLAUDE.md,
AGENTS.md, DEVELOPER.md, copilot-instructions, and 6 SKILL files with
the apply+verify shape `make py_format && make py_format_check`.
`make py_format` is NOT a strict superset of pylint — it only applies
auto-fixable violations; `make py_format_check` catches the rest.
Basedpyright baseline regenerated: ruff format reflowed multi-line
signatures in ~70 files, shifting type-error column positions. The
basedpyright baseline matches by (file path, error code, range), so
column shifts caused 19 entries to mis-align. Net diff is small
(154 lines in/out of the 13MB baseline.json) — purely positional.
Verified locally:
- make py_format_check → All checks passed
- nox --no-venv -s static-checks → 0 errors, 0 warnings, 0 notes
* chore(ingestion): finish ruff swap — nox lint session + skill docs
Three remaining stale-tooling references after Stage 2c:
- `ingestion/noxfile.py` `lint` session was still calling `black --check`,
`isort --check-only`, `pycln --diff`. Those tools aren't installed
anywhere (we dropped them from dev deps). Replace with the ruff
equivalents that mirror `make py_format_check`.
- `skills/standards/code_style.md`: stack listed as `black + isort +
pycln`; line length claimed 88 (black default). Both wrong: stack is
ruff, line length is 120.
- `skills/connector-building/SKILL.md`: `make py_format` comment said
`# black + isort + pycln`. Same swap.
* chore(ingestion): keep main's baseline + globally ignore TRY400
Per gitar-bot's review on PR #27774:
1. Main's PR #27728 promoted ~60 `logger.warning()` → `logger.error()`
inside `except` blocks. Those changes landed on main with their own
baseline updates. Our PR doesn't promote anything — the merge from
origin/main brought those `error` calls along with their baseline
entries.
The bot interpreted the `# noqa: TRY400` we added next to those lines
as us silencing the rule case-by-case. Cleaner: globally ignore
TRY400 in pyproject.toml, with a comment explaining why the codebase's
`logger.error(...)` + separate `logger.debug(traceback.format_exc())`
pattern is intentional. Strip ~430 per-line `# noqa: TRY400` markers
from source.
2. Document that `S101` in `per-file-ignores` is a forward-looking
entry — flake8-bandit (`S`) is not yet selected, so the rule is
no-op today; the entry stays so when `S` lands later, tests don't
immediately error.
Reverts the platform pin and Linux Docker–generated baseline. Keep
main's baseline intact and let CI surface the exact column-shifted
entries; the team will decide whether to fix in-place (revert format
on affected files) or add per-line `# pyright: ignore` markers.
* chore(ingestion): regen baseline for new connector type debt
Main's baseline was stale relative to recently-added connectors
(McpConnection, CustomDriveConnection) that lack common attributes
like `hostPort`, `database`, `catalog` etc. — all sites that access
those attributes via the union-typed `serviceConnection.root.config`
fire `reportAttributeAccessIssue` errors that aren't baselined.
71 errors + 58 warnings absorbed. Local macOS regen; pushing to see
CI's drift count. Per the basedpyright-baseline-and-ci PR experience,
macOS↔Linux column drift on this size of regen has historically been
1-7 residuals.
|
||
|
IceS2
|
84ed278720
|
chore(ingestion): enable basedpyright across the codebase via baseline (#27755)
* chore(ingestion): enable basedpyright across the codebase via baseline
Removes the ~25 paths from `[tool.basedpyright] ignore` (which excluded
roughly 90% of the codebase from type checking) and grandfathers the
existing violations into a baseline file. New violations in any
previously-ignored file now fail CI.
Changes:
- ingestion/pyproject.toml: drop the entire `ignore = [...]` block
- ingestion/setup.py: bump `basedpyright~=1.14` to `~=1.39.0`
- ingestion/.basedpyright/baseline.json (new, ~13MB): captures the
starting violation set (~18.8K errors + ~37.4K warnings) so the
migration is behavior-preserving. Regenerate with
`cd ingestion && basedpyright -p pyproject.toml --baselinefile
.basedpyright/baseline.json --writebaseline`. basedpyright analysis
has minor non-determinism (similar to ruff's), so re-running
--writebaseline a few times converges the baseline.
- ingestion/noxfile.py: pass `--baselinefile .basedpyright/baseline.json`
to the basedpyright invocation in the `static-checks` session so CI
honors the grandfathering. CI already runs the session via
`cd ingestion && nox --no-venv -s static-checks` (py-tests.yml).
- ingestion/Makefile: `make static-checks` now delegates to
`nox -s static-checks` so local invocations match CI exactly. Also
drops the dead Python 3.9 / OM_SKIP_SDK_PY39 branch (we require
Python >=3.10 since the previous modernization PR).
- .gitignore: add `.serena/` (local language-server cache)
* chore(ingestion): add nox to the dev dependency set
The static-checks Makefile target and the py-tests CI job both delegate
to `nox -s static-checks`, but nox was being installed as a separate
side step (`pip install nox` in `install_dev_env`, `uv pip install nox`
in the test-environment composite action). Listing it in dev extras
means a plain `pip install ingestion[dev]` brings it in.
* chore(ingestion): pin basedpyright analysis to py3.10; CI runs once
Following the basedpyright + multi-Python-version research:
- ingestion/pyproject.toml: add `pythonVersion = "3.10"` to
[tool.basedpyright] so type-checking always analyzes for the lowest
supported Python version. Forward-incompatible code (tomllib usage,
PEP 695 generics, etc.) is caught at type-check time regardless of
which Python interpreter runs the checker.
- .github/workflows/py-tests.yml: gate the "Run Static Checks" step on
`matrix.py-version == '3.10'`. With pythonVersion pinned, results are
identical across the matrix; running once avoids redundant work and
keeps the baseline file deterministic. Unit tests still run on the
full 3.10/3.11/3.12 matrix to verify runtime compatibility.
- ingestion/.basedpyright/baseline.json: regenerated cleanly with the
new pythonVersion config (~18.8K errors / ~37.3K warnings, similar
scale to the previous baseline). Aligns with the canonical
type-check-on-floor / test-on-matrix pattern used by Pydantic, CPython,
and other major Python projects.
* chore(ingestion): pin basedpyright pythonPlatform to Linux + regen baseline
CI's previous run still surfaced ~9 issues (2 errors + 7 warnings) that
weren't in the baseline. Root cause: my local environment differs from
CI's in three ways that affect type inference — Python interpreter
(3.11 vs 3.10), platform (Darwin vs Linux), and pip-resolved package
versions (couchbase, avro, trino, sqlalchemy stubs all differ slightly).
This commit closes the platform gap and regenerates the baseline from a
fresh CI-equivalent environment:
- ingestion/pyproject.toml: add `pythonPlatform = "Linux"` to
[tool.basedpyright] so type-checking uses the Linux subset of stdlib /
third-party stubs regardless of where the analyzer runs.
- ingestion/.basedpyright/baseline.json: regenerated against a fresh
Python 3.10 venv installed via `uv pip install ingestion[test]` (the
same install path CI's setup-openmetadata-test-environment composite
action uses). New scale: ~18.7K errors / ~37.5K warnings — same
ballpark as the previous baseline, with column positions now matching
CI's environment.
Local-developer note: when running `make static-checks` from a venv
that doesn't mirror CI exactly (e.g. macOS, Python 3.11, different
package versions), you may see drift errors. The supported workflow for
regenerating the baseline is to mirror CI:
python3.10 -m venv /tmp/ci-mirror
source /tmp/ci-mirror/bin/activate
uv pip install --upgrade pip "setuptools<81"
uv pip install --no-build-isolation "cx_Oracle>=8.3.0,<9"
uv pip install -e "ingestion[test]"
uv pip install "basedpyright~=1.39.0" nox
cd ingestion && basedpyright -p pyproject.toml \
--baselinefile .basedpyright/baseline.json --writebaseline
* chore(ingestion): drop pythonPlatform pin and regen baseline from CI-mirror
The previous attempt added `pythonPlatform = "Linux"` thinking it would
make the local-generated baseline match CI. It did the opposite — Linux
platform stubs activate additional conditional code paths that weren't
analyzed before, so CI saw 101 errors instead of the prior 2 errors.
Reverting:
- Drop `pythonPlatform = "Linux"` from [tool.basedpyright]. Without it,
basedpyright analyzes for the host platform; on CI's ubuntu-latest
runner that's Linux automatically, but type-stub coverage stays the
same as before (matching the
|
||
|
IceS2
|
1fa0c79d27
|
chore(github): migrate issue templates to structured forms (#27710)
* chore(github): migrate issue templates to structured forms - Convert bug_report, feature_request, doc_update to GitHub issue forms (YAML) - Add connector_bug form with free-text Connector field - Drop epic and feature_task templates (stale since 2022, no usage evidence) - Add auto-label workflow that maps the Connector field to a namespaced connector:<name> label, falling back to connector:other on 0 or 2+ matches - Labels are applied exclusively and auto-created with a grey "Connector" description when missing * chore(github): drop redundant pipeline type field from connector_bug form Feature area already covers metadata/lineage/profiler/usage distinction. * fix(github): address PR review feedback - bug_report.yml: add labels: ["bug"] for pattern consistency - label-connector.yml: add contents: read permission (needed by checkout) - label_connector.py: raise on unexpected HTTP status; accept 404 for idempotent GET-label and DELETE-label-from-issue; stop echoing the raw Connector field value into workflow logs |
||
|
Mayur Singal
|
878421a644
|
fix: enable subprocess coverage tracking for CLI E2E tests (#27329)
* fix: enable subprocess coverage tracking for CLI E2E tests
CLI E2E tests run connectors via `subprocess.Popen("metadata ingest")`
but the subprocess coverage data was silently lost. Two issues:
1. Missing `parallel = true` in coverage config — parent pytest process
and child subprocess both wrote to the same `.coverage` file, causing
data collision. With parallel mode, each process writes to its own
`.coverage.<pid>` file that `coverage combine` can merge.
2. `COVERAGE_PROCESS_START` used a relative path (`ingestion/pyproject.toml`)
in sitecustomize.py. Resolved to absolute using `GITHUB_WORKSPACE`.
Evidence: Metabase (zero unit tests, only E2E) shows 53.6% on SonarCloud
with client.py at 17.2% — inspection of .coverage.metabase confirms only
import-time + in-process setup lines are present, with zero method body
coverage from the subprocess execution.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: remove -a (append) flags incompatible with parallel coverage mode
`coverage run -a` and `coverage combine -a` conflict with `parallel = true`
in the coverage config. In parallel mode each process writes to its own
`.coverage.<pid>` file, and `coverage combine` merges them — no append needed.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* MINOR: Fix snowflake e2e (#26677)
* MINOR: Fix snowflake e2e
* fix pyformat
* improve snowflake test
* fix count
* mark flaky auto classification test
* improve test address comment
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
||
|
Sid
|
0a98f5bf32
|
test(playwright): add nightly SSO login spec starting (#27164)
* test(playwright): add nightly SSO login spec starting with Okta
Extends Playwright coverage end-to-end for SSO login flows. Today's SSO
coverage (Features/SSOConfiguration.spec.ts) only asserts the config
form UI. This adds a new suite that configures OpenMetadata to an
external identity provider, drives a real login through the provider's
hosted UI, and validates the resulting session against the OM API.
Phase 1 ships Okta only (integrator-9351624.okta.com). Additional
providers (Auth0, Azure, Cognito, SAML, Google) plug into the same
dispatcher by adding a ProviderHelper implementation.
## What's new
- playwright/e2e/Auth/SSOLogin.spec.ts — two-test suite tagged @sso
1. Asserts the SSO sign-in button renders on /signin with the correct
brand label and that the basic-auth form is not shown.
2. Clicks the button, drives the provider's login widget, follows the
OAuth callback, completes first-run self-signup when needed,
lands on /my-data, then verifies the JWT by calling
GET /api/v1/users/loggedInUser and asserting the returned email
matches SSO_USERNAME.
- playwright/utils/ssoAuth.ts — provider-agnostic orchestration:
applyProviderConfig (PUT /api/v1/system/security/config),
restoreBasicAuth, buildAuthContextFromJwt, verifyLoggedInUserMatches.
Composes existing getApiContext/getAuthContext/getToken helpers — no
token extraction or HTTP plumbing is reimplemented.
- playwright/utils/sso-providers/{index,okta}.ts — ProviderHelper
interface plus the Okta Identity Engine widget driver. Defaults the
dev tenant values from the committed openmetadata.yaml snippet so the
spec only needs SSO_USERNAME/SSO_PASSWORD to run locally.
- playwright/constant/ssoAuth.ts — env var key constants,
PROVIDER_BUTTON_TEXT map, and the BASIC_AUTH_CONFIG payload used for
cleanup.
- playwright.config.ts — new 'sso-auth' project matching
playwright/e2e/Auth/**/*.spec.ts with its own serial workers, and
'**/Auth/**' added to the chromium project's testIgnore so these
tests never run in the default suite.
## How provider switching works
beforeAll logs in as admin via basic auth, captures the admin JWT via
getToken(page) BEFORE the swap, then PUTs the Okta config. The admin
JWT survives the provider swap because OM's internal JWKS stays in
publicKeyUrls and the admin user's isAdmin flag is persisted in the DB.
afterAll rebuilds an API context from that JWT and restores basic auth,
making the spec fully idempotent — the same OM instance can run the
suite repeatedly without any manual cleanup.
## Running locally
export SSO_PROVIDER_TYPE=okta
export SSO_USERNAME='<okta-test-user>'
export SSO_PASSWORD='<okta-test-password>'
npx playwright test playwright/e2e/Auth/SSOLogin.spec.ts \
--project=sso-auth --workers=1
Verified end-to-end against integrator-9351624.okta.com — both tests
pass in ~12s on an already-provisioned user, ~14s on first-run
self-signup. Cleanup leaves the server in basic-auth mode.
## Notes for reviewers
- The existing .github/workflows/playwright-sso-tests.yml already wires
up the CI matrix and secret names; this change intentionally does
NOT enable the cron schedule. That lands in a follow-up once one
provider is stable for a few nightly runs.
- OKTA_SSO_CLIENT_ID / OKTA_SSO_DOMAIN / OKTA_SSO_PRINCIPAL_DOMAIN env
vars can override the baked-in dev tenant defaults if a different
Okta tenant is used in CI.
* ci: add dedicated SSO Login Nightly workflow
Adds .github/workflows/playwright-sso-login-nightly.yml, a standalone
workflow that runs the new SSOLogin spec nightly at 03:00 UTC instead
of piggy-backing on playwright-sso-tests.yml.
The existing playwright-sso-tests.yml is left untouched — it still
covers the SSO configuration form UI via SSOConfiguration.spec.ts and
its matrix/secrets wiring is unchanged. The new workflow complements
it with a real end-to-end login round-trip:
- Schedule: cron '0 3 * * *'
- Provider matrix: okta only for Phase 1 (extended as helpers ship)
- Invokes playwright/e2e/Auth/SSOLogin.spec.ts under the new
sso-auth Playwright project with workers=1
- Wires provider credentials via secrets with the existing
{PROVIDER}_SSO_USERNAME / {PROVIDER}_SSO_PASSWORD convention plus
optional OKTA_SSO_CLIENT_ID / OKTA_SSO_DOMAIN /
OKTA_SSO_PRINCIPAL_DOMAIN overrides
- Uses the shared setup-openmetadata-test-environment composite
action, PostgreSQL, ingestion disabled — matching the existing SSO
tests workflow
- Uploads the HTML report as an artifact on every run and cleans up
the docker stack in a final always-run step
* refactor(playwright): simplify ssoAuth helpers
- verifyLoggedInUserMatches now asserts directly on the lowercased
email field instead of building a candidate array and feeding it a
long stringified failure message. The assertion failure already
shows expected vs received, so the wrapper string was just noise.
- Drop buildAuthContextFromJwt — it was a one-line wrapper around
getAuthContext. The spec calls getAuthContext directly now.
* refactor(playwright): address SSO suite review feedback
- Extract OM_BASE_URL from PLAYWRIGHT_TEST_BASE_URL (with the same
http://localhost:8585 default as playwright.config.ts) and export
it from constant/ssoAuth.ts. okta.ts and BASIC_AUTH_CONFIG both
consume it, so callbackUrl, the OM JWKS entry in publicKeyUrls, and
the basic-auth restore payload all match the test target — including
CI runs against non-default hosts.
- Drop PROVIDER_BUTTON_TEXT. It was exported but never imported; the
ProviderHelper.expectedButtonText field is the only source of truth
for the SSO sign-in button label and the spec already reads from it.
- Restore the OM convention adminPrincipals: ['admin'] in the Okta
config (matches conf/openmetadata.yaml's AUTHORIZER_ADMIN_PRINCIPALS
default). The previous code was granting admin to whichever IdP user
ran the suite — verifyLoggedInUserMatches only needs an authenticated
session, not admin, so the elevation was unnecessary. This also drops
the now-unused requireEnv on SSO_USERNAME inside okta.ts; the spec
itself still gates on the env var via test.skip.
- Set workers: 1 on the sso-auth Playwright project. fullyParallel:
false alone wasn't enough — the global workers: 3 on CI could still
fan out across multiple Auth/**/*.spec.ts files in the future. The
explicit limit enforces full isolation as more provider specs land.
* ci: avoid CodeQL "Excessive Secrets Exposure" in SSO Login Nightly
Replaces the dynamic secret lookup
secrets[format('{0}_SSO_USERNAME', upper(matrix.provider))]
with a static reference
secrets.OKTA_SSO_USERNAME
CodeQL flagged the dynamic indexing because GitHub Actions can only
mask & scope secrets that are referenced statically. With a computed
key, the runner has no way to know which single secret is needed and
conservatively materializes EVERY org and repo secret into the step's
environment — even though the test only reads OKTA_SSO_*. Static
references let GitHub expose only the two credentials this step
actually uses.
Phase 1's matrix is okta-only so the change is two lines. The added
inline comment documents the convention for future providers: add a
sibling step gated by `if: matrix.provider == '<provider>'` with that
provider's static secret references — do not bring back the
secrets[format(...)] pattern.
* refactor(playwright): capture/restore real security config in SSO suite
- Snapshot /system/security/config in beforeAll, restore exact payload in
afterAll instead of PUTting a hand-rolled basic-auth baseline (preserves
allowedDomains, forceSecureSessionCookie, adminPrincipals, etc.)
- Strip ldap/saml subtrees from the snapshot: GET returns empty-string
placeholders the PUT validator rejects
- Require OKTA_SSO_{CLIENT_ID,DOMAIN,PRINCIPAL_DOMAIN} via getRequiredEnv;
no more hardcoded tenant defaults
- Fail fast in beforeAll if admin JWT capture returns empty string so the
server is never left stuck in SSO mode
- Shrink Okta provider override to just the fields Okta needs; sibling
authorizer fields come from the captured snapshot
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* ci(sso-login): extract per-provider composite action
Restructures the nightly workflow so provider credentials stay statically
referenced for CodeQL while making it trivial to add new providers:
- New composite action .github/actions/sso-login-run bundles all shared
setup + test-run logic; pulls non-secret provider config from the
caller's vars context dynamically (${PROVIDER_UPPER}_SSO_*)
- playwright-sso-login-nightly.yml becomes a thin dispatcher with one
real job per provider. Each job declares environment: test so it can
resolve its password via a static secrets.<PROVIDER>_SSO_PASSWORD
reference (no secrets[format(...)] dynamic lookup, CodeQL clean)
- Adding a provider = copy the okta job stanza, swap the secret name,
add the provider to the dispatch input choices, register the helper
in sso-providers/index.ts
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* refactor(playwright): move Okta tenant config to a repo constant
The Okta tenant identifiers (clientId, domain, principalDomain) are
non-secret OAuth public values — visible on the hosted login page
during any sign-in. Keeping them in GitHub environment variables cost
setup friction (5 env vars to configure locally, each a potential typo)
without any security benefit. Move them back to a committed OKTA_TENANT
constant in okta.ts where a reviewer can see exactly which tenant the
suite is exercising.
Net effect:
- Local runs only need SSO_PROVIDER_TYPE, SSO_USERNAME, SSO_PASSWORD.
- The test environment in GH Actions keeps OKTA_SSO_USERNAME (variable)
and OKTA_SSO_PASSWORD (secret); the three tenant variables are no
longer consumed.
- Composite action drops the jq-based dynamic var extraction; the
caller passes sso_username directly.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* ci(sso-login): move timeout-minutes from composite step to job level
Composite actions don't support timeout-minutes on individual steps —
that's a runner job field only. Move the 30-minute test timeout up to
the dispatcher job and bump to 45 minutes to cover docker + maven setup.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* ci(sso-login): consolidate dispatcher + composite action into one file
Collapse the dispatcher workflow + composite action split into a single
~115-line workflow using a strategy matrix and dynamic
vars[format(...)] / secrets[format(...)] credential resolution keyed on
the matrix provider name.
Trade-off:
- CodeQL "Excessive Secrets Exposure" (low severity) will re-flag the
dynamic secret lookup. Accepted in exchange for a single source of
truth and true zero-workflow-churn multi-provider support.
Onboarding a new provider is now:
1. Add its name to the matrix array + dispatch options list.
2. Add <PROVIDER>_SSO_USERNAME (variable) + <PROVIDER>_SSO_PASSWORD
(secret) in the test environment.
3. Register the helper in sso-providers/index.ts.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* ci(sso-login): drop provider-prefix bash step; use case-insensitive lookup
GitHub secret and variable names are case-insensitive, so
format('{0}_SSO_PASSWORD', matrix.provider) with the lowercase matrix
value resolves correctly against the uppercase conventional names like
OKTA_SSO_PASSWORD. That removes the need for a separate "Compute
provider prefix" step and its cross-step env-context plumbing.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* ci(sso-login): drop redundant case-insensitivity comment
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* ci(sso-login): pin playwright install to 1.57.0 to match package.json
The previous 1.51.1 pin was stale vs. the @playwright/test version in
package.json. The mismatch caused browser cache path divergence — the
install step wrote browsers under 1.51.1's cache and the test run
looked for them under 1.57.0's cache and failed with "browsers not
installed."
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* refactor(playwright): address SSO suite review comments [skip ci]
- Drive Okta tenant (clientId, domain, principalDomain) from env vars,
falling back to the existing nightly tenant values as defaults
- Use redirectToHomePage as the final assertion in the SSO login step
- Document why the /signup vs /my-data branch is conditional
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* saml
* test(playwright): add SAML providers to SSO login nightly
Extend the nightly SSO login matrix with Azure AD SAML and a self-contained
Keycloak SAML fixture (Azure-profile + Google-profile realms), so the suite
exercises the full SAML flow end-to-end without relying on a hosted IdP.
- docker/local-sso/keycloak-saml: Keycloak 26.3.3 compose + pre-imported
realms bound to OM at localhost:8585, port-overridable via
KEYCLOAK_SAML_PORT.
- playwright sso-providers: azure-saml helper (hosted tenant, non-secret
federation metadata committed) and keycloak-saml factory that fetches the
realm's IdP X509 at runtime.
- SSO assertion matches OM's actual SAML sign-in label ("Sign in with
SAML SSO"), since providerName isn't propagated into the store for the
SAML provider branch of getAuthConfig.
- Workflow starts/stops the Keycloak stack only for keycloak-* matrix rows
and injects the fixture credentials inline.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* refactor(playwright): fetch Azure SAML IdP cert at runtime
Drop the committed Azure Federated SSO X509 certificate and the
AZURE_SAML_IDP_CERTIFICATE env fallback from the azure-saml provider.
The cert now comes from Azure's federation metadata XML endpoint at test
start, mirroring how the Keycloak provider resolves its realm cert, so the
suite stays aligned with Azure's ~3-year cert rotations automatically.
- New saml-metadata.ts exporting fetchIdpX509Certificate(descriptorUrl,
label), reused by azure-saml and keycloak-saml.
- azure-saml.buildConfigPayload is now async and pulls the cert from
https://login.microsoftonline.com/<tenantId>/federationmetadata/2007-06/federationmetadata.xml
before building the SAML payload.
- keycloak-saml drops its inline cert-fetching helpers and delegates to
the shared util.
- Trim narration comments across the SSO suite to keep only the
non-obvious rationale.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* refactor(playwright): drop hosted Azure SAML provider
The nightly Keycloak SAML fixture with Azure-profile attribute claims
exercises the same OM SAML code path as the hosted Azure AD tenant. The
hosted provider added external tenant/cert coupling without unique
coverage, so this removes it.
Drops the azure-saml helper, its env keys (AZURE_SAML_TENANT_ID /
AZURE_SAML_PRINCIPAL_DOMAIN), the dispatcher registration, and the
workflow dispatch option. Keycloak Azure/Google realms remain.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* test(playwright): cover SSO session lifecycle end-to-end
Extends the SSO login spec beyond "can you log in" to the full session
round-trip: reload survives, same-context tabs inherit auth, sidebar
logout (with modal confirm) lands on /signin, and post-logout refresh
stays signed out.
Adds a describe-scoped userContext/userPage created in beforeAll so
tests 2-6 inherit the IdP-backed session; test 1 keeps its fresh
fixture for the unauthenticated assertion. Cleanup closes the user
context before restoring the server security config.
Verified locally against keycloak-azure-saml and keycloak-google-saml
realms: 6 passed each (was 2).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* remove slow from individual spec
* remove slow from beforeAll
* style(playwright): fix SSOLogin spec prettier issues
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* test(playwright): tighten SSO sign-in locator and await logout response
Address Copilot review comments on PR #27164:
- Use button.signin-button to match the pattern in SSOAuthentication.spec.ts.
- Await /api/v1/users/logout POST alongside the /signin navigation in
the logout test to remove the race against the server response.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix
* Update openmetadata-ui/src/main/resources/ui/playwright/e2e/Auth/SSOLogin.spec.ts
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* fix
* test(playwright): resolve SSO creds via env vars, drop keycloak-google-saml
Route Keycloak credentials through the same `vars[format(...)]` /
`secrets[format(...)]` indirection as Okta via an `env_prefix` matrix
column, removing the hardcoded fixture literals from the workflow.
Password lookup falls back `vars || secrets` so fixture passwords can
live as vars while real provider secrets stay in secrets.
Also drop the keycloak-google-saml variant — same IdP and realm shape
as the Azure variant, so it adds CI cost without meaningful coverage.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* test(playwright): post SSO login nightly results to Slack
Adds a per-provider Slack notification step mirroring the pattern used
by the postgresql/mysql nightly workflows — reuses the existing
`slack-cli.config.json` and `playwright-slack-report` CLI against the
`results.json` that the global JSON reporter already emits.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(playwright): drop logout response wait in SSO spec
OktaAuthenticator.logout clears tokens locally with no backend call, and
GenericAuthenticator (SAML) hits `GET /auth/logout` — neither triggers
the `POST /api/v1/users/logout` the test was waiting on. The listener
never matched, so `Promise.all` hung past the 180s test timeout even
though the page had already navigated to /signin.
Rely on `waitForURL('**/signin')` + the signin button assertion, which
are the actual cross-provider success signals.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Siddhant <siddhant@MacBook-Pro-457.local>
Co-authored-by: Siddhant <siddhant@MacBook-Pro-529.local>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: Siddhant <siddhant@MacBook-Pro-621.local>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
|
||
|
Aniket Katkar
|
12ce3b614d
|
Chore(UI): consolidated UI checkstyle fix commands and modify workflow comment (#27402)
* feat: add consolidated UI checkstyle commands for all and changed files * update prt to pr * test commit to fail ui-checkstyle * update the comment * Revert "test commit to fail ui-checkstyle" This reverts commit |
||
|
Teddy
|
50c17502cf
|
MINOR - Enable merge group GH event (#27371)
* chore: added merge_group for github merge queue * chore: remove unnecessary merger group on team labeler * fix: added gates for merge queue and pull request events |
||
|
Pere Miquel Brull
|
1dedc0cf15
|
Add k8s-operator unit tests to PR CI (#27387)
* Add k8s-operator unit tests to PR CI pipeline The k8s operator tests only ran during manual release builds. Add a path-filtered job so they run on PRs touching openmetadata-k8s-operator/**, following the same Detect Changes pattern used by the service unit tests. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Remove -DfailIfNoTests=false — we want to catch missing tests Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Fix k8s-operator tests: add surefire includes and remove unnecessary stub Parent POM surefire includes only match org.openmetadata.service.*, so operator tests under org.openmetadata.operator.* were silently skipped. Override with **/*Test.java in the operator pom.xml. Also remove unused KubernetesClient mock stub from CronOMJobReconcilerTest.setUp — no test reaches the code path that calls context.getClient(), causing UnnecessaryStubbingException. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Rename k8s-operator to k8s_operator in workflow outputs Hyphens in output names are parsed as subtraction in GitHub Actions expressions dot notation, so the job condition would never trigger. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Fix filesystem paths — underscore rename only applies to output keys The replace_all incorrectly changed directory names from openmetadata-k8s-operator to openmetadata-k8s_operator. Only the GitHub Actions output key needs the underscore; all file paths must use the actual hyphenated directory name. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Drop -am flag from k8s-operator test command openmetadata-service is a provided-scope dependency, so -am tries to compile it including shaded ES/OS jars that aren't available in a clean CI environment. The operator module compiles fine on its own. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Fix invalid YAML in conf/openmetadata.yaml The CSP policy line has unescaped colons inside the value which the YAML parser interprets as mapping indicators. Use a folded block scalar (>-) so the value is parsed as a plain string. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Build k8s-operator deps before running tests The operator depends on openmetadata-service (provided scope) which won't be in the Maven cache on a cold CI runner. Build with -am -DskipTests first, then run operator tests separately — same pattern as docker-k8s-operator.yml. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Reintroduce lenient client mock to prevent flaky NPE The reconcile flow is time-dependent — tests using "0 * * * *" can reach context.getClient() near the top of the hour. Stub the full client.resources().inNamespace().resource().create() chain as lenient so early-return tests aren't penalized but happy-path tests won't NPE. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Revert conf/openmetadata.yaml — fix belongs in a separate PR Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
||
|
Harsh Vador
|
f4c939869d
|
ci(security): add Retire.js workflow to detect bundled JS vulnerabilities (#27315)
* ci(security): add Retire.js workflow to detect bundled JS vulnerabilities * address gitar * add om existing security scan workflow * address gitar * add slack support & remove PR check * address gitar * change job name * address comment * address comment |
||
|
Sriharsha Chintalapani
|
bb0daa180e
|
RDF, cleanup relations and remove unnecessary bindings, add distributed mode for RDF reindex (#26902)
* RDF, cleanup relations and remove unnecessary bindings, add distributed mode for RDF reindex * Update generated TypeScript types * Address comments from copilot * Update generated TypeScript types * fix test issues * Fix minor UI bugs * Add the missing filters * Fix RDF export API error * Add export functionality * Fix ui-checkstyle * Fix java checkstyle * Fix unit tests * Fix and increase the coverage for KnowledgeGraph.spec.ts * Fix tests * Remove rdf as default in playwright and local docker * fix ui-checkstyle * Address comments * Potential fix for pull request finding 'CodeQL / Artifact poisoning' Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> * Address copilot comments * Address copilot comments * FIx tests * FIx docker * Update openmetadata-service/src/main/java/org/openmetadata/service/apps/bundles/rdf/distributed/DistributedRdfIndexCoordinator.java Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Address copilot review comments: license headers, JSON escaping, type safety, border-color, stop semantics Agent-Logs-Url: https://github.com/open-metadata/OpenMetadata/sessions/c026e52e-162b-4c9a-9874-43791d4aaac1 Co-authored-by: harshach <38649+harshach@users.noreply.github.com> * Show error toast for unsupported export format in KnowledgeGraph Agent-Logs-Url: https://github.com/open-metadata/OpenMetadata/sessions/c026e52e-162b-4c9a-9874-43791d4aaac1 Co-authored-by: harshach <38649+harshach@users.noreply.github.com> * Fix docker * Fix docker for playwright * Fix docker for playwright * Fix tests * Fix tests * Fix docker * Fix docker * Fix glossary and pagination spec flakiness * update the missing translations * Fix docker * Fix docker * Fix integration test * Fix fuseki not starting * Fixed the run local docker script * worked on comments * Fix flakiness in knowledge graph tests * Fix checkstyle --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Aniket Katkar <aniketkatkar97@gmail.com> Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: harshach <38649+harshach@users.noreply.github.com> |
||
|
Chirag Madlani
|
4f7be5f014
|
fix(ci): filter blob pattern causing failure to sonarcloud (#27357)
* fix(ci): filter blob pattern causing failure to sonarcloud * fix(ci): add missing backslash continuation in sonar-scanner command Agent-Logs-Url: https://github.com/open-metadata/OpenMetadata/sessions/88d229f2-81dd-4662-8295-a3bb0df03815 Co-authored-by: chirag-madlani <12962843+chirag-madlani@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> |
||
|
Aniket Katkar
|
3428dfbd6a
|
Chore(UI): Fix rbac tests not running on PR checks (#26994)
* Fix rbac tests not running on PR checks * update the dependency * Update the SearchRBAC dependency |
||
|
Pere Miquel Brull
|
f6258819e7
|
ci: reduce checkout history footprint in PR workflows (#27221)
* ci: reduce checkout history footprint in PR workflows Optimize actions/checkout usage to avoid downloading the full repo blob history on every PR run. The repo is large, so cloning everything just to run tests wastes minutes of CI time per job. - py-operator-build-test.yml: drop fetch-depth: 0 (no history needed) - openmetadata-service-unit-tests.yml: drop fetch-depth: 0 (Sonar is explicitly skipped via -Dsonar.skip=true); shallow-fetch PR base ref - airflow-apis-tests.yml, py-tests.yml, yarn-coverage.yml: add filter: blob:none to Sonar jobs so commits/trees remain available for blame while blobs are fetched lazily on demand - ui-checkstyle.yml: add filter: blob:none to all jobs that rely on tj-actions/changed-files (needs commit/tree metadata, not blobs) * ci: drop fetch-depth: 0 from jobs that don't walk history Follow-up audit after the initial pass. Four jobs were still declaring fetch-depth: 0 (plus filter: blob:none in two cases) without actually needing any history beyond HEAD. ui-checkstyle.yml - i18n-sync: runs 'yarn i18n' then 'git status --porcelain'. git status compares the working tree to HEAD; no history walk. Default depth 1 is sufficient. - app-docs: same pattern with 'yarn generate:app-docs'. py-sonarcloud-nightly.yml - py-unit-tests: only uploads a coverage artifact, no Sonar invocation. - py-integration-tests: same. - py-combine-coverage: does run SonarSource/sonarqube-scan-action, so it genuinely needs the commit graph — added filter: blob:none for parity with the PR Sonar jobs. * ci: remove unused 'Fetch PR base branch' step from service unit tests Copilot review flagged that the step was using --depth=1 while the main checkout is also shallow, which would break any merge-base operation. On investigation, nothing downstream actually uses the base ref: the only command that runs after the checkout is 'mvn ... -Dsonar.skip=true', which has no git dependency. The step was preserved defensively in the previous commit, but it's dead code — cleanest fix is to delete it. |
||
|
Chirag Madlani
|
917a36c6a4
|
Potential fix for code scanning alert no. 1842: Artifact poisoning (#27220)
* Potential fix for code scanning alert no. 1842: Artifact poisoning Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> * Pin Yarn version to 1.22.18 to fix artifact poisoning alert Agent-Logs-Url: https://github.com/open-metadata/OpenMetadata/sessions/29aebdb5-eef0-4a2a-be01-489deef48d2b Co-authored-by: chirag-madlani <12962843+chirag-madlani@users.noreply.github.com> * Fix artifact poisoning in update-playwright-e2e-docs.yml: replace npm install -g yarn with pinned corepack Agent-Logs-Url: https://github.com/open-metadata/OpenMetadata/sessions/550fba5a-bb13-45da-a144-b67599c9eaa4 Co-authored-by: chirag-madlani <12962843+chirag-madlani@users.noreply.github.com> * Remove corepack prepare to eliminate artifact poisoning: use only corepack enable (bundled yarn) Agent-Logs-Url: https://github.com/open-metadata/OpenMetadata/sessions/90f6ed8d-3f2b-4c3d-9a34-cd1f57c4d89c Co-authored-by: chirag-madlani <12962843+chirag-madlani@users.noreply.github.com> --------- Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> |
||
|
Sriharsha Chintalapani
|
b2b49db75e
|
MSAL Token Renewal Fix — Safari Session Loss (#27214)
* MSAL Token Renewal Fix — Safari Session Loss * MSAL Token Renewal Fix — Safari Session Loss * MSAL Token Renewal Fix — Safari Session Loss * apply lint * MSAL Token Renewal Fix — OIDC fix * wait for token update * fix unit tests * Add SSO playwright tests * Add tests --------- Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com> |
||
|
Mohit Yadav
|
3ec31e3e68
|
Make OpeNMetadata Service Unit Test Required (#27099) |