Commit graph

290 commits

Author SHA1 Message Date
github-actions
81bae5cd39 chore(release): Prepare Branch for 1.12.2 2026-02-24 23:08:12 +00:00
github-actions
d52db56a85 chore(release): Prepare Branch for 1.12.1 2026-02-23 12:05:27 +00:00
ulixius9
c0c0b2b4e4 Merge branch 'main' into 1.12.0 2026-02-10 16:30:46 +05:30
Mohit Yadav
fa3b7b9305
[Search] Upgrade Clients (#25719)
* Upgrade Clients

* Update clients in docker files

* Fix Tests

* Fix integration test

* Fix Review Comments

* Fix More review comments :-
  1. ElasticSearchClient.java - Added keep-alive timeout configuration
  2. OpenSearchClient.java - Added keep-alive timeout configuration
  3. OpenMetadataOperations.java - Added logging for caught exception
  4. SigV4Hc5RequestSigningInterceptor.java - Now throws exception instead of silently returning

* Fix More review comments :-
  1. ElasticSearchClient.java - Added keep-alive timeout configuration
  2. OpenSearchClient.java - Added keep-alive timeout configuration
  3. OpenMetadataOperations.java - Added logging for caught exception
  4. SigV4Hc5RequestSigningInterceptor.java - Now throws exception instead of silently returning

Co-authored-by: mohityadav766 <mohityadav766@users.noreply.github.com>

* upgrade to 9.3.0 vs 3.4.0 server since earlier had bug

* fix version in pom

* Fix Review Comments

* FIX IAM OpenSearch FIx

---------

Co-authored-by: Gitar <noreply@gitar.ai>
Co-authored-by: mohityadav766 <mohityadav766@users.noreply.github.com>
2026-02-07 18:54:13 +05:30
github-actions
463bfd06e6 chore(release): Prepare Branch for 1.12.0 2026-02-03 16:49:03 +05:30
Pere Miquel Brull
7a3746c00f
FIX - Server passes secret prefixes to ingestion (#25527)
* FIX Query Runner - Server passes secret prefixes to ingestion

* FIX Query Runner - Server passes secret prefixes to ingestion
2026-01-28 10:35:13 +01:00
Mohit Yadav
0129f274ed
ReApply changes Fix Stats Issue and Add Tests (#25521)
* Fix Issue and Add Tests

* Update generated TypeScript types

* Fix CI jest failure

---------

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2026-01-26 21:10:23 +05:30
mohitdeuex
c006bdb2b0 Revert "Fix stats and Improve Search with Insights (#25495)"
This reverts commit 19725a7130.
2026-01-24 11:53:51 +05:30
Mohit Yadav
19725a7130
Fix stats and Improve Search with Insights (#25495)
* Fix Stats

* Add Warning logs and reindex failure analysis

* Add Search Insights in Preferences

* Add Label

* Fix Full Error not available

* Add check for reindex run
2026-01-24 10:27:46 +05:30
Sriharsha Chintalapani
89f627da81
Distributed Search Indexing with Push Notifications (#24939)
* Add Distributed Indexing in Multi-Server scenarios

* Add Distributed Indexing in Multi-Server scenarios

* Update generated TypeScript types

* Handle Servers leaving and joining

* Update generated TypeScript types

* spotless fix

* Refactor Code for Single Server and Multiple Server

* Add Metrics and Search Index Orphaned Cleanup

* Add Language

* Add Test settings

* Add Test data

* Add Test data

* Update generated TypeScript types

* Add Load Test for more entities

* Add Stats fix

* Add server information

* Fix Staging INdex unavailable to DistributedJobParticipant

* Fix Stats issue

* Align Tests

* Fix Stats and Error Handling

* participant stat fix

* Fix coordinator stats

* Add E2E failure tests

* Fix Stats for Reader and Sink

* Added flush for sinking stats

* Add language label

* Fix Entity Build Errors

* Missing commit

* Update generated TypeScript types

* Change runId to serverId

* Fix test failures

---------

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Mohit Yadav <105265192+mohityadav766@users.noreply.github.com>
Co-authored-by: mohitdeuex <mohit.y@deuexsolutions.com>
2026-01-23 06:12:05 +05:30
Pere Miquel Brull
fa4373054e
Finish K8sPipelineClient Implementation (#25172)
* config cleanup

* add missing configs

* fix auto pilot

* fix lifecycle

* fix logs and tests

* fix test

* move integration tests

* fix

* fix

* Address code review feedback

- Fix UsageWorkflowConfig to set stageFileLocation instead of queryLogFilePath
- Add error handling for parseInt in IngestionLogHandler to catch NumberFormatException

* fix

* fix lifecycle

* prepare cronOMJob

* remove PR target

* fix

* fix

* fix

* fix

* fix

* fix tests

* fix review

* fix review

* fix review

* fix

---------

Co-authored-by: Gitar <gitar@gitar.ai>
Co-authored-by: Gitar <noreply@gitar.ai>
Co-authored-by: pmbrull <pmbrull@users.noreply.github.com>
2026-01-15 08:17:55 +01:00
Sriharsha Chintalapani
f5cf3190c4
Add OpenSearch IAM auth; Add multi host listing capability in the existing config for search (#25204)
* Add OpenSearch IAM auth; Add multi host listing capability in the existing config for search

* Update generated TypeScript types

* Issue #22768: OpenSearch IAM auth; multi-host config

* Update generated TypeScript types

* Unify AWS config across different services

* Update generated TypeScript types

---------

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Mohit Yadav <105265192+mohityadav766@users.noreply.github.com>
2026-01-14 12:35:53 +05:30
Sriharsha Chintalapani
2c8a45d2a8
Upgrade to Dropwizard 5x and Jetty 12.1 (#24776)
* Add support for Dropwizard 5.0 and Jetty 12.1.x

* Dropwizard 5x and Jetty 12.1 upgrade

* Fix test behavior

* Fix rdf tests

* revert enableVirtualThreads

* fix tests

* Fix Tests

* Fix tests

* Switch to jersey-jetty-connector for Jetty 12 compatibility

- Replace jersey-apache-connector with jersey-jetty-connector
- Jersey 3.1.4+ jersey-jetty-connector supports Jetty 12.0.x+
- Use JettyConnectorProvider and JettyHttpClientSupplier for HTTP client
- Keep reasonable timeouts (30s connect, 2min read) to prevent CI hangs
- Set SYNC_LISTENER_RESPONSE_MAX_SIZE for large responses

This fixes the 1,093 InterruptedException test failures caused by
using the default Jersey client (HttpURLConnection-based) which doesn't
handle concurrent test execution properly.

* Fix: Start Jetty HttpClient before use

Jetty 12 HttpClient implements LifeCycle and must be explicitly
started with httpClient.start() before use. This fixes the 163
InterruptedException test failures.

* Fix: Force jetty-client to 12.1.1 for jersey-jetty-connector

jersey-jetty-connector brings transitive jetty-client:12.0.22 but
Dropwizard 5.0 uses Jetty 12.1.1. The ClientConnector.newTransport()
API changed between 12.0.x and 12.1.x, causing NoSuchMethodError.

Fix: Exclude transitive jetty-client and add explicit 12.1.x dependency.

* Use Java 11+ HttpClient connector for tests (jersey-jnh-connector)

Switch from the broken jersey-jetty-connector (incompatible with Jetty 12.1.x)
to jersey-jnh-connector which uses Java's built-in java.net.http.HttpClient.
This connector:
- Natively supports all HTTP methods including PATCH
- Works with Java 21
- No external dependencies required
- Avoids compatibility issues with Jetty versions

* Use Apache HttpClient 5.x connector for tests (jersey-apache5-connector)

Switch from jersey-jetty-connector (incompatible with Jetty 12.1.x)
to jersey-apache5-connector which uses Apache HttpClient 5.x.
This connector:
- Supports all HTTP methods including PATCH
- Lenient with empty PUT request bodies
- Has proper timeout support to prevent indefinite hangs
- Works with Jetty 12.1.x

* Fix  tests

* Fix  docker compose

* Fix tests

* Fix tests - make url compatible

* Add URL parsing

* Fix URL decode

* fix tests

* fix test

* fix tests

* Fix integration with new dropwizard-5x changes

---------

Co-authored-by: Karan Hotchandani <33024356+karanh37@users.noreply.github.com>
Co-authored-by: karanh37 <karanh37@gmail.com>
Co-authored-by: Mohit Yadav <105265192+mohityadav766@users.noreply.github.com>
2026-01-12 12:18:29 -08:00
Suman Maharana
2741d277ad
Fix Trivy scans (#24867)
* Fix Trivy scans

* remove comms

* fixes

* fixed incompatible changes

* revert dependency conflicts

* update airflow to 3.1.5

* fix airflow not showing debug logs

---------

Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
Co-authored-by: Keshav Mohta <68001229+keshavmohta09@users.noreply.github.com>
2025-12-19 16:27:12 +01:00
Sriharsha Chintalapani
e71715ad6c
Single RDF knowledge graph for all entities (#24839)
* Single RDF knowledge graph for all entities

* Fix RDF Resource Test

* fix test

* fix test

* Add support for TagLabel objects

---------

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
Co-authored-by: lautel <laura92cp2@gmail.com>
2025-12-18 16:33:15 +01:00
Pere Miquel Brull
6fdc3539bb
MINOR - Prepare extra validations for system repository health (#24846)
* MINOR - Prepare extra validations for system repository health

* Update generated TypeScript types

---------

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2025-12-18 07:37:37 +01:00
Karan Hotchandani
c8501f2f4f
preparing 1.12 branch (#24870) 2025-12-17 18:36:03 +05:30
Mayur Singal
cee0fd2c77
MINOR: Add option to disable ingestion in run_local_docker (#24676) 2025-12-03 12:35:29 +00:00
Teddy
11c2d2f6a9
MINOR - Airflow serialized limit (#24617)
* chore: set lower DAG serialization defaults

* chore: increase timeout for testSuite

* chore: update DAG processor interval postgres

* chore: lower DAG parse interval and delay

* fix: remove internal parsing and trigger dag parsing automatically on deploy
2025-12-01 11:09:19 +01:00
Mayur Singal
acb1be97f4
Fix #23096: Add Airflow 3.x support (#24338)
* Fix #23096: Add Airflow 3.x support

* airflow auth fixes

* fix airflow tests

* fix airflow 3 ingestion

* pyformat

* fix pytest

* pyformat

* bump version

* fix version

* fix mlflow

* custom pydoris

* fix airflow tests

* fix spotless

* final test fixs

* playwrite debug

* fix pytests

* checkstyle fix

* fit get status api and revert playwrite debug

* fix airflow version

---------

Co-authored-by: Ashish Gupta <ashish@getcollate.io>
2025-11-21 12:28:28 +01:00
Eugenio
f528616b2f
Identify when Airflow DAG import errors in CI (#23589) 2025-09-26 15:30:15 +02:00
Himanshu Khairajani
8c4cebea13
fix #21555: Automator - Separating terms and tags in action config (#22970)
* Separating terms and tags in action config

* Update generated TypeScript types

* add: migration files for separate tags and terms

* chore: java formatting

* yaml formatting

* Chore: updated the release number

* updated to v194 as per release cycle

* updated to v195 as per release cycle

---------

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: --global <--global>
2025-08-28 10:50:48 +02:00
Mohit Yadav
c0d7a574d7
chore(release): Prepare Branch for 1.10.0-SNAPSHOT (#23034)
Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
2025-08-21 21:43:01 +05:30
Pere Miquel Brull
dbe8d5ed00
CI - YAML formatting issue (#23047)
* CI

* CI
2025-08-21 18:05:15 +02:00
Sriharsha Chintalapani
a6d544a5d8
RDF Ontology, Json LD, DCAT vocabulary support by mapping OM Schemas to RDF (#22852)
* Support for RDF, SPARQL, SQL-TO-SPARQL

* Tests are working

* Add  RDF relations tests

* improve Knowledge Graph UI, tags , glossary term relations

* Lang translations

* Fix level depth querying

* Add semantic search interfaces , integration into search

* cleanup

* Update generated TypeScript types

* Fix styling

* remove duplicated ttl file

* model generator cleanup

* Update OM - DCAT vocab

* Update DataProduct Schema

* Improve JsonLD Translator

* Update generated TypeScript types

* Fix Tests

* Fix java checkstyle

* Add RDF workflows

* fix unit tests

* fix e2e

---------

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
2025-08-17 18:36:26 -07:00
Mohit Yadav
b92e9d0e06
chore(release): Prepare Branch for 1.9.0-SNAPSHOT (#22742)
Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
2025-08-04 20:00:25 +05:30
Mohit Yadav
0b2321e976
Added Session Age for Cookies (#22166)
* - Added Session Age for Cookies

* Make OIDC Session Expiry Configurable

* Update generated TypeScript types

* Updated Docker Files

* Update Session to 7 days

---------

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2025-07-08 15:07:52 +05:30
Sriharsha Chintalapani
d10178f050
Fix java 21 in docker (#21746) 2025-06-12 18:24:25 -07:00
Sriharsha Chintalapani
8bb055fc9e
Fix #21506: Upgrade to Java 21 (#21507)
* Fix #21506: Upgrade to Java 21

* Fix #1655: Upgrade to Java 21
2025-06-11 22:06:08 -07:00
Akash Jain
799e3ca900
chore(docker-compose): Bump indices.query.bool.max_clause_count=4096 (#21301) 2025-05-27 14:29:05 +02:00
Imri Paran
d91273a30d
Fix 20325: Trigger external apps with config (#20397)
* wip

* feat: trigger external apps with override config

- Added in openmetadata-airflow-apis functionality to trigger DAG with feature.
- Modified openmetadata-airflow-apis application runner to accept override config from params.
- Added overloaded runPipeline with `Map<String,Object> config` to allow triggering apps with configuration. We might want to expand this to all ingestion pipelines. For now its just for apps.
- Implemented an example external app that can be used to test functionality of external apps. The app can be enabled by setting the `ENABLE_APP_HelloPipelines=true` environment variable.

* fix class doc for application

* fixed README for airflow apis

* fixes

* set HelloPipelines to disabeld by default

* fixed basedpywright errros

* fixed app schema

* reduced airflow client runPipeline to an overload with null config
removed duplicate call to runPipeline in AppResource

* Update openmetadata-docs/content/v1.7.x-SNAPSHOT/developers/applications/index.md

Co-authored-by: Matias Puerta <matias@getcollate.io>

* deleted documentation file

---------

Co-authored-by: Matias Puerta <matias@getcollate.io>
2025-05-06 17:41:24 +07:00
Mohit Yadav
20f17a3367
Fixes #16062: Added prompt config to allow config (#20959)
* Fixes #16062
Make prompt=login as optional

* update null or empty
2025-04-25 08:37:25 +05:30
Ashish Gupta
73aaa34b75
update the snapshot to 1.8.0 (#20925) 2025-04-24 10:46:36 +05:30
Akash Jain
0f6d0523d8
feat: Bump Versions to 1.7.0-SNAPSHOT on Main Branch (#20847)
* feat: Bump Versions to 1.7.0-SNAPSHOT on Main Branch

* fix(script): Add a condition for "-SNAPSHOT" is version update script
2025-04-16 15:21:01 +05:30
Mohit Yadav
3a01ad7da5
[Fix-20125] OIDC: Allow max_age to be optional (#20721)
* Make Max Age Optional

* spotless fix
2025-04-09 15:09:57 +05:30
Mohit Yadav
c28f3274d1
Adds new param to docker files (#20338) 2025-03-19 18:13:22 +05:30
Pere Miquel Brull
69c9102da1
MINOR - Bump Ingestion versions (#19836)
* MINOR - Bump Ingestion versions

* MINOR - Bump Ingestion versions

* fix

* fix db_scheme for airflow +2.9.1

* fix
2025-02-18 07:56:46 +01:00
Ethan
48700ae9ea
Fixes #18075: Dockerfile lint warning (#18077)
* fix docker warning

* for running actions

---------

Co-authored-by: Akash Jain <15995028+akash-jain-10@users.noreply.github.com>
2025-02-04 15:28:36 +05:30
Chirag Madlani
a43835df32
Revert "fixes #18820: updated docker compose files (#18821)" (#19297)
This reverts commit 69dd8b99f9.
2025-01-09 15:34:07 +05:30
tarunpandey23
69dd8b99f9
fixes #18820: updated docker compose files (#18821) 2025-01-09 10:50:00 +05:30
Ethan
e708a3242e
feat: update version (#18259) 2024-10-17 16:18:37 -07:00
Imri Paran
be82086e25
MINOR: add column case sensitivity parameter (#18115)
* fix(data-quality): table diff

- added handling for case-insensitive columns
- added handling for different numeric types (int/float/Decimal)
- added handling of boolean test case parameters

* add migrations for table diff

* add migrations for table diff

* removed cross type diff for now. it appears to be flaky

* fixed migrations

* use casefold() instead of lower()

* - implemented utils.get_test_case_param_value
- fixed params for case sensitive column

* handle bool test case parameters

* format

* testing

* format

* list -> List

* list -> List

* - change caseSensitiveColumns default to fase
- added migration to stay backward compatible

* - removed migration files
- updated logging message for table diff migration

* changed bool test case parameters default to always be false

* format

* docs: data diff

- added the caseSensitiveColumns parameter

requires: https://github.com/open-metadata/OpenMetadata/pull/18115

* fixed test_get_bool_test_case_param
2024-10-15 16:29:43 +02:00
Chirag Madlani
b0563ccf98
revert quicktype bump since for CI issue (#17934)
* enable logging for debugging

* remove node-gyp

* add node-gyp globally before installing deps

* reduce quick type to 10

* revert quicktype and node-gyp changes for CI

* fix unit tests
2024-09-20 19:30:06 +05:30
Pere Miquel Brull
6a1cd0ef8b
GEN-1493 - Fix paginate_es in opensearch (#17858)
* GEN-1493 - Fix opensearch pagination

* GEN-1494 - Add CI for py-tests with Postgres and Opensearch

* GEN-1494 - Add CI for py-tests with Postgres and Opensearch
2024-09-17 14:21:10 +02:00
Prajwal214
1c90eaaf3d
Docs: Updated Domain in Docker Compose & Docs (#17603)
* Minor: Updating Domain in Docker Compose & Docs

* replace openmetadata to open-metadata

---------

Co-authored-by: Prajwal Pandit <prajwalpandit@Prajwals-MacBook-Air.local>
Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
Co-authored-by: Shailesh Parmar <shailesh.parmar.webdev@gmail.com>
2024-09-09 11:47:09 +05:30
Ayush Shah
9245c8965f
fix: Bump Ingestion and Dockerfile versions (#17624) 2024-08-29 12:03:30 +05:30
Mohit Yadav
05baba9dc1
Change Default Value of Batch Size and Payload Size (#17391) 2024-08-12 11:18:18 +05:30
Mohit Yadav
aa5b9b2985
Increase Payload Size to 100MIB for ELastic Search (#17366)
* Increase Payload Size to 100MIB

* Increase batchSize Size to 500

* Add Payload Size as option in openmetadataoperations , and searchIndexJob

* //Comment

* Add Logs
2024-08-11 21:58:55 +05:30
sonika-shah
8d5661a7bf
Support Elasticsearch cluster alias for E2E tests  (#17117)
* Fix backend tests and have index alias with clusterAlias appended

* Fix backend tests and have index alias with clusterAlias appended

* Fix failing tests

* alias setup

* fix suggestion not working due to alias

* fix getIndexOrAliasName method for multiple indexes

* update openmetadata.yaml

* update childAliases with clusterAlias

---------

Co-authored-by: Ashish Gupta <ashish@getcollate.io>
Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
2024-08-10 11:26:49 -07:00
Sriharsha Chintalapani
ed2115067a
fix: docker/development/Dockerfile to reduce vulnerabilities (#16855)
The following vulnerabilities are fixed with an upgrade:
- https://snyk.io/vuln/SNYK-ALPINE319-OPENSSL-7413523
- https://snyk.io/vuln/SNYK-ALPINE319-OPENSSL-7413523
- https://snyk.io/vuln/SNYK-ALPINE319-OPENSSL-7413527
- https://snyk.io/vuln/SNYK-ALPINE319-OPENSSL-7413527

Co-authored-by: snyk-bot <snyk-bot@snyk.io>
2024-07-31 21:26:51 +05:30