Commit graph

86 commits

Author SHA1 Message Date
Mohit Yadav
fa3b7b9305
[Search] Upgrade Clients (#25719)
* Upgrade Clients

* Update clients in docker files

* Fix Tests

* Fix integration test

* Fix Review Comments

* Fix More review comments :-
  1. ElasticSearchClient.java - Added keep-alive timeout configuration
  2. OpenSearchClient.java - Added keep-alive timeout configuration
  3. OpenMetadataOperations.java - Added logging for caught exception
  4. SigV4Hc5RequestSigningInterceptor.java - Now throws exception instead of silently returning

* Fix More review comments :-
  1. ElasticSearchClient.java - Added keep-alive timeout configuration
  2. OpenSearchClient.java - Added keep-alive timeout configuration
  3. OpenMetadataOperations.java - Added logging for caught exception
  4. SigV4Hc5RequestSigningInterceptor.java - Now throws exception instead of silently returning

Co-authored-by: mohityadav766 <mohityadav766@users.noreply.github.com>

* upgrade to 9.3.0 vs 3.4.0 server since earlier had bug

* fix version in pom

* Fix Review Comments

* FIX IAM OpenSearch FIx

---------

Co-authored-by: Gitar <noreply@gitar.ai>
Co-authored-by: mohityadav766 <mohityadav766@users.noreply.github.com>
2026-02-07 18:54:13 +05:30
Pere Miquel Brull
7a3746c00f
FIX - Server passes secret prefixes to ingestion (#25527)
* FIX Query Runner - Server passes secret prefixes to ingestion

* FIX Query Runner - Server passes secret prefixes to ingestion
2026-01-28 10:35:13 +01:00
Mohit Yadav
0129f274ed
ReApply changes Fix Stats Issue and Add Tests (#25521)
* Fix Issue and Add Tests

* Update generated TypeScript types

* Fix CI jest failure

---------

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2026-01-26 21:10:23 +05:30
mohitdeuex
c006bdb2b0 Revert "Fix stats and Improve Search with Insights (#25495)"
This reverts commit 19725a7130.
2026-01-24 11:53:51 +05:30
Mohit Yadav
19725a7130
Fix stats and Improve Search with Insights (#25495)
* Fix Stats

* Add Warning logs and reindex failure analysis

* Add Search Insights in Preferences

* Add Label

* Fix Full Error not available

* Add check for reindex run
2026-01-24 10:27:46 +05:30
Sriharsha Chintalapani
89f627da81
Distributed Search Indexing with Push Notifications (#24939)
* Add Distributed Indexing in Multi-Server scenarios

* Add Distributed Indexing in Multi-Server scenarios

* Update generated TypeScript types

* Handle Servers leaving and joining

* Update generated TypeScript types

* spotless fix

* Refactor Code for Single Server and Multiple Server

* Add Metrics and Search Index Orphaned Cleanup

* Add Language

* Add Test settings

* Add Test data

* Add Test data

* Update generated TypeScript types

* Add Load Test for more entities

* Add Stats fix

* Add server information

* Fix Staging INdex unavailable to DistributedJobParticipant

* Fix Stats issue

* Align Tests

* Fix Stats and Error Handling

* participant stat fix

* Fix coordinator stats

* Add E2E failure tests

* Fix Stats for Reader and Sink

* Added flush for sinking stats

* Add language label

* Fix Entity Build Errors

* Missing commit

* Update generated TypeScript types

* Change runId to serverId

* Fix test failures

---------

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Mohit Yadav <105265192+mohityadav766@users.noreply.github.com>
Co-authored-by: mohitdeuex <mohit.y@deuexsolutions.com>
2026-01-23 06:12:05 +05:30
Pere Miquel Brull
fa4373054e
Finish K8sPipelineClient Implementation (#25172)
* config cleanup

* add missing configs

* fix auto pilot

* fix lifecycle

* fix logs and tests

* fix test

* move integration tests

* fix

* fix

* Address code review feedback

- Fix UsageWorkflowConfig to set stageFileLocation instead of queryLogFilePath
- Add error handling for parseInt in IngestionLogHandler to catch NumberFormatException

* fix

* fix lifecycle

* prepare cronOMJob

* remove PR target

* fix

* fix

* fix

* fix

* fix

* fix tests

* fix review

* fix review

* fix review

* fix

---------

Co-authored-by: Gitar <gitar@gitar.ai>
Co-authored-by: Gitar <noreply@gitar.ai>
Co-authored-by: pmbrull <pmbrull@users.noreply.github.com>
2026-01-15 08:17:55 +01:00
Sriharsha Chintalapani
f5cf3190c4
Add OpenSearch IAM auth; Add multi host listing capability in the existing config for search (#25204)
* Add OpenSearch IAM auth; Add multi host listing capability in the existing config for search

* Update generated TypeScript types

* Issue #22768: OpenSearch IAM auth; multi-host config

* Update generated TypeScript types

* Unify AWS config across different services

* Update generated TypeScript types

---------

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Mohit Yadav <105265192+mohityadav766@users.noreply.github.com>
2026-01-14 12:35:53 +05:30
Sriharsha Chintalapani
2c8a45d2a8
Upgrade to Dropwizard 5x and Jetty 12.1 (#24776)
* Add support for Dropwizard 5.0 and Jetty 12.1.x

* Dropwizard 5x and Jetty 12.1 upgrade

* Fix test behavior

* Fix rdf tests

* revert enableVirtualThreads

* fix tests

* Fix Tests

* Fix tests

* Switch to jersey-jetty-connector for Jetty 12 compatibility

- Replace jersey-apache-connector with jersey-jetty-connector
- Jersey 3.1.4+ jersey-jetty-connector supports Jetty 12.0.x+
- Use JettyConnectorProvider and JettyHttpClientSupplier for HTTP client
- Keep reasonable timeouts (30s connect, 2min read) to prevent CI hangs
- Set SYNC_LISTENER_RESPONSE_MAX_SIZE for large responses

This fixes the 1,093 InterruptedException test failures caused by
using the default Jersey client (HttpURLConnection-based) which doesn't
handle concurrent test execution properly.

* Fix: Start Jetty HttpClient before use

Jetty 12 HttpClient implements LifeCycle and must be explicitly
started with httpClient.start() before use. This fixes the 163
InterruptedException test failures.

* Fix: Force jetty-client to 12.1.1 for jersey-jetty-connector

jersey-jetty-connector brings transitive jetty-client:12.0.22 but
Dropwizard 5.0 uses Jetty 12.1.1. The ClientConnector.newTransport()
API changed between 12.0.x and 12.1.x, causing NoSuchMethodError.

Fix: Exclude transitive jetty-client and add explicit 12.1.x dependency.

* Use Java 11+ HttpClient connector for tests (jersey-jnh-connector)

Switch from the broken jersey-jetty-connector (incompatible with Jetty 12.1.x)
to jersey-jnh-connector which uses Java's built-in java.net.http.HttpClient.
This connector:
- Natively supports all HTTP methods including PATCH
- Works with Java 21
- No external dependencies required
- Avoids compatibility issues with Jetty versions

* Use Apache HttpClient 5.x connector for tests (jersey-apache5-connector)

Switch from jersey-jetty-connector (incompatible with Jetty 12.1.x)
to jersey-apache5-connector which uses Apache HttpClient 5.x.
This connector:
- Supports all HTTP methods including PATCH
- Lenient with empty PUT request bodies
- Has proper timeout support to prevent indefinite hangs
- Works with Jetty 12.1.x

* Fix  tests

* Fix  docker compose

* Fix tests

* Fix tests - make url compatible

* Add URL parsing

* Fix URL decode

* fix tests

* fix test

* fix tests

* Fix integration with new dropwizard-5x changes

---------

Co-authored-by: Karan Hotchandani <33024356+karanh37@users.noreply.github.com>
Co-authored-by: karanh37 <karanh37@gmail.com>
Co-authored-by: Mohit Yadav <105265192+mohityadav766@users.noreply.github.com>
2026-01-12 12:18:29 -08:00
Suman Maharana
2741d277ad
Fix Trivy scans (#24867)
* Fix Trivy scans

* remove comms

* fixes

* fixed incompatible changes

* revert dependency conflicts

* update airflow to 3.1.5

* fix airflow not showing debug logs

---------

Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
Co-authored-by: Keshav Mohta <68001229+keshavmohta09@users.noreply.github.com>
2025-12-19 16:27:12 +01:00
Sriharsha Chintalapani
e71715ad6c
Single RDF knowledge graph for all entities (#24839)
* Single RDF knowledge graph for all entities

* Fix RDF Resource Test

* fix test

* fix test

* Add support for TagLabel objects

---------

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
Co-authored-by: lautel <laura92cp2@gmail.com>
2025-12-18 16:33:15 +01:00
Pere Miquel Brull
6fdc3539bb
MINOR - Prepare extra validations for system repository health (#24846)
* MINOR - Prepare extra validations for system repository health

* Update generated TypeScript types

---------

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2025-12-18 07:37:37 +01:00
Teddy
11c2d2f6a9
MINOR - Airflow serialized limit (#24617)
* chore: set lower DAG serialization defaults

* chore: increase timeout for testSuite

* chore: update DAG processor interval postgres

* chore: lower DAG parse interval and delay

* fix: remove internal parsing and trigger dag parsing automatically on deploy
2025-12-01 11:09:19 +01:00
Mayur Singal
acb1be97f4
Fix #23096: Add Airflow 3.x support (#24338)
* Fix #23096: Add Airflow 3.x support

* airflow auth fixes

* fix airflow tests

* fix airflow 3 ingestion

* pyformat

* fix pytest

* pyformat

* bump version

* fix version

* fix mlflow

* custom pydoris

* fix airflow tests

* fix spotless

* final test fixs

* playwrite debug

* fix pytests

* checkstyle fix

* fit get status api and revert playwrite debug

* fix airflow version

---------

Co-authored-by: Ashish Gupta <ashish@getcollate.io>
2025-11-21 12:28:28 +01:00
Sriharsha Chintalapani
a6d544a5d8
RDF Ontology, Json LD, DCAT vocabulary support by mapping OM Schemas to RDF (#22852)
* Support for RDF, SPARQL, SQL-TO-SPARQL

* Tests are working

* Add  RDF relations tests

* improve Knowledge Graph UI, tags , glossary term relations

* Lang translations

* Fix level depth querying

* Add semantic search interfaces , integration into search

* cleanup

* Update generated TypeScript types

* Fix styling

* remove duplicated ttl file

* model generator cleanup

* Update OM - DCAT vocab

* Update DataProduct Schema

* Improve JsonLD Translator

* Update generated TypeScript types

* Fix Tests

* Fix java checkstyle

* Add RDF workflows

* fix unit tests

* fix e2e

---------

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
2025-08-17 18:36:26 -07:00
Mohit Yadav
0b2321e976
Added Session Age for Cookies (#22166)
* - Added Session Age for Cookies

* Make OIDC Session Expiry Configurable

* Update generated TypeScript types

* Updated Docker Files

* Update Session to 7 days

---------

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2025-07-08 15:07:52 +05:30
Sriharsha Chintalapani
8bb055fc9e
Fix #21506: Upgrade to Java 21 (#21507)
* Fix #21506: Upgrade to Java 21

* Fix #1655: Upgrade to Java 21
2025-06-11 22:06:08 -07:00
Akash Jain
799e3ca900
chore(docker-compose): Bump indices.query.bool.max_clause_count=4096 (#21301) 2025-05-27 14:29:05 +02:00
Imri Paran
d91273a30d
Fix 20325: Trigger external apps with config (#20397)
* wip

* feat: trigger external apps with override config

- Added in openmetadata-airflow-apis functionality to trigger DAG with feature.
- Modified openmetadata-airflow-apis application runner to accept override config from params.
- Added overloaded runPipeline with `Map<String,Object> config` to allow triggering apps with configuration. We might want to expand this to all ingestion pipelines. For now its just for apps.
- Implemented an example external app that can be used to test functionality of external apps. The app can be enabled by setting the `ENABLE_APP_HelloPipelines=true` environment variable.

* fix class doc for application

* fixed README for airflow apis

* fixes

* set HelloPipelines to disabeld by default

* fixed basedpywright errros

* fixed app schema

* reduced airflow client runPipeline to an overload with null config
removed duplicate call to runPipeline in AppResource

* Update openmetadata-docs/content/v1.7.x-SNAPSHOT/developers/applications/index.md

Co-authored-by: Matias Puerta <matias@getcollate.io>

* deleted documentation file

---------

Co-authored-by: Matias Puerta <matias@getcollate.io>
2025-05-06 17:41:24 +07:00
Mohit Yadav
20f17a3367
Fixes #16062: Added prompt config to allow config (#20959)
* Fixes #16062
Make prompt=login as optional

* update null or empty
2025-04-25 08:37:25 +05:30
Mohit Yadav
3a01ad7da5
[Fix-20125] OIDC: Allow max_age to be optional (#20721)
* Make Max Age Optional

* spotless fix
2025-04-09 15:09:57 +05:30
Mohit Yadav
c28f3274d1
Adds new param to docker files (#20338) 2025-03-19 18:13:22 +05:30
Pere Miquel Brull
69c9102da1
MINOR - Bump Ingestion versions (#19836)
* MINOR - Bump Ingestion versions

* MINOR - Bump Ingestion versions

* fix

* fix db_scheme for airflow +2.9.1

* fix
2025-02-18 07:56:46 +01:00
Chirag Madlani
a43835df32
Revert "fixes #18820: updated docker compose files (#18821)" (#19297)
This reverts commit 69dd8b99f9.
2025-01-09 15:34:07 +05:30
tarunpandey23
69dd8b99f9
fixes #18820: updated docker compose files (#18821) 2025-01-09 10:50:00 +05:30
Ethan
e708a3242e
feat: update version (#18259) 2024-10-17 16:18:37 -07:00
Pere Miquel Brull
6a1cd0ef8b
GEN-1493 - Fix paginate_es in opensearch (#17858)
* GEN-1493 - Fix opensearch pagination

* GEN-1494 - Add CI for py-tests with Postgres and Opensearch

* GEN-1494 - Add CI for py-tests with Postgres and Opensearch
2024-09-17 14:21:10 +02:00
Prajwal214
1c90eaaf3d
Docs: Updated Domain in Docker Compose & Docs (#17603)
* Minor: Updating Domain in Docker Compose & Docs

* replace openmetadata to open-metadata

---------

Co-authored-by: Prajwal Pandit <prajwalpandit@Prajwals-MacBook-Air.local>
Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
Co-authored-by: Shailesh Parmar <shailesh.parmar.webdev@gmail.com>
2024-09-09 11:47:09 +05:30
Mohit Yadav
05baba9dc1
Change Default Value of Batch Size and Payload Size (#17391) 2024-08-12 11:18:18 +05:30
Mohit Yadav
aa5b9b2985
Increase Payload Size to 100MIB for ELastic Search (#17366)
* Increase Payload Size to 100MIB

* Increase batchSize Size to 500

* Add Payload Size as option in openmetadataoperations , and searchIndexJob

* //Comment

* Add Logs
2024-08-11 21:58:55 +05:30
sonika-shah
8d5661a7bf
Support Elasticsearch cluster alias for E2E tests  (#17117)
* Fix backend tests and have index alias with clusterAlias appended

* Fix backend tests and have index alias with clusterAlias appended

* Fix failing tests

* alias setup

* fix suggestion not working due to alias

* fix getIndexOrAliasName method for multiple indexes

* update openmetadata.yaml

* update childAliases with clusterAlias

---------

Co-authored-by: Ashish Gupta <ashish@getcollate.io>
Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
2024-08-10 11:26:49 -07:00
Sriharsha Chintalapani
ed2115067a
fix: docker/development/Dockerfile to reduce vulnerabilities (#16855)
The following vulnerabilities are fixed with an upgrade:
- https://snyk.io/vuln/SNYK-ALPINE319-OPENSSL-7413523
- https://snyk.io/vuln/SNYK-ALPINE319-OPENSSL-7413523
- https://snyk.io/vuln/SNYK-ALPINE319-OPENSSL-7413527
- https://snyk.io/vuln/SNYK-ALPINE319-OPENSSL-7413527

Co-authored-by: snyk-bot <snyk-bot@snyk.io>
2024-07-31 21:26:51 +05:30
k.nakagaki
9a31a35296
Fixes 9875: supporting gcp secret manager (#16505)
* Split ExternalSecretsManagerTest to new ExternalSecretsManagerTest and AWSBasedSecretsManagerTest

* implement SecretsManagerFactory to create GCPSecretsManager

* implement GCPSecretsManager

* implements gcp secret manager.

* Fix it for the GCP's rule.

* create a template of GCP

* fix compile error

* implements to use project_id

* add library for the google cloud secret manager

* add test code for using google credential in the docker container

* modify docker-compose.yml for GCP

* add google_crc32c module

* modify ways to get project id

* create a new docker-compose.yml for Google Cloud

* create a new document

* create compose file for gcp secret manager

* fix invalid styles and formats

* downgrade google library to avoid conflicting protoc versions
2024-06-28 21:09:02 -07:00
Mohit Yadav
37e1472fa3
Add Jwt Claim Mapping Env (#16780) 2024-06-24 23:15:19 +05:30
Mohit Yadav
a5295396bd
Add WebConf Env to Compose (#16713) 2024-06-19 12:34:33 +05:30
Shailesh Parmar
1d4e3e0825
Revert alias setup (#16610)
* Revert "fix(CI): update `run_local_docker.sh` script with cluster alias ES endpoint (#16604)"

This reverts commit baab52076f.

* Revert "chore(CI): Enable Elasticsearch cluster alias by default (#16599)"

This reverts commit a3524e14d3.
2024-06-11 19:14:25 +05:30
Akash Jain
a3524e14d3
chore(CI): Enable Elasticsearch cluster alias by default (#16599) 2024-06-11 11:34:24 +05:30
sonika-shah
c010939072
Added payload size check logic for Elasticsearch (#15540)
* Added payload size check logic for Elasticsearch

* Added payload size check logic for Elasticsearch - made review changes

* Added payload size check logic for Elasticsearch - added env in docker files

* Optimize payload logic, fix search application error stats issue
2024-04-15 16:25:06 +05:30
Shin-ichi Hashiba
ba7b5d248a
delete duplicate env (#15709) 2024-04-01 18:21:51 +05:30
Mohit Yadav
e8f8271b77
- Add docker vars (#15619)
- Modified Azure refresh token logic
2024-03-20 11:47:25 +05:30
Imri Paran
aade838020
Fixes #15388: Use native backup tools (#15393)
* feat: use native backup tools

1. added mysqldump 8.3 to the ingestion container.
2. documented how to use native tools to back up and restore.
3. added deprecated message on the cli backup and restore.

* added deprecation notice for 1.3 backup

* removed 1.3.x deprecation notice

* added another backup page in 1.3 introducing SQL dump tools

* added --set-gtid-purged=OFF to the mysql dump process
2024-03-12 06:23:05 +01:00
Pere Miquel Brull
62c0cc7563
#13985 - Azure KV Secrets Manager (#15192)
* #13985 - Azure KV Secrets Manager

* Format

* #13985 - Azure KV Secrets Manager

* #13985 - Azure KV Secrets Manager

* Simplify credentials loading

* Simplify credentials loading

* Simplify credentials loading
2024-02-20 07:18:35 +01:00
Akash Jain
065a8e30da
fix: docker-compose failed randomly without healthcheck of elasticsearch. #14854 (#15034) 2024-02-05 18:28:00 +05:30
Akash Jain
8574e9a742
feat(docker-compose): Use openmetadata-ops.sh script for migrations (#14990) 2024-02-05 11:22:53 +05:30
Akash Jain
69e6f50769
chore(docker-compose): fix DB_PARAMS for Docker Compose files (#14931) 2024-01-29 19:28:07 +05:30
Dhruvin Maniar
8b5faae720
minor: fix alpine package to avoid vulnerabilities (#14926) 2024-01-29 11:33:24 +00:00
Akash Jain
cb893cdd65
fix(CI): Optimize Server Docker Image Layers (#14835) 2024-01-24 16:03:18 +05:30
Akash Jain
07eb1ccdf5
fix: Add SAML and LDAP env configs; disabled by default (#14453) 2023-12-25 18:08:38 -08:00
Pere Miquel Brull
d8984d267e
#14340 & #13849 - Clean secret ID and improve encrypt/decrypt exception management (#14356)
* Fix supported characters in SM

* Update SM

* Fixes

* Fixes

* Improve class conversion exceptions

* Comments

* Rename noop to db secrets manager providee

* Update sm

* Fix

* db SM

* db SM

* Fix test

* UI

* Update openmetadata-ui/src/main/resources/ui/src/mocks/IngestionListTable.mock.ts

* update default
2023-12-18 06:43:20 +01:00
Akash Jain
6eefd51180
fix: AIRFLOW_DB_PROPERTIES default value in docker compose files (#14274) 2023-12-06 19:56:19 +05:30