Commit graph

4 commits

Author SHA1 Message Date
Pere Miquel Brull
cfd71e8bd3
Fix k8s operator exit handler pod loop and TTL cleanup, add tolerations (#26971)
* Fix k8s operator exit handler pod loop and TTL cleanup, add tolerations support (#26772)

Fix two bugs in the OMJob operator:
- Exit handler pods were recreated indefinitely because findExitHandlerPod()
  lacked the name-based fallback that findMainPod() already had, causing
  label propagation delays to trigger repeated pod creation events
- Terminal phase handler never rescheduled for TTL-based cleanup, so pods
  were never cleaned up after ttlSecondsAfterFinished expired

Add tolerations support for ingestion pod scheduling across the full stack:
- Operator: OMJobPodSpec field, PodManager.buildPod(), CRD schema
- Server: OMJob model, K8sPipelineClientConfig parsing, K8sPipelineClient
  builder, K8sJobUtils serialization

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Add K8S_TOLERATIONS env var mapping in openmetadata.yaml

Adds the tolerations config binding so the server picks up the
K8S_TOLERATIONS env var set by the Helm chart secret.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Add tolerations to k8s test values for local validation

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix cleanup

* Address PR review: remove redundant pod lookup and guard null items

- Remove redundant server-created pod selector fallback in findMainPod()
  since buildPodSelector() now matches all pods by omjob-name alone
- Add null guard for getItems() in deletePods() to prevent NPE
- Update local test values for namespace and image config

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-14 09:42:54 +02:00
Mohit Yadav
fa3b7b9305
[Search] Upgrade Clients (#25719)
* Upgrade Clients

* Update clients in docker files

* Fix Tests

* Fix integration test

* Fix Review Comments

* Fix More review comments :-
  1. ElasticSearchClient.java - Added keep-alive timeout configuration
  2. OpenSearchClient.java - Added keep-alive timeout configuration
  3. OpenMetadataOperations.java - Added logging for caught exception
  4. SigV4Hc5RequestSigningInterceptor.java - Now throws exception instead of silently returning

* Fix More review comments :-
  1. ElasticSearchClient.java - Added keep-alive timeout configuration
  2. OpenSearchClient.java - Added keep-alive timeout configuration
  3. OpenMetadataOperations.java - Added logging for caught exception
  4. SigV4Hc5RequestSigningInterceptor.java - Now throws exception instead of silently returning

Co-authored-by: mohityadav766 <mohityadav766@users.noreply.github.com>

* upgrade to 9.3.0 vs 3.4.0 server since earlier had bug

* fix version in pom

* Fix Review Comments

* FIX IAM OpenSearch FIx

---------

Co-authored-by: Gitar <noreply@gitar.ai>
Co-authored-by: mohityadav766 <mohityadav766@users.noreply.github.com>
2026-02-07 18:54:13 +05:30
Pere Miquel Brull
7a3746c00f
FIX - Server passes secret prefixes to ingestion (#25527)
* FIX Query Runner - Server passes secret prefixes to ingestion

* FIX Query Runner - Server passes secret prefixes to ingestion
2026-01-28 10:35:13 +01:00
Pere Miquel Brull
fa4373054e
Finish K8sPipelineClient Implementation (#25172)
* config cleanup

* add missing configs

* fix auto pilot

* fix lifecycle

* fix logs and tests

* fix test

* move integration tests

* fix

* fix

* Address code review feedback

- Fix UsageWorkflowConfig to set stageFileLocation instead of queryLogFilePath
- Add error handling for parseInt in IngestionLogHandler to catch NumberFormatException

* fix

* fix lifecycle

* prepare cronOMJob

* remove PR target

* fix

* fix

* fix

* fix

* fix

* fix tests

* fix review

* fix review

* fix review

* fix

---------

Co-authored-by: Gitar <gitar@gitar.ai>
Co-authored-by: Gitar <noreply@gitar.ai>
Co-authored-by: pmbrull <pmbrull@users.noreply.github.com>
2026-01-15 08:17:55 +01:00