Commit graph

7 commits

Author SHA1 Message Date
Sriharsha Chintalapani
410c852f4a
Add Json Logging (#26357)
* Add Json Logging

* Fix comments

* Fix tests

* Centralize junit.platform.version in root pom

* Fix test-config-mcp.yaml - update to JSON logging

* Fix logback.xml to use LOG_LEVEL for backward compatibility

* Reverted to text format for test env  test-config-mcp.yaml

* Add the ability to switch between text/json logging

* Fix comments

* Fix json logging

* Address Comments

* Address Comments

---------

Co-authored-by: sonika-shah <58761340+sonika-shah@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: Mohit Yadav <105265192+mohityadav766@users.noreply.github.com>
2026-03-31 16:15:07 -07:00
Mohit Yadav
b59aa7fc44
Improve indexing (#26154)
* Add Prometheus metrics for reindexing pipeline via Micrometer                                                       Bridge the existing reindexing atomic counters to Prometheus so operators     can alert on failures, latency spikes, and backpressure without relying      solely on database-flushed stats.

  - Add ReindexingMetrics singleton (initialize/getInstance pattern matching
    CacheMetrics) with job lifecycle counters, stage success/failed/warnings
    counters, bulk request timers with SLA buckets, payload size distribution,
    backpressure and promotion counters, and active/pending gauges
  - Register in MicrometerBundle after StreamableLogsMetrics
  - Instrument ReindexingOrchestrator.run() with job started/completed/failed/stopped
  - Bridge StageStatsTracker.flush() deltas to Prometheus per stage and entity type
  - Add bulk request latency timer and payload size recording in OpenSearchBulkSink
  - Record backpressure events in SearchIndexExecutor.handleBackpressure()
  - Record promotion success/failure in DefaultRecreateHandler
  - Add ReindexingMetricsTest with 24 tests covering all metric types

* Add Improvements

* Auto Gene

* Use Auto Config in distributed

* Fix Partition Claim Spread

* Make partition use config

* Correct total count

* Fix Wait time to 5 mins

* Revert om yaml

* Fix Sink sync

* Add Failure Handling at different stages

* Update script to create entities

* Move to scripts

* Add usage and fix script

* Fix Script

* Update generated TypeScript types

* Fix Staging miss

* Fix Stats reconcilation issue

* Revert workflow handler

* Fix Partition worker early sync

* Update Logs

* Update logs EntityRepository

* Error failure test

* Review Comments fix

* Fix Non Distributed live feed

* Fix Non Distributed stats feed

* Fix Review comments

* Fix Time Series cutt off

* Update generated TypeScript types

* Md

* Benchmark addition

* Fix date time warning

* Update load test to do benchmark analysis

* Disagnostic and update perf test

* Move load test to bin

* Fix Review Comments

* Add numeric values

* Move to localhost by default

* Fix Perf test issues

* Review Comments

* Add Preflight Fixes

* Add Preflight fixes for stale entry

* Remove stale entry on ApplicationHandler

---------

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2026-03-03 16:39:27 +05:30
Mohit Yadav
fa3b7b9305
[Search] Upgrade Clients (#25719)
* Upgrade Clients

* Update clients in docker files

* Fix Tests

* Fix integration test

* Fix Review Comments

* Fix More review comments :-
  1. ElasticSearchClient.java - Added keep-alive timeout configuration
  2. OpenSearchClient.java - Added keep-alive timeout configuration
  3. OpenMetadataOperations.java - Added logging for caught exception
  4. SigV4Hc5RequestSigningInterceptor.java - Now throws exception instead of silently returning

* Fix More review comments :-
  1. ElasticSearchClient.java - Added keep-alive timeout configuration
  2. OpenSearchClient.java - Added keep-alive timeout configuration
  3. OpenMetadataOperations.java - Added logging for caught exception
  4. SigV4Hc5RequestSigningInterceptor.java - Now throws exception instead of silently returning

Co-authored-by: mohityadav766 <mohityadav766@users.noreply.github.com>

* upgrade to 9.3.0 vs 3.4.0 server since earlier had bug

* fix version in pom

* Fix Review Comments

* FIX IAM OpenSearch FIx

---------

Co-authored-by: Gitar <noreply@gitar.ai>
Co-authored-by: mohityadav766 <mohityadav766@users.noreply.github.com>
2026-02-07 18:54:13 +05:30
Mohit Yadav
0129f274ed
ReApply changes Fix Stats Issue and Add Tests (#25521)
* Fix Issue and Add Tests

* Update generated TypeScript types

* Fix CI jest failure

---------

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2026-01-26 21:10:23 +05:30
mohitdeuex
c006bdb2b0 Revert "Fix stats and Improve Search with Insights (#25495)"
This reverts commit 19725a7130.
2026-01-24 11:53:51 +05:30
Mohit Yadav
19725a7130
Fix stats and Improve Search with Insights (#25495)
* Fix Stats

* Add Warning logs and reindex failure analysis

* Add Search Insights in Preferences

* Add Label

* Fix Full Error not available

* Add check for reindex run
2026-01-24 10:27:46 +05:30
Sriharsha Chintalapani
89f627da81
Distributed Search Indexing with Push Notifications (#24939)
* Add Distributed Indexing in Multi-Server scenarios

* Add Distributed Indexing in Multi-Server scenarios

* Update generated TypeScript types

* Handle Servers leaving and joining

* Update generated TypeScript types

* spotless fix

* Refactor Code for Single Server and Multiple Server

* Add Metrics and Search Index Orphaned Cleanup

* Add Language

* Add Test settings

* Add Test data

* Add Test data

* Update generated TypeScript types

* Add Load Test for more entities

* Add Stats fix

* Add server information

* Fix Staging INdex unavailable to DistributedJobParticipant

* Fix Stats issue

* Align Tests

* Fix Stats and Error Handling

* participant stat fix

* Fix coordinator stats

* Add E2E failure tests

* Fix Stats for Reader and Sink

* Added flush for sinking stats

* Add language label

* Fix Entity Build Errors

* Missing commit

* Update generated TypeScript types

* Change runId to serverId

* Fix test failures

---------

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Mohit Yadav <105265192+mohityadav766@users.noreply.github.com>
Co-authored-by: mohitdeuex <mohit.y@deuexsolutions.com>
2026-01-23 06:12:05 +05:30