OpenMetadata is a unified metadata platform for data discovery, data observability, and data governance powered by a central metadata repository, in-depth column level lineage, and seamless team collaboration.
Find a file
Mohit Yadav b59aa7fc44
Improve indexing (#26154)
* Add Prometheus metrics for reindexing pipeline via Micrometer                                                       Bridge the existing reindexing atomic counters to Prometheus so operators     can alert on failures, latency spikes, and backpressure without relying      solely on database-flushed stats.

  - Add ReindexingMetrics singleton (initialize/getInstance pattern matching
    CacheMetrics) with job lifecycle counters, stage success/failed/warnings
    counters, bulk request timers with SLA buckets, payload size distribution,
    backpressure and promotion counters, and active/pending gauges
  - Register in MicrometerBundle after StreamableLogsMetrics
  - Instrument ReindexingOrchestrator.run() with job started/completed/failed/stopped
  - Bridge StageStatsTracker.flush() deltas to Prometheus per stage and entity type
  - Add bulk request latency timer and payload size recording in OpenSearchBulkSink
  - Record backpressure events in SearchIndexExecutor.handleBackpressure()
  - Record promotion success/failure in DefaultRecreateHandler
  - Add ReindexingMetricsTest with 24 tests covering all metric types

* Add Improvements

* Auto Gene

* Use Auto Config in distributed

* Fix Partition Claim Spread

* Make partition use config

* Correct total count

* Fix Wait time to 5 mins

* Revert om yaml

* Fix Sink sync

* Add Failure Handling at different stages

* Update script to create entities

* Move to scripts

* Add usage and fix script

* Fix Script

* Update generated TypeScript types

* Fix Staging miss

* Fix Stats reconcilation issue

* Revert workflow handler

* Fix Partition worker early sync

* Update Logs

* Update logs EntityRepository

* Error failure test

* Review Comments fix

* Fix Non Distributed live feed

* Fix Non Distributed stats feed

* Fix Review comments

* Fix Time Series cutt off

* Update generated TypeScript types

* Md

* Benchmark addition

* Fix date time warning

* Update load test to do benchmark analysis

* Disagnostic and update perf test

* Move load test to bin

* Fix Review Comments

* Add numeric values

* Move to localhost by default

* Fix Perf test issues

* Review Comments

* Add Preflight Fixes

* Add Preflight fixes for stale entry

* Remove stale entry on ApplicationHandler

---------

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2026-03-03 16:39:27 +05:30
.claude/skills TEST - Add Data Contract ODCS tests (#25588) 2026-02-04 15:10:39 +01:00
.github ISSUE #20036 - sqlalchemy 2.0 migration (#26031) 2026-03-02 13:07:47 -08:00
bin Improve indexing (#26154) 2026-03-03 16:39:27 +05:30
bootstrap Fix #26178: Add support for IAM auth for redshift (#26179) 2026-03-02 21:57:28 +05:30
common preparing 1.12 branch (#24870) 2025-12-17 18:36:03 +05:30
conf Add Semantic Search core to OSS (#25792) 2026-02-12 10:01:28 +01:00
docker Improve indexing (#26154) 2026-03-03 16:39:27 +05:30
docs Feature/search indexing issues (#25594) 2026-01-29 18:50:39 +05:30
examples/python-sdk/data-quality Create documentation resources for Data Quality as Code (closes #23800) (#24169) 2025-11-11 10:25:42 +00:00
ingestion Add data sanity checks for auto classification to prevent OOM (#25977) 2026-03-03 04:51:34 +00:00
openmetadata-airflow-apis ISSUE #20036 - sqlalchemy 2.0 migration (#26031) 2026-03-02 13:07:47 -08:00
openmetadata-clients chore(deps): bump org.mozilla:rhino (#24681) 2026-01-08 16:11:07 +05:30
openmetadata-dist preparing 1.12 branch (#24870) 2025-12-17 18:36:03 +05:30
openmetadata-integration-tests Fix #19596: Messaging Service Connection can be accessed via API for Without Access (#26196) 2026-03-03 01:02:39 -08:00
openmetadata-k8s-operator MINOR - Add Operator Tests (#25343) 2026-01-19 14:20:37 +01:00
openmetadata-mcp add MCP tool integration tests (#26146) 2026-02-28 04:34:46 +05:30
openmetadata-sdk Improvement #26033: StorageServices missing form the entities SDK in the python client (#26164) 2026-03-02 07:04:55 -08:00
openmetadata-service Improve indexing (#26154) 2026-03-03 16:39:27 +05:30
openmetadata-shaded-deps Reduced version to 3.4 (#26017) 2026-02-20 19:28:21 +05:30
openmetadata-spec Improve indexing (#26154) 2026-03-03 16:39:27 +05:30
openmetadata-ui Improve indexing (#26154) 2026-03-03 16:39:27 +05:30
openmetadata-ui-core-components Chore(deps-dev): Bump storybook (#26134) 2026-03-02 20:44:41 +00:00
scripts CI - Slack link monitor w/ playwright (#25641) 2026-01-30 10:23:52 +01:00
.git-blame-ignore-revs Minor: update git-blmae-ignore-revs, and uncomment ClassificationResourceTest tests code (#14431) 2023-12-18 19:16:29 -08:00
.gitignore Skip and warn when autoclassification values are too long (#25660) 2026-02-02 14:59:25 +01:00
.nojekyll shahsank3t published a site update 2021-08-04 06:23:29 +00:00
.pre-commit-config.yaml feature/pii-processor-improvement (#21248) 2025-05-19 17:52:17 +00:00
.pylintrc ISSUE #21101 - Implement BQ Partitioned Tests (#21348) 2025-05-22 17:22:05 +02:00
.snyk Ignore _openmetadata_testutils from snyk (#21168) 2025-05-13 18:01:05 +05:30
APPLICATION.md ISSUE #1534 - Profiler Refactor for Metadata Extraction Application (#23200) 2025-09-05 13:07:04 +02:00
CLAUDE.md Fix #26178: Add support for IAM auth for redshift (#26179) 2026-03-02 21:57:28 +05:30
CODE_OF_CONDUCT.md Fix #412 - Add code of conduct for OpenMetadata community 2021-09-06 18:57:17 -07:00
CONTRIBUTING.md addded more detail on issue creation in contributors page (#16583) 2024-06-09 14:02:36 -07:00
generate_ts.sh Feature: Generate TS From JSON (#19823) 2025-02-25 18:18:02 +05:30
INCIDENT_RESPONSE.md Add threat model and incident response (#23603) 2025-09-28 13:17:23 -07:00
LICENSE OpenMetadata snapshot release 0.3 2021-08-01 14:27:44 -07:00
Makefile MINOR: Add OpenAPI version update functionality in Makefile and script (#24604) 2026-01-14 14:11:56 +05:30
NOTICE OpenMetadata snapshot release 0.3 2021-08-01 14:27:44 -07:00
package.json chore(ui): bump quicktype to resolve vulnerabilities (#17979) 2024-09-25 15:09:34 +05:30
pom.xml Security vulnerability fixes for 1.12.0 (#26030) 2026-02-21 21:16:15 +01:00
README.md Update README.md for column-level consistency (#24670) 2025-12-03 07:59:18 -08:00
SECURITY.md Update vulnerability reporting instructions in SECURITY.md (#25651) 2026-01-30 14:03:09 -08:00
tests.txt Implement Modern Fluent API Pattern for OpenMetadata Java Client (#23239) 2025-09-29 16:07:02 -07:00
THREAT_MODEL.md Add threat model and incident response (#23603) 2025-09-28 13:17:23 -07:00
yarn.lock Chore(deps): Bump minimatch from 3.1.2 to 3.1.5 (#26157) 2026-03-03 06:22:12 +00:00



Logo

Empower your Data Journey with OpenMetadata

Commit Activity Release

What is OpenMetadata?

OpenMetadata is a unified metadata platform for data discovery, data observability, and data governance powered by a central metadata repository, in-depth column-level lineage, and seamless team collaboration. It is one of the fastest-growing open-source projects with a vibrant community and adoption by a diverse set of companies in a variety of industry verticals. Based on Open Metadata Standards and APIs, supporting connectors to a wide range of data services, OpenMetadata enables end-to-end metadata management, giving you the freedom to unlock the value of your data assets.


Contents:

OpenMetadata Consists of Four Main Components:

  • Metadata Schemas: These are the core definitions and vocabulary for metadata based on common abstractions and types. They also allow for custom extensions and properties to suit different use cases and domains.
  • Metadata Store: This is the central repository for storing and managing the metadata graph, which connects data assets, users, and tool-generated metadata in a unified way.
  • Metadata APIs: These are the interfaces for producing and consuming metadata, built on top of the metadata schemas. They enable seamless integration of user interfaces and tools, systems, and services with the metadata store.
  • Ingestion Framework: This is a pluggable framework for ingesting metadata from various sources and tools to the metadata store. It supports about 84+ connectors for data warehouses, databases, dashboard services, messaging services, pipeline services, and more.

Key Features of OpenMetadata

Data Discovery: Find and explore all your data assets in a single place using various strategies, such as keyword search, data associations, and advanced queries. You can search across tables, topics, dashboards, pipelines, and services.

12


Data Collaboration: Communicate, converse, and cooperate with other users and teams on data assets. You can get event notifications, send alerts, add announcements, create tasks, and use conversation threads.

11


Data Quality and Profiler: Measure and monitor the quality with no-code to build trust in your data. You can define and run data quality tests, group them into test suites, and view the results in an interactive dashboard. With powerful collaboration, make data quality a shared responsibility in your organization.

8


Data Governance: Enforce data policies and standards across your organization. You can define data domains and data products, assign owners and stakeholders, and classify data assets using tags and terms. Use powerful automation features to auto-classify your data.

10


Data Insights and KPIs: Use reports and platform analytics to understand how your organization's data is doing. Data Insights provides a single-pane view of all the key metrics to reflect the state of your data best. Define the Key Performance Indicators (KPIs) and set goals within OpenMetadata to work towards better documentation, ownership, and tiering. Alerts can be set against the KPIs to be received on a specified schedule.

9


Data Lineage: Track and visualize the origin and transformation of your data assets end-to-end. You can view column-level lineage, filter queries, and edit lineage manually using a no-code editor.

Data Documentation: Document your data assets and metadata entities using rich text, images, and links. You can also add comments and annotations and generate data dictionaries and data catalogs.

Data Observability: Monitor the health and performance of your data assets and pipelines. You can view metrics such as data freshness, data volume, data quality, and data latency. You can also set up alerts and notifications for any anomalies or failures.

Data Security: Secure your data and metadata using various authentication and authorization mechanisms. You can integrate with different identity providers for single sign-on and define roles and policies for access control.

Webhooks: Integrate with external applications and services using webhooks. You can register URLs to receive metadata event notifications and integrate with Slack, Microsoft Teams, and Google Chat.

Connectors: Ingest metadata from various sources and tools using connectors. OpenMetadata supports about 84+ connectors for data warehouses, databases, dashboard services, messaging services, pipeline services, and more.

Try our Sandbox

Take a look and play with sample data at http://sandbox.open-metadata.org

Install and Run OpenMetadata

Get up and running in a few minutes. See the OpenMetadata documentation for installation instructions.

Documentation and Support

We're here to help and make OpenMetadata even better! Check out OpenMetadata documentation for a complete description of OpenMetadata's features. Join our Slack Community to get in touch with us if you want to chat, need help, or discuss new feature requirements.

Contributors

We ❤️ all contributions, big and small! Check out our CONTRIBUTING guide to get started, and let us know how we can help.

Don't want to miss anything? Give the project a 🚀

A HUGE THANK YOU to all our supporters!

Stargazers

Stargazers of @open-metadata/OpenMetadata repo

License

OpenMetadata is released under Apache License, Version 2.0