d2c64d8ac7
* fix(workflows): make Flowable schema upgrades idempotent to survive partial migrations Fixes #26048. When the server crashed mid-startup during a Flowable schema upgrade, the DB was left in a partially-migrated state. On restart, Flowable re-ran the same DDL and failed on already-existing objects (indexes, tables, columns), permanently wedging both the server and migrate --force. Changes: 1. WorkflowHandler: webserver now uses DB_SCHEMA_UPDATE_FALSE — it validates the schema but never runs DDL. Only migrate CLI uses DB_SCHEMA_UPDATE_TRUE. 2. OpenMetadataOperations: explicit WorkflowHandler.initialize(config, true) inside the migrate command so Flowable DDL always runs during migration. 3. WorkflowHandler: catches FlowableWrongDbException on webserver startup and rethrows with an actionable message directing the operator to run migrate. 4. IdempotentDdlDataSource + IdempotentDdlStatement: JDBC DataSource wrapper used exclusively in migration context. Intercepts execute(sql) for CREATE INDEX, CREATE TABLE, and ALTER TABLE ADD COLUMN and pre-checks existence via standard DatabaseMetaData (getIndexInfo, getTables, getColumns) before executing. If the object already exists it logs a skip and returns — no SQL state codes, no string matching, works on MySQL and PostgreSQL. Unit tests cover schema-update mode selection in both contexts. * fix(workflows): address review comments on idempotent DDL wrapper - Extract shouldSkip() helper; apply idempotency checks to all execute() and executeUpdate() overloads, not just execute(String) - Tighten ALTER TABLE regex with negative lookahead to exclude SQL keywords (CONSTRAINT, PRIMARY, UNIQUE, FOREIGN, CHECK, INDEX, KEY) from being matched as column names - IdempotentDdlDataSource now wraps a DataSource delegate instead of calling DriverManager directly; uses migrationDataSource() helper in WorkflowHandler to resolve from existing DataSource or JDBC params - Fix InvocationTargetException wrapping in Connection proxy — unwrap cause so callers receive the original SQLException - Wrap all createStatement() variants in the proxy, not just the no-arg form - Contextual error message in WorkflowHandler — distinguish between server startup and migration context - Add IdempotentDdlStatementTest: 11 tests covering skip/execute for CREATE INDEX, CREATE UNIQUE INDEX, CREATE TABLE, ALTER TABLE ADD COLUMN, keyword-guarded ALTER TABLE, executeUpdate overload, and pass-through * fix(workflows): include DB/library versions in FlowableWrongDbException message * test(workflows): add IdempotentDdlDataSourceTest for proxy wrapping and exception surfacing * test(workflows): assert exception identity in proxy exception-surfacing tests * fix(workflows): catalog-aware identifier normalization in IdempotentDdlStatement On MySQL with lower_case_table_names=0 (default on Linux), table names are stored as-is and catalog=null metadata lookups can miss existing objects. - Use connection.getCatalog() for all getIndexInfo/getTables/getColumns calls - Normalize identifiers via DatabaseMetaData.storesLowerCaseIdentifiers() / storesUpperCaseIdentifiers() instead of unconditional toLowerCase() - stripIdentifierQuotes() handles backtick, double-quote and bracket quoting - extractObjectName() handles schema-qualified names (schema.table) - columnExists now iterates and normalizes COLUMN_NAME from ResultSet - Test: added MySQL uppercase storage case to IdempotentDdlStatementTest * fix(workflows): null guard in shouldSkip, drop-create Flowable init, robust test indexing - shouldSkip() returns false immediately for null SQL, preserving JDBC contract (delegate handles null and throws the driver's own error) - drop-create command now calls WorkflowHandler.initialize(config, true) after native migrations so it produces a fully startable DB including Flowable tables - WorkflowHandlerSchemaUpdateTest: replace brittle get(1) with getLast() so the test is not sensitive to how many StandaloneProcessEngineConfiguration instances are constructed before initializeNewProcessEngine runs |
||
|---|---|---|
| .claude | ||
| .devcontainer | ||
| .github | ||
| bin | ||
| bootstrap | ||
| common | ||
| conf | ||
| docker | ||
| docs | ||
| examples/python-sdk/data-quality | ||
| ingestion | ||
| openmetadata-airflow-apis | ||
| openmetadata-clients | ||
| openmetadata-dist | ||
| openmetadata-integration-tests | ||
| openmetadata-k8s-operator | ||
| openmetadata-mcp | ||
| openmetadata-sdk | ||
| openmetadata-service | ||
| openmetadata-shaded-deps | ||
| openmetadata-spec | ||
| openmetadata-ui | ||
| openmetadata-ui-core-components | ||
| perf-tests | ||
| scripts | ||
| skills | ||
| .dockerignore | ||
| .git-blame-ignore-revs | ||
| .gitignore | ||
| .nojekyll | ||
| .pre-commit-config.yaml | ||
| .pylintrc | ||
| .snyk | ||
| APPLICATION.md | ||
| CLAUDE.md | ||
| CODE_OF_CONDUCT.md | ||
| CONTRIBUTING.md | ||
| DEVELOPER.md | ||
| generate_ts.sh | ||
| INCIDENT_RESPONSE.md | ||
| LICENSE | ||
| Makefile | ||
| NOTICE | ||
| package.json | ||
| pom.xml | ||
| README.md | ||
| SECURITY.md | ||
| THREAT_MODEL.md | ||
| yarn.lock | ||
Empower your Data Journey with OpenMetadata
What is OpenMetadata?
OpenMetadata is a unified metadata platform for data discovery, data observability, and data governance powered by a central metadata repository, in-depth column-level lineage, and seamless team collaboration. It is one of the fastest-growing open-source projects with a vibrant community and adoption by a diverse set of companies in a variety of industry verticals. Based on Open Metadata Standards and APIs, supporting connectors to a wide range of data services, OpenMetadata enables end-to-end metadata management, giving you the freedom to unlock the value of your data assets.
Contents:
OpenMetadata Consists of Four Main Components:
- Metadata Schemas: These are the core definitions and vocabulary for metadata based on common abstractions and types. They also allow for custom extensions and properties to suit different use cases and domains.
- Metadata Store: This is the central repository for storing and managing the metadata graph, which connects data assets, users, and tool-generated metadata in a unified way.
- Metadata APIs: These are the interfaces for producing and consuming metadata, built on top of the metadata schemas. They enable seamless integration of user interfaces and tools, systems, and services with the metadata store.
- Ingestion Framework: This is a pluggable framework for ingesting metadata from various sources and tools to the metadata store. It supports about 84+ connectors for data warehouses, databases, dashboard services, messaging services, pipeline services, and more.
Key Features of OpenMetadata
Data Discovery: Find and explore all your data assets in a single place using various strategies, such as keyword search, data associations, and advanced queries. You can search across tables, topics, dashboards, pipelines, and services.
Data Collaboration: Communicate, converse, and cooperate with other users and teams on data assets. You can get event notifications, send alerts, add announcements, create tasks, and use conversation threads.
Data Quality and Profiler: Measure and monitor the quality with no-code to build trust in your data. You can define and run data quality tests, group them into test suites, and view the results in an interactive dashboard. With powerful collaboration, make data quality a shared responsibility in your organization.
Data Governance: Enforce data policies and standards across your organization. You can define data domains and data products, assign owners and stakeholders, and classify data assets using tags and terms. Use powerful automation features to auto-classify your data.
Data Insights and KPIs: Use reports and platform analytics to understand how your organization's data is doing. Data Insights provides a single-pane view of all the key metrics to reflect the state of your data best. Define the Key Performance Indicators (KPIs) and set goals within OpenMetadata to work towards better documentation, ownership, and tiering. Alerts can be set against the KPIs to be received on a specified schedule.
Data Lineage: Track and visualize the origin and transformation of your data assets end-to-end. You can view column-level lineage, filter queries, and edit lineage manually using a no-code editor.
Data Documentation: Document your data assets and metadata entities using rich text, images, and links. You can also add comments and annotations and generate data dictionaries and data catalogs.
Data Observability: Monitor the health and performance of your data assets and pipelines. You can view metrics such as data freshness, data volume, data quality, and data latency. You can also set up alerts and notifications for any anomalies or failures.
Data Security: Secure your data and metadata using various authentication and authorization mechanisms. You can integrate with different identity providers for single sign-on and define roles and policies for access control.
Webhooks: Integrate with external applications and services using webhooks. You can register URLs to receive metadata event notifications and integrate with Slack, Microsoft Teams, and Google Chat.
Connectors: Ingest metadata from various sources and tools using connectors. OpenMetadata supports about 84+ connectors for data warehouses, databases, dashboard services, messaging services, pipeline services, and more.
Try our Sandbox
Take a look and play with sample data at http://sandbox.open-metadata.org
Install and Run OpenMetadata
Get up and running in a few minutes. See the OpenMetadata documentation for installation instructions.
Documentation and Support
We're here to help and make OpenMetadata even better! Check out OpenMetadata documentation for a complete description of OpenMetadata's features. Join our Slack Community to get in touch with us if you want to chat, need help, or discuss new feature requirements.
Contributors
We ❤️ all contributions, big and small! Check out our CONTRIBUTING guide to get started, and let us know how we can help.
Don't want to miss anything? Give the project a ⭐ 🚀
A HUGE THANK YOU to all our supporters!
Stargazers
License
OpenMetadata is released under Apache License, Version 2.0