Elgato_dark/OpenMetadata
1
0
Fork 0
mirror of https://github.com/open-metadata/OpenMetadata synced 2026-05-24 09:39:11 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 6
OpenMetadata/ingestion
History
Mohit Tilala 3d6fd71de3
Fixes #27950: [Datalake] JSON columns incorrectly typed as STRING for empty dict values (#27951) 
* fix: datalake JSON columns incorrectly typed as STRING for empty dict values

* fix: wrap df_row_val with str() for strptime and parse calls to satisfy type checker

* fix: address static check type errors and review comments in datalake utils

* Restore debug logging, fix dead-code fallback, strengthen tests

* Replace lexicographic max() with explicit type precedence in fetch_col_types
2026-05-11 18:02:06 +05:30
..
.basedpyright chore(ingestion): refresh basedpyright config; standard mode + ratchet scaffold (#27794) 2026-04-28 12:58:23 +02:00
examples chore(ingestion): drop pylint, expand ruff (#27774) 2026-04-28 07:21:59 +02:00
operators Migrate Databricks from sqlalchemy-databricks to databricks-sqlalchemy (#26896) 2026-05-04 18:53:24 +05:30
pipelines Openlineage: Added Kinesis Support #24752 (#26050) 2026-02-26 14:20:46 +05:30
src Fixes #27950: [Datalake] JSON columns incorrectly typed as STRING for empty dict values (#27951) 2026-05-11 18:02:06 +05:30
tests Fixes #27950: [Datalake] JSON columns incorrectly typed as STRING for empty dict values (#27951) 2026-05-11 18:02:06 +05:30
__init__.py ci/nox-setup-testing (#21377) 2025-05-27 10:56:52 +02:00
airflow-constraints-2.10.5.txt Chore: Remove iceberg standalone connector (#26365) 2026-04-02 14:55:23 +00:00
airflow-constraints-3.1.2.txt Fix #23096: Add Airflow 3.x support (#24338) 2025-11-21 12:28:28 +01:00
airflow-constraints-3.1.5.txt Fix Trivy scans (#24867) 2025-12-19 16:27:12 +01:00
airflow-constraints-3.1.7.txt Fixes: Airflow 3.1.7 (#26849) 2026-03-30 21:27:08 +00:00
Dockerfile MINOR: imagemagick purge (#27752) 2026-04-28 15:02:33 +05:30
Dockerfile.ci Migrate Databricks from sqlalchemy-databricks to databricks-sqlalchemy (#26896) 2026-05-04 18:53:24 +05:30
ingestion_dependency.sh Fix #23096: Add Airflow 3.x support (#24338) 2025-11-21 12:28:28 +01:00
LICENSE Docs - Ingestion License (#17893) 2024-09-17 08:58:53 -07:00
Makefile chore(ingestion): drop pylint, expand ruff (#27774) 2026-04-28 07:21:59 +02:00
noxfile.py chore(ingestion): drop pylint, expand ruff (#27774) 2026-04-28 07:21:59 +02:00
pyproject.toml chore(ingestion): refresh basedpyright config; standard mode + ratchet scaffold (#27794) 2026-04-28 12:58:23 +02:00
README.md Refactor: remove doc changes from OM repo (#22019) 2025-08-20 14:28:48 +05:30
setup.py feat(ingestion): add QuestDB database connector (#27604) 2026-05-11 13:02:32 +05:30
sonar-project.properties MINOR: Fix sonar coverage (#25276) 2026-01-16 08:35:00 +01:00

README.md

This guide will help you setup the Ingestion framework and connectors

This guide will help you setup the Ingestion framework and connectors

Python version 3.9+

OpenMetadata Ingestion is a simple framework to build connectors and ingest metadata of various systems through OpenMetadata APIs. It could be used in an orchestration framework(e.g. Apache Airflow) to ingest metadata. Prerequisites

  • Python >= 3.9.x

Docs

Please refer to the documentation here https://docs.open-metadata.org/connectors

TopologyRunner

All the Ingestion Workflows run through the TopologyRunner.

The flow is depicted in the images below.

TopologyRunner Standard Flow

image

TopologyRunner Multithread Flow

image