OpenMetadata/ingestion/tests/integration
Sriharsha Chintalapani 09af9fc801
Fixes #4003: bulk + async restore for large entity hierarchies (#27997)
* fix(restore): bulk + async restore for large entity hierarchies

EntityRepository.restoreEntity walked descendants synchronously, taking
4+ minutes on a 12k-table database and exceeding typical proxy timeouts.
restoreChildren now groups CONTAINS children by type and dispatches one
bulkRestoreSubtree per type, batching DB writes, version history,
change events, and cache invalidation; the existing ES cascade handles
descendant index updates in one update_by_query.

Adds an async option (?async=true) on the deep-hierarchy restore
endpoints that returns 202 Accepted with a job id and runs the restore
on AsyncService, emitting WebSocket notifications on
restoreEntityChannel. Java SDK adds .restore().async().execute() fluent
builders on Tables/Databases plus restoreServerAsync on
EntityServiceBase; Python SDK mirrors this with
restore_request().with_async().execute() and restore_async() helpers
on BaseEntity, exposing a new AsyncJobResponse type.

Tests: EntityRepositoryRestoreTest verifies the per-type grouping and
bulk dispatch path; RestoreFluentAPITest covers the Java SDK fluent
behavior; RestoreHierarchyIT exercises sync and async restore against a
real DB→schemas→tables tree end-to-end; test_restore_async.py covers
the Python SDK paths.

Fixes #4003
2026-05-20 17:57:40 -07:00
..
airflow chore(ingestion): drop pylint, expand ruff (#27774) 2026-04-28 07:21:59 +02:00
alationsink chore(ingestion): drop pylint, expand ruff (#27774) 2026-04-28 07:21:59 +02:00
amundsen chore(ingestion): drop pylint, expand ruff (#27774) 2026-04-28 07:21:59 +02:00
atlas chore(ingestion): drop pylint, expand ruff (#27774) 2026-04-28 07:21:59 +02:00
auto_classification Fixes 21329: exclude temporal table period columns from autoClassification sampling (#27960) 2026-05-11 11:45:40 +05:30
automations chore(ingestion): drop pylint, expand ruff (#27774) 2026-04-28 07:21:59 +02:00
cassandra chore(ingestion): migrate to ruff for format + isort + unused-import (#27739) 2026-04-27 10:05:28 +02:00
cockroach chore(ingestion): drop pylint, expand ruff (#27774) 2026-04-28 07:21:59 +02:00
connections chore(ingestion): drop pylint, expand ruff (#27774) 2026-04-28 07:21:59 +02:00
data_quality chore(ingestion): drop pylint, expand ruff (#27774) 2026-04-28 07:21:59 +02:00
datalake Fixes #4003: bulk + async restore for large entity hierarchies (#27997) 2026-05-20 17:57:40 -07:00
fivetran chore(ingestion): drop pylint, expand ruff (#27774) 2026-04-28 07:21:59 +02:00
great_expectations chore(ingestion): drop pylint, expand ruff (#27774) 2026-04-28 07:21:59 +02:00
kafka chore(ingestion): drop pylint, expand ruff (#27774) 2026-04-28 07:21:59 +02:00
lineage chore(ingestion): drop pylint, expand ruff (#27774) 2026-04-28 07:21:59 +02:00
mcp MCP services (#23623) 2026-04-01 22:15:20 +05:30
mongodb chore(ingestion): drop pylint, expand ruff (#27774) 2026-04-28 07:21:59 +02:00
mysql ISSUE #3032 (#27912) 2026-05-07 09:01:18 -07:00
ometa Refactor(ingestion): introduce ClassifiableEntityAdapter to eliminate scattered isinstance checks (#27716) 2026-05-13 09:37:58 +02:00
orm_profiler chore(ingestion): drop pylint, expand ruff (#27774) 2026-04-28 07:21:59 +02:00
postgres chore(ingestion): drop pylint, expand ruff (#27774) 2026-04-28 07:21:59 +02:00
powerbi chore(ingestion): migrate to ruff for format + isort + unused-import (#27739) 2026-04-27 10:05:28 +02:00
profiler ISSUE #3032 (#27912) 2026-05-07 09:01:18 -07:00
s3 Containers: batch container data-model column tag retrieval to avoid subtree fan-out (#27836) 2026-04-30 20:55:55 -07:00
sas chore(ingestion): drop pylint, expand ruff (#27774) 2026-04-28 07:21:59 +02:00
sdk Fixes #4003: bulk + async restore for large entity hierarchies (#27997) 2026-05-20 17:57:40 -07:00
sftp chore(ingestion): drop pylint, expand ruff (#27774) 2026-04-28 07:21:59 +02:00
sources chore(ingestion): drop pylint, expand ruff (#27774) 2026-04-28 07:21:59 +02:00
sql_server chore(ingestion): drop pylint, expand ruff (#27774) 2026-04-28 07:21:59 +02:00
ssrs chore(ingestion): migrate to ruff for format + isort + unused-import (#27739) 2026-04-27 10:05:28 +02:00
superset chore(ingestion): drop pylint, expand ruff (#27774) 2026-04-28 07:21:59 +02:00
test_suite chore(ingestion): drop pylint, expand ruff (#27774) 2026-04-28 07:21:59 +02:00
trino ISSUE #3032 (#27912) 2026-05-07 09:01:18 -07:00
usage chore(ingestion): drop pylint, expand ruff (#27774) 2026-04-28 07:21:59 +02:00
workflow chore(ingestion): drop pylint, expand ruff (#27774) 2026-04-28 07:21:59 +02:00
__init__.py License header update (#1498) 2021-12-01 12:46:28 +05:30
conftest.py Fixes #4003: bulk + async restore for large entity hierarchies (#27997) 2026-05-20 17:57:40 -07:00
containers.py chore(ingestion): drop pylint, expand ruff (#27774) 2026-04-28 07:21:59 +02:00
integration_base.py chore(ingestion): drop pylint, expand ruff (#27774) 2026-04-28 07:21:59 +02:00