OpenMetadata/bootstrap/sql/migrations/native/1.12.7/postgres/schemaChanges.sql
sonika-shah f43d762f72
Some checks are pending
Integration Tests - MySQL + Elasticsearch / Detect Changes (push) Waiting to run
Integration Tests - MySQL + Elasticsearch / integration-tests-mysql-elasticsearch (push) Blocked by required conditions
Integration Tests - PostgreSQL + OpenSearch / Detect Changes (push) Waiting to run
Integration Tests - PostgreSQL + OpenSearch / integration-tests-postgres-opensearch (push) Blocked by required conditions
Java Checkstyle / java-checkstyle (push) Waiting to run
Maven Collate Tests / maven-collate-ci (push) Waiting to run
OpenMetadata Service Unit Tests / openmetadata-service-unit-tests-status (push) Blocked by required conditions
OpenMetadata Service Unit Tests / Detect Changes (push) Waiting to run
OpenMetadata Service Unit Tests / openmetadata-service-unit-tests (mysql) (push) Blocked by required conditions
OpenMetadata Service Unit Tests / openmetadata-service-unit-tests (postgresql) (push) Blocked by required conditions
OpenMetadata Service Unit Tests / k8s_operator-unit-tests (push) Blocked by required conditions
Fixes #27158: ingestion slowdown from tag_usage seq-scan on Postgres (#27745)
* Fixes #27158: restore tag_usage prefix-LIKE index on Postgres

The 1.11.0 perf migration (#23054) added four `WHERE state = 1` partial
indexes on tag_usage; #24063 dropped the matching `state = 1` predicate
from getTagsInternalByPrefix (Suggested-state rows are valid for both
classification and glossary derivation), leaving every partial index
inapplicable. Postgres fell back to a parallel seq scan; MySQL was
unaffected because its 1.11.0 indexes were never partial.

Adds a non-partial single-col btree on targetfqnhash_lower (mirrors
MySQL's idx_targetfqnhash_lower) and rebuilds the four partials as
non-partial -- same shape, same INCLUDE columns, predicate coupling
removed so future query changes can't silently invalidate them.

Verified end-to-end against a local Postgres with 50k rows: seq scan
reproduced before the fix (matches reporter's EXPLAIN), bitmap index
scan after, both for inline and prepared-statement paths.

* Move #27158 tag_usage fix to 1.12.7 + add tagfqn_lower index

Ship the fix in the 1.12.7 release line so customers on 1.12.x get it
without waiting for 2.0.x. Also closes the second structural gap with
MySQL: a non-partial single-col btree on tagfqn_lower mirroring MySQL's
idx_tagfqn_lower (1.11.0). 1.12.8 directory removed; 1.12.7's existing
schemaChanges.sql now carries the tag_usage indexes alongside the
entity_extension migration.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-05 10:30:11 +05:30

32 lines
No EOL
1.6 KiB
SQL

-- Issue #27158: tag_usage seq-scan on Postgres. #24063 dropped the
-- `state = 1` predicate that 1.11.0's partial indexes required.
-- Fix: add single-col indexes on the `_lower` columns, and drop the
-- `WHERE state = 1` filter from the partials so changes can't invalidate them.
DROP INDEX CONCURRENTLY IF EXISTS idx_tag_usage_targetfqnhash_lower_pattern;
CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_tag_usage_targetfqnhash_lower_pattern
ON tag_usage (targetfqnhash_lower text_pattern_ops);
DROP INDEX CONCURRENTLY IF EXISTS idx_tag_usage_tagfqn_lower_pattern;
CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_tag_usage_tagfqn_lower_pattern
ON tag_usage (tagfqn_lower text_pattern_ops);
DROP INDEX CONCURRENTLY IF EXISTS idx_tag_usage_target_prefix_covering;
CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_tag_usage_target_prefix_covering
ON tag_usage (source, targetfqnhash_lower text_pattern_ops)
INCLUDE (tagFQN, labelType, state);
DROP INDEX CONCURRENTLY IF EXISTS idx_tag_usage_tagfqn_prefix_covering;
CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_tag_usage_tagfqn_prefix_covering
ON tag_usage (source, tagfqn_lower text_pattern_ops)
INCLUDE (targetFQNHash, labelType, state);
DROP INDEX CONCURRENTLY IF EXISTS idx_tag_usage_join_source;
CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_tag_usage_join_source
ON tag_usage (tagFQNHash, source)
INCLUDE (targetFQNHash, tagFQN, labelType, state);
CREATE EXTENSION IF NOT EXISTS pg_trgm;
DROP INDEX CONCURRENTLY IF EXISTS gin_tag_usage_targetfqn_trgm;
CREATE INDEX CONCURRENTLY IF NOT EXISTS gin_tag_usage_targetfqn_trgm
ON tag_usage USING GIN (targetFQNHash gin_trgm_ops);