OpenMetadata/bootstrap/sql/migrations/native
Pere Miquel Brull 7485c5b421
feat: add ContextMemory entity (Context Center memories) (#28224)
* feat(spec): add ContextMemory + CreateContextMemory JSON schemas

* feat(jdbi3): add ContextMemoryDAO

* feat: register contextMemory entity type constant

* feat(service): add ContextMemory repository, resource, mapper

* feat(bootstrap): add context_memory table DDL

* test(service): ContextMemory resource CRUD test

* fix(context-memory): address review (relationship types, stable FQN, status msg, test name)

- storeRelationships: rootMemory -> Relationship.CONTAINS, parentMemory -> Relationship.HAS
  so the root-ancestor and direct-parent hierarchies are distinguishable.
- setFullyQualifiedName: derive from the immutable name only (drop mutable
  primaryEntity/owner derivation that destabilized nameHash on update).
- validateStatusTransition: separate "no transitions defined" from "disallowed transition".
- Rename ContextMemoryResourceTest -> ContextMemoryStatusTransitionTest (pure unit test).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* test(context-memory): add ContextMemoryIT + SDK ContextMemoryService

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* fix(spec): register contextMemory in EntityLink.g4 ENTITY_TYPE grammar

EntityLinkGrammarTest.testAllEntityTypesHaveGrammarOrExclusion enumerates every
Entity.java constant and requires each to be in the EntityLink grammar or the
test's exclusion list. ContextMemory is a normal EntityRepository-backed
top-level entity (like learningResource / contextFile), so it belongs in the
ENTITY_TYPE rule.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* test(context-memory): override owner ITs for creator-as-owner default

ContextMemoryMapper.defaultOwners() intentionally assigns the creating
user as owner when the create request omits owners. BaseEntityIT's
patch_entityUpdateOwner_200 and patch_entityUpdateOwnerFromNull_200
assert "no owner initially" for any supportsOwners entity, so both
failed for ContextMemory.

Override both in ContextMemoryIT: keep the PATCH-replace-owner contract,
change only the precondition to expect the creator as the sole initial
owner (asserted by count, not a hardcoded principal). Mapper unchanged.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* Update generated TypeScript types

Add the generated ContextMemory TS types (entity/context/contextMemory.ts,
api/context/createContextMemory.ts). The schemas were on the branch but their
generated types were missing, failing the TypeScript Type Generation check on
this fork PR.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(context-memory): address review (relationship cleanup, owner scope, validations)

Copilot review on the ContextMemory entity:
- #1 record primaryEntity/relatedEntities/root/parent/source*/machineRepresentation
  in version history; usageCount/lastUsedAt documented as untracked telemetry
- #2 clear stale HAS/RELATED_TO/CONTAINS edges before re-adding in storeRelationships
- #4 default creator as owner only on create; PUT without owners no longer
  silently replaces previously set owners
- #5 schema documents that any status is allowed at creation; transitions
  enforced only on update
- #6 setFullyQualifiedName via FullyQualifiedName.build with skip-if-set guard
- #7 validate shared principal type is user/team/domain
- #8 reject self-reference for parentMemory/rootMemory
- #10 inline Entity.CONTEXT_MEMORY, drop redundant constant

Regenerate ContextMemory TS types for the schema doc change; add IT coverage
for the self-reference and invalid-shared-principal validations.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(context-memory): don't blanket-delete relationships (domain data loss)

The #2 cleanup via deleteTo(memory, CONTEXT_MEMORY, HAS, null) also matched the
framework's domain --HAS--> memory edge (storeDomains runs before
storeRelationships in storeRelationshipsInternal, on every create and update),
silently dropping domain assignments.

storeRelationships is now add-only (addRelationship upserts, so re-running on
update is idempotent). Stale-edge cleanup moved to ContextMemoryUpdater using
the framework's updateFromRelationship(s) helpers, which delete only the
specific changed refs and record the version change. parentMemory now uses
Relationship.PARENT_OF (distinct from primaryEntity's HAS and the framework's
domain HAS) so the parent edge can be maintained without collision.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore(bootstrap): move context_memory DDL from 2.0.1 to 2.0.0

The context_memory table belongs in the 2.0.0 migration. Relocated the
MySQL and Postgres DDL verbatim; the 2.0.1 schemaChanges.sql files are
restored to their original task_migration_mapping-only content.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore(bootstrap): add ENGINE=InnoDB to context_memory MySQL DDL

Explicit engine clause, consistent with the task/search-index tables in the
same migration and robust to any server default change.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(context-memory): preserve sanitized/validated fields; validate relatedEntities

Review follow-ups:
- ContextMemoryMapper no longer re-sets description/owners/domains/tags/displayName
  after copy(). copy() sanitizes description (stored-XSS) and validates owners and
  domains; re-setting the raw request values bypassed both. Only ContextMemory-
  specific fields are set now.
- prepare() now assigns the result of EntityUtil.populateEntityReferences back onto
  relatedEntities so orphaned/invalid refs are filtered instead of persisted.
- ContextMemoryIT Javadoc now references ContextMemoryRepository#setCreatorAsDefaultOwner
  (the defaultOwners mapper method no longer exists).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-19 18:10:46 +02:00
..
1.1.0 Fix postgres migration files (#12923) 2023-08-18 14:54:43 +02:00
1.1.1 Fix postgres migration files (#12923) 2023-08-18 14:54:43 +02:00
1.1.2 Issue 8930 - Update profiler timestamp from seconds to milliseconds (#12948) 2023-08-25 08:47:16 +02:00
1.1.5 only add collation to hash columns (#13201) 2023-09-15 12:49:11 +05:30
1.1.6 Add 1.1.6 migrations dir (#13305) 2023-09-22 09:45:00 +02:00
1.1.7 Prep v1.1.7 migrations to address test cases & suites (#13345) 2023-09-27 11:49:21 +02:00
1.2.0 Migration Fixes (#16131) 2024-05-07 22:07:25 +05:30
1.2.1 fix: comment in sql migration (#13979) 2023-11-15 10:32:11 +01:00
1.2.3 Minor: Fix migration location for unity catalog (#14339) 2024-01-03 18:26:11 +05:30
1.2.4 Fix #13982: Fix userFQN encoding while creating mentions (#14496) 2023-12-25 17:28:13 -08:00
1.3.0 Migration Fixes (#16131) 2024-05-07 22:07:25 +05:30
1.3.1 fix: move migration to 1.3.1 (#15463) 2024-03-05 15:30:43 +01:00
1.3.2 Remove SQls from 1.3.2 (#15917) 2024-04-16 18:51:03 +05:30
1.3.3 Move migration for apps to 1.3.3 all together (#15944) 2024-04-18 14:26:05 +05:30
1.4.0 ISSUE #2681 - Add Missing test parameters in PSQL (#25323) 2026-01-16 12:09:15 +01:00
1.4.2 Fix Test Suite Filter (#16615) 2024-06-12 10:40:05 +05:30
1.4.4 Fix #16788: Optimize feed query performance issues introduced in 1.4.2 (#16862) 2024-07-01 19:58:47 -07:00
1.4.5 MINOR - Clean automations_workflow in 1.4.5 (#17006) 2024-07-12 13:54:46 +02:00
1.4.6 Move Migration to 1.4.6 (#17095) 2024-07-19 12:16:53 +05:30
1.4.7 Migrate NameHash (#17317) 2024-08-06 18:41:37 +05:30
1.5.0 Improve count/feed api performance for 1.5 (#17576) 2024-08-23 11:20:34 -07:00
1.5.6 [Search] Indexing Fixes (#18048) 2024-09-30 23:39:27 +05:30
1.5.7 migration: fix duplicate param key insertion (#20802) 2025-04-15 14:10:51 +02:00
1.5.9 MINOR - Move appName migration to 1.5.9 (#18435) 2024-10-28 16:29:56 +01:00
1.5.11 Fix Search Index Contention (#18605) 2024-11-12 20:36:23 +05:30
1.5.15 Domain Policy Update to be non-system (#19060) 2024-12-15 01:18:12 +05:30
1.6.0 Feat# Implementation of Custom Workflows (#23023) 2025-10-08 18:57:44 +05:30
1.6.2 Improvement #19065 : Support removing existing enumKeys (for enum type custom property) (#19054) 2025-01-07 19:25:59 -08:00
1.6.3 Cleanup App data (#19571) 2025-01-28 19:22:33 +05:30
1.6.7 MINOR: chore: added missing timestamp indexes for time series tables (#20373) 2025-03-24 07:43:07 +01:00
1.7.0 Add cleanup apps_extension_time_series (#20857) 2025-04-16 14:54:11 +05:30
1.7.1 Escape ? to causing issues in jdbi binding (#21381) 2025-05-23 17:13:45 +05:30
1.7.2 FIX - Automation Workflows should not be updated by the SM & cleanup migration (#21435) 2025-06-03 12:17:14 +02:00
1.7.4 Disabled bot creating activity feeds (#21773) 2025-06-14 19:21:00 +05:30
1.8.0 Add Data Contracts Specification and APIs (#21164) 2025-06-04 06:36:28 +02:00
1.8.1 Fix #20621: User Status Tracking in the System (#21911) 2025-07-02 14:37:36 -07:00
1.8.2 Fix #20145: Implemented Prefix For Dashboard Service (#21585) 2025-07-08 18:54:35 +02:00
1.8.4 MINOR - Add columns.description in search settings (#22299) 2025-07-15 09:21:57 +02:00
1.8.5 Added missing migration sql files [1.8.5 and 1.10.2] (#24399) 2025-11-18 08:02:35 +01:00
1.8.7 Feature: Security Service (#22450) 2025-07-31 06:38:21 +02:00
1.8.8 Feature: Security Service (#22450) 2025-07-31 06:38:21 +02:00
1.8.9 Feature: Security Service (#22450) 2025-07-31 06:38:21 +02:00
1.9.0 MINOR - Add Tests & fix migrations (#22714) 2025-08-03 15:19:54 +02:00
1.9.2 Add missing domain migrations for entity version history (#23032) 2025-08-21 14:33:37 +05:30
1.9.5 MINOR - Move migrations to 1.9.5 (#23095) 2025-08-28 09:23:21 +02:00
1.9.6 ISSUE #1534 - Profiler Refactor for Metadata Extraction Application (#23200) 2025-09-05 13:07:04 +02:00
1.9.9 Minor fix broken 1.9.8 migrations (#23487) 2025-09-22 13:13:25 +00:00
1.9.10 Fixes #23356: Databricks & UnityCatalog OAuth and Azure AD Auth (#23561) 2025-10-03 19:53:19 +05:30
1.9.11 add entityType.keyword aggregation in searchSettings.json (#23559) 2025-09-25 17:04:49 +05:30
1.10.0 Move migrations to 1.11.x (#24074) 2025-10-30 01:02:45 +05:30
1.10.2 Added missing migration sql files [1.8.5 and 1.10.2] (#24399) 2025-11-18 08:02:35 +01:00
1.10.3 MINOR: dbt migration fix (#23980) 2025-10-23 12:54:34 +02:00
1.10.4 chore: move dbt migration to 1.11 (#24076) 2025-11-03 08:46:47 +01:00
1.10.5 TRUNCATE Flowable History Tables in both 1.10.5 and 1.10.7 Migration (#24323) 2025-11-13 21:05:31 +00:00
1.10.6 Fixes #24132: Airbyte Cloud Support (#24261) 2025-11-11 16:24:09 +05:30
1.10.7 TRUNCATE Flowable History Tables in both 1.10.5 and 1.10.7 Migration (#24323) 2025-11-13 21:05:31 +00:00
1.10.8 Fix email configuration templates default value from 'collate' to 'openmetadata' (#24352) 2025-11-17 08:39:41 +01:00
1.11.0 Moved AI Application and LLM Model entities migrations to 1.12.0 (#25659) 2026-02-02 08:50:37 +01:00
1.11.1 chore: realign main migration with 1.11.1 branch (#24938) 2025-12-22 09:03:28 +01:00
1.11.2 Fix #24578: Datamodels not visible if . in service name (#24779) 2025-12-27 10:00:26 -08:00
1.11.4 Fix search percentile rank scoring (#24859) 2025-12-23 18:06:27 +00:00
1.11.5 Tagging explanation (#24817) 2026-01-08 17:02:40 +01:00
1.11.6 Fix: remove overrideLineage config from database service metadata pipeline (#25379) 2026-01-20 09:08:26 +05:30
1.11.8 Fixes #24546: Add sobjectNames field for multi-object selection in Salesforce connector (#24547) 2026-02-02 16:05:59 +01:00
1.11.9 Add bulk apis for pipeline status (#25731) 2026-02-10 18:14:06 +05:30
1.11.11 Fix-20713: Add support for metadata ingestion using local file in REST connector (#26036) 2026-02-23 21:50:26 +05:30
1.11.12 Fix #26178: Add support for IAM auth for redshift (#26179) 2026-03-02 21:57:28 +05:30
1.12.0 fix(lineage): service nodes appearing in entity lineage view and empty By Service view (#27258) 2026-04-17 00:55:16 -07:00
1.12.1 Continuous indexing to handle failures (#26111) 2026-03-18 16:23:04 +05:30
1.12.2 Fixes #26225: Add index and FORCE INDEX for listLastTestCaseResultsForTestSuite (MySQL) (#26235) 2026-03-06 07:55:41 -08:00
1.12.4 Move Migration to 1.12.4 from 1.12.3 (#26629) 2026-03-20 09:41:15 +00:00
1.12.5 Update indexing schedule (#27204) 2026-04-10 19:15:08 +05:30
1.12.6 fix(lineage): service nodes appearing in entity lineage view and empty By Service view (#27258) 2026-04-17 00:55:16 -07:00
1.12.7 Fixes #27158: ingestion slowdown from tag_usage seq-scan on Postgres (#27745) 2026-05-05 10:30:11 +05:30
1.12.8 Add migrations to ensure PII are really enabled (#27921) 2026-05-08 15:39:29 +00:00
1.12.9 fix(profiler): N+1 / missing-index regression on /tables/.../columns?fields=profile (#3488) (#27746) 2026-05-14 06:56:39 -07:00
1.13.0 fix(glossary): preserve all relation types between same term pair (#28172) 2026-05-16 11:37:21 -07:00
2.0.0 feat: add ContextMemory entity (Context Center memories) (#28224) 2026-05-19 18:10:46 +02:00
2.0.1 fix(rdf): converge Fuseki state on weekly rebuilds and isolate API latency (#28117) 2026-05-15 17:36:06 -07:00