OpenMetadata/openspec/changes/incident-lifecycle-workflow/design.md

Context

The incident manager today is a switch statement in TestCaseResolutionStatusRepository.storeInternal() with no extension points. This design moves incident Thread/task lifecycle into the governance workflows framework using an event-driven, signal-based architecture where every TCRS event starts a short-lived process that reads state and acts.

Key files in the current system:

Component	File	Role
State machine	TestCaseResolutionStatusRepository.storeInternal()	Switch on New/Ack/Assigned/Resolved
Incident creation	TestCaseResolutionStatusRepository.getOrCreateIncident()	Creates New record, calls storeInternal()
Severity inference	TestCaseResolutionStatusRepository.inferIncidentSeverity()	Called inside storeInternal() before insert
Task creation	TestCaseResolutionStatusRepository.openOrAssignTask()	Creates Thread on Ack, patches assignee on Assigned
Resolution	TestCaseResolutionStatusRepository.resolveTask()	Closes Thread task via closeTaskWithoutWorkflow()
incidentId linking	TestCaseResultRepository.setTestCaseResultIncidentId()	Calls getOrCreateIncident() synchronously
Workflow engine	WorkflowHandler.java	Flowable ProcessEngine singleton
Node registration	NodeFactory.java	Switch on NodeSubType
Event routing	WorkflowEventConsumer.sendMessage()	Routes ChangeEvents to Flowable signals
Event dispatcher	EntityLifecycleEventDispatcher	Synchronous observer-based event system
Trigger filter	FilterEntityImpl.java	Evaluates JSON Logic against full entity (fetched with Include.ALL)

---

Goals / Non-Goals

Goals:

• Move incident Thread/task lifecycle into a governance workflow
• Enable auto-assign on incident creation (configurable, default off)
• Ship a default branching workflow that handles open/close
• Introduce reusable openTask and closeTask node types
• Extend event pipeline to broadcast TCRS events to workflows
• Handle re-open from Resolved to any non-Resolved status

Non-Goals:

• Auto-close on test pass (Slice 2)
• TTL / stale incident expiration (Slice 3)
• Timer subprocess or signal-based timer interruption (Slice 3)
• Cleanup timer for orphaned processes (follow-up work)
• Changing the Ack/Assigned assignee-patching logic
• Custom lifecycle states
• UI changes

---

Decisions

D1: Event-driven architecture — storeInternal broadcasts, workflow reacts

Decision: storeInternal() is decoupled from Flowable. After persisting a TCRS record, it broadcasts an event via EntityLifecycleEventDispatcher. A registered handler in WorkflowHandler receives the event and broadcasts a Flowable signal. The workflow starts from the signal and handles lifecycle actions.

Event flow:

storeInternal() persists TCRS record
  → EntityLifecycleEventDispatcher.postCreate(tcrsRecord)
    → WorkflowHandler.onTcrsEvent(tcrsRecord)
      → runtimeService.signalEventReceived("tcrs_{testCaseFQN}", variables)
        → Flowable starts new process instance(s) in matching workflow(s)

Why EntityLifecycleEventDispatcher (not ChangeEvents):

• TCRS is a time-series entity that does NOT emit ChangeEvents
• EntityLifecycleEventDispatcher is synchronous (same thread) — no async delay
• The API call returns only after the workflow process completes (short-lived processes reach end event before the signal call returns)
• Already used for search indexing; natural extension point

Signal variables carried:

• status: the TCRS status (New, Ack, Assigned, Resolved)
• testCaseFQN: the test case fully qualified name
• stateId: the incident state ID
• entityLink: the entity link from the TCRS record

What stays in storeInternal():

• Record persistence (unchanged)
• Severity inference (unchanged, see D6)
• Ack case: assignee patching on existing Thread (see D5)
• Assigned case: assignee patching (unchanged)
• Event broadcast (new)

What moves to the workflow:

• Thread/task creation (currently in openOrAssignTask() Ack case)
• Thread/task closure (currently in resolveTask())

D2: Generic nodes — openTask and closeTask (replaces HumanInterventionTask)

Decision: Instead of a monolithic HumanInterventionTask, build two generic, composable nodes following the three-layer pattern:

openTask:

Layer	Class	Responsibility	Flowable dependency
Task	OpenTask	Builds ServiceTask BPMN element. References delegate class name.	Yes (BPMN model only)
Delegate	OpenTaskDelegate implements JavaDelegate	Receives config via Expression fields. Sets taskCreated process variable.	Yes (thin adapter)
Impl	OpenTaskImpl	Idempotent Thread/task creation. No Flowable imports.	None

closeTask:

Layer	Class	Responsibility	Flowable dependency
Task	CloseTask	Builds ServiceTask BPMN element. References delegate class name.	Yes (BPMN model only)
Delegate	CloseTaskDelegate implements JavaDelegate	Receives config via Expression fields. Sets taskClosed process variable.	Yes (thin adapter)
Impl	CloseTaskImpl	Thread/task closure. No Flowable imports.	None

Reference: CreateAndRunIngestionPipelineTask → CreateIngestionPipelineDelegate → CreateIngestionPipelineImpl

openTask config:

{
  "template": "incident",
  "taskType": "RequestTestCaseFailureResolution",
  "responsibles": { "source": "tableOwner" }
}

• template: identifies the task template (for future extensibility)
• taskType: the TaskType enum value for Thread creation
• responsibles (optional): auto-assign config. Omitted = unassigned (default)
  • { "source": "tableOwner" } → resolve table entity owner
  • { "source": "specificUser", "target": "user.fqn" } → specific user

closeTask config:

{
  "template": "incident",
  "taskType": "RequestTestCaseFailureResolution"
}

Idempotency:

• openTask: queries for existing open Thread/task for the entity. If found → no-op, sets taskCreated=false. If not found → creates Thread, sets taskCreated=true.
• closeTask: queries for open Thread/task. If found → closes it, sets taskClosed=true. If not found → no-op, sets taskClosed=false.

Runs as governance-bot — the WorkflowEventConsumer already skips events from governance-bot, preventing infinite loops.

D3: Signal architecture — broadcast for fan-out, no queries

Decision: Use Flowable signals (broadcast) for workflow triggering. Signals are the right primitive because:

1. Multiple workflows can react to the same TCRS event (fan-out)
2. No need to query Flowable for existing processes (signals are fire-and-forget)
3. Each workflow's signal start event independently creates a new process instance

Slice 1 signal:

• tcrs_{testCaseFQN} — broadcast on every TCRS event. Starts new processes.

Slice 3 signal (deferred):

• tcrs_closed_{testCaseFQN} — broadcast by closeTask. Caught by signal boundary events on timer subprocesses to interrupt/cancel timers.

Why signals, not messages:

• Messages are unicast (delivered to one specific execution). Would require knowing which process to target → Flowable queries.
• Signals are broadcast (all listeners receive). No routing logic needed.
• The "double fire" concern (signal catches both start events and intermediate catches) is handled by using different signal names for different purposes.

Flowable does NOT enforce business key uniqueness. Multiple process instances can share the same business key. This is fine — short-lived processes complete quickly and don't accumulate.

D4: Short-lived processes — every event is an episode

Decision: Every TCRS event starts a new short-lived process instance. The process reads state, acts, and ends. No long-lived processes in Slice 1.

BPMN structure for default workflow:

[SignalStartEvent: "tcrs_{testCaseFQN}"]
  → [ExclusiveGateway: statusGateway]
      condition: ${status != "Resolved"}
        → [ServiceTask: openTask] → [EndEvent]
      condition: ${status == "Resolved"}
        → [ServiceTask: closeTask] → [EndEvent]
      default:
        → [EndEvent: skipEnd]

Process lifecycle per event:

TCRS Event	Gateway route	Node action	Process duration
New (first failure)	NOT Resolved	openTask creates Thread	Milliseconds
Ack	NOT Resolved	openTask no-op (Thread exists)	Milliseconds
Assigned	NOT Resolved	openTask no-op (Thread exists)	Milliseconds
Resolved	Resolved	closeTask closes Thread	Milliseconds
Resolved→New (re-open)	NOT Resolved	openTask creates new Thread	Milliseconds
Resolved→Ack (re-open)	NOT Resolved	openTask creates new Thread	Milliseconds

Why short-lived:

• No Flowable state accumulation (no ACT_RU_* rows between events)
• No need to correlate messages to running processes
• No orphan cleanup needed
• Idempotent nodes make repeated execution safe

Slice 3 evolution: The openTask branch gains a timer subprocess:

  → [ServiceTask: openTask]
    → [SubProcess: timerChain]
        [Timer: 24h] → [Notify] → [Timer: 7d] → [AutoClose] → [End]
        SignalBoundaryEvent (interrupting): "tcrs_closed_{fqn}"
    → [EndEvent]

The timer subprocess makes the New/Ack branch long-lived. When Resolved arrives, a new short-lived process runs closeTask AND broadcasts tcrs_closed_{fqn}, which interrupts the timer subprocess via its signal boundary event. All timers are cancelled, the old process ends cleanly.

Only Resolved kills timers. Ack/Assigned events don't interrupt the timer subprocess — reminders continue counting from the original failure time. This is a deliberate product choice: "remind the (now assigned) person" is still useful.

D5: Ack backward compatibility

Decision: Modify openOrAssignTask() for the Ack case to check if a Thread/task already exists before creating one.

Why: With the workflow creating the Thread/task on the New event (immediately on test failure), by the time a user Acks, the task already exists. Today, Ack unconditionally calls createTask(), which would create a duplicate.

Change: In openOrAssignTask(), the Ack case mirrors the Assigned pattern:

case Ack -> {
    Thread existingTask = getIncidentTask(incidentStatus);
    if (existingTask == null) {
        createTask(incidentStatus, Collections.singletonList(incidentStatus.getUpdatedBy()));
    } else {
        patchTaskAssignee(existingTask, incidentStatus.getUpdatedBy(),
            incidentStatus.getUpdatedBy().getName());
    }
}

This is backward-compatible: if the workflow didn't run (Flowable was down), the Ack fallback creates the task.

D6: Severity inference stays in repository

Decision: inferIncidentSeverity() is already called inside storeInternal() during record creation. No change needed — severity is inferred when the record is created, before the workflow fires.

Confirmed by code trace: getOrCreateIncident() → createNewRecord() → storeInternal() → inferIncidentSeverity().

D7: Bootstrap mechanism for default workflow

Decision: Add a JSON file to the existing bootstrap directory at openmetadata-service/src/main/resources/json/data/governance/workflows/.

The server's WorkflowDefinitionResource.initialize() calls initSeedDataFromResources() on startup, which loads all JSON files matching .*json/data/workflowDefinition/.*\.json$ and persists them via createOrUpdate semantics.

Two workflows already bootstrap this way:

• GlossaryApprovalWorkflow.json
• RecognizerFeedbackReviewWorkflow.json

We add IncidentLifecycleWorkflow.json following the same pattern.

D8: Schema changes

New files:

• openmetadata-spec/.../governance/workflows/elements/nodes/automatedTask/openTask.json — openTask config schema
• openmetadata-spec/.../governance/workflows/elements/nodes/automatedTask/closeTask.json — closeTask config schema
• openmetadata-service/.../governance/workflows/elements/nodes/automatedTask/OpenTask.java — Task layer
• openmetadata-service/.../governance/workflows/elements/nodes/automatedTask/CloseTask.java — Task layer
• openmetadata-service/.../governance/workflows/elements/nodes/automatedTask/impl/OpenTaskDelegate.java — Delegate
• openmetadata-service/.../governance/workflows/elements/nodes/automatedTask/impl/OpenTaskImpl.java — Impl
• openmetadata-service/.../governance/workflows/elements/nodes/automatedTask/impl/CloseTaskDelegate.java — Delegate
• openmetadata-service/.../governance/workflows/elements/nodes/automatedTask/impl/CloseTaskImpl.java — Impl
• openmetadata-service/src/main/resources/json/data/governance/workflows/IncidentLifecycleWorkflow.json — Bootstrap data

Modified files:

• openmetadata-spec/.../governance/workflows/elements/nodeSubType.json — Add openTask, closeTask
• openmetadata-service/.../governance/workflows/elements/NodeFactory.java — Add switch cases for openTask, closeTask
• openmetadata-service/.../governance/workflows/WorkflowHandler.java — Add TCRS event handler, signal broadcasting
• openmetadata-service/.../EntityLifecycleEventDispatcher.java (or equivalent) — Register TCRS event broadcasting
• openmetadata-service/.../jdbi3/TestCaseResolutionStatusRepository.java — Add event broadcast in storeInternal(), modify openOrAssignTask() Ack case, remove Thread creation from New path, remove Thread closure from resolveTask()
• openmetadata-service/.../jdbi3/TestCaseRepository.java — Add event broadcast in resolution paths

D9: Timer subplot architecture (Slice 3 prep)

Decision (deferred to Slice 3, documented here for architectural clarity):

When timers are added (reminders, TTL), the openTask branch gains an embedded subprocess:

[openTask] → [SubProcess: timerChain]
    ┌──────────────────────────────────────────────────┐
    │ [Timer: 24h] → [Notify] → [Timer: 7d] → [Close] │
    │                                                    │
    │ SignalBoundaryEvent (interrupting):                │
    │   "tcrs_closed_{testCaseFQN}"                     │
    └──────────────────────────────────────────────────┘

The signal boundary event catches tcrs_closed_{fqn} (broadcast by closeTask in the Resolved branch's process). Since Flowable signals are broadcast, the same closeTask execution simultaneously:

1. Completes its own short-lived process (Resolved branch)
2. Interrupts any running timer subprocess in an older process (via signal boundary)

No Flowable queries needed. The two-signal pattern (tcrs_{fqn} for fan-out, tcrs_closed_{fqn} for termination) cleanly separates concerns.

---

Risks / Trade-offs

[EntityLifecycleEventDispatcher is synchronous] → The workflow process runs in the same thread as the API call. If the workflow takes too long, the API response is delayed. Mitigation: All processes in Slice 1 are short-lived (milliseconds). The idempotent nodes do a DB query + conditional insert — comparable to the current storeInternal() logic. For Slice 3, timer subprocesses enter a wait state quickly (the timer setup is fast, only the wait is long).

[Multiple processes for same business key] → Flowable does not enforce business key uniqueness. Multiple short-lived processes could theoretically run concurrently for the same FQN (e.g., rapid Ack + Assigned). Mitigation: Idempotent nodes make this safe — the second process simply no-ops. DB-level row locking on Thread creation prevents duplicates.

[TCRS event pipeline is new infrastructure] → Extending EntityLifecycleEventDispatcher for TCRS is new code. Mitigation: The pattern already exists for other entities (search indexing). The extension is small — one handler registration + one method call.

[Ack backward compatibility] → If the workflow fails to create the Thread/task (Flowable down, config error), the Ack path falls back to creating the task — because getIncidentTask() returns null and createTask() is called. This is a natural fallback, not a designed dual-path.

[Signal name collision] → Signal names include the test case FQN, which can be long. Mitigation: Flowable stores signal names as strings with no length limit in ACT_RU_EVENT_SUBSCR. FQNs are already bounded by entity naming constraints.

---

Open Questions

• EntityLifecycleEventDispatcher extension: What's the cleanest integration? A new TcrsEventHandler interface, a method on WorkflowHandler, or a lambda registration?
• Thread/task creation details: Does OpenTaskImpl create the Thread directly via FeedRepository.create(), or does it call TestCaseResolutionStatusRepository.createTask()? The latter has coupled logic (severity, metrics) that may not apply.
• Signal payload sufficiency: Can signal variables carry enough context (status, FQN, stateId, entityLink) to avoid a DB read in openTask/closeTask, or should each node read from DB independently for freshness?
• storeInternal removal scope: How much of the New and Resolved cases can we remove vs. keep as fallback? If the dispatcher fails to fire, the old code path could serve as a safety net during rollout.