Recovery

Real labs fail. A device errors, a barcode doesn't match, a method takes longer than its timing budget. Orca Framework's recovery model lets you respond without losing the rest of the workflow.

Two distinct mechanisms — they look similar but solve different problems:

• Incidents — runtime-detected events worth tracking. Some pause threads; some are informational. Operators acknowledge them.
• Recovery decisions: when an action errors with FailurePolicy.PAUSE, the thread error-pauses. The operator picks RETRY / RETRY_OP / ABORT_ACTION / ABORT_METHOD / ABORT_THREAD to advance.

Action-error PAUSE plus the five decisions is documented in Failure policies. This page covers the incident model and the special-case recovery flows.

Incidents

An incident is a structured record: what happened, where, what kind of thing it is, and what the suggested recovery is. The runtime records it; the operator acknowledges it.

from orca.runtime.incident_store import (
    SystemIncident,        # the record
    IncidentCategory,      # what kind
    IncidentSeverity,      # how bad
    RecoveryAction,        # suggested response
)

Field	What it holds
id	UUID for the incident. ID-prefix lookups work in the CLI.
timestamp	When it fired (seconds since epoch).
category	One of 18 values; see the taxonomy.
severity	INFO / WARNING / ERROR / CRITICAL.
execution_id	The execution that triggered it (may be None for system-level).
thread_id	The thread that triggered it (may be None for execution-level).
message	One-line human-readable summary.
detail	Typed dataclass matching the category (see below).
recovery_action	Suggested response — THREAD_RECOVER_RETRY, RESTART_EXECUTION, etc. Not enforced; the operator chooses.
acknowledged	True once an operator has acked it.

The detail field is a typed dataclass per category — e.g. a BARCODE_MISMATCH incident carries BarcodeMismatchDetail(expected_barcode, scanned_barcode, context). No dict[str, Any] — wire mis-shapes are caught at compile time.

Incident categories

Category	When it fires	Detail dataclass	Typical recovery
VARIABLE_RESOLUTION	An action read a variable that has no value anywhere up the resolution chain.	VariableResolutionDetail	Fix the value (var set), then recover --decision retry.
VARIABLE_VALIDATION	A set attempt was rejected by VariableDefinition.validate_value (wrong type, out of range, not in allowed values).	VariableValidationDetail	Adjust the value to fit the definition.
CO_LABWARE_TIMEOUT	A multi-labware action waited too long for co-labware to arrive.	CoLabwareTimeoutDetail	Investigate why the partner didn't arrive; consider extending the co-labware timeout or aborting.
AUTO_SPAWN_FAILED	The engine couldn't auto-spawn a contributor labware (e.g. no available source).	AutoSpawnFailedDetail	MANUAL_SPAWN — operator places the labware explicitly.
BARCODE_MISMATCH	A barcode scan didn't match the expected value.	BarcodeMismatchDetail	Verify the physical plate; either re-scan or edit-barcode.
RESERVATION_DEADLOCK	The engine found a resolvable reservation knot and cleared it automatically (a thread was rerouted or aborted).	DeadlockDetail (reroute_applied=True)	Informational; no action required.
UNRESOLVABLE_DEADLOCK	Same as above, but no automated resolution was possible. The execution paused all threads.	UnresolvableDeadlockContext	See the special case.
RECOVERABLE_TIMEOUT	A device command exceeded its advertised max_seconds. The execution paused; the operator decides extend / abort / mark-complete.	RecoverableTimeoutContext	See the special case.
ACTION_FAILED	An action raised under the default PAUSE policy; the thread error-pauses.	ActionFailedContext	THREAD_RECOVER_RETRY: fix the cause, then recover --decision retry (or abort the action/method/thread).
MOVE_FAILED	A routing move raised under the default PAUSE policy; the thread error-pauses (the move-side mirror of ACTION_FAILED). No incident is recorded when a coordination signal, e.g. gateway control or HALT, preempts the move.	MoveFailedContext	THREAD_RECOVER_RETRY: make the path safe, then recover --decision retry.
DEVICE_INIT_FAILED	A driver's initialize hook raised at startup.	DeviceInitFailedDetail	Fix the driver / device, then orca device initialize <id> to retry.
DEVICE_BUSY_EXHAUSTED	A method retried acquiring a device too many times without success.	DeviceBusyExhaustedDetail	Investigate what's hanging on the device; reservation cancel if necessary.
PLUGIN_HANDLER_EXCEPTION	A registered plugin raised in its event handler.	PluginHandlerExceptionDetail	PLUGIN_DISABLE if recurring; otherwise ack and move on.
EVENT_HANDLER_EXCEPTION	An @orca.workflow wf.on(...) handler raised.	EventHandlerExceptionDetail	Fix the handler in code.
UNRESOLVED_ANCHOR_INSERT	An insert-method / insert-action named an anchor that didn't exist in the thread's queue.	UnresolvedAnchorInsertDetail	Re-issue the insert with a valid anchor.
SYSTEM_STALL	Every live thread is internally blocked in a co-labware or a reservation wait (with at least one co-labware waiter), none is in flight, and it stays that way across a couple of ticks. The execution is paused.	SystemStallDetail(stalled_thread_ids, waits)	See the special case.
ORPHANED_BACKLOG	A receiver thread died (aborted or stopped) still owing contributions; the engine quarantines the slot and pauses the affected threads.	OrphanedBacklogContext	RESUME_EXECUTION: resume the execution to accept the partial fill (undelivered contributions are abandoned).
OTHER	Catch-all for unclassified runtime events.	OtherIncidentDetail(message_extra)	Read the message; act accordingly.

The CLI surfaces this taxonomy via orca incident list --category <category>. REST: GET /api/incidents?category=<category>. MCP: incidents_list.

Severity levels

class IncidentSeverity(str, Enum):
    INFO = "INFO"        # informational; usually auto-resolved
    WARNING = "WARNING"  # operator attention recommended but not required
    ERROR = "ERROR"      # operator attention required
    CRITICAL = "CRITICAL" # production-stopping; usually a deployment problem

In practice the engine records only WARNING and ERROR today. UNRESOLVABLE_DEADLOCK and BARCODE_MISMATCH are ERROR. INFO and CRITICAL are defined in the enum but no code path currently emits them.

Suggested recovery actions

class RecoveryAction(str, Enum):
    THREAD_RECOVER_RETRY = "THREAD_RECOVER_RETRY"  # fix the cause + recover with retry
    THREAD_RECOVER_ABORT = "THREAD_RECOVER_ABORT"  # skip the action that triggered it
    PLUGIN_DISABLE       = "PLUGIN_DISABLE"        # quarantine a misbehaving plugin
    MANUAL_SPAWN         = "MANUAL_SPAWN"          # for AUTO_SPAWN_FAILED
    RESTART_EXECUTION    = "RESTART_EXECUTION"     # stop + remove + resubmit
    RESUME_EXECUTION     = "RESUME_EXECUTION"      # resume to accept a partial fill (ORPHANED_BACKLOG)
    NONE                 = "NONE"                  # informational; no action possible

Not enforced — the operator picks what they actually do. recovery_action is the runtime's suggestion based on category; the CLI surfaces it via orca incident get <id>.

The special cases

RECOVERABLE_TIMEOUT

Fires when an in-flight device command exceeds its advertised max_seconds without a response. The engine owns this flow: the RecoverableTimeoutCoordinator parks the dispatch on an asyncio.Event and pauses the execution while the operator deliberates. It is a first-class orca-core concept, not a hosted-only one.

Three first-class operator decisions, surfaced via orca incident recoverable-timeout, the runtime methods recoverable_timeout_{extend,abort,mark_complete}, and the daemon REST routes /incidents/{id}/recoverable_timeout/{extend,abort,mark_complete}:

Decision	When
extend --by <seconds>	The command is still progressing and likely to finish. Add to the budget. Repeatable.
abort	The command is wedged. Cancel it; the thread error-pauses normally. PHYSICAL danger, --reason required.
mark-complete --response @<json>	The command actually completed (you confirmed by inspection) but the driver never returned. Forge the driver response from JSON so downstream actions can proceed. PHYSICAL danger.

A late real response from the driver/transport auto-acknowledges the incident and resumes the paused execution, so you don't lose anything if the driver finally answers while the operator is deliberating.

Detail dataclass:

@dataclass(frozen=True)
class RecoverableTimeoutContext:
    device_id: str
    command: str
    command_id: str
    elapsed_seconds: float
    max_seconds: float

UNRESOLVABLE_DEADLOCK

The engine found a reservation knot (a set of threads none of which can proceed) that it couldn't break by rerouting or auto-aborting. It detects this by AND-OR knot detection, which catches knots spanning more than one simple cycle. Every thread in the affected execution is paused so the operator can investigate from a frozen state.

Detail dataclass: UnresolvableDeadlockContext (engine-side error context doubles as the persisted detail).

There's no "recover from deadlock" CLI verb; the answer is to inspect what each thread was waiting on and either execution stop the whole thing or surgery-target one thread (thread abort-method, thread abort-thread) so the others can proceed.

SYSTEM_STALL

A multi-labware action waits for its co-labware to arrive, and that co-labware wait is unbounded by default (there is no wall-clock cap). The structural stall detector is the backstop that surfaces a genuinely wedged wait. It fires when every live thread is internally blocked in a co-labware wait OR a reservation wait, with at least one thread waiting on co-labware, none in flight, and the state holds stable across a couple of ticks (default 2). A reservation-only wait never stalls on its own.

The detail is SystemStallDetail(stalled_thread_ids, waits), naming which threads are stuck and what each is waiting on. From there the operator investigates why a partner never arrived and either frees it (extend or abort the upstream timeout, spawn a missing contributor) or execution stops the run. SYSTEM_STALL is distinct from UNRESOLVABLE_DEADLOCK: a deadlock is a reservation knot, a stall is forward progress simply never happening. Known residual: a state blocked only on a system-held or manual operator hold can trip the detector even though the operator can release it without any thread acting; the cost is bounded (one incident and a pause, cleared by releasing the hold and resuming).

Action-error recovery (recap)

When an action raises and the policy is PAUSE (the default), the thread error-pauses with an incident. Operator advances via:

orca execution thread recover <exec> <tid> <decision>

Where <decision> is one of: retry, retry-op, abort-action, abort-method, abort-thread. Full semantics in Failure policies.

Acknowledging incidents

Incidents are immutable — acknowledging replaces the record with a frozen copy carrying acknowledged=True. Ack does NOT advance the thread. A thread error-paused by an incident stays paused until you recover --decision <choice>; the ack just clears the alert.

orca incident ack <id>
orca incident ack-all --category RECOVERABLE_TIMEOUT

Manual pause and resume

Independent of incidents:

runtime.pause_thread(execution_id, thread_id)
runtime.resume_thread(execution_id, thread_id)

runtime.pause_all_threads(execution_id)
runtime.resume_all_threads(execution_id)

CLI: orca execution pause/resume and orca execution thread pause.

Mechanism	Use for
Manual pause	Planned holds — maintenance window, inspection, operator break.
Incident + recover	Unplanned failures — device errors, mismatches, timeouts.

resume does NOT un-pause an error-paused thread. Error pauses are lifted by recover with a decision; manual pauses are lifted by resume. The CLI surfaces this distinction in its verb names.

Pitfalls

• Acknowledge ≠ recover. Acking an incident clears the alert but does NOT advance the error-paused thread. You still need thread recover --decision <choice>.
• ABORT_ACTION skips ONE action, not the whole method. If the failing action was structurally necessary (the seal step in a thaw-seal-freeze flow), the next action may produce garbage.
• Reservations stay held during PAUSE. Other threads waiting on the same device queue behind. To free the device, the operator either picks ABORT_THREAD or uses reservation cancel explicitly.
• RETRY re-runs with the SAME args. Transient comms blip — fine. Logic error in arg computation — will fail the same way. Use ABORT_ACTION and edit the code.
• UNRESOLVABLE_DEADLOCK freezes every thread in the execution. Don't recover one thread expecting the others to follow; surgery the deadlock root (the thread holding the contended reservation), then resume the others.
• RecoveryAction is a suggestion, not an enforcement. The runtime puts its best guess on each incident, but the operator's decision is what actually happens. Don't write tooling that assumes the suggestion will be followed.
• runtime.resume_thread will NOT lift an error pause. Use runtime.recover_thread(exec, tid, decision) for that.

Where to go next

• Failure policies — the per-action PAUSE-vs-ABORT decision that creates error-paused threads.
• CLI: recover — the operator surface for triage and recovery decisions.
• Events — INCIDENT.* events and audit trail.