Recovery

A device errors, a move drops a plate, a driver stops answering. Orca Framework stops the labware thread that hit the fault and waits for a person, so you can answer it without losing the rest of the workflow. This page is written for that person: what a paused thread looks like, how to see why, what you can tell it to do, and what each of those answers does. The incident record behind it all is further down.

A paused thread waits for you

Two different things stop a thread, and they are lifted differently.

• An error pause. An action or a move raised under FailurePolicy.PAUSE, which is the default. The thread stops where it is, the runtime files an incident, and the thread keeps its reservations, so other threads waiting on that device queue behind it. It waits until an operator sends a recovery decision. resume does not lift it. The policy that produced it is in Failure policies.
• A manual pause. Somebody paused the thread or the execution for a maintenance window or an inspection. resume lifts it, and no decision is needed. See Manual pause and resume.

Acknowledging the incident does not advance the thread either. Ack clears the alert. A recovery decision moves the thread.

Why it stopped

One call answers it for the whole run:

blockers = await runtime.blockers()

Each entry names its kind, a headline, and the remedies that clear it, worst first. The recovery verbs on a remedy are read off the same table that refuses them, so a verb offered here can never be refused on the call. An unacknowledged incident is not a blocker; a blocker carries its incident_id. See Runtime.

Per thread, read the snapshot:

orca execution detail <exec-id>                 # every error-paused thread in the execution
orca execution thread detail <exec-id> <tid>    # the same for one thread

pause_site says where the thread stopped. honoured_decisions lists exactly the verbs that site will accept.

The incidents themselves:

orca incident list --unacknowledged
orca incident get <id>

The recovery decisions

Six verbs, in escalation order, cheapest first.

from orca.workflow_models.status_enums import RecoveryDecision

Decision	What happens
RETRY	Re-run the whole action from the start with the same args. Every device call the action already completed runs again.
RETRY_OP	Re-run only the failed device operation, leaving the rest of the action body suspended at its await. Valid only while the thread is paused inside a device call.
CONTINUE	The operator says the situation is dealt with, so the run carries on.
ABORT_ACTION	Discard the failing action as work that never happened; advance to the next action in the method.
ABORT_METHOD	Abandon the rest of the current method; advance to the next method in the thread.
ABORT_THREAD	Mark the thread aborted. Reservations release. The execution finishes at ABORTED, not COMPLETED.

orca execution thread recover <exec> <tid> <decision>

Where <decision> is retry, retry-op, continue, abort-action, abort-method or abort-thread. Daemon REST: POST /operations/recover-thread.

CONTINUE and ABORT_ACTION are not the same answer

Both advance to the next action. They say different things. ABORT_ACTION discards the action as work that never happened. CONTINUE records it as operator-confirmed rather than executed. Neither promises the work got done.

CONTINUE means the operator has dealt with it and the run may carry on: the call really did succeed, or they fixed the situation by hand.

At a failed move, CONTINUE means the operator carried the labware to the target themselves. It is refused there until the ledger says the labware IS at the target, because the run has to plan the next action from somewhere. The refusal names where the ledger currently has the labware and tells you the sequence: put it at the target, record that with orca labware edit-location, then CONTINUE. If it is somewhere else, record that instead and RETRY, and the move replans from wherever you said.

CONTINUE is refused entirely outside an errored action and a ledger-confirmed move. There is nothing to declare finished at a pause site where no action or move is bound.

What each pause site honours

A thread stops in nine places, and they do not accept the same verbs. RETRY_OP re-runs a suspended device call, so it needs one suspended. ABORT_ACTION discards a bound action, so it needs one bound.

The engine says which verbs it will take, and refuses the rest. ThreadSnapshot.honoured_decisions carries them, and the refusal happens before anything is delivered, so a wrong pick costs a message rather than the thread. Read the field and offer exactly those; do not send a decision to find out.

Pause site	Where the thread stopped	Honours
ACTION_BODY	An action's Python raised.	RETRY, CONTINUE, ABORT_ACTION, ABORT_METHOD, ABORT_THREAD
DEVICE_OP	Suspended inside one device call.	all six
MOVE	A transporter move raised.	RETRY, CONTINUE, ABORT_THREAD
THREAD_STEP	The thread body itself raised, outside any action.	RETRY, ABORT_THREAD
WAIT_EVENT_TIMEOUT	A wait_event ran out of time.	RETRY, ABORT_THREAD
ACTION_RESOLUTION	Failed before any action was bound.	RETRY, ABORT_ACTION, ABORT_METHOD, ABORT_THREAD
MOVE_RESOLUTION	No route to plan, usually because the labware is recorded somewhere nothing serving this move can reach.	RETRY, ABORT_THREAD
DEADLOCK	The reservation graph could not be resolved.	RETRY, ABORT_ACTION, ABORT_METHOD, ABORT_THREAD
SPAWN_CAPACITY	An auto-spawn hit a capacity limit before an action existed.	RETRY, ABORT_THREAD

A verb does not always do what its name suggests. At ACTION_RESOLUTION and DEADLOCK there is no bound action, so ABORT_ACTION and ABORT_METHOD cannot discard one: they end the thread. That is deliberate. The alternative was leaving the owner of a rendezvous and all its contributors parked at PAUSED forever. Every refusal message names what the site lacks, so an operator reading it can work out why.

At MOVE, the action-level aborts are refused rather than widened, because discarding an action there would leave the labware where the next action does not expect it.

Incidents

An incident is a structured record: what happened, where, what kind of thing it is, and what the suggested recovery is. The runtime records it. The operator acknowledges it. Some incidents pause threads. Some are informational.

from orca.runtime.incident_store import (
    SystemIncident,        # the record
    IncidentCategory,      # what kind
    IncidentSeverity,      # how bad
    RecoveryAction,        # suggested response
)

Field	What it holds
id	UUID for the incident. ID-prefix lookups work in the CLI (4 characters is enough).
timestamp	When it fired (seconds since epoch).
category	See the taxonomy.
severity	INFO / WARNING / ERROR / CRITICAL.
execution_id	The execution that triggered it. None for system-level.
thread_id	The thread that triggered it. None for execution-level.
message	One-line human-readable summary.
detail	Typed dataclass matching the category.
recovery_action	Suggested response. Not enforced; the operator chooses.
acknowledged	True once an operator has acked it.

The detail field is a typed dataclass per category. A DECK_RECONCILE_CONFLICT incident carries DeckReconcileConflictDetail(device_name, labware_id, labware_name, position_id, reason, driver_site, blocking_labware_name, detail). No dict[str, Any], so a wire mis-shape is caught at type-check time.

Every recorded incident also fires an INCIDENT.<id>.<category> event, with the summary fields on it. A sink that wants to react to faults subscribes to the event stream rather than polling the store. See Events.

Severity levels

class IncidentSeverity(str, Enum):
    INFO = "INFO"          # informational; usually auto-resolved
    WARNING = "WARNING"    # operator attention recommended but not required
    ERROR = "ERROR"        # operator attention required
    CRITICAL = "CRITICAL"  # production-stopping; usually a deployment problem

The engine records only WARNING and ERROR today. INFO and CRITICAL are defined in the enum but no code path emits them.

Suggested recovery actions

class RecoveryAction(str, Enum):
    THREAD_RECOVER_RETRY    = "THREAD_RECOVER_RETRY"     # fix the cause, recover with retry
    THREAD_RECOVER_RETRY_OP = "THREAD_RECOVER_RETRY_OP"  # re-run just the failed device call
    THREAD_RECOVER_ABORT    = "THREAD_RECOVER_ABORT"     # skip the action that triggered it
    PLUGIN_DISABLE          = "PLUGIN_DISABLE"           # quarantine a misbehaving plugin
    MANUAL_SPAWN            = "MANUAL_SPAWN"             # for AUTO_SPAWN_FAILED
    RESTART_EXECUTION       = "RESTART_EXECUTION"        # stop + remove + resubmit
    RESUME_EXECUTION        = "RESUME_EXECUTION"         # resume to accept a partial fill
    NONE                    = "NONE"                     # informational; no action possible

Not enforced. The operator picks what they actually do. recovery_action is the runtime's suggestion based on category; the CLI surfaces it via orca incident get <id>.

Incident categories

These are the categories the engine records today, each from a declare_* entry point on the runtime.

Category	When it fires	Severity	Suggested recovery
ACTION_FAILED	An action raised under the default PAUSE policy; the thread error-pauses.	ERROR	THREAD_RECOVER_RETRY_OP if the thread is suspended inside a device call, otherwise THREAD_RECOVER_RETRY.
MOVE_FAILED	A routing move raised under PAUSE; the thread error-pauses. The move-side mirror of ACTION_FAILED.	ERROR	THREAD_RECOVER_RETRY. A move is not an action body suspended in a call, so RETRY_OP is refused there.
ACTION_CONTINUED	An operator answered CONTINUE at an errored action. The ACTION_FAILED record stays unacknowledged: continuing is not resolving.	WARNING	NONE. Nothing to recover, something to be aware of: state past that point is unverified.
MOVE_CONTINUED	An operator finished a failed move by hand and answered CONTINUE.	WARNING	NONE. Says which arrival rests on a person's word rather than a completed place.
THREAD_DIED	A thread stopped on an error that escaped every handler, so there is no paused thread to recover.	ERROR	NONE. The incident names where the labware was left, because clearing it by hand is the only way it moves.
UNRESOLVABLE_DEADLOCK	A reservation knot the engine could not break. Every thread in the execution is paused.	ERROR	THREAD_RECOVER_ABORT. See the special case.
SYSTEM_STALL	Every live thread is internally blocked and none is in flight, stably. The execution is paused.	ERROR	RESTART_EXECUTION. See the special case.
RECOVERABLE_TIMEOUT	A device command exceeded its advertised max_seconds with no answer. The threads pause while an operator decides.	WARNING	NONE; three dedicated decisions instead. See the special case.
ORPHANED_BACKLOG	A receiver thread died still owing contributions. The engine quarantines the slot and pauses the affected threads.	ERROR	RESUME_EXECUTION: resume to accept the partial fill. Undelivered contributions are abandoned.
DEVICE_INIT_FAILED	A driver's bring-up raised during lazy first-execution initialization.	ERROR	RESTART_EXECUTION. Bring-up is not resumable from there: fix the device and resubmit.
DECK_RECONCILE_CONFLICT	The ledger and a liquid handler's driver disagree about one labware.	ERROR	NONE. Neither side is changed; the operator settles it. See Runtime state.
LEDGER_CONTRADICTED	An ad-hoc operator command only makes sense if the record was wrong. The command is believed and folded.	WARNING	NONE. The fold is now wrong by an amount nothing can compute, so look at the labware and state its contents.
UNRESOLVED_ANCHOR_INSERT	A Before / After insert was still pending when the lane closed. anchor_reached says whether the anchor came past and the insert still never ran, or the anchor never appeared at all.	WARNING	NONE. Informational: it makes an otherwise-silent drop queryable.

Ten further categories are defined in IncidentCategory and carry persistence mappings, but nothing in the engine emits them today: VARIABLE_RESOLUTION, VARIABLE_VALIDATION, CO_LABWARE_TIMEOUT, AUTO_SPAWN_FAILED, BARCODE_MISMATCH, RESERVATION_DEADLOCK, DEVICE_BUSY_EXHAUSTED, PLUGIN_HANDLER_EXCEPTION, EVENT_HANDLER_EXCEPTION, OTHER. Don't build a surface that assumes one of them will arrive.

CLI: orca incident list --category <category>. Daemon REST: GET /incidents?category=<category>.

The special cases

RECOVERABLE_TIMEOUT

Fires when an in-flight device command exceeds its advertised max_seconds without a response. Drivers that declare no timing get a default of 600 seconds. The engine owns the flow: the coordinator parks the dispatch on an event and pauses the threads while the operator deliberates.

Three operator decisions, surfaced via orca incident recoverable-timeout, the runtime methods recoverable_timeout_{extend,abort,mark_complete}, and the daemon routes POST /incidents/{id}/recoverable_timeout/{extend,abort,mark_complete}:

Decision	When
extend --additional-seconds <n>	The command is still progressing and likely to finish. Add to the budget. Repeatable. The execution stays paused.
abort --operator <who> --reason <why>	The command is wedged. Cancel it; the thread error-pauses normally and the failure policy fires. PHYSICAL danger; operator and reason are required.
mark-complete --operator <who> --reason <why>	The command actually completed (you confirmed by inspection) but the driver never returned. Synthesizes success so downstream actions proceed, and clears the device fault, because someone looked. PHYSICAL danger.

All three acknowledge the incident and resume the threads the timeout paused.

If the driver's real answer arrives while the operator is deliberating, an extend picks it up immediately and uses it: nothing is lost. abort and mark-complete cancel the in-flight call and discard whatever it was about to return, because the operator's decision is authoritative. The gateway sends a cancel for the command id so a late orphan response cannot resolve a settled call. For a remote driver, the physical command keeps running until the wire-level cancel contract reaches the on-prem agent.

@dataclass(frozen=True)
class RecoverableTimeoutContext:
    device_id: str
    command: str
    command_id: str
    elapsed_seconds: float
    max_seconds: float

UNRESOLVABLE_DEADLOCK

The engine found a reservation knot (a set of threads none of which can proceed) that it could not break by rerouting or auto-aborting. It detects this with AND-OR knot detection, which catches knots spanning more than one simple cycle. Every thread in the affected execution is paused so the operator can investigate from a frozen state.

There is no "recover from deadlock" verb. Inspect what each thread was waiting on, then either stop the whole execution or surgically end one thread so the others can proceed. Threads paused at DEADLOCK honour ABORT_ACTION and ABORT_METHOD, both of which end the thread there.

SYSTEM_STALL

A co-labware wait is unbounded by design, because a partner may legitimately run a multi-hour action first. The structural stall detector is the backstop.

It fires when every live thread is internally blocked in a co-labware wait or a reservation wait, with at least one thread waiting on co-labware, none in flight, and that exact shape holds across consecutive checks. A reservation-only wait never stalls on its own: reservation contention lasting minutes or hours is normal, and genuine reservation cycles belong to the deadlock detector. Reservation waits join the rule only while the whole system is quiescent, so a busy sibling execution can never mask a pure co-labware stall in another.

Operator waits (PAUSED, awaiting a manual place or remove, AWAITING_EVENT) are legitimate indefinite waits on the outside world and block a verdict by design.

SystemStallDetail(stalled_thread_ids, waits) names which threads are stuck and what each is waiting on. The execution is paused with reason="system", and anything awaiting it through runtime.wait(...) gets a SystemStallError immediately rather than blocking blind. Resuming clears the episode.

One residual false positive: a state blocked only on a system-held or manual operator hold can trip the detector even though the operator can release it without any thread acting. The cost is bounded: one incident and a pause, cleared by releasing the hold and resuming.

THREAD_DIED

Every fault the lab can produce parks a thread for a recovery decision instead. Reaching THREAD_DIED means the error escaped that handling. There is no paused thread to recover and no prompt: the run carries on without this thread and its labware sits where the crash left it. The incident names that position, because clearing it by hand is the only way it moves.

Manual pause and resume

Independent of incidents:

runtime.pause_thread(execution_id, thread_id)
runtime.resume_thread(execution_id, thread_id)

runtime.pause_execution(execution_id)     # latch + fan out
runtime.resume_execution(execution_id)
runtime.pause_all_threads(execution_id)   # fan out only
runtime.resume_all_threads(execution_id)

CLI: orca execution pause/resume and orca execution thread pause.

Mechanism	Use for
Manual pause	Planned holds: maintenance window, inspection, operator break.
Incident + recover	Unplanned failures: device errors, failed moves, timeouts.

resume does NOT un-pause an error-paused thread. An error pause is lifted by a recovery decision; a manual pause is lifted by resume. resume_all_threads reports the difference in its counters: resumed, pause_cancelled, error_skipped, completed_skipped.

A stop ends an error-paused thread

An error-paused thread watches the recovery channel and nothing else, so asking it to stop used to reach nobody: only cancelling its task ended one, and every shutdown that asked politely first waited out its full drain timeout.

A cooperative stop now resolves that pause with ABORT_THREAD, which is the same unwind an operator would get. The action gives its device back and records the operations it is dropping, instead of abandoning both. A contributor following a peer's action is excluded: its own wait already races the stop, and feeding it a decision would consume the one its owner is waiting for.

resume_all_threads also refuses outright, returning all zeros, while the execution-level pause latch is set. That is what makes an operator's stop outrank a system pause: the recoverable-timeout coordinator resumes threads directly and cannot lift a latch a person set.

Acknowledging incidents

Incidents are immutable. Acknowledging replaces the record with a frozen copy carrying acknowledged=True. Ack does NOT advance the thread. A thread error-paused by an incident stays paused until you recover it with a decision; the ack just clears the alert.

orca incident ack <id>
orca incident ack-all --category RECOVERABLE_TIMEOUT

Recording an incident from your own code

The runtime's declare_* methods are the entry points, and the engine calls them automatically for the categories above. They are public because a hosted layer or a plugin sometimes needs to declare on the engine's behalf.

Method	Records
declare_action_failure(execution_id, thread_id, context)	ACTION_FAILED
declare_action_continued(execution_id, thread_id, context)	ACTION_CONTINUED
declare_move_failure(execution_id, thread_id, context)	MOVE_FAILED
declare_move_continued(execution_id, thread_id, context)	MOVE_CONTINUED
declare_thread_death(execution_id, thread_id, context)	THREAD_DIED
declare_unresolvable_deadlock(execution_id, context)	UNRESOLVABLE_DEADLOCK and pauses every thread in the execution
declare_device_init_failure(execution_id, error)	DEVICE_INIT_FAILED
declare_orphaned_backlog(execution_id, context)	ORPHANED_BACKLOG
declare_recoverable_timeout(execution_id, context)	RECOVERABLE_TIMEOUT
declare_unresolved_anchor_insert(execution_id, thread_id, anchor_name, direction, target_type, item_name, anchor_reached)	UNRESOLVED_ANCHOR_INSERT

Only declare_unresolvable_deadlock fans a pause out to the whole execution. declare_action_failure and declare_move_failure add the queryable record next to a thread that already paused itself.

Pitfalls

• Acknowledge is not recover. Acking clears the alert but does NOT advance the error-paused thread.
• Do not send a decision to find out whether it applies. Read honoured_decisions off the thread first.
• ABORT_ACTION at a pause site with nothing bound ends the thread. At ACTION_RESOLUTION and DEADLOCK it is not "skip one action".
• ABORT_ACTION skips ONE action, not the whole method, where an action IS bound. If the failing action was structurally necessary, the next one may produce garbage.
• Reservations stay held during PAUSE. Other threads waiting on the same device queue behind. To free it, ABORT_THREAD or reservation cancel.
• RETRY re-runs with the SAME args, so every device call the action already completed runs again. Fine for a transient comms blip; wrong for a logic error in how the args were computed. Use ABORT_ACTION and fix the workflow code instead.
• CONTINUE at a move is refused until the ledger agrees. Record the position first.
• UNRESOLVABLE_DEADLOCK freezes every thread in the execution. Don't recover one and expect the others to follow; deal with the thread holding the contended reservation, then resume the rest.
• RecoveryAction is a suggestion, not an enforcement. Don't write tooling that assumes it will be followed.
• Ten incident categories are defined but never emitted. Don't build a surface around one.

See also

• Failure policies: the per-action pause-or-abort decision that creates error-paused threads.
• Runtime state: deck conflicts, contradicted ledgers, and the unsettled worklist.
• CLI: recover: the operator surface for triage and recovery decisions.
• Runtime: the one list of blockers, and pausing an execution.
• Events: INCIDENT.* and DEVICE.* events.