Recover from errors

A labware thread has stopped on an error and somebody has to say what happens to it. When something goes wrong (a device error, a timeout, a labware mismatch) the runtime files an incident and pauses the affected thread. These verbs are the operator's recovery surface.

See Recovery for the incident model itself. This page is the CLI.

Triage

orca incident list                                   # every incident, acknowledged or not
orca incident list --unacknowledged                  # only the open ones
orca incident list --category UNRESOLVABLE_DEADLOCK  # filter by category
orca incident list --execution <exec-id>             # scope to one execution
orca incident get <id>                               # full detail
orca incident ack <id>                               # mark acknowledged
orca incident ack-all [--category <c>]

list returns every incident by default. There is no --state flag; pass --unacknowledged to narrow to the open ones. --category and --execution filter further and combine with AND.

 id         category       severity   message                                ack
 7a3f1c0e   ACTION_FAILED  ERROR      shake command failed: drive overcur...  False

Ids resolve from a prefix of four characters or more.

get

incident 7a3f1c0e
  category: ACTION_FAILED
  severity: ERROR
  message: shake command failed: drive overcurrent
  acknowledged: False
  timestamp: 1780000042.1
  execution_id: 9b2e4108
  thread_id: 4c1a9d20
  recovery_action: THREAD_RECOVER_RETRY

--json returns the full incident, including the typed per-category detail. The shape is the same across CLI, REST and MCP.

recovery_action is a suggestion, not an enforcement. What a thread will actually accept is on the thread's own snapshot; see below.

The categories you meet most

The full table is in Recovery: incident categories. The ones that show up on operator surfaces:

Category	What it means
ACTION_FAILED	An action raised on a device operation.
MOVE_FAILED	A routing move raised and left the thread paused. The move-side mirror of ACTION_FAILED.
ACTION_CONTINUED	An operator carried on past an errored action. A warning, not a failure: it says the action's side effects are unknown and state past that point is unverified.
MOVE_CONTINUED	An operator finished a failed move by hand. Says which arrival rests on a person's word rather than on a completed place.
BARCODE_MISMATCH	The runtime's barcode expectation did not match the scan.
CO_LABWARE_TIMEOUT	A multi-labware action waited too long for co-labware to arrive.
AUTO_SPAWN_FAILED	The engine could not auto-spawn a contributor labware. Recover with thread spawn.
DEVICE_INIT_FAILED	A driver's initialize hook raised.
DEVICE_BUSY_EXHAUSTED	A method exhausted its reservation-acquisition retries.
RECOVERABLE_TIMEOUT	A device command overran its timing budget. See below.
DECK_RECONCILE_CONFLICT	The ledger and a liquid handler's own deck disagreed about where labware is. Filed by reconcile-deck, one per difference.
LEDGER_CONTRADICTED	An ad-hoc command contradicted what the record held at the time it was sent.
THREAD_DIED	A thread stopped on an uncaught error.
SYSTEM_STALL	Every live thread is internally blocked on a co-labware or reservation wait, at least one on co-labware, and none in flight. The execution is paused. This is a stall.
ORPHANED_BACKLOG	A convergence receiver died still owing contributions; the execution is paused. Recover with execution resume to accept the partial fill.
UNRESOLVABLE_DEADLOCK	The engine hit a reservation knot no automated resolver can break, a deadlock; every thread in the execution is paused.

ACTION_CONTINUED and MOVE_CONTINUED do not replace the failure they follow. The original ACTION_FAILED or MOVE_FAILED stays as it is, because continuing is not resolving.

Recover an error-paused thread

orca execution thread recover <exec> <tid> <decision>

<decision> is positional, one of six values:

Decision	What happens
retry	Re-run the whole action body from the top, re-resolving variables. Every device call it already completed runs a second time.
retry-op	Re-run only the device call that failed. The action body stays suspended at its await and the work it already did stays done.
continue	The work is dealt with; carry on to the next action. See below.
abort-action	Discard the action as work that did not happen; advance to the next action.
abort-method	Abandon the rest of the current method; advance to the next method.
abort-thread	End this thread. The execution finishes at ABORTED, not COMPLETED.

The CLI maps these to the recovery decision enum (RETRY, RETRY_OP, CONTINUE, ABORT_ACTION, ABORT_METHOD, ABORT_THREAD). See Failure policies.

recover is not resume. resume lifts a manual pause. recover lifts an error pause and tells the engine which way out.

The pause site decides which verbs it honours

A thread stops at one of nine pause sites, and not every verb applies everywhere. retry-op re-runs a suspended device call, so it needs one suspended. abort-action discards a bound action, so it needs one bound.

Read the thread before choosing. honoured_decisions on the thread's snapshot lists exactly the verbs that site will accept, and it comes from the same rule that refuses one on the call, so a verb can never be offered there and refused here. pause_site names where the thread stopped.

orca execution detail <exec-id>                 # prints the block for every error-paused thread
orca execution thread detail <exec-id> <tid>    # the same for one thread

A refused verb costs a message, not the thread: the refusal lands before anything is delivered, the thread stays paused, and you pick again. The refusal message names what the site lacks.

Pause site	Honours
ACTION_BODY	Everything except retry-op. No device call is suspended.
DEVICE_OP	All six.
MOVE	retry, continue, abort-thread. A move is not a method action, so the action-level aborts have nothing to discard and would leave the labware where the next action does not expect it.
THREAD_STEP	retry, abort-thread.
WAIT_EVENT_TIMEOUT	retry, abort-thread.
MOVE_RESOLUTION	retry, abort-thread. No route could be planned, so there is no move to call finished. Correct the position and try again.
SPAWN_CAPACITY	retry, abort-thread. The spawn runs before an action is resolved. Free capacity and try again.
ACTION_RESOLUTION	retry, abort-action, abort-method, abort-thread.
DEADLOCK	retry, abort-action, abort-method, abort-thread.

At ACTION_RESOLUTION and DEADLOCK the narrower aborts end the thread. There is no bound action for them to discard, so they cannot do what their names suggest. They are honoured anyway, deliberately: the alternative was leaving the owner and its contributors parked at PAUSED forever. This is the one place a verb does not mean what it says, and it is the reason to read honoured_decisions rather than reason from the name.

A thread that is no longer error-paused honours nothing. A thread ended by abort-thread keeps its pause site, so it can still say what aborted it.

The table above is a rendering of the engine's own rule, not a second copy of it: HONOURED_DECISIONS in orca-core is what refuses a verb, and honoured_decisions on the snapshot is served from it. Read the field, not this page, when the two could differ.

continue

continue says a person has dealt with it and the run may proceed. It makes no claim that the failed step's work happened. The ledger records the step as operator-confirmed, not executed, and files an ACTION_CONTINUED or MOVE_CONTINUED incident saying state past that point is unverified.

• After an action errored, use it when the call actually succeeded (the driver never returned) or when you fixed the situation by hand.
• After a move failed, it means you carried the labware to the move's target yourself. It is refused until the ledger says the labware is at the target, because the run has to resolve the next action from somewhere. Run orca labware edit-location first.

It is refused anywhere other than an errored action or a ledger-confirmed move.

continue also clears any fault standing on the device it was paused against, as retry and retry-op do: recovering the thread is a statement that the machine has been looked at.

After a decision

A recovery decision clears one condition. It says nothing about the rest, and the classic mistake is clearing a device fault, expecting the arm to move, and not noticing the thread it faulted is still paused. Re-read the whole list:

orca status               # against a cloud backend: one list of everything stopping the run
orca runtime status       # is the runtime built at all
orca incident list --unacknowledged
orca state unsettled      # what nobody has stated, and the verb for each row

See Monitor for how each of those reads, and State for what makes a row unsettled.

Pitfalls

• abort-action skips one action, not the method. If the broken action was structurally necessary (sealing the plate, say), the next action may fail too.
• abort-method advances to the next method. If the failing method was a pre-step for the rest of the thread, skipping it can strand the labware.
• abort-thread releases reservations and leaves labware where it physically is. Follow with labware where and reconcile with labware edit-location.
• A decision does not acknowledge the incident. The original stays in the log with what the operator did recorded beside it.
• A replacement staged for the paused step still needs a decision. replace-method stages the substitute to run next; the failed step is dropped only when you recover with abort-method or abort-action, after making the cell physically safe.

Recoverable timeouts

A device command that overruns its advertised max_seconds without returning is not a failure. The runtime files a RECOVERABLE_TIMEOUT incident, pauses every thread in the execution, and parks the held call on an event until an operator decides. The thread that sent the command is already blocked on it. The siblings are paused so they do not race ahead while you deliberate.

orca incident recoverable-timeout extend        <id> --additional-seconds <n>
orca incident recoverable-timeout abort         <id> --operator <name> --reason "..."
orca incident recoverable-timeout mark-complete <id> --operator <name> --reason "..."

Decision	When
extend	The command is still running and likely to finish. Re-arms the command timer for --additional-seconds more and acknowledges the incident. No operator or reason.
abort	The command is wedged. Fails the held command with CommandTimeoutError; the workflow propagates the failure and the paused threads resume into the error state.
mark-complete	The command actually finished but the driver never returned. Synthesizes a success response so downstream actions can proceed. Dangerous.

--operator and --reason are both required and must be non-empty on abort and mark-complete. They become the audit-trail entry. extend requires --additional-seconds greater than zero.

These verbs need a full incident UUID. The sub-app's own --help text still says "cloud only"; that string is stale. All three dispatch through the resolved client, so they reach the local daemon and an Orca AI deployment alike.

Pitfalls

• mark-complete is the most surgical and the most dangerous. You are telling the runtime a command succeeded with no proof. Downstream consumers expecting a typed response shape may fail at the workflow layer. Use it only after physically inspecting the result.
• extend is repeatable. A device that is slow but progressing can be extended repeatedly, and each extension is logged. If the new timer expires before a real response, a fresh RECOVERABLE_TIMEOUT incident is declared.
• Abort and mark-complete stop orca awaiting the call, not the instrument. For a remote driver the physical command keeps running until the wire-cancel contract propagates the cancellation to the device agent.

When the labware record is past saving

When the runtime's view of labware has gone wrong beyond individual correction, the clearing verbs are on the control page: taking labware off the platform. They clear the record only and dispose of nothing physically.

See also

• Recovery: the incident model, and what each decision does to the engine.
• Failure policies: what decides whether an error pauses a thread at all.
• Mid-flight inspection: read pause_site and honoured_decisions before you choose.
• Operator interventions: correcting a plate's location, clearing a fault, thread surgery.
• State: settling what nobody has stated once the thread is moving again.