Recover from errors

When something goes wrong — a device error, a timeout, a labware mismatch — the runtime fires an incident and pauses the affected thread. These verbs are the operator's recovery surface.

See Recovery for the incident model itself; this page is just the CLI.

Triage

orca incident list                                  # every incident (ack'd and not)
orca incident list --unacknowledged                 # only unacknowledged
orca incident list --category UNRESOLVABLE_DEADLOCK  # filter by category
orca incident list --execution <exec-id>            # scope to one execution
orca incident get <id>                              # full detail
orca incident ack <id>                              # mark acknowledged
orca incident ack-all --category RECOVERABLE_TIMEOUT

list returns ALL incidents by default. There is no --state flag; pass --unacknowledged to narrow to the open ones. --category and --execution filter further and AND-combine.

Eighteen categories the engine emits; see Recovery: incident categories for the full table with detail dataclasses and suggested recovery per category. The most common in operator surfaces:

Category	What it means
ACTION_FAILED	An action raised on a device operation.
MOVE_FAILED	A routing move raised and left the thread paused (the move-side mirror of ACTION_FAILED).
BARCODE_MISMATCH	Runtime expectation about a barcode didn't match the scan.
CO_LABWARE_TIMEOUT	A multi-labware action waited too long for co-labware to arrive.
AUTO_SPAWN_FAILED	The engine couldn't auto-spawn a contributor labware.
DEVICE_INIT_FAILED	A driver's initialize hook raised.
DEVICE_BUSY_EXHAUSTED	A method exhausted its reservation-acquisition retries.
RECOVERABLE_TIMEOUT	A device command exceeded its timing budget; operator decides extend / abort / mark-complete.
SYSTEM_STALL	Every live thread is internally blocked on a co-labware or reservation wait (at least one on co-labware), none in flight; the execution is paused.
ORPHANED_BACKLOG	A convergence receiver died still owing contributions; the execution is paused. Recover with execution resume to accept the partial fill.
UNRESOLVABLE_DEADLOCK	The engine hit a reservation knot no automated resolver can break; every thread in the execution is paused.

get example

incident 7a3f1c0e
  category:         ACTION_FAILED
  severity:         ERROR
  message:          "shake command failed: drive overcurrent"
  acknowledged:     false
  timestamp:        2026-05-26 14:00:42
  execution_id:     9b2e4108
  thread_id:        tid-3
  recovery_action:  (pending operator decision)

--json returns the full IncidentDetailDTO. Stable shape across CLI / REST / MCP.

Recover an error-paused thread

Once an incident is acknowledged (or in some cases without acknowledgement), tell the thread how to proceed:

orca execution thread recover <exec> <tid> <decision>

<decision> is a positional argument, one of five values:

Decision	What happens
retry	Re-run the failing action from the start. Same args.
retry-op	Re-run ONLY the failed device operation, leaving the rest of the action body suspended at its await. Valid only while the thread is paused on a device-operation seam.
abort-action	Skip the failing action; advance to the next action in the method.
abort-method	Abandon the rest of the current method; advance to the next method in the thread.
abort-thread	Mark the thread aborted; release its reservations.

The CLI maps these to the RecoveryDecision enum (RETRY, RETRY_OP, ABORT_ACTION, ABORT_METHOD, ABORT_THREAD). See Failure policies for the full semantics.

recover is NOT resume. resume lifts a manual pause. recover lifts an ERROR pause and tells the engine which way out.

Pitfalls

• abort-action skips ONE ACTION, not the whole method. If the broken action was structurally necessary (e.g. "seal the plate"), the next action may also fail.
• abort-method advances to the next METHOD in the thread, not the next action. If the failing method's actions were a pre-step for the rest of the thread, skipping the whole method may strand the labware.
• abort-thread releases reservations but leaves labware where it physically is. Follow up with labware where and reconcile with labware edit-location if needed.
• retry does NOT re-acknowledge the incident. The original incident stays in the log with the operator's decision attached.

Recoverable timeouts

A RECOVERABLE_TIMEOUT is fired when a device command exceeds its timing budget but the system can't unilaterally decide what to do — only the operator can. Three first-class decisions:

orca incident recoverable-timeout extend        <id> --additional-seconds <n>
orca incident recoverable-timeout abort         <id> --operator <name> --reason "..."
orca incident recoverable-timeout mark-complete <id> --operator <name> --reason "..."

Decision	When
extend	The command is still running and likely to finish. Grant --additional-seconds <n> more budget. extend takes no operator/reason; it just re-arms the command timer and acknowledges the incident.
abort	The command is wedged. Fails the held command with CommandTimeoutError; the workflow propagates the failure and paused threads resume into the error state. --operator and --reason required.
mark-complete	The command actually completed (you confirmed by inspection) but the driver never returned. Synthesizes a success response so downstream actions can proceed. DANGEROUS. --operator and --reason required.

--operator <name> + --reason "..." are required on abort and mark-complete (not extend). They become the audit-trail entry.

Pitfalls

• mark-complete is the most surgical and the most dangerous: you're telling the runtime "this command succeeded" without proof. Downstream consumers expecting a typed response shape may fail at the workflow layer. Only use it when you've physically inspected the result.
• extend is repeatable. A device that's slow but progressing can be extended multiple times. Each extension is logged. If the new timer expires before a real response, a fresh RECOVERABLE_TIMEOUT incident is declared.
• These verbs work against both backends. The held command's in-flight future lives on whichever control plane is running it, so the decisions reach the local daemon and a Swarm deployment alike. (Note: orca-core's own --help text still labels the sub-app "cloud only" — that string is stale; the verbs dispatch through the resolved client either way.) Cloud has no listing endpoint on the CLI, so these verbs require full incident UUIDs.

W15 panic chain

When labware state has gone hopelessly wrong (e.g. a stop mid-run left ten plates in unknown locations and reconciling individually is hopeless), the panic chain wipes labware state for a submission, one plate, or the whole deployment.

orca labware clear-submission <submission-id> --force   # nuke labware for one submission
orca labware discharge        <labware-id>    --force   # forget one plate
orca labware clear-all        --force --yes             # nuke all labware

--force bypasses the active-execution / active-thread refusal on each verb. clear-all additionally prompts on an interactive TTY; on a non-TTY (CI, scripts) it requires --yes. There is no --confirm or --reason flag on these. They do NOT physically dispose of anything — they only clear the runtime's record. Physical disposition is the operator's job. All three work against both backends.

Pitfalls

• clear-all refuses on a non-TTY without --yes. Pass --yes (or -y) explicitly in scripts that genuinely intend the wipe; without it on a non-TTY the verb exits non-zero rather than wiping a live deck.
• discharge removes a single labware instance from the runtime. Any thread that was tracking that labware will fail on its next reference.
• None of these are for "the workflow ran weird, let me reset and retry". They're for "the system view is unrecoverably wrong, start over."