Failure policies

When an @orca.action raises, the engine decides what to do based on the action's failure policy. There are two values and one default.

from orca.workflow_models.status_enums import FailurePolicy

Value	What happens when the action raises
PAUSE	The thread pauses with an incident. Reservations are held. Operator picks a recovery decision to advance. Default.
ABORT	The thread is killed. Reservations released. No operator input requested.

Note: there is no RETRY value. Automatic retry is deliberately not a policy — if a class of failure is retryable, the action body should catch and retry itself. Operator-driven retry is one of the recovery decisions after a PAUSE.

Where to set it

Two surfaces, action wins.

On an action

@orca.action(device=sealer, inputs=[sample_plate], failure_policy=FailurePolicy.ABORT)
async def seal(ctx: ActionContext):
    await ctx.device().seal(temperature=100, duration=60)

The action's policy is what the engine reads when the action raises. If you set it on the action, you don't need to set it on the method.

On a method

@orca.method(failure_policy=FailurePolicy.ABORT)
async def critical_step(ctx: MethodContext):
    yield delicate_action

The method-level policy is the legacy authoring path. Today the engine still reads from the executing action (see current_failure_policy on MethodInstance), so an action-level setting overrides whatever the method declared.

Recovery decisions

When PAUSE fires, the thread sits with an incident. An operator advances it with one of five decisions:

from orca.workflow_models.status_enums import RecoveryDecision

Decision	What happens
RETRY	Re-run the whole action from the start with the same args.
RETRY_OP	Re-run only the failed device operation, leaving the rest of the action body suspended at its await. Valid only while the thread is paused on a device-operation seam.
ABORT_ACTION	Skip the failing action; advance to the next action in the method.
ABORT_METHOD	Abandon the rest of the current method; advance to the next method in the thread.
ABORT_THREAD	Mark the thread aborted. Reservations release.

CLI: orca execution thread recover <exec> <tid> <decision> where <decision> is one of retry, retry-op, abort-action, abort-method, abort-thread. Daemon REST: POST /operations/recover-thread.

Engine override: OverrideWithPauseError

A few exception classes signal "this isn't a method failure — some external coordination (operator hitting halt, gateway holding a device, maintenance window) has taken state that prevents the method from continuing." These exceptions inherit from OverrideWithPauseError and force PAUSE regardless of what FailurePolicy the action declared.

This is engine-internal — workflow authors don't write these, drivers and the runtime do. The reason it surfaces in user-facing docs: if you declared FailurePolicy.ABORT and a thread ends up PAUSED anyway, the cause is an OverrideWithPauseError somewhere. The incident detail will show which exception.

Shared-action failures

When several threads converge on ONE shared action (a pooled step where multiple contributors meet on the same device operation) and that action fails, the engine pauses the WHOLE action group as a unit, not just the thread that raised. The operator sees one stuck action, not a scattered set of paused threads.

Recovery of a shared action is group-wide: the first recovery decision wins and propagates to every participant. No participant can half-resume, and recovering more than one paused participant is idempotent (the terminal outcome is published once and read by all). So you make one decision on the shared action and the whole group advances together.

Pitfalls

• Default is PAUSE, not ABORT. A method that doesn't specify a policy will pause on first error and sit there until an operator recovers. For unattended runs (CI, batch jobs) consider setting FailurePolicy.ABORT explicitly so the run terminates instead of waiting forever.
• ABORT_ACTION skips ONE action. If the action that failed was structurally necessary (the seal step in a thaw → seal → freeze flow), the next action may also fail or produce garbage. The operator deciding ABORT_ACTION is asserting "the next step can proceed without this one."
• Reservations stay held during PAUSE. A thread paused on the shaker keeps the shaker reservation. Other threads waiting for that device queue behind. If you intend to free the device during recovery investigation, the operator should ABORT_THREAD or use reservation cancel explicitly.
• RETRY re-runs the action with the SAME args. If the failure was a transient comms blip, that's fine. If it was a logic error in how the args were computed, retry will fail the same way — use ABORT_ACTION and edit the workflow code.

Where to go next

• Recovery — full incident model and per-category operator decisions.
• CLI: recover — the operator surface for the recovery decisions.