Failure policies

An action that raises stops its labware thread and waits for a person. That is the default. The alternative is to kill the thread instead, and you choose per action. This page is for whoever writes the action and has to pick.

The default is pause

Policy	What happens when the action raises
PAUSE	The thread pauses and the runtime files an incident. Reservations stay held. An operator picks a recovery decision to advance it. This is the default.
ABORT	The thread is killed. Reservations are released. Nobody is asked anything.

from orca.workflow_models.status_enums import FailurePolicy

There is no RETRY policy. Automatic retry is deliberately not one: if a class of failure is retryable, the action body should catch it and retry itself. Operator-driven retry is one of the recovery decisions after a pause.

For an unattended run (CI, an overnight batch) declare FailurePolicy.ABORT so the run ends instead of waiting for an operator who is not there.

Where to set it

Set it on the action. That is the only declaration the engine reads when an action raises.

@orca.action(device=sealer, inputs=[sample_plate], failure_policy=FailurePolicy.ABORT)
async def seal(ctx: ActionContext):
    await ctx.device().seal(temperature=100, duration=60)

Every action template starts at FailurePolicy.PAUSE. The decorator overrides that only when you pass one. On an error the engine reads current_failure_policy, which is the executing action's own policy and nothing else.

The keyword on a method is metadata

A method also accepts failure_policy:

@orca.method(failure_policy=FailurePolicy.ABORT)
async def critical_step(ctx: MethodContext):
    yield delicate_action

It does not govern an action error. The method's policy is stored on the method instance and reported back on method listings. No code path consults it when an action raises. An action inside critical_step still pauses unless that action declares ABORT itself. Treat the method-level keyword as metadata, not as a default its actions inherit.

Those two are the whole list of declaration sites. There is no thread-level, workflow-level, config-file or runtime-mutable failure policy.

Recovery decisions

A paused thread advances when an operator sends one of six decisions.

from orca.workflow_models.status_enums import RecoveryDecision

RETRY, RETRY_OP, CONTINUE, ABORT_ACTION, ABORT_METHOD, ABORT_THREAD, in that order, which is the escalation ladder: cheapest first. What each one does is in Recovery.

A decision is refused if the pause site does not honour it. A thread stops in nine different places and they accept different verbs. Send one that does not apply and the call is refused, the thread stays paused, and the message says which verbs it will take. Read honoured_decisions off the thread rather than sending a decision to find out. The table is in Recovery.

At two pause sites the narrower aborts end the thread. Where nothing is bound yet, ABORT_ACTION and ABORT_METHOD cannot discard an action, and the engine deliberately ends the thread instead of leaving a rendezvous parked forever.

CLI: orca execution thread recover <exec> <tid> <decision> where <decision> is one of retry, retry-op, continue, abort-action, abort-method, abort-thread. Daemon REST: POST /operations/recover-thread.

Some errors force a pause

A few exception classes say that this is not a method failure. Some external coordination has taken state that stops the method continuing: an operator took control of a device, a gateway is holding it, a mover is already carrying something. These inherit from OverrideWithPauseError and force PAUSE regardless of what FailurePolicy the action declared.

from orca.workflow_models.error_policy_overrides import OverrideWithPauseError

That marker class is the entire API. There is no registry and no function: an exception opts in by inheriting from it. Three do today, all in orca.resource_models.device_error: MoverAlreadyHoldingError, DeviceUnderExternalControlError and DeviceStateDivergenceError.

Drivers and the runtime raise these. Workflow authors do not. It is here for one reason: if you declared FailurePolicy.ABORT and a thread ends up paused anyway, an OverrideWithPauseError is why. These do not record an incident, so read the thread's last_error rather than the incident list to see which one it was.

The four kinds of action error, and why only a method failure follows the declared policy, are named in orca/workflow_models/error_policy_overrides.py.

When threads share the failing action

Several threads can converge on ONE shared action: a pooled step where several contributors meet on the same device operation. When that action fails, the engine pauses the WHOLE action group as a unit, not just the thread that raised. The operator sees one stuck action instead of a scatter of paused threads.

Recovery is group-wide. The first recovery decision wins and propagates to every participant. No participant can half-resume, and recovering more than one paused participant is idempotent: the terminal outcome is published once and read by all. You make one decision on the shared action and the whole group advances together.

A group's decision is delivered on the group's own channel, and only from a thread whose pause site is the group's. A contributor that failed its own move keeps its own channel, so it cannot swallow a decision meant for the owner. A contributor also reads the group's live pause site rather than the one it stamped when it first saw the pause, because the owner can fail inside a device call, retry, and fail again in the action body around it.

Pitfalls

• The default is PAUSE, not ABORT. An action that declares no policy pauses on its first error and sits there until an operator recovers it.
• Reservations stay held during a pause. A thread paused on the shaker keeps the shaker reservation, and other threads waiting for that device queue behind it. To free the device during an investigation, ABORT_THREAD or use reservation cancel.
• The keyword on a method does not reach its actions. Set the policy on every action you want to abort.

See also

• Recovery: what a paused thread looks like, the six decisions, and what each pause site honours.
• CLI: recover: the operator surface for triage and recovery.
• Actions: what an action is, and the rest of what the decorator takes.
• Methods: how actions are sequenced into a method.