Runtime state and the unsettled worklist

The runtime keeps a record of the physical world: where each labware is, what it holds, what is mounted on a liquid handler's channels. Every routing decision and every pipetting step is planned from that record, so it matters more that the record is honest than that it is complete. This page is for whoever has to decide whether a number is still true before acting on it.

Two rules make it honest, and everything on this page follows from them.

One owner per fact. Each physical fact is answered by exactly one ledger and nowhere else. A read that finds nothing answers UNKNOWN. It never falls back to a template's declaration, a driver's report, or a default, because each of those turns a guess into an answer that looks measured. A template declaration is the ledger's opening entry, written once at birth and never consulted again.

A fact is either known, or it is on the worklist. There is no third case. Every fact an operator can be asked about is a fact an operator can edit.

These are ADR-014 in the orca-core repository, and a CI guard enforces the first one.

The facts and their owners

Fact	Owner	Read it with
Where a labware is	the placement ledger	runtime.labware.get_by_id(...), orca labware where
What a labware holds (well volumes, tips on a rack)	the contents ledger, folded from operation history	runtime.labware.get_well_volumes(...) / get_tip_state(...)
What a liquid handler's channels are carrying	the mounted-tips ledger	runtime.devices.get_mounted_tips(...), orca device mounted-tips

Contents are not stored as a number. They are folded on every read from the append-only operation history: every pick-up, aspirate and dispense any action performed. A durable ops-history store is therefore what decides whether volumes and tip state survive at all.

Provenance: three states

Every answer carries a provenance saying how well it is known.

Provenance	Meaning	Does an operator have to look?
UNKNOWN	Nothing has ever stated this. A read means exactly that, and never "empty".	Yes, if you are about to act on it.
KNOWN	There is a value, and nothing has gone unobserved since.	No.
STALE	There is a value, and a stretch went unobserved since anyone last knew. It is still the best value there is.	Yes. This is the only state worth asking about.

Three states, because there are only two questions worth asking: is there a value at all, and does somebody have to go and look. A boolean cannot tell "confirmed empty" from "nobody ever told me", and that is exactly the pair you have to separate before acting on a number.

Alongside provenance, every answer carries a Source saying which layer produced it: OPERATOR, OPERATION, DECLARATION or NONE, highest precedence first. Keeping the two apart stops a new provenance state appearing every time a new kind of writer does.

What makes a fact stale

The runtime writes an observation gap whenever a stretch of time passed with nothing recording. It moves no volume and no tip: the fold is unchanged, but an attestation made before the gap stops counting as current.

Cause	When
RUNTIME_RESTART	The runtime booted. Everything it re-placed is stale.
DEVICE_RECONNECT	An on-prem agent reconnected and rebuilt its drivers. Everything standing on that device is stale, and so is its head.
ERROR_PAUSE	A thread stopped on an error.
OPERATOR_CONTRADICTED	An operator command only made sense if the record was wrong. One moment, not a stretch.
OPERATIONS_DROPPED	An action was abandoned holding operations it had really performed, and they went with it. One moment, not a stretch.

Only an operator's word closes a gap. A confirm or a restatement clears it; the system's own fold cannot, because the fold is what the gap called into question. A later gap does not replace an earlier one either: an abort leaves the count short by the work it threw away, and a restart afterwards says nothing about that shortfall.

OPERATIONS_DROPPED is the one gap a confirm cannot close, because agreeing with a number would freeze a shortfall nothing can compute. A contradicted record is deliberately not in that company: the command that proved the record wrong is itself folded, so an operator who looks can find the rest of the record right and say so.

One more path to STALE, and it is not a gap. An action that has aspirated and not yet finished is still holding those records, so the read is behind. The numbers stay what the record says, because an unfinished action may yet be retried and counting its operations twice is a different wrong answer. The read stops calling itself known.

The unsettled worklist

Scattered across surfaces, unsettled facts read as unrelated warnings. Together they are the worklist an operator walking up to a paused system actually wants.

subjects = await runtime.unsettled_state()

orca state unsettled

Each entry names one thing and the verb that answers it:

Field	What it holds
subject	A labware name, or a device name for a head.
subject_id	The id the verb takes when that is not the subject itself: a labware id for a labware row, null for a head, whose verbs take the device name. Two live instances can share a name, so a row naming only the name could not be acted on.
subject_kind	labware or device_head, so a surface can group without parsing.
provenance	UNKNOWN or STALE. KNOWN is settled and never listed.
detail	What is unsettled, in words.
settle_with	The verb that answers it, or null when no verb does.

Daemon REST: POST /operations/unsettled.

The verb the row names

settle_with is one of six operation names, and the worklist picks which by asking what the record actually holds:

It names	When
confirm-tip-state, confirm-well-volumes, confirm-mounted-tips	The record is still the best value there is. This is the ordinary stale row: a restart, a reconnect, an error pause, a contradicted record.
set-tip-state, set-well-volumes, set-mounted-tips	Nobody has ever said; or an aborted action lost work; or the record has nothing the confirm verb could agree with.
null	An unfinished action is holding operations. No verb settles it.

Naming set-tip-state for a rack that merely went unwatched would send an operator to retype ninety-six positions the record already has right, so a stale row names confirm. The noun follows the subject: a rack gets the tip verbs, a plate the volume verbs, a liquid handler head the mounted-tips verbs. Naming a verb that would refuse the subject is worse than naming none.

What it lists, and what it does not

It covers labware contents and liquid handler heads. It does not list positions: where a labware is has its own operator surface (orca labware edit-location) and its own conflict path, below.

An ordinary destination plate is not listed either. A plate whose template declared nothing to put in it, and which nothing has happened to yet, reads UNKNOWN and is skipped. Listing every one of those would make the worklist unreadable.

The CLI verbs

Operation the row names	CLI
confirm-tip-state	orca labware confirm-tips <id>
set-tip-state	orca labware set-tips <id> ...
confirm-well-volumes	orca labware confirm-volumes <id>
set-well-volumes	orca labware set-volume <id> ...
confirm-mounted-tips	orca device confirm-mounted-tips <name>
set-mounted-tips	orca device set-mounted-tips <name> ...

Confirming appends the current projection as a durable operator baseline, so a later reader sees what was agreed and when. Stating is an absolute overwrite of what you name.

When a confirm is refused

Confirming means agreeing with the record, and there are three states the record is known not to describe. All three raise rather than writing a baseline.

Nothing is tracked. Agreeing with nothing would assert an empty rack nobody stated. Use the set verb to declare the layout or the volumes.

An unfinished action is holding operations. The action has aspirated or picked up tips, and the record only hears about it when the action ends. The read is behind, and a confirm would freeze the number it is behind by. Stating the contents by hand is worse: the action's own operations fold on top of a number that already accounted for them. Settle the action first (retry, continue or abort) and the record catches up on its own. This is the case where settle_with is null. See Recovery.

An aborted action lost work it had really done. The operations happened, nothing wrote them down, and nothing ever will. The labware looks exactly as the record describes it to anyone glancing at the shelf, and the record is short by whatever the abort discarded, so a confirm would mark a wrong number checked. Only counting and stating settles it, which is why the worklist names the set verb here.

That last one is the only observation gap where agreeing is refused. Every other gap is settled by looking: confirm if the record is right, state it if it is not.

What comes back after a restart

At start() the runtime re-places every persisted labware onto the system graph, so it resumes from the physical state it shut down with rather than an empty deck. A plate the arm was carrying is persisted as being in the jaws, not at the slot it was lifted off, because after a restart the arm is still holding it.

Fact	Comes back as
Labware positions	Re-placed and routable.
Well volumes	Re-folded from operation history, marked STALE.
Tip layout on a rack	Re-folded from operation history, marked STALE.
What each liquid handler head carries	Re-folded, marked STALE.
Labware with no opening entry at all	Given one, so it is not permanently unreadable. Already-seeded labware is untouched, so a restart can never refill a consumed rack.

Everything that returns lands on the worklist, because the runtime was off and a hand could have swapped, emptied or refilled any of it. The numbers stand; they stop counting as confirmed. An operator confirms or corrects, and the list empties.

Whether any of this survives depends on which stores you made durable. The shipped defaults are in-process. See Stores.

An agent reconnecting does the same thing at a smaller scale: everything standing on that device, plus its head, gets an observation gap, and the runtime re-pushes the ledger's world onto the rebuilt driver.

Deck reconciliation

A liquid handler's driver keeps its own picture of what is on its deck. That picture is a projection: the runtime put it there. So when the two disagree, the record is normally the one to believe. The disagreement is worth a person's attention either way.

Two verbs, and the difference matters:

Verb	What it does
runtime.devices.compare_deck(name) / orca device compare-deck	Read-only. Reports the disagreements and files nothing.
runtime.devices.reconcile_deck(name) / orca device reconcile-deck	Compares first, files an incident per disagreement, then pushes the ledger's world onto the driver.

compare_deck files nothing on purpose: it is the read an assistant or a UI runs routinely, and filing on every call would bury the incident queue. reconcile_deck is the operator verb that also settles the driver.

Reconciliation also runs on its own when a device initializes, connects, or its agent reconnects.

A driver reporting an entirely empty deck is not treated as a pile of conflicts. That is a rebuilt session, not a disagreement, and re-seeding is the correct answer.

Conflicts become incidents

Each disagreement is recorded as a DECK_RECONCILE_CONFLICT incident naming both sides. Nothing resolves itself and neither side is changed: the operator settles it.

Reason	What the two sides say
DRIVER_SITE_DIFFERS	The driver has the labware at one site, the ledger at another. Something moved it outside orca.
MISSING_FROM_DRIVER	The ledger places it on this deck; the driver has it nowhere. Either it left the deck, or the driver's session was rebuilt and lost it.
UNKNOWN_TO_LEDGER	The driver has labware the ledger does not place on this deck.
SITE_NOT_IN_LAYOUT	The labware sits at a site the active deck layout no longer provides, so it was never projected.
HELD_BY_GRIPPER	The ledger has it in the jaws. A deck addresses labware by site and the jaws are not one, so both models cannot be right.
LEDGER_TARGET_OCCUPIED	An operator command put labware where the ledger already had something. The command ran; only the record of it did not.
CONTENTS_DIFFER	The two disagree about what the labware holds, not where it is.
INTERRUPTED_MOVE	A move stopped part-way. Look before commanding motion.

For most of these the answer is the same: put the labware where it really is with orca labware edit-location, or discharge it after physically removing it. UNKNOWN_TO_LEDGER instead wants register-labware or removal at the instrument. CONTENTS_DIFFER wants set-tips or set-volume.

A contradicted ledger

An ad-hoc operator device command can only make sense if the record was wrong: picking up tips from spots the record says are empty, for instance. The command is believed and folded, because the operator was at the bench and the record was not.

What the command cannot supply is the rest of the labware. The command proved one position wrong at one moment and says nothing about the others, so the runtime records a LEDGER_CONTRADICTED incident and the labware goes on the worklist.

Go and look at it. The pick that proved the record wrong is itself folded, so the record may well be right about everything else: if it is, confirm settles it, and if it is not, state the contents. This is the same contract every other stale row has, and it is why the worklist names the confirm verb on a contradicted row.

One list of what is stopping the run

The worklist says what the record cannot answer. The blocker list says what is stopping work, which is a different question with a different answer.

blockers = await runtime.blockers()

Recovery state lives in several places: a thread's pause fields, a device fault, an execution's pause latch, a reservation nobody will release, a park waiting on a person. Clearing one says nothing about the rest. That is how an operator clears a device fault, expects the arm to move, and finds the thread it faulted still paused with nothing saying so.

blockers() derives one list from all of that on every read, worst first, so it cannot drift. Each entry carries the remedies that clear it, with the recovery verbs read off the same table that refuses them, so a verb offered here can never be refused on the call. A blocker on a device whose command never answered is flagged may_still_be_moving, because nothing told the instrument to stop.

An unacknowledged incident is not a blocker. It is the record of one, and counting both would report every error pause twice; a blocker carries its incident_id instead.

The list is derived on every read, so re-reading it after clearing one blocker is what tells you the run is actually free. In the framework this is the Python call above; the local daemon has no route for it. A hosted deployment serves the same list over REST, and orca status --backend cloud renders it worst first with the recommended remedy per row.

Pitfalls

• UNKNOWN is not empty. A read that finds nothing says nobody has ever stated this. Acting on it as though the plate were empty is the failure this model exists to prevent.
• STALE is still the best value there is. Nothing blocks on it. A prompt on every move is a prompt nobody reads.
• Read settle_with rather than assuming. A stale row usually names confirm, not set, and it is null when an unfinished action holds the subject.
• A contradicted record can still be confirmed. Look at the labware; if the record is right, agree with it. Only an abort that lost work refuses the confirm.
• Confirm is refused when nothing is tracked. Use the set verb to state the layout or the volumes instead.
• Do not confirm a subject whose action is still unfinished. Settle the action first; the numbers catch up.
• compare_deck files nothing. If you want the conflicts recorded, use reconcile_deck.
• A deck conflict changes neither side. It is a record for a person, not a repair.
• The worklist does not cover positions. A wrong position surfaces as a deck conflict or is corrected with edit-location.

See also

• Labware: templates, instances, and how labware is tracked through a run.
• Stores: which stores make this state durable, and what the defaults actually persist.
• Recovery: incidents, pause sites, and settling an unfinished action.
• Submissions: the unsettled list reported at submit time.
• CLI: monitor: reading positions, contents and history from the terminal.