Runtime

The runtime is the long-lived service that runs your lab. You start it once, submit workflows to it as you need them, and it runs them side by side against the same instruments.

Start it and run a workflow

Five lines, the same five the Quick Start ends with:

build = await orca.build_system(
    name="my_lab", topology=topology, workflow=workflow, stores=stores,
)
runtime = SystemRuntime(build.system, event_bus=build.event_bus)
await runtime.start()
submission = await runtime.submit(workflow, mode=WorkflowRunMode.PURE_SIM)
status = await runtime.wait_for_execution(submission)

1. build_system wires your topology, your stores and the event bus together and resolves the routing graph. It is async, so await it.
2. SystemRuntime(...) wraps the built system in the service that schedules and runs work. Only build.system is required. Pass event_bus as well, or per-workflow events never reach system sinks.
3. start() brings the runtime up so it can accept submissions.
4. submit(...) hands the workflow over and returns a handle immediately, while the run proceeds in the background. mode says which world this run drives: simulators on your laptop, or real instruments. See Sim hierarchy.
5. wait_for_execution(...) blocks until that run finishes and hands back the final status. Check it. status.status is an ExecutionPhase, and a run that paused for an operator and was never answered does not come back COMPLETED.

The imports and a full runnable file are in the Quick Start. From a terminal, the same job is orca run.

A script waits for one run and exits. A service does not: it starts the runtime once, keeps the process alive, and submits work as it arrives. Three things you get by keeping it up:

1. Many submissions over time. Real labs run several workflows in a day, sometimes overlapping. A persistent runtime accepts them as they arrive and runs them concurrently against the same lab.
2. State survives the work. Labware positions, well volumes, tip state, incidents and run history are meaningless if they vanish when a script exits. The runtime writes each of these through a store and re-seeds from it at the next boot. See Runtime state.
3. You can talk to it. Dashboards, monitoring and external orchestrators subscribe to the same event bus and reach the same operations, in process or over the local daemon.

Submitting more work

Two ways to submit, differing only in how you name the workflow:

# By object: the built workflow you just defined.
submission = await runtime.submit(workflow, mode=WorkflowRunMode.PURE_SIM)

# By name: a template already registered on the system.
record = await runtime.submit_workflow(
    "my_assay",
    variables={"plate_count": 4},
    mode=WorkflowRunMode.PURE_SIM,
)

variables overrides variable values for this run only. See Variables. What you are handing over is a workflow, which runs one or more labware threads. See Labware threads.

To register another template after construction, add it on the system: build.system.add_workflow_template(another_workflow).

mode is required on every submission, and it lives on the submission, not on the runtime. There is no deployment-level fallback, and omitting it raises RunModeRequiredError. Before booting the execution the runtime checks every device declared in the topology against the requested mode, so a live submission against a topology with disconnected hardware fails fast rather than part way through the run. The three modes are covered in Sim hierarchy.

submit_workflow returns an ExecutionRecord. For submissions that carry several labware groups, or that join a run already in flight, see Submissions.

The three states

runtime.state is one of three values:

RuntimeState	What it means
CREATED	SystemRuntime(...) returned. Templates can be registered. No work runs.
RUNNING	await runtime.start() returned. Submissions are accepted, executions can run.
STOPPED	await runtime.shutdown() returned.

start() does three things. It re-places every persisted labware back onto the system graph, so the runtime resumes from the physical state it shut down with. It registers the sink that keeps execution records current. It starts the stall detector. Devices are configured and initialized by the first thread that touches them, so boot does not walk every deck in the topology.

shutdown() is an abort, not a graceful drain of the work itself. It ends every live execution. The ordering it uses matters if you are embedding the runtime in a service.

Pausing, resuming, recovering

When something goes wrong, the runtime pauses the affected labware thread and records an incident. An operator reads the incident and answers with a recovery decision:

from orca.workflow_models.status_enums import RecoveryDecision

# Pause one thread (execution_id first, then thread_id)
runtime.pause_thread(execution_id, thread_id)

# Resume a thread a person paused
runtime.resume_thread(execution_id, thread_id)

# Answer an error-paused thread with an operator decision
runtime.recover_thread(execution_id, thread_id, RecoveryDecision.RETRY)

These three calls are synchronous. A thread refuses any decision its pause site does not honour, and says which ones it will take. See Recovery for the decisions and what each one means, and orca execution for the terminal equivalents.

Pausing and resuming one thread is far more common than stopping the whole runtime.

Watching a run

Every status change inside the runtime fires an event on the event bus. Register a sink with runtime.register_sink(sink) to persist events, forward them to a dashboard, or feed them to another system. See Events. To watch from a terminal instead, use orca monitor.

Advanced

Everything below is for embedding the runtime in a service, plugging in your own persistence, or building an operator surface on top of it.

Constructor arguments

SystemRuntime.__init__ takes one required argument and fourteen optional ones. Most of them exist so a hosted deployment can inject its own persistence without the framework knowing anything about it. Standing the runtime up yourself, you pass very few.

Passed by nearly everyone:

Argument	Default when omitted
system	required, from build.system
event_bus	None, so per-workflow events never reach system sinks. Pass build.event_bus.

Passed when you want a specific behavior:

Argument	Default when omitted	Who normally passes it
labware_store	InMemoryLabwareStore()	anyone who wants labware positions to survive a process restart
profile_store	NullDeploymentProfileStore()	anyone submitting with a named deployment profile
gateway_registry	NullGatewayRegistry()	deployments with remote devices
connection_source	NullDeviceConnectionSource()	deployments with remote devices
stall_check_interval	10.0 seconds; None disables the detector	anyone tuning stall detection
stall_required_stable_ticks	2	anyone tuning stall detection

Injected by a hosted deployment that brings its own database:

Argument	Default when omitted
incident_service	IncidentService over in-memory SQLite
execution_record_service	ExecutionRecordService over in-memory SQLite
access_config_service	AccessConfigService over in-memory SQLite
move_defaults_service	MoveDefaultsService over in-memory SQLite
grip_profile_service	GripProfileService over in-memory SQLite
ops_history_store	the store already on system.ops_history
labware_catalog_store	a seeded SQLite-backed LabwareCatalogService, built lazily

The runtime disposes only the services it built itself. An injected service belongs to whoever injected it, and shutdown() leaves it alone. See Stores for the interfaces behind each of these.

The facade surface

The runtime's real API is a set of facades reached as properties. Each returns the same instance every time, so a hosted deployment can layer its own scoping over one of them without touching the rest.

Property	What you do through it
runtime.variables	Read and set variable values for a running execution.
runtime.labware	Where a labware is, what it holds, edit or reset a position, set and confirm well volumes and tip state, discharge or clear labware.
runtime.devices	Device status, connect, initialize, ad-hoc commands, deck reconciliation, mounted tips, take and release control, clear a fault.
runtime.threads	Pause, resume, recover, skip, abort, insert and replace steps on a live labware thread.
runtime.registry	System info; list devices, transporters, movers, pools, locations, labware templates, methods and thread templates; add and remove workflow templates; list and cancel reservations.
runtime.incidents	List, get and acknowledge incidents.
runtime.submissions	submit_group(...), close_execution(...), list and get submissions.
runtime.teachpoints	Named transporter positions.
runtime.deck_layouts	Named PLR deck configurations per liquid handler.
runtime.topology	The declared device set.
runtime.gateway	Devices reachable over the gateway right now.
runtime.device_registry	The two together, keyed by device name, with live connect and init state.
runtime.ops_history	Append-only operation history: list and search.

Alongside those, runtime.execution_records, runtime.labware_catalog_store, runtime.access_config_store, runtime.move_defaults_service, runtime.grip_profile_service and runtime.profile_store hand back the service instances themselves. They are exposed so an operator surface built on top shares the same instance the runtime resolves against, rather than a second copy that drifts.

What is stopping the run

Recovery state lives in several places: a thread's pause fields, a device fault, an execution's pause latch, a reservation nobody will release, a park waiting on a person. Clearing one says nothing about the rest.

blockers = await runtime.blockers()

blockers() returns one list of everything stopping the run, worst first, derived on every read so it cannot drift. Each entry names its kind (DEVICE_FAULT, THREAD_ERROR_PAUSE, EXECUTION_PAUSED, WAITING_ON_A_PARKED_HOLDER, MANUAL_PLACE, MANUAL_REMOVE, MANUAL_STEP, DEVICE_EXTERNAL_CONTROL, RUNTIME_NOT_BUILT), a severity, a headline, and the remedies that clear it. A blocker on a device whose command never answered carries may_still_be_moving, because nothing told the instrument to stop.

An unacknowledged incident is not a blocker. It is the record of one, and a blocker carries its incident_id instead.

The companion read is await runtime.unsettled_state(), which lists what the record cannot answer rather than what is stopping work. See Runtime state for the unsettled worklist.

Pausing a whole execution

runtime.pause_execution(execution_id)                # sets the latch, fans out
runtime.pause_all_threads(execution_id)              # fans out only
runtime.resume_execution(execution_id)
runtime.resume_all_threads(execution_id)

pause_execution sets an execution-level latch on top of the fan-out. The latch is what refuses a JOIN_EXISTING submission into a paused execution, and it is orthogonal to the execution's phase: a paused execution is still ACCEPTING or DRAINING. pause_all_threads fans the pause out without the latch.

Both return three counters, so a caller can tell an operator whether anything pauseable was found: pausing, already_paused, terminal_skipped. Threads spawned after the call are born paused, because a parent already blocked on a reservation can still spawn contributors on its way to its own safe point.

Both calls take reason, which is "manual" by default and "system" when the runtime pauses on its own behalf: a detected stall, an unresolvable deadlock, a quarantined slot. The reason travels to each thread and out onto the thread's status, so a surface can say whether a person stopped the run or the engine did. A recoverable device timeout is a third case: it fans the pause out to the threads without setting the execution latch, so an operator's own stop still outranks it.

Stall detection

A co-labware wait is unbounded by design, because a partner thread may legitimately run a multi-hour action before it can move its labware. A wall-clock cap cannot tell that apart from a real wedge, so the runtime asks a structural question instead: is every live thread internally blocked, with none in flight?

The detector runs every stall_check_interval seconds (10 by default; pass None to disable it) and declares a stall only when the same blocked shape holds for stall_required_stable_ticks consecutive checks (2 by default). Any thread genuinely acting resets the count, so a slow but healthy run never trips it, whatever the interval.

On a stall the runtime records a SYSTEM_STALL incident and pauses the execution with reason="system". Anything awaiting that execution through runtime.wait(...) or runtime.wait_for_execution(...) gets a SystemStallError immediately, rather than blocking blind until some outer timeout. The execution stays paused and recoverable, and resuming clears the episode.

Shutdown ordering

shutdown() is abortive, and it drains before it closes anything. It aborts every live execution and waits for the thread tasks to unwind, then stops the tick loops, then waits for any cancels still travelling to the device bridges, then flushes the pending labware-position writes, and only then closes the services it owns. Doing it the other way round left threads running against stores that had already been disposed.

Every one of those waits is bounded, because shutdown has to end even when something will not come back: 30 seconds for the executions, the background teardown and the position queue, 5 seconds for the cancels. A wait that gives up says so in the log, and the position-queue warning names how many positions the next boot will not know about. Cooperative parking is not implemented: this is an abort, not a graceful drain of the work itself.

What the runtime persists

The runtime persists state through pluggable interfaces. Every one of them has a working default that ships with the framework: in-memory for labware, embedded SQLite for incidents, execution records, access configs, move defaults, grip profiles and the labware catalog, and JSONL for operation history. A hosted deployment injects database-backed implementations behind the same interfaces as a separate product layer.

There is no IStateStore. Live execution state lives on the runtime itself, and carrying an in-flight execution across a restart is not something the framework does. See Stores for the full interface contracts and how to plug in your own.

Three ways to reach the runtime

The same operations are available on three surfaces. Pick whichever fits your caller.

Surface	How to reach it	Best for
Python SystemRuntime API	Construct SystemRuntime(...) directly in your process.	In-process services, embedded testing, in-process orchestration loops.
orca.daemon REST + SSE	orca start spawns a uvicorn-hosted FastAPI app on 127.0.0.1:<port>. Roughly 150 endpoints covering executions, submissions, devices, incidents, labware state, variables, teachpoints, deck layouts, ops history and plugins, plus an SSE event stream.	Out-of-process control on the same host. Local UIs, side-car automation.
orca CLI	Ships with the framework package. Talks to the local daemon over its REST API. See CLI.	Human operators, scripts, ad-hoc inspection.

For remote access over a network, including authenticated REST and MCP, wrap the daemon in your own service layer or use Orca AI, which provides exactly that on top of Orca Framework.

See also

• Quick Start: the full runnable file these five lines come from.
• Submissions: variables, labware groups, batching, and stopping an execution.
• Sim hierarchy: the three run modes and how to develop without hardware.
• Recovery: incidents, pause sites and the operator decisions that clear them.
• Runtime state: what the record knows, what it does not, and what an operator has to settle.