Runtime

The SystemRuntime is the long-lived service that operates your lab. You stand it up once at process start; it accepts workflow submissions, runs them concurrently, and stays alive until the process exits. Think of it the way you'd think of a FastAPI app or an Airflow scheduler — a persistent process that handles incoming work.

Why a persistent runtime

Three reasons:

Many submissions over time. Real labs run multiple workflows in a day, sometimes overlapping. A persistent runtime accepts submissions as they arrive, schedules them, and runs them concurrently against the same physical lab.
State survives the work. Labware tracking, incidents, run history — all of this is meaningless if it goes away when a script exits. The runtime persists state through pluggable stores.
External integration. A long-lived runtime is something you can talk to. Dashboards, monitoring systems, and external orchestrators subscribe to the same event bus and reach the same operations through either the in-process Python API or the local daemon's REST + SSE.

Three control surfaces

The runtime exposes the same operations across three surfaces; pick whichever fits your caller.

Surface	How to reach it	Best for
Python `SystemRuntime` API	Construct `SystemRuntime(...)` directly in your process.	In-process services, embedded testing, in-process orchestration loops.
`orca.daemon` REST + SSE	`orca start` spawns a uvicorn-hosted FastAPI app on `127.0.0.1:<port>`. ~70 endpoints covering executions, submissions, devices, incidents, audit, variables, teachpoints, deck layouts, labware, plugins. SSE event stream included.	Out-of-process control on the same host. Local UIs, side-car automation.
`orca` CLI	Ships with the framework package. Talks to the local daemon over its REST API. See CLI.	Human operators, scripts, ad-hoc inspection.

For remote/network access — including authenticated REST + MCP — wrap the daemon in your own service layer or use Swarm, which provides exactly this on top of Orca Framework.

Building a runtime

import asyncio
import orca.orca as orca
from orca.runtime.system_runtime import SystemRuntime
from orca.runtime.run_modes import WorkflowRunMode
from orca.runtime.store_factory import InMemoryRuntimeStoreFactory

async def main():
    stores = InMemoryRuntimeStoreFactory()
    topology = build_topology(stores)
    workflow = build_workflow(topology)

    # build_system is async, so await it.
    build = await orca.build_system(
        name="my_lab",
        topology=topology,
        stores=stores,
        workflow=workflow,        # register the initial template
    )

    runtime = SystemRuntime(
        system=build.system,
        event_bus=build.event_bus,
        # Optional services; each defaults to an in-memory SQLite-backed
        # implementation when omitted (labware_store, incident_service,
        # execution_record_service, access_config_service, profile_store,
        # ops_history_store, labware_catalog_store, ...).
    )

    # To register more templates after construction, add them on the System:
    #   build.system.add_workflow_template(another_workflow)

    # Start the runtime. Devices initialize lazily on first dispatch.
    await runtime.start()

    # Two ways to submit: submit_workflow(name) uses a registered template's name
    # (here); submit(template) takes a built workflow object (see Quick Start).
    # Both require `mode`; there is no deployment fallback.
    submission = await runtime.submit_workflow(
        "my_assay",
        variables={"plate_count": 1},
        mode=WorkflowRunMode.PURE_SIM,
    )

    # Keep the process alive while the runtime serves submissions.
    await asyncio.Event().wait()

asyncio.run(main())

In production you'd usually wrap this in a service framework (FastAPI, gRPC, etc.) so external systems can submit through a network protocol.

Lifecycle

A runtime moves through a small number of states:

State	What it means
Constructed	`SystemRuntime(...)` returned. Templates can be registered. No work runs.
Started	`await runtime.start()` returned. Submissions are accepted, executions can run. Device init is deferred to first dispatch.
Paused (global)	All threads paused at the next safe point. Useful for live recovery or maintenance.
Stopped	Runtime tearing down. New submissions rejected; in-flight work is given a chance to drain.

You usually only worry about start and stop. Pausing/resuming individual threads is more common than pausing the whole runtime.

Submitting work

submission = await runtime.submit_workflow(
    "my_assay",                            # workflow template name
    variables={"plate_count": 4},          # variable overrides
    mode=WorkflowRunMode.PURE_SIM,         # REQUIRED: per-submission run mode
)

Run mode lives on the submission, not on the runtime. Every submit declares its own:

Mode	Devices	Use it for
`WorkflowRunMode.PURE_SIM`	Pure software simulators. No hardware contact.	Developing and testing assays on a laptop.
`WorkflowRunMode.DEVICE_SIM`	Real driver code paths against PyLabRobot Chatterbox simulators.	Validating driver behavior without instruments.
`WorkflowRunMode.LIVE`	Real hardware drivers.	Production runs.

mode is required — there is no deployment-level fallback. Omitting it raises RunModeRequiredError. The runtime validates every device in the topology against the requested mode before booting the execution, so a LIVE submission against a topology with disconnected hardware fails fast.

submit_workflow returns an ExecutionRecord you can use to track progress, cancel, or inspect later. For more advanced submission shapes — labware groups, batching with in-flight executions, joining receivers — see Submissions.

Pausing, resuming, recovering

When something goes wrong, the runtime pauses the affected thread and creates an incident. An operator can inspect the incident and choose a recovery action:

from orca.workflow_models.status_enums import RecoveryDecision

# Pause a specific thread (execution_id first, then thread_id)
runtime.pause_thread(execution_id, thread_id)

# Resume a manually paused thread
runtime.resume_thread(execution_id, thread_id)

# Recover an error-paused thread with an operator decision
runtime.recover_thread(execution_id, thread_id, RecoveryDecision.RETRY)

These calls are synchronous. recover_thread takes a RecoveryDecision (RETRY, RETRY_OP, ABORT_ACTION, ABORT_METHOD, ABORT_THREAD); there is no recover_incident / RetryDecision.

See Recovery for the full incident model and the decisions available at each pause site.

Stores

The runtime persists state through pluggable interfaces. The framework defaults each to an embedded SQLite-backed implementation (with in-memory and file-backed variants); a hosted deployment plugs in database-backed implementations as a separate product layer.

Interface	Holds
`ILabwareStore`	Labware tracking: which plates are in the system, where they are, what's happened to them.
`IIncidentStore`	Incidents and their resolutions (a Protocol, injected via `IncidentService`).

There is no IStateStore. Execution state lives on the runtime itself; persisting it across restarts is a hosted concern, not a framework interface. See Stores for the full interface contracts, the ships-with defaults, and how to plug in custom implementations.

Events

Every status change inside the runtime fires an event on the EventBus. Sinks subscribe to events and can persist them, forward them to dashboards, or feed them to external systems. See Events for the event taxonomy and how to register a sink.

Where to go next

Submissions — submission shapes (groups, batching, JOIN_EXISTING).
Recovery — incidents, pause sites, recovery decisions.
Events — event taxonomy and integration sinks.

Why a persistent runtime​

Three control surfaces​

Building a runtime​

Lifecycle​

Submitting work​

Pausing, resuming, recovering​

Stores​

Events​

Where to go next​