A production runtime for long-horizon agents

Abstract

Most agent frameworks are notebooks pretending to be services. They run in a loop until they crash, they share memory across requests by accident, and the only test that ever ran is the demo. That works for prototypes. It does not work for an autonomous-lab agent that holds keys to a liquid handler, or a chip-design agent that submits jobs to a multi-million-dollar simulation farm. This paper describes the production runtime that every Aether ^[1] discipline runs on — the substrate behind the wet-lab, the chip-design, the CAE and the coding agent — and the design choices that distinguish a deployable runtime from a notebook-shaped framework. We document five design properties (long-running processes, deterministic memory, capability-gated tool use, evaluation as a first-class object, blue/green deploys), discuss what they buy in practice, and describe the failure modes the runtime is designed to remove.

1. Why notebooks fail in production

The dominant pattern in the open agent-framework ecosystem is a Python script that calls a foundation model in a loop, manages its own state in a dictionary, and exposes a simple HTTP wrapper for invocation. This pattern is fast to prototype and useful for demos. It also has six failure modes that compound in production.

• Crashes are terminal. The agent's state is in process memory; a crash loses it. A six-hour molecular-dynamics campaign that crashes at hour five starts over.
• Sessions bleed. Without an enforced session boundary, memory from one user's session can leak into another's. The fix is at the framework layer, not at the application layer.
• Memory is opaque. "What did the agent know when it made this decision?" is unanswerable without a memory store with provenance ^[2].
• Tool use is unconstrained. An agent that can call any tool can compose actions in unsafe combinations ^[3]. The fix is capability gating at the runtime, not at the call site.
• Evaluation is afterthought. Agents go to production without regression tests, then the operations team writes ad-hoc retrospective evals after the first incident.
• Deploys are dangerous. Pushing a small change without versioned eval can break invariants the customer was depending on; rollback is manual and slow.

Every one of these is a runtime-level concern. The application-layer code cannot fix them; the foundation-layer choices have to be different. The next five sections describe what we did differently.

2. First-class long-running processes

An agent in the Aether runtime is a first-class process. It has an identity, an audit trail, a policy attached, a memory binding and a versioned configuration. It can run for seconds or for months; it can checkpoint, pause, and resume across hosts. Resuming a six-hour molecular-dynamics campaign after a machine failure is a single API call.

The mechanism is straightforward. The agent's state — context, memory references, tool-call cursor, in-progress reasoning chain — is serialised to durable storage on a configurable cadence (default: every tool-call boundary, plus every 30 seconds during long pure-reasoning stretches). On host failure, the runtime restarts the agent on a healthy host from the most recent serialised state. The application-layer agent code does not need to know about the failure; the resumption appears as a transparent continuation.

Checkpoint and resume are not just for failure recovery. They are also the mechanism for human-in-the-loop pause/approve patterns, for cross-zone migration during maintenance windows, and for forking exploratory branches of a long-horizon investigation. The same primitive serves all four use cases.

3. Deterministic memory

Memory in the runtime is episodic and semantic, versioned, queryable, and exportable. Every memory write has provenance — which decision wrote it, which tool call produced the value, which policy was active ^[2]. "The model just forgot" is not a thing that can happen, because forgetting requires explicit revocation, and revocation is itself a versioned event.

Memory is queried through a hybrid policy described in the memory paper ^[2]. The agent does not specify which index to hit; it asks for memory satisfying some criterion, and the runtime decides whether vector similarity, symbolic traversal, or temporal scan answers it best. This indirection is deliberate — it lets us evolve the underlying retrieval strategy without changing every agent that uses it.

4. Capability-gated tool use

Every tool the agent wants to call declares the capability it needs. Capabilities are granted by customer policy, and grants are audited. The agent can read production logs without ever being able to delete a row; it can submit a tape-out job without being able to cancel an unrelated one.

The capability model is hierarchical and least-privilege by default. Tool authors declare the narrowest capability that supports the tool's intended use; customer policy maps capabilities to authorised principals. New capability grants require an explicit policy update, which is itself audited and (in regulated environments) requires a documented review.

This is the mechanism that makes the runtime's safety architecture ^[3] practically enforceable. Refusal corpora handle the request-shape level; capability gating handles the tool-call level; memory provenance handles the after-the-fact audit. Together they raise the cost of unsafe composition meaningfully above what any single layer can support.

5. Evaluation as a first-class object

Evaluations live in the runtime, not on the side. Each agent ships with a regression suite that runs on every commit, plus replay of historical traces to verify the agent still handles them the same way. Synthetic adversarial cases run against every release; corpus-versioned refusal scenarios ^[4] are part of the standard eval suite.

Evals are typed and versioned alongside the agent code. A test case captures the agent's input, the policy that was active, the expected output, and the rubric for grading partial matches. New cases are added by the engineer who fixed the bug that exposed the case; the corpus grows monotonically and every release runs against the union.

Critically, eval results are part of the deploy-time contract. A pull request that breaks the eval cannot merge; a release that regresses the eval cannot promote (Section 6). The eval suite is not advisory.

6. Blue/green for agents

Agent versions promote behind a percentage rollout, gated by the eval suite. Rollback is one click. Promotion is automatic when the eval threshold holds for the configured window. Most agent failures we have seen in production were "somebody pushed a small change and the eval suite did not exist" — we removed that failure mode at the runtime layer.

Concretely: the runtime maintains two simultaneous versions per agent — a stable version handling the current traffic and a candidate version receiving a configurable fraction (default: 5%). The candidate's eval suite runs continuously against live traffic; if the threshold holds for the configured window (default: 24 hours), the runtime promotes the candidate to stable and the old stable version is retired. If the threshold breaks, the candidate is automatically rolled back and the operator is paged.

Customers in regulated environments can override the auto-promotion with a manual approval gate. The mechanism is the same; only the promotion trigger changes.

7. Observability

Every decision, every tool call, every memory write is captured in an immutable trace, exportable as OpenTelemetry spans, SIEM events or JSONL. Traces are searchable and filterable by every dimension that matters in production — agent version, policy version, customer, principal, capability used, eval-suite version at the time of the call.

We make a deliberate choice not to mine traces for product analytics. The traces are operational and audit artefacts; customer privacy and the contractual data-handling commitment dominate any incremental value from aggregating customer traces. Operators see their own data; we see only what we need to support the deployment.

8. What this is not

This is not a claim that the runtime is novel. The design choices above are well-understood individually in the broader infrastructure community. The contribution is the integration — applying the standards of production service infrastructure to long-horizon agents specifically, where the application-layer patterns have lagged the infrastructure-layer patterns by several years.

It is also not a claim that the runtime is finished. Several capabilities are still being built (planner-layer auditing for composition attacks ^[3], richer eval-suite tooling, cross-tenant federation primitives for multi-site deployments). We publish the current design honestly rather than presenting it as a finished system.

9. Conclusion

Long-horizon agents need long-horizon infrastructure. The Apex agent runtime applies the standards of production service infrastructure — long-running processes, durable state, observable behaviour, eval-gated deploys, capability-gated tool use — to the agent class. The runtime is the substrate that the safety story ^[3], the memory story ^[2] and the customer-grade audit story sit on. Without it, none of those higher-level guarantees are practically enforceable; with it, they are.