research/durable-agent-runtime/pi-durable.md

# pi-durable — Durable Agent Harness

> Part of [[durable-agent-runtime/index|Durable Agent Runtime]]
> Researched 2026-10-20 against `earendil-works/pi` main (`48c8f32`, 2026-10-20) and the published npm package `@earendil-works/pi-durable@1.3.0` (2026-10-15). Status: **experimental**, API changes between releases.

## TL;DR

`@earendil-works/pi-durable` is an MIT-licensed TypeScript **library that turns an agent harness into a durable state machine**. Conversations, transcript entries, tool calls, model turns, and application state are committed to storage before they are published; a new process opening the same storage resumes unfinished work from per-task checkpoints.

It is not a sandbox, VM, or durable-object platform. It is the layer *above* the substrate: storage-agnostic (memory / SQLite / JSONL / Cloudflare Durable Object SQLite) and execution-environment-agnostic (local Node, remote VM, container). That makes it the clearest example so far of a distinct layer in this topic's taxonomy: the durable harness library, orthogonal to where compute runs.

The important caveat: durability here is **at-least-once execution of phases, not exactly-once external effects**. Effects must be memoized or made idempotent by the application. Tool calls get an explicit replay policy (`safe` / `unsafe`); an interrupted unsafe tool is reported to the model as `interrupted`, never silently repeated.

## Where it sits

| Layer | Examples | pi-durable's relationship |
| --- | --- | --- |
| Durable object / actor platform | celld, rivet, Cloudflare Durable Objects | pi-durable can *run on* a DO (SQLite-backed) or Node; it does not provide the platform |
| Sandbox / VM runtime | hypeman, E2B, Amp orbs | pi-durable does not isolate tools; an `ExecutionEnv` can point at one |
| Durable harness library | **pi-durable**, Temporal-style durable execution applied to agents | This is the new layer: commits the loop's state, not the compute |
| Harness / loop library | pi, nanocodex, Codex, Code Mode | pi-durable is the durable variant of the harness itself |

The convergence thesis in [[durable-agent-runtime/analysis|Analysis]] — compute is disposable, state is durable elsewhere — still holds. pi-durable sharpens it: *the harness process is also disposable; the conversation + task graph is the durable object.*

## Architecture

### Record model

The normative spec (`packages/durable/docs/spec.md`, ~5,600 lines) defines six primitives:

- **Session** — owns one mutation line: conversations, entries, tasks, submissions, documents. Every commit is atomic across all of them.
- **Conversation** — a transcript scope; can fork another at an entry. Root conversation is created on first use and found again after restart.
- **Entry** — one immutable transcript record: `pi.user`, `pi.assistant`, `pi.tool-result`, `pi.system`, `pi.reset`, `pi.compaction`, plus application kinds. Entries are never mutated or reused.
- **Document** — mutable typed JSON state stored next to the transcript, implemented on **Chord** (`@earendil-works/chord`) as an operation log with periodic complete bases. Built-ins: `pi.agent`, `pi.provider`, `pi.live`, `pi.inbox`, `pi.usage`.
- **Task** — a durable state machine with an explicit phase checkpoint. Every step commits before moving on.
- **Submission** — an awaitable input or write; `requestId` makes retries idempotent.

Required invariants (spec §1) worth quoting because they are the design:

1. One Session commit is atomic across all record and document writes.
2. A document update is published only after its storage commit succeeds.
3. All visible progress is durable; there is no volatile publication path.
4. External effects do not run inside the Session mutation transaction.
5. Entries and IDs are immutable.
6. Commit/close observers capture immutable state synchronously; user callbacks run later, off the mutation line.
7. A storage error is fatal to the open Session — it publishes nothing and must be reopened.

### Task model

A task is `defineTask({ name, version, initial, phases, abort, migrate? })`. Each phase receives `(task, runtime, context)` and ends by committing a replacement checkpoint, a `waiting` state, a sleep, or a terminal outcome. Recovery never resumes a phase mid-way: it reruns the phase from its top, so post-wait work belongs in the next phase and its inputs in the checkpoint. On `main`, a task sleeps by ending its phase (`runtime.sleepUntil(...)` or returning `{ status: "sleeping", until, checkpoint }`); the published 1.3.0 still has the older `runtime.sleep()`, which waits *inside* a phase — removed in `[Unreleased]` precisely because a restart reran the phase and restarted the wait (see version gap below).

Task statuses in the spec: `pending`, `running`, `waiting` (parked on other tasks, `allSettled` or `failFast`), `completing` (outcome decided, owned work still draining), `terminal`. Main adds `sleeping`; published 1.3.0 does not.

Built-in tasks:

- `pi.generation` — prepares the system prompt, calls the model, owns the round's `pi.tool` tasks, hands the run to the next generation.
- `pi.tool` — one call per task; intent is committed before `execute()`.
- `pi.compaction` — summarizes older entries; runs as a task, can run in background while the conversation continues.

Usage/cost is not a task: it is the built-in `pi.usage` document, incremented by generation, tools, and compaction, with subtree accounting for owned conversations.

### Structured concurrency

Tasks and conversations form one ownership tree (spec §5.5):

- Every task names its owner: its conversation or another task. Owner edges are immutable.
- **Abort flows down.** A live owner's cancellation intent (abort mark or held non-`completed` outcome) marks all ordinary owned work; abort handlers run bottom-up so each task undoes its own effects.
- **Owned work does not outlive its owner's finish.** A task that finishes while owned work is live becomes `completing`; terminal only after the subtree drains.
- **Background work is a boundary**: it survives the parent's abort and does not keep the parent busy (used for persistent subagents).
- This is the same shape as structured concurrency in languages (Kotlin/Nursery), applied to agent tasks and subagent conversations.

### Durability semantics — the part that matters

The core pattern is the **effect sandwich** (spec §5.2):

```text
commit intent phase
perform external effect
commit outcome or next phase
```

Reopening in an intent phase means the effect *may* have happened. The handler must retry safely, poll an external handle, or record interruption. There is no magic.

Concrete guarantees, per mechanism:

- **Submissions**: a `requestId` deduplicates admission within a conversation. A client can retry after a crash and get the original submission back. This is the cleanest exactly-once primitive in the package.
- **Model turns**: streaming partials are committed at most every `settings.progress.partialIntervalMs` (default 100 ms). On recovery, a leftover partial is converted to an **aborted** `pi.assistant` entry and the request is resent from the same committed context, with the same model, thinking level, stream options, and provider `sessionId` (prompt cache/session affinity). So: no token-level resume; duplicate model spend is possible; the partial answer survives in the transcript as aborted.
- **Tool calls**: intent (final arguments + replay policy) is committed before `execute()`.
  - `replay: "safe"` → the call reruns on recovery.
  - default `unsafe` → it does **not** rerun; the model receives a `pi.tool-result` with diagnostic code `interrupted` and the durable partial output committed so far.
  - This is a deliberate product decision, not a limitation to paper over.
- **Application state**: `runtime.memo(name, value)` is a first-writer-wins durable memo. A rerun sees the memo and can skip the effect. Memos disappear when the task becomes terminal.
- **Code Mode**: scripts run deterministically (virtual clock, seeded randomness, UTC) and every tool result is journaled before delivery. On restart the script reruns and receives journaled results in order; a changed path, sandbox fingerprint, or expired deadline fails the call instead.
- **Compaction**: durable task; a manual/background summary is placed at a turn boundary; a summary that would cut before the current context settles `stale`.
- **Storage errors**: fatal to the Session. Nothing after the error publishes; reopen to recover. Custom storage backends are responsible for retrying their own transient failures.

### Storage and deployment

Shipped backends (`README` §Storage):

- **Memory** — tests only.
- **SQLite** (Node `node:sqlite`) — WAL, `synchronous = NORMAL`: commits survive process crashes; the newest commit may be lost on power/host failure.
- **JSONL** — append-only file per record, optional `fsync`; keeps the whole history in memory too, so use SQLite for long-running agents.
- **Cloudflare Durable Object SQLite** — commits are the DO's storage transactions. Main's README adds `onWake`/`wake()` so the DO arms its own alarm and hibernates while tasks sleep (unreleased in 1.3.0).

Constraints:

- **One process owns a storage at a time.** No cross-process locking. Other clients attach to that process (views/watches over an application-chosen transport).
- Node `>= 22.19`, ESM, TypeScript types shipped.
- Portable SQLite/JSONL cores avoid Node APIs so they run on Bun or in Workers with a small adapter.

Observed SQLite schema in a 1.3.0 test database (9 tables): `conversations`, `entries`, `tasks`, `submissions`, `documents`, `document_revisions`, `record_ids`, `durable_metadata`, `durable_schema`. Entries and task records are JSON in `TEXT` columns with `json_valid` checks; indexes cover conversation/kind/status lookups and document addresses. This is a compact, inspectable store — a transcript can be reconstructed with plain SQL.

### Extensions, tools, environment

- **Registry**: extensions are named bundles of tools, prompt sections, hooks, wrappers, and tasks. Conversations store extension/tool **names**, never code, so they survive restarts and pick up whatever the new process installs. Registry changes are live; an in-flight call finishes on the code it started with.
- **Tools**: TypeBox-validated arguments; three result channels — `output` (model), `structuredOutput` (programs/nested calls), `details` (UI). Nested tool calls (`api.executeTool`) are their own tasks with a key for replay reattachment.
- **Code Mode**: model writes JavaScript in a QuickJS sandbox; tool calls become nested durable calls; journaled replay as above.
- **Environment**: `env({ cwd, conversationId })` builds an `ExecutionEnv` per tool call. Built-in `NodeExecutionEnv` gives full host FS/shell. A remote/container env can put tools on another machine. **No sandbox is included by default** — isolation is entirely the env implementation's job.
- **Observability**: `pi.durable.*` OpenTelemetry-style spans/events/metrics with content-free attributes (IDs and control metadata only, never messages/arguments/results). `watch()`, `viewState()`, and `watchEvents()` expose committed state for multi-client UIs.

## Empirical checks (run 2026-10-20)

Environment: Node 22.22.2, fresh install of `pi-durable@1.3.0` + `pi-ai@1.3.0` + `chord@1.3.0` (91 packages, 111 MB `node_modules`).

1. **Basic chat via faux provider** — worked as documented: submit → `pi.generation` → `pi.system` + `pi.assistant` entries; transcript `['pi.user', 'pi.system', 'pi.assistant']`.
2. **Crash mid-phase, memo vs no memo** — custom task, `SIGKILL` after committing a memo but before committing the outcome; reopen with `resume()` + `waitForIdle()`:
   - with `runtime.memo("effect", true)`: rerun logged `EFFECT-SKIPPED-BY-MEMO`, task completed; effect ran exactly once total.
   - without memo: rerun logged `EFFECT-RAN` again; effect ran twice total. **Confirmed at-least-once phase execution.**
3. **Crash mid-tool-call** — faux model issues a tool call; tool `SIGKILL`s the process during `execute()`:
   - `replay: "unsafe"` (default): on reopen the tool did **not** rerun; the transcript got a `pi.tool-result` with `diagnostics: [{ severity: "error", code: "interrupted", message: "Tool slow was interrupted and may have partially run" }]`; generation continued with the next model turn.
   - `replay: "safe"`: the tool reran (`TOOL-RAN` printed again) and completed; generation continued.
4. **Cold-open wait race in 1.3.0** — after `SIGKILL`, calling `harness.waitForTask(id)` immediately after `Harness.open()` returned the stored **pending** record instead of waiting (its type says `SettledTask`, i.e. terminal). `resume()` + `waitForIdle()` + then `waitForTask()` returned the terminal outcome. Likely a reconciliation race (`#live` not populated when the wait registers). Observed behavior, reported here as an implementation caveat; needs upstream confirmation before calling it a bug.
5. **Version gap confirmed** — installed 1.3.0 contains no `sleepUntil`, `onWake`, `wake()`, `restartLimit`, or `sleeping` status, although the README on `main` documents them. The 1.3.0 SQLite `tasks` table check constraint only allows `pending/running/waiting/completing/terminal`. **Do not build against main's README thinking it is the published API.**

## Claims vs. guarantees

- **"Committed before anything is shown"** is directionally true but not literal: streamed partials are committed every ~100 ms, and the spec's own footgun list says a crash may lose the current uncommitted throttle window. Committed state is durable; the last 100 ms of visible tokens may not be.
- **"Pick up where it stopped"** means *from the last committed phase boundary*, not from an arbitrary instruction. A phase reruns from its top; side effects after the checkpoint repeat unless memoized/idempotent.
- **"Exactly-once"** applies to **submission admission** (`requestId`), not to tool effects, model calls, or webhooks. There is no outbox/transactional-messaging layer.
- **Failure detection is fast**: a storage error fails the whole Session; code errors fail only their unit (task phase faults the task, tool throws → error result, hook throws → reported and skipped, commit callback throws → rollback).
- **Crash loops**: `settings.restartLimit` (default 5) parks a task that crashes 5 times without a new checkpoint — added on main, not in 1.3.0. This is a necessary guard for deterministic crashes; without it, a poison task resumes forever.

## Non-goals (spec §13)

No whole-Session DOM; no visible-undurable publication; no independent event journal; no session-scoped rewindable documents; no automatic checkpoint heuristic; no automatic third-party view mounting; no CRDT/offline multi-writer merge; no SQL translation of arbitrary Chord ops; no JSONL global compaction or corruption repair; no forced termination of non-cooperative extension code inside one process.

## Maturity and adoption

- `pi-durable` 1.0.0 shipped **2026-10-01**; latest published **1.3.0 on 2026-10-15**. 16 npm versions in ~4 weeks; `[Unreleased]` already contains breaking changes (removal of `TaskRuntime.sleep()`, schema v2 migration that cannot roll back to 1.3.0).
- npm: **387,665 downloads in the week ending 2026-10-18**; 740,046 since first publish (2026-09-19). For context, `pi-ai` did 4.65 M that week and `chord` 1.91 M.
- GitHub: the `pi` monorepo has **116,619 stars / 14,877 forks** (2026-10-20), MIT. `packages/durable` has 175 path commits as of that date, the oldest returned by the API from 2026-09-18 (the package was split out of an earlier "Pico" package); 1.0.0 released 2026-10-01.
- Vendor claim: ~15,000 lines of source excluding tests (~150k GPT tokens / ~250k Claude tokens); storage backends ~3,000 lines. That was accurate at launch: we count **17,662 raw lines / 16,412 non-blank** across 60 `src/**/*.ts` files at tag `v1.0.0`. On `main` at 2026-10-20 it is **26,028 raw lines / 24,342 non-blank** across 84 files — roughly +47% in 19 days.
- Language: TypeScript only. The launch FAQ says a Rust port is "very easy" and "not ruled out", but not planned — relevant to our Rust-core preference.

## Why this matters for the taxonomy

pi-durable demonstrates that **durability can be a property of the harness library**, not only of the substrate. celld/rivet answer "where does state live and how does compute wake?"; pi-durable answers "what must the loop commit, in what order, and what happens on reopen?" The two are composable: the package's natural production shape is one Session per Cloudflare Durable Object (or one Node process) with SQLite as the commit log.

It also imports durable-execution vocabulary into agent infrastructure without pretending to be Temporal:

- effect sandwich ≈ activity with a durable intent record
- memo ≈ idempotency key with storage
- submission `requestId` ≈ client-supplied idempotency key
- code-mode journal ≈ deterministic replay of a workflow
- task ownership tree ≈ structured concurrency / cancellation scopes

## Implications for our stack (hermes-vm)

What to copy into Hermes-style tooling, regardless of language:

1. **Commit-before-publish + `requestId` at every API boundary.** Kanban/webhook/task admission should be idempotent by a caller-supplied key; if our webhook path can be retried without one, it can duplicate work — worth verifying against the current handler.
2. **Effect sandwich for tools.** Separate "intent committed" from "effect performed" from "outcome committed"; make destructive tools (`bash`, deploy, send-message) `unsafe` by default and surface `interrupted` to the model instead of retrying.
3. **Task ownership tree with bottom-up abort.** The semantic we'd want behind `delegate_task`: a parent abort should cascade to children, and a parent should not report done while children run.
4. **Crash-loop guard.** A deterministic crash must not resume forever; park after N restarts and expose `retryTask`.
5. **Content-free telemetry.** IDs and control metadata only — a good default for our own traces.
6. **Compaction as a durable task**, not an in-memory step that can be lost mid-summary.

What not to adopt: another runtime. hermes-vm already is the durable host (systemd, SQLite, kanban, session DBs); pi-durable solves a problem we have only if we build a *multi-client, long-running agent service* in TypeScript. If we do, the package itself (not a reimplementation) is the fastest path: SQLite on the VM, `NodeExecutionEnv`, and the `CodingTools`/Code Mode extensions.

If we want the semantics in a Rust core + Python adapters, pi-durable is a strong **reference design** but not a dependency: the spec is the deliverable, and the storage contract (9 tables, atomic commit, JSON records) is small enough to reimplement.

## Open questions

- No Postgres/KV adapter shipped; the vendor says the interface is small, but no reference implementation exists. A multi-node deployment would need one plus a leader/lease model — explicitly out of scope today.
- Multi-client transport is application-owned: `watch()` gives committed operations, not a wire protocol. How much work is a real Slack/Web UI adapter?
- Security: the default Node env is full host access. There is no permission model, no per-tool sandbox, no policy engine beyond hooks. How does this compare with hypeman/E2B for untrusted tools?
- Cost control: model requests are resent after interruption; no token-level resume. How much does this cost at 100k-token contexts? No published benchmarks for storage or recovery latency (bench scripts exist: `bench:storage`, `bench:scheduler-fanout`, `bench:document-append`, etc.).
- Does `restartLimit`/`sleeping`/`onWake` ship in 1.4, and does the DO path become production-grade?
- Is the source size workable for a reimplementation? At `main` it is 26k raw lines, but much of that is TypeScript types and exhaustive JSDoc; the storage/task core is materially smaller. Worth a focused read of `src/harness/scheduler.ts` (108 KB) before committing to "small enough to reimplement".

## Sources

Primary (code/spec, highest trust):

- [`packages/durable/README.md`](https://github.com/earendil-works/pi/blob/main/packages/durable/README.md) — user-facing API, storage, errors, examples. Accessed 2026-10-20.
- [`packages/durable/docs/spec.md`](https://github.com/earendil-works/pi/blob/main/packages/durable/docs/spec.md) — normative specification: terms/invariants (§1), records (§2), documents (§3), transactions (§4), tasks/effect sandwich/structured concurrency/sleep (§5), submissions (§6), extensions/tools (§7), built-in tasks (§8), views (§9), storage contract (§10), backends (§11), footguns (§12), non-goals (§13).
- [`packages/durable/CHANGELOG.md`](https://github.com/earendil-works/pi/blob/main/packages/durable/CHANGELOG.md) — release history; 1.0.0 2026-10-01 → 1.3.0 2026-10-15; unreleased breaking changes.
- [`packages/durable/package.json`](https://github.com/earendil-works/pi/blob/main/packages/durable/package.json) — version 1.3.0, MIT, Node >= 22.19, deps (pi-ai, chord, pi-codemode, pi-telemetry, TypeBox, Photon, diff).
- [`packages/chord/README.md`](https://github.com/earendil-works/pi/blob/main/packages/chord/README.md) — the document-state substrate: objects, references, observable JSON, deltas, hot reload, SQLite adapters.
- Source tree inspection: `src/harness` (scheduler 108 KB, tool 41 KB, generation 34 KB), `src/session` (transaction 42 KB), `src/storage` (memory 33 KB, sqlite, jsonl), `src/codemode`, `src/tools`, `src/env`.
- GitHub API: repo metadata (116,619 stars, MIT) and commit history for `packages/durable` (175 path commits, oldest 2026-09-18; package split out of "Pico"), accessed 2026-10-20.
- npm registry API: `@earendil-works/pi-durable` dist-tags (latest 1.3.0), first publish 2026-09-19, 16 versions; downloads last week 387,665.

Vendor/community (reported, not independently verified):

- [Earendil Engineering, "Pi Durable" (2026-10-01)](https://earendil.com/posts/pi-durable/) — launch post: positioning ("harness that runs anywhere"), architecture tour, multi-client/multiplayer claims, ~15k LOC claim, FAQ on TypeScript/Rust.
- [explainx.ai, "Pi 1.0 + Durable" (Oct 2026)](https://www.explainx.ai/blog/earendil-pi-1-durable-minimal-harness-2026) — secondary coverage; useful only for reception framing.
- [MindStudio, "Pi Durable: What It Is and Why It Matters"](https://www.mindstudio.ai/blog/pi-durable-long-running-agents) — secondary summary; not used for factual claims.

Verified by execution in this research (reproducible):

- Install `@earendil-works/pi-durable@1.3.0` with Node 22.22.2; run faux-provider chat; kill -9 mid-phase with and without `runtime.memo`; kill -9 inside `replay: "safe"` and default `unsafe` tool calls; inspect SQLite schema. Scripts and DBs under `/tmp/pi-durable-research/test/` on the research host; results summarized in "Empirical checks" above.