c978db1437
- Add agent-infra umbrella topic (hub for state, harness, model-access, collaboration lenses) - Split harness-engineering out of durable-agent-runtime; fold in Cloudflare Code Mode finding - Narrow durable-agent-runtime to state/durability lens; retitle - Tag codex-app-server-byo-auth as agent-infra subtopic - Document umbrella-topic convention in README
42 lines
3.6 KiB
Markdown
42 lines
3.6 KiB
Markdown
# Analysis — Harness Engineering
|
|
|
|
> Part of [[harness-engineering/index|Harness Engineering]]
|
|
|
|
## Code as the tool interface
|
|
|
|
Three independent sources converge on the same insight:
|
|
|
|
- **Cloudflare Code Mode**: present MCP tools as a generated TypeScript API and let the LLM write code that calls it. LLMs are trained on billions of lines of real code but only contrived tool-call token sequences. Writing code also skips feeding every intermediate tool result back through the context window.
|
|
- **nanocodex**: typed `Turn`/`TurnResult` values + event stream over WebSocket instead of stdout parsing. "stdout parsing for agent loops breaks constantly in edge cases; typed events give reliable state checkpoints and clean prompt cache hits."
|
|
- **Prime Agent RLM**: the model's only tool is a persistent IPython REPL; context is a variable, subagents are function calls.
|
|
|
|
The unifying pattern: **the harness should look like a programming environment, not a tool-call menu.** This matters directly for how we build agent environments on hermes-vm (and why pi's minimalism / 4-tool philosophy won).
|
|
|
|
## Cloudflare Code Mode mechanics (2026-08-11)
|
|
|
|
Code Mode = let the model write code instead of calling tools one-by-one. The model gets ONE tool (code execution); its code becomes a compact plan that calls the real tools, processes results, and returns only what the model needs.
|
|
|
|
**Why it beats direct tool calling**
|
|
- Direct tool calls: model picks tool → result goes back into context → model picks next tool → round-trip each time. Intermediate data pollutes context even when the model only needs a fragment.
|
|
- Code Mode: all intermediate logic (compose calls, loop over results, filter/transform, branch) runs inside ONE sandboxed execution. Only the final value comes back to the model. Less context, fewer round-trips, more reusable.
|
|
|
|
**Key features**
|
|
- **Tools as typed methods** — configured tools become a TypeScript API the model codes against
|
|
- **Progressive discovery** — `codemode.search()` / `codemode.describe()` pull connector definitions on demand instead of loading a huge catalog into context every request (solves the "21-tool MCP eats 13.7k tokens" problem)
|
|
- **Execution safety** — generated code runs in an isolated V8 sandbox (not containers); the sandbox's only exit is the typed tool API, so it can't leak keys or touch the network except through approved connectors
|
|
- **Integration surfaces** — durable runtime (connectors + approvals + replay + snippets), AI SDK, TanStack AI, browser (isolated iframe), MCP (expose MCP tools inside Code Mode), OpenAPI (derive typed methods from an OpenAPI spec)
|
|
|
|
**When to use (their guidance)**
|
|
- Direct tool calls: simple tasks, small fixed tool set
|
|
- Code Mode: composed/dependent calls, large tool catalogs, loops/branching/filtering, reusable logic — i.e. multi-step agent work
|
|
|
|
**Caveats**
|
|
- Marked **experimental** — fine to prototype with, caution for production
|
|
|
|
## Implications for our Effect TS stack
|
|
|
|
The MCP + OpenAPI surfaces are directly relevant: our Effect TS services could expose a typed API that agents *write code against* instead of raw tool-call schemas. Same convergence as the Codex app-server thread/turn/item streaming model and its typed client schemas (`codex app-server generate-ts`). Watch item: "Code Mode pattern for Effect TS".
|
|
|
|
## Verdict
|
|
|
|
Harness engineering is where most of the leverage lives for a single-box agent platform: typed events, streaming items, and a code-shaped tool interface cost little to adopt and remove the classic agent-loop failure modes (stdout parsing, context pollution, tool-catalog bloat).
|