research/harness-engineering/analysis.md

Analysis — Harness Engineering

Part of harness-engineering/index

Code as the tool interface

Three independent sources converge on the same insight:

• Cloudflare Code Mode: present MCP tools as a generated TypeScript API and let the LLM write code that calls it. LLMs are trained on billions of lines of real code but only contrived tool-call token sequences. Writing code also skips feeding every intermediate tool result back through the context window.
• nanocodex: typed Turn/TurnResult values + event stream over WebSocket instead of stdout parsing. "stdout parsing for agent loops breaks constantly in edge cases; typed events give reliable state checkpoints and clean prompt cache hits."
• Prime Agent RLM: the model's only tool is a persistent IPython REPL; context is a variable, subagents are function calls.

The unifying pattern: the harness should look like a programming environment, not a tool-call menu. This matters directly for how we build agent environments on hermes-vm (and why pi's minimalism / 4-tool philosophy won).

Cloudflare Code Mode mechanics (2026-08-11)

Code Mode = let the model write code instead of calling tools one-by-one. The model gets ONE tool (code execution); its code becomes a compact plan that calls the real tools, processes results, and returns only what the model needs.

Why it beats direct tool calling

• Direct tool calls: model picks tool → result goes back into context → model picks next tool → round-trip each time. Intermediate data pollutes context even when the model only needs a fragment.
• Code Mode: all intermediate logic (compose calls, loop over results, filter/transform, branch) runs inside ONE sandboxed execution. Only the final value comes back to the model. Less context, fewer round-trips, more reusable.

Key features

• Tools as typed methods — configured tools become a TypeScript API the model codes against
• Progressive discovery — codemode.search() / codemode.describe() pull connector definitions on demand instead of loading a huge catalog into context every request (solves the "21-tool MCP eats 13.7k tokens" problem)
• Execution safety — generated code runs in an isolated V8 sandbox (not containers); the sandbox's only exit is the typed tool API, so it can't leak keys or touch the network except through approved connectors
• Integration surfaces — durable runtime (connectors + approvals + replay + snippets), AI SDK, TanStack AI, browser (isolated iframe), MCP (expose MCP tools inside Code Mode), OpenAPI (derive typed methods from an OpenAPI spec)

When to use (their guidance)

• Direct tool calls: simple tasks, small fixed tool set
• Code Mode: composed/dependent calls, large tool catalogs, loops/branching/filtering, reusable logic — i.e. multi-step agent work

Caveats

• Marked experimental — fine to prototype with, caution for production

Implications for our Effect TS stack

The MCP + OpenAPI surfaces are directly relevant: our Effect TS services could expose a typed API that agents write code against instead of raw tool-call schemas. Same convergence as the Codex app-server thread/turn/item streaming model and its typed client schemas (codex app-server generate-ts). Watch item: "Code Mode pattern for Effect TS".

Verdict

Harness engineering is where most of the leverage lives for a single-box agent platform: typed events, streaming items, and a code-shaped tool interface cost little to adopt and remove the classic agent-loop failure modes (stdout parsing, context pollution, tool-catalog bloat).