0xrsydn/research
1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Restructure agent research under agent-infra umbrella

Browse source 
- Add agent-infra umbrella topic (hub for state, harness, model-access, collaboration lenses)
- Split harness-engineering out of durable-agent-runtime; fold in Cloudflare Code Mode finding
- Narrow durable-agent-runtime to state/durability lens; retitle
- Tag codex-app-server-byo-auth as agent-infra subtopic
- Document umbrella-topic convention in README
This commit is contained in:
hermes 2026-08-11 08:16:18 +00:00
commit c978db1437
8 changed files with 172 additions and 39 deletions
Download patch file Download diff file Expand all files Collapse all files

42
harness-engineering/analysis.md Normal file
View file

@ -0,0 +1,42 @@
# Analysis — Harness Engineering
> Part of [[harness-engineering/index|Harness Engineering]]
## Code as the tool interface
Three independent sources converge on the same insight:
- **Cloudflare Code Mode**: present MCP tools as a generated TypeScript API and let the LLM write code that calls it. LLMs are trained on billions of lines of real code but only contrived tool-call token sequences. Writing code also skips feeding every intermediate tool result back through the context window.
- **nanocodex**: typed `Turn`/`TurnResult` values + event stream over WebSocket instead of stdout parsing. "stdout parsing for agent loops breaks constantly in edge cases; typed events give reliable state checkpoints and clean prompt cache hits."
- **Prime Agent RLM**: the model's only tool is a persistent IPython REPL; context is a variable, subagents are function calls.
The unifying pattern: **the harness should look like a programming environment, not a tool-call menu.** This matters directly for how we build agent environments on hermes-vm (and why pi's minimalism / 4-tool philosophy won).
## Cloudflare Code Mode mechanics (2026-08-11)
Code Mode = let the model write code instead of calling tools one-by-one. The model gets ONE tool (code execution); its code becomes a compact plan that calls the real tools, processes results, and returns only what the model needs.
**Why it beats direct tool calling**
- Direct tool calls: model picks tool → result goes back into context → model picks next tool → round-trip each time. Intermediate data pollutes context even when the model only needs a fragment.
- Code Mode: all intermediate logic (compose calls, loop over results, filter/transform, branch) runs inside ONE sandboxed execution. Only the final value comes back to the model. Less context, fewer round-trips, more reusable.
**Key features**
- **Tools as typed methods** — configured tools become a TypeScript API the model codes against
- **Progressive discovery** — `codemode.search()` / `codemode.describe()` pull connector definitions on demand instead of loading a huge catalog into context every request (solves the "21-tool MCP eats 13.7k tokens" problem)
- **Execution safety** — generated code runs in an isolated V8 sandbox (not containers); the sandbox's only exit is the typed tool API, so it can't leak keys or touch the network except through approved connectors
- **Integration surfaces** — durable runtime (connectors + approvals + replay + snippets), AI SDK, TanStack AI, browser (isolated iframe), MCP (expose MCP tools inside Code Mode), OpenAPI (derive typed methods from an OpenAPI spec)
**When to use (their guidance)**
- Direct tool calls: simple tasks, small fixed tool set
- Code Mode: composed/dependent calls, large tool catalogs, loops/branching/filtering, reusable logic — i.e. multi-step agent work
**Caveats**
- Marked **experimental** — fine to prototype with, caution for production
## Implications for our Effect TS stack
The MCP + OpenAPI surfaces are directly relevant: our Effect TS services could expose a typed API that agents *write code against* instead of raw tool-call schemas. Same convergence as the Codex app-server thread/turn/item streaming model and its typed client schemas (`codex app-server generate-ts`). Watch item: "Code Mode pattern for Effect TS".
## Verdict
Harness engineering is where most of the leverage lives for a single-box agent platform: typed events, streaming items, and a code-shaped tool interface cost little to adopt and remove the classic agent-loop failure modes (stdout parsing, context pollution, tool-catalog bloat).

46
harness-engineering/index.md Normal file
View file

@ -0,0 +1,46 @@
---
title: Harness Engineering — Agent Loops & Code-as-Tool-Interface
description: The agent harness layer — how the agent loop is built and how the model interacts with tools. Covers nanocodex (agent as in-process library), pi minimalism, Codex protocols, and code-as-tool-interface (Cloudflare Code Mode: tools as typed API, progressive discovery, V8 sandbox).
status: active
category: technical
tags: [harness, agent-loop, code-as-tool, codemode, cloudflare, nanocodex, pi, codex, tool-calling, typed-events, agent-infra]
draft: true
created: 2026-08-11
updated: 2026-08-11
origin: buzz://d8a718be-031f-4a6e-9f5c-a55466641654/c66f017974f748bd6db019c913d3a6f59de349e0cbd90ae38c222e9640521d63
---
# Harness Engineering — Agent Loops & Code-as-Tool-Interface
## Summary
The **harness** is the layer that runs the agent loop: it connects the model to tools, manages context, and streams events. Split out of durable-agent-runtime (2026-08-11) because it is a distinct lens — durability is about state; harness engineering is about the loop and the tool interface. Central 2026 finding: **code-as-tool-interface** — the harness should look like a programming environment, not a tool-call menu.
## Key Findings
- **Code as the tool interface is the converging pattern** (three independent sources):
- Cloudflare Code Mode: MCP tools become a generated TypeScript API; the model writes code that calls it
- nanocodex: typed `Turn`/`TurnResult` events over WebSocket beat stdout parsing
- Prime Agent RLM: a persistent IPython REPL is the model's only tool
- **Why code beats direct tool calls**: LLMs are trained on real code, not synthetic tool-call tokens; intermediate results don't pollute context; composed logic is reusable.
- **Cloudflare Code Mode mechanics** (see analysis): tools as typed methods, progressive discovery (`codemode.search()` / `codemode.describe()`), execution safety via V8 isolate, six integration surfaces, experimental status.
- **Minimalism wins for small harnesses**: pi's 4-tool philosophy; don't load a huge MCP catalog into context.
## Sources
Primary: https://developers.cloudflare.com/agents/tools/codemode/ (2026-08-11); https://learn.chatgpt.com/docs/app-server; nanocodex repo; Prime Agent RLM (arXiv 2605.09998). See [[durable-agent-runtime/sources|Durable Agent Runtime sources]] for shared references.
## Detail Files
- [[harness-engineering/analysis|Analysis]] — code-as-tool-interface deep-dive, Cloudflare Code Mode mechanics, implications for our Effect TS stack
## Related Research
- [[agent-infra/index|Agent Infrastructure]] — umbrella topic
- [[durable-agent-runtime/index|Durable Agent Runtime]] — state/durability lens (formerly contained this topic)
- [[codex-app-server-byo-auth/index|Codex App Server & BYO Auth]] — Codex app-server as an embedding protocol surface
## Next Steps
- [ ] Prototype the "Code Mode pattern for Effect TS": expose our MCP/REST tools as a typed API agents write code against
- [ ] Evaluate which harness primitives (typed events, streaming items, code sandbox) belong in our own agent tooling