test(mcp): agent smoke test across five coding agents, tool snapshot

- tests/agents/agent-smoke.sh runs tests/agents/cases.json through
  Claude Code, Codex, pi, Hermes Agent, and opencode headless against a
  local idx-mcp build, each with an isolated config, and checks the
  tools called, their key arguments, and (mock data) facts in the
  answer. Opt-in; never part of the gate.
- crates/idx-mcp/tests/golden/tools.json pins tool names, descriptions,
  and input schemas, so changes to what agents see are reviewed.
- nix develop .#agents adds Claude Code, Codex, and opencode from
  nixpkgs after the host PATH.

First run: 30/30 (5 agents x 6 cases, mock data).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
hermes 2026-10-14 20:59:56 +00:00
commit 9d8914b2c9
7 changed files with 758 additions and 1 deletions

View file

@ -22,6 +22,7 @@ crates/idx-cli/ binary `idx`
└── tests/ cli.rs (behavior), golden.rs + golden/ (output parity)
crates/idx-mcp/ binary `idx-mcp`: stdio MCP tools (server.rs), inputs (params.rs)
└── tests/mcp.rs JSON-RPC over stdio against the mock provider
tests/agents/ opt-in agent smoke test: real coding agents against idx-mcp
scripts/ smoke, installer, and ownership snapshot publisher
```
@ -52,4 +53,5 @@ Docs:
nix flake check -L # canonical gate (fmt, clippy, tests, package)
nix develop --command cargo test <name> # focused loop
scripts/live-smoke.sh --compare <old> # live drift check, A/B against an older build
tests/agents/agent-smoke.sh # after changing idx-mcp tools: real agents, opt-in
```