- Require bash 4.4+ up front, and match answer facts with jq regexes
(Oniguruma, (?i) everywhere) instead of GNU-only grep -P.
- Unparseable agent output fails that run instead of aborting the
matrix; check_run now also reports unparseable tool calls, which
previously passed the tool check silently.
- Copy the Hermes model block with awk up to the next top-level key, so
a final or differently ordered block is handled.
- --keep keeps run output but deletes the copied agent credentials.
- docs: full path to the tool golden.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- tests/agents/agent-smoke.sh runs tests/agents/cases.json through
Claude Code, Codex, pi, Hermes Agent, and opencode headless against a
local idx-mcp build, each with an isolated config, and checks the
tools called, their key arguments, and (mock data) facts in the
answer. Opt-in; never part of the gate.
- crates/idx-mcp/tests/golden/tools.json pins tool names, descriptions,
and input schemas, so changes to what agents see are reviewed.
- nix develop .#agents adds Claude Code, Codex, and opencode from
nixpkgs after the host PATH.
First run: 30/30 (5 agents x 6 cases, mock data).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
