From a Codex review of the harness:
- Drop target/debug from PATH in --npx mode (the dev shell adds it), so
a missing package bin cannot fall through to the local build, and fail
an agent whose npm cache holds no installed idx-mcp binary.
- The live `changes` case must diff the two latest releases, read from
the synced snapshot, not any two dates.
- Hermes: `hermes -z` waits ~15s for MCP tools regardless of
connect_timeout, so a slow cold install lost the first case. Warm up
with `hermes mcp test idx`, which installs through Hermes' own npx and
honors connect_timeout (now 300s).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V3syn2fN2LNo5pvTSqc6eh
opencode 2 (@opencode/cli) moved MCP servers to `mcp.servers`, turns
Code Mode on by default, replaced `permission` with ordered
`permissions` rules, and dropped `run --pure` for `--standalone` (runs
otherwise go through a shared background service). Pick the config and
flag from `opencode --version`.
Its `execute` tool stays allowed: with it denied, opencode 2.0.30 never
offers the idx tools to the model in `opencode run`, though the server
connects before the first step.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V3syn2fN2LNo5pvTSqc6eh
opencode kept its built-in tools, so it could route around the server
under test: on published 0.3.1 it answered a quote from web search, and
on the live `changes` case it queried the ownership SQLite file with
bash and timed out. Deny every tool but `idx_*`, as Claude, pi, and
Hermes already are.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V3syn2fN2LNo5pvTSqc6eh
The `changes` case names the fixture releases (2026-01-30, 2026-02-27),
which the live snapshot does not have, so under --live/--npx it failed
agents that correctly declined to answer about missing releases and
passed ones that called the tool anyway. A case's `live` fields now
replace the mock ones; this one asks about the two most recent releases.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V3syn2fN2LNo5pvTSqc6eh
All agents shared one npm cache, so only the first ran a cold
`npx -y idx-mcp` install and the others reused its installed binary,
never exercising their own first-run install.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V3syn2fN2LNo5pvTSqc6eh
- agent-smoke `--npx <spec>` registers `npx -y <spec>` (the published package
or an `npm pack` tarball) with a fresh npm cache, instead of the local build,
so each agent's own npx handling and a cold install are tested.
- `nix develop .#agents-fhs` (Linux) runs agents in an FHS sandbox: Hermes
replaces a bare `npx` with its own generic-Linux Node, which NixOS cannot
load. `agent-fhs <cmd>` runs anything inside it.
- Hermes: only the idx toolset (with its terminal it repaired a broken
server mid-run and passed), the Node toolchain fetched before the first
case (`hermes -z` waits ~15s for MCP tools), JSON config.yaml (as profiles
write it), linked plugins for plugin model providers, and a copy of its
mcp-stderr.log per run.
- A failed run names its cause inline, since output is deleted without --keep.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WB13omP6h7YeyJx7GNtUnF
Findings from a Codex (gpt-6.1-sol, xhigh) review of v0.2.4..HEAD, each
verified in the code before fixing:
- high: ticker symbols reached cache file paths unchecked, so a symbol
like `../../REPORT` read (and deleted, if unparseable) .json files
outside the cache, reachable through MCP get_quotes. resolve_symbol now
accepts only ticker characters, and cache keys are encoded as single
safe path components.
- high: sync decided before downloading and replaced the database
without rechecking; an import that landed meanwhile was erased. It now
rechecks the local state before installing, and idx-mcp runs
ownership_sync exclusively with ownership reads.
- replaces_sqlite_sha256 never downgrades history: the client requires
the rebuild to keep at least the local release count, and the
publisher lists a replaced build only then.
- cache writes are atomic (temp file + rename), so concurrent readers
never see, and delete, a partial entry.
- idx-mcp caps symbols (20, deduplicated) and limits, and runs at most 4
tool calls at once.
- each Yahoo client gets its own cookie jar, paired with its crumb.
- ZIP entries (XLSX, KSEI archives) are read with a 256 MiB bound;
non-ASCII archive dates and huge percentages are parse errors instead
of panics or overflow.
- agent-smoke --keep deletes every copied agent config, not only auth
files.
CLI goldens unchanged; the MCP tool golden changes only to state the
new caps.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Require bash 4.4+ up front, and match answer facts with jq regexes
(Oniguruma, (?i) everywhere) instead of GNU-only grep -P.
- Unparseable agent output fails that run instead of aborting the
matrix; check_run now also reports unparseable tool calls, which
previously passed the tool check silently.
- Copy the Hermes model block with awk up to the next top-level key, so
a final or differently ordered block is handled.
- --keep keeps run output but deletes the copied agent credentials.
- docs: full path to the tool golden.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- tests/agents/agent-smoke.sh runs tests/agents/cases.json through
Claude Code, Codex, pi, Hermes Agent, and opencode headless against a
local idx-mcp build, each with an isolated config, and checks the
tools called, their key arguments, and (mock data) facts in the
answer. Opt-in; never part of the gate.
- crates/idx-mcp/tests/golden/tools.json pins tool names, descriptions,
and input schemas, so changes to what agents see are reviewed.
- nix develop .#agents adds Claude Code, Codex, and opencode from
nixpkgs after the host PATH.
First run: 30/30 (5 agents x 6 cases, mock data).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
