test(agents): run agents through npx -y idx-mcp, as users configure it

- agent-smoke `--npx <spec>` registers `npx -y <spec>` (the published package
  or an `npm pack` tarball) with a fresh npm cache, instead of the local build,
  so each agent's own npx handling and a cold install are tested.
- `nix develop .#agents-fhs` (Linux) runs agents in an FHS sandbox: Hermes
  replaces a bare `npx` with its own generic-Linux Node, which NixOS cannot
  load. `agent-fhs <cmd>` runs anything inside it.
- Hermes: only the idx toolset (with its terminal it repaired a broken
  server mid-run and passed), the Node toolchain fetched before the first
  case (`hermes -z` waits ~15s for MCP tools), JSON config.yaml (as profiles
  write it), linked plugins for plugin model providers, and a copy of its
  mcp-stderr.log per run.
- A failed run names its cause inline, since output is deleted without --keep.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WB13omP6h7YeyJx7GNtUnF
This commit is contained in:
hermes 2026-10-16 04:40:40 +00:00
commit ca218ef2ea
3 changed files with 121 additions and 30 deletions

View file

@ -86,8 +86,19 @@ and agent behavior varies between runs, so it is never part of the gate.
nix develop .#agents --command tests/agents/agent-smoke.sh # every agent found
nix develop .#agents --command tests/agents/agent-smoke.sh --agents claude,codex --runs 3
tests/agents/agent-smoke.sh --live --case shareholders --keep # real data
tests/agents/agent-smoke.sh --npx idx-mcp # the published npm package
tests/agents/agent-smoke.sh --npx "$(cd npm/idx-mcp && realpath "$(npm pack --silent)")" # this checkout's wrapper
```
`--npx <spec>` registers `npx -y <spec>` the way users configure it, so each
agent's own `npx` handling and a cold npm install are part of the test. It
runs the release binary, so it implies `--live`; `IDX_BINARY_URL` swaps in a
local one. Hermes Agent replaces a bare `npx` with its own generic-Linux
Node, which NixOS cannot load; on NixOS use `nix develop .#agents-fhs`, which
runs every agent in an FHS sandbox (and `agent-fhs hermes` for an interactive
session). Hermes logs a server's stderr to `$HERMES_HOME/logs/mcp-stderr.log`;
each run keeps a copy next to its output as `<case>.<run>.mcp-stderr`.
- Each agent gets an isolated config that registers only the local server;
your own agent settings and MCP servers are not loaded or changed. Logins
are copied from the agent's home (Codex `auth.json`, pi and Hermes auth
@ -95,7 +106,8 @@ tests/agents/agent-smoke.sh --live --case shareholders --keep # real da
output but still deletes every copied agent config.
- Tools are auto-approved for the run (Claude `dontAsk` with
`mcp__idx__*`, Codex `default_tools_approval_mode="approve"`); Claude and pi
also get no built-in tools, so they can only answer through MCP.
also get no built-in tools, and Hermes only the `idx` toolset, so they
can only answer through MCP.
- `IDX_AGENT_MODEL_<AGENT>` picks a model (for example
`IDX_AGENT_MODEL_CLAUDE=sonnet`). `IDX_AGENT_EXEC_PREFIX` prefixes every
agent command, for hosts that inject API keys through a wrapper.