mirror of
https://github.com/0xrsydn/idx-cli.git
synced 2026-11-07 09:45:36 +00:00
test(agents): run agents through npx -y idx-mcp, as users configure it
- agent-smoke `--npx <spec>` registers `npx -y <spec>` (the published package or an `npm pack` tarball) with a fresh npm cache, instead of the local build, so each agent's own npx handling and a cold install are tested. - `nix develop .#agents-fhs` (Linux) runs agents in an FHS sandbox: Hermes replaces a bare `npx` with its own generic-Linux Node, which NixOS cannot load. `agent-fhs <cmd>` runs anything inside it. - Hermes: only the idx toolset (with its terminal it repaired a broken server mid-run and passed), the Node toolchain fetched before the first case (`hermes -z` waits ~15s for MCP tools), JSON config.yaml (as profiles write it), linked plugins for plugin model providers, and a copy of its mcp-stderr.log per run. - A failed run names its cause inline, since output is deleted without --keep. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WB13omP6h7YeyJx7GNtUnF
This commit is contained in:
parent
bdbcf62a7b
commit
ca218ef2ea
3 changed files with 121 additions and 30 deletions
|
|
@ -86,8 +86,19 @@ and agent behavior varies between runs, so it is never part of the gate.
|
|||
nix develop .#agents --command tests/agents/agent-smoke.sh # every agent found
|
||||
nix develop .#agents --command tests/agents/agent-smoke.sh --agents claude,codex --runs 3
|
||||
tests/agents/agent-smoke.sh --live --case shareholders --keep # real data
|
||||
tests/agents/agent-smoke.sh --npx idx-mcp # the published npm package
|
||||
tests/agents/agent-smoke.sh --npx "$(cd npm/idx-mcp && realpath "$(npm pack --silent)")" # this checkout's wrapper
|
||||
```
|
||||
|
||||
`--npx <spec>` registers `npx -y <spec>` the way users configure it, so each
|
||||
agent's own `npx` handling and a cold npm install are part of the test. It
|
||||
runs the release binary, so it implies `--live`; `IDX_BINARY_URL` swaps in a
|
||||
local one. Hermes Agent replaces a bare `npx` with its own generic-Linux
|
||||
Node, which NixOS cannot load; on NixOS use `nix develop .#agents-fhs`, which
|
||||
runs every agent in an FHS sandbox (and `agent-fhs hermes` for an interactive
|
||||
session). Hermes logs a server's stderr to `$HERMES_HOME/logs/mcp-stderr.log`;
|
||||
each run keeps a copy next to its output as `<case>.<run>.mcp-stderr`.
|
||||
|
||||
- Each agent gets an isolated config that registers only the local server;
|
||||
your own agent settings and MCP servers are not loaded or changed. Logins
|
||||
are copied from the agent's home (Codex `auth.json`, pi and Hermes auth
|
||||
|
|
@ -95,7 +106,8 @@ tests/agents/agent-smoke.sh --live --case shareholders --keep # real da
|
|||
output but still deletes every copied agent config.
|
||||
- Tools are auto-approved for the run (Claude `dontAsk` with
|
||||
`mcp__idx__*`, Codex `default_tools_approval_mode="approve"`); Claude and pi
|
||||
also get no built-in tools, so they can only answer through MCP.
|
||||
also get no built-in tools, and Hermes only the `idx` toolset, so they
|
||||
can only answer through MCP.
|
||||
- `IDX_AGENT_MODEL_<AGENT>` picks a model (for example
|
||||
`IDX_AGENT_MODEL_CLAUDE=sonnet`). `IDX_AGENT_EXEC_PREFIX` prefixes every
|
||||
agent command, for hosts that inject API keys through a wrapper.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue