0xrsydn/idx-cli
1
mirror of https://github.com/0xrsydn/idx-cli.git synced 2026-11-07 09:45:36 +00:00
Code Issues Projects Releases 1 Packages Wiki Activity

test(agents): a live variant of the ownership-changes case

Browse source 
The `changes` case names the fixture releases (2026-01-30, 2026-02-27),
which the live snapshot does not have, so under --live/--npx it failed
agents that correctly declined to answer about missing releases and
passed ones that called the tool anyway. A case's `live` fields now
replace the mock ones; this one asks about the two most recent releases.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V3syn2fN2LNo5pvTSqc6eh
This commit is contained in:
hermes 2026-10-16 06:09:00 +00:00
commit 7ebf63ed42
3 changed files with 10 additions and 2 deletions
Download patch file Download diff file Expand all files Collapse all files

2
docs/SMOKE.md
View file

@ -111,6 +111,8 @@ each run keeps a copy next to its output as `<case>.<run>.mcp-stderr`.
- `IDX_AGENT_MODEL_<AGENT>` picks a model (for example
`IDX_AGENT_MODEL_CLAUDE=sonnet`). `IDX_AGENT_EXEC_PREFIX` prefixes every
agent command, for hosts that inject API keys through a wrapper.
- With `--live` (and `--npx`), a case's `live` fields replace its mock ones,
for prompts that name data only the fixtures have, such as release dates.
- The run fails only when an agent fails a case in every run; use `--runs` to
measure pass rates. `--keep` keeps raw agent output and `results.jsonl`.
- `nix develop .#agents` adds Claude Code, Codex, and opencode from nixpkgs