0xrsydn/idx-cli
1
mirror of https://github.com/0xrsydn/idx-cli.git synced 2026-11-07 09:45:36 +00:00
Code Issues Projects Releases 1 Packages Wiki Activity

test(agents): a live variant of the ownership-changes case

Browse source 
The `changes` case names the fixture releases (2026-01-30, 2026-02-27),
which the live snapshot does not have, so under --live/--npx it failed
agents that correctly declined to answer about missing releases and
passed ones that called the tool anyway. A case's `live` fields now
replace the mock ones; this one asks about the two most recent releases.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V3syn2fN2LNo5pvTSqc6eh
This commit is contained in:
hermes 2026-10-16 06:09:00 +00:00
commit 7ebf63ed42
3 changed files with 10 additions and 2 deletions
Download patch file Download diff file Expand all files Collapse all files

2
docs/SMOKE.md
View file

@ -111,6 +111,8 @@ each run keeps a copy next to its output as `<case>.<run>.mcp-stderr`.
- `IDX_AGENT_MODEL_<AGENT>` picks a model (for example - `IDX_AGENT_MODEL_<AGENT>` picks a model (for example
`IDX_AGENT_MODEL_CLAUDE=sonnet`). `IDX_AGENT_EXEC_PREFIX` prefixes every `IDX_AGENT_MODEL_CLAUDE=sonnet`). `IDX_AGENT_EXEC_PREFIX` prefixes every
agent command, for hosts that inject API keys through a wrapper. agent command, for hosts that inject API keys through a wrapper.
- With `--live` (and `--npx`), a case's `live` fields replace its mock ones,
for prompts that name data only the fixtures have, such as release dates.
- The run fails only when an agent fails a case in every run; use `--runs` to - The run fails only when an agent fails a case in every run; use `--runs` to
measure pass rates. `--keep` keeps raw agent output and `results.jsonl`. measure pass rates. `--keep` keeps raw agent output and `results.jsonl`.
- `nix develop .#agents` adds Claude Code, Codex, and opencode from nixpkgs - `nix develop .#agents` adds Claude Code, Codex, and opencode from nixpkgs

4
tests/agents/agent-smoke.sh
View file

@ -339,7 +339,9 @@ for agent in "${agents[@]}"; do
fi fi
"setup_$agent" "setup_$agent"
for id in "${case_ids[@]}"; do for id in "${case_ids[@]}"; do
case_json="$(jq -c --arg id "$id" '.[] | select(.id == $id)' "$cases_file")" # A case's "live" fields replace the mock ones, e.g. release dates.
case_json="$(jq -c --arg id "$id" --argjson live "$live" '.[] | select(.id == $id)
| if $live == 1 then . + (.live // {}) else . end | del(.live)' "$cases_file")"
prompt="$(jq -r '.prompt' <<< "$case_json") Use the idx MCP tools." prompt="$(jq -r '.prompt' <<< "$case_json") Use the idx MCP tools."
for ((run = 1; run <= runs; run++)); do for ((run = 1; run <= runs; run++)); do
out="$work/runs/$agent/$id.$run" out="$work/runs/$agent/$id.$run"

6
tests/agents/cases.json
View file

@ -33,6 +33,10 @@
"id": "changes", "id": "changes",
"prompt": "Which shareholders changed between the 2026-01-30 and 2026-02-27 KSEI releases?", "prompt": "Which shareholders changed between the 2026-01-30 and 2026-02-27 KSEI releases?",
"expect_tools": [{"name": "get_ownership_changes", "args": {"from": "2026-01-30", "to": "2026-02-27"}}], "expect_tools": [{"name": "get_ownership_changes", "args": {"from": "2026-01-30", "to": "2026-02-27"}}],
"expect_text": [] "expect_text": [],
"live": {
"prompt": "Which shareholders changed between the two most recent KSEI releases?",
"expect_tools": [{"name": "get_ownership_changes"}]
}
} }
] ]