idx-cli/tests/agents/cases.json
hermes 687c9ad83f test(agents): close ways an --npx run could pass or fail wrongly
From a Codex review of the harness:
- Drop target/debug from PATH in --npx mode (the dev shell adds it), so
  a missing package bin cannot fall through to the local build, and fail
  an agent whose npm cache holds no installed idx-mcp binary.
- The live `changes` case must diff the two latest releases, read from
  the synced snapshot, not any two dates.
- Hermes: `hermes -z` waits ~15s for MCP tools regardless of
  connect_timeout, so a slow cold install lost the first case. Warm up
  with `hermes mcp test idx`, which installs through Hermes' own npx and
  honors connect_timeout (now 300s).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V3syn2fN2LNo5pvTSqc6eh
2026-10-16 08:03:39 +00:00

42 lines
1.5 KiB
JSON

[
{
"id": "shareholders",
"prompt": "Who are the shareholders above 5% of BBCA?",
"expect_tools": [{"name": "get_shareholders", "args": {"ticker": "BBCA"}}],
"expect_text": ["(?i)dwimuria", "54[.,]9"]
},
{
"id": "quote",
"prompt": "What is the latest BBCA share price?",
"expect_tools": [{"name": "get_quotes", "args": {"symbols": ["BBCA"]}}],
"expect_text": ["(?i)bbca"]
},
{
"id": "compare",
"prompt": "Compare the fundamentals of BBCA and BBRI.",
"expect_tools": [{"name": "compare_stocks", "args": {"symbols": ["BBCA", "BBRI"]}}],
"expect_text": ["(?i)bbri"]
},
{
"id": "technical",
"prompt": "Give me a technical analysis of BBCA: trend, RSI, and MACD.",
"expect_tools": [{"name": "analyze_stock", "args": {"symbol": "BBCA"}}],
"expect_text": ["(?i)rsi"]
},
{
"id": "investor",
"prompt": "Which IDX stocks does Anthoni Salim hold above 1%?",
"expect_tools": [{"any": ["get_investor_holdings", "search_investors"]}],
"expect_text": ["(?i)bbca"]
},
{
"id": "changes",
"prompt": "Which shareholders changed between the 2026-01-30 and 2026-02-27 KSEI releases?",
"expect_tools": [{"name": "get_ownership_changes", "args": {"from": "2026-01-30", "to": "2026-02-27"}}],
"expect_text": [],
"live": {
"prompt": "Which shareholders changed between the two most recent KSEI releases?",
"expect_tools": [{"name": "get_ownership_changes", "args": {"from": "{previous_release}", "to": "{latest_release}"}}]
}
}
]