test: replace mock smoke matrix with a live-only drift check

The 43-case mock smoke duplicated tests/cli.rs, which already runs the
real binary against fixture providers. Replace the flake's smoke-mock
check with a small packaged-cli check (version JSON, bash completions,
one mock quote), and cut scripts/live-smoke.sh from 672 to ~170 lines:
it now only runs shipped commands against live MSN/Yahoo/IDX and
validates output shape. Rewrite docs/SMOKE.md around the Nix gate and
the drift check.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
hermes 2026-10-14 07:45:51 +00:00
commit 86c108f6eb
7 changed files with 188 additions and 771 deletions

View file

@ -119,7 +119,7 @@ let conn = Connection::open(path)
- **Unit tests:** retain precise parser, normalization, calculation, and boundary regressions; avoid pinning incidental wording, enum counts, or internal wiring.
- **Integration tests:** prefer CLI workflows, real temporary/in-memory SQLite, and deterministic local provider fixtures for user-visible contracts and state transitions.
- **Output contracts:** parse JSON from stdout on success and stderr on failure; errors must leave stdout empty. Assert meaningful values, types, and stable error codes rather than substrings or diagnostic wording.
- **Stateful workflows:** compare cached data across offline/failure transitions, and verify failed snapshot updates preserve both database bytes and query results. Keep packaged-binary smoke representative; put exhaustive boundaries in integration tests.
- **Stateful workflows:** compare cached data across offline/failure transitions, and verify failed snapshot updates preserve both database bytes and query results. Put behavioral coverage in `tests/cli.rs`, not shell smoke.
- **Fixtures:** `tests/fixtures/*.json` — real API responses, sanitized
- **No live API calls in CI** — `IDX_USE_MOCK_PROVIDER=1`
- **Test naming:** `test_<function>_<scenario>` (e.g., `test_parse_id_number_with_dots`)