mirror of
https://github.com/0xrsydn/idx-cli.git
synced 2026-11-07 01:35:36 +00:00
test: add golden output parity gate and live A/B compare
Phase 0 of docs/DESIGN.md, before any refactor code: - tests/golden.rs runs every case in tests/golden/cases.toml (126 outputs across help, completions, config, cache, all stocks commands on MSN and Yahoo mocks, error paths, and ownership queries on a seeded fixture DB) and compares exit code, stdout, and stderr byte for byte. Regenerate only with UPDATE_GOLDEN=1 for a reviewed output change. - scripts/live-smoke.sh --compare <old-idx> runs each live case with both binaries and fails on differing exit codes, added/missing/retyped JSON keys, or changed values for slow-moving data. - Exclude golden files from the whitespace/EOF pre-commit fixers, which would otherwise rewrite pinned help output. Both gates were fault-injected: a one-word table change fails exactly the three history goldens, and a binary dropping profile.long_name fails only the profile live comparison. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
parent
29796f0f9d
commit
fe627496dd
133 changed files with 6033 additions and 23 deletions
|
|
@ -162,7 +162,8 @@ is another 20-30% smaller than pretty JSON.
|
|||
- stdout carries only data; diagnostics and warnings go to stderr. Already true.
|
||||
- Errors in `json`/`jsonl` mode are one JSON object on stderr with stdout empty.
|
||||
Already true.
|
||||
- No ANSI color when stdout is not a terminal.
|
||||
- No ANSI color when stdout is not a terminal. Today some table headers emit
|
||||
bold codes even when piped.
|
||||
- Commands that take symbols accept `-` to read them from stdin, one per line or
|
||||
whitespace-separated:
|
||||
`idx stocks screen --filter low-pe | jq -r '.[].symbol' | idx stocks compare -`
|
||||
|
|
@ -264,14 +265,16 @@ Before any refactor code lands, capture the current CLI as golden files:
|
|||
fixture-built ownership DB
|
||||
- the documented error cases (invalid input, unsupported provider, offline
|
||||
without cache, empty ownership DB)
|
||||
- stored as `tests/golden/<case>.{stdout,stderr,exit}` and compared byte for
|
||||
byte by one data-driven test in `tests/golden.rs`
|
||||
- cases listed in `tests/golden/cases.toml`; each result stored as
|
||||
`tests/golden/<case>.<format>.golden` (exit code, stdout, stderr) and
|
||||
compared byte for byte by one data-driven test in `tests/golden.rs`
|
||||
|
||||
Regenerating goldens (`UPDATE_GOLDEN=1`) is only allowed for an intentional,
|
||||
reviewed output change, and the diff must be in the same commit with the reason
|
||||
in the message. Phases 1-3 must not change any golden file. Phase 0 changes only
|
||||
what it documents: default format resolution, JSON whitespace when piped, and
|
||||
exit codes; JSON content and table bytes stay identical.
|
||||
in the message. Phases 2-4 must not change any golden file. Phase 1 changes only
|
||||
what it documents: default format resolution, JSON whitespace when piped, ANSI
|
||||
color when piped (table output emits bold headers into pipes today), and exit
|
||||
codes; JSON content and table text stay identical.
|
||||
|
||||
`tests/cli.rs` stays the behavioral suite and must keep passing unchanged.
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue