0xrsydn/idx-cli
1
mirror of https://github.com/0xrsydn/idx-cli.git synced 2026-11-07 09:45:36 +00:00
Code Issues Projects Releases 1 Packages Wiki Activity

test: add golden output parity gate and live A/B compare

Browse source 
Phase 0 of docs/DESIGN.md, before any refactor code:

- tests/golden.rs runs every case in tests/golden/cases.toml (126
  outputs across help, completions, config, cache, all stocks commands
  on MSN and Yahoo mocks, error paths, and ownership queries on a seeded
  fixture DB) and compares exit code, stdout, and stderr byte for byte.
  Regenerate only with UPDATE_GOLDEN=1 for a reviewed output change.
- scripts/live-smoke.sh --compare <old-idx> runs each live case with
  both binaries and fails on differing exit codes, added/missing/retyped
  JSON keys, or changed values for slow-moving data.
- Exclude golden files from the whitespace/EOF pre-commit fixers, which
  would otherwise rewrite pinned help output.

Both gates were fault-injected: a one-word table change fails exactly
the three history goldens, and a binary dropping profile.long_name fails
only the profile live comparison.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
hermes 2026-10-14 08:33:25 +00:00
commit fe627496dd
133 changed files with 6033 additions and 23 deletions
Download patch file Download diff file Expand all files Collapse all files

15
docs/SMOKE.md
View file

@ -8,7 +8,7 @@ offline with pinned tools:
| Check | Covers |
| --- | --- |
| `fmt`, `clippy` | formatting and lints (`--all-targets`) |
| `test` | unit/module tests and `tests/cli.rs` CLI integration tests |
| `test` | unit/module tests, `tests/cli.rs` CLI integration tests, and `tests/golden.rs` output parity |
| `default`, `package` | release build and `cargo package` verification |
| `packaged-cli` | the wrapped Nix binary starts, emits JSON, and generates completions |
| `ownership-publisher`, `publisher-test` | packaged publisher entry point and its regression scenarios |
@ -19,6 +19,12 @@ fixture-backed mock providers (`IDX_USE_MOCK_PROVIDER=1`) and covers output
contracts, cache/offline transitions, provider routing, error codes, ownership
imports and queries, and snapshot sync.
`tests/golden.rs` pins the exact output (exit code, stdout, stderr) of every
command listed in `tests/golden/cases.toml`, in table and JSON. A mismatch means
user-visible output changed. Regenerate with
`UPDATE_GOLDEN=1 cargo test --test golden` only when the change is intended,
and commit the golden diff with the reason.
For a focused loop, run one check or filter Cargo tests:
```bash
@ -39,6 +45,7 @@ validates each JSON output shape. It is opt-in and never part of the gate.
nix develop --command scripts/live-smoke.sh # stocks + ownership
nix develop --command scripts/live-smoke.sh --group stocks --symbol BBRI
nix develop --command scripts/live-smoke.sh --bin ./result/bin/idx
nix develop --command scripts/live-smoke.sh --compare ./old/bin/idx # A/B against an older build
scripts/live-smoke.sh --dry-run
```
@ -48,6 +55,12 @@ scripts/live-smoke.sh --dry-run
DB and lists `releases`. IDX sits behind Cloudflare and returns `403` to many
datacenter IPs, including GitHub-hosted runners; run it from a network IDX accepts.
`--compare <old-idx>` runs every case with both binaries back to back, each
with its own config and cache, and fails on a different exit code, any added,
missing, or retyped JSON key, or changed values for `profile`, `financials`,
`earnings`, and ownership results. Build the old side with, for example,
`nix build "git+file://$PWD?ref=main" -o old`.
Each run isolates config, cache, and data under `tmp/live-smoke/<timestamp>/`
and keeps per-case `.stdout`, `.stderr`, and `.validation` logs there.