fe627496dd
Phase 0 of docs/DESIGN.md, before any refactor code: - tests/golden.rs runs every case in tests/golden/cases.toml (126 outputs across help, completions, config, cache, all stocks commands on MSN and Yahoo mocks, error paths, and ownership queries on a seeded fixture DB) and compares exit code, stdout, and stderr byte for byte. Regenerate only with UPDATE_GOLDEN=1 for a reviewed output change. - scripts/live-smoke.sh --compare <old-idx> runs each live case with both binaries and fails on differing exit codes, added/missing/retyped JSON keys, or changed values for slow-moving data. - Exclude golden files from the whitespace/EOF pre-commit fixers, which would otherwise rewrite pinned help output. Both gates were fault-injected: a one-word table change fails exactly the three history goldens, and a binary dropping profile.long_name fails only the profile live comparison. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
3.2 KiB
Testing and Smoke Checks
Automated gate
nix flake check -L is the canonical local, pre-push, and CI gate. It runs
offline with pinned tools:
| Check | Covers |
|---|---|
fmt, clippy |
formatting and lints (--all-targets) |
test |
unit/module tests, tests/cli.rs CLI integration tests, and tests/golden.rs output parity |
default, package |
release build and cargo package verification |
packaged-cli |
the wrapped Nix binary starts, emits JSON, and generates completions |
ownership-publisher, publisher-test |
packaged publisher entry point and its regression scenarios |
install-script-sh, install-script-dash, shellcheck |
installer regressions and shell lint |
tests/cli.rs is the behavioral suite: it runs the real idx binary against
fixture-backed mock providers (IDX_USE_MOCK_PROVIDER=1) and covers output
contracts, cache/offline transitions, provider routing, error codes, ownership
imports and queries, and snapshot sync.
tests/golden.rs pins the exact output (exit code, stdout, stderr) of every
command listed in tests/golden/cases.toml, in table and JSON. A mismatch means
user-visible output changed. Regenerate with
UPDATE_GOLDEN=1 cargo test --test golden only when the change is intended,
and commit the golden diff with the reason.
For a focused loop, run one check or filter Cargo tests:
nix build .#checks.x86_64-linux.publisher-test -L
nix develop --command cargo test full_screener
CI cache keys include platform, dependency/toolchain inputs, and checked source contents. The native macOS installer job runs separately to exercise BSD tools.
Live drift check
Fixtures cannot tell when MSN, Yahoo, or IDX change their responses.
scripts/live-smoke.sh runs shipped commands against the real upstreams and
validates each JSON output shape. It is opt-in and never part of the gate.
nix develop --command scripts/live-smoke.sh # stocks + ownership
nix develop --command scripts/live-smoke.sh --group stocks --symbol BBRI
nix develop --command scripts/live-smoke.sh --bin ./result/bin/idx
nix develop --command scripts/live-smoke.sh --compare ./old/bin/idx # A/B against an older build
scripts/live-smoke.sh --dry-run
stocks: everystockscommand for one ticker, pluscompare BUMI,ADRO,AIMS, whose MSN key ratios contain non-finite values.ownership:discover, then imports the newest supported report into a temp DB and listsreleases. IDX sits behind Cloudflare and returns403to many datacenter IPs, including GitHub-hosted runners; run it from a network IDX accepts.
--compare <old-idx> runs every case with both binaries back to back, each
with its own config and cache, and fails on a different exit code, any added,
missing, or retyped JSON key, or changed values for profile, financials,
earnings, and ownership results. Build the old side with, for example,
nix build "git+file://$PWD?ref=main" -o old.
Each run isolates config, cache, and data under tmp/live-smoke/<timestamp>/
and keeps per-case .stdout, .stderr, and .validation logs there.
scripts/audit-msn-fundamentals.sh is a heavier sweep of valuation output
across the whole IDX MSN symbol map, for provider-health audits.