e78d9fcecd
Phase 3a of docs/DESIGN.md. MockProvider moves to api/mock.rs, and it, the embedded MSN fixture bodies, and the IDX_MOCK_* hooks compile only with idx-core's new `testing` feature. idx-cli enables it as a dev-dependency, so `cargo test` keeps the fixture-backed provider while release binaries no longer embed fixture data or honor IDX_USE_MOCK_PROVIDER. Parser helpers used only by the mock are gated the same way. The packaged-cli flake check no longer relies on the mock; it checks an offline cache miss (JSON CACHEMISS, exit 3) instead. Goldens unchanged; clippy is clean with and without `testing`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
3.3 KiB
Testing and Smoke Checks
Automated gate
nix flake check -L is the canonical local, pre-push, and CI gate. It runs
offline with pinned tools:
| Check | Covers |
|---|---|
fmt, clippy |
formatting and lints (--all-targets) |
test |
unit/module tests, tests/cli.rs CLI integration tests, and tests/golden.rs output parity |
default, package |
release build and cargo package verification |
packaged-cli |
the wrapped Nix release binary emits JSON, generates completions, and returns a JSON CACHEMISS error with exit 3 offline |
ownership-publisher, publisher-test |
packaged publisher entry point and its regression scenarios |
install-script-sh, install-script-dash, shellcheck |
installer regressions and shell lint |
tests/cli.rs is the behavioral suite: it runs the real idx binary against
fixture-backed mock providers (IDX_USE_MOCK_PROVIDER=1) and covers output
contracts, cache/offline transitions, provider routing, error codes, ownership
imports and queries, and snapshot sync.
tests/golden.rs pins the exact output (exit code, stdout, stderr) of every
command listed in tests/golden/cases.toml, in table and JSON. A mismatch means
user-visible output changed. Regenerate with
UPDATE_GOLDEN=1 cargo test --test golden only when the change is intended,
and commit the golden diff with the reason.
For a focused loop, run one check or filter Cargo tests:
nix build .#checks.x86_64-linux.publisher-test -L
nix develop --command cargo test full_screener
CI cache keys include platform, dependency/toolchain inputs, and checked source contents. The native macOS installer job runs separately to exercise BSD tools.
Live drift check
Fixtures cannot tell when MSN, Yahoo, or IDX change their responses.
scripts/live-smoke.sh runs shipped commands against the real upstreams and
validates each JSON output shape. It is opt-in and never part of the gate.
nix develop --command scripts/live-smoke.sh # stocks + ownership
nix develop --command scripts/live-smoke.sh --group stocks --symbol BBRI
nix develop --command scripts/live-smoke.sh --bin ./result/bin/idx
nix develop --command scripts/live-smoke.sh --compare ./old/bin/idx # A/B against an older build
scripts/live-smoke.sh --dry-run
stocks: everystockscommand for one ticker, pluscompare BUMI,ADRO,AIMS, whose MSN key ratios contain non-finite values.ownership:discover, then imports the newest supported report into a temp DB and listsreleases. IDX sits behind Cloudflare and returns403to many datacenter IPs, including GitHub-hosted runners; run it from a network IDX accepts.
--compare <old-idx> runs every case with both binaries back to back, each
with its own config and cache, and fails on a different exit code, any added,
missing, or retyped JSON key, or changed values for profile, financials,
earnings, and ownership results. Build the old side with, for example,
nix build "git+file://$PWD?ref=main" -o old.
Each run isolates config, cache, and data under tmp/live-smoke/<timestamp>/
and keeps per-case .stdout, .stderr, and .validation logs there.
scripts/audit-msn-fundamentals.sh is a heavier sweep of valuation output
across the whole IDX MSN symbol map, for provider-health audits.