docs(testing): document Nix gates and behavioral coverage

Describe canonical local and CI verification, cache reuse, output assertions, workflow boundaries, and observed regression results without treating test counts as coverage targets.
This commit is contained in:
hermes 2026-10-03 16:23:16 +00:00
commit cd410708c0
5 changed files with 105 additions and 25 deletions

View file

@ -5,8 +5,8 @@ Use `scripts/live-smoke.sh` as the reusable smoke runner for shipped CLI surface
The script:
- builds `target/debug/idx` once by default
- isolates config, cache, and data under `tmp/live-smoke/<timestamp>/`
- records one log file per case under `tmp/live-smoke/<timestamp>/logs/`
- supports fast mock-backed runs and slower live-network passes
- records separate `.stdout` and `.stderr` files per case under `tmp/live-smoke/<timestamp>/logs/`, plus `.validation` diagnostics for JSON checks
- supports fast mock-backed runs and slower live-network passes; `jq` is required and provided by `nix develop`
## Common Runs
@ -24,17 +24,44 @@ scripts/audit-msn-fundamentals.sh --tickers BUMI,ADRO,AIMS
## Nix CI Checks
`nix flake check -L` runs formatting, Clippy, Rust tests, crate packaging, the
application and ownership-publisher builds, and the offline mock smoke. Crane
shares dependency build artifacts between the Rust derivations. CI restores the
Nix store from the GitHub Actions cache, keyed on the flake and Cargo inputs.
`nix flake check -L` is the canonical local, pre-push, and Linux CI gate. Its
eleven checks cover:
The publisher regression script remains a separate CI step:
- Rust formatting, Clippy (`--all-targets`), and all unit/CLI tests
- the optimized application build and verified crate packaging
- the packaged ownership publisher's `--help` entry point
- the offline mock CLI smoke matrix
- publisher regression scenarios, including the SQLite artifact builder
- installer regressions under `sh` and `dash`, plus installer and smoke-runner ShellCheck
All check tools are pinned by `flake.lock`. Installer fixtures use resolved
shell paths, and generated publisher stubs use the running Bash interpreter,
so the regression suites also work inside the Nix sandbox.
Crane shares dev-profile dependency artifacts between tests, Clippy, and
`cargo package` verification; the shipped application uses a separate release
dependency cache. Package verification still builds the packaged crate.
Integration-test edits do not invalidate the application build, and script
checks consume only their relevant source files. README/license edits affect
the packaging check, not the Rust test/application derivations.
CI cache keys include platform, dependency/toolchain inputs, and checked source
contents. Source changes can therefore save new outputs; restore prefixes first
reuse the same dependency/toolchain cache, then another cache for the platform.
The GitHub cache is opportunistic and subject to eviction, not a shared public
binary cache. Obsolete PR runs are cancelled; main-branch runs are not.
Use individual checks when investigating a failure (replace the system as
needed), or targeted Cargo commands inside the pinned development shell:
```bash
scripts/publish-ownership-snapshot-test.sh
nix build .#checks.x86_64-linux.publisher-test -L
nix develop --command cargo test full_screener
```
The native macOS installer job remains separate to exercise BSD/macOS tools.
Live-network smoke is deliberately outside the sandboxed offline gate.
For a live full-list screener check, use an isolated config/cache and run:
```bash
@ -48,6 +75,37 @@ candidates. The two offline results must match prefixes of the same cached full
list. Provider counts and prices vary; live HTTP failures are not offline-test
failures.
## Coverage Boundaries
The mock preset has 43 cases: 7 general, 19 stock, 9 cache, 2 routing, 4 error,
and 2 empty/unsupported ownership checks. This is a packaged-binary safety net,
not a claim that every important behavior is covered by shell smoke.
- All fifteen stock commands run in JSON mode with structural and fixture-value
assertions. Four representative table renders check displayed data.
- JSON checks require exactly one document in the correct stream. Failed
commands must leave stdout empty and expose the expected stable error code.
- Cache scenarios use zero TTL, force provider failures, and compare parsed
offline/stale results with the warmed data; diagnostics must stay on stderr.
- Redundant table launches, duplicate routing cases, and repeated setup checks
are omitted rather than counted as extra correctness coverage.
The CLI integration suite carries deeper workflows under the same Nix gate:
cache misses, provider isolation, filtered reports, non-finite values,
XLSX/ZIP import and ownership queries, snapshot install/no-op/force, and rejected
updates preserving database bytes and query results.
Screener regressions generate 503 distinct candidates, with ranking leaders
beyond the first 500. They exercise top/worst performers, volume, and market-cap
ordering, a small first request followed by complete offline reads, and rejected
missing/incomplete totals without cache poisoning. The raw fixture override
`IDX_MOCK_MSN_SCREENER_FIXTURE` is honored only with `IDX_USE_MOCK_PROVIDER`;
responses honor the requested batch size so the complete-fetch path is exercised.
Mocks do not verify current upstream availability, authentication, or response
drift. Live provider checks remain a separate, opt-in operation.
## Modes
@ -60,8 +118,8 @@ failures.
- `general`: `version`, `completions`, `config`, and `cache` command basics
- `live-table`: all shipped `stocks` commands in live table mode
- `live-json`: all shipped `stocks` commands in live JSON mode
- `mock`: all shipped `stocks` commands against the mock provider in both table and JSON mode
- `cache`: cache warm, `--offline`, and stale-cache fallback checks for quote, technical, and MSN `profile`
- `mock`: all shipped `stocks` commands against the mock provider in JSON mode, plus representative quote/history/fundamental/financials table renders
- `cache`: exact data equality across warm, `--offline`, and stale-cache fallback reads for quote, technical, and MSN `profile`
- `routing`: Yahoo/MSN provider routing plus explicit MSN history behavior
- `errors`: JSON error contract and invalid flag/input checks
- `live-nonfinite`: opt-in live MSN fundamentals checks for known non-finite ticker payloads (`BUMI`, `ADRO`, `AIMS`)
@ -78,4 +136,4 @@ failures.
- `ownership sync` is still primarily covered by fixture-backed CLI tests rather than the reusable smoke runner.
- The new `ownership-import` group is intentionally opt-in for explicit `--group ownership-import` runs or `--mode full`; it discovers live URLs first, imports the supported `above1` attachment into the temp DB, then asserts the current `above5` and `investor-type` URLs fail with explicit unsupported-schema UX.
- The new `live-nonfinite` group is intentionally opt-in only. As of `2026-04-13`, the known real repro tickers are `BUMI`, `ADRO`, and `AIMS`.
- When a case fails, inspect the per-case log in `tmp/live-smoke/.../logs/` before updating `TODO.md` or `FEATURE_SPEC.md`.
- When a case fails, inspect its `.stdout`, `.stderr`, and (for JSON cases) `.validation` files in `tmp/live-smoke/.../logs/` before updating `TODO.md` or `FEATURE_SPEC.md`.