docs(testing): document Nix gates and behavioral coverage

Describe canonical local and CI verification, cache reuse, output assertions, workflow boundaries, and observed regression results without treating test counts as coverage targets.

FEATURE_SPEC.md

@@ -26,7 +26,7 @@ Use `TODO.md` as the execution log and smoke-history record.
 
 ## Verified Current State
 
-- Current automated coverage is `180` tests: `117` unit and `63` integration.
+- The canonical automated gate is `nix flake check -L`: unit/module and CLI integration tests, packaged-binary smoke, packaging, lint, and shell regressions. Per-run results belong in `TODO.md`, not a fixed test-count target here.
 - Reusable smoke coverage exists via `scripts/live-smoke.sh`; command groups are documented in `docs/SMOKE.md`.
 - The latest smoke notes in `TODO.md` report passing live table and JSON checks for all shipped `stocks` commands.
 - Cache/offline parity, JSON startup-error handling, screener input validation, and the recent MSN output cleanups have already been completed.
@@ -183,10 +183,7 @@ Priority order:
 
 Do not treat a core refactor or new provider feature as complete until all of the following are true:
 
-- `cargo build` passes
+- `nix flake check -L` passes, including the packaged-binary mock smoke and the relevant behavioral integration scenarios
-- `cargo clippy -- -D warnings` passes
-- `cargo test` passes
-- `scripts/live-smoke.sh --mode mock` passes
 - the relevant live smoke groups pass for changed user-facing behavior
 - `TODO.md` is updated with any new smoke finding, regression, or behavior change
 

README.md

@@ -151,18 +151,31 @@ Precedence order:
 
 ## Development
 
+The canonical local and CI verification command is:
+
 ```bash
-nix develop
+nix flake check -L
-cargo test
 ```
 
-Prek hooks are configured for quality gates:
+It covers Rust formatting, Clippy, tests, crate package verification, release
+builds, offline CLI smoke, publisher regressions, installer tests with `sh` and
+`dash`, and ShellCheck for installer/smoke scripts. The checks use pinned tools and cached Nix
+outputs; an unchanged successful check is reused rather than rerun.
 
-- pre-commit: `cargo fmt --check` + `cargo clippy -- -D warnings`
+For a focused edit/debug loop, use the same pinned toolchain:
-- pre-push: `cargo test`
 
-Script test suites (no live network): `scripts/install-sh-test.sh`,
+```bash
-`scripts/publish-ownership-snapshot-test.sh`, and `scripts/npm-smoke.sh`.
+nix develop --command cargo test full_screener
+```
+
+Configured Prek hooks keep whitespace/conflict checks at pre-commit and run
+`nix flake check -L` at pre-push. There is no separate full Cargo test/lint gate
+to repeat after a successful flake check.
+
+CI additionally checks the installer with native macOS tools for portability.
+Live-provider smoke remains opt-in; see [docs/SMOKE.md](docs/SMOKE.md) for check
+selection and caching details. The npm distribution harness remains available
+separately as `scripts/npm-smoke.sh`.
 
 ## License
 

TODO.md

@@ -162,6 +162,13 @@
 - [ ] Additional providers (Alpha Vantage, Twelve Data, IDX official)
 
 ## 🔬 Latest Smoke Findings (2026-04-02)
+- [x] Behavioral coverage consolidation on `2026-10-02`: reduced the mock smoke matrix from 62 launch-heavy cases to 43 checks with separate stdout/stderr, single-document JSON validation, fixture values, and exact cache-data comparisons. Consolidated CLI tests from 89 to 62 stronger scenarios; retained all 173 unit/module regressions.
+- [x] Strengthened CLI workflows cover cache misses/expiry/provider isolation, filtered and non-finite financial data, XLSX/ZIP ownership imports and queries, snapshot install/no-op/force, rejected updates preserving database bytes and query results, and 503-candidate screener ranking/full offline caches across four filters.
+- [x] The stricter JSON checks exposed and fixed stderr pollution in unsupported `ownership import --fetch-bing` and totally failed `stocks compare`; partial comparisons still return available data and warnings, with `--quiet` suppressing those diagnostics.
+- [x] Verification: all eleven checks passed via `nix flake check -L --keep-going` on x86_64-linux (173 unit/module tests, 62 CLI tests, 43 packaged-binary smoke cases). Throwaway fault injection proved malformed JSON, extra documents, wrong-stream output, and changed cache data are rejected. Direct packaged-CLI runs confirmed the 503-candidate/offline-prefix boundary, incomplete-total rejection without cache population, and both clean JSON error fixes. Temporary verification fixtures and injectors were removed.
+- [x] Nix verification consolidation on `2026-10-02`: all eleven flake checks pass on x86_64-linux, including publisher regressions, installer `sh`/`dash` scenarios, ShellCheck, 173 unit/module tests, 89 CLI tests, and 62 mock smoke cases. CI and the pre-push hook use `nix flake check -L`; native macOS installer coverage is retained.
+- [x] Checks and crate verification now share dev-profile dependency artifacts, while the installed application retains a separate release cache. Package verification logs confirm only `idx-cli` recompiles instead of rebuilding third-party dependencies. Narrowed source inputs and content-aware CI cache keys avoid unrelated rebuilds and allow source revisions to save new cached outputs.
+- [x] Cache verification on `2026-10-02`: README/docs edits invalidated only the package check; the other ten derivations stayed unchanged. The real Prek pre-push gate passed, and an unchanged `nix flake check -L` completed in 11.69 seconds with no builds or compilation.
 - [x] Combined PR verification on `2026-10-02`: JSON/offline-cache fixes, Nix/Crane CI, and market-data fixes pass `nix flake check -L` on x86_64-linux (173 unit tests, 89 CLI tests, 62 mock smoke cases), plus the publisher and `sh` installer regression scripts.
 - [x] Fixed the integration review's screener truncation: full-list queries now re-request MSN's reported total rather than cap candidates at 500, reject incomplete totals, and use separate `all` cache keys. Live US top-performers returned 600 unique stocks; offline limits 3 and 25 matched the full result's prefixes. One upstream HTTP 499 occurred before the successful live check.
 - [x] Removed the committed `result-1` Nix output symlink and ignored numbered result links.

docs/CONVENTIONS.md

@@ -114,11 +114,16 @@ let conn = Connection::open(path)
 - **Money (USD):** whole dollars (i64) for Bing data.
 
 ## Testing
-- **Unit tests:** pure functions (parsers, normalizers, signals)
+- **Canonical verification:** `nix flake check -L` locally and before pushing; CI uses the same gate. Do not repeat full Cargo build/lint/test runs after a successful flake check.
-- **Integration tests:** in-memory SQLite (`Connection::open_in_memory()`), mock providers
+- **Focused debugging:** use `nix develop --command cargo test <filter>` when iterating on one failure.
+- **Unit tests:** retain precise parser, normalization, calculation, and boundary regressions; avoid pinning incidental wording, enum counts, or internal wiring.
+- **Integration tests:** prefer CLI workflows, real temporary/in-memory SQLite, and deterministic local provider fixtures for user-visible contracts and state transitions.
+- **Output contracts:** parse JSON from stdout on success and stderr on failure; errors must leave stdout empty. Assert meaningful values, types, and stable error codes rather than substrings or diagnostic wording.
+- **Stateful workflows:** compare cached data across offline/failure transitions, and verify failed snapshot updates preserve both database bytes and query results. Keep packaged-binary smoke representative; put exhaustive boundaries in integration tests.
 - **Fixtures:** `tests/fixtures/*.json` — real API responses, sanitized
 - **No live API calls in CI** — `IDX_USE_MOCK_PROVIDER=1`
 - **Test naming:** `test_<function>_<scenario>` (e.g., `test_parse_id_number_with_dots`)
+- **Coverage over counts:** replacing redundant unit assertions with stronger integration scenarios must retain meaningful error, boundary, and data-integrity coverage.
 
 ## Git / VCS
 - **jj (Jujutsu)** as local workflow, colocated with git

docs/SMOKE.md

@@ -5,8 +5,8 @@ Use `scripts/live-smoke.sh` as the reusable smoke runner for shipped CLI surface
 The script:
 - builds `target/debug/idx` once by default
 - isolates config, cache, and data under `tmp/live-smoke/<timestamp>/`
-- records one log file per case under `tmp/live-smoke/<timestamp>/logs/`
+- records separate `.stdout` and `.stderr` files per case under `tmp/live-smoke/<timestamp>/logs/`, plus `.validation` diagnostics for JSON checks
-- supports fast mock-backed runs and slower live-network passes
+- supports fast mock-backed runs and slower live-network passes; `jq` is required and provided by `nix develop`
 
 ## Common Runs
 
@@ -24,17 +24,44 @@ scripts/audit-msn-fundamentals.sh --tickers BUMI,ADRO,AIMS
 
 ## Nix CI Checks
 
-`nix flake check -L` runs formatting, Clippy, Rust tests, crate packaging, the
+`nix flake check -L` is the canonical local, pre-push, and Linux CI gate. Its
-application and ownership-publisher builds, and the offline mock smoke. Crane
+eleven checks cover:
-shares dependency build artifacts between the Rust derivations. CI restores the
-Nix store from the GitHub Actions cache, keyed on the flake and Cargo inputs.
 
-The publisher regression script remains a separate CI step:
+- Rust formatting, Clippy (`--all-targets`), and all unit/CLI tests
+- the optimized application build and verified crate packaging
+- the packaged ownership publisher's `--help` entry point
+- the offline mock CLI smoke matrix
+- publisher regression scenarios, including the SQLite artifact builder
+- installer regressions under `sh` and `dash`, plus installer and smoke-runner ShellCheck
+
+All check tools are pinned by `flake.lock`. Installer fixtures use resolved
+shell paths, and generated publisher stubs use the running Bash interpreter,
+so the regression suites also work inside the Nix sandbox.
+
+Crane shares dev-profile dependency artifacts between tests, Clippy, and
+`cargo package` verification; the shipped application uses a separate release
+dependency cache. Package verification still builds the packaged crate.
+Integration-test edits do not invalidate the application build, and script
+checks consume only their relevant source files. README/license edits affect
+the packaging check, not the Rust test/application derivations.
+
+CI cache keys include platform, dependency/toolchain inputs, and checked source
+contents. Source changes can therefore save new outputs; restore prefixes first
+reuse the same dependency/toolchain cache, then another cache for the platform.
+The GitHub cache is opportunistic and subject to eviction, not a shared public
+binary cache. Obsolete PR runs are cancelled; main-branch runs are not.
+
+Use individual checks when investigating a failure (replace the system as
+needed), or targeted Cargo commands inside the pinned development shell:
 
 ```bash
-scripts/publish-ownership-snapshot-test.sh
+nix build .#checks.x86_64-linux.publisher-test -L
+nix develop --command cargo test full_screener
 ```
 
+The native macOS installer job remains separate to exercise BSD/macOS tools.
+Live-network smoke is deliberately outside the sandboxed offline gate.
+
 For a live full-list screener check, use an isolated config/cache and run:
 
 ```bash
@@ -48,6 +75,37 @@ candidates. The two offline results must match prefixes of the same cached full
 list. Provider counts and prices vary; live HTTP failures are not offline-test
 failures.
 
+## Coverage Boundaries
+
+The mock preset has 43 cases: 7 general, 19 stock, 9 cache, 2 routing, 4 error,
+and 2 empty/unsupported ownership checks. This is a packaged-binary safety net,
+not a claim that every important behavior is covered by shell smoke.
+
+- All fifteen stock commands run in JSON mode with structural and fixture-value
+  assertions. Four representative table renders check displayed data.
+- JSON checks require exactly one document in the correct stream. Failed
+  commands must leave stdout empty and expose the expected stable error code.
+- Cache scenarios use zero TTL, force provider failures, and compare parsed
+  offline/stale results with the warmed data; diagnostics must stay on stderr.
+- Redundant table launches, duplicate routing cases, and repeated setup checks
+  are omitted rather than counted as extra correctness coverage.
+
+The CLI integration suite carries deeper workflows under the same Nix gate:
+cache misses, provider isolation, filtered reports, non-finite values,
+XLSX/ZIP import and ownership queries, snapshot install/no-op/force, and rejected
+updates preserving database bytes and query results.
+
+Screener regressions generate 503 distinct candidates, with ranking leaders
+beyond the first 500. They exercise top/worst performers, volume, and market-cap
+ordering, a small first request followed by complete offline reads, and rejected
+missing/incomplete totals without cache poisoning. The raw fixture override
+`IDX_MOCK_MSN_SCREENER_FIXTURE` is honored only with `IDX_USE_MOCK_PROVIDER`;
+responses honor the requested batch size so the complete-fetch path is exercised.
+
+Mocks do not verify current upstream availability, authentication, or response
+drift. Live provider checks remain a separate, opt-in operation.
+
+
 
 ## Modes
 
@@ -60,8 +118,8 @@ failures.
 - `general`: `version`, `completions`, `config`, and `cache` command basics
 - `live-table`: all shipped `stocks` commands in live table mode
 - `live-json`: all shipped `stocks` commands in live JSON mode
-- `mock`: all shipped `stocks` commands against the mock provider in both table and JSON mode
+- `mock`: all shipped `stocks` commands against the mock provider in JSON mode, plus representative quote/history/fundamental/financials table renders
-- `cache`: cache warm, `--offline`, and stale-cache fallback checks for quote, technical, and MSN `profile`
+- `cache`: exact data equality across warm, `--offline`, and stale-cache fallback reads for quote, technical, and MSN `profile`
 - `routing`: Yahoo/MSN provider routing plus explicit MSN history behavior
 - `errors`: JSON error contract and invalid flag/input checks
 - `live-nonfinite`: opt-in live MSN fundamentals checks for known non-finite ticker payloads (`BUMI`, `ADRO`, `AIMS`)
@@ -78,4 +136,4 @@ failures.
 - `ownership sync` is still primarily covered by fixture-backed CLI tests rather than the reusable smoke runner.
 - The new `ownership-import` group is intentionally opt-in for explicit `--group ownership-import` runs or `--mode full`; it discovers live URLs first, imports the supported `above1` attachment into the temp DB, then asserts the current `above5` and `investor-type` URLs fail with explicit unsupported-schema UX.
 - The new `live-nonfinite` group is intentionally opt-in only. As of `2026-04-13`, the known real repro tickers are `BUMI`, `ADRO`, and `AIMS`.
-- When a case fails, inspect the per-case log in `tmp/live-smoke/.../logs/` before updating `TODO.md` or `FEATURE_SPEC.md`.
+- When a case fails, inspect its `.stdout`, `.stderr`, and (for JSON cases) `.validation` files in `tmp/live-smoke/.../logs/` before updating `TODO.md` or `FEATURE_SPEC.md`.