diff --git a/FEATURE_SPEC.md b/FEATURE_SPEC.md index 055fa1e..dba4adc 100644 --- a/FEATURE_SPEC.md +++ b/FEATURE_SPEC.md @@ -26,7 +26,7 @@ Use `TODO.md` as the execution log and smoke-history record. ## Verified Current State -- Current automated coverage is `180` tests: `117` unit and `63` integration. +- The canonical automated gate is `nix flake check -L`: unit/module and CLI integration tests, packaged-binary smoke, packaging, lint, and shell regressions. Per-run results belong in `TODO.md`, not a fixed test-count target here. - Reusable smoke coverage exists via `scripts/live-smoke.sh`; command groups are documented in `docs/SMOKE.md`. - The latest smoke notes in `TODO.md` report passing live table and JSON checks for all shipped `stocks` commands. - Cache/offline parity, JSON startup-error handling, screener input validation, and the recent MSN output cleanups have already been completed. @@ -183,10 +183,7 @@ Priority order: Do not treat a core refactor or new provider feature as complete until all of the following are true: -- `cargo build` passes -- `cargo clippy -- -D warnings` passes -- `cargo test` passes -- `scripts/live-smoke.sh --mode mock` passes +- `nix flake check -L` passes, including the packaged-binary mock smoke and the relevant behavioral integration scenarios - the relevant live smoke groups pass for changed user-facing behavior - `TODO.md` is updated with any new smoke finding, regression, or behavior change diff --git a/README.md b/README.md index 46e3e77..b077238 100644 --- a/README.md +++ b/README.md @@ -151,18 +151,31 @@ Precedence order: ## Development +The canonical local and CI verification command is: + ```bash -nix develop -cargo test +nix flake check -L ``` -Prek hooks are configured for quality gates: +It covers Rust formatting, Clippy, tests, crate package verification, release +builds, offline CLI smoke, publisher regressions, installer tests with `sh` and +`dash`, and ShellCheck for installer/smoke scripts. The checks use pinned tools and cached Nix +outputs; an unchanged successful check is reused rather than rerun. -- pre-commit: `cargo fmt --check` + `cargo clippy -- -D warnings` -- pre-push: `cargo test` +For a focused edit/debug loop, use the same pinned toolchain: -Script test suites (no live network): `scripts/install-sh-test.sh`, -`scripts/publish-ownership-snapshot-test.sh`, and `scripts/npm-smoke.sh`. +```bash +nix develop --command cargo test full_screener +``` + +Configured Prek hooks keep whitespace/conflict checks at pre-commit and run +`nix flake check -L` at pre-push. There is no separate full Cargo test/lint gate +to repeat after a successful flake check. + +CI additionally checks the installer with native macOS tools for portability. +Live-provider smoke remains opt-in; see [docs/SMOKE.md](docs/SMOKE.md) for check +selection and caching details. The npm distribution harness remains available +separately as `scripts/npm-smoke.sh`. ## License diff --git a/TODO.md b/TODO.md index 93a3a1a..6727fb8 100644 --- a/TODO.md +++ b/TODO.md @@ -162,6 +162,13 @@ - [ ] Additional providers (Alpha Vantage, Twelve Data, IDX official) ## 🔬 Latest Smoke Findings (2026-04-02) +- [x] Behavioral coverage consolidation on `2026-10-02`: reduced the mock smoke matrix from 62 launch-heavy cases to 43 checks with separate stdout/stderr, single-document JSON validation, fixture values, and exact cache-data comparisons. Consolidated CLI tests from 89 to 62 stronger scenarios; retained all 173 unit/module regressions. +- [x] Strengthened CLI workflows cover cache misses/expiry/provider isolation, filtered and non-finite financial data, XLSX/ZIP ownership imports and queries, snapshot install/no-op/force, rejected updates preserving database bytes and query results, and 503-candidate screener ranking/full offline caches across four filters. +- [x] The stricter JSON checks exposed and fixed stderr pollution in unsupported `ownership import --fetch-bing` and totally failed `stocks compare`; partial comparisons still return available data and warnings, with `--quiet` suppressing those diagnostics. +- [x] Verification: all eleven checks passed via `nix flake check -L --keep-going` on x86_64-linux (173 unit/module tests, 62 CLI tests, 43 packaged-binary smoke cases). Throwaway fault injection proved malformed JSON, extra documents, wrong-stream output, and changed cache data are rejected. Direct packaged-CLI runs confirmed the 503-candidate/offline-prefix boundary, incomplete-total rejection without cache population, and both clean JSON error fixes. Temporary verification fixtures and injectors were removed. +- [x] Nix verification consolidation on `2026-10-02`: all eleven flake checks pass on x86_64-linux, including publisher regressions, installer `sh`/`dash` scenarios, ShellCheck, 173 unit/module tests, 89 CLI tests, and 62 mock smoke cases. CI and the pre-push hook use `nix flake check -L`; native macOS installer coverage is retained. +- [x] Checks and crate verification now share dev-profile dependency artifacts, while the installed application retains a separate release cache. Package verification logs confirm only `idx-cli` recompiles instead of rebuilding third-party dependencies. Narrowed source inputs and content-aware CI cache keys avoid unrelated rebuilds and allow source revisions to save new cached outputs. +- [x] Cache verification on `2026-10-02`: README/docs edits invalidated only the package check; the other ten derivations stayed unchanged. The real Prek pre-push gate passed, and an unchanged `nix flake check -L` completed in 11.69 seconds with no builds or compilation. - [x] Combined PR verification on `2026-10-02`: JSON/offline-cache fixes, Nix/Crane CI, and market-data fixes pass `nix flake check -L` on x86_64-linux (173 unit tests, 89 CLI tests, 62 mock smoke cases), plus the publisher and `sh` installer regression scripts. - [x] Fixed the integration review's screener truncation: full-list queries now re-request MSN's reported total rather than cap candidates at 500, reject incomplete totals, and use separate `all` cache keys. Live US top-performers returned 600 unique stocks; offline limits 3 and 25 matched the full result's prefixes. One upstream HTTP 499 occurred before the successful live check. - [x] Removed the committed `result-1` Nix output symlink and ignored numbered result links. diff --git a/docs/CONVENTIONS.md b/docs/CONVENTIONS.md index a8786dc..5965e6c 100644 --- a/docs/CONVENTIONS.md +++ b/docs/CONVENTIONS.md @@ -114,11 +114,16 @@ let conn = Connection::open(path) - **Money (USD):** whole dollars (i64) for Bing data. ## Testing -- **Unit tests:** pure functions (parsers, normalizers, signals) -- **Integration tests:** in-memory SQLite (`Connection::open_in_memory()`), mock providers +- **Canonical verification:** `nix flake check -L` locally and before pushing; CI uses the same gate. Do not repeat full Cargo build/lint/test runs after a successful flake check. +- **Focused debugging:** use `nix develop --command cargo test ` when iterating on one failure. +- **Unit tests:** retain precise parser, normalization, calculation, and boundary regressions; avoid pinning incidental wording, enum counts, or internal wiring. +- **Integration tests:** prefer CLI workflows, real temporary/in-memory SQLite, and deterministic local provider fixtures for user-visible contracts and state transitions. +- **Output contracts:** parse JSON from stdout on success and stderr on failure; errors must leave stdout empty. Assert meaningful values, types, and stable error codes rather than substrings or diagnostic wording. +- **Stateful workflows:** compare cached data across offline/failure transitions, and verify failed snapshot updates preserve both database bytes and query results. Keep packaged-binary smoke representative; put exhaustive boundaries in integration tests. - **Fixtures:** `tests/fixtures/*.json` — real API responses, sanitized - **No live API calls in CI** — `IDX_USE_MOCK_PROVIDER=1` - **Test naming:** `test__` (e.g., `test_parse_id_number_with_dots`) +- **Coverage over counts:** replacing redundant unit assertions with stronger integration scenarios must retain meaningful error, boundary, and data-integrity coverage. ## Git / VCS - **jj (Jujutsu)** as local workflow, colocated with git diff --git a/docs/SMOKE.md b/docs/SMOKE.md index 55755e4..4d1d31b 100644 --- a/docs/SMOKE.md +++ b/docs/SMOKE.md @@ -5,8 +5,8 @@ Use `scripts/live-smoke.sh` as the reusable smoke runner for shipped CLI surface The script: - builds `target/debug/idx` once by default - isolates config, cache, and data under `tmp/live-smoke//` -- records one log file per case under `tmp/live-smoke//logs/` -- supports fast mock-backed runs and slower live-network passes +- records separate `.stdout` and `.stderr` files per case under `tmp/live-smoke//logs/`, plus `.validation` diagnostics for JSON checks +- supports fast mock-backed runs and slower live-network passes; `jq` is required and provided by `nix develop` ## Common Runs @@ -24,17 +24,44 @@ scripts/audit-msn-fundamentals.sh --tickers BUMI,ADRO,AIMS ## Nix CI Checks -`nix flake check -L` runs formatting, Clippy, Rust tests, crate packaging, the -application and ownership-publisher builds, and the offline mock smoke. Crane -shares dependency build artifacts between the Rust derivations. CI restores the -Nix store from the GitHub Actions cache, keyed on the flake and Cargo inputs. +`nix flake check -L` is the canonical local, pre-push, and Linux CI gate. Its +eleven checks cover: -The publisher regression script remains a separate CI step: +- Rust formatting, Clippy (`--all-targets`), and all unit/CLI tests +- the optimized application build and verified crate packaging +- the packaged ownership publisher's `--help` entry point +- the offline mock CLI smoke matrix +- publisher regression scenarios, including the SQLite artifact builder +- installer regressions under `sh` and `dash`, plus installer and smoke-runner ShellCheck + +All check tools are pinned by `flake.lock`. Installer fixtures use resolved +shell paths, and generated publisher stubs use the running Bash interpreter, +so the regression suites also work inside the Nix sandbox. + +Crane shares dev-profile dependency artifacts between tests, Clippy, and +`cargo package` verification; the shipped application uses a separate release +dependency cache. Package verification still builds the packaged crate. +Integration-test edits do not invalidate the application build, and script +checks consume only their relevant source files. README/license edits affect +the packaging check, not the Rust test/application derivations. + +CI cache keys include platform, dependency/toolchain inputs, and checked source +contents. Source changes can therefore save new outputs; restore prefixes first +reuse the same dependency/toolchain cache, then another cache for the platform. +The GitHub cache is opportunistic and subject to eviction, not a shared public +binary cache. Obsolete PR runs are cancelled; main-branch runs are not. + +Use individual checks when investigating a failure (replace the system as +needed), or targeted Cargo commands inside the pinned development shell: ```bash -scripts/publish-ownership-snapshot-test.sh +nix build .#checks.x86_64-linux.publisher-test -L +nix develop --command cargo test full_screener ``` +The native macOS installer job remains separate to exercise BSD/macOS tools. +Live-network smoke is deliberately outside the sandboxed offline gate. + For a live full-list screener check, use an isolated config/cache and run: ```bash @@ -48,6 +75,37 @@ candidates. The two offline results must match prefixes of the same cached full list. Provider counts and prices vary; live HTTP failures are not offline-test failures. +## Coverage Boundaries + +The mock preset has 43 cases: 7 general, 19 stock, 9 cache, 2 routing, 4 error, +and 2 empty/unsupported ownership checks. This is a packaged-binary safety net, +not a claim that every important behavior is covered by shell smoke. + +- All fifteen stock commands run in JSON mode with structural and fixture-value + assertions. Four representative table renders check displayed data. +- JSON checks require exactly one document in the correct stream. Failed + commands must leave stdout empty and expose the expected stable error code. +- Cache scenarios use zero TTL, force provider failures, and compare parsed + offline/stale results with the warmed data; diagnostics must stay on stderr. +- Redundant table launches, duplicate routing cases, and repeated setup checks + are omitted rather than counted as extra correctness coverage. + +The CLI integration suite carries deeper workflows under the same Nix gate: +cache misses, provider isolation, filtered reports, non-finite values, +XLSX/ZIP import and ownership queries, snapshot install/no-op/force, and rejected +updates preserving database bytes and query results. + +Screener regressions generate 503 distinct candidates, with ranking leaders +beyond the first 500. They exercise top/worst performers, volume, and market-cap +ordering, a small first request followed by complete offline reads, and rejected +missing/incomplete totals without cache poisoning. The raw fixture override +`IDX_MOCK_MSN_SCREENER_FIXTURE` is honored only with `IDX_USE_MOCK_PROVIDER`; +responses honor the requested batch size so the complete-fetch path is exercised. + +Mocks do not verify current upstream availability, authentication, or response +drift. Live provider checks remain a separate, opt-in operation. + + ## Modes @@ -60,8 +118,8 @@ failures. - `general`: `version`, `completions`, `config`, and `cache` command basics - `live-table`: all shipped `stocks` commands in live table mode - `live-json`: all shipped `stocks` commands in live JSON mode -- `mock`: all shipped `stocks` commands against the mock provider in both table and JSON mode -- `cache`: cache warm, `--offline`, and stale-cache fallback checks for quote, technical, and MSN `profile` +- `mock`: all shipped `stocks` commands against the mock provider in JSON mode, plus representative quote/history/fundamental/financials table renders +- `cache`: exact data equality across warm, `--offline`, and stale-cache fallback reads for quote, technical, and MSN `profile` - `routing`: Yahoo/MSN provider routing plus explicit MSN history behavior - `errors`: JSON error contract and invalid flag/input checks - `live-nonfinite`: opt-in live MSN fundamentals checks for known non-finite ticker payloads (`BUMI`, `ADRO`, `AIMS`) @@ -78,4 +136,4 @@ failures. - `ownership sync` is still primarily covered by fixture-backed CLI tests rather than the reusable smoke runner. - The new `ownership-import` group is intentionally opt-in for explicit `--group ownership-import` runs or `--mode full`; it discovers live URLs first, imports the supported `above1` attachment into the temp DB, then asserts the current `above5` and `investor-type` URLs fail with explicit unsupported-schema UX. - The new `live-nonfinite` group is intentionally opt-in only. As of `2026-04-13`, the known real repro tickers are `BUMI`, `ADRO`, and `AIMS`. -- When a case fails, inspect the per-case log in `tmp/live-smoke/.../logs/` before updating `TODO.md` or `FEATURE_SPEC.md`. +- When a case fails, inspect its `.stdout`, `.stderr`, and (for JSON cases) `.validation` files in `tmp/live-smoke/.../logs/` before updating `TODO.md` or `FEATURE_SPEC.md`.