0xrsydn/idx-cli
1
mirror of https://github.com/0xrsydn/idx-cli.git synced 2026-11-07 09:45:36 +00:00
Code Issues Projects Releases 1 Packages Wiki Activity

docs(testing): document Nix gates and behavioral coverage

Browse source 
Describe canonical local and CI verification, cache reuse, output assertions, workflow boundaries, and observed regression results without treating test counts as coverage targets.
This commit is contained in:
hermes 2026-10-03 16:23:16 +00:00
commit cd410708c0
5 changed files with 105 additions and 25 deletions
Download patch file Download diff file Expand all files Collapse all files

7
TODO.md
View file

@ -162,6 +162,13 @@
- [ ] Additional providers (Alpha Vantage, Twelve Data, IDX official)
## 🔬 Latest Smoke Findings (2026-04-02)
- [x] Behavioral coverage consolidation on `2026-10-02`: reduced the mock smoke matrix from 62 launch-heavy cases to 43 checks with separate stdout/stderr, single-document JSON validation, fixture values, and exact cache-data comparisons. Consolidated CLI tests from 89 to 62 stronger scenarios; retained all 173 unit/module regressions.
- [x] Strengthened CLI workflows cover cache misses/expiry/provider isolation, filtered and non-finite financial data, XLSX/ZIP ownership imports and queries, snapshot install/no-op/force, rejected updates preserving database bytes and query results, and 503-candidate screener ranking/full offline caches across four filters.
- [x] The stricter JSON checks exposed and fixed stderr pollution in unsupported `ownership import --fetch-bing` and totally failed `stocks compare`; partial comparisons still return available data and warnings, with `--quiet` suppressing those diagnostics.
- [x] Verification: all eleven checks passed via `nix flake check -L --keep-going` on x86_64-linux (173 unit/module tests, 62 CLI tests, 43 packaged-binary smoke cases). Throwaway fault injection proved malformed JSON, extra documents, wrong-stream output, and changed cache data are rejected. Direct packaged-CLI runs confirmed the 503-candidate/offline-prefix boundary, incomplete-total rejection without cache population, and both clean JSON error fixes. Temporary verification fixtures and injectors were removed.
- [x] Nix verification consolidation on `2026-10-02`: all eleven flake checks pass on x86_64-linux, including publisher regressions, installer `sh`/`dash` scenarios, ShellCheck, 173 unit/module tests, 89 CLI tests, and 62 mock smoke cases. CI and the pre-push hook use `nix flake check -L`; native macOS installer coverage is retained.
- [x] Checks and crate verification now share dev-profile dependency artifacts, while the installed application retains a separate release cache. Package verification logs confirm only `idx-cli` recompiles instead of rebuilding third-party dependencies. Narrowed source inputs and content-aware CI cache keys avoid unrelated rebuilds and allow source revisions to save new cached outputs.
- [x] Cache verification on `2026-10-02`: README/docs edits invalidated only the package check; the other ten derivations stayed unchanged. The real Prek pre-push gate passed, and an unchanged `nix flake check -L` completed in 11.69 seconds with no builds or compilation.
- [x] Combined PR verification on `2026-10-02`: JSON/offline-cache fixes, Nix/Crane CI, and market-data fixes pass `nix flake check -L` on x86_64-linux (173 unit tests, 89 CLI tests, 62 mock smoke cases), plus the publisher and `sh` installer regression scripts.
- [x] Fixed the integration review's screener truncation: full-list queries now re-request MSN's reported total rather than cap candidates at 500, reject incomplete totals, and use separate `all` cache keys. Live US top-performers returned 600 unique stocks; offline limits 3 and 25 matched the full result's prefixes. One upstream HTTP 499 occurred before the successful live check.
- [x] Removed the committed `result-1` Nix output symlink and ignored numbered result links.