mirror of
https://github.com/0xrsydn/idx-cli.git
synced 2026-11-07 01:35:36 +00:00
test: replace mock smoke matrix with a live-only drift check
The 43-case mock smoke duplicated tests/cli.rs, which already runs the real binary against fixture providers. Replace the flake's smoke-mock check with a small packaged-cli check (version JSON, bash completions, one mock quote), and cut scripts/live-smoke.sh from 672 to ~170 lines: it now only runs shipped commands against live MSN/Yahoo/IDX and validates output shape. Rewrite docs/SMOKE.md around the Nix gate and the drift check. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
hermes
parent
1631058b8c
commit
86c108f6eb
7 changed files with 188 additions and 771 deletions
|
|
@ -119,7 +119,7 @@ let conn = Connection::open(path)
|
|||
- **Unit tests:** retain precise parser, normalization, calculation, and boundary regressions; avoid pinning incidental wording, enum counts, or internal wiring.
|
||||
- **Integration tests:** prefer CLI workflows, real temporary/in-memory SQLite, and deterministic local provider fixtures for user-visible contracts and state transitions.
|
||||
- **Output contracts:** parse JSON from stdout on success and stderr on failure; errors must leave stdout empty. Assert meaningful values, types, and stable error codes rather than substrings or diagnostic wording.
|
||||
- **Stateful workflows:** compare cached data across offline/failure transitions, and verify failed snapshot updates preserve both database bytes and query results. Keep packaged-binary smoke representative; put exhaustive boundaries in integration tests.
|
||||
- **Stateful workflows:** compare cached data across offline/failure transitions, and verify failed snapshot updates preserve both database bytes and query results. Put behavioral coverage in `tests/cli.rs`, not shell smoke.
|
||||
- **Fixtures:** `tests/fixtures/*.json` — real API responses, sanitized
|
||||
- **No live API calls in CI** — `IDX_USE_MOCK_PROVIDER=1`
|
||||
- **Test naming:** `test_<function>_<scenario>` (e.g., `test_parse_id_number_with_dots`)
|
||||
|
|
|
|||
154
docs/SMOKE.md
154
docs/SMOKE.md
|
|
@ -1,135 +1,55 @@
|
|||
# Smoke Checks
|
||||
# Testing and Smoke Checks
|
||||
|
||||
Use `scripts/live-smoke.sh` as the reusable smoke runner for shipped CLI surfaces.
|
||||
## Automated gate
|
||||
|
||||
The script:
|
||||
- builds `target/debug/idx` once by default
|
||||
- isolates config, cache, and data under `tmp/live-smoke/<timestamp>/`
|
||||
- records separate `.stdout` and `.stderr` files per case under `tmp/live-smoke/<timestamp>/logs/`, plus `.validation` diagnostics for JSON checks
|
||||
- supports fast mock-backed runs and slower live-network passes; `jq` is required and provided by `nix develop`
|
||||
`nix flake check -L` is the canonical local, pre-push, and CI gate. It runs
|
||||
offline with pinned tools:
|
||||
|
||||
## Common Runs
|
||||
| Check | Covers |
|
||||
| --- | --- |
|
||||
| `fmt`, `clippy` | formatting and lints (`--all-targets`) |
|
||||
| `test` | unit/module tests and `tests/cli.rs` CLI integration tests |
|
||||
| `default`, `package` | release build and `cargo package` verification |
|
||||
| `packaged-cli` | the wrapped Nix binary starts, emits JSON, and generates completions |
|
||||
| `ownership-publisher`, `publisher-test` | packaged publisher entry point and its regression scenarios |
|
||||
| `install-script-sh`, `install-script-dash`, `shellcheck` | installer regressions and shell lint |
|
||||
|
||||
```bash
|
||||
scripts/live-smoke.sh
|
||||
scripts/live-smoke.sh --mode full
|
||||
scripts/live-smoke.sh --mode mock
|
||||
scripts/live-smoke.sh --group live-table --group live-json
|
||||
scripts/live-smoke.sh --group live-nonfinite
|
||||
scripts/live-smoke.sh --group cache --symbol BBRI
|
||||
scripts/live-smoke.sh --dry-run --mode full
|
||||
scripts/live-smoke.sh --bin ./tmp/release-install/bin/idx --no-build --mode mock
|
||||
scripts/audit-msn-fundamentals.sh --tickers BUMI,ADRO,AIMS
|
||||
```
|
||||
`tests/cli.rs` is the behavioral suite: it runs the real `idx` binary against
|
||||
fixture-backed mock providers (`IDX_USE_MOCK_PROVIDER=1`) and covers output
|
||||
contracts, cache/offline transitions, provider routing, error codes, ownership
|
||||
imports and queries, and snapshot sync.
|
||||
|
||||
## Nix CI Checks
|
||||
|
||||
`nix flake check -L` is the canonical local, pre-push, and Linux CI gate. Its
|
||||
eleven checks cover:
|
||||
|
||||
- Rust formatting, Clippy (`--all-targets`), and all unit/CLI tests
|
||||
- the optimized application build and verified crate packaging
|
||||
- the packaged ownership publisher's `--help` entry point
|
||||
- the offline mock CLI smoke matrix
|
||||
- publisher regression scenarios, including the SQLite artifact builder
|
||||
- installer regressions under `sh` and `dash`, plus installer and smoke-runner ShellCheck
|
||||
|
||||
All check tools are pinned by `flake.lock`. Installer fixtures use resolved
|
||||
shell paths, and generated publisher stubs use the running Bash interpreter,
|
||||
so the regression suites also work inside the Nix sandbox.
|
||||
|
||||
Crane shares dev-profile dependency artifacts between tests, Clippy, and
|
||||
`cargo package` verification; the shipped application uses a separate release
|
||||
dependency cache. Package verification still builds the packaged crate.
|
||||
Integration-test edits do not invalidate the application build, and script
|
||||
checks consume only their relevant source files. README/license edits affect
|
||||
the packaging check, not the Rust test/application derivations.
|
||||
|
||||
CI cache keys include platform, dependency/toolchain inputs, and checked source
|
||||
contents. Source changes can therefore save new outputs; restore prefixes first
|
||||
reuse the same dependency/toolchain cache, then another cache for the platform.
|
||||
The GitHub cache is opportunistic and subject to eviction, not a shared public
|
||||
binary cache. Obsolete PR runs are cancelled; main-branch runs are not.
|
||||
|
||||
Use individual checks when investigating a failure (replace the system as
|
||||
needed), or targeted Cargo commands inside the pinned development shell:
|
||||
For a focused loop, run one check or filter Cargo tests:
|
||||
|
||||
```bash
|
||||
nix build .#checks.x86_64-linux.publisher-test -L
|
||||
nix develop --command cargo test full_screener
|
||||
```
|
||||
|
||||
The native macOS installer job remains separate to exercise BSD/macOS tools.
|
||||
Live-network smoke is deliberately outside the sandboxed offline gate.
|
||||
CI cache keys include platform, dependency/toolchain inputs, and checked source
|
||||
contents. The native macOS installer job runs separately to exercise BSD tools.
|
||||
|
||||
For a live full-list screener check, use an isolated config/cache and run:
|
||||
## Live drift check
|
||||
|
||||
Fixtures cannot tell when MSN, Yahoo, or IDX change their responses.
|
||||
`scripts/live-smoke.sh` runs shipped commands against the real upstreams and
|
||||
validates each JSON output shape. It is opt-in and never part of the gate.
|
||||
|
||||
```bash
|
||||
idx -o json stocks screen --region us --filter top-performers --limit 600
|
||||
idx --offline -o json stocks screen --region us --filter top-performers --limit 3
|
||||
idx --offline -o json stocks screen --region us --filter top-performers --limit 25
|
||||
nix develop --command scripts/live-smoke.sh # stocks + ownership
|
||||
nix develop --command scripts/live-smoke.sh --group stocks --symbol BBRI
|
||||
nix develop --command scripts/live-smoke.sh --bin ./result/bin/idx
|
||||
scripts/live-smoke.sh --dry-run
|
||||
```
|
||||
|
||||
The first command should not stop at the old 500-row cap when MSN reports more
|
||||
candidates. The two offline results must match prefixes of the same cached full
|
||||
list. Provider counts and prices vary; live HTTP failures are not offline-test
|
||||
failures.
|
||||
- `stocks`: every `stocks` command for one ticker, plus `compare BUMI,ADRO,AIMS`,
|
||||
whose MSN key ratios contain non-finite values.
|
||||
- `ownership`: `discover`, then imports the newest supported report into a temp
|
||||
DB and lists `releases`. IDX sits behind Cloudflare and returns `403` to many
|
||||
datacenter IPs, including GitHub-hosted runners; run it from a network IDX accepts.
|
||||
|
||||
## Coverage Boundaries
|
||||
Each run isolates config, cache, and data under `tmp/live-smoke/<timestamp>/`
|
||||
and keeps per-case `.stdout`, `.stderr`, and `.validation` logs there.
|
||||
|
||||
The mock preset has 43 cases: 7 general, 19 stock, 9 cache, 2 routing, 4 error,
|
||||
and 2 empty/unsupported ownership checks. This is a packaged-binary safety net,
|
||||
not a claim that every important behavior is covered by shell smoke.
|
||||
|
||||
- All fifteen stock commands run in JSON mode with structural and fixture-value
|
||||
assertions. Four representative table renders check displayed data.
|
||||
- JSON checks require exactly one document in the correct stream. Failed
|
||||
commands must leave stdout empty and expose the expected stable error code.
|
||||
- Cache scenarios use zero TTL, force provider failures, and compare parsed
|
||||
offline/stale results with the warmed data; diagnostics must stay on stderr.
|
||||
- Redundant table launches, duplicate routing cases, and repeated setup checks
|
||||
are omitted rather than counted as extra correctness coverage.
|
||||
|
||||
The CLI integration suite carries deeper workflows under the same Nix gate:
|
||||
cache misses, provider isolation, filtered reports, non-finite values,
|
||||
XLSX/ZIP import and ownership queries, snapshot install/no-op/force, and rejected
|
||||
updates preserving database bytes and query results.
|
||||
|
||||
Screener regressions generate 503 distinct candidates, with ranking leaders
|
||||
beyond the first 500. They exercise top/worst performers, volume, and market-cap
|
||||
ordering, a small first request followed by complete offline reads, and rejected
|
||||
missing/incomplete totals without cache poisoning. The raw fixture override
|
||||
`IDX_MOCK_MSN_SCREENER_FIXTURE` is honored only with `IDX_USE_MOCK_PROVIDER`;
|
||||
responses honor the requested batch size so the complete-fetch path is exercised.
|
||||
|
||||
Mocks do not verify current upstream availability, authentication, or response
|
||||
drift. Live provider checks remain a separate, opt-in operation.
|
||||
|
||||
## Modes
|
||||
|
||||
- `live` runs the default real-network baseline: `general`, `live-table`, and `ownership`
|
||||
- `mock` runs the deterministic baseline: `general`, `mock`, `cache`, `routing`, `errors`, and `ownership`
|
||||
- `full` runs every group, including live JSON output checks and ownership-import verification
|
||||
|
||||
## Groups
|
||||
|
||||
- `general`: `version`, `completions`, `config`, and `cache` command basics
|
||||
- `live-table`: all shipped `stocks` commands in live table mode
|
||||
- `live-json`: all shipped `stocks` commands in live JSON mode
|
||||
- `mock`: all shipped `stocks` commands against the mock provider in JSON mode, plus representative quote/history/fundamental/financials table renders
|
||||
- `cache`: exact data equality across warm, `--offline`, and stale-cache fallback reads for quote, technical, and MSN `profile`
|
||||
- `routing`: Yahoo/MSN provider routing plus explicit MSN history behavior
|
||||
- `errors`: JSON error contract and invalid flag/input checks
|
||||
- `live-nonfinite`: opt-in live MSN fundamentals checks for known non-finite ticker payloads (`BUMI`, `ADRO`, `AIMS`)
|
||||
- `ownership`: safe ownership smoke checks that do not require imported ownership data
|
||||
- `ownership-import`: live discovery/import hardening checks for supported `above1` import plus expected unsupported legacy-family failures
|
||||
|
||||
## Notes
|
||||
|
||||
- The runner forces `IDX_OUTPUT=table` so table cases stay stable; JSON checks pass `-o json` explicitly.
|
||||
- Cache-group warm cases clear the smoke cache first so warm/offline/stale sequences start clean.
|
||||
- Use `--bin <path> --no-build` to validate an installed binary instead of `target/debug/idx`.
|
||||
- `ownership-import` and `live-nonfinite` are opt-in (explicit `--group` or `--mode full`). Known non-finite repro tickers as of `2026-04-13`: `BUMI`, `ADRO`, `AIMS`.
|
||||
- The baseline `ownership` group needs no imported data; `ownership sync` is covered by fixture-backed CLI tests instead.
|
||||
- `scripts/audit-msn-fundamentals.sh` is a heavier provider-health sweep over the IDX MSN symbol map, not a baseline check.
|
||||
- When a case fails, inspect its `.stdout`, `.stderr`, and `.validation` files under `tmp/live-smoke/.../logs/`.
|
||||
`scripts/audit-msn-fundamentals.sh` is a heavier sweep of valuation output
|
||||
across the whole IDX MSN symbol map, for provider-health audits.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue