mirror of
https://github.com/0xrsydn/idx-cli.git
synced 2026-11-07 09:45:36 +00:00
test: replace mock smoke matrix with a live-only drift check
The 43-case mock smoke duplicated tests/cli.rs, which already runs the real binary against fixture providers. Replace the flake's smoke-mock check with a small packaged-cli check (version JSON, bash completions, one mock quote), and cut scripts/live-smoke.sh from 672 to ~170 lines: it now only runs shipped commands against live MSN/Yahoo/IDX and validates output shape. Rewrite docs/SMOKE.md around the Nix gate and the drift check. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
hermes
parent
1631058b8c
commit
86c108f6eb
7 changed files with 188 additions and 771 deletions
|
|
@ -60,13 +60,13 @@ curl -fsSL https://crates.io/api/v1/crates/idx-cli
|
||||||
For a real release or release-hygiene fix, verify the repo state first.
|
For a real release or release-hygiene fix, verify the repo state first.
|
||||||
|
|
||||||
Run the canonical gate (fmt, clippy, tests, release build, `cargo package`
|
Run the canonical gate (fmt, clippy, tests, release build, `cargo package`
|
||||||
verification, and mock smoke):
|
verification, and a packaged-binary check):
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
nix flake check -L
|
nix flake check -L
|
||||||
```
|
```
|
||||||
|
|
||||||
Run targeted smoke when the release changes user-facing CLI behavior. For
|
Run `scripts/live-smoke.sh` when the release changes provider-facing behavior. For
|
||||||
ownership-sync changes, re-verify a clean-machine flow:
|
ownership-sync changes, re-verify a clean-machine flow:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
|
|
|
||||||
|
|
@ -21,7 +21,7 @@ Docs:
|
||||||
- `docs/ARCHITECTURE.md`: providers, ownership ingest/discovery, error and output contract
|
- `docs/ARCHITECTURE.md`: providers, ownership ingest/discovery, error and output contract
|
||||||
- `docs/CONVENTIONS.md`: types, naming, testing, and git rules
|
- `docs/CONVENTIONS.md`: types, naming, testing, and git rules
|
||||||
- `docs/OWNERSHIP_SYNC.md` / `docs/OWNERSHIP_PUBLISH.md`: snapshot consumer contract / maintainer publishing
|
- `docs/OWNERSHIP_SYNC.md` / `docs/OWNERSHIP_PUBLISH.md`: snapshot consumer contract / maintainer publishing
|
||||||
- `docs/SMOKE.md`: smoke runner and Nix check coverage
|
- `docs/SMOKE.md`: Nix checks, test layout, and the live drift check
|
||||||
- `ROADMAP.md`: open work
|
- `ROADMAP.md`: open work
|
||||||
|
|
||||||
## Rules
|
## Rules
|
||||||
|
|
@ -36,7 +36,7 @@ Docs:
|
||||||
|
|
||||||
## Verify
|
## Verify
|
||||||
```bash
|
```bash
|
||||||
nix flake check -L # canonical gate (fmt, clippy, tests, package, smoke)
|
nix flake check -L # canonical gate (fmt, clippy, tests, package)
|
||||||
nix develop --command cargo test <name> # focused loop
|
nix develop --command cargo test <name> # focused loop
|
||||||
scripts/live-smoke.sh --mode full # opt-in live providers
|
scripts/live-smoke.sh # opt-in live provider drift check
|
||||||
```
|
```
|
||||||
|
|
|
||||||
18
README.md
18
README.md
|
|
@ -146,7 +146,7 @@ Precedence order:
|
||||||
- [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md): providers, ownership ingest, and discovery
|
- [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md): providers, ownership ingest, and discovery
|
||||||
- [docs/OWNERSHIP_SYNC.md](docs/OWNERSHIP_SYNC.md): snapshot manifest and sync rules
|
- [docs/OWNERSHIP_SYNC.md](docs/OWNERSHIP_SYNC.md): snapshot manifest and sync rules
|
||||||
- [docs/OWNERSHIP_PUBLISH.md](docs/OWNERSHIP_PUBLISH.md): maintainer snapshot publishing
|
- [docs/OWNERSHIP_PUBLISH.md](docs/OWNERSHIP_PUBLISH.md): maintainer snapshot publishing
|
||||||
- [docs/SMOKE.md](docs/SMOKE.md): smoke tests against live providers
|
- [docs/SMOKE.md](docs/SMOKE.md): test layout and the live provider drift check
|
||||||
- [ROADMAP.md](ROADMAP.md): planned work
|
- [ROADMAP.md](ROADMAP.md): planned work
|
||||||
|
|
||||||
## Development
|
## Development
|
||||||
|
|
@ -157,10 +157,10 @@ The canonical local and CI verification command is:
|
||||||
nix flake check -L
|
nix flake check -L
|
||||||
```
|
```
|
||||||
|
|
||||||
It covers Rust formatting, Clippy, tests, crate package verification, release
|
It covers Rust formatting, Clippy, unit and CLI integration tests, crate
|
||||||
builds, offline CLI smoke, publisher regressions, installer tests with `sh` and
|
package verification, the packaged binary, publisher regressions, installer
|
||||||
`dash`, and ShellCheck for installer/smoke scripts. The checks use pinned tools and cached Nix
|
tests with `sh` and `dash`, and ShellCheck. The checks use pinned tools and
|
||||||
outputs; an unchanged successful check is reused rather than rerun.
|
cached Nix outputs; an unchanged successful check is reused rather than rerun.
|
||||||
|
|
||||||
For a focused edit/debug loop, use the same pinned toolchain:
|
For a focused edit/debug loop, use the same pinned toolchain:
|
||||||
|
|
||||||
|
|
@ -172,10 +172,10 @@ Configured Prek hooks keep whitespace/conflict checks at pre-commit and run
|
||||||
`nix flake check -L` at pre-push. There is no separate full Cargo test/lint gate
|
`nix flake check -L` at pre-push. There is no separate full Cargo test/lint gate
|
||||||
to repeat after a successful flake check.
|
to repeat after a successful flake check.
|
||||||
|
|
||||||
CI additionally checks the installer with native macOS tools for portability.
|
CI additionally checks the installer with native macOS tools. The live
|
||||||
Live-provider smoke remains opt-in; see [docs/SMOKE.md](docs/SMOKE.md) for check
|
provider drift check (`scripts/live-smoke.sh`) is opt-in; see
|
||||||
selection and caching details. The npm distribution harness remains available
|
[docs/SMOKE.md](docs/SMOKE.md). The npm wrapper has its own harness,
|
||||||
separately as `scripts/npm-smoke.sh`.
|
`scripts/npm-smoke.sh`.
|
||||||
|
|
||||||
## License
|
## License
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -119,7 +119,7 @@ let conn = Connection::open(path)
|
||||||
- **Unit tests:** retain precise parser, normalization, calculation, and boundary regressions; avoid pinning incidental wording, enum counts, or internal wiring.
|
- **Unit tests:** retain precise parser, normalization, calculation, and boundary regressions; avoid pinning incidental wording, enum counts, or internal wiring.
|
||||||
- **Integration tests:** prefer CLI workflows, real temporary/in-memory SQLite, and deterministic local provider fixtures for user-visible contracts and state transitions.
|
- **Integration tests:** prefer CLI workflows, real temporary/in-memory SQLite, and deterministic local provider fixtures for user-visible contracts and state transitions.
|
||||||
- **Output contracts:** parse JSON from stdout on success and stderr on failure; errors must leave stdout empty. Assert meaningful values, types, and stable error codes rather than substrings or diagnostic wording.
|
- **Output contracts:** parse JSON from stdout on success and stderr on failure; errors must leave stdout empty. Assert meaningful values, types, and stable error codes rather than substrings or diagnostic wording.
|
||||||
- **Stateful workflows:** compare cached data across offline/failure transitions, and verify failed snapshot updates preserve both database bytes and query results. Keep packaged-binary smoke representative; put exhaustive boundaries in integration tests.
|
- **Stateful workflows:** compare cached data across offline/failure transitions, and verify failed snapshot updates preserve both database bytes and query results. Put behavioral coverage in `tests/cli.rs`, not shell smoke.
|
||||||
- **Fixtures:** `tests/fixtures/*.json` — real API responses, sanitized
|
- **Fixtures:** `tests/fixtures/*.json` — real API responses, sanitized
|
||||||
- **No live API calls in CI** — `IDX_USE_MOCK_PROVIDER=1`
|
- **No live API calls in CI** — `IDX_USE_MOCK_PROVIDER=1`
|
||||||
- **Test naming:** `test_<function>_<scenario>` (e.g., `test_parse_id_number_with_dots`)
|
- **Test naming:** `test_<function>_<scenario>` (e.g., `test_parse_id_number_with_dots`)
|
||||||
|
|
|
||||||
154
docs/SMOKE.md
154
docs/SMOKE.md
|
|
@ -1,135 +1,55 @@
|
||||||
# Smoke Checks
|
# Testing and Smoke Checks
|
||||||
|
|
||||||
Use `scripts/live-smoke.sh` as the reusable smoke runner for shipped CLI surfaces.
|
## Automated gate
|
||||||
|
|
||||||
The script:
|
`nix flake check -L` is the canonical local, pre-push, and CI gate. It runs
|
||||||
- builds `target/debug/idx` once by default
|
offline with pinned tools:
|
||||||
- isolates config, cache, and data under `tmp/live-smoke/<timestamp>/`
|
|
||||||
- records separate `.stdout` and `.stderr` files per case under `tmp/live-smoke/<timestamp>/logs/`, plus `.validation` diagnostics for JSON checks
|
|
||||||
- supports fast mock-backed runs and slower live-network passes; `jq` is required and provided by `nix develop`
|
|
||||||
|
|
||||||
## Common Runs
|
| Check | Covers |
|
||||||
|
| --- | --- |
|
||||||
|
| `fmt`, `clippy` | formatting and lints (`--all-targets`) |
|
||||||
|
| `test` | unit/module tests and `tests/cli.rs` CLI integration tests |
|
||||||
|
| `default`, `package` | release build and `cargo package` verification |
|
||||||
|
| `packaged-cli` | the wrapped Nix binary starts, emits JSON, and generates completions |
|
||||||
|
| `ownership-publisher`, `publisher-test` | packaged publisher entry point and its regression scenarios |
|
||||||
|
| `install-script-sh`, `install-script-dash`, `shellcheck` | installer regressions and shell lint |
|
||||||
|
|
||||||
```bash
|
`tests/cli.rs` is the behavioral suite: it runs the real `idx` binary against
|
||||||
scripts/live-smoke.sh
|
fixture-backed mock providers (`IDX_USE_MOCK_PROVIDER=1`) and covers output
|
||||||
scripts/live-smoke.sh --mode full
|
contracts, cache/offline transitions, provider routing, error codes, ownership
|
||||||
scripts/live-smoke.sh --mode mock
|
imports and queries, and snapshot sync.
|
||||||
scripts/live-smoke.sh --group live-table --group live-json
|
|
||||||
scripts/live-smoke.sh --group live-nonfinite
|
|
||||||
scripts/live-smoke.sh --group cache --symbol BBRI
|
|
||||||
scripts/live-smoke.sh --dry-run --mode full
|
|
||||||
scripts/live-smoke.sh --bin ./tmp/release-install/bin/idx --no-build --mode mock
|
|
||||||
scripts/audit-msn-fundamentals.sh --tickers BUMI,ADRO,AIMS
|
|
||||||
```
|
|
||||||
|
|
||||||
## Nix CI Checks
|
For a focused loop, run one check or filter Cargo tests:
|
||||||
|
|
||||||
`nix flake check -L` is the canonical local, pre-push, and Linux CI gate. Its
|
|
||||||
eleven checks cover:
|
|
||||||
|
|
||||||
- Rust formatting, Clippy (`--all-targets`), and all unit/CLI tests
|
|
||||||
- the optimized application build and verified crate packaging
|
|
||||||
- the packaged ownership publisher's `--help` entry point
|
|
||||||
- the offline mock CLI smoke matrix
|
|
||||||
- publisher regression scenarios, including the SQLite artifact builder
|
|
||||||
- installer regressions under `sh` and `dash`, plus installer and smoke-runner ShellCheck
|
|
||||||
|
|
||||||
All check tools are pinned by `flake.lock`. Installer fixtures use resolved
|
|
||||||
shell paths, and generated publisher stubs use the running Bash interpreter,
|
|
||||||
so the regression suites also work inside the Nix sandbox.
|
|
||||||
|
|
||||||
Crane shares dev-profile dependency artifacts between tests, Clippy, and
|
|
||||||
`cargo package` verification; the shipped application uses a separate release
|
|
||||||
dependency cache. Package verification still builds the packaged crate.
|
|
||||||
Integration-test edits do not invalidate the application build, and script
|
|
||||||
checks consume only their relevant source files. README/license edits affect
|
|
||||||
the packaging check, not the Rust test/application derivations.
|
|
||||||
|
|
||||||
CI cache keys include platform, dependency/toolchain inputs, and checked source
|
|
||||||
contents. Source changes can therefore save new outputs; restore prefixes first
|
|
||||||
reuse the same dependency/toolchain cache, then another cache for the platform.
|
|
||||||
The GitHub cache is opportunistic and subject to eviction, not a shared public
|
|
||||||
binary cache. Obsolete PR runs are cancelled; main-branch runs are not.
|
|
||||||
|
|
||||||
Use individual checks when investigating a failure (replace the system as
|
|
||||||
needed), or targeted Cargo commands inside the pinned development shell:
|
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
nix build .#checks.x86_64-linux.publisher-test -L
|
nix build .#checks.x86_64-linux.publisher-test -L
|
||||||
nix develop --command cargo test full_screener
|
nix develop --command cargo test full_screener
|
||||||
```
|
```
|
||||||
|
|
||||||
The native macOS installer job remains separate to exercise BSD/macOS tools.
|
CI cache keys include platform, dependency/toolchain inputs, and checked source
|
||||||
Live-network smoke is deliberately outside the sandboxed offline gate.
|
contents. The native macOS installer job runs separately to exercise BSD tools.
|
||||||
|
|
||||||
For a live full-list screener check, use an isolated config/cache and run:
|
## Live drift check
|
||||||
|
|
||||||
|
Fixtures cannot tell when MSN, Yahoo, or IDX change their responses.
|
||||||
|
`scripts/live-smoke.sh` runs shipped commands against the real upstreams and
|
||||||
|
validates each JSON output shape. It is opt-in and never part of the gate.
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
idx -o json stocks screen --region us --filter top-performers --limit 600
|
nix develop --command scripts/live-smoke.sh # stocks + ownership
|
||||||
idx --offline -o json stocks screen --region us --filter top-performers --limit 3
|
nix develop --command scripts/live-smoke.sh --group stocks --symbol BBRI
|
||||||
idx --offline -o json stocks screen --region us --filter top-performers --limit 25
|
nix develop --command scripts/live-smoke.sh --bin ./result/bin/idx
|
||||||
|
scripts/live-smoke.sh --dry-run
|
||||||
```
|
```
|
||||||
|
|
||||||
The first command should not stop at the old 500-row cap when MSN reports more
|
- `stocks`: every `stocks` command for one ticker, plus `compare BUMI,ADRO,AIMS`,
|
||||||
candidates. The two offline results must match prefixes of the same cached full
|
whose MSN key ratios contain non-finite values.
|
||||||
list. Provider counts and prices vary; live HTTP failures are not offline-test
|
- `ownership`: `discover`, then imports the newest supported report into a temp
|
||||||
failures.
|
DB and lists `releases`. IDX sits behind Cloudflare and returns `403` to many
|
||||||
|
datacenter IPs, including GitHub-hosted runners; run it from a network IDX accepts.
|
||||||
|
|
||||||
## Coverage Boundaries
|
Each run isolates config, cache, and data under `tmp/live-smoke/<timestamp>/`
|
||||||
|
and keeps per-case `.stdout`, `.stderr`, and `.validation` logs there.
|
||||||
|
|
||||||
The mock preset has 43 cases: 7 general, 19 stock, 9 cache, 2 routing, 4 error,
|
`scripts/audit-msn-fundamentals.sh` is a heavier sweep of valuation output
|
||||||
and 2 empty/unsupported ownership checks. This is a packaged-binary safety net,
|
across the whole IDX MSN symbol map, for provider-health audits.
|
||||||
not a claim that every important behavior is covered by shell smoke.
|
|
||||||
|
|
||||||
- All fifteen stock commands run in JSON mode with structural and fixture-value
|
|
||||||
assertions. Four representative table renders check displayed data.
|
|
||||||
- JSON checks require exactly one document in the correct stream. Failed
|
|
||||||
commands must leave stdout empty and expose the expected stable error code.
|
|
||||||
- Cache scenarios use zero TTL, force provider failures, and compare parsed
|
|
||||||
offline/stale results with the warmed data; diagnostics must stay on stderr.
|
|
||||||
- Redundant table launches, duplicate routing cases, and repeated setup checks
|
|
||||||
are omitted rather than counted as extra correctness coverage.
|
|
||||||
|
|
||||||
The CLI integration suite carries deeper workflows under the same Nix gate:
|
|
||||||
cache misses, provider isolation, filtered reports, non-finite values,
|
|
||||||
XLSX/ZIP import and ownership queries, snapshot install/no-op/force, and rejected
|
|
||||||
updates preserving database bytes and query results.
|
|
||||||
|
|
||||||
Screener regressions generate 503 distinct candidates, with ranking leaders
|
|
||||||
beyond the first 500. They exercise top/worst performers, volume, and market-cap
|
|
||||||
ordering, a small first request followed by complete offline reads, and rejected
|
|
||||||
missing/incomplete totals without cache poisoning. The raw fixture override
|
|
||||||
`IDX_MOCK_MSN_SCREENER_FIXTURE` is honored only with `IDX_USE_MOCK_PROVIDER`;
|
|
||||||
responses honor the requested batch size so the complete-fetch path is exercised.
|
|
||||||
|
|
||||||
Mocks do not verify current upstream availability, authentication, or response
|
|
||||||
drift. Live provider checks remain a separate, opt-in operation.
|
|
||||||
|
|
||||||
## Modes
|
|
||||||
|
|
||||||
- `live` runs the default real-network baseline: `general`, `live-table`, and `ownership`
|
|
||||||
- `mock` runs the deterministic baseline: `general`, `mock`, `cache`, `routing`, `errors`, and `ownership`
|
|
||||||
- `full` runs every group, including live JSON output checks and ownership-import verification
|
|
||||||
|
|
||||||
## Groups
|
|
||||||
|
|
||||||
- `general`: `version`, `completions`, `config`, and `cache` command basics
|
|
||||||
- `live-table`: all shipped `stocks` commands in live table mode
|
|
||||||
- `live-json`: all shipped `stocks` commands in live JSON mode
|
|
||||||
- `mock`: all shipped `stocks` commands against the mock provider in JSON mode, plus representative quote/history/fundamental/financials table renders
|
|
||||||
- `cache`: exact data equality across warm, `--offline`, and stale-cache fallback reads for quote, technical, and MSN `profile`
|
|
||||||
- `routing`: Yahoo/MSN provider routing plus explicit MSN history behavior
|
|
||||||
- `errors`: JSON error contract and invalid flag/input checks
|
|
||||||
- `live-nonfinite`: opt-in live MSN fundamentals checks for known non-finite ticker payloads (`BUMI`, `ADRO`, `AIMS`)
|
|
||||||
- `ownership`: safe ownership smoke checks that do not require imported ownership data
|
|
||||||
- `ownership-import`: live discovery/import hardening checks for supported `above1` import plus expected unsupported legacy-family failures
|
|
||||||
|
|
||||||
## Notes
|
|
||||||
|
|
||||||
- The runner forces `IDX_OUTPUT=table` so table cases stay stable; JSON checks pass `-o json` explicitly.
|
|
||||||
- Cache-group warm cases clear the smoke cache first so warm/offline/stale sequences start clean.
|
|
||||||
- Use `--bin <path> --no-build` to validate an installed binary instead of `target/debug/idx`.
|
|
||||||
- `ownership-import` and `live-nonfinite` are opt-in (explicit `--group` or `--mode full`). Known non-finite repro tickers as of `2026-04-13`: `BUMI`, `ADRO`, `AIMS`.
|
|
||||||
- The baseline `ownership` group needs no imported data; `ownership sync` is covered by fixture-backed CLI tests instead.
|
|
||||||
- `scripts/audit-msn-fundamentals.sh` is a heavier provider-health sweep over the IDX MSN symbol map, not a baseline check.
|
|
||||||
- When a case fails, inspect its `.stdout`, `.stderr`, and `.validation` files under `tmp/live-smoke/.../logs/`.
|
|
||||||
|
|
|
||||||
16
flake.nix
16
flake.nix
|
|
@ -189,19 +189,19 @@
|
||||||
doInstallCargoArtifacts = false;
|
doInstallCargoArtifacts = false;
|
||||||
installPhaseCommand = "touch $out";
|
installPhaseCommand = "touch $out";
|
||||||
});
|
});
|
||||||
smoke-mock = pkgs.runCommand "idx-smoke-mock"
|
# Sanity check of the packaged (wrapped) binary; CLI behavior is
|
||||||
{ nativeBuildInputs = with pkgs; [ bash coreutils diffutils findutils gnugrep gnused gawk jq ]; }
|
# covered by tests/cli.rs.
|
||||||
|
packaged-cli = pkgs.runCommand "idx-packaged-cli"
|
||||||
|
{ nativeBuildInputs = with pkgs; [ bash jq ]; }
|
||||||
''
|
''
|
||||||
mkdir -p scripts
|
|
||||||
cp ${./scripts/live-smoke.sh} scripts/live-smoke.sh
|
|
||||||
chmod u+w scripts/live-smoke.sh
|
|
||||||
patchShebangs scripts/live-smoke.sh
|
|
||||||
# The mock provider reads fixtures relative to the working dir.
|
# The mock provider reads fixtures relative to the working dir.
|
||||||
mkdir -p tests
|
mkdir -p tests
|
||||||
cp -r ${./tests/fixtures} tests/fixtures
|
cp -r ${./tests/fixtures} tests/fixtures
|
||||||
export HOME="$TMPDIR"
|
export HOME="$TMPDIR"
|
||||||
${idxPackage}/bin/idx version
|
${idxPackage}/bin/idx -o json version | jq -e '.version | type == "string"'
|
||||||
bash scripts/live-smoke.sh --bin ${idxPackage}/bin/idx --no-build --mode mock
|
${idxPackage}/bin/idx completions bash | bash -n
|
||||||
|
IDX_USE_MOCK_PROVIDER=1 ${idxPackage}/bin/idx -o json stocks quote BBCA \
|
||||||
|
| jq -e '.[0].price | type == "number"'
|
||||||
touch "$out"
|
touch "$out"
|
||||||
'';
|
'';
|
||||||
install-script-sh = installerCheck "sh";
|
install-script-sh = installerCheck "sh";
|
||||||
|
|
|
||||||
|
|
@ -1,672 +1,169 @@
|
||||||
#!/usr/bin/env bash
|
#!/usr/bin/env bash
|
||||||
|
# Live provider drift check: runs shipped commands against real MSN, Yahoo,
|
||||||
|
# and IDX endpoints and validates the JSON output shape. Deterministic CLI
|
||||||
|
# behavior is covered by tests/cli.rs; this only answers "do the upstreams
|
||||||
|
# still look the way our parsers expect?"
|
||||||
|
|
||||||
set -u
|
set -u
|
||||||
set -o pipefail
|
set -o pipefail
|
||||||
|
|
||||||
ROOT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
ROOT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||||
MODE="live"
|
|
||||||
BIN_PATH=""
|
|
||||||
WORKDIR=""
|
|
||||||
SYMBOL="BBCA"
|
SYMBOL="BBCA"
|
||||||
COMPARE_SYMBOLS="BBCA,BBRI"
|
BIN_PATH=""
|
||||||
NEWS_LIMIT="5"
|
|
||||||
SCREEN_LIMIT="10"
|
|
||||||
DRY_RUN=0
|
DRY_RUN=0
|
||||||
BUILD=1
|
declare -a GROUPS_SELECTED=()
|
||||||
|
|
||||||
declare -a GROUP_FILTERS=()
|
|
||||||
declare -a CASES=()
|
|
||||||
declare -a FAILURES=()
|
declare -a FAILURES=()
|
||||||
|
TOTAL=0
|
||||||
TOTAL_CASES=0
|
PASSED=0
|
||||||
PASSED_CASES=0
|
|
||||||
FAILED_CASES=0
|
|
||||||
SEP=$'\x1f'
|
|
||||||
|
|
||||||
CONFIG_HOME=""
|
|
||||||
CACHE_HOME=""
|
|
||||||
DATA_HOME=""
|
|
||||||
OWNERSHIP_DB=""
|
|
||||||
LOG_DIR=""
|
|
||||||
OWNERSHIP_ABOVE1_URL=""
|
|
||||||
OWNERSHIP_ABOVE5_URL=""
|
|
||||||
OWNERSHIP_INVESTOR_TYPE_URL=""
|
|
||||||
|
|
||||||
declare -a BASE_ENV=()
|
|
||||||
|
|
||||||
usage() {
|
usage() {
|
||||||
cat <<'EOF'
|
cat <<'EOF'
|
||||||
Usage: scripts/live-smoke.sh [options]
|
Usage: scripts/live-smoke.sh [options]
|
||||||
|
|
||||||
Reusable smoke runner for shipped idx-cli command surfaces.
|
|
||||||
|
|
||||||
Options:
|
Options:
|
||||||
--mode <live|mock|full> Preset case selection. Default: live
|
--group <stocks|ownership> Restrict to a group. Repeatable. Default: both
|
||||||
--group <name> Restrict to one or more groups. Repeatable.
|
--symbol <ticker> Primary ticker. Default: BBCA
|
||||||
--symbol <ticker> Primary ticker for stock cases. Default: BBCA
|
--bin <path> Use an existing idx binary (skips cargo build)
|
||||||
--compare-symbols <list> Compare symbols. Default: BBCA,BBRI
|
--dry-run Print cases without running them
|
||||||
--news-limit <n> News limit. Default: 5
|
--help Show this help
|
||||||
--screen-limit <n> Screener limit. Default: 10
|
|
||||||
--workdir <path> Artifact root. Default: tmp/live-smoke/<timestamp>
|
|
||||||
--bin <path> Use an existing idx binary instead of target/debug/idx
|
|
||||||
--no-build Skip cargo build
|
|
||||||
--dry-run Print selected cases without running them
|
|
||||||
--help Show this help
|
|
||||||
|
|
||||||
Groups:
|
|
||||||
general version/completions/config/cache basics
|
|
||||||
live-table live network stock commands in table mode
|
|
||||||
live-json live network stock commands in JSON mode
|
|
||||||
mock mock-backed stock JSON contracts + representative table checks
|
|
||||||
cache deterministic cache/offline/stale-cache checks
|
|
||||||
routing provider routing and explicit unsupported checks
|
|
||||||
errors JSON error contract and invalid-flag checks
|
|
||||||
live-nonfinite opt-in live MSN fundamentals checks for known non-finite tickers
|
|
||||||
ownership ownership commands that are safe without imported data
|
|
||||||
ownership-import live ownership discovery/import hardening checks
|
|
||||||
|
|
||||||
Examples:
|
|
||||||
scripts/live-smoke.sh
|
|
||||||
scripts/live-smoke.sh --mode full
|
|
||||||
scripts/live-smoke.sh --mode mock --group cache
|
|
||||||
scripts/live-smoke.sh --group live-table --group live-json --symbol BBRI
|
|
||||||
scripts/live-smoke.sh --group live-nonfinite
|
|
||||||
EOF
|
EOF
|
||||||
}
|
}
|
||||||
|
|
||||||
add_group_filters() {
|
while [[ $# -gt 0 ]]; do
|
||||||
local raw="$1"
|
case "$1" in
|
||||||
local part
|
--group) GROUPS_SELECTED+=("${2:-}"); shift 2 ;;
|
||||||
local old_ifs="$IFS"
|
--symbol) SYMBOL="${2:-}"; shift 2 ;;
|
||||||
IFS=','
|
--bin) BIN_PATH="${2:-}"; shift 2 ;;
|
||||||
for part in $raw; do
|
--dry-run) DRY_RUN=1; shift ;;
|
||||||
if [[ -n "$part" ]]; then
|
--help|-h) usage; exit 0 ;;
|
||||||
GROUP_FILTERS+=("$part")
|
*) echo "unknown argument: $1" >&2; usage >&2; exit 2 ;;
|
||||||
fi
|
|
||||||
done
|
|
||||||
IFS="$old_ifs"
|
|
||||||
}
|
|
||||||
|
|
||||||
mode_allows_group() {
|
|
||||||
local group="$1"
|
|
||||||
case "$MODE" in
|
|
||||||
live)
|
|
||||||
case "$group" in
|
|
||||||
general|live-table|ownership) return 0 ;;
|
|
||||||
*) return 1 ;;
|
|
||||||
esac
|
|
||||||
;;
|
|
||||||
mock)
|
|
||||||
case "$group" in
|
|
||||||
general|mock|cache|routing|errors|ownership) return 0 ;;
|
|
||||||
*) return 1 ;;
|
|
||||||
esac
|
|
||||||
;;
|
|
||||||
full)
|
|
||||||
return 0
|
|
||||||
;;
|
|
||||||
*)
|
|
||||||
return 1
|
|
||||||
;;
|
|
||||||
esac
|
esac
|
||||||
}
|
done
|
||||||
|
[[ ${#GROUPS_SELECTED[@]} -eq 0 ]] && GROUPS_SELECTED=(stocks ownership)
|
||||||
|
|
||||||
group_requested() {
|
for group in "${GROUPS_SELECTED[@]}"; do
|
||||||
local group="$1"
|
case "$group" in
|
||||||
local selected
|
stocks|ownership) ;;
|
||||||
|
*) echo "invalid --group '$group' (expected stocks or ownership)" >&2; exit 2 ;;
|
||||||
|
esac
|
||||||
|
done
|
||||||
|
|
||||||
if [[ ${#GROUP_FILTERS[@]} -eq 0 ]]; then
|
selected() {
|
||||||
return 0
|
local group
|
||||||
fi
|
for group in "${GROUPS_SELECTED[@]}"; do
|
||||||
|
[[ "$group" == "$1" ]] && return 0
|
||||||
for selected in "${GROUP_FILTERS[@]}"; do
|
|
||||||
if [[ "$selected" == "$group" ]]; then
|
|
||||||
return 0
|
|
||||||
fi
|
|
||||||
done
|
done
|
||||||
|
|
||||||
return 1
|
return 1
|
||||||
}
|
}
|
||||||
|
|
||||||
should_run_group() {
|
WORKDIR="$ROOT_DIR/tmp/live-smoke/$(date +%Y%m%d-%H%M%S)"
|
||||||
local group="$1"
|
LOG_DIR="$WORKDIR/logs"
|
||||||
if [[ ${#GROUP_FILTERS[@]} -gt 0 ]]; then
|
BASE_ENV=(
|
||||||
group_requested "$group"
|
"XDG_CONFIG_HOME=$WORKDIR/config"
|
||||||
return
|
"XDG_CACHE_HOME=$WORKDIR/cache"
|
||||||
fi
|
"XDG_DATA_HOME=$WORKDIR/data"
|
||||||
|
"IDX_PROVIDER=msn"
|
||||||
|
"IDX_HISTORY_PROVIDER=auto"
|
||||||
|
"IDX_NO_COLOR=1"
|
||||||
|
)
|
||||||
|
|
||||||
mode_allows_group "$group"
|
# jq filter describing the JSON shape each command must keep.
|
||||||
}
|
contract() {
|
||||||
|
case "$1" in
|
||||||
add_case() {
|
"stocks quote "*|"stocks screen"*)
|
||||||
CASES+=("$1$SEP$2$SEP$3$SEP$4$SEP$5$SEP$6$SEP${7:-}")
|
echo 'type == "array" and length > 0 and all(.[]; (.symbol | type == "string") and (.price | type == "number"))' ;;
|
||||||
}
|
"stocks history "*)
|
||||||
|
echo 'type == "array" and length > 0 and all(.[]; (.date | type == "string") and (.close | type == "number"))' ;;
|
||||||
register_cases() {
|
"stocks technical "*)
|
||||||
local live_env="IDX_PROVIDER=msn IDX_HISTORY_PROVIDER=auto"
|
echo '(.current_price | type == "number") and (.signals.overall | type == "string")' ;;
|
||||||
local mock_env="IDX_PROVIDER=msn IDX_HISTORY_PROVIDER=auto IDX_USE_MOCK_PROVIDER=1"
|
"stocks growth "*|"stocks valuation "*|"stocks risk "*)
|
||||||
local yahoo_mock_env="IDX_PROVIDER=yahoo IDX_HISTORY_PROVIDER=auto IDX_USE_MOCK_PROVIDER=1"
|
echo '.overall_signal | type == "string"' ;;
|
||||||
local quote_zero_ttl_env="IDX_PROVIDER=msn IDX_USE_MOCK_PROVIDER=1 IDX_CACHE_QUOTE_TTL=0"
|
"stocks fundamental "*)
|
||||||
local profile_zero_ttl_env="IDX_PROVIDER=msn IDX_USE_MOCK_PROVIDER=1 IDX_CACHE_FUNDAMENTAL_TTL=0"
|
echo '(.growth.overall_signal | type == "string") and (.valuation.overall_signal | type == "string") and (.risk.overall_signal | type == "string")' ;;
|
||||||
local live_symbol="$SYMBOL"
|
"stocks compare "*)
|
||||||
local compare_args="$COMPARE_SYMBOLS"
|
echo 'type == "array" and length > 0 and all(.[]; (.symbol | type == "string") and (.valuation | type == "object"))' ;;
|
||||||
local history_args="stocks history $live_symbol --period 3mo"
|
"stocks profile "*)
|
||||||
local screen_args="stocks screen --filter top-performers --limit $SCREEN_LIMIT"
|
echo '(.symbol | type == "string") and has("long_name") and (.officers | type == "array")' ;;
|
||||||
local news_args="stocks news $live_symbol --limit $NEWS_LIMIT"
|
"stocks financials "*)
|
||||||
|
echo '(.instrument.symbol | type == "string") and has("income_statement") and has("balance_sheet") and has("cash_flow")' ;;
|
||||||
add_case "general" "version" "0" "" "-o json version" "" '.version | type == "string" and test("^[0-9]+\\.[0-9]+\\.[0-9]+")'
|
"stocks earnings "*)
|
||||||
add_case "general" "completions-bash" "0" "" "completions bash" ""
|
echo '(.history | type == "array") and (.forecast | type == "array")' ;;
|
||||||
add_case "general" "config-set-output-json" "0" "" "config set general.output json" ""
|
"stocks sentiment "*)
|
||||||
add_case "general" "config-get-output" "0" "" "config get general.output" "json"
|
echo '.statistics | type == "array"' ;;
|
||||||
add_case "general" "config-set-output-table" "0" "" "config set general.output table" ""
|
"stocks insights "*)
|
||||||
add_case "general" "cache-info" "0" "" "-o json cache info" "" '.files | type == "number" and . >= 0'
|
echo '(.highlights | type == "array") and (.risks | type == "array")' ;;
|
||||||
add_case "general" "cache-clear" "0" "" "-o json cache clear" "" '.removed | type == "number" and . >= 0'
|
"stocks news "*)
|
||||||
|
echo 'type == "array" and all(.[]; (.title | type == "string") and (.url | type == "string"))' ;;
|
||||||
add_case "live-table" "quote" "0" "$live_env" "stocks quote $live_symbol" ""
|
"ownership discover"*)
|
||||||
add_case "live-table" "history" "0" "$live_env" "$history_args" ""
|
echo 'type == "array" and length > 0 and (.[0].pdf_url | type == "string")' ;;
|
||||||
add_case "live-table" "technical" "0" "$live_env" "stocks technical $live_symbol" ""
|
"ownership releases")
|
||||||
add_case "live-table" "growth" "0" "$live_env" "stocks growth $live_symbol" ""
|
echo 'type == "array" and length == 1' ;;
|
||||||
add_case "live-table" "valuation" "0" "$live_env" "stocks valuation $live_symbol" ""
|
|
||||||
add_case "live-table" "risk" "0" "$live_env" "stocks risk $live_symbol" ""
|
|
||||||
add_case "live-table" "fundamental" "0" "$live_env" "stocks fundamental $live_symbol" ""
|
|
||||||
add_case "live-table" "compare" "0" "$live_env" "stocks compare $compare_args" ""
|
|
||||||
add_case "live-table" "profile" "0" "$live_env" "stocks profile $live_symbol" ""
|
|
||||||
add_case "live-table" "financials" "0" "$live_env" "stocks financials $live_symbol" ""
|
|
||||||
add_case "live-table" "earnings" "0" "$live_env" "stocks earnings $live_symbol" ""
|
|
||||||
add_case "live-table" "sentiment" "0" "$live_env" "stocks sentiment $live_symbol" ""
|
|
||||||
add_case "live-table" "insights" "0" "$live_env" "stocks insights $live_symbol" ""
|
|
||||||
add_case "live-table" "news" "0" "$live_env" "$news_args" ""
|
|
||||||
add_case "live-table" "screen" "0" "$live_env" "$screen_args" ""
|
|
||||||
|
|
||||||
add_case "live-json" "quote" "0" "$live_env" "-o json stocks quote $live_symbol" ""
|
|
||||||
add_case "live-json" "history" "0" "$live_env" "-o json $history_args" ""
|
|
||||||
add_case "live-json" "technical" "0" "$live_env" "-o json stocks technical $live_symbol" ""
|
|
||||||
add_case "live-json" "growth" "0" "$live_env" "-o json stocks growth $live_symbol" ""
|
|
||||||
add_case "live-json" "valuation" "0" "$live_env" "-o json stocks valuation $live_symbol" ""
|
|
||||||
add_case "live-json" "risk" "0" "$live_env" "-o json stocks risk $live_symbol" ""
|
|
||||||
add_case "live-json" "fundamental" "0" "$live_env" "-o json stocks fundamental $live_symbol" ""
|
|
||||||
add_case "live-json" "compare" "0" "$live_env" "-o json stocks compare $compare_args" ""
|
|
||||||
add_case "live-json" "profile" "0" "$live_env" "-o json stocks profile $live_symbol" ""
|
|
||||||
add_case "live-json" "financials" "0" "$live_env" "-o json stocks financials $live_symbol" ""
|
|
||||||
add_case "live-json" "earnings" "0" "$live_env" "-o json stocks earnings $live_symbol" ""
|
|
||||||
add_case "live-json" "sentiment" "0" "$live_env" "-o json stocks sentiment $live_symbol" ""
|
|
||||||
add_case "live-json" "insights" "0" "$live_env" "-o json stocks insights $live_symbol" ""
|
|
||||||
add_case "live-json" "news" "0" "$live_env" "-o json $news_args" ""
|
|
||||||
add_case "live-json" "screen" "0" "$live_env" "-o json $screen_args" ""
|
|
||||||
|
|
||||||
add_case "live-nonfinite" "valuation-bumi" "0" "$live_env" "-o json stocks valuation BUMI" ""
|
|
||||||
add_case "live-nonfinite" "valuation-adro" "0" "$live_env" "-o json stocks valuation ADRO" ""
|
|
||||||
add_case "live-nonfinite" "valuation-aims" "0" "$live_env" "-o json stocks valuation AIMS" ""
|
|
||||||
add_case "live-nonfinite" "compare-bad-tickers" "0" "$live_env" "-o json stocks compare BUMI,ADRO,AIMS" "" 'any(.[]; .symbol == "BUMI.JK")'
|
|
||||||
|
|
||||||
add_case "mock" "quote-table" "0" "$mock_env" "stocks quote $live_symbol" "9,875"
|
|
||||||
add_case "mock" "quote-json" "0" "$mock_env" "-o json stocks quote $live_symbol" "" '.[0].price == 9875 and .[0].change == 117'
|
|
||||||
add_case "mock" "history-table" "0" "$mock_env" "$history_args" "2024-03-03"
|
|
||||||
add_case "mock" "history-json" "0" "$mock_env" "-o json $history_args" "" '.[-1].close == 9925'
|
|
||||||
add_case "mock" "technical-json" "0" "$mock_env" "-o json stocks technical $live_symbol" "" '.current_price == 9925 and .sma200 == null'
|
|
||||||
add_case "mock" "growth-json" "0" "$mock_env" "-o json stocks growth $live_symbol" "" '.revenue_growth_pct == 8.1 and .earnings_growth_pct == 12.1'
|
|
||||||
add_case "mock" "valuation-json" "0" "$mock_env" "-o json stocks valuation $live_symbol" "" '.pe_trailing == 25.4 and .roe_pct == 19.8'
|
|
||||||
add_case "mock" "risk-json" "0" "$mock_env" "-o json stocks risk $live_symbol" "" '.debt_to_equity == 0.42 and .roa_pct == 3.1'
|
|
||||||
add_case "mock" "fundamental-table" "0" "$mock_env" "stocks fundamental $live_symbol" "25.40"
|
|
||||||
add_case "mock" "fundamental-json" "0" "$mock_env" "-o json stocks fundamental $live_symbol" "" '.growth.revenue_growth_pct == 8.1 and .valuation.pe_trailing == 25.4 and .risk.debt_to_equity == 0.42'
|
|
||||||
add_case "mock" "compare-json" "0" "$mock_env" "-o json stocks compare $compare_args" "" 'length > 0 and all(.[]; .growth.revenue_growth_pct == 8.1 and .valuation.pe_trailing == 25.4)'
|
|
||||||
add_case "mock" "profile-json" "0" "$mock_env" "-o json stocks profile $live_symbol" "" '.country == "Indonesia" and .employees == 27937'
|
|
||||||
add_case "mock" "financials-table" "0" "$mock_env" "stocks financials $live_symbol" "400,000,000"
|
|
||||||
add_case "mock" "financials-json" "0" "$mock_env" "-o json stocks financials $live_symbol" "" '.income_statement.values.netIncome == 400000000 and .cash_flow.values.operatingCashFlow == 600000000'
|
|
||||||
add_case "mock" "earnings-json" "0" "$mock_env" "-o json stocks earnings $live_symbol" "" '.eps_last_year == 1200 and .forecast[0].eps_forecast == 1300 and .history[0].eps_actual == 1250'
|
|
||||||
add_case "mock" "sentiment-json" "0" "$mock_env" "-o json stocks sentiment $live_symbol" "" '.statistics[0] == {"time_range":"1D","bullish":10,"bearish":2,"neutral":3}'
|
|
||||||
add_case "mock" "insights-json" "0" "$mock_env" "-o json stocks insights $live_symbol" "" '.last_updated == "2026-03-26T04:14:57.9197955Z"'
|
|
||||||
add_case "mock" "news-json" "0" "$mock_env" "-o json $news_args" "" "map(.id) == ([\"news-1\"][:$NEWS_LIMIT])"
|
|
||||||
add_case "mock" "screen-json" "0" "$mock_env" "-o json $screen_args" "" "map(.symbol) == ([\"BBRI.JK\", \"BBCA.JK\"][:$SCREEN_LIMIT])"
|
|
||||||
|
|
||||||
add_case "cache" "quote-warm" "0" "$quote_zero_ttl_env" "-o json stocks quote $live_symbol" ""
|
|
||||||
add_case "cache" "quote-offline" "0" "$quote_zero_ttl_env IDX_MOCK_ERROR=1" "-o json --offline stocks quote $live_symbol" ""
|
|
||||||
add_case "cache" "quote-stale-fallback" "0" "$quote_zero_ttl_env IDX_MOCK_ERROR=1" "-o json stocks quote $live_symbol" ""
|
|
||||||
add_case "cache" "technical-warm" "0" "$quote_zero_ttl_env" "-o json stocks technical $live_symbol" ""
|
|
||||||
add_case "cache" "technical-offline" "0" "$quote_zero_ttl_env IDX_MOCK_ERROR=1" "-o json --offline stocks technical $live_symbol" ""
|
|
||||||
add_case "cache" "technical-stale-fallback" "0" "$quote_zero_ttl_env IDX_MOCK_ERROR=1" "-o json stocks technical $live_symbol" ""
|
|
||||||
add_case "cache" "profile-warm" "0" "$profile_zero_ttl_env" "-o json stocks profile $live_symbol" ""
|
|
||||||
add_case "cache" "profile-offline" "0" "$profile_zero_ttl_env IDX_MOCK_ERROR=1" "-o json --offline stocks profile $live_symbol" ""
|
|
||||||
add_case "cache" "profile-stale-fallback" "0" "$profile_zero_ttl_env IDX_MOCK_ERROR=1" "-o json stocks profile $live_symbol" ""
|
|
||||||
|
|
||||||
add_case "routing" "yahoo-quote" "0" "$yahoo_mock_env" "-o json --no-cache stocks quote $live_symbol" "" '.[0].price == 9875'
|
|
||||||
add_case "routing" "explicit-msn-history" "0" "$mock_env" "-o json --no-cache stocks history $live_symbol --period 3mo --history-provider msn" "" '.[0].close == 8000 and all(.[]; .volume == 0 and .open == .close and .high == .close and .low == .close)'
|
|
||||||
|
|
||||||
add_case "errors" "invalid-provider-json" "1" "IDX_PROVIDER=bogus" "-o json version" "" '.code == "CONFIGERROR"'
|
|
||||||
add_case "errors" "profile-provider-gate-json" "1" "IDX_PROVIDER=yahoo" "-o json stocks profile $live_symbol" "" '.code == "UNSUPPORTED"'
|
|
||||||
add_case "errors" "invalid-screen-region-json" "1" "$mock_env" "-o json stocks screen --region eu" "" '.code == "INVALIDINPUT"'
|
|
||||||
add_case "errors" "offline-no-cache" "1" "IDX_USE_MOCK_PROVIDER=1" "-o json --offline --no-cache stocks quote $live_symbol" "" '.code == "INVALIDINPUT"'
|
|
||||||
|
|
||||||
add_case "ownership" "releases-empty" "0" "" "-o json ownership releases" "" '. == []'
|
|
||||||
add_case "ownership" "fetch-bing-unsupported" "1" "" "-o json ownership import --fetch-bing $live_symbol" "" '.code == "UNSUPPORTED"'
|
|
||||||
|
|
||||||
add_case "ownership-import" "discover-default-above1" "0" "" "ownership discover --limit 1" "supported"
|
|
||||||
add_case "ownership-import" "import-supported-above1" "0" "" "ownership import --url $OWNERSHIP_ABOVE1_URL" "Imported "
|
|
||||||
add_case "ownership-import" "releases-after-import" "0" "" "ownership releases" "$OWNERSHIP_ABOVE1_URL"
|
|
||||||
add_case "ownership-import" "import-legacy-above5" "1" "" "ownership import --url $OWNERSHIP_ABOVE5_URL" "legacy IDX \`above5\` ownership PDFs are not supported for import"
|
|
||||||
add_case "ownership-import" "import-legacy-investor-type" "1" "" "ownership import --url $OWNERSHIP_INVESTOR_TYPE_URL" "legacy IDX \`investor-type\` ownership PDFs are not supported for import"
|
|
||||||
}
|
|
||||||
|
|
||||||
parse_args() {
|
|
||||||
while [[ $# -gt 0 ]]; do
|
|
||||||
case "$1" in
|
|
||||||
--mode)
|
|
||||||
MODE="${2:-}"
|
|
||||||
shift 2
|
|
||||||
;;
|
|
||||||
--group)
|
|
||||||
add_group_filters "${2:-}"
|
|
||||||
shift 2
|
|
||||||
;;
|
|
||||||
--symbol)
|
|
||||||
SYMBOL="${2:-}"
|
|
||||||
shift 2
|
|
||||||
;;
|
|
||||||
--compare-symbols)
|
|
||||||
COMPARE_SYMBOLS="${2:-}"
|
|
||||||
shift 2
|
|
||||||
;;
|
|
||||||
--news-limit)
|
|
||||||
NEWS_LIMIT="${2:-}"
|
|
||||||
shift 2
|
|
||||||
;;
|
|
||||||
--screen-limit)
|
|
||||||
SCREEN_LIMIT="${2:-}"
|
|
||||||
shift 2
|
|
||||||
;;
|
|
||||||
--workdir)
|
|
||||||
WORKDIR="${2:-}"
|
|
||||||
shift 2
|
|
||||||
;;
|
|
||||||
--bin)
|
|
||||||
BIN_PATH="${2:-}"
|
|
||||||
BUILD=0
|
|
||||||
shift 2
|
|
||||||
;;
|
|
||||||
--no-build)
|
|
||||||
BUILD=0
|
|
||||||
shift
|
|
||||||
;;
|
|
||||||
--dry-run)
|
|
||||||
DRY_RUN=1
|
|
||||||
shift
|
|
||||||
;;
|
|
||||||
--help|-h)
|
|
||||||
usage
|
|
||||||
exit 0
|
|
||||||
;;
|
|
||||||
*)
|
|
||||||
echo "unknown argument: $1" >&2
|
|
||||||
usage >&2
|
|
||||||
exit 2
|
|
||||||
;;
|
|
||||||
esac
|
|
||||||
done
|
|
||||||
|
|
||||||
case "$MODE" in
|
|
||||||
live|mock|full) ;;
|
|
||||||
*)
|
*)
|
||||||
echo "invalid --mode '$MODE' (expected live, mock, or full)" >&2
|
echo 'true' ;;
|
||||||
exit 2
|
|
||||||
;;
|
|
||||||
esac
|
esac
|
||||||
}
|
}
|
||||||
|
|
||||||
prepare_paths() {
|
run_case() {
|
||||||
if [[ -z "$WORKDIR" ]]; then
|
local name="$1" cmd="$2" filter log
|
||||||
WORKDIR="$ROOT_DIR/tmp/live-smoke/$(date +%Y%m%d-%H%M%S)"
|
local -a args
|
||||||
|
TOTAL=$((TOTAL + 1))
|
||||||
|
printf '[%02d] %s\n' "$TOTAL" "$name"
|
||||||
|
if (( DRY_RUN )); then
|
||||||
|
printf ' idx -o json %s\n' "$cmd"
|
||||||
|
return
|
||||||
fi
|
fi
|
||||||
|
|
||||||
CONFIG_HOME="$WORKDIR/config"
|
log="$LOG_DIR/$(printf '%02d' "$TOTAL")_$name"
|
||||||
CACHE_HOME="$WORKDIR/cache"
|
read -r -a args <<< "$cmd"
|
||||||
DATA_HOME="$WORKDIR/data"
|
if ! env "${BASE_ENV[@]}" "$BIN_PATH" -o json "${args[@]}" >"$log.stdout" 2>"$log.stderr"; then
|
||||||
OWNERSHIP_DB="$WORKDIR/ownership/ownership.db"
|
FAILURES+=("$name: non-zero exit (see $log.stderr)")
|
||||||
LOG_DIR="$WORKDIR/logs"
|
echo " FAIL exit"
|
||||||
|
return
|
||||||
BASE_ENV=(
|
|
||||||
"XDG_CONFIG_HOME=$CONFIG_HOME"
|
|
||||||
"XDG_CACHE_HOME=$CACHE_HOME"
|
|
||||||
"XDG_DATA_HOME=$DATA_HOME"
|
|
||||||
"IDX_OUTPUT=table"
|
|
||||||
"IDX_NO_COLOR=1"
|
|
||||||
)
|
|
||||||
|
|
||||||
if (( DRY_RUN == 0 )); then
|
|
||||||
mkdir -p "$CONFIG_HOME" "$CACHE_HOME" "$DATA_HOME" "$(dirname "$OWNERSHIP_DB")" "$LOG_DIR"
|
|
||||||
fi
|
fi
|
||||||
|
filter="$(contract "$cmd")"
|
||||||
|
if ! jq -e -s "length == 1 and (.[0] | $filter)" "$log.stdout" >/dev/null 2>"$log.validation"; then
|
||||||
|
FAILURES+=("$name: output shape changed (see $log.stdout)")
|
||||||
|
echo " FAIL shape"
|
||||||
|
return
|
||||||
|
fi
|
||||||
|
PASSED=$((PASSED + 1))
|
||||||
|
echo " PASS"
|
||||||
}
|
}
|
||||||
|
|
||||||
build_binary() {
|
if (( ! DRY_RUN )); then
|
||||||
local build_log
|
command -v jq >/dev/null 2>&1 || { echo "jq is required; run inside nix develop" >&2; exit 1; }
|
||||||
local stale_input
|
mkdir -p "$LOG_DIR"
|
||||||
|
|
||||||
if [[ -z "$BIN_PATH" ]]; then
|
if [[ -z "$BIN_PATH" ]]; then
|
||||||
BIN_PATH="$ROOT_DIR/target/debug/idx"
|
BIN_PATH="$ROOT_DIR/target/debug/idx"
|
||||||
|
(cd "$ROOT_DIR" && cargo build --quiet --bin idx) || { echo "cargo build failed" >&2; exit 1; }
|
||||||
fi
|
fi
|
||||||
|
env "${BASE_ENV[@]}" "$BIN_PATH" config set ownership.db_path "$WORKDIR/ownership.db" >/dev/null
|
||||||
|
fi
|
||||||
|
|
||||||
if (( DRY_RUN )); then
|
if selected stocks; then
|
||||||
return 0
|
for cmd in quote technical growth valuation risk fundamental profile financials earnings sentiment insights; do
|
||||||
fi
|
run_case "$cmd" "stocks $cmd $SYMBOL"
|
||||||
|
|
||||||
if (( BUILD )); then
|
|
||||||
if ! command -v cargo >/dev/null 2>&1; then
|
|
||||||
if [[ -x "$BIN_PATH" ]]; then
|
|
||||||
printf 'info: cargo not found; reusing existing binary at %s\n' "$BIN_PATH"
|
|
||||||
BUILD=0
|
|
||||||
else
|
|
||||||
echo "cargo not found; run inside nix develop or pass --bin/--no-build with an existing idx binary" >&2
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
fi
|
|
||||||
fi
|
|
||||||
|
|
||||||
if (( BUILD == 0 )) && [[ "$BIN_PATH" == "$ROOT_DIR/target/debug/idx" && -x "$BIN_PATH" ]]; then
|
|
||||||
stale_input="$(
|
|
||||||
find \
|
|
||||||
"$ROOT_DIR/src" \
|
|
||||||
"$ROOT_DIR/tests" \
|
|
||||||
"$ROOT_DIR/Cargo.toml" \
|
|
||||||
"$ROOT_DIR/Cargo.lock" \
|
|
||||||
"$ROOT_DIR/scripts/live-smoke.sh" \
|
|
||||||
-type f \
|
|
||||||
-newer "$BIN_PATH" \
|
|
||||||
-print \
|
|
||||||
-quit \
|
|
||||||
2>/dev/null
|
|
||||||
)"
|
|
||||||
if [[ -n "$stale_input" ]]; then
|
|
||||||
echo "refusing to run smoke checks against stale $BIN_PATH; newer input detected at $stale_input" >&2
|
|
||||||
echo "rebuild first or omit --no-build so the runner refreshes target/debug/idx" >&2
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
fi
|
|
||||||
|
|
||||||
if (( BUILD )); then
|
|
||||||
build_log="$LOG_DIR/build.log"
|
|
||||||
if ! (
|
|
||||||
cd "$ROOT_DIR" || exit 1
|
|
||||||
cargo build --quiet --bin idx
|
|
||||||
) >"$build_log" 2>&1; then
|
|
||||||
echo "build failed; see $build_log" >&2
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
fi
|
|
||||||
|
|
||||||
if [[ ! -x "$BIN_PATH" ]]; then
|
|
||||||
echo "idx binary not found or not executable: $BIN_PATH" >&2
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
}
|
|
||||||
|
|
||||||
bootstrap_case() {
|
|
||||||
local label="$1"
|
|
||||||
shift
|
|
||||||
local log_file="$LOG_DIR/bootstrap-${label}.log"
|
|
||||||
|
|
||||||
if (( DRY_RUN )); then
|
|
||||||
printf 'bootstrap %-18s %s\n' "$label" "$*"
|
|
||||||
return 0
|
|
||||||
fi
|
|
||||||
|
|
||||||
if ! (
|
|
||||||
cd "$ROOT_DIR" || exit 1
|
|
||||||
env "${BASE_ENV[@]}" "$BIN_PATH" "$@"
|
|
||||||
) >"$log_file" 2>&1; then
|
|
||||||
echo "bootstrap failed for $label; see $log_file" >&2
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
}
|
|
||||||
|
|
||||||
prepare_environment() {
|
|
||||||
bootstrap_case "config-init" config init
|
|
||||||
bootstrap_case "ownership-db" config set ownership.db_path "$OWNERSHIP_DB"
|
|
||||||
}
|
|
||||||
|
|
||||||
discover_ownership_import_url() {
|
|
||||||
local family="$1"
|
|
||||||
local log_file="$LOG_DIR/bootstrap-discover-${family}.log"
|
|
||||||
local url
|
|
||||||
|
|
||||||
if (( DRY_RUN )); then
|
|
||||||
case "$family" in
|
|
||||||
above1) printf 'https://example.invalid/above1-lamp1.pdf' ;;
|
|
||||||
above5) printf 'https://example.invalid/above5-lamp1.pdf' ;;
|
|
||||||
investor-type) printf 'https://example.invalid/investor-type-lamp1.pdf' ;;
|
|
||||||
*) return 1 ;;
|
|
||||||
esac
|
|
||||||
return 0
|
|
||||||
fi
|
|
||||||
|
|
||||||
if ! (
|
|
||||||
cd "$ROOT_DIR" || exit 1
|
|
||||||
env "${BASE_ENV[@]}" "$BIN_PATH" -o json ownership discover --family "$family" --limit 1
|
|
||||||
) >"$log_file" 2>&1; then
|
|
||||||
echo "bootstrap failed for ownership-discover-$family; see $log_file" >&2
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
|
|
||||||
url="$(grep -m1 '"pdf_url"' "$log_file" | sed -E 's/.*"pdf_url": "([^"]+)".*/\1/')"
|
|
||||||
if [[ -z "$url" ]]; then
|
|
||||||
echo "bootstrap failed for ownership-discover-$family: could not parse pdf_url from $log_file" >&2
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
|
|
||||||
printf '%s' "$url"
|
|
||||||
}
|
|
||||||
|
|
||||||
prepare_ownership_import_inputs() {
|
|
||||||
if ! should_run_group "ownership-import"; then
|
|
||||||
return 0
|
|
||||||
fi
|
|
||||||
|
|
||||||
OWNERSHIP_ABOVE1_URL="$(discover_ownership_import_url above1)"
|
|
||||||
OWNERSHIP_ABOVE5_URL="$(discover_ownership_import_url above5)"
|
|
||||||
OWNERSHIP_INVESTOR_TYPE_URL="$(discover_ownership_import_url investor-type)"
|
|
||||||
}
|
|
||||||
|
|
||||||
cache_case_starts_fresh() {
|
|
||||||
local group="$1"
|
|
||||||
local label="$2"
|
|
||||||
|
|
||||||
[[ "$group" == "cache" && "$label" == *"-warm" ]]
|
|
||||||
}
|
|
||||||
|
|
||||||
reset_cache_for_case() {
|
|
||||||
local label="$1"
|
|
||||||
|
|
||||||
# Cache checks need a clean starting point so the warm/offline/stale trio
|
|
||||||
# validates TTL-sensitive fallback behavior instead of reusing prior group state.
|
|
||||||
bootstrap_case "cache-reset-${label}" cache clear
|
|
||||||
}
|
|
||||||
|
|
||||||
sanitize_name() {
|
|
||||||
printf '%s' "$1" | tr -cs 'A-Za-z0-9._-' '_'
|
|
||||||
}
|
|
||||||
|
|
||||||
command_display() {
|
|
||||||
local env_spec="$1"
|
|
||||||
local cmd_spec="$2"
|
|
||||||
|
|
||||||
printf 'env %s' "${BASE_ENV[*]}"
|
|
||||||
if [[ -n "$env_spec" ]]; then
|
|
||||||
printf ' %s' "$env_spec"
|
|
||||||
fi
|
|
||||||
printf ' %s %s' "$BIN_PATH" "$cmd_spec"
|
|
||||||
}
|
|
||||||
|
|
||||||
json_contract() {
|
|
||||||
if [[ "$2" -ne 0 ]]; then
|
|
||||||
printf '%s\n' '.error == true and (.code | type == "string") and (.message | type == "string")'
|
|
||||||
return
|
|
||||||
fi
|
|
||||||
|
|
||||||
case "$1" in
|
|
||||||
*"stocks quote "*|*"stocks screen"*)
|
|
||||||
printf '%s\n' 'type == "array" and all(.[]; (.symbol | type == "string") and (.price | type == "number") and has("change_pct"))' ;;
|
|
||||||
*"stocks history "*)
|
|
||||||
printf '%s\n' 'type == "array" and all(.[]; (.date | type == "string") and (.close | type == "number") and (.volume | type == "number"))' ;;
|
|
||||||
*"stocks technical "*)
|
|
||||||
printf '%s\n' '(.symbol | type == "string") and (.current_price | type == "number") and has("sma200") and (.signals.overall | type == "string")' ;;
|
|
||||||
*"stocks growth "*)
|
|
||||||
printf '%s\n' 'has("revenue_growth_pct") and has("earnings_growth_pct") and (.overall_signal | type == "string")' ;;
|
|
||||||
*"stocks valuation "*)
|
|
||||||
printf '%s\n' 'has("pe_trailing") and has("roe_pct") and (.overall_signal | type == "string")' ;;
|
|
||||||
*"stocks risk "*)
|
|
||||||
printf '%s\n' 'has("debt_to_equity") and has("roa_pct") and (.overall_signal | type == "string")' ;;
|
|
||||||
*"stocks fundamental "*)
|
|
||||||
printf '%s\n' '(.symbol | type == "string") and (.growth.overall_signal | type == "string") and (.valuation.overall_signal | type == "string") and (.risk.overall_signal | type == "string")' ;;
|
|
||||||
*"stocks compare "*)
|
|
||||||
printf '%s\n' 'type == "array" and all(.[]; (.symbol | type == "string") and (.growth | type == "object") and (.valuation | type == "object") and (.risk | type == "object"))' ;;
|
|
||||||
*"stocks profile "*)
|
|
||||||
printf '%s\n' '(.symbol | type == "string") and has("long_name") and has("country") and (.officers | type == "array")' ;;
|
|
||||||
*"stocks financials "*)
|
|
||||||
printf '%s\n' '(.instrument.symbol | type == "string") and has("income_statement") and has("balance_sheet") and has("cash_flow")' ;;
|
|
||||||
*"stocks earnings "*)
|
|
||||||
printf '%s\n' '(.symbol | type == "string") and (.history | type == "array") and (.forecast | type == "array")' ;;
|
|
||||||
*"stocks sentiment "*)
|
|
||||||
printf '%s\n' '(.symbol | type == "string") and (.statistics | type == "array") and all(.statistics[]; (.bullish | type == "number") and (.bearish | type == "number") and (.neutral | type == "number"))' ;;
|
|
||||||
*"stocks insights "*)
|
|
||||||
printf '%s\n' '(.symbol | type == "string") and (.highlights | type == "array") and (.risks | type == "array")' ;;
|
|
||||||
*"stocks news "*)
|
|
||||||
printf '%s\n' 'type == "array" and all(.[]; (.symbol | type == "string") and (.title | type == "string") and (.url | type == "string"))' ;;
|
|
||||||
*)
|
|
||||||
printf '%s\n' 'type == "object" or type == "array"' ;;
|
|
||||||
esac
|
|
||||||
}
|
|
||||||
|
|
||||||
run_case_line() {
|
|
||||||
local line="$1"
|
|
||||||
local group label expected_exit env_spec cmd_spec expect_spec json_expect
|
|
||||||
local safe_name log_file stdout_file stderr_file output_file status contract baseline_file
|
|
||||||
local failure=""
|
|
||||||
local -a env_parts cmd_parts
|
|
||||||
|
|
||||||
IFS="$SEP" read -r group label expected_exit env_spec cmd_spec expect_spec json_expect <<< "$line"
|
|
||||||
|
|
||||||
if ! should_run_group "$group"; then
|
|
||||||
return 0
|
|
||||||
fi
|
|
||||||
|
|
||||||
TOTAL_CASES=$((TOTAL_CASES + 1))
|
|
||||||
safe_name="$(sanitize_name "${group}-${label}")"
|
|
||||||
log_file="$(printf '%s/%03d_%s' "$LOG_DIR" "$TOTAL_CASES" "$safe_name")"
|
|
||||||
stdout_file="$log_file.stdout"
|
|
||||||
stderr_file="$log_file.stderr"
|
|
||||||
|
|
||||||
printf '[%02d] %-10s %s\n' "$TOTAL_CASES" "$group" "$label"
|
|
||||||
|
|
||||||
if cache_case_starts_fresh "$group" "$label"; then
|
|
||||||
reset_cache_for_case "$label"
|
|
||||||
fi
|
|
||||||
|
|
||||||
if (( DRY_RUN )); then
|
|
||||||
printf ' %s\n' "$(command_display "$env_spec" "$cmd_spec")"
|
|
||||||
return 0
|
|
||||||
fi
|
|
||||||
|
|
||||||
env_parts=()
|
|
||||||
cmd_parts=()
|
|
||||||
|
|
||||||
if [[ -n "$env_spec" ]]; then
|
|
||||||
read -r -a env_parts <<< "$env_spec"
|
|
||||||
fi
|
|
||||||
read -r -a cmd_parts <<< "$cmd_spec"
|
|
||||||
|
|
||||||
(
|
|
||||||
cd "$ROOT_DIR" || exit 1
|
|
||||||
env "${BASE_ENV[@]}" "${env_parts[@]}" "$BIN_PATH" "${cmd_parts[@]}"
|
|
||||||
) >"$stdout_file" 2>"$stderr_file"
|
|
||||||
status=$?
|
|
||||||
|
|
||||||
output_file="$stdout_file"
|
|
||||||
if [[ "$status" -ne 0 ]]; then
|
|
||||||
output_file="$stderr_file"
|
|
||||||
fi
|
|
||||||
|
|
||||||
if [[ "$status" -ne "$expected_exit" ]]; then
|
|
||||||
failure="exit $status, expected $expected_exit"
|
|
||||||
elif [[ "$status" -ne 0 && -s "$stdout_file" ]]; then
|
|
||||||
failure="error command wrote to stdout"
|
|
||||||
elif [[ -n "$expect_spec" ]] && ! grep -Fq -- "$expect_spec" "$output_file"; then
|
|
||||||
failure="missing '$expect_spec' in expected output stream"
|
|
||||||
elif [[ " $cmd_spec " == *" -o json "* ]]; then
|
|
||||||
contract="$(json_contract "$cmd_spec" "$status")"
|
|
||||||
if ! jq -e -s "length == 1 and (.[0] | ($contract) and (${json_expect:-true}))" \
|
|
||||||
"$output_file" > /dev/null 2>"$log_file.validation"; then
|
|
||||||
failure="invalid JSON document or output contract"
|
|
||||||
fi
|
|
||||||
elif [[ "$label" == "completions-bash" ]] && ! bash -n "$stdout_file"; then
|
|
||||||
failure="invalid Bash completion script"
|
|
||||||
fi
|
|
||||||
|
|
||||||
if [[ -z "$failure" && "$group" == "cache" ]]; then
|
|
||||||
baseline_file="$LOG_DIR/cache-${label%%-*}-baseline.json"
|
|
||||||
if [[ "$label" == *"-warm" ]]; then
|
|
||||||
jq -S . "$stdout_file" >"$baseline_file"
|
|
||||||
elif ! jq -S . "$stdout_file" | cmp -s "$baseline_file" -; then
|
|
||||||
failure="offline/stale data differs from the warmed cache"
|
|
||||||
elif [[ ! -s "$stderr_file" ]]; then
|
|
||||||
failure="expired-cache diagnostic missing from stderr"
|
|
||||||
fi
|
|
||||||
fi
|
|
||||||
|
|
||||||
if [[ -n "$failure" ]]; then
|
|
||||||
FAILED_CASES=$((FAILED_CASES + 1))
|
|
||||||
FAILURES+=("$group/$label ($failure) -> $log_file.{stdout,stderr,validation}")
|
|
||||||
printf ' FAIL %s logs=%s.{stdout,stderr,validation}\n' "$failure" "$log_file"
|
|
||||||
return 0
|
|
||||||
fi
|
|
||||||
|
|
||||||
PASSED_CASES=$((PASSED_CASES + 1))
|
|
||||||
printf ' PASS logs=%s.{stdout,stderr}\n' "$log_file"
|
|
||||||
}
|
|
||||||
|
|
||||||
main() {
|
|
||||||
local case_line
|
|
||||||
|
|
||||||
parse_args "$@"
|
|
||||||
prepare_paths
|
|
||||||
if (( DRY_RUN == 0 )) && ! command -v jq >/dev/null 2>&1; then
|
|
||||||
echo "jq is required for smoke output validation; run inside nix develop" >&2
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
|
|
||||||
printf 'mode: %s\n' "$MODE"
|
|
||||||
printf 'workdir: %s\n' "$WORKDIR"
|
|
||||||
printf 'symbol: %s\n' "$SYMBOL"
|
|
||||||
printf 'compare symbols: %s\n' "$COMPARE_SYMBOLS"
|
|
||||||
if [[ ${#GROUP_FILTERS[@]} -gt 0 ]]; then
|
|
||||||
printf 'group filter: %s\n' "${GROUP_FILTERS[*]}"
|
|
||||||
fi
|
|
||||||
|
|
||||||
build_binary
|
|
||||||
prepare_environment
|
|
||||||
prepare_ownership_import_inputs
|
|
||||||
register_cases
|
|
||||||
|
|
||||||
if should_run_group "ownership-import"; then
|
|
||||||
printf 'ownership above1 url: %s\n' "$OWNERSHIP_ABOVE1_URL"
|
|
||||||
printf 'ownership above5 url: %s\n' "$OWNERSHIP_ABOVE5_URL"
|
|
||||||
printf 'ownership investor-type url: %s\n' "$OWNERSHIP_INVESTOR_TYPE_URL"
|
|
||||||
fi
|
|
||||||
|
|
||||||
for case_line in "${CASES[@]}"; do
|
|
||||||
run_case_line "$case_line"
|
|
||||||
done
|
done
|
||||||
|
run_case history "stocks history $SYMBOL --period 3mo"
|
||||||
|
run_case news "stocks news $SYMBOL --limit 5"
|
||||||
|
run_case screen "stocks screen --filter top-performers --limit 10"
|
||||||
|
run_case compare "stocks compare $SYMBOL,BBRI"
|
||||||
|
# BUMI/ADRO/AIMS return non-finite MSN key ratios (Infinity/NaN).
|
||||||
|
run_case compare-nonfinite "stocks compare BUMI,ADRO,AIMS"
|
||||||
|
fi
|
||||||
|
|
||||||
if (( DRY_RUN )); then
|
if selected ownership; then
|
||||||
printf '\ndry-run complete: %d case(s) selected\n' "$TOTAL_CASES"
|
run_case discover "ownership discover --limit 1"
|
||||||
exit 0
|
if (( ! DRY_RUN )) && [[ -s "$LOG_DIR/$(printf '%02d' "$TOTAL")_discover.stdout" ]]; then
|
||||||
|
url="$(jq -r '.[0].pdf_url' "$LOG_DIR/$(printf '%02d' "$TOTAL")_discover.stdout")"
|
||||||
|
run_case import "ownership import --url $url"
|
||||||
|
run_case releases "ownership releases"
|
||||||
fi
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
printf '\nsummary: %d passed, %d failed, %d total\n' "$PASSED_CASES" "$FAILED_CASES" "$TOTAL_CASES"
|
(( DRY_RUN )) && { echo "dry-run: $TOTAL case(s)"; exit 0; }
|
||||||
printf 'logs: %s\n' "$LOG_DIR"
|
printf '\nsummary: %d passed, %d failed\nlogs: %s\n' "$PASSED" "$((TOTAL - PASSED))" "$LOG_DIR"
|
||||||
|
if (( ${#FAILURES[@]} > 0 )); then
|
||||||
if (( FAILED_CASES > 0 )); then
|
printf ' - %s\n' "${FAILURES[@]}"
|
||||||
printf 'failures:\n'
|
exit 1
|
||||||
printf ' - %s\n' "${FAILURES[@]}"
|
fi
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
}
|
|
||||||
|
|
||||||
main "$@"
|
|
||||||
|
|
|
||||||
Loading…
Add table
Add a link
Reference in a new issue