feat: login-free Stockbit intraday and quote provider

Read Stockbit's public symbol page (__NEXT_DATA__) with a plain ureq
request and an honest user agent: `stocks intraday` (today by the
minute, any provider), `--provider stockbit` quotes with the best
bid/offer only while the market is open, and MCP `get_intraday`.

- UPSTREAMBLOCKED error (exit 4) for a CDN 403; never retried
- requests paced 1 s apart across idx processes, 5 s snapshot cache
- global --provider flag (yahoo, msn, stockbit)
- roadmap: L1 ships from Stockbit; depth stays a login follow-on
This commit is contained in:
hermes 2026-10-23 04:58:32 +00:00
commit 74e5bb0536
65 changed files with 2320 additions and 381 deletions

View file

@ -55,6 +55,9 @@ command parses arguments, makes one idx-core call, and renders the result.
(`BrokerSummaryProvider`). See [Broker Summaries](#broker-summaries).
- **IDX (idx.co.id)** — the daily trading summary behind `stocks flow
--source idx` (`TradingSummaryProvider`). See [Foreign Flow](#foreign-flow).
- **Stockbit (public page)** — `stocks intraday` whichever provider is
selected (`IntradayProvider`), and quotes under the opt-in
`--provider stockbit`. See [Stockbit](#stockbit).
### MSN Endpoints
- `https://assets.msn.com/service/`: quotes, charts, equities, earnings,
@ -151,6 +154,41 @@ WIB as the fallback.
range fails on one. `foreign_net_value` is net shares at the session's
average price (value over volume), an estimate.
### Stockbit
- `GET https://stockbit.com/symbol/{CODE}` needs no login. Next.js
server-renders the quote and today's chart into the page's `__NEXT_DATA__`
JSON; `api/stockbit/parse.rs` reads it purely. `props.pageProps.company`
is the quote (numbers mostly as strings, an index price as a float string),
`orderbook.{bid,offer}.price` the best bid and offer, `market_hour.status`
the market's state, and `status` `STATUS_ACTIVE|SUSPENDED|DELISTED`.
`chart.prices[]` holds one price per minute in WIB with no offset, after a
synthetic first row at the previous close; per-minute OHLC and volume are
empty. An unknown code is HTTP 404 with the 404 page.
- Stockbit's JSON API (`exodus.stockbit.com`: chart, order book) answers 401
without a user's bearer token; it is not used.
- Bid and offer are kept only while `market_hour.status` is `open`, so a
closing book never reads as a live one. Observed 2026-10-23: `open` during
trading (bid/offer set), `break` from 11:30 on a Friday, `close` after
hours; any other value fails closed. In session the chart gained a point
every minute and the last trade was within a minute of the clock, although
Stockbit's terms call its data delayed at least 10 minutes. Order-book
sizes are dropped: the page does not say whether they are lots or shares.
`as_of` comes from `date`/`time`
only when the stock traded (a suspended stock carries today's date).
`session_note` repeats what the page says (suspended, delisted) or "no
trades this session"; it is never inferred.
- CloudFront in front of the page answers a static 403 to some user agents
(`curl/`, `axios/`, `Go-http-client/`, empty) and does not fingerprint TLS,
so a plain ureq request with `idx-cli/<version> (+repo)` passes. A 403 is
`UPSTREAMBLOCKED` and is not retried. Requests are at least 1 s apart
across the user's idx processes (the last request's time sits in
`<cache>/stockbit.pace` under a file lock, so a shell loop of `idx` calls is
paced too), and snapshots are cached for 5 s (at most `quote_ttl`), which
coalesces an agent's quote-then-intraday burst across MCP calls.
- Stockbit's terms call its data delayed at least 10 minutes and for
personal use; frontends say so. Fixtures (`tests/fixtures/stockbit_*`) are
pages recorded 2026-10-22 after the close, cut to the fields read.
### Capability Gating
```
MarketDataProvider = QuoteProvider + FundamentalsProvider
@ -161,8 +199,13 @@ Factory functions:
history_provider(kind, mode, verbose) → Result<(ProviderKind, Box<dyn HistoryProvider>)>
broker_provider() → Box<dyn BrokerSummaryProvider>
trading_summary_provider() → Box<dyn TradingSummaryProvider>
intraday_provider() → Box<dyn IntradayProvider>
```
`ProviderKind::Stockbit` implements quotes only: fundamentals are
`UNSUPPORTED`, the MSN-only capabilities say "requires --provider msn", and
`auto` history falls back to Yahoo.
## Ownership Module (SQLite-backed)
Unlike the `stocks` module (live-fetch), ownership is **import-then-query**:
@ -239,7 +282,9 @@ CLI flags > environment variables > config file > defaults
bidirectional overrides replaced before it reaches a terminal table.
- Exit codes by class: `0` success, `2` `INVALIDINPUT`/`CONFIGERROR` and clap usage
errors, `3` `SYMBOLNOTFOUND`/`NOMARKETDATA`/`CACHEMISS`, `4` `HTTP`/`RATELIMITED`/
`PROVIDERUNAVAILABLE`/`OFFLINE`/`AUTHERROR`, `5` `UNSUPPORTED`, `1` everything else
`PROVIDERUNAVAILABLE`/`OFFLINE`/`AUTHERROR`/`UPSTREAMBLOCKED`, `5` `UNSUPPORTED`,
`1` everything else. `UPSTREAMBLOCKED` is an upstream refusing this client
outright (a CDN 403 page); retrying at once does not help.
- A totally failed `stocks compare` emits one structured error in JSON mode; per-symbol warnings are retained for partial success, not printed ahead of a total-failure error.
## CLI Output and Offline Cache

View file

@ -184,7 +184,7 @@ is another 20-30% smaller than pretty JSON.
| 1 | other failure | `IO`, `DATABASEERROR`, `PARSEERROR`, ... |
| 2 | invalid usage or input | clap errors, `INVALIDINPUT`, `CONFIGERROR` |
| 3 | not found or no data | `SYMBOLNOTFOUND`, `NOMARKETDATA`, `CACHEMISS` |
| 4 | upstream unavailable, retry later | `HTTP`, `RATELIMITED`, `PROVIDERUNAVAILABLE`, `OFFLINE`, `AUTHERROR` |
| 4 | upstream unavailable or blocked, retry later | `HTTP`, `RATELIMITED`, `PROVIDERUNAVAILABLE`, `OFFLINE`, `AUTHERROR`, `UPSTREAMBLOCKED` |
| 5 | unsupported | `UNSUPPORTED` |
- JSON keeps today's top-level shape: no `{data, meta}` wrapper, so `jq` paths

View file

@ -42,12 +42,12 @@ contents. The native macOS installer job runs separately to exercise BSD tools.
## Live drift check
Fixtures cannot tell when MSN, Yahoo, or IDX change their responses.
Fixtures cannot tell when MSN, Yahoo, IPOT, IDX, or Stockbit change their responses.
`scripts/live-smoke.sh` runs shipped commands against the real upstreams and
validates each JSON output shape. It is opt-in and never part of the gate.
```bash
nix develop --command scripts/live-smoke.sh # stocks + ownership
nix develop --command scripts/live-smoke.sh # stocks + stockbit + ownership
nix develop --command scripts/live-smoke.sh --group stocks --symbol BBRI
nix develop --command scripts/live-smoke.sh --bin ./result/bin/idx
nix develop --command scripts/live-smoke.sh --compare ./old/bin/idx # A/B against an older build
@ -58,10 +58,16 @@ scripts/live-smoke.sh --dry-run
`flow` and `flow --source idx` on the new binary only, so an older `--compare` build that predates
them is fine), plus `compare BUMI,ADRO,AIMS`,
whose MSN key ratios contain non-finite values.
- `stockbit`: `intraday` for the ticker and `^JKSE`, and
`--provider stockbit stocks quote`, on the new binary only. A run outside the
session sees `market_status: close` and no bid/offer.
- `ownership`: `discover`, then imports the newest supported report into a temp
DB and lists `releases`. IDX sits behind Cloudflare and returns `403` to many
datacenter IPs, including GitHub-hosted runners; run it from a network IDX accepts.
A case whose upstream refuses us (`UPSTREAMBLOCKED`, an HTTP 403) prints
`BLOCKED (HTTP 403)` and counts as a failure, never as a pass.
`--compare <old-idx>` runs every case with both binaries back to back, each
with its own config and cache, and fails on a different exit code, any added,
missing, or retyped JSON key, or changed values for `profile`, `financials`,