# Architecture ## Crates The repo is a Cargo workspace with one shared version: - `crates/idx-core` (library): the `Idx` entry point with `market()`, `financial()`, `analysis()`, and `investor()` (cache policy and `Fetched` in `client.rs`; ownership queries, import, sync, and discovery in `investor.rs`), providers (`api/`), `analysis/`, `cache`, `config` (`CoreConfig`), `ownership/` (SQLite, parsers, discovery, snapshot sync), `error`, and `runtime` (quiet/info/warn). It owns the shared test fixtures in `tests/fixtures/`; the fixture-backed mock provider needs the `testing` feature. - `crates/idx-cli` (binary `idx`): clap commands (`cli/`), output format and color resolution (`config.rs`, `IdxConfig`), and rendering (`output/`). Its `tests/` hold the CLI integration and golden output suites. `crates/idx-cli/src/cli/` holds no fetching, caching, or database logic: each command parses arguments, makes one idx-core call, and renders the result. ## Domain Map ``` ┌─────────────────────────────────────────────────────────┐ │ CLI Layer (idx-cli) │ │ main.rs → cli/stocks.rs, cli/ownership.rs, cli/... │ │ Clap derive structs, command dispatch, arg validation │ └──────────────────────┬──────────────────────────────────┘ │ ┌──────────────┼──────────────┐ │ │ │ ┌───────▼───────┐ ┌────▼────┐ ┌──────▼──────┐ │ API Layer │ │ Analysis│ │ Ownership │ │ (providers) │ │ Module │ │ Module │ │ │ │ │ │ │ │ MarketData │ │ techni- │ │ SQLite DB │ │ Provider trait│ │ cal.rs │ │ KSEI parser │ │ │ │ signals │ │ Bing client │ │ Yahoo (OHLCV) │ │ fund.rs │ │ entity res. │ │ MSN (rich) │ │ │ │ FTS5 search │ └───────┬───────┘ └────┬────┘ └──────┬──────┘ │ │ │ ┌───────▼──────────────▼──────────────▼──────┐ │ Output Layer │ │ table.rs (comfy-table) │ json.rs (serde) │ └─────────────────────────────────────────────┘ ``` ## Provider Architecture ### Dual Provider Model - **MSN Finance** — default provider. Rich data: quotes, fundamentals, profile, earnings, financials, sentiment, insights, news, screener, and explicit price-only chart history for supported IDX windows. - **Yahoo Finance** — default/auto history source. Reliable OHLCV data via `/v8/finance/chart/`. - **IPOT (Indo Premier)** — broker summaries, whichever provider is selected (`BrokerSummaryProvider`). See [Broker Summaries](#broker-summaries). - **IDX (idx.co.id)** — the daily trading summary behind `stocks flow --source idx` (`TradingSummaryProvider`). See [Foreign Flow](#foreign-flow). - **Stockbit (public page)** — `stocks intraday` whichever provider is selected (`IntradayProvider`), and quotes under the opt-in `--provider stockbit`. See [Stockbit](#stockbit). ### MSN Endpoints - `https://assets.msn.com/service/`: quotes, charts, equities, earnings, sentiment, screener - `https://api.msn.com/msn/v0/pages/finance/`: key ratios, insights, news feed The public API key embedded in MSN Money lives in `crates/idx-core/src/api/msn/client.rs`. ### Hybrid History Strategy When `history_provider = auto` (default): 1. Use Yahoo for history because it provides full OHLCV candles. 2. Keep logging when provider selection falls back from MSN to Yahoo. 3. Allow explicit `--history-provider msn` for supported MSN chart windows (`1mo`, `3mo`, `1y` with `1d` interval). MSN Charts are price-only for IDX. The CLI normalizes them into `Ohlc` rows by using the chart price as open/high/low/close and `0` volume. MSN candle dates use WIB (UTC+7); Yahoo candle dates use `meta.gmtoffset`, with WIB as the fallback. ### Quote and Screener Data - Quotes expose nullable `as_of` last-trade timestamps in WIB, and `behind_session` when `as_of` is before the latest weekday whose session has ended (16:15 WIB). It is computed on return, not cached; MSN can lag the close by hours, and on an exchange holiday every quote reads as behind. Only `.JK` quotes, `--region id` screens and `stocks intraday` are marked; other exchanges keep other hours. Table output notes behind quotes, and calls those over seven days old not current. - `change`, `change_pct`, and `volume` are nullable when the provider omits them; JSON consumers must not assume these fields are always numeric. - Listed MSN instruments without a price return `NOMARKETDATA`; unknown symbols still return `SYMBOLNOTFOUND`. Empty sentiment responses represent no votes. - Performers, high-volume, and large-cap screens fetch the complete candidate list before local ranking and `--limit`. MSN ignores `pageIndex` for these lists, so the provider uses the response's `count` to request the full list when the initial batch is incomplete, and rejects a still-truncated response. - Full-list screen caches use an `all` key, separate from older limited results. ### Broker Summaries - `GET https://www.indopremier.com/module/saham/include/data-brokersummary.php` takes `code`, `start`/`end` (`MM/DD/YYYY`), `fd` (`all|F|D`) and `board` (`all|RG|TN|NG`). It needs no login and answers an HTML fragment: the top 10 buyers and top 10 sellers ranked by lots (gross, so a broker can be on both sides), plus range totals. `api/ipot/parse.rs` parses it purely, with html5ever (`scraper`), so it reads the cells, date inputs and footer a browser would build: a hand-written reader kept disagreeing with HTML on crafted markup in the 0.4.0 review. A table whose rows don't line up (missing cells, ranks out of order, a broker after a blank row, a second body) rejects the page, and the dates it echoes must match the request. - Figures of a million and more arrive abbreviated (`1.6 M`); they are scaled in decimal and the summary's `rounded` flag says so. `F. NVal` ignores the investor and board filters, so `foreign_net_value` is null for a summary of one board. With `fd=F` or `fd=D` the footer covers only that side's buying (BBCA 2026-10-16: foreign 1.7 T and domestic 4.9 T of 6.6 T), so `total_value` is null and the footer is `buy_value`. On all boards the side's `sell_value` is its buying less the foreign net (foreign: 1.7 + 2.3 = 4.0 T, against IDX's 3.95 T regular) or plus it (domestic); with a board filter it is null. - An unknown code is a 200 with `Invalid stock code!` (`SYMBOLNOTFOUND`). The server also answers spurious nginx 404s for seconds at a time, so the client retries 404 and 5xx with backoff. A range without trades is an empty summary. - Data is end of day: IDX hides broker codes during trading. Summaries that ended before today are cached for 7 days, today's for 5 minutes. Broker names come from `broker/codes.tsv` (IDX `GetBrokerCodeList`). - Bandarmology (`analysis/bandar.rs`, method `top_concentration_v1`) is pure over one summary: for the top 1/3/5/10 per side, the buyers' share of the value bought minus the sellers' share of the value sold (for all investors, buy value minus sell value in basis points of value traded), with thresholds ±500/±1500 bps taken from fzn0x/idx. A tier with a missing figure has no reading; missing values are never summed as zero. A change to the rules gets a new method name. ### Foreign Flow - The default source is IPOT: per session, the all-investor broker summary gives the value traded and the foreign one (`fd=F`) gives foreign buying (its footer), the foreign net, and selling (buying less the net), all boards, through the broker summary cache. IPOT's foreign net matched IDX's estimate within 0.3 B on BBCA 2026-10-12 to 16. An empty all-investor summary counts as no session. - `--source idx`: `GET https://www.idx.co.id/primary/TradingSummary/GetStockSummary` `?length=9999&start=0&date=YYYYMMDD` answers one JSON row per stock for the session: prices, volume, value, `ForeignBuy`/`ForeignSell` (shares) and non-regular volume and value. `Volume`, `Value` and the foreign figures are the regular board only: IPOT's regular-board total for BBCA on 2026-10-16 matches `Value`, and in a week of every stock no foreign figure exceeded regular volume. `non_regular_bps` is therefore over regular plus non-regular value. A day without trading has no rows; rows dated another day are dropped. `api/idx/parse.rs` parses it purely. - Cloudflare rejects most TLS fingerprints (in October 2026 only `curl_safari184` passed), so the request uses curl-impersonate's profile fallback with a 30 s limit. A challenge page fails that profile, not the parse. - Each session is cached whole (every stock) by date: 30 days once past, 5 minutes for today. `Flow::foreign` walks weekdays without requesting weekends; offline, `--days` skips uncached sessions while a `--from/--to` range fails on one. `foreign_net_value` is net shares at the session's average price (value over volume), an estimate. ### Stockbit - `GET https://stockbit.com/symbol/{CODE}` needs no login. Next.js server-renders the quote and today's chart into the page's `__NEXT_DATA__` JSON; `api/stockbit/parse.rs` reads it purely. `props.pageProps.company` is the quote (numbers mostly as strings, an index price as a float string), `orderbook.{bid,offer}.price` the best bid and offer, `market_hour.status` the market's state, and `status` `STATUS_ACTIVE|SUSPENDED|DELISTED`. `chart.prices[]` holds one price per minute in WIB with no offset, after a synthetic first row at the previous close; per-minute OHLC and volume are empty. An unknown code is HTTP 404 with the 404 page. - Stockbit's JSON API (`exodus.stockbit.com`) answers 401 without a user's bearer token; it is used only with the user's own session (below). - Bid and offer are kept only while `market_hour.status` is `open`, so a closing book never reads as a live one. Observed 2026-10-23: `open` during trading (bid/offer set), `break` from 11:30 on a Friday, `close` after hours; any other value fails closed. In session the chart gained a point every minute and the last trade was within a minute of the clock, although Stockbit's terms call its data delayed at least 10 minutes. Order-book sizes are not shown; they are shares (BBCA's best bid on 2026-10-23 was 1,357,500 on the page and 13,575 lots in the session API's book). `as_of` comes from `date`/`time` only when the stock traded (a suspended stock carries today's date). `session_note` repeats what the page says (suspended, delisted) or "no trades this session"; it is never inferred. - CloudFront in front of the page answers a static 403 to some user agents (`curl/`, `axios/`, `Go-http-client/`, empty) and does not fingerprint TLS, so a plain ureq request with `idx-cli/ (+repo)` passes. A 403 is `UPSTREAMBLOCKED` and is not retried. Requests are at least 1 s apart across the user's idx processes (the last request's time sits in `/stockbit.pace` under a file lock, so a shell loop of `idx` calls is paced too), and snapshots are cached for 5 s (at most `quote_ttl`), which coalesces an agent's quote-then-intraday burst across MCP calls. - Stockbit's terms call its data delayed at least 10 minutes and for personal use; frontends say so. Fixtures (`tests/fixtures/stockbit_*`) are pages recorded 2026-10-22 after the close, cut to the fields read. ### Stockbit session (opt-in login) - `idx stockbit login` stores the user's session from a pasted `credentialStorage` cookie (or a bare refresh token) in `stockbit-session.json`, written atomically with mode 0600, in `auth::session_dir()`: the config directory, or on macOS the other default location when the session or profile is there, because a shell with `XDG_CONFIG_HOME=~/.config` and an app started from the Dock resolve to different places (a custom `XDG_CONFIG_HOME` stands alone, which keeps isolated runs off the user's session); `IDX_STOCKBIT_REFRESH_TOKEN` / `IDX_STOCKBIT_ACCESS_TOKEN` fill fields the file lacks. Stockbit publishes no OAuth client a third party can register, and its sign-in is a first-party flow guarded by reCAPTCHA, so idx cannot ask for a token itself; every way in ends with the cookie's two tokens. - What the cookie is (read from Stockbit's web bundle, 2026-10-29): the web app's persisted credential store, written by script, host-only on `stockbit.com`, for 30 days, as percent-encoded `{"state":{"access","refresh","user"},"version":0}`. Every sign-in (password, Google or Apple, a new device's one-time code, recovery) and the app's own `POST /login/refresh` go through one `setCredentials`, so the cookie always holds the newest pair. A signed-out page writes the store without tokens, so a cookie counts only when `auth::session_from_paste` finds tokens in it. The app refreshes by itself when it loads with a stale access token, which is why idx never leaves a session in a browser it does not own. - `api/stockbit/signin.rs` is the one sign-in flow; `idx stockbit login` waits on it in the foreground, and idx-mcp's `stockbit_login` runs it on a thread (`start_background`) that `stockbit_status` reports on (`sign_in`: waiting, signed_in or failed, with the error's code). The thread's warnings go into `sign_in.warning` through `runtime::on_warning`, since nobody reads idx-mcp's stderr, and the error that ends a browser wait names a refusal it hit. A thread does not outlive `main`, so idx-mcp calls `signin::shutdown` when stdin closes or on SIGTERM or Ctrl-C, which closes a waiting browser (SIGKILL cannot). `Method::Auto` uses the browser unless the shell is over SSH or has no display, and falls back to the page only when the browser cannot start (none found, or it exits at launch); a sign-in that started and then timed out or was closed is an error. Both ways end in `signin::accept`, which is `stockbit::login`: store, check with one request, restore the previous session if refused. A stored session that still works means nothing to sign in to: both frontends verify it with one request first, which is the common case, because every refresh extends the session. - The browser way (`api/stockbit/browser.rs`): Chrome, Chromium, Edge, Brave or Helium on `stockbit.com/login`, with a profile of its own at `/stockbit-browser` (mode 0700) and `--remote-debugging-port=0`. A profile belongs to one browser (its cookies use that browser's key, and an older Chromium may refuse a newer profile), so `/idx-browser` records the executable and the next run reuses it; `IDX_STOCKBIT_BROWSER` overrides that (and is the only browser the mock provider will start). The browser writes the port and WebSocket path to `/DevToolsActivePort`; idx connects and polls `Storage.getCookies`. On a session `accept` takes, it opens a blank tab, closes every other one (no Stockbit script may write the cookie back), deletes `credentialStorage` by the domain and path the browser reported, reads the cookies again to confirm it is gone, and asks the browser to close. Stockbit's device cookies stay, so a later sign-in needs no new one-time code. The window waits 15 minutes; closing it or the browser exiting ends the wait at once. The password, one-time codes and reCAPTCHA stay in the browser. Google refused its own sign-in in this window on 2026-10-31 ("This browser or app may not be secure"), most likely because of the debugging flag, so password sign-in is the supported way here and Google users go through the page. - The page way (`api/stockbit/loopback.rs`), for `--no-browser`, SSH, no display or no browser: a page on 127.0.0.1 (7878 when free, so an `ssh -L` forward stays the same) with a bookmarklet that reads the cookie in the user's own browser and opens the page with it in the URL fragment, or a clipboard button; a per-run secret guards the path, the page is `no-store` with a strict CSP, and the server stops on success or after five minutes. Piped stdin keeps the paste path. Sign-in progress goes to stderr (as JSON lines with `-o json`), so stdout holds only the result. - `api/stockbit/auth.rs` reads expiries from the JWTs (`exp`, unverified) and refreshes an access token within an hour of expiry with `POST /login/refresh` (bearer = refresh token), which rotates both tokens. Because a refresh token is spent once, refreshes run under an exclusive lock on `stockbit-session.lock` and re-read the file first, so concurrent idx processes never spend the same token. A data request answered 401 (or a JSON 403) refreshes once and retries once; a second refusal is `AUTHERROR`. Tokens never reach output, errors, logs or cache keys (`Session`'s `Debug` redacts them). - `api/stockbit/exodus.rs` (endpoints and units mapped by fzn0x/idx, MIT; its recordings are `tests/fixtures/stockbit_exodus_*`): `/company-price-feed/v2/orderbook/companies/{CODE}` gives the depth ladder (`volume` in shares, `que_num` orders), OHLC, value, frequency and foreign buy/sell/net (kept only when `has_foreign_bs` is true: on 2026-11-03 in session it was false and the figures were the previous session's); `/company-price-feed/market-time/session` the market state of the `regular` board (`STATE_NAME_SESSION_2` while trading, `STATE_NAME_BREAK`, `STATE_NAME_MARKET_CLOSED`; cached 30 s per process; the `fca` board's own state is not read); `/charts/{CODE}/daily` daily candles for `1m`/`3m`/`1y` windows (volume in lots, returned in shares). Requests share the public page's cross-process pacing. - With a session, `--provider stockbit` quotes come from the order book (bid and offer only in a `STATE_NAME_SESSION_` state; session 2 was observed on 2026-11-03, session 1 is assumed to match), `stocks orderbook` works, and `auto` history under the stockbit provider uses Stockbit's daily windows (`5d` to `1y`, trimmed from `1m`, `3m` or `1y`), Yahoo otherwise. Without one, quotes use the public page and the order book is `AUTHERROR`. - Corporate actions (`corporate/`, `api/stockbit/corpaction.rs`): `/corpaction/{type}` per kind (`dividend`, `stock_dividend`, `bonus`, `rightissue`, `warrant`, `stocksplit`, `reversesplit`, `rups`, `pubex`, `ipo`, `tenderoffer`), with `?symbol=` for one stock's history or without for the market (about a year back and what is announced ahead). Each type has its own field prefix and the list key is not always the path's, so entitlements are read by field suffix (`_cumdate`, `_exdate`, ...). One request per kind, cached 6 hours under `stockbit-corpaction` (`:` or `:market`); `stocks actions` / `get_corporate_actions` sort newest first, `stocks calendar` / `get_corporate_calendar` keep what `CalendarRange::includes`: a `CorporateAction::date` (the cum date for entitlements) in the range, or a rights, tender or IPO `period` that overlaps it (not a warrant's, which runs for years), sorted oldest first. Endpoint notes and the wider design: `docs/STOCKBIT.md`. ### Capability Gating ``` MarketDataProvider = QuoteProvider + FundamentalsProvider HistoryProvider = separate trait, not all providers implement Factory functions: default_provider(kind) → Box history_provider(source, verbose) → Box (source from history_source) broker_provider() → Box trading_summary_provider() → Box intraday_provider() → Box orderbook_provider() → Box corporate_action_provider() → Box ``` `ProviderKind::Stockbit` serves quotes (and, with a session, daily history): fundamentals are `UNSUPPORTED`, the MSN-only capabilities say "requires --provider msn", and `auto` history uses Yahoo unless a session covers the window. ## Ownership Module (SQLite-backed) Unlike the `stocks` module (live-fetch), ownership is **import-then-query**: 1. `idx ownership import` — ETL pipeline: fetch XLSX/PDF → parse → normalize → load SQLite 2. `idx ownership sync` — install a maintained SQLite snapshot via manifest + checksum validation 3. All query commands read from local `~/.local/share/idx/ownership.db` 4. Fully offline after import/sync ### Data Sources - **KSEI** — official ≥1% shareholder registry, published monthly by IDX. Through May 2026 it was a PDF announcement; from the 2026-05-29 report onward it is an XLSX workbook on the IDX "Data Kepemilikan Saham" page - **KSEI archive** — monthly ZIP/TXT balance-position matrix, used as a local fallback/backstop import path - **Bing Finance** — global institutional ownership. The schema, `--source bing` queries, and `crates/idx-core/src/api/msn/bing.rs` exist, but `ownership import --fetch-bing` is intentionally unsupported, so no ingest path populates Bing rows today ### Discovery `idx ownership discover` merges two IDX sources: - the announcement API (`GetAllAnnouncement`) for the legacy PDF announcements - the Data Kepemilikan Saham page (`/id/perusahaan-tercatat/data-kepemilikan-saham/`), whose server-rendered Nuxt payload lists every XLSX with a description such as `Pemegang Saham di Atas 1% per 31 Agustus 2026`; the as-of date comes from that text Only the above-1% XLSX is `supported`; the daily above-5% and investor-type workbooks are listed as `unsupported`. Both URLs can be overridden for tests with `IDX_OWNERSHIP_ANNOUNCEMENT_API_URL` and `IDX_OWNERSHIP_DATA_PAGE_URL`. IDX sits behind Cloudflare, which accepts some curl-impersonate TLS profiles and rejects others per path (for example `curl_chrome142` gets 403 on the data page). IDX fetches therefore try several profiles until the response validates (JSON, page payload, `%PDF`, or `PK` zip magic). `IDX_CURL_IMPERSONATE_BIN` pins a single profile. ### Parser Pipeline ``` KSEI XLSX → zip + quick-xml (sharedStrings + sheet1) → exact 12-column header check → KseiHoldingDraft (Excel serial dates, numeric shares, percent → bps) → SQLite INSERT (within transaction) KSEI PDF → mutool stext (XML with coordinates) → quick-xml parse → KseiRawRow → normalize (ID locale numbers, dates, entity names) → KseiHolding → SQLite INSERT (within transaction) KSEI archive ZIP/TXT → pipe-delimited balance-position rows → map investor-type/locality buckets into synthetic aggregate holders → SQLite INSERT (within transaction) ``` ## Data Flow Patterns ### Live Query (stocks module) ``` CLI args → resolve symbol → provider.quote/history/fundamentals → render table/json ``` ### Import-Query (ownership module) ``` Import: XLSX/PDF/archive → parse → normalize → resolve entities → SQLite INSERT Query: CLI args → SQLite SELECT → render table/json (no network) ``` ## Configuration Precedence ``` CLI flags > environment variables > config file > defaults ``` ## Error Strategy - `IdxError` enum (thiserror) with structured error codes - Table mode: human-readable error on stderr - JSON and JSONL modes: `{"error": true, "code": "...", "message": "..."}` on stderr, stdout empty. This includes argument errors (`INVALIDINPUT`, exit 2) when `-o`/`--output`, `IDX_OUTPUT`, or auto mode with stdout not a terminal selects JSON; `--help` and `--version` print normally. - Untrusted text (provider strings, KSEI names) has control characters and bidirectional overrides replaced before it reaches a terminal table. - Exit codes by class: `0` success, `2` `INVALIDINPUT`/`CONFIGERROR` and clap usage errors, `3` `SYMBOLNOTFOUND`/`NOMARKETDATA`/`CACHEMISS`, `4` `HTTP`/`RATELIMITED`/ `PROVIDERUNAVAILABLE`/`OFFLINE`/`AUTHERROR`/`UPSTREAMBLOCKED`, `5` `UNSUPPORTED`, `1` everything else. `UPSTREAMBLOCKED` is an upstream refusing this client outright (a CDN 403 page); retrying at once does not help. - A totally failed `stocks compare` emits one structured error in JSON mode; per-symbol warnings are retained for partial success, not printed ahead of a total-failure error. ## CLI Output and Offline Cache - Output format resolves as `-o` flag > `IDX_OUTPUT` > `general.output` > `auto`. `auto` is a table when stdout is a terminal and JSON otherwise (`std::io::IsTerminal`). - JSON is pretty-printed only on a terminal; piped output is compact. `jsonl` prints each array item on its own line with the same key order as `json`, and a single object as one line. - ANSI styling goes through `output::style` and is disabled when stdout is not a terminal or color is turned off. - `version`, `cache info`, `cache clear`, and `ownership resolve map|merge` emit JSON objects when JSON output is selected. - `stocks quote` and `stocks compare` require at least one symbol. - Offline reads warn on stderr when serving cache entries past their TTL, including the cache fetch timestamp. `--quiet` suppresses these warnings. In JSON mode warnings and info are JSON lines, `{"level": "warning", "message": "..."}`, so stderr stays parseable. Staleness is not yet represented by a versioned JSON result envelope.