mirror of
https://github.com/0xrsydn/idx-cli.git
synced 2026-11-07 09:45:36 +00:00
425 lines
26 KiB
Markdown
425 lines
26 KiB
Markdown
# Architecture
|
|
|
|
## Crates
|
|
|
|
The repo is a Cargo workspace with one shared version:
|
|
|
|
- `crates/idx-core` (library): the `Idx` entry point with `market()`,
|
|
`financial()`, `analysis()`, and `investor()` (cache policy and `Fetched<T>` in
|
|
`client.rs`; ownership queries, import, sync, and discovery in `investor.rs`),
|
|
providers (`api/`), `analysis/`, `cache`, `config` (`CoreConfig`),
|
|
`ownership/` (SQLite, parsers, discovery, snapshot sync), `error`, and
|
|
`runtime` (quiet/info/warn). It owns the shared test fixtures in
|
|
`tests/fixtures/`; the fixture-backed mock provider needs the `testing` feature.
|
|
- `crates/idx-cli` (binary `idx`): clap commands (`cli/`), output format and
|
|
color resolution (`config.rs`, `IdxConfig`), and rendering (`output/`). Its
|
|
`tests/` hold the CLI integration and golden output suites.
|
|
|
|
`crates/idx-cli/src/cli/` holds no fetching, caching, or database logic: each
|
|
command parses arguments, makes one idx-core call, and renders the result.
|
|
|
|
## Domain Map
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────────────┐
|
|
│ CLI Layer (idx-cli) │
|
|
│ main.rs → cli/stocks.rs, cli/ownership.rs, cli/... │
|
|
│ Clap derive structs, command dispatch, arg validation │
|
|
└──────────────────────┬──────────────────────────────────┘
|
|
│
|
|
┌──────────────┼──────────────┐
|
|
│ │ │
|
|
┌───────▼───────┐ ┌────▼────┐ ┌──────▼──────┐
|
|
│ API Layer │ │ Analysis│ │ Ownership │
|
|
│ (providers) │ │ Module │ │ Module │
|
|
│ │ │ │ │ │
|
|
│ MarketData │ │ techni- │ │ SQLite DB │
|
|
│ Provider trait│ │ cal.rs │ │ KSEI parser │
|
|
│ │ │ signals │ │ Bing client │
|
|
│ Yahoo (OHLCV) │ │ fund.rs │ │ entity res. │
|
|
│ MSN (rich) │ │ │ │ FTS5 search │
|
|
└───────┬───────┘ └────┬────┘ └──────┬──────┘
|
|
│ │ │
|
|
┌───────▼──────────────▼──────────────▼──────┐
|
|
│ Output Layer │
|
|
│ table.rs (comfy-table) │ json.rs (serde) │
|
|
└─────────────────────────────────────────────┘
|
|
```
|
|
|
|
## Provider Architecture
|
|
|
|
### Dual Provider Model
|
|
- **MSN Finance** — default provider. Rich data: quotes, fundamentals, profile, earnings, financials, sentiment, insights, news, screener, and explicit price-only chart history for supported IDX windows.
|
|
- **Yahoo Finance** — default/auto history source. Reliable OHLCV data via `/v8/finance/chart/`.
|
|
- **IPOT (Indo Premier)** — broker summaries, whichever provider is selected
|
|
(`BrokerSummaryProvider`). See [Broker Summaries](#broker-summaries).
|
|
- **IDX (idx.co.id)** — the daily trading summary behind `stocks flow
|
|
--source idx` (`TradingSummaryProvider`). See [Foreign Flow](#foreign-flow).
|
|
- **Stockbit (public page)** — `stocks intraday` whichever provider is
|
|
selected (`IntradayProvider`), and quotes under the opt-in
|
|
`--provider stockbit`. See [Stockbit](#stockbit).
|
|
|
|
### MSN Endpoints
|
|
- `https://assets.msn.com/service/`: quotes, charts, equities, earnings,
|
|
sentiment, screener
|
|
- `https://api.msn.com/msn/v0/pages/finance/`: key ratios, insights, news feed
|
|
|
|
The public API key embedded in MSN Money lives in `crates/idx-core/src/api/msn/client.rs`.
|
|
|
|
### Hybrid History Strategy
|
|
When `history_provider = auto` (default):
|
|
1. Use Yahoo for history because it provides full OHLCV candles.
|
|
2. Keep logging when provider selection falls back from MSN to Yahoo.
|
|
3. Allow explicit `--history-provider msn` for supported MSN chart windows (`1mo`, `3mo`, `1y` with `1d` interval).
|
|
|
|
MSN Charts are price-only for IDX. The CLI normalizes them into `Ohlc` rows by
|
|
using the chart price as open/high/low/close and `0` volume.
|
|
MSN candle dates use WIB (UTC+7); Yahoo candle dates use `meta.gmtoffset`, with
|
|
WIB as the fallback.
|
|
|
|
### Quote and Screener Data
|
|
- Quotes expose nullable `as_of` last-trade timestamps in WIB, and
|
|
`behind_session` when `as_of` is before the latest weekday whose session has
|
|
ended (16:15 WIB). It is computed on return, not cached; MSN can lag the close
|
|
by hours, and on an exchange holiday every quote reads as behind. Only `.JK`
|
|
quotes, `--region id` screens and `stocks intraday` are marked; other exchanges keep other hours.
|
|
Table output notes behind quotes, and calls those over seven days old not
|
|
current.
|
|
- `change`, `change_pct`, and `volume` are nullable when the provider omits them;
|
|
JSON consumers must not assume these fields are always numeric.
|
|
- Listed MSN instruments without a price return `NOMARKETDATA`; unknown symbols
|
|
still return `SYMBOLNOTFOUND`. Empty sentiment responses represent no votes.
|
|
- Performers, high-volume, and large-cap screens fetch the complete candidate
|
|
list before local ranking and `--limit`. MSN ignores `pageIndex` for these
|
|
lists, so the provider uses the response's `count` to request the full list
|
|
when the initial batch is incomplete, and rejects a still-truncated response.
|
|
- Full-list screen caches use an `all` key, separate from older limited results.
|
|
|
|
### Broker Summaries
|
|
- `GET https://www.indopremier.com/module/saham/include/data-brokersummary.php`
|
|
takes `code`, `start`/`end` (`MM/DD/YYYY`), `fd` (`all|F|D`) and `board`
|
|
(`all|RG|TN|NG`). It needs no login and answers an HTML fragment: the top 10
|
|
buyers and top 10 sellers ranked by lots (gross, so a broker can be on both
|
|
sides), plus range totals. `api/ipot/parse.rs` parses it purely, with
|
|
html5ever (`scraper`), so it reads the cells, date inputs and footer a
|
|
browser would build: a hand-written reader kept disagreeing with HTML on
|
|
crafted markup in the 0.4.0 review. A table whose rows don't line up
|
|
(missing cells, ranks out of order, a broker after a blank row, a second
|
|
body) rejects the page, and the dates it echoes must match the request.
|
|
- Figures of a million and more arrive abbreviated (`1.6 M`); they are scaled
|
|
in decimal and the summary's `rounded` flag says so. `F. NVal` ignores the
|
|
investor and board filters, so `foreign_net_value` is null for a summary of
|
|
one board. With `fd=F` or `fd=D` the footer covers only that
|
|
side's buying (BBCA 2026-10-16: foreign 1.7 T and domestic 4.9 T of 6.6 T),
|
|
so `total_value` is null and the footer is `buy_value`. On all boards the
|
|
side's `sell_value` is its buying less the foreign net (foreign: 1.7 + 2.3
|
|
= 4.0 T, against IDX's 3.95 T regular) or plus it (domestic); with a board
|
|
filter it is null.
|
|
- An unknown code is a 200 with `Invalid stock code!` (`SYMBOLNOTFOUND`). The
|
|
server also answers spurious nginx 404s for seconds at a time, so the client
|
|
retries 404 and 5xx with backoff. A range without trades is an empty summary.
|
|
- Data is end of day: IDX hides broker codes during trading. Summaries that
|
|
ended before today are cached for 7 days, today's for 5 minutes. Broker
|
|
names come from `broker/codes.tsv` (IDX `GetBrokerCodeList`).
|
|
- Bandarmology (`analysis/bandar.rs`, method `top_concentration_v1`) is pure
|
|
over one summary: for the top 1/3/5/10 per side, the buyers' share of the
|
|
value bought minus the sellers' share of the value sold (for all investors,
|
|
buy value minus sell value in basis points of value traded), with thresholds ±500/±1500 bps taken from
|
|
fzn0x/idx. A tier with a missing figure has no reading; missing values are
|
|
never summed as zero. A change to the rules gets a new method name.
|
|
|
|
### Foreign Flow
|
|
- The default source is IPOT: per session, the all-investor broker summary
|
|
gives the value traded and the foreign one (`fd=F`) gives foreign buying
|
|
(its footer), the foreign net, and selling (buying less the net), all
|
|
boards, through the broker summary cache. IPOT's foreign net matched IDX's
|
|
estimate within 0.3 B on BBCA 2026-10-12 to 16. An empty all-investor
|
|
summary counts as no session.
|
|
- `--source idx`: `GET https://www.idx.co.id/primary/TradingSummary/GetStockSummary`
|
|
`?length=9999&start=0&date=YYYYMMDD` answers one JSON row per stock for the
|
|
session: prices, volume, value, `ForeignBuy`/`ForeignSell` (shares) and
|
|
non-regular volume and value. `Volume`, `Value` and the foreign figures are
|
|
the regular board only: IPOT's regular-board total for BBCA on 2026-10-16
|
|
matches `Value`, and in a week of every stock no foreign figure exceeded
|
|
regular volume. `non_regular_bps` is therefore over regular plus
|
|
non-regular value. A day without trading has no rows; rows dated
|
|
another day are dropped. `api/idx/parse.rs` parses it purely.
|
|
- Cloudflare rejects most TLS fingerprints (in October 2026 only
|
|
`curl_safari184` passed), so the request uses curl-impersonate's profile
|
|
fallback with a 30 s limit. A challenge page fails that profile, not the
|
|
parse.
|
|
- Each session is cached whole (every stock) by date: 30 days once past,
|
|
5 minutes for today. `Flow::foreign` walks weekdays without requesting
|
|
weekends; offline, `--days` skips uncached sessions while a `--from/--to`
|
|
range fails on one. `foreign_net_value` is net shares at the session's
|
|
average price (value over volume), an estimate.
|
|
|
|
### Stockbit
|
|
- `GET https://stockbit.com/symbol/{CODE}` needs no login. Next.js
|
|
server-renders the quote and today's chart into the page's `__NEXT_DATA__`
|
|
JSON; `api/stockbit/parse.rs` reads it purely. `props.pageProps.company`
|
|
is the quote (numbers mostly as strings, an index price as a float string),
|
|
`orderbook.{bid,offer}.price` the best bid and offer, `market_hour.status`
|
|
the market's state, and `status` `STATUS_ACTIVE|SUSPENDED|DELISTED`.
|
|
`chart.prices[]` holds one price per minute in WIB with no offset, after a
|
|
synthetic first row at the previous close; per-minute OHLC and volume are
|
|
empty. An unknown code is HTTP 404 with the 404 page.
|
|
- Stockbit's JSON API (`exodus.stockbit.com`) answers 401 without a user's
|
|
bearer token; it is used only with the user's own session (below).
|
|
- Bid and offer are kept only while `market_hour.status` is `open`, so a
|
|
closing book never reads as a live one. Observed 2026-10-23: `open` during
|
|
trading (bid/offer set), `break` from 11:30 on a Friday, `close` after
|
|
hours; any other value fails closed. In session the chart gained a point
|
|
every minute and the last trade was within a minute of the clock, although
|
|
Stockbit's terms call its data delayed at least 10 minutes. Order-book
|
|
sizes are not shown; they are shares (BBCA's best bid on 2026-10-23 was
|
|
1,357,500 on the page and 13,575 lots in the session API's book). `as_of`
|
|
comes from `date`/`time`
|
|
only when the stock traded (a suspended stock carries today's date).
|
|
`session_note` repeats what the page says (suspended, delisted) or "no
|
|
trades this session"; it is never inferred.
|
|
- CloudFront in front of the page answers a static 403 to some user agents
|
|
(`curl/`, `axios/`, `Go-http-client/`, empty) and does not fingerprint TLS,
|
|
so a plain ureq request with `idx-cli/<version> (+repo)` passes. A 403 is
|
|
`UPSTREAMBLOCKED` and is not retried. Requests are at least 1 s apart
|
|
across the user's idx processes (the last request's time sits in
|
|
`<cache>/stockbit.pace` under a file lock, so a shell loop of `idx` calls is
|
|
paced too), and snapshots are cached for 5 s (at most `quote_ttl`), which
|
|
coalesces an agent's quote-then-intraday burst across MCP calls.
|
|
- Stockbit's terms call its data delayed at least 10 minutes and for
|
|
personal use; frontends say so. Fixtures (`tests/fixtures/stockbit_*`) are
|
|
pages recorded 2026-10-22 after the close, cut to the fields read.
|
|
|
|
### Stockbit session (opt-in login)
|
|
- `idx stockbit login` stores the user's session from a pasted
|
|
`credentialStorage` cookie (or a bare refresh token) in
|
|
`stockbit-session.json`, written atomically with mode 0600, in
|
|
`auth::session_dir()`: the config directory, or on macOS the other default
|
|
location when the session or profile is there, because a shell with
|
|
`XDG_CONFIG_HOME=~/.config` and an app started from the Dock resolve to
|
|
different places (a custom `XDG_CONFIG_HOME` stands alone, which keeps
|
|
isolated runs off the user's session);
|
|
`IDX_STOCKBIT_REFRESH_TOKEN` / `IDX_STOCKBIT_ACCESS_TOKEN` fill fields the
|
|
file lacks. Stockbit publishes no OAuth client a third party can register,
|
|
and its sign-in is a first-party flow guarded by reCAPTCHA, so idx cannot
|
|
ask for a token itself; every way in ends with the cookie's two tokens.
|
|
- What the cookie is (read from Stockbit's web bundle, 2026-10-29): the web
|
|
app's persisted credential store, written by script, host-only on
|
|
`stockbit.com`, for 30 days, as percent-encoded
|
|
`{"state":{"access","refresh","user"},"version":0}`. Every sign-in
|
|
(password, Google or Apple, a new device's one-time code, recovery) and the
|
|
app's own `POST /login/refresh` go through one `setCredentials`, so the
|
|
cookie always holds the newest pair. A signed-out page writes the store
|
|
without tokens, so a cookie counts only when `auth::session_from_paste`
|
|
finds tokens in it. The app refreshes by itself when it loads with a stale
|
|
access token, which is why idx never leaves a session in a browser it does
|
|
not own.
|
|
- `api/stockbit/signin.rs` is the one sign-in flow; `idx stockbit login` waits
|
|
on it in the foreground, and idx-mcp's `stockbit_login` runs it on a thread
|
|
(`start_background`) that `stockbit_status` reports on (`sign_in`: waiting,
|
|
signed_in or failed, with the error's code). A thread does not outlive
|
|
`main`, so idx-mcp calls `signin::shutdown` when stdin closes or on SIGTERM
|
|
or Ctrl-C, which closes a waiting browser (SIGKILL cannot). `Method::Auto` uses the browser
|
|
unless the shell is over SSH or has no display, and falls back to the page
|
|
only when the browser cannot start (none found, or it exits at launch); a
|
|
sign-in that started and then timed out or was closed is an error. Both
|
|
ways end in `signin::accept`, which is `stockbit::login`: store, check with
|
|
one request, restore the previous session if refused. A stored session that
|
|
still works means nothing to sign in to: both frontends verify it with one
|
|
request first, which is the common case, because every refresh extends the
|
|
session.
|
|
- The browser way (`api/stockbit/browser.rs`): Chrome, Chromium, Edge, Brave
|
|
or Helium on `stockbit.com/login`, with a profile of its own at
|
|
`<session dir>/stockbit-browser` (mode 0700) and
|
|
`--remote-debugging-port=0`. A profile belongs to one browser (its cookies
|
|
use that browser's key, and an older Chromium may refuse a newer profile),
|
|
so `<profile>/idx-browser` records the executable and the next run reuses
|
|
it; `IDX_STOCKBIT_BROWSER` overrides that (and is the only browser the mock
|
|
provider will start). The
|
|
browser writes the port and WebSocket path to `<profile>/DevToolsActivePort`;
|
|
idx connects and polls `Storage.getCookies`. On a session `accept` takes, it
|
|
opens a blank tab, closes every other one (no Stockbit script may write the
|
|
cookie back), deletes `credentialStorage` by the domain and path the browser
|
|
reported, reads the cookies again to confirm it is gone, and asks the
|
|
browser to close. Stockbit's device cookies stay, so a later
|
|
sign-in needs no new one-time code. The window waits 15 minutes; closing it
|
|
or the browser exiting ends the wait at once. The password, one-time codes
|
|
and reCAPTCHA stay in the browser. Google refused its own sign-in in this
|
|
window on 2026-10-31 ("This browser or app may not be secure"), most likely
|
|
because of the debugging flag, so password sign-in is the supported way
|
|
here and Google users go through the page.
|
|
- The page way (`api/stockbit/loopback.rs`), for `--no-browser`, SSH, no
|
|
display or no browser: a page on 127.0.0.1 (7878 when free, so an `ssh -L`
|
|
forward stays the same) with a bookmarklet that reads the cookie in the
|
|
user's own browser and opens the page with it in the URL fragment, or a
|
|
clipboard button; a per-run secret guards the path, the page is `no-store`
|
|
with a strict CSP, and the server stops on success or after five minutes.
|
|
Piped stdin keeps the paste path. Sign-in progress goes to stderr (as JSON
|
|
lines with `-o json`), so stdout holds only the result.
|
|
- `api/stockbit/auth.rs` reads expiries from the JWTs (`exp`, unverified)
|
|
and refreshes an access token within an hour of expiry with
|
|
`POST /login/refresh` (bearer = refresh token), which rotates both tokens.
|
|
Because a refresh token is spent once, refreshes run under an exclusive
|
|
lock on `stockbit-session.lock` and re-read the file first, so concurrent
|
|
idx processes never spend the same token. A data request answered 401 (or
|
|
a JSON 403) refreshes once and retries once; a second refusal is
|
|
`AUTHERROR`. Tokens never reach output, errors, logs or cache keys
|
|
(`Session`'s `Debug` redacts them).
|
|
- `api/stockbit/exodus.rs` (endpoints and units mapped by fzn0x/idx, MIT;
|
|
its recordings are `tests/fixtures/stockbit_exodus_*`):
|
|
`/company-price-feed/v2/orderbook/companies/{CODE}` gives the depth ladder
|
|
(`volume` in shares, `que_num` orders), OHLC, value, frequency and foreign
|
|
buy/sell/net (kept only when `has_foreign_bs` is true: on 2026-11-03 in
|
|
session it was false and the figures were the previous session's);
|
|
`/company-price-feed/market-time/session` the market state of the `regular`
|
|
board (`STATE_NAME_SESSION_2` while trading, `STATE_NAME_BREAK`,
|
|
`STATE_NAME_MARKET_CLOSED`; cached 30 s per process; the `fca` board's own
|
|
state is not read); `/charts/{CODE}/daily`
|
|
daily candles for `1m`/`3m`/`1y` windows (volume in lots, returned in
|
|
shares). Requests share the public page's cross-process pacing.
|
|
- With a session, `--provider stockbit` quotes come from the order book (bid
|
|
and offer only in a `STATE_NAME_SESSION_<n>` state; session 2 was observed
|
|
on 2026-11-03, session 1 is assumed to match), `stocks orderbook` works, and `auto` history under the stockbit
|
|
provider uses Stockbit's daily windows (`5d` to `1y`, trimmed from `1m`,
|
|
`3m` or `1y`), Yahoo otherwise. Without one, quotes use the public page and
|
|
the order book is `AUTHERROR`.
|
|
- Corporate actions (`corporate/`, `api/stockbit/corpaction.rs`):
|
|
`/corpaction/{type}` per kind (`dividend`, `stock_dividend`, `bonus`,
|
|
`rightissue`, `warrant`, `stocksplit`, `reversesplit`, `rups`, `pubex`,
|
|
`ipo`, `tenderoffer`), with `?symbol=` for one stock's history or without
|
|
for the market (about a year back and what is announced ahead). Each type
|
|
has its own field prefix and the list key is not always the path's, so
|
|
entitlements are read by field suffix (`_cumdate`, `_exdate`, ...). One
|
|
request per kind, cached 6 hours under `stockbit-corpaction`
|
|
(`<kind>:<SYMBOL.JK>` or `<kind>:market`); `stocks actions` /
|
|
`get_corporate_actions` sort newest first, `stocks calendar` /
|
|
`get_corporate_calendar` filter by `CorporateAction::date` (the cum date
|
|
for entitlements) and sort oldest first. Endpoint notes and the wider
|
|
design: `docs/STOCKBIT.md`.
|
|
- Shareholder movements (`movements`, `api/stockbit/movement.rs`) read
|
|
`/insider/company/majorholder`: IDX insider disclosures and KSEI 5% holder
|
|
changes, 50 rows a page, filtered by Stockbit (`symbol`, `date_start`,
|
|
`date_end`, `insider` (holder id), `action_type`, `source_type`). Each page
|
|
is cached an hour under `stockbit-movements`, keyed by every filter and the
|
|
page. `ownership movements` and `get_shareholder_movements` read up to 10
|
|
pages; the KSEI register in SQLite is not involved.
|
|
|
|
### Capability Gating
|
|
```
|
|
MarketDataProvider = QuoteProvider + FundamentalsProvider
|
|
HistoryProvider = separate trait, not all providers implement
|
|
|
|
Factory functions:
|
|
default_provider(kind) → Box<dyn MarketDataProvider>
|
|
history_provider(source, verbose) → Box<dyn HistoryProvider> (source from history_source)
|
|
broker_provider() → Box<dyn BrokerSummaryProvider>
|
|
trading_summary_provider() → Box<dyn TradingSummaryProvider>
|
|
intraday_provider() → Box<dyn IntradayProvider>
|
|
orderbook_provider() → Box<dyn OrderBookProvider>
|
|
corporate_action_provider() → Box<dyn CorporateActionProvider>
|
|
shareholder_movement_provider() → Box<dyn ShareholderMovementProvider>
|
|
```
|
|
|
|
`ProviderKind::Stockbit` serves quotes (and, with a session, daily history):
|
|
fundamentals are `UNSUPPORTED`, the MSN-only capabilities say "requires
|
|
--provider msn", and `auto` history uses Yahoo unless a session covers the
|
|
window.
|
|
|
|
## Ownership Module (SQLite-backed)
|
|
|
|
Unlike the `stocks` module (live-fetch), ownership is **import-then-query**:
|
|
|
|
1. `idx ownership import` — ETL pipeline: fetch XLSX/PDF → parse → normalize → load SQLite
|
|
2. `idx ownership sync` — install a maintained SQLite snapshot via manifest + checksum validation
|
|
3. All query commands read from local `~/.local/share/idx/ownership.db`
|
|
4. Fully offline after import/sync
|
|
|
|
### Data Sources
|
|
- **KSEI** — official ≥1% shareholder registry, published monthly by IDX. Through May 2026 it was a PDF announcement; from the 2026-05-29 report onward it is an XLSX workbook on the IDX "Data Kepemilikan Saham" page
|
|
- **KSEI archive** — monthly ZIP/TXT balance-position matrix, used as a local fallback/backstop import path
|
|
- **Bing Finance** — global institutional ownership. The schema, `--source bing`
|
|
queries, and `crates/idx-core/src/api/msn/bing.rs` exist, but `ownership import --fetch-bing`
|
|
is intentionally unsupported, so no ingest path populates Bing rows today
|
|
|
|
### Discovery
|
|
`idx ownership discover` merges two IDX sources:
|
|
- the announcement API (`GetAllAnnouncement`) for the legacy PDF announcements
|
|
- the Data Kepemilikan Saham page (`/id/perusahaan-tercatat/data-kepemilikan-saham/`),
|
|
whose server-rendered Nuxt payload lists every XLSX with a description such as
|
|
`Pemegang Saham di Atas 1% per 31 Agustus 2026`; the as-of date comes from that text
|
|
|
|
Only the above-1% XLSX is `supported`; the daily above-5% and investor-type workbooks
|
|
are listed as `unsupported`. Both URLs can be overridden for tests with
|
|
`IDX_OWNERSHIP_ANNOUNCEMENT_API_URL` and `IDX_OWNERSHIP_DATA_PAGE_URL`.
|
|
|
|
IDX sits behind Cloudflare, which accepts some curl-impersonate TLS profiles and
|
|
rejects others per path (for example `curl_chrome142` gets 403 on the data page).
|
|
IDX fetches therefore try several profiles until the response validates (JSON, page
|
|
payload, `%PDF`, or `PK` zip magic). `IDX_CURL_IMPERSONATE_BIN` pins a single profile.
|
|
|
|
### Parser Pipeline
|
|
```
|
|
KSEI XLSX → zip + quick-xml (sharedStrings + sheet1) → exact 12-column header check
|
|
→ KseiHoldingDraft (Excel serial dates, numeric shares, percent → bps)
|
|
→ SQLite INSERT (within transaction)
|
|
|
|
KSEI PDF → mutool stext (XML with coordinates) → quick-xml parse → KseiRawRow
|
|
→ normalize (ID locale numbers, dates, entity names) → KseiHolding
|
|
→ SQLite INSERT (within transaction)
|
|
|
|
KSEI archive ZIP/TXT → pipe-delimited balance-position rows
|
|
→ map investor-type/locality buckets into synthetic aggregate holders
|
|
→ SQLite INSERT (within transaction)
|
|
```
|
|
|
|
## Data Flow Patterns
|
|
|
|
### Live Query (stocks module)
|
|
```
|
|
CLI args → resolve symbol → provider.quote/history/fundamentals → render table/json
|
|
```
|
|
|
|
### Import-Query (ownership module)
|
|
```
|
|
Import: XLSX/PDF/archive → parse → normalize → resolve entities → SQLite INSERT
|
|
Query: CLI args → SQLite SELECT → render table/json (no network)
|
|
```
|
|
|
|
## Configuration Precedence
|
|
```
|
|
CLI flags > environment variables > config file > defaults
|
|
```
|
|
|
|
## Error Strategy
|
|
- `IdxError` enum (thiserror) with structured error codes
|
|
- Table mode: human-readable error on stderr
|
|
- JSON and JSONL modes: `{"error": true, "code": "...", "message": "..."}` on stderr, stdout empty.
|
|
This includes argument errors (`INVALIDINPUT`, exit 2) when `-o`/`--output`,
|
|
`IDX_OUTPUT`, or auto mode with stdout not a terminal selects JSON; `--help`
|
|
and `--version` print normally.
|
|
- Untrusted text (provider strings, KSEI names) has control characters and
|
|
bidirectional overrides replaced before it reaches a terminal table.
|
|
- Exit codes by class: `0` success, `2` `INVALIDINPUT`/`CONFIGERROR` and clap usage
|
|
errors, `3` `SYMBOLNOTFOUND`/`NOMARKETDATA`/`CACHEMISS`, `4` `HTTP`/`RATELIMITED`/
|
|
`PROVIDERUNAVAILABLE`/`OFFLINE`/`AUTHERROR`/`UPSTREAMBLOCKED`, `5` `UNSUPPORTED`,
|
|
`1` everything else. `UPSTREAMBLOCKED` is an upstream refusing this client
|
|
outright (a CDN 403 page); retrying at once does not help.
|
|
- A totally failed `stocks compare` emits one structured error in JSON mode; per-symbol warnings are retained for partial success, not printed ahead of a total-failure error.
|
|
|
|
## CLI Output and Offline Cache
|
|
- Output format resolves as `-o` flag > `IDX_OUTPUT` > `general.output` > `auto`.
|
|
`auto` is a table when stdout is a terminal and JSON otherwise
|
|
(`std::io::IsTerminal`).
|
|
- JSON is pretty-printed only on a terminal; piped output is compact. `jsonl`
|
|
prints each array item on its own line with the same key order as `json`,
|
|
and a single object as one line.
|
|
- ANSI styling goes through `output::style` and is disabled when stdout is not a
|
|
terminal or color is turned off.
|
|
- `version`, `cache info`, `cache clear`, and `ownership resolve map|merge` emit
|
|
JSON objects when JSON output is selected.
|
|
- `stocks quote` and `stocks compare` require at least one symbol.
|
|
- Offline reads warn on stderr when serving cache entries past their TTL,
|
|
including the cache fetch timestamp. `--quiet` suppresses these warnings.
|
|
In JSON mode warnings and info are JSON lines,
|
|
`{"level": "warning", "message": "..."}`, so stderr stays parseable.
|
|
Staleness is not yet represented by a versioned JSON result envelope.
|