idx-cli/docs/ARCHITECTURE.md

Architecture

Crates

The repo is a Cargo workspace with one shared version:

• crates/idx-core (library): the Idx entry point with market(), financial(), analysis(), and investor() (cache policy and Fetched<T> in client.rs; ownership queries, import, sync, and discovery in investor.rs), providers (api/), analysis/, cache, config (CoreConfig), ownership/ (SQLite, parsers, discovery, snapshot sync), error, and runtime (quiet/info/warn). It owns the shared test fixtures in tests/fixtures/; the fixture-backed mock provider needs the testing feature.
• crates/idx-cli (binary idx): clap commands (cli/), output format and color resolution (config.rs, IdxConfig), and rendering (output/). Its tests/ hold the CLI integration and golden output suites.

crates/idx-cli/src/cli/ holds no fetching, caching, or database logic: each command parses arguments, makes one idx-core call, and renders the result.

Domain Map

┌─────────────────────────────────────────────────────────┐
│                 CLI Layer (idx-cli)                      │
│  main.rs → cli/stocks.rs, cli/ownership.rs, cli/...     │
│  Clap derive structs, command dispatch, arg validation   │
└──────────────────────┬──────────────────────────────────┘
                       │
        ┌──────────────┼──────────────┐
        │              │              │
┌───────▼───────┐ ┌────▼────┐ ┌──────▼──────┐
│  API Layer    │ │ Analysis│ │  Ownership  │
│  (providers)  │ │  Module │ │   Module    │
│               │ │         │ │             │
│ MarketData    │ │ techni- │ │ SQLite DB   │
│ Provider trait│ │ cal.rs  │ │ KSEI parser │
│               │ │ signals │ │ Bing client │
│ Yahoo (OHLCV) │ │ fund.rs │ │ entity res. │
│ MSN (rich)   │ │         │ │ FTS5 search │
└───────┬───────┘ └────┬────┘ └──────┬──────┘
        │              │              │
┌───────▼──────────────▼──────────────▼──────┐
│              Output Layer                    │
│  table.rs (comfy-table) │ json.rs (serde)   │
└─────────────────────────────────────────────┘

Provider Architecture

Dual Provider Model

• MSN Finance — default provider. Rich data: quotes, fundamentals, profile, earnings, financials, sentiment, insights, news, screener, and explicit price-only chart history for supported IDX windows.
• Yahoo Finance — default/auto history source. Reliable OHLCV data via /v8/finance/chart/.

MSN Endpoints

• https://assets.msn.com/service/: quotes, charts, equities, earnings, sentiment, screener
• https://api.msn.com/msn/v0/pages/finance/: key ratios, insights, news feed

The public API key embedded in MSN Money lives in crates/idx-core/src/api/msn/client.rs.

Hybrid History Strategy

When history_provider = auto (default):

1. Use Yahoo for history because it provides full OHLCV candles.
2. Keep logging when provider selection falls back from MSN to Yahoo.
3. Allow explicit --history-provider msn for supported MSN chart windows (1mo, 3mo, 1y with 1d interval).

MSN Charts are price-only for IDX. The CLI normalizes them into Ohlc rows by using the chart price as open/high/low/close and 0 volume. MSN candle dates use WIB (UTC+7); Yahoo candle dates use meta.gmtoffset, with WIB as the fallback.

Quote and Screener Data

• Quotes expose nullable as_of last-trade timestamps in WIB. Table output flags quotes whose last trade was more than seven days ago.
• change, change_pct, and volume are nullable when the provider omits them; JSON consumers must not assume these fields are always numeric.
• Listed MSN instruments without a price return NOMARKETDATA; unknown symbols still return SYMBOLNOTFOUND. Empty sentiment responses represent no votes.
• Performers, high-volume, and large-cap screens fetch the complete candidate list before local ranking and --limit. MSN ignores pageIndex for these lists, so the provider uses the response's count to request the full list when the initial batch is incomplete, and rejects a still-truncated response.
• Full-list screen caches use an all key, separate from older limited results.

Capability Gating

MarketDataProvider = QuoteProvider + FundamentalsProvider
HistoryProvider    = separate trait, not all providers implement

Factory functions:
  default_provider(kind) → Box<dyn MarketDataProvider>
  history_provider(kind, mode, verbose) → Result<(ProviderKind, Box<dyn HistoryProvider>)>

Ownership Module (SQLite-backed)

Unlike the stocks module (live-fetch), ownership is import-then-query:

1. idx ownership import — ETL pipeline: fetch XLSX/PDF → parse → normalize → load SQLite
2. idx ownership sync — install a maintained SQLite snapshot via manifest + checksum validation
3. All query commands read from local ~/.local/share/idx/ownership.db
4. Fully offline after import/sync

Data Sources

• KSEI — official ≥1% shareholder registry, published monthly by IDX. Through May 2026 it was a PDF announcement; from the 2026-05-29 report onward it is an XLSX workbook on the IDX "Data Kepemilikan Saham" page
• KSEI archive — monthly ZIP/TXT balance-position matrix, used as a local fallback/backstop import path
• Bing Finance — global institutional ownership. The schema, --source bing queries, and crates/idx-core/src/api/msn/bing.rs exist, but ownership import --fetch-bing is intentionally unsupported, so no ingest path populates Bing rows today

Discovery

idx ownership discover merges two IDX sources:

• the announcement API (GetAllAnnouncement) for the legacy PDF announcements
• the Data Kepemilikan Saham page (/id/perusahaan-tercatat/data-kepemilikan-saham/), whose server-rendered Nuxt payload lists every XLSX with a description such as Pemegang Saham di Atas 1% per 31 Agustus 2026; the as-of date comes from that text

Only the above-1% XLSX is supported; the daily above-5% and investor-type workbooks are listed as unsupported. Both URLs can be overridden for tests with IDX_OWNERSHIP_ANNOUNCEMENT_API_URL and IDX_OWNERSHIP_DATA_PAGE_URL.

IDX sits behind Cloudflare, which accepts some curl-impersonate TLS profiles and rejects others per path (for example curl_chrome142 gets 403 on the data page). IDX fetches therefore try several profiles until the response validates (JSON, page payload, %PDF, or PK zip magic). IDX_CURL_IMPERSONATE_BIN pins a single profile.

Parser Pipeline

KSEI XLSX → zip + quick-xml (sharedStrings + sheet1) → exact 12-column header check
  → KseiHoldingDraft (Excel serial dates, numeric shares, percent → bps)
  → SQLite INSERT (within transaction)

KSEI PDF → mutool stext (XML with coordinates) → quick-xml parse → KseiRawRow
  → normalize (ID locale numbers, dates, entity names) → KseiHolding
  → SQLite INSERT (within transaction)

KSEI archive ZIP/TXT → pipe-delimited balance-position rows
  → map investor-type/locality buckets into synthetic aggregate holders
  → SQLite INSERT (within transaction)

Data Flow Patterns

Live Query (stocks module)

CLI args → resolve symbol → provider.quote/history/fundamentals → render table/json

Import-Query (ownership module)

Import: XLSX/PDF/archive → parse → normalize → resolve entities → SQLite INSERT
Query:  CLI args → SQLite SELECT → render table/json (no network)

Configuration Precedence

CLI flags > environment variables > config file > defaults

Error Strategy

• IdxError enum (thiserror) with structured error codes
• Table mode: human-readable error on stderr
• JSON and JSONL modes: {"error": true, "code": "...", "message": "..."} on stderr, stdout empty. This includes argument errors (INVALIDINPUT, exit 2) when -o/--output, IDX_OUTPUT, or auto mode with stdout not a terminal selects JSON; --help and --version print normally.
• Untrusted text (provider strings, KSEI names) has control characters and bidirectional overrides replaced before it reaches a terminal table.
• Exit codes by class: 0 success, 2 INVALIDINPUT/CONFIGERROR and clap usage errors, 3 SYMBOLNOTFOUND/NOMARKETDATA/CACHEMISS, 4 HTTP/RATELIMITED/ PROVIDERUNAVAILABLE/OFFLINE/AUTHERROR, 5 UNSUPPORTED, 1 everything else
• A totally failed stocks compare emits one structured error in JSON mode; per-symbol warnings are retained for partial success, not printed ahead of a total-failure error.

CLI Output and Offline Cache

• Output format resolves as -o flag > IDX_OUTPUT > general.output > auto. auto is a table when stdout is a terminal and JSON otherwise (std::io::IsTerminal).
• JSON is pretty-printed only on a terminal; piped output is compact. jsonl prints each array item on its own line with the same key order as json, and a single object as one line.
• ANSI styling goes through output::style and is disabled when stdout is not a terminal or color is turned off.
• version, cache info, cache clear, and ownership resolve map|merge emit JSON objects when JSON output is selected.
• stocks quote and stocks compare require at least one symbol.
• Offline reads warn on stderr when serving cache entries past their TTL, including the cache fetch timestamp. --quiet suppresses these warnings. In JSON mode warnings and info are JSON lines, {"level": "warning", "message": "..."}, so stderr stays parseable. Staleness is not yet represented by a versioned JSON result envelope.