mirror of https://github.com/0xrsydn/idx-cli.git synced 2026-11-07 01:35:36 +00:00

hermes 2c85acb829 feat(ownership): import IDX above-1% XLSX holder register

Since June 2026 IDX publishes the monthly "Pemegang Saham di Atas 1%"
list as an XLSX workbook on the Data Kepemilikan Saham page instead of
a PDF announcement. Add a parser for it and accept `.xlsx` in
`ownership import --file`.

- src/ownership/xlsx.rs: zip + quick-xml reader (shared strings incl.
  rich text, sheet cells), exact 12-column header match so the daily
  above-5% and investor-type/classification workbooks are rejected,
  Excel serial dates, numeric shares, percent -> bps with float-noise
  rounding. Builds KseiHoldingDraft directly and reuses the entity
  normalizers, so investor types stay uppercase like the PDF path.
- Fixtures: trimmed excerpts of the real 2026-08-31 above-1% and
  2026-09-23 above-5% workbooks.
- Unit + CLI tests: import, ticker, releases, above-5% rejection.
- Docs: ARCHITECTURE parser pipeline, AGENTS ownership inputs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017a5Cv5Z1NzdYKgKnTiAF4E

2026-09-24 20:48:07 +00:00

5.1 KiB

Raw Blame History

Architecture

Domain Map

┌─────────────────────────────────────────────────────────┐
│                      CLI Layer                           │
│  main.rs → cli/stocks.rs, cli/ownership.rs, cli/...     │
│  Clap derive structs, command dispatch, arg validation   │
└──────────────────────┬──────────────────────────────────┘
                       │
        ┌──────────────┼──────────────┐
        │              │              │
┌───────▼───────┐ ┌────▼────┐ ┌──────▼──────┐
│  API Layer    │ │ Analysis│ │  Ownership  │
│  (providers)  │ │  Module │ │   Module    │
│               │ │         │ │             │
│ MarketData    │ │ techni- │ │ SQLite DB   │
│ Provider trait│ │ cal.rs  │ │ KSEI parser │
│               │ │ signals │ │ Bing client │
│ Yahoo (OHLCV) │ │ fund.rs │ │ entity res. │
│ MSN (rich)   │ │         │ │ FTS5 search │
└───────┬───────┘ └────┬────┘ └──────┬──────┘
        │              │              │
┌───────▼──────────────▼──────────────▼──────┐
│              Output Layer                    │
│  table.rs (comfy-table) │ json.rs (serde)   │
└─────────────────────────────────────────────┘

Provider Architecture

Dual Provider Model

MSN Finance — default provider. Rich data: quotes, fundamentals, profile, earnings, financials, sentiment, insights, news, screener, and explicit price-only chart history for supported IDX windows.
Yahoo Finance — default/auto history source. Reliable OHLCV data via /v8/finance/chart/.

Hybrid History Strategy

When history_provider = auto (default):

Use Yahoo for history because it provides full OHLCV candles.
Keep logging when provider selection falls back from MSN to Yahoo.
Allow explicit --history-provider msn for supported MSN chart windows (1mo, 3mo, 1y with 1d interval).

MSN Charts are price-only for IDX. The CLI normalizes them into Ohlc rows by using the chart price as open/high/low/close and 0 volume.

Capability Gating

MarketDataProvider = QuoteProvider + FundamentalsProvider
HistoryProvider    = separate trait, not all providers implement

Factory functions:
  default_provider(kind) → Box<dyn MarketDataProvider>
  history_provider(kind, mode, verbose) → Result<(ProviderKind, Box<dyn HistoryProvider>)>

Ownership Module (SQLite-backed)

Unlike the stocks module (live-fetch), ownership is import-then-query:

idx ownership import — ETL pipeline: fetch PDF/API → parse → normalize → load SQLite
idx ownership sync — install a maintained SQLite snapshot via manifest + checksum validation
All query commands read from local ~/.local/share/idx/ownership.db
Fully offline after import/sync

Data Sources

KSEI — official ≥1% shareholder registry, published monthly by IDX. Through May 2026 it was a PDF announcement; from the 2026-05-29 report onward it is an XLSX workbook on the IDX "Data Kepemilikan Saham" page
KSEI archive — monthly ZIP/TXT balance-position matrix, used as a local fallback/backstop import path
Bing Finance — global institutional ownership (REST API, quarterly)

Parser Pipeline

KSEI XLSX → zip + quick-xml (sharedStrings + sheet1) → exact 12-column header check
  → KseiHoldingDraft (Excel serial dates, numeric shares, percent → bps)
  → SQLite INSERT (within transaction)

KSEI PDF → mutool stext (XML with coordinates) → quick-xml parse → KseiRawRow
  → normalize (ID locale numbers, dates, entity names) → KseiHolding
  → SQLite INSERT (within transaction)

KSEI archive ZIP/TXT → pipe-delimited balance-position rows
  → map investor-type/locality buckets into synthetic aggregate holders
  → SQLite INSERT (within transaction)

Data Flow Patterns

Live Query (stocks module)

CLI args → resolve symbol → provider.quote/history/fundamentals → render table/json

Import-Query (ownership module)

Import: PDF/API → parse → normalize → resolve entities → SQLite INSERT
Query:  CLI args → SQLite SELECT → render table/json (no network)

Configuration Precedence

CLI flags > environment variables > config file > defaults

Error Strategy

IdxError enum (thiserror) with structured error codes
Table mode: human-readable error on stderr
JSON mode: {"error": true, "code": "...", "message": "..."}
Exit code 0 on success, non-zero on failure

5.1 KiB Raw Blame History