mirror of https://github.com/0xrsydn/idx-cli.git synced 2026-11-07 09:45:36 +00:00

hermes 29e416109c feat(ownership): discover above-1% XLSX on the IDX data page

IDX stopped posting the monthly above-1% list as an announcement after
the 2026-05-29 report; June/July/August are only on the Data
Kepemilikan Saham page. `ownership discover` could not see them, and
IDX's Cloudflare now rejects curl_chrome142 on that page and on the
/Media/ XLSX files.

- remote: parse the page's Nuxt payload (Description/Prospectus pairs,
  / escapes), classify above-1% XLSX as supported and above-5% /
  investor-type workbooks as unsupported, derive as_of_date from the
  Indonesian description. Keep all above-1% entries, only the newest
  of the other series. Merge with the announcement results.
- discover JSON gains `format` (xlsx|pdf) and `as_of_date`; page
  entries use the as-of date as publish_date (the page shows none).
- import --url accepts .xlsx and downloads via the XLSX validator.
- curl_impersonate: run_owned_with_fallback tries several browser
  profiles until the body validates (JSON / page payload / %PDF / PK);
  IDX_CURL_IMPERSONATE_BIN still pins one.
- Tests: page fixture parsing, date parsing, payload validation, CLI
  discover via a local page server; existing discover tests now point
  IDX_OWNERSHIP_DATA_PAGE_URL at a closed port so no test hits IDX.
- Docs: ARCHITECTURE discovery section, AGENTS ownership inputs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017a5Cv5Z1NzdYKgKnTiAF4E

2026-09-24 21:09:05 +00:00

3.4 KiB

Raw Blame History

AGENTS.md

Project

idx-cli is a Rust CLI for Indonesian stock market analysis and ownership workflows.

The repo currently has two main product areas:

stocks: live market data and analysis
ownership: import/sync once, then query locally from SQLite

Stack

Rust stable
clap 4 for CLI
ureq 3 for HTTP
comfy-table and owo-colors for output
rusqlite + bundled SQLite/FTS5 for ownership
TOML config in ~/.config/idx/config.toml
file cache in ~/.cache/idx/

Source Map

src/
├── main.rs
├── cli/
│   ├── stocks.rs
│   ├── ownership.rs
│   ├── config.rs
│   └── cache.rs
├── api/
│   ├── mod.rs
│   ├── types.rs
│   ├── yahoo/
│   └── msn/
├── analysis/
├── ownership/
│   ├── archive.rs
│   ├── db.rs
│   ├── entities.rs
│   ├── parser.rs
│   ├── remote.rs
│   ├── snapshot.rs
│   └── types.rs
├── output/
├── cache.rs
├── config.rs
└── error.rs

Provider Model

MSN is the primary/default provider for quotes, fundamentals, profile, earnings, financials, sentiment, insights, news, and screener data.
Yahoo is the fallback provider for history/OHLCV because MSN history for IDX is still not supported.
Current config knobs:
- IDX_PROVIDER=msn|yahoo
- IDX_HISTORY_PROVIDER=auto|yahoo|msn

Ownership Model

Ownership is local-first after bootstrap.

Preferred bootstrap/update path:

idx ownership sync
if no snapshot manifest is available: idx ownership discover then idx ownership import --url <xlsx-or-pdf-url>
local --file imports remain available for manual/fallback use

Ownership input paths:

primary remote source: IDX above1 holder register — monthly XLSX on the Data Kepemilikan Saham page (since the 2026-05-29 report), legacy PDF announcements before that; ownership discover checks both
maintained snapshot path: ownership sync
local file imports: IDX above-1% .xlsx (current format since June 2026), legacy .pdf, plus local archive .zip / .txt

Important scope note:

archive ZIP/TXT ingest is a fallback/backstop path, not the primary product ingest surface
ownership import --fetch-bing is still intentionally unsupported

Working Principles

Keep data access provider-driven where possible; avoid adding new ad hoc fetch paths at the CLI layer.
Prefer pure parse/normalize transforms over hidden state.
Use fixtures in tests; do not hit live network in automated tests.
Preserve the output contract: table to stdout, JSON with --output json, errors to stderr.
Treat ownership schema/query compatibility as important: releases, ticker, and changes should keep working across ingest paths.

Verification

Core verification:

nix develop
cargo build
cargo clippy -- -D warnings
cargo test

Smoke tooling:

scripts/live-smoke.sh
scripts/live-smoke.sh --mode mock
scripts/live-smoke.sh --mode full

Read First

Start with these repo docs before making changes:

FEATURE_SPEC.md — active implementation backlog and remaining core gaps
TODO.md — execution tracker and smoke notes
docs/ARCHITECTURE.md — provider and ownership flow
docs/OWNERSHIP_SYNC.md — snapshot sync contract
docs/SMOKE.md — reusable smoke commands
docs/CONVENTIONS.md — repo conventions

3.4 KiB Raw Blame History