From a36648fd89e23ae1a08e26e1700dc4187904c623 Mon Sep 17 00:00:00 2001 From: 0xrsydn Date: Mon, 30 Mar 2026 11:57:14 +0700 Subject: [PATCH 1/2] ownership: add IDX above1 discovery and import Refactor KSEI PDF parsing for the live above-1 holder-register layout, add IDX announcement discovery plus browser-impersonated PDF fetches, and update the ownership docs to mark Batch 1 complete and scope Batch 2 around above1 hardening and unsupported legacy inputs. --- .gitignore | 1 + FEATURE_SPEC.md | 541 +++++++------- TODO.md | 114 ++- src/api/yahoo/client.rs | 58 +- src/cli/ownership.rs | 112 ++- src/curl_impersonate.rs | 67 ++ src/main.rs | 1 + src/ownership/entities.rs | 2 +- src/ownership/mod.rs | 1 + src/ownership/parser.rs | 520 ++++++++++---- src/ownership/remote.rs | 664 ++++++++++++++++++ tests/cli.rs | 129 ++++ .../idx_announcement_kepemilikan.json | 48 ++ 13 files changed, 1755 insertions(+), 503 deletions(-) create mode 100644 src/curl_impersonate.rs create mode 100644 src/ownership/remote.rs create mode 100644 tests/fixtures/idx_announcement_kepemilikan.json diff --git a/.gitignore b/.gitignore index 8076db4..aa5c97a 100644 --- a/.gitignore +++ b/.gitignore @@ -2,6 +2,7 @@ node_modules/ target/ .direnv/ result +tmp/ # Internal docs (specs, research, business strategy) docs-internal/ diff --git a/FEATURE_SPEC.md b/FEATURE_SPEC.md index 8844dcb..5d3de1a 100644 --- a/FEATURE_SPEC.md +++ b/FEATURE_SPEC.md @@ -1,320 +1,249 @@ -# Feature Spec: MSN Finance Full Coverage +# Feature Spec: Current Coverage and Remaining Core Work -**Branch:** `feat/msn-full` -**Status:** Draft — pending review -**Reference:** `origin/dev/rubick` (Go implementation by rubick) +**Status:** Active - updated against the current repo state on 2026-03-28 +**Current focus:** Close the remaining core gaps before expanding into new market and watchlist surfaces +**Reference:** `origin/dev/rubick` (endpoint reference only, not the current implementation plan) --- -## Background +## Purpose -The Rust CLI currently supports two MSN endpoints: -- `Finance/Quotes` → `quote()` -- `api.msn.com/keyratios` → `fundamentals()` +This is no longer a "port MSN from zero" spec. -The rubick Go project (friend's scraper) demonstrates a much wider set of MSN Finance endpoints covering equities, financials, earnings, charts, sentiment, insights, and news — all using the same public API key. This spec defines the full porting roadmap from Go → Rust. +The CLI already ships MSN-backed support for: +- `profile` +- `financials` +- `earnings` +- `sentiment` +- `insights` +- `news` +- `screen` + +Use this document to track what is still open in core functionality. +Use `TODO.md` as the execution log and smoke-history record. + +--- + +## Verified Current State + +- Current automated coverage is `153` tests: `102` unit and `51` integration. +- Reusable smoke coverage exists via `scripts/live-smoke.sh`; command groups are documented in `docs/SMOKE.md`. +- The latest smoke notes in `TODO.md` report passing live table and JSON checks for all shipped `stocks` commands. +- Cache/offline parity, JSON startup-error handling, screener input validation, and the recent MSN output cleanups have already been completed. +- KSEI ownership import/query is now verified end-to-end from a local March 2026 PDF file into SQLite, with direct CLI reads through `ownership releases` and `ownership ticker`. +- Remote ownership ingestion should now be treated as an IDX PDF discovery-and-fetch problem first, not a hardcoded KSEI PDF URL problem: direct IDX announcement PDFs exist, but the hashed asset URL must be discovered from an IDX listing/announcement surface and fetched with browser-like behavior. +- The CLI now has an explicit `idx ownership discover` surface that enumerates the latest hashed BEI ownership report URLs. +- Live verification on `2026-03-29` confirmed that the parser-compatible discovered source is currently the `Pemegang Saham di atas 1% (KSEI)` `lamp1` attachment, and `idx ownership import --url` now works end to end for that discovered BEI PDF. +- The currently discoverable `above 5%` and `investor-type` BEI families remain different schemas and should now be treated as legacy / unsupported input outside the current holder-register parser contract. + +This means the main gap is no longer endpoint coverage. +The remaining work is architecture cleanup, a few correctness edge cases, and selective UX expansion on top of already-shipped commands. + +--- + +## Coverage Snapshot + +| Area | CLI | Status | Notes | +| --- | --- | --- | --- | +| Quotes | `idx stocks quote` | Implemented | Cached, smoke-tested, and covered by integration tests | +| History | `idx stocks history` | Partial | Works today via Yahoo/history-provider routing; explicit MSN history remains unsupported for IDX | +| Technical | `idx stocks technical` | Implemented | Uses the cached history path | +| Growth | `idx stocks growth` | Implemented | Shipped and exercised | +| Valuation | `idx stocks valuation` | Implemented | Shipped and exercised | +| Risk | `idx stocks risk` | Implemented | Shipped and exercised | +| Fundamental | `idx stocks fundamental` | Implemented | Shipped and exercised | +| Compare | `idx stocks compare` | Implemented | Shipped and exercised | +| Company profile | `idx stocks profile` | Implemented | Cache/offline parity and fixture-backed coverage are in place | +| Financial statements | `idx stocks financials` | Implemented with gaps | Output cleanup landed; statement filter flags and richer period controls are still missing | +| Earnings | `idx stocks earnings` | Implemented with gaps | History/forecast split is rendered; filter flags are still missing | +| Sentiment | `idx stocks sentiment` | Implemented | Fixture-backed CLI coverage exists | +| Insights | `idx stocks insights` | Implemented | Summary/highlights/risks/`last_updated` mapping was corrected and tested | +| News | `idx stocks news` | Implemented | Fixture-backed CLI coverage exists | +| Screener | `idx stocks screen` | Implemented with gaps | Validation landed; expression/preset workflow is still future work | +| MSN charts | `idx stocks history --history-provider msn` | Missing | Explicit MSN history still returns unsupported for IDX | +| KSEI ownership import/query | `idx ownership import --file`, `idx ownership import --url`, `idx ownership releases`, `idx ownership ticker` | Implemented with gaps | Local PDF import and SQLite-backed query flow are verified against the March 2026 KSEI release; remote IDX import now works for the discovered `above 1%` `lamp1` BEI attachment, while legacy `above 5%` and `investor-type` BEI report families still need explicit unsupported-input handling | +| Bing ownership CLI | `idx ownership import --fetch-bing` | Not implemented | Client groundwork exists, CLI import path is still deferred | + +--- + +## Resolved Since The Previous Revision + +The following items should no longer be treated as active backlog in this spec: + +- CLI truth-pass baseline for shipped `stocks` commands +- Unified cache, offline, stale-cache, and `--no-cache` handling across the core and MSN-only stock commands +- Rejection of the conflicting `--offline --no-cache` flag combination +- JSON-aware startup/config failures, not just runtime failures +- Validation for `stocks screen --filter` and `--region` +- `profile` output remapping to prefer company/localized fields +- `insights` output remapping for summary, highlights, risks, and `last_updated` +- Signed-number and table-label cleanup for `financials` +- Table-mode cleanup for `earnings` +- Baseline parser and CLI regression coverage for the shipped MSN-only command set +- KSEI ownership parser hardening for the March 2026 live PDF layout, including the merged `DATE + SHARE_CODE` segment and `D`/`A` locality markers +- Real KSEI ownership CLI verification from local file import into SQLite (`7261` rows across `955` tickers on `2026-03-28`) + +If any of the above regress, capture that in `TODO.md` as a new finding rather than reopening the old section here wholesale. + +--- + +## Remaining Core Gaps + +### P0 - Correctness and architecture + +#### 1. Unify provider and capability flow + +Current state: +- `src/cli/stocks.rs` still directly constructs `MsnProvider::new(false)` for `profile`, `financials`, `earnings`, `sentiment`, `insights`, `news`, and `screen`. +- This keeps a split execution path alive even though cache/offline behavior is now mostly unified around `fetch_msn_with_cache`. + +Why it matters: +- The architecture doc describes provider/capability-based flow, but the CLI still special-cases MSN-only commands at the handler layer. +- Future features will be harder to extend cleanly if this split remains. + +Done when: +- Normal command handlers stop constructing `MsnProvider` directly. +- Provider selection and capability checks are centralized and consistent with `docs/ARCHITECTURE.md`. + +#### 2. Fix screener row hygiene for incomplete data + +Current state: +- `src/api/msn/map.rs` still defaults missing screener price data to `0.0` when constructing `Quote` rows. + +Why it matters: +- A zero-priced row is not the same thing as a valid priced stock. +- This can silently admit incomplete market rows instead of rejecting or filtering them. + +Done when: +- Screener rows without usable price data are filtered or rejected explicitly. +- Regression tests cover the chosen behavior. + +#### 3. Decide the fundamentals fallback policy + +Current state: +- Fundamentals can still fall back to industry-level metrics when company metrics are absent. + +Why it matters: +- This can produce analysis that looks precise but is actually based on peer or category data. + +Decision needed: +- Allow the fallback and annotate it clearly, or +- reject the fallback and surface missing company data explicitly. + +Done when: +- The policy is explicit in code and reflected in output semantics. + +#### 4. Harden Yahoo reliability edge cases + +Open issues: +- Yahoo can still return `429` from datacenter IPs intermittently. +- SMA200 trend output can still show "Insufficient data" when fewer than `200` candles are returned. + +Done when: +- The retry/fallback story for Yahoo failures is deliberate and documented. +- SMA200 behavior is either improved or clearly documented as expected. + +#### 5. Automate IDX ownership source discovery and fetch + +Current state: +- `idx ownership import --file ` is now working and verified against the March 2026 KSEI release. +- Direct IDX announcement PDFs exist, but the monthly file path is not safely hardcodable because the asset filename is hashed. +- The official BEI listing page for this feed is `https://www.idx.co.id/id/berita/pengumuman/`, and the page’s own Nuxt client fetches announcement data from `GET /primary/NewsAnnouncement/GetAllAnnouncement`. +- The BEI endpoint is now reverse-engineered enough for `idx ownership discover` to locate the current `Pemegang Saham di atas 1% (KSEI)`, `Pemegang Saham di atas 5% (KSEI)`, and `Kepemilikan Saham Perusahaan Tercatat Berdasarkan Tipe Investor` hashed PDFs. +- Live verification on `2026-03-29` shows that the discoverable `above 1%` `lamp1` attachment matches the raw KSEI holder-register shape that the current parser imports successfully. +- Live verification on `2026-03-29` also shows that the currently discoverable `above 5%` and `investor-type` BEI families do not match the holder-register schema the current parser imports. +- Product direction as of `2026-03-30` is to standardize on the `above 1%` holder-register structure for remote import; the other discovered families are legacy inputs that should be rejected clearly rather than parsed. +- Plain `curl`/`ureq` requests to IDX-hosted PDFs still get `403` from Cloudflare, while `curl-impersonate` inside the project `nix develop` environment has already been verified to return a real PDF for a March 2026 ownership source URL. +- The KSEI archive remains a secondary upstream and currently exposes monthly ZIP files that can be used later as fallback or cross-check input, but it is no longer the primary roadmap target. + +Why it matters: +- The ownership feature now parses and stores live ownership data correctly from both local files and the currently discoverable `above 1%` BEI `lamp1` attachment. +- The remaining gap is not basic remote import anymore; it is hardening the supported `above 1%` path and clearly rejecting other discovered BEI schemas that do not match the holder-register contract. + +Done when: +- The CLI can discover the latest parser-compatible IDX ownership PDF URL from an IDX listing/announcement surface without hardcoded monthly paths. +- Remote fetches use the same browser-impersonation strategy already established elsewhere in the repo instead of the current bare `ureq` path. +- `idx ownership import --url` can fetch and parse the current `above 1%` IDX-hosted PDF reliably. +- Unsupported BEI report families are classified and rejected explicitly instead of failing later with a generic zero-row parse error. + +Roadmap: +1. Keep discovery and import flows centered on the parser-compatible `above 1%` family and its `lamp1` attachment. +2. Reuse the existing `curl-impersonate` pattern for browser-like PDF fetches. +3. Add schema classification / unsupported-input UX for the legacy `above 5%` and `investor-type` BEI PDFs. +4. Decide whether the default discover output should be `above1`-first, with legacy families retained only for diagnostic use. +5. Publish maintained SQLite snapshot artifacts and add `idx ownership sync` after remote IDX import is stable. +6. Optionally add KSEI ZIP/TXT ingest later as fallback or cross-check input. + +### P1 - UX and output contract cleanup + +Tasks: +- Add `financials` filters such as `--statement income|balance|cashflow`. +- Add `earnings` filters such as `--forecast|--history` and `--annual|--quarterly`. +- Review JSON payload consistency where symbol or context fields are still sparse. +- Decide whether `screen` stays under `stocks` long term or graduates into a richer dedicated surface later. + +Done when: +- Existing shipped commands are easier to drive without changing product scope. + +### P2 - Next feature work after the above is green + +Priority order: + +1. MSN Charts / `Finance/Charts` + - Reuse the existing `idx stocks history` command. + - Decide how to handle price-only timeframes safely. + +2. Bing ownership CLI integration + - Reuse the existing client groundwork in `src/api/msn/bing.rs`. + - Define the import shape and output contract for `idx ownership import --fetch-bing`. + +3. Ownership sync and snapshot distribution + - Keep IDX PDF discovery/fetch as the first milestone before starting this work. + - Publish maintained SQLite snapshot artifacts and add `idx ownership sync`. + - Treat the KSEI archive (`https://web.ksei.co.id/archive_download/holding_composition`) as fallback/backstop input, not the primary product ingest path. + +4. Richer financial statements + - Decide whether to stay with the current single-period model or add multi-period fetch support. + +5. New user-facing surfaces from `TODO.md` + - `market summary` + - `market movers` + - `market sectors` + - `screen query` + - `screen presets` + - `watchlist` + - `alerts` + +--- + +## Verification Gate + +Do not treat a core refactor or new provider feature as complete until all of the following are true: + +- `cargo build` passes +- `cargo clippy -- -D warnings` passes +- `cargo test` passes +- `scripts/live-smoke.sh --mode mock` passes +- the relevant live smoke groups pass for changed user-facing behavior +- `TODO.md` is updated with any new smoke finding, regression, or behavior change + +Keep the detailed reusable smoke commands in `docs/SMOKE.md`. +Do not duplicate per-run results in this spec. + +--- + +## Endpoint Reference Appendix MSN API key (public, embedded in MSN Money website): + ``` 0QfOX3Vn51YCzitbLaRkTTBadtWpgTN8NZLW0C1SEM ``` Base URLs: -- `https://assets.msn.com/service/` — core market data (Quotes, Charts, Equities, Earnings, Sentiment, Screener) -- `https://api.msn.com/msn/v0/pages/finance/` — extended data (keyratios, insights, newsfeed) -- `https://services.bingapis.com/contentservices-finance.hedgefunddataprovider/api/v1/` — Bing ownership data +- `https://assets.msn.com/service/` - core market data (Quotes, Charts, Equities, Earnings, Sentiment, Screener) +- `https://api.msn.com/msn/v0/pages/finance/` - extended data (key ratios, insights, news feed) +- `https://services.bingapis.com/contentservices-finance.hedgefunddataprovider/api/v1/` - Bing ownership data ---- - -## Endpoints to Implement - -### P0 — Core Completeness - -#### 1. `Finance/Equities` — Company Profile -- **Method:** GET -- **URL:** `{MSN_ASSETS_BASE_URL}Finance/Equities?apikey={key}&ids={id}&wrapodata=false` -- **Returns:** `EquityData` — company name, description, sector, industry, website, employees, address, officers/executives -- **CLI use:** `idx stock profile BBCA` or folded into `info` subcommand -- **Rust struct:** -```rust -pub struct EquityData { - pub id: String, - pub symbol: String, - pub short_name: String, - pub long_name: String, - pub description: String, - pub sector: String, - pub industry: String, - pub website: String, - pub employees: i64, - pub address: String, - pub city: String, - pub country: String, - pub phone: String, - pub officers: Vec, -} - -pub struct Officer { - pub name: String, - pub title: String, - pub age: Option, - pub year_born: Option, - pub total_pay: Option, -} -``` -- **Complexity:** Low - ---- - -#### 2. `Finance/Equities/financialstatements` — Financial Statements -- **Method:** GET -- **URL:** `{MSN_ASSETS_BASE_URL}Finance/Equities/financialstatements?apikey={key}&ids={id}&wrapodata=false` -- **Returns:** Balance sheet, cash flow, income statement — each as a map of `{field: value}` keyed by line item name, with period metadata (reportDate, endDate, currency, source) -- **CLI use:** `idx stock financials BBCA [--statement income|balance|cashflow]` -- **Note:** Fields are dynamic (map-based), not fixed columns — render as table with row=line item, col=period if multiple periods returned -- **Rust struct:** -```rust -pub struct FinancialStatements { - pub instrument: InstrumentInfo, - pub balance_sheet: Option, - pub cash_flow: Option, - pub income_statement: Option, -} - -pub struct BalanceSheet { - pub current_assets: HashMap, - pub long_term_assets: HashMap, - pub current_liabilities: HashMap, - pub equity: HashMap, - pub currency: String, - pub report_date: String, - pub end_date: String, -} - -// Similar pattern for CashFlow (financing/investing/operating) and IncomeStatement -``` -- **Complexity:** Medium (dynamic maps → table rendering) - ---- - -### P1 — High Analyst Value - -#### 3. `Finance/Events/Earnings` — Earnings History & Forecast -- **Method:** GET -- **URL:** `{MSN_ASSETS_BASE_URL}Finance/Events/Earnings?apikey={key}&ids={id}&wrapodata=false` -- **Returns:** - - `EpsLastYear`, `RevenueLastYear` - - `Forecast.annual` — 2 forward years: EpsForecast, RevenueForecast, GAAP/Normalized consensus - - `Forecast.quarterly` — next 4 quarters with same fields + EarningReleaseDate - - `History.annual` — 5 years: EpsActual, EpsSurprise, EpsSurprisePercent, RevenueActual, RevenueSurprise - - `History.quarterly` — ~12 quarters of actuals + surprises -- **CLI use:** `idx stock earnings BBCA [--forecast|--history] [--annual|--quarterly]` -- **Rust struct:** -```rust -pub struct EarningsReport { - pub eps_last_year: f64, - pub revenue_last_year: f64, - pub forecast: EarningsForecast, - pub history: EarningsHistory, -} - -pub struct EarningsData { - pub eps_actual: Option, - pub eps_forecast: Option, - pub eps_surprise: Option, - pub eps_surprise_pct: Option, - pub revenue_actual: Option, - pub revenue_forecast: Option, - pub revenue_surprise: Option, - pub earning_release_date: Option, - pub period_type: String, // e.g. "Q42025", "2025" -} -``` -- **Complexity:** Medium (nested map keyed by period string) - ---- - -#### 4. `Finance/Charts` — Price Chart / OHLCV History -- **Method:** GET -- **URL:** `{MSN_ASSETS_BASE_URL}Finance/Charts?apikey={key}&ids={id}&chartType={type}&wrapodata=false` -- **Chart types:** `1D`, `1W`, `1M`, `3M`, `6M`, `1Y`, `3Y`, `5Y`, `MAX` -- **Returns:** Series of `ChartPoint { time, open, high, low, close, price, volume }` -- **Note:** This unblocks the `history()` provider method — current implementation explicitly returns `Unsupported`. MSN charts don't guarantee OHLCV on all timeframes (1D is often price-only), so parse defensively. -- **CLI use:** `idx stock history BBCA --period 3M` (existing command, just needs this wired up) -- **Rust struct:** -```rust -pub struct ChartPoint { - pub time: String, - pub open: Option, - pub high: Option, - pub low: Option, - pub close: Option, - pub price: f64, - pub volume: Option, -} -``` -- **Complexity:** Medium (parse series array, handle missing OHLCV gracefully) - ---- - -### P2 — Enrichment Layer - -#### 5. `Finance/SentimentBrowser` — Crowd Sentiment -- **Method:** GET -- **URL:** `{MSN_ASSETS_BASE_URL}Finance/SentimentBrowser?apikey={key}&ids={id}&wrapodata=false` -- **Returns:** Per-period sentiment stats: bullish/bearish/neutral counts, time range name (e.g., "1D", "1W", "1M") -- **CLI use:** `idx stock sentiment BBCA` -- **Rust struct:** -```rust -pub struct SentimentData { - pub symbol: String, - pub statistics: Vec, -} - -pub struct SentimentPeriod { - pub time_range: String, // "1D", "1W", "1M" - pub bullish: i32, - pub bearish: i32, - pub neutral: i32, -} -``` -- **Complexity:** Low - ---- - -#### 6. `api.msn.com/insights` — AI-Generated Insights -- **Method:** GET -- **URL:** `{MSN_API_BASE_URL}insights?apikey={key}&ids={id}&wrapodata=false` -- **Returns:** Summary text, highlights array, risks array, last updated timestamp -- **CLI use:** `idx stock insights BBCA` -- **Rust struct:** -```rust -pub struct InsightData { - pub id: String, - pub summary: String, - pub highlights: Vec, - pub risks: Vec, - pub last_updated: String, -} -``` -- **Complexity:** Low - ---- - -#### 7. `MSN/Feed/me` — Stock News Feed -- **Method:** GET -- **URL:** `{MSN_API_BASE_URL}` + entity feed params with stock ID -- **Returns:** News cards: title, URL, abstract, provider name, publish time, read time -- **CLI use:** `idx stock news BBCA [--limit 10]` -- **Rust struct:** -```rust -pub struct NewsItem { - pub id: String, - pub title: String, - pub url: String, - pub description: String, - pub provider: String, - pub published_at: String, - pub read_time_min: Option, -} -``` -- **Complexity:** Medium (URL construction + response parsing needs rubick reference) - ---- - -#### 8. `Finance/Screener` — IDX Universe Screener -- **Method:** POST -- **URL:** `{MSN_ASSETS_BASE_URL}Finance/Screener?apikey={key}&wrapodata=false` -- **Body:** `{ filter: [{key, keyGroup, isRange}], order: {key, dir}, returnValueType: [...], screenerType: "...", limit: 50 }` -- **Returns:** List of stocks with quote data (price, change, market cap, volume, 52w hi/lo, YTD return) -- **CLI use:** `idx screen [--preset top-gainers|top-losers|most-active|...]` -- **Complexity:** Medium (POST body construction, preset filter definitions) - ---- - -### P3 — Optional / Future - -#### 9. Bing Ownership API — Institutional Holders -- **Base:** `https://services.bingapis.com/contentservices-finance.hedgefunddataprovider/api/v1/` -- **Endpoints:** - - `GetSecurityTopShareHolders` - - `GetSecurityTopBuyers` / `GetSecurityTopSellers` - - `GetSecurityTopNewShareHolders` / `GetSecurityTopExitedShareHolders` -- **CLI use:** `idx stock holders BBCA [--buyers|--sellers|--new|--exited]` -- **Note:** Separate base URL, may need different auth/headers than MSN. Validate working before implementing. -- **Complexity:** Low-Medium - ---- - -## Implementation Plan - -### Phase 1 — Extend `src/api/msn/` -1. Add `fetch_equities(symbol)` to `client.rs` -2. Add `fetch_financial_statements(symbol)` to `client.rs` -3. Add `fetch_earnings(symbol)` to `client.rs` -4. Add `fetch_charts(symbol, period)` to `client.rs` -5. Add corresponding parse functions to `parse.rs` -6. Expose via new methods on `MsnProvider` in `mod.rs` - -### Phase 2 — New Rust structs in `src/api/msn/types.rs` (new file) -- Extract shared types (currently inline in `parse.rs`) into dedicated `types.rs` -- Add all new structs listed above - -### Phase 3 — Wire CLI commands in `src/cli/stocks.rs` -New subcommands to add: -``` -idx stock profile # Company info + officers -idx stock financials # Income / balance / cashflow -idx stock earnings # EPS history + forecast -idx stock sentiment # Crowd sentiment -idx stock insights # AI highlights + risks -idx stock news # News feed -idx screen # IDX screener (separate top-level command) -``` - -And unblock existing: -``` -idx stock history # Wire MSN charts (currently Unsupported) -``` - -### Phase 4 — Output formatting -- Table output for financials (line item rows, period columns) -- Compact output for earnings (actual vs forecast vs surprise %) -- JSON output flag `--json` should work for all new commands - ---- - -## Open Questions - -1. **Chart OHLCV completeness** — rubick notes that MSN charts don't always return full OHLCV on short timeframes (e.g., 1D is price-only). Do we want to keep `history()` returning `Unsupported` for MSN and add a separate `charts()` method, or silently map price → close for compatibility? - -2. **Financial statements period count** — The API returns one period per call (most recent). Do we want to add a bulk-fetch loop (e.g., fetch last 4 quarters separately) or just expose single-period for now? - -3. **News feed URL construction** — needs exact param structure from rubick's `GetNewsFeed()` Go implementation. Worth a closer look before implementing. - -4. **Screener presets** — rubick defines filter key constants (e.g., `"st_list_topperfs"`, `"st_reg_id"`). Need to decide which presets to expose as CLI flags and what the default screener view looks like. - -5. **Provider trait extension** — `quote()`, `fundamentals()`, `history()` are currently defined on `Provider` trait. New methods (earnings, profile, etc.) are MSN-specific — do we extend the trait or expose them as inherent methods on `MsnProvider` only? - ---- - -## Files to Touch - -``` -src/api/msn/ - client.rs — add fetch_* methods - mod.rs — expose new provider methods - parse.rs — add parse_* functions - types.rs — NEW: shared type definitions - -src/cli/ - stocks.rs — add new subcommands + output formatting - -tests/ - cli.rs — integration tests for new commands - fixtures/ — add response fixtures for new endpoints -``` - ---- - -*Drafted by Ciphercat based on rubick Go implementation analysis + live MSN API verification.* +Keep this appendix for endpoint discovery and future work. +Use the sections above as the actual implementation plan. diff --git a/TODO.md b/TODO.md index 3acaac7..2957507 100644 --- a/TODO.md +++ b/TODO.md @@ -31,9 +31,83 @@ - [x] `stocks compare ` — side-by-side multi-symbol comparison - [x] `analysis/fundamental.rs` — fundamental analysis module (ported from idx-mcp) - [x] Yahoo quoteSummary endpoint parser (/v10/finance/quoteSummary) -- [x] 41 tests passing (22 unit + 19 integration) +- [x] 142 tests passing (94 unit + 48 integration) + +## ✅ Completed (2026-03-26 — hardening pass) +- [x] Unified cache, offline, stale-cache, and `--no-cache` behavior across core and MSN-only stock commands +- [x] Rejected the conflicting `--offline --no-cache` flag combination explicitly +- [x] Routed startup/config failures through the same JSON-aware error path as runtime failures +- [x] Validated `stocks screen --filter` and `--region`; invalid values now return errors +- [x] Added regression tests for startup JSON errors, MSN profile offline/stale-cache behavior, and screener validation + +## ✅ Completed (2026-03-26 — MSN output cleanup) +- [x] Re-mapped live MSN `profile` output to prefer localized/company fields for name, description, sector, industry, website, address, and phone +- [x] Reworked `stocks insights` output to derive a real summary, split highlights vs risks from evaluation status, and populate `last_updated` +- [x] Fixed signed-number formatting so `stocks financials` table output no longer mangles negative values +- [x] Added live-like fixtures and regression coverage for `profile`, `insights`, and signed table formatting ## 🚧 Next Up +- [x] Build a reusable live smoke script/checklist for all shipped CLI commands (`scripts/live-smoke.sh`, `docs/SMOKE.md`) +- [x] Re-run full live smoke for MSN-only commands after the cache/offline, mapping, and formatting fixes (`scripts/live-smoke.sh --mode full --group live-table --group live-json`) +- [x] Review remaining noisy table output in `financials` and `earnings` beyond the signed-number fix +- [x] Expand parser/CLI regression coverage for the rest of the MSN-only command set +- [x] Keep the live-smoke notes below in sync with real command output after each hardening pass + +## 🚧 Ownership Roadmap Reset (2026-03-29) + +### Batch 1 — IDX discovery + remote PDF import +- [x] Verify and document the IDX announcement/listing page that exposes the monthly ownership PDF link + - Verified official BEI listing page: `https://www.idx.co.id/id/berita/pengumuman/` + - Verified listing page JSON source used by the site: `GET /primary/NewsAnnouncement/GetAllAnnouncement?keywords=...` +- [x] Implement discovery/crawler logic for the hashed IDX PDF asset URL instead of hardcoding monthly paths +- [x] Expose discovery as an explicit `idx ownership discover` CLI surface so the live BEI feed can be inspected without coupling it to import +- [x] Extract a reusable browser-impersonated fetch helper for ownership downloads by reusing the repo's `curl-impersonate` pattern +- [x] Wire `ownership import --url` to the impersonated fetch path for IDX-hosted PDFs +- [x] Keep the current PDF parser/import path as the first production ingest route +- [x] Add tests and fixtures for IDX announcement-page parsing plus remote PDF fetch failure modes +- [x] Batch 1 verification: `cargo build` +- [x] Batch 1 verification: `cargo clippy -- -D warnings` +- [x] Batch 1 verification: `cargo test` +- [x] Batch 1 verification: fixture-backed parser/downloader tests cover announcement discovery, hashed URL extraction, and downloader failure cases +- [x] Batch 1 verification: live `idx ownership discover --family above1 --limit 2` in `nix develop` resolves the current BEI hashed URLs for the parser-compatible `above 1%` report family +- [x] Batch 1 verification: live end-to-end import succeeds for one discovered BEI ownership PDF URL +- [x] Batch 1 verification: end-to-end import into a temp ownership DB via `idx ownership import --url ...`, followed by `idx ownership releases` and one ticker query + - Resolved root cause on `2026-03-29`: + - the parser was already correct for the holder-register schema; the missing piece was discovery support for the `Pemegang Saham di atas 1% (KSEI)` family + - the parser-compatible source is the `lamp1` attachment `https://www.idx.co.id/StaticData/NewsAndAnnouncement/ANNOUNCEMENTSTOCK/From_EREP/202603/b9b638e5a8_8928aca255.pdf` + - the `above 5%` and `investor-type` BEI families remain different schemas; as of `2026-03-30`, they should be treated as legacy / unsupported input rather than new parser targets + +### Batch 2 — Above-1 hardening and unsupported-input UX +- [ ] Standardize the supported remote-import contract on the `Pemegang Saham di atas 1% (KSEI)` holder-register layout and its `lamp1` attachment shape +- [ ] Add BEI PDF schema classification before parse/import so `ownership import --url` can reject non-holder-register PDFs before the parser runs +- [ ] Improve CLI error messages and fallback behavior for discovery failure, fetch failure, invalid remote content, and known-but-unsupported legacy BEI schema variants +- [ ] Capture live-like fixtures for the current discoverable `investor-type` and `above 5%` BEI attachments (or their `mutool` `stext` extracts) so unsupported-input detection and failure UX are regression-tested +- [ ] Decide and document whether `ownership discover` should default to `above1` output while keeping legacy families available only for diagnostic use +- [ ] Decide and document whether `ownership import --url` accepts only direct PDF URLs or can also accept an IDX listing page as input +- [ ] Add regression coverage for Cloudflare/HTML responses, missing announcement links, duplicate release imports, and unsupported BEI schema detections +- [ ] Batch 2 verification: `cargo build` +- [ ] Batch 2 verification: `cargo clippy -- -D warnings` +- [ ] Batch 2 verification: `cargo test` +- [ ] Batch 2 verification: ownership-focused smoke checks cover successful remote import plus expected failure UX + +### Batch 3 — Snapshot publishing + sync +- [ ] Design maintained SQLite snapshot publishing after remote IDX import is stable +- [ ] Add `idx ownership sync` +- [ ] Define manifest/checksum/update semantics and local DB replacement rules +- [ ] Add regression coverage for manifest parsing, checksum validation, no-op sync, and forced refresh +- [ ] Batch 3 verification: `cargo build` +- [ ] Batch 3 verification: `cargo clippy -- -D warnings` +- [ ] Batch 3 verification: `cargo test` +- [ ] Batch 3 verification: sync installs into an empty temp data dir, preserves query behavior, and no-ops when already current + +### Batch 4 — KSEI ZIP/TXT fallback and cross-check path +- [ ] Keep KSEI ZIP/TXT ingest as fallback and validation/backstop work, not the first milestone +- [ ] Define whether the KSEI archive is only a maintainer fallback or a user-facing alternative import source +- [ ] Add cross-check coverage between IDX-PDF-derived output and KSEI-archive-derived output for at least one monthly release +- [ ] Batch 4 verification: `cargo build` +- [ ] Batch 4 verification: `cargo clippy -- -D warnings` +- [ ] Batch 4 verification: `cargo test` +- [ ] Batch 4 verification: fallback ingest produces a compatible SQLite state for `ownership releases`, `ticker`, and `changes` ## 📋 Backlog (per SPEC.md) - [ ] `market summary` — IHSG index, market breadth @@ -47,9 +121,47 @@ - [ ] CSV/TSV output formats - [ ] Additional providers (Alpha Vantage, Twelve Data, IDX official) +## 🔬 Latest Smoke Findings (2026-03-28) +- [x] Live smoke passed for shipped `stocks` commands: `quote`, `history`, `technical`, `growth`, `valuation`, `risk`, `fundamental`, `compare`, `profile`, `financials`, `earnings`, `sentiment`, `insights`, `news`, `screen` +- [x] Yahoo routing verified for live `quote` and `history` +- [x] `stocks history --history-provider msn` correctly fails for IDX as unsupported +- [x] `ownership releases` works with a writable `ownership.db_path` and an empty DB +- [x] Regression coverage now verifies offline/cache parity for MSN-only commands (`stocks profile BBCA`) +- [x] `--offline --no-cache` now fails fast as an invalid flag combination instead of serving cache +- [x] Config/startup failures now honor the JSON error contract (`IDX_PROVIDER=bogus idx -o json version`) +- [x] Invalid `stocks screen --filter/--region` values now return validation errors +- [x] Added reusable smoke runner/checklist for shipped CLI surfaces (`scripts/live-smoke.sh`, `docs/SMOKE.md`) +- [x] Reusable smoke runner passes on deterministic mock suites: `general`/`cache`/`routing`/`errors`/`ownership` = 32/32 and shipped `stocks` mock matrix = 30/30 (`tmp/live-smoke/20260326-194907`, `tmp/live-smoke/20260326-194914`) +- [x] Full live MSN-only smoke now passes via runner for both table and JSON surfaces: 30/30 (`tmp/live-smoke/20260326-195853`) +- [x] Targeted no-cache live checks confirm `profile`, `insights`, and `financials` fixes against real MSN responses +- [x] Live `stocks profile BBCA` no-cache output now populates company/localized fields instead of the sparse top-level fallback +- [x] Live `stocks insights BBCA` JSON now returns a mixed-signal summary plus non-empty `last_updated` +- [x] Live `stocks financials BBCA` table no longer renders malformed negative numbers +- [x] Re-run full live smoke to reconfirm all MSN-only commands after the latest hardening fixes +- [x] `stocks financials BBCA` table now trims ISO timestamps from section headers and humanizes raw line-item keys +- [x] `stocks earnings BBCA` table now splits history vs forecast and formats annual periods, revenue values, and dates for table mode +- [x] Fixture-backed parser and CLI JSON regression coverage now covers the remaining MSN-only `sentiment`, `news`, and `screen` commands +- [x] Fresh post-coverage live MSN smoke rerun still passes for table and JSON surfaces: 30/30 (`tmp/live-smoke/20260327-163403`) +- [ ] `ownership import --fetch-bing` is still deferred and returns unsupported +- [x] Real KSEI PDF import from local file now works again: `ownership import --file /Users/rasyidanakbar/Downloads/ksei_raw_data.pdf` imported `7261` rows for `955` tickers on `2026-03-28`, replacing the previous `6`-row/`1`-ticker failure mode +- [x] KSEI parser no longer depends on the old hardcoded column bounds fixture layout; it now reconstructs rows from `mutool` line output and handles the live `DATE + SHARE_CODE` merged segment plus `D`/`A` locality markers +- [x] A real IDX-hosted March 2026 ownership PDF URL was verified on `2026-03-29`, but only through `curl-impersonate` inside `nix develop`; plain `curl` still returns `403` from Cloudflare for the same asset +- [x] The current repo now has a concrete IDX-first roadmap: discover the hashed PDF URL from IDX pages, fetch it with browser impersonation, and only then layer snapshot publishing and `ownership sync` +- [x] The KSEI archive (`https://web.ksei.co.id/archive_download/holding_composition`) was verified as a secondary upstream that exposes monthly ZIP files and remains useful for fallback/cross-check work +- [x] Remote IDX discovery now targets the official BEI `Pengumuman` feed (`/id/berita/pengumuman/`) and its backing JSON endpoint (`/primary/NewsAnnouncement/GetAllAnnouncement`), which exposes hashed attachment URLs for ownership-related reports +- [x] New `ownership discover` CLI output now shows the latest hashed BEI ownership URLs across the current `above 5%`, `above 1%`, and `investor-type` report families +- [x] `ownership import --url https://www.idx.co.id/...pdf` now uses the same browser-impersonated download path as Yahoo auth +- [x] Reverse-engineering the current BEI feed shows the discoverable `Pemegang Saham di atas 1% (KSEI)` `lamp1` attachment matches the known-good raw KSEI holder-register layout and now imports successfully +- [x] Live `ownership import --url` against the discovered `above 1%` BEI `lamp1` attachment (`b9b638e5a8_8928aca255.pdf`) now succeeds end to end: `7261` rows for `955` tickers on `2026-03-29`, followed by successful `ownership releases` and `ownership ticker AADI --source ksei` +- [x] Product scope decision on `2026-03-30`: standardize supported remote import on the discoverable `above 1%` holder-register family; treat `above 5%` and `investor-type` PDFs as legacy / unsupported input +- [x] Live `ownership import --url` checks against the currently discoverable `above 5%` and `investor-type` BEI PDFs still fail with `no KSEI rows parsed from PDF`, which is expected until unsupported-family detection / rejection UX lands +- [x] Live `mutool` inspection of the current discoverable `investor-type` BEI attachment shows a stock-level aggregate matrix (`DATE`, `STOCK_CODE`, `NUMBER_OF_SHARES`, investor-type columns, holder-size buckets), not the holder-level KSEI register schema the current parser imports +- [x] Live `mutool` inspection of the current discoverable `above 5%` BEI attachment shows a member/tampungan report (`INVS`, member names, `KSEI UNTUK CLOSED MEMBER-...` labels), not the raw KSEI holder-register layout + ## 🐛 Known Issues - [ ] Yahoo Finance returns 429 from datacenter IPs occasionally - [ ] SMA200 trend shows "Insufficient data" if Yahoo returns < 200 candles +- [x] KSEI ownership parser no longer leaks adjacent columns into `INVESTOR_NAME` on the March 2026 live PDF; direct CLI verification now returns clean holders like `AGUNG PERKASA INVESTINDO` / `CP` / `L` - [x] Yahoo quoteSummary crumb auth — fixed via curl-impersonate-chrome (curl_chrome131) - fc.yahoo.com → A3 cookie + query1 getcrumb → crumb, both sent to quoteSummary - Requires nixpkgs#curl-impersonate-chrome in PATH (added to flake.nix + clan-private) diff --git a/src/api/yahoo/client.rs b/src/api/yahoo/client.rs index b320f30..eb913f8 100644 --- a/src/api/yahoo/client.rs +++ b/src/api/yahoo/client.rs @@ -1,9 +1,10 @@ use std::path::{Path, PathBuf}; -use std::process::{Command, Output}; +use std::process::Command; use std::sync::Mutex; use std::time::Duration; use crate::api::types::{Interval, Period}; +use crate::curl_impersonate; use crate::error::IdxError; use super::raw_types::{ChartResponse, QuoteSummaryResponse}; @@ -117,56 +118,8 @@ impl YahooClient { Ok(cookies.join("; ")) } - fn chrome_curl_binary() -> Option<&'static str> { - // curl-impersonate-chrome ships per-version binaries (curl_chrome131 etc). - // Try latest versions first; no --impersonate flag needed; the binary is the impersonation. - const CANDIDATES: &[&str] = &[ - "curl_chrome136", - "curl_chrome133a", - "curl_chrome131", - "curl_chrome124", - "curl_chrome120", - "curl_chrome116", - ]; - CANDIDATES - .iter() - .copied() - .find(|bin| Command::new(bin).arg("--version").output().is_ok()) - } - - fn run_curl(stage: &str, binary: &str, args: &[&str]) -> Result { - let output = Command::new(binary).args(args).output().map_err(|e| { - if e.kind() == std::io::ErrorKind::NotFound { - return IdxError::Http(format!( - "curl-impersonate binary '{binary}' not found; install nixpkgs#curl-impersonate-chrome" - )); - } - IdxError::Http(format!("failed to run {binary} for Yahoo {stage}: {e}")) - })?; - - if output.status.success() { - return Ok(output); - } - - let stderr = String::from_utf8_lossy(&output.stderr); - let detail = stderr.trim(); - Err(IdxError::Http(format!( - "Yahoo {stage} {binary} failed (status {}): {}", - output.status, - if detail.is_empty() { - "no output" - } else { - detail - } - ))) - } - fn fetch_crumb_via_curl(&self) -> Result { - let binary = Self::chrome_curl_binary().ok_or_else(|| { - IdxError::Http( - "no curl_chrome* binary found; install nixpkgs#curl-impersonate-chrome".to_string(), - ) - })?; + let binary = curl_impersonate::chrome_curl_binary()?; let cookie_jar = Self::cookie_jar_path(); let cookie_jar_str = cookie_jar.to_str().ok_or_else(|| { @@ -190,9 +143,8 @@ impl YahooClient { .output(); // Step 2: fetch crumb with cookie jar (Chrome TLS fingerprint + A3 cookie). - let output = Self::run_curl( - "crumb fetch", - binary, + let output = curl_impersonate::run( + "Yahoo crumb fetch", &["--silent", "--cookie", cookie_jar_str, CRUMB_FETCH_URL], )?; diff --git a/src/cli/ownership.rs b/src/cli/ownership.rs index ca3da5d..e4e1001 100644 --- a/src/cli/ownership.rs +++ b/src/cli/ownership.rs @@ -18,7 +18,7 @@ use crate::output::table::format_idr; use crate::ownership::types::{ ChangeType, FlowSignal, HolderRow, KseiHolding, OwnershipRelease, OwnershipSource, }; -use crate::ownership::{db, entities, graph, parser, search}; +use crate::ownership::{db, entities, graph, parser, remote, search}; #[derive(Debug, Args)] pub struct OwnershipCmd { @@ -28,6 +28,8 @@ pub struct OwnershipCmd { #[derive(Debug, Subcommand)] pub enum OwnershipCommand { + /// Discover the latest IDX-hosted ownership report URLs. + Discover(DiscoverArgs), /// Import ownership data from KSEI PDF or Bing API. Import(ImportArgs), /// Show all holders for a ticker (KSEI + Bing combined). @@ -52,9 +54,19 @@ pub enum OwnershipCommand { Releases, } +#[derive(Debug, Args)] +pub struct DiscoverArgs { + /// Report family to discover: all, above1, above5, or investor-type. + #[arg(long, default_value = "all")] + pub family: String, + /// Maximum number of discovered report URLs to print. + #[arg(long, default_value_t = 6)] + pub limit: usize, +} + #[derive(Debug, Args)] pub struct ImportArgs { - /// URL to KSEI ownership PDF. + /// URL to a remote ownership PDF. #[arg(long)] pub url: Option, /// Path to local KSEI PDF file. @@ -159,6 +171,7 @@ pub enum ResolveCommand { pub fn handle(cmd: &OwnershipCommand, config: &IdxConfig) -> Result<(), IdxError> { match cmd { + OwnershipCommand::Discover(args) => handle_discover(args, config), OwnershipCommand::Import(args) => handle_import(args, config), OwnershipCommand::Ticker(args) => handle_ticker(args, config), OwnershipCommand::Entity(args) => handle_entity(args, config), @@ -173,6 +186,45 @@ pub fn handle(cmd: &OwnershipCommand, config: &IdxConfig) -> Result<(), IdxError } } +fn handle_discover(args: &DiscoverArgs, config: &IdxConfig) -> Result<(), IdxError> { + if args.limit == 0 { + return Err(IdxError::ParseError( + "--limit must be greater than 0".to_string(), + )); + } + + let family = parse_discovery_family(&args.family)?; + let reports = remote::discover_idx_ownership_reports(family, args.limit)?; + + if matches!(config.output, OutputFormat::Json) { + return json::print_json(&reports); + } + + let mut table = Table::new(); + table + .load_preset(UTF8_FULL) + .set_content_arrangement(ContentArrangement::Dynamic) + .set_header(vec!["DATE", "FAMILY", "KIND", "FILE", "TITLE", "URL"]); + + for report in reports { + table.add_row(vec![ + Cell::new(report.publish_date.split('T').next().unwrap_or("-")), + Cell::new(report.family.label()), + Cell::new(if report.is_attachment { + "attachment" + } else { + "main" + }), + Cell::new(report.original_filename.unwrap_or_else(|| "-".to_string())), + Cell::new(report.title), + Cell::new(report.pdf_url), + ]); + } + + println!("{table}"); + Ok(()) +} + fn handle_ticker(args: &TickerArgs, config: &IdxConfig) -> Result<(), IdxError> { let conn = db::open_db(config)?; let symbol = args.symbol.trim().to_uppercase(); @@ -660,19 +712,19 @@ fn handle_import(args: &ImportArgs, config: &IdxConfig) -> Result<(), IdxError> } } - let Some(pdf_path) = resolve_pdf_input(args)? else { + let Some(pdf_input) = resolve_pdf_input(args)? else { return Ok(()); }; let conn = db::open_db(config)?; - let sha256 = sha256_file(&pdf_path)?; + let sha256 = sha256_file(&pdf_input.pdf_path)?; if !args.force && db::release_exists(&conn, &sha256)? { println!("Release already imported (sha256: {sha256}). Use --force to re-import."); return Ok(()); } - let raw_rows = parser::parse_ksei_pdf(&pdf_path)?; + let raw_rows = parser::parse_ksei_pdf(&pdf_input.pdf_path)?; if raw_rows.is_empty() { return Err(IdxError::ParseError( "no KSEI rows parsed from PDF".to_string(), @@ -718,7 +770,7 @@ fn handle_import(args: &ImportArgs, config: &IdxConfig) -> Result<(), IdxError> let release = OwnershipRelease { id: 0, - source_url: args.url.clone(), + source_url: pdf_input.source_url, sha256, as_of_date, row_count: inserted_rows, @@ -769,7 +821,30 @@ fn handle_releases(config: &IdxConfig) -> Result<(), IdxError> { Ok(()) } -fn resolve_pdf_input(args: &ImportArgs) -> Result, IdxError> { +fn parse_discovery_family(raw: &str) -> Result, IdxError> { + match raw.trim().to_ascii_lowercase().as_str() { + "all" => Ok(None), + "above1" | "above-1" | "above_1" => { + Ok(Some(remote::OwnershipReportFamily::AboveOnePercent)) + } + "above5" | "above-5" | "above_5" => { + Ok(Some(remote::OwnershipReportFamily::AboveFivePercent)) + } + "investor-type" | "investor_type" | "investortype" => { + Ok(Some(remote::OwnershipReportFamily::InvestorTypeBreakdown)) + } + _ => Err(IdxError::ParseError( + "invalid --family, expected: all|above1|above5|investor-type".to_string(), + )), + } +} + +struct ResolvedPdfInput { + pdf_path: PathBuf, + source_url: Option, +} + +fn resolve_pdf_input(args: &ImportArgs) -> Result, IdxError> { if let Some(path) = &args.file { if !path.exists() { return Err(IdxError::Io(format!( @@ -777,13 +852,19 @@ fn resolve_pdf_input(args: &ImportArgs) -> Result, IdxError> { path.display() ))); } - return Ok(Some(path.clone())); + return Ok(Some(ResolvedPdfInput { + pdf_path: path.clone(), + source_url: None, + })); } if let Some(url) = &args.url { let target = cache_pdf_path(url)?; download_pdf(url, &target)?; - return Ok(Some(target)); + return Ok(Some(ResolvedPdfInput { + pdf_path: target, + source_url: Some(url.clone()), + })); } Ok(None) @@ -815,6 +896,10 @@ fn cache_pdf_path(url: &str) -> Result { } fn download_pdf(url: &str, target: &Path) -> Result<(), IdxError> { + if is_idx_url(url) { + return remote::download_idx_pdf(url, target); + } + let response = ureq::get(url) .header( "User-Agent", @@ -830,6 +915,7 @@ fn download_pdf(url: &str, target: &Path) -> Result<(), IdxError> { let bytes = body .read_to_vec() .map_err(|e| IdxError::Http(format!("failed reading PDF body: {e}")))?; + remote::validate_pdf_payload(&bytes)?; fs::write(target, &bytes).map_err(|e| { IdxError::Io(format!( @@ -841,6 +927,14 @@ fn download_pdf(url: &str, target: &Path) -> Result<(), IdxError> { Ok(()) } +fn is_idx_url(url: &str) -> bool { + let normalized = url.trim().to_ascii_lowercase(); + normalized.starts_with("https://www.idx.co.id/") + || normalized.starts_with("http://www.idx.co.id/") + || normalized.starts_with("https://idx.co.id/") + || normalized.starts_with("http://idx.co.id/") +} + fn sha256_file(path: &Path) -> Result { let bytes = fs::read(path).map_err(|e| { IdxError::Io(format!( diff --git a/src/curl_impersonate.rs b/src/curl_impersonate.rs new file mode 100644 index 0000000..3735feb --- /dev/null +++ b/src/curl_impersonate.rs @@ -0,0 +1,67 @@ +use std::process::{Command, Output}; + +use crate::error::IdxError; + +const CANDIDATES: &[&str] = &[ + "curl_chrome142", + "curl_chrome136", + "curl_chrome133a", + "curl_chrome131", + "curl_chrome124", + "curl_chrome120", + "curl_chrome116", +]; +const OVERRIDE_ENV: &str = "IDX_CURL_IMPERSONATE_BIN"; + +pub fn chrome_curl_binary() -> Result { + if let Ok(value) = std::env::var(OVERRIDE_ENV) { + let trimmed = value.trim(); + if !trimmed.is_empty() { + return Ok(trimmed.to_string()); + } + } + + CANDIDATES + .iter() + .copied() + .find(|bin| Command::new(bin).arg("--version").output().is_ok()) + .map(str::to_string) + .ok_or_else(|| { + IdxError::Http(format!( + "no curl_chrome* binary found; set {OVERRIDE_ENV} or install nixpkgs#curl-impersonate-chrome" + )) + }) +} + +pub fn run(stage: &str, args: &[&str]) -> Result { + let owned_args: Vec = args.iter().map(|arg| (*arg).to_string()).collect(); + run_owned(stage, &owned_args) +} + +pub fn run_owned(stage: &str, args: &[String]) -> Result { + let binary = chrome_curl_binary()?; + let output = Command::new(&binary).args(args).output().map_err(|e| { + if e.kind() == std::io::ErrorKind::NotFound { + return IdxError::Http(format!( + "curl-impersonate binary '{binary}' not found; set {OVERRIDE_ENV} or install nixpkgs#curl-impersonate-chrome" + )); + } + IdxError::Http(format!("failed to run {binary} for {stage}: {e}")) + })?; + + if output.status.success() { + return Ok(output); + } + + let stderr = String::from_utf8_lossy(&output.stderr); + let detail = stderr.trim(); + Err(IdxError::Http(format!( + "{stage} {binary} failed (status {}): {}", + output.status, + if detail.is_empty() { + "no output" + } else { + detail + } + ))) +} diff --git a/src/main.rs b/src/main.rs index bac5ff0..94a0f6b 100644 --- a/src/main.rs +++ b/src/main.rs @@ -3,6 +3,7 @@ mod api; mod cache; mod cli; mod config; +mod curl_impersonate; mod error; mod output; #[cfg(feature = "ownership")] diff --git a/src/ownership/entities.rs b/src/ownership/entities.rs index a6847d8..a3fc227 100644 --- a/src/ownership/entities.rs +++ b/src/ownership/entities.rs @@ -215,7 +215,7 @@ fn normalize_investor_type(raw: &str) -> Option { fn normalize_locality(raw: &str) -> Option { match raw.trim().to_uppercase().as_str() { - "L" => Some(Locality::Local), + "L" | "D" => Some(Locality::Local), "F" | "A" => Some(Locality::Foreign), _ => None, } diff --git a/src/ownership/mod.rs b/src/ownership/mod.rs index c2c67c7..07b2902 100644 --- a/src/ownership/mod.rs +++ b/src/ownership/mod.rs @@ -2,5 +2,6 @@ pub mod db; pub mod entities; pub mod graph; pub mod parser; +pub mod remote; pub mod search; pub mod types; diff --git a/src/ownership/parser.rs b/src/ownership/parser.rs index a414f2b..c6896c7 100644 --- a/src/ownership/parser.rs +++ b/src/ownership/parser.rs @@ -3,32 +3,36 @@ use std::path::Path; use std::process::Command; use quick_xml::Reader; -use quick_xml::events::Event; +use quick_xml::events::{BytesStart, Event}; use crate::error::IdxError; use crate::ownership::types::KseiRawRow; -/// Character grid for a single PDF page: y-coord → column-index → sorted (x, char) pairs. -type PageGrid = HashMap>>; - const Y_TOLERANCE: f32 = 0.8; +const HEADER_MATCH_MIN: usize = 4; -/// Inclusive-left, exclusive-right X ranges for each KSEI data column. -const COLUMN_BOUNDS: [(f32, f32); 12] = [ - (15.0, 52.0), // date - (52.0, 70.0), // share_code - (70.0, 167.0), // issuer_name - (167.0, 432.0), // investor_name - (432.0, 463.0), // investor_type - (463.0, 497.0), // local_foreign - (497.0, 558.0), // nationality - (558.0, 615.0), // domicile - (615.0, 653.0), // holdings_scripless - (653.0, 692.0), // holdings_scrip - (692.0, 745.0), // total_holding_shares - (745.0, 800.0), // percentage +const HEADER_LABELS: &[&str] = &[ + "DATE", + "SHARECODE", + "ISSUERNAME", + "INVESTORNAME", + "INVESTORTYPE", + "LOCALFOREIGN", + "NATIONALITY", + "DOMICILE", + "HOLDINGSSCRIPLESS", + "HOLDINGSSCRIP", + "TOTALHOLDINGSHARES", + "PERCENTAGE", ]; +#[derive(Debug, Clone)] +struct PageLine { + x: f32, + y: f32, + text: String, +} + /// Parse a KSEI ownership PDF into raw rows. /// Shells out to `mutool` for XML extraction, then parses with quick-xml. pub fn parse_ksei_pdf(path: &Path) -> Result, IdxError> { @@ -65,76 +69,24 @@ pub fn parse_stext_xml(xml: &str) -> Result, IdxError> { reader.config_mut().trim_text(false); let mut rows: Vec = Vec::new(); - let mut current_page: Option = None; + let mut current_page: Option> = None; let mut buf = Vec::new(); loop { match reader.read_event_into(&mut buf) { Ok(Event::Start(e)) if e.name().as_ref() == b"page" => { - current_page = Some(HashMap::new()); + current_page = Some(Vec::new()); } - Ok(Event::Empty(e)) if e.name().as_ref() == b"char" => { - if let Some(page) = current_page.as_mut() { - let mut x: Option = None; - let mut y: Option = None; - let mut c: Option = None; - - for attr_result in e.attributes().with_checks(false) { - let attr = attr_result.map_err(|err| { - IdxError::PdfParseError(format!("invalid XML attribute: {err}")) - })?; - - match attr.key.as_ref() { - b"x" => { - let s = attr.decode_and_unescape_value(reader.decoder()).map_err( - |err| { - IdxError::PdfParseError(format!( - "invalid XML x attribute: {err}" - )) - }, - )?; - x = s.parse::().ok(); - } - b"y" => { - let s = attr.decode_and_unescape_value(reader.decoder()).map_err( - |err| { - IdxError::PdfParseError(format!( - "invalid XML y attribute: {err}" - )) - }, - )?; - y = s.parse::().ok(); - } - b"c" => { - let s = attr.decode_and_unescape_value(reader.decoder()).map_err( - |err| { - IdxError::PdfParseError(format!( - "invalid XML char attribute: {err}" - )) - }, - )?; - c = s.chars().next(); - } - _ => {} - } - } - - if let (Some(x_val), Some(y_val), Some(ch)) = (x, y, c) - && let Some(col_idx) = x_to_column(x_val) - { - let yb = y_bucket(y_val); - let xi = (x_val * 100.0).round() as i32; - page.entry(yb) - .or_default() - .entry(col_idx) - .or_default() - .push((xi, ch)); - } + Ok(Event::Start(e)) if e.name().as_ref() == b"line" => { + if let Some(page) = current_page.as_mut() + && let Some(line) = parse_line_attrs(&reader, &e)? + { + page.push(line); } } Ok(Event::End(e)) if e.name().as_ref() == b"page" => { if let Some(page) = current_page.take() { - rows.extend(extract_rows_from_page(page)); + rows.extend(extract_rows_from_page(&page)); } } Ok(Event::Eof) => break, @@ -164,67 +116,263 @@ pub fn check_mutool() -> Result<(), IdxError> { Ok(()) } -fn extract_rows_from_page(page: PageGrid) -> Vec { - let mut page_rows = Vec::new(); +fn parse_line_attrs( + reader: &Reader<&[u8]>, + event: &BytesStart<'_>, +) -> Result, IdxError> { + let mut bbox: Option = None; + let mut text: Option = None; - let mut y_keys: Vec = page.keys().copied().collect(); + for attr_result in event.attributes().with_checks(false) { + let attr = attr_result + .map_err(|err| IdxError::PdfParseError(format!("invalid XML attribute: {err}")))?; + + match attr.key.as_ref() { + b"bbox" => { + bbox = Some( + attr.decode_and_unescape_value(reader.decoder()) + .map_err(|err| { + IdxError::PdfParseError(format!("invalid XML bbox attribute: {err}")) + })? + .to_string(), + ); + } + b"text" => { + text = Some( + attr.decode_and_unescape_value(reader.decoder()) + .map_err(|err| { + IdxError::PdfParseError(format!("invalid XML text attribute: {err}")) + })? + .to_string(), + ); + } + _ => {} + } + } + + let Some(text) = text.map(|value| normalize_spaces(&value)) else { + return Ok(None); + }; + if text.is_empty() { + return Ok(None); + } + + let Some(bbox) = bbox else { + return Ok(None); + }; + let mut parts = bbox.split_whitespace(); + let x = parts.next().and_then(|value| value.parse::().ok()); + let y = parts.next().and_then(|value| value.parse::().ok()); + + match (x, y) { + (Some(x), Some(y)) => Ok(Some(PageLine { x, y, text })), + _ => Ok(None), + } +} + +fn extract_rows_from_page(page: &[PageLine]) -> Vec { + let mut rows_by_y: HashMap> = HashMap::new(); + for line in page { + rows_by_y + .entry(y_bucket(line.y)) + .or_default() + .push(line.clone()); + } + + let mut y_keys: Vec = rows_by_y.keys().copied().collect(); y_keys.sort_unstable(); + let mut rows = Vec::new(); for y in y_keys { - let mut row = KseiRawRow { - date: String::new(), - share_code: String::new(), - issuer_name: String::new(), - investor_name: String::new(), - investor_type: String::new(), - local_foreign: String::new(), - nationality: String::new(), - domicile: String::new(), - holdings_scripless: String::new(), - holdings_scrip: String::new(), - total_holding_shares: String::new(), - percentage: String::new(), + let Some(lines) = rows_by_y.get(&y) else { + continue; }; - if let Some(col_map) = page.get(&y) { - for (col_idx, chars) in col_map { - let mut sorted = chars.clone(); - sorted.sort_by_key(|(x, _)| *x); - let text = normalize_spaces(&sorted.iter().map(|(_, c)| c).collect::()); - assign_column(&mut row, *col_idx, text); - } + let mut sorted = lines.clone(); + sorted.sort_by(|left, right| left.x.total_cmp(&right.x)); + + let texts: Vec = sorted.into_iter().map(|line| line.text).collect(); + if is_header_row(&texts) { + continue; } + let row = parse_row_segments(&texts); if is_data_row(&row) { - page_rows.push(row); + rows.push(row); } } - page_rows + rows } -fn assign_column(row: &mut KseiRawRow, col_idx: usize, value: String) { - match col_idx { - 0 => row.date = value, - 1 => row.share_code = value, - 2 => row.issuer_name = value, - 3 => row.investor_name = value, - 4 => row.investor_type = value, - 5 => row.local_foreign = value, - 6 => row.nationality = value, - 7 => row.domicile = value, - 8 => row.holdings_scripless = value, - 9 => row.holdings_scrip = value, - 10 => row.total_holding_shares = value, - 11 => row.percentage = value, - _ => {} - } -} - -fn x_to_column(x: f32) -> Option { - COLUMN_BOUNDS +fn is_header_row(texts: &[String]) -> bool { + texts .iter() - .position(|(left, right)| x >= *left && x < *right) + .filter(|text| HEADER_LABELS.contains(&normalize_header_label(text).as_str())) + .count() + >= HEADER_MATCH_MIN +} + +fn parse_row_segments(texts: &[String]) -> KseiRawRow { + let mut row = KseiRawRow { + date: String::new(), + share_code: String::new(), + issuer_name: String::new(), + investor_name: String::new(), + investor_type: String::new(), + local_foreign: String::new(), + nationality: String::new(), + domicile: String::new(), + holdings_scripless: String::new(), + holdings_scrip: String::new(), + total_holding_shares: String::new(), + percentage: String::new(), + }; + + let mut remaining: Vec = texts + .iter() + .map(|text| normalize_spaces(text)) + .filter(|text| !text.is_empty()) + .collect(); + if remaining.is_empty() { + return row; + } + + if let Some((date, share_code)) = split_date_and_share(&remaining[0]) { + row.date = date; + row.share_code = share_code; + let _ = remaining.remove(0); + } + + if row.date.is_empty() { + return row; + } + + if row.share_code.is_empty() + && remaining + .first() + .is_some_and(|segment| is_share_code_like(segment)) + { + row.share_code = remaining.remove(0); + } + + if remaining + .last() + .is_some_and(|segment| is_percentage_like(segment)) + { + row.percentage = remaining.pop().unwrap_or_default(); + } + if remaining + .last() + .is_some_and(|segment| is_id_number_like(segment)) + { + row.total_holding_shares = remaining.pop().unwrap_or_default(); + } + if remaining + .last() + .is_some_and(|segment| is_id_number_like(segment)) + { + row.holdings_scrip = remaining.pop().unwrap_or_default(); + } + if remaining + .last() + .is_some_and(|segment| is_id_number_like(segment)) + { + row.holdings_scripless = remaining.pop().unwrap_or_default(); + } + + let mut geo_fields = Vec::new(); + while remaining.len() > 2 { + let Some(candidate) = remaining.last().cloned() else { + break; + }; + + if row.local_foreign.is_empty() && is_locality_marker(&candidate) { + row.local_foreign = remaining.pop().unwrap_or_default(); + continue; + } + + if row.investor_type.is_empty() && is_investor_type_marker(&candidate) { + row.investor_type = remaining.pop().unwrap_or_default(); + continue; + } + + if geo_fields.len() < 2 { + geo_fields.push(remaining.pop().unwrap_or_default()); + continue; + } + + break; + } + + geo_fields.reverse(); + if let Some(first) = geo_fields.first() { + row.nationality = first.clone(); + } + if geo_fields.len() > 1 { + row.domicile = geo_fields[1..].join(" "); + } + + if let Some(first) = remaining.first() { + row.issuer_name = first.clone(); + } + if remaining.len() > 1 { + row.investor_name = remaining[1..].join(" "); + } + + row +} + +fn split_date_and_share(segment: &str) -> Option<(String, String)> { + let trimmed = segment.trim(); + if trimmed.len() < 11 { + return None; + } + + let date = trimmed.get(..11)?.to_string(); + if !is_ksei_date(&date) { + return None; + } + + let share_code = trimmed.get(11..).unwrap_or_default().trim().to_string(); + Some((date, share_code)) +} + +fn is_share_code_like(value: &str) -> bool { + let trimmed = value.trim(); + (3..=8).contains(&trimmed.len()) + && trimmed + .chars() + .all(|ch| ch.is_ascii_uppercase() || ch.is_ascii_digit()) +} + +fn is_id_number_like(value: &str) -> bool { + let trimmed = value.trim(); + !trimmed.is_empty() + && trimmed != "-" + && trimmed.chars().all(|ch| ch.is_ascii_digit() || ch == '.') +} + +fn is_investor_type_marker(value: &str) -> bool { + let trimmed = value.trim(); + !trimmed.is_empty() && trimmed.len() <= 4 && trimmed.chars().all(|ch| ch.is_ascii_uppercase()) +} + +fn is_locality_marker(value: &str) -> bool { + matches!( + value.trim().to_ascii_uppercase().as_str(), + "L" | "F" | "A" | "D" | "LOCAL" | "FOREIGN" | "ASING" | "DOMESTIC" + ) +} + +fn normalize_header_label(text: &str) -> String { + let mut normalized = String::new(); + for ch in text.chars() { + if ch.is_ascii_alphanumeric() { + normalized.push(ch.to_ascii_uppercase()); + } + } + normalized } fn y_bucket(y: f32) -> i32 { @@ -236,7 +384,7 @@ fn normalize_spaces(input: &str) -> String { let mut prev_space = false; for ch in input.chars() { - if ch == ' ' { + if ch.is_whitespace() { if !prev_space { out.push(' '); } @@ -311,24 +459,37 @@ fn is_percentage_like(s: &str) -> bool { #[cfg(test)] mod tests { - use std::fs; use std::path::Path; use super::{check_mutool, parse_ksei_pdf, parse_stext_xml}; #[test] - fn test_parse_stext_xml_fixture_extracts_rows() { - let fixture_path = Path::new("tests/fixtures/ksei_stext_sample.xml"); - let xml = fs::read_to_string(fixture_path).expect("failed to read stext fixture"); + fn test_parse_stext_xml_live_like_lines_extract_rows() { + let xml = build_live_like_stext_xml(); - let rows = parse_stext_xml(&xml).expect("failed to parse fixture XML"); + let rows = parse_stext_xml(&xml).expect("failed to parse live-like fixture XML"); assert_eq!(rows.len(), 3); let first = &rows[0]; assert_eq!(first.date, "27-Feb-2026"); - assert_eq!(first.share_code, "BBCA"); - assert_eq!(first.investor_name, "PT DWIMURIA INVESTAMA ANDALAN"); - assert_eq!(first.percentage, "54,94"); + assert_eq!(first.share_code, "AADI"); + assert_eq!(first.issuer_name, "ADARO ANDALAN INDONESIA Tbk"); + assert_eq!(first.investor_name, "ADARO STRATEGIC INVESTMENTS"); + assert_eq!(first.investor_type, "CP"); + assert_eq!(first.local_foreign, "D"); + assert_eq!(first.nationality, "INDONESIA"); + assert_eq!(first.holdings_scripless, "3.200.142.830"); + assert_eq!(first.holdings_scrip, "0"); + assert_eq!(first.total_holding_shares, "3.200.142.830"); + assert_eq!(first.percentage, "66,18"); + + let last = &rows[2]; + assert_eq!(last.share_code, "BBRI"); + assert_eq!(last.investor_name, "PT NUSANTARA CAPITAL"); + assert_eq!(last.investor_type, "ID"); + assert_eq!(last.local_foreign, "A"); + assert_eq!(last.nationality, "SINGAPORE"); + assert_eq!(last.percentage, "15,00"); } #[test] @@ -352,4 +513,97 @@ mod tests { rows.len() ); } + + fn build_live_like_stext_xml() -> String { + let mut xml = String::from(r#""#); + + append_page( + &mut xml, + "page1", + &[ + (31.56, 86.33, "DATE"), + (56.64, 86.33, "SHARE_CODE"), + (121.22, 86.33, "ISSUER_NAME"), + (267.17, 86.33, "INVESTOR_NAME"), + (390.89, 86.33, "INVESTOR_TYPE"), + (434.11, 86.33, "LOCAL_FOREIGN"), + (475.87, 86.33, "NATIONALITY"), + (525.19, 86.33, "DOMICILE"), + (574.63, 86.33, "HOLDINGS_SCRIPLESS"), + (629.98, 86.33, "HOLDINGS_SCRIP"), + (680.02, 86.33, "TOTAL_HOLDING_SHARES"), + (741.22, 86.33, "PERCENTAGE"), + (28.68, 91.01, "27-Feb-2026 AADI"), + (85.10, 91.01, "ADARO ANDALAN INDONESIA Tbk"), + (179.30, 91.01, "ADARO STRATEGIC INVESTMENTS"), + (381.41, 91.01, "CP"), + (424.75, 91.01, "D"), + (504.07, 91.01, "INDONESIA"), + (597.58, 91.01, "3.200.142.830"), + (630.10, 91.01, "0"), + (680.12, 91.01, "3.200.142.830"), + (741.30, 91.01, "66,18"), + (28.68, 96.01, "27-Feb-2026 AADI"), + (85.10, 96.01, "ADARO ANDALAN INDONESIA Tbk"), + (179.30, 96.01, "PUBLIC"), + (381.41, 96.01, "OT"), + (424.75, 96.01, "A"), + (504.07, 96.01, "SINGAPORE"), + (597.58, 96.01, "500.000.000"), + (630.10, 96.01, "0"), + (680.12, 96.01, "500.000.000"), + (741.30, 96.01, "10,34"), + ], + ); + + append_page( + &mut xml, + "page2", + &[ + (28.68, 20.00, "27-Feb-2026 BBRI"), + (85.10, 20.00, "BANK RAKYAT INDONESIA Tbk"), + (179.30, 20.00, "PT NUSANTARA CAPITAL"), + (381.41, 20.00, "ID"), + (424.75, 20.00, "A"), + (504.07, 20.00, "SINGAPORE"), + (597.58, 20.00, "1.250.000.000"), + (630.10, 20.00, "0"), + (680.12, 20.00, "1.250.000.000"), + (741.30, 20.00, "15,00"), + ], + ); + + xml.push_str(""); + xml + } + + fn append_page(xml: &mut String, id: &str, lines: &[(f32, f32, &str)]) { + xml.push_str(&format!(r#""#)); + for (x, y, text) in lines { + append_line(xml, *x, *y, text); + } + xml.push_str(""); + } + + fn append_line(xml: &mut String, x: f32, y: f32, text: &str) { + let width = x + (text.len() as f32 * 2.0); + let height = y + 3.48; + xml.push_str(&format!( + r#""#, + escape_xml_attr(text) + )); + } + + fn escape_xml_attr(text: &str) -> String { + text.chars() + .map(|ch| match ch { + '&' => "&".to_string(), + '<' => "<".to_string(), + '>' => ">".to_string(), + '"' => """.to_string(), + '\'' => "'".to_string(), + other => other.to_string(), + }) + .collect::() + } } diff --git a/src/ownership/remote.rs b/src/ownership/remote.rs new file mode 100644 index 0000000..6e99a71 --- /dev/null +++ b/src/ownership/remote.rs @@ -0,0 +1,664 @@ +use std::fs; +use std::path::Path; + +use serde::{Deserialize, Serialize}; + +use crate::curl_impersonate; +use crate::error::IdxError; + +pub const IDX_ANNOUNCEMENT_LISTING_URL: &str = "https://www.idx.co.id/id/berita/pengumuman/"; +const DEFAULT_IDX_ANNOUNCEMENT_API_URL: &str = + "https://www.idx.co.id/primary/NewsAnnouncement/GetAllAnnouncement"; +const IDX_ANNOUNCEMENT_API_ENV: &str = "IDX_OWNERSHIP_ANNOUNCEMENT_API_URL"; +const IDX_ANNOUNCEMENT_LISTING_ENV: &str = "IDX_OWNERSHIP_ANNOUNCEMENT_PAGE_URL"; +const IDX_ANNOUNCEMENT_PAGE_SIZE: usize = 10; +const USER_AGENT: &str = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"; + +#[derive(Debug, Clone, Deserialize)] +pub struct AnnouncementPage { + #[serde(rename = "Items", default)] + pub items: Vec, + #[serde(rename = "ItemCount")] + pub item_count: Option, + #[serde(rename = "PageCount")] + pub page_count: Option, +} + +#[derive(Debug, Clone, Deserialize)] +pub struct AnnouncementItem { + #[serde(rename = "PublishDate")] + pub publish_date: String, + #[serde(rename = "Title")] + pub title: String, + #[serde(rename = "AnnouncementType")] + pub announcement_type: Option, + #[serde(rename = "Code")] + pub code: Option, + #[serde(rename = "Attachments", default)] + pub attachments: Vec, + #[serde(rename = "PdfPath")] + pub pdf_path: Option, +} + +#[derive(Debug, Clone, Deserialize)] +pub struct AnnouncementAttachment { + #[serde(rename = "FullSavePath")] + pub full_save_path: String, + #[serde(rename = "OriginalFilename")] + pub original_filename: Option, + #[serde(rename = "PDFFilename")] + pub pdf_filename: Option, + #[serde(rename = "IsAttachment")] + pub is_attachment: Option, +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize)] +#[serde(rename_all = "snake_case")] +pub enum OwnershipReportFamily { + AboveOnePercent, + AboveFivePercent, + InvestorTypeBreakdown, +} + +impl OwnershipReportFamily { + pub fn cli_name(self) -> &'static str { + match self { + Self::AboveOnePercent => "above1", + Self::AboveFivePercent => "above5", + Self::InvestorTypeBreakdown => "investor-type", + } + } + + pub fn label(self) -> &'static str { + match self { + Self::AboveOnePercent => "Above 1%", + Self::AboveFivePercent => "Above 5%", + Self::InvestorTypeBreakdown => "Investor Type Breakdown", + } + } +} + +#[derive(Debug, Clone, PartialEq, Eq, Serialize)] +pub struct DiscoveredOwnershipPdf { + pub family: OwnershipReportFamily, + pub listing_page_url: String, + pub query_url: String, + pub pdf_url: String, + pub title: String, + pub publish_date: String, + pub code: Option, + pub original_filename: Option, + pub is_attachment: bool, +} + +struct DiscoveryQuery { + family: OwnershipReportFamily, + keywords: &'static str, + title_needles: &'static [&'static str], +} + +const DISCOVERY_QUERIES: &[DiscoveryQuery] = &[ + DiscoveryQuery { + family: OwnershipReportFamily::AboveOnePercent, + keywords: "pemegang saham di atas 1", + title_needles: &["PEMEGANG SAHAM DI ATAS 1", "SHAREHOLDERS ABOVE 1"], + }, + DiscoveryQuery { + family: OwnershipReportFamily::AboveFivePercent, + keywords: "pemegang saham di atas 5", + title_needles: &["PEMEGANG SAHAM DI ATAS 5", "SHAREHOLDERS ABOVE 5"], + }, + DiscoveryQuery { + family: OwnershipReportFamily::InvestorTypeBreakdown, + keywords: "kepemilikan saham perusahaan tercatat", + title_needles: &[ + "KEPEMILIKAN SAHAM PERUSAHAAN TERCATAT BERDASARKAN TIPE INVESTOR", + "DATA KSEI TERKAIT KEPEMILIKAN SAHAM PERUSAHAAN TERCATAT", + ], + }, +]; + +pub fn announcement_listing_url() -> String { + std::env::var(IDX_ANNOUNCEMENT_LISTING_ENV) + .ok() + .filter(|value| !value.trim().is_empty()) + .unwrap_or_else(|| IDX_ANNOUNCEMENT_LISTING_URL.to_string()) +} + +pub fn announcement_api_url() -> String { + std::env::var(IDX_ANNOUNCEMENT_API_ENV) + .ok() + .filter(|value| !value.trim().is_empty()) + .unwrap_or_else(|| DEFAULT_IDX_ANNOUNCEMENT_API_URL.to_string()) +} + +pub fn build_announcement_query_url( + keywords: &str, + page_number: usize, + page_size: usize, +) -> String { + format!( + "{}?keywords={}&pageNumber={page_number}&pageSize={page_size}&lang=id", + announcement_api_url(), + percent_encode(keywords), + ) +} + +pub fn parse_announcement_page(raw: &str) -> Result { + let trimmed = raw.trim(); + if trimmed.is_empty() { + return Err(IdxError::Http( + "IDX ownership discovery returned an empty announcement payload".to_string(), + )); + } + let normalized = trimmed.to_ascii_lowercase(); + if normalized.starts_with(" Result, IdxError> { + let Some(query) = DISCOVERY_QUERIES + .iter() + .find(|query| query.family == family) + else { + return Err(IdxError::Http(format!( + "failed to discover IDX ownership reports from {query_url}: unknown report family" + ))); + }; + + let mut matches: Vec<&AnnouncementItem> = page + .items + .iter() + .filter(|item| item_matches_family(item, query)) + .collect(); + matches.sort_by(|left, right| right.publish_date.cmp(&left.publish_date)); + + let Some(item) = matches.into_iter().next() else { + return Err(IdxError::Http(format!( + "failed to discover IDX ownership reports from {query_url}: no matching announcement found" + ))); + }; + + let mut attachments = item_pdf_attachments(item) + .into_iter() + .map(|attachment| { + let is_attachment = attachment_is_attachment(&attachment); + let original_filename = attachment + .original_filename + .clone() + .or(attachment.pdf_filename.clone()) + .map(|value| value.trim().to_string()); + + DiscoveredOwnershipPdf { + family, + listing_page_url: announcement_listing_url(), + query_url: query_url.to_string(), + pdf_url: attachment.full_save_path, + title: item.title.clone(), + publish_date: item.publish_date.clone(), + code: clean_option(item.code.as_deref()), + original_filename, + is_attachment, + } + }) + .collect::>(); + + attachments.sort_by(|left, right| { + left.is_attachment + .cmp(&right.is_attachment) + .then_with(|| left.original_filename.cmp(&right.original_filename)) + }); + + if attachments.is_empty() { + return Err(IdxError::Http(format!( + "failed to discover IDX ownership reports from {query_url}: matching announcement had no PDF attachments" + ))); + } + + Ok(attachments) +} + +pub fn discover_idx_ownership_reports( + family_filter: Option, + limit: usize, +) -> Result, IdxError> { + let mut discovered = Vec::new(); + let mut errors = Vec::new(); + + for query in DISCOVERY_QUERIES + .iter() + .filter(|query| family_filter.is_none_or(|family| family == query.family)) + { + let query_url = build_announcement_query_url(query.keywords, 1, IDX_ANNOUNCEMENT_PAGE_SIZE); + match fetch_text( + "IDX ownership announcement discovery", + &query_url, + &json_headers(), + ) + .and_then(|raw| parse_announcement_page(&raw)) + .and_then(|page| select_latest_ownership_reports(&page, &query_url, query.family)) + { + Ok(mut reports) => discovered.append(&mut reports), + Err(err) => errors.push(err.to_string()), + } + } + + discovered.sort_by(|left, right| { + right + .publish_date + .cmp(&left.publish_date) + .then_with(|| left.is_attachment.cmp(&right.is_attachment)) + .then_with(|| left.original_filename.cmp(&right.original_filename)) + }); + + if limit < discovered.len() { + discovered.truncate(limit); + } + + if discovered.is_empty() { + let detail = if errors.is_empty() { + "no matching ownership reports found".to_string() + } else { + errors.join("; ") + }; + return Err(IdxError::Http(format!( + "failed to discover IDX ownership reports from {}: {detail}", + announcement_listing_url() + ))); + } + + Ok(discovered) +} + +pub fn download_idx_pdf(url: &str, target: &Path) -> Result<(), IdxError> { + let bytes = fetch_bytes("IDX ownership PDF download", url, &pdf_headers())?; + validate_pdf_payload(&bytes)?; + + fs::write(target, &bytes).map_err(|e| { + IdxError::Io(format!( + "failed writing cached PDF {}: {e}", + target.display() + )) + })?; + + Ok(()) +} + +pub fn validate_pdf_payload(bytes: &[u8]) -> Result<(), IdxError> { + if bytes.is_empty() { + return Err(IdxError::Http( + "IDX ownership download returned an empty response".to_string(), + )); + } + + let trimmed = bytes + .iter() + .skip_while(|byte| byte.is_ascii_whitespace()) + .copied() + .collect::>(); + if trimmed.starts_with(b"%PDF-") { + return Ok(()); + } + + let preview = String::from_utf8_lossy(&trimmed[..trimmed.len().min(256)]).to_ascii_lowercase(); + if preview.contains(" Result { + let bytes = fetch_bytes(stage, url, headers)?; + String::from_utf8(bytes) + .map_err(|e| IdxError::Http(format!("failed to decode {stage} response as utf-8: {e}"))) +} + +fn fetch_bytes(stage: &str, url: &str, headers: &[(String, String)]) -> Result, IdxError> { + let mut args = vec![ + "--silent".to_string(), + "--show-error".to_string(), + "--location".to_string(), + "--fail".to_string(), + "--compressed".to_string(), + ]; + for (name, value) in headers { + args.push("--header".to_string()); + args.push(format!("{name}: {value}")); + } + args.push(url.to_string()); + + let output = curl_impersonate::run_owned(stage, &args)?; + Ok(output.stdout) +} + +fn json_headers() -> Vec<(String, String)> { + vec![ + ("User-Agent".to_string(), USER_AGENT.to_string()), + ( + "Accept".to_string(), + "application/json,text/plain,*/*".to_string(), + ), + ( + "Accept-Language".to_string(), + "id-ID,id;q=0.9,en-US;q=0.8,en;q=0.7".to_string(), + ), + ("Referer".to_string(), announcement_listing_url()), + ] +} + +fn pdf_headers() -> Vec<(String, String)> { + vec![ + ("User-Agent".to_string(), USER_AGENT.to_string()), + ( + "Accept".to_string(), + "application/pdf,application/octet-stream,*/*;q=0.8".to_string(), + ), + ( + "Accept-Language".to_string(), + "id-ID,id;q=0.9,en-US;q=0.8,en;q=0.7".to_string(), + ), + ("Referer".to_string(), announcement_listing_url()), + ] +} + +fn item_matches_family(item: &AnnouncementItem, query: &DiscoveryQuery) -> bool { + let title = item.title.to_ascii_uppercase(); + query + .title_needles + .iter() + .any(|needle| title.contains(needle)) +} + +fn item_pdf_attachments(item: &AnnouncementItem) -> Vec { + let attachments = if item.attachments.is_empty() { + item.pdf_path + .as_deref() + .and_then(parse_pdf_path) + .unwrap_or_default() + } else { + item.attachments.clone() + }; + + attachments + .into_iter() + .filter(|attachment| { + attachment + .full_save_path + .to_ascii_lowercase() + .ends_with(".pdf") + }) + .collect() +} + +fn attachment_label(attachment: &AnnouncementAttachment) -> String { + attachment + .original_filename + .as_deref() + .or(attachment.pdf_filename.as_deref()) + .unwrap_or_default() + .trim() + .to_ascii_lowercase() +} + +fn parse_pdf_path(raw: &str) -> Option> { + serde_json::from_str::>(raw).ok() +} + +fn attachment_is_attachment(attachment: &AnnouncementAttachment) -> bool { + match attachment.is_attachment.as_ref() { + Some(serde_json::Value::Bool(value)) => *value, + Some(serde_json::Value::Number(value)) => value.as_i64() == Some(1), + Some(serde_json::Value::String(value)) => { + matches!(value.trim(), "1" | "true" | "TRUE" | "True") + } + _ => attachment_label(attachment).contains("lamp"), + } +} + +fn clean_option(value: Option<&str>) -> Option { + value + .map(str::trim) + .filter(|value| !value.is_empty()) + .map(str::to_string) +} + +fn percent_encode(value: &str) -> String { + let mut encoded = String::with_capacity(value.len()); + for byte in value.bytes() { + match byte { + b'A'..=b'Z' | b'a'..=b'z' | b'0'..=b'9' | b'-' | b'_' | b'.' | b'~' => { + encoded.push(byte as char) + } + b' ' => encoded.push_str("%20"), + other => encoded.push_str(&format!("%{other:02X}")), + } + } + encoded +} + +#[cfg(test)] +mod tests { + use super::{ + AnnouncementPage, IDX_ANNOUNCEMENT_LISTING_URL, OwnershipReportFamily, + build_announcement_query_url, parse_announcement_page, select_latest_ownership_reports, + validate_pdf_payload, + }; + use crate::error::IdxError; + + #[test] + fn builds_announcement_query_url() { + let url = build_announcement_query_url("pemegang saham di atas 5", 1, 10); + assert_eq!( + url, + "https://www.idx.co.id/primary/NewsAnnouncement/GetAllAnnouncement?keywords=pemegang%20saham%20di%20atas%205&pageNumber=1&pageSize=10&lang=id" + ); + } + + #[test] + fn parses_fixture_and_selects_latest_above_5_reports() { + let raw = include_str!("../../tests/fixtures/idx_announcement_kepemilikan.json"); + let page = parse_announcement_page(raw).expect("announcement fixture should parse"); + let discovered = select_latest_ownership_reports( + &page, + "https://www.idx.co.id/primary/NewsAnnouncement/GetAllAnnouncement?keywords=pemegang%20saham%20di%20atas%205&pageNumber=1&pageSize=10&lang=id", + OwnershipReportFamily::AboveFivePercent, + ) + .expect("should select ownership reports"); + + assert_eq!(discovered.len(), 2); + assert_eq!( + discovered[0].family, + OwnershipReportFamily::AboveFivePercent + ); + assert_eq!(discovered[0].listing_page_url, IDX_ANNOUNCEMENT_LISTING_URL); + assert_eq!(discovered[0].publish_date, "2026-03-27T16:34:20"); + assert_eq!( + discovered[0].pdf_url, + "https://www.idx.co.id/StaticData/NewsAndAnnouncement/ANNOUNCEMENTSTOCK/From_EREP/202603/5d31bb6f49_announcement.pdf" + ); + assert_eq!( + discovered[0].original_filename.as_deref(), + Some("20260327_Semua Emiten Saham_Pengumuman Bursa_32055594.pdf") + ); + assert!(!discovered[0].is_attachment); + assert_eq!( + discovered[1].pdf_url, + "https://www.idx.co.id/StaticData/NewsAndAnnouncement/ANNOUNCEMENTSTOCK/From_EREP/202603/4f5c4efc6f_bf70f249ac_lamp1.pdf" + ); + assert_eq!( + discovered[1].original_filename.as_deref(), + Some("20260327_Semua Emiten Saham_Pengumuman Bursa_32055594_lamp1.pdf") + ); + assert!(discovered[1].is_attachment); + } + + #[test] + fn selects_above_1_reports() { + let raw = r#"{ + "Items": [ + { + "PublishDate": "2026-03-10T12:09:09", + "Title": "Pemegang Saham di atas 1% (KSEI)", + "AnnouncementType": "", + "Code": " Semua Emiten Saham ", + "Attachments": [ + { + "PDFFilename": "d67ebf37e6_10d4080288.pdf", + "FullSavePath": "https://www.idx.co.id/StaticData/NewsAndAnnouncement/ANNOUNCEMENTSTOCK/From_EREP/202603/d67ebf37e6_10d4080288.pdf", + "IsAttachment": 0, + "OriginalFilename": "20260310_Semua Emiten Saham_Pengumuman Bursa_32052554.pdf" + }, + { + "PDFFilename": "b9b638e5a8_8928aca255.pdf", + "FullSavePath": "https://www.idx.co.id/StaticData/NewsAndAnnouncement/ANNOUNCEMENTSTOCK/From_EREP/202603/b9b638e5a8_8928aca255.pdf", + "IsAttachment": 1, + "OriginalFilename": "20260310_Semua Emiten Saham_Pengumuman Bursa_32052554_lamp1.pdf" + } + ], + "PdfPath": "" + } + ], + "ItemCount": 1, + "PageCount": 1 + }"#; + + let page = parse_announcement_page(raw).expect("above-1 fixture should parse"); + let discovered = select_latest_ownership_reports( + &page, + "https://www.idx.co.id/primary/NewsAnnouncement/GetAllAnnouncement?keywords=pemegang%20saham%20di%20atas%201&pageNumber=1&pageSize=10&lang=id", + OwnershipReportFamily::AboveOnePercent, + ) + .expect("should select above-1 reports"); + + assert_eq!(discovered.len(), 2); + assert_eq!(discovered[0].family, OwnershipReportFamily::AboveOnePercent); + assert_eq!(discovered[0].code.as_deref(), Some("Semua Emiten Saham")); + assert_eq!( + discovered[0].original_filename.as_deref(), + Some("20260310_Semua Emiten Saham_Pengumuman Bursa_32052554.pdf") + ); + assert!(!discovered[0].is_attachment); + assert_eq!( + discovered[1].pdf_url, + "https://www.idx.co.id/StaticData/NewsAndAnnouncement/ANNOUNCEMENTSTOCK/From_EREP/202603/b9b638e5a8_8928aca255.pdf" + ); + assert!(discovered[1].is_attachment); + } + + #[test] + fn selects_investor_type_breakdown_reports() { + let raw = r#"{ + "Items": [ + { + "PublishDate": "2026-03-02T16:14:37", + "Title": "Data KSEI terkait Kepemilikan Saham Perusahaan Tercatat Berdasarkan Tipe Investor per 27 Februari 2026", + "AnnouncementType": "", + "Code": " ", + "Attachments": [ + { + "PDFFilename": "20260302_Pengumuman Bursa_32040089.pdf", + "FullSavePath": "https://www.idx.co.id/StaticData/NewsAndAnnouncement/ANNOUNCEMENTSTOCK/From_EREP/202603/43025665c8_fda8953269.pdf", + "IsAttachment": 0, + "OriginalFilename": "20260302_Pengumuman Bursa_32040089.pdf" + }, + { + "PDFFilename": "20260302_Pengumuman Bursa_32040089_lamp1.pdf", + "FullSavePath": "https://www.idx.co.id/StaticData/NewsAndAnnouncement/ANNOUNCEMENTSTOCK/From_EREP/202603/db5b5a86e1_2e2b3d976a.pdf", + "IsAttachment": 1, + "OriginalFilename": "20260302_Pengumuman Bursa_32040089_lamp1.pdf" + } + ], + "PdfPath": "" + } + ], + "ItemCount": 1, + "PageCount": 1 + }"#; + + let page = parse_announcement_page(raw).expect("investor-type fixture should parse"); + let discovered = select_latest_ownership_reports( + &page, + "https://www.idx.co.id/primary/NewsAnnouncement/GetAllAnnouncement?keywords=kepemilikan%20saham%20perusahaan%20tercatat&pageNumber=1&pageSize=10&lang=id", + OwnershipReportFamily::InvestorTypeBreakdown, + ) + .expect("should select investor-type reports"); + + assert_eq!(discovered.len(), 2); + assert_eq!( + discovered[0].original_filename.as_deref(), + Some("20260302_Pengumuman Bursa_32040089.pdf") + ); + assert_eq!( + discovered[1].original_filename.as_deref(), + Some("20260302_Pengumuman Bursa_32040089_lamp1.pdf") + ); + } + + #[test] + fn falls_back_to_pdf_path_when_attachments_are_missing() { + let raw = r#"{ + "Items": [ + { + "PublishDate": "2026-03-04T08:54:00", + "Title": "Pemegang Saham di atas 5% (KSEI)", + "AnnouncementType": "", + "Code": "Semua Emiten Saham", + "Attachments": [], + "PdfPath": "[{\"PDFFilename\":\"20260305_LKS_KSEI_000043_lamp1.pdf\",\"FullSavePath\":\"https://www.idx.co.id/StaticData/NewsAndAnnouncement/ANNOUNCEMENTSTOCK/From_EREP/202603/ec835d7451_c30c886a1a.pdf\",\"IsAttachment\":\"1\",\"OriginalFilename\":\"20260305_LKS_KSEI_000043_lamp1.pdf\"}]" + } + ], + "ItemCount": 1, + "PageCount": 1 + }"#; + + let page: AnnouncementPage = parse_announcement_page(raw).expect("fallback page parses"); + let discovered = select_latest_ownership_reports( + &page, + "https://www.idx.co.id/primary/NewsAnnouncement/GetAllAnnouncement?keywords=pemegang%20saham%20di%20atas%205&pageNumber=1&pageSize=10&lang=id", + OwnershipReportFamily::AboveFivePercent, + ) + .expect("fallback attachment should be parsed"); + + assert_eq!(discovered.len(), 1); + assert_eq!( + discovered[0].pdf_url, + "https://www.idx.co.id/StaticData/NewsAndAnnouncement/ANNOUNCEMENTSTOCK/From_EREP/202603/ec835d7451_c30c886a1a.pdf" + ); + assert_eq!(discovered[0].code.as_deref(), Some("Semua Emiten Saham")); + assert!(discovered[0].is_attachment); + } + + #[test] + fn rejects_html_instead_of_announcement_json() { + let err = parse_announcement_page("blocked") + .expect_err("html response must fail"); + assert!(matches!(err, IdxError::Http(_))); + } + + #[test] + fn accepts_pdf_header() { + validate_pdf_payload(b"%PDF-1.7\n1 0 obj\n").expect("pdf header should pass"); + } + + #[test] + fn rejects_html_instead_of_pdf() { + let err = validate_pdf_payload(b"blocked") + .expect_err("html body must fail"); + assert!(matches!(err, IdxError::Http(_))); + } +} diff --git a/tests/cli.rs b/tests/cli.rs index 1645a8b..0b7c91a 100644 --- a/tests/cli.rs +++ b/tests/cli.rs @@ -1,5 +1,8 @@ use std::fs; +use std::io::{Read, Write}; +use std::net::TcpListener; use std::path::{Path, PathBuf}; +use std::thread; use assert_cmd::Command; use predicates::prelude::*; @@ -38,6 +41,30 @@ fn test_bin(name: &str) -> Command { bin_with_root(&root) } +fn spawn_single_response_server(content_type: &str, body: impl Into>) -> String { + let listener = TcpListener::bind("127.0.0.1:0").expect("bind local test server"); + let addr = listener.local_addr().expect("local addr"); + let content_type = content_type.to_string(); + let body = body.into(); + + thread::spawn(move || { + let (mut stream, _) = listener.accept().expect("accept test connection"); + let mut buf = [0u8; 2048]; + let _ = stream.read(&mut buf); + + let headers = format!( + "HTTP/1.1 200 OK\r\nContent-Type: {content_type}\r\nContent-Length: {}\r\nConnection: close\r\n\r\n", + body.len() + ); + stream + .write_all(headers.as_bytes()) + .expect("write response headers"); + stream.write_all(&body).expect("write response body"); + }); + + format!("http://{addr}") +} + #[test] fn help_works() { test_bin("help").arg("--help").assert().success(); @@ -580,6 +607,108 @@ fn ownership_import_fetch_bing_reports_unsupported() { )); } +#[test] +fn ownership_discover_lists_fixture_candidates() { + let body = fs::read_to_string("tests/fixtures/idx_announcement_kepemilikan.json") + .expect("read ownership discovery fixture"); + let json_url = spawn_single_response_server("application/json", body); + + test_bin("ownership-discover") + .env("IDX_CURL_IMPERSONATE_BIN", "curl") + .env("IDX_OWNERSHIP_ANNOUNCEMENT_API_URL", &json_url) + .env( + "IDX_OWNERSHIP_ANNOUNCEMENT_PAGE_URL", + "http://127.0.0.1/pengumuman", + ) + .args([ + "ownership", + "discover", + "--family", + "above5", + "--limit", + "2", + ]) + .assert() + .success() + .stdout(predicate::str::contains("Above 5%")) + .stdout(predicate::str::contains( + "20260327_Semua Emiten Saham_Pengumuman Bursa_32055594.pdf", + )) + .stdout(predicate::str::contains( + "20260327_Semua Emiten Saham_Pengumuman Bursa_32055594_lamp1.pdf", + )); +} + +#[test] +fn ownership_discover_supports_above1_family() { + let body = r#"{ + "Items": [ + { + "PublishDate": "2026-03-10T12:09:09", + "Title": "Pemegang Saham di atas 1% (KSEI)", + "AnnouncementType": "", + "Code": "Semua Emiten Saham", + "Attachments": [ + { + "PDFFilename": "d67ebf37e6_10d4080288.pdf", + "FullSavePath": "https://www.idx.co.id/StaticData/NewsAndAnnouncement/ANNOUNCEMENTSTOCK/From_EREP/202603/d67ebf37e6_10d4080288.pdf", + "IsAttachment": 0, + "OriginalFilename": "20260310_Semua Emiten Saham_Pengumuman Bursa_32052554.pdf" + }, + { + "PDFFilename": "b9b638e5a8_8928aca255.pdf", + "FullSavePath": "https://www.idx.co.id/StaticData/NewsAndAnnouncement/ANNOUNCEMENTSTOCK/From_EREP/202603/b9b638e5a8_8928aca255.pdf", + "IsAttachment": 1, + "OriginalFilename": "20260310_Semua Emiten Saham_Pengumuman Bursa_32052554_lamp1.pdf" + } + ], + "PdfPath": "" + } + ], + "ItemCount": 1, + "PageSize": 10, + "PageNumber": 1, + "PageCount": 1 + }"#; + let json_url = spawn_single_response_server("application/json", body.to_string()); + + test_bin("ownership-discover-above1") + .env("IDX_CURL_IMPERSONATE_BIN", "curl") + .env("IDX_OWNERSHIP_ANNOUNCEMENT_API_URL", &json_url) + .env( + "IDX_OWNERSHIP_ANNOUNCEMENT_PAGE_URL", + "http://127.0.0.1/pengumuman", + ) + .args([ + "ownership", + "discover", + "--family", + "above1", + "--limit", + "2", + ]) + .assert() + .success() + .stdout(predicate::str::contains("Above 1%")) + .stdout(predicate::str::contains( + "20260310_Semua Emiten Saham_Pengumuman Bursa_32052554_lamp1.pdf", + )); +} + +#[test] +fn ownership_import_url_rejects_html_response_before_pdf_parse() { + let html_url = spawn_single_response_server( + "text/html; charset=utf-8", + "blocked", + ); + + test_bin("ownership-import-url-html") + .args(["ownership", "import", "--url", &html_url]) + .assert() + .failure() + .stderr(predicate::str::contains("returned HTML instead of a PDF")); +} + #[test] fn technical_serves_stale_cache_on_provider_failure_with_warning() { let root = test_env_dir("technical-stale"); diff --git a/tests/fixtures/idx_announcement_kepemilikan.json b/tests/fixtures/idx_announcement_kepemilikan.json new file mode 100644 index 0000000..a92d5c5 --- /dev/null +++ b/tests/fixtures/idx_announcement_kepemilikan.json @@ -0,0 +1,48 @@ +{ + "Items": [ + { + "Id": "20260327163420-Peng-LKS-00043/BEI.PLP/03-2026_id-id", + "AnnouncementNo": "Peng-LKS-00043/BEI.PLP/03-2026", + "PublishDate": "2026-03-27T16:34:20", + "Title": "Pemegang Saham di atas 5% (KSEI)", + "AnnouncementType": "", + "Code": "Semua Emiten Saham", + "Attachments": [ + { + "PDFFilename": "20260327_Semua Emiten Saham_Pengumuman Bursa_32055594.pdf", + "FullSavePath": "https://www.idx.co.id/StaticData/NewsAndAnnouncement/ANNOUNCEMENTSTOCK/From_EREP/202603/5d31bb6f49_announcement.pdf", + "IsAttachment": 0, + "OriginalFilename": "20260327_Semua Emiten Saham_Pengumuman Bursa_32055594.pdf" + }, + { + "PDFFilename": "20260327_Semua Emiten Saham_Pengumuman Bursa_32055594_lamp1.pdf", + "FullSavePath": "https://www.idx.co.id/StaticData/NewsAndAnnouncement/ANNOUNCEMENTSTOCK/From_EREP/202603/4f5c4efc6f_bf70f249ac_lamp1.pdf", + "IsAttachment": 1, + "OriginalFilename": "20260327_Semua Emiten Saham_Pengumuman Bursa_32055594_lamp1.pdf" + } + ], + "PdfPath": "[{\"PDFFilename\":\"20260327_Semua Emiten Saham_Pengumuman Bursa_32055594.pdf\",\"FullSavePath\":\"https://www.idx.co.id/StaticData/NewsAndAnnouncement/ANNOUNCEMENTSTOCK/From_EREP/202603/5d31bb6f49_announcement.pdf\",\"IsAttachment\":\"0\",\"OriginalFilename\":\"20260327_Semua Emiten Saham_Pengumuman Bursa_32055594.pdf\"},{\"PDFFilename\":\"20260327_Semua Emiten Saham_Pengumuman Bursa_32055594_lamp1.pdf\",\"FullSavePath\":\"https://www.idx.co.id/StaticData/NewsAndAnnouncement/ANNOUNCEMENTSTOCK/From_EREP/202603/4f5c4efc6f_bf70f249ac_lamp1.pdf\",\"IsAttachment\":\"1\",\"OriginalFilename\":\"20260327_Semua Emiten Saham_Pengumuman Bursa_32055594_lamp1.pdf\"}]" + }, + { + "Id": "20260328114739-LK/28032026/0001/1_id-id", + "AnnouncementNo": "LK/28032026/0001/1", + "PublishDate": "2026-03-28T11:47:39", + "Title": "LAPORAN KEPEMILIKAN ATAU SETIAP PERUBAHAN KEPEMILIKAN SAHAM PERUSAHAAN TERBUKA", + "AnnouncementType": "LKS", + "Code": "CASH", + "Attachments": [ + { + "PDFFilename": "LK-28032026-2468-00.pdf-0", + "FullSavePath": "https://www.idx.co.id/StaticData/NewsAndAnnouncement/ANNOUNCEMENTSTOCK/From_KSEI/LK-28032026-2468-00.pdf-0.pdf", + "IsAttachment": 1, + "OriginalFilename": "LK-28032026-2468-00.pdf" + } + ], + "PdfPath": "[{\"PDFFilename\":\"LK-28032026-2468-00.pdf-0\",\"FullSavePath\":\"https://www.idx.co.id/StaticData/NewsAndAnnouncement/ANNOUNCEMENTSTOCK/From_KSEI/LK-28032026-2468-00.pdf-0.pdf\",\"IsAttachment\":1,\"OriginalFilename\":\"LK-28032026-2468-00.pdf\"}]" + } + ], + "ItemCount": 2, + "PageSize": 10, + "PageNumber": 1, + "PageCount": 1 +} From 9d406ba3eedfa90d2cabf5523a8575ee5d7f75d1 Mon Sep 17 00:00:00 2001 From: 0xrsydn Date: Mon, 30 Mar 2026 11:58:53 +0700 Subject: [PATCH 2/2] Add IDX above1 ownership discovery and import --- AGENTS.md | 31 ++++++++++++++++++++++++------- 1 file changed, 24 insertions(+), 7 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index af599f4..644db20 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -22,7 +22,8 @@ src/ ├── cli/ # Command handlers (clap derive structs) │ ├── stocks.rs # stocks quote/history/technical/fundamental/... │ ├── config.rs # config get/set/init/path -│ └── cache.rs # cache info/clear +│ ├── cache.rs # cache info/clear +│ └── ownership.rs # ownership import/query commands ├── api/ # Data providers (trait-based abstraction) │ ├── mod.rs # MarketDataProvider trait + factory functions │ ├── types.rs # All domain types (Quote, Ohlc, Fundamentals, ...) @@ -43,6 +44,22 @@ src/ - **Yahoo** = automatic fallback for history/OHLCV (MSN doesn't support IDX history) - Configurable: `IDX_PROVIDER=msn|yahoo`, `IDX_HISTORY_PROVIDER=auto|yahoo|msn` +## Current Status +- Automated coverage is healthy: `cargo test` currently passes with 122 tests (86 unit, 36 integration). +- Live `stocks` commands are implemented and smoke-tested for: `quote`, `history`, `technical`, `growth`, `valuation`, `risk`, `fundamental`, `compare`, `profile`, `financials`, `earnings`, `sentiment`, `insights`, `news`, `screen`. +- `stocks history --history-provider msn` is intentionally unsupported for IDX; `auto` falls back to Yahoo. +- `ownership import --fetch-bing` is still intentionally unsupported; Bing client groundwork exists but the CLI path is deferred. + +## Known Hardening Gaps +- MSN-only commands still bypass the shared cache/offline path in `src/cli/stocks.rs`; `--offline` is not reliable for `profile`/`financials`/`earnings`/`sentiment`/`insights`/`news`/`screen`. +- Core quote flow has a verified `--offline --no-cache` bug: stale cache can still be served. +- Startup/config failures do not yet honor the JSON error contract; runtime failures do. +- `stocks screen --filter` and `--region` still silently coerce invalid values instead of rejecting them. +- Some live MSN output is incomplete or misleading: + - `profile` can return sparse fields. + - `insights.last_updated` is still empty. + - `financials` table output has malformed negative-number formatting in some rows. + ## Development ```bash nix develop # enter dev shell @@ -64,15 +81,15 @@ cargo test # all tests pass 1. **Schema-driven** — define types first, build logic around them. Types are the spec. 2. **Functional approach** — pure parse/transform functions (`parse_*`, `normalize_*`), no hidden state. 3. **Data types heavy** — rich enums, newtypes, composite structs. Precision via integer representations (basis points for %, i64 for shares). -4. **Provider abstraction** — all data access through traits, never call Yahoo/MSN directly from commands. +4. **Provider abstraction first** — all data access should flow through traits/factories. Note: current MSN-only stock commands still instantiate `MsnProvider` directly in `src/cli/stocks.rs`; removing that split path is an active hardening target. 5. **Sync only** — no tokio/async. CLI tool, ureq is sufficient. 6. **Test with fixtures** — never hit live APIs in tests. Mock provider + fixture JSON. 7. **Output contract** — table to stdout (humans), `--output json` (machines), errors to stderr. 8. **Feature-gated modules** — `ownership` feature for SQLite dep, keeps base binary lean. ## Docs -Detailed specs live in `docs-internal/` (gitignored — internal strategy): -- `docs-internal/SPEC.md` — system design, command tree, milestones -- `docs-internal/TODO.md` — sprint breakdown -- `docs-internal/ownership/SPEC.md` — ownership module architecture -- `docs-internal/ownership/TODO.md` — ownership sprint plan +Start with the repo-visible docs: +- `FEATURE_SPEC.md` — current hardening backlog and CLI truth-pass expectations +- `TODO.md` — working task list, including latest smoke findings +- `docs/ARCHITECTURE.md` — provider/capability design and error strategy +- `docs/CONVENTIONS.md` — repo conventions