mirror of
https://github.com/0xrsydn/idx-cli.git
synced 2026-08-07 01:33:52 +00:00
ownership: add IDX above1 discovery and import
Refactor KSEI PDF parsing for the live above-1 holder-register layout, add IDX announcement discovery plus browser-impersonated PDF fetches, and update the ownership docs to mark Batch 1 complete and scope Batch 2 around above1 hardening and unsupported legacy inputs.
This commit is contained in:
parent
b431a8c251
commit
a36648fd89
13 changed files with 1716 additions and 464 deletions
1
.gitignore
vendored
1
.gitignore
vendored
|
|
@ -2,6 +2,7 @@ node_modules/
|
||||||
target/
|
target/
|
||||||
.direnv/
|
.direnv/
|
||||||
result
|
result
|
||||||
|
tmp/
|
||||||
|
|
||||||
# Internal docs (specs, research, business strategy)
|
# Internal docs (specs, research, business strategy)
|
||||||
docs-internal/
|
docs-internal/
|
||||||
|
|
|
||||||
541
FEATURE_SPEC.md
541
FEATURE_SPEC.md
|
|
@ -1,320 +1,249 @@
|
||||||
# Feature Spec: MSN Finance Full Coverage
|
# Feature Spec: Current Coverage and Remaining Core Work
|
||||||
|
|
||||||
**Branch:** `feat/msn-full`
|
**Status:** Active - updated against the current repo state on 2026-03-28
|
||||||
**Status:** Draft — pending review
|
**Current focus:** Close the remaining core gaps before expanding into new market and watchlist surfaces
|
||||||
**Reference:** `origin/dev/rubick` (Go implementation by rubick)
|
**Reference:** `origin/dev/rubick` (endpoint reference only, not the current implementation plan)
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Background
|
## Purpose
|
||||||
|
|
||||||
The Rust CLI currently supports two MSN endpoints:
|
This is no longer a "port MSN from zero" spec.
|
||||||
- `Finance/Quotes` → `quote()`
|
|
||||||
- `api.msn.com/keyratios` → `fundamentals()`
|
|
||||||
|
|
||||||
The rubick Go project (friend's scraper) demonstrates a much wider set of MSN Finance endpoints covering equities, financials, earnings, charts, sentiment, insights, and news — all using the same public API key. This spec defines the full porting roadmap from Go → Rust.
|
The CLI already ships MSN-backed support for:
|
||||||
|
- `profile`
|
||||||
|
- `financials`
|
||||||
|
- `earnings`
|
||||||
|
- `sentiment`
|
||||||
|
- `insights`
|
||||||
|
- `news`
|
||||||
|
- `screen`
|
||||||
|
|
||||||
|
Use this document to track what is still open in core functionality.
|
||||||
|
Use `TODO.md` as the execution log and smoke-history record.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Verified Current State
|
||||||
|
|
||||||
|
- Current automated coverage is `153` tests: `102` unit and `51` integration.
|
||||||
|
- Reusable smoke coverage exists via `scripts/live-smoke.sh`; command groups are documented in `docs/SMOKE.md`.
|
||||||
|
- The latest smoke notes in `TODO.md` report passing live table and JSON checks for all shipped `stocks` commands.
|
||||||
|
- Cache/offline parity, JSON startup-error handling, screener input validation, and the recent MSN output cleanups have already been completed.
|
||||||
|
- KSEI ownership import/query is now verified end-to-end from a local March 2026 PDF file into SQLite, with direct CLI reads through `ownership releases` and `ownership ticker`.
|
||||||
|
- Remote ownership ingestion should now be treated as an IDX PDF discovery-and-fetch problem first, not a hardcoded KSEI PDF URL problem: direct IDX announcement PDFs exist, but the hashed asset URL must be discovered from an IDX listing/announcement surface and fetched with browser-like behavior.
|
||||||
|
- The CLI now has an explicit `idx ownership discover` surface that enumerates the latest hashed BEI ownership report URLs.
|
||||||
|
- Live verification on `2026-03-29` confirmed that the parser-compatible discovered source is currently the `Pemegang Saham di atas 1% (KSEI)` `lamp1` attachment, and `idx ownership import --url` now works end to end for that discovered BEI PDF.
|
||||||
|
- The currently discoverable `above 5%` and `investor-type` BEI families remain different schemas and should now be treated as legacy / unsupported input outside the current holder-register parser contract.
|
||||||
|
|
||||||
|
This means the main gap is no longer endpoint coverage.
|
||||||
|
The remaining work is architecture cleanup, a few correctness edge cases, and selective UX expansion on top of already-shipped commands.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Coverage Snapshot
|
||||||
|
|
||||||
|
| Area | CLI | Status | Notes |
|
||||||
|
| --- | --- | --- | --- |
|
||||||
|
| Quotes | `idx stocks quote` | Implemented | Cached, smoke-tested, and covered by integration tests |
|
||||||
|
| History | `idx stocks history` | Partial | Works today via Yahoo/history-provider routing; explicit MSN history remains unsupported for IDX |
|
||||||
|
| Technical | `idx stocks technical` | Implemented | Uses the cached history path |
|
||||||
|
| Growth | `idx stocks growth` | Implemented | Shipped and exercised |
|
||||||
|
| Valuation | `idx stocks valuation` | Implemented | Shipped and exercised |
|
||||||
|
| Risk | `idx stocks risk` | Implemented | Shipped and exercised |
|
||||||
|
| Fundamental | `idx stocks fundamental` | Implemented | Shipped and exercised |
|
||||||
|
| Compare | `idx stocks compare` | Implemented | Shipped and exercised |
|
||||||
|
| Company profile | `idx stocks profile` | Implemented | Cache/offline parity and fixture-backed coverage are in place |
|
||||||
|
| Financial statements | `idx stocks financials` | Implemented with gaps | Output cleanup landed; statement filter flags and richer period controls are still missing |
|
||||||
|
| Earnings | `idx stocks earnings` | Implemented with gaps | History/forecast split is rendered; filter flags are still missing |
|
||||||
|
| Sentiment | `idx stocks sentiment` | Implemented | Fixture-backed CLI coverage exists |
|
||||||
|
| Insights | `idx stocks insights` | Implemented | Summary/highlights/risks/`last_updated` mapping was corrected and tested |
|
||||||
|
| News | `idx stocks news` | Implemented | Fixture-backed CLI coverage exists |
|
||||||
|
| Screener | `idx stocks screen` | Implemented with gaps | Validation landed; expression/preset workflow is still future work |
|
||||||
|
| MSN charts | `idx stocks history --history-provider msn` | Missing | Explicit MSN history still returns unsupported for IDX |
|
||||||
|
| KSEI ownership import/query | `idx ownership import --file`, `idx ownership import --url`, `idx ownership releases`, `idx ownership ticker` | Implemented with gaps | Local PDF import and SQLite-backed query flow are verified against the March 2026 KSEI release; remote IDX import now works for the discovered `above 1%` `lamp1` BEI attachment, while legacy `above 5%` and `investor-type` BEI report families still need explicit unsupported-input handling |
|
||||||
|
| Bing ownership CLI | `idx ownership import --fetch-bing` | Not implemented | Client groundwork exists, CLI import path is still deferred |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Resolved Since The Previous Revision
|
||||||
|
|
||||||
|
The following items should no longer be treated as active backlog in this spec:
|
||||||
|
|
||||||
|
- CLI truth-pass baseline for shipped `stocks` commands
|
||||||
|
- Unified cache, offline, stale-cache, and `--no-cache` handling across the core and MSN-only stock commands
|
||||||
|
- Rejection of the conflicting `--offline --no-cache` flag combination
|
||||||
|
- JSON-aware startup/config failures, not just runtime failures
|
||||||
|
- Validation for `stocks screen --filter` and `--region`
|
||||||
|
- `profile` output remapping to prefer company/localized fields
|
||||||
|
- `insights` output remapping for summary, highlights, risks, and `last_updated`
|
||||||
|
- Signed-number and table-label cleanup for `financials`
|
||||||
|
- Table-mode cleanup for `earnings`
|
||||||
|
- Baseline parser and CLI regression coverage for the shipped MSN-only command set
|
||||||
|
- KSEI ownership parser hardening for the March 2026 live PDF layout, including the merged `DATE + SHARE_CODE` segment and `D`/`A` locality markers
|
||||||
|
- Real KSEI ownership CLI verification from local file import into SQLite (`7261` rows across `955` tickers on `2026-03-28`)
|
||||||
|
|
||||||
|
If any of the above regress, capture that in `TODO.md` as a new finding rather than reopening the old section here wholesale.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Remaining Core Gaps
|
||||||
|
|
||||||
|
### P0 - Correctness and architecture
|
||||||
|
|
||||||
|
#### 1. Unify provider and capability flow
|
||||||
|
|
||||||
|
Current state:
|
||||||
|
- `src/cli/stocks.rs` still directly constructs `MsnProvider::new(false)` for `profile`, `financials`, `earnings`, `sentiment`, `insights`, `news`, and `screen`.
|
||||||
|
- This keeps a split execution path alive even though cache/offline behavior is now mostly unified around `fetch_msn_with_cache`.
|
||||||
|
|
||||||
|
Why it matters:
|
||||||
|
- The architecture doc describes provider/capability-based flow, but the CLI still special-cases MSN-only commands at the handler layer.
|
||||||
|
- Future features will be harder to extend cleanly if this split remains.
|
||||||
|
|
||||||
|
Done when:
|
||||||
|
- Normal command handlers stop constructing `MsnProvider` directly.
|
||||||
|
- Provider selection and capability checks are centralized and consistent with `docs/ARCHITECTURE.md`.
|
||||||
|
|
||||||
|
#### 2. Fix screener row hygiene for incomplete data
|
||||||
|
|
||||||
|
Current state:
|
||||||
|
- `src/api/msn/map.rs` still defaults missing screener price data to `0.0` when constructing `Quote` rows.
|
||||||
|
|
||||||
|
Why it matters:
|
||||||
|
- A zero-priced row is not the same thing as a valid priced stock.
|
||||||
|
- This can silently admit incomplete market rows instead of rejecting or filtering them.
|
||||||
|
|
||||||
|
Done when:
|
||||||
|
- Screener rows without usable price data are filtered or rejected explicitly.
|
||||||
|
- Regression tests cover the chosen behavior.
|
||||||
|
|
||||||
|
#### 3. Decide the fundamentals fallback policy
|
||||||
|
|
||||||
|
Current state:
|
||||||
|
- Fundamentals can still fall back to industry-level metrics when company metrics are absent.
|
||||||
|
|
||||||
|
Why it matters:
|
||||||
|
- This can produce analysis that looks precise but is actually based on peer or category data.
|
||||||
|
|
||||||
|
Decision needed:
|
||||||
|
- Allow the fallback and annotate it clearly, or
|
||||||
|
- reject the fallback and surface missing company data explicitly.
|
||||||
|
|
||||||
|
Done when:
|
||||||
|
- The policy is explicit in code and reflected in output semantics.
|
||||||
|
|
||||||
|
#### 4. Harden Yahoo reliability edge cases
|
||||||
|
|
||||||
|
Open issues:
|
||||||
|
- Yahoo can still return `429` from datacenter IPs intermittently.
|
||||||
|
- SMA200 trend output can still show "Insufficient data" when fewer than `200` candles are returned.
|
||||||
|
|
||||||
|
Done when:
|
||||||
|
- The retry/fallback story for Yahoo failures is deliberate and documented.
|
||||||
|
- SMA200 behavior is either improved or clearly documented as expected.
|
||||||
|
|
||||||
|
#### 5. Automate IDX ownership source discovery and fetch
|
||||||
|
|
||||||
|
Current state:
|
||||||
|
- `idx ownership import --file <local-pdf>` is now working and verified against the March 2026 KSEI release.
|
||||||
|
- Direct IDX announcement PDFs exist, but the monthly file path is not safely hardcodable because the asset filename is hashed.
|
||||||
|
- The official BEI listing page for this feed is `https://www.idx.co.id/id/berita/pengumuman/`, and the page’s own Nuxt client fetches announcement data from `GET /primary/NewsAnnouncement/GetAllAnnouncement`.
|
||||||
|
- The BEI endpoint is now reverse-engineered enough for `idx ownership discover` to locate the current `Pemegang Saham di atas 1% (KSEI)`, `Pemegang Saham di atas 5% (KSEI)`, and `Kepemilikan Saham Perusahaan Tercatat Berdasarkan Tipe Investor` hashed PDFs.
|
||||||
|
- Live verification on `2026-03-29` shows that the discoverable `above 1%` `lamp1` attachment matches the raw KSEI holder-register shape that the current parser imports successfully.
|
||||||
|
- Live verification on `2026-03-29` also shows that the currently discoverable `above 5%` and `investor-type` BEI families do not match the holder-register schema the current parser imports.
|
||||||
|
- Product direction as of `2026-03-30` is to standardize on the `above 1%` holder-register structure for remote import; the other discovered families are legacy inputs that should be rejected clearly rather than parsed.
|
||||||
|
- Plain `curl`/`ureq` requests to IDX-hosted PDFs still get `403` from Cloudflare, while `curl-impersonate` inside the project `nix develop` environment has already been verified to return a real PDF for a March 2026 ownership source URL.
|
||||||
|
- The KSEI archive remains a secondary upstream and currently exposes monthly ZIP files that can be used later as fallback or cross-check input, but it is no longer the primary roadmap target.
|
||||||
|
|
||||||
|
Why it matters:
|
||||||
|
- The ownership feature now parses and stores live ownership data correctly from both local files and the currently discoverable `above 1%` BEI `lamp1` attachment.
|
||||||
|
- The remaining gap is not basic remote import anymore; it is hardening the supported `above 1%` path and clearly rejecting other discovered BEI schemas that do not match the holder-register contract.
|
||||||
|
|
||||||
|
Done when:
|
||||||
|
- The CLI can discover the latest parser-compatible IDX ownership PDF URL from an IDX listing/announcement surface without hardcoded monthly paths.
|
||||||
|
- Remote fetches use the same browser-impersonation strategy already established elsewhere in the repo instead of the current bare `ureq` path.
|
||||||
|
- `idx ownership import --url` can fetch and parse the current `above 1%` IDX-hosted PDF reliably.
|
||||||
|
- Unsupported BEI report families are classified and rejected explicitly instead of failing later with a generic zero-row parse error.
|
||||||
|
|
||||||
|
Roadmap:
|
||||||
|
1. Keep discovery and import flows centered on the parser-compatible `above 1%` family and its `lamp1` attachment.
|
||||||
|
2. Reuse the existing `curl-impersonate` pattern for browser-like PDF fetches.
|
||||||
|
3. Add schema classification / unsupported-input UX for the legacy `above 5%` and `investor-type` BEI PDFs.
|
||||||
|
4. Decide whether the default discover output should be `above1`-first, with legacy families retained only for diagnostic use.
|
||||||
|
5. Publish maintained SQLite snapshot artifacts and add `idx ownership sync` after remote IDX import is stable.
|
||||||
|
6. Optionally add KSEI ZIP/TXT ingest later as fallback or cross-check input.
|
||||||
|
|
||||||
|
### P1 - UX and output contract cleanup
|
||||||
|
|
||||||
|
Tasks:
|
||||||
|
- Add `financials` filters such as `--statement income|balance|cashflow`.
|
||||||
|
- Add `earnings` filters such as `--forecast|--history` and `--annual|--quarterly`.
|
||||||
|
- Review JSON payload consistency where symbol or context fields are still sparse.
|
||||||
|
- Decide whether `screen` stays under `stocks` long term or graduates into a richer dedicated surface later.
|
||||||
|
|
||||||
|
Done when:
|
||||||
|
- Existing shipped commands are easier to drive without changing product scope.
|
||||||
|
|
||||||
|
### P2 - Next feature work after the above is green
|
||||||
|
|
||||||
|
Priority order:
|
||||||
|
|
||||||
|
1. MSN Charts / `Finance/Charts`
|
||||||
|
- Reuse the existing `idx stocks history` command.
|
||||||
|
- Decide how to handle price-only timeframes safely.
|
||||||
|
|
||||||
|
2. Bing ownership CLI integration
|
||||||
|
- Reuse the existing client groundwork in `src/api/msn/bing.rs`.
|
||||||
|
- Define the import shape and output contract for `idx ownership import --fetch-bing`.
|
||||||
|
|
||||||
|
3. Ownership sync and snapshot distribution
|
||||||
|
- Keep IDX PDF discovery/fetch as the first milestone before starting this work.
|
||||||
|
- Publish maintained SQLite snapshot artifacts and add `idx ownership sync`.
|
||||||
|
- Treat the KSEI archive (`https://web.ksei.co.id/archive_download/holding_composition`) as fallback/backstop input, not the primary product ingest path.
|
||||||
|
|
||||||
|
4. Richer financial statements
|
||||||
|
- Decide whether to stay with the current single-period model or add multi-period fetch support.
|
||||||
|
|
||||||
|
5. New user-facing surfaces from `TODO.md`
|
||||||
|
- `market summary`
|
||||||
|
- `market movers`
|
||||||
|
- `market sectors`
|
||||||
|
- `screen query`
|
||||||
|
- `screen presets`
|
||||||
|
- `watchlist`
|
||||||
|
- `alerts`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Verification Gate
|
||||||
|
|
||||||
|
Do not treat a core refactor or new provider feature as complete until all of the following are true:
|
||||||
|
|
||||||
|
- `cargo build` passes
|
||||||
|
- `cargo clippy -- -D warnings` passes
|
||||||
|
- `cargo test` passes
|
||||||
|
- `scripts/live-smoke.sh --mode mock` passes
|
||||||
|
- the relevant live smoke groups pass for changed user-facing behavior
|
||||||
|
- `TODO.md` is updated with any new smoke finding, regression, or behavior change
|
||||||
|
|
||||||
|
Keep the detailed reusable smoke commands in `docs/SMOKE.md`.
|
||||||
|
Do not duplicate per-run results in this spec.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Endpoint Reference Appendix
|
||||||
|
|
||||||
MSN API key (public, embedded in MSN Money website):
|
MSN API key (public, embedded in MSN Money website):
|
||||||
|
|
||||||
```
|
```
|
||||||
0QfOX3Vn51YCzitbLaRkTTBadtWpgTN8NZLW0C1SEM
|
0QfOX3Vn51YCzitbLaRkTTBadtWpgTN8NZLW0C1SEM
|
||||||
```
|
```
|
||||||
|
|
||||||
Base URLs:
|
Base URLs:
|
||||||
- `https://assets.msn.com/service/` — core market data (Quotes, Charts, Equities, Earnings, Sentiment, Screener)
|
- `https://assets.msn.com/service/` - core market data (Quotes, Charts, Equities, Earnings, Sentiment, Screener)
|
||||||
- `https://api.msn.com/msn/v0/pages/finance/` — extended data (keyratios, insights, newsfeed)
|
- `https://api.msn.com/msn/v0/pages/finance/` - extended data (key ratios, insights, news feed)
|
||||||
- `https://services.bingapis.com/contentservices-finance.hedgefunddataprovider/api/v1/` — Bing ownership data
|
- `https://services.bingapis.com/contentservices-finance.hedgefunddataprovider/api/v1/` - Bing ownership data
|
||||||
|
|
||||||
---
|
Keep this appendix for endpoint discovery and future work.
|
||||||
|
Use the sections above as the actual implementation plan.
|
||||||
## Endpoints to Implement
|
|
||||||
|
|
||||||
### P0 — Core Completeness
|
|
||||||
|
|
||||||
#### 1. `Finance/Equities` — Company Profile
|
|
||||||
- **Method:** GET
|
|
||||||
- **URL:** `{MSN_ASSETS_BASE_URL}Finance/Equities?apikey={key}&ids={id}&wrapodata=false`
|
|
||||||
- **Returns:** `EquityData` — company name, description, sector, industry, website, employees, address, officers/executives
|
|
||||||
- **CLI use:** `idx stock profile BBCA` or folded into `info` subcommand
|
|
||||||
- **Rust struct:**
|
|
||||||
```rust
|
|
||||||
pub struct EquityData {
|
|
||||||
pub id: String,
|
|
||||||
pub symbol: String,
|
|
||||||
pub short_name: String,
|
|
||||||
pub long_name: String,
|
|
||||||
pub description: String,
|
|
||||||
pub sector: String,
|
|
||||||
pub industry: String,
|
|
||||||
pub website: String,
|
|
||||||
pub employees: i64,
|
|
||||||
pub address: String,
|
|
||||||
pub city: String,
|
|
||||||
pub country: String,
|
|
||||||
pub phone: String,
|
|
||||||
pub officers: Vec<Officer>,
|
|
||||||
}
|
|
||||||
|
|
||||||
pub struct Officer {
|
|
||||||
pub name: String,
|
|
||||||
pub title: String,
|
|
||||||
pub age: Option<i32>,
|
|
||||||
pub year_born: Option<i32>,
|
|
||||||
pub total_pay: Option<i64>,
|
|
||||||
}
|
|
||||||
```
|
|
||||||
- **Complexity:** Low
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
#### 2. `Finance/Equities/financialstatements` — Financial Statements
|
|
||||||
- **Method:** GET
|
|
||||||
- **URL:** `{MSN_ASSETS_BASE_URL}Finance/Equities/financialstatements?apikey={key}&ids={id}&wrapodata=false`
|
|
||||||
- **Returns:** Balance sheet, cash flow, income statement — each as a map of `{field: value}` keyed by line item name, with period metadata (reportDate, endDate, currency, source)
|
|
||||||
- **CLI use:** `idx stock financials BBCA [--statement income|balance|cashflow]`
|
|
||||||
- **Note:** Fields are dynamic (map-based), not fixed columns — render as table with row=line item, col=period if multiple periods returned
|
|
||||||
- **Rust struct:**
|
|
||||||
```rust
|
|
||||||
pub struct FinancialStatements {
|
|
||||||
pub instrument: InstrumentInfo,
|
|
||||||
pub balance_sheet: Option<BalanceSheet>,
|
|
||||||
pub cash_flow: Option<CashFlow>,
|
|
||||||
pub income_statement: Option<IncomeStatement>,
|
|
||||||
}
|
|
||||||
|
|
||||||
pub struct BalanceSheet {
|
|
||||||
pub current_assets: HashMap<String, f64>,
|
|
||||||
pub long_term_assets: HashMap<String, f64>,
|
|
||||||
pub current_liabilities: HashMap<String, f64>,
|
|
||||||
pub equity: HashMap<String, f64>,
|
|
||||||
pub currency: String,
|
|
||||||
pub report_date: String,
|
|
||||||
pub end_date: String,
|
|
||||||
}
|
|
||||||
|
|
||||||
// Similar pattern for CashFlow (financing/investing/operating) and IncomeStatement
|
|
||||||
```
|
|
||||||
- **Complexity:** Medium (dynamic maps → table rendering)
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### P1 — High Analyst Value
|
|
||||||
|
|
||||||
#### 3. `Finance/Events/Earnings` — Earnings History & Forecast
|
|
||||||
- **Method:** GET
|
|
||||||
- **URL:** `{MSN_ASSETS_BASE_URL}Finance/Events/Earnings?apikey={key}&ids={id}&wrapodata=false`
|
|
||||||
- **Returns:**
|
|
||||||
- `EpsLastYear`, `RevenueLastYear`
|
|
||||||
- `Forecast.annual` — 2 forward years: EpsForecast, RevenueForecast, GAAP/Normalized consensus
|
|
||||||
- `Forecast.quarterly` — next 4 quarters with same fields + EarningReleaseDate
|
|
||||||
- `History.annual` — 5 years: EpsActual, EpsSurprise, EpsSurprisePercent, RevenueActual, RevenueSurprise
|
|
||||||
- `History.quarterly` — ~12 quarters of actuals + surprises
|
|
||||||
- **CLI use:** `idx stock earnings BBCA [--forecast|--history] [--annual|--quarterly]`
|
|
||||||
- **Rust struct:**
|
|
||||||
```rust
|
|
||||||
pub struct EarningsReport {
|
|
||||||
pub eps_last_year: f64,
|
|
||||||
pub revenue_last_year: f64,
|
|
||||||
pub forecast: EarningsForecast,
|
|
||||||
pub history: EarningsHistory,
|
|
||||||
}
|
|
||||||
|
|
||||||
pub struct EarningsData {
|
|
||||||
pub eps_actual: Option<f64>,
|
|
||||||
pub eps_forecast: Option<f64>,
|
|
||||||
pub eps_surprise: Option<f64>,
|
|
||||||
pub eps_surprise_pct: Option<f64>,
|
|
||||||
pub revenue_actual: Option<f64>,
|
|
||||||
pub revenue_forecast: Option<f64>,
|
|
||||||
pub revenue_surprise: Option<f64>,
|
|
||||||
pub earning_release_date: Option<String>,
|
|
||||||
pub period_type: String, // e.g. "Q42025", "2025"
|
|
||||||
}
|
|
||||||
```
|
|
||||||
- **Complexity:** Medium (nested map keyed by period string)
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
#### 4. `Finance/Charts` — Price Chart / OHLCV History
|
|
||||||
- **Method:** GET
|
|
||||||
- **URL:** `{MSN_ASSETS_BASE_URL}Finance/Charts?apikey={key}&ids={id}&chartType={type}&wrapodata=false`
|
|
||||||
- **Chart types:** `1D`, `1W`, `1M`, `3M`, `6M`, `1Y`, `3Y`, `5Y`, `MAX`
|
|
||||||
- **Returns:** Series of `ChartPoint { time, open, high, low, close, price, volume }`
|
|
||||||
- **Note:** This unblocks the `history()` provider method — current implementation explicitly returns `Unsupported`. MSN charts don't guarantee OHLCV on all timeframes (1D is often price-only), so parse defensively.
|
|
||||||
- **CLI use:** `idx stock history BBCA --period 3M` (existing command, just needs this wired up)
|
|
||||||
- **Rust struct:**
|
|
||||||
```rust
|
|
||||||
pub struct ChartPoint {
|
|
||||||
pub time: String,
|
|
||||||
pub open: Option<f64>,
|
|
||||||
pub high: Option<f64>,
|
|
||||||
pub low: Option<f64>,
|
|
||||||
pub close: Option<f64>,
|
|
||||||
pub price: f64,
|
|
||||||
pub volume: Option<i64>,
|
|
||||||
}
|
|
||||||
```
|
|
||||||
- **Complexity:** Medium (parse series array, handle missing OHLCV gracefully)
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### P2 — Enrichment Layer
|
|
||||||
|
|
||||||
#### 5. `Finance/SentimentBrowser` — Crowd Sentiment
|
|
||||||
- **Method:** GET
|
|
||||||
- **URL:** `{MSN_ASSETS_BASE_URL}Finance/SentimentBrowser?apikey={key}&ids={id}&wrapodata=false`
|
|
||||||
- **Returns:** Per-period sentiment stats: bullish/bearish/neutral counts, time range name (e.g., "1D", "1W", "1M")
|
|
||||||
- **CLI use:** `idx stock sentiment BBCA`
|
|
||||||
- **Rust struct:**
|
|
||||||
```rust
|
|
||||||
pub struct SentimentData {
|
|
||||||
pub symbol: String,
|
|
||||||
pub statistics: Vec<SentimentPeriod>,
|
|
||||||
}
|
|
||||||
|
|
||||||
pub struct SentimentPeriod {
|
|
||||||
pub time_range: String, // "1D", "1W", "1M"
|
|
||||||
pub bullish: i32,
|
|
||||||
pub bearish: i32,
|
|
||||||
pub neutral: i32,
|
|
||||||
}
|
|
||||||
```
|
|
||||||
- **Complexity:** Low
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
#### 6. `api.msn.com/insights` — AI-Generated Insights
|
|
||||||
- **Method:** GET
|
|
||||||
- **URL:** `{MSN_API_BASE_URL}insights?apikey={key}&ids={id}&wrapodata=false`
|
|
||||||
- **Returns:** Summary text, highlights array, risks array, last updated timestamp
|
|
||||||
- **CLI use:** `idx stock insights BBCA`
|
|
||||||
- **Rust struct:**
|
|
||||||
```rust
|
|
||||||
pub struct InsightData {
|
|
||||||
pub id: String,
|
|
||||||
pub summary: String,
|
|
||||||
pub highlights: Vec<String>,
|
|
||||||
pub risks: Vec<String>,
|
|
||||||
pub last_updated: String,
|
|
||||||
}
|
|
||||||
```
|
|
||||||
- **Complexity:** Low
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
#### 7. `MSN/Feed/me` — Stock News Feed
|
|
||||||
- **Method:** GET
|
|
||||||
- **URL:** `{MSN_API_BASE_URL}` + entity feed params with stock ID
|
|
||||||
- **Returns:** News cards: title, URL, abstract, provider name, publish time, read time
|
|
||||||
- **CLI use:** `idx stock news BBCA [--limit 10]`
|
|
||||||
- **Rust struct:**
|
|
||||||
```rust
|
|
||||||
pub struct NewsItem {
|
|
||||||
pub id: String,
|
|
||||||
pub title: String,
|
|
||||||
pub url: String,
|
|
||||||
pub description: String,
|
|
||||||
pub provider: String,
|
|
||||||
pub published_at: String,
|
|
||||||
pub read_time_min: Option<i32>,
|
|
||||||
}
|
|
||||||
```
|
|
||||||
- **Complexity:** Medium (URL construction + response parsing needs rubick reference)
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
#### 8. `Finance/Screener` — IDX Universe Screener
|
|
||||||
- **Method:** POST
|
|
||||||
- **URL:** `{MSN_ASSETS_BASE_URL}Finance/Screener?apikey={key}&wrapodata=false`
|
|
||||||
- **Body:** `{ filter: [{key, keyGroup, isRange}], order: {key, dir}, returnValueType: [...], screenerType: "...", limit: 50 }`
|
|
||||||
- **Returns:** List of stocks with quote data (price, change, market cap, volume, 52w hi/lo, YTD return)
|
|
||||||
- **CLI use:** `idx screen [--preset top-gainers|top-losers|most-active|...]`
|
|
||||||
- **Complexity:** Medium (POST body construction, preset filter definitions)
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### P3 — Optional / Future
|
|
||||||
|
|
||||||
#### 9. Bing Ownership API — Institutional Holders
|
|
||||||
- **Base:** `https://services.bingapis.com/contentservices-finance.hedgefunddataprovider/api/v1/`
|
|
||||||
- **Endpoints:**
|
|
||||||
- `GetSecurityTopShareHolders`
|
|
||||||
- `GetSecurityTopBuyers` / `GetSecurityTopSellers`
|
|
||||||
- `GetSecurityTopNewShareHolders` / `GetSecurityTopExitedShareHolders`
|
|
||||||
- **CLI use:** `idx stock holders BBCA [--buyers|--sellers|--new|--exited]`
|
|
||||||
- **Note:** Separate base URL, may need different auth/headers than MSN. Validate working before implementing.
|
|
||||||
- **Complexity:** Low-Medium
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Implementation Plan
|
|
||||||
|
|
||||||
### Phase 1 — Extend `src/api/msn/`
|
|
||||||
1. Add `fetch_equities(symbol)` to `client.rs`
|
|
||||||
2. Add `fetch_financial_statements(symbol)` to `client.rs`
|
|
||||||
3. Add `fetch_earnings(symbol)` to `client.rs`
|
|
||||||
4. Add `fetch_charts(symbol, period)` to `client.rs`
|
|
||||||
5. Add corresponding parse functions to `parse.rs`
|
|
||||||
6. Expose via new methods on `MsnProvider` in `mod.rs`
|
|
||||||
|
|
||||||
### Phase 2 — New Rust structs in `src/api/msn/types.rs` (new file)
|
|
||||||
- Extract shared types (currently inline in `parse.rs`) into dedicated `types.rs`
|
|
||||||
- Add all new structs listed above
|
|
||||||
|
|
||||||
### Phase 3 — Wire CLI commands in `src/cli/stocks.rs`
|
|
||||||
New subcommands to add:
|
|
||||||
```
|
|
||||||
idx stock profile <SYMBOL> # Company info + officers
|
|
||||||
idx stock financials <SYMBOL> # Income / balance / cashflow
|
|
||||||
idx stock earnings <SYMBOL> # EPS history + forecast
|
|
||||||
idx stock sentiment <SYMBOL> # Crowd sentiment
|
|
||||||
idx stock insights <SYMBOL> # AI highlights + risks
|
|
||||||
idx stock news <SYMBOL> # News feed
|
|
||||||
idx screen # IDX screener (separate top-level command)
|
|
||||||
```
|
|
||||||
|
|
||||||
And unblock existing:
|
|
||||||
```
|
|
||||||
idx stock history <SYMBOL> # Wire MSN charts (currently Unsupported)
|
|
||||||
```
|
|
||||||
|
|
||||||
### Phase 4 — Output formatting
|
|
||||||
- Table output for financials (line item rows, period columns)
|
|
||||||
- Compact output for earnings (actual vs forecast vs surprise %)
|
|
||||||
- JSON output flag `--json` should work for all new commands
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Open Questions
|
|
||||||
|
|
||||||
1. **Chart OHLCV completeness** — rubick notes that MSN charts don't always return full OHLCV on short timeframes (e.g., 1D is price-only). Do we want to keep `history()` returning `Unsupported` for MSN and add a separate `charts()` method, or silently map price → close for compatibility?
|
|
||||||
|
|
||||||
2. **Financial statements period count** — The API returns one period per call (most recent). Do we want to add a bulk-fetch loop (e.g., fetch last 4 quarters separately) or just expose single-period for now?
|
|
||||||
|
|
||||||
3. **News feed URL construction** — needs exact param structure from rubick's `GetNewsFeed()` Go implementation. Worth a closer look before implementing.
|
|
||||||
|
|
||||||
4. **Screener presets** — rubick defines filter key constants (e.g., `"st_list_topperfs"`, `"st_reg_id"`). Need to decide which presets to expose as CLI flags and what the default screener view looks like.
|
|
||||||
|
|
||||||
5. **Provider trait extension** — `quote()`, `fundamentals()`, `history()` are currently defined on `Provider` trait. New methods (earnings, profile, etc.) are MSN-specific — do we extend the trait or expose them as inherent methods on `MsnProvider` only?
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Files to Touch
|
|
||||||
|
|
||||||
```
|
|
||||||
src/api/msn/
|
|
||||||
client.rs — add fetch_* methods
|
|
||||||
mod.rs — expose new provider methods
|
|
||||||
parse.rs — add parse_* functions
|
|
||||||
types.rs — NEW: shared type definitions
|
|
||||||
|
|
||||||
src/cli/
|
|
||||||
stocks.rs — add new subcommands + output formatting
|
|
||||||
|
|
||||||
tests/
|
|
||||||
cli.rs — integration tests for new commands
|
|
||||||
fixtures/ — add response fixtures for new endpoints
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
*Drafted by Ciphercat based on rubick Go implementation analysis + live MSN API verification.*
|
|
||||||
|
|
|
||||||
114
TODO.md
114
TODO.md
|
|
@ -31,9 +31,83 @@
|
||||||
- [x] `stocks compare <SYM1,SYM2,...>` — side-by-side multi-symbol comparison
|
- [x] `stocks compare <SYM1,SYM2,...>` — side-by-side multi-symbol comparison
|
||||||
- [x] `analysis/fundamental.rs` — fundamental analysis module (ported from idx-mcp)
|
- [x] `analysis/fundamental.rs` — fundamental analysis module (ported from idx-mcp)
|
||||||
- [x] Yahoo quoteSummary endpoint parser (/v10/finance/quoteSummary)
|
- [x] Yahoo quoteSummary endpoint parser (/v10/finance/quoteSummary)
|
||||||
- [x] 41 tests passing (22 unit + 19 integration)
|
- [x] 142 tests passing (94 unit + 48 integration)
|
||||||
|
|
||||||
|
## ✅ Completed (2026-03-26 — hardening pass)
|
||||||
|
- [x] Unified cache, offline, stale-cache, and `--no-cache` behavior across core and MSN-only stock commands
|
||||||
|
- [x] Rejected the conflicting `--offline --no-cache` flag combination explicitly
|
||||||
|
- [x] Routed startup/config failures through the same JSON-aware error path as runtime failures
|
||||||
|
- [x] Validated `stocks screen --filter` and `--region`; invalid values now return errors
|
||||||
|
- [x] Added regression tests for startup JSON errors, MSN profile offline/stale-cache behavior, and screener validation
|
||||||
|
|
||||||
|
## ✅ Completed (2026-03-26 — MSN output cleanup)
|
||||||
|
- [x] Re-mapped live MSN `profile` output to prefer localized/company fields for name, description, sector, industry, website, address, and phone
|
||||||
|
- [x] Reworked `stocks insights` output to derive a real summary, split highlights vs risks from evaluation status, and populate `last_updated`
|
||||||
|
- [x] Fixed signed-number formatting so `stocks financials` table output no longer mangles negative values
|
||||||
|
- [x] Added live-like fixtures and regression coverage for `profile`, `insights`, and signed table formatting
|
||||||
|
|
||||||
## 🚧 Next Up
|
## 🚧 Next Up
|
||||||
|
- [x] Build a reusable live smoke script/checklist for all shipped CLI commands (`scripts/live-smoke.sh`, `docs/SMOKE.md`)
|
||||||
|
- [x] Re-run full live smoke for MSN-only commands after the cache/offline, mapping, and formatting fixes (`scripts/live-smoke.sh --mode full --group live-table --group live-json`)
|
||||||
|
- [x] Review remaining noisy table output in `financials` and `earnings` beyond the signed-number fix
|
||||||
|
- [x] Expand parser/CLI regression coverage for the rest of the MSN-only command set
|
||||||
|
- [x] Keep the live-smoke notes below in sync with real command output after each hardening pass
|
||||||
|
|
||||||
|
## 🚧 Ownership Roadmap Reset (2026-03-29)
|
||||||
|
|
||||||
|
### Batch 1 — IDX discovery + remote PDF import
|
||||||
|
- [x] Verify and document the IDX announcement/listing page that exposes the monthly ownership PDF link
|
||||||
|
- Verified official BEI listing page: `https://www.idx.co.id/id/berita/pengumuman/`
|
||||||
|
- Verified listing page JSON source used by the site: `GET /primary/NewsAnnouncement/GetAllAnnouncement?keywords=...`
|
||||||
|
- [x] Implement discovery/crawler logic for the hashed IDX PDF asset URL instead of hardcoding monthly paths
|
||||||
|
- [x] Expose discovery as an explicit `idx ownership discover` CLI surface so the live BEI feed can be inspected without coupling it to import
|
||||||
|
- [x] Extract a reusable browser-impersonated fetch helper for ownership downloads by reusing the repo's `curl-impersonate` pattern
|
||||||
|
- [x] Wire `ownership import --url` to the impersonated fetch path for IDX-hosted PDFs
|
||||||
|
- [x] Keep the current PDF parser/import path as the first production ingest route
|
||||||
|
- [x] Add tests and fixtures for IDX announcement-page parsing plus remote PDF fetch failure modes
|
||||||
|
- [x] Batch 1 verification: `cargo build`
|
||||||
|
- [x] Batch 1 verification: `cargo clippy -- -D warnings`
|
||||||
|
- [x] Batch 1 verification: `cargo test`
|
||||||
|
- [x] Batch 1 verification: fixture-backed parser/downloader tests cover announcement discovery, hashed URL extraction, and downloader failure cases
|
||||||
|
- [x] Batch 1 verification: live `idx ownership discover --family above1 --limit 2` in `nix develop` resolves the current BEI hashed URLs for the parser-compatible `above 1%` report family
|
||||||
|
- [x] Batch 1 verification: live end-to-end import succeeds for one discovered BEI ownership PDF URL
|
||||||
|
- [x] Batch 1 verification: end-to-end import into a temp ownership DB via `idx ownership import --url ...`, followed by `idx ownership releases` and one ticker query
|
||||||
|
- Resolved root cause on `2026-03-29`:
|
||||||
|
- the parser was already correct for the holder-register schema; the missing piece was discovery support for the `Pemegang Saham di atas 1% (KSEI)` family
|
||||||
|
- the parser-compatible source is the `lamp1` attachment `https://www.idx.co.id/StaticData/NewsAndAnnouncement/ANNOUNCEMENTSTOCK/From_EREP/202603/b9b638e5a8_8928aca255.pdf`
|
||||||
|
- the `above 5%` and `investor-type` BEI families remain different schemas; as of `2026-03-30`, they should be treated as legacy / unsupported input rather than new parser targets
|
||||||
|
|
||||||
|
### Batch 2 — Above-1 hardening and unsupported-input UX
|
||||||
|
- [ ] Standardize the supported remote-import contract on the `Pemegang Saham di atas 1% (KSEI)` holder-register layout and its `lamp1` attachment shape
|
||||||
|
- [ ] Add BEI PDF schema classification before parse/import so `ownership import --url` can reject non-holder-register PDFs before the parser runs
|
||||||
|
- [ ] Improve CLI error messages and fallback behavior for discovery failure, fetch failure, invalid remote content, and known-but-unsupported legacy BEI schema variants
|
||||||
|
- [ ] Capture live-like fixtures for the current discoverable `investor-type` and `above 5%` BEI attachments (or their `mutool` `stext` extracts) so unsupported-input detection and failure UX are regression-tested
|
||||||
|
- [ ] Decide and document whether `ownership discover` should default to `above1` output while keeping legacy families available only for diagnostic use
|
||||||
|
- [ ] Decide and document whether `ownership import --url` accepts only direct PDF URLs or can also accept an IDX listing page as input
|
||||||
|
- [ ] Add regression coverage for Cloudflare/HTML responses, missing announcement links, duplicate release imports, and unsupported BEI schema detections
|
||||||
|
- [ ] Batch 2 verification: `cargo build`
|
||||||
|
- [ ] Batch 2 verification: `cargo clippy -- -D warnings`
|
||||||
|
- [ ] Batch 2 verification: `cargo test`
|
||||||
|
- [ ] Batch 2 verification: ownership-focused smoke checks cover successful remote import plus expected failure UX
|
||||||
|
|
||||||
|
### Batch 3 — Snapshot publishing + sync
|
||||||
|
- [ ] Design maintained SQLite snapshot publishing after remote IDX import is stable
|
||||||
|
- [ ] Add `idx ownership sync`
|
||||||
|
- [ ] Define manifest/checksum/update semantics and local DB replacement rules
|
||||||
|
- [ ] Add regression coverage for manifest parsing, checksum validation, no-op sync, and forced refresh
|
||||||
|
- [ ] Batch 3 verification: `cargo build`
|
||||||
|
- [ ] Batch 3 verification: `cargo clippy -- -D warnings`
|
||||||
|
- [ ] Batch 3 verification: `cargo test`
|
||||||
|
- [ ] Batch 3 verification: sync installs into an empty temp data dir, preserves query behavior, and no-ops when already current
|
||||||
|
|
||||||
|
### Batch 4 — KSEI ZIP/TXT fallback and cross-check path
|
||||||
|
- [ ] Keep KSEI ZIP/TXT ingest as fallback and validation/backstop work, not the first milestone
|
||||||
|
- [ ] Define whether the KSEI archive is only a maintainer fallback or a user-facing alternative import source
|
||||||
|
- [ ] Add cross-check coverage between IDX-PDF-derived output and KSEI-archive-derived output for at least one monthly release
|
||||||
|
- [ ] Batch 4 verification: `cargo build`
|
||||||
|
- [ ] Batch 4 verification: `cargo clippy -- -D warnings`
|
||||||
|
- [ ] Batch 4 verification: `cargo test`
|
||||||
|
- [ ] Batch 4 verification: fallback ingest produces a compatible SQLite state for `ownership releases`, `ticker`, and `changes`
|
||||||
|
|
||||||
## 📋 Backlog (per SPEC.md)
|
## 📋 Backlog (per SPEC.md)
|
||||||
- [ ] `market summary` — IHSG index, market breadth
|
- [ ] `market summary` — IHSG index, market breadth
|
||||||
|
|
@ -47,9 +121,47 @@
|
||||||
- [ ] CSV/TSV output formats
|
- [ ] CSV/TSV output formats
|
||||||
- [ ] Additional providers (Alpha Vantage, Twelve Data, IDX official)
|
- [ ] Additional providers (Alpha Vantage, Twelve Data, IDX official)
|
||||||
|
|
||||||
|
## 🔬 Latest Smoke Findings (2026-03-28)
|
||||||
|
- [x] Live smoke passed for shipped `stocks` commands: `quote`, `history`, `technical`, `growth`, `valuation`, `risk`, `fundamental`, `compare`, `profile`, `financials`, `earnings`, `sentiment`, `insights`, `news`, `screen`
|
||||||
|
- [x] Yahoo routing verified for live `quote` and `history`
|
||||||
|
- [x] `stocks history --history-provider msn` correctly fails for IDX as unsupported
|
||||||
|
- [x] `ownership releases` works with a writable `ownership.db_path` and an empty DB
|
||||||
|
- [x] Regression coverage now verifies offline/cache parity for MSN-only commands (`stocks profile BBCA`)
|
||||||
|
- [x] `--offline --no-cache` now fails fast as an invalid flag combination instead of serving cache
|
||||||
|
- [x] Config/startup failures now honor the JSON error contract (`IDX_PROVIDER=bogus idx -o json version`)
|
||||||
|
- [x] Invalid `stocks screen --filter/--region` values now return validation errors
|
||||||
|
- [x] Added reusable smoke runner/checklist for shipped CLI surfaces (`scripts/live-smoke.sh`, `docs/SMOKE.md`)
|
||||||
|
- [x] Reusable smoke runner passes on deterministic mock suites: `general`/`cache`/`routing`/`errors`/`ownership` = 32/32 and shipped `stocks` mock matrix = 30/30 (`tmp/live-smoke/20260326-194907`, `tmp/live-smoke/20260326-194914`)
|
||||||
|
- [x] Full live MSN-only smoke now passes via runner for both table and JSON surfaces: 30/30 (`tmp/live-smoke/20260326-195853`)
|
||||||
|
- [x] Targeted no-cache live checks confirm `profile`, `insights`, and `financials` fixes against real MSN responses
|
||||||
|
- [x] Live `stocks profile BBCA` no-cache output now populates company/localized fields instead of the sparse top-level fallback
|
||||||
|
- [x] Live `stocks insights BBCA` JSON now returns a mixed-signal summary plus non-empty `last_updated`
|
||||||
|
- [x] Live `stocks financials BBCA` table no longer renders malformed negative numbers
|
||||||
|
- [x] Re-run full live smoke to reconfirm all MSN-only commands after the latest hardening fixes
|
||||||
|
- [x] `stocks financials BBCA` table now trims ISO timestamps from section headers and humanizes raw line-item keys
|
||||||
|
- [x] `stocks earnings BBCA` table now splits history vs forecast and formats annual periods, revenue values, and dates for table mode
|
||||||
|
- [x] Fixture-backed parser and CLI JSON regression coverage now covers the remaining MSN-only `sentiment`, `news`, and `screen` commands
|
||||||
|
- [x] Fresh post-coverage live MSN smoke rerun still passes for table and JSON surfaces: 30/30 (`tmp/live-smoke/20260327-163403`)
|
||||||
|
- [ ] `ownership import --fetch-bing` is still deferred and returns unsupported
|
||||||
|
- [x] Real KSEI PDF import from local file now works again: `ownership import --file /Users/rasyidanakbar/Downloads/ksei_raw_data.pdf` imported `7261` rows for `955` tickers on `2026-03-28`, replacing the previous `6`-row/`1`-ticker failure mode
|
||||||
|
- [x] KSEI parser no longer depends on the old hardcoded column bounds fixture layout; it now reconstructs rows from `mutool` line output and handles the live `DATE + SHARE_CODE` merged segment plus `D`/`A` locality markers
|
||||||
|
- [x] A real IDX-hosted March 2026 ownership PDF URL was verified on `2026-03-29`, but only through `curl-impersonate` inside `nix develop`; plain `curl` still returns `403` from Cloudflare for the same asset
|
||||||
|
- [x] The current repo now has a concrete IDX-first roadmap: discover the hashed PDF URL from IDX pages, fetch it with browser impersonation, and only then layer snapshot publishing and `ownership sync`
|
||||||
|
- [x] The KSEI archive (`https://web.ksei.co.id/archive_download/holding_composition`) was verified as a secondary upstream that exposes monthly ZIP files and remains useful for fallback/cross-check work
|
||||||
|
- [x] Remote IDX discovery now targets the official BEI `Pengumuman` feed (`/id/berita/pengumuman/`) and its backing JSON endpoint (`/primary/NewsAnnouncement/GetAllAnnouncement`), which exposes hashed attachment URLs for ownership-related reports
|
||||||
|
- [x] New `ownership discover` CLI output now shows the latest hashed BEI ownership URLs across the current `above 5%`, `above 1%`, and `investor-type` report families
|
||||||
|
- [x] `ownership import --url https://www.idx.co.id/...pdf` now uses the same browser-impersonated download path as Yahoo auth
|
||||||
|
- [x] Reverse-engineering the current BEI feed shows the discoverable `Pemegang Saham di atas 1% (KSEI)` `lamp1` attachment matches the known-good raw KSEI holder-register layout and now imports successfully
|
||||||
|
- [x] Live `ownership import --url` against the discovered `above 1%` BEI `lamp1` attachment (`b9b638e5a8_8928aca255.pdf`) now succeeds end to end: `7261` rows for `955` tickers on `2026-03-29`, followed by successful `ownership releases` and `ownership ticker AADI --source ksei`
|
||||||
|
- [x] Product scope decision on `2026-03-30`: standardize supported remote import on the discoverable `above 1%` holder-register family; treat `above 5%` and `investor-type` PDFs as legacy / unsupported input
|
||||||
|
- [x] Live `ownership import --url` checks against the currently discoverable `above 5%` and `investor-type` BEI PDFs still fail with `no KSEI rows parsed from PDF`, which is expected until unsupported-family detection / rejection UX lands
|
||||||
|
- [x] Live `mutool` inspection of the current discoverable `investor-type` BEI attachment shows a stock-level aggregate matrix (`DATE`, `STOCK_CODE`, `NUMBER_OF_SHARES`, investor-type columns, holder-size buckets), not the holder-level KSEI register schema the current parser imports
|
||||||
|
- [x] Live `mutool` inspection of the current discoverable `above 5%` BEI attachment shows a member/tampungan report (`INVS`, member names, `KSEI UNTUK CLOSED MEMBER-...` labels), not the raw KSEI holder-register layout
|
||||||
|
|
||||||
## 🐛 Known Issues
|
## 🐛 Known Issues
|
||||||
- [ ] Yahoo Finance returns 429 from datacenter IPs occasionally
|
- [ ] Yahoo Finance returns 429 from datacenter IPs occasionally
|
||||||
- [ ] SMA200 trend shows "Insufficient data" if Yahoo returns < 200 candles
|
- [ ] SMA200 trend shows "Insufficient data" if Yahoo returns < 200 candles
|
||||||
|
- [x] KSEI ownership parser no longer leaks adjacent columns into `INVESTOR_NAME` on the March 2026 live PDF; direct CLI verification now returns clean holders like `AGUNG PERKASA INVESTINDO` / `CP` / `L`
|
||||||
- [x] Yahoo quoteSummary crumb auth — fixed via curl-impersonate-chrome (curl_chrome131)
|
- [x] Yahoo quoteSummary crumb auth — fixed via curl-impersonate-chrome (curl_chrome131)
|
||||||
- fc.yahoo.com → A3 cookie + query1 getcrumb → crumb, both sent to quoteSummary
|
- fc.yahoo.com → A3 cookie + query1 getcrumb → crumb, both sent to quoteSummary
|
||||||
- Requires nixpkgs#curl-impersonate-chrome in PATH (added to flake.nix + clan-private)
|
- Requires nixpkgs#curl-impersonate-chrome in PATH (added to flake.nix + clan-private)
|
||||||
|
|
|
||||||
|
|
@ -1,9 +1,10 @@
|
||||||
use std::path::{Path, PathBuf};
|
use std::path::{Path, PathBuf};
|
||||||
use std::process::{Command, Output};
|
use std::process::Command;
|
||||||
use std::sync::Mutex;
|
use std::sync::Mutex;
|
||||||
use std::time::Duration;
|
use std::time::Duration;
|
||||||
|
|
||||||
use crate::api::types::{Interval, Period};
|
use crate::api::types::{Interval, Period};
|
||||||
|
use crate::curl_impersonate;
|
||||||
use crate::error::IdxError;
|
use crate::error::IdxError;
|
||||||
|
|
||||||
use super::raw_types::{ChartResponse, QuoteSummaryResponse};
|
use super::raw_types::{ChartResponse, QuoteSummaryResponse};
|
||||||
|
|
@ -117,56 +118,8 @@ impl YahooClient {
|
||||||
Ok(cookies.join("; "))
|
Ok(cookies.join("; "))
|
||||||
}
|
}
|
||||||
|
|
||||||
fn chrome_curl_binary() -> Option<&'static str> {
|
|
||||||
// curl-impersonate-chrome ships per-version binaries (curl_chrome131 etc).
|
|
||||||
// Try latest versions first; no --impersonate flag needed; the binary is the impersonation.
|
|
||||||
const CANDIDATES: &[&str] = &[
|
|
||||||
"curl_chrome136",
|
|
||||||
"curl_chrome133a",
|
|
||||||
"curl_chrome131",
|
|
||||||
"curl_chrome124",
|
|
||||||
"curl_chrome120",
|
|
||||||
"curl_chrome116",
|
|
||||||
];
|
|
||||||
CANDIDATES
|
|
||||||
.iter()
|
|
||||||
.copied()
|
|
||||||
.find(|bin| Command::new(bin).arg("--version").output().is_ok())
|
|
||||||
}
|
|
||||||
|
|
||||||
fn run_curl(stage: &str, binary: &str, args: &[&str]) -> Result<Output, IdxError> {
|
|
||||||
let output = Command::new(binary).args(args).output().map_err(|e| {
|
|
||||||
if e.kind() == std::io::ErrorKind::NotFound {
|
|
||||||
return IdxError::Http(format!(
|
|
||||||
"curl-impersonate binary '{binary}' not found; install nixpkgs#curl-impersonate-chrome"
|
|
||||||
));
|
|
||||||
}
|
|
||||||
IdxError::Http(format!("failed to run {binary} for Yahoo {stage}: {e}"))
|
|
||||||
})?;
|
|
||||||
|
|
||||||
if output.status.success() {
|
|
||||||
return Ok(output);
|
|
||||||
}
|
|
||||||
|
|
||||||
let stderr = String::from_utf8_lossy(&output.stderr);
|
|
||||||
let detail = stderr.trim();
|
|
||||||
Err(IdxError::Http(format!(
|
|
||||||
"Yahoo {stage} {binary} failed (status {}): {}",
|
|
||||||
output.status,
|
|
||||||
if detail.is_empty() {
|
|
||||||
"no output"
|
|
||||||
} else {
|
|
||||||
detail
|
|
||||||
}
|
|
||||||
)))
|
|
||||||
}
|
|
||||||
|
|
||||||
fn fetch_crumb_via_curl(&self) -> Result<String, IdxError> {
|
fn fetch_crumb_via_curl(&self) -> Result<String, IdxError> {
|
||||||
let binary = Self::chrome_curl_binary().ok_or_else(|| {
|
let binary = curl_impersonate::chrome_curl_binary()?;
|
||||||
IdxError::Http(
|
|
||||||
"no curl_chrome* binary found; install nixpkgs#curl-impersonate-chrome".to_string(),
|
|
||||||
)
|
|
||||||
})?;
|
|
||||||
|
|
||||||
let cookie_jar = Self::cookie_jar_path();
|
let cookie_jar = Self::cookie_jar_path();
|
||||||
let cookie_jar_str = cookie_jar.to_str().ok_or_else(|| {
|
let cookie_jar_str = cookie_jar.to_str().ok_or_else(|| {
|
||||||
|
|
@ -190,9 +143,8 @@ impl YahooClient {
|
||||||
.output();
|
.output();
|
||||||
|
|
||||||
// Step 2: fetch crumb with cookie jar (Chrome TLS fingerprint + A3 cookie).
|
// Step 2: fetch crumb with cookie jar (Chrome TLS fingerprint + A3 cookie).
|
||||||
let output = Self::run_curl(
|
let output = curl_impersonate::run(
|
||||||
"crumb fetch",
|
"Yahoo crumb fetch",
|
||||||
binary,
|
|
||||||
&["--silent", "--cookie", cookie_jar_str, CRUMB_FETCH_URL],
|
&["--silent", "--cookie", cookie_jar_str, CRUMB_FETCH_URL],
|
||||||
)?;
|
)?;
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -18,7 +18,7 @@ use crate::output::table::format_idr;
|
||||||
use crate::ownership::types::{
|
use crate::ownership::types::{
|
||||||
ChangeType, FlowSignal, HolderRow, KseiHolding, OwnershipRelease, OwnershipSource,
|
ChangeType, FlowSignal, HolderRow, KseiHolding, OwnershipRelease, OwnershipSource,
|
||||||
};
|
};
|
||||||
use crate::ownership::{db, entities, graph, parser, search};
|
use crate::ownership::{db, entities, graph, parser, remote, search};
|
||||||
|
|
||||||
#[derive(Debug, Args)]
|
#[derive(Debug, Args)]
|
||||||
pub struct OwnershipCmd {
|
pub struct OwnershipCmd {
|
||||||
|
|
@ -28,6 +28,8 @@ pub struct OwnershipCmd {
|
||||||
|
|
||||||
#[derive(Debug, Subcommand)]
|
#[derive(Debug, Subcommand)]
|
||||||
pub enum OwnershipCommand {
|
pub enum OwnershipCommand {
|
||||||
|
/// Discover the latest IDX-hosted ownership report URLs.
|
||||||
|
Discover(DiscoverArgs),
|
||||||
/// Import ownership data from KSEI PDF or Bing API.
|
/// Import ownership data from KSEI PDF or Bing API.
|
||||||
Import(ImportArgs),
|
Import(ImportArgs),
|
||||||
/// Show all holders for a ticker (KSEI + Bing combined).
|
/// Show all holders for a ticker (KSEI + Bing combined).
|
||||||
|
|
@ -52,9 +54,19 @@ pub enum OwnershipCommand {
|
||||||
Releases,
|
Releases,
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[derive(Debug, Args)]
|
||||||
|
pub struct DiscoverArgs {
|
||||||
|
/// Report family to discover: all, above1, above5, or investor-type.
|
||||||
|
#[arg(long, default_value = "all")]
|
||||||
|
pub family: String,
|
||||||
|
/// Maximum number of discovered report URLs to print.
|
||||||
|
#[arg(long, default_value_t = 6)]
|
||||||
|
pub limit: usize,
|
||||||
|
}
|
||||||
|
|
||||||
#[derive(Debug, Args)]
|
#[derive(Debug, Args)]
|
||||||
pub struct ImportArgs {
|
pub struct ImportArgs {
|
||||||
/// URL to KSEI ownership PDF.
|
/// URL to a remote ownership PDF.
|
||||||
#[arg(long)]
|
#[arg(long)]
|
||||||
pub url: Option<String>,
|
pub url: Option<String>,
|
||||||
/// Path to local KSEI PDF file.
|
/// Path to local KSEI PDF file.
|
||||||
|
|
@ -159,6 +171,7 @@ pub enum ResolveCommand {
|
||||||
|
|
||||||
pub fn handle(cmd: &OwnershipCommand, config: &IdxConfig) -> Result<(), IdxError> {
|
pub fn handle(cmd: &OwnershipCommand, config: &IdxConfig) -> Result<(), IdxError> {
|
||||||
match cmd {
|
match cmd {
|
||||||
|
OwnershipCommand::Discover(args) => handle_discover(args, config),
|
||||||
OwnershipCommand::Import(args) => handle_import(args, config),
|
OwnershipCommand::Import(args) => handle_import(args, config),
|
||||||
OwnershipCommand::Ticker(args) => handle_ticker(args, config),
|
OwnershipCommand::Ticker(args) => handle_ticker(args, config),
|
||||||
OwnershipCommand::Entity(args) => handle_entity(args, config),
|
OwnershipCommand::Entity(args) => handle_entity(args, config),
|
||||||
|
|
@ -173,6 +186,45 @@ pub fn handle(cmd: &OwnershipCommand, config: &IdxConfig) -> Result<(), IdxError
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
fn handle_discover(args: &DiscoverArgs, config: &IdxConfig) -> Result<(), IdxError> {
|
||||||
|
if args.limit == 0 {
|
||||||
|
return Err(IdxError::ParseError(
|
||||||
|
"--limit must be greater than 0".to_string(),
|
||||||
|
));
|
||||||
|
}
|
||||||
|
|
||||||
|
let family = parse_discovery_family(&args.family)?;
|
||||||
|
let reports = remote::discover_idx_ownership_reports(family, args.limit)?;
|
||||||
|
|
||||||
|
if matches!(config.output, OutputFormat::Json) {
|
||||||
|
return json::print_json(&reports);
|
||||||
|
}
|
||||||
|
|
||||||
|
let mut table = Table::new();
|
||||||
|
table
|
||||||
|
.load_preset(UTF8_FULL)
|
||||||
|
.set_content_arrangement(ContentArrangement::Dynamic)
|
||||||
|
.set_header(vec!["DATE", "FAMILY", "KIND", "FILE", "TITLE", "URL"]);
|
||||||
|
|
||||||
|
for report in reports {
|
||||||
|
table.add_row(vec![
|
||||||
|
Cell::new(report.publish_date.split('T').next().unwrap_or("-")),
|
||||||
|
Cell::new(report.family.label()),
|
||||||
|
Cell::new(if report.is_attachment {
|
||||||
|
"attachment"
|
||||||
|
} else {
|
||||||
|
"main"
|
||||||
|
}),
|
||||||
|
Cell::new(report.original_filename.unwrap_or_else(|| "-".to_string())),
|
||||||
|
Cell::new(report.title),
|
||||||
|
Cell::new(report.pdf_url),
|
||||||
|
]);
|
||||||
|
}
|
||||||
|
|
||||||
|
println!("{table}");
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
fn handle_ticker(args: &TickerArgs, config: &IdxConfig) -> Result<(), IdxError> {
|
fn handle_ticker(args: &TickerArgs, config: &IdxConfig) -> Result<(), IdxError> {
|
||||||
let conn = db::open_db(config)?;
|
let conn = db::open_db(config)?;
|
||||||
let symbol = args.symbol.trim().to_uppercase();
|
let symbol = args.symbol.trim().to_uppercase();
|
||||||
|
|
@ -660,19 +712,19 @@ fn handle_import(args: &ImportArgs, config: &IdxConfig) -> Result<(), IdxError>
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
let Some(pdf_path) = resolve_pdf_input(args)? else {
|
let Some(pdf_input) = resolve_pdf_input(args)? else {
|
||||||
return Ok(());
|
return Ok(());
|
||||||
};
|
};
|
||||||
|
|
||||||
let conn = db::open_db(config)?;
|
let conn = db::open_db(config)?;
|
||||||
|
|
||||||
let sha256 = sha256_file(&pdf_path)?;
|
let sha256 = sha256_file(&pdf_input.pdf_path)?;
|
||||||
if !args.force && db::release_exists(&conn, &sha256)? {
|
if !args.force && db::release_exists(&conn, &sha256)? {
|
||||||
println!("Release already imported (sha256: {sha256}). Use --force to re-import.");
|
println!("Release already imported (sha256: {sha256}). Use --force to re-import.");
|
||||||
return Ok(());
|
return Ok(());
|
||||||
}
|
}
|
||||||
|
|
||||||
let raw_rows = parser::parse_ksei_pdf(&pdf_path)?;
|
let raw_rows = parser::parse_ksei_pdf(&pdf_input.pdf_path)?;
|
||||||
if raw_rows.is_empty() {
|
if raw_rows.is_empty() {
|
||||||
return Err(IdxError::ParseError(
|
return Err(IdxError::ParseError(
|
||||||
"no KSEI rows parsed from PDF".to_string(),
|
"no KSEI rows parsed from PDF".to_string(),
|
||||||
|
|
@ -718,7 +770,7 @@ fn handle_import(args: &ImportArgs, config: &IdxConfig) -> Result<(), IdxError>
|
||||||
|
|
||||||
let release = OwnershipRelease {
|
let release = OwnershipRelease {
|
||||||
id: 0,
|
id: 0,
|
||||||
source_url: args.url.clone(),
|
source_url: pdf_input.source_url,
|
||||||
sha256,
|
sha256,
|
||||||
as_of_date,
|
as_of_date,
|
||||||
row_count: inserted_rows,
|
row_count: inserted_rows,
|
||||||
|
|
@ -769,7 +821,30 @@ fn handle_releases(config: &IdxConfig) -> Result<(), IdxError> {
|
||||||
Ok(())
|
Ok(())
|
||||||
}
|
}
|
||||||
|
|
||||||
fn resolve_pdf_input(args: &ImportArgs) -> Result<Option<PathBuf>, IdxError> {
|
fn parse_discovery_family(raw: &str) -> Result<Option<remote::OwnershipReportFamily>, IdxError> {
|
||||||
|
match raw.trim().to_ascii_lowercase().as_str() {
|
||||||
|
"all" => Ok(None),
|
||||||
|
"above1" | "above-1" | "above_1" => {
|
||||||
|
Ok(Some(remote::OwnershipReportFamily::AboveOnePercent))
|
||||||
|
}
|
||||||
|
"above5" | "above-5" | "above_5" => {
|
||||||
|
Ok(Some(remote::OwnershipReportFamily::AboveFivePercent))
|
||||||
|
}
|
||||||
|
"investor-type" | "investor_type" | "investortype" => {
|
||||||
|
Ok(Some(remote::OwnershipReportFamily::InvestorTypeBreakdown))
|
||||||
|
}
|
||||||
|
_ => Err(IdxError::ParseError(
|
||||||
|
"invalid --family, expected: all|above1|above5|investor-type".to_string(),
|
||||||
|
)),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
struct ResolvedPdfInput {
|
||||||
|
pdf_path: PathBuf,
|
||||||
|
source_url: Option<String>,
|
||||||
|
}
|
||||||
|
|
||||||
|
fn resolve_pdf_input(args: &ImportArgs) -> Result<Option<ResolvedPdfInput>, IdxError> {
|
||||||
if let Some(path) = &args.file {
|
if let Some(path) = &args.file {
|
||||||
if !path.exists() {
|
if !path.exists() {
|
||||||
return Err(IdxError::Io(format!(
|
return Err(IdxError::Io(format!(
|
||||||
|
|
@ -777,13 +852,19 @@ fn resolve_pdf_input(args: &ImportArgs) -> Result<Option<PathBuf>, IdxError> {
|
||||||
path.display()
|
path.display()
|
||||||
)));
|
)));
|
||||||
}
|
}
|
||||||
return Ok(Some(path.clone()));
|
return Ok(Some(ResolvedPdfInput {
|
||||||
|
pdf_path: path.clone(),
|
||||||
|
source_url: None,
|
||||||
|
}));
|
||||||
}
|
}
|
||||||
|
|
||||||
if let Some(url) = &args.url {
|
if let Some(url) = &args.url {
|
||||||
let target = cache_pdf_path(url)?;
|
let target = cache_pdf_path(url)?;
|
||||||
download_pdf(url, &target)?;
|
download_pdf(url, &target)?;
|
||||||
return Ok(Some(target));
|
return Ok(Some(ResolvedPdfInput {
|
||||||
|
pdf_path: target,
|
||||||
|
source_url: Some(url.clone()),
|
||||||
|
}));
|
||||||
}
|
}
|
||||||
|
|
||||||
Ok(None)
|
Ok(None)
|
||||||
|
|
@ -815,6 +896,10 @@ fn cache_pdf_path(url: &str) -> Result<PathBuf, IdxError> {
|
||||||
}
|
}
|
||||||
|
|
||||||
fn download_pdf(url: &str, target: &Path) -> Result<(), IdxError> {
|
fn download_pdf(url: &str, target: &Path) -> Result<(), IdxError> {
|
||||||
|
if is_idx_url(url) {
|
||||||
|
return remote::download_idx_pdf(url, target);
|
||||||
|
}
|
||||||
|
|
||||||
let response = ureq::get(url)
|
let response = ureq::get(url)
|
||||||
.header(
|
.header(
|
||||||
"User-Agent",
|
"User-Agent",
|
||||||
|
|
@ -830,6 +915,7 @@ fn download_pdf(url: &str, target: &Path) -> Result<(), IdxError> {
|
||||||
let bytes = body
|
let bytes = body
|
||||||
.read_to_vec()
|
.read_to_vec()
|
||||||
.map_err(|e| IdxError::Http(format!("failed reading PDF body: {e}")))?;
|
.map_err(|e| IdxError::Http(format!("failed reading PDF body: {e}")))?;
|
||||||
|
remote::validate_pdf_payload(&bytes)?;
|
||||||
|
|
||||||
fs::write(target, &bytes).map_err(|e| {
|
fs::write(target, &bytes).map_err(|e| {
|
||||||
IdxError::Io(format!(
|
IdxError::Io(format!(
|
||||||
|
|
@ -841,6 +927,14 @@ fn download_pdf(url: &str, target: &Path) -> Result<(), IdxError> {
|
||||||
Ok(())
|
Ok(())
|
||||||
}
|
}
|
||||||
|
|
||||||
|
fn is_idx_url(url: &str) -> bool {
|
||||||
|
let normalized = url.trim().to_ascii_lowercase();
|
||||||
|
normalized.starts_with("https://www.idx.co.id/")
|
||||||
|
|| normalized.starts_with("http://www.idx.co.id/")
|
||||||
|
|| normalized.starts_with("https://idx.co.id/")
|
||||||
|
|| normalized.starts_with("http://idx.co.id/")
|
||||||
|
}
|
||||||
|
|
||||||
fn sha256_file(path: &Path) -> Result<String, IdxError> {
|
fn sha256_file(path: &Path) -> Result<String, IdxError> {
|
||||||
let bytes = fs::read(path).map_err(|e| {
|
let bytes = fs::read(path).map_err(|e| {
|
||||||
IdxError::Io(format!(
|
IdxError::Io(format!(
|
||||||
|
|
|
||||||
67
src/curl_impersonate.rs
Normal file
67
src/curl_impersonate.rs
Normal file
|
|
@ -0,0 +1,67 @@
|
||||||
|
use std::process::{Command, Output};
|
||||||
|
|
||||||
|
use crate::error::IdxError;
|
||||||
|
|
||||||
|
const CANDIDATES: &[&str] = &[
|
||||||
|
"curl_chrome142",
|
||||||
|
"curl_chrome136",
|
||||||
|
"curl_chrome133a",
|
||||||
|
"curl_chrome131",
|
||||||
|
"curl_chrome124",
|
||||||
|
"curl_chrome120",
|
||||||
|
"curl_chrome116",
|
||||||
|
];
|
||||||
|
const OVERRIDE_ENV: &str = "IDX_CURL_IMPERSONATE_BIN";
|
||||||
|
|
||||||
|
pub fn chrome_curl_binary() -> Result<String, IdxError> {
|
||||||
|
if let Ok(value) = std::env::var(OVERRIDE_ENV) {
|
||||||
|
let trimmed = value.trim();
|
||||||
|
if !trimmed.is_empty() {
|
||||||
|
return Ok(trimmed.to_string());
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
CANDIDATES
|
||||||
|
.iter()
|
||||||
|
.copied()
|
||||||
|
.find(|bin| Command::new(bin).arg("--version").output().is_ok())
|
||||||
|
.map(str::to_string)
|
||||||
|
.ok_or_else(|| {
|
||||||
|
IdxError::Http(format!(
|
||||||
|
"no curl_chrome* binary found; set {OVERRIDE_ENV} or install nixpkgs#curl-impersonate-chrome"
|
||||||
|
))
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn run(stage: &str, args: &[&str]) -> Result<Output, IdxError> {
|
||||||
|
let owned_args: Vec<String> = args.iter().map(|arg| (*arg).to_string()).collect();
|
||||||
|
run_owned(stage, &owned_args)
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn run_owned(stage: &str, args: &[String]) -> Result<Output, IdxError> {
|
||||||
|
let binary = chrome_curl_binary()?;
|
||||||
|
let output = Command::new(&binary).args(args).output().map_err(|e| {
|
||||||
|
if e.kind() == std::io::ErrorKind::NotFound {
|
||||||
|
return IdxError::Http(format!(
|
||||||
|
"curl-impersonate binary '{binary}' not found; set {OVERRIDE_ENV} or install nixpkgs#curl-impersonate-chrome"
|
||||||
|
));
|
||||||
|
}
|
||||||
|
IdxError::Http(format!("failed to run {binary} for {stage}: {e}"))
|
||||||
|
})?;
|
||||||
|
|
||||||
|
if output.status.success() {
|
||||||
|
return Ok(output);
|
||||||
|
}
|
||||||
|
|
||||||
|
let stderr = String::from_utf8_lossy(&output.stderr);
|
||||||
|
let detail = stderr.trim();
|
||||||
|
Err(IdxError::Http(format!(
|
||||||
|
"{stage} {binary} failed (status {}): {}",
|
||||||
|
output.status,
|
||||||
|
if detail.is_empty() {
|
||||||
|
"no output"
|
||||||
|
} else {
|
||||||
|
detail
|
||||||
|
}
|
||||||
|
)))
|
||||||
|
}
|
||||||
|
|
@ -3,6 +3,7 @@ mod api;
|
||||||
mod cache;
|
mod cache;
|
||||||
mod cli;
|
mod cli;
|
||||||
mod config;
|
mod config;
|
||||||
|
mod curl_impersonate;
|
||||||
mod error;
|
mod error;
|
||||||
mod output;
|
mod output;
|
||||||
#[cfg(feature = "ownership")]
|
#[cfg(feature = "ownership")]
|
||||||
|
|
|
||||||
|
|
@ -215,7 +215,7 @@ fn normalize_investor_type(raw: &str) -> Option<InvestorTypeCode> {
|
||||||
|
|
||||||
fn normalize_locality(raw: &str) -> Option<Locality> {
|
fn normalize_locality(raw: &str) -> Option<Locality> {
|
||||||
match raw.trim().to_uppercase().as_str() {
|
match raw.trim().to_uppercase().as_str() {
|
||||||
"L" => Some(Locality::Local),
|
"L" | "D" => Some(Locality::Local),
|
||||||
"F" | "A" => Some(Locality::Foreign),
|
"F" | "A" => Some(Locality::Foreign),
|
||||||
_ => None,
|
_ => None,
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -2,5 +2,6 @@ pub mod db;
|
||||||
pub mod entities;
|
pub mod entities;
|
||||||
pub mod graph;
|
pub mod graph;
|
||||||
pub mod parser;
|
pub mod parser;
|
||||||
|
pub mod remote;
|
||||||
pub mod search;
|
pub mod search;
|
||||||
pub mod types;
|
pub mod types;
|
||||||
|
|
|
||||||
|
|
@ -3,32 +3,36 @@ use std::path::Path;
|
||||||
use std::process::Command;
|
use std::process::Command;
|
||||||
|
|
||||||
use quick_xml::Reader;
|
use quick_xml::Reader;
|
||||||
use quick_xml::events::Event;
|
use quick_xml::events::{BytesStart, Event};
|
||||||
|
|
||||||
use crate::error::IdxError;
|
use crate::error::IdxError;
|
||||||
use crate::ownership::types::KseiRawRow;
|
use crate::ownership::types::KseiRawRow;
|
||||||
|
|
||||||
/// Character grid for a single PDF page: y-coord → column-index → sorted (x, char) pairs.
|
|
||||||
type PageGrid = HashMap<i32, HashMap<usize, Vec<(i32, char)>>>;
|
|
||||||
|
|
||||||
const Y_TOLERANCE: f32 = 0.8;
|
const Y_TOLERANCE: f32 = 0.8;
|
||||||
|
const HEADER_MATCH_MIN: usize = 4;
|
||||||
|
|
||||||
/// Inclusive-left, exclusive-right X ranges for each KSEI data column.
|
const HEADER_LABELS: &[&str] = &[
|
||||||
const COLUMN_BOUNDS: [(f32, f32); 12] = [
|
"DATE",
|
||||||
(15.0, 52.0), // date
|
"SHARECODE",
|
||||||
(52.0, 70.0), // share_code
|
"ISSUERNAME",
|
||||||
(70.0, 167.0), // issuer_name
|
"INVESTORNAME",
|
||||||
(167.0, 432.0), // investor_name
|
"INVESTORTYPE",
|
||||||
(432.0, 463.0), // investor_type
|
"LOCALFOREIGN",
|
||||||
(463.0, 497.0), // local_foreign
|
"NATIONALITY",
|
||||||
(497.0, 558.0), // nationality
|
"DOMICILE",
|
||||||
(558.0, 615.0), // domicile
|
"HOLDINGSSCRIPLESS",
|
||||||
(615.0, 653.0), // holdings_scripless
|
"HOLDINGSSCRIP",
|
||||||
(653.0, 692.0), // holdings_scrip
|
"TOTALHOLDINGSHARES",
|
||||||
(692.0, 745.0), // total_holding_shares
|
"PERCENTAGE",
|
||||||
(745.0, 800.0), // percentage
|
|
||||||
];
|
];
|
||||||
|
|
||||||
|
#[derive(Debug, Clone)]
|
||||||
|
struct PageLine {
|
||||||
|
x: f32,
|
||||||
|
y: f32,
|
||||||
|
text: String,
|
||||||
|
}
|
||||||
|
|
||||||
/// Parse a KSEI ownership PDF into raw rows.
|
/// Parse a KSEI ownership PDF into raw rows.
|
||||||
/// Shells out to `mutool` for XML extraction, then parses with quick-xml.
|
/// Shells out to `mutool` for XML extraction, then parses with quick-xml.
|
||||||
pub fn parse_ksei_pdf(path: &Path) -> Result<Vec<KseiRawRow>, IdxError> {
|
pub fn parse_ksei_pdf(path: &Path) -> Result<Vec<KseiRawRow>, IdxError> {
|
||||||
|
|
@ -65,76 +69,24 @@ pub fn parse_stext_xml(xml: &str) -> Result<Vec<KseiRawRow>, IdxError> {
|
||||||
reader.config_mut().trim_text(false);
|
reader.config_mut().trim_text(false);
|
||||||
|
|
||||||
let mut rows: Vec<KseiRawRow> = Vec::new();
|
let mut rows: Vec<KseiRawRow> = Vec::new();
|
||||||
let mut current_page: Option<PageGrid> = None;
|
let mut current_page: Option<Vec<PageLine>> = None;
|
||||||
let mut buf = Vec::new();
|
let mut buf = Vec::new();
|
||||||
|
|
||||||
loop {
|
loop {
|
||||||
match reader.read_event_into(&mut buf) {
|
match reader.read_event_into(&mut buf) {
|
||||||
Ok(Event::Start(e)) if e.name().as_ref() == b"page" => {
|
Ok(Event::Start(e)) if e.name().as_ref() == b"page" => {
|
||||||
current_page = Some(HashMap::new());
|
current_page = Some(Vec::new());
|
||||||
}
|
}
|
||||||
Ok(Event::Empty(e)) if e.name().as_ref() == b"char" => {
|
Ok(Event::Start(e)) if e.name().as_ref() == b"line" => {
|
||||||
if let Some(page) = current_page.as_mut() {
|
if let Some(page) = current_page.as_mut()
|
||||||
let mut x: Option<f32> = None;
|
&& let Some(line) = parse_line_attrs(&reader, &e)?
|
||||||
let mut y: Option<f32> = None;
|
|
||||||
let mut c: Option<char> = None;
|
|
||||||
|
|
||||||
for attr_result in e.attributes().with_checks(false) {
|
|
||||||
let attr = attr_result.map_err(|err| {
|
|
||||||
IdxError::PdfParseError(format!("invalid XML attribute: {err}"))
|
|
||||||
})?;
|
|
||||||
|
|
||||||
match attr.key.as_ref() {
|
|
||||||
b"x" => {
|
|
||||||
let s = attr.decode_and_unescape_value(reader.decoder()).map_err(
|
|
||||||
|err| {
|
|
||||||
IdxError::PdfParseError(format!(
|
|
||||||
"invalid XML x attribute: {err}"
|
|
||||||
))
|
|
||||||
},
|
|
||||||
)?;
|
|
||||||
x = s.parse::<f32>().ok();
|
|
||||||
}
|
|
||||||
b"y" => {
|
|
||||||
let s = attr.decode_and_unescape_value(reader.decoder()).map_err(
|
|
||||||
|err| {
|
|
||||||
IdxError::PdfParseError(format!(
|
|
||||||
"invalid XML y attribute: {err}"
|
|
||||||
))
|
|
||||||
},
|
|
||||||
)?;
|
|
||||||
y = s.parse::<f32>().ok();
|
|
||||||
}
|
|
||||||
b"c" => {
|
|
||||||
let s = attr.decode_and_unescape_value(reader.decoder()).map_err(
|
|
||||||
|err| {
|
|
||||||
IdxError::PdfParseError(format!(
|
|
||||||
"invalid XML char attribute: {err}"
|
|
||||||
))
|
|
||||||
},
|
|
||||||
)?;
|
|
||||||
c = s.chars().next();
|
|
||||||
}
|
|
||||||
_ => {}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
if let (Some(x_val), Some(y_val), Some(ch)) = (x, y, c)
|
|
||||||
&& let Some(col_idx) = x_to_column(x_val)
|
|
||||||
{
|
{
|
||||||
let yb = y_bucket(y_val);
|
page.push(line);
|
||||||
let xi = (x_val * 100.0).round() as i32;
|
|
||||||
page.entry(yb)
|
|
||||||
.or_default()
|
|
||||||
.entry(col_idx)
|
|
||||||
.or_default()
|
|
||||||
.push((xi, ch));
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
Ok(Event::End(e)) if e.name().as_ref() == b"page" => {
|
Ok(Event::End(e)) if e.name().as_ref() == b"page" => {
|
||||||
if let Some(page) = current_page.take() {
|
if let Some(page) = current_page.take() {
|
||||||
rows.extend(extract_rows_from_page(page));
|
rows.extend(extract_rows_from_page(&page));
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
Ok(Event::Eof) => break,
|
Ok(Event::Eof) => break,
|
||||||
|
|
@ -164,13 +116,104 @@ pub fn check_mutool() -> Result<(), IdxError> {
|
||||||
Ok(())
|
Ok(())
|
||||||
}
|
}
|
||||||
|
|
||||||
fn extract_rows_from_page(page: PageGrid) -> Vec<KseiRawRow> {
|
fn parse_line_attrs(
|
||||||
let mut page_rows = Vec::new();
|
reader: &Reader<&[u8]>,
|
||||||
|
event: &BytesStart<'_>,
|
||||||
|
) -> Result<Option<PageLine>, IdxError> {
|
||||||
|
let mut bbox: Option<String> = None;
|
||||||
|
let mut text: Option<String> = None;
|
||||||
|
|
||||||
let mut y_keys: Vec<i32> = page.keys().copied().collect();
|
for attr_result in event.attributes().with_checks(false) {
|
||||||
|
let attr = attr_result
|
||||||
|
.map_err(|err| IdxError::PdfParseError(format!("invalid XML attribute: {err}")))?;
|
||||||
|
|
||||||
|
match attr.key.as_ref() {
|
||||||
|
b"bbox" => {
|
||||||
|
bbox = Some(
|
||||||
|
attr.decode_and_unescape_value(reader.decoder())
|
||||||
|
.map_err(|err| {
|
||||||
|
IdxError::PdfParseError(format!("invalid XML bbox attribute: {err}"))
|
||||||
|
})?
|
||||||
|
.to_string(),
|
||||||
|
);
|
||||||
|
}
|
||||||
|
b"text" => {
|
||||||
|
text = Some(
|
||||||
|
attr.decode_and_unescape_value(reader.decoder())
|
||||||
|
.map_err(|err| {
|
||||||
|
IdxError::PdfParseError(format!("invalid XML text attribute: {err}"))
|
||||||
|
})?
|
||||||
|
.to_string(),
|
||||||
|
);
|
||||||
|
}
|
||||||
|
_ => {}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
let Some(text) = text.map(|value| normalize_spaces(&value)) else {
|
||||||
|
return Ok(None);
|
||||||
|
};
|
||||||
|
if text.is_empty() {
|
||||||
|
return Ok(None);
|
||||||
|
}
|
||||||
|
|
||||||
|
let Some(bbox) = bbox else {
|
||||||
|
return Ok(None);
|
||||||
|
};
|
||||||
|
let mut parts = bbox.split_whitespace();
|
||||||
|
let x = parts.next().and_then(|value| value.parse::<f32>().ok());
|
||||||
|
let y = parts.next().and_then(|value| value.parse::<f32>().ok());
|
||||||
|
|
||||||
|
match (x, y) {
|
||||||
|
(Some(x), Some(y)) => Ok(Some(PageLine { x, y, text })),
|
||||||
|
_ => Ok(None),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn extract_rows_from_page(page: &[PageLine]) -> Vec<KseiRawRow> {
|
||||||
|
let mut rows_by_y: HashMap<i32, Vec<PageLine>> = HashMap::new();
|
||||||
|
for line in page {
|
||||||
|
rows_by_y
|
||||||
|
.entry(y_bucket(line.y))
|
||||||
|
.or_default()
|
||||||
|
.push(line.clone());
|
||||||
|
}
|
||||||
|
|
||||||
|
let mut y_keys: Vec<i32> = rows_by_y.keys().copied().collect();
|
||||||
y_keys.sort_unstable();
|
y_keys.sort_unstable();
|
||||||
|
|
||||||
|
let mut rows = Vec::new();
|
||||||
for y in y_keys {
|
for y in y_keys {
|
||||||
|
let Some(lines) = rows_by_y.get(&y) else {
|
||||||
|
continue;
|
||||||
|
};
|
||||||
|
|
||||||
|
let mut sorted = lines.clone();
|
||||||
|
sorted.sort_by(|left, right| left.x.total_cmp(&right.x));
|
||||||
|
|
||||||
|
let texts: Vec<String> = sorted.into_iter().map(|line| line.text).collect();
|
||||||
|
if is_header_row(&texts) {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
let row = parse_row_segments(&texts);
|
||||||
|
if is_data_row(&row) {
|
||||||
|
rows.push(row);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
rows
|
||||||
|
}
|
||||||
|
|
||||||
|
fn is_header_row(texts: &[String]) -> bool {
|
||||||
|
texts
|
||||||
|
.iter()
|
||||||
|
.filter(|text| HEADER_LABELS.contains(&normalize_header_label(text).as_str()))
|
||||||
|
.count()
|
||||||
|
>= HEADER_MATCH_MIN
|
||||||
|
}
|
||||||
|
|
||||||
|
fn parse_row_segments(texts: &[String]) -> KseiRawRow {
|
||||||
let mut row = KseiRawRow {
|
let mut row = KseiRawRow {
|
||||||
date: String::new(),
|
date: String::new(),
|
||||||
share_code: String::new(),
|
share_code: String::new(),
|
||||||
|
|
@ -186,45 +229,150 @@ fn extract_rows_from_page(page: PageGrid) -> Vec<KseiRawRow> {
|
||||||
percentage: String::new(),
|
percentage: String::new(),
|
||||||
};
|
};
|
||||||
|
|
||||||
if let Some(col_map) = page.get(&y) {
|
let mut remaining: Vec<String> = texts
|
||||||
for (col_idx, chars) in col_map {
|
|
||||||
let mut sorted = chars.clone();
|
|
||||||
sorted.sort_by_key(|(x, _)| *x);
|
|
||||||
let text = normalize_spaces(&sorted.iter().map(|(_, c)| c).collect::<String>());
|
|
||||||
assign_column(&mut row, *col_idx, text);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
if is_data_row(&row) {
|
|
||||||
page_rows.push(row);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
page_rows
|
|
||||||
}
|
|
||||||
|
|
||||||
fn assign_column(row: &mut KseiRawRow, col_idx: usize, value: String) {
|
|
||||||
match col_idx {
|
|
||||||
0 => row.date = value,
|
|
||||||
1 => row.share_code = value,
|
|
||||||
2 => row.issuer_name = value,
|
|
||||||
3 => row.investor_name = value,
|
|
||||||
4 => row.investor_type = value,
|
|
||||||
5 => row.local_foreign = value,
|
|
||||||
6 => row.nationality = value,
|
|
||||||
7 => row.domicile = value,
|
|
||||||
8 => row.holdings_scripless = value,
|
|
||||||
9 => row.holdings_scrip = value,
|
|
||||||
10 => row.total_holding_shares = value,
|
|
||||||
11 => row.percentage = value,
|
|
||||||
_ => {}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
fn x_to_column(x: f32) -> Option<usize> {
|
|
||||||
COLUMN_BOUNDS
|
|
||||||
.iter()
|
.iter()
|
||||||
.position(|(left, right)| x >= *left && x < *right)
|
.map(|text| normalize_spaces(text))
|
||||||
|
.filter(|text| !text.is_empty())
|
||||||
|
.collect();
|
||||||
|
if remaining.is_empty() {
|
||||||
|
return row;
|
||||||
|
}
|
||||||
|
|
||||||
|
if let Some((date, share_code)) = split_date_and_share(&remaining[0]) {
|
||||||
|
row.date = date;
|
||||||
|
row.share_code = share_code;
|
||||||
|
let _ = remaining.remove(0);
|
||||||
|
}
|
||||||
|
|
||||||
|
if row.date.is_empty() {
|
||||||
|
return row;
|
||||||
|
}
|
||||||
|
|
||||||
|
if row.share_code.is_empty()
|
||||||
|
&& remaining
|
||||||
|
.first()
|
||||||
|
.is_some_and(|segment| is_share_code_like(segment))
|
||||||
|
{
|
||||||
|
row.share_code = remaining.remove(0);
|
||||||
|
}
|
||||||
|
|
||||||
|
if remaining
|
||||||
|
.last()
|
||||||
|
.is_some_and(|segment| is_percentage_like(segment))
|
||||||
|
{
|
||||||
|
row.percentage = remaining.pop().unwrap_or_default();
|
||||||
|
}
|
||||||
|
if remaining
|
||||||
|
.last()
|
||||||
|
.is_some_and(|segment| is_id_number_like(segment))
|
||||||
|
{
|
||||||
|
row.total_holding_shares = remaining.pop().unwrap_or_default();
|
||||||
|
}
|
||||||
|
if remaining
|
||||||
|
.last()
|
||||||
|
.is_some_and(|segment| is_id_number_like(segment))
|
||||||
|
{
|
||||||
|
row.holdings_scrip = remaining.pop().unwrap_or_default();
|
||||||
|
}
|
||||||
|
if remaining
|
||||||
|
.last()
|
||||||
|
.is_some_and(|segment| is_id_number_like(segment))
|
||||||
|
{
|
||||||
|
row.holdings_scripless = remaining.pop().unwrap_or_default();
|
||||||
|
}
|
||||||
|
|
||||||
|
let mut geo_fields = Vec::new();
|
||||||
|
while remaining.len() > 2 {
|
||||||
|
let Some(candidate) = remaining.last().cloned() else {
|
||||||
|
break;
|
||||||
|
};
|
||||||
|
|
||||||
|
if row.local_foreign.is_empty() && is_locality_marker(&candidate) {
|
||||||
|
row.local_foreign = remaining.pop().unwrap_or_default();
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
if row.investor_type.is_empty() && is_investor_type_marker(&candidate) {
|
||||||
|
row.investor_type = remaining.pop().unwrap_or_default();
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
if geo_fields.len() < 2 {
|
||||||
|
geo_fields.push(remaining.pop().unwrap_or_default());
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
|
||||||
|
geo_fields.reverse();
|
||||||
|
if let Some(first) = geo_fields.first() {
|
||||||
|
row.nationality = first.clone();
|
||||||
|
}
|
||||||
|
if geo_fields.len() > 1 {
|
||||||
|
row.domicile = geo_fields[1..].join(" ");
|
||||||
|
}
|
||||||
|
|
||||||
|
if let Some(first) = remaining.first() {
|
||||||
|
row.issuer_name = first.clone();
|
||||||
|
}
|
||||||
|
if remaining.len() > 1 {
|
||||||
|
row.investor_name = remaining[1..].join(" ");
|
||||||
|
}
|
||||||
|
|
||||||
|
row
|
||||||
|
}
|
||||||
|
|
||||||
|
fn split_date_and_share(segment: &str) -> Option<(String, String)> {
|
||||||
|
let trimmed = segment.trim();
|
||||||
|
if trimmed.len() < 11 {
|
||||||
|
return None;
|
||||||
|
}
|
||||||
|
|
||||||
|
let date = trimmed.get(..11)?.to_string();
|
||||||
|
if !is_ksei_date(&date) {
|
||||||
|
return None;
|
||||||
|
}
|
||||||
|
|
||||||
|
let share_code = trimmed.get(11..).unwrap_or_default().trim().to_string();
|
||||||
|
Some((date, share_code))
|
||||||
|
}
|
||||||
|
|
||||||
|
fn is_share_code_like(value: &str) -> bool {
|
||||||
|
let trimmed = value.trim();
|
||||||
|
(3..=8).contains(&trimmed.len())
|
||||||
|
&& trimmed
|
||||||
|
.chars()
|
||||||
|
.all(|ch| ch.is_ascii_uppercase() || ch.is_ascii_digit())
|
||||||
|
}
|
||||||
|
|
||||||
|
fn is_id_number_like(value: &str) -> bool {
|
||||||
|
let trimmed = value.trim();
|
||||||
|
!trimmed.is_empty()
|
||||||
|
&& trimmed != "-"
|
||||||
|
&& trimmed.chars().all(|ch| ch.is_ascii_digit() || ch == '.')
|
||||||
|
}
|
||||||
|
|
||||||
|
fn is_investor_type_marker(value: &str) -> bool {
|
||||||
|
let trimmed = value.trim();
|
||||||
|
!trimmed.is_empty() && trimmed.len() <= 4 && trimmed.chars().all(|ch| ch.is_ascii_uppercase())
|
||||||
|
}
|
||||||
|
|
||||||
|
fn is_locality_marker(value: &str) -> bool {
|
||||||
|
matches!(
|
||||||
|
value.trim().to_ascii_uppercase().as_str(),
|
||||||
|
"L" | "F" | "A" | "D" | "LOCAL" | "FOREIGN" | "ASING" | "DOMESTIC"
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
fn normalize_header_label(text: &str) -> String {
|
||||||
|
let mut normalized = String::new();
|
||||||
|
for ch in text.chars() {
|
||||||
|
if ch.is_ascii_alphanumeric() {
|
||||||
|
normalized.push(ch.to_ascii_uppercase());
|
||||||
|
}
|
||||||
|
}
|
||||||
|
normalized
|
||||||
}
|
}
|
||||||
|
|
||||||
fn y_bucket(y: f32) -> i32 {
|
fn y_bucket(y: f32) -> i32 {
|
||||||
|
|
@ -236,7 +384,7 @@ fn normalize_spaces(input: &str) -> String {
|
||||||
let mut prev_space = false;
|
let mut prev_space = false;
|
||||||
|
|
||||||
for ch in input.chars() {
|
for ch in input.chars() {
|
||||||
if ch == ' ' {
|
if ch.is_whitespace() {
|
||||||
if !prev_space {
|
if !prev_space {
|
||||||
out.push(' ');
|
out.push(' ');
|
||||||
}
|
}
|
||||||
|
|
@ -311,24 +459,37 @@ fn is_percentage_like(s: &str) -> bool {
|
||||||
|
|
||||||
#[cfg(test)]
|
#[cfg(test)]
|
||||||
mod tests {
|
mod tests {
|
||||||
use std::fs;
|
|
||||||
use std::path::Path;
|
use std::path::Path;
|
||||||
|
|
||||||
use super::{check_mutool, parse_ksei_pdf, parse_stext_xml};
|
use super::{check_mutool, parse_ksei_pdf, parse_stext_xml};
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn test_parse_stext_xml_fixture_extracts_rows() {
|
fn test_parse_stext_xml_live_like_lines_extract_rows() {
|
||||||
let fixture_path = Path::new("tests/fixtures/ksei_stext_sample.xml");
|
let xml = build_live_like_stext_xml();
|
||||||
let xml = fs::read_to_string(fixture_path).expect("failed to read stext fixture");
|
|
||||||
|
|
||||||
let rows = parse_stext_xml(&xml).expect("failed to parse fixture XML");
|
let rows = parse_stext_xml(&xml).expect("failed to parse live-like fixture XML");
|
||||||
assert_eq!(rows.len(), 3);
|
assert_eq!(rows.len(), 3);
|
||||||
|
|
||||||
let first = &rows[0];
|
let first = &rows[0];
|
||||||
assert_eq!(first.date, "27-Feb-2026");
|
assert_eq!(first.date, "27-Feb-2026");
|
||||||
assert_eq!(first.share_code, "BBCA");
|
assert_eq!(first.share_code, "AADI");
|
||||||
assert_eq!(first.investor_name, "PT DWIMURIA INVESTAMA ANDALAN");
|
assert_eq!(first.issuer_name, "ADARO ANDALAN INDONESIA Tbk");
|
||||||
assert_eq!(first.percentage, "54,94");
|
assert_eq!(first.investor_name, "ADARO STRATEGIC INVESTMENTS");
|
||||||
|
assert_eq!(first.investor_type, "CP");
|
||||||
|
assert_eq!(first.local_foreign, "D");
|
||||||
|
assert_eq!(first.nationality, "INDONESIA");
|
||||||
|
assert_eq!(first.holdings_scripless, "3.200.142.830");
|
||||||
|
assert_eq!(first.holdings_scrip, "0");
|
||||||
|
assert_eq!(first.total_holding_shares, "3.200.142.830");
|
||||||
|
assert_eq!(first.percentage, "66,18");
|
||||||
|
|
||||||
|
let last = &rows[2];
|
||||||
|
assert_eq!(last.share_code, "BBRI");
|
||||||
|
assert_eq!(last.investor_name, "PT NUSANTARA CAPITAL");
|
||||||
|
assert_eq!(last.investor_type, "ID");
|
||||||
|
assert_eq!(last.local_foreign, "A");
|
||||||
|
assert_eq!(last.nationality, "SINGAPORE");
|
||||||
|
assert_eq!(last.percentage, "15,00");
|
||||||
}
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
|
|
@ -352,4 +513,97 @@ mod tests {
|
||||||
rows.len()
|
rows.len()
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
fn build_live_like_stext_xml() -> String {
|
||||||
|
let mut xml = String::from(r#"<?xml version="1.0"?><document>"#);
|
||||||
|
|
||||||
|
append_page(
|
||||||
|
&mut xml,
|
||||||
|
"page1",
|
||||||
|
&[
|
||||||
|
(31.56, 86.33, "DATE"),
|
||||||
|
(56.64, 86.33, "SHARE_CODE"),
|
||||||
|
(121.22, 86.33, "ISSUER_NAME"),
|
||||||
|
(267.17, 86.33, "INVESTOR_NAME"),
|
||||||
|
(390.89, 86.33, "INVESTOR_TYPE"),
|
||||||
|
(434.11, 86.33, "LOCAL_FOREIGN"),
|
||||||
|
(475.87, 86.33, "NATIONALITY"),
|
||||||
|
(525.19, 86.33, "DOMICILE"),
|
||||||
|
(574.63, 86.33, "HOLDINGS_SCRIPLESS"),
|
||||||
|
(629.98, 86.33, "HOLDINGS_SCRIP"),
|
||||||
|
(680.02, 86.33, "TOTAL_HOLDING_SHARES"),
|
||||||
|
(741.22, 86.33, "PERCENTAGE"),
|
||||||
|
(28.68, 91.01, "27-Feb-2026 AADI"),
|
||||||
|
(85.10, 91.01, "ADARO ANDALAN INDONESIA Tbk"),
|
||||||
|
(179.30, 91.01, "ADARO STRATEGIC INVESTMENTS"),
|
||||||
|
(381.41, 91.01, "CP"),
|
||||||
|
(424.75, 91.01, "D"),
|
||||||
|
(504.07, 91.01, "INDONESIA"),
|
||||||
|
(597.58, 91.01, "3.200.142.830"),
|
||||||
|
(630.10, 91.01, "0"),
|
||||||
|
(680.12, 91.01, "3.200.142.830"),
|
||||||
|
(741.30, 91.01, "66,18"),
|
||||||
|
(28.68, 96.01, "27-Feb-2026 AADI"),
|
||||||
|
(85.10, 96.01, "ADARO ANDALAN INDONESIA Tbk"),
|
||||||
|
(179.30, 96.01, "PUBLIC"),
|
||||||
|
(381.41, 96.01, "OT"),
|
||||||
|
(424.75, 96.01, "A"),
|
||||||
|
(504.07, 96.01, "SINGAPORE"),
|
||||||
|
(597.58, 96.01, "500.000.000"),
|
||||||
|
(630.10, 96.01, "0"),
|
||||||
|
(680.12, 96.01, "500.000.000"),
|
||||||
|
(741.30, 96.01, "10,34"),
|
||||||
|
],
|
||||||
|
);
|
||||||
|
|
||||||
|
append_page(
|
||||||
|
&mut xml,
|
||||||
|
"page2",
|
||||||
|
&[
|
||||||
|
(28.68, 20.00, "27-Feb-2026 BBRI"),
|
||||||
|
(85.10, 20.00, "BANK RAKYAT INDONESIA Tbk"),
|
||||||
|
(179.30, 20.00, "PT NUSANTARA CAPITAL"),
|
||||||
|
(381.41, 20.00, "ID"),
|
||||||
|
(424.75, 20.00, "A"),
|
||||||
|
(504.07, 20.00, "SINGAPORE"),
|
||||||
|
(597.58, 20.00, "1.250.000.000"),
|
||||||
|
(630.10, 20.00, "0"),
|
||||||
|
(680.12, 20.00, "1.250.000.000"),
|
||||||
|
(741.30, 20.00, "15,00"),
|
||||||
|
],
|
||||||
|
);
|
||||||
|
|
||||||
|
xml.push_str("</document>");
|
||||||
|
xml
|
||||||
|
}
|
||||||
|
|
||||||
|
fn append_page(xml: &mut String, id: &str, lines: &[(f32, f32, &str)]) {
|
||||||
|
xml.push_str(&format!(r#"<page id="{id}" width="792" height="612">"#));
|
||||||
|
for (x, y, text) in lines {
|
||||||
|
append_line(xml, *x, *y, text);
|
||||||
|
}
|
||||||
|
xml.push_str("</page>");
|
||||||
|
}
|
||||||
|
|
||||||
|
fn append_line(xml: &mut String, x: f32, y: f32, text: &str) {
|
||||||
|
let width = x + (text.len() as f32 * 2.0);
|
||||||
|
let height = y + 3.48;
|
||||||
|
xml.push_str(&format!(
|
||||||
|
r#"<line bbox="{x:.2} {y:.2} {width:.2} {height:.2}" text="{}"></line>"#,
|
||||||
|
escape_xml_attr(text)
|
||||||
|
));
|
||||||
|
}
|
||||||
|
|
||||||
|
fn escape_xml_attr(text: &str) -> String {
|
||||||
|
text.chars()
|
||||||
|
.map(|ch| match ch {
|
||||||
|
'&' => "&".to_string(),
|
||||||
|
'<' => "<".to_string(),
|
||||||
|
'>' => ">".to_string(),
|
||||||
|
'"' => """.to_string(),
|
||||||
|
'\'' => "'".to_string(),
|
||||||
|
other => other.to_string(),
|
||||||
|
})
|
||||||
|
.collect::<String>()
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
|
||||||
664
src/ownership/remote.rs
Normal file
664
src/ownership/remote.rs
Normal file
|
|
@ -0,0 +1,664 @@
|
||||||
|
use std::fs;
|
||||||
|
use std::path::Path;
|
||||||
|
|
||||||
|
use serde::{Deserialize, Serialize};
|
||||||
|
|
||||||
|
use crate::curl_impersonate;
|
||||||
|
use crate::error::IdxError;
|
||||||
|
|
||||||
|
pub const IDX_ANNOUNCEMENT_LISTING_URL: &str = "https://www.idx.co.id/id/berita/pengumuman/";
|
||||||
|
const DEFAULT_IDX_ANNOUNCEMENT_API_URL: &str =
|
||||||
|
"https://www.idx.co.id/primary/NewsAnnouncement/GetAllAnnouncement";
|
||||||
|
const IDX_ANNOUNCEMENT_API_ENV: &str = "IDX_OWNERSHIP_ANNOUNCEMENT_API_URL";
|
||||||
|
const IDX_ANNOUNCEMENT_LISTING_ENV: &str = "IDX_OWNERSHIP_ANNOUNCEMENT_PAGE_URL";
|
||||||
|
const IDX_ANNOUNCEMENT_PAGE_SIZE: usize = 10;
|
||||||
|
const USER_AGENT: &str = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36";
|
||||||
|
|
||||||
|
#[derive(Debug, Clone, Deserialize)]
|
||||||
|
pub struct AnnouncementPage {
|
||||||
|
#[serde(rename = "Items", default)]
|
||||||
|
pub items: Vec<AnnouncementItem>,
|
||||||
|
#[serde(rename = "ItemCount")]
|
||||||
|
pub item_count: Option<usize>,
|
||||||
|
#[serde(rename = "PageCount")]
|
||||||
|
pub page_count: Option<usize>,
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Debug, Clone, Deserialize)]
|
||||||
|
pub struct AnnouncementItem {
|
||||||
|
#[serde(rename = "PublishDate")]
|
||||||
|
pub publish_date: String,
|
||||||
|
#[serde(rename = "Title")]
|
||||||
|
pub title: String,
|
||||||
|
#[serde(rename = "AnnouncementType")]
|
||||||
|
pub announcement_type: Option<String>,
|
||||||
|
#[serde(rename = "Code")]
|
||||||
|
pub code: Option<String>,
|
||||||
|
#[serde(rename = "Attachments", default)]
|
||||||
|
pub attachments: Vec<AnnouncementAttachment>,
|
||||||
|
#[serde(rename = "PdfPath")]
|
||||||
|
pub pdf_path: Option<String>,
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Debug, Clone, Deserialize)]
|
||||||
|
pub struct AnnouncementAttachment {
|
||||||
|
#[serde(rename = "FullSavePath")]
|
||||||
|
pub full_save_path: String,
|
||||||
|
#[serde(rename = "OriginalFilename")]
|
||||||
|
pub original_filename: Option<String>,
|
||||||
|
#[serde(rename = "PDFFilename")]
|
||||||
|
pub pdf_filename: Option<String>,
|
||||||
|
#[serde(rename = "IsAttachment")]
|
||||||
|
pub is_attachment: Option<serde_json::Value>,
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize)]
|
||||||
|
#[serde(rename_all = "snake_case")]
|
||||||
|
pub enum OwnershipReportFamily {
|
||||||
|
AboveOnePercent,
|
||||||
|
AboveFivePercent,
|
||||||
|
InvestorTypeBreakdown,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl OwnershipReportFamily {
|
||||||
|
pub fn cli_name(self) -> &'static str {
|
||||||
|
match self {
|
||||||
|
Self::AboveOnePercent => "above1",
|
||||||
|
Self::AboveFivePercent => "above5",
|
||||||
|
Self::InvestorTypeBreakdown => "investor-type",
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn label(self) -> &'static str {
|
||||||
|
match self {
|
||||||
|
Self::AboveOnePercent => "Above 1%",
|
||||||
|
Self::AboveFivePercent => "Above 5%",
|
||||||
|
Self::InvestorTypeBreakdown => "Investor Type Breakdown",
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Debug, Clone, PartialEq, Eq, Serialize)]
|
||||||
|
pub struct DiscoveredOwnershipPdf {
|
||||||
|
pub family: OwnershipReportFamily,
|
||||||
|
pub listing_page_url: String,
|
||||||
|
pub query_url: String,
|
||||||
|
pub pdf_url: String,
|
||||||
|
pub title: String,
|
||||||
|
pub publish_date: String,
|
||||||
|
pub code: Option<String>,
|
||||||
|
pub original_filename: Option<String>,
|
||||||
|
pub is_attachment: bool,
|
||||||
|
}
|
||||||
|
|
||||||
|
struct DiscoveryQuery {
|
||||||
|
family: OwnershipReportFamily,
|
||||||
|
keywords: &'static str,
|
||||||
|
title_needles: &'static [&'static str],
|
||||||
|
}
|
||||||
|
|
||||||
|
const DISCOVERY_QUERIES: &[DiscoveryQuery] = &[
|
||||||
|
DiscoveryQuery {
|
||||||
|
family: OwnershipReportFamily::AboveOnePercent,
|
||||||
|
keywords: "pemegang saham di atas 1",
|
||||||
|
title_needles: &["PEMEGANG SAHAM DI ATAS 1", "SHAREHOLDERS ABOVE 1"],
|
||||||
|
},
|
||||||
|
DiscoveryQuery {
|
||||||
|
family: OwnershipReportFamily::AboveFivePercent,
|
||||||
|
keywords: "pemegang saham di atas 5",
|
||||||
|
title_needles: &["PEMEGANG SAHAM DI ATAS 5", "SHAREHOLDERS ABOVE 5"],
|
||||||
|
},
|
||||||
|
DiscoveryQuery {
|
||||||
|
family: OwnershipReportFamily::InvestorTypeBreakdown,
|
||||||
|
keywords: "kepemilikan saham perusahaan tercatat",
|
||||||
|
title_needles: &[
|
||||||
|
"KEPEMILIKAN SAHAM PERUSAHAAN TERCATAT BERDASARKAN TIPE INVESTOR",
|
||||||
|
"DATA KSEI TERKAIT KEPEMILIKAN SAHAM PERUSAHAAN TERCATAT",
|
||||||
|
],
|
||||||
|
},
|
||||||
|
];
|
||||||
|
|
||||||
|
pub fn announcement_listing_url() -> String {
|
||||||
|
std::env::var(IDX_ANNOUNCEMENT_LISTING_ENV)
|
||||||
|
.ok()
|
||||||
|
.filter(|value| !value.trim().is_empty())
|
||||||
|
.unwrap_or_else(|| IDX_ANNOUNCEMENT_LISTING_URL.to_string())
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn announcement_api_url() -> String {
|
||||||
|
std::env::var(IDX_ANNOUNCEMENT_API_ENV)
|
||||||
|
.ok()
|
||||||
|
.filter(|value| !value.trim().is_empty())
|
||||||
|
.unwrap_or_else(|| DEFAULT_IDX_ANNOUNCEMENT_API_URL.to_string())
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn build_announcement_query_url(
|
||||||
|
keywords: &str,
|
||||||
|
page_number: usize,
|
||||||
|
page_size: usize,
|
||||||
|
) -> String {
|
||||||
|
format!(
|
||||||
|
"{}?keywords={}&pageNumber={page_number}&pageSize={page_size}&lang=id",
|
||||||
|
announcement_api_url(),
|
||||||
|
percent_encode(keywords),
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn parse_announcement_page(raw: &str) -> Result<AnnouncementPage, IdxError> {
|
||||||
|
let trimmed = raw.trim();
|
||||||
|
if trimmed.is_empty() {
|
||||||
|
return Err(IdxError::Http(
|
||||||
|
"IDX ownership discovery returned an empty announcement payload".to_string(),
|
||||||
|
));
|
||||||
|
}
|
||||||
|
let normalized = trimmed.to_ascii_lowercase();
|
||||||
|
if normalized.starts_with("<!doctype html") || normalized.starts_with("<html") {
|
||||||
|
return Err(IdxError::Http(
|
||||||
|
"IDX ownership discovery returned HTML instead of announcement JSON".to_string(),
|
||||||
|
));
|
||||||
|
}
|
||||||
|
|
||||||
|
serde_json::from_str(trimmed)
|
||||||
|
.map_err(|e| IdxError::ParseError(format!("failed to parse IDX announcement JSON: {e}")))
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn select_latest_ownership_reports(
|
||||||
|
page: &AnnouncementPage,
|
||||||
|
query_url: &str,
|
||||||
|
family: OwnershipReportFamily,
|
||||||
|
) -> Result<Vec<DiscoveredOwnershipPdf>, IdxError> {
|
||||||
|
let Some(query) = DISCOVERY_QUERIES
|
||||||
|
.iter()
|
||||||
|
.find(|query| query.family == family)
|
||||||
|
else {
|
||||||
|
return Err(IdxError::Http(format!(
|
||||||
|
"failed to discover IDX ownership reports from {query_url}: unknown report family"
|
||||||
|
)));
|
||||||
|
};
|
||||||
|
|
||||||
|
let mut matches: Vec<&AnnouncementItem> = page
|
||||||
|
.items
|
||||||
|
.iter()
|
||||||
|
.filter(|item| item_matches_family(item, query))
|
||||||
|
.collect();
|
||||||
|
matches.sort_by(|left, right| right.publish_date.cmp(&left.publish_date));
|
||||||
|
|
||||||
|
let Some(item) = matches.into_iter().next() else {
|
||||||
|
return Err(IdxError::Http(format!(
|
||||||
|
"failed to discover IDX ownership reports from {query_url}: no matching announcement found"
|
||||||
|
)));
|
||||||
|
};
|
||||||
|
|
||||||
|
let mut attachments = item_pdf_attachments(item)
|
||||||
|
.into_iter()
|
||||||
|
.map(|attachment| {
|
||||||
|
let is_attachment = attachment_is_attachment(&attachment);
|
||||||
|
let original_filename = attachment
|
||||||
|
.original_filename
|
||||||
|
.clone()
|
||||||
|
.or(attachment.pdf_filename.clone())
|
||||||
|
.map(|value| value.trim().to_string());
|
||||||
|
|
||||||
|
DiscoveredOwnershipPdf {
|
||||||
|
family,
|
||||||
|
listing_page_url: announcement_listing_url(),
|
||||||
|
query_url: query_url.to_string(),
|
||||||
|
pdf_url: attachment.full_save_path,
|
||||||
|
title: item.title.clone(),
|
||||||
|
publish_date: item.publish_date.clone(),
|
||||||
|
code: clean_option(item.code.as_deref()),
|
||||||
|
original_filename,
|
||||||
|
is_attachment,
|
||||||
|
}
|
||||||
|
})
|
||||||
|
.collect::<Vec<_>>();
|
||||||
|
|
||||||
|
attachments.sort_by(|left, right| {
|
||||||
|
left.is_attachment
|
||||||
|
.cmp(&right.is_attachment)
|
||||||
|
.then_with(|| left.original_filename.cmp(&right.original_filename))
|
||||||
|
});
|
||||||
|
|
||||||
|
if attachments.is_empty() {
|
||||||
|
return Err(IdxError::Http(format!(
|
||||||
|
"failed to discover IDX ownership reports from {query_url}: matching announcement had no PDF attachments"
|
||||||
|
)));
|
||||||
|
}
|
||||||
|
|
||||||
|
Ok(attachments)
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn discover_idx_ownership_reports(
|
||||||
|
family_filter: Option<OwnershipReportFamily>,
|
||||||
|
limit: usize,
|
||||||
|
) -> Result<Vec<DiscoveredOwnershipPdf>, IdxError> {
|
||||||
|
let mut discovered = Vec::new();
|
||||||
|
let mut errors = Vec::new();
|
||||||
|
|
||||||
|
for query in DISCOVERY_QUERIES
|
||||||
|
.iter()
|
||||||
|
.filter(|query| family_filter.is_none_or(|family| family == query.family))
|
||||||
|
{
|
||||||
|
let query_url = build_announcement_query_url(query.keywords, 1, IDX_ANNOUNCEMENT_PAGE_SIZE);
|
||||||
|
match fetch_text(
|
||||||
|
"IDX ownership announcement discovery",
|
||||||
|
&query_url,
|
||||||
|
&json_headers(),
|
||||||
|
)
|
||||||
|
.and_then(|raw| parse_announcement_page(&raw))
|
||||||
|
.and_then(|page| select_latest_ownership_reports(&page, &query_url, query.family))
|
||||||
|
{
|
||||||
|
Ok(mut reports) => discovered.append(&mut reports),
|
||||||
|
Err(err) => errors.push(err.to_string()),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
discovered.sort_by(|left, right| {
|
||||||
|
right
|
||||||
|
.publish_date
|
||||||
|
.cmp(&left.publish_date)
|
||||||
|
.then_with(|| left.is_attachment.cmp(&right.is_attachment))
|
||||||
|
.then_with(|| left.original_filename.cmp(&right.original_filename))
|
||||||
|
});
|
||||||
|
|
||||||
|
if limit < discovered.len() {
|
||||||
|
discovered.truncate(limit);
|
||||||
|
}
|
||||||
|
|
||||||
|
if discovered.is_empty() {
|
||||||
|
let detail = if errors.is_empty() {
|
||||||
|
"no matching ownership reports found".to_string()
|
||||||
|
} else {
|
||||||
|
errors.join("; ")
|
||||||
|
};
|
||||||
|
return Err(IdxError::Http(format!(
|
||||||
|
"failed to discover IDX ownership reports from {}: {detail}",
|
||||||
|
announcement_listing_url()
|
||||||
|
)));
|
||||||
|
}
|
||||||
|
|
||||||
|
Ok(discovered)
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn download_idx_pdf(url: &str, target: &Path) -> Result<(), IdxError> {
|
||||||
|
let bytes = fetch_bytes("IDX ownership PDF download", url, &pdf_headers())?;
|
||||||
|
validate_pdf_payload(&bytes)?;
|
||||||
|
|
||||||
|
fs::write(target, &bytes).map_err(|e| {
|
||||||
|
IdxError::Io(format!(
|
||||||
|
"failed writing cached PDF {}: {e}",
|
||||||
|
target.display()
|
||||||
|
))
|
||||||
|
})?;
|
||||||
|
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn validate_pdf_payload(bytes: &[u8]) -> Result<(), IdxError> {
|
||||||
|
if bytes.is_empty() {
|
||||||
|
return Err(IdxError::Http(
|
||||||
|
"IDX ownership download returned an empty response".to_string(),
|
||||||
|
));
|
||||||
|
}
|
||||||
|
|
||||||
|
let trimmed = bytes
|
||||||
|
.iter()
|
||||||
|
.skip_while(|byte| byte.is_ascii_whitespace())
|
||||||
|
.copied()
|
||||||
|
.collect::<Vec<_>>();
|
||||||
|
if trimmed.starts_with(b"%PDF-") {
|
||||||
|
return Ok(());
|
||||||
|
}
|
||||||
|
|
||||||
|
let preview = String::from_utf8_lossy(&trimmed[..trimmed.len().min(256)]).to_ascii_lowercase();
|
||||||
|
if preview.contains("<!doctype html") || preview.contains("<html") {
|
||||||
|
return Err(IdxError::Http(
|
||||||
|
"IDX ownership download returned HTML instead of a PDF".to_string(),
|
||||||
|
));
|
||||||
|
}
|
||||||
|
|
||||||
|
Err(IdxError::Http(
|
||||||
|
"IDX ownership download did not look like a PDF".to_string(),
|
||||||
|
))
|
||||||
|
}
|
||||||
|
|
||||||
|
fn fetch_text(stage: &str, url: &str, headers: &[(String, String)]) -> Result<String, IdxError> {
|
||||||
|
let bytes = fetch_bytes(stage, url, headers)?;
|
||||||
|
String::from_utf8(bytes)
|
||||||
|
.map_err(|e| IdxError::Http(format!("failed to decode {stage} response as utf-8: {e}")))
|
||||||
|
}
|
||||||
|
|
||||||
|
fn fetch_bytes(stage: &str, url: &str, headers: &[(String, String)]) -> Result<Vec<u8>, IdxError> {
|
||||||
|
let mut args = vec![
|
||||||
|
"--silent".to_string(),
|
||||||
|
"--show-error".to_string(),
|
||||||
|
"--location".to_string(),
|
||||||
|
"--fail".to_string(),
|
||||||
|
"--compressed".to_string(),
|
||||||
|
];
|
||||||
|
for (name, value) in headers {
|
||||||
|
args.push("--header".to_string());
|
||||||
|
args.push(format!("{name}: {value}"));
|
||||||
|
}
|
||||||
|
args.push(url.to_string());
|
||||||
|
|
||||||
|
let output = curl_impersonate::run_owned(stage, &args)?;
|
||||||
|
Ok(output.stdout)
|
||||||
|
}
|
||||||
|
|
||||||
|
fn json_headers() -> Vec<(String, String)> {
|
||||||
|
vec![
|
||||||
|
("User-Agent".to_string(), USER_AGENT.to_string()),
|
||||||
|
(
|
||||||
|
"Accept".to_string(),
|
||||||
|
"application/json,text/plain,*/*".to_string(),
|
||||||
|
),
|
||||||
|
(
|
||||||
|
"Accept-Language".to_string(),
|
||||||
|
"id-ID,id;q=0.9,en-US;q=0.8,en;q=0.7".to_string(),
|
||||||
|
),
|
||||||
|
("Referer".to_string(), announcement_listing_url()),
|
||||||
|
]
|
||||||
|
}
|
||||||
|
|
||||||
|
fn pdf_headers() -> Vec<(String, String)> {
|
||||||
|
vec![
|
||||||
|
("User-Agent".to_string(), USER_AGENT.to_string()),
|
||||||
|
(
|
||||||
|
"Accept".to_string(),
|
||||||
|
"application/pdf,application/octet-stream,*/*;q=0.8".to_string(),
|
||||||
|
),
|
||||||
|
(
|
||||||
|
"Accept-Language".to_string(),
|
||||||
|
"id-ID,id;q=0.9,en-US;q=0.8,en;q=0.7".to_string(),
|
||||||
|
),
|
||||||
|
("Referer".to_string(), announcement_listing_url()),
|
||||||
|
]
|
||||||
|
}
|
||||||
|
|
||||||
|
fn item_matches_family(item: &AnnouncementItem, query: &DiscoveryQuery) -> bool {
|
||||||
|
let title = item.title.to_ascii_uppercase();
|
||||||
|
query
|
||||||
|
.title_needles
|
||||||
|
.iter()
|
||||||
|
.any(|needle| title.contains(needle))
|
||||||
|
}
|
||||||
|
|
||||||
|
fn item_pdf_attachments(item: &AnnouncementItem) -> Vec<AnnouncementAttachment> {
|
||||||
|
let attachments = if item.attachments.is_empty() {
|
||||||
|
item.pdf_path
|
||||||
|
.as_deref()
|
||||||
|
.and_then(parse_pdf_path)
|
||||||
|
.unwrap_or_default()
|
||||||
|
} else {
|
||||||
|
item.attachments.clone()
|
||||||
|
};
|
||||||
|
|
||||||
|
attachments
|
||||||
|
.into_iter()
|
||||||
|
.filter(|attachment| {
|
||||||
|
attachment
|
||||||
|
.full_save_path
|
||||||
|
.to_ascii_lowercase()
|
||||||
|
.ends_with(".pdf")
|
||||||
|
})
|
||||||
|
.collect()
|
||||||
|
}
|
||||||
|
|
||||||
|
fn attachment_label(attachment: &AnnouncementAttachment) -> String {
|
||||||
|
attachment
|
||||||
|
.original_filename
|
||||||
|
.as_deref()
|
||||||
|
.or(attachment.pdf_filename.as_deref())
|
||||||
|
.unwrap_or_default()
|
||||||
|
.trim()
|
||||||
|
.to_ascii_lowercase()
|
||||||
|
}
|
||||||
|
|
||||||
|
fn parse_pdf_path(raw: &str) -> Option<Vec<AnnouncementAttachment>> {
|
||||||
|
serde_json::from_str::<Vec<AnnouncementAttachment>>(raw).ok()
|
||||||
|
}
|
||||||
|
|
||||||
|
fn attachment_is_attachment(attachment: &AnnouncementAttachment) -> bool {
|
||||||
|
match attachment.is_attachment.as_ref() {
|
||||||
|
Some(serde_json::Value::Bool(value)) => *value,
|
||||||
|
Some(serde_json::Value::Number(value)) => value.as_i64() == Some(1),
|
||||||
|
Some(serde_json::Value::String(value)) => {
|
||||||
|
matches!(value.trim(), "1" | "true" | "TRUE" | "True")
|
||||||
|
}
|
||||||
|
_ => attachment_label(attachment).contains("lamp"),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn clean_option(value: Option<&str>) -> Option<String> {
|
||||||
|
value
|
||||||
|
.map(str::trim)
|
||||||
|
.filter(|value| !value.is_empty())
|
||||||
|
.map(str::to_string)
|
||||||
|
}
|
||||||
|
|
||||||
|
fn percent_encode(value: &str) -> String {
|
||||||
|
let mut encoded = String::with_capacity(value.len());
|
||||||
|
for byte in value.bytes() {
|
||||||
|
match byte {
|
||||||
|
b'A'..=b'Z' | b'a'..=b'z' | b'0'..=b'9' | b'-' | b'_' | b'.' | b'~' => {
|
||||||
|
encoded.push(byte as char)
|
||||||
|
}
|
||||||
|
b' ' => encoded.push_str("%20"),
|
||||||
|
other => encoded.push_str(&format!("%{other:02X}")),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
encoded
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod tests {
|
||||||
|
use super::{
|
||||||
|
AnnouncementPage, IDX_ANNOUNCEMENT_LISTING_URL, OwnershipReportFamily,
|
||||||
|
build_announcement_query_url, parse_announcement_page, select_latest_ownership_reports,
|
||||||
|
validate_pdf_payload,
|
||||||
|
};
|
||||||
|
use crate::error::IdxError;
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn builds_announcement_query_url() {
|
||||||
|
let url = build_announcement_query_url("pemegang saham di atas 5", 1, 10);
|
||||||
|
assert_eq!(
|
||||||
|
url,
|
||||||
|
"https://www.idx.co.id/primary/NewsAnnouncement/GetAllAnnouncement?keywords=pemegang%20saham%20di%20atas%205&pageNumber=1&pageSize=10&lang=id"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn parses_fixture_and_selects_latest_above_5_reports() {
|
||||||
|
let raw = include_str!("../../tests/fixtures/idx_announcement_kepemilikan.json");
|
||||||
|
let page = parse_announcement_page(raw).expect("announcement fixture should parse");
|
||||||
|
let discovered = select_latest_ownership_reports(
|
||||||
|
&page,
|
||||||
|
"https://www.idx.co.id/primary/NewsAnnouncement/GetAllAnnouncement?keywords=pemegang%20saham%20di%20atas%205&pageNumber=1&pageSize=10&lang=id",
|
||||||
|
OwnershipReportFamily::AboveFivePercent,
|
||||||
|
)
|
||||||
|
.expect("should select ownership reports");
|
||||||
|
|
||||||
|
assert_eq!(discovered.len(), 2);
|
||||||
|
assert_eq!(
|
||||||
|
discovered[0].family,
|
||||||
|
OwnershipReportFamily::AboveFivePercent
|
||||||
|
);
|
||||||
|
assert_eq!(discovered[0].listing_page_url, IDX_ANNOUNCEMENT_LISTING_URL);
|
||||||
|
assert_eq!(discovered[0].publish_date, "2026-03-27T16:34:20");
|
||||||
|
assert_eq!(
|
||||||
|
discovered[0].pdf_url,
|
||||||
|
"https://www.idx.co.id/StaticData/NewsAndAnnouncement/ANNOUNCEMENTSTOCK/From_EREP/202603/5d31bb6f49_announcement.pdf"
|
||||||
|
);
|
||||||
|
assert_eq!(
|
||||||
|
discovered[0].original_filename.as_deref(),
|
||||||
|
Some("20260327_Semua Emiten Saham_Pengumuman Bursa_32055594.pdf")
|
||||||
|
);
|
||||||
|
assert!(!discovered[0].is_attachment);
|
||||||
|
assert_eq!(
|
||||||
|
discovered[1].pdf_url,
|
||||||
|
"https://www.idx.co.id/StaticData/NewsAndAnnouncement/ANNOUNCEMENTSTOCK/From_EREP/202603/4f5c4efc6f_bf70f249ac_lamp1.pdf"
|
||||||
|
);
|
||||||
|
assert_eq!(
|
||||||
|
discovered[1].original_filename.as_deref(),
|
||||||
|
Some("20260327_Semua Emiten Saham_Pengumuman Bursa_32055594_lamp1.pdf")
|
||||||
|
);
|
||||||
|
assert!(discovered[1].is_attachment);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn selects_above_1_reports() {
|
||||||
|
let raw = r#"{
|
||||||
|
"Items": [
|
||||||
|
{
|
||||||
|
"PublishDate": "2026-03-10T12:09:09",
|
||||||
|
"Title": "Pemegang Saham di atas 1% (KSEI)",
|
||||||
|
"AnnouncementType": "",
|
||||||
|
"Code": " Semua Emiten Saham ",
|
||||||
|
"Attachments": [
|
||||||
|
{
|
||||||
|
"PDFFilename": "d67ebf37e6_10d4080288.pdf",
|
||||||
|
"FullSavePath": "https://www.idx.co.id/StaticData/NewsAndAnnouncement/ANNOUNCEMENTSTOCK/From_EREP/202603/d67ebf37e6_10d4080288.pdf",
|
||||||
|
"IsAttachment": 0,
|
||||||
|
"OriginalFilename": "20260310_Semua Emiten Saham_Pengumuman Bursa_32052554.pdf"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"PDFFilename": "b9b638e5a8_8928aca255.pdf",
|
||||||
|
"FullSavePath": "https://www.idx.co.id/StaticData/NewsAndAnnouncement/ANNOUNCEMENTSTOCK/From_EREP/202603/b9b638e5a8_8928aca255.pdf",
|
||||||
|
"IsAttachment": 1,
|
||||||
|
"OriginalFilename": "20260310_Semua Emiten Saham_Pengumuman Bursa_32052554_lamp1.pdf"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"PdfPath": ""
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"ItemCount": 1,
|
||||||
|
"PageCount": 1
|
||||||
|
}"#;
|
||||||
|
|
||||||
|
let page = parse_announcement_page(raw).expect("above-1 fixture should parse");
|
||||||
|
let discovered = select_latest_ownership_reports(
|
||||||
|
&page,
|
||||||
|
"https://www.idx.co.id/primary/NewsAnnouncement/GetAllAnnouncement?keywords=pemegang%20saham%20di%20atas%201&pageNumber=1&pageSize=10&lang=id",
|
||||||
|
OwnershipReportFamily::AboveOnePercent,
|
||||||
|
)
|
||||||
|
.expect("should select above-1 reports");
|
||||||
|
|
||||||
|
assert_eq!(discovered.len(), 2);
|
||||||
|
assert_eq!(discovered[0].family, OwnershipReportFamily::AboveOnePercent);
|
||||||
|
assert_eq!(discovered[0].code.as_deref(), Some("Semua Emiten Saham"));
|
||||||
|
assert_eq!(
|
||||||
|
discovered[0].original_filename.as_deref(),
|
||||||
|
Some("20260310_Semua Emiten Saham_Pengumuman Bursa_32052554.pdf")
|
||||||
|
);
|
||||||
|
assert!(!discovered[0].is_attachment);
|
||||||
|
assert_eq!(
|
||||||
|
discovered[1].pdf_url,
|
||||||
|
"https://www.idx.co.id/StaticData/NewsAndAnnouncement/ANNOUNCEMENTSTOCK/From_EREP/202603/b9b638e5a8_8928aca255.pdf"
|
||||||
|
);
|
||||||
|
assert!(discovered[1].is_attachment);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn selects_investor_type_breakdown_reports() {
|
||||||
|
let raw = r#"{
|
||||||
|
"Items": [
|
||||||
|
{
|
||||||
|
"PublishDate": "2026-03-02T16:14:37",
|
||||||
|
"Title": "Data KSEI terkait Kepemilikan Saham Perusahaan Tercatat Berdasarkan Tipe Investor per 27 Februari 2026",
|
||||||
|
"AnnouncementType": "",
|
||||||
|
"Code": " ",
|
||||||
|
"Attachments": [
|
||||||
|
{
|
||||||
|
"PDFFilename": "20260302_Pengumuman Bursa_32040089.pdf",
|
||||||
|
"FullSavePath": "https://www.idx.co.id/StaticData/NewsAndAnnouncement/ANNOUNCEMENTSTOCK/From_EREP/202603/43025665c8_fda8953269.pdf",
|
||||||
|
"IsAttachment": 0,
|
||||||
|
"OriginalFilename": "20260302_Pengumuman Bursa_32040089.pdf"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"PDFFilename": "20260302_Pengumuman Bursa_32040089_lamp1.pdf",
|
||||||
|
"FullSavePath": "https://www.idx.co.id/StaticData/NewsAndAnnouncement/ANNOUNCEMENTSTOCK/From_EREP/202603/db5b5a86e1_2e2b3d976a.pdf",
|
||||||
|
"IsAttachment": 1,
|
||||||
|
"OriginalFilename": "20260302_Pengumuman Bursa_32040089_lamp1.pdf"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"PdfPath": ""
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"ItemCount": 1,
|
||||||
|
"PageCount": 1
|
||||||
|
}"#;
|
||||||
|
|
||||||
|
let page = parse_announcement_page(raw).expect("investor-type fixture should parse");
|
||||||
|
let discovered = select_latest_ownership_reports(
|
||||||
|
&page,
|
||||||
|
"https://www.idx.co.id/primary/NewsAnnouncement/GetAllAnnouncement?keywords=kepemilikan%20saham%20perusahaan%20tercatat&pageNumber=1&pageSize=10&lang=id",
|
||||||
|
OwnershipReportFamily::InvestorTypeBreakdown,
|
||||||
|
)
|
||||||
|
.expect("should select investor-type reports");
|
||||||
|
|
||||||
|
assert_eq!(discovered.len(), 2);
|
||||||
|
assert_eq!(
|
||||||
|
discovered[0].original_filename.as_deref(),
|
||||||
|
Some("20260302_Pengumuman Bursa_32040089.pdf")
|
||||||
|
);
|
||||||
|
assert_eq!(
|
||||||
|
discovered[1].original_filename.as_deref(),
|
||||||
|
Some("20260302_Pengumuman Bursa_32040089_lamp1.pdf")
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn falls_back_to_pdf_path_when_attachments_are_missing() {
|
||||||
|
let raw = r#"{
|
||||||
|
"Items": [
|
||||||
|
{
|
||||||
|
"PublishDate": "2026-03-04T08:54:00",
|
||||||
|
"Title": "Pemegang Saham di atas 5% (KSEI)",
|
||||||
|
"AnnouncementType": "",
|
||||||
|
"Code": "Semua Emiten Saham",
|
||||||
|
"Attachments": [],
|
||||||
|
"PdfPath": "[{\"PDFFilename\":\"20260305_LKS_KSEI_000043_lamp1.pdf\",\"FullSavePath\":\"https://www.idx.co.id/StaticData/NewsAndAnnouncement/ANNOUNCEMENTSTOCK/From_EREP/202603/ec835d7451_c30c886a1a.pdf\",\"IsAttachment\":\"1\",\"OriginalFilename\":\"20260305_LKS_KSEI_000043_lamp1.pdf\"}]"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"ItemCount": 1,
|
||||||
|
"PageCount": 1
|
||||||
|
}"#;
|
||||||
|
|
||||||
|
let page: AnnouncementPage = parse_announcement_page(raw).expect("fallback page parses");
|
||||||
|
let discovered = select_latest_ownership_reports(
|
||||||
|
&page,
|
||||||
|
"https://www.idx.co.id/primary/NewsAnnouncement/GetAllAnnouncement?keywords=pemegang%20saham%20di%20atas%205&pageNumber=1&pageSize=10&lang=id",
|
||||||
|
OwnershipReportFamily::AboveFivePercent,
|
||||||
|
)
|
||||||
|
.expect("fallback attachment should be parsed");
|
||||||
|
|
||||||
|
assert_eq!(discovered.len(), 1);
|
||||||
|
assert_eq!(
|
||||||
|
discovered[0].pdf_url,
|
||||||
|
"https://www.idx.co.id/StaticData/NewsAndAnnouncement/ANNOUNCEMENTSTOCK/From_EREP/202603/ec835d7451_c30c886a1a.pdf"
|
||||||
|
);
|
||||||
|
assert_eq!(discovered[0].code.as_deref(), Some("Semua Emiten Saham"));
|
||||||
|
assert!(discovered[0].is_attachment);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn rejects_html_instead_of_announcement_json() {
|
||||||
|
let err = parse_announcement_page("<!doctype html><html><body>blocked</body></html>")
|
||||||
|
.expect_err("html response must fail");
|
||||||
|
assert!(matches!(err, IdxError::Http(_)));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn accepts_pdf_header() {
|
||||||
|
validate_pdf_payload(b"%PDF-1.7\n1 0 obj\n").expect("pdf header should pass");
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn rejects_html_instead_of_pdf() {
|
||||||
|
let err = validate_pdf_payload(b"<!doctype html><html><body>blocked</body></html>")
|
||||||
|
.expect_err("html body must fail");
|
||||||
|
assert!(matches!(err, IdxError::Http(_)));
|
||||||
|
}
|
||||||
|
}
|
||||||
129
tests/cli.rs
129
tests/cli.rs
|
|
@ -1,5 +1,8 @@
|
||||||
use std::fs;
|
use std::fs;
|
||||||
|
use std::io::{Read, Write};
|
||||||
|
use std::net::TcpListener;
|
||||||
use std::path::{Path, PathBuf};
|
use std::path::{Path, PathBuf};
|
||||||
|
use std::thread;
|
||||||
|
|
||||||
use assert_cmd::Command;
|
use assert_cmd::Command;
|
||||||
use predicates::prelude::*;
|
use predicates::prelude::*;
|
||||||
|
|
@ -38,6 +41,30 @@ fn test_bin(name: &str) -> Command {
|
||||||
bin_with_root(&root)
|
bin_with_root(&root)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
fn spawn_single_response_server(content_type: &str, body: impl Into<Vec<u8>>) -> String {
|
||||||
|
let listener = TcpListener::bind("127.0.0.1:0").expect("bind local test server");
|
||||||
|
let addr = listener.local_addr().expect("local addr");
|
||||||
|
let content_type = content_type.to_string();
|
||||||
|
let body = body.into();
|
||||||
|
|
||||||
|
thread::spawn(move || {
|
||||||
|
let (mut stream, _) = listener.accept().expect("accept test connection");
|
||||||
|
let mut buf = [0u8; 2048];
|
||||||
|
let _ = stream.read(&mut buf);
|
||||||
|
|
||||||
|
let headers = format!(
|
||||||
|
"HTTP/1.1 200 OK\r\nContent-Type: {content_type}\r\nContent-Length: {}\r\nConnection: close\r\n\r\n",
|
||||||
|
body.len()
|
||||||
|
);
|
||||||
|
stream
|
||||||
|
.write_all(headers.as_bytes())
|
||||||
|
.expect("write response headers");
|
||||||
|
stream.write_all(&body).expect("write response body");
|
||||||
|
});
|
||||||
|
|
||||||
|
format!("http://{addr}")
|
||||||
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn help_works() {
|
fn help_works() {
|
||||||
test_bin("help").arg("--help").assert().success();
|
test_bin("help").arg("--help").assert().success();
|
||||||
|
|
@ -580,6 +607,108 @@ fn ownership_import_fetch_bing_reports_unsupported() {
|
||||||
));
|
));
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn ownership_discover_lists_fixture_candidates() {
|
||||||
|
let body = fs::read_to_string("tests/fixtures/idx_announcement_kepemilikan.json")
|
||||||
|
.expect("read ownership discovery fixture");
|
||||||
|
let json_url = spawn_single_response_server("application/json", body);
|
||||||
|
|
||||||
|
test_bin("ownership-discover")
|
||||||
|
.env("IDX_CURL_IMPERSONATE_BIN", "curl")
|
||||||
|
.env("IDX_OWNERSHIP_ANNOUNCEMENT_API_URL", &json_url)
|
||||||
|
.env(
|
||||||
|
"IDX_OWNERSHIP_ANNOUNCEMENT_PAGE_URL",
|
||||||
|
"http://127.0.0.1/pengumuman",
|
||||||
|
)
|
||||||
|
.args([
|
||||||
|
"ownership",
|
||||||
|
"discover",
|
||||||
|
"--family",
|
||||||
|
"above5",
|
||||||
|
"--limit",
|
||||||
|
"2",
|
||||||
|
])
|
||||||
|
.assert()
|
||||||
|
.success()
|
||||||
|
.stdout(predicate::str::contains("Above 5%"))
|
||||||
|
.stdout(predicate::str::contains(
|
||||||
|
"20260327_Semua Emiten Saham_Pengumuman Bursa_32055594.pdf",
|
||||||
|
))
|
||||||
|
.stdout(predicate::str::contains(
|
||||||
|
"20260327_Semua Emiten Saham_Pengumuman Bursa_32055594_lamp1.pdf",
|
||||||
|
));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn ownership_discover_supports_above1_family() {
|
||||||
|
let body = r#"{
|
||||||
|
"Items": [
|
||||||
|
{
|
||||||
|
"PublishDate": "2026-03-10T12:09:09",
|
||||||
|
"Title": "Pemegang Saham di atas 1% (KSEI)",
|
||||||
|
"AnnouncementType": "",
|
||||||
|
"Code": "Semua Emiten Saham",
|
||||||
|
"Attachments": [
|
||||||
|
{
|
||||||
|
"PDFFilename": "d67ebf37e6_10d4080288.pdf",
|
||||||
|
"FullSavePath": "https://www.idx.co.id/StaticData/NewsAndAnnouncement/ANNOUNCEMENTSTOCK/From_EREP/202603/d67ebf37e6_10d4080288.pdf",
|
||||||
|
"IsAttachment": 0,
|
||||||
|
"OriginalFilename": "20260310_Semua Emiten Saham_Pengumuman Bursa_32052554.pdf"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"PDFFilename": "b9b638e5a8_8928aca255.pdf",
|
||||||
|
"FullSavePath": "https://www.idx.co.id/StaticData/NewsAndAnnouncement/ANNOUNCEMENTSTOCK/From_EREP/202603/b9b638e5a8_8928aca255.pdf",
|
||||||
|
"IsAttachment": 1,
|
||||||
|
"OriginalFilename": "20260310_Semua Emiten Saham_Pengumuman Bursa_32052554_lamp1.pdf"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"PdfPath": ""
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"ItemCount": 1,
|
||||||
|
"PageSize": 10,
|
||||||
|
"PageNumber": 1,
|
||||||
|
"PageCount": 1
|
||||||
|
}"#;
|
||||||
|
let json_url = spawn_single_response_server("application/json", body.to_string());
|
||||||
|
|
||||||
|
test_bin("ownership-discover-above1")
|
||||||
|
.env("IDX_CURL_IMPERSONATE_BIN", "curl")
|
||||||
|
.env("IDX_OWNERSHIP_ANNOUNCEMENT_API_URL", &json_url)
|
||||||
|
.env(
|
||||||
|
"IDX_OWNERSHIP_ANNOUNCEMENT_PAGE_URL",
|
||||||
|
"http://127.0.0.1/pengumuman",
|
||||||
|
)
|
||||||
|
.args([
|
||||||
|
"ownership",
|
||||||
|
"discover",
|
||||||
|
"--family",
|
||||||
|
"above1",
|
||||||
|
"--limit",
|
||||||
|
"2",
|
||||||
|
])
|
||||||
|
.assert()
|
||||||
|
.success()
|
||||||
|
.stdout(predicate::str::contains("Above 1%"))
|
||||||
|
.stdout(predicate::str::contains(
|
||||||
|
"20260310_Semua Emiten Saham_Pengumuman Bursa_32052554_lamp1.pdf",
|
||||||
|
));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn ownership_import_url_rejects_html_response_before_pdf_parse() {
|
||||||
|
let html_url = spawn_single_response_server(
|
||||||
|
"text/html; charset=utf-8",
|
||||||
|
"<!doctype html><html><body>blocked</body></html>",
|
||||||
|
);
|
||||||
|
|
||||||
|
test_bin("ownership-import-url-html")
|
||||||
|
.args(["ownership", "import", "--url", &html_url])
|
||||||
|
.assert()
|
||||||
|
.failure()
|
||||||
|
.stderr(predicate::str::contains("returned HTML instead of a PDF"));
|
||||||
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn technical_serves_stale_cache_on_provider_failure_with_warning() {
|
fn technical_serves_stale_cache_on_provider_failure_with_warning() {
|
||||||
let root = test_env_dir("technical-stale");
|
let root = test_env_dir("technical-stale");
|
||||||
|
|
|
||||||
48
tests/fixtures/idx_announcement_kepemilikan.json
vendored
Normal file
48
tests/fixtures/idx_announcement_kepemilikan.json
vendored
Normal file
|
|
@ -0,0 +1,48 @@
|
||||||
|
{
|
||||||
|
"Items": [
|
||||||
|
{
|
||||||
|
"Id": "20260327163420-Peng-LKS-00043/BEI.PLP/03-2026_id-id",
|
||||||
|
"AnnouncementNo": "Peng-LKS-00043/BEI.PLP/03-2026",
|
||||||
|
"PublishDate": "2026-03-27T16:34:20",
|
||||||
|
"Title": "Pemegang Saham di atas 5% (KSEI)",
|
||||||
|
"AnnouncementType": "",
|
||||||
|
"Code": "Semua Emiten Saham",
|
||||||
|
"Attachments": [
|
||||||
|
{
|
||||||
|
"PDFFilename": "20260327_Semua Emiten Saham_Pengumuman Bursa_32055594.pdf",
|
||||||
|
"FullSavePath": "https://www.idx.co.id/StaticData/NewsAndAnnouncement/ANNOUNCEMENTSTOCK/From_EREP/202603/5d31bb6f49_announcement.pdf",
|
||||||
|
"IsAttachment": 0,
|
||||||
|
"OriginalFilename": "20260327_Semua Emiten Saham_Pengumuman Bursa_32055594.pdf"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"PDFFilename": "20260327_Semua Emiten Saham_Pengumuman Bursa_32055594_lamp1.pdf",
|
||||||
|
"FullSavePath": "https://www.idx.co.id/StaticData/NewsAndAnnouncement/ANNOUNCEMENTSTOCK/From_EREP/202603/4f5c4efc6f_bf70f249ac_lamp1.pdf",
|
||||||
|
"IsAttachment": 1,
|
||||||
|
"OriginalFilename": "20260327_Semua Emiten Saham_Pengumuman Bursa_32055594_lamp1.pdf"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"PdfPath": "[{\"PDFFilename\":\"20260327_Semua Emiten Saham_Pengumuman Bursa_32055594.pdf\",\"FullSavePath\":\"https://www.idx.co.id/StaticData/NewsAndAnnouncement/ANNOUNCEMENTSTOCK/From_EREP/202603/5d31bb6f49_announcement.pdf\",\"IsAttachment\":\"0\",\"OriginalFilename\":\"20260327_Semua Emiten Saham_Pengumuman Bursa_32055594.pdf\"},{\"PDFFilename\":\"20260327_Semua Emiten Saham_Pengumuman Bursa_32055594_lamp1.pdf\",\"FullSavePath\":\"https://www.idx.co.id/StaticData/NewsAndAnnouncement/ANNOUNCEMENTSTOCK/From_EREP/202603/4f5c4efc6f_bf70f249ac_lamp1.pdf\",\"IsAttachment\":\"1\",\"OriginalFilename\":\"20260327_Semua Emiten Saham_Pengumuman Bursa_32055594_lamp1.pdf\"}]"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"Id": "20260328114739-LK/28032026/0001/1_id-id",
|
||||||
|
"AnnouncementNo": "LK/28032026/0001/1",
|
||||||
|
"PublishDate": "2026-03-28T11:47:39",
|
||||||
|
"Title": "LAPORAN KEPEMILIKAN ATAU SETIAP PERUBAHAN KEPEMILIKAN SAHAM PERUSAHAAN TERBUKA",
|
||||||
|
"AnnouncementType": "LKS",
|
||||||
|
"Code": "CASH",
|
||||||
|
"Attachments": [
|
||||||
|
{
|
||||||
|
"PDFFilename": "LK-28032026-2468-00.pdf-0",
|
||||||
|
"FullSavePath": "https://www.idx.co.id/StaticData/NewsAndAnnouncement/ANNOUNCEMENTSTOCK/From_KSEI/LK-28032026-2468-00.pdf-0.pdf",
|
||||||
|
"IsAttachment": 1,
|
||||||
|
"OriginalFilename": "LK-28032026-2468-00.pdf"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"PdfPath": "[{\"PDFFilename\":\"LK-28032026-2468-00.pdf-0\",\"FullSavePath\":\"https://www.idx.co.id/StaticData/NewsAndAnnouncement/ANNOUNCEMENTSTOCK/From_KSEI/LK-28032026-2468-00.pdf-0.pdf\",\"IsAttachment\":1,\"OriginalFilename\":\"LK-28032026-2468-00.pdf\"}]"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"ItemCount": 2,
|
||||||
|
"PageSize": 10,
|
||||||
|
"PageNumber": 1,
|
||||||
|
"PageCount": 1
|
||||||
|
}
|
||||||
Loading…
Add table
Add a link
Reference in a new issue