mirror of
https://github.com/0xrsydn/idx-cli.git
synced 2026-11-07 09:45:36 +00:00
feat(ownership): discover above-1% XLSX on the IDX data page
IDX stopped posting the monthly above-1% list as an announcement after the 2026-05-29 report; June/July/August are only on the Data Kepemilikan Saham page. `ownership discover` could not see them, and IDX's Cloudflare now rejects curl_chrome142 on that page and on the /Media/ XLSX files. - remote: parse the page's Nuxt payload (Description/Prospectus pairs, / escapes), classify above-1% XLSX as supported and above-5% / investor-type workbooks as unsupported, derive as_of_date from the Indonesian description. Keep all above-1% entries, only the newest of the other series. Merge with the announcement results. - discover JSON gains `format` (xlsx|pdf) and `as_of_date`; page entries use the as-of date as publish_date (the page shows none). - import --url accepts .xlsx and downloads via the XLSX validator. - curl_impersonate: run_owned_with_fallback tries several browser profiles until the body validates (JSON / page payload / %PDF / PK); IDX_CURL_IMPERSONATE_BIN still pins one. - Tests: page fixture parsing, date parsing, payload validation, CLI discover via a local page server; existing discover tests now point IDX_OWNERSHIP_DATA_PAGE_URL at a closed port so no test hits IDX. - Docs: ARCHITECTURE discovery section, AGENTS ownership inputs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017a5Cv5Z1NzdYKgKnTiAF4E
This commit is contained in:
parent
06d0a7a2ae
commit
29e416109c
7 changed files with 592 additions and 27 deletions
|
|
@ -67,6 +67,22 @@ Unlike the `stocks` module (live-fetch), ownership is **import-then-query**:
|
|||
- **KSEI archive** — monthly ZIP/TXT balance-position matrix, used as a local fallback/backstop import path
|
||||
- **Bing Finance** — global institutional ownership (REST API, quarterly)
|
||||
|
||||
### Discovery
|
||||
`idx ownership discover` merges two IDX sources:
|
||||
- the announcement API (`GetAllAnnouncement`) for the legacy PDF announcements
|
||||
- the Data Kepemilikan Saham page (`/id/perusahaan-tercatat/data-kepemilikan-saham/`),
|
||||
whose server-rendered Nuxt payload lists every XLSX with a description such as
|
||||
`Pemegang Saham di Atas 1% per 31 Agustus 2026`; the as-of date comes from that text
|
||||
|
||||
Only the above-1% XLSX is `supported`; the daily above-5% and investor-type workbooks
|
||||
are listed as `unsupported`. Both URLs can be overridden for tests with
|
||||
`IDX_OWNERSHIP_ANNOUNCEMENT_API_URL` and `IDX_OWNERSHIP_DATA_PAGE_URL`.
|
||||
|
||||
IDX sits behind Cloudflare, which accepts some curl-impersonate TLS profiles and
|
||||
rejects others per path (for example `curl_chrome142` gets 403 on the data page).
|
||||
IDX fetches therefore try several profiles until the response validates (JSON, page
|
||||
payload, `%PDF`, or `PK` zip magic). `IDX_CURL_IMPERSONATE_BIN` pins a single profile.
|
||||
|
||||
### Parser Pipeline
|
||||
```
|
||||
KSEI XLSX → zip + quick-xml (sharedStrings + sheet1) → exact 12-column header check
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue