From d2ad5c142b97220c1bde8099bf3311cddb02dcfa Mon Sep 17 00:00:00 2001 From: hermes Date: Fri, 16 Oct 2026 08:45:34 +0000 Subject: [PATCH] docs: research IDX news and disclosures, propose a published feed MSN has no company news for many tickers; IDX's disclosures cover every issuer. idx.co.id is behind a Cloudflare challenge that plain curl and curl-impersonate fail from the maintainer host, while the local Camofox browser gets through. Record the working endpoints (per-issuer and all-issuer disclosures, exchange news), their shapes and quirks, the options, and a proposed design: a maintainer publisher writes static JSON to a GitHub release, as the ownership snapshot does, and idx-core merges it into news. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01V3syn2fN2LNo5pvTSqc6eh --- AGENTS.md | 1 + ROADMAP.md | 2 + docs/IDX_NEWS.md | 115 +++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 118 insertions(+) create mode 100644 docs/IDX_NEWS.md diff --git a/AGENTS.md b/AGENTS.md index 6eb9efa..ed234e9 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -33,6 +33,7 @@ Docs: - `docs/OWNERSHIP_SYNC.md` / `docs/OWNERSHIP_PUBLISH.md`: snapshot consumer contract / maintainer publishing - `docs/SMOKE.md`: Nix checks, test layout, and the live drift check - `docs/DESIGN.md`: idx-core / idx-mcp design and parity gates +- `docs/IDX_NEWS.md`: IDX news and disclosures research, proposed publisher design - `ROADMAP.md`: open work ## Rules diff --git a/ROADMAP.md b/ROADMAP.md index f4d7f89..51672e6 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -11,6 +11,8 @@ history lives in git. - `screen presets` / `screen run `: built-in screener presets (graduate `screen` out of `stocks` only if these grow into a richer workflow) - Multi-period financial statements (today `financials` is single-period) +- IDX company disclosures and exchange news in `news` / `get_company`, through + a maintainer-published feed (`docs/IDX_NEWS.md`) ## Workflow - `watchlist`: named symbol sets usable by other commands diff --git a/docs/IDX_NEWS.md b/docs/IDX_NEWS.md new file mode 100644 index 0000000..9894fbb --- /dev/null +++ b/docs/IDX_NEWS.md @@ -0,0 +1,115 @@ +# IDX News and Disclosures + +Research notes and a proposed design for adding IDX's own news and company +disclosures next to MSN news. Nothing here ships yet; `idx stocks news` and the +`get_company` news section use MSN only (see `docs/ARCHITECTURE.md`). +Measured 2026-10-16. + +## Why + +MSN's feed for a stock is often general market news. After relevance ranking, +10 items for ADRO, BREN and CUAN held no company items even 50 deep, and BBRI +one. IDX publishes what MSN lacks: every issuer's disclosures (keterbukaan +informasi), such as monthly shareholder registers, financial statements, +material transactions, and replies to IDX queries, plus the exchange's own +news. + +## What blocks direct access + +`www.idx.co.id` is behind a Cloudflare challenge. From the maintainer host +(a datacenter VM), every page and API below returns `403` to plain curl and to +`curl_chrome142` (curl-impersonate), the client ownership discovery uses. +Untested from residential networks. + +The Camofox stealth browser on the maintainer host (`127.0.0.1:9377`, a local +service, not shipped) gets through: + +1. Open a tab on a page that passes the challenge. The ownership page + `/id/perusahaan-tercatat/data-kepemilikan-saham/` passed in ~12 s; the + disclosure page `/id/perusahaan-tercatat/keterbukaan-informasi/` stayed on + "Performing security verification". +2. `POST /tabs/{id}/navigate` the same tab to an API URL; the clearance cookie + carries over. +3. `GET /tabs/{id}/snapshot` returns one line, `- text: ""`: + `sed 's/^- text: //' | jq 'fromjson'`. +4. Close the tab. Camofox has no fetch or evaluate endpoint, so JSON comes only + through the snapshot. + +## Endpoints + +All under `https://www.idx.co.id`, all JSON once past the challenge. + +| Endpoint | Returns | +| --- | --- | +| `/primary/ListedCompany/GetAnnouncement?kodeEmiten=ADRO&emitenType=*&indexFrom=0&pageSize=10&dateFrom=19010101&dateTo=20991231&lang=id&keyword=` | one issuer's disclosures, newest first (ADRO 188, BREN 177) | +| `/primary/NewsAnnouncement/GetAllAnnouncement?keywords=&pageNumber=1&pageSize=100&dateFrom=20261016&dateTo=20261016&lang=id` | every issuer's disclosures in a date range (78 on 2026-10-16) | +| `/primary/NewsAnnouncement/GetNewsSearch?pageNumber=1&pageSize=10&keyword=&lang=id` | exchange news behind `/id/berita/berita` and `/en/news/news` (11,390 items) | + +Shapes: + +- `GetAnnouncement`: `{ResultCount, Replies: [{pengumuman: {Kode_Emiten, + JudulPengumuman, PerihalPengumuman, TglPengumuman, NoPengumuman, ...}, + attachments: [{FullSavePath, OriginalFilename, ...}]}]}`. `Kode_Emiten` is + space-padded (`"ADRO "`). +- `GetAllAnnouncement`: `{ItemCount, PageCount, Items: [{Code, Title, Perihal, + PublishDate, AnnouncementNo, Attachments: [{FullSavePath, ...}]}]}`. Attachment + PDFs are under `/StaticData/NewsAndAnnouncement/...`. +- `GetNewsSearch`: `{ItemCount, Items: [{Id, PublishedDate, Title, Summary, + Tags, IsHeadline, ImageUrl, Links: [{Href: "/api/api/publication/news/"}]}]}`. + +Quirks: + +- `GetAllAnnouncement` ignores `kodeEmiten` and returns every issuer; filter on + `Code`, or use `GetAnnouncement` per issuer. +- `GetNewsSearch` returned the same `en-us` items for `lang=id` and `lang=en`. +- Guessed paths (`/primary/NewsAnnouncement/GetNews`, `/primary/News/GetNews`) + return `503`, not `404`. +- Exchange news is about IDX itself (meetings, events, regulation), not about + a given company; it suits a market-news section, not per-ticker news. + +## Options + +| Option | Verdict | +| --- | --- | +| Users fetch IDX directly (curl-impersonate) | Blocked from datacenter IPs; may work from homes, but unreliable. At most a best-effort fallback. | +| Ship a browser with idx | No: heavy, fragile, and out of place in a CLI and MCP server. | +| Hosted API that fetches IDX on request (e.g. a Cloudflare Worker) | Gets the same challenge; needs a browser anyway. | +| **Maintainer publisher → static JSON → users** | **Recommended.** Same model as the ownership snapshot. | + +## Proposed design + +**Publisher** (maintainer host with Camofox, on a schedule): + +1. Warm a Camofox tab on a page that passes the challenge. +2. Page `GetAllAnnouncement` for the window since the last run (100 per page; + a trading day is ~80) and `GetNewsSearch` for new exchange news. +3. Normalize to a small schema: `{ticker, title, subject, published_at, + number, attachments: [url]}` for disclosures; `{id, title, summary, + published_at, tags, url}` for news. Trim padding, convert WIB timestamps to + RFC 3339 with `+07:00`. +4. Keep a rolling window, e.g. 90 days of disclosures (~6,000 items, a few MB + of JSON; per-ticker shards if size matters) and the latest few hundred news + items. +5. Publish to a dedicated GitHub release, `idx-news-current`, as a manifest + (stable URL, the commit point) plus content-addressed JSON, as + `docs/OWNERSHIP_PUBLISH.md` does. Run hourly on trading days; a freshness + check alerts when the manifest is stale. + +**Consumer** (idx-core): + +- A provider that fetches the manifest and JSON over plain HTTPS (no + Cloudflare on GitHub release assets), cached like other providers. +- `stocks news ` / `get_company` news: merge the ticker's IDX + disclosures with MSN articles, each item marked with `source: idx | msn`; + disclosures count as `relevance: company`. +- Possibly `idx market news` (and an MCP tool) for exchange news. +- If the feed is unreachable, fall back to MSN only, as today. + +## Open questions + +- IDX terms of use for redistributing disclosure metadata and links; links to + IDX-hosted PDFs rather than copies keep the footprint small. +- Publisher reliability: Camofox is a single local service; the challenge + behavior can change. The freshness check and MSN fallback bound the impact. +- Whether `GetAnnouncement` per issuer is needed for backfill beyond the + rolling window.