idx-cli/docs/IDX_NEWS.md

# IDX News and Disclosures

Research notes and a proposed design for adding IDX's own news and company
disclosures next to MSN news. Nothing here ships yet; `idx stocks news` and the
`get_company` news section use MSN only (see `docs/ARCHITECTURE.md`).
Measured 2026-10-16.

## Why

MSN's feed for a stock is often general market news. After relevance ranking,
10 items for ADRO, BREN and CUAN held no company items even 50 deep, and BBRI
one. IDX publishes what MSN lacks: every issuer's disclosures (keterbukaan
informasi), such as monthly shareholder registers, financial statements,
material transactions, and replies to IDX queries, plus the exchange's own
news.

## What blocks direct access

`www.idx.co.id` is behind a Cloudflare challenge. From the maintainer host
(a datacenter VM), every page and API below returns `403` to plain curl and to
`curl_chrome142` (curl-impersonate), the client ownership discovery uses.
Untested from residential networks.

The Camofox stealth browser on the maintainer host (`127.0.0.1:9377`, a local
service, not shipped) gets through:

1. Open a tab on a page that passes the challenge. The ownership page
   `/id/perusahaan-tercatat/data-kepemilikan-saham/` passed in ~12 s; the
   disclosure page `/id/perusahaan-tercatat/keterbukaan-informasi/` stayed on
   "Performing security verification".
2. `POST /tabs/{id}/navigate` the same tab to an API URL; the clearance cookie
   carries over.
3. `GET /tabs/{id}/snapshot` returns one line, `- text: "<escaped JSON>"`:
   `sed 's/^- text: //' | jq 'fromjson'`.
4. Close the tab. Camofox has no fetch or evaluate endpoint, so JSON comes only
   through the snapshot.

## Endpoints

All under `https://www.idx.co.id`, all JSON once past the challenge.

| Endpoint | Returns |
| --- | --- |
| `/primary/ListedCompany/GetAnnouncement?kodeEmiten=ADRO&emitenType=*&indexFrom=0&pageSize=10&dateFrom=19010101&dateTo=20991231&lang=id&keyword=` | one issuer's disclosures, newest first (ADRO 188, BREN 177) |
| `/primary/NewsAnnouncement/GetAllAnnouncement?keywords=&pageNumber=1&pageSize=100&dateFrom=20261016&dateTo=20261016&lang=id` | every issuer's disclosures in a date range (78 on 2026-10-16) |
| `/primary/NewsAnnouncement/GetNewsSearch?pageNumber=1&pageSize=10&keyword=&lang=id` | exchange news behind `/id/berita/berita` and `/en/news/news` (11,390 items) |

Shapes:

- `GetAnnouncement`: `{ResultCount, Replies: [{pengumuman: {Kode_Emiten,
  JudulPengumuman, PerihalPengumuman, TglPengumuman, NoPengumuman, ...},
  attachments: [{FullSavePath, OriginalFilename, ...}]}]}`. `Kode_Emiten` is
  space-padded (`"ADRO "`).
- `GetAllAnnouncement`: `{ItemCount, PageCount, Items: [{Code, Title, Perihal,
  PublishDate, AnnouncementNo, Attachments: [{FullSavePath, ...}]}]}`. Attachment
  PDFs are under `/StaticData/NewsAndAnnouncement/...`.
- `GetNewsSearch`: `{ItemCount, Items: [{Id, PublishedDate, Title, Summary,
  Tags, IsHeadline, ImageUrl, Links: [{Href: "/api/api/publication/news/<Id>"}]}]}`.

Quirks:

- `GetAllAnnouncement` ignores `kodeEmiten` and returns every issuer; filter on
  `Code`, or use `GetAnnouncement` per issuer.
- `GetNewsSearch` returned the same `en-us` items for `lang=id` and `lang=en`.
- Guessed paths (`/primary/NewsAnnouncement/GetNews`, `/primary/News/GetNews`)
  return `503`, not `404`.
- Exchange news is about IDX itself (meetings, events, regulation), not about
  a given company; it suits a market-news section, not per-ticker news.

## Options

| Option | Verdict |
| --- | --- |
| Users fetch IDX directly (curl-impersonate) | Blocked from datacenter IPs; may work from homes, but unreliable. At most a best-effort fallback. |
| Ship a browser with idx | No: heavy, fragile, and out of place in a CLI and MCP server. |
| Hosted API that fetches IDX on request (e.g. a Cloudflare Worker) | Gets the same challenge; needs a browser anyway. |
| **Maintainer publisher → static JSON → users** | **Recommended.** Same model as the ownership snapshot. |

## Proposed design

**Publisher** (maintainer host with Camofox, on a schedule):

1. Warm a Camofox tab on a page that passes the challenge.
2. Page `GetAllAnnouncement` for the window since the last run (100 per page;
   a trading day is ~80) and `GetNewsSearch` for new exchange news.
3. Normalize to a small schema: `{ticker, title, subject, published_at,
   number, attachments: [url]}` for disclosures; `{id, title, summary,
   published_at, tags, url}` for news. Trim padding, convert WIB timestamps to
   RFC 3339 with `+07:00`.
4. Keep a rolling window, e.g. 90 days of disclosures (~6,000 items, a few MB
   of JSON; per-ticker shards if size matters) and the latest few hundred news
   items.
5. Publish to a dedicated GitHub release, `idx-news-current`, as a manifest
   (stable URL, the commit point) plus content-addressed JSON, as
   `docs/OWNERSHIP_PUBLISH.md` does. Run hourly on trading days; a freshness
   check alerts when the manifest is stale.

**Consumer** (idx-core):

- A provider that fetches the manifest and JSON over plain HTTPS (no
  Cloudflare on GitHub release assets), cached like other providers.
- `stocks news <TICKER>` / `get_company` news: merge the ticker's IDX
  disclosures with MSN articles, each item marked with `source: idx | msn`;
  disclosures count as `relevance: company`.
- Possibly `idx market news` (and an MCP tool) for exchange news.
- If the feed is unreachable, fall back to MSN only, as today.

## Open questions

- IDX terms of use for redistributing disclosure metadata and links; links to
  IDX-hosted PDFs rather than copies keep the footprint small.
- Publisher reliability: Camofox is a single local service; the challenge
  behavior can change. The freshness check and MSN fallback bound the impact.
- Whether `GetAnnouncement` per issuer is needed for backfill beyond the
  rolling window.