docs: research IDX news and disclosures, propose a published feed

MSN has no company news for many tickers; IDX's disclosures cover every
issuer. idx.co.id is behind a Cloudflare challenge that plain curl and
curl-impersonate fail from the maintainer host, while the local Camofox
browser gets through. Record the working endpoints (per-issuer and
all-issuer disclosures, exchange news), their shapes and quirks, the
options, and a proposed design: a maintainer publisher writes static JSON
to a GitHub release, as the ownership snapshot does, and idx-core merges
it into news.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V3syn2fN2LNo5pvTSqc6eh

AGENTS.md

@@ -33,6 +33,7 @@ Docs:
 - `docs/OWNERSHIP_SYNC.md` / `docs/OWNERSHIP_PUBLISH.md`: snapshot consumer contract / maintainer publishing
 - `docs/SMOKE.md`: Nix checks, test layout, and the live drift check
 - `docs/DESIGN.md`: idx-core / idx-mcp design and parity gates
+- `docs/IDX_NEWS.md`: IDX news and disclosures research, proposed publisher design
 - `ROADMAP.md`: open work
 
 ## Rules

ROADMAP.md

@@ -11,6 +11,8 @@ history lives in git.
 - `screen presets` / `screen run <PRESET>`: built-in screener presets
   (graduate `screen` out of `stocks` only if these grow into a richer workflow)
 - Multi-period financial statements (today `financials` is single-period)
+- IDX company disclosures and exchange news in `news` / `get_company`, through
+  a maintainer-published feed (`docs/IDX_NEWS.md`)
 
 ## Workflow
 - `watchlist`: named symbol sets usable by other commands

docs/IDX_NEWS.md

@@ -0,0 +1,115 @@
+# IDX News and Disclosures
+
+Research notes and a proposed design for adding IDX's own news and company
+disclosures next to MSN news. Nothing here ships yet; `idx stocks news` and the
+`get_company` news section use MSN only (see `docs/ARCHITECTURE.md`).
+Measured 2026-10-16.
+
+## Why
+
+MSN's feed for a stock is often general market news. After relevance ranking,
+10 items for ADRO, BREN and CUAN held no company items even 50 deep, and BBRI
+one. IDX publishes what MSN lacks: every issuer's disclosures (keterbukaan
+informasi), such as monthly shareholder registers, financial statements,
+material transactions, and replies to IDX queries, plus the exchange's own
+news.
+
+## What blocks direct access
+
+`www.idx.co.id` is behind a Cloudflare challenge. From the maintainer host
+(a datacenter VM), every page and API below returns `403` to plain curl and to
+`curl_chrome142` (curl-impersonate), the client ownership discovery uses.
+Untested from residential networks.
+
+The Camofox stealth browser on the maintainer host (`127.0.0.1:9377`, a local
+service, not shipped) gets through:
+
+1. Open a tab on a page that passes the challenge. The ownership page
+   `/id/perusahaan-tercatat/data-kepemilikan-saham/` passed in ~12 s; the
+   disclosure page `/id/perusahaan-tercatat/keterbukaan-informasi/` stayed on
+   "Performing security verification".
+2. `POST /tabs/{id}/navigate` the same tab to an API URL; the clearance cookie
+   carries over.
+3. `GET /tabs/{id}/snapshot` returns one line, `- text: "<escaped JSON>"`:
+   `sed 's/^- text: //' | jq 'fromjson'`.
+4. Close the tab. Camofox has no fetch or evaluate endpoint, so JSON comes only
+   through the snapshot.
+
+## Endpoints
+
+All under `https://www.idx.co.id`, all JSON once past the challenge.
+
+| Endpoint | Returns |
+| --- | --- |
+| `/primary/ListedCompany/GetAnnouncement?kodeEmiten=ADRO&emitenType=*&indexFrom=0&pageSize=10&dateFrom=19010101&dateTo=20991231&lang=id&keyword=` | one issuer's disclosures, newest first (ADRO 188, BREN 177) |
+| `/primary/NewsAnnouncement/GetAllAnnouncement?keywords=&pageNumber=1&pageSize=100&dateFrom=20261016&dateTo=20261016&lang=id` | every issuer's disclosures in a date range (78 on 2026-10-16) |
+| `/primary/NewsAnnouncement/GetNewsSearch?pageNumber=1&pageSize=10&keyword=&lang=id` | exchange news behind `/id/berita/berita` and `/en/news/news` (11,390 items) |
+
+Shapes:
+
+- `GetAnnouncement`: `{ResultCount, Replies: [{pengumuman: {Kode_Emiten,
+  JudulPengumuman, PerihalPengumuman, TglPengumuman, NoPengumuman, ...},
+  attachments: [{FullSavePath, OriginalFilename, ...}]}]}`. `Kode_Emiten` is
+  space-padded (`"ADRO "`).
+- `GetAllAnnouncement`: `{ItemCount, PageCount, Items: [{Code, Title, Perihal,
+  PublishDate, AnnouncementNo, Attachments: [{FullSavePath, ...}]}]}`. Attachment
+  PDFs are under `/StaticData/NewsAndAnnouncement/...`.
+- `GetNewsSearch`: `{ItemCount, Items: [{Id, PublishedDate, Title, Summary,
+  Tags, IsHeadline, ImageUrl, Links: [{Href: "/api/api/publication/news/<Id>"}]}]}`.
+
+Quirks:
+
+- `GetAllAnnouncement` ignores `kodeEmiten` and returns every issuer; filter on
+  `Code`, or use `GetAnnouncement` per issuer.
+- `GetNewsSearch` returned the same `en-us` items for `lang=id` and `lang=en`.
+- Guessed paths (`/primary/NewsAnnouncement/GetNews`, `/primary/News/GetNews`)
+  return `503`, not `404`.
+- Exchange news is about IDX itself (meetings, events, regulation), not about
+  a given company; it suits a market-news section, not per-ticker news.
+
+## Options
+
+| Option | Verdict |
+| --- | --- |
+| Users fetch IDX directly (curl-impersonate) | Blocked from datacenter IPs; may work from homes, but unreliable. At most a best-effort fallback. |
+| Ship a browser with idx | No: heavy, fragile, and out of place in a CLI and MCP server. |
+| Hosted API that fetches IDX on request (e.g. a Cloudflare Worker) | Gets the same challenge; needs a browser anyway. |
+| **Maintainer publisher → static JSON → users** | **Recommended.** Same model as the ownership snapshot. |
+
+## Proposed design
+
+**Publisher** (maintainer host with Camofox, on a schedule):
+
+1. Warm a Camofox tab on a page that passes the challenge.
+2. Page `GetAllAnnouncement` for the window since the last run (100 per page;
+   a trading day is ~80) and `GetNewsSearch` for new exchange news.
+3. Normalize to a small schema: `{ticker, title, subject, published_at,
+   number, attachments: [url]}` for disclosures; `{id, title, summary,
+   published_at, tags, url}` for news. Trim padding, convert WIB timestamps to
+   RFC 3339 with `+07:00`.
+4. Keep a rolling window, e.g. 90 days of disclosures (~6,000 items, a few MB
+   of JSON; per-ticker shards if size matters) and the latest few hundred news
+   items.
+5. Publish to a dedicated GitHub release, `idx-news-current`, as a manifest
+   (stable URL, the commit point) plus content-addressed JSON, as
+   `docs/OWNERSHIP_PUBLISH.md` does. Run hourly on trading days; a freshness
+   check alerts when the manifest is stale.
+
+**Consumer** (idx-core):
+
+- A provider that fetches the manifest and JSON over plain HTTPS (no
+  Cloudflare on GitHub release assets), cached like other providers.
+- `stocks news <TICKER>` / `get_company` news: merge the ticker's IDX
+  disclosures with MSN articles, each item marked with `source: idx | msn`;
+  disclosures count as `relevance: company`.
+- Possibly `idx market news` (and an MCP tool) for exchange news.
+- If the feed is unreachable, fall back to MSN only, as today.
+
+## Open questions
+
+- IDX terms of use for redistributing disclosure metadata and links; links to
+  IDX-hosted PDFs rather than copies keep the footprint small.
+- Publisher reliability: Camofox is a single local service; the challenge
+  behavior can change. The freshness check and MSN fallback bound the impact.
+- Whether `GetAnnouncement` per issuer is needed for backfill beyond the
+  rolling window.