mirror of
https://github.com/0xrsydn/idx-cli.git
synced 2026-11-07 01:35:36 +00:00
docs: news sources research: one publisher, Google News, greencloud test
- Why not a browser per user: a full engine per install and fetch, each user facing the challenge, IDX seeing thousands of scrapers; one scheduled publisher serving static files instead (diagram). Hosting left open (GitHub release, Cloudflare Pages/R2, a Worker). - The IDX challenge is JavaScript-based: curl fails from a second datacenter (greencloud-vps, Singapore) too, so IP is not what curl lacks. - Other sources measured: Google News RSS covers companies MSN misses (BREN 75/100 titles naming the ticker in 7 days), plain HTTP; Yahoo search answers 429. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01V3syn2fN2LNo5pvTSqc6eh
This commit is contained in:
hermes
parent
d2ad5c142b
commit
2d0bceade2
3 changed files with 77 additions and 25 deletions
|
|
@ -1,9 +1,9 @@
|
|||
# IDX News and Disclosures
|
||||
# News Sources Research
|
||||
|
||||
Research notes and a proposed design for adding IDX's own news and company
|
||||
disclosures next to MSN news. Nothing here ships yet; `idx stocks news` and the
|
||||
`get_company` news section use MSN only (see `docs/ARCHITECTURE.md`).
|
||||
Measured 2026-10-16.
|
||||
Research notes on news beyond MSN: IDX's own news and company disclosures,
|
||||
and other public sources. Nothing here ships yet; `idx stocks news` and the
|
||||
`get_company` news section use MSN only (see `docs/ARCHITECTURE.md`), and MSN
|
||||
is enough for now. Measured 2026-10-16 from the maintainer host unless noted.
|
||||
|
||||
## Why
|
||||
|
||||
|
|
@ -16,10 +16,14 @@ news.
|
|||
|
||||
## What blocks direct access
|
||||
|
||||
`www.idx.co.id` is behind a Cloudflare challenge. From the maintainer host
|
||||
(a datacenter VM), every page and API below returns `403` to plain curl and to
|
||||
`curl_chrome142` (curl-impersonate), the client ownership discovery uses.
|
||||
Untested from residential networks.
|
||||
`www.idx.co.id` is behind a Cloudflare JavaScript challenge ("Just a
|
||||
moment..."). Every page and API below returns `403` with that challenge to
|
||||
plain curl and to `curl_chrome142` (curl-impersonate, the client ownership
|
||||
discovery uses), from the maintainer host and from a second datacenter
|
||||
(greencloud-vps, Singapore, AS8888 xTom). A client that cannot run JavaScript
|
||||
fails the challenge whatever its IP, so a residential network is unlikely to
|
||||
help curl; whether a real browser passes as easily from other IPs is
|
||||
untested.
|
||||
|
||||
The Camofox stealth browser on the maintainer host (`127.0.0.1:9377`, a local
|
||||
service, not shipped) gets through:
|
||||
|
|
@ -71,10 +75,25 @@ Quirks:
|
|||
|
||||
| Option | Verdict |
|
||||
| --- | --- |
|
||||
| Users fetch IDX directly (curl-impersonate) | Blocked from datacenter IPs; may work from homes, but unreliable. At most a best-effort fallback. |
|
||||
| Ship a browser with idx | No: heavy, fragile, and out of place in a CLI and MCP server. |
|
||||
| Users fetch IDX directly (curl-impersonate) | Fails the JavaScript challenge; not viable. |
|
||||
| Each user runs a stealth browser (idx launches one per fetch) | No: a full browser engine per install and per fetch (hundreds of MB, seconds to start), every user facing the challenge on their own, and IDX seeing thousands of scrapers. |
|
||||
| Hosted API that fetches IDX on request (e.g. a Cloudflare Worker) | Gets the same challenge; needs a browser anyway. |
|
||||
| **Maintainer publisher → static JSON → users** | **Recommended.** Same model as the ownership snapshot. |
|
||||
| **One publisher fetches on a schedule → static JSON → users** | **Recommended.** Same model as the ownership snapshot. |
|
||||
|
||||
One browser, in one place, fetches for everyone; users download small static
|
||||
files:
|
||||
|
||||
```text
|
||||
one place, on a schedule every user
|
||||
┌──────────────────────────────────────┐ ┌─────────────────────────┐
|
||||
│ publisher │ │ idx / idx-mcp │
|
||||
│ stealth browser, ~2 requests/hour │ ───> │ plain HTTPS GET of a │
|
||||
│ -> normalize -> JSON per ticker │ JSON │ few-KB JSON file, │
|
||||
│ -> upload to a static host │ │ cached; MSN fallback │
|
||||
└──────────────────────────────────────┘ └─────────────────────────┘
|
||||
```
|
||||
|
||||
The browser's cost does not grow with users, and IDX sees one polite client.
|
||||
|
||||
## Proposed design
|
||||
|
||||
|
|
@ -90,26 +109,58 @@ Quirks:
|
|||
4. Keep a rolling window, e.g. 90 days of disclosures (~6,000 items, a few MB
|
||||
of JSON; per-ticker shards if size matters) and the latest few hundred news
|
||||
items.
|
||||
5. Publish to a dedicated GitHub release, `idx-news-current`, as a manifest
|
||||
(stable URL, the commit point) plus content-addressed JSON, as
|
||||
`docs/OWNERSHIP_PUBLISH.md` does. Run hourly on trading days; a freshness
|
||||
check alerts when the manifest is stale.
|
||||
5. Publish to a static host (to be chosen, see below). Run hourly on trading
|
||||
days; a freshness check alerts when the feed is stale.
|
||||
|
||||
**Hosting** (undecided; the project website may serve it later):
|
||||
|
||||
| Host | Fit |
|
||||
| --- | --- |
|
||||
| GitHub release asset, like `ownership-snapshot-current` | Proven here; suits one bundle (manifest + content-addressed JSON), less so many per-ticker files overwritten hourly. |
|
||||
| Cloudflare Pages or R2 (the website is on Pages) | A CDN, per-ticker files, cheap frequent overwrites, cache headers for freshness. |
|
||||
| A small API (a Worker over R2/KV) | Only for queries static files cannot answer, such as all of today's disclosures matching a keyword. |
|
||||
|
||||
**Consumer** (idx-core):
|
||||
|
||||
- A provider that fetches the manifest and JSON over plain HTTPS (no
|
||||
Cloudflare on GitHub release assets), cached like other providers.
|
||||
- A provider that fetches the JSON over plain HTTPS (no challenge on the
|
||||
static host), cached like other providers.
|
||||
- `stocks news <TICKER>` / `get_company` news: merge the ticker's IDX
|
||||
disclosures with MSN articles, each item marked with `source: idx | msn`;
|
||||
disclosures count as `relevance: company`.
|
||||
- Possibly `idx market news` (and an MCP tool) for exchange news.
|
||||
- If the feed is unreachable, fall back to MSN only, as today.
|
||||
|
||||
## Other news sources
|
||||
|
||||
| Source | Finding |
|
||||
| --- | --- |
|
||||
| MSN (shipped) | Per-stock entity feed, tagged by MSN instrument id. Strong for large caps (BBCA 10/10 company items); for ADRO, BREN and CUAN no company items even 50 deep. Telkom articles are tagged with its NYSE ADR (`TLK`), not `TLKM`. |
|
||||
| Google News RSS, `https://news.google.com/rss/search?q=<TICKER>+saham+when:7d&hl=id&gl=ID&ceid=ID:id` | Plain HTTP, no key, no browser; each user can fetch directly. Far better per-company coverage: in 7 days, titles naming the ticker: BREN 75/100, GOTO 75/100, CUAN 54/100, BBRI 37/83, TLKM 17/51, ADRO 14/55. Caught the OJK fine on DSSA, BREN and CUAN that MSN missed. Caveats: mixed source quality (real outlets beside low-quality sites, so an allowlist or ranking), links are Google redirect URLs, the feeds are offered for personal use, and some tickers are words ("CUAN" means profit), so match company names too, as `parse_news` does. |
|
||||
| Yahoo Finance search (`query1/2.finance.yahoo.com/v1/finance/search?q=<T>.JK&newsCount=10`) | `429` from the maintainer host on both hosts; not usable as is. |
|
||||
| Publisher RSS (Kontan, Bisnis, CNBC Indonesia, Katadata) | Not tested; whole-site feeds, so ticker filtering would be ours. |
|
||||
|
||||
If MSN falls short, Google News RSS is the cheapest upgrade: no
|
||||
infrastructure, and it fits the existing `relevance` labeling. IDX
|
||||
disclosures are the authoritative source but need the publisher above.
|
||||
|
||||
## Camofox notes
|
||||
|
||||
- It is a shared local service (`127.0.0.1:9377`), not something to ship; it
|
||||
has no per-tab proxy setting, so testing other egress IPs means a separate
|
||||
browser on that host.
|
||||
- JSON comes only through the accessibility snapshot. A raw-content or
|
||||
evaluate endpoint would make a publisher sturdier.
|
||||
- Use your own `userId`, close your own tabs, never `POST /stop`.
|
||||
|
||||
## Open questions
|
||||
|
||||
- IDX terms of use for redistributing disclosure metadata and links; links to
|
||||
IDX-hosted PDFs rather than copies keep the footprint small.
|
||||
- Publisher reliability: Camofox is a single local service; the challenge
|
||||
behavior can change. The freshness check and MSN fallback bound the impact.
|
||||
- IDX terms of use for automated access and redistributing disclosure
|
||||
metadata; IDX also sells official data feeds, the legitimate source if
|
||||
scraping is not allowed. Links to IDX-hosted PDFs rather than copies keep the
|
||||
footprint small.
|
||||
- Where the publisher runs (this host with Camofox, or a VPS with its own
|
||||
stealth browser), and where its files are hosted.
|
||||
- Publisher reliability: one browser process; the challenge can change. The
|
||||
freshness check and MSN fallback bound the impact.
|
||||
- Whether `GetAnnouncement` per issuer is needed for backfill beyond the
|
||||
rolling window.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue