When Cloudflare answers idx.co.id with a challenge, curl fails with only
`curl: (22)`, which exits 4 like any other HTTP failure, and every profile
is tried twice. A challenge needs a browser, so no profile can solve it. The
source is monthly, so the publisher only needs one run per month to get
through: treat a challenge as "retry tomorrow" and alert on stale data.
- Add IdxError::CloudflareChallenge (CLOUDFLARECHALLENGE, exit 6), detected
from the cf-mitigated header. curl reports it on stderr through --write-out,
because --fail drops the response.
- A URL with a challenge gets one round of profiles; other failures keep two.
This also covers `stocks flow --source idx`, which now exits 6 on a challenge.
- -v logs which profile fetched each URL.
- Discovery keeps error types, so a challenge survives an announcement miss.
A source answering with something unparseable still is not called a challenge.
- `ownership discover --require-data-page` fails when the data page cannot be
read. Default discover is unchanged: it warns and lists older announcements.
- The publisher and builder use that flag, so a failed data page can no longer
end in a false "up-to-date". A challenge ends in `RESULT: blocked stage=<stage>`,
exit 6; every other failure stays `RESULT: FAILED`.
- The systemd example treats exit 6 as success and runs the freshness check, so
the unit fails on FAILED or on stale data.
- live-smoke reports a challenge as CHALLENGED, a failure and never a pass.
Golden change: completions-bash.golden gains `--require-data-page` in the
discover options, one word on one line. It is the direct result of the new flag;
no other golden file changes.
Live check on 2026-11-05 against release/v0.5.0: 19 passed, 2 failed. Both
failures are real challenges (flow-idx, discover), each exit 6 with empty stdout.
The import, releases and ticker cases did not run, because discover did not pass.
Refs #83
`idx stocks flow` now reads IPOT by default, which needs nothing installed:
per session, the all-investor broker summary gives the value traded and
the foreign one gives foreign buying (its footer), the foreign net and
selling (buying less the net), across all boards, through the broker
summary cache. IPOT's foreign net matched IDX's estimate within 0.3 B on
BBCA 2026-10-12 to 16.
`--source idx` (and `source` on get_foreign_flow) keeps IDX's trading
summary: exact share counts, close and the non-regular share, behind
curl-impersonate and Cloudflare.
- ForeignFlowDay gains foreign_buy_value, foreign_sell_value and
foreign_share_bps for both sources (estimated at the session average
for IDX); ForeignFlow gains source, the value totals and rounded.
- The table lays out each source's figures; the live smoke runs both.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V3syn2fN2LNo5pvTSqc6eh
`idx stocks flow SYMBOL` shows what foreign investors bought and sold of a
stock, session by session, from IDX's own daily trading summary: foreign
buy, sell and net volume, net value at the session's average price, and
the share of value traded outside the regular board, with range totals.
Latest 10 sessions by default (--days, at most 60) or --from/--to. The MCP
server exposes it as get_foreign_flow.
- api/idx: pure parser for GetStockSummary (rows dated another session
are dropped) and a provider over curl-impersonate's profile fallback
with a 30 s limit; Cloudflare passes only some fingerprints.
- flow: each session is cached whole by date, so other stocks' flows for
the same sessions need no request. Weekends are never requested.
- Idx::idx_code is shared by broker and flow for the .JK-only check.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V3syn2fN2LNo5pvTSqc6eh
Phase 0 of docs/DESIGN.md, before any refactor code:
- tests/golden.rs runs every case in tests/golden/cases.toml (126
outputs across help, completions, config, cache, all stocks commands
on MSN and Yahoo mocks, error paths, and ownership queries on a seeded
fixture DB) and compares exit code, stdout, and stderr byte for byte.
Regenerate only with UPDATE_GOLDEN=1 for a reviewed output change.
- scripts/live-smoke.sh --compare <old-idx> runs each live case with
both binaries and fails on differing exit codes, added/missing/retyped
JSON keys, or changed values for slow-moving data.
- Exclude golden files from the whitespace/EOF pre-commit fixers, which
would otherwise rewrite pinned help output.
Both gates were fault-injected: a one-word table change fails exactly
the three history goldens, and a binary dropping profile.long_name fails
only the profile live comparison.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The 43-case mock smoke duplicated tests/cli.rs, which already runs the
real binary against fixture providers. Replace the flake's smoke-mock
check with a small packaged-cli check (version JSON, bash completions,
one mock quote), and cut scripts/live-smoke.sh from 672 to ~170 lines:
it now only runs shipped commands against live MSN/Yahoo/IDX and
validates output shape. Rewrite docs/SMOKE.md around the Nix gate and
the drift check.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Run deterministic installer and publisher checks in Nix, align dev and release dependency artifacts, narrow derivation inputs, and use content-aware CI cache keys. Replace duplicate Cargo gates with a Nix pre-push gate; validate JSON streams and cache data in representative packaged-binary smoke cases.
Clear the smoke cache before each cache-group warm case so the following offline and stale-cache checks always start from a fresh baseline instead of earlier group state.
Document the runner behavior in the smoke guide to make the cache-group sequencing explicit.
