fix(ownership): report a Cloudflare challenge as its own error and result

When Cloudflare answers idx.co.id with a challenge, curl fails with only
`curl: (22)`, which exits 4 like any other HTTP failure, and every profile
is tried twice. A challenge needs a browser, so no profile can solve it. The
source is monthly, so the publisher only needs one run per month to get
through: treat a challenge as "retry tomorrow" and alert on stale data.

- Add IdxError::CloudflareChallenge (CLOUDFLARECHALLENGE, exit 6), detected
  from the cf-mitigated header. curl reports it on stderr through --write-out,
  because --fail drops the response.
- A URL with a challenge gets one round of profiles; other failures keep two.
  This also covers `stocks flow --source idx`, which now exits 6 on a challenge.
- -v logs which profile fetched each URL.
- Discovery keeps error types, so a challenge survives an announcement miss.
  A source answering with something unparseable still is not called a challenge.
- `ownership discover --require-data-page` fails when the data page cannot be
  read. Default discover is unchanged: it warns and lists older announcements.
- The publisher and builder use that flag, so a failed data page can no longer
  end in a false "up-to-date". A challenge ends in `RESULT: blocked stage=<stage>`,
  exit 6; every other failure stays `RESULT: FAILED`.
- The systemd example treats exit 6 as success and runs the freshness check, so
  the unit fails on FAILED or on stale data.
- live-smoke reports a challenge as CHALLENGED, a failure and never a pass.

Golden change: completions-bash.golden gains `--require-data-page` in the
discover options, one word on one line. It is the direct result of the new flag;
no other golden file changes.

Live check on 2026-11-05 against release/v0.5.0: 19 passed, 2 failed. Both
failures are real challenges (flow-idx, discover), each exit 6 with empty stdout.
The import, releases and ticker cases did not run, because discover did not pass.

Refs #83
This commit is contained in:
hermes 2026-11-05 08:54:58 +00:00
commit b236c7f9d9
23 changed files with 797 additions and 54 deletions

View file

@ -48,8 +48,10 @@ check needs no token.
## What It Does
1. `idx -o json ownership discover --family above1 --limit 50` (newest supported
report first; XLSX from the Data Kepemilikan Saham page since June 2026)
1. `idx -v -o json ownership discover --family above1 --limit 50 --require-data-page`
(newest supported report first; XLSX from the Data Kepemilikan Saham page since
June 2026, so a run that cannot read that page stops rather than miss the newest
report). `-v` logs which curl-impersonate profile fetched each URL.
2. verifies the discovered report is the supported import path
3. imports it plus `--history <n>` earlier months, oldest first, into an
isolated temp DB
@ -91,12 +93,24 @@ The publisher is idempotent and prints exactly one final line:
| --- | --- | --- |
| `RESULT: published <as-of>` | 0 | a newer snapshot, or larger history, was uploaded |
| `RESULT: up-to-date <as-of>` | 0 | nothing to upload |
| `RESULT: blocked stage=<stage>` | 6 | IDX answered with a Cloudflare challenge in `discover` or `build-snapshot`; nothing was uploaded |
| `RESULT: FAILED stage=<stage> (exit N)` | N | failed in `arguments`, `build`, `preflight`, `discover`, `build-snapshot`, `stage-output`, `upload`, or `publish` |
`idx-ownership-freshness` is the independent alarm: it exits 1 when the published
`latest_as_of_date` is older than `--max-age-days` (default 40). The source lands
2-3 days after month end, so 40 days only trips when publishing has stopped.
`blocked` means retry on the next run, not that something broke. The source is
monthly, so one run getting through per month is enough, and stale data is what
should alert:
| Result | Snapshot | Action |
| --- | --- | --- |
| `published` or `up-to-date` | fresh | none |
| `blocked` | fresh | none; the next daily run retries |
| `blocked` | stale | alert: retry, or import the file by hand |
| `FAILED` | either | alert at once |
## History Coverage
`--history <n>` (maximum 1000) requests the latest report plus up to `n` earlier
@ -135,16 +149,18 @@ on a self-hosted machine that IDX accepts:
- `contrib/systemd/idx-ownership-snapshot-publish.service` and `.timer` run the
packaged publisher daily at 06:00 UTC with `GH_TOKEN` from
`/etc/idx-ownership-snapshot.env`. Wire `OnFailure=` to a notification.
`/etc/idx-ownership-snapshot.env`. The service treats exit 6 (`blocked`) as success
and then runs the freshness check, so the unit fails on `FAILED` or on stale data.
An unreadable manifest also fails it. Wire `OnFailure=` to a notification.
- On NixOS, add this repo as a flake input and use
`inputs.idx-cli.packages.${system}.ownership-publisher`; `nix flake update idx-cli`
picks up fixes.
`.github/workflows/publish-ownership-snapshot.yml` is a manual
(`workflow_dispatch`) fallback that runs the same packaged publisher with inputs
passed as environment variables.
passed as environment variables. There a `blocked` run shows as a failed job.
Observed XLSX upload times (HTTP Last-Modified) were the 2nd-3rd of each month
around 02:00-04:00 UTC; a daily idempotent run makes the exact day irrelevant.
If the monthly source has not appeared yet, the run should fail loudly and retry
later rather than publish stale data.
If the monthly source has not appeared yet, the run should not publish stale data:
it ends `up-to-date` until the new report is listed, and a later daily run picks it up.