mirror of
https://github.com/0xrsydn/idx-cli.git
synced 2026-11-07 09:45:36 +00:00
fix(ownership): report a Cloudflare challenge as its own error and result
When Cloudflare answers idx.co.id with a challenge, curl fails with only `curl: (22)`, which exits 4 like any other HTTP failure, and every profile is tried twice. A challenge needs a browser, so no profile can solve it. The source is monthly, so the publisher only needs one run per month to get through: treat a challenge as "retry tomorrow" and alert on stale data. - Add IdxError::CloudflareChallenge (CLOUDFLARECHALLENGE, exit 6), detected from the cf-mitigated header. curl reports it on stderr through --write-out, because --fail drops the response. - A URL with a challenge gets one round of profiles; other failures keep two. This also covers `stocks flow --source idx`, which now exits 6 on a challenge. - -v logs which profile fetched each URL. - Discovery keeps error types, so a challenge survives an announcement miss. A source answering with something unparseable still is not called a challenge. - `ownership discover --require-data-page` fails when the data page cannot be read. Default discover is unchanged: it warns and lists older announcements. - The publisher and builder use that flag, so a failed data page can no longer end in a false "up-to-date". A challenge ends in `RESULT: blocked stage=<stage>`, exit 6; every other failure stays `RESULT: FAILED`. - The systemd example treats exit 6 as success and runs the freshness check, so the unit fails on FAILED or on stale data. - live-smoke reports a challenge as CHALLENGED, a failure and never a pass. Golden change: completions-bash.golden gains `--require-data-page` in the discover options, one word on one line. It is the direct result of the new flag; no other golden file changes. Live check on 2026-11-05 against release/v0.5.0: 19 passed, 2 failed. Both failures are real challenges (flow-idx, discover), each exit 6 with empty stdout. The import, releases and ticker cases did not run, because discover did not pass. Refs #83
This commit is contained in:
parent
50d9f92550
commit
b236c7f9d9
23 changed files with 797 additions and 54 deletions
|
|
@ -48,8 +48,10 @@ check needs no token.
|
|||
|
||||
## What It Does
|
||||
|
||||
1. `idx -o json ownership discover --family above1 --limit 50` (newest supported
|
||||
report first; XLSX from the Data Kepemilikan Saham page since June 2026)
|
||||
1. `idx -v -o json ownership discover --family above1 --limit 50 --require-data-page`
|
||||
(newest supported report first; XLSX from the Data Kepemilikan Saham page since
|
||||
June 2026, so a run that cannot read that page stops rather than miss the newest
|
||||
report). `-v` logs which curl-impersonate profile fetched each URL.
|
||||
2. verifies the discovered report is the supported import path
|
||||
3. imports it plus `--history <n>` earlier months, oldest first, into an
|
||||
isolated temp DB
|
||||
|
|
@ -91,12 +93,24 @@ The publisher is idempotent and prints exactly one final line:
|
|||
| --- | --- | --- |
|
||||
| `RESULT: published <as-of>` | 0 | a newer snapshot, or larger history, was uploaded |
|
||||
| `RESULT: up-to-date <as-of>` | 0 | nothing to upload |
|
||||
| `RESULT: blocked stage=<stage>` | 6 | IDX answered with a Cloudflare challenge in `discover` or `build-snapshot`; nothing was uploaded |
|
||||
| `RESULT: FAILED stage=<stage> (exit N)` | N | failed in `arguments`, `build`, `preflight`, `discover`, `build-snapshot`, `stage-output`, `upload`, or `publish` |
|
||||
|
||||
`idx-ownership-freshness` is the independent alarm: it exits 1 when the published
|
||||
`latest_as_of_date` is older than `--max-age-days` (default 40). The source lands
|
||||
2-3 days after month end, so 40 days only trips when publishing has stopped.
|
||||
|
||||
`blocked` means retry on the next run, not that something broke. The source is
|
||||
monthly, so one run getting through per month is enough, and stale data is what
|
||||
should alert:
|
||||
|
||||
| Result | Snapshot | Action |
|
||||
| --- | --- | --- |
|
||||
| `published` or `up-to-date` | fresh | none |
|
||||
| `blocked` | fresh | none; the next daily run retries |
|
||||
| `blocked` | stale | alert: retry, or import the file by hand |
|
||||
| `FAILED` | either | alert at once |
|
||||
|
||||
## History Coverage
|
||||
|
||||
`--history <n>` (maximum 1000) requests the latest report plus up to `n` earlier
|
||||
|
|
@ -135,16 +149,18 @@ on a self-hosted machine that IDX accepts:
|
|||
|
||||
- `contrib/systemd/idx-ownership-snapshot-publish.service` and `.timer` run the
|
||||
packaged publisher daily at 06:00 UTC with `GH_TOKEN` from
|
||||
`/etc/idx-ownership-snapshot.env`. Wire `OnFailure=` to a notification.
|
||||
`/etc/idx-ownership-snapshot.env`. The service treats exit 6 (`blocked`) as success
|
||||
and then runs the freshness check, so the unit fails on `FAILED` or on stale data.
|
||||
An unreadable manifest also fails it. Wire `OnFailure=` to a notification.
|
||||
- On NixOS, add this repo as a flake input and use
|
||||
`inputs.idx-cli.packages.${system}.ownership-publisher`; `nix flake update idx-cli`
|
||||
picks up fixes.
|
||||
|
||||
`.github/workflows/publish-ownership-snapshot.yml` is a manual
|
||||
(`workflow_dispatch`) fallback that runs the same packaged publisher with inputs
|
||||
passed as environment variables.
|
||||
passed as environment variables. There a `blocked` run shows as a failed job.
|
||||
|
||||
Observed XLSX upload times (HTTP Last-Modified) were the 2nd-3rd of each month
|
||||
around 02:00-04:00 UTC; a daily idempotent run makes the exact day irrelevant.
|
||||
If the monthly source has not appeared yet, the run should fail loudly and retry
|
||||
later rather than publish stale data.
|
||||
If the monthly source has not appeared yet, the run should not publish stale data:
|
||||
it ends `up-to-date` until the new report is listed, and a later daily run picks it up.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue