0xrsydn/idx-cli
1
mirror of https://github.com/0xrsydn/idx-cli.git synced 2026-11-07 09:45:36 +00:00
Code Issues Projects Releases 1 Packages Wiki Activity

fix(ownership): report a Cloudflare challenge as its own error and result

Browse source 
When Cloudflare answers idx.co.id with a challenge, curl fails with only
`curl: (22)`, which exits 4 like any other HTTP failure, and every profile
is tried twice. A challenge needs a browser, so no profile can solve it. The
source is monthly, so the publisher only needs one run per month to get
through: treat a challenge as "retry tomorrow" and alert on stale data.

- Add IdxError::CloudflareChallenge (CLOUDFLARECHALLENGE, exit 6), detected
  from the cf-mitigated header. curl reports it on stderr through --write-out,
  because --fail drops the response.
- A URL with a challenge gets one round of profiles; other failures keep two.
  This also covers `stocks flow --source idx`, which now exits 6 on a challenge.
- -v logs which profile fetched each URL.
- Discovery keeps error types, so a challenge survives an announcement miss.
  A source answering with something unparseable still is not called a challenge.
- `ownership discover --require-data-page` fails when the data page cannot be
  read. Default discover is unchanged: it warns and lists older announcements.
- The publisher and builder use that flag, so a failed data page can no longer
  end in a false "up-to-date". A challenge ends in `RESULT: blocked stage=<stage>`,
  exit 6; every other failure stays `RESULT: FAILED`.
- The systemd example treats exit 6 as success and runs the freshness check, so
  the unit fails on FAILED or on stale data.
- live-smoke reports a challenge as CHALLENGED, a failure and never a pass.

Golden change: completions-bash.golden gains `--require-data-page` in the
discover options, one word on one line. It is the direct result of the new flag;
no other golden file changes.

Live check on 2026-11-05 against release/v0.5.0: 19 passed, 2 failed. Both
failures are real challenges (flow-idx, discover), each exit 6 with empty stdout.
The import, releases and ticker cases did not run, because discover did not pass.

Refs #83
This commit is contained in:
hermes 2026-11-05 08:54:58 +00:00
commit b236c7f9d9
23 changed files with 797 additions and 54 deletions
Download patch file Download diff file Expand all files Collapse all files

26
docs/ARCHITECTURE.md
View file

@ -144,7 +144,7 @@ WIB as the fallback.
- Cloudflare rejects most TLS fingerprints (in October 2026 only
`curl_safari184` passed), so the request uses curl-impersonate's profile
fallback with a 30 s limit. A challenge page fails that profile, not the
parse.
parse; if no profile passes, the error is `CLOUDFLARECHALLENGE`.
- Each session is cached whole (every stock) by date: 30 days once past,
5 minutes for today. `Flow::foreign` walks weekdays without requesting
weekends; offline, `--days` skips uncached sessions while a `--from/--to`
@ -190,10 +190,23 @@ Only the above-1% XLSX is `supported`; the daily above-5% and investor-type work
are listed as `unsupported`. Both URLs can be overridden for tests with
`IDX_OWNERSHIP_ANNOUNCEMENT_API_URL` and `IDX_OWNERSHIP_DATA_PAGE_URL`.
IDX sits behind Cloudflare, which accepts some curl-impersonate TLS profiles and
rejects others per path (for example `curl_chrome142` gets 403 on the data page).
IDX fetches therefore try several profiles until the response validates (JSON, page
payload, `%PDF`, or `PK` zip magic). `IDX_CURL_IMPERSONATE_BIN` pins a single profile.
IDX sits behind Cloudflare, which can accept some curl-impersonate TLS profiles and
reject others, varying by path and over time. IDX fetches therefore try several
profiles until the response validates (JSON, page payload, `%PDF`, or `PK` zip
magic). `IDX_CURL_IMPERSONATE_BIN` pins a single profile.
At times every profile gets a challenge (`cf-mitigated: challenge`, seen on
2026-11-05), which only a browser can solve. curl reports that header on stderr
through `--write-out`, because `--fail` drops the response. A profile that gets a
challenge fails with `CLOUDFLARECHALLENGE` (exit 6): the remaining profiles still
run once, but a URL with a challenge gets no second round. Other failures keep two
rounds. `-v` logs which profile fetched each URL.
Discovery that finds nothing reports `CLOUDFLARECHALLENGE` if a source was
challenged, unless another source answered with something unparseable, which a retry
cannot fix. By default a failed data page is only a warning when announcements
returned results, and those can miss the newest reports;
`ownership discover --require-data-page` makes it an error.
### Parser Pipeline
```
@ -240,7 +253,8 @@ CLI flags > environment variables > config file > defaults
bidirectional overrides replaced before it reaches a terminal table.
- Exit codes by class: `0` success, `2` `INVALIDINPUT`/`CONFIGERROR` and clap usage
errors, `3` `SYMBOLNOTFOUND`/`NOMARKETDATA`/`CACHEMISS`, `4` `HTTP`/`RATELIMITED`/
`PROVIDERUNAVAILABLE`/`OFFLINE`/`AUTHERROR`, `5` `UNSUPPORTED`, `1` everything else
`PROVIDERUNAVAILABLE`/`OFFLINE`/`AUTHERROR`, `5` `UNSUPPORTED`, `6` `CLOUDFLARECHALLENGE`,
`1` everything else
- A totally failed `stocks compare` emits one structured error in JSON mode; per-symbol warnings are retained for partial success, not printed ahead of a total-failure error.
## CLI Output and Offline Cache