When Cloudflare answers idx.co.id with a challenge, curl fails with only
`curl: (22)`, which exits 4 like any other HTTP failure, and every profile
is tried twice. A challenge needs a browser, so no profile can solve it. The
source is monthly, so the publisher only needs one run per month to get
through: treat a challenge as "retry tomorrow" and alert on stale data.
- Add IdxError::CloudflareChallenge (CLOUDFLARECHALLENGE, exit 6), detected
from the cf-mitigated header. curl reports it on stderr through --write-out,
because --fail drops the response.
- A URL with a challenge gets one round of profiles; other failures keep two.
This also covers `stocks flow --source idx`, which now exits 6 on a challenge.
- -v logs which profile fetched each URL.
- Discovery keeps error types, so a challenge survives an announcement miss.
A source answering with something unparseable still is not called a challenge.
- `ownership discover --require-data-page` fails when the data page cannot be
read. Default discover is unchanged: it warns and lists older announcements.
- The publisher and builder use that flag, so a failed data page can no longer
end in a false "up-to-date". A challenge ends in `RESULT: blocked stage=<stage>`,
exit 6; every other failure stays `RESULT: FAILED`.
- The systemd example treats exit 6 as success and runs the freshness check, so
the unit fails on FAILED or on stale data.
- live-smoke reports a challenge as CHALLENGED, a failure and never a pass.
Golden change: completions-bash.golden gains `--require-data-page` in the
discover options, one word on one line. It is the direct result of the new flag;
no other golden file changes.
Live check on 2026-11-05 against release/v0.5.0: 19 passed, 2 failed. Both
failures are real challenges (flow-idx, discover), each exit 6 with empty stdout.
The import, releases and ticker cases did not run, because discover did not pass.
Refs #83
From the Codex re-gate of the 0.3.3 fixes: NUL-separated discovery
fields need jq 1.7 (--raw-output0) and bash 4.4 (mapfile -d); check both
up front with a clear message instead of failing midway on older tools.
ci.yml actions are pinned to commit SHAs like the release workflows.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V3syn2fN2LNo5pvTSqc6eh
From the full-repo Codex audit:
- The publisher read discovery fields one per line, so a report title
with newlines shifted every later field and published a manifest
without source.publish_date, which every client rejects (blocking all
syncs). Read them NUL-separated, and refuse to write a manifest missing
a field clients require.
- The snapshot builder copied the SQLite file with cp; committed data
still in the -wal file of an open database would be missing. Checkpoint
first (a no-op for the publisher's closed database, so the bytes and
hash are unchanged) and refuse to copy when that cannot complete.
- install.sh took the tag from a greedy match, which on a compact
single-line releases response picks the last tag, not the newest (it
installed v0.2.3 over v0.2.4 in the new test). Split fields first.
- scripts/audit-msn-fundamentals.sh read the symbol map from a path that
no longer exists and silently checked no stocks.
- release.yml granted contents: write to every job; only publish gets it
now. Actions in the release and snapshot workflows are pinned to commit
SHAs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V3syn2fN2LNo5pvTSqc6eh
Addresses review on #31:
- the snapshot download limit is the manifest's size_bytes (+1, since
ureq fails once a body reaches its limit), capped at 1 GiB; a body
larger than advertised fails during download. New CLI test; the
>10 MiB test also caught the off-by-one.
- a refused (oversized) standalone build removes its sqlite and
manifest from --output-dir.
- docs: downloads are buffered, not streamed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017a5Cv5Z1NzdYKgKnTiAF4E
On 2026-09-25 a snapshot built with --history 5 (15.3 MB) was
published, and `idx ownership sync` failed for every client:
"the response body is larger than request limit: 10485760" (ureq's
default body cap). It was republished by hand with --history 2 (8.2 MB).
- snapshot download: raise the ureq body limit to 1 GiB; the file is
still verified against the manifest's size and SHA-256. The PDF/XLSX
report downloads get a 256 MiB limit for the same reason.
- CLI test: sync a >10 MiB snapshot over HTTP (fails without the fix
with the exact production error).
- build-latest-ownership-snapshot.sh: --max-snapshot-bytes (default
10485760, the limit of released clients <= v0.2.3); an oversized
snapshot fails the build, so the published one stays untouched.
- publish-ownership-snapshot.sh passes --max-snapshot-bytes through;
publisher tests cover the failure and that the release is unchanged.
- docs: size table and guidance.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017a5Cv5Z1NzdYKgKnTiAF4E
Addresses review on #27: canonicalize --history ("08" no longer hits
bash octal parsing), run the history jq in a plain command
substitution so a jq failure aborts under set -e, and restore the
schedule-guidance paragraph in OWNERSHIP_SELF_HOSTED.md.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017a5Cv5Z1NzdYKgKnTiAF4E
- build-latest-ownership-snapshot.sh: discover with a wider limit and
add --history <n> to import the n previous supported above-1% XLSX
months (one per as-of date, oldest first) before the latest, so
`idx ownership changes` works straight from a synced snapshot.
- publish-ownership-snapshot.sh: pass --history through (for the
clan-private systemd timer via extraArgs).
- publish-ownership-snapshot.yml: `history` input (default 5).
- Docs: publish/self-hosted flow now describes the XLSX source,
observed upload days (2nd-3rd of the month), and --history.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017a5Cv5Z1NzdYKgKnTiAF4E
