When Cloudflare answers idx.co.id with a challenge, curl fails with only
`curl: (22)`, which exits 4 like any other HTTP failure, and every profile
is tried twice. A challenge needs a browser, so no profile can solve it. The
source is monthly, so the publisher only needs one run per month to get
through: treat a challenge as "retry tomorrow" and alert on stale data.
- Add IdxError::CloudflareChallenge (CLOUDFLARECHALLENGE, exit 6), detected
from the cf-mitigated header. curl reports it on stderr through --write-out,
because --fail drops the response.
- A URL with a challenge gets one round of profiles; other failures keep two.
This also covers `stocks flow --source idx`, which now exits 6 on a challenge.
- -v logs which profile fetched each URL.
- Discovery keeps error types, so a challenge survives an announcement miss.
A source answering with something unparseable still is not called a challenge.
- `ownership discover --require-data-page` fails when the data page cannot be
read. Default discover is unchanged: it warns and lists older announcements.
- The publisher and builder use that flag, so a failed data page can no longer
end in a false "up-to-date". A challenge ends in `RESULT: blocked stage=<stage>`,
exit 6; every other failure stays `RESULT: FAILED`.
- The systemd example treats exit 6 as success and runs the freshness check, so
the unit fails on FAILED or on stale data.
- live-smoke reports a challenge as CHALLENGED, a failure and never a pass.
Golden change: completions-bash.golden gains `--require-data-page` in the
discover options, one word on one line. It is the direct result of the new flag;
no other golden file changes.
Live check on 2026-11-05 against release/v0.5.0: 19 passed, 2 failed. Both
failures are real challenges (flow-idx, discover), each exit 6 with empty stdout.
The import, releases and ticker cases did not run, because discover did not pass.
Refs #83
A snapshot republished for the same as-of date (the TRUE ticker fix)
kept the date, latest release, and release count, so sync saw nothing
to do and existing installs kept the bad data without --force.
The manifest gains an optional snapshot.replaces_sqlite_sha256 listing
earlier builds for that date. Queries never write to a synced file, so
a local database byte-identical to a listed build is replaced; one that
matches nothing (local imports or edits) still needs --force. The
publisher fills the list on a same-date rebuild. Older clients ignore
the field.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Remove FEATURE_SPEC.md, TODO.md, and docs/AGENT_INTEGRATION.md; their
core gaps have shipped. Open work moves to a short ROADMAP.md.
- Remove the CLAUDE.md symlink and rewrite AGENTS.md as a quick guide:
navigation, rules, and the nix flake check gate.
- Merge OWNERSHIP_SELF_HOSTED.md into OWNERSHIP_PUBLISH.md and drop
private host integration notes.
- ARCHITECTURE: add MSN endpoints, clarify Bing ingest is unsupported.
- OWNERSHIP_SYNC: note source.pdf_url also carries XLSX URLs.
- systemd example: --history 2 so snapshots stay under the 10 MiB
client cap.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- build-latest-ownership-snapshot.sh: discover with a wider limit and
add --history <n> to import the n previous supported above-1% XLSX
months (one per as-of date, oldest first) before the latest, so
`idx ownership changes` works straight from a synced snapshot.
- publish-ownership-snapshot.sh: pass --history through (for the
clan-private systemd timer via extraArgs).
- publish-ownership-snapshot.yml: `history` input (default 5).
- Docs: publish/self-hosted flow now describes the XLSX source,
observed upload days (2nd-3rd of the month), and --history.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017a5Cv5Z1NzdYKgKnTiAF4E
