fix(ownership): report a Cloudflare challenge as its own error and result

When Cloudflare answers idx.co.id with a challenge, curl fails with only
`curl: (22)`, which exits 4 like any other HTTP failure, and every profile
is tried twice. A challenge needs a browser, so no profile can solve it. The
source is monthly, so the publisher only needs one run per month to get
through: treat a challenge as "retry tomorrow" and alert on stale data.

- Add IdxError::CloudflareChallenge (CLOUDFLARECHALLENGE, exit 6), detected
  from the cf-mitigated header. curl reports it on stderr through --write-out,
  because --fail drops the response.
- A URL with a challenge gets one round of profiles; other failures keep two.
  This also covers `stocks flow --source idx`, which now exits 6 on a challenge.
- -v logs which profile fetched each URL.
- Discovery keeps error types, so a challenge survives an announcement miss.
  A source answering with something unparseable still is not called a challenge.
- `ownership discover --require-data-page` fails when the data page cannot be
  read. Default discover is unchanged: it warns and lists older announcements.
- The publisher and builder use that flag, so a failed data page can no longer
  end in a false "up-to-date". A challenge ends in `RESULT: blocked stage=<stage>`,
  exit 6; every other failure stays `RESULT: FAILED`.
- The systemd example treats exit 6 as success and runs the freshness check, so
  the unit fails on FAILED or on stale data.
- live-smoke reports a challenge as CHALLENGED, a failure and never a pass.

Golden change: completions-bash.golden gains `--require-data-page` in the
discover options, one word on one line. It is the direct result of the new flag;
no other golden file changes.

Live check on 2026-11-05 against release/v0.5.0: 19 passed, 2 failed. Both
failures are real challenges (flow-idx, discover), each exit 6 with empty stdout.
The import, releases and ticker cases did not run, because discover did not pass.

Refs #83

crates/idx-core/src/curl_impersonate.rs

@@ -44,9 +44,11 @@ pub fn run_owned(stage: &str, args: &[String]) -> Result<Output, IdxError> {
     run_binary(stage, &binary, args)
 }
 
-/// Browser profiles tried, in order, when a site's bot protection rejects
-/// some TLS fingerprints but not others (IDX's Cloudflare rules block
-/// `curl_chrome142` on some pages while Firefox/Safari profiles pass).
+/// Browser profiles tried, in order, because a site's bot protection can reject
+/// some TLS fingerprints but not others. On IDX which profiles pass changes over
+/// time and by path, and at times none does (on 2026-11-05 Chrome, Firefox and
+/// Safari profiles all got a challenge). A profile only changes the fingerprint,
+/// so it cannot solve a challenge that is active for this client.
 const FALLBACK_PROFILES: &[&str] = &[
     "curl_chrome142",
     "curl_firefox144",
@@ -60,9 +62,47 @@ const FALLBACK_PROFILES: &[&str] = &[
 ];
 const FALLBACK_ROUNDS: usize = 2;
 
+/// Marks the line curl adds to stderr describing the response. `--fail` drops
+/// the response itself, and stdout must stay the body alone.
+const PROBE_PREFIX: &str = "idx-curl-probe";
+const PROBE_WRITE_OUT: &str =
+    "%{stderr}idx-curl-probe status=%{http_code} cf-mitigated=%header{cf-mitigated}\\n";
+
+/// What the probe line says about a response.
+#[derive(Debug, PartialEq, Eq)]
+struct ResponseProbe {
+    status: String,
+    challenged: bool,
+}
+
+/// Separates the probe line from curl's own stderr. Without a probe line (the
+/// caller did not ask for one) stderr comes back unchanged.
+fn split_probe(stderr: &[u8]) -> (Option<ResponseProbe>, Vec<u8>) {
+    let mut probe = None;
+    let mut rest = Vec::with_capacity(stderr.len());
+    for line in stderr.split_inclusive(|byte| *byte == b'\n') {
+        let parsed = std::str::from_utf8(line)
+            .ok()
+            .and_then(|text| text.trim_end().strip_prefix(PROBE_PREFIX))
+            .and_then(|fields| fields.strip_prefix(" status="))
+            .and_then(|fields| fields.split_once(" cf-mitigated="));
+        match parsed {
+            Some((status, mitigated)) => {
+                probe = Some(ResponseProbe {
+                    status: status.to_string(),
+                    challenged: mitigated.trim().eq_ignore_ascii_case("challenge"),
+                });
+            }
+            None => rest.extend_from_slice(line),
+        }
+    }
+    (probe, rest)
+}
+
 /// Run curl with each available impersonation profile until `accept` approves
 /// the response body. `IDX_CURL_IMPERSONATE_BIN`, when set, is the only
-/// profile used. Returns the first accepted output, or the last error.
+/// profile used. Returns the first accepted output, or an error: a Cloudflare
+/// challenge from any profile, otherwise the last error.
 pub fn run_owned_with_fallback(
     stage: &str,
     args: &[String],
@@ -83,17 +123,39 @@ pub fn run_owned_with_fallback(
         )));
     }
 
+    run_profiles(stage, &profiles, args, accept)
+}
+
+/// A challenge can differ by profile, so one round tries them all, but a repeat
+/// straight away is unlikely to change the answer: it ends after that round.
+fn run_profiles(
+    stage: &str,
+    profiles: &[String],
+    args: &[String],
+    accept: &dyn Fn(&[u8]) -> Result<(), IdxError>,
+) -> Result<Output, IdxError> {
+    let mut probed_args = vec!["--write-out".to_string(), PROBE_WRITE_OUT.to_string()];
+    probed_args.extend_from_slice(args);
+
     let mut last_error = None;
+    let mut challenge = None;
     for _ in 0..FALLBACK_ROUNDS {
-        for binary in &profiles {
-            match run_binary(stage, binary, args) {
+        for binary in profiles {
+            match run_binary(stage, binary, &probed_args) {
                 Ok(output) => match accept(&output.stdout) {
-                    Ok(()) => return Ok(output),
+                    Ok(()) => {
+                        crate::runtime::verbose(format!("{stage}: fetched with {binary}"));
+                        return Ok(output);
+                    }
                     Err(err) => last_error = Some(err),
                 },
+                Err(err @ IdxError::CloudflareChallenge(_)) => challenge = Some(err),
                 Err(err) => last_error = Some(err),
             }
         }
+        if let Some(err) = challenge {
+            return Err(err);
+        }
     }
     Err(last_error.expect("at least one profile was tried"))
 }
@@ -166,12 +228,20 @@ fn run_binary_limited(
         .wait()
         .map_err(|e| IdxError::Http(format!("failed to wait for {binary} for {stage}: {e}")))?;
     let stderr = stderr_reader.join().unwrap_or_default();
+    let (probe, stderr) = split_probe(&stderr);
     let output = Output {
         status,
         stdout,
         stderr,
     };
 
+    if let Some(probe) = probe.filter(|probe| probe.challenged) {
+        return Err(IdxError::CloudflareChallenge(format!(
+            "{stage}: Cloudflare answered {binary} with a challenge (HTTP {}, cf-mitigated: challenge), which curl cannot solve; retry later",
+            probe.status
+        )));
+    }
+
     if output.status.success() {
         return Ok(output);
     }

crates/idx-core/src/error.rs

@@ -34,6 +34,10 @@ pub enum IdxError {
     InvalidInput(String),
     #[error("no market data: {0}")]
     NoMarketData(String),
+    /// Cloudflare answered with a challenge only a browser can solve, on every
+    /// client tried. Nothing is wrong with the request, so a later run may pass.
+    #[error("cloudflare challenge: {0}")]
+    CloudflareChallenge(String),
 }
 
 #[derive(Debug, Clone, Copy, Serialize, PartialEq, Eq)]
@@ -53,6 +57,7 @@ pub enum ErrorCode {
     PdfParseError,
     InvalidInput,
     NoMarketData,
+    CloudflareChallenge,
 }
 
 /// The stable wire form frontends emit, e.g. `SYMBOLNOTFOUND`.
@@ -80,6 +85,7 @@ impl IdxError {
             Self::PdfParseError(_) => ErrorCode::PdfParseError,
             Self::InvalidInput(_) => ErrorCode::InvalidInput,
             Self::NoMarketData(_) => ErrorCode::NoMarketData,
+            Self::CloudflareChallenge(_) => ErrorCode::CloudflareChallenge,
         }
     }
 
@@ -95,6 +101,7 @@ impl IdxError {
             | ErrorCode::Offline
             | ErrorCode::AuthError => 4,
             ErrorCode::Unsupported => 5,
+            ErrorCode::CloudflareChallenge => 6,
             ErrorCode::Io
             | ErrorCode::ParseError
             | ErrorCode::DatabaseError

crates/idx-core/src/investor.rs

@@ -14,7 +14,9 @@ use sha2::{Digest, Sha256};
 
 use crate::config::CoreConfig;
 use crate::error::IdxError;
-use crate::ownership::remote::{self, DiscoveredOwnershipPdf, OwnershipReportFamily};
+use crate::ownership::remote::{
+    self, DataPagePolicy, DiscoveredOwnershipPdf, OwnershipReportFamily,
+};
 use crate::ownership::snapshot::{self, OwnershipSyncResult};
 use crate::ownership::types::{
     ChangeRow, ConcentrationMetrics, CrossHolderRow, Entity, EntityHoldings, GraphEdge, GraphNode,
@@ -264,8 +266,9 @@ impl Investor {
         &self,
         family: Option<OwnershipReportFamily>,
         limit: usize,
+        data_page: DataPagePolicy,
     ) -> Result<Vec<DiscoveredOwnershipPdf>, IdxError> {
-        remote::discover_idx_ownership_reports(family, limit)
+        remote::discover_idx_ownership_reports(family, limit, data_page)
     }
 
     /// Installs or refreshes the maintained snapshot. `manifest` overrides

crates/idx-core/src/lib.rs

@@ -27,6 +27,7 @@ pub mod runtime {
 
     static QUIET: AtomicBool = AtomicBool::new(false);
     static JSON: AtomicBool = AtomicBool::new(false);
+    static VERBOSE: AtomicBool = AtomicBool::new(false);
 
     pub fn set_quiet(quiet: bool) {
         QUIET.store(quiet, Ordering::Relaxed);
@@ -36,6 +37,10 @@ pub mod runtime {
         QUIET.load(Ordering::Relaxed)
     }
 
+    pub fn set_verbose(verbose: bool) {
+        VERBOSE.store(verbose, Ordering::Relaxed);
+    }
+
     /// In JSON mode diagnostics are JSON lines, so stderr stays
     /// machine-readable alongside a JSON error.
     pub fn set_json(json: bool) {
@@ -61,6 +66,13 @@ pub mod runtime {
         emit("info", message);
     }
 
+    /// Detail worth keeping in a job's log but not worth showing by default.
+    pub fn verbose(message: impl Display) {
+        if VERBOSE.load(Ordering::Relaxed) {
+            emit("info", message);
+        }
+    }
+
     pub fn warn(message: impl Display) {
         emit("warning", message);
     }

crates/idx-core/src/ownership/remote.rs

@@ -281,12 +281,24 @@ pub fn select_latest_ownership_reports(
     Ok(attachments)
 }
 
+/// Whether discovery may answer without the data page, the only place the
+/// newest above-1% reports are listed since June 2026.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub enum DataPagePolicy {
+    /// Warn and return what the announcements list, which can miss the newest
+    /// reports.
+    BestEffort,
+    /// Fail with the data page's error, so old results cannot pass as current.
+    Required,
+}
+
 pub fn discover_idx_ownership_reports(
     family_filter: Option<OwnershipReportFamily>,
     limit: usize,
+    data_page: DataPagePolicy,
 ) -> Result<Vec<DiscoveredOwnershipPdf>, IdxError> {
     let mut discovered = Vec::new();
-    let mut errors = Vec::new();
+    let mut errors: Vec<IdxError> = Vec::new();
 
     for query in DISCOVERY_QUERIES
         .iter()
@@ -303,7 +315,7 @@ pub fn discover_idx_ownership_reports(
         .and_then(|page| select_latest_ownership_reports(&page, &query_url, query.family))
         {
             Ok(mut reports) => discovered.append(&mut reports),
-            Err(err) => errors.push(err.to_string()),
+            Err(err) => errors.push(err),
         }
     }
 
@@ -320,10 +332,10 @@ pub fn discover_idx_ownership_reports(
                 // The page loaded (it has its file list) but nothing parsed:
                 // the payload shape changed. Say so instead of silently
                 // falling back to announcement-only results.
-                page_error = Some(format!(
+                page_error = Some(IdxError::ParseError(format!(
                     "IDX share ownership data page {} loaded but no report entries could be parsed; the page layout may have changed",
                     share_ownership_page_url()
-                ));
+                )));
             }
             page_reports
                 .retain(|report| family_filter.is_none_or(|family| family == report.family));
@@ -343,7 +355,13 @@ pub fn discover_idx_ownership_reports(
             });
             discovered.extend(page_reports);
         }
-        Err(err) => page_error = Some(err.to_string()),
+        Err(err) => page_error = Some(err),
+    }
+
+    if data_page == DataPagePolicy::Required
+        && let Some(err) = page_error
+    {
+        return Err(err);
     }
 
     discovered.sort_by(|left, right| {
@@ -369,21 +387,50 @@ pub fn discover_idx_ownership_reports(
     errors.extend(page_error);
 
     if discovered.is_empty() {
-        let detail = if errors.is_empty() {
-            "no matching ownership reports found".to_string()
-        } else {
-            errors.join("; ")
-        };
-        return Err(IdxError::Http(format!(
-            "failed to discover IDX ownership reports from {} or {}: {detail}",
-            announcement_listing_url(),
-            share_ownership_page_url()
-        )));
+        return Err(discovery_failure(
+            &errors,
+            &announcement_listing_url(),
+            &share_ownership_page_url(),
+        ));
     }
 
     Ok(discovered)
 }
 
+/// The error for a discovery that found nothing. A challenge names the cause
+/// unless a source answered with something unparseable: a retry cannot fix that,
+/// so it stays a plain failure. Announcement misses do not hide a challenge.
+fn discovery_failure(errors: &[IdxError], listing_url: &str, page_url: &str) -> IdxError {
+    let challenged = errors
+        .iter()
+        .any(|err| matches!(err, IdxError::CloudflareChallenge(_)));
+    let unparseable = errors
+        .iter()
+        .any(|err| matches!(err, IdxError::ParseError(_)));
+
+    let detail = if errors.is_empty() {
+        "no matching ownership reports found".to_string()
+    } else {
+        errors
+            .iter()
+            .map(|err| match err {
+                IdxError::CloudflareChallenge(message) => message.clone(),
+                other => other.to_string(),
+            })
+            .collect::<Vec<_>>()
+            .join("; ")
+    };
+    let message = format!(
+        "failed to discover IDX ownership reports from {listing_url} or {page_url}: {detail}"
+    );
+
+    if challenged && !unparseable {
+        IdxError::CloudflareChallenge(message)
+    } else {
+        IdxError::Http(message)
+    }
+}
+
 pub fn download_idx_pdf(url: &str, target: &Path) -> Result<(), IdxError> {
     let bytes = fetch_bytes(
         "IDX ownership PDF download",