fix(ownership): sync snapshots over 10 MiB; refuse oversized publishes

On 2026-09-25 a snapshot built with --history 5 (15.3 MB) was
published, and `idx ownership sync` failed for every client:
"the response body is larger than request limit: 10485760" (ureq's
default body cap). It was republished by hand with --history 2 (8.2 MB).

- snapshot download: raise the ureq body limit to 1 GiB; the file is
  still verified against the manifest's size and SHA-256. The PDF/XLSX
  report downloads get a 256 MiB limit for the same reason.
- CLI test: sync a >10 MiB snapshot over HTTP (fails without the fix
  with the exact production error).
- build-latest-ownership-snapshot.sh: --max-snapshot-bytes (default
  10485760, the limit of released clients <= v0.2.3); an oversized
  snapshot fails the build, so the published one stays untouched.
- publish-ownership-snapshot.sh passes --max-snapshot-bytes through;
  publisher tests cover the failure and that the release is unchanged.
- docs: size table and guidance.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017a5Cv5Z1NzdYKgKnTiAF4E
This commit is contained in:
hermes 2026-09-25 22:22:48 +00:00
commit de58d04da9
7 changed files with 156 additions and 5 deletions

View file

@ -21,6 +21,10 @@ Options:
--history <n> Also import the <n> previous monthly above-1% XLSX
reports (oldest first) so `ownership changes` works
from the snapshot (default: 0)
--max-snapshot-bytes <n>
Refuse to emit a snapshot larger than this
(default: 10485760, the download limit of idx
clients <= v0.2.3; exceeding it breaks their sync)
--keep-workdir Keep the temp workdir instead of deleting it
--help Show this help
EOF
@ -33,6 +37,7 @@ REPO_FULL_NAME="0xrsydn/idx-cli"
RELEASE_TAG="ownership-snapshot-current"
KEEP_WORKDIR="0"
HISTORY="0"
MAX_SNAPSHOT_BYTES="10485760"
while [[ $# -gt 0 ]]; do
case "$1" in
@ -60,6 +65,10 @@ while [[ $# -gt 0 ]]; do
HISTORY="${2:-}"
shift 2
;;
--max-snapshot-bytes)
MAX_SNAPSHOT_BYTES="${2:-}"
shift 2
;;
--keep-workdir)
KEEP_WORKDIR="1"
shift
@ -82,6 +91,11 @@ if [[ -z "$OUTPUT_DIR" ]]; then
exit 2
fi
if ! [[ "$MAX_SNAPSHOT_BYTES" =~ ^[1-9][0-9]{0,17}$ ]]; then
echo "--max-snapshot-bytes must be a positive integer" >&2
exit 2
fi
if ! [[ "$HISTORY" =~ ^[0-9]+$ ]]; then
echo "--history must be a non-negative integer" >&2
exit 2
@ -254,6 +268,19 @@ IMPORTED_ROW_COUNT="${release_fields[2]:-}"
printf 'Building snapshot artifact and manifest...\n'
"$BASIC_BUILDER" --db "$DB_PATH" --output-dir "$OUTPUT_DIR" --base-url "$BASE_URL"
# Released clients cannot download a snapshot above their HTTP body limit, so
# publishing one would break `idx ownership sync` for every existing install.
SNAPSHOT_BYTES="$(jq -r '.snapshot.size_bytes // empty' "$MANIFEST_PATH")"
if ! [[ "$SNAPSHOT_BYTES" =~ ^[0-9]+$ ]]; then
echo "built manifest has no valid snapshot.size_bytes" >&2
exit 1
fi
if (( SNAPSHOT_BYTES > MAX_SNAPSHOT_BYTES )); then
printf 'snapshot is %s bytes (%s monthly releases), above --max-snapshot-bytes %s; lower --history\n' \
"$SNAPSHOT_BYTES" "$(jq -r '.snapshot.release_count' "$MANIFEST_PATH")" "$MAX_SNAPSHOT_BYTES" >&2
exit 1
fi
TMP_MANIFEST_PATH="$WORKDIR/ownership-snapshot-manifest.json"
jq \
--arg family "$DISCOVERED_FAMILY" \