idx-cli/docs/OWNERSHIP_PUBLISH.md

Ownership Snapshot Publishing

Maintainer workflow for producing and publishing the snapshot that idx ownership sync consumes. The consumer contract (manifest schema and replacement rules) is in docs/OWNERSHIP_SYNC.md.

Release Layout

Snapshot assets live on a dedicated GitHub release, separate from versioned app releases:

• tag: ownership-snapshot-current
• manifest: ownership-snapshot-manifest.json (stable URL, the commit point)
• SQLite: ownership-snapshot-YYYY-MM-DD-<sha256>.sqlite (immutable, content-addressed)

https://github.com/0xrsydn/idx-cli/releases/download/ownership-snapshot-current/ownership-snapshot-manifest.json

Never use .../releases/latest/download/...: the repo's latest release is an app release, not the snapshot.

Running the Publisher

Packaged (recommended for hosts; no checkout or cargo build needed):

nix build github:0xrsydn/idx-cli#ownership-publisher
./result/bin/idx-ownership-publish --output-dir /var/lib/idx/ownership-snapshot/current --history 2
./result/bin/idx-ownership-freshness --max-age-days 40

From a checkout:

nix develop --command scripts/publish-ownership-snapshot.sh \
  --build \
  --idx-bin ./target/debug/idx \
  --output-dir dist/ownership-snapshot \
  --repo 0xrsydn/idx-cli \
  --release-tag ownership-snapshot-current

idx-ownership-publish wraps scripts/publish-ownership-snapshot.sh with a pinned idx (including curl-impersonate and mupdf) plus jq, gh, sqlite, and curl. It needs GH_TOKEN with permission to upload release assets; the freshness check needs no token.

What It Does

1. idx -v -o json ownership discover --family above1 --limit 50 --require-data-page (newest supported report first; XLSX from the Data Kepemilikan Saham page since June 2026, so a run that cannot read that page stops rather than miss the newest report). -v logs which curl-impersonate profile fetched each URL.
2. verifies the discovered report is the supported import path
3. imports it plus --history <n> earlier months, oldest first, into an isolated temp DB
4. checks the imported release metadata is non-empty and tied to the source URL
5. builds the SQLite artifact and manifest (scripts/build-latest-ownership-snapshot.sh, which calls scripts/build-ownership-snapshot.sh) and records source provenance in the manifest
6. uploads using the protocol below

Publication Protocol

The manifest is the commit point, so uploads are ordered:

1. Upload the content-addressed SQLite asset. Same content means same name, so retries are idempotent.
2. Read the release assets back and verify the full SHA-256 and size (GitHub's sha256: digest when present, otherwise download and hash). Fail before touching the manifest if this does not match.
3. Upload the manifest last and verify its snapshot.sqlite_sha256.

A partial upload exits non-zero and is repaired by the next run; previously published manifests stay valid because old SQLite assets are retained. GitHub does not replace the manifest transactionally, so a failed replacement can leave it unavailable until a retry succeeds.

The publisher treats the published snapshot as current only when the manifest's download_url names this repo and tag and the referenced asset matches its SHA-256 and size. A missing or corrupt asset, a mismatched URL, an unreadable manifest, or any GitHub auth/network error means "publish", never "up to date". --force uploads regardless. A forced rebuild for the already-published as-of date adds the replaced build's sqlite_sha256 to replaces_sqlite_sha256 (keeping the last 20), so clients still holding it update on their next sync.

Run Result

The publisher is idempotent and prints exactly one final line:

Last line	Exit	Meaning
RESULT: published <as-of>	0	a newer snapshot, or larger history, was uploaded
RESULT: up-to-date <as-of>	0	nothing to upload
RESULT: blocked stage=<stage>	6	IDX answered with a Cloudflare challenge in discover or build-snapshot; nothing was uploaded
RESULT: FAILED stage=<stage> (exit N)	N	failed in arguments, build, preflight, discover, build-snapshot, stage-output, upload, or publish

idx-ownership-freshness is the independent alarm: it exits 1 when the published latest_as_of_date is older than --max-age-days (default 40). The source lands 2-3 days after month end, so 40 days only trips when publishing has stopped.

blocked means retry on the next run, not that something broke. The source is monthly, so one run getting through per month is enough, and stale data is what should alert:

Result	Snapshot	Action
published or up-to-date	fresh	none
blocked	fresh	none; the next daily run retries
blocked	stale	alert: retry, or import the file by hand
FAILED	either	alert at once

History Coverage

--history <n> (maximum 1000) requests the latest report plus up to n earlier monthly XLSX reports:

• desired releases = 1 + min(--history, available earlier XLSX months)
• same as-of date with at least the desired release count: no-op
• same as-of date with fewer releases: republish to add history
• a request larger than the source's history is accepted once, not rebuilt forever

Each extra month adds about 2.4 MB, and history is what lets idx ownership changes work straight from a synced snapshot.

Snapshot Size Limit

idx clients up to v0.2.3 cap the snapshot download at 10 MiB (10,485,760 bytes) and accept only uncompressed SQLite. A larger snapshot breaks idx ownership sync for every existing install (this happened on 2026-09-25 with --history 5).

--history	months	size (2026-08-31)
2	3	8,171,520
3	4	10,510,336 (too large)
5	6	15,290,368 (too large)

The builder refuses snapshots above --max-snapshot-bytes (default 10485760), ending with RESULT: FAILED stage=build-snapshot and leaving the published snapshot unchanged. Newer clients accept up to the manifest's size_bytes (at most 1 GiB); raise both limits only once pre-v0.2.4 clients are rare.

Automation

IDX has returned 403 to GitHub-hosted runners, so unattended publishing runs on a self-hosted machine that IDX accepts:

• contrib/systemd/idx-ownership-snapshot-publish.service and .timer run the packaged publisher daily at 06:00 UTC with GH_TOKEN from /etc/idx-ownership-snapshot.env. The service treats exit 6 (blocked) as success and then runs the freshness check, so the unit fails on FAILED or on stale data. An unreadable manifest also fails it. Wire OnFailure= to a notification.
• On NixOS, add this repo as a flake input and use inputs.idx-cli.packages.${system}.ownership-publisher; nix flake update idx-cli picks up fixes.

.github/workflows/publish-ownership-snapshot.yml is a manual (workflow_dispatch) fallback that runs the same packaged publisher with inputs passed as environment variables. There a blocked run shows as a failed job.

Observed XLSX upload times (HTTP Last-Modified) were the 2nd-3rd of each month around 02:00-04:00 UTC; a daily idempotent run makes the exact day irrelevant. If the monthly source has not appeared yet, the run should not publish stale data: it ends up-to-date until the new report is listed, and a later daily run picks it up.