idx-cli/docs/OWNERSHIP_SELF_HOSTED.md

Self-Hosted Ownership Snapshot Publish

This document covers the recommended self-hosted automation path for publishing the monthly ownership snapshot consumed by idx ownership sync.

Use this path when:

• the repo's public release assets should stay on GitHub Releases
• IDX blocks GitHub-hosted Actions with 403
• you have a VPS or other trusted machine that can reach the IDX source

Why Self-Hosted

The current GitHub-hosted workflow remains useful as a manual reference, but it is not reliable enough for unattended publishing because IDX has returned 403 to GitHub-hosted runners. The self-hosted path keeps the end-user experience simple:

• maintainers publish from a trusted machine
• end users still run plain idx ownership sync

No Nix setup is required for end users. Nix is only an implementation detail for the maintainer host if that host already uses NixOS.

Packaged Publisher (recommended)

The flake exposes a self-contained publisher, so the host needs no checkout, nix develop, or cargo build at run time:

nix build github:0xrsydn/idx-cli#ownership-publisher
./result/bin/idx-ownership-publish --output-dir /var/lib/idx/ownership-snapshot/current --history 5
./result/bin/idx-ownership-freshness --max-age-days 40

The publisher needs GH_TOKEN in its environment: a token that can create and upload assets on the target repo's releases (the examples read it from /etc/idx-ownership-snapshot.env; clan-private uses a Clan var). The freshness check needs no token.

idx-ownership-publish wraps scripts/publish-ownership-snapshot.sh with a pinned idx (itself wrapped with curl-impersonate and mupdf) plus jq, gh, sqlite and curl. On NixOS, add this repo as a flake input and run inputs.idx-cli.packages.${system}.ownership-publisher; bump the input (nix flake update idx-cli) to pick up fixes.

Run Semantics

The publisher is idempotent and prints exactly one final line:

Last line	Exit	Meaning
RESULT: published <as-of>	0	a newer snapshot, or a larger history, was uploaded
RESULT: up-to-date <as-of>	0	the published snapshot already has the latest report and requested history; nothing uploaded
RESULT: FAILED stage=<stage> (exit N)	N	failed in arguments, build, preflight, discover, build-snapshot, stage-output, upload, or publish

It checks the published manifest and verifies that the manifest's download_url names this repo and release tag, and that the referenced SQLite asset matches the manifest SHA-256 and size, before it skips anything. It uses GitHub's sha256: digest metadata when available and downloads legacy assets without a digest to hash them. A missing or corrupt asset, a mismatched URL, an unreadable manifest, or a GitHub authentication/network error is treated as "publish", never as up to date. The upload path verifies the full SHA-256 of the uploaded asset before it publishes the manifest. For XLSX sources it also compares the requested --history coverage with the months the source actually exposes, so a history increase is published once and a request larger than the available history does not rebuild forever. --history is bounded (maximum 1000) so an enormous value cannot wrap and look already satisfied. The checks use discovery only, so a daily run stays cheap. --force uploads regardless.

The SQLite asset is immutable and content-addressed. The publisher uploads and verifies it before it uploads the manifest, which is the commit point consumers follow. A partial upload therefore exits non-zero and is repaired by the next run.

idx-ownership-freshness is the independent alarm: it exits 1 when the published latest_as_of_date is older than --max-age-days (default 40). The source is monthly and lands 2-3 days after month end, so 40 days only trips when publishing has actually stopped working. Wire both to an OnFailure= notification.

Manual Command (from a checkout)

nix develop --command scripts/publish-ownership-snapshot.sh \
  --build \
  --idx-bin ./target/debug/idx \
  --output-dir /var/lib/idx/ownership-snapshot/current \
  --repo 0xrsydn/idx-cli \
  --release-tag ownership-snapshot-current

systemd Units

Examples: contrib/systemd/idx-ownership-snapshot-publish.service and contrib/systemd/idx-ownership-snapshot-publish.timer (daily at 06:00 UTC).

Observed XLSX upload times (HTTP Last-Modified): 2026-06-03, 2026-07-02 03:55Z, 2026-08-02 01:50Z, 2026-09-02 03:51Z. With a daily, idempotent run the exact day no longer matters.

Pass --history <n> to include earlier months so idx ownership changes works straight from the synced snapshot (about 2.4 MB per extra month).

Snapshot size limit

idx clients up to v0.2.3 cap the snapshot download at 10 MiB (10,485,760 bytes) and only accept uncompressed SQLite. A larger snapshot makes idx ownership sync fail for every existing install. This happened on 2026-09-25 with --history 5 (15.3 MB).

--history	months	size (2026-08-31)
2	3	8,171,520
3	4	10,510,336 (too large)
5	6	15,290,368 (too large)

The builder therefore refuses snapshots above --max-snapshot-bytes (default 10485760). The run ends RESULT: FAILED stage=build-snapshot and the published snapshot stays unchanged. Newer clients stream downloads up to 1 GiB; raise --max-snapshot-bytes and --history only after clients without the 10 MiB cap are widespread.

Clan Integration Later

The clean split for clan-private is:

1. keep the workflow logic in this repo
2. keep host-specific secrets and enablement in clan-private

Recommended integration shape on greencloud-vps:

• store a checked-out copy of idx-cli on the host, for example at /srv/idx-cli
• provide GH_TOKEN through Clan vars or another secret mechanism
• add a small NixOS module that installs/enables the service and timer
• set the module's WorkingDirectory, env file path, and publish directory

The existing pattern in clan-private/modules/workspace-backup.nix is a good fit: a oneshot systemd service plus a systemd.timer.

Operational Notes

• Keep ownership-snapshot-current separate from versioned app releases.
• Prefer manual runs first, then enable the timer after a few successful publishes.
• If the monthly source has not appeared yet, the service should fail loudly and be retried later; do not silently publish stale assumptions.