Merge pull request #19 from 0xrsydn/codex/self-hosted-ownership-publish

Draft self-hosted ownership snapshot publish flow

contrib/systemd/idx-ownership-snapshot-publish.service

@@ -0,0 +1,11 @@
+[Unit]
+Description=Publish idx ownership snapshot to GitHub Releases
+Wants=network-online.target
+After=network-online.target
+
+[Service]
+Type=oneshot
+WorkingDirectory=/srv/idx-cli
+Environment=HOME=/var/lib/idx-ownership-snapshot
+EnvironmentFile=/etc/idx-ownership-snapshot.env
+ExecStart=/bin/sh -lc 'nix develop --command scripts/publish-ownership-snapshot.sh --build --idx-bin ./target/debug/idx --output-dir /var/lib/idx-ownership-snapshot/current --repo 0xrsydn/idx-cli --release-tag ownership-snapshot-current'

contrib/systemd/idx-ownership-snapshot-publish.timer

@@ -0,0 +1,11 @@
+[Unit]
+Description=Run monthly idx ownership snapshot publish job
+
+[Timer]
+OnCalendar=*-*-02 09:00:00
+Persistent=true
+RandomizedDelaySec=30m
+Unit=idx-ownership-snapshot-publish.service
+
+[Install]
+WantedBy=timers.target

docs/OWNERSHIP_PUBLISH.md

@@ -93,3 +93,18 @@ Current behavior:
 
 This is intentionally manual-first. Add a schedule only after a few successful
 publish runs confirm the live source remains stable enough.
+
+## Self-Hosted Automation
+
+GitHub-hosted Actions are still not sufficient for this job on their own. Live
+verification has shown IDX returning `403` to GitHub-hosted runners during
+discovery/import.
+
+The recommended unattended path is therefore:
+
+- keep the public artifacts on GitHub Releases
+- run the publish job from a self-hosted machine that IDX accepts
+- use a `systemd` oneshot service plus `systemd.timer` on that machine
+
+See `docs/OWNERSHIP_SELF_HOSTED.md` for the reusable helper script, sample
+systemd units, and the recommended split between this repo and `clan-private`.

docs/OWNERSHIP_SELF_HOSTED.md

@@ -0,0 +1,95 @@
+# Self-Hosted Ownership Snapshot Publish
+
+This document covers the recommended self-hosted automation path for publishing
+the monthly ownership snapshot consumed by `idx ownership sync`.
+
+Use this path when:
+
+- the repo's public release assets should stay on GitHub Releases
+- IDX blocks GitHub-hosted Actions with `403`
+- you have a VPS or other trusted machine that can reach the IDX source
+
+## Why Self-Hosted
+
+The current GitHub-hosted workflow remains useful as a manual reference, but it
+is not reliable enough for unattended publishing because IDX has returned `403`
+to GitHub-hosted runners. The self-hosted path keeps the end-user experience
+simple:
+
+- maintainers publish from a trusted machine
+- end users still run plain `idx ownership sync`
+
+No Nix setup is required for end users. Nix is only an implementation detail for
+the maintainer host if that host already uses NixOS.
+
+## Manual Command
+
+From a checked-out repo with GitHub auth already configured:
+
+```bash
+nix develop --command scripts/publish-ownership-snapshot.sh \
+  --build \
+  --idx-bin ./target/debug/idx \
+  --output-dir /var/lib/idx/ownership-snapshot/current \
+  --repo 0xrsydn/idx-cli \
+  --release-tag ownership-snapshot-current
+```
+
+That helper:
+
+1. optionally builds `idx`
+2. discovers the latest supported IDX/KSEI PDF
+3. imports it into an isolated temp DB
+4. emits the SQLite snapshot and manifest
+5. ensures the stable GitHub release exists
+6. uploads the manifest and SQLite asset with `--clobber`
+
+## systemd Service
+
+Example service: [contrib/systemd/idx-ownership-snapshot-publish.service](/Users/rasyidanakbar/Development/myApp/idx-cli/contrib/systemd/idx-ownership-snapshot-publish.service)
+
+Important assumptions:
+
+- the repo checkout lives at `/srv/idx-cli`
+- a writable publish directory exists at `/var/lib/idx-ownership-snapshot/current`
+- `GH_TOKEN` is provided via an env file such as `/etc/idx-ownership-snapshot.env`
+- the host can run `nix develop`
+
+## systemd Timer
+
+Example timer: [contrib/systemd/idx-ownership-snapshot-publish.timer](/Users/rasyidanakbar/Development/myApp/idx-cli/contrib/systemd/idx-ownership-snapshot-publish.timer)
+
+The sample timer uses:
+
+- `OnCalendar=*-*-02 09:00:00`
+- `Persistent=true`
+- `RandomizedDelaySec=30m`
+
+That is intentionally conservative. The ownership source is monthly, but the
+exact publish day can drift. Start with an early-month schedule and adjust after
+observing a few real runs.
+
+## Clan Integration Later
+
+The clean split for `clan-private` is:
+
+1. keep the workflow logic in this repo
+2. keep host-specific secrets and enablement in `clan-private`
+
+Recommended integration shape on `greencloud-vps`:
+
+- store a checked-out copy of `idx-cli` on the host, for example at `/srv/idx-cli`
+- provide `GH_TOKEN` through Clan vars or another secret mechanism
+- add a small NixOS module that installs/enables the service and timer
+- set the module's `WorkingDirectory`, env file path, and publish directory
+
+The existing pattern in `clan-private/modules/workspace-backup.nix` is a good
+fit: a oneshot `systemd` service plus a `systemd.timer`.
+
+## Operational Notes
+
+- Keep `ownership-snapshot-current` separate from versioned app releases.
+- Prefer manual runs first, then enable the timer after a few successful
+  publishes.
+- If the monthly source has not appeared yet, the service should fail loudly and
+  be retried later; do not silently publish stale assumptions.

scripts/publish-ownership-snapshot.sh

@@ -0,0 +1,182 @@
+#!/usr/bin/env bash
+
+set -euo pipefail
+
+usage() {
+    cat <<'EOF'
+Usage: scripts/publish-ownership-snapshot.sh --output-dir <dir> [options]
+
+Build and publish the latest supported IDX/KSEI ownership snapshot to the stable
+GitHub release used by `idx ownership sync`.
+
+Options:
+  --idx-bin <path>       idx binary to use (default: ./target/debug/idx)
+  --output-dir <dir>     Directory to write the copied snapshot and manifest
+  --repo <owner/name>    GitHub repo used for release upload and public URLs
+                         (default: 0xrsydn/idx-cli)
+  --release-tag <tag>    Stable GitHub release tag used for snapshot assets
+                         (default: ownership-snapshot-current)
+  --build                Run `cargo build` before publishing
+  --keep-workdir         Keep the temp workdir created by the builder helper
+  --help                 Show this help
+EOF
+}
+
+IDX_BIN="./target/debug/idx"
+OUTPUT_DIR=""
+REPO_FULL_NAME="0xrsydn/idx-cli"
+RELEASE_TAG="ownership-snapshot-current"
+BUILD_FIRST="0"
+KEEP_WORKDIR="0"
+PUBLISH_WORKDIR=""
+
+while [[ $# -gt 0 ]]; do
+    case "$1" in
+        --idx-bin)
+            IDX_BIN="${2:-}"
+            shift 2
+            ;;
+        --output-dir)
+            OUTPUT_DIR="${2:-}"
+            shift 2
+            ;;
+        --repo)
+            REPO_FULL_NAME="${2:-}"
+            shift 2
+            ;;
+        --release-tag)
+            RELEASE_TAG="${2:-}"
+            shift 2
+            ;;
+        --build)
+            BUILD_FIRST="1"
+            shift
+            ;;
+        --keep-workdir)
+            KEEP_WORKDIR="1"
+            shift
+            ;;
+        --help|-h)
+            usage
+            exit 0
+            ;;
+        *)
+            echo "unknown argument: $1" >&2
+            usage >&2
+            exit 2
+            ;;
+    esac
+done
+
+if [[ -z "$OUTPUT_DIR" ]]; then
+    echo "--output-dir is required" >&2
+    usage >&2
+    exit 2
+fi
+
+if ! command -v gh >/dev/null 2>&1; then
+    echo "gh is required for GitHub release upload" >&2
+    exit 1
+fi
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+BUILDER="$SCRIPT_DIR/build-latest-ownership-snapshot.sh"
+
+if [[ ! -x "$BUILDER" ]]; then
+    echo "required helper script is missing or not executable: $BUILDER" >&2
+    exit 1
+fi
+
+mkdir -p "$OUTPUT_DIR"
+
+PUBLISH_WORKDIR="$(mktemp -d "${TMPDIR:-/tmp}/idx-ownership-publish.XXXXXX")"
+cleanup() {
+    if [[ -z "$PUBLISH_WORKDIR" ]]; then
+        return
+    fi
+
+    if [[ "$KEEP_WORKDIR" == "1" ]]; then
+        printf 'Kept publish workdir: %s\n' "$PUBLISH_WORKDIR"
+        return
+    fi
+
+    rm -rf "$PUBLISH_WORKDIR"
+}
+trap cleanup EXIT
+
+if [[ "$BUILD_FIRST" == "1" ]]; then
+    printf 'Building idx...\n'
+    cargo build
+fi
+
+if ! "$IDX_BIN" version >/dev/null 2>&1; then
+    echo "failed to run idx binary: $IDX_BIN" >&2
+    echo "build the CLI first or pass --idx-bin <path>" >&2
+    exit 1
+fi
+
+build_args=(
+    --idx-bin "$IDX_BIN"
+    --output-dir "$PUBLISH_WORKDIR"
+    --repo "$REPO_FULL_NAME"
+    --release-tag "$RELEASE_TAG"
+)
+
+if [[ "$KEEP_WORKDIR" == "1" ]]; then
+    build_args+=(--keep-workdir)
+fi
+
+printf 'Preparing latest ownership snapshot artifacts...\n'
+"$BUILDER" "${build_args[@]}"
+
+STAGED_MANIFEST_PATH="$PUBLISH_WORKDIR/ownership-snapshot-manifest.json"
+if [[ ! -f "$STAGED_MANIFEST_PATH" ]]; then
+    echo "manifest was not generated: $STAGED_MANIFEST_PATH" >&2
+    exit 1
+fi
+
+shopt -s nullglob
+sqlite_matches=("$PUBLISH_WORKDIR"/ownership-snapshot-*.sqlite)
+existing_snapshot_paths=("$OUTPUT_DIR"/ownership-snapshot-*.sqlite)
+shopt -u nullglob
+
+if [[ "${#sqlite_matches[@]}" -ne 1 ]]; then
+    echo "expected exactly one SQLite artifact in $PUBLISH_WORKDIR" >&2
+    exit 1
+fi
+
+STAGED_SQLITE_PATH="${sqlite_matches[0]}"
+
+rm -f "$OUTPUT_DIR/ownership-snapshot-manifest.json"
+if [[ "${#existing_snapshot_paths[@]}" -gt 0 ]]; then
+    rm -f "${existing_snapshot_paths[@]}"
+fi
+
+cp "$STAGED_MANIFEST_PATH" "$OUTPUT_DIR/ownership-snapshot-manifest.json"
+cp "$STAGED_SQLITE_PATH" "$OUTPUT_DIR/"
+
+MANIFEST_PATH="$OUTPUT_DIR/ownership-snapshot-manifest.json"
+SQLITE_PATH="$OUTPUT_DIR/$(basename "$STAGED_SQLITE_PATH")"
+
+if gh release view "$RELEASE_TAG" --repo "$REPO_FULL_NAME" >/dev/null 2>&1; then
+    printf 'Release %s already exists.\n' "$RELEASE_TAG"
+else
+    printf 'Creating stable snapshot release %s...\n' "$RELEASE_TAG"
+    gh release create "$RELEASE_TAG" \
+        --repo "$REPO_FULL_NAME" \
+        --target "$(git rev-parse HEAD)" \
+        --title "Ownership Snapshot Current" \
+        --notes "Stable release for idx ownership snapshot artifacts consumed by \`idx ownership sync\`." \
+        --latest=false
+fi
+
+printf 'Uploading snapshot assets to %s...\n' "$RELEASE_TAG"
+gh release upload "$RELEASE_TAG" \
+    "$MANIFEST_PATH" \
+    "$SQLITE_PATH" \
+    --repo "$REPO_FULL_NAME" \
+    --clobber
+
+printf 'Published manifest: https://github.com/%s/releases/download/%s/ownership-snapshot-manifest.json\n' \
+    "$REPO_FULL_NAME" "$RELEASE_TAG"
+printf 'Published SQLite: %s\n' "$SQLITE_PATH"