fix(ownership): verify snapshot assets before publishing manifests

This commit is contained in:
hermes 2026-09-25 10:05:07 +00:00
commit 9f4ed5b4a5
11 changed files with 1129 additions and 116 deletions

View file

@ -162,6 +162,14 @@
- [ ] Additional providers (Alpha Vantage, Twelve Data, IDX official)
## 🔬 Latest Smoke Findings (2026-04-02)
- [x] Publish hardening batch on `2026-09-25`: `scripts/publish-ownership-snapshot.sh` now uploads an immutable content-addressed SQLite asset (date + sha256) and verifies it before publishing the manifest as the final commit point, so a partial upload no longer leaves a manifest that points at a missing asset; the manifest is trusted for a no-op only when the referenced asset matches the manifest SHA-256 and size at the configured repo/tag, and GitHub auth/network errors never become "up-to-date"
- [x] Publish integrity follow-up on `2026-09-25`: no-op and upload verification now check the full SHA-256, not just the filename and size; the publisher uses GitHub's `sha256:` asset digest when present and falls back to `gh release download` + local hashing for legacy assets, rejects a `download_url` that names a different repo/tag, and refuses to publish the manifest when the uploaded asset's SHA-256 does not match
- [x] `--history` is bounded (maximum `1000`, leading zeros stripped before conversion) so an enormous value cannot wrap negative and look already satisfied
- [x] The GitHub publish workflow now builds the packaged `idx-ownership-publisher` and routes publication through it with inputs passed as environment variables; the workflow artifact upload uses `if-no-files-found: ignore` so a legitimate up-to-date run does not fail
- [x] `--keep-workdir` no longer prints after the `RESULT:` line: the retained path is announced when the workdir is created and the EXIT cleanup is silent
- [x] `--history` changes are now honored for same-date data: the publisher compares `1 + min(--history, available XLSX months)` against the published `release_count` before and after building, validates `--history` before early exits, and does not rebuild on every run when the source has fewer historical months than requested
- [x] New offline regression harness `scripts/publish-ownership-snapshot-test.sh` (wired into `.github/workflows/ci.yml`) covers successful publish/no-op, partial SQLite upload and retry, manifest upload failure and retry, forced same-date refresh preserving the previous artifact, a missing referenced asset forcing republish, same-size remote corruption with and without digest metadata, upload corruption rejected before the manifest commit (digest and digest-less), a manifest URL for a different repo/tag, legacy PDF post-build no-op, history upgrade, limited available history, `--history` bounds, `--keep-workdir` output ordering, and auth/network failure not becoming up-to-date
- [x] Verification on `2026-09-25`: `bash -n` on the changed shell scripts and `scripts/publish-ownership-snapshot-test.sh` all passed
- [x] `ownership sync` now falls back to the built-in published manifest URL on `2026-04-12`, so plain `idx ownership sync` works on a clean machine once the public snapshot release is available
- [x] Ownership snapshot publishing now has a live-source maintainer path on `2026-04-11`: `scripts/build-latest-ownership-snapshot.sh` discovers the current supported `above1` IDX/KSEI PDF, imports it into an isolated temp DB, emits release-ready SQLite + manifest artifacts, and records the source IDX metadata in the manifest
- [x] Verification on `2026-04-11`: `nix develop --command cargo test`, `bash -n scripts/build-latest-ownership-snapshot.sh`, and a real `nix develop --command scripts/build-latest-ownership-snapshot.sh --idx-bin ./target/debug/idx --output-dir tmp/ownership-snapshot-live` run all passed; the current live source produced `7214` rows for `955` tickers with `as_of_date=2026-03-31`