mirror of
https://github.com/0xrsydn/idx-cli.git
synced 2026-11-07 01:35:36 +00:00
docs: prune stale planning docs and lean out agent guide
- Remove FEATURE_SPEC.md, TODO.md, and docs/AGENT_INTEGRATION.md; their core gaps have shipped. Open work moves to a short ROADMAP.md. - Remove the CLAUDE.md symlink and rewrite AGENTS.md as a quick guide: navigation, rules, and the nix flake check gate. - Merge OWNERSHIP_SELF_HOSTED.md into OWNERSHIP_PUBLISH.md and drop private host integration notes. - ARCHITECTURE: add MSN endpoints, clarify Bing ingest is unsupported. - OWNERSHIP_SYNC: note source.pdf_url also carries XLSX URLs. - systemd example: --history 2 so snapshots stay under the 10 MiB client cap. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
hermes
parent
cd410708c0
commit
1631058b8c
15 changed files with 206 additions and 1033 deletions
|
|
@ -1,149 +0,0 @@
|
|||
# Agent Integration Roadmap
|
||||
|
||||
**Status:** Planning — prerequisites are the remaining P0/P1 core gaps in `FEATURE_SPEC.md`
|
||||
**Goal:** Make `idx-cli` a first-class tool for AI agent workflows (Claude Code, OpenClaw, MCP clients)
|
||||
|
||||
---
|
||||
|
||||
## Current Foundation
|
||||
|
||||
The CLI already has the building blocks:
|
||||
|
||||
- Structured JSON output (`-o json`) on every command
|
||||
- Trait-based provider abstraction (mockable, extensible)
|
||||
- Rich typed domain models (`Quote`, `Fundamentals`, `CompanyProfile`, `OwnershipHolder`, etc.)
|
||||
- Structured error codes in JSON mode
|
||||
- File-based cache with TTL, offline, and stale-cache fallback
|
||||
- Feature-gated modules (ownership behind `--features ownership`)
|
||||
|
||||
---
|
||||
|
||||
## Tier 1 — MCP Server Mode (highest leverage)
|
||||
|
||||
Add `idx mcp-serve` to expose existing commands as MCP tools over stdio.
|
||||
|
||||
- Use `rust-mcp-sdk` crate for the stdio JSON-RPC transport
|
||||
- Each existing command maps to one MCP tool (e.g. `get_stock_quote`, `get_financials`, `screen_stocks`, `get_ownership_ticker`)
|
||||
- Existing JSON serialization becomes the tool response body — minimal new code
|
||||
- Tool input schemas derive from clap structs or parallel types
|
||||
- Configure in Claude Code:
|
||||
```json
|
||||
{ "mcpServers": { "idx": { "command": "idx", "args": ["mcp-serve"] } } }
|
||||
```
|
||||
|
||||
**Why this first:** One feature makes `idx-cli` instantly usable from Claude Code, Claude Desktop, OpenClaw, and any MCP-compatible client. No wrapper scripts needed.
|
||||
|
||||
**Reference:** `financial-datasets/mcp-server` (1.7k stars), `tradingview-mcp` (500 stars), `twsemcp` (Taiwan Stock Exchange MCP — closest regional analog), `rust-mcp-sdk` (Rust MCP crate).
|
||||
|
||||
---
|
||||
|
||||
## Tier 2 — Composability
|
||||
|
||||
### Stdin symbol input
|
||||
|
||||
```bash
|
||||
idx stocks screen --filter pe_lt:15 -o json | jq -r '.[].symbol' | idx stocks fundamental --stdin -o json
|
||||
```
|
||||
|
||||
Agents generate symbol lists dynamically. Piping enables chained workflows without temp files.
|
||||
|
||||
### JSONL output
|
||||
|
||||
```bash
|
||||
idx -o jsonl stocks quote BBCA BBRI
|
||||
```
|
||||
|
||||
Newline-delimited JSON — better for streaming and piping than pretty-printed arrays.
|
||||
|
||||
### Batch mode
|
||||
|
||||
```bash
|
||||
idx batch commands.jsonl
|
||||
```
|
||||
|
||||
Where each line is a command spec:
|
||||
```json
|
||||
{"command": "stocks/quote", "symbols": ["BBCA", "BBRI"]}
|
||||
{"command": "ownership/ticker", "symbol": "BBCA", "source": "ksei"}
|
||||
```
|
||||
|
||||
Reduces process spawn overhead when agents run many queries in sequence.
|
||||
|
||||
---
|
||||
|
||||
## Tier 3 — Schema Export
|
||||
|
||||
```bash
|
||||
idx schema stocks/quote
|
||||
```
|
||||
|
||||
Dumps the JSON Schema for a command's output. Agents use this to understand available fields without trial-and-error. Document which fields are guaranteed present vs. optional.
|
||||
|
||||
---
|
||||
|
||||
## Tier 4 — Agent-Oriented Domain Features
|
||||
|
||||
### Watchlist as named symbol sets
|
||||
|
||||
```bash
|
||||
idx watchlist create banks BBCA,BBRI,BMRI,BNGA
|
||||
idx stocks quote --watchlist banks
|
||||
```
|
||||
|
||||
Agents build and reuse curated lists across sessions.
|
||||
|
||||
### Composite analysis
|
||||
|
||||
```bash
|
||||
idx stocks analyze BBCA -o json
|
||||
```
|
||||
|
||||
Returns quote + fundamentals + ownership + sentiment in one call. Saves agents 4+ tool calls and reduces token overhead.
|
||||
|
||||
### Diff / change detection
|
||||
|
||||
```bash
|
||||
idx stocks quote BBCA --diff
|
||||
```
|
||||
|
||||
Compares against cached previous values and surfaces deltas. Agents monitor without maintaining their own state.
|
||||
|
||||
### CSV/TSV export
|
||||
|
||||
```bash
|
||||
idx -o csv stocks screen --filter pe_lt:15
|
||||
```
|
||||
|
||||
For agent pipelines that need tabular data (pandas, spreadsheets, database import).
|
||||
|
||||
---
|
||||
|
||||
## Tier 5 — Integration Hooks
|
||||
|
||||
- **Webhook primitives** — `idx watch BBCA --threshold price_gt:5000 --webhook http://...` for event-driven agent architectures
|
||||
- **Session context** — `idx context set research-session-1` to tag cache entries, enabling isolated research sessions that don't pollute each other
|
||||
|
||||
---
|
||||
|
||||
## Execution Order
|
||||
|
||||
| Priority | Feature | Effort | Impact |
|
||||
| --- | --- | --- | --- |
|
||||
| 1 | MCP server mode | Medium | Unlocks all MCP clients |
|
||||
| 2 | Stdin piping + JSONL | Small | Composability for chaining |
|
||||
| 3 | Schema export | Small | Agent self-discovery |
|
||||
| 4 | Watchlist + composite | Medium | Reduces agent round-trips |
|
||||
| 5 | CSV export | Small | Tabular pipeline support |
|
||||
| 6 | Webhooks + sessions | Large | Event-driven architecture |
|
||||
|
||||
---
|
||||
|
||||
## Prerequisites
|
||||
|
||||
Close the remaining core gaps first (`FEATURE_SPEC.md` P0/P1):
|
||||
|
||||
1. Unify provider and capability flow
|
||||
2. Fix screener row hygiene
|
||||
3. Decide fundamentals fallback policy
|
||||
4. Harden Yahoo reliability edge cases
|
||||
5. Add `financials` and `earnings` filter flags
|
||||
|
|
@ -34,6 +34,13 @@
|
|||
- **MSN Finance** — default provider. Rich data: quotes, fundamentals, profile, earnings, financials, sentiment, insights, news, screener, and explicit price-only chart history for supported IDX windows.
|
||||
- **Yahoo Finance** — default/auto history source. Reliable OHLCV data via `/v8/finance/chart/`.
|
||||
|
||||
### MSN Endpoints
|
||||
- `https://assets.msn.com/service/`: quotes, charts, equities, earnings,
|
||||
sentiment, screener
|
||||
- `https://api.msn.com/msn/v0/pages/finance/`: key ratios, insights, news feed
|
||||
|
||||
The public API key embedded in MSN Money lives in `src/api/msn/client.rs`.
|
||||
|
||||
### Hybrid History Strategy
|
||||
When `history_provider = auto` (default):
|
||||
1. Use Yahoo for history because it provides full OHLCV candles.
|
||||
|
|
@ -73,7 +80,7 @@ Factory functions:
|
|||
|
||||
Unlike the `stocks` module (live-fetch), ownership is **import-then-query**:
|
||||
|
||||
1. `idx ownership import` — ETL pipeline: fetch PDF/API → parse → normalize → load SQLite
|
||||
1. `idx ownership import` — ETL pipeline: fetch XLSX/PDF → parse → normalize → load SQLite
|
||||
2. `idx ownership sync` — install a maintained SQLite snapshot via manifest + checksum validation
|
||||
3. All query commands read from local `~/.local/share/idx/ownership.db`
|
||||
4. Fully offline after import/sync
|
||||
|
|
@ -81,7 +88,9 @@ Unlike the `stocks` module (live-fetch), ownership is **import-then-query**:
|
|||
### Data Sources
|
||||
- **KSEI** — official ≥1% shareholder registry, published monthly by IDX. Through May 2026 it was a PDF announcement; from the 2026-05-29 report onward it is an XLSX workbook on the IDX "Data Kepemilikan Saham" page
|
||||
- **KSEI archive** — monthly ZIP/TXT balance-position matrix, used as a local fallback/backstop import path
|
||||
- **Bing Finance** — global institutional ownership (REST API, quarterly)
|
||||
- **Bing Finance** — global institutional ownership. The schema, `--source bing`
|
||||
queries, and `src/api/msn/bing.rs` exist, but `ownership import --fetch-bing`
|
||||
is intentionally unsupported, so no ingest path populates Bing rows today
|
||||
|
||||
### Discovery
|
||||
`idx ownership discover` merges two IDX sources:
|
||||
|
|
@ -123,7 +132,7 @@ CLI args → resolve symbol → provider.quote/history/fundamentals → render t
|
|||
|
||||
### Import-Query (ownership module)
|
||||
```
|
||||
Import: PDF/API → parse → normalize → resolve entities → SQLite INSERT
|
||||
Import: XLSX/PDF/archive → parse → normalize → resolve entities → SQLite INSERT
|
||||
Query: CLI args → SQLite SELECT → render table/json (no network)
|
||||
```
|
||||
|
||||
|
|
|
|||
|
|
@ -126,7 +126,6 @@ let conn = Connection::open(path)
|
|||
- **Coverage over counts:** replacing redundant unit assertions with stronger integration scenarios must retain meaningful error, boundary, and data-integrity coverage.
|
||||
|
||||
## Git / VCS
|
||||
- **jj (Jujutsu)** as local workflow, colocated with git
|
||||
- Push via `nix develop --command git push` (for prek hooks)
|
||||
- Branch naming: `feat/<name>`, `fix/<name>`
|
||||
- Branch naming: `feat/<name>`, `fix/<name>`, `docs/<name>`
|
||||
- Commit messages: conventional commits (`feat:`, `fix:`, `refactor:`, `docs:`)
|
||||
|
|
|
|||
|
|
@ -1,170 +1,148 @@
|
|||
# Ownership Snapshot Publishing
|
||||
|
||||
This document covers the maintainer workflow for producing and publishing the
|
||||
ownership snapshot artifacts consumed by `idx ownership sync`.
|
||||
Maintainer workflow for producing and publishing the snapshot that
|
||||
`idx ownership sync` consumes. The consumer contract (manifest schema and
|
||||
replacement rules) is in `docs/OWNERSHIP_SYNC.md`.
|
||||
|
||||
## Goals
|
||||
## Release Layout
|
||||
|
||||
- Build the snapshot from the currently discoverable supported IDX/KSEI source,
|
||||
not from an ad hoc local DB.
|
||||
- Keep the published manifest URL stable for end users.
|
||||
- Record source provenance in the manifest so the published SQLite artifact can
|
||||
be traced back to the live IDX announcement and PDF URL used to build it.
|
||||
|
||||
## Recommended GitHub Releases Layout
|
||||
|
||||
Use a dedicated stable release tag for snapshot assets:
|
||||
Snapshot assets live on a dedicated GitHub release, separate from versioned app
|
||||
releases:
|
||||
|
||||
- tag: `ownership-snapshot-current`
|
||||
- manifest asset: `ownership-snapshot-manifest.json`
|
||||
- SQLite asset: `ownership-snapshot-YYYY-MM-DD-<sha256>.sqlite` (immutable, content-addressed)
|
||||
|
||||
Recommended public manifest URL:
|
||||
- manifest: `ownership-snapshot-manifest.json` (stable URL, the commit point)
|
||||
- SQLite: `ownership-snapshot-YYYY-MM-DD-<sha256>.sqlite` (immutable, content-addressed)
|
||||
|
||||
```text
|
||||
https://github.com/0xrsydn/idx-cli/releases/download/ownership-snapshot-current/ownership-snapshot-manifest.json
|
||||
```
|
||||
|
||||
Avoid `.../releases/latest/download/...` if normal app releases and snapshot
|
||||
publishes share the same repository. The repo's "latest" release can drift away
|
||||
from the ownership snapshot release.
|
||||
Never use `.../releases/latest/download/...`: the repo's latest release is an
|
||||
app release, not the snapshot.
|
||||
|
||||
## Manual Maintainer Flow
|
||||
## Running the Publisher
|
||||
|
||||
Run the publisher helper inside `nix develop` so `mutool` and the
|
||||
`curl-impersonate` helper are available:
|
||||
Packaged (recommended for hosts; no checkout or cargo build needed):
|
||||
|
||||
```bash
|
||||
nix build github:0xrsydn/idx-cli#ownership-publisher
|
||||
./result/bin/idx-ownership-publish --output-dir /var/lib/idx/ownership-snapshot/current --history 2
|
||||
./result/bin/idx-ownership-freshness --max-age-days 40
|
||||
```
|
||||
|
||||
From a checkout:
|
||||
|
||||
```bash
|
||||
nix develop --command cargo build
|
||||
nix develop --command scripts/publish-ownership-snapshot.sh \
|
||||
--build \
|
||||
--idx-bin ./target/debug/idx \
|
||||
--output-dir dist/ownership-snapshot \
|
||||
--repo 0xrsydn/idx-cli \
|
||||
--release-tag ownership-snapshot-current
|
||||
```
|
||||
|
||||
The helper performs these steps:
|
||||
`idx-ownership-publish` wraps `scripts/publish-ownership-snapshot.sh` with a
|
||||
pinned `idx` (including curl-impersonate and mupdf) plus jq, gh, sqlite, and
|
||||
curl. It needs `GH_TOKEN` with permission to upload release assets; the freshness
|
||||
check needs no token.
|
||||
|
||||
## What It Does
|
||||
|
||||
1. `idx -o json ownership discover --family above1 --limit 50` (newest supported
|
||||
report first; XLSX from the Data Kepemilikan Saham page since June 2026)
|
||||
2. verifies the discovered report is the current supported import path
|
||||
3. imports that report (plus `--history <n>` earlier months, oldest first) into
|
||||
an isolated temp ownership DB
|
||||
4. checks that the imported release metadata is non-empty and tied to the same
|
||||
source URL
|
||||
5. runs `scripts/build-ownership-snapshot.sh` to emit the SQLite artifact and
|
||||
base manifest
|
||||
6. enriches the manifest with `source` provenance metadata
|
||||
|
||||
The resulting manifest records:
|
||||
|
||||
- `source.family`
|
||||
- `source.listing_page_url`
|
||||
- `source.query_url`
|
||||
- `source.pdf_url`
|
||||
- `source.title`
|
||||
- `source.publish_date`
|
||||
- `source.original_filename`
|
||||
|
||||
That metadata is additive. Existing sync clients can still parse the manifest.
|
||||
2. verifies the discovered report is the supported import path
|
||||
3. imports it plus `--history <n>` earlier months, oldest first, into an
|
||||
isolated temp DB
|
||||
4. checks the imported release metadata is non-empty and tied to the source URL
|
||||
5. builds the SQLite artifact and manifest (`scripts/build-latest-ownership-snapshot.sh`,
|
||||
which calls `scripts/build-ownership-snapshot.sh`) and records `source`
|
||||
provenance in the manifest
|
||||
6. uploads using the protocol below
|
||||
|
||||
## Publication Protocol
|
||||
|
||||
The manifest is the commit point that `idx ownership sync` follows. The
|
||||
publisher therefore never uploads the manifest and the SQLite asset in one
|
||||
non-atomic call. It uses this order:
|
||||
The manifest is the commit point, so uploads are ordered:
|
||||
|
||||
1. Build the snapshot and stage the manifest plus one content-addressed SQLite
|
||||
file.
|
||||
2. Upload the SQLite asset first. The name includes the artifact's SHA-256, so
|
||||
the same content always has the same name and a retry is idempotent.
|
||||
3. Read the release assets back and confirm the SQLite asset exists with the
|
||||
expected SHA-256 and size. Use GitHub's `sha256:` digest metadata when it is
|
||||
present; for legacy assets without a digest, download the asset through `gh`
|
||||
and hash it. Fail before touching the manifest if verification fails.
|
||||
4. Upload the manifest last and verify that its `snapshot.sqlite_sha256`
|
||||
matches the staged manifest.
|
||||
1. Upload the content-addressed SQLite asset. Same content means same name, so
|
||||
retries are idempotent.
|
||||
2. Read the release assets back and verify the full SHA-256 and size (GitHub's
|
||||
`sha256:` digest when present, otherwise download and hash). Fail before
|
||||
touching the manifest if this does not match.
|
||||
3. Upload the manifest last and verify its `snapshot.sqlite_sha256`.
|
||||
|
||||
Consequences:
|
||||
A partial upload exits non-zero and is repaired by the next run; previously
|
||||
published manifests stay valid because old SQLite assets are retained. GitHub
|
||||
does not replace the manifest transactionally, so a failed replacement can leave
|
||||
it unavailable until a retry succeeds.
|
||||
|
||||
- A partial upload (manifest uploaded, SQLite missing) can no longer happen.
|
||||
- If either upload fails, the job exits non-zero and the next run retries.
|
||||
Previously downloaded manifests remain valid because the old SQLite asset is retained.
|
||||
- GitHub does not replace the manifest transactionally. A failed replacement
|
||||
can leave the manifest unavailable until a retry succeeds.
|
||||
- The publisher treats a published manifest as current only when `download_url`
|
||||
names this repo and release tag and the referenced asset matches the manifest
|
||||
SHA-256 and size. A missing or corrupt asset, a mismatched URL, or any GitHub
|
||||
authentication/network error means "publish", never "up to date".
|
||||
The publisher treats the published snapshot as current only when the manifest's
|
||||
`download_url` names this repo and tag and the referenced asset matches its
|
||||
SHA-256 and size. A missing or corrupt asset, a mismatched URL, an unreadable
|
||||
manifest, or any GitHub auth/network error means "publish", never "up to date".
|
||||
`--force` uploads regardless.
|
||||
|
||||
## Run Result
|
||||
|
||||
The publisher is idempotent and prints exactly one final line:
|
||||
|
||||
| Last line | Exit | Meaning |
|
||||
| --- | --- | --- |
|
||||
| `RESULT: published <as-of>` | 0 | a newer snapshot, or larger history, was uploaded |
|
||||
| `RESULT: up-to-date <as-of>` | 0 | nothing to upload |
|
||||
| `RESULT: FAILED stage=<stage> (exit N)` | N | failed in `arguments`, `build`, `preflight`, `discover`, `build-snapshot`, `stage-output`, `upload`, or `publish` |
|
||||
|
||||
`idx-ownership-freshness` is the independent alarm: it exits 1 when the published
|
||||
`latest_as_of_date` is older than `--max-age-days` (default 40). The source lands
|
||||
2-3 days after month end, so 40 days only trips when publishing has stopped.
|
||||
|
||||
## History Coverage
|
||||
|
||||
`--history <n>` requests the latest report plus up to `<n>` earlier monthly
|
||||
XLSX reports. The publisher compares the request with the months the source
|
||||
actually exposes:
|
||||
`--history <n>` (maximum 1000) requests the latest report plus up to `n` earlier
|
||||
monthly XLSX reports:
|
||||
|
||||
- desired releases = `1 + min(--history, available earlier XLSX months)`
|
||||
- a published snapshot with the same as-of date and at least the desired number
|
||||
of releases is a no-op
|
||||
- a published snapshot with the same as-of date but fewer releases is
|
||||
republished to add history
|
||||
- when the source has fewer historical months than requested, the publisher
|
||||
accepts the smaller coverage and does not rebuild on every run
|
||||
- same as-of date with at least the desired release count: no-op
|
||||
- same as-of date with fewer releases: republish to add history
|
||||
- a request larger than the source's history is accepted once, not rebuilt forever
|
||||
|
||||
## Upload Step
|
||||
Each extra month adds about 2.4 MB, and history is what lets
|
||||
`idx ownership changes` work straight from a synced snapshot.
|
||||
|
||||
Prefer the publisher helper, which implements the protocol above:
|
||||
## Snapshot Size Limit
|
||||
|
||||
```bash
|
||||
scripts/publish-ownership-snapshot.sh \
|
||||
--idx-bin ./target/debug/idx \
|
||||
--output-dir dist/ownership-snapshot \
|
||||
--repo 0xrsydn/idx-cli \
|
||||
--release-tag ownership-snapshot-current
|
||||
```
|
||||
idx clients up to v0.2.3 cap the snapshot download at **10 MiB** (10,485,760
|
||||
bytes) and accept only uncompressed SQLite. A larger snapshot breaks
|
||||
`idx ownership sync` for every existing install (this happened on 2026-09-25
|
||||
with `--history 5`).
|
||||
|
||||
If you upload by hand, upload these files to the `ownership-snapshot-current`
|
||||
GitHub release in this order:
|
||||
| `--history` | months | size (2026-08-31) |
|
||||
| --- | --- | --- |
|
||||
| 2 | 3 | 8,171,520 |
|
||||
| 3 | 4 | 10,510,336 (too large) |
|
||||
| 5 | 6 | 15,290,368 (too large) |
|
||||
|
||||
1. `dist/ownership-snapshot/ownership-snapshot-YYYY-MM-DD-<sha256>.sqlite`
|
||||
2. `dist/ownership-snapshot/ownership-snapshot-manifest.json`
|
||||
The builder refuses snapshots above `--max-snapshot-bytes` (default 10485760),
|
||||
ending with `RESULT: FAILED stage=build-snapshot` and leaving the published
|
||||
snapshot unchanged. Newer clients accept up to the manifest's `size_bytes` (at
|
||||
most 1 GiB); raise both limits only once pre-v0.2.4 clients are rare.
|
||||
|
||||
Never upload the manifest before its SQLite asset is present. Only after this
|
||||
manual flow is reliable should the repo automate it in GitHub Actions.
|
||||
## Automation
|
||||
|
||||
## GitHub Actions Workflow
|
||||
IDX has returned `403` to GitHub-hosted runners, so unattended publishing runs
|
||||
on a self-hosted machine that IDX accepts:
|
||||
|
||||
The repo now includes a manual workflow at
|
||||
`.github/workflows/publish-ownership-snapshot.yml`.
|
||||
- `contrib/systemd/idx-ownership-snapshot-publish.service` and `.timer` run the
|
||||
packaged publisher daily at 06:00 UTC with `GH_TOKEN` from
|
||||
`/etc/idx-ownership-snapshot.env`. Wire `OnFailure=` to a notification.
|
||||
- On NixOS, add this repo as a flake input and use
|
||||
`inputs.idx-cli.packages.${system}.ownership-publisher`; `nix flake update idx-cli`
|
||||
picks up fixes.
|
||||
|
||||
Current behavior:
|
||||
`.github/workflows/publish-ownership-snapshot.yml` is a manual
|
||||
(`workflow_dispatch`) fallback that runs the same packaged publisher with inputs
|
||||
passed as environment variables.
|
||||
|
||||
- trigger: `workflow_dispatch` only
|
||||
- builds the packaged publisher with `nix build .#ownership-publisher`
|
||||
- runs `idx-ownership-publish` from that package, which performs discovery,
|
||||
import, snapshot build, and the full upload protocol in one pass
|
||||
- uploads any locally produced artifacts as a workflow artifact copy with
|
||||
`if-no-files-found: ignore`, so a legitimate up-to-date run still succeeds
|
||||
- does not run a separate release-create or asset-upload step
|
||||
|
||||
This is intentionally manual-first. Add a schedule only after a few successful
|
||||
publish runs confirm the live source remains stable enough.
|
||||
|
||||
The workflow passes its inputs to the shell through environment variables and
|
||||
routes publication through the same fixed publisher used by self-hosted hosts,
|
||||
so it cannot drift from the CLI protocol.
|
||||
|
||||
## Self-Hosted Automation
|
||||
|
||||
GitHub-hosted Actions are still not sufficient for this job on their own. Live
|
||||
verification has shown IDX returning `403` to GitHub-hosted runners during
|
||||
discovery/import.
|
||||
|
||||
The recommended unattended path is therefore:
|
||||
|
||||
- keep the public artifacts on GitHub Releases
|
||||
- run the publish job from a self-hosted machine that IDX accepts
|
||||
- use a `systemd` oneshot service plus `systemd.timer` on that machine
|
||||
|
||||
See `docs/OWNERSHIP_SELF_HOSTED.md` for the reusable helper script, sample
|
||||
systemd units, and the recommended split between this repo and `clan-private`.
|
||||
Observed XLSX upload times (HTTP Last-Modified) were the 2nd-3rd of each month
|
||||
around 02:00-04:00 UTC; a daily idempotent run makes the exact day irrelevant.
|
||||
If the monthly source has not appeared yet, the run should fail loudly and retry
|
||||
later rather than publish stale data.
|
||||
|
|
|
|||
|
|
@ -1,149 +0,0 @@
|
|||
# Self-Hosted Ownership Snapshot Publish
|
||||
|
||||
This document covers the recommended self-hosted automation path for publishing
|
||||
the monthly ownership snapshot consumed by `idx ownership sync`.
|
||||
|
||||
Use this path when:
|
||||
|
||||
- the repo's public release assets should stay on GitHub Releases
|
||||
- IDX blocks GitHub-hosted Actions with `403`
|
||||
- you have a VPS or other trusted machine that can reach the IDX source
|
||||
|
||||
## Why Self-Hosted
|
||||
|
||||
The current GitHub-hosted workflow remains useful as a manual reference, but it
|
||||
is not reliable enough for unattended publishing because IDX has returned `403`
|
||||
to GitHub-hosted runners. The self-hosted path keeps the end-user experience
|
||||
simple:
|
||||
|
||||
- maintainers publish from a trusted machine
|
||||
- end users still run plain `idx ownership sync`
|
||||
|
||||
No Nix setup is required for end users. Nix is only an implementation detail for
|
||||
the maintainer host if that host already uses NixOS.
|
||||
|
||||
## Packaged Publisher (recommended)
|
||||
|
||||
The flake exposes a self-contained publisher, so the host needs no checkout,
|
||||
`nix develop`, or cargo build at run time:
|
||||
|
||||
```bash
|
||||
nix build github:0xrsydn/idx-cli#ownership-publisher
|
||||
./result/bin/idx-ownership-publish --output-dir /var/lib/idx/ownership-snapshot/current --history 5
|
||||
./result/bin/idx-ownership-freshness --max-age-days 40
|
||||
```
|
||||
|
||||
The publisher needs `GH_TOKEN` in its environment: a token that can create
|
||||
and upload assets on the target repo's releases (the examples read it from
|
||||
`/etc/idx-ownership-snapshot.env`; clan-private uses a Clan var). The
|
||||
freshness check needs no token.
|
||||
|
||||
`idx-ownership-publish` wraps `scripts/publish-ownership-snapshot.sh` with a
|
||||
pinned `idx` (itself wrapped with curl-impersonate and mupdf) plus jq, gh,
|
||||
sqlite and curl. On NixOS, add this repo as a flake input and run
|
||||
`inputs.idx-cli.packages.${system}.ownership-publisher`; bump the input
|
||||
(`nix flake update idx-cli`) to pick up fixes.
|
||||
|
||||
## Run Semantics
|
||||
|
||||
The publisher is idempotent and prints exactly one final line:
|
||||
|
||||
| Last line | Exit | Meaning |
|
||||
| --- | --- | --- |
|
||||
| `RESULT: published <as-of>` | 0 | a newer snapshot, or a larger history, was uploaded |
|
||||
| `RESULT: up-to-date <as-of>` | 0 | the published snapshot already has the latest report and requested history; nothing uploaded |
|
||||
| `RESULT: FAILED stage=<stage> (exit N)` | N | failed in `arguments`, `build`, `preflight`, `discover`, `build-snapshot`, `stage-output`, `upload`, or `publish` |
|
||||
|
||||
It checks the published manifest and verifies that the manifest's `download_url`
|
||||
names this repo and release tag, and that the referenced SQLite asset matches
|
||||
the manifest SHA-256 and size, before it skips anything. It uses GitHub's
|
||||
`sha256:` digest metadata when available and downloads legacy assets without a
|
||||
digest to hash them. A missing or corrupt asset, a mismatched URL, an unreadable
|
||||
manifest, or a GitHub authentication/network error is treated as "publish",
|
||||
never as up to date. The upload path verifies the full SHA-256 of the uploaded
|
||||
asset before it publishes the manifest.
|
||||
For XLSX sources it also compares the requested `--history` coverage with the
|
||||
months the source actually exposes, so a history increase is published once and
|
||||
a request larger than the available history does not rebuild forever. `--history`
|
||||
is bounded (maximum 1000) so an enormous value cannot wrap and look already
|
||||
satisfied. The checks use discovery only, so a daily run stays cheap. `--force`
|
||||
uploads regardless.
|
||||
|
||||
The SQLite asset is immutable and content-addressed. The publisher uploads and
|
||||
verifies it before it uploads the manifest, which is the commit point consumers
|
||||
follow. A partial upload therefore exits non-zero and is repaired by the next
|
||||
run.
|
||||
|
||||
`idx-ownership-freshness` is the independent alarm: it exits 1 when the
|
||||
published `latest_as_of_date` is older than `--max-age-days` (default 40). The
|
||||
source is monthly and lands 2-3 days after month end, so 40 days only trips
|
||||
when publishing has actually stopped working. Wire both to an `OnFailure=`
|
||||
notification.
|
||||
|
||||
## Manual Command (from a checkout)
|
||||
|
||||
```bash
|
||||
nix develop --command scripts/publish-ownership-snapshot.sh \
|
||||
--build \
|
||||
--idx-bin ./target/debug/idx \
|
||||
--output-dir /var/lib/idx/ownership-snapshot/current \
|
||||
--repo 0xrsydn/idx-cli \
|
||||
--release-tag ownership-snapshot-current
|
||||
```
|
||||
|
||||
## systemd Units
|
||||
|
||||
Examples: `contrib/systemd/idx-ownership-snapshot-publish.service` and
|
||||
`contrib/systemd/idx-ownership-snapshot-publish.timer` (daily at 06:00 UTC).
|
||||
|
||||
Observed XLSX upload times (HTTP Last-Modified): 2026-06-03,
|
||||
2026-07-02 03:55Z, 2026-08-02 01:50Z, 2026-09-02 03:51Z. With a daily,
|
||||
idempotent run the exact day no longer matters.
|
||||
|
||||
Pass `--history <n>` to include earlier months so `idx ownership changes`
|
||||
works straight from the synced snapshot (about 2.4 MB per extra month).
|
||||
|
||||
### Snapshot size limit
|
||||
|
||||
idx clients up to v0.2.3 cap the snapshot download at **10 MiB** (10,485,760
|
||||
bytes) and only accept uncompressed SQLite. A larger snapshot makes
|
||||
`idx ownership sync` fail for every existing install. This happened on
|
||||
2026-09-25 with `--history 5` (15.3 MB).
|
||||
|
||||
| `--history` | months | size (2026-08-31) |
|
||||
| --- | --- | --- |
|
||||
| 2 | 3 | 8,171,520 |
|
||||
| 3 | 4 | 10,510,336 (too large) |
|
||||
| 5 | 6 | 15,290,368 (too large) |
|
||||
|
||||
The builder therefore refuses snapshots above `--max-snapshot-bytes`
|
||||
(default 10485760). The run ends `RESULT: FAILED stage=build-snapshot` and the
|
||||
published snapshot stays unchanged. Newer clients accept snapshots up to the
|
||||
manifest's `size_bytes` (at most 1 GiB, buffered in memory);
|
||||
raise `--max-snapshot-bytes` and `--history` only after clients without the
|
||||
10 MiB cap are widespread.
|
||||
|
||||
## Clan Integration Later
|
||||
|
||||
The clean split for `clan-private` is:
|
||||
|
||||
1. keep the workflow logic in this repo
|
||||
2. keep host-specific secrets and enablement in `clan-private`
|
||||
|
||||
Recommended integration shape on `greencloud-vps`:
|
||||
|
||||
- store a checked-out copy of `idx-cli` on the host, for example at `/srv/idx-cli`
|
||||
- provide `GH_TOKEN` through Clan vars or another secret mechanism
|
||||
- add a small NixOS module that installs/enables the service and timer
|
||||
- set the module's `WorkingDirectory`, env file path, and publish directory
|
||||
|
||||
The existing pattern in `clan-private/modules/workspace-backup.nix` is a good
|
||||
fit: a oneshot `systemd` service plus a `systemd.timer`.
|
||||
|
||||
## Operational Notes
|
||||
|
||||
- Keep `ownership-snapshot-current` separate from versioned app releases.
|
||||
- Prefer manual runs first, then enable the timer after a few successful
|
||||
publishes.
|
||||
- If the monthly source has not appeared yet, the service should fail loudly and
|
||||
be retried later; do not silently publish stale assumptions.
|
||||
|
|
@ -4,8 +4,7 @@
|
|||
|
||||
The command is intentionally manifest-driven so the repo can publish snapshots in GitHub releases, object storage, or a local filesystem path without changing the CLI.
|
||||
|
||||
The recommended GitHub publish layout and maintainer workflow are documented in
|
||||
`docs/OWNERSHIP_PUBLISH.md`.
|
||||
Publishing snapshots is a maintainer task; see `docs/OWNERSHIP_PUBLISH.md`.
|
||||
|
||||
## Consumer Inputs
|
||||
|
||||
|
|
@ -55,7 +54,8 @@ Current schema version: `1`
|
|||
```
|
||||
|
||||
Semantics:
|
||||
- `source` is optional provenance metadata describing the IDX/KSEI PDF used to build the snapshot.
|
||||
- `source` is optional provenance metadata describing the IDX/KSEI report used to build the snapshot.
|
||||
`source.pdf_url` is a legacy field name: since June 2026 it holds the XLSX URL.
|
||||
- `download_url` points to the SQLite artifact itself. The filename is an
|
||||
implementation detail: published assets are immutable and content-addressed
|
||||
(`ownership-snapshot-YYYY-MM-DD-<sha256>.sqlite`), so consumers must always
|
||||
|
|
@ -76,20 +76,3 @@ Without `--force`, sync behaves conservatively:
|
|||
- local DB diverges at the same latest date: no-op and require `--force`
|
||||
|
||||
Replacement is staged through a temp file and validated before the local DB is swapped in.
|
||||
|
||||
## Publisher Workflow
|
||||
|
||||
Use `scripts/build-ownership-snapshot.sh` to copy a vetted ownership DB into a publishable artifact plus manifest:
|
||||
|
||||
```bash
|
||||
scripts/build-ownership-snapshot.sh \
|
||||
--db /path/to/ownership.db \
|
||||
--output-dir dist/ownership-snapshot \
|
||||
--base-url https://example.com/idx/ownership
|
||||
```
|
||||
|
||||
If `--base-url` is omitted, the generated manifest uses the local artifact path as `download_url`, which is useful for local testing.
|
||||
|
||||
For the maintainer flow that discovers the latest supported IDX/KSEI source
|
||||
first and then builds GitHub-release-ready assets, use
|
||||
`scripts/build-latest-ownership-snapshot.sh`.
|
||||
|
|
|
|||
|
|
@ -105,8 +105,6 @@ responses honor the requested batch size so the complete-fetch path is exercised
|
|||
Mocks do not verify current upstream availability, authentication, or response
|
||||
drift. Live provider checks remain a separate, opt-in operation.
|
||||
|
||||
|
||||
|
||||
## Modes
|
||||
|
||||
- `live` runs the default real-network baseline: `general`, `live-table`, and `ownership`
|
||||
|
|
@ -128,12 +126,10 @@ drift. Live provider checks remain a separate, opt-in operation.
|
|||
|
||||
## Notes
|
||||
|
||||
- The runner forces `IDX_OUTPUT=table` as its default environment so table cases stay stable; JSON checks use `-o json` explicitly.
|
||||
- Cache-group warm cases clear the smoke cache before they run so each warm/offline/stale sequence starts clean and stale-cache assertions are not masked by earlier groups.
|
||||
- Use `--bin <path> --no-build` when you want to validate an installed binary instead of the workspace `target/debug/idx` build.
|
||||
- Use `scripts/audit-msn-fundamentals.sh` for a full CLI valuation sweep across the IDX MSN symbol map. It is intentionally separate from the reusable smoke runner because it is a heavier provider-health audit, not a stable baseline check.
|
||||
- Ownership commands that need imported data are intentionally not part of the baseline runner yet. The current baseline only covers `ownership releases` and the known unsupported `ownership import --fetch-bing`.
|
||||
- `ownership sync` is still primarily covered by fixture-backed CLI tests rather than the reusable smoke runner.
|
||||
- The new `ownership-import` group is intentionally opt-in for explicit `--group ownership-import` runs or `--mode full`; it discovers live URLs first, imports the supported `above1` attachment into the temp DB, then asserts the current `above5` and `investor-type` URLs fail with explicit unsupported-schema UX.
|
||||
- The new `live-nonfinite` group is intentionally opt-in only. As of `2026-04-13`, the known real repro tickers are `BUMI`, `ADRO`, and `AIMS`.
|
||||
- When a case fails, inspect its `.stdout`, `.stderr`, and (for JSON cases) `.validation` files in `tmp/live-smoke/.../logs/` before updating `TODO.md` or `FEATURE_SPEC.md`.
|
||||
- The runner forces `IDX_OUTPUT=table` so table cases stay stable; JSON checks pass `-o json` explicitly.
|
||||
- Cache-group warm cases clear the smoke cache first so warm/offline/stale sequences start clean.
|
||||
- Use `--bin <path> --no-build` to validate an installed binary instead of `target/debug/idx`.
|
||||
- `ownership-import` and `live-nonfinite` are opt-in (explicit `--group` or `--mode full`). Known non-finite repro tickers as of `2026-04-13`: `BUMI`, `ADRO`, `AIMS`.
|
||||
- The baseline `ownership` group needs no imported data; `ownership sync` is covered by fixture-backed CLI tests instead.
|
||||
- `scripts/audit-msn-fundamentals.sh` is a heavier provider-health sweep over the IDX MSN symbol map, not a baseline check.
|
||||
- When a case fails, inspect its `.stdout`, `.stderr`, and `.validation` files under `tmp/live-smoke/.../logs/`.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue