From 1631058b8cb0302c899fda3ac9ff7cf5833701f8 Mon Sep 17 00:00:00 2001 From: hermes Date: Wed, 14 Oct 2026 07:39:21 +0000 Subject: [PATCH] docs: prune stale planning docs and lean out agent guide - Remove FEATURE_SPEC.md, TODO.md, and docs/AGENT_INTEGRATION.md; their core gaps have shipped. Open work moves to a short ROADMAP.md. - Remove the CLAUDE.md symlink and rewrite AGENTS.md as a quick guide: navigation, rules, and the nix flake check gate. - Merge OWNERSHIP_SELF_HOSTED.md into OWNERSHIP_PUBLISH.md and drop private host integration notes. - ARCHITECTURE: add MSN endpoints, clarify Bing ingest is unsupported. - OWNERSHIP_SYNC: note source.pdf_url also carries XLSX URLs. - systemd example: --history 2 so snapshots stay under the 10 MiB client cap. Co-Authored-By: Claude Opus 5.5 --- .codex/skills/idx-release/SKILL.md | 10 +- AGENTS.md | 122 +++------ CLAUDE.md | 1 - FEATURE_SPEC.md | 207 -------------- README.md | 4 +- ROADMAP.md | 33 +++ TODO.md | 257 ------------------ .../idx-ownership-snapshot-publish.service | 2 +- docs/AGENT_INTEGRATION.md | 149 ---------- docs/ARCHITECTURE.md | 15 +- docs/CONVENTIONS.md | 3 +- docs/OWNERSHIP_PUBLISH.md | 226 +++++++-------- docs/OWNERSHIP_SELF_HOSTED.md | 149 ---------- docs/OWNERSHIP_SYNC.md | 23 +- docs/SMOKE.md | 18 +- 15 files changed, 196 insertions(+), 1023 deletions(-) delete mode 120000 CLAUDE.md delete mode 100644 FEATURE_SPEC.md create mode 100644 ROADMAP.md delete mode 100644 TODO.md delete mode 100644 docs/AGENT_INTEGRATION.md delete mode 100644 docs/OWNERSHIP_SELF_HOSTED.md diff --git a/.codex/skills/idx-release/SKILL.md b/.codex/skills/idx-release/SKILL.md index b26feff..035ebe4 100644 --- a/.codex/skills/idx-release/SKILL.md +++ b/.codex/skills/idx-release/SKILL.md @@ -16,7 +16,7 @@ ownership snapshot artifacts as a separate release track. Read these files before mutating release state: - `Cargo.toml` for the package version -- `TODO.md` for the latest smoke findings and release-readiness notes +- `ROADMAP.md` for known limits that may belong in release notes - `README.md` if install or publish notes may need updates - `docs/OWNERSHIP_SYNC.md` and `docs/OWNERSHIP_PUBLISH.md` only when checking ownership-sync behavior or keeping snapshot and app releases separated @@ -59,13 +59,11 @@ curl -fsSL https://crates.io/api/v1/crates/idx-cli For a real release or release-hygiene fix, verify the repo state first. -Run the standard checks inside `nix develop`: +Run the canonical gate (fmt, clippy, tests, release build, `cargo package` +verification, and mock smoke): ```bash -nix develop --command cargo build -nix develop --command cargo clippy -- -D warnings -nix develop --command cargo test -nix develop --command cargo package --locked +nix flake check -L ``` Run targeted smoke when the release changes user-facing CLI behavior. For diff --git a/AGENTS.md b/AGENTS.md index 3a4a034..001441e 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,102 +1,42 @@ # AGENTS.md -## Project -`idx-cli` is a Rust CLI for Indonesian stock market analysis and ownership workflows. +`idx-cli` is a Rust CLI (binary `idx`) for Indonesian stock market analysis +(`stocks`) and KSEI shareholder ownership (`ownership`). -The repo currently has two main product areas: -- `stocks`: live market data and analysis -- `ownership`: import/sync once, then query locally from SQLite - -## Stack -- Rust stable -- `clap` 4 for CLI -- `ureq` 3 for HTTP -- `comfy-table` and `owo-colors` for output -- `rusqlite` + bundled SQLite/FTS5 for ownership -- TOML config in `~/.config/idx/config.toml` -- file cache in `~/.cache/idx/` - -## Source Map +## Navigation ```text src/ -├── main.rs -├── cli/ -│ ├── stocks.rs -│ ├── ownership.rs -│ ├── config.rs -│ └── cache.rs -├── api/ -│ ├── mod.rs -│ ├── types.rs -│ ├── yahoo/ -│ └── msn/ -├── analysis/ -├── ownership/ -│ ├── archive.rs -│ ├── db.rs -│ ├── entities.rs -│ ├── parser.rs -│ ├── remote.rs -│ ├── snapshot.rs -│ └── types.rs -├── output/ -├── cache.rs -├── config.rs -└── error.rs +├── main.rs, config.rs, cache.rs, error.rs, curl_impersonate.rs +├── cli/ clap commands: stocks, ownership, config, cache +├── api/ providers: msn/ (default), yahoo/ (history/OHLCV) +├── analysis/ technical, fundamental, signals +├── ownership/ xlsx + pdf parsers, archive (ZIP/TXT), remote discovery, +│ snapshot sync, db (SQLite/FTS5), entities, search, graph +└── output/ table.rs, json.rs +tests/cli.rs CLI integration tests; fixtures in tests/fixtures/ +scripts/ smoke, installer, and ownership snapshot publisher ``` -## Provider Model -- `MSN` is the primary/default provider for quotes, fundamentals, profile, earnings, financials, sentiment, insights, news, and screener data. -- `Yahoo` is the fallback provider for history/OHLCV because MSN history for IDX is still not supported. -- Current config knobs: - - `IDX_PROVIDER=msn|yahoo` - - `IDX_HISTORY_PROVIDER=auto|yahoo|msn` +Docs: +- `docs/ARCHITECTURE.md`: providers, ownership ingest/discovery, error and output contract +- `docs/CONVENTIONS.md`: types, naming, testing, and git rules +- `docs/OWNERSHIP_SYNC.md` / `docs/OWNERSHIP_PUBLISH.md`: snapshot consumer contract / maintainer publishing +- `docs/SMOKE.md`: smoke runner and Nix check coverage +- `ROADMAP.md`: open work -## Ownership Model -Ownership is local-first after bootstrap. +## Rules +1. Keep data access provider-driven; no ad hoc fetches in `cli/`. +2. Keep parse/normalize functions pure. +3. Tests use fixtures and `IDX_USE_MOCK_PROVIDER=1`; never hit the live network. +4. Output contract: tables on stdout, one JSON document with `-o json`, errors + on stderr (JSON-shaped in JSON mode), stdout empty on failure. +5. Keep ownership queries (`releases`, `ticker`, `changes`, ...) working across + every ingest path: XLSX, legacy PDF, archive ZIP/TXT, and snapshot sync. +6. `ownership import --fetch-bing` is intentionally unsupported. -Preferred bootstrap/update path: -1. `idx ownership sync` -2. if no snapshot manifest is available: `idx ownership discover` then `idx ownership import --url ` -3. local `--file` imports remain available for manual/fallback use - -Ownership input paths: -- primary remote source: IDX `above1` holder register — monthly XLSX on the Data Kepemilikan Saham page (since the 2026-05-29 report), legacy PDF announcements before that; `ownership discover` checks both -- maintained snapshot path: `ownership sync` -- local file imports: IDX above-1% `.xlsx` (current format since June 2026), legacy `.pdf`, plus local archive `.zip` / `.txt` - -Important scope note: -- archive ZIP/TXT ingest is a fallback/backstop path, not the primary product ingest surface -- `ownership import --fetch-bing` is still intentionally unsupported - -## Working Principles -1. Keep data access provider-driven where possible; avoid adding new ad hoc fetch paths at the CLI layer. -2. Prefer pure parse/normalize transforms over hidden state. -3. Use fixtures in tests; do not hit live network in automated tests. -4. Preserve the output contract: table to stdout, JSON with `--output json`, errors to stderr. -5. Treat ownership schema/query compatibility as important: `releases`, `ticker`, and `changes` should keep working across ingest paths. - -## Verification -Core verification: +## Verify ```bash -nix develop -cargo build -cargo clippy -- -D warnings -cargo test +nix flake check -L # canonical gate (fmt, clippy, tests, package, smoke) +nix develop --command cargo test # focused loop +scripts/live-smoke.sh --mode full # opt-in live providers ``` - -Smoke tooling: -```bash -scripts/live-smoke.sh -scripts/live-smoke.sh --mode mock -scripts/live-smoke.sh --mode full -``` - -## Read First -Start with these repo docs before making changes: -- `FEATURE_SPEC.md` — active implementation backlog and remaining core gaps -- `TODO.md` — execution tracker and smoke notes -- `docs/ARCHITECTURE.md` — provider and ownership flow -- `docs/OWNERSHIP_SYNC.md` — snapshot sync contract -- `docs/SMOKE.md` — reusable smoke commands -- `docs/CONVENTIONS.md` — repo conventions diff --git a/CLAUDE.md b/CLAUDE.md deleted file mode 120000 index 47dc3e3..0000000 --- a/CLAUDE.md +++ /dev/null @@ -1 +0,0 @@ -AGENTS.md \ No newline at end of file diff --git a/FEATURE_SPEC.md b/FEATURE_SPEC.md deleted file mode 100644 index dba4adc..0000000 --- a/FEATURE_SPEC.md +++ /dev/null @@ -1,207 +0,0 @@ -# Feature Spec: Current Coverage and Remaining Core Work - -**Status:** Active - updated against the current repo state on 2026-03-31 -**Current focus:** Close the remaining core gaps before expanding into new market and watchlist surfaces -**Reference:** `origin/dev/rubick` (endpoint reference only, not the current implementation plan) - ---- - -## Purpose - -This is no longer a "port MSN from zero" spec. - -The CLI already ships MSN-backed support for: -- `profile` -- `financials` -- `earnings` -- `sentiment` -- `insights` -- `news` -- `screen` - -Use this document to track what is still open in core functionality. -Use `TODO.md` as the execution log and smoke-history record. - ---- - -## Verified Current State - -- The canonical automated gate is `nix flake check -L`: unit/module and CLI integration tests, packaged-binary smoke, packaging, lint, and shell regressions. Per-run results belong in `TODO.md`, not a fixed test-count target here. -- Reusable smoke coverage exists via `scripts/live-smoke.sh`; command groups are documented in `docs/SMOKE.md`. -- The latest smoke notes in `TODO.md` report passing live table and JSON checks for all shipped `stocks` commands. -- Cache/offline parity, JSON startup-error handling, screener input validation, and the recent MSN output cleanups have already been completed. -- KSEI ownership import/query is now verified end-to-end from a local March 2026 PDF file into SQLite, with direct CLI reads through `ownership releases` and `ownership ticker`. -- Remote ownership ingestion should now be treated as an IDX PDF discovery-and-fetch problem first, not a hardcoded KSEI PDF URL problem: direct IDX announcement PDFs exist, but the hashed asset URL must be discovered from an IDX listing/announcement surface and fetched with browser-like behavior. -- The CLI now has an explicit `idx ownership discover` surface that enumerates the latest hashed BEI ownership report URLs and defaults to the supported `above1` family. -- Live verification on `2026-03-29` confirmed that the parser-compatible discovered source is currently the `Pemegang Saham di atas 1% (KSEI)` `lamp1` attachment, and `idx ownership import --url` now works end to end for that discovered BEI PDF. -- The currently discoverable `above 5%` and `investor-type` BEI families remain different schemas and are now classified and rejected explicitly during import instead of falling through to a generic zero-row parse failure. -- Ownership smoke coverage now includes a dedicated `ownership-import` group that verifies live supported import plus expected unsupported-family failures. -- `idx ownership sync` now installs maintained SQLite snapshots via a manifest-driven contract with checksum validation, conservative local replacement rules, and fixture-backed regression coverage for install/no-op/force-refresh behavior. -- The KSEI archive ZIP/TXT path is now implemented as a local-file maintainer fallback/backstop, with cross-check coverage against the `above1` PDF contract for a shared monthly release. - -This means the main gap is no longer endpoint coverage. -The remaining work is architecture cleanup, a few correctness edge cases, and selective UX expansion on top of already-shipped commands. - ---- - -## Coverage Snapshot - -| Area | CLI | Status | Notes | -| --- | --- | --- | --- | -| Quotes | `idx stocks quote` | Implemented | Cached, smoke-tested, and covered by integration tests | -| History | `idx stocks history` | Implemented with provider-specific limits | Yahoo remains the default/auto OHLCV source; explicit MSN history works for supported price-only chart windows | -| Technical | `idx stocks technical` | Implemented | Uses the cached history path | -| Growth | `idx stocks growth` | Implemented | Shipped and exercised | -| Valuation | `idx stocks valuation` | Implemented | Shipped and exercised | -| Risk | `idx stocks risk` | Implemented | Shipped and exercised | -| Fundamental | `idx stocks fundamental` | Implemented | Shipped and exercised | -| Compare | `idx stocks compare` | Implemented | Shipped and exercised | -| Company profile | `idx stocks profile` | Implemented | Cache/offline parity and fixture-backed coverage are in place | -| Financial statements | `idx stocks financials` | Implemented | Output cleanup landed, and statement filters now support `--statement income|balance|cashflow` | -| Earnings | `idx stocks earnings` | Implemented | History/forecast split is rendered, and filters now support `--forecast|--history` with `--annual|--quarterly` | -| Sentiment | `idx stocks sentiment` | Implemented | Fixture-backed CLI coverage exists | -| Insights | `idx stocks insights` | Implemented | Summary/highlights/risks/`last_updated` mapping was corrected and tested | -| News | `idx stocks news` | Implemented | Fixture-backed CLI coverage exists | -| Screener | `idx stocks screen` | Implemented with gaps | Validation landed; expression/preset workflow is still future work | -| MSN charts | `idx stocks history --history-provider msn` | Implemented with limits | Supports `--period 1mo|3mo|1y --interval 1d`; MSN provides price-only chart series, so OHLC is synthesized and volume is `0` | -| KSEI ownership import/query | `idx ownership import --file`, `idx ownership import --url`, `idx ownership releases`, `idx ownership ticker` | Implemented | Local PDF import and SQLite-backed query flow are verified against the March 2026 KSEI release; remote IDX import now works for the discovered `above 1%` `lamp1` BEI attachment, and legacy `above 5%` / `investor-type` BEI report families are rejected explicitly | -| KSEI archive fallback import | `idx ownership import --file <.zip|.txt>` | Implemented as fallback | Local archive ZIP/TXT ingest maps investor-type/locality buckets into synthetic aggregate holders for validation/backstop use, not the primary ingest surface | -| Ownership snapshot sync | `idx ownership sync` | Implemented | Manifest-driven SQLite snapshot install with checksum validation, conservative replacement/no-op rules, and publisher helper script | - ---- - -## Resolved Since The Previous Revision - -The following items should no longer be treated as active backlog in this spec: - -- CLI truth-pass baseline for shipped `stocks` commands -- Unified cache, offline, stale-cache, and `--no-cache` handling across the core and MSN-only stock commands -- Rejection of the conflicting `--offline --no-cache` flag combination -- JSON-aware startup/config failures, not just runtime failures -- Validation for `stocks screen --filter` and `--region` -- `profile` output remapping to prefer company/localized fields -- `insights` output remapping for summary, highlights, risks, and `last_updated` -- Signed-number and table-label cleanup for `financials` -- Table-mode cleanup for `earnings` -- Baseline parser and CLI regression coverage for the shipped MSN-only command set -- KSEI ownership parser hardening for the March 2026 live PDF layout, including the merged `DATE + SHARE_CODE` segment and `D`/`A` locality markers -- Real KSEI ownership CLI verification from local file import into SQLite (`7261` rows across `955` tickers on `2026-03-28`) -- Ownership remote-import hardening for the `above1` contract, including direct-PDF-only `--url` input, discovery status output, explicit legacy-schema rejection, and live ownership-import smoke coverage -- Ownership snapshot publishing/sync contract, including `idx ownership sync`, manifest/checksum validation, conservative local replacement rules, and the `scripts/build-ownership-snapshot.sh` publisher helper; the publisher now uses an immutable content-addressed SQLite asset, uploads and verifies it before the manifest commit point, and makes history-aware no-op decisions -- KSEI archive ZIP/TXT fallback ingest for local `.zip`/`.txt` files, including cross-check coverage against the `above1` PDF holder-register fixture and compatible `releases`/`ticker`/`changes` verification - -If any of the above regress, capture that in `TODO.md` as a new finding rather than reopening the old section here wholesale. - ---- - -## Remaining Core Gaps - -### P0 - Correctness and architecture - -#### 1. Unify provider and capability flow - -Current state: -- `src/cli/stocks.rs` still directly constructs `MsnProvider::new(false)` for `profile`, `financials`, `earnings`, `sentiment`, `insights`, `news`, and `screen`. -- This keeps a split execution path alive even though cache/offline behavior is now mostly unified around `fetch_msn_with_cache`. - -Why it matters: -- The architecture doc describes provider/capability-based flow, but the CLI still special-cases MSN-only commands at the handler layer. -- Future features will be harder to extend cleanly if this split remains. - -Done when: -- Normal command handlers stop constructing `MsnProvider` directly. -- Provider selection and capability checks are centralized and consistent with `docs/ARCHITECTURE.md`. - -#### 2. Fix screener row hygiene for incomplete data - -Current state: -- `src/api/msn/map.rs` still defaults missing screener price data to `0.0` when constructing `Quote` rows. - -Why it matters: -- A zero-priced row is not the same thing as a valid priced stock. -- This can silently admit incomplete market rows instead of rejecting or filtering them. - -Done when: -- Screener rows without usable price data are filtered or rejected explicitly. -- Regression tests cover the chosen behavior. - -#### 3. Decide the fundamentals fallback policy - -Current state: -- Fundamentals can still fall back to industry-level metrics when company metrics are absent. - -Why it matters: -- This can produce analysis that looks precise but is actually based on peer or category data. - -Decision needed: -- Allow the fallback and annotate it clearly, or -- reject the fallback and surface missing company data explicitly. - -Done when: -- The policy is explicit in code and reflected in output semantics. - -#### 4. Harden Yahoo reliability edge cases - -Open issues: -- Yahoo can still return `429` from datacenter IPs intermittently. -- SMA200 trend output can still show "Insufficient data" when fewer than `200` candles are returned. - -Done when: -- The retry/fallback story for Yahoo failures is deliberate and documented. -- SMA200 behavior is either improved or clearly documented as expected. - -### P1 - UX and output contract cleanup - -Completed on `2026-04-02`: -- `financials` now supports `--statement income|balance|cashflow`. -- `earnings` now supports `--forecast|--history` and `--annual|--quarterly`. -- JSON payload context is less sparse for MSN `earnings`, `insights`, and `news`; each now carries the resolved symbol. -- `screen` stays under `stocks` for now; revisit a dedicated surface only if `screen query` / `screen presets` grow into a richer workflow. - -Done when: -- Existing shipped commands are easier to drive without changing product scope. - -### P2 - Next feature work after the above is green - -Priority order: - -1. Richer financial statements - - Decide whether to stay with the current single-period model or add multi-period fetch support. - -2. New user-facing surfaces from `TODO.md` - - `market summary` - - `market movers` - - `market sectors` - - `screen query` - - `screen presets` - - `watchlist` - - `alerts` - ---- - -## Verification Gate - -Do not treat a core refactor or new provider feature as complete until all of the following are true: - -- `nix flake check -L` passes, including the packaged-binary mock smoke and the relevant behavioral integration scenarios -- the relevant live smoke groups pass for changed user-facing behavior -- `TODO.md` is updated with any new smoke finding, regression, or behavior change - -Keep the detailed reusable smoke commands in `docs/SMOKE.md`. -Do not duplicate per-run results in this spec. - ---- - -## Endpoint Reference Appendix - -MSN API key (public, embedded in MSN Money website): - -``` -0QfOX3Vn51YCzitbLaRkTTBadtWpgTN8NZLW0C1SEM -``` - -Base URLs: -- `https://assets.msn.com/service/` - core market data (Quotes, Charts, Equities, Earnings, Sentiment, Screener) -- `https://api.msn.com/msn/v0/pages/finance/` - extended data (key ratios, insights, news feed) -Keep this appendix for endpoint discovery and future work. -Use the sections above as the actual implementation plan. diff --git a/README.md b/README.md index b077238..744a049 100644 --- a/README.md +++ b/README.md @@ -145,9 +145,9 @@ Precedence order: - [docs/NPM_DISTRIBUTION.md](docs/NPM_DISTRIBUTION.md): npm package and release flow - [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md): providers, ownership ingest, and discovery - [docs/OWNERSHIP_SYNC.md](docs/OWNERSHIP_SYNC.md): snapshot manifest and sync rules -- [docs/OWNERSHIP_PUBLISH.md](docs/OWNERSHIP_PUBLISH.md) and - [docs/OWNERSHIP_SELF_HOSTED.md](docs/OWNERSHIP_SELF_HOSTED.md): maintainer snapshot publishing +- [docs/OWNERSHIP_PUBLISH.md](docs/OWNERSHIP_PUBLISH.md): maintainer snapshot publishing - [docs/SMOKE.md](docs/SMOKE.md): smoke tests against live providers +- [ROADMAP.md](ROADMAP.md): planned work ## Development diff --git a/ROADMAP.md b/ROADMAP.md new file mode 100644 index 0000000..8813789 --- /dev/null +++ b/ROADMAP.md @@ -0,0 +1,33 @@ +# Roadmap + +Open work only. Shipped behavior is described in `README.md` and `docs/`; +history lives in git. + +## Stocks +- `market summary`: IHSG index and market breadth +- `market movers`: top gainers, losers, and volume +- `market sectors`: sector performance overview +- `screen query ""`: filter stocks by expression +- `screen presets` / `screen run `: built-in screener presets + (graduate `screen` out of `stocks` only if these grow into a richer workflow) +- Multi-period financial statements (today `financials` is single-period) + +## Workflow +- `watchlist`: named symbol sets usable by other commands +- `alerts`: price alerts with a daemon +- CSV/TSV and JSONL output +- Stdin symbol input for chaining (`... | idx stocks fundamental --stdin`) +- Additional providers (Alpha Vantage, Twelve Data, IDX official) + +## Agent integration +- MCP server exposing the existing commands as tools (planned as `idx-mcp`) +- JSON Schema export for command output +- Composite `analyze` command (quote, fundamentals, ownership, sentiment) + +## Known limits +- Yahoo can return `429` from datacenter IPs; requests retry with backoff but + can still fail. +- `stocks technical` reports the SMA200 trend as unavailable when fewer than + 200 daily candles are returned. +- MSN history for IDX is price-only (`1mo`, `3mo`, `1y`); Yahoo stays the + default OHLCV source. diff --git a/TODO.md b/TODO.md deleted file mode 100644 index 6727fb8..0000000 --- a/TODO.md +++ /dev/null @@ -1,257 +0,0 @@ -# idx-cli TODO - -## ✅ Completed (v0.1.0) -- [x] Provider abstraction (MarketDataProvider trait, Yahoo impl) -- [x] `stocks quote ` — real-time quotes, multi-symbol -- [x] `stocks history ` — historical OHLC data -- [x] File cache with TTL + offline mode + stale-cache fallback -- [x] Config system (TOML file, env vars, CLI flags, precedence) -- [x] Output modes: table (human) + JSON (agent) -- [x] 52-week range bar in quote table -- [x] `cache info` / `cache clear` -- [x] `config init` / `get` / `set` / `path` -- [x] Pre-commit hooks (fmt, clippy, test) -- [x] GitHub Actions CI -- [x] Nix flake + devshell -- [x] MIT license -- [x] crates.io metadata - -## ✅ Completed (v0.1.1) -- [x] Technical analysis module (SMA, EMA, RSI, MACD, volume ratio) -- [x] Signal interpretation (bullish/bearish/neutral consensus) -- [x] `stocks technical ` — full technical analysis command -- [x] Colored signal output (green/red/yellow) -- [x] 1-year lookback for SMA200 coverage - -## ✅ Completed (v0.1.1 — fundamental suite) -- [x] `stocks growth ` — revenue/earnings growth with signals -- [x] `stocks valuation ` — PE, PB, ROE, margins, EV/EBITDA with signals -- [x] `stocks risk ` — D/E, current ratio, ROA with signals -- [x] `stocks fundamental ` — composite growth + valuation + risk -- [x] `stocks compare ` — side-by-side multi-symbol comparison -- [x] `analysis/fundamental.rs` — fundamental analysis module (ported from idx-mcp) -- [x] Yahoo quoteSummary endpoint parser (/v10/finance/quoteSummary) -- [x] 142 tests passing (94 unit + 48 integration) - -## ✅ Completed (2026-03-26 — hardening pass) -- [x] Unified cache, offline, stale-cache, and `--no-cache` behavior across core and MSN-only stock commands -- [x] Rejected the conflicting `--offline --no-cache` flag combination explicitly -- [x] Routed startup/config failures through the same JSON-aware error path as runtime failures -- [x] Validated `stocks screen --filter` and `--region`; invalid values now return errors -- [x] Added regression tests for startup JSON errors, MSN profile offline/stale-cache behavior, and screener validation - -## ✅ Completed (2026-03-26 — MSN output cleanup) -- [x] Re-mapped live MSN `profile` output to prefer localized/company fields for name, description, sector, industry, website, address, and phone -- [x] Reworked `stocks insights` output to derive a real summary, split highlights vs risks from evaluation status, and populate `last_updated` -- [x] Fixed signed-number formatting so `stocks financials` table output no longer mangles negative values -- [x] Added live-like fixtures and regression coverage for `profile`, `insights`, and signed table formatting - -## 🚧 Next Up -- [x] Build a reusable live smoke script/checklist for all shipped CLI commands (`scripts/live-smoke.sh`, `docs/SMOKE.md`) -- [x] Re-run full live smoke for MSN-only commands after the cache/offline, mapping, and formatting fixes (`scripts/live-smoke.sh --mode full --group live-table --group live-json`) -- [x] Review remaining noisy table output in `financials` and `earnings` beyond the signed-number fix -- [x] Expand parser/CLI regression coverage for the rest of the MSN-only command set -- [x] Keep the live-smoke notes below in sync with real command output after each hardening pass - -## 🚧 Ownership Roadmap Reset (2026-03-29) - -### Batch 1 — IDX discovery + remote PDF import -- [x] Verify and document the IDX announcement/listing page that exposes the monthly ownership PDF link - - Verified official BEI listing page: `https://www.idx.co.id/id/berita/pengumuman/` - - Verified listing page JSON source used by the site: `GET /primary/NewsAnnouncement/GetAllAnnouncement?keywords=...` -- [x] Implement discovery/crawler logic for the hashed IDX PDF asset URL instead of hardcoding monthly paths -- [x] Expose discovery as an explicit `idx ownership discover` CLI surface so the live BEI feed can be inspected without coupling it to import -- [x] Extract a reusable browser-impersonated fetch helper for ownership downloads by reusing the repo's `curl-impersonate` pattern -- [x] Wire `ownership import --url` to the impersonated fetch path for IDX-hosted PDFs -- [x] Keep the current PDF parser/import path as the first production ingest route -- [x] Add tests and fixtures for IDX announcement-page parsing plus remote PDF fetch failure modes -- [x] Batch 1 verification: `cargo build` -- [x] Batch 1 verification: `cargo clippy -- -D warnings` -- [x] Batch 1 verification: `cargo test` -- [x] Batch 1 verification: fixture-backed parser/downloader tests cover announcement discovery, hashed URL extraction, and downloader failure cases -- [x] Batch 1 verification: live `idx ownership discover --family above1 --limit 2` in `nix develop` resolves the current BEI hashed URLs for the parser-compatible `above 1%` report family -- [x] Batch 1 verification: live end-to-end import succeeds for one discovered BEI ownership PDF URL -- [x] Batch 1 verification: end-to-end import into a temp ownership DB via `idx ownership import --url ...`, followed by `idx ownership releases` and one ticker query - - Resolved root cause on `2026-03-29`: - - the parser was already correct for the holder-register schema; the missing piece was discovery support for the `Pemegang Saham di atas 1% (KSEI)` family - - the parser-compatible source is the `lamp1` attachment `https://www.idx.co.id/StaticData/NewsAndAnnouncement/ANNOUNCEMENTSTOCK/From_EREP/202603/b9b638e5a8_8928aca255.pdf` - - the `above 5%` and `investor-type` BEI families remain different schemas; as of `2026-03-30`, they should be treated as legacy / unsupported input rather than new parser targets - - Post-merge reconfirmation on `2026-03-30`: - - live `idx ownership discover --family above1 --limit 2` still resolves the `2026-03-10` `above 1%` BEI pair and keeps the `lamp1` attachment as the parser-compatible source - - live `idx ownership import --url` into an isolated temp DB succeeded again from `b9b638e5a8_8928aca255.pdf`: `7261` rows for `955` tickers, `1` release, `5200` distinct raw investor names, and `5176` canonical entities for `as_of_date=2026-02-27` - - post-import sanity checks still pass for `idx ownership releases` and `idx ownership ticker AADI --source ksei` - - parser coverage now includes a compact real `above1` `mutool 1.27.0` `stext` excerpt fixture, not only the synthetic live-like line fixture - -### Batch 2 — Above-1 hardening and unsupported-input UX -- [x] Standardize the supported remote-import contract on the `Pemegang Saham di atas 1% (KSEI)` holder-register layout and its `lamp1` attachment shape -- [x] Add BEI PDF schema classification before parse/import so `ownership import --url` can reject non-holder-register PDFs before the parser runs -- [x] Improve CLI error messages and fallback behavior for discovery failure, fetch failure, invalid remote content, and known-but-unsupported legacy BEI schema variants -- [x] Capture live-like fixtures for the current discoverable `investor-type` and `above 5%` BEI attachments (or their `mutool` `stext` extracts) so unsupported-input detection and failure UX are regression-tested -- [x] Decide and document whether `ownership discover` should default to `above1` output while keeping legacy families available only for diagnostic use -- [x] Decide and document whether `ownership import --url` accepts only direct PDF URLs or can also accept an IDX listing page as input -- [x] Add regression coverage for Cloudflare/HTML responses, missing announcement links, duplicate release imports, and unsupported BEI schema detections -- [x] Batch 2 verification: `cargo build` -- [x] Batch 2 verification: `cargo clippy -- -D warnings` -- [x] Batch 2 verification: `cargo test` -- [x] Batch 2 verification: ownership-focused smoke checks cover successful remote import plus expected failure UX - -### Batch 3 — Snapshot publishing + sync -- [x] Design maintained SQLite snapshot publishing after remote IDX import is stable -- [x] Add `idx ownership sync` -- [x] Define manifest/checksum/update semantics and local DB replacement rules -- [x] Add regression coverage for manifest parsing, checksum validation, no-op sync, and forced refresh -- [x] Batch 3 verification: `cargo build` -- [x] Batch 3 verification: `cargo clippy -- -D warnings` -- [x] Batch 3 verification: `cargo test` -- [x] Batch 3 verification: sync installs into an empty temp data dir, preserves query behavior, and no-ops when already current - -### Batch 4 — KSEI ZIP/TXT fallback and cross-check path -- [x] Keep KSEI ZIP/TXT ingest as fallback and validation/backstop work, not the first milestone -- [x] Define whether the KSEI archive is only a maintainer fallback or a user-facing alternative import source -- [x] Add cross-check coverage between IDX-PDF-derived output and KSEI-archive-derived output for at least one monthly release -- [x] Batch 4 verification: `cargo build` -- [x] Batch 4 verification: `cargo clippy -- -D warnings` -- [x] Batch 4 verification: `cargo test` -- [x] Batch 4 verification: fallback ingest produces a compatible SQLite state for `ownership releases`, `ticker`, and `changes` - -## 🎯 Current Core Work (from FEATURE_SPEC.md) - -### P0 — Correctness and architecture -- [x] Unify provider and capability flow so `src/cli/stocks.rs` stops constructing `MsnProvider` directly for MSN-only command handlers -- [x] Fix screener row hygiene so incomplete MSN screener rows with missing price data are filtered or rejected instead of defaulting to `0.0` -- [x] Decide and implement the fundamentals fallback policy when company metrics are missing -- [x] Harden Yahoo reliability edge cases: intermittent `429` handling and documented/intentional SMA200 behavior with fewer than `200` candles - -### P1 — UX and output contract cleanup -- [x] Add `stocks financials` filters such as `--statement income|balance|cashflow` -- [x] Add `stocks earnings` filters such as `--forecast|--history` and `--annual|--quarterly` -- [x] Review JSON payload consistency where symbol or context fields are still sparse -- [x] Decide whether `screen` stays under `stocks` long term or graduates into a richer dedicated surface later - -### P2 — Deferred but real work -- [x] Add MSN chart/history support through `stocks history --history-provider msn` for supported price-only daily chart windows (`1mo`, `3mo`, `1y`) -- [ ] Decide whether richer financial statements should stay single-period or grow into multi-period fetch support - -## 🚀 Publish Readiness (2026-04-03) - -### P1 — Release blockers -- [x] Keep the installed binary name as `idx`; `Cargo.toml` already ships `[[bin]] name = "idx"` while the package remains `idx-cli` -- [x] Expose a default Nix package/app so the documented `nix run github:0xrsydn/idx-cli` path actually works -- [x] Restrict packaged crate contents so internal repo files like `.claude/`, `CLAUDE.md`, `AGENTS.md`, and agent-planning docs are not shipped to crates.io -- [x] Document real runtime dependencies and platform assumptions for `cargo install idx-cli`, including `curl-impersonate-chrome` for Yahoo auth and `mutool` for ownership PDF parsing -- [x] Add release/install verification to CI, at minimum `cargo package` and an install smoke path; include Nix app/package verification if the README keeps advertising `nix run` - -### P2 — Pre-publish cleanup -- [x] Wire `--quiet` so non-essential `info:` / `warning:` output is actually suppressed -- [x] Fail fast on invalid `IDX_CACHE_QUOTE_TTL` / `IDX_CACHE_FUNDAMENTAL_TTL` env values instead of silently ignoring them -- [x] Propagate `-v` into the history/technical provider path so verbose Yahoo diagnostics can surface -- [x] Honor XDG overrides consistently for ownership DB/raw download defaults, not only config/cache -- [x] Tighten `scripts/live-smoke.sh` so it does not silently validate a stale `target/debug/idx` binary -- [x] Remove or fix the README reference to the non-existent `skills/` directory - -## 📋 Backlog (per SPEC.md) -- [ ] `market summary` — IHSG index, market breadth -- [ ] `market movers` — top gainers/losers/volume -- [ ] `market sectors` — sector performance overview -- [ ] `screen query ""` — filter stocks by expression -- [ ] `screen presets` / `screen run ` — built-in screener presets -- [ ] `watchlist` commands — create, manage, live watch -- [ ] `alerts` system (v0.2+) — price alerts with daemon -- [x] `completions ` — shell completion generation -- [ ] CSV/TSV output formats -- [ ] Additional providers (Alpha Vantage, Twelve Data, IDX official) - -## 🔬 Latest Smoke Findings (2026-04-02) -- [x] Behavioral coverage consolidation on `2026-10-02`: reduced the mock smoke matrix from 62 launch-heavy cases to 43 checks with separate stdout/stderr, single-document JSON validation, fixture values, and exact cache-data comparisons. Consolidated CLI tests from 89 to 62 stronger scenarios; retained all 173 unit/module regressions. -- [x] Strengthened CLI workflows cover cache misses/expiry/provider isolation, filtered and non-finite financial data, XLSX/ZIP ownership imports and queries, snapshot install/no-op/force, rejected updates preserving database bytes and query results, and 503-candidate screener ranking/full offline caches across four filters. -- [x] The stricter JSON checks exposed and fixed stderr pollution in unsupported `ownership import --fetch-bing` and totally failed `stocks compare`; partial comparisons still return available data and warnings, with `--quiet` suppressing those diagnostics. -- [x] Verification: all eleven checks passed via `nix flake check -L --keep-going` on x86_64-linux (173 unit/module tests, 62 CLI tests, 43 packaged-binary smoke cases). Throwaway fault injection proved malformed JSON, extra documents, wrong-stream output, and changed cache data are rejected. Direct packaged-CLI runs confirmed the 503-candidate/offline-prefix boundary, incomplete-total rejection without cache population, and both clean JSON error fixes. Temporary verification fixtures and injectors were removed. -- [x] Nix verification consolidation on `2026-10-02`: all eleven flake checks pass on x86_64-linux, including publisher regressions, installer `sh`/`dash` scenarios, ShellCheck, 173 unit/module tests, 89 CLI tests, and 62 mock smoke cases. CI and the pre-push hook use `nix flake check -L`; native macOS installer coverage is retained. -- [x] Checks and crate verification now share dev-profile dependency artifacts, while the installed application retains a separate release cache. Package verification logs confirm only `idx-cli` recompiles instead of rebuilding third-party dependencies. Narrowed source inputs and content-aware CI cache keys avoid unrelated rebuilds and allow source revisions to save new cached outputs. -- [x] Cache verification on `2026-10-02`: README/docs edits invalidated only the package check; the other ten derivations stayed unchanged. The real Prek pre-push gate passed, and an unchanged `nix flake check -L` completed in 11.69 seconds with no builds or compilation. -- [x] Combined PR verification on `2026-10-02`: JSON/offline-cache fixes, Nix/Crane CI, and market-data fixes pass `nix flake check -L` on x86_64-linux (173 unit tests, 89 CLI tests, 62 mock smoke cases), plus the publisher and `sh` installer regression scripts. -- [x] Fixed the integration review's screener truncation: full-list queries now re-request MSN's reported total rather than cap candidates at 500, reject incomplete totals, and use separate `all` cache keys. Live US top-performers returned 600 unique stocks; offline limits 3 and 25 matched the full result's prefixes. One upstream HTTP 499 occurred before the successful live check. -- [x] Removed the committed `result-1` Nix output symlink and ignored numbered result links. -- [x] Publish hardening batch on `2026-09-25`: `scripts/publish-ownership-snapshot.sh` now uploads an immutable content-addressed SQLite asset (date + sha256) and verifies it before publishing the manifest as the final commit point, so a partial upload no longer leaves a manifest that points at a missing asset; the manifest is trusted for a no-op only when the referenced asset matches the manifest SHA-256 and size at the configured repo/tag, and GitHub auth/network errors never become "up-to-date" -- [x] Publish integrity follow-up on `2026-09-25`: no-op and upload verification now check the full SHA-256, not just the filename and size; the publisher uses GitHub's `sha256:` asset digest when present and falls back to `gh release download` + local hashing for legacy assets, rejects a `download_url` that names a different repo/tag, and refuses to publish the manifest when the uploaded asset's SHA-256 does not match -- [x] `--history` is bounded (maximum `1000`, leading zeros stripped before conversion) so an enormous value cannot wrap negative and look already satisfied -- [x] The GitHub publish workflow now builds the packaged `idx-ownership-publisher` and routes publication through it with inputs passed as environment variables; the workflow artifact upload uses `if-no-files-found: ignore` so a legitimate up-to-date run does not fail -- [x] `--keep-workdir` no longer prints after the `RESULT:` line: the retained path is announced when the workdir is created and the EXIT cleanup is silent -- [x] `--history` changes are now honored for same-date data: the publisher compares `1 + min(--history, available XLSX months)` against the published `release_count` before and after building, validates `--history` before early exits, and does not rebuild on every run when the source has fewer historical months than requested -- [x] New offline regression harness `scripts/publish-ownership-snapshot-test.sh` (wired into `.github/workflows/ci.yml`) covers successful publish/no-op, partial SQLite upload and retry, manifest upload failure and retry, forced same-date refresh preserving the previous artifact, a missing referenced asset forcing republish, same-size remote corruption with and without digest metadata, upload corruption rejected before the manifest commit (digest and digest-less), a manifest URL for a different repo/tag, legacy PDF post-build no-op, history upgrade, limited available history, `--history` bounds, `--keep-workdir` output ordering, and auth/network failure not becoming up-to-date -- [x] Verification on `2026-09-25`: `bash -n` on the changed shell scripts and `scripts/publish-ownership-snapshot-test.sh` all passed -- [x] `ownership sync` now falls back to the built-in published manifest URL on `2026-04-12`, so plain `idx ownership sync` works on a clean machine once the public snapshot release is available -- [x] Ownership snapshot publishing now has a live-source maintainer path on `2026-04-11`: `scripts/build-latest-ownership-snapshot.sh` discovers the current supported `above1` IDX/KSEI PDF, imports it into an isolated temp DB, emits release-ready SQLite + manifest artifacts, and records the source IDX metadata in the manifest -- [x] Verification on `2026-04-11`: `nix develop --command cargo test`, `bash -n scripts/build-latest-ownership-snapshot.sh`, and a real `nix develop --command scripts/build-latest-ownership-snapshot.sh --idx-bin ./target/debug/idx --output-dir tmp/ownership-snapshot-live` run all passed; the current live source produced `7214` rows for `955` tickers with `as_of_date=2026-03-31` -- [x] Final release-hygiene pass on `2026-04-06`: crate metadata now declares `rust-version = 1.85`, README install docs now spell out Cargo helper-runtime expectations plus persistent `nix profile install`, and CI install smoke now runs the mock smoke matrix against the installed binary instead of only checking `idx version` -- [x] Verification on `2026-04-06`: `nix develop --command cargo build`, `nix develop --command cargo clippy -- -D warnings`, `nix develop --command cargo test`, `nix develop --command cargo package --allow-dirty --locked`, and `scripts/live-smoke.sh --bin ./tmp/release-install/bin/idx --no-build --mode mock` all passed -- [x] Publish review blocker batch on `2026-04-05`: the default Nix app/package path builds again without relying on an untracked runtime module, `ownership import --force` now re-imports the same release SHA atomically, and current ownership views (`ticker`, `entity`, `cross-holders`, `concentration`, `graph`) now scope KSEI data to the latest imported release instead of blending historical snapshots -- [x] Verification on `2026-04-05`: `nix develop --command cargo build`, `nix develop --command cargo clippy -- -D warnings`, `nix develop --command cargo test`, `nix build .#default`, `nix develop --command cargo package --allow-dirty --locked`, fresh `cargo install --path . --locked --root tmp/release-install`, `./tmp/release-install/bin/idx version`, and `nix run .#default -- version` all passed -- [x] Publish P2 cleanup batch on `2026-04-03`: `--quiet` now suppresses non-essential CLI warnings/info, invalid cache TTL env vars fail during startup, `-v` reaches Yahoo history diagnostics, ownership default DB/raw paths honor XDG overrides, and `scripts/live-smoke.sh` now refuses a stale `target/debug/idx` when build refresh is skipped -- [x] Verification on `2026-04-03`: `nix develop --command cargo build`, `nix develop --command cargo clippy -- -D warnings`, `nix develop --command cargo test`, `bash -n scripts/live-smoke.sh`, and `scripts/live-smoke.sh --mode mock --dry-run` all passed -- [x] Publish blocker batch on `2026-04-03`: crate packaging now excludes repo-internal agent harness files, the flake exports a default package/app for `nix run`, README install docs now describe runtime helper dependencies, and CI now checks package/install surfaces -- [x] Verification on `2026-04-03`: `nix develop --command cargo build`, `nix develop --command cargo clippy -- -D warnings`, `nix develop --command cargo test`, `nix develop --command cargo package --allow-dirty --locked`, fresh `cargo install --path . --locked --root tmp/release-install`, `./tmp/release-install/bin/idx version`, `nix build .#default`, and `nix run .#default -- version` all passed -- [x] P0 provider/capability routing is now centralized through `SelectedProvider`; `stocks` handlers no longer construct `MsnProvider` directly for MSN-only commands -- [x] MSN fundamentals now reject industry-only metrics with an explicit unsupported error; the normal MSN mock/live path uses company metrics only -- [x] MSN screener parsing now drops rows with missing/invalid/non-positive price data instead of synthesizing `0.0` price rows -- [x] Yahoo `429` retry/backoff is now centralized and regression-tested for chart and quote-summary fetches -- [x] `stocks technical` table output now says `Trend unavailable (need at least 200 daily candles)` when fewer than `200` daily candles are available -- [x] Verification on `2026-04-02`: `nix develop --command cargo build`, `nix develop --command cargo clippy -- -D warnings`, `nix develop --command cargo test`, `scripts/live-smoke.sh --mode mock`, and `scripts/live-smoke.sh --group live-table --group live-json --group routing --group cache --group errors` all passed (`tmp/live-smoke/20260402-122215`, `tmp/live-smoke/20260402-122218`) -- [x] Live smoke passed for shipped `stocks` commands: `quote`, `history`, `technical`, `growth`, `valuation`, `risk`, `fundamental`, `compare`, `profile`, `financials`, `earnings`, `sentiment`, `insights`, `news`, `screen` -- [x] Yahoo routing verified for live `quote` and `history` -- [x] `stocks history --history-provider msn` now works for supported MSN price-only chart windows; Yahoo remains the auto history source for full OHLCV -- [x] `ownership releases` works with a writable `ownership.db_path` and an empty DB -- [x] Regression coverage now verifies offline/cache parity for MSN-only commands (`stocks profile BBCA`) -- [x] `--offline --no-cache` now fails fast as an invalid flag combination instead of serving cache -- [x] Config/startup failures now honor the JSON error contract (`IDX_PROVIDER=bogus idx -o json version`) -- [x] Invalid `stocks screen --filter/--region` values now return validation errors -- [x] Added reusable smoke runner/checklist for shipped CLI surfaces (`scripts/live-smoke.sh`, `docs/SMOKE.md`) -- [x] Reusable smoke runner passes on deterministic mock suites: `general`/`cache`/`routing`/`errors`/`ownership` = 32/32 and shipped `stocks` mock matrix = 30/30 (`tmp/live-smoke/20260326-194907`, `tmp/live-smoke/20260326-194914`) -- [x] Full live MSN-only smoke now passes via runner for both table and JSON surfaces: 30/30 (`tmp/live-smoke/20260326-195853`) -- [x] Targeted no-cache live checks confirm `profile`, `insights`, and `financials` fixes against real MSN responses -- [x] Live `stocks profile BBCA` no-cache output now populates company/localized fields instead of the sparse top-level fallback -- [x] Live `stocks insights BBCA` JSON now returns a mixed-signal summary plus non-empty `last_updated` -- [x] Live `stocks financials BBCA` table no longer renders malformed negative numbers -- [x] Re-run full live smoke to reconfirm all MSN-only commands after the latest hardening fixes -- [x] `stocks financials BBCA` table now trims ISO timestamps from section headers and humanizes raw line-item keys -- [x] `stocks earnings BBCA` table now splits history vs forecast and formats annual periods, revenue values, and dates for table mode -- [x] `stocks financials` now supports section filters via `--statement income|balance|cashflow`, with filtered table output and JSON sections rendered as `null` when intentionally excluded -- [x] `stocks earnings` now supports `--forecast|--history` and `--annual|--quarterly`, so table and JSON output can be scoped without changing the cached source payload -- [x] JSON payload context is now less sparse for MSN `earnings`, `insights`, and `news`; each payload now carries the resolved stock symbol even when sourced from older cache entries -- [x] `stocks financials` JSON now normalizes `instrument.symbol` to the exchange-qualified ticker (for example `BBCA.JK`), and older cached payloads are backfilled at read time so users do not need to clear cache after upgrading -- [x] Product decision on `2026-04-02`: keep `screen` under `stocks` for now; revisit a dedicated surface only when `screen query` / `screen presets` graduate from backlog into a richer workflow -- [x] Fixture-backed parser and CLI JSON regression coverage now covers the remaining MSN-only `sentiment`, `news`, and `screen` commands -- [x] Fresh post-coverage live MSN smoke rerun still passes for table and JSON surfaces: 30/30 (`tmp/live-smoke/20260327-163403`) -- [x] MSN key-ratios parsing now tolerates stringified non-finite numeric sentinels (`Infinity`, `-Infinity`, `NaN`) as missing data instead of failing the entire fundamentals payload -- [x] Live direct CLI repros on `2026-04-13` confirmed the original failure class on `BUMI`, `ADRO`, and `AIMS`; an opt-in `scripts/live-smoke.sh --group live-nonfinite` group now covers those real ticker paths -- [x] Added `scripts/audit-msn-fundamentals.sh` for a full CLI valuation sweep across the IDX MSN symbol map; keep it as a provider-health audit, not part of the default smoke baseline -- [x] Real KSEI PDF import from local file now works again: `ownership import --file /Users/rasyidanakbar/Downloads/ksei_raw_data.pdf` imported `7261` rows for `955` tickers on `2026-03-28`, replacing the previous `6`-row/`1`-ticker failure mode -- [x] KSEI parser no longer depends on the old hardcoded column bounds fixture layout; it now reconstructs rows from `mutool` line output and handles the live `DATE + SHARE_CODE` merged segment plus `D`/`A` locality markers -- [x] A real IDX-hosted March 2026 ownership PDF URL was verified on `2026-03-29`, but only through `curl-impersonate` inside `nix develop`; plain `curl` still returns `403` from Cloudflare for the same asset -- [x] The current repo now has a concrete IDX-first roadmap: discover the hashed PDF URL from IDX pages, fetch it with browser impersonation, and only then layer snapshot publishing and `ownership sync` -- [x] The KSEI archive (`https://web.ksei.co.id/archive_download/holding_composition`) was verified as a secondary upstream that exposes monthly ZIP files and remains useful for fallback/cross-check work -- [x] Remote IDX discovery now targets the official BEI `Pengumuman` feed (`/id/berita/pengumuman/`) and its backing JSON endpoint (`/primary/NewsAnnouncement/GetAllAnnouncement`), which exposes hashed attachment URLs for ownership-related reports -- [x] New `ownership discover` CLI output now shows the latest hashed BEI ownership URLs across the current `above 5%`, `above 1%`, and `investor-type` report families -- [x] `ownership import --url https://www.idx.co.id/...pdf` now uses the same browser-impersonated download path as Yahoo auth -- [x] Reverse-engineering the current BEI feed shows the discoverable `Pemegang Saham di atas 1% (KSEI)` `lamp1` attachment matches the known-good raw KSEI holder-register layout and now imports successfully -- [x] Live `ownership import --url` against the discovered `above 1%` BEI `lamp1` attachment (`b9b638e5a8_8928aca255.pdf`) now succeeds end to end: `7261` rows for `955` tickers on `2026-03-29`, followed by successful `ownership releases` and `ownership ticker AADI --source ksei` -- [x] Product scope decision on `2026-03-30`: standardize supported remote import on the discoverable `above 1%` holder-register family; treat `above 5%` and `investor-type` PDFs as legacy / unsupported input -- [x] Live `ownership import --url` checks against the currently discoverable `above 5%` and `investor-type` BEI PDFs still fail with `no KSEI rows parsed from PDF`, which is expected until unsupported-family detection / rejection UX lands -- [x] Live `mutool` inspection of the current discoverable `investor-type` BEI attachment shows a stock-level aggregate matrix (`DATE`, `STOCK_CODE`, `NUMBER_OF_SHARES`, investor-type columns, holder-size buckets), not the holder-level KSEI register schema the current parser imports -- [x] Live `mutool` inspection of the current discoverable `above 5%` BEI attachment shows a member/tampungan report (`INVS`, member names, `KSEI UNTUK CLOSED MEMBER-...` labels), not the raw KSEI holder-register layout -- [x] `ownership discover` now defaults to the supported `above1` family, surfaces per-URL importability status, and orders the importable `lamp1` attachment first so `--limit 1` yields the current supported PDF URL -- [x] `ownership import --url` now accepts only direct PDF URLs; IDX listing/announcement page URLs fail fast with guidance to run `ownership discover` -- [x] Valid-but-unsupported BEI PDFs now fail before row parsing with explicit schema-aware errors (`announcement_wrapper`, legacy `above5`, legacy `investor-type`) instead of the old generic `no KSEI rows parsed from PDF` path -- [x] Regression coverage now covers default `ownership discover` behavior, status visibility, listing-page rejection, duplicate SHA imports, and explicit unsupported-schema detection with compact `stext` fixtures plus fake-`mutool` CLI tests -- [x] New `ownership-import` smoke coverage now discovers the current live `above1`/`above5`/`investor-type` URLs, imports the supported `above1` attachment successfully, and confirms the legacy families fail with explicit unsupported-schema UX (`tmp/live-smoke/20260330-160201`) -- [x] New ownership snapshot sync coverage now verifies manifest parsing, checksum validation, install into an empty temp data dir, preserved query behavior for `releases`/`ticker`/`changes`, no-op sync when current, and `--force` refresh via fixture-backed local manifests on `2026-03-31` -- [x] KSEI archive ZIP/TXT fallback import now works through `ownership import --file` for local `.zip` and `.txt` inputs, using synthetic investor-type/locality aggregate holders as a maintainer backstop rather than the primary product ingest -- [x] Batch 4 coverage now cross-checks the live-like `2026-02-27` `above1` PDF fixture against the matching KSEI archive bucket excerpt and verifies fallback import/query behavior for `releases`, `ticker`, and `changes` - -## 🐛 Known Issues -- [ ] Yahoo Finance returns 429 from datacenter IPs occasionally -- [ ] SMA200 trend shows "Insufficient data" if Yahoo returns < 200 candles -- [x] KSEI ownership parser no longer leaks adjacent columns into `INVESTOR_NAME` on the March 2026 live PDF; direct CLI verification now returns clean holders like `AGUNG PERKASA INVESTINDO` / `CP` / `L` -- [x] Yahoo quoteSummary crumb auth — fixed via curl-impersonate-chrome (curl_chrome131) - - fc.yahoo.com → A3 cookie + query1 getcrumb → crumb, both sent to quoteSummary - - Requires nixpkgs#curl-impersonate-chrome in PATH (added to flake.nix + clan-private) diff --git a/contrib/systemd/idx-ownership-snapshot-publish.service b/contrib/systemd/idx-ownership-snapshot-publish.service index 8064e24..51fb6ee 100644 --- a/contrib/systemd/idx-ownership-snapshot-publish.service +++ b/contrib/systemd/idx-ownership-snapshot-publish.service @@ -9,4 +9,4 @@ Environment=HOME=/var/lib/idx-ownership-snapshot EnvironmentFile=/etc/idx-ownership-snapshot.env # Build once with `nix build github:0xrsydn/idx-cli#ownership-publisher -o /opt/idx-ownership-publisher`. # Idempotent: exits 0 with "RESULT: up-to-date" when nothing newer exists. -ExecStart=/opt/idx-ownership-publisher/bin/idx-ownership-publish --output-dir /var/lib/idx-ownership-snapshot/current --repo 0xrsydn/idx-cli --release-tag ownership-snapshot-current --history 5 +ExecStart=/opt/idx-ownership-publisher/bin/idx-ownership-publish --output-dir /var/lib/idx-ownership-snapshot/current --repo 0xrsydn/idx-cli --release-tag ownership-snapshot-current --history 2 diff --git a/docs/AGENT_INTEGRATION.md b/docs/AGENT_INTEGRATION.md deleted file mode 100644 index 61e3e99..0000000 --- a/docs/AGENT_INTEGRATION.md +++ /dev/null @@ -1,149 +0,0 @@ -# Agent Integration Roadmap - -**Status:** Planning — prerequisites are the remaining P0/P1 core gaps in `FEATURE_SPEC.md` -**Goal:** Make `idx-cli` a first-class tool for AI agent workflows (Claude Code, OpenClaw, MCP clients) - ---- - -## Current Foundation - -The CLI already has the building blocks: - -- Structured JSON output (`-o json`) on every command -- Trait-based provider abstraction (mockable, extensible) -- Rich typed domain models (`Quote`, `Fundamentals`, `CompanyProfile`, `OwnershipHolder`, etc.) -- Structured error codes in JSON mode -- File-based cache with TTL, offline, and stale-cache fallback -- Feature-gated modules (ownership behind `--features ownership`) - ---- - -## Tier 1 — MCP Server Mode (highest leverage) - -Add `idx mcp-serve` to expose existing commands as MCP tools over stdio. - -- Use `rust-mcp-sdk` crate for the stdio JSON-RPC transport -- Each existing command maps to one MCP tool (e.g. `get_stock_quote`, `get_financials`, `screen_stocks`, `get_ownership_ticker`) -- Existing JSON serialization becomes the tool response body — minimal new code -- Tool input schemas derive from clap structs or parallel types -- Configure in Claude Code: - ```json - { "mcpServers": { "idx": { "command": "idx", "args": ["mcp-serve"] } } } - ``` - -**Why this first:** One feature makes `idx-cli` instantly usable from Claude Code, Claude Desktop, OpenClaw, and any MCP-compatible client. No wrapper scripts needed. - -**Reference:** `financial-datasets/mcp-server` (1.7k stars), `tradingview-mcp` (500 stars), `twsemcp` (Taiwan Stock Exchange MCP — closest regional analog), `rust-mcp-sdk` (Rust MCP crate). - ---- - -## Tier 2 — Composability - -### Stdin symbol input - -```bash -idx stocks screen --filter pe_lt:15 -o json | jq -r '.[].symbol' | idx stocks fundamental --stdin -o json -``` - -Agents generate symbol lists dynamically. Piping enables chained workflows without temp files. - -### JSONL output - -```bash -idx -o jsonl stocks quote BBCA BBRI -``` - -Newline-delimited JSON — better for streaming and piping than pretty-printed arrays. - -### Batch mode - -```bash -idx batch commands.jsonl -``` - -Where each line is a command spec: -```json -{"command": "stocks/quote", "symbols": ["BBCA", "BBRI"]} -{"command": "ownership/ticker", "symbol": "BBCA", "source": "ksei"} -``` - -Reduces process spawn overhead when agents run many queries in sequence. - ---- - -## Tier 3 — Schema Export - -```bash -idx schema stocks/quote -``` - -Dumps the JSON Schema for a command's output. Agents use this to understand available fields without trial-and-error. Document which fields are guaranteed present vs. optional. - ---- - -## Tier 4 — Agent-Oriented Domain Features - -### Watchlist as named symbol sets - -```bash -idx watchlist create banks BBCA,BBRI,BMRI,BNGA -idx stocks quote --watchlist banks -``` - -Agents build and reuse curated lists across sessions. - -### Composite analysis - -```bash -idx stocks analyze BBCA -o json -``` - -Returns quote + fundamentals + ownership + sentiment in one call. Saves agents 4+ tool calls and reduces token overhead. - -### Diff / change detection - -```bash -idx stocks quote BBCA --diff -``` - -Compares against cached previous values and surfaces deltas. Agents monitor without maintaining their own state. - -### CSV/TSV export - -```bash -idx -o csv stocks screen --filter pe_lt:15 -``` - -For agent pipelines that need tabular data (pandas, spreadsheets, database import). - ---- - -## Tier 5 — Integration Hooks - -- **Webhook primitives** — `idx watch BBCA --threshold price_gt:5000 --webhook http://...` for event-driven agent architectures -- **Session context** — `idx context set research-session-1` to tag cache entries, enabling isolated research sessions that don't pollute each other - ---- - -## Execution Order - -| Priority | Feature | Effort | Impact | -| --- | --- | --- | --- | -| 1 | MCP server mode | Medium | Unlocks all MCP clients | -| 2 | Stdin piping + JSONL | Small | Composability for chaining | -| 3 | Schema export | Small | Agent self-discovery | -| 4 | Watchlist + composite | Medium | Reduces agent round-trips | -| 5 | CSV export | Small | Tabular pipeline support | -| 6 | Webhooks + sessions | Large | Event-driven architecture | - ---- - -## Prerequisites - -Close the remaining core gaps first (`FEATURE_SPEC.md` P0/P1): - -1. Unify provider and capability flow -2. Fix screener row hygiene -3. Decide fundamentals fallback policy -4. Harden Yahoo reliability edge cases -5. Add `financials` and `earnings` filter flags diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index bbfc9f2..e16dec0 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -34,6 +34,13 @@ - **MSN Finance** — default provider. Rich data: quotes, fundamentals, profile, earnings, financials, sentiment, insights, news, screener, and explicit price-only chart history for supported IDX windows. - **Yahoo Finance** — default/auto history source. Reliable OHLCV data via `/v8/finance/chart/`. +### MSN Endpoints +- `https://assets.msn.com/service/`: quotes, charts, equities, earnings, + sentiment, screener +- `https://api.msn.com/msn/v0/pages/finance/`: key ratios, insights, news feed + +The public API key embedded in MSN Money lives in `src/api/msn/client.rs`. + ### Hybrid History Strategy When `history_provider = auto` (default): 1. Use Yahoo for history because it provides full OHLCV candles. @@ -73,7 +80,7 @@ Factory functions: Unlike the `stocks` module (live-fetch), ownership is **import-then-query**: -1. `idx ownership import` — ETL pipeline: fetch PDF/API → parse → normalize → load SQLite +1. `idx ownership import` — ETL pipeline: fetch XLSX/PDF → parse → normalize → load SQLite 2. `idx ownership sync` — install a maintained SQLite snapshot via manifest + checksum validation 3. All query commands read from local `~/.local/share/idx/ownership.db` 4. Fully offline after import/sync @@ -81,7 +88,9 @@ Unlike the `stocks` module (live-fetch), ownership is **import-then-query**: ### Data Sources - **KSEI** — official ≥1% shareholder registry, published monthly by IDX. Through May 2026 it was a PDF announcement; from the 2026-05-29 report onward it is an XLSX workbook on the IDX "Data Kepemilikan Saham" page - **KSEI archive** — monthly ZIP/TXT balance-position matrix, used as a local fallback/backstop import path -- **Bing Finance** — global institutional ownership (REST API, quarterly) +- **Bing Finance** — global institutional ownership. The schema, `--source bing` + queries, and `src/api/msn/bing.rs` exist, but `ownership import --fetch-bing` + is intentionally unsupported, so no ingest path populates Bing rows today ### Discovery `idx ownership discover` merges two IDX sources: @@ -123,7 +132,7 @@ CLI args → resolve symbol → provider.quote/history/fundamentals → render t ### Import-Query (ownership module) ``` -Import: PDF/API → parse → normalize → resolve entities → SQLite INSERT +Import: XLSX/PDF/archive → parse → normalize → resolve entities → SQLite INSERT Query: CLI args → SQLite SELECT → render table/json (no network) ``` diff --git a/docs/CONVENTIONS.md b/docs/CONVENTIONS.md index 5965e6c..d50ebff 100644 --- a/docs/CONVENTIONS.md +++ b/docs/CONVENTIONS.md @@ -126,7 +126,6 @@ let conn = Connection::open(path) - **Coverage over counts:** replacing redundant unit assertions with stronger integration scenarios must retain meaningful error, boundary, and data-integrity coverage. ## Git / VCS -- **jj (Jujutsu)** as local workflow, colocated with git - Push via `nix develop --command git push` (for prek hooks) -- Branch naming: `feat/`, `fix/` +- Branch naming: `feat/`, `fix/`, `docs/` - Commit messages: conventional commits (`feat:`, `fix:`, `refactor:`, `docs:`) diff --git a/docs/OWNERSHIP_PUBLISH.md b/docs/OWNERSHIP_PUBLISH.md index 8112004..0b82831 100644 --- a/docs/OWNERSHIP_PUBLISH.md +++ b/docs/OWNERSHIP_PUBLISH.md @@ -1,170 +1,148 @@ # Ownership Snapshot Publishing -This document covers the maintainer workflow for producing and publishing the -ownership snapshot artifacts consumed by `idx ownership sync`. +Maintainer workflow for producing and publishing the snapshot that +`idx ownership sync` consumes. The consumer contract (manifest schema and +replacement rules) is in `docs/OWNERSHIP_SYNC.md`. -## Goals +## Release Layout -- Build the snapshot from the currently discoverable supported IDX/KSEI source, - not from an ad hoc local DB. -- Keep the published manifest URL stable for end users. -- Record source provenance in the manifest so the published SQLite artifact can - be traced back to the live IDX announcement and PDF URL used to build it. - -## Recommended GitHub Releases Layout - -Use a dedicated stable release tag for snapshot assets: +Snapshot assets live on a dedicated GitHub release, separate from versioned app +releases: - tag: `ownership-snapshot-current` -- manifest asset: `ownership-snapshot-manifest.json` -- SQLite asset: `ownership-snapshot-YYYY-MM-DD-.sqlite` (immutable, content-addressed) - -Recommended public manifest URL: +- manifest: `ownership-snapshot-manifest.json` (stable URL, the commit point) +- SQLite: `ownership-snapshot-YYYY-MM-DD-.sqlite` (immutable, content-addressed) ```text https://github.com/0xrsydn/idx-cli/releases/download/ownership-snapshot-current/ownership-snapshot-manifest.json ``` -Avoid `.../releases/latest/download/...` if normal app releases and snapshot -publishes share the same repository. The repo's "latest" release can drift away -from the ownership snapshot release. +Never use `.../releases/latest/download/...`: the repo's latest release is an +app release, not the snapshot. -## Manual Maintainer Flow +## Running the Publisher -Run the publisher helper inside `nix develop` so `mutool` and the -`curl-impersonate` helper are available: +Packaged (recommended for hosts; no checkout or cargo build needed): + +```bash +nix build github:0xrsydn/idx-cli#ownership-publisher +./result/bin/idx-ownership-publish --output-dir /var/lib/idx/ownership-snapshot/current --history 2 +./result/bin/idx-ownership-freshness --max-age-days 40 +``` + +From a checkout: ```bash -nix develop --command cargo build nix develop --command scripts/publish-ownership-snapshot.sh \ + --build \ --idx-bin ./target/debug/idx \ --output-dir dist/ownership-snapshot \ --repo 0xrsydn/idx-cli \ --release-tag ownership-snapshot-current ``` -The helper performs these steps: +`idx-ownership-publish` wraps `scripts/publish-ownership-snapshot.sh` with a +pinned `idx` (including curl-impersonate and mupdf) plus jq, gh, sqlite, and +curl. It needs `GH_TOKEN` with permission to upload release assets; the freshness +check needs no token. + +## What It Does 1. `idx -o json ownership discover --family above1 --limit 50` (newest supported report first; XLSX from the Data Kepemilikan Saham page since June 2026) -2. verifies the discovered report is the current supported import path -3. imports that report (plus `--history ` earlier months, oldest first) into - an isolated temp ownership DB -4. checks that the imported release metadata is non-empty and tied to the same - source URL -5. runs `scripts/build-ownership-snapshot.sh` to emit the SQLite artifact and - base manifest -6. enriches the manifest with `source` provenance metadata - -The resulting manifest records: - -- `source.family` -- `source.listing_page_url` -- `source.query_url` -- `source.pdf_url` -- `source.title` -- `source.publish_date` -- `source.original_filename` - -That metadata is additive. Existing sync clients can still parse the manifest. +2. verifies the discovered report is the supported import path +3. imports it plus `--history ` earlier months, oldest first, into an + isolated temp DB +4. checks the imported release metadata is non-empty and tied to the source URL +5. builds the SQLite artifact and manifest (`scripts/build-latest-ownership-snapshot.sh`, + which calls `scripts/build-ownership-snapshot.sh`) and records `source` + provenance in the manifest +6. uploads using the protocol below ## Publication Protocol -The manifest is the commit point that `idx ownership sync` follows. The -publisher therefore never uploads the manifest and the SQLite asset in one -non-atomic call. It uses this order: +The manifest is the commit point, so uploads are ordered: -1. Build the snapshot and stage the manifest plus one content-addressed SQLite - file. -2. Upload the SQLite asset first. The name includes the artifact's SHA-256, so - the same content always has the same name and a retry is idempotent. -3. Read the release assets back and confirm the SQLite asset exists with the - expected SHA-256 and size. Use GitHub's `sha256:` digest metadata when it is - present; for legacy assets without a digest, download the asset through `gh` - and hash it. Fail before touching the manifest if verification fails. -4. Upload the manifest last and verify that its `snapshot.sqlite_sha256` - matches the staged manifest. +1. Upload the content-addressed SQLite asset. Same content means same name, so + retries are idempotent. +2. Read the release assets back and verify the full SHA-256 and size (GitHub's + `sha256:` digest when present, otherwise download and hash). Fail before + touching the manifest if this does not match. +3. Upload the manifest last and verify its `snapshot.sqlite_sha256`. -Consequences: +A partial upload exits non-zero and is repaired by the next run; previously +published manifests stay valid because old SQLite assets are retained. GitHub +does not replace the manifest transactionally, so a failed replacement can leave +it unavailable until a retry succeeds. -- A partial upload (manifest uploaded, SQLite missing) can no longer happen. -- If either upload fails, the job exits non-zero and the next run retries. - Previously downloaded manifests remain valid because the old SQLite asset is retained. -- GitHub does not replace the manifest transactionally. A failed replacement - can leave the manifest unavailable until a retry succeeds. -- The publisher treats a published manifest as current only when `download_url` - names this repo and release tag and the referenced asset matches the manifest - SHA-256 and size. A missing or corrupt asset, a mismatched URL, or any GitHub - authentication/network error means "publish", never "up to date". +The publisher treats the published snapshot as current only when the manifest's +`download_url` names this repo and tag and the referenced asset matches its +SHA-256 and size. A missing or corrupt asset, a mismatched URL, an unreadable +manifest, or any GitHub auth/network error means "publish", never "up to date". +`--force` uploads regardless. + +## Run Result + +The publisher is idempotent and prints exactly one final line: + +| Last line | Exit | Meaning | +| --- | --- | --- | +| `RESULT: published ` | 0 | a newer snapshot, or larger history, was uploaded | +| `RESULT: up-to-date ` | 0 | nothing to upload | +| `RESULT: FAILED stage= (exit N)` | N | failed in `arguments`, `build`, `preflight`, `discover`, `build-snapshot`, `stage-output`, `upload`, or `publish` | + +`idx-ownership-freshness` is the independent alarm: it exits 1 when the published +`latest_as_of_date` is older than `--max-age-days` (default 40). The source lands +2-3 days after month end, so 40 days only trips when publishing has stopped. ## History Coverage -`--history ` requests the latest report plus up to `` earlier monthly -XLSX reports. The publisher compares the request with the months the source -actually exposes: +`--history ` (maximum 1000) requests the latest report plus up to `n` earlier +monthly XLSX reports: - desired releases = `1 + min(--history, available earlier XLSX months)` -- a published snapshot with the same as-of date and at least the desired number - of releases is a no-op -- a published snapshot with the same as-of date but fewer releases is - republished to add history -- when the source has fewer historical months than requested, the publisher - accepts the smaller coverage and does not rebuild on every run +- same as-of date with at least the desired release count: no-op +- same as-of date with fewer releases: republish to add history +- a request larger than the source's history is accepted once, not rebuilt forever -## Upload Step +Each extra month adds about 2.4 MB, and history is what lets +`idx ownership changes` work straight from a synced snapshot. -Prefer the publisher helper, which implements the protocol above: +## Snapshot Size Limit -```bash -scripts/publish-ownership-snapshot.sh \ - --idx-bin ./target/debug/idx \ - --output-dir dist/ownership-snapshot \ - --repo 0xrsydn/idx-cli \ - --release-tag ownership-snapshot-current -``` +idx clients up to v0.2.3 cap the snapshot download at **10 MiB** (10,485,760 +bytes) and accept only uncompressed SQLite. A larger snapshot breaks +`idx ownership sync` for every existing install (this happened on 2026-09-25 +with `--history 5`). -If you upload by hand, upload these files to the `ownership-snapshot-current` -GitHub release in this order: +| `--history` | months | size (2026-08-31) | +| --- | --- | --- | +| 2 | 3 | 8,171,520 | +| 3 | 4 | 10,510,336 (too large) | +| 5 | 6 | 15,290,368 (too large) | -1. `dist/ownership-snapshot/ownership-snapshot-YYYY-MM-DD-.sqlite` -2. `dist/ownership-snapshot/ownership-snapshot-manifest.json` +The builder refuses snapshots above `--max-snapshot-bytes` (default 10485760), +ending with `RESULT: FAILED stage=build-snapshot` and leaving the published +snapshot unchanged. Newer clients accept up to the manifest's `size_bytes` (at +most 1 GiB); raise both limits only once pre-v0.2.4 clients are rare. -Never upload the manifest before its SQLite asset is present. Only after this -manual flow is reliable should the repo automate it in GitHub Actions. +## Automation -## GitHub Actions Workflow +IDX has returned `403` to GitHub-hosted runners, so unattended publishing runs +on a self-hosted machine that IDX accepts: -The repo now includes a manual workflow at -`.github/workflows/publish-ownership-snapshot.yml`. +- `contrib/systemd/idx-ownership-snapshot-publish.service` and `.timer` run the + packaged publisher daily at 06:00 UTC with `GH_TOKEN` from + `/etc/idx-ownership-snapshot.env`. Wire `OnFailure=` to a notification. +- On NixOS, add this repo as a flake input and use + `inputs.idx-cli.packages.${system}.ownership-publisher`; `nix flake update idx-cli` + picks up fixes. -Current behavior: +`.github/workflows/publish-ownership-snapshot.yml` is a manual +(`workflow_dispatch`) fallback that runs the same packaged publisher with inputs +passed as environment variables. -- trigger: `workflow_dispatch` only -- builds the packaged publisher with `nix build .#ownership-publisher` -- runs `idx-ownership-publish` from that package, which performs discovery, - import, snapshot build, and the full upload protocol in one pass -- uploads any locally produced artifacts as a workflow artifact copy with - `if-no-files-found: ignore`, so a legitimate up-to-date run still succeeds -- does not run a separate release-create or asset-upload step - -This is intentionally manual-first. Add a schedule only after a few successful -publish runs confirm the live source remains stable enough. - -The workflow passes its inputs to the shell through environment variables and -routes publication through the same fixed publisher used by self-hosted hosts, -so it cannot drift from the CLI protocol. - -## Self-Hosted Automation - -GitHub-hosted Actions are still not sufficient for this job on their own. Live -verification has shown IDX returning `403` to GitHub-hosted runners during -discovery/import. - -The recommended unattended path is therefore: - -- keep the public artifacts on GitHub Releases -- run the publish job from a self-hosted machine that IDX accepts -- use a `systemd` oneshot service plus `systemd.timer` on that machine - -See `docs/OWNERSHIP_SELF_HOSTED.md` for the reusable helper script, sample -systemd units, and the recommended split between this repo and `clan-private`. +Observed XLSX upload times (HTTP Last-Modified) were the 2nd-3rd of each month +around 02:00-04:00 UTC; a daily idempotent run makes the exact day irrelevant. +If the monthly source has not appeared yet, the run should fail loudly and retry +later rather than publish stale data. diff --git a/docs/OWNERSHIP_SELF_HOSTED.md b/docs/OWNERSHIP_SELF_HOSTED.md deleted file mode 100644 index 6b1d95e..0000000 --- a/docs/OWNERSHIP_SELF_HOSTED.md +++ /dev/null @@ -1,149 +0,0 @@ -# Self-Hosted Ownership Snapshot Publish - -This document covers the recommended self-hosted automation path for publishing -the monthly ownership snapshot consumed by `idx ownership sync`. - -Use this path when: - -- the repo's public release assets should stay on GitHub Releases -- IDX blocks GitHub-hosted Actions with `403` -- you have a VPS or other trusted machine that can reach the IDX source - -## Why Self-Hosted - -The current GitHub-hosted workflow remains useful as a manual reference, but it -is not reliable enough for unattended publishing because IDX has returned `403` -to GitHub-hosted runners. The self-hosted path keeps the end-user experience -simple: - -- maintainers publish from a trusted machine -- end users still run plain `idx ownership sync` - -No Nix setup is required for end users. Nix is only an implementation detail for -the maintainer host if that host already uses NixOS. - -## Packaged Publisher (recommended) - -The flake exposes a self-contained publisher, so the host needs no checkout, -`nix develop`, or cargo build at run time: - -```bash -nix build github:0xrsydn/idx-cli#ownership-publisher -./result/bin/idx-ownership-publish --output-dir /var/lib/idx/ownership-snapshot/current --history 5 -./result/bin/idx-ownership-freshness --max-age-days 40 -``` - -The publisher needs `GH_TOKEN` in its environment: a token that can create -and upload assets on the target repo's releases (the examples read it from -`/etc/idx-ownership-snapshot.env`; clan-private uses a Clan var). The -freshness check needs no token. - -`idx-ownership-publish` wraps `scripts/publish-ownership-snapshot.sh` with a -pinned `idx` (itself wrapped with curl-impersonate and mupdf) plus jq, gh, -sqlite and curl. On NixOS, add this repo as a flake input and run -`inputs.idx-cli.packages.${system}.ownership-publisher`; bump the input -(`nix flake update idx-cli`) to pick up fixes. - -## Run Semantics - -The publisher is idempotent and prints exactly one final line: - -| Last line | Exit | Meaning | -| --- | --- | --- | -| `RESULT: published ` | 0 | a newer snapshot, or a larger history, was uploaded | -| `RESULT: up-to-date ` | 0 | the published snapshot already has the latest report and requested history; nothing uploaded | -| `RESULT: FAILED stage= (exit N)` | N | failed in `arguments`, `build`, `preflight`, `discover`, `build-snapshot`, `stage-output`, `upload`, or `publish` | - -It checks the published manifest and verifies that the manifest's `download_url` -names this repo and release tag, and that the referenced SQLite asset matches -the manifest SHA-256 and size, before it skips anything. It uses GitHub's -`sha256:` digest metadata when available and downloads legacy assets without a -digest to hash them. A missing or corrupt asset, a mismatched URL, an unreadable -manifest, or a GitHub authentication/network error is treated as "publish", -never as up to date. The upload path verifies the full SHA-256 of the uploaded -asset before it publishes the manifest. -For XLSX sources it also compares the requested `--history` coverage with the -months the source actually exposes, so a history increase is published once and -a request larger than the available history does not rebuild forever. `--history` -is bounded (maximum 1000) so an enormous value cannot wrap and look already -satisfied. The checks use discovery only, so a daily run stays cheap. `--force` -uploads regardless. - -The SQLite asset is immutable and content-addressed. The publisher uploads and -verifies it before it uploads the manifest, which is the commit point consumers -follow. A partial upload therefore exits non-zero and is repaired by the next -run. - -`idx-ownership-freshness` is the independent alarm: it exits 1 when the -published `latest_as_of_date` is older than `--max-age-days` (default 40). The -source is monthly and lands 2-3 days after month end, so 40 days only trips -when publishing has actually stopped working. Wire both to an `OnFailure=` -notification. - -## Manual Command (from a checkout) - -```bash -nix develop --command scripts/publish-ownership-snapshot.sh \ - --build \ - --idx-bin ./target/debug/idx \ - --output-dir /var/lib/idx/ownership-snapshot/current \ - --repo 0xrsydn/idx-cli \ - --release-tag ownership-snapshot-current -``` - -## systemd Units - -Examples: `contrib/systemd/idx-ownership-snapshot-publish.service` and -`contrib/systemd/idx-ownership-snapshot-publish.timer` (daily at 06:00 UTC). - -Observed XLSX upload times (HTTP Last-Modified): 2026-06-03, -2026-07-02 03:55Z, 2026-08-02 01:50Z, 2026-09-02 03:51Z. With a daily, -idempotent run the exact day no longer matters. - -Pass `--history ` to include earlier months so `idx ownership changes` -works straight from the synced snapshot (about 2.4 MB per extra month). - -### Snapshot size limit - -idx clients up to v0.2.3 cap the snapshot download at **10 MiB** (10,485,760 -bytes) and only accept uncompressed SQLite. A larger snapshot makes -`idx ownership sync` fail for every existing install. This happened on -2026-09-25 with `--history 5` (15.3 MB). - -| `--history` | months | size (2026-08-31) | -| --- | --- | --- | -| 2 | 3 | 8,171,520 | -| 3 | 4 | 10,510,336 (too large) | -| 5 | 6 | 15,290,368 (too large) | - -The builder therefore refuses snapshots above `--max-snapshot-bytes` -(default 10485760). The run ends `RESULT: FAILED stage=build-snapshot` and the -published snapshot stays unchanged. Newer clients accept snapshots up to the -manifest's `size_bytes` (at most 1 GiB, buffered in memory); -raise `--max-snapshot-bytes` and `--history` only after clients without the -10 MiB cap are widespread. - -## Clan Integration Later - -The clean split for `clan-private` is: - -1. keep the workflow logic in this repo -2. keep host-specific secrets and enablement in `clan-private` - -Recommended integration shape on `greencloud-vps`: - -- store a checked-out copy of `idx-cli` on the host, for example at `/srv/idx-cli` -- provide `GH_TOKEN` through Clan vars or another secret mechanism -- add a small NixOS module that installs/enables the service and timer -- set the module's `WorkingDirectory`, env file path, and publish directory - -The existing pattern in `clan-private/modules/workspace-backup.nix` is a good -fit: a oneshot `systemd` service plus a `systemd.timer`. - -## Operational Notes - -- Keep `ownership-snapshot-current` separate from versioned app releases. -- Prefer manual runs first, then enable the timer after a few successful - publishes. -- If the monthly source has not appeared yet, the service should fail loudly and - be retried later; do not silently publish stale assumptions. diff --git a/docs/OWNERSHIP_SYNC.md b/docs/OWNERSHIP_SYNC.md index 725b98a..bfdd045 100644 --- a/docs/OWNERSHIP_SYNC.md +++ b/docs/OWNERSHIP_SYNC.md @@ -4,8 +4,7 @@ The command is intentionally manifest-driven so the repo can publish snapshots in GitHub releases, object storage, or a local filesystem path without changing the CLI. -The recommended GitHub publish layout and maintainer workflow are documented in -`docs/OWNERSHIP_PUBLISH.md`. +Publishing snapshots is a maintainer task; see `docs/OWNERSHIP_PUBLISH.md`. ## Consumer Inputs @@ -55,7 +54,8 @@ Current schema version: `1` ``` Semantics: -- `source` is optional provenance metadata describing the IDX/KSEI PDF used to build the snapshot. +- `source` is optional provenance metadata describing the IDX/KSEI report used to build the snapshot. + `source.pdf_url` is a legacy field name: since June 2026 it holds the XLSX URL. - `download_url` points to the SQLite artifact itself. The filename is an implementation detail: published assets are immutable and content-addressed (`ownership-snapshot-YYYY-MM-DD-.sqlite`), so consumers must always @@ -76,20 +76,3 @@ Without `--force`, sync behaves conservatively: - local DB diverges at the same latest date: no-op and require `--force` Replacement is staged through a temp file and validated before the local DB is swapped in. - -## Publisher Workflow - -Use `scripts/build-ownership-snapshot.sh` to copy a vetted ownership DB into a publishable artifact plus manifest: - -```bash -scripts/build-ownership-snapshot.sh \ - --db /path/to/ownership.db \ - --output-dir dist/ownership-snapshot \ - --base-url https://example.com/idx/ownership -``` - -If `--base-url` is omitted, the generated manifest uses the local artifact path as `download_url`, which is useful for local testing. - -For the maintainer flow that discovers the latest supported IDX/KSEI source -first and then builds GitHub-release-ready assets, use -`scripts/build-latest-ownership-snapshot.sh`. diff --git a/docs/SMOKE.md b/docs/SMOKE.md index 4d1d31b..267b58e 100644 --- a/docs/SMOKE.md +++ b/docs/SMOKE.md @@ -105,8 +105,6 @@ responses honor the requested batch size so the complete-fetch path is exercised Mocks do not verify current upstream availability, authentication, or response drift. Live provider checks remain a separate, opt-in operation. - - ## Modes - `live` runs the default real-network baseline: `general`, `live-table`, and `ownership` @@ -128,12 +126,10 @@ drift. Live provider checks remain a separate, opt-in operation. ## Notes -- The runner forces `IDX_OUTPUT=table` as its default environment so table cases stay stable; JSON checks use `-o json` explicitly. -- Cache-group warm cases clear the smoke cache before they run so each warm/offline/stale sequence starts clean and stale-cache assertions are not masked by earlier groups. -- Use `--bin --no-build` when you want to validate an installed binary instead of the workspace `target/debug/idx` build. -- Use `scripts/audit-msn-fundamentals.sh` for a full CLI valuation sweep across the IDX MSN symbol map. It is intentionally separate from the reusable smoke runner because it is a heavier provider-health audit, not a stable baseline check. -- Ownership commands that need imported data are intentionally not part of the baseline runner yet. The current baseline only covers `ownership releases` and the known unsupported `ownership import --fetch-bing`. -- `ownership sync` is still primarily covered by fixture-backed CLI tests rather than the reusable smoke runner. -- The new `ownership-import` group is intentionally opt-in for explicit `--group ownership-import` runs or `--mode full`; it discovers live URLs first, imports the supported `above1` attachment into the temp DB, then asserts the current `above5` and `investor-type` URLs fail with explicit unsupported-schema UX. -- The new `live-nonfinite` group is intentionally opt-in only. As of `2026-04-13`, the known real repro tickers are `BUMI`, `ADRO`, and `AIMS`. -- When a case fails, inspect its `.stdout`, `.stderr`, and (for JSON cases) `.validation` files in `tmp/live-smoke/.../logs/` before updating `TODO.md` or `FEATURE_SPEC.md`. +- The runner forces `IDX_OUTPUT=table` so table cases stay stable; JSON checks pass `-o json` explicitly. +- Cache-group warm cases clear the smoke cache first so warm/offline/stale sequences start clean. +- Use `--bin --no-build` to validate an installed binary instead of `target/debug/idx`. +- `ownership-import` and `live-nonfinite` are opt-in (explicit `--group` or `--mode full`). Known non-finite repro tickers as of `2026-04-13`: `BUMI`, `ADRO`, `AIMS`. +- The baseline `ownership` group needs no imported data; `ownership sync` is covered by fixture-backed CLI tests instead. +- `scripts/audit-msn-fundamentals.sh` is a heavier provider-health sweep over the IDX MSN symbol map, not a baseline check. +- When a case fails, inspect its `.stdout`, `.stderr`, and `.validation` files under `tmp/live-smoke/.../logs/`.