mirror of
https://github.com/0xrsydn/idx-cli.git
synced 2026-11-07 09:45:36 +00:00
docs: prune stale planning docs and lean out agent guide
- Remove FEATURE_SPEC.md, TODO.md, and docs/AGENT_INTEGRATION.md; their core gaps have shipped. Open work moves to a short ROADMAP.md. - Remove the CLAUDE.md symlink and rewrite AGENTS.md as a quick guide: navigation, rules, and the nix flake check gate. - Merge OWNERSHIP_SELF_HOSTED.md into OWNERSHIP_PUBLISH.md and drop private host integration notes. - ARCHITECTURE: add MSN endpoints, clarify Bing ingest is unsupported. - OWNERSHIP_SYNC: note source.pdf_url also carries XLSX URLs. - systemd example: --history 2 so snapshots stay under the 10 MiB client cap. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
hermes
parent
cd410708c0
commit
1631058b8c
15 changed files with 206 additions and 1033 deletions
|
|
@ -16,7 +16,7 @@ ownership snapshot artifacts as a separate release track.
|
||||||
Read these files before mutating release state:
|
Read these files before mutating release state:
|
||||||
|
|
||||||
- `Cargo.toml` for the package version
|
- `Cargo.toml` for the package version
|
||||||
- `TODO.md` for the latest smoke findings and release-readiness notes
|
- `ROADMAP.md` for known limits that may belong in release notes
|
||||||
- `README.md` if install or publish notes may need updates
|
- `README.md` if install or publish notes may need updates
|
||||||
- `docs/OWNERSHIP_SYNC.md` and `docs/OWNERSHIP_PUBLISH.md` only when checking
|
- `docs/OWNERSHIP_SYNC.md` and `docs/OWNERSHIP_PUBLISH.md` only when checking
|
||||||
ownership-sync behavior or keeping snapshot and app releases separated
|
ownership-sync behavior or keeping snapshot and app releases separated
|
||||||
|
|
@ -59,13 +59,11 @@ curl -fsSL https://crates.io/api/v1/crates/idx-cli
|
||||||
|
|
||||||
For a real release or release-hygiene fix, verify the repo state first.
|
For a real release or release-hygiene fix, verify the repo state first.
|
||||||
|
|
||||||
Run the standard checks inside `nix develop`:
|
Run the canonical gate (fmt, clippy, tests, release build, `cargo package`
|
||||||
|
verification, and mock smoke):
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
nix develop --command cargo build
|
nix flake check -L
|
||||||
nix develop --command cargo clippy -- -D warnings
|
|
||||||
nix develop --command cargo test
|
|
||||||
nix develop --command cargo package --locked
|
|
||||||
```
|
```
|
||||||
|
|
||||||
Run targeted smoke when the release changes user-facing CLI behavior. For
|
Run targeted smoke when the release changes user-facing CLI behavior. For
|
||||||
|
|
|
||||||
122
AGENTS.md
122
AGENTS.md
|
|
@ -1,102 +1,42 @@
|
||||||
# AGENTS.md
|
# AGENTS.md
|
||||||
|
|
||||||
## Project
|
`idx-cli` is a Rust CLI (binary `idx`) for Indonesian stock market analysis
|
||||||
`idx-cli` is a Rust CLI for Indonesian stock market analysis and ownership workflows.
|
(`stocks`) and KSEI shareholder ownership (`ownership`).
|
||||||
|
|
||||||
The repo currently has two main product areas:
|
## Navigation
|
||||||
- `stocks`: live market data and analysis
|
|
||||||
- `ownership`: import/sync once, then query locally from SQLite
|
|
||||||
|
|
||||||
## Stack
|
|
||||||
- Rust stable
|
|
||||||
- `clap` 4 for CLI
|
|
||||||
- `ureq` 3 for HTTP
|
|
||||||
- `comfy-table` and `owo-colors` for output
|
|
||||||
- `rusqlite` + bundled SQLite/FTS5 for ownership
|
|
||||||
- TOML config in `~/.config/idx/config.toml`
|
|
||||||
- file cache in `~/.cache/idx/`
|
|
||||||
|
|
||||||
## Source Map
|
|
||||||
```text
|
```text
|
||||||
src/
|
src/
|
||||||
├── main.rs
|
├── main.rs, config.rs, cache.rs, error.rs, curl_impersonate.rs
|
||||||
├── cli/
|
├── cli/ clap commands: stocks, ownership, config, cache
|
||||||
│ ├── stocks.rs
|
├── api/ providers: msn/ (default), yahoo/ (history/OHLCV)
|
||||||
│ ├── ownership.rs
|
├── analysis/ technical, fundamental, signals
|
||||||
│ ├── config.rs
|
├── ownership/ xlsx + pdf parsers, archive (ZIP/TXT), remote discovery,
|
||||||
│ └── cache.rs
|
│ snapshot sync, db (SQLite/FTS5), entities, search, graph
|
||||||
├── api/
|
└── output/ table.rs, json.rs
|
||||||
│ ├── mod.rs
|
tests/cli.rs CLI integration tests; fixtures in tests/fixtures/
|
||||||
│ ├── types.rs
|
scripts/ smoke, installer, and ownership snapshot publisher
|
||||||
│ ├── yahoo/
|
|
||||||
│ └── msn/
|
|
||||||
├── analysis/
|
|
||||||
├── ownership/
|
|
||||||
│ ├── archive.rs
|
|
||||||
│ ├── db.rs
|
|
||||||
│ ├── entities.rs
|
|
||||||
│ ├── parser.rs
|
|
||||||
│ ├── remote.rs
|
|
||||||
│ ├── snapshot.rs
|
|
||||||
│ └── types.rs
|
|
||||||
├── output/
|
|
||||||
├── cache.rs
|
|
||||||
├── config.rs
|
|
||||||
└── error.rs
|
|
||||||
```
|
```
|
||||||
|
|
||||||
## Provider Model
|
Docs:
|
||||||
- `MSN` is the primary/default provider for quotes, fundamentals, profile, earnings, financials, sentiment, insights, news, and screener data.
|
- `docs/ARCHITECTURE.md`: providers, ownership ingest/discovery, error and output contract
|
||||||
- `Yahoo` is the fallback provider for history/OHLCV because MSN history for IDX is still not supported.
|
- `docs/CONVENTIONS.md`: types, naming, testing, and git rules
|
||||||
- Current config knobs:
|
- `docs/OWNERSHIP_SYNC.md` / `docs/OWNERSHIP_PUBLISH.md`: snapshot consumer contract / maintainer publishing
|
||||||
- `IDX_PROVIDER=msn|yahoo`
|
- `docs/SMOKE.md`: smoke runner and Nix check coverage
|
||||||
- `IDX_HISTORY_PROVIDER=auto|yahoo|msn`
|
- `ROADMAP.md`: open work
|
||||||
|
|
||||||
## Ownership Model
|
## Rules
|
||||||
Ownership is local-first after bootstrap.
|
1. Keep data access provider-driven; no ad hoc fetches in `cli/`.
|
||||||
|
2. Keep parse/normalize functions pure.
|
||||||
|
3. Tests use fixtures and `IDX_USE_MOCK_PROVIDER=1`; never hit the live network.
|
||||||
|
4. Output contract: tables on stdout, one JSON document with `-o json`, errors
|
||||||
|
on stderr (JSON-shaped in JSON mode), stdout empty on failure.
|
||||||
|
5. Keep ownership queries (`releases`, `ticker`, `changes`, ...) working across
|
||||||
|
every ingest path: XLSX, legacy PDF, archive ZIP/TXT, and snapshot sync.
|
||||||
|
6. `ownership import --fetch-bing` is intentionally unsupported.
|
||||||
|
|
||||||
Preferred bootstrap/update path:
|
## Verify
|
||||||
1. `idx ownership sync`
|
|
||||||
2. if no snapshot manifest is available: `idx ownership discover` then `idx ownership import --url <xlsx-or-pdf-url>`
|
|
||||||
3. local `--file` imports remain available for manual/fallback use
|
|
||||||
|
|
||||||
Ownership input paths:
|
|
||||||
- primary remote source: IDX `above1` holder register — monthly XLSX on the Data Kepemilikan Saham page (since the 2026-05-29 report), legacy PDF announcements before that; `ownership discover` checks both
|
|
||||||
- maintained snapshot path: `ownership sync`
|
|
||||||
- local file imports: IDX above-1% `.xlsx` (current format since June 2026), legacy `.pdf`, plus local archive `.zip` / `.txt`
|
|
||||||
|
|
||||||
Important scope note:
|
|
||||||
- archive ZIP/TXT ingest is a fallback/backstop path, not the primary product ingest surface
|
|
||||||
- `ownership import --fetch-bing` is still intentionally unsupported
|
|
||||||
|
|
||||||
## Working Principles
|
|
||||||
1. Keep data access provider-driven where possible; avoid adding new ad hoc fetch paths at the CLI layer.
|
|
||||||
2. Prefer pure parse/normalize transforms over hidden state.
|
|
||||||
3. Use fixtures in tests; do not hit live network in automated tests.
|
|
||||||
4. Preserve the output contract: table to stdout, JSON with `--output json`, errors to stderr.
|
|
||||||
5. Treat ownership schema/query compatibility as important: `releases`, `ticker`, and `changes` should keep working across ingest paths.
|
|
||||||
|
|
||||||
## Verification
|
|
||||||
Core verification:
|
|
||||||
```bash
|
```bash
|
||||||
nix develop
|
nix flake check -L # canonical gate (fmt, clippy, tests, package, smoke)
|
||||||
cargo build
|
nix develop --command cargo test <name> # focused loop
|
||||||
cargo clippy -- -D warnings
|
scripts/live-smoke.sh --mode full # opt-in live providers
|
||||||
cargo test
|
|
||||||
```
|
```
|
||||||
|
|
||||||
Smoke tooling:
|
|
||||||
```bash
|
|
||||||
scripts/live-smoke.sh
|
|
||||||
scripts/live-smoke.sh --mode mock
|
|
||||||
scripts/live-smoke.sh --mode full
|
|
||||||
```
|
|
||||||
|
|
||||||
## Read First
|
|
||||||
Start with these repo docs before making changes:
|
|
||||||
- `FEATURE_SPEC.md` — active implementation backlog and remaining core gaps
|
|
||||||
- `TODO.md` — execution tracker and smoke notes
|
|
||||||
- `docs/ARCHITECTURE.md` — provider and ownership flow
|
|
||||||
- `docs/OWNERSHIP_SYNC.md` — snapshot sync contract
|
|
||||||
- `docs/SMOKE.md` — reusable smoke commands
|
|
||||||
- `docs/CONVENTIONS.md` — repo conventions
|
|
||||||
|
|
|
||||||
|
|
@ -1 +0,0 @@
|
||||||
AGENTS.md
|
|
||||||
207
FEATURE_SPEC.md
207
FEATURE_SPEC.md
|
|
@ -1,207 +0,0 @@
|
||||||
# Feature Spec: Current Coverage and Remaining Core Work
|
|
||||||
|
|
||||||
**Status:** Active - updated against the current repo state on 2026-03-31
|
|
||||||
**Current focus:** Close the remaining core gaps before expanding into new market and watchlist surfaces
|
|
||||||
**Reference:** `origin/dev/rubick` (endpoint reference only, not the current implementation plan)
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Purpose
|
|
||||||
|
|
||||||
This is no longer a "port MSN from zero" spec.
|
|
||||||
|
|
||||||
The CLI already ships MSN-backed support for:
|
|
||||||
- `profile`
|
|
||||||
- `financials`
|
|
||||||
- `earnings`
|
|
||||||
- `sentiment`
|
|
||||||
- `insights`
|
|
||||||
- `news`
|
|
||||||
- `screen`
|
|
||||||
|
|
||||||
Use this document to track what is still open in core functionality.
|
|
||||||
Use `TODO.md` as the execution log and smoke-history record.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Verified Current State
|
|
||||||
|
|
||||||
- The canonical automated gate is `nix flake check -L`: unit/module and CLI integration tests, packaged-binary smoke, packaging, lint, and shell regressions. Per-run results belong in `TODO.md`, not a fixed test-count target here.
|
|
||||||
- Reusable smoke coverage exists via `scripts/live-smoke.sh`; command groups are documented in `docs/SMOKE.md`.
|
|
||||||
- The latest smoke notes in `TODO.md` report passing live table and JSON checks for all shipped `stocks` commands.
|
|
||||||
- Cache/offline parity, JSON startup-error handling, screener input validation, and the recent MSN output cleanups have already been completed.
|
|
||||||
- KSEI ownership import/query is now verified end-to-end from a local March 2026 PDF file into SQLite, with direct CLI reads through `ownership releases` and `ownership ticker`.
|
|
||||||
- Remote ownership ingestion should now be treated as an IDX PDF discovery-and-fetch problem first, not a hardcoded KSEI PDF URL problem: direct IDX announcement PDFs exist, but the hashed asset URL must be discovered from an IDX listing/announcement surface and fetched with browser-like behavior.
|
|
||||||
- The CLI now has an explicit `idx ownership discover` surface that enumerates the latest hashed BEI ownership report URLs and defaults to the supported `above1` family.
|
|
||||||
- Live verification on `2026-03-29` confirmed that the parser-compatible discovered source is currently the `Pemegang Saham di atas 1% (KSEI)` `lamp1` attachment, and `idx ownership import --url` now works end to end for that discovered BEI PDF.
|
|
||||||
- The currently discoverable `above 5%` and `investor-type` BEI families remain different schemas and are now classified and rejected explicitly during import instead of falling through to a generic zero-row parse failure.
|
|
||||||
- Ownership smoke coverage now includes a dedicated `ownership-import` group that verifies live supported import plus expected unsupported-family failures.
|
|
||||||
- `idx ownership sync` now installs maintained SQLite snapshots via a manifest-driven contract with checksum validation, conservative local replacement rules, and fixture-backed regression coverage for install/no-op/force-refresh behavior.
|
|
||||||
- The KSEI archive ZIP/TXT path is now implemented as a local-file maintainer fallback/backstop, with cross-check coverage against the `above1` PDF contract for a shared monthly release.
|
|
||||||
|
|
||||||
This means the main gap is no longer endpoint coverage.
|
|
||||||
The remaining work is architecture cleanup, a few correctness edge cases, and selective UX expansion on top of already-shipped commands.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Coverage Snapshot
|
|
||||||
|
|
||||||
| Area | CLI | Status | Notes |
|
|
||||||
| --- | --- | --- | --- |
|
|
||||||
| Quotes | `idx stocks quote` | Implemented | Cached, smoke-tested, and covered by integration tests |
|
|
||||||
| History | `idx stocks history` | Implemented with provider-specific limits | Yahoo remains the default/auto OHLCV source; explicit MSN history works for supported price-only chart windows |
|
|
||||||
| Technical | `idx stocks technical` | Implemented | Uses the cached history path |
|
|
||||||
| Growth | `idx stocks growth` | Implemented | Shipped and exercised |
|
|
||||||
| Valuation | `idx stocks valuation` | Implemented | Shipped and exercised |
|
|
||||||
| Risk | `idx stocks risk` | Implemented | Shipped and exercised |
|
|
||||||
| Fundamental | `idx stocks fundamental` | Implemented | Shipped and exercised |
|
|
||||||
| Compare | `idx stocks compare` | Implemented | Shipped and exercised |
|
|
||||||
| Company profile | `idx stocks profile` | Implemented | Cache/offline parity and fixture-backed coverage are in place |
|
|
||||||
| Financial statements | `idx stocks financials` | Implemented | Output cleanup landed, and statement filters now support `--statement income|balance|cashflow` |
|
|
||||||
| Earnings | `idx stocks earnings` | Implemented | History/forecast split is rendered, and filters now support `--forecast|--history` with `--annual|--quarterly` |
|
|
||||||
| Sentiment | `idx stocks sentiment` | Implemented | Fixture-backed CLI coverage exists |
|
|
||||||
| Insights | `idx stocks insights` | Implemented | Summary/highlights/risks/`last_updated` mapping was corrected and tested |
|
|
||||||
| News | `idx stocks news` | Implemented | Fixture-backed CLI coverage exists |
|
|
||||||
| Screener | `idx stocks screen` | Implemented with gaps | Validation landed; expression/preset workflow is still future work |
|
|
||||||
| MSN charts | `idx stocks history --history-provider msn` | Implemented with limits | Supports `--period 1mo|3mo|1y --interval 1d`; MSN provides price-only chart series, so OHLC is synthesized and volume is `0` |
|
|
||||||
| KSEI ownership import/query | `idx ownership import --file`, `idx ownership import --url`, `idx ownership releases`, `idx ownership ticker` | Implemented | Local PDF import and SQLite-backed query flow are verified against the March 2026 KSEI release; remote IDX import now works for the discovered `above 1%` `lamp1` BEI attachment, and legacy `above 5%` / `investor-type` BEI report families are rejected explicitly |
|
|
||||||
| KSEI archive fallback import | `idx ownership import --file <.zip|.txt>` | Implemented as fallback | Local archive ZIP/TXT ingest maps investor-type/locality buckets into synthetic aggregate holders for validation/backstop use, not the primary ingest surface |
|
|
||||||
| Ownership snapshot sync | `idx ownership sync` | Implemented | Manifest-driven SQLite snapshot install with checksum validation, conservative replacement/no-op rules, and publisher helper script |
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Resolved Since The Previous Revision
|
|
||||||
|
|
||||||
The following items should no longer be treated as active backlog in this spec:
|
|
||||||
|
|
||||||
- CLI truth-pass baseline for shipped `stocks` commands
|
|
||||||
- Unified cache, offline, stale-cache, and `--no-cache` handling across the core and MSN-only stock commands
|
|
||||||
- Rejection of the conflicting `--offline --no-cache` flag combination
|
|
||||||
- JSON-aware startup/config failures, not just runtime failures
|
|
||||||
- Validation for `stocks screen --filter` and `--region`
|
|
||||||
- `profile` output remapping to prefer company/localized fields
|
|
||||||
- `insights` output remapping for summary, highlights, risks, and `last_updated`
|
|
||||||
- Signed-number and table-label cleanup for `financials`
|
|
||||||
- Table-mode cleanup for `earnings`
|
|
||||||
- Baseline parser and CLI regression coverage for the shipped MSN-only command set
|
|
||||||
- KSEI ownership parser hardening for the March 2026 live PDF layout, including the merged `DATE + SHARE_CODE` segment and `D`/`A` locality markers
|
|
||||||
- Real KSEI ownership CLI verification from local file import into SQLite (`7261` rows across `955` tickers on `2026-03-28`)
|
|
||||||
- Ownership remote-import hardening for the `above1` contract, including direct-PDF-only `--url` input, discovery status output, explicit legacy-schema rejection, and live ownership-import smoke coverage
|
|
||||||
- Ownership snapshot publishing/sync contract, including `idx ownership sync`, manifest/checksum validation, conservative local replacement rules, and the `scripts/build-ownership-snapshot.sh` publisher helper; the publisher now uses an immutable content-addressed SQLite asset, uploads and verifies it before the manifest commit point, and makes history-aware no-op decisions
|
|
||||||
- KSEI archive ZIP/TXT fallback ingest for local `.zip`/`.txt` files, including cross-check coverage against the `above1` PDF holder-register fixture and compatible `releases`/`ticker`/`changes` verification
|
|
||||||
|
|
||||||
If any of the above regress, capture that in `TODO.md` as a new finding rather than reopening the old section here wholesale.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Remaining Core Gaps
|
|
||||||
|
|
||||||
### P0 - Correctness and architecture
|
|
||||||
|
|
||||||
#### 1. Unify provider and capability flow
|
|
||||||
|
|
||||||
Current state:
|
|
||||||
- `src/cli/stocks.rs` still directly constructs `MsnProvider::new(false)` for `profile`, `financials`, `earnings`, `sentiment`, `insights`, `news`, and `screen`.
|
|
||||||
- This keeps a split execution path alive even though cache/offline behavior is now mostly unified around `fetch_msn_with_cache`.
|
|
||||||
|
|
||||||
Why it matters:
|
|
||||||
- The architecture doc describes provider/capability-based flow, but the CLI still special-cases MSN-only commands at the handler layer.
|
|
||||||
- Future features will be harder to extend cleanly if this split remains.
|
|
||||||
|
|
||||||
Done when:
|
|
||||||
- Normal command handlers stop constructing `MsnProvider` directly.
|
|
||||||
- Provider selection and capability checks are centralized and consistent with `docs/ARCHITECTURE.md`.
|
|
||||||
|
|
||||||
#### 2. Fix screener row hygiene for incomplete data
|
|
||||||
|
|
||||||
Current state:
|
|
||||||
- `src/api/msn/map.rs` still defaults missing screener price data to `0.0` when constructing `Quote` rows.
|
|
||||||
|
|
||||||
Why it matters:
|
|
||||||
- A zero-priced row is not the same thing as a valid priced stock.
|
|
||||||
- This can silently admit incomplete market rows instead of rejecting or filtering them.
|
|
||||||
|
|
||||||
Done when:
|
|
||||||
- Screener rows without usable price data are filtered or rejected explicitly.
|
|
||||||
- Regression tests cover the chosen behavior.
|
|
||||||
|
|
||||||
#### 3. Decide the fundamentals fallback policy
|
|
||||||
|
|
||||||
Current state:
|
|
||||||
- Fundamentals can still fall back to industry-level metrics when company metrics are absent.
|
|
||||||
|
|
||||||
Why it matters:
|
|
||||||
- This can produce analysis that looks precise but is actually based on peer or category data.
|
|
||||||
|
|
||||||
Decision needed:
|
|
||||||
- Allow the fallback and annotate it clearly, or
|
|
||||||
- reject the fallback and surface missing company data explicitly.
|
|
||||||
|
|
||||||
Done when:
|
|
||||||
- The policy is explicit in code and reflected in output semantics.
|
|
||||||
|
|
||||||
#### 4. Harden Yahoo reliability edge cases
|
|
||||||
|
|
||||||
Open issues:
|
|
||||||
- Yahoo can still return `429` from datacenter IPs intermittently.
|
|
||||||
- SMA200 trend output can still show "Insufficient data" when fewer than `200` candles are returned.
|
|
||||||
|
|
||||||
Done when:
|
|
||||||
- The retry/fallback story for Yahoo failures is deliberate and documented.
|
|
||||||
- SMA200 behavior is either improved or clearly documented as expected.
|
|
||||||
|
|
||||||
### P1 - UX and output contract cleanup
|
|
||||||
|
|
||||||
Completed on `2026-04-02`:
|
|
||||||
- `financials` now supports `--statement income|balance|cashflow`.
|
|
||||||
- `earnings` now supports `--forecast|--history` and `--annual|--quarterly`.
|
|
||||||
- JSON payload context is less sparse for MSN `earnings`, `insights`, and `news`; each now carries the resolved symbol.
|
|
||||||
- `screen` stays under `stocks` for now; revisit a dedicated surface only if `screen query` / `screen presets` grow into a richer workflow.
|
|
||||||
|
|
||||||
Done when:
|
|
||||||
- Existing shipped commands are easier to drive without changing product scope.
|
|
||||||
|
|
||||||
### P2 - Next feature work after the above is green
|
|
||||||
|
|
||||||
Priority order:
|
|
||||||
|
|
||||||
1. Richer financial statements
|
|
||||||
- Decide whether to stay with the current single-period model or add multi-period fetch support.
|
|
||||||
|
|
||||||
2. New user-facing surfaces from `TODO.md`
|
|
||||||
- `market summary`
|
|
||||||
- `market movers`
|
|
||||||
- `market sectors`
|
|
||||||
- `screen query`
|
|
||||||
- `screen presets`
|
|
||||||
- `watchlist`
|
|
||||||
- `alerts`
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Verification Gate
|
|
||||||
|
|
||||||
Do not treat a core refactor or new provider feature as complete until all of the following are true:
|
|
||||||
|
|
||||||
- `nix flake check -L` passes, including the packaged-binary mock smoke and the relevant behavioral integration scenarios
|
|
||||||
- the relevant live smoke groups pass for changed user-facing behavior
|
|
||||||
- `TODO.md` is updated with any new smoke finding, regression, or behavior change
|
|
||||||
|
|
||||||
Keep the detailed reusable smoke commands in `docs/SMOKE.md`.
|
|
||||||
Do not duplicate per-run results in this spec.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Endpoint Reference Appendix
|
|
||||||
|
|
||||||
MSN API key (public, embedded in MSN Money website):
|
|
||||||
|
|
||||||
```
|
|
||||||
0QfOX3Vn51YCzitbLaRkTTBadtWpgTN8NZLW0C1SEM
|
|
||||||
```
|
|
||||||
|
|
||||||
Base URLs:
|
|
||||||
- `https://assets.msn.com/service/` - core market data (Quotes, Charts, Equities, Earnings, Sentiment, Screener)
|
|
||||||
- `https://api.msn.com/msn/v0/pages/finance/` - extended data (key ratios, insights, news feed)
|
|
||||||
Keep this appendix for endpoint discovery and future work.
|
|
||||||
Use the sections above as the actual implementation plan.
|
|
||||||
|
|
@ -145,9 +145,9 @@ Precedence order:
|
||||||
- [docs/NPM_DISTRIBUTION.md](docs/NPM_DISTRIBUTION.md): npm package and release flow
|
- [docs/NPM_DISTRIBUTION.md](docs/NPM_DISTRIBUTION.md): npm package and release flow
|
||||||
- [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md): providers, ownership ingest, and discovery
|
- [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md): providers, ownership ingest, and discovery
|
||||||
- [docs/OWNERSHIP_SYNC.md](docs/OWNERSHIP_SYNC.md): snapshot manifest and sync rules
|
- [docs/OWNERSHIP_SYNC.md](docs/OWNERSHIP_SYNC.md): snapshot manifest and sync rules
|
||||||
- [docs/OWNERSHIP_PUBLISH.md](docs/OWNERSHIP_PUBLISH.md) and
|
- [docs/OWNERSHIP_PUBLISH.md](docs/OWNERSHIP_PUBLISH.md): maintainer snapshot publishing
|
||||||
[docs/OWNERSHIP_SELF_HOSTED.md](docs/OWNERSHIP_SELF_HOSTED.md): maintainer snapshot publishing
|
|
||||||
- [docs/SMOKE.md](docs/SMOKE.md): smoke tests against live providers
|
- [docs/SMOKE.md](docs/SMOKE.md): smoke tests against live providers
|
||||||
|
- [ROADMAP.md](ROADMAP.md): planned work
|
||||||
|
|
||||||
## Development
|
## Development
|
||||||
|
|
||||||
|
|
|
||||||
33
ROADMAP.md
Normal file
33
ROADMAP.md
Normal file
|
|
@ -0,0 +1,33 @@
|
||||||
|
# Roadmap
|
||||||
|
|
||||||
|
Open work only. Shipped behavior is described in `README.md` and `docs/`;
|
||||||
|
history lives in git.
|
||||||
|
|
||||||
|
## Stocks
|
||||||
|
- `market summary`: IHSG index and market breadth
|
||||||
|
- `market movers`: top gainers, losers, and volume
|
||||||
|
- `market sectors`: sector performance overview
|
||||||
|
- `screen query "<EXPR>"`: filter stocks by expression
|
||||||
|
- `screen presets` / `screen run <PRESET>`: built-in screener presets
|
||||||
|
(graduate `screen` out of `stocks` only if these grow into a richer workflow)
|
||||||
|
- Multi-period financial statements (today `financials` is single-period)
|
||||||
|
|
||||||
|
## Workflow
|
||||||
|
- `watchlist`: named symbol sets usable by other commands
|
||||||
|
- `alerts`: price alerts with a daemon
|
||||||
|
- CSV/TSV and JSONL output
|
||||||
|
- Stdin symbol input for chaining (`... | idx stocks fundamental --stdin`)
|
||||||
|
- Additional providers (Alpha Vantage, Twelve Data, IDX official)
|
||||||
|
|
||||||
|
## Agent integration
|
||||||
|
- MCP server exposing the existing commands as tools (planned as `idx-mcp`)
|
||||||
|
- JSON Schema export for command output
|
||||||
|
- Composite `analyze` command (quote, fundamentals, ownership, sentiment)
|
||||||
|
|
||||||
|
## Known limits
|
||||||
|
- Yahoo can return `429` from datacenter IPs; requests retry with backoff but
|
||||||
|
can still fail.
|
||||||
|
- `stocks technical` reports the SMA200 trend as unavailable when fewer than
|
||||||
|
200 daily candles are returned.
|
||||||
|
- MSN history for IDX is price-only (`1mo`, `3mo`, `1y`); Yahoo stays the
|
||||||
|
default OHLCV source.
|
||||||
257
TODO.md
257
TODO.md
|
|
@ -1,257 +0,0 @@
|
||||||
# idx-cli TODO
|
|
||||||
|
|
||||||
## ✅ Completed (v0.1.0)
|
|
||||||
- [x] Provider abstraction (MarketDataProvider trait, Yahoo impl)
|
|
||||||
- [x] `stocks quote <SYMBOL...>` — real-time quotes, multi-symbol
|
|
||||||
- [x] `stocks history <SYMBOL>` — historical OHLC data
|
|
||||||
- [x] File cache with TTL + offline mode + stale-cache fallback
|
|
||||||
- [x] Config system (TOML file, env vars, CLI flags, precedence)
|
|
||||||
- [x] Output modes: table (human) + JSON (agent)
|
|
||||||
- [x] 52-week range bar in quote table
|
|
||||||
- [x] `cache info` / `cache clear`
|
|
||||||
- [x] `config init` / `get` / `set` / `path`
|
|
||||||
- [x] Pre-commit hooks (fmt, clippy, test)
|
|
||||||
- [x] GitHub Actions CI
|
|
||||||
- [x] Nix flake + devshell
|
|
||||||
- [x] MIT license
|
|
||||||
- [x] crates.io metadata
|
|
||||||
|
|
||||||
## ✅ Completed (v0.1.1)
|
|
||||||
- [x] Technical analysis module (SMA, EMA, RSI, MACD, volume ratio)
|
|
||||||
- [x] Signal interpretation (bullish/bearish/neutral consensus)
|
|
||||||
- [x] `stocks technical <SYMBOL>` — full technical analysis command
|
|
||||||
- [x] Colored signal output (green/red/yellow)
|
|
||||||
- [x] 1-year lookback for SMA200 coverage
|
|
||||||
|
|
||||||
## ✅ Completed (v0.1.1 — fundamental suite)
|
|
||||||
- [x] `stocks growth <SYMBOL>` — revenue/earnings growth with signals
|
|
||||||
- [x] `stocks valuation <SYMBOL>` — PE, PB, ROE, margins, EV/EBITDA with signals
|
|
||||||
- [x] `stocks risk <SYMBOL>` — D/E, current ratio, ROA with signals
|
|
||||||
- [x] `stocks fundamental <SYMBOL>` — composite growth + valuation + risk
|
|
||||||
- [x] `stocks compare <SYM1,SYM2,...>` — side-by-side multi-symbol comparison
|
|
||||||
- [x] `analysis/fundamental.rs` — fundamental analysis module (ported from idx-mcp)
|
|
||||||
- [x] Yahoo quoteSummary endpoint parser (/v10/finance/quoteSummary)
|
|
||||||
- [x] 142 tests passing (94 unit + 48 integration)
|
|
||||||
|
|
||||||
## ✅ Completed (2026-03-26 — hardening pass)
|
|
||||||
- [x] Unified cache, offline, stale-cache, and `--no-cache` behavior across core and MSN-only stock commands
|
|
||||||
- [x] Rejected the conflicting `--offline --no-cache` flag combination explicitly
|
|
||||||
- [x] Routed startup/config failures through the same JSON-aware error path as runtime failures
|
|
||||||
- [x] Validated `stocks screen --filter` and `--region`; invalid values now return errors
|
|
||||||
- [x] Added regression tests for startup JSON errors, MSN profile offline/stale-cache behavior, and screener validation
|
|
||||||
|
|
||||||
## ✅ Completed (2026-03-26 — MSN output cleanup)
|
|
||||||
- [x] Re-mapped live MSN `profile` output to prefer localized/company fields for name, description, sector, industry, website, address, and phone
|
|
||||||
- [x] Reworked `stocks insights` output to derive a real summary, split highlights vs risks from evaluation status, and populate `last_updated`
|
|
||||||
- [x] Fixed signed-number formatting so `stocks financials` table output no longer mangles negative values
|
|
||||||
- [x] Added live-like fixtures and regression coverage for `profile`, `insights`, and signed table formatting
|
|
||||||
|
|
||||||
## 🚧 Next Up
|
|
||||||
- [x] Build a reusable live smoke script/checklist for all shipped CLI commands (`scripts/live-smoke.sh`, `docs/SMOKE.md`)
|
|
||||||
- [x] Re-run full live smoke for MSN-only commands after the cache/offline, mapping, and formatting fixes (`scripts/live-smoke.sh --mode full --group live-table --group live-json`)
|
|
||||||
- [x] Review remaining noisy table output in `financials` and `earnings` beyond the signed-number fix
|
|
||||||
- [x] Expand parser/CLI regression coverage for the rest of the MSN-only command set
|
|
||||||
- [x] Keep the live-smoke notes below in sync with real command output after each hardening pass
|
|
||||||
|
|
||||||
## 🚧 Ownership Roadmap Reset (2026-03-29)
|
|
||||||
|
|
||||||
### Batch 1 — IDX discovery + remote PDF import
|
|
||||||
- [x] Verify and document the IDX announcement/listing page that exposes the monthly ownership PDF link
|
|
||||||
- Verified official BEI listing page: `https://www.idx.co.id/id/berita/pengumuman/`
|
|
||||||
- Verified listing page JSON source used by the site: `GET /primary/NewsAnnouncement/GetAllAnnouncement?keywords=...`
|
|
||||||
- [x] Implement discovery/crawler logic for the hashed IDX PDF asset URL instead of hardcoding monthly paths
|
|
||||||
- [x] Expose discovery as an explicit `idx ownership discover` CLI surface so the live BEI feed can be inspected without coupling it to import
|
|
||||||
- [x] Extract a reusable browser-impersonated fetch helper for ownership downloads by reusing the repo's `curl-impersonate` pattern
|
|
||||||
- [x] Wire `ownership import --url` to the impersonated fetch path for IDX-hosted PDFs
|
|
||||||
- [x] Keep the current PDF parser/import path as the first production ingest route
|
|
||||||
- [x] Add tests and fixtures for IDX announcement-page parsing plus remote PDF fetch failure modes
|
|
||||||
- [x] Batch 1 verification: `cargo build`
|
|
||||||
- [x] Batch 1 verification: `cargo clippy -- -D warnings`
|
|
||||||
- [x] Batch 1 verification: `cargo test`
|
|
||||||
- [x] Batch 1 verification: fixture-backed parser/downloader tests cover announcement discovery, hashed URL extraction, and downloader failure cases
|
|
||||||
- [x] Batch 1 verification: live `idx ownership discover --family above1 --limit 2` in `nix develop` resolves the current BEI hashed URLs for the parser-compatible `above 1%` report family
|
|
||||||
- [x] Batch 1 verification: live end-to-end import succeeds for one discovered BEI ownership PDF URL
|
|
||||||
- [x] Batch 1 verification: end-to-end import into a temp ownership DB via `idx ownership import --url ...`, followed by `idx ownership releases` and one ticker query
|
|
||||||
- Resolved root cause on `2026-03-29`:
|
|
||||||
- the parser was already correct for the holder-register schema; the missing piece was discovery support for the `Pemegang Saham di atas 1% (KSEI)` family
|
|
||||||
- the parser-compatible source is the `lamp1` attachment `https://www.idx.co.id/StaticData/NewsAndAnnouncement/ANNOUNCEMENTSTOCK/From_EREP/202603/b9b638e5a8_8928aca255.pdf`
|
|
||||||
- the `above 5%` and `investor-type` BEI families remain different schemas; as of `2026-03-30`, they should be treated as legacy / unsupported input rather than new parser targets
|
|
||||||
- Post-merge reconfirmation on `2026-03-30`:
|
|
||||||
- live `idx ownership discover --family above1 --limit 2` still resolves the `2026-03-10` `above 1%` BEI pair and keeps the `lamp1` attachment as the parser-compatible source
|
|
||||||
- live `idx ownership import --url` into an isolated temp DB succeeded again from `b9b638e5a8_8928aca255.pdf`: `7261` rows for `955` tickers, `1` release, `5200` distinct raw investor names, and `5176` canonical entities for `as_of_date=2026-02-27`
|
|
||||||
- post-import sanity checks still pass for `idx ownership releases` and `idx ownership ticker AADI --source ksei`
|
|
||||||
- parser coverage now includes a compact real `above1` `mutool 1.27.0` `stext` excerpt fixture, not only the synthetic live-like line fixture
|
|
||||||
|
|
||||||
### Batch 2 — Above-1 hardening and unsupported-input UX
|
|
||||||
- [x] Standardize the supported remote-import contract on the `Pemegang Saham di atas 1% (KSEI)` holder-register layout and its `lamp1` attachment shape
|
|
||||||
- [x] Add BEI PDF schema classification before parse/import so `ownership import --url` can reject non-holder-register PDFs before the parser runs
|
|
||||||
- [x] Improve CLI error messages and fallback behavior for discovery failure, fetch failure, invalid remote content, and known-but-unsupported legacy BEI schema variants
|
|
||||||
- [x] Capture live-like fixtures for the current discoverable `investor-type` and `above 5%` BEI attachments (or their `mutool` `stext` extracts) so unsupported-input detection and failure UX are regression-tested
|
|
||||||
- [x] Decide and document whether `ownership discover` should default to `above1` output while keeping legacy families available only for diagnostic use
|
|
||||||
- [x] Decide and document whether `ownership import --url` accepts only direct PDF URLs or can also accept an IDX listing page as input
|
|
||||||
- [x] Add regression coverage for Cloudflare/HTML responses, missing announcement links, duplicate release imports, and unsupported BEI schema detections
|
|
||||||
- [x] Batch 2 verification: `cargo build`
|
|
||||||
- [x] Batch 2 verification: `cargo clippy -- -D warnings`
|
|
||||||
- [x] Batch 2 verification: `cargo test`
|
|
||||||
- [x] Batch 2 verification: ownership-focused smoke checks cover successful remote import plus expected failure UX
|
|
||||||
|
|
||||||
### Batch 3 — Snapshot publishing + sync
|
|
||||||
- [x] Design maintained SQLite snapshot publishing after remote IDX import is stable
|
|
||||||
- [x] Add `idx ownership sync`
|
|
||||||
- [x] Define manifest/checksum/update semantics and local DB replacement rules
|
|
||||||
- [x] Add regression coverage for manifest parsing, checksum validation, no-op sync, and forced refresh
|
|
||||||
- [x] Batch 3 verification: `cargo build`
|
|
||||||
- [x] Batch 3 verification: `cargo clippy -- -D warnings`
|
|
||||||
- [x] Batch 3 verification: `cargo test`
|
|
||||||
- [x] Batch 3 verification: sync installs into an empty temp data dir, preserves query behavior, and no-ops when already current
|
|
||||||
|
|
||||||
### Batch 4 — KSEI ZIP/TXT fallback and cross-check path
|
|
||||||
- [x] Keep KSEI ZIP/TXT ingest as fallback and validation/backstop work, not the first milestone
|
|
||||||
- [x] Define whether the KSEI archive is only a maintainer fallback or a user-facing alternative import source
|
|
||||||
- [x] Add cross-check coverage between IDX-PDF-derived output and KSEI-archive-derived output for at least one monthly release
|
|
||||||
- [x] Batch 4 verification: `cargo build`
|
|
||||||
- [x] Batch 4 verification: `cargo clippy -- -D warnings`
|
|
||||||
- [x] Batch 4 verification: `cargo test`
|
|
||||||
- [x] Batch 4 verification: fallback ingest produces a compatible SQLite state for `ownership releases`, `ticker`, and `changes`
|
|
||||||
|
|
||||||
## 🎯 Current Core Work (from FEATURE_SPEC.md)
|
|
||||||
|
|
||||||
### P0 — Correctness and architecture
|
|
||||||
- [x] Unify provider and capability flow so `src/cli/stocks.rs` stops constructing `MsnProvider` directly for MSN-only command handlers
|
|
||||||
- [x] Fix screener row hygiene so incomplete MSN screener rows with missing price data are filtered or rejected instead of defaulting to `0.0`
|
|
||||||
- [x] Decide and implement the fundamentals fallback policy when company metrics are missing
|
|
||||||
- [x] Harden Yahoo reliability edge cases: intermittent `429` handling and documented/intentional SMA200 behavior with fewer than `200` candles
|
|
||||||
|
|
||||||
### P1 — UX and output contract cleanup
|
|
||||||
- [x] Add `stocks financials` filters such as `--statement income|balance|cashflow`
|
|
||||||
- [x] Add `stocks earnings` filters such as `--forecast|--history` and `--annual|--quarterly`
|
|
||||||
- [x] Review JSON payload consistency where symbol or context fields are still sparse
|
|
||||||
- [x] Decide whether `screen` stays under `stocks` long term or graduates into a richer dedicated surface later
|
|
||||||
|
|
||||||
### P2 — Deferred but real work
|
|
||||||
- [x] Add MSN chart/history support through `stocks history --history-provider msn` for supported price-only daily chart windows (`1mo`, `3mo`, `1y`)
|
|
||||||
- [ ] Decide whether richer financial statements should stay single-period or grow into multi-period fetch support
|
|
||||||
|
|
||||||
## 🚀 Publish Readiness (2026-04-03)
|
|
||||||
|
|
||||||
### P1 — Release blockers
|
|
||||||
- [x] Keep the installed binary name as `idx`; `Cargo.toml` already ships `[[bin]] name = "idx"` while the package remains `idx-cli`
|
|
||||||
- [x] Expose a default Nix package/app so the documented `nix run github:0xrsydn/idx-cli` path actually works
|
|
||||||
- [x] Restrict packaged crate contents so internal repo files like `.claude/`, `CLAUDE.md`, `AGENTS.md`, and agent-planning docs are not shipped to crates.io
|
|
||||||
- [x] Document real runtime dependencies and platform assumptions for `cargo install idx-cli`, including `curl-impersonate-chrome` for Yahoo auth and `mutool` for ownership PDF parsing
|
|
||||||
- [x] Add release/install verification to CI, at minimum `cargo package` and an install smoke path; include Nix app/package verification if the README keeps advertising `nix run`
|
|
||||||
|
|
||||||
### P2 — Pre-publish cleanup
|
|
||||||
- [x] Wire `--quiet` so non-essential `info:` / `warning:` output is actually suppressed
|
|
||||||
- [x] Fail fast on invalid `IDX_CACHE_QUOTE_TTL` / `IDX_CACHE_FUNDAMENTAL_TTL` env values instead of silently ignoring them
|
|
||||||
- [x] Propagate `-v` into the history/technical provider path so verbose Yahoo diagnostics can surface
|
|
||||||
- [x] Honor XDG overrides consistently for ownership DB/raw download defaults, not only config/cache
|
|
||||||
- [x] Tighten `scripts/live-smoke.sh` so it does not silently validate a stale `target/debug/idx` binary
|
|
||||||
- [x] Remove or fix the README reference to the non-existent `skills/` directory
|
|
||||||
|
|
||||||
## 📋 Backlog (per SPEC.md)
|
|
||||||
- [ ] `market summary` — IHSG index, market breadth
|
|
||||||
- [ ] `market movers` — top gainers/losers/volume
|
|
||||||
- [ ] `market sectors` — sector performance overview
|
|
||||||
- [ ] `screen query "<EXPR>"` — filter stocks by expression
|
|
||||||
- [ ] `screen presets` / `screen run <PRESET>` — built-in screener presets
|
|
||||||
- [ ] `watchlist` commands — create, manage, live watch
|
|
||||||
- [ ] `alerts` system (v0.2+) — price alerts with daemon
|
|
||||||
- [x] `completions <SHELL>` — shell completion generation
|
|
||||||
- [ ] CSV/TSV output formats
|
|
||||||
- [ ] Additional providers (Alpha Vantage, Twelve Data, IDX official)
|
|
||||||
|
|
||||||
## 🔬 Latest Smoke Findings (2026-04-02)
|
|
||||||
- [x] Behavioral coverage consolidation on `2026-10-02`: reduced the mock smoke matrix from 62 launch-heavy cases to 43 checks with separate stdout/stderr, single-document JSON validation, fixture values, and exact cache-data comparisons. Consolidated CLI tests from 89 to 62 stronger scenarios; retained all 173 unit/module regressions.
|
|
||||||
- [x] Strengthened CLI workflows cover cache misses/expiry/provider isolation, filtered and non-finite financial data, XLSX/ZIP ownership imports and queries, snapshot install/no-op/force, rejected updates preserving database bytes and query results, and 503-candidate screener ranking/full offline caches across four filters.
|
|
||||||
- [x] The stricter JSON checks exposed and fixed stderr pollution in unsupported `ownership import --fetch-bing` and totally failed `stocks compare`; partial comparisons still return available data and warnings, with `--quiet` suppressing those diagnostics.
|
|
||||||
- [x] Verification: all eleven checks passed via `nix flake check -L --keep-going` on x86_64-linux (173 unit/module tests, 62 CLI tests, 43 packaged-binary smoke cases). Throwaway fault injection proved malformed JSON, extra documents, wrong-stream output, and changed cache data are rejected. Direct packaged-CLI runs confirmed the 503-candidate/offline-prefix boundary, incomplete-total rejection without cache population, and both clean JSON error fixes. Temporary verification fixtures and injectors were removed.
|
|
||||||
- [x] Nix verification consolidation on `2026-10-02`: all eleven flake checks pass on x86_64-linux, including publisher regressions, installer `sh`/`dash` scenarios, ShellCheck, 173 unit/module tests, 89 CLI tests, and 62 mock smoke cases. CI and the pre-push hook use `nix flake check -L`; native macOS installer coverage is retained.
|
|
||||||
- [x] Checks and crate verification now share dev-profile dependency artifacts, while the installed application retains a separate release cache. Package verification logs confirm only `idx-cli` recompiles instead of rebuilding third-party dependencies. Narrowed source inputs and content-aware CI cache keys avoid unrelated rebuilds and allow source revisions to save new cached outputs.
|
|
||||||
- [x] Cache verification on `2026-10-02`: README/docs edits invalidated only the package check; the other ten derivations stayed unchanged. The real Prek pre-push gate passed, and an unchanged `nix flake check -L` completed in 11.69 seconds with no builds or compilation.
|
|
||||||
- [x] Combined PR verification on `2026-10-02`: JSON/offline-cache fixes, Nix/Crane CI, and market-data fixes pass `nix flake check -L` on x86_64-linux (173 unit tests, 89 CLI tests, 62 mock smoke cases), plus the publisher and `sh` installer regression scripts.
|
|
||||||
- [x] Fixed the integration review's screener truncation: full-list queries now re-request MSN's reported total rather than cap candidates at 500, reject incomplete totals, and use separate `all` cache keys. Live US top-performers returned 600 unique stocks; offline limits 3 and 25 matched the full result's prefixes. One upstream HTTP 499 occurred before the successful live check.
|
|
||||||
- [x] Removed the committed `result-1` Nix output symlink and ignored numbered result links.
|
|
||||||
- [x] Publish hardening batch on `2026-09-25`: `scripts/publish-ownership-snapshot.sh` now uploads an immutable content-addressed SQLite asset (date + sha256) and verifies it before publishing the manifest as the final commit point, so a partial upload no longer leaves a manifest that points at a missing asset; the manifest is trusted for a no-op only when the referenced asset matches the manifest SHA-256 and size at the configured repo/tag, and GitHub auth/network errors never become "up-to-date"
|
|
||||||
- [x] Publish integrity follow-up on `2026-09-25`: no-op and upload verification now check the full SHA-256, not just the filename and size; the publisher uses GitHub's `sha256:` asset digest when present and falls back to `gh release download` + local hashing for legacy assets, rejects a `download_url` that names a different repo/tag, and refuses to publish the manifest when the uploaded asset's SHA-256 does not match
|
|
||||||
- [x] `--history` is bounded (maximum `1000`, leading zeros stripped before conversion) so an enormous value cannot wrap negative and look already satisfied
|
|
||||||
- [x] The GitHub publish workflow now builds the packaged `idx-ownership-publisher` and routes publication through it with inputs passed as environment variables; the workflow artifact upload uses `if-no-files-found: ignore` so a legitimate up-to-date run does not fail
|
|
||||||
- [x] `--keep-workdir` no longer prints after the `RESULT:` line: the retained path is announced when the workdir is created and the EXIT cleanup is silent
|
|
||||||
- [x] `--history` changes are now honored for same-date data: the publisher compares `1 + min(--history, available XLSX months)` against the published `release_count` before and after building, validates `--history` before early exits, and does not rebuild on every run when the source has fewer historical months than requested
|
|
||||||
- [x] New offline regression harness `scripts/publish-ownership-snapshot-test.sh` (wired into `.github/workflows/ci.yml`) covers successful publish/no-op, partial SQLite upload and retry, manifest upload failure and retry, forced same-date refresh preserving the previous artifact, a missing referenced asset forcing republish, same-size remote corruption with and without digest metadata, upload corruption rejected before the manifest commit (digest and digest-less), a manifest URL for a different repo/tag, legacy PDF post-build no-op, history upgrade, limited available history, `--history` bounds, `--keep-workdir` output ordering, and auth/network failure not becoming up-to-date
|
|
||||||
- [x] Verification on `2026-09-25`: `bash -n` on the changed shell scripts and `scripts/publish-ownership-snapshot-test.sh` all passed
|
|
||||||
- [x] `ownership sync` now falls back to the built-in published manifest URL on `2026-04-12`, so plain `idx ownership sync` works on a clean machine once the public snapshot release is available
|
|
||||||
- [x] Ownership snapshot publishing now has a live-source maintainer path on `2026-04-11`: `scripts/build-latest-ownership-snapshot.sh` discovers the current supported `above1` IDX/KSEI PDF, imports it into an isolated temp DB, emits release-ready SQLite + manifest artifacts, and records the source IDX metadata in the manifest
|
|
||||||
- [x] Verification on `2026-04-11`: `nix develop --command cargo test`, `bash -n scripts/build-latest-ownership-snapshot.sh`, and a real `nix develop --command scripts/build-latest-ownership-snapshot.sh --idx-bin ./target/debug/idx --output-dir tmp/ownership-snapshot-live` run all passed; the current live source produced `7214` rows for `955` tickers with `as_of_date=2026-03-31`
|
|
||||||
- [x] Final release-hygiene pass on `2026-04-06`: crate metadata now declares `rust-version = 1.85`, README install docs now spell out Cargo helper-runtime expectations plus persistent `nix profile install`, and CI install smoke now runs the mock smoke matrix against the installed binary instead of only checking `idx version`
|
|
||||||
- [x] Verification on `2026-04-06`: `nix develop --command cargo build`, `nix develop --command cargo clippy -- -D warnings`, `nix develop --command cargo test`, `nix develop --command cargo package --allow-dirty --locked`, and `scripts/live-smoke.sh --bin ./tmp/release-install/bin/idx --no-build --mode mock` all passed
|
|
||||||
- [x] Publish review blocker batch on `2026-04-05`: the default Nix app/package path builds again without relying on an untracked runtime module, `ownership import --force` now re-imports the same release SHA atomically, and current ownership views (`ticker`, `entity`, `cross-holders`, `concentration`, `graph`) now scope KSEI data to the latest imported release instead of blending historical snapshots
|
|
||||||
- [x] Verification on `2026-04-05`: `nix develop --command cargo build`, `nix develop --command cargo clippy -- -D warnings`, `nix develop --command cargo test`, `nix build .#default`, `nix develop --command cargo package --allow-dirty --locked`, fresh `cargo install --path . --locked --root tmp/release-install`, `./tmp/release-install/bin/idx version`, and `nix run .#default -- version` all passed
|
|
||||||
- [x] Publish P2 cleanup batch on `2026-04-03`: `--quiet` now suppresses non-essential CLI warnings/info, invalid cache TTL env vars fail during startup, `-v` reaches Yahoo history diagnostics, ownership default DB/raw paths honor XDG overrides, and `scripts/live-smoke.sh` now refuses a stale `target/debug/idx` when build refresh is skipped
|
|
||||||
- [x] Verification on `2026-04-03`: `nix develop --command cargo build`, `nix develop --command cargo clippy -- -D warnings`, `nix develop --command cargo test`, `bash -n scripts/live-smoke.sh`, and `scripts/live-smoke.sh --mode mock --dry-run` all passed
|
|
||||||
- [x] Publish blocker batch on `2026-04-03`: crate packaging now excludes repo-internal agent harness files, the flake exports a default package/app for `nix run`, README install docs now describe runtime helper dependencies, and CI now checks package/install surfaces
|
|
||||||
- [x] Verification on `2026-04-03`: `nix develop --command cargo build`, `nix develop --command cargo clippy -- -D warnings`, `nix develop --command cargo test`, `nix develop --command cargo package --allow-dirty --locked`, fresh `cargo install --path . --locked --root tmp/release-install`, `./tmp/release-install/bin/idx version`, `nix build .#default`, and `nix run .#default -- version` all passed
|
|
||||||
- [x] P0 provider/capability routing is now centralized through `SelectedProvider`; `stocks` handlers no longer construct `MsnProvider` directly for MSN-only commands
|
|
||||||
- [x] MSN fundamentals now reject industry-only metrics with an explicit unsupported error; the normal MSN mock/live path uses company metrics only
|
|
||||||
- [x] MSN screener parsing now drops rows with missing/invalid/non-positive price data instead of synthesizing `0.0` price rows
|
|
||||||
- [x] Yahoo `429` retry/backoff is now centralized and regression-tested for chart and quote-summary fetches
|
|
||||||
- [x] `stocks technical` table output now says `Trend unavailable (need at least 200 daily candles)` when fewer than `200` daily candles are available
|
|
||||||
- [x] Verification on `2026-04-02`: `nix develop --command cargo build`, `nix develop --command cargo clippy -- -D warnings`, `nix develop --command cargo test`, `scripts/live-smoke.sh --mode mock`, and `scripts/live-smoke.sh --group live-table --group live-json --group routing --group cache --group errors` all passed (`tmp/live-smoke/20260402-122215`, `tmp/live-smoke/20260402-122218`)
|
|
||||||
- [x] Live smoke passed for shipped `stocks` commands: `quote`, `history`, `technical`, `growth`, `valuation`, `risk`, `fundamental`, `compare`, `profile`, `financials`, `earnings`, `sentiment`, `insights`, `news`, `screen`
|
|
||||||
- [x] Yahoo routing verified for live `quote` and `history`
|
|
||||||
- [x] `stocks history --history-provider msn` now works for supported MSN price-only chart windows; Yahoo remains the auto history source for full OHLCV
|
|
||||||
- [x] `ownership releases` works with a writable `ownership.db_path` and an empty DB
|
|
||||||
- [x] Regression coverage now verifies offline/cache parity for MSN-only commands (`stocks profile BBCA`)
|
|
||||||
- [x] `--offline --no-cache` now fails fast as an invalid flag combination instead of serving cache
|
|
||||||
- [x] Config/startup failures now honor the JSON error contract (`IDX_PROVIDER=bogus idx -o json version`)
|
|
||||||
- [x] Invalid `stocks screen --filter/--region` values now return validation errors
|
|
||||||
- [x] Added reusable smoke runner/checklist for shipped CLI surfaces (`scripts/live-smoke.sh`, `docs/SMOKE.md`)
|
|
||||||
- [x] Reusable smoke runner passes on deterministic mock suites: `general`/`cache`/`routing`/`errors`/`ownership` = 32/32 and shipped `stocks` mock matrix = 30/30 (`tmp/live-smoke/20260326-194907`, `tmp/live-smoke/20260326-194914`)
|
|
||||||
- [x] Full live MSN-only smoke now passes via runner for both table and JSON surfaces: 30/30 (`tmp/live-smoke/20260326-195853`)
|
|
||||||
- [x] Targeted no-cache live checks confirm `profile`, `insights`, and `financials` fixes against real MSN responses
|
|
||||||
- [x] Live `stocks profile BBCA` no-cache output now populates company/localized fields instead of the sparse top-level fallback
|
|
||||||
- [x] Live `stocks insights BBCA` JSON now returns a mixed-signal summary plus non-empty `last_updated`
|
|
||||||
- [x] Live `stocks financials BBCA` table no longer renders malformed negative numbers
|
|
||||||
- [x] Re-run full live smoke to reconfirm all MSN-only commands after the latest hardening fixes
|
|
||||||
- [x] `stocks financials BBCA` table now trims ISO timestamps from section headers and humanizes raw line-item keys
|
|
||||||
- [x] `stocks earnings BBCA` table now splits history vs forecast and formats annual periods, revenue values, and dates for table mode
|
|
||||||
- [x] `stocks financials` now supports section filters via `--statement income|balance|cashflow`, with filtered table output and JSON sections rendered as `null` when intentionally excluded
|
|
||||||
- [x] `stocks earnings` now supports `--forecast|--history` and `--annual|--quarterly`, so table and JSON output can be scoped without changing the cached source payload
|
|
||||||
- [x] JSON payload context is now less sparse for MSN `earnings`, `insights`, and `news`; each payload now carries the resolved stock symbol even when sourced from older cache entries
|
|
||||||
- [x] `stocks financials` JSON now normalizes `instrument.symbol` to the exchange-qualified ticker (for example `BBCA.JK`), and older cached payloads are backfilled at read time so users do not need to clear cache after upgrading
|
|
||||||
- [x] Product decision on `2026-04-02`: keep `screen` under `stocks` for now; revisit a dedicated surface only when `screen query` / `screen presets` graduate from backlog into a richer workflow
|
|
||||||
- [x] Fixture-backed parser and CLI JSON regression coverage now covers the remaining MSN-only `sentiment`, `news`, and `screen` commands
|
|
||||||
- [x] Fresh post-coverage live MSN smoke rerun still passes for table and JSON surfaces: 30/30 (`tmp/live-smoke/20260327-163403`)
|
|
||||||
- [x] MSN key-ratios parsing now tolerates stringified non-finite numeric sentinels (`Infinity`, `-Infinity`, `NaN`) as missing data instead of failing the entire fundamentals payload
|
|
||||||
- [x] Live direct CLI repros on `2026-04-13` confirmed the original failure class on `BUMI`, `ADRO`, and `AIMS`; an opt-in `scripts/live-smoke.sh --group live-nonfinite` group now covers those real ticker paths
|
|
||||||
- [x] Added `scripts/audit-msn-fundamentals.sh` for a full CLI valuation sweep across the IDX MSN symbol map; keep it as a provider-health audit, not part of the default smoke baseline
|
|
||||||
- [x] Real KSEI PDF import from local file now works again: `ownership import --file /Users/rasyidanakbar/Downloads/ksei_raw_data.pdf` imported `7261` rows for `955` tickers on `2026-03-28`, replacing the previous `6`-row/`1`-ticker failure mode
|
|
||||||
- [x] KSEI parser no longer depends on the old hardcoded column bounds fixture layout; it now reconstructs rows from `mutool` line output and handles the live `DATE + SHARE_CODE` merged segment plus `D`/`A` locality markers
|
|
||||||
- [x] A real IDX-hosted March 2026 ownership PDF URL was verified on `2026-03-29`, but only through `curl-impersonate` inside `nix develop`; plain `curl` still returns `403` from Cloudflare for the same asset
|
|
||||||
- [x] The current repo now has a concrete IDX-first roadmap: discover the hashed PDF URL from IDX pages, fetch it with browser impersonation, and only then layer snapshot publishing and `ownership sync`
|
|
||||||
- [x] The KSEI archive (`https://web.ksei.co.id/archive_download/holding_composition`) was verified as a secondary upstream that exposes monthly ZIP files and remains useful for fallback/cross-check work
|
|
||||||
- [x] Remote IDX discovery now targets the official BEI `Pengumuman` feed (`/id/berita/pengumuman/`) and its backing JSON endpoint (`/primary/NewsAnnouncement/GetAllAnnouncement`), which exposes hashed attachment URLs for ownership-related reports
|
|
||||||
- [x] New `ownership discover` CLI output now shows the latest hashed BEI ownership URLs across the current `above 5%`, `above 1%`, and `investor-type` report families
|
|
||||||
- [x] `ownership import --url https://www.idx.co.id/...pdf` now uses the same browser-impersonated download path as Yahoo auth
|
|
||||||
- [x] Reverse-engineering the current BEI feed shows the discoverable `Pemegang Saham di atas 1% (KSEI)` `lamp1` attachment matches the known-good raw KSEI holder-register layout and now imports successfully
|
|
||||||
- [x] Live `ownership import --url` against the discovered `above 1%` BEI `lamp1` attachment (`b9b638e5a8_8928aca255.pdf`) now succeeds end to end: `7261` rows for `955` tickers on `2026-03-29`, followed by successful `ownership releases` and `ownership ticker AADI --source ksei`
|
|
||||||
- [x] Product scope decision on `2026-03-30`: standardize supported remote import on the discoverable `above 1%` holder-register family; treat `above 5%` and `investor-type` PDFs as legacy / unsupported input
|
|
||||||
- [x] Live `ownership import --url` checks against the currently discoverable `above 5%` and `investor-type` BEI PDFs still fail with `no KSEI rows parsed from PDF`, which is expected until unsupported-family detection / rejection UX lands
|
|
||||||
- [x] Live `mutool` inspection of the current discoverable `investor-type` BEI attachment shows a stock-level aggregate matrix (`DATE`, `STOCK_CODE`, `NUMBER_OF_SHARES`, investor-type columns, holder-size buckets), not the holder-level KSEI register schema the current parser imports
|
|
||||||
- [x] Live `mutool` inspection of the current discoverable `above 5%` BEI attachment shows a member/tampungan report (`INVS`, member names, `KSEI UNTUK CLOSED MEMBER-...` labels), not the raw KSEI holder-register layout
|
|
||||||
- [x] `ownership discover` now defaults to the supported `above1` family, surfaces per-URL importability status, and orders the importable `lamp1` attachment first so `--limit 1` yields the current supported PDF URL
|
|
||||||
- [x] `ownership import --url` now accepts only direct PDF URLs; IDX listing/announcement page URLs fail fast with guidance to run `ownership discover`
|
|
||||||
- [x] Valid-but-unsupported BEI PDFs now fail before row parsing with explicit schema-aware errors (`announcement_wrapper`, legacy `above5`, legacy `investor-type`) instead of the old generic `no KSEI rows parsed from PDF` path
|
|
||||||
- [x] Regression coverage now covers default `ownership discover` behavior, status visibility, listing-page rejection, duplicate SHA imports, and explicit unsupported-schema detection with compact `stext` fixtures plus fake-`mutool` CLI tests
|
|
||||||
- [x] New `ownership-import` smoke coverage now discovers the current live `above1`/`above5`/`investor-type` URLs, imports the supported `above1` attachment successfully, and confirms the legacy families fail with explicit unsupported-schema UX (`tmp/live-smoke/20260330-160201`)
|
|
||||||
- [x] New ownership snapshot sync coverage now verifies manifest parsing, checksum validation, install into an empty temp data dir, preserved query behavior for `releases`/`ticker`/`changes`, no-op sync when current, and `--force` refresh via fixture-backed local manifests on `2026-03-31`
|
|
||||||
- [x] KSEI archive ZIP/TXT fallback import now works through `ownership import --file` for local `.zip` and `.txt` inputs, using synthetic investor-type/locality aggregate holders as a maintainer backstop rather than the primary product ingest
|
|
||||||
- [x] Batch 4 coverage now cross-checks the live-like `2026-02-27` `above1` PDF fixture against the matching KSEI archive bucket excerpt and verifies fallback import/query behavior for `releases`, `ticker`, and `changes`
|
|
||||||
|
|
||||||
## 🐛 Known Issues
|
|
||||||
- [ ] Yahoo Finance returns 429 from datacenter IPs occasionally
|
|
||||||
- [ ] SMA200 trend shows "Insufficient data" if Yahoo returns < 200 candles
|
|
||||||
- [x] KSEI ownership parser no longer leaks adjacent columns into `INVESTOR_NAME` on the March 2026 live PDF; direct CLI verification now returns clean holders like `AGUNG PERKASA INVESTINDO` / `CP` / `L`
|
|
||||||
- [x] Yahoo quoteSummary crumb auth — fixed via curl-impersonate-chrome (curl_chrome131)
|
|
||||||
- fc.yahoo.com → A3 cookie + query1 getcrumb → crumb, both sent to quoteSummary
|
|
||||||
- Requires nixpkgs#curl-impersonate-chrome in PATH (added to flake.nix + clan-private)
|
|
||||||
|
|
@ -9,4 +9,4 @@ Environment=HOME=/var/lib/idx-ownership-snapshot
|
||||||
EnvironmentFile=/etc/idx-ownership-snapshot.env
|
EnvironmentFile=/etc/idx-ownership-snapshot.env
|
||||||
# Build once with `nix build github:0xrsydn/idx-cli#ownership-publisher -o /opt/idx-ownership-publisher`.
|
# Build once with `nix build github:0xrsydn/idx-cli#ownership-publisher -o /opt/idx-ownership-publisher`.
|
||||||
# Idempotent: exits 0 with "RESULT: up-to-date" when nothing newer exists.
|
# Idempotent: exits 0 with "RESULT: up-to-date" when nothing newer exists.
|
||||||
ExecStart=/opt/idx-ownership-publisher/bin/idx-ownership-publish --output-dir /var/lib/idx-ownership-snapshot/current --repo 0xrsydn/idx-cli --release-tag ownership-snapshot-current --history 5
|
ExecStart=/opt/idx-ownership-publisher/bin/idx-ownership-publish --output-dir /var/lib/idx-ownership-snapshot/current --repo 0xrsydn/idx-cli --release-tag ownership-snapshot-current --history 2
|
||||||
|
|
|
||||||
|
|
@ -1,149 +0,0 @@
|
||||||
# Agent Integration Roadmap
|
|
||||||
|
|
||||||
**Status:** Planning — prerequisites are the remaining P0/P1 core gaps in `FEATURE_SPEC.md`
|
|
||||||
**Goal:** Make `idx-cli` a first-class tool for AI agent workflows (Claude Code, OpenClaw, MCP clients)
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Current Foundation
|
|
||||||
|
|
||||||
The CLI already has the building blocks:
|
|
||||||
|
|
||||||
- Structured JSON output (`-o json`) on every command
|
|
||||||
- Trait-based provider abstraction (mockable, extensible)
|
|
||||||
- Rich typed domain models (`Quote`, `Fundamentals`, `CompanyProfile`, `OwnershipHolder`, etc.)
|
|
||||||
- Structured error codes in JSON mode
|
|
||||||
- File-based cache with TTL, offline, and stale-cache fallback
|
|
||||||
- Feature-gated modules (ownership behind `--features ownership`)
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Tier 1 — MCP Server Mode (highest leverage)
|
|
||||||
|
|
||||||
Add `idx mcp-serve` to expose existing commands as MCP tools over stdio.
|
|
||||||
|
|
||||||
- Use `rust-mcp-sdk` crate for the stdio JSON-RPC transport
|
|
||||||
- Each existing command maps to one MCP tool (e.g. `get_stock_quote`, `get_financials`, `screen_stocks`, `get_ownership_ticker`)
|
|
||||||
- Existing JSON serialization becomes the tool response body — minimal new code
|
|
||||||
- Tool input schemas derive from clap structs or parallel types
|
|
||||||
- Configure in Claude Code:
|
|
||||||
```json
|
|
||||||
{ "mcpServers": { "idx": { "command": "idx", "args": ["mcp-serve"] } } }
|
|
||||||
```
|
|
||||||
|
|
||||||
**Why this first:** One feature makes `idx-cli` instantly usable from Claude Code, Claude Desktop, OpenClaw, and any MCP-compatible client. No wrapper scripts needed.
|
|
||||||
|
|
||||||
**Reference:** `financial-datasets/mcp-server` (1.7k stars), `tradingview-mcp` (500 stars), `twsemcp` (Taiwan Stock Exchange MCP — closest regional analog), `rust-mcp-sdk` (Rust MCP crate).
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Tier 2 — Composability
|
|
||||||
|
|
||||||
### Stdin symbol input
|
|
||||||
|
|
||||||
```bash
|
|
||||||
idx stocks screen --filter pe_lt:15 -o json | jq -r '.[].symbol' | idx stocks fundamental --stdin -o json
|
|
||||||
```
|
|
||||||
|
|
||||||
Agents generate symbol lists dynamically. Piping enables chained workflows without temp files.
|
|
||||||
|
|
||||||
### JSONL output
|
|
||||||
|
|
||||||
```bash
|
|
||||||
idx -o jsonl stocks quote BBCA BBRI
|
|
||||||
```
|
|
||||||
|
|
||||||
Newline-delimited JSON — better for streaming and piping than pretty-printed arrays.
|
|
||||||
|
|
||||||
### Batch mode
|
|
||||||
|
|
||||||
```bash
|
|
||||||
idx batch commands.jsonl
|
|
||||||
```
|
|
||||||
|
|
||||||
Where each line is a command spec:
|
|
||||||
```json
|
|
||||||
{"command": "stocks/quote", "symbols": ["BBCA", "BBRI"]}
|
|
||||||
{"command": "ownership/ticker", "symbol": "BBCA", "source": "ksei"}
|
|
||||||
```
|
|
||||||
|
|
||||||
Reduces process spawn overhead when agents run many queries in sequence.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Tier 3 — Schema Export
|
|
||||||
|
|
||||||
```bash
|
|
||||||
idx schema stocks/quote
|
|
||||||
```
|
|
||||||
|
|
||||||
Dumps the JSON Schema for a command's output. Agents use this to understand available fields without trial-and-error. Document which fields are guaranteed present vs. optional.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Tier 4 — Agent-Oriented Domain Features
|
|
||||||
|
|
||||||
### Watchlist as named symbol sets
|
|
||||||
|
|
||||||
```bash
|
|
||||||
idx watchlist create banks BBCA,BBRI,BMRI,BNGA
|
|
||||||
idx stocks quote --watchlist banks
|
|
||||||
```
|
|
||||||
|
|
||||||
Agents build and reuse curated lists across sessions.
|
|
||||||
|
|
||||||
### Composite analysis
|
|
||||||
|
|
||||||
```bash
|
|
||||||
idx stocks analyze BBCA -o json
|
|
||||||
```
|
|
||||||
|
|
||||||
Returns quote + fundamentals + ownership + sentiment in one call. Saves agents 4+ tool calls and reduces token overhead.
|
|
||||||
|
|
||||||
### Diff / change detection
|
|
||||||
|
|
||||||
```bash
|
|
||||||
idx stocks quote BBCA --diff
|
|
||||||
```
|
|
||||||
|
|
||||||
Compares against cached previous values and surfaces deltas. Agents monitor without maintaining their own state.
|
|
||||||
|
|
||||||
### CSV/TSV export
|
|
||||||
|
|
||||||
```bash
|
|
||||||
idx -o csv stocks screen --filter pe_lt:15
|
|
||||||
```
|
|
||||||
|
|
||||||
For agent pipelines that need tabular data (pandas, spreadsheets, database import).
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Tier 5 — Integration Hooks
|
|
||||||
|
|
||||||
- **Webhook primitives** — `idx watch BBCA --threshold price_gt:5000 --webhook http://...` for event-driven agent architectures
|
|
||||||
- **Session context** — `idx context set research-session-1` to tag cache entries, enabling isolated research sessions that don't pollute each other
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Execution Order
|
|
||||||
|
|
||||||
| Priority | Feature | Effort | Impact |
|
|
||||||
| --- | --- | --- | --- |
|
|
||||||
| 1 | MCP server mode | Medium | Unlocks all MCP clients |
|
|
||||||
| 2 | Stdin piping + JSONL | Small | Composability for chaining |
|
|
||||||
| 3 | Schema export | Small | Agent self-discovery |
|
|
||||||
| 4 | Watchlist + composite | Medium | Reduces agent round-trips |
|
|
||||||
| 5 | CSV export | Small | Tabular pipeline support |
|
|
||||||
| 6 | Webhooks + sessions | Large | Event-driven architecture |
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Prerequisites
|
|
||||||
|
|
||||||
Close the remaining core gaps first (`FEATURE_SPEC.md` P0/P1):
|
|
||||||
|
|
||||||
1. Unify provider and capability flow
|
|
||||||
2. Fix screener row hygiene
|
|
||||||
3. Decide fundamentals fallback policy
|
|
||||||
4. Harden Yahoo reliability edge cases
|
|
||||||
5. Add `financials` and `earnings` filter flags
|
|
||||||
|
|
@ -34,6 +34,13 @@
|
||||||
- **MSN Finance** — default provider. Rich data: quotes, fundamentals, profile, earnings, financials, sentiment, insights, news, screener, and explicit price-only chart history for supported IDX windows.
|
- **MSN Finance** — default provider. Rich data: quotes, fundamentals, profile, earnings, financials, sentiment, insights, news, screener, and explicit price-only chart history for supported IDX windows.
|
||||||
- **Yahoo Finance** — default/auto history source. Reliable OHLCV data via `/v8/finance/chart/`.
|
- **Yahoo Finance** — default/auto history source. Reliable OHLCV data via `/v8/finance/chart/`.
|
||||||
|
|
||||||
|
### MSN Endpoints
|
||||||
|
- `https://assets.msn.com/service/`: quotes, charts, equities, earnings,
|
||||||
|
sentiment, screener
|
||||||
|
- `https://api.msn.com/msn/v0/pages/finance/`: key ratios, insights, news feed
|
||||||
|
|
||||||
|
The public API key embedded in MSN Money lives in `src/api/msn/client.rs`.
|
||||||
|
|
||||||
### Hybrid History Strategy
|
### Hybrid History Strategy
|
||||||
When `history_provider = auto` (default):
|
When `history_provider = auto` (default):
|
||||||
1. Use Yahoo for history because it provides full OHLCV candles.
|
1. Use Yahoo for history because it provides full OHLCV candles.
|
||||||
|
|
@ -73,7 +80,7 @@ Factory functions:
|
||||||
|
|
||||||
Unlike the `stocks` module (live-fetch), ownership is **import-then-query**:
|
Unlike the `stocks` module (live-fetch), ownership is **import-then-query**:
|
||||||
|
|
||||||
1. `idx ownership import` — ETL pipeline: fetch PDF/API → parse → normalize → load SQLite
|
1. `idx ownership import` — ETL pipeline: fetch XLSX/PDF → parse → normalize → load SQLite
|
||||||
2. `idx ownership sync` — install a maintained SQLite snapshot via manifest + checksum validation
|
2. `idx ownership sync` — install a maintained SQLite snapshot via manifest + checksum validation
|
||||||
3. All query commands read from local `~/.local/share/idx/ownership.db`
|
3. All query commands read from local `~/.local/share/idx/ownership.db`
|
||||||
4. Fully offline after import/sync
|
4. Fully offline after import/sync
|
||||||
|
|
@ -81,7 +88,9 @@ Unlike the `stocks` module (live-fetch), ownership is **import-then-query**:
|
||||||
### Data Sources
|
### Data Sources
|
||||||
- **KSEI** — official ≥1% shareholder registry, published monthly by IDX. Through May 2026 it was a PDF announcement; from the 2026-05-29 report onward it is an XLSX workbook on the IDX "Data Kepemilikan Saham" page
|
- **KSEI** — official ≥1% shareholder registry, published monthly by IDX. Through May 2026 it was a PDF announcement; from the 2026-05-29 report onward it is an XLSX workbook on the IDX "Data Kepemilikan Saham" page
|
||||||
- **KSEI archive** — monthly ZIP/TXT balance-position matrix, used as a local fallback/backstop import path
|
- **KSEI archive** — monthly ZIP/TXT balance-position matrix, used as a local fallback/backstop import path
|
||||||
- **Bing Finance** — global institutional ownership (REST API, quarterly)
|
- **Bing Finance** — global institutional ownership. The schema, `--source bing`
|
||||||
|
queries, and `src/api/msn/bing.rs` exist, but `ownership import --fetch-bing`
|
||||||
|
is intentionally unsupported, so no ingest path populates Bing rows today
|
||||||
|
|
||||||
### Discovery
|
### Discovery
|
||||||
`idx ownership discover` merges two IDX sources:
|
`idx ownership discover` merges two IDX sources:
|
||||||
|
|
@ -123,7 +132,7 @@ CLI args → resolve symbol → provider.quote/history/fundamentals → render t
|
||||||
|
|
||||||
### Import-Query (ownership module)
|
### Import-Query (ownership module)
|
||||||
```
|
```
|
||||||
Import: PDF/API → parse → normalize → resolve entities → SQLite INSERT
|
Import: XLSX/PDF/archive → parse → normalize → resolve entities → SQLite INSERT
|
||||||
Query: CLI args → SQLite SELECT → render table/json (no network)
|
Query: CLI args → SQLite SELECT → render table/json (no network)
|
||||||
```
|
```
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -126,7 +126,6 @@ let conn = Connection::open(path)
|
||||||
- **Coverage over counts:** replacing redundant unit assertions with stronger integration scenarios must retain meaningful error, boundary, and data-integrity coverage.
|
- **Coverage over counts:** replacing redundant unit assertions with stronger integration scenarios must retain meaningful error, boundary, and data-integrity coverage.
|
||||||
|
|
||||||
## Git / VCS
|
## Git / VCS
|
||||||
- **jj (Jujutsu)** as local workflow, colocated with git
|
|
||||||
- Push via `nix develop --command git push` (for prek hooks)
|
- Push via `nix develop --command git push` (for prek hooks)
|
||||||
- Branch naming: `feat/<name>`, `fix/<name>`
|
- Branch naming: `feat/<name>`, `fix/<name>`, `docs/<name>`
|
||||||
- Commit messages: conventional commits (`feat:`, `fix:`, `refactor:`, `docs:`)
|
- Commit messages: conventional commits (`feat:`, `fix:`, `refactor:`, `docs:`)
|
||||||
|
|
|
||||||
|
|
@ -1,170 +1,148 @@
|
||||||
# Ownership Snapshot Publishing
|
# Ownership Snapshot Publishing
|
||||||
|
|
||||||
This document covers the maintainer workflow for producing and publishing the
|
Maintainer workflow for producing and publishing the snapshot that
|
||||||
ownership snapshot artifacts consumed by `idx ownership sync`.
|
`idx ownership sync` consumes. The consumer contract (manifest schema and
|
||||||
|
replacement rules) is in `docs/OWNERSHIP_SYNC.md`.
|
||||||
|
|
||||||
## Goals
|
## Release Layout
|
||||||
|
|
||||||
- Build the snapshot from the currently discoverable supported IDX/KSEI source,
|
Snapshot assets live on a dedicated GitHub release, separate from versioned app
|
||||||
not from an ad hoc local DB.
|
releases:
|
||||||
- Keep the published manifest URL stable for end users.
|
|
||||||
- Record source provenance in the manifest so the published SQLite artifact can
|
|
||||||
be traced back to the live IDX announcement and PDF URL used to build it.
|
|
||||||
|
|
||||||
## Recommended GitHub Releases Layout
|
|
||||||
|
|
||||||
Use a dedicated stable release tag for snapshot assets:
|
|
||||||
|
|
||||||
- tag: `ownership-snapshot-current`
|
- tag: `ownership-snapshot-current`
|
||||||
- manifest asset: `ownership-snapshot-manifest.json`
|
- manifest: `ownership-snapshot-manifest.json` (stable URL, the commit point)
|
||||||
- SQLite asset: `ownership-snapshot-YYYY-MM-DD-<sha256>.sqlite` (immutable, content-addressed)
|
- SQLite: `ownership-snapshot-YYYY-MM-DD-<sha256>.sqlite` (immutable, content-addressed)
|
||||||
|
|
||||||
Recommended public manifest URL:
|
|
||||||
|
|
||||||
```text
|
```text
|
||||||
https://github.com/0xrsydn/idx-cli/releases/download/ownership-snapshot-current/ownership-snapshot-manifest.json
|
https://github.com/0xrsydn/idx-cli/releases/download/ownership-snapshot-current/ownership-snapshot-manifest.json
|
||||||
```
|
```
|
||||||
|
|
||||||
Avoid `.../releases/latest/download/...` if normal app releases and snapshot
|
Never use `.../releases/latest/download/...`: the repo's latest release is an
|
||||||
publishes share the same repository. The repo's "latest" release can drift away
|
app release, not the snapshot.
|
||||||
from the ownership snapshot release.
|
|
||||||
|
|
||||||
## Manual Maintainer Flow
|
## Running the Publisher
|
||||||
|
|
||||||
Run the publisher helper inside `nix develop` so `mutool` and the
|
Packaged (recommended for hosts; no checkout or cargo build needed):
|
||||||
`curl-impersonate` helper are available:
|
|
||||||
|
```bash
|
||||||
|
nix build github:0xrsydn/idx-cli#ownership-publisher
|
||||||
|
./result/bin/idx-ownership-publish --output-dir /var/lib/idx/ownership-snapshot/current --history 2
|
||||||
|
./result/bin/idx-ownership-freshness --max-age-days 40
|
||||||
|
```
|
||||||
|
|
||||||
|
From a checkout:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
nix develop --command cargo build
|
|
||||||
nix develop --command scripts/publish-ownership-snapshot.sh \
|
nix develop --command scripts/publish-ownership-snapshot.sh \
|
||||||
|
--build \
|
||||||
--idx-bin ./target/debug/idx \
|
--idx-bin ./target/debug/idx \
|
||||||
--output-dir dist/ownership-snapshot \
|
--output-dir dist/ownership-snapshot \
|
||||||
--repo 0xrsydn/idx-cli \
|
--repo 0xrsydn/idx-cli \
|
||||||
--release-tag ownership-snapshot-current
|
--release-tag ownership-snapshot-current
|
||||||
```
|
```
|
||||||
|
|
||||||
The helper performs these steps:
|
`idx-ownership-publish` wraps `scripts/publish-ownership-snapshot.sh` with a
|
||||||
|
pinned `idx` (including curl-impersonate and mupdf) plus jq, gh, sqlite, and
|
||||||
|
curl. It needs `GH_TOKEN` with permission to upload release assets; the freshness
|
||||||
|
check needs no token.
|
||||||
|
|
||||||
|
## What It Does
|
||||||
|
|
||||||
1. `idx -o json ownership discover --family above1 --limit 50` (newest supported
|
1. `idx -o json ownership discover --family above1 --limit 50` (newest supported
|
||||||
report first; XLSX from the Data Kepemilikan Saham page since June 2026)
|
report first; XLSX from the Data Kepemilikan Saham page since June 2026)
|
||||||
2. verifies the discovered report is the current supported import path
|
2. verifies the discovered report is the supported import path
|
||||||
3. imports that report (plus `--history <n>` earlier months, oldest first) into
|
3. imports it plus `--history <n>` earlier months, oldest first, into an
|
||||||
an isolated temp ownership DB
|
isolated temp DB
|
||||||
4. checks that the imported release metadata is non-empty and tied to the same
|
4. checks the imported release metadata is non-empty and tied to the source URL
|
||||||
source URL
|
5. builds the SQLite artifact and manifest (`scripts/build-latest-ownership-snapshot.sh`,
|
||||||
5. runs `scripts/build-ownership-snapshot.sh` to emit the SQLite artifact and
|
which calls `scripts/build-ownership-snapshot.sh`) and records `source`
|
||||||
base manifest
|
provenance in the manifest
|
||||||
6. enriches the manifest with `source` provenance metadata
|
6. uploads using the protocol below
|
||||||
|
|
||||||
The resulting manifest records:
|
|
||||||
|
|
||||||
- `source.family`
|
|
||||||
- `source.listing_page_url`
|
|
||||||
- `source.query_url`
|
|
||||||
- `source.pdf_url`
|
|
||||||
- `source.title`
|
|
||||||
- `source.publish_date`
|
|
||||||
- `source.original_filename`
|
|
||||||
|
|
||||||
That metadata is additive. Existing sync clients can still parse the manifest.
|
|
||||||
|
|
||||||
## Publication Protocol
|
## Publication Protocol
|
||||||
|
|
||||||
The manifest is the commit point that `idx ownership sync` follows. The
|
The manifest is the commit point, so uploads are ordered:
|
||||||
publisher therefore never uploads the manifest and the SQLite asset in one
|
|
||||||
non-atomic call. It uses this order:
|
|
||||||
|
|
||||||
1. Build the snapshot and stage the manifest plus one content-addressed SQLite
|
1. Upload the content-addressed SQLite asset. Same content means same name, so
|
||||||
file.
|
retries are idempotent.
|
||||||
2. Upload the SQLite asset first. The name includes the artifact's SHA-256, so
|
2. Read the release assets back and verify the full SHA-256 and size (GitHub's
|
||||||
the same content always has the same name and a retry is idempotent.
|
`sha256:` digest when present, otherwise download and hash). Fail before
|
||||||
3. Read the release assets back and confirm the SQLite asset exists with the
|
touching the manifest if this does not match.
|
||||||
expected SHA-256 and size. Use GitHub's `sha256:` digest metadata when it is
|
3. Upload the manifest last and verify its `snapshot.sqlite_sha256`.
|
||||||
present; for legacy assets without a digest, download the asset through `gh`
|
|
||||||
and hash it. Fail before touching the manifest if verification fails.
|
|
||||||
4. Upload the manifest last and verify that its `snapshot.sqlite_sha256`
|
|
||||||
matches the staged manifest.
|
|
||||||
|
|
||||||
Consequences:
|
A partial upload exits non-zero and is repaired by the next run; previously
|
||||||
|
published manifests stay valid because old SQLite assets are retained. GitHub
|
||||||
|
does not replace the manifest transactionally, so a failed replacement can leave
|
||||||
|
it unavailable until a retry succeeds.
|
||||||
|
|
||||||
- A partial upload (manifest uploaded, SQLite missing) can no longer happen.
|
The publisher treats the published snapshot as current only when the manifest's
|
||||||
- If either upload fails, the job exits non-zero and the next run retries.
|
`download_url` names this repo and tag and the referenced asset matches its
|
||||||
Previously downloaded manifests remain valid because the old SQLite asset is retained.
|
SHA-256 and size. A missing or corrupt asset, a mismatched URL, an unreadable
|
||||||
- GitHub does not replace the manifest transactionally. A failed replacement
|
manifest, or any GitHub auth/network error means "publish", never "up to date".
|
||||||
can leave the manifest unavailable until a retry succeeds.
|
`--force` uploads regardless.
|
||||||
- The publisher treats a published manifest as current only when `download_url`
|
|
||||||
names this repo and release tag and the referenced asset matches the manifest
|
## Run Result
|
||||||
SHA-256 and size. A missing or corrupt asset, a mismatched URL, or any GitHub
|
|
||||||
authentication/network error means "publish", never "up to date".
|
The publisher is idempotent and prints exactly one final line:
|
||||||
|
|
||||||
|
| Last line | Exit | Meaning |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| `RESULT: published <as-of>` | 0 | a newer snapshot, or larger history, was uploaded |
|
||||||
|
| `RESULT: up-to-date <as-of>` | 0 | nothing to upload |
|
||||||
|
| `RESULT: FAILED stage=<stage> (exit N)` | N | failed in `arguments`, `build`, `preflight`, `discover`, `build-snapshot`, `stage-output`, `upload`, or `publish` |
|
||||||
|
|
||||||
|
`idx-ownership-freshness` is the independent alarm: it exits 1 when the published
|
||||||
|
`latest_as_of_date` is older than `--max-age-days` (default 40). The source lands
|
||||||
|
2-3 days after month end, so 40 days only trips when publishing has stopped.
|
||||||
|
|
||||||
## History Coverage
|
## History Coverage
|
||||||
|
|
||||||
`--history <n>` requests the latest report plus up to `<n>` earlier monthly
|
`--history <n>` (maximum 1000) requests the latest report plus up to `n` earlier
|
||||||
XLSX reports. The publisher compares the request with the months the source
|
monthly XLSX reports:
|
||||||
actually exposes:
|
|
||||||
|
|
||||||
- desired releases = `1 + min(--history, available earlier XLSX months)`
|
- desired releases = `1 + min(--history, available earlier XLSX months)`
|
||||||
- a published snapshot with the same as-of date and at least the desired number
|
- same as-of date with at least the desired release count: no-op
|
||||||
of releases is a no-op
|
- same as-of date with fewer releases: republish to add history
|
||||||
- a published snapshot with the same as-of date but fewer releases is
|
- a request larger than the source's history is accepted once, not rebuilt forever
|
||||||
republished to add history
|
|
||||||
- when the source has fewer historical months than requested, the publisher
|
|
||||||
accepts the smaller coverage and does not rebuild on every run
|
|
||||||
|
|
||||||
## Upload Step
|
Each extra month adds about 2.4 MB, and history is what lets
|
||||||
|
`idx ownership changes` work straight from a synced snapshot.
|
||||||
|
|
||||||
Prefer the publisher helper, which implements the protocol above:
|
## Snapshot Size Limit
|
||||||
|
|
||||||
```bash
|
idx clients up to v0.2.3 cap the snapshot download at **10 MiB** (10,485,760
|
||||||
scripts/publish-ownership-snapshot.sh \
|
bytes) and accept only uncompressed SQLite. A larger snapshot breaks
|
||||||
--idx-bin ./target/debug/idx \
|
`idx ownership sync` for every existing install (this happened on 2026-09-25
|
||||||
--output-dir dist/ownership-snapshot \
|
with `--history 5`).
|
||||||
--repo 0xrsydn/idx-cli \
|
|
||||||
--release-tag ownership-snapshot-current
|
|
||||||
```
|
|
||||||
|
|
||||||
If you upload by hand, upload these files to the `ownership-snapshot-current`
|
| `--history` | months | size (2026-08-31) |
|
||||||
GitHub release in this order:
|
| --- | --- | --- |
|
||||||
|
| 2 | 3 | 8,171,520 |
|
||||||
|
| 3 | 4 | 10,510,336 (too large) |
|
||||||
|
| 5 | 6 | 15,290,368 (too large) |
|
||||||
|
|
||||||
1. `dist/ownership-snapshot/ownership-snapshot-YYYY-MM-DD-<sha256>.sqlite`
|
The builder refuses snapshots above `--max-snapshot-bytes` (default 10485760),
|
||||||
2. `dist/ownership-snapshot/ownership-snapshot-manifest.json`
|
ending with `RESULT: FAILED stage=build-snapshot` and leaving the published
|
||||||
|
snapshot unchanged. Newer clients accept up to the manifest's `size_bytes` (at
|
||||||
|
most 1 GiB); raise both limits only once pre-v0.2.4 clients are rare.
|
||||||
|
|
||||||
Never upload the manifest before its SQLite asset is present. Only after this
|
## Automation
|
||||||
manual flow is reliable should the repo automate it in GitHub Actions.
|
|
||||||
|
|
||||||
## GitHub Actions Workflow
|
IDX has returned `403` to GitHub-hosted runners, so unattended publishing runs
|
||||||
|
on a self-hosted machine that IDX accepts:
|
||||||
|
|
||||||
The repo now includes a manual workflow at
|
- `contrib/systemd/idx-ownership-snapshot-publish.service` and `.timer` run the
|
||||||
`.github/workflows/publish-ownership-snapshot.yml`.
|
packaged publisher daily at 06:00 UTC with `GH_TOKEN` from
|
||||||
|
`/etc/idx-ownership-snapshot.env`. Wire `OnFailure=` to a notification.
|
||||||
|
- On NixOS, add this repo as a flake input and use
|
||||||
|
`inputs.idx-cli.packages.${system}.ownership-publisher`; `nix flake update idx-cli`
|
||||||
|
picks up fixes.
|
||||||
|
|
||||||
Current behavior:
|
`.github/workflows/publish-ownership-snapshot.yml` is a manual
|
||||||
|
(`workflow_dispatch`) fallback that runs the same packaged publisher with inputs
|
||||||
|
passed as environment variables.
|
||||||
|
|
||||||
- trigger: `workflow_dispatch` only
|
Observed XLSX upload times (HTTP Last-Modified) were the 2nd-3rd of each month
|
||||||
- builds the packaged publisher with `nix build .#ownership-publisher`
|
around 02:00-04:00 UTC; a daily idempotent run makes the exact day irrelevant.
|
||||||
- runs `idx-ownership-publish` from that package, which performs discovery,
|
If the monthly source has not appeared yet, the run should fail loudly and retry
|
||||||
import, snapshot build, and the full upload protocol in one pass
|
later rather than publish stale data.
|
||||||
- uploads any locally produced artifacts as a workflow artifact copy with
|
|
||||||
`if-no-files-found: ignore`, so a legitimate up-to-date run still succeeds
|
|
||||||
- does not run a separate release-create or asset-upload step
|
|
||||||
|
|
||||||
This is intentionally manual-first. Add a schedule only after a few successful
|
|
||||||
publish runs confirm the live source remains stable enough.
|
|
||||||
|
|
||||||
The workflow passes its inputs to the shell through environment variables and
|
|
||||||
routes publication through the same fixed publisher used by self-hosted hosts,
|
|
||||||
so it cannot drift from the CLI protocol.
|
|
||||||
|
|
||||||
## Self-Hosted Automation
|
|
||||||
|
|
||||||
GitHub-hosted Actions are still not sufficient for this job on their own. Live
|
|
||||||
verification has shown IDX returning `403` to GitHub-hosted runners during
|
|
||||||
discovery/import.
|
|
||||||
|
|
||||||
The recommended unattended path is therefore:
|
|
||||||
|
|
||||||
- keep the public artifacts on GitHub Releases
|
|
||||||
- run the publish job from a self-hosted machine that IDX accepts
|
|
||||||
- use a `systemd` oneshot service plus `systemd.timer` on that machine
|
|
||||||
|
|
||||||
See `docs/OWNERSHIP_SELF_HOSTED.md` for the reusable helper script, sample
|
|
||||||
systemd units, and the recommended split between this repo and `clan-private`.
|
|
||||||
|
|
|
||||||
|
|
@ -1,149 +0,0 @@
|
||||||
# Self-Hosted Ownership Snapshot Publish
|
|
||||||
|
|
||||||
This document covers the recommended self-hosted automation path for publishing
|
|
||||||
the monthly ownership snapshot consumed by `idx ownership sync`.
|
|
||||||
|
|
||||||
Use this path when:
|
|
||||||
|
|
||||||
- the repo's public release assets should stay on GitHub Releases
|
|
||||||
- IDX blocks GitHub-hosted Actions with `403`
|
|
||||||
- you have a VPS or other trusted machine that can reach the IDX source
|
|
||||||
|
|
||||||
## Why Self-Hosted
|
|
||||||
|
|
||||||
The current GitHub-hosted workflow remains useful as a manual reference, but it
|
|
||||||
is not reliable enough for unattended publishing because IDX has returned `403`
|
|
||||||
to GitHub-hosted runners. The self-hosted path keeps the end-user experience
|
|
||||||
simple:
|
|
||||||
|
|
||||||
- maintainers publish from a trusted machine
|
|
||||||
- end users still run plain `idx ownership sync`
|
|
||||||
|
|
||||||
No Nix setup is required for end users. Nix is only an implementation detail for
|
|
||||||
the maintainer host if that host already uses NixOS.
|
|
||||||
|
|
||||||
## Packaged Publisher (recommended)
|
|
||||||
|
|
||||||
The flake exposes a self-contained publisher, so the host needs no checkout,
|
|
||||||
`nix develop`, or cargo build at run time:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
nix build github:0xrsydn/idx-cli#ownership-publisher
|
|
||||||
./result/bin/idx-ownership-publish --output-dir /var/lib/idx/ownership-snapshot/current --history 5
|
|
||||||
./result/bin/idx-ownership-freshness --max-age-days 40
|
|
||||||
```
|
|
||||||
|
|
||||||
The publisher needs `GH_TOKEN` in its environment: a token that can create
|
|
||||||
and upload assets on the target repo's releases (the examples read it from
|
|
||||||
`/etc/idx-ownership-snapshot.env`; clan-private uses a Clan var). The
|
|
||||||
freshness check needs no token.
|
|
||||||
|
|
||||||
`idx-ownership-publish` wraps `scripts/publish-ownership-snapshot.sh` with a
|
|
||||||
pinned `idx` (itself wrapped with curl-impersonate and mupdf) plus jq, gh,
|
|
||||||
sqlite and curl. On NixOS, add this repo as a flake input and run
|
|
||||||
`inputs.idx-cli.packages.${system}.ownership-publisher`; bump the input
|
|
||||||
(`nix flake update idx-cli`) to pick up fixes.
|
|
||||||
|
|
||||||
## Run Semantics
|
|
||||||
|
|
||||||
The publisher is idempotent and prints exactly one final line:
|
|
||||||
|
|
||||||
| Last line | Exit | Meaning |
|
|
||||||
| --- | --- | --- |
|
|
||||||
| `RESULT: published <as-of>` | 0 | a newer snapshot, or a larger history, was uploaded |
|
|
||||||
| `RESULT: up-to-date <as-of>` | 0 | the published snapshot already has the latest report and requested history; nothing uploaded |
|
|
||||||
| `RESULT: FAILED stage=<stage> (exit N)` | N | failed in `arguments`, `build`, `preflight`, `discover`, `build-snapshot`, `stage-output`, `upload`, or `publish` |
|
|
||||||
|
|
||||||
It checks the published manifest and verifies that the manifest's `download_url`
|
|
||||||
names this repo and release tag, and that the referenced SQLite asset matches
|
|
||||||
the manifest SHA-256 and size, before it skips anything. It uses GitHub's
|
|
||||||
`sha256:` digest metadata when available and downloads legacy assets without a
|
|
||||||
digest to hash them. A missing or corrupt asset, a mismatched URL, an unreadable
|
|
||||||
manifest, or a GitHub authentication/network error is treated as "publish",
|
|
||||||
never as up to date. The upload path verifies the full SHA-256 of the uploaded
|
|
||||||
asset before it publishes the manifest.
|
|
||||||
For XLSX sources it also compares the requested `--history` coverage with the
|
|
||||||
months the source actually exposes, so a history increase is published once and
|
|
||||||
a request larger than the available history does not rebuild forever. `--history`
|
|
||||||
is bounded (maximum 1000) so an enormous value cannot wrap and look already
|
|
||||||
satisfied. The checks use discovery only, so a daily run stays cheap. `--force`
|
|
||||||
uploads regardless.
|
|
||||||
|
|
||||||
The SQLite asset is immutable and content-addressed. The publisher uploads and
|
|
||||||
verifies it before it uploads the manifest, which is the commit point consumers
|
|
||||||
follow. A partial upload therefore exits non-zero and is repaired by the next
|
|
||||||
run.
|
|
||||||
|
|
||||||
`idx-ownership-freshness` is the independent alarm: it exits 1 when the
|
|
||||||
published `latest_as_of_date` is older than `--max-age-days` (default 40). The
|
|
||||||
source is monthly and lands 2-3 days after month end, so 40 days only trips
|
|
||||||
when publishing has actually stopped working. Wire both to an `OnFailure=`
|
|
||||||
notification.
|
|
||||||
|
|
||||||
## Manual Command (from a checkout)
|
|
||||||
|
|
||||||
```bash
|
|
||||||
nix develop --command scripts/publish-ownership-snapshot.sh \
|
|
||||||
--build \
|
|
||||||
--idx-bin ./target/debug/idx \
|
|
||||||
--output-dir /var/lib/idx/ownership-snapshot/current \
|
|
||||||
--repo 0xrsydn/idx-cli \
|
|
||||||
--release-tag ownership-snapshot-current
|
|
||||||
```
|
|
||||||
|
|
||||||
## systemd Units
|
|
||||||
|
|
||||||
Examples: `contrib/systemd/idx-ownership-snapshot-publish.service` and
|
|
||||||
`contrib/systemd/idx-ownership-snapshot-publish.timer` (daily at 06:00 UTC).
|
|
||||||
|
|
||||||
Observed XLSX upload times (HTTP Last-Modified): 2026-06-03,
|
|
||||||
2026-07-02 03:55Z, 2026-08-02 01:50Z, 2026-09-02 03:51Z. With a daily,
|
|
||||||
idempotent run the exact day no longer matters.
|
|
||||||
|
|
||||||
Pass `--history <n>` to include earlier months so `idx ownership changes`
|
|
||||||
works straight from the synced snapshot (about 2.4 MB per extra month).
|
|
||||||
|
|
||||||
### Snapshot size limit
|
|
||||||
|
|
||||||
idx clients up to v0.2.3 cap the snapshot download at **10 MiB** (10,485,760
|
|
||||||
bytes) and only accept uncompressed SQLite. A larger snapshot makes
|
|
||||||
`idx ownership sync` fail for every existing install. This happened on
|
|
||||||
2026-09-25 with `--history 5` (15.3 MB).
|
|
||||||
|
|
||||||
| `--history` | months | size (2026-08-31) |
|
|
||||||
| --- | --- | --- |
|
|
||||||
| 2 | 3 | 8,171,520 |
|
|
||||||
| 3 | 4 | 10,510,336 (too large) |
|
|
||||||
| 5 | 6 | 15,290,368 (too large) |
|
|
||||||
|
|
||||||
The builder therefore refuses snapshots above `--max-snapshot-bytes`
|
|
||||||
(default 10485760). The run ends `RESULT: FAILED stage=build-snapshot` and the
|
|
||||||
published snapshot stays unchanged. Newer clients accept snapshots up to the
|
|
||||||
manifest's `size_bytes` (at most 1 GiB, buffered in memory);
|
|
||||||
raise `--max-snapshot-bytes` and `--history` only after clients without the
|
|
||||||
10 MiB cap are widespread.
|
|
||||||
|
|
||||||
## Clan Integration Later
|
|
||||||
|
|
||||||
The clean split for `clan-private` is:
|
|
||||||
|
|
||||||
1. keep the workflow logic in this repo
|
|
||||||
2. keep host-specific secrets and enablement in `clan-private`
|
|
||||||
|
|
||||||
Recommended integration shape on `greencloud-vps`:
|
|
||||||
|
|
||||||
- store a checked-out copy of `idx-cli` on the host, for example at `/srv/idx-cli`
|
|
||||||
- provide `GH_TOKEN` through Clan vars or another secret mechanism
|
|
||||||
- add a small NixOS module that installs/enables the service and timer
|
|
||||||
- set the module's `WorkingDirectory`, env file path, and publish directory
|
|
||||||
|
|
||||||
The existing pattern in `clan-private/modules/workspace-backup.nix` is a good
|
|
||||||
fit: a oneshot `systemd` service plus a `systemd.timer`.
|
|
||||||
|
|
||||||
## Operational Notes
|
|
||||||
|
|
||||||
- Keep `ownership-snapshot-current` separate from versioned app releases.
|
|
||||||
- Prefer manual runs first, then enable the timer after a few successful
|
|
||||||
publishes.
|
|
||||||
- If the monthly source has not appeared yet, the service should fail loudly and
|
|
||||||
be retried later; do not silently publish stale assumptions.
|
|
||||||
|
|
@ -4,8 +4,7 @@
|
||||||
|
|
||||||
The command is intentionally manifest-driven so the repo can publish snapshots in GitHub releases, object storage, or a local filesystem path without changing the CLI.
|
The command is intentionally manifest-driven so the repo can publish snapshots in GitHub releases, object storage, or a local filesystem path without changing the CLI.
|
||||||
|
|
||||||
The recommended GitHub publish layout and maintainer workflow are documented in
|
Publishing snapshots is a maintainer task; see `docs/OWNERSHIP_PUBLISH.md`.
|
||||||
`docs/OWNERSHIP_PUBLISH.md`.
|
|
||||||
|
|
||||||
## Consumer Inputs
|
## Consumer Inputs
|
||||||
|
|
||||||
|
|
@ -55,7 +54,8 @@ Current schema version: `1`
|
||||||
```
|
```
|
||||||
|
|
||||||
Semantics:
|
Semantics:
|
||||||
- `source` is optional provenance metadata describing the IDX/KSEI PDF used to build the snapshot.
|
- `source` is optional provenance metadata describing the IDX/KSEI report used to build the snapshot.
|
||||||
|
`source.pdf_url` is a legacy field name: since June 2026 it holds the XLSX URL.
|
||||||
- `download_url` points to the SQLite artifact itself. The filename is an
|
- `download_url` points to the SQLite artifact itself. The filename is an
|
||||||
implementation detail: published assets are immutable and content-addressed
|
implementation detail: published assets are immutable and content-addressed
|
||||||
(`ownership-snapshot-YYYY-MM-DD-<sha256>.sqlite`), so consumers must always
|
(`ownership-snapshot-YYYY-MM-DD-<sha256>.sqlite`), so consumers must always
|
||||||
|
|
@ -76,20 +76,3 @@ Without `--force`, sync behaves conservatively:
|
||||||
- local DB diverges at the same latest date: no-op and require `--force`
|
- local DB diverges at the same latest date: no-op and require `--force`
|
||||||
|
|
||||||
Replacement is staged through a temp file and validated before the local DB is swapped in.
|
Replacement is staged through a temp file and validated before the local DB is swapped in.
|
||||||
|
|
||||||
## Publisher Workflow
|
|
||||||
|
|
||||||
Use `scripts/build-ownership-snapshot.sh` to copy a vetted ownership DB into a publishable artifact plus manifest:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
scripts/build-ownership-snapshot.sh \
|
|
||||||
--db /path/to/ownership.db \
|
|
||||||
--output-dir dist/ownership-snapshot \
|
|
||||||
--base-url https://example.com/idx/ownership
|
|
||||||
```
|
|
||||||
|
|
||||||
If `--base-url` is omitted, the generated manifest uses the local artifact path as `download_url`, which is useful for local testing.
|
|
||||||
|
|
||||||
For the maintainer flow that discovers the latest supported IDX/KSEI source
|
|
||||||
first and then builds GitHub-release-ready assets, use
|
|
||||||
`scripts/build-latest-ownership-snapshot.sh`.
|
|
||||||
|
|
|
||||||
|
|
@ -105,8 +105,6 @@ responses honor the requested batch size so the complete-fetch path is exercised
|
||||||
Mocks do not verify current upstream availability, authentication, or response
|
Mocks do not verify current upstream availability, authentication, or response
|
||||||
drift. Live provider checks remain a separate, opt-in operation.
|
drift. Live provider checks remain a separate, opt-in operation.
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
## Modes
|
## Modes
|
||||||
|
|
||||||
- `live` runs the default real-network baseline: `general`, `live-table`, and `ownership`
|
- `live` runs the default real-network baseline: `general`, `live-table`, and `ownership`
|
||||||
|
|
@ -128,12 +126,10 @@ drift. Live provider checks remain a separate, opt-in operation.
|
||||||
|
|
||||||
## Notes
|
## Notes
|
||||||
|
|
||||||
- The runner forces `IDX_OUTPUT=table` as its default environment so table cases stay stable; JSON checks use `-o json` explicitly.
|
- The runner forces `IDX_OUTPUT=table` so table cases stay stable; JSON checks pass `-o json` explicitly.
|
||||||
- Cache-group warm cases clear the smoke cache before they run so each warm/offline/stale sequence starts clean and stale-cache assertions are not masked by earlier groups.
|
- Cache-group warm cases clear the smoke cache first so warm/offline/stale sequences start clean.
|
||||||
- Use `--bin <path> --no-build` when you want to validate an installed binary instead of the workspace `target/debug/idx` build.
|
- Use `--bin <path> --no-build` to validate an installed binary instead of `target/debug/idx`.
|
||||||
- Use `scripts/audit-msn-fundamentals.sh` for a full CLI valuation sweep across the IDX MSN symbol map. It is intentionally separate from the reusable smoke runner because it is a heavier provider-health audit, not a stable baseline check.
|
- `ownership-import` and `live-nonfinite` are opt-in (explicit `--group` or `--mode full`). Known non-finite repro tickers as of `2026-04-13`: `BUMI`, `ADRO`, `AIMS`.
|
||||||
- Ownership commands that need imported data are intentionally not part of the baseline runner yet. The current baseline only covers `ownership releases` and the known unsupported `ownership import --fetch-bing`.
|
- The baseline `ownership` group needs no imported data; `ownership sync` is covered by fixture-backed CLI tests instead.
|
||||||
- `ownership sync` is still primarily covered by fixture-backed CLI tests rather than the reusable smoke runner.
|
- `scripts/audit-msn-fundamentals.sh` is a heavier provider-health sweep over the IDX MSN symbol map, not a baseline check.
|
||||||
- The new `ownership-import` group is intentionally opt-in for explicit `--group ownership-import` runs or `--mode full`; it discovers live URLs first, imports the supported `above1` attachment into the temp DB, then asserts the current `above5` and `investor-type` URLs fail with explicit unsupported-schema UX.
|
- When a case fails, inspect its `.stdout`, `.stderr`, and `.validation` files under `tmp/live-smoke/.../logs/`.
|
||||||
- The new `live-nonfinite` group is intentionally opt-in only. As of `2026-04-13`, the known real repro tickers are `BUMI`, `ADRO`, and `AIMS`.
|
|
||||||
- When a case fails, inspect its `.stdout`, `.stderr`, and (for JSON cases) `.validation` files in `tmp/live-smoke/.../logs/` before updating `TODO.md` or `FEATURE_SPEC.md`.
|
|
||||||
|
|
|
||||||
Loading…
Add table
Add a link
Reference in a new issue