From 535ef64a0cb808018103490ca803b4f83dec34a5 Mon Sep 17 00:00:00 2001 From: hermes Date: Wed, 14 Oct 2026 08:18:17 +0000 Subject: [PATCH] docs: add idx-core / idx-mcp system design Accepted design for extracting idx-core as the shared data layer, the agent-native CLI output contract, idx-mcp tools, packaging, and the parity gates (golden output tests plus live A/B check) that the refactor must pass before merging to main. Co-Authored-By: Claude Opus 5.5 --- AGENTS.md | 4 + docs/DESIGN.md | 320 +++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 324 insertions(+) create mode 100644 docs/DESIGN.md diff --git a/AGENTS.md b/AGENTS.md index f4367cd..b137161 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -22,6 +22,7 @@ Docs: - `docs/CONVENTIONS.md`: types, naming, testing, and git rules - `docs/OWNERSHIP_SYNC.md` / `docs/OWNERSHIP_PUBLISH.md`: snapshot consumer contract / maintainer publishing - `docs/SMOKE.md`: Nix checks, test layout, and the live drift check +- `docs/DESIGN.md`: idx-core / idx-mcp design and parity gates - `ROADMAP.md`: open work ## Rules @@ -33,6 +34,9 @@ Docs: 5. Keep ownership queries (`releases`, `ticker`, `changes`, ...) working across every ingest path: XLSX, legacy PDF, archive ZIP/TXT, and snapshot sync. 6. `ownership import --fetch-bing` is intentionally unsupported. +7. Output parity is a contract: never edit golden files to make a change pass. + After changing CLI behavior, run the live check and report its summary; + report IDX `403`s explicitly rather than counting them as passes. ## Verify ```bash diff --git a/docs/DESIGN.md b/docs/DESIGN.md new file mode 100644 index 0000000..1125aae --- /dev/null +++ b/docs/DESIGN.md @@ -0,0 +1,320 @@ +# System Design: idx-core and its frontends + +**Status:** Accepted 2026-10-14. Implementation on `refactor/idx-core`; nothing implemented yet. + +## Goal + +Turn idx-cli's data layer into `idx-core`, a library that is the single way this +project retrieves Indonesian market, financial, and investor data. The CLI, an +MCP server, and later an HTTP API become thin frontends over it. + +Non-goals for now: new data sources, a hosted service, async rewrites of the +provider clients. + +## Architecture + +```text + Upstreams idx-core (library) Frontends +┌──────────────┐ ┌──────────────────────────────────────┐ ┌──────────────┐ +│ MSN Finance │──▶│ sources/ HTTP clients + parsers │ │ idx-cli │ +│ Yahoo │──▶│ cache/ TTL, offline, stale │──▶│ clap, table │ +│ IDX / KSEI │──▶│ store/ ownership SQLite │ ├──────────────┤ +└──────────────┘ │ market/ quotes, history, screen │──▶│ idx-mcp │ + │ financial/ fundamentals, reports │ │ MCP tools │ + │ investor/ ownership queries + sync │ ├──────────────┤ + │ analysis/ technical, fundamental │──▶│ idx-api │ + └──────────────────────────────────────┘ │ (later) │ + └──────────────┘ +``` + +Rule: frontends parse input and render output. Every fetch, cache decision, +filter, sort, and calculation lives in `idx-core`. + +## Data Domains + +| Domain | Data | Source today | Status | +| --- | --- | --- | --- | +| Market | quotes, OHLCV history, screener | MSN (default), Yahoo (OHLCV) | shipped | +| Market | IHSG index, breadth, movers, sectors | not chosen yet | `ROADMAP.md` | +| Financial | key ratios, statements, earnings, profile | MSN | shipped (single-period statements) | +| Financial | sentiment, insights, news | MSN | shipped | +| Investor | KSEI holders above 1%, entity holdings, changes, concentration, cross-holders, graph | IDX XLSX (legacy PDF), synced snapshot | shipped | +| Investor | institutional holders (Bing) | schema only | ingest unsupported | +| Analysis | technical signals, growth/valuation/risk | derived locally | shipped | + +Two access patterns, and the core API should keep them visibly separate: + +- **Live data** (market, financial): fetched from upstreams, cached on disk by TTL. +- **Local data** (investor): read from a local SQLite database that is + populated by `sync` or `import`. Queries never touch the network. + +All upstreams are unofficial or scraped. They change without notice, rate limit +datacenter IPs (Yahoo `429`), and sit behind Cloudflare (IDX `403`). The core +must report which source and how fresh each result is, and must never present +stale or fallback data as current. + +## idx-core API + +A single entry point holds configuration, providers, cache, and the DB path. +Domain methods hang off it. + +```rust +let idx = Idx::builder() + .config(CoreConfig::load()?) // provider, TTLs, paths; no output settings + .build()?; + +// market +let quotes: Fetched> = idx.market().quotes(&["BBCA", "BBRI"], CachePolicy::Default)?; +let candles: Fetched> = idx.market().history("BBCA", Period::M3, Interval::D1, CachePolicy::Default)?; +let screen: Fetched> = idx.market().screen(ScreenQuery { filter, region, limit })?; + +// financial +let report: Fetched = idx.financial().statements("BBCA", StatementFilter::all())?; +let earnings: Fetched = idx.financial().earnings("BBCA", EarningsFilter::default())?; + +// analysis (fetches its own inputs through the same cache) +let tech: Fetched = idx.analysis().technical("BBCA")?; + +// investor (local DB) +let holders: TickerOwnership = idx.investor().ticker("BBCA")?; +let status: OwnershipStatus = idx.investor().status()?; // releases, latest as-of, empty? +let sync: OwnershipSyncResult = idx.investor().sync(SyncOptions::default())?; +``` + +Key decisions: + +- **Blocking API.** `ureq` and `rusqlite` are blocking, and so is the CLI. + Async frontends (MCP, HTTP) call core through `spawn_blocking`. Revisit only + if an HTTP API needs high concurrency. +- **`Fetched` envelope** for live data: `{ data, source, fetched_at, stale }`. + This replaces today's stderr-only stale-cache warning, which an agent or API + client cannot see. +- **`CachePolicy`**: `Default | Offline | Refresh` replaces the `--offline` / + `--no-cache` flag handling now spread through `cli/stocks.rs`. +- **Explicit configuration.** Core reads no environment variables. `IDX_*` env + parsing, `config.toml` loading, and XDG resolution move to a `CoreConfig::load()` + helper that frontends call; tests construct `CoreConfig` directly. +- **Mock provider behind a `testing` feature.** Today `MockProvider` ships in the + release binary and reads fixtures relative to the working directory. +- **Typed inputs.** Screener filters, regions, statement and earnings filters + become enums with `FromStr`, so validation is identical across frontends. + +## What Moves Out of `src/cli/` + +| From | To | +| --- | --- | +| `fetch_with_cache`, `cache_bucket`, offline/stale handling (`cli/stocks.rs`) | `idx-core` cache layer, behind `CachePolicy` | +| `build_technical_report`, `fetch_fundamental_analysis_report` | `analysis` | +| `filter_financial_statements`, `filter_earnings_report` | `financial` | +| `screener_filter_key`, `screener_region_key`, `sort_screener_quotes`, full-list fetch | `market::screen` | +| ownership command orchestration (`cli/ownership.rs`) | `investor` | +| `IdxConfig.output`, `no_color` | stay in `idx-cli` | + +After the move, `cli/stocks.rs` and `cli/ownership.rs` should be argument +parsing plus a call into core plus rendering. + +## Error Model + +`IdxError` and its stable codes (`SYMBOLNOTFOUND`, `NOMARKETDATA`, `RATELIMITED`, +`UNSUPPORTED`, `INVALIDINPUT`, `OFFLINE`, `CACHEMISS`, `HTTP`, ...) live in core. +Each frontend maps them once: + +| Frontend | Mapping | +| --- | --- | +| CLI | unchanged: stderr, JSON `{error, code, message}` in JSON mode, non-zero exit | +| MCP | tool result with `isError: true` and the same `{code, message}` body | +| HTTP | status by class (`INVALIDINPUT` 400, `SYMBOLNOTFOUND` 404, `RATELIMITED` 429, upstream failures 502) plus the same body | + +## Agent-Native CLI Output + +Agents mostly drive the CLI through a shell and compose it with `jq`, `grep`, +`sed`, Python, and other commands. The CLI should be a well-behaved Unix filter +first and a pretty terminal tool second. + +### Output resolution + +`-o` flag > `IDX_OUTPUT` > `config.toml` > **auto**. Auto picks: + +- stdout is a terminal: `table` +- stdout is a pipe, file, or captured by a harness: `json` + +idx cannot tell whether an agent is calling it, only whether stdout is a +terminal (`std::io::IsTerminal`). Agent harnesses capture output, so they get +JSON with no flag. Two edge cases, both handled by the explicit overrides: + +- a human piping to `less` or `grep` gets JSON; use `-o table` +- a harness that runs commands inside a pseudo-terminal gets tables; set + `IDX_OUTPUT=json` in its environment + +### Formats + +| Format | Shape | Use | +| --- | --- | --- | +| `table` | human-readable | terminals | +| `json` | one document; pretty on a terminal, compact otherwise | default for pipes; `jq`, `json.load` | +| `jsonl` | list results: one record per line; single objects: one line | `head`, `grep`, streaming, `pandas.read_json(..., lines=True)` | + +Measured on fixtures, table output is 2-5x the bytes of JSON, and compact JSON +is another 20-30% smaller than pretty JSON. + +### Composability contract + +- stdout carries only data; diagnostics and warnings go to stderr. Already true. +- Errors in `json`/`jsonl` mode are one JSON object on stderr with stdout empty. + Already true. +- No ANSI color when stdout is not a terminal. +- Commands that take symbols accept `-` to read them from stdin, one per line or + whitespace-separated: + `idx stocks screen --filter low-pe | jq -r '.[].symbol' | idx stocks compare -` +- Exit codes by error class (today every error exits `1`): + + | Exit | Meaning | Codes | + | --- | --- | --- | + | 0 | success, including partial `compare` results | | + | 1 | other failure | `IO`, `DATABASEERROR`, `PARSEERROR`, ... | + | 2 | invalid usage or input | clap errors, `INVALIDINPUT`, `CONFIGERROR` | + | 3 | not found or no data | `SYMBOLNOTFOUND`, `NOMARKETDATA`, `CACHEMISS` | + | 4 | upstream unavailable, retry later | `HTTP`, `RATELIMITED`, `PROVIDERUNAVAILABLE`, `OFFLINE` | + | 5 | unsupported | `UNSUPPORTED` | + +- JSON keeps today's top-level shape: no `{data, meta}` wrapper, so `jq` paths + stay short and JSONL works per record. Freshness metadata (`Fetched`) is + exposed by MCP and the API; the CLI keeps its stderr stale-cache warning. +- Field names, units (basis points, whole IDR, WIB timestamps), and nullability + are part of the contract and documented in `docs/ARCHITECTURE.md`. + +Examples: + +```bash +idx stocks quote BBCA BBRI | jq -r '.[] | "\(.symbol) \(.price)"' +idx -o jsonl stocks history BBCA --period 1y | tail -5 +idx ownership ticker BBCA | jq '.holders | map(select(.percentage_bps > 500))' +idx -o jsonl stocks screen --filter top-performers \ + | python -c 'import sys, json; [print(json.loads(l)["symbol"]) for l in sys.stdin]' +``` + +This is a breaking change for anyone parsing table output from a pipe. Ship it +in a minor version with a release note; `general.output = "table"` restores the +old default. + +## Frontend: idx-mcp + +Stdio transport first, using the `rmcp` crate. Read-only by default. + +Tools are grouped by question rather than mirroring every CLI command, which +keeps the tool list short for agents: + +| Tool | Inputs | Backed by | +| --- | --- | --- | +| `get_quotes` | `symbols[]` | `market().quotes` | +| `get_price_history` | `symbol`, `period`, `interval` | `market().history` | +| `screen_stocks` | `filter`, `region`, `limit` | `market().screen` | +| `get_company` | `symbol`, `sections[]` (profile, financials, earnings, sentiment, insights, news) | `financial()` | +| `analyze_stock` | `symbol`, `kinds[]` (technical, growth, valuation, risk) | `analysis()` | +| `compare_stocks` | `symbols[]` | `analysis()` | +| `get_shareholders` | `ticker` | `investor().ticker` | +| `get_investor_holdings` | `entity` | `investor().entity` | +| `search_investors` | `query` | `investor().search` | +| `get_ownership_changes` | `from`, `to`, `ticker?` | `investor().changes` | +| `ownership_status` | none | `investor().status` | + +Ownership bootstrap: if the local DB is empty, investor tools return an error +telling the agent to call `ownership_sync`. That tool is opt-in through a +server flag, because it downloads several MB. + +## Frontend: idx-api (later) + +An HTTP server over the same core, e.g. `GET /v1/stocks/{symbol}/quote`. Before +building it, decide whether it is self-hosted only. Publicly re-serving MSN, +Yahoo, or IDX data raises terms-of-use and rate-limit questions that a local CLI +does not. + +## Packaging + +```text +Cargo workspace (this repo) +├── crates/idx-core library, published to crates.io +├── crates/idx-cli binary `idx` (package name unchanged) +├── crates/idx-mcp binary `idx-mcp` +npm/ +├── idx-cli existing wrapper +└── idx-mcp same postinstall download + SHA256SUMS +``` + +- One version for every crate and npm package, released from one `vX.Y.Z` tag. + Release assets add `idx-mcp-` next to the existing `idx-`. +- Nix: `packages.default` (idx) and `packages.idx-mcp`; both run through + `nix flake check`. +- The names `idx-core`, `idx-mcp`, and `idx-api` were free on crates.io and npm + on 2026-10-14. +- Bundling later is a packaging choice: an `idx mcp` subcommand behind a Cargo + feature, or an npm package that installs both binaries. + +## Parity and Verification + +The refactor must keep every shipped command's output identical. Two gates +enforce that, and neither may be skipped or weakened to make a change pass. + +### Golden output tests (offline, in `nix flake check`) + +Before any refactor code lands, capture the current CLI as golden files: + +- every `stocks`, `ownership`, `config`, `cache`, `version`, and `completions` + command, in both `table` and `json`, against the mock provider fixtures and a + fixture-built ownership DB +- the documented error cases (invalid input, unsupported provider, offline + without cache, empty ownership DB) +- stored as `tests/golden/.{stdout,stderr,exit}` and compared byte for + byte by one data-driven test in `tests/golden.rs` + +Regenerating goldens (`UPDATE_GOLDEN=1`) is only allowed for an intentional, +reviewed output change, and the diff must be in the same commit with the reason +in the message. Phases 1-3 must not change any golden file. Phase 0 changes only +what it documents: default format resolution, JSON whitespace when piped, and +exit codes; JSON content and table bytes stay identical. + +`tests/cli.rs` stays the behavioral suite and must keep passing unchanged. + +### Live A/B check (opt-in, before merging the refactor) + +Fixtures cannot show that the new code still talks to real upstreams correctly. +`scripts/live-smoke.sh --compare ` runs every command with the +pre-refactor binary and the new binary back to back against live MSN, Yahoo, and +IDX, then compares: + +- exit codes and JSON structure (keys and types) for every command +- values for slow-moving data (profile, financials, earnings, ownership) +- volatile fields (price, change, volume, news items) for type only + +Any coding agent that changes CLI behavior must run the live check and report +its summary before calling the work done. IDX ownership cases need a network +that Cloudflare does not block; when they return `403`, report that explicitly +instead of treating it as a pass. + +## Migration Plan + +Work happens on the `refactor/idx-core` branch. Each phase is its own commit +series with `nix flake check -L` green; `main` receives the branch only after +every phase passes both parity gates. + +0. **Golden baseline.** Add `tests/golden.rs` and the golden files from the + current `main`, plus `live-smoke.sh --compare`. No source changes. +1. **Agent-native output.** `auto` output, compact JSON, `jsonl`, stdin `-`, + and exit-code classes. Touches only `config.rs`, `output/`, and `main.rs`. +2. **Workspace split, no behavior change.** Move code into `crates/idx-core` + and `crates/idx-cli`. +3. **Pull logic into core.** Move the table above into core, add `CachePolicy`, + `Fetched`, `CoreConfig`, and typed filter enums. +4. **idx-mcp.** Implement the tool table over stdio, with fixture-backed tests. +5. **Release.** Workspace publishing, npm `idx-mcp`, release assets, flake package. + +## Open Questions + +1. Are the exit-code classes right, and should `compare` with partial results + still exit `0`? +2. MCP tool granularity: is the grouped table above right, or should it be one + tool per CLI command? +3. Should `ownership_sync` be available to agents by default? +4. Which source should back IHSG index, breadth, and sector data, and should it + land before or after idx-mcp? +5. Is idx-api self-hosted only?