# Feature Spec: Current Coverage and Remaining Core Work **Status:** Active - updated against the current repo state on 2026-03-31 **Current focus:** Close the remaining core gaps before expanding into new market and watchlist surfaces **Reference:** `origin/dev/rubick` (endpoint reference only, not the current implementation plan) --- ## Purpose This is no longer a "port MSN from zero" spec. The CLI already ships MSN-backed support for: - `profile` - `financials` - `earnings` - `sentiment` - `insights` - `news` - `screen` Use this document to track what is still open in core functionality. Use `TODO.md` as the execution log and smoke-history record. --- ## Verified Current State - Current automated coverage is `176` tests: `115` unit and `61` integration. - Reusable smoke coverage exists via `scripts/live-smoke.sh`; command groups are documented in `docs/SMOKE.md`. - The latest smoke notes in `TODO.md` report passing live table and JSON checks for all shipped `stocks` commands. - Cache/offline parity, JSON startup-error handling, screener input validation, and the recent MSN output cleanups have already been completed. - KSEI ownership import/query is now verified end-to-end from a local March 2026 PDF file into SQLite, with direct CLI reads through `ownership releases` and `ownership ticker`. - Remote ownership ingestion should now be treated as an IDX PDF discovery-and-fetch problem first, not a hardcoded KSEI PDF URL problem: direct IDX announcement PDFs exist, but the hashed asset URL must be discovered from an IDX listing/announcement surface and fetched with browser-like behavior. - The CLI now has an explicit `idx ownership discover` surface that enumerates the latest hashed BEI ownership report URLs and defaults to the supported `above1` family. - Live verification on `2026-03-29` confirmed that the parser-compatible discovered source is currently the `Pemegang Saham di atas 1% (KSEI)` `lamp1` attachment, and `idx ownership import --url` now works end to end for that discovered BEI PDF. - The currently discoverable `above 5%` and `investor-type` BEI families remain different schemas and are now classified and rejected explicitly during import instead of falling through to a generic zero-row parse failure. - Ownership smoke coverage now includes a dedicated `ownership-import` group that verifies live supported import plus expected unsupported-family failures. - `idx ownership sync` now installs maintained SQLite snapshots via a manifest-driven contract with checksum validation, conservative local replacement rules, and fixture-backed regression coverage for install/no-op/force-refresh behavior. This means the main gap is no longer endpoint coverage. The remaining work is architecture cleanup, a few correctness edge cases, and selective UX expansion on top of already-shipped commands. --- ## Coverage Snapshot | Area | CLI | Status | Notes | | --- | --- | --- | --- | | Quotes | `idx stocks quote` | Implemented | Cached, smoke-tested, and covered by integration tests | | History | `idx stocks history` | Partial | Works today via Yahoo/history-provider routing; explicit MSN history remains unsupported for IDX | | Technical | `idx stocks technical` | Implemented | Uses the cached history path | | Growth | `idx stocks growth` | Implemented | Shipped and exercised | | Valuation | `idx stocks valuation` | Implemented | Shipped and exercised | | Risk | `idx stocks risk` | Implemented | Shipped and exercised | | Fundamental | `idx stocks fundamental` | Implemented | Shipped and exercised | | Compare | `idx stocks compare` | Implemented | Shipped and exercised | | Company profile | `idx stocks profile` | Implemented | Cache/offline parity and fixture-backed coverage are in place | | Financial statements | `idx stocks financials` | Implemented with gaps | Output cleanup landed; statement filter flags and richer period controls are still missing | | Earnings | `idx stocks earnings` | Implemented with gaps | History/forecast split is rendered; filter flags are still missing | | Sentiment | `idx stocks sentiment` | Implemented | Fixture-backed CLI coverage exists | | Insights | `idx stocks insights` | Implemented | Summary/highlights/risks/`last_updated` mapping was corrected and tested | | News | `idx stocks news` | Implemented | Fixture-backed CLI coverage exists | | Screener | `idx stocks screen` | Implemented with gaps | Validation landed; expression/preset workflow is still future work | | MSN charts | `idx stocks history --history-provider msn` | Missing | Explicit MSN history still returns unsupported for IDX | | KSEI ownership import/query | `idx ownership import --file`, `idx ownership import --url`, `idx ownership releases`, `idx ownership ticker` | Implemented | Local PDF import and SQLite-backed query flow are verified against the March 2026 KSEI release; remote IDX import now works for the discovered `above 1%` `lamp1` BEI attachment, and legacy `above 5%` / `investor-type` BEI report families are rejected explicitly | | Ownership snapshot sync | `idx ownership sync` | Implemented | Manifest-driven SQLite snapshot install with checksum validation, conservative replacement/no-op rules, and publisher helper script | | Bing ownership CLI | `idx ownership import --fetch-bing` | Not implemented | Client groundwork exists, CLI import path is still deferred | --- ## Resolved Since The Previous Revision The following items should no longer be treated as active backlog in this spec: - CLI truth-pass baseline for shipped `stocks` commands - Unified cache, offline, stale-cache, and `--no-cache` handling across the core and MSN-only stock commands - Rejection of the conflicting `--offline --no-cache` flag combination - JSON-aware startup/config failures, not just runtime failures - Validation for `stocks screen --filter` and `--region` - `profile` output remapping to prefer company/localized fields - `insights` output remapping for summary, highlights, risks, and `last_updated` - Signed-number and table-label cleanup for `financials` - Table-mode cleanup for `earnings` - Baseline parser and CLI regression coverage for the shipped MSN-only command set - KSEI ownership parser hardening for the March 2026 live PDF layout, including the merged `DATE + SHARE_CODE` segment and `D`/`A` locality markers - Real KSEI ownership CLI verification from local file import into SQLite (`7261` rows across `955` tickers on `2026-03-28`) - Ownership remote-import hardening for the `above1` contract, including direct-PDF-only `--url` input, discovery status output, explicit legacy-schema rejection, and live ownership-import smoke coverage - Ownership snapshot publishing/sync contract, including `idx ownership sync`, manifest/checksum validation, conservative local replacement rules, and the `scripts/build-ownership-snapshot.sh` publisher helper If any of the above regress, capture that in `TODO.md` as a new finding rather than reopening the old section here wholesale. --- ## Remaining Core Gaps ### P0 - Correctness and architecture #### 1. Unify provider and capability flow Current state: - `src/cli/stocks.rs` still directly constructs `MsnProvider::new(false)` for `profile`, `financials`, `earnings`, `sentiment`, `insights`, `news`, and `screen`. - This keeps a split execution path alive even though cache/offline behavior is now mostly unified around `fetch_msn_with_cache`. Why it matters: - The architecture doc describes provider/capability-based flow, but the CLI still special-cases MSN-only commands at the handler layer. - Future features will be harder to extend cleanly if this split remains. Done when: - Normal command handlers stop constructing `MsnProvider` directly. - Provider selection and capability checks are centralized and consistent with `docs/ARCHITECTURE.md`. #### 2. Fix screener row hygiene for incomplete data Current state: - `src/api/msn/map.rs` still defaults missing screener price data to `0.0` when constructing `Quote` rows. Why it matters: - A zero-priced row is not the same thing as a valid priced stock. - This can silently admit incomplete market rows instead of rejecting or filtering them. Done when: - Screener rows without usable price data are filtered or rejected explicitly. - Regression tests cover the chosen behavior. #### 3. Decide the fundamentals fallback policy Current state: - Fundamentals can still fall back to industry-level metrics when company metrics are absent. Why it matters: - This can produce analysis that looks precise but is actually based on peer or category data. Decision needed: - Allow the fallback and annotate it clearly, or - reject the fallback and surface missing company data explicitly. Done when: - The policy is explicit in code and reflected in output semantics. #### 4. Harden Yahoo reliability edge cases Open issues: - Yahoo can still return `429` from datacenter IPs intermittently. - SMA200 trend output can still show "Insufficient data" when fewer than `200` candles are returned. Done when: - The retry/fallback story for Yahoo failures is deliberate and documented. - SMA200 behavior is either improved or clearly documented as expected. #### 5. Ownership fallback ingest and cross-check path Current state: - Remote IDX discovery/fetch is already in place for the supported `above1` holder-register family. - `idx ownership sync` now covers the maintained-snapshot distribution path on top of that import flow. - The KSEI ZIP/TXT archive remains the main unresolved ownership follow-up and should stay a fallback/cross-check input, not the primary ingest path. Why it matters: - The core ownership product path is now stable: local import, remote IDX discovery/import, and maintained snapshot sync all exist. - The remaining ownership work is about fallback resilience and validation against a secondary upstream. Done when: - The KSEI ZIP/TXT archive ingest shape is defined. - At least one monthly archive release is cross-checked against the IDX-PDF-derived SQLite state. - Fallback ingest produces query-compatible data for `ownership releases`, `ticker`, and `changes`. Roadmap: 1. Add KSEI ZIP/TXT ingest later as fallback or cross-check input. 2. Decide whether that fallback stays maintainer-only or becomes a user-facing alternative path. ### P1 - UX and output contract cleanup Tasks: - Add `financials` filters such as `--statement income|balance|cashflow`. - Add `earnings` filters such as `--forecast|--history` and `--annual|--quarterly`. - Review JSON payload consistency where symbol or context fields are still sparse. - Decide whether `screen` stays under `stocks` long term or graduates into a richer dedicated surface later. Done when: - Existing shipped commands are easier to drive without changing product scope. ### P2 - Next feature work after the above is green Priority order: 1. MSN Charts / `Finance/Charts` - Reuse the existing `idx stocks history` command. - Decide how to handle price-only timeframes safely. 2. Bing ownership CLI integration - Reuse the existing client groundwork in `src/api/msn/bing.rs`. - Define the import shape and output contract for `idx ownership import --fetch-bing`. 3. Ownership fallback ingest and cross-check work - Treat the KSEI archive (`https://web.ksei.co.id/archive_download/holding_composition`) as fallback/backstop input, not the primary product ingest path. - Define the ZIP/TXT ingest shape and verify compatibility with the IDX-PDF-derived SQLite contract. 4. Richer financial statements - Decide whether to stay with the current single-period model or add multi-period fetch support. 5. New user-facing surfaces from `TODO.md` - `market summary` - `market movers` - `market sectors` - `screen query` - `screen presets` - `watchlist` - `alerts` --- ## Verification Gate Do not treat a core refactor or new provider feature as complete until all of the following are true: - `cargo build` passes - `cargo clippy -- -D warnings` passes - `cargo test` passes - `scripts/live-smoke.sh --mode mock` passes - the relevant live smoke groups pass for changed user-facing behavior - `TODO.md` is updated with any new smoke finding, regression, or behavior change Keep the detailed reusable smoke commands in `docs/SMOKE.md`. Do not duplicate per-run results in this spec. --- ## Endpoint Reference Appendix MSN API key (public, embedded in MSN Money website): ``` 0QfOX3Vn51YCzitbLaRkTTBadtWpgTN8NZLW0C1SEM ``` Base URLs: - `https://assets.msn.com/service/` - core market data (Quotes, Charts, Equities, Earnings, Sentiment, Screener) - `https://api.msn.com/msn/v0/pages/finance/` - extended data (key ratios, insights, news feed) - `https://services.bingapis.com/contentservices-finance.hedgefunddataprovider/api/v1/` - Bing ownership data Keep this appendix for endpoint discovery and future work. Use the sections above as the actual implementation plan.