idx-cli/FEATURE_SPEC.md

Feature Spec: Current Coverage and Remaining Core Work

Status: Active - updated against the current repo state on 2026-03-28 Current focus: Close the remaining core gaps before expanding into new market and watchlist surfaces Reference: origin/dev/rubick (endpoint reference only, not the current implementation plan)

---

Purpose

This is no longer a "port MSN from zero" spec.

The CLI already ships MSN-backed support for:

• profile
• financials
• earnings
• sentiment
• insights
• news
• screen

Use this document to track what is still open in core functionality. Use TODO.md as the execution log and smoke-history record.

---

Verified Current State

• Current automated coverage is 153 tests: 102 unit and 51 integration.
• Reusable smoke coverage exists via scripts/live-smoke.sh; command groups are documented in docs/SMOKE.md.
• The latest smoke notes in TODO.md report passing live table and JSON checks for all shipped stocks commands.
• Cache/offline parity, JSON startup-error handling, screener input validation, and the recent MSN output cleanups have already been completed.
• KSEI ownership import/query is now verified end-to-end from a local March 2026 PDF file into SQLite, with direct CLI reads through ownership releases and ownership ticker.
• Remote ownership ingestion should now be treated as an IDX PDF discovery-and-fetch problem first, not a hardcoded KSEI PDF URL problem: direct IDX announcement PDFs exist, but the hashed asset URL must be discovered from an IDX listing/announcement surface and fetched with browser-like behavior.
• The CLI now has an explicit idx ownership discover surface that enumerates the latest hashed BEI ownership report URLs.
• Live verification on 2026-03-29 confirmed that the parser-compatible discovered source is currently the Pemegang Saham di atas 1% (KSEI) lamp1 attachment, and idx ownership import --url now works end to end for that discovered BEI PDF.
• The currently discoverable above 5% and investor-type BEI families remain different schemas and should now be treated as legacy / unsupported input outside the current holder-register parser contract.

This means the main gap is no longer endpoint coverage. The remaining work is architecture cleanup, a few correctness edge cases, and selective UX expansion on top of already-shipped commands.

---

Coverage Snapshot

Area	CLI	Status	Notes
Quotes	idx stocks quote	Implemented	Cached, smoke-tested, and covered by integration tests
History	idx stocks history	Partial	Works today via Yahoo/history-provider routing; explicit MSN history remains unsupported for IDX
Technical	idx stocks technical	Implemented	Uses the cached history path
Growth	idx stocks growth	Implemented	Shipped and exercised
Valuation	idx stocks valuation	Implemented	Shipped and exercised
Risk	idx stocks risk	Implemented	Shipped and exercised
Fundamental	idx stocks fundamental	Implemented	Shipped and exercised
Compare	idx stocks compare	Implemented	Shipped and exercised
Company profile	idx stocks profile	Implemented	Cache/offline parity and fixture-backed coverage are in place
Financial statements	idx stocks financials	Implemented with gaps	Output cleanup landed; statement filter flags and richer period controls are still missing
Earnings	idx stocks earnings	Implemented with gaps	History/forecast split is rendered; filter flags are still missing
Sentiment	idx stocks sentiment	Implemented	Fixture-backed CLI coverage exists
Insights	idx stocks insights	Implemented	Summary/highlights/risks/last_updated mapping was corrected and tested
News	idx stocks news	Implemented	Fixture-backed CLI coverage exists
Screener	idx stocks screen	Implemented with gaps	Validation landed; expression/preset workflow is still future work
MSN charts	idx stocks history --history-provider msn	Missing	Explicit MSN history still returns unsupported for IDX
KSEI ownership import/query	idx ownership import --file, idx ownership import --url, idx ownership releases, idx ownership ticker	Implemented with gaps	Local PDF import and SQLite-backed query flow are verified against the March 2026 KSEI release; remote IDX import now works for the discovered above 1% lamp1 BEI attachment, while legacy above 5% and investor-type BEI report families still need explicit unsupported-input handling
Bing ownership CLI	idx ownership import --fetch-bing	Not implemented	Client groundwork exists, CLI import path is still deferred

---

Resolved Since The Previous Revision

The following items should no longer be treated as active backlog in this spec:

• CLI truth-pass baseline for shipped stocks commands
• Unified cache, offline, stale-cache, and --no-cache handling across the core and MSN-only stock commands
• Rejection of the conflicting --offline --no-cache flag combination
• JSON-aware startup/config failures, not just runtime failures
• Validation for stocks screen --filter and --region
• profile output remapping to prefer company/localized fields
• insights output remapping for summary, highlights, risks, and last_updated
• Signed-number and table-label cleanup for financials
• Table-mode cleanup for earnings
• Baseline parser and CLI regression coverage for the shipped MSN-only command set
• KSEI ownership parser hardening for the March 2026 live PDF layout, including the merged DATE + SHARE_CODE segment and D/A locality markers
• Real KSEI ownership CLI verification from local file import into SQLite (7261 rows across 955 tickers on 2026-03-28)

If any of the above regress, capture that in TODO.md as a new finding rather than reopening the old section here wholesale.

---

Remaining Core Gaps

P0 - Correctness and architecture

1. Unify provider and capability flow

Current state:

• src/cli/stocks.rs still directly constructs MsnProvider::new(false) for profile, financials, earnings, sentiment, insights, news, and screen.
• This keeps a split execution path alive even though cache/offline behavior is now mostly unified around fetch_msn_with_cache.

Why it matters:

• The architecture doc describes provider/capability-based flow, but the CLI still special-cases MSN-only commands at the handler layer.
• Future features will be harder to extend cleanly if this split remains.

Done when:

• Normal command handlers stop constructing MsnProvider directly.
• Provider selection and capability checks are centralized and consistent with docs/ARCHITECTURE.md.

2. Fix screener row hygiene for incomplete data

Current state:

• src/api/msn/map.rs still defaults missing screener price data to 0.0 when constructing Quote rows.

Why it matters:

• A zero-priced row is not the same thing as a valid priced stock.
• This can silently admit incomplete market rows instead of rejecting or filtering them.

Done when:

• Screener rows without usable price data are filtered or rejected explicitly.
• Regression tests cover the chosen behavior.

3. Decide the fundamentals fallback policy

Current state:

• Fundamentals can still fall back to industry-level metrics when company metrics are absent.

Why it matters:

• This can produce analysis that looks precise but is actually based on peer or category data.

Decision needed:

• Allow the fallback and annotate it clearly, or
• reject the fallback and surface missing company data explicitly.

Done when:

• The policy is explicit in code and reflected in output semantics.

4. Harden Yahoo reliability edge cases

Open issues:

• Yahoo can still return 429 from datacenter IPs intermittently.
• SMA200 trend output can still show "Insufficient data" when fewer than 200 candles are returned.

Done when:

• The retry/fallback story for Yahoo failures is deliberate and documented.
• SMA200 behavior is either improved or clearly documented as expected.

5. Automate IDX ownership source discovery and fetch

Current state:

• idx ownership import --file <local-pdf> is now working and verified against the March 2026 KSEI release.
• Direct IDX announcement PDFs exist, but the monthly file path is not safely hardcodable because the asset filename is hashed.
• The official BEI listing page for this feed is https://www.idx.co.id/id/berita/pengumuman/, and the page’s own Nuxt client fetches announcement data from GET /primary/NewsAnnouncement/GetAllAnnouncement.
• The BEI endpoint is now reverse-engineered enough for idx ownership discover to locate the current Pemegang Saham di atas 1% (KSEI), Pemegang Saham di atas 5% (KSEI), and Kepemilikan Saham Perusahaan Tercatat Berdasarkan Tipe Investor hashed PDFs.
• Live verification on 2026-03-29 shows that the discoverable above 1% lamp1 attachment matches the raw KSEI holder-register shape that the current parser imports successfully.
• Live verification on 2026-03-29 also shows that the currently discoverable above 5% and investor-type BEI families do not match the holder-register schema the current parser imports.
• Product direction as of 2026-03-30 is to standardize on the above 1% holder-register structure for remote import; the other discovered families are legacy inputs that should be rejected clearly rather than parsed.
• Plain curl/ureq requests to IDX-hosted PDFs still get 403 from Cloudflare, while curl-impersonate inside the project nix develop environment has already been verified to return a real PDF for a March 2026 ownership source URL.
• The KSEI archive remains a secondary upstream and currently exposes monthly ZIP files that can be used later as fallback or cross-check input, but it is no longer the primary roadmap target.

Why it matters:

• The ownership feature now parses and stores live ownership data correctly from both local files and the currently discoverable above 1% BEI lamp1 attachment.
• The remaining gap is not basic remote import anymore; it is hardening the supported above 1% path and clearly rejecting other discovered BEI schemas that do not match the holder-register contract.

Done when:

• The CLI can discover the latest parser-compatible IDX ownership PDF URL from an IDX listing/announcement surface without hardcoded monthly paths.
• Remote fetches use the same browser-impersonation strategy already established elsewhere in the repo instead of the current bare ureq path.
• idx ownership import --url can fetch and parse the current above 1% IDX-hosted PDF reliably.
• Unsupported BEI report families are classified and rejected explicitly instead of failing later with a generic zero-row parse error.

Roadmap:

1. Keep discovery and import flows centered on the parser-compatible above 1% family and its lamp1 attachment.
2. Reuse the existing curl-impersonate pattern for browser-like PDF fetches.
3. Add schema classification / unsupported-input UX for the legacy above 5% and investor-type BEI PDFs.
4. Decide whether the default discover output should be above1-first, with legacy families retained only for diagnostic use.
5. Publish maintained SQLite snapshot artifacts and add idx ownership sync after remote IDX import is stable.
6. Optionally add KSEI ZIP/TXT ingest later as fallback or cross-check input.

P1 - UX and output contract cleanup

Tasks:

• Add financials filters such as --statement income|balance|cashflow.
• Add earnings filters such as --forecast|--history and --annual|--quarterly.
• Review JSON payload consistency where symbol or context fields are still sparse.
• Decide whether screen stays under stocks long term or graduates into a richer dedicated surface later.

Done when:

• Existing shipped commands are easier to drive without changing product scope.

P2 - Next feature work after the above is green

Priority order:

1. MSN Charts / Finance/Charts

   • Reuse the existing idx stocks history command.
   • Decide how to handle price-only timeframes safely.

2. Bing ownership CLI integration

   • Reuse the existing client groundwork in src/api/msn/bing.rs.
   • Define the import shape and output contract for idx ownership import --fetch-bing.

3. Ownership sync and snapshot distribution

   • Keep IDX PDF discovery/fetch as the first milestone before starting this work.
   • Publish maintained SQLite snapshot artifacts and add idx ownership sync.
   • Treat the KSEI archive (https://web.ksei.co.id/archive_download/holding_composition) as fallback/backstop input, not the primary product ingest path.

4. Richer financial statements

   • Decide whether to stay with the current single-period model or add multi-period fetch support.

5. New user-facing surfaces from TODO.md

   • market summary
   • market movers
   • market sectors
   • screen query
   • screen presets
   • watchlist
   • alerts

---

Verification Gate

Do not treat a core refactor or new provider feature as complete until all of the following are true:

• cargo build passes
• cargo clippy -- -D warnings passes
• cargo test passes
• scripts/live-smoke.sh --mode mock passes
• the relevant live smoke groups pass for changed user-facing behavior
• TODO.md is updated with any new smoke finding, regression, or behavior change

Keep the detailed reusable smoke commands in docs/SMOKE.md. Do not duplicate per-run results in this spec.

---

Endpoint Reference Appendix

MSN API key (public, embedded in MSN Money website):

0QfOX3Vn51YCzitbLaRkTTBadtWpgTN8NZLW0C1SEM

Base URLs:

• https://assets.msn.com/service/ - core market data (Quotes, Charts, Equities, Earnings, Sentiment, Screener)
• https://api.msn.com/msn/v0/pages/finance/ - extended data (key ratios, insights, news feed)
• https://services.bingapis.com/contentservices-finance.hedgefunddataprovider/api/v1/ - Bing ownership data

Keep this appendix for endpoint discovery and future work. Use the sections above as the actual implementation plan.