Phase 0 of docs/DESIGN.md, before any refactor code:
- tests/golden.rs runs every case in tests/golden/cases.toml (126
outputs across help, completions, config, cache, all stocks commands
on MSN and Yahoo mocks, error paths, and ownership queries on a seeded
fixture DB) and compares exit code, stdout, and stderr byte for byte.
Regenerate only with UPDATE_GOLDEN=1 for a reviewed output change.
- scripts/live-smoke.sh --compare <old-idx> runs each live case with
both binaries and fails on differing exit codes, added/missing/retyped
JSON keys, or changed values for slow-moving data.
- Exclude golden files from the whitespace/EOF pre-commit fixers, which
would otherwise rewrite pinned help output.
Both gates were fault-injected: a one-word table change fails exactly
the three history goldens, and a binary dropping profile.long_name fails
only the profile live comparison.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- --offline now warns when it serves a cache entry past its TTL; before,
arbitrarily old data was returned with no signal at all
- version, cache info and cache clear honour -o json
- ownership resolve map/merge honour -o json
- stocks quote/compare require at least one symbol instead of printing []
- cache schema-mismatch notice goes through runtime::info so --quiet
silences it, instead of a raw debug eprintln
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Q2PufCUdvVZoVW1Dy53js
Addresses review on #31:
- the snapshot download limit is the manifest's size_bytes (+1, since
ureq fails once a body reaches its limit), capped at 1 GiB; a body
larger than advertised fails during download. New CLI test; the
>10 MiB test also caught the off-by-one.
- a refused (oversized) standalone build removes its sqlite and
manifest from --output-dir.
- docs: downloads are buffered, not streamed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017a5Cv5Z1NzdYKgKnTiAF4E
On 2026-09-25 a snapshot built with --history 5 (15.3 MB) was
published, and `idx ownership sync` failed for every client:
"the response body is larger than request limit: 10485760" (ureq's
default body cap). It was republished by hand with --history 2 (8.2 MB).
- snapshot download: raise the ureq body limit to 1 GiB; the file is
still verified against the manifest's size and SHA-256. The PDF/XLSX
report downloads get a 256 MiB limit for the same reason.
- CLI test: sync a >10 MiB snapshot over HTTP (fails without the fix
with the exact production error).
- build-latest-ownership-snapshot.sh: --max-snapshot-bytes (default
10485760, the limit of released clients <= v0.2.3); an oversized
snapshot fails the build, so the published one stays untouched.
- publish-ownership-snapshot.sh passes --max-snapshot-bytes through;
publisher tests cover the failure and that the release is unchanged.
- docs: size table and guidance.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017a5Cv5Z1NzdYKgKnTiAF4E
Addresses review on #26:
- strip ?query/#fragment before the .xlsx/.pdf extension check
- skip page entries without a parseable as-of date instead of emitting
an empty publish_date
- warn on stderr when the data page fails or parses to zero entries
while announcement results still exist, instead of silently
degrading to announcement-only results
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017a5Cv5Z1NzdYKgKnTiAF4E
IDX stopped posting the monthly above-1% list as an announcement after
the 2026-05-29 report; June/July/August are only on the Data
Kepemilikan Saham page. `ownership discover` could not see them, and
IDX's Cloudflare now rejects curl_chrome142 on that page and on the
/Media/ XLSX files.
- remote: parse the page's Nuxt payload (Description/Prospectus pairs,
/ escapes), classify above-1% XLSX as supported and above-5% /
investor-type workbooks as unsupported, derive as_of_date from the
Indonesian description. Keep all above-1% entries, only the newest
of the other series. Merge with the announcement results.
- discover JSON gains `format` (xlsx|pdf) and `as_of_date`; page
entries use the as-of date as publish_date (the page shows none).
- import --url accepts .xlsx and downloads via the XLSX validator.
- curl_impersonate: run_owned_with_fallback tries several browser
profiles until the body validates (JSON / page payload / %PDF / PK);
IDX_CURL_IMPERSONATE_BIN still pins one.
- Tests: page fixture parsing, date parsing, payload validation, CLI
discover via a local page server; existing discover tests now point
IDX_OWNERSHIP_DATA_PAGE_URL at a closed port so no test hits IDX.
- Docs: ARCHITECTURE discovery section, AGENTS ownership inputs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017a5Cv5Z1NzdYKgKnTiAF4E
Since June 2026 IDX publishes the monthly "Pemegang Saham di Atas 1%"
list as an XLSX workbook on the Data Kepemilikan Saham page instead of
a PDF announcement. Add a parser for it and accept `.xlsx` in
`ownership import --file`.
- src/ownership/xlsx.rs: zip + quick-xml reader (shared strings incl.
rich text, sheet cells), exact 12-column header match so the daily
above-5% and investor-type/classification workbooks are rejected,
Excel serial dates, numeric shares, percent -> bps with float-noise
rounding. Builds KseiHoldingDraft directly and reuses the entity
normalizers, so investor types stay uppercase like the PDF path.
- Fixtures: trimmed excerpts of the real 2026-08-31 above-1% and
2026-09-23 above-5% workbooks.
- Unit + CLI tests: import, ticker, releases, above-5% rejection.
- Docs: ARCHITECTURE parser pipeline, AGENTS ownership inputs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017a5Cv5Z1NzdYKgKnTiAF4E
Newer KSEI above1 PDFs use spaces in column headers (SHARE CODE,
INVESTOR NAME, INVESTOR CLASSIFICATION) instead of the underscored
form (SHARE_CODE, INVESTOR_NAME, INVESTOR_TYPE) that the parser
schema markers expected. This caused valid holder-register PDFs to
be misclassified as LegacyAboveFivePercent (false positive on
REKENING TAMPUNGAN KSEI text in the data), blocking import.
Changes:
- Add normalize_stext_for_classification() that replaces spaces with
underscores inside text="..." attributes before schema marker
matching, so both old and new header layouts are recognized
- Add INVESTOR_CLASSIFICATION to HOLDER_REGISTER_SCHEMA_MARKERS and
HEADER_LABELS to match the renamed column
- Update ANNOUNCEMENT_WRAPPER and ABOVE_FIVE marker constants to use
underscores (matching the normalized text)
- Add test fixture and test case for the spaced-header layout
- Bump version to 0.2.3
Tested against the live June 2026 KSEI PDF (as_of 2026-05-29):
idx ownership import --url <latest-pdf-url>
-> Imported 7124 rows for 956 tickers (as of 2026-05-29)
The April 2026 IDX ownership PDF omits the zero-valued scrip column for
some holders, causing the parser to leave holdings_scripless/scrip as
empty strings. This broke normalize_ksei_row which called parse_id_number
on empty input.
Two-layer fix:
- Parser: pop_numeric_tail handles 2-column (missing scrip) rows by
defaulting scrip to "0"
- Normalizer: parse_id_number_or_zero backstop treats empty component
share fields as 0 while still requiring total_shares
Also includes AGENTS.md refresh, formatting cleanup (rustfmt), and
version bump to 0.2.2.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add a compact parser fixture excerpted from the real 2026-03-10 IDX above1 lamp1 mutool stext output and assert the first live row still parses as expected.
This keeps the post-merge ownership parser coverage grounded in the supported above1 layout instead of relying only on synthetic line fixtures.
Refactor KSEI PDF parsing for the live above-1 holder-register layout, add IDX announcement discovery plus browser-impersonated PDF fetches, and update the ownership docs to mark Batch 1 complete and scope Batch 2 around above1 hardening and unsupported legacy inputs.
MSN Finance/Charts does not serve OHLCV data for XIDX stocks (returns 404).
Following the FP principle of not exposing capabilities a provider cannot fulfil:
- Remove impl HistoryProvider for MsnProvider entirely
- Remove fetch_charts from MsnClient
- Remove parse_chart_history, resample_history, trim_history_to_period from map.rs
- Remove MsnChart, ChartSeries, RawChart from raw_types.rs
- Remove parse_history_from_str, parse_close_only_history from parse.rs
- Decouple HistoryProvider from MarketDataProvider trait bound
- Add history_provider() factory: returns None for MSN, Some(Yahoo) for Yahoo
- CLI gates History/Technical on history_provider(), fails fast for MSN
- MSN mock returns Err(Unsupported); tests verify the behaviour explicitly
- financials: fix incomeStatement serde rename (was incomeStatements, API sends singular)
- financials: flatten nested sub-objects (income/revenue/expense/cash) in parse_statement_section
- insights: rewrite RawInsight to match actual API shape ({insights:[{insightName,insightStatement,category}]})
- insights: group insight items into highlights (non-risk) and risks by category
- screener: build Quote directly in parse_screener_results, skip stocks with no price
- profile: use short_name fallback when long_name is null, hide empty fields
- history: map Finance/Charts 404 to Unsupported with clear IDX-specific message
- tests: update MSN history/technical tests to use mock provider
Add MsnProvider implementing MarketDataProvider trait with quote,
fundamentals, and history support. Includes provider-aware config
(file/env/CLI), cache namespace isolation per provider, symbol ID
mapping via embedded TSV, OHLCV resampling, and comprehensive unit
+ integration tests with MSN fixture data.
Key changes:
- MsnProvider with quote, key-ratios, and chart endpoints
- ProviderKind enum (yahoo/msn) with config hierarchy support
- Provider-namespaced cache buckets to prevent cross-provider poisoning
- Provider-aware MockProvider loading correct fixtures per provider
- Integration tests for config round-trip and cache isolation
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add analysis module: SMA, EMA, RSI(14), MACD(12,26,9), volume ratio
- Add signal interpretation: bullish/bearish/neutral with consensus voting
- Wire up 'stocks technical <SYMBOL>' CLI subcommand
- Table output with colored signals + JSON output support
- Cache/offline/stale-cache fallback (same pattern as quote/history)
- Fetch 1 year of daily data for SMA200 coverage (~250 trading days)
- Add TechnicalReport, MacdSnapshot, VolumeSnapshot structs
- Add 4 new unit tests + 3 integration tests (30 total passing)
