- --offline now warns when it serves a cache entry past its TTL; before,
arbitrarily old data was returned with no signal at all
- version, cache info and cache clear honour -o json
- ownership resolve map/merge honour -o json
- stocks quote/compare require at least one symbol instead of printing []
- cache schema-mismatch notice goes through runtime::info so --quiet
silences it, instead of a raw debug eprintln
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Q2PufCUdvVZoVW1Dy53js
Addresses review on #31:
- the snapshot download limit is the manifest's size_bytes (+1, since
ureq fails once a body reaches its limit), capped at 1 GiB; a body
larger than advertised fails during download. New CLI test; the
>10 MiB test also caught the off-by-one.
- a refused (oversized) standalone build removes its sqlite and
manifest from --output-dir.
- docs: downloads are buffered, not streamed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017a5Cv5Z1NzdYKgKnTiAF4E
On 2026-09-25 a snapshot built with --history 5 (15.3 MB) was
published, and `idx ownership sync` failed for every client:
"the response body is larger than request limit: 10485760" (ureq's
default body cap). It was republished by hand with --history 2 (8.2 MB).
- snapshot download: raise the ureq body limit to 1 GiB; the file is
still verified against the manifest's size and SHA-256. The PDF/XLSX
report downloads get a 256 MiB limit for the same reason.
- CLI test: sync a >10 MiB snapshot over HTTP (fails without the fix
with the exact production error).
- build-latest-ownership-snapshot.sh: --max-snapshot-bytes (default
10485760, the limit of released clients <= v0.2.3); an oversized
snapshot fails the build, so the published one stays untouched.
- publish-ownership-snapshot.sh passes --max-snapshot-bytes through;
publisher tests cover the failure and that the release is unchanged.
- docs: size table and guidance.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017a5Cv5Z1NzdYKgKnTiAF4E
Addresses review on #26:
- strip ?query/#fragment before the .xlsx/.pdf extension check
- skip page entries without a parseable as-of date instead of emitting
an empty publish_date
- warn on stderr when the data page fails or parses to zero entries
while announcement results still exist, instead of silently
degrading to announcement-only results
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017a5Cv5Z1NzdYKgKnTiAF4E
IDX stopped posting the monthly above-1% list as an announcement after
the 2026-05-29 report; June/July/August are only on the Data
Kepemilikan Saham page. `ownership discover` could not see them, and
IDX's Cloudflare now rejects curl_chrome142 on that page and on the
/Media/ XLSX files.
- remote: parse the page's Nuxt payload (Description/Prospectus pairs,
/ escapes), classify above-1% XLSX as supported and above-5% /
investor-type workbooks as unsupported, derive as_of_date from the
Indonesian description. Keep all above-1% entries, only the newest
of the other series. Merge with the announcement results.
- discover JSON gains `format` (xlsx|pdf) and `as_of_date`; page
entries use the as-of date as publish_date (the page shows none).
- import --url accepts .xlsx and downloads via the XLSX validator.
- curl_impersonate: run_owned_with_fallback tries several browser
profiles until the body validates (JSON / page payload / %PDF / PK);
IDX_CURL_IMPERSONATE_BIN still pins one.
- Tests: page fixture parsing, date parsing, payload validation, CLI
discover via a local page server; existing discover tests now point
IDX_OWNERSHIP_DATA_PAGE_URL at a closed port so no test hits IDX.
- Docs: ARCHITECTURE discovery section, AGENTS ownership inputs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017a5Cv5Z1NzdYKgKnTiAF4E
Addresses review on #25: sheet XML omits blank rows, so error messages
now use each <row r=...> number instead of the position; the excerpt
test pins the exact row count so dropped rows fail loudly.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017a5Cv5Z1NzdYKgKnTiAF4E
The February and March 2026 above-1% workbooks name the investor
column INVESTOR_TYPE (codes such as CP, ID); April onward uses
INVESTOR_CLASSIFICATION (words). Treat both as the same field; every
other column stays strict.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017a5Cv5Z1NzdYKgKnTiAF4E
Since June 2026 IDX publishes the monthly "Pemegang Saham di Atas 1%"
list as an XLSX workbook on the Data Kepemilikan Saham page instead of
a PDF announcement. Add a parser for it and accept `.xlsx` in
`ownership import --file`.
- src/ownership/xlsx.rs: zip + quick-xml reader (shared strings incl.
rich text, sheet cells), exact 12-column header match so the daily
above-5% and investor-type/classification workbooks are rejected,
Excel serial dates, numeric shares, percent -> bps with float-noise
rounding. Builds KseiHoldingDraft directly and reuses the entity
normalizers, so investor types stay uppercase like the PDF path.
- Fixtures: trimmed excerpts of the real 2026-08-31 above-1% and
2026-09-23 above-5% workbooks.
- Unit + CLI tests: import, ticker, releases, above-5% rejection.
- Docs: ARCHITECTURE parser pipeline, AGENTS ownership inputs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017a5Cv5Z1NzdYKgKnTiAF4E
- sanitize_current_ratio: and_then if/else -> filter
- screener/graph sorts: sort_by with reversed cmp -> sort_by_key(Reverse)
- insert_ksei_holdings_rows: manual Result match -> ?
Fixes the Clippy step on ubuntu-latest (rust 1.97) that the nix
toolchain (1.93.1) did not flag.
Newer KSEI above1 PDFs use spaces in column headers (SHARE CODE,
INVESTOR NAME, INVESTOR CLASSIFICATION) instead of the underscored
form (SHARE_CODE, INVESTOR_NAME, INVESTOR_TYPE) that the parser
schema markers expected. This caused valid holder-register PDFs to
be misclassified as LegacyAboveFivePercent (false positive on
REKENING TAMPUNGAN KSEI text in the data), blocking import.
Changes:
- Add normalize_stext_for_classification() that replaces spaces with
underscores inside text="..." attributes before schema marker
matching, so both old and new header layouts are recognized
- Add INVESTOR_CLASSIFICATION to HOLDER_REGISTER_SCHEMA_MARKERS and
HEADER_LABELS to match the renamed column
- Update ANNOUNCEMENT_WRAPPER and ABOVE_FIVE marker constants to use
underscores (matching the normalized text)
- Add test fixture and test case for the spaced-header layout
- Bump version to 0.2.3
Tested against the live June 2026 KSEI PDF (as_of 2026-05-29):
idx ownership import --url <latest-pdf-url>
-> Imported 7124 rows for 956 tickers (as of 2026-05-29)
The April 2026 IDX ownership PDF omits the zero-valued scrip column for
some holders, causing the parser to leave holdings_scripless/scrip as
empty strings. This broke normalize_ksei_row which called parse_id_number
on empty input.
Two-layer fix:
- Parser: pop_numeric_tail handles 2-column (missing scrip) rows by
defaulting scrip to "0"
- Normalizer: parse_id_number_or_zero backstop treats empty component
share fields as 0 while still requiring total_shares
Also includes AGENTS.md refresh, formatting cleanup (rustfmt), and
version bump to 0.2.2.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add a compact parser fixture excerpted from the real 2026-03-10 IDX above1 lamp1 mutool stext output and assert the first live row still parses as expected.
This keeps the post-merge ownership parser coverage grounded in the supported above1 layout instead of relying only on synthetic line fixtures.
Refactor KSEI PDF parsing for the live above-1 holder-register layout, add IDX announcement discovery plus browser-impersonated PDF fetches, and update the ownership docs to mark Batch 1 complete and scope Batch 2 around above1 hardening and unsupported legacy inputs.
Reject invalid values for known config keys in 'idx config set':
- general.provider: only yahoo|msn
- general.history_provider: only auto|yahoo|msn
- general.output: only table|json
- general.color: only true|false
- cache TTL fields: non-negative integers only
- Unknown keys: rejected with clear error listing valid keys
Issue #9: Previously, corrupted JSON in cache files would cause commands
to fail with a ParseError. Now:
- IO errors and parse errors are caught in read_entry()
- A warning is printed to stderr with the data_type/symbol
- The corrupted file is deleted
- Ok(None) is returned (treated as cache miss)
This applies to both get() and get_stale() since they share read_entry().
Added tests:
- corrupted_cache_entry_returns_none_and_deletes_file
- corrupted_cache_entry_get_stale_returns_none_and_deletes_file
Both verify the corrupted file is cleaned up after the graceful failure.
- Change resolve_symbol return type from String to Result<String, IdxError>
- Add early validation for empty/whitespace-only ticker input
- Return Err(IdxError::InvalidInput) for empty tickers
- Add InvalidInput variant to IdxError enum
- Update all 15 callers in cli/stocks.rs to handle Result with ? propagation
- Update tests: empty/whitespace tickers now return error
Fixes#7
MSN Finance/Charts does not serve OHLCV data for XIDX stocks (returns 404).
Following the FP principle of not exposing capabilities a provider cannot fulfil:
- Remove impl HistoryProvider for MsnProvider entirely
- Remove fetch_charts from MsnClient
- Remove parse_chart_history, resample_history, trim_history_to_period from map.rs
- Remove MsnChart, ChartSeries, RawChart from raw_types.rs
- Remove parse_history_from_str, parse_close_only_history from parse.rs
- Decouple HistoryProvider from MarketDataProvider trait bound
- Add history_provider() factory: returns None for MSN, Some(Yahoo) for Yahoo
- CLI gates History/Technical on history_provider(), fails fast for MSN
- MSN mock returns Err(Unsupported); tests verify the behaviour explicitly
- financials: fix incomeStatement serde rename (was incomeStatements, API sends singular)
- financials: flatten nested sub-objects (income/revenue/expense/cash) in parse_statement_section
- insights: rewrite RawInsight to match actual API shape ({insights:[{insightName,insightStatement,category}]})
- insights: group insight items into highlights (non-risk) and risks by category
- screener: build Quote directly in parse_screener_results, skip stocks with no price
- profile: use short_name fallback when long_name is null, hide empty fields
- history: map Finance/Charts 404 to Unsupported with clear IDX-specific message
- tests: update MSN history/technical tests to use mock provider
Add MsnProvider implementing MarketDataProvider trait with quote,
fundamentals, and history support. Includes provider-aware config
(file/env/CLI), cache namespace isolation per provider, symbol ID
mapping via embedded TSV, OHLCV resampling, and comprehensive unit
+ integration tests with MSN fixture data.
Key changes:
- MsnProvider with quote, key-ratios, and chart endpoints
- ProviderKind enum (yahoo/msn) with config hierarchy support
- Provider-namespaced cache buckets to prevent cross-provider poisoning
- Provider-aware MockProvider loading correct fixtures per provider
- Integration tests for config round-trip and cache isolation
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
