- sanitize_current_ratio: and_then if/else -> filter
- screener/graph sorts: sort_by with reversed cmp -> sort_by_key(Reverse)
- insert_ksei_holdings_rows: manual Result match -> ?
Fixes the Clippy step on ubuntu-latest (rust 1.97) that the nix
toolchain (1.93.1) did not flag.
Newer KSEI above1 PDFs use spaces in column headers (SHARE CODE,
INVESTOR NAME, INVESTOR CLASSIFICATION) instead of the underscored
form (SHARE_CODE, INVESTOR_NAME, INVESTOR_TYPE) that the parser
schema markers expected. This caused valid holder-register PDFs to
be misclassified as LegacyAboveFivePercent (false positive on
REKENING TAMPUNGAN KSEI text in the data), blocking import.
Changes:
- Add normalize_stext_for_classification() that replaces spaces with
underscores inside text="..." attributes before schema marker
matching, so both old and new header layouts are recognized
- Add INVESTOR_CLASSIFICATION to HOLDER_REGISTER_SCHEMA_MARKERS and
HEADER_LABELS to match the renamed column
- Update ANNOUNCEMENT_WRAPPER and ABOVE_FIVE marker constants to use
underscores (matching the normalized text)
- Add test fixture and test case for the spaced-header layout
- Bump version to 0.2.3
Tested against the live June 2026 KSEI PDF (as_of 2026-05-29):
idx ownership import --url <latest-pdf-url>
-> Imported 7124 rows for 956 tickers (as of 2026-05-29)
The April 2026 IDX ownership PDF omits the zero-valued scrip column for
some holders, causing the parser to leave holdings_scripless/scrip as
empty strings. This broke normalize_ksei_row which called parse_id_number
on empty input.
Two-layer fix:
- Parser: pop_numeric_tail handles 2-column (missing scrip) rows by
defaulting scrip to "0"
- Normalizer: parse_id_number_or_zero backstop treats empty component
share fields as 0 while still requiring total_shares
Also includes AGENTS.md refresh, formatting cleanup (rustfmt), and
version bump to 0.2.2.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add a compact parser fixture excerpted from the real 2026-03-10 IDX above1 lamp1 mutool stext output and assert the first live row still parses as expected.
This keeps the post-merge ownership parser coverage grounded in the supported above1 layout instead of relying only on synthetic line fixtures.
Refactor KSEI PDF parsing for the live above-1 holder-register layout, add IDX announcement discovery plus browser-impersonated PDF fetches, and update the ownership docs to mark Batch 1 complete and scope Batch 2 around above1 hardening and unsupported legacy inputs.
Reject invalid values for known config keys in 'idx config set':
- general.provider: only yahoo|msn
- general.history_provider: only auto|yahoo|msn
- general.output: only table|json
- general.color: only true|false
- cache TTL fields: non-negative integers only
- Unknown keys: rejected with clear error listing valid keys
Issue #9: Previously, corrupted JSON in cache files would cause commands
to fail with a ParseError. Now:
- IO errors and parse errors are caught in read_entry()
- A warning is printed to stderr with the data_type/symbol
- The corrupted file is deleted
- Ok(None) is returned (treated as cache miss)
This applies to both get() and get_stale() since they share read_entry().
Added tests:
- corrupted_cache_entry_returns_none_and_deletes_file
- corrupted_cache_entry_get_stale_returns_none_and_deletes_file
Both verify the corrupted file is cleaned up after the graceful failure.
- Change resolve_symbol return type from String to Result<String, IdxError>
- Add early validation for empty/whitespace-only ticker input
- Return Err(IdxError::InvalidInput) for empty tickers
- Add InvalidInput variant to IdxError enum
- Update all 15 callers in cli/stocks.rs to handle Result with ? propagation
- Update tests: empty/whitespace tickers now return error
Fixes#7
MSN Finance/Charts does not serve OHLCV data for XIDX stocks (returns 404).
Following the FP principle of not exposing capabilities a provider cannot fulfil:
- Remove impl HistoryProvider for MsnProvider entirely
- Remove fetch_charts from MsnClient
- Remove parse_chart_history, resample_history, trim_history_to_period from map.rs
- Remove MsnChart, ChartSeries, RawChart from raw_types.rs
- Remove parse_history_from_str, parse_close_only_history from parse.rs
- Decouple HistoryProvider from MarketDataProvider trait bound
- Add history_provider() factory: returns None for MSN, Some(Yahoo) for Yahoo
- CLI gates History/Technical on history_provider(), fails fast for MSN
- MSN mock returns Err(Unsupported); tests verify the behaviour explicitly
- financials: fix incomeStatement serde rename (was incomeStatements, API sends singular)
- financials: flatten nested sub-objects (income/revenue/expense/cash) in parse_statement_section
- insights: rewrite RawInsight to match actual API shape ({insights:[{insightName,insightStatement,category}]})
- insights: group insight items into highlights (non-risk) and risks by category
- screener: build Quote directly in parse_screener_results, skip stocks with no price
- profile: use short_name fallback when long_name is null, hide empty fields
- history: map Finance/Charts 404 to Unsupported with clear IDX-specific message
- tests: update MSN history/technical tests to use mock provider
Add MsnProvider implementing MarketDataProvider trait with quote,
fundamentals, and history support. Includes provider-aware config
(file/env/CLI), cache namespace isolation per provider, symbol ID
mapping via embedded TSV, OHLCV resampling, and comprehensive unit
+ integration tests with MSN fixture data.
Key changes:
- MsnProvider with quote, key-ratios, and chart endpoints
- ProviderKind enum (yahoo/msn) with config hierarchy support
- Provider-namespaced cache buckets to prevent cross-provider poisoning
- Provider-aware MockProvider loading correct fixtures per provider
- Integration tests for config round-trip and cache isolation
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Use curl_chrome131 (and fallback chain) instead of 'curl-impersonate --impersonate'
The curl-impersonate-chrome package ships per-version binaries, not a generic binary
- Add Unknown(serde_json::Value) catch-all variant to QuoteSummaryValue
Yahoo returns {} for empty fields and null/strings that broke deserialization
- Keep cookie jar flow: fc.yahoo.com (404 but writes A3 cookie) + getcrumb + quoteSummary
- Pass cookie header from jar to quoteSummary request via ureq
Live result: stocks fundamental/growth/valuation/risk/compare now working
BBCA: ROE 21.14% excellent, Net Margin 53.28% excellent, Growth mixed
Yahoo Finance blocks standard ureq/rustls via TLS fingerprinting (JA3/JA4).
Reverse-engineered from yfinance source: they use curl_cffi with Chrome impersonation.
Fix:
- Fetch cookie via curl-impersonate from fc.yahoo.com
- Fetch crumb via curl-impersonate from query1.finance.yahoo.com/v1/test/getcrumb
(correct URL: getcrumb, not getCrumb or csrfToken)
- Parse Netscape cookie jar format, send cookies as header to quoteSummary
- 401 retry: clear crumb + cookie jar, re-auth on next attempt
- Crumb validation: reject HTML, empty, rate-limit responses
- Add curl-impersonate-chrome to flake.nix devShell
- Add analysis module: SMA, EMA, RSI(14), MACD(12,26,9), volume ratio
- Add signal interpretation: bullish/bearish/neutral with consensus voting
- Wire up 'stocks technical <SYMBOL>' CLI subcommand
- Table output with colored signals + JSON output support
- Cache/offline/stale-cache fallback (same pattern as quote/history)
- Fetch 1 year of daily data for SMA200 coverage (~250 trading days)
- Add TechnicalReport, MacdSnapshot, VolumeSnapshot structs
- Add 4 new unit tests + 3 integration tests (30 total passing)
