Addresses review on #31:
- the snapshot download limit is the manifest's size_bytes (+1, since
ureq fails once a body reaches its limit), capped at 1 GiB; a body
larger than advertised fails during download. New CLI test; the
>10 MiB test also caught the off-by-one.
- a refused (oversized) standalone build removes its sqlite and
manifest from --output-dir.
- docs: downloads are buffered, not streamed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017a5Cv5Z1NzdYKgKnTiAF4E
On 2026-09-25 a snapshot built with --history 5 (15.3 MB) was
published, and `idx ownership sync` failed for every client:
"the response body is larger than request limit: 10485760" (ureq's
default body cap). It was republished by hand with --history 2 (8.2 MB).
- snapshot download: raise the ureq body limit to 1 GiB; the file is
still verified against the manifest's size and SHA-256. The PDF/XLSX
report downloads get a 256 MiB limit for the same reason.
- CLI test: sync a >10 MiB snapshot over HTTP (fails without the fix
with the exact production error).
- build-latest-ownership-snapshot.sh: --max-snapshot-bytes (default
10485760, the limit of released clients <= v0.2.3); an oversized
snapshot fails the build, so the published one stays untouched.
- publish-ownership-snapshot.sh passes --max-snapshot-bytes through;
publisher tests cover the failure and that the release is unchanged.
- docs: size table and guidance.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017a5Cv5Z1NzdYKgKnTiAF4E
Addresses review on #26:
- strip ?query/#fragment before the .xlsx/.pdf extension check
- skip page entries without a parseable as-of date instead of emitting
an empty publish_date
- warn on stderr when the data page fails or parses to zero entries
while announcement results still exist, instead of silently
degrading to announcement-only results
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017a5Cv5Z1NzdYKgKnTiAF4E
IDX stopped posting the monthly above-1% list as an announcement after
the 2026-05-29 report; June/July/August are only on the Data
Kepemilikan Saham page. `ownership discover` could not see them, and
IDX's Cloudflare now rejects curl_chrome142 on that page and on the
/Media/ XLSX files.
- remote: parse the page's Nuxt payload (Description/Prospectus pairs,
/ escapes), classify above-1% XLSX as supported and above-5% /
investor-type workbooks as unsupported, derive as_of_date from the
Indonesian description. Keep all above-1% entries, only the newest
of the other series. Merge with the announcement results.
- discover JSON gains `format` (xlsx|pdf) and `as_of_date`; page
entries use the as-of date as publish_date (the page shows none).
- import --url accepts .xlsx and downloads via the XLSX validator.
- curl_impersonate: run_owned_with_fallback tries several browser
profiles until the body validates (JSON / page payload / %PDF / PK);
IDX_CURL_IMPERSONATE_BIN still pins one.
- Tests: page fixture parsing, date parsing, payload validation, CLI
discover via a local page server; existing discover tests now point
IDX_OWNERSHIP_DATA_PAGE_URL at a closed port so no test hits IDX.
- Docs: ARCHITECTURE discovery section, AGENTS ownership inputs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017a5Cv5Z1NzdYKgKnTiAF4E
Addresses review on #25: sheet XML omits blank rows, so error messages
now use each <row r=...> number instead of the position; the excerpt
test pins the exact row count so dropped rows fail loudly.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017a5Cv5Z1NzdYKgKnTiAF4E
The February and March 2026 above-1% workbooks name the investor
column INVESTOR_TYPE (codes such as CP, ID); April onward uses
INVESTOR_CLASSIFICATION (words). Treat both as the same field; every
other column stays strict.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017a5Cv5Z1NzdYKgKnTiAF4E
Since June 2026 IDX publishes the monthly "Pemegang Saham di Atas 1%"
list as an XLSX workbook on the Data Kepemilikan Saham page instead of
a PDF announcement. Add a parser for it and accept `.xlsx` in
`ownership import --file`.
- src/ownership/xlsx.rs: zip + quick-xml reader (shared strings incl.
rich text, sheet cells), exact 12-column header match so the daily
above-5% and investor-type/classification workbooks are rejected,
Excel serial dates, numeric shares, percent -> bps with float-noise
rounding. Builds KseiHoldingDraft directly and reuses the entity
normalizers, so investor types stay uppercase like the PDF path.
- Fixtures: trimmed excerpts of the real 2026-08-31 above-1% and
2026-09-23 above-5% workbooks.
- Unit + CLI tests: import, ticker, releases, above-5% rejection.
- Docs: ARCHITECTURE parser pipeline, AGENTS ownership inputs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017a5Cv5Z1NzdYKgKnTiAF4E
- sanitize_current_ratio: and_then if/else -> filter
- screener/graph sorts: sort_by with reversed cmp -> sort_by_key(Reverse)
- insert_ksei_holdings_rows: manual Result match -> ?
Fixes the Clippy step on ubuntu-latest (rust 1.97) that the nix
toolchain (1.93.1) did not flag.
Newer KSEI above1 PDFs use spaces in column headers (SHARE CODE,
INVESTOR NAME, INVESTOR CLASSIFICATION) instead of the underscored
form (SHARE_CODE, INVESTOR_NAME, INVESTOR_TYPE) that the parser
schema markers expected. This caused valid holder-register PDFs to
be misclassified as LegacyAboveFivePercent (false positive on
REKENING TAMPUNGAN KSEI text in the data), blocking import.
Changes:
- Add normalize_stext_for_classification() that replaces spaces with
underscores inside text="..." attributes before schema marker
matching, so both old and new header layouts are recognized
- Add INVESTOR_CLASSIFICATION to HOLDER_REGISTER_SCHEMA_MARKERS and
HEADER_LABELS to match the renamed column
- Update ANNOUNCEMENT_WRAPPER and ABOVE_FIVE marker constants to use
underscores (matching the normalized text)
- Add test fixture and test case for the spaced-header layout
- Bump version to 0.2.3
Tested against the live June 2026 KSEI PDF (as_of 2026-05-29):
idx ownership import --url <latest-pdf-url>
-> Imported 7124 rows for 956 tickers (as of 2026-05-29)
The April 2026 IDX ownership PDF omits the zero-valued scrip column for
some holders, causing the parser to leave holdings_scripless/scrip as
empty strings. This broke normalize_ksei_row which called parse_id_number
on empty input.
Two-layer fix:
- Parser: pop_numeric_tail handles 2-column (missing scrip) rows by
defaulting scrip to "0"
- Normalizer: parse_id_number_or_zero backstop treats empty component
share fields as 0 while still requiring total_shares
Also includes AGENTS.md refresh, formatting cleanup (rustfmt), and
version bump to 0.2.2.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add a compact parser fixture excerpted from the real 2026-03-10 IDX above1 lamp1 mutool stext output and assert the first live row still parses as expected.
This keeps the post-merge ownership parser coverage grounded in the supported above1 layout instead of relying only on synthetic line fixtures.
Refactor KSEI PDF parsing for the live above-1 holder-register layout, add IDX announcement discovery plus browser-impersonated PDF fetches, and update the ownership docs to mark Batch 1 complete and scope Batch 2 around above1 hardening and unsupported legacy inputs.
