fix(core): parse IPOT pages with html5ever

Six rounds of the 0.4.0 review found crafted markup that the hand-written
reader read differently from HTML (cells in comments or attributes, stray
or spaced tags, a decoy date attribute or total, foreign content). Parse
the page with scraper (html5ever) instead: dates are the inputs' values,
rows are the single tbody's tr children with nine td each, totals are the
tfoot spans, all as a browser builds them. Rows that don't line up still
reject the page. scraper's CSS crates (selectors, cssparser) are MPL-2.0.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V3syn2fN2LNo5pvTSqc6eh
This commit is contained in:
hermes 2026-10-18 12:54:34 +00:00
commit 79b15d51b9
4 changed files with 517 additions and 321 deletions

View file

@ -92,7 +92,12 @@ WIB as the fallback.
takes `code`, `start`/`end` (`MM/DD/YYYY`), `fd` (`all|F|D`) and `board`
(`all|RG|TN|NG`). It needs no login and answers an HTML fragment: the top 10
buyers and top 10 sellers ranked by lots (gross, so a broker can be on both
sides), plus range totals. `api/ipot/parse.rs` parses it purely.
sides), plus range totals. `api/ipot/parse.rs` parses it purely, with
html5ever (`scraper`), so it reads the cells, date inputs and footer a
browser would build: a hand-written reader kept disagreeing with HTML on
crafted markup in the 0.4.0 review. A table whose rows don't line up
(missing cells, ranks out of order, a broker after a blank row, a second
body) rejects the page, and the dates it echoes must match the request.
- Figures of a million and more arrive abbreviated (`1.6 M`); they are scaled
in decimal and the summary's `rounded` flag says so. `F. NVal` ignores the
investor and board filters, so `foreign_net_value` is null for a summary of