Commit graph

323 commits

Author SHA1 Message Date
hermes
7d55be80c5 fix(core): bind IPOT class tokens to the elements that carry them
Classes are allowed per element type, not page-wide, and the table and
each footer element on the way to a total must carry exactly IPOT's
classes, so Bootstrap's visible-xs (hidden at desktop widths) on a total
or the table rejects the page (Pullfrog review).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V3syn2fN2LNo5pvTSqc6eh
2026-10-18 14:35:02 +00:00
hermes
3c5f9bb8a4 fix(core): an IPOT page without a readable date range is rejected
Ancestor classes compare as sets (repeats removed too), and a page whose
echoed range can't be read is a parse error rather than a summary with
no dates, so the parser returns the exact range or nothing (Codex
re-check, round 13). The broker layer already rejected missing dates.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V3syn2fN2LNo5pvTSqc6eh
2026-10-18 14:25:45 +00:00
hermes
bf77073877 fix(core): close the round-12 audit of the IPOT parser
From Codex's exhaustive audit of every allowed element, attribute and
returned value:
- class tokens split on ASCII whitespace as HTML does, so a no-break
  space stays in the token (and fails the class allowlist);
- a date input's line breaks are dropped before parsing, as a text input
  drops them;
- ancestor placement compares class token sets, not exact strings;
- the echoed range needs both date inputs, start then end in document
  order, each with IPOT's id and name; otherwise no dates;
- a side with a blank broker code may show no figures.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V3syn2fN2LNo5pvTSqc6eh
2026-10-18 14:15:51 +00:00
hermes
d15fc4833a fix(core): header cells take only align; broker kind from the class token
A header cell spanning columns would move the labels over the figures,
so header cells may carry only align. The broker kind comes from the
span's single class token as the browser splits it, so spacing in the
attribute no longer drops it (Codex re-check, round 11).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V3syn2fN2LNo5pvTSqc6eh
2026-10-18 13:58:59 +00:00
hermes
828d16c92e fix(core): match IPOT's unknown-code reply exactly; check column labels
An unknown code is only the reply that is exactly "Invalid stock code!",
not any page with that text (e.g. inside a script). The header row must
be IPOT's nine labels in order, so a page labelling the columns otherwise
is rejected rather than read by position (Codex re-check, round 10).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V3syn2fN2LNo5pvTSqc6eh
2026-10-18 13:49:15 +00:00
hermes
ee4fe4b852 fix(core): read IPOT's four footer spans exactly; bound the page size
The footer div must hold exactly IPOT's four text-only spans, in order and
each starting with its label, with nothing else shown between them, so a
total split from or across spans no longer reads as a smaller figure.
Pages over 64 KiB or 2,000 tags (IPOT's is about 8 KB and under 500) are
rejected before parsing, which bounds tree building on deep nesting
(Codex re-check, round 9).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V3syn2fN2LNo5pvTSqc6eh
2026-10-18 13:40:01 +00:00
hermes
63c4d9342f fix(core): accept only IPOT's own markup, in its own places
Replace the list of hiding mechanisms with what the page is: every element
must be one IPOT uses, with only its attributes and class tokens, and with
input type, the footer div's style, colspan and button type pinned; the
table and date inputs must sit at IPOT's exact div paths; the table is
thead, tbody, tfoot; totals come from tfoot > tr > th > div > span only;
and a cell's span holds all the cell shows. A popover, an escaped inline
style, a hidden input, a total in details or a code beside its span now
reject the page, and the footer is walked directly (linear). Tests now
state the property: a page is rejected or read as shown, never as other
data (Codex re-check, round 8).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V3syn2fN2LNo5pvTSqc6eh
2026-10-18 13:30:08 +00:00
hermes
723da4a5a6 fix(core): the IPOT table and date inputs must sit in plain divs
Inside details, dialog, noscript, object, select and the like they may
not show, so the table there rejects the page and such a date input is
no echoed date.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V3syn2fN2LNo5pvTSqc6eh
2026-10-18 13:18:30 +00:00
hermes
6cfceafe04 fix(core): read IPOT cells and footer only in the shape IPOT writes
On the html5ever DOM, a body cell must be a td (align only) holding text
or one span with a single text-* class holding text; footer totals must be
text-only spans, each label once; each date input must be unique; and a
template or an element hidden by markup (hidden, inline display:none or
visibility:hidden) anywhere rejects the page. So script or hidden text
inside a figure, a hidden total, a decoy span before a broker code, or a
duplicate date input can no longer be read as data, and footer text is
read once per span (Codex re-check, round 7).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V3syn2fN2LNo5pvTSqc6eh
2026-10-18 13:09:13 +00:00
hermes
79b15d51b9 fix(core): parse IPOT pages with html5ever
Six rounds of the 0.4.0 review found crafted markup that the hand-written
reader read differently from HTML (cells in comments or attributes, stray
or spaced tags, a decoy date attribute or total, foreign content). Parse
the page with scraper (html5ever) instead: dates are the inputs' values,
rows are the single tbody's tr children with nine td each, totals are the
tfoot spans, all as a browser builds them. Rows that don't line up still
reject the page. scraper's CSS crates (selectors, cssparser) are MPL-2.0.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V3syn2fN2LNo5pvTSqc6eh
2026-10-18 12:54:34 +00:00
hermes
1c6ec6eb28 fix(core): read the IPOT page as a browser would around its table
Comments are removed before parsing, and an unclosed or abrupt one
(<!-->, <!--->, --!>) or other <! / <? markup rejects the page; the page
is read only up to its first raw-text element (it ends with one empty
script). The echoed dates are read before the table and the totals after
it, cell text may not hold '>', and the row loop no longer rescans
trailing whitespace per row (Codex re-check, round 5).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V3syn2fN2LNo5pvTSqc6eh
2026-10-18 12:34:16 +00:00
hermes
cecfbab235 fix(core): accept only the exact markup IPOT serves in its table body
Replace the tag scanner with a strict grammar: whole <tr> rows of nine
<td> cells, each holding text or one <span>, attributes as name="value"
without quotes or angle brackets in the value, and no character references
besides &nbsp;. Markup in that form reads the same in any HTML parser, so
a quoted '>', '</tr >', '<td-x' or '&#50;' can no longer make the parser
see other cells than the page shows; such pages are rejected (Codex
re-check, round 4).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V3syn2fN2LNo5pvTSqc6eh
2026-10-18 12:24:44 +00:00
hermes
186d9557d6 fix(core): the IPOT table body holds plain row markup only
One tbody start and end in the page, in any case, and inside it only tr,
td and span tags with no '<' before a tag closes. A stray </TbOdY> no
longer cuts the table short, and cells inside a comment or an attribute
are no longer read as brokers (Codex re-check).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V3syn2fN2LNo5pvTSqc6eh
2026-10-18 12:15:40 +00:00
hermes
30afcc109d fix(core): read the IPOT table body as whole rows only
The body (one tbody, tags matched case-insensitively) must be whole rows:
each one <tr> holding nine opened and nine closed cells, ranked in order,
with nothing else between rows. A row with missing close tags no longer
merges into its neighbour, and an uppercase row is no longer skipped
(Codex re-check).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V3syn2fN2LNo5pvTSqc6eh
2026-10-18 12:05:13 +00:00
hermes
106590d151 fix(core): a row with no parsed cells rejects the page too
Every piece of the table body that holds a row must parse as nine cells;
ranks count the rows read, not the pieces. A rank-1 row without its cell
ends was skipped and ranks 2-10 still matched (Codex re-review).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V3syn2fN2LNo5pvTSqc6eh
2026-10-18 11:56:25 +00:00
hermes
2ae025be5d test(agents): broker, bandar and foreign flow cases
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V3syn2fN2LNo5pvTSqc6eh
2026-10-18 11:48:00 +00:00
hermes
6f211a398f fix(mcp): analyze_bandar lists the tiers top accepts
The schema said uint8 0-255; only 1, 3, 5 and 10 are valid.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V3syn2fN2LNo5pvTSqc6eh
2026-10-18 11:48:00 +00:00
hermes
d82f7c6be9 fix(core): reject broker and trading pages that would misreport a session
From the 0.4.0 release review (Codex):
- IPOT pages with a short row, a rank out of order or a broker after a
  blank row are parse errors; skipping a row shifted later brokers up a
  rank and could flip a bandar reading (-2425 bps to -66 bps).
- A tbody end before its start is a parse error, not a panic.
- An IPOT page that echoes other dates than requested is rejected before
  it is cached, so one session's figures never pass as another's.
- An IDX trading summary with rows for other days only is an error, not
  an empty session cached for 30 days.
- Empty sessions count toward a flow's fetched_at and stale.
- The bandar imbalance uses checked i128 arithmetic: past i128 there is
  no reading instead of a wrapped one.
- The bandar table prints "No trades." only when no broker traded.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V3syn2fN2LNo5pvTSqc6eh
2026-10-18 11:48:00 +00:00
brutaljokerz
d6ff056b72
Merge pull request #59 from 0xrsydn/release/v0.3.3 v0.3.3
release: 0.3.3
2026-10-18 17:56:00 +07:00
hermes
8b507d78e2 feat: foreign flow from IPOT by default (--source ipot|idx)
`idx stocks flow` now reads IPOT by default, which needs nothing installed:
per session, the all-investor broker summary gives the value traded and
the foreign one gives foreign buying (its footer), the foreign net and
selling (buying less the net), across all boards, through the broker
summary cache. IPOT's foreign net matched IDX's estimate within 0.3 B on
BBCA 2026-10-12 to 16.

`--source idx` (and `source` on get_foreign_flow) keeps IDX's trading
summary: exact share counts, close and the non-regular share, behind
curl-impersonate and Cloudflare.

- ForeignFlowDay gains foreign_buy_value, foreign_sell_value and
  foreign_share_bps for both sources (estimated at the session average
  for IDX); ForeignFlow gains source, the value totals and rounded.
- The table lays out each source's figures; the live smoke runs both.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V3syn2fN2LNo5pvTSqc6eh
2026-10-18 06:40:22 +00:00
hermes
b137dbf540 fix(core): version the broker summary cache bucket
The cache stores normalized summaries, so one written before buy_value and
sell_value existed read back without them: bandar gave null readings (the
table said "No trades.") for a week, and an investor-filtered entry lacked
its totals. Summaries now cache under ipot-broker-v2, so an old-shape entry
is never served; offline it is a cache miss, online it is fetched again. A
CLI test seeds an old entry and checks both, in JSON and table output.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V3syn2fN2LNo5pvTSqc6eh
2026-10-18 06:40:15 +00:00
hermes
5f64319157 feat: bandar and broker totals for one investor side
IPOT's investor filter splits each broker row by client domicile, but its
footer then gives only that side's buying (BBCA 2026-10-16: foreign 1.7 T,
domestic 4.9 T of 6.6 T). The foreign net gives the selling on all boards:
foreign 1.7 + 2.3 = 4.0 T (IDX: about 3.95 T regular), domestic
4.9 - 2.3 = 2.6 T.

- BrokerSummary gains buy_value and sell_value: the total for all
  investors, the footer and the derived selling for one side (selling is
  null with a board filter, where the net covers other boards).
- Bandar weighs the top N buyers' share of the value bought against the
  top N sellers' share of the value sold, so `--investor foreign|domestic`
  reads that side alone; for all investors the readings are unchanged.
  One side on one board is refused, as there is no selling to weigh.
- Tables print what the side bought and sold; fixtures for foreign and
  domestic on all boards were recorded from IPOT.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V3syn2fN2LNo5pvTSqc6eh
2026-10-18 06:08:02 +00:00
hermes
83783684cb fix(core): non-regular share is of all trading
IDX's Value is regular-board turnover, so NonRegularValue / Value gave
non-regular over regular (BBRI 2026-10-14: 50.91%) instead of the share of
all trading (33.74%). non_regular_bps now divides by regular plus
non-regular value, with a regression test on that fixture. The docs now say
volume, value and the foreign figures are the regular board: IPOT's
regular total matches Value, and in a week of every stock no foreign figure
exceeded regular volume, even with non-regular volume above twice it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V3syn2fN2LNo5pvTSqc6eh
2026-10-18 05:41:55 +00:00
hermes
29f143fa1b feat: foreign flow (idx stocks flow, get_foreign_flow)
`idx stocks flow SYMBOL` shows what foreign investors bought and sold of a
stock, session by session, from IDX's own daily trading summary: foreign
buy, sell and net volume, net value at the session's average price, and
the share of value traded outside the regular board, with range totals.
Latest 10 sessions by default (--days, at most 60) or --from/--to. The MCP
server exposes it as get_foreign_flow.

- api/idx: pure parser for GetStockSummary (rows dated another session
  are dropped) and a provider over curl-impersonate's profile fallback
  with a 30 s limit; Cloudflare passes only some fingerprints.
- flow: each session is cached whole by date, so other stocks' flows for
  the same sessions need no request. Weekends are never requested.
- Idx::idx_code is shared by broker and flow for the .JK-only check.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V3syn2fN2LNo5pvTSqc6eh
2026-10-18 05:39:12 +00:00
hermes
4c2f0acff3 fix(core): symmetric bandar thresholds; bandar needs all investors
- Basis points truncate toward zero instead of flooring, so a reading of
  -4.9952% stays neutral like its mirror image and -14.9952% stays
  distribution; tests cover both signs at both thresholds through
  bandar_tier. Goldens move by the rounding (BBCA top 5: -2426 -> -2425).
- With one investor side selected the summary has no total to weigh
  brokers against, so bandar and analyze_bandar reject it (INVALIDINPUT)
  instead of reading every tier as missing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V3syn2fN2LNo5pvTSqc6eh
2026-10-18 05:39:05 +00:00
hermes
970da26a0e feat: bandarmology (idx stocks bandar, analyze_bandar)
`idx stocks bandar SYMBOL...` reads the broker summary: for the top 1, 3,
5 and 10 brokers per side, buy value minus sell value in basis points of
value traded, with a signal from big accumulation to big distribution on
the headline tier (--top, default 5). Several symbols give a screen ranked
by that figure (at most 100, one request each), which pipes from
`stocks screen`. The MCP server exposes it as analyze_bandar (20 symbols).

- analysis/bandar.rs is pure and versioned (`top_concentration_v1`):
  integer basis points, no figure missing counted as zero, buy/sell VWAP
  from exact averages, and the brokers on both sides listed because their
  gross figures overstate their direction.
- Thresholds (±5%/±15%) follow fzn0x/idx (MIT); docs call them heuristics.
- Invalid input fails a screen at once; other errors fail one symbol and
  are warned about on stderr.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V3syn2fN2LNo5pvTSqc6eh
2026-10-18 05:34:37 +00:00
hermes
ae7a1e2b5c fix(cli): no totals line for one investor side
A foreign- or domestic-only summary has no totals now, so the table says
why instead of printing "Total - IDR". The CLI test checks the nulls and
the table.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V3syn2fN2LNo5pvTSqc6eh
2026-10-18 05:34:29 +00:00
hermes
860d4daf77 feat(cli,mcp): idx stocks broker and get_broker_summary
`idx stocks broker SYMBOL` shows the top 10 buyers and sellers of a stock
by lots for the latest session with trades, one `--date`, or a
`--from`/`--to` range, filtered by `--investor` and `--board`. The MCP
server exposes the same as get_broker_summary.

- Range and filter parsing lives in idx-core (BrokerRange::from_args,
  FromStr for the filters), so both frontends accept the same input.
- Without dates, the latest session is found by walking back over
  weekdays; offline, uncached days are skipped.
- Table output lists foreign and state-owned brokers, and says when the
  source rounded figures.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V3syn2fN2LNo5pvTSqc6eh
2026-10-18 05:33:12 +00:00
hermes
7c8151a3a6 fix(core): no broker totals for one investor side
With fd=F or fd=D, IPOT's footer covers only that side's buying: BBCA's
foreign/regular summary for 2026-10-16 gives T. Val 1.6 T while its top
seller alone sold 2.0 T. Passing it through as the range total gave
consumers a denominator smaller than one listed broker, so total_value,
total_lots and average_price are now null for an investor-filtered summary.
The page-to-summary step is a pure function, tested on both fixtures.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V3syn2fN2LNo5pvTSqc6eh
2026-10-18 05:32:29 +00:00
hermes
2c93c73b6a feat(core): broker summaries from IPOT
Adds a BrokerSummaryProvider capability served by Indo Premier's public
broker summary page: the top 10 buyers and sellers of a stock by lots over
a date range, with totals. No login or key.

- api/ipot: pure parser (decimal scaling of `1.6 M`-style figures), a
  client that retries the server's spurious 404s, and a fixture mock.
- broker: Broker::summary/latest over the cache (7 days for closed ranges,
  5 minutes for today), range checks, and IDX member names bundled from
  GetBrokerCodeList.
- foreign_net_value is null for filtered summaries: the source does not
  apply the filters to it.

The endpoint's behavior was first mapped by fzn0x/idx (MIT).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V3syn2fN2LNo5pvTSqc6eh
2026-10-18 04:28:38 +00:00
hermes
ebdba13a41 chore: release 0.3.3
Patch release for the full-repo audit of 0.3.2: ownership query
correctness (changes, alias search, graph, investor lookup), snapshot
sync safety, request and lock time limits, the CLI output contract and
terminal-safe output, fundamental rating logic, publisher and installer
robustness, and zip off a yanked release (#51).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V3syn2fN2LNo5pvTSqc6eh
2026-10-18 02:16:18 +00:00
hermes
9f714dfed2 fix(publisher): require the jq and bash it uses; pin CI actions
From the Codex re-gate of the 0.3.3 fixes: NUL-separated discovery
fields need jq 1.7 (--raw-output0) and bash 4.4 (mapfile -d); check both
up front with a clear message instead of failing midway on older tools.
ci.yml actions are pinned to commit SHAs like the release workflows.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V3syn2fN2LNo5pvTSqc6eh
2026-10-18 02:16:18 +00:00
hermes
304709f3ac fix(release): robust publisher fields, WAL-safe copy, installer tag order, pinned actions
From the full-repo Codex audit:
- The publisher read discovery fields one per line, so a report title
  with newlines shifted every later field and published a manifest
  without source.publish_date, which every client rejects (blocking all
  syncs). Read them NUL-separated, and refuse to write a manifest missing
  a field clients require.
- The snapshot builder copied the SQLite file with cp; committed data
  still in the -wal file of an open database would be missing. Checkpoint
  first (a no-op for the publisher's closed database, so the bytes and
  hash are unchanged) and refuse to copy when that cannot complete.
- install.sh took the tag from a greedy match, which on a compact
  single-line releases response picks the last tag, not the newest (it
  installed v0.2.3 over v0.2.4 in the new test). Split fields first.
- scripts/audit-msn-fundamentals.sh read the symbol map from a path that
  no longer exists and silently checked no stocks.
- release.yml granted contents: write to every job; only publish gets it
  now. Actions in the release and snapshot workflows are pinned to commit
  SHAs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V3syn2fN2LNo5pvTSqc6eh
2026-10-18 02:16:18 +00:00
hermes
49b2d3aaba fix(analysis): ratings need evidence; negative equity is high risk
From the full-repo Codex audit:
- The overall fundamental rating was "healthy" whenever growth was
  growing and valuation and risk were merely not "expensive" / not "high
  risk", so with no valuation or risk data at all (partial MSN
  fundamentals) it still read "healthy". It now needs valuation
  undervalued or fairly valued and risk low or moderate.
- Risk recognized only heavy leverage or a weak current ratio as high
  risk; negative equity fell through to "moderate risk", and a known
  adverse signal with the other metric missing became "incomplete data".
  Any known adverse signal now makes it high risk.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V3syn2fN2LNo5pvTSqc6eh
2026-10-18 02:16:18 +00:00
hermes
5cff14bf90 docs(cli): help for config keys and output, changes dates, alias search
`idx config --help` lists the keys and says when it prints JSON;
`ownership changes` says both dates must be imported releases and that
accounts are summed; `ownership search` and `entity` describe alias and
substring matching. Regenerated the CLI reference page.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V3syn2fN2LNo5pvTSqc6eh
2026-10-18 02:16:11 +00:00
hermes
57eccc5843 fix(cli): idx config prints JSON only when JSON is asked for
Auto output stays plain text when piped, as in 0.3.2, so scripts that
read $(idx config path) or $(idx config get KEY) keep working; -o json,
IDX_OUTPUT=json, or general.output = "json" print JSON. The config
goldens return to their 0.3.2 content.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V3syn2fN2LNo5pvTSqc6eh
2026-10-18 02:11:31 +00:00
hermes
c8e79b8f7b fix(cli): keep a recovered -o when a retry reaches --help or --version
Dropping an unknown argument can expose a later --help, and clap then
returns no matches, losing an -o already read. Keep the last pass that
read one (Pullfrog review on #56).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V3syn2fN2LNo5pvTSqc6eh
2026-10-17 13:24:28 +00:00
hermes
70e923334d fix(cli): read an -o that follows an unknown argument
clap stops parsing at an unknown argument, so `idx config get --bogus
-o json` lost the -o. Drop each unknown argument and parse again
before reading the output flag (Pullfrog review on #56).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V3syn2fN2LNo5pvTSqc6eh
2026-10-17 13:24:28 +00:00
hermes
0f2806f9f5 fix(cli): pick the argument-error format the way clap picks -o
The raw scan took the first -o and missed clustered short flags, so
`idx -o table config -o json get` and `idx -qojson config get` printed
plain-text errors. Resolve -o with clap itself, ignoring the error being
reported (Pullfrog review on #56).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V3syn2fN2LNo5pvTSqc6eh
2026-10-17 13:24:28 +00:00
hermes
927d4224e9 fix(cli): sanitize errors and diagnostics at their choke points
From the narrow Codex confirmation: provider, KSEI, and manifest text
still reached the terminal through error messages (a Yahoo error
description, a manifest's `kind`), idx-core diagnostics, the ownership
heading's ticker code and Bing date, the technical heading and
comparison symbol, and printed paths. Sanitize where every error
(emit_error) and every diagnostic (runtime) is printed instead of at each
caller, plus the remaining direct prints. terminal_safe moves to
idx_core::text so the runtime can use it; the CLI re-exports it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V3syn2fN2LNo5pvTSqc6eh
2026-10-17 13:24:28 +00:00
hermes
75b8819d95 fix(cli): sanitize every provider or user value printed outside tables
From the second Codex re-gate: the empty-sentiment line printed MSN's
symbol raw, so a hostile response could clear the screen. Sweep every
print in the table and ownership renderers that interpolates data: the
sentiment and analysis headings, stale-quote notes, sync reasons, flow
period, search query, alias mapping, and DOT graph output.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V3syn2fN2LNo5pvTSqc6eh
2026-10-17 13:24:28 +00:00
hermes
e3df327f31 fix(cli): keep idx's own colors; sanitize remaining provider text; no UTF-8 panic
From the Codex re-gate of the 0.3.3 fixes:
- Colored signal cells (Bullish/Bearish, ratings, ownership change and
  flow labels) were painted before the new cell sanitizer, which turned
  their escape codes into U+FFFD on every color terminal; the goldens run
  without color and missed it. Labels idx produces itself go into cells
  untouched (`painted`), with the label made safe before styling.
- Statement dates and currency, and the entity name in `ownership
  entity`, still reached the terminal unsanitized.
- The JSON-mode argument handler read std::env::args(), which panics on
  a non-UTF-8 argument (exit 101 instead of 2); use args_os.
- A missing subcommand printed help as plain text in JSON mode; it is an
  error (exit 2), so it is a JSON error there too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V3syn2fN2LNo5pvTSqc6eh
2026-10-17 13:24:28 +00:00
hermes
ec2e870c6a fix(cli): output contract, terminal-safe text, char-safe truncation, cache symlinks
From the full-repo Codex audit:
- Provider and KSEI text reached the terminal raw, so an escape sequence
  in a news title or insight could clear or rewrite the screen. Every
  table cell, insight line, and graph label now has control characters
  and bidirectional overrides replaced.
- Table rendering sliced strings by byte (a description at 200, a URL at
  72, an earnings period), panicking when a multi-byte character such as
  "’" or "é" fell on the cut. Truncate by character.
- In JSON mode, argument errors were clap's plain text and warnings were
  plain lines ahead of the JSON error. Argument errors are now the usual
  JSON error (INVALIDINPUT, exit 2) when JSON output is selected, and
  diagnostics are JSON lines; --help and --version are unchanged.
- `config` commands ignored the output format; they follow it like every
  other command (JSON when piped or with -o json, text on a terminal).
- Ownership argument mistakes (--limit 0, --format, missing or
  conflicting import sources, --family, an empty graph root) returned
  PARSEERROR / exit 1; they are INVALIDINPUT / exit 2 as documented.
- `cache clear` followed symlinked directories and deleted files outside
  the cache; it now removes links without following them.

Output change, intentional: goldens regenerated for the JSON argument
error and the five config commands, which run with stdout piped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V3syn2fN2LNo5pvTSqc6eh
2026-10-17 13:24:28 +00:00
hermes
57169c75f2 revert: keep curl in idx's process group so Ctrl+C reaches it
Reverts a360d0b. A separate process group let an overflow kill the curl
behind a profile script, but Ctrl+C then reached only idx and left a
stalled transfer running (Pullfrog review on #55). Cancellation is the
common path; on overflow, curl still ends on its next write to the
closed pipe, and the error returns at once either way.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V3syn2fN2LNo5pvTSqc6eh
2026-10-17 13:24:20 +00:00
hermes
a360d0b429 fix(net): kill curl itself, not only its profile script, on overflow
Run curl in its own process group and kill the group, so the curl
behind a curl_* wrapper script ends at once instead of at its next
write or --max-time (Pullfrog review on #55).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V3syn2fN2LNo5pvTSqc6eh
2026-10-17 13:04:39 +00:00
hermes
6b0ef687d0 fix(net): return the size error without waiting on a wrapped curl
The curl_* profiles run curl without exec, so killing the wrapper left
curl holding stderr, and joining stderr waited up to --max-time. Close
stdout and return; curl ends on its next write (Pullfrog review on #55).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V3syn2fN2LNo5pvTSqc6eh
2026-10-17 12:46:01 +00:00
hermes
b2b511e9c2 fix(net): cap curl output while reading it, not after
Command::output() buffered the whole response before the size check, and
curl before 8.20 does not apply --max-filesize to decoded bodies, so a
small compressed response could exhaust memory. Read stdout up to the
limit, then kill and reap curl (Pullfrog review on #55).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V3syn2fN2LNo5pvTSqc6eh
2026-10-17 12:26:26 +00:00
hermes
46cf69db6e fix(quotes): keep negative price changes
From the Codex re-gate of the 0.3.3 fixes: the price sanity check also
applied to MSN's priceChange, so a falling stock without a previous close
got change: null. Bound a change by its size instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V3syn2fN2LNo5pvTSqc6eh
2026-10-17 12:26:26 +00:00
hermes
66207d24b6 fix(net): bounded request time, curl size and time caps, lock waits, sane prices
From the full-repo Codex audit:
- MSN and Yahoo clients set connect and body timeouts but none for the
  response headers, so a server that stalled before answering hung the
  call; four such calls take every MCP call slot. Add a 30 s overall
  deadline. Every curl-impersonate run gets --max-time 600 and a 256 MiB
  --max-filesize, and output above that is rejected (curl cannot cap a
  response without a length); the Yahoo cookie fetch gets --max-time.
- Ownership commands blocked on the database lock without limit, so a
  long sync or import, or a stuck process, held MCP call slots
  indefinitely. Wait up to 30 s, then fail with a "busy, try again" error.
- A price of 1e20 saturated `as i64` and the change subtraction then
  overflowed. Prices that are not finite, negative, or above 1e12 IDR are
  treated as missing, and a quote's own price as an error.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V3syn2fN2LNo5pvTSqc6eh
2026-10-17 12:26:26 +00:00
hermes
94aec74b09 fix(sync): restore an old install's backup before deciding to replace
A pre-0.3.3 install interrupted between its renames left only a backup.
Restoring it inside the install step came after the decision, so an
older snapshot replaced a newer database without --force, and a failed
download left no database (Pullfrog review on #54).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V3syn2fN2LNo5pvTSqc6eh
2026-10-17 12:26:18 +00:00