Phase 3c of docs/DESIGN.md. `ownership` handlers are now argument
parsing, one idx-core call, and rendering.
- investor.rs: Investor (also Idx::investor()) with releases, ticker,
entity, search, cross_holders, concentration, flow, changes, graph,
unresolved, map_alias, merge_entities, discover, sync, and import.
Import orchestration (input resolution, download, SHA-256 dedup,
PDF/XLSX/archive parsing, entity resolution, release write) moves
from the CLI unchanged.
- Typed inputs: ImportSource, HolderSource, ConcentrationRank, and
parse_report_family, with the same error messages.
- CoreConfig gains an ownership_db override; db::open_db_at opens an
explicit path. idx-cli no longer depends on rusqlite, sha2, or ureq
at runtime.
- Behavior note: an invalid `ticker --source` or `concentration --by`
is now rejected before the database file is opened (and created);
the error and exit code are unchanged.
- Fix: building idx-cli without the ownership feature no longer warns
about the ownership-only `blue` style.
All 143 goldens unchanged; 170 core + 7 CLI unit tests and 62 CLI tests
pass; clippy clean for default, release, testing, and no-default
feature sets. Live A/B against main: 16/16 stocks identical; ownership
discover got IDX 403 on both (exit 4 vs 1 is the phase 1 change).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Phase 3a of docs/DESIGN.md. MockProvider moves to api/mock.rs, and it,
the embedded MSN fixture bodies, and the IDX_MOCK_* hooks compile only
with idx-core's new `testing` feature. idx-cli enables it as a
dev-dependency, so `cargo test` keeps the fixture-backed provider while
release binaries no longer embed fixture data or honor
IDX_USE_MOCK_PROVIDER. Parser helpers used only by the mock are gated
the same way.
The packaged-cli flake check no longer relies on the mock; it checks
an offline cache miss (JSON CACHEMISS, exit 3) instead.
Goldens unchanged; clippy is clean with and without `testing`.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Phase 2 of docs/DESIGN.md, no behavior change:
- crates/idx-core (library): api, analysis, cache, config file handling
and provider selection, curl_impersonate, error, ownership, runtime,
plus the shared test fixtures. clap derives are behind an optional
`clap` feature so the library does not require clap.
- crates/idx-cli (binary `idx`): cli/, output/, and IdxConfig (output
format and color resolution). Package and binary names are unchanged.
- config.rs is split: file read/write and validation move to core; a
CLI test keeps core's OUTPUT_VALUES in sync with OutputSetting.
- ownership::db::open_db/db_path drop their unused IdxConfig argument.
- One workspace version; shared dependency versions; no third-party
version changes in Cargo.lock. LICENSE is symlinked into each crate.
- flake: workspace version, new source paths, and a per-crate package
check (idx-cli is built from its tarball against packaged idx-core,
since idx-core is not on crates.io yet). CI cache key, release path
filter, and the golden whitespace-hook exclusion follow the new layout.
Parity: all 143 golden outputs match byte for byte without
regeneration; 157 core + 17 CLI unit tests (173 before plus the new
OUTPUT_VALUES sync test), 62 CLI tests pass. Live A/B against main:
16/16 stocks cases identical; ownership discover got IDX 403 on both
(exit 4 vs 1 is the phase 1 exit-code change).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Phase 0 of docs/DESIGN.md, before any refactor code:
- tests/golden.rs runs every case in tests/golden/cases.toml (126
outputs across help, completions, config, cache, all stocks commands
on MSN and Yahoo mocks, error paths, and ownership queries on a seeded
fixture DB) and compares exit code, stdout, and stderr byte for byte.
Regenerate only with UPDATE_GOLDEN=1 for a reviewed output change.
- scripts/live-smoke.sh --compare <old-idx> runs each live case with
both binaries and fails on differing exit codes, added/missing/retyped
JSON keys, or changed values for slow-moving data.
- Exclude golden files from the whitespace/EOF pre-commit fixers, which
would otherwise rewrite pinned help output.
Both gates were fault-injected: a one-word table change fails exactly
the three history goldens, and a binary dropping profile.long_name fails
only the profile live comparison.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Accepted design for extracting idx-core as the shared data layer, the
agent-native CLI output contract, idx-mcp tools, packaging, and the
parity gates (golden output tests plus live A/B check) that the
refactor must pass before merging to main.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The 43-case mock smoke duplicated tests/cli.rs, which already runs the
real binary against fixture providers. Replace the flake's smoke-mock
check with a small packaged-cli check (version JSON, bash completions,
one mock quote), and cut scripts/live-smoke.sh from 672 to ~170 lines:
it now only runs shipped commands against live MSN/Yahoo/IDX and
validates output shape. Rewrite docs/SMOKE.md around the Nix gate and
the drift check.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Remove FEATURE_SPEC.md, TODO.md, and docs/AGENT_INTEGRATION.md; their
core gaps have shipped. Open work moves to a short ROADMAP.md.
- Remove the CLAUDE.md symlink and rewrite AGENTS.md as a quick guide:
navigation, rules, and the nix flake check gate.
- Merge OWNERSHIP_SELF_HOSTED.md into OWNERSHIP_PUBLISH.md and drop
private host integration notes.
- ARCHITECTURE: add MSN endpoints, clarify Bing ingest is unsupported.
- OWNERSHIP_SYNC: note source.pdf_url also carries XLSX URLs.
- systemd example: --history 2 so snapshots stay under the 10 MiB
client cap.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
IDX stopped posting the monthly above-1% list as an announcement after
the 2026-05-29 report; June/July/August are only on the Data
Kepemilikan Saham page. `ownership discover` could not see them, and
IDX's Cloudflare now rejects curl_chrome142 on that page and on the
/Media/ XLSX files.
- remote: parse the page's Nuxt payload (Description/Prospectus pairs,
/ escapes), classify above-1% XLSX as supported and above-5% /
investor-type workbooks as unsupported, derive as_of_date from the
Indonesian description. Keep all above-1% entries, only the newest
of the other series. Merge with the announcement results.
- discover JSON gains `format` (xlsx|pdf) and `as_of_date`; page
entries use the as-of date as publish_date (the page shows none).
- import --url accepts .xlsx and downloads via the XLSX validator.
- curl_impersonate: run_owned_with_fallback tries several browser
profiles until the body validates (JSON / page payload / %PDF / PK);
IDX_CURL_IMPERSONATE_BIN still pins one.
- Tests: page fixture parsing, date parsing, payload validation, CLI
discover via a local page server; existing discover tests now point
IDX_OWNERSHIP_DATA_PAGE_URL at a closed port so no test hits IDX.
- Docs: ARCHITECTURE discovery section, AGENTS ownership inputs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017a5Cv5Z1NzdYKgKnTiAF4E
Since June 2026 IDX publishes the monthly "Pemegang Saham di Atas 1%"
list as an XLSX workbook on the Data Kepemilikan Saham page instead of
a PDF announcement. Add a parser for it and accept `.xlsx` in
`ownership import --file`.
- src/ownership/xlsx.rs: zip + quick-xml reader (shared strings incl.
rich text, sheet cells), exact 12-column header match so the daily
above-5% and investor-type/classification workbooks are rejected,
Excel serial dates, numeric shares, percent -> bps with float-noise
rounding. Builds KseiHoldingDraft directly and reuses the entity
normalizers, so investor types stay uppercase like the PDF path.
- Fixtures: trimmed excerpts of the real 2026-08-31 above-1% and
2026-09-23 above-5% workbooks.
- Unit + CLI tests: import, ticker, releases, above-5% rejection.
- Docs: ARCHITECTURE parser pipeline, AGENTS ownership inputs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017a5Cv5Z1NzdYKgKnTiAF4E
The April 2026 IDX ownership PDF omits the zero-valued scrip column for
some holders, causing the parser to leave holdings_scripless/scrip as
empty strings. This broke normalize_ksei_row which called parse_id_number
on empty input.
Two-layer fix:
- Parser: pop_numeric_tail handles 2-column (missing scrip) rows by
defaulting scrip to "0"
- Normalizer: parse_id_number_or_zero backstop treats empty component
share fields as 0 while still requiring total_shares
Also includes AGENTS.md refresh, formatting cleanup (rustfmt), and
version bump to 0.2.2.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
