0xrsydn/idx-cli
1
mirror of https://github.com/0xrsydn/idx-cli.git synced 2026-08-07 01:33:52 +00:00
Code Issues Projects Releases 1 Packages Wiki Activity

feat(ownership): complete ownership intelligence module

Browse source 
Sprints 0-8: Types, schema, KSEI parser, entity resolution, DB CRUD,
Bing API client, import pipeline, query commands, graph traversal,
changes diff, entity resolution CLI, FTS5 search.

- KSEI PDF parser (mutool stext + quick-xml, 99.2% accuracy)
- 7 query commands: ticker, entity, search, cross-holders, concentration, flow, releases
- Ownership graph with recursive CTE (ASCII tree + Graphviz DOT)
- Release diff/changes between KSEI snapshots
- Entity resolution CLI: unresolved, map, merge
- FTS5 trigram search on entity names
- Feature-gated under 'ownership' (default-on)
- 82 tests passing
This commit is contained in:
Ciphercat 2026-03-07 17:31:04 +00:00
commit 6671e22976
23 changed files with 5351 additions and 47 deletions
Download patch file Download diff file Expand all files Collapse all files

1
.gitignore vendored
View file

@ -13,3 +13,4 @@ research/
# Added by cargo # Added by cargo
/target /target
out.txt

92
AGENTS.md
View file

@ -1,78 +1,78 @@
# AGENTS.md # AGENTS.md
## Project ## Project
`idx-cli` — CLI tool for Indonesian stock market (IDX) analysis. Built in Rust for humans and AI agents. Single binary, zero runtime deps. `idx-cli` — CLI tool for Indonesian stock market (IDX) analysis. Built in Rust for humans and AI agents. Single binary, schema-driven, functional architecture.
## Stack ## Stack
- **Language:** Rust (stable, via rust-overlay) - **Language:** Rust (stable, via rust-overlay)
- **CLI:** clap 4 (derive) - **CLI:** clap 4 (derive)
- **HTTP:** ureq 3 (sync, no async runtime) - **HTTP:** ureq 3 (sync, no async runtime)
- **Output:** comfy-table, owo-colors - **Output:** comfy-table, owo-colors
- **DB:** rusqlite (bundled SQLite, FTS5) — ownership module
- **Config:** TOML (`~/.config/idx/config.toml`) - **Config:** TOML (`~/.config/idx/config.toml`)
- **Cache:** JSON file-based (`~/.cache/idx/`) - **Cache:** JSON file-based (`~/.cache/idx/`)
- **Testing:** cargo nextest, assert_cmd, predicates - **Testing:** cargo test, assert_cmd, predicates
- **Hooks:** prek (pre-commit: fmt+clippy, pre-push: test) - **Hooks:** prek (pre-commit: fmt+clippy, pre-push: test)
- **VCS:** jj (Jujutsu, colocated with git)
## Structure ## Structure
``` ```
src/ src/
├── main.rs # Entry point, clap setup, command dispatch ├── main.rs # Entry point, command dispatch
├── cli/ # Command definitions (clap structs + handlers) ├── cli/ # Command handlers (clap derive structs)
│ ├── stocks.rs # stocks quote, history commands │ ├── stocks.rs # stocks quote/history/technical/fundamental/...
│ ├── config.rs # config get/set/init/path │ ├── config.rs # config get/set/init/path
│ └── cache.rs # cache info/clear │ └── cache.rs # cache info/clear
├── api/ # Data provider abstraction + implementations ├── api/ # Data providers (trait-based abstraction)
│ ├── mod.rs # MarketDataProvider trait │ ├── mod.rs # MarketDataProvider trait + factory functions
│ ├── yahoo.rs # Yahoo Finance provider (query2 endpoint) │ ├── types.rs # All domain types (Quote, Ohlc, Fundamentals, ...)
│ └── types.rs # Quote, OHLC, Period, Interval types │ ├── yahoo/ # Yahoo Finance provider (history/OHLCV)
├── output/ # Rendering layer (table, json) │ └── msn/ # MSN Finance provider (quotes, fundamentals, ++)
│ ├── table.rs # comfy-table + owo-colors ├── analysis/ # Technical & fundamental analysis (pure functions)
│ └── json.rs # serde_json pretty print ├── ownership/ # Ownership intelligence module (SQLite-backed)
│ ├── types.rs # Ownership domain types
│ └── db.rs # Schema, migrations, queries
├── output/ # Rendering (table, json)
├── cache.rs # File-based TTL cache ├── cache.rs # File-based TTL cache
├── config.rs # Config loading + merge (flags > env > file > defaults) ├── config.rs # Config loading (flags > env > file > defaults)
└── error.rs # IdxError enum (thiserror) └── error.rs # IdxError enum (thiserror)
tests/
├── cli.rs # Integration tests (assert_cmd, mock provider)
docs-internal/ # (gitignored) Specs, research, business strategy
``` ```
## Providers
- **MSN** = default provider (quotes, fundamentals, profile, earnings, financials, sentiment, insights, news, screener)
- **Yahoo** = automatic fallback for history/OHLCV (MSN doesn't support IDX history)
- Configurable: `IDX_PROVIDER=msn|yahoo`, `IDX_HISTORY_PROVIDER=auto|yahoo|msn`
## Development ## Development
```bash ```bash
# Enter dev shell (requires Nix + direnv) nix develop # enter dev shell
direnv allow # or: nix develop cargo build # build
cargo run -- stocks quote BBCA # run
# Build cargo run -- -o json stocks history BBCA # JSON output
cargo build cargo test # test
cargo fmt --check && cargo clippy -- -D warnings # lint
# Run
cargo run -- stocks quote BBCA
cargo run -- -o json stocks quote BBCA,BBRI
cargo run -- stocks history BBCA --period 3mo
# Test
cargo nextest run # or: cargo test
# Lint
cargo fmt --check
cargo clippy -- -D warnings
``` ```
## Docs
- `docs-internal/SPEC.md` — system design, command tree, milestones (gitignored)
- `docs-internal/TODO.md` — task breakdown with checklist (gitignored)
- `docs-internal/OWNERSHIP_FEATURE_DESIGN.md` — ownership intelligence feature design (gitignored)
## Verification ## Verification
```bash ```bash
cargo build # must compile cargo build # must compile
cargo clippy -- -D warnings # zero warnings cargo clippy -- -D warnings # zero warnings
cargo nextest run # all tests pass cargo test # all tests pass
``` ```
Hooks enforce this: prek runs fmt+clippy on commit, tests on push.
## Rules ## Principles
1. **Provider abstraction** — all data access goes through `MarketDataProvider` trait, never call Yahoo directly from commands 1. **Schema-driven** — define types first, build logic around them. Types are the spec.
2. **Sync only** — no tokio/async, this is a CLI tool using ureq 2. **Functional approach** — pure parse/transform functions (`parse_*`, `normalize_*`), no hidden state.
3. **Test with fixtures** — never hit live APIs in tests, use mock provider + fixture JSON 3. **Data types heavy** — rich enums, newtypes, composite structs. Precision via integer representations (basis points for %, i64 for shares).
4. **Output contract** — table mode to stdout for humans, `--json` for machines, errors to stderr 4. **Provider abstraction** — all data access through traits, never call Yahoo/MSN directly from commands.
5. **Symbol resolution** — always normalize symbols (`BBCA` → `BBCA.JK`) before API calls 5. **Sync only** — no tokio/async. CLI tool, ureq is sufficient.
6. **Test with fixtures** — never hit live APIs in tests. Mock provider + fixture JSON.
7. **Output contract** — table to stdout (humans), `--output json` (machines), errors to stderr.
8. **Feature-gated modules** — `ownership` feature for SQLite dep, keeps base binary lean.
## Docs
Detailed specs live in `docs-internal/` (gitignored — internal strategy):
- `docs-internal/SPEC.md` — system design, command tree, milestones
- `docs-internal/TODO.md` — sprint breakdown
- `docs-internal/ownership/SPEC.md` — ownership module architecture
- `docs-internal/ownership/TODO.md` — ownership sprint plan

221
Cargo.lock generated
View file

@ -8,6 +8,18 @@ version = "2.0.1"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "320119579fcad9c21884f5c4861d16174d0e06250625266f50fe6898340abefa" checksum = "320119579fcad9c21884f5c4861d16174d0e06250625266f50fe6898340abefa"
[[package]]
name = "ahash"
version = "0.8.12"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5a15f179cd60c4584b8a8c596927aadc462e27f2ca70c04e0071964a73ba7a75"
dependencies = [
"cfg-if",
"once_cell",
"version_check",
"zerocopy",
]
[[package]] [[package]]
name = "aho-corasick" name = "aho-corasick"
version = "1.1.4" version = "1.1.4"
@ -109,6 +121,15 @@ version = "2.11.0"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "843867be96c8daad0d758b57df9392b6d8d271134fce549de6ce169ff98a92af" checksum = "843867be96c8daad0d758b57df9392b6d8d271134fce549de6ce169ff98a92af"
[[package]]
name = "block-buffer"
version = "0.10.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3078c7629b62d3f0439517fa394996acacc5cbc91c5a20d8c658e77abd503a71"
dependencies = [
"generic-array",
]
[[package]] [[package]]
name = "bstr" name = "bstr"
version = "1.12.1" version = "1.12.1"
@ -263,6 +284,15 @@ version = "0.8.7"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "773648b94d0e5d620f64f280777445740e61fe701025087ec8b57f45c791888b" checksum = "773648b94d0e5d620f64f280777445740e61fe701025087ec8b57f45c791888b"
[[package]]
name = "cpufeatures"
version = "0.2.17"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "59ed5838eebb26a2bb2e58f6d5b5316989ae9d08bab10e0e6d103e656d1b0280"
dependencies = [
"libc",
]
[[package]] [[package]]
name = "crc32fast" name = "crc32fast"
version = "1.5.0" version = "1.5.0"
@ -295,6 +325,37 @@ dependencies = [
"winapi", "winapi",
] ]
[[package]]
name = "crypto-common"
version = "0.1.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "78c8292055d1c1df0cce5d180393dc8cce0abec0a7102adb6c7b1eef6016d60a"
dependencies = [
"generic-array",
"typenum",
]
[[package]]
name = "csv"
version = "1.4.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "52cd9d68cf7efc6ddfaaee42e7288d3a99d613d4b50f76ce9827ae0c6e14f938"
dependencies = [
"csv-core",
"itoa",
"ryu",
"serde_core",
]
[[package]]
name = "csv-core"
version = "0.1.13"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "704a3c26996a80471189265814dbc2c257598b96b8a7feae2d31ace646bb9782"
dependencies = [
"memchr",
]
[[package]] [[package]]
name = "deranged" name = "deranged"
version = "0.5.8" version = "0.5.8"
@ -310,6 +371,16 @@ version = "0.4.0"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6184e33543162437515c2e2b48714794e37845ec9851711914eec9d308f6ebe8" checksum = "6184e33543162437515c2e2b48714794e37845ec9851711914eec9d308f6ebe8"
[[package]]
name = "digest"
version = "0.10.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9ed9a281f7bc9b7576e61468ba615a66a5c8cfdff42420a70aa82701a3b1e292"
dependencies = [
"block-buffer",
"crypto-common",
]
[[package]] [[package]]
name = "directories" name = "directories"
version = "5.0.1" version = "5.0.1"
@ -367,6 +438,18 @@ dependencies = [
"windows-sys 0.61.2", "windows-sys 0.61.2",
] ]
[[package]]
name = "fallible-iterator"
version = "0.3.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2acce4a10f12dc2fb14a218589d4f1f62ef011b2d0cc4b3cb1bba8e94da14649"
[[package]]
name = "fallible-streaming-iterator"
version = "0.1.9"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7360491ce676a36bf9bb3c56c1aa791658183a54d2744120f27285738d90465a"
[[package]] [[package]]
name = "fastrand" name = "fastrand"
version = "2.3.0" version = "2.3.0"
@ -407,6 +490,16 @@ dependencies = [
"percent-encoding", "percent-encoding",
] ]
[[package]]
name = "generic-array"
version = "0.14.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "85649ca51fd72272d7821adaf274ad91c288277713d9c18820d8499a7ff69e9a"
dependencies = [
"typenum",
"version_check",
]
[[package]] [[package]]
name = "getrandom" name = "getrandom"
version = "0.2.17" version = "0.2.17"
@ -418,12 +511,30 @@ dependencies = [
"wasi", "wasi",
] ]
[[package]]
name = "hashbrown"
version = "0.14.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e5274423e17b7c9fc20b6e7e208532f9b19825d82dfd615708b70edd83df41f1"
dependencies = [
"ahash",
]
[[package]] [[package]]
name = "hashbrown" name = "hashbrown"
version = "0.16.1" version = "0.16.1"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "841d1cc9bed7f9236f321df977030373f4a4163ae1a7dbfe1a51a2c1a51d9100" checksum = "841d1cc9bed7f9236f321df977030373f4a4163ae1a7dbfe1a51a2c1a51d9100"
[[package]]
name = "hashlink"
version = "0.9.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6ba4ff7128dee98c7dc9794b6a411377e1404dba1c97deb8d1a55297bd25d8af"
dependencies = [
"hashbrown 0.14.5",
]
[[package]] [[package]]
name = "heck" name = "heck"
version = "0.5.0" version = "0.5.0"
@ -585,8 +696,11 @@ dependencies = [
"fastrand", "fastrand",
"owo-colors", "owo-colors",
"predicates", "predicates",
"quick-xml",
"rusqlite",
"serde", "serde",
"serde_json", "serde_json",
"sha2",
"thiserror 2.0.18", "thiserror 2.0.18",
"toml", "toml",
"ureq", "ureq",
@ -599,7 +713,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7714e70437a7dc3ac8eb7e6f8df75fd8eb422675fc7678aff7364301092b1017" checksum = "7714e70437a7dc3ac8eb7e6f8df75fd8eb422675fc7678aff7364301092b1017"
dependencies = [ dependencies = [
"equivalent", "equivalent",
"hashbrown", "hashbrown 0.16.1",
] ]
[[package]] [[package]]
@ -639,6 +753,17 @@ dependencies = [
"libc", "libc",
] ]
[[package]]
name = "libsqlite3-sys"
version = "0.30.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2e99fb7a497b1e3339bc746195567ed8d3e24945ecd636e3619d20b9de9e9149"
dependencies = [
"cc",
"pkg-config",
"vcpkg",
]
[[package]] [[package]]
name = "linux-raw-sys" name = "linux-raw-sys"
version = "0.12.1" version = "0.12.1"
@ -762,6 +887,12 @@ version = "2.3.2"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9b4f627cb1b25917193a259e49bdad08f671f8d9708acfd5fe0a8c1455d87220" checksum = "9b4f627cb1b25917193a259e49bdad08f671f8d9708acfd5fe0a8c1455d87220"
[[package]]
name = "pkg-config"
version = "0.3.32"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7edddbd0b52d732b21ad9a5fab5c704c14cd949e5e9a1ec5929a24fded1b904c"
[[package]] [[package]]
name = "potential_utf" name = "potential_utf"
version = "0.1.4" version = "0.1.4"
@ -816,6 +947,15 @@ dependencies = [
"unicode-ident", "unicode-ident",
] ]
[[package]]
name = "quick-xml"
version = "0.37.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "331e97a1af0bf59823e6eadffe373d7b27f485be8748f71471c662c1f269b7fb"
dependencies = [
"memchr",
]
[[package]] [[package]]
name = "quote" name = "quote"
version = "1.0.45" version = "1.0.45"
@ -888,6 +1028,26 @@ dependencies = [
"windows-sys 0.52.0", "windows-sys 0.52.0",
] ]
[[package]]
name = "rusqlite"
version = "0.32.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7753b721174eb8ff87a9a0e799e2d7bc3749323e773db92e0984debb00019d6e"
dependencies = [
"bitflags",
"chrono",
"csv",
"fallible-iterator",
"fallible-streaming-iterator",
"hashlink",
"libsqlite3-sys",
"serde_json",
"smallvec",
"time",
"url",
"uuid",
]
[[package]] [[package]]
name = "rustix" name = "rustix"
version = "1.1.4" version = "1.1.4"
@ -942,6 +1102,12 @@ version = "1.0.22"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b39cdef0fa800fc44525c84ccb54a029961a8215f9619753635a9c0d2538d46d" checksum = "b39cdef0fa800fc44525c84ccb54a029961a8215f9619753635a9c0d2538d46d"
[[package]]
name = "ryu"
version = "1.0.23"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9774ba4a74de5f7b1c1451ed6cd5285a32eddb5cccb8cc655a4e50009e06477f"
[[package]] [[package]]
name = "scopeguard" name = "scopeguard"
version = "1.2.0" version = "1.2.0"
@ -1000,6 +1166,17 @@ dependencies = [
"serde", "serde",
] ]
[[package]]
name = "sha2"
version = "0.10.9"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a7507d819769d01a365ab707794a4084392c824f54a7a6a7862f8c3d0892b283"
dependencies = [
"cfg-if",
"cpufeatures",
"digest",
]
[[package]] [[package]]
name = "shlex" name = "shlex"
version = "1.3.0" version = "1.3.0"
@ -1186,6 +1363,12 @@ version = "0.1.2"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5d99f8c9a7727884afe522e9bd5edbfc91a3312b36a77b5fb8926e4c31a41801" checksum = "5d99f8c9a7727884afe522e9bd5edbfc91a3312b36a77b5fb8926e4c31a41801"
[[package]]
name = "typenum"
version = "1.19.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "562d481066bde0658276a35467c4af00bdc6ee726305698a55b86e61d7ad82bb"
[[package]] [[package]]
name = "unicode-ident" name = "unicode-ident"
version = "1.0.24" version = "1.0.24"
@ -1273,6 +1456,22 @@ version = "0.2.2"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "06abde3611657adf66d383f00b093d7faecc7fa57071cce2578660c9f1010821" checksum = "06abde3611657adf66d383f00b093d7faecc7fa57071cce2578660c9f1010821"
[[package]]
name = "uuid"
version = "1.22.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a68d3c8f01c0cfa54a75291d83601161799e4a89a39e0929f4b0354d88757a37"
dependencies = [
"js-sys",
"wasm-bindgen",
]
[[package]]
name = "vcpkg"
version = "0.2.15"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "accd4ea62f7bb7a82fe23066fb0957d48ef677f6eeb8215f372f52e48bb32426"
[[package]] [[package]]
name = "version_check" name = "version_check"
version = "0.9.5" version = "0.9.5"
@ -1615,6 +1814,26 @@ dependencies = [
"synstructure", "synstructure",
] ]
[[package]]
name = "zerocopy"
version = "0.8.40"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a789c6e490b576db9f7e6b6d661bcc9799f7c0ac8352f56ea20193b2681532e5"
dependencies = [
"zerocopy-derive",
]
[[package]]
name = "zerocopy-derive"
version = "0.8.40"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f65c489a7071a749c849713807783f70672b28094011623e200cb86dcb835953"
dependencies = [
"proc-macro2",
"quote",
"syn",
]
[[package]] [[package]]
name = "zerofrom" name = "zerofrom"
version = "0.1.6" version = "0.1.6"

7
Cargo.toml
View file

@ -9,6 +9,10 @@ homepage = "https://github.com/0xrsydn/idx-cli"
keywords = ["idx", "stocks", "indonesia", "cli", "finance"] keywords = ["idx", "stocks", "indonesia", "cli", "finance"]
categories = ["command-line-utilities", "finance"] categories = ["command-line-utilities", "finance"]
[features]
default = ["ownership"]
ownership = ["dep:rusqlite", "dep:quick-xml", "dep:sha2"]
[dependencies] [dependencies]
clap = { version = "4", features = ["derive"] } clap = { version = "4", features = ["derive"] }
clap_complete = "4" clap_complete = "4"
@ -22,6 +26,9 @@ directories = "5"
chrono = { version = "0.4", features = ["serde"] } chrono = { version = "0.4", features = ["serde"] }
thiserror = "2" thiserror = "2"
fastrand = "2" fastrand = "2"
rusqlite = { version = "0.32", features = ["bundled-full"], optional = true }
quick-xml = { version = "0.37", optional = true }
sha2 = { version = "0.10", optional = true }
[dev-dependencies] [dev-dependencies]
assert_cmd = "2" assert_cmd = "2"

94
docs/ARCHITECTURE.md Normal file
View file

@ -0,0 +1,94 @@
# Architecture
## Domain Map
```
┌─────────────────────────────────────────────────────────┐
│ CLI Layer │
│ main.rs → cli/stocks.rs, cli/ownership.rs, cli/... │
│ Clap derive structs, command dispatch, arg validation │
└──────────────────────┬──────────────────────────────────┘
│
┌──────────────┼──────────────┐
│ │ │
┌───────▼───────┐ ┌────▼────┐ ┌──────▼──────┐
│ API Layer │ │ Analysis│ │ Ownership │
│ (providers) │ │ Module │ │ Module │
│ │ │ │ │ │
│ MarketData │ │ techni- │ │ SQLite DB │
│ Provider trait│ │ cal.rs │ │ KSEI parser │
│ │ │ signals │ │ Bing client │
│ Yahoo (OHLCV) │ │ fund.rs │ │ entity res. │
│ MSN (rich) │ │ │ │ FTS5 search │
└───────┬───────┘ └────┬────┘ └──────┬──────┘
│ │ │
┌───────▼──────────────▼──────────────▼──────┐
│ Output Layer │
│ table.rs (comfy-table) │ json.rs (serde) │
└─────────────────────────────────────────────┘
```
## Provider Architecture
### Dual Provider Model
- **MSN Finance** — default provider. Rich data: quotes, fundamentals, profile, earnings, financials, sentiment, insights, news, screener. No history for IDX stocks.
- **Yahoo Finance** — history fallback. Reliable OHLCV data via `/v8/finance/chart/`.
### Hybrid History Strategy
When `history_provider = auto` (default):
1. Check if current provider supports `HistoryProvider` trait
2. MSN doesn't → transparently fallback to Yahoo
3. Log info message: `"history provider fallback active (msn -> yahoo)"`
### Capability Gating
```
MarketDataProvider = QuoteProvider + FundamentalsProvider
HistoryProvider = separate trait, not all providers implement
Factory functions:
default_provider(kind) → Box<dyn MarketDataProvider>
history_provider(kind, mode, verbose) → Result<(ProviderKind, Box<dyn HistoryProvider>)>
```
## Ownership Module (SQLite-backed)
Unlike the `stocks` module (live-fetch), ownership is **import-then-query**:
1. `idx ownership import` — ETL pipeline: fetch PDF/API → parse → normalize → load SQLite
2. All query commands read from local `~/.local/share/idx/ownership.db`
3. Fully offline after import
### Data Sources
- **KSEI** — official ≥1% shareholder registry (monthly PDF from IDX)
- **Bing Finance** — global institutional ownership (REST API, quarterly)
### Parser Pipeline
```
KSEI PDF → mutool stext (XML with coordinates) → quick-xml parse → KseiRawRow
→ normalize (ID locale numbers, dates, entity names) → KseiHolding
→ SQLite INSERT (within transaction)
```
## Data Flow Patterns
### Live Query (stocks module)
```
CLI args → resolve symbol → provider.quote/history/fundamentals → render table/json
```
### Import-Query (ownership module)
```
Import: PDF/API → parse → normalize → resolve entities → SQLite INSERT
Query: CLI args → SQLite SELECT → render table/json (no network)
```
## Configuration Precedence
```
CLI flags > environment variables > config file > defaults
```
## Error Strategy
- `IdxError` enum (thiserror) with structured error codes
- Table mode: human-readable error on stderr
- JSON mode: `{"error": true, "code": "...", "message": "..."}`
- Exit code 0 on success, non-zero on failure

127
docs/CONVENTIONS.md Normal file
View file

@ -0,0 +1,127 @@
# Conventions
## Design Philosophy
### Schema-Driven Development
Define data types FIRST, then build logic around them. Types are the spec.
```rust
// ✅ Good: rich type with documented fields, integer precision
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct KseiHolding {
/// Ownership percentage in basis points: 41.10% → 4110.
pub percentage_bps: i64,
/// Total shares held (absolute count).
pub total_shares: i64,
}
// ❌ Bad: stringly-typed, float precision
pub struct Holding {
pub percentage: f64, // what unit? what precision?
pub shares: String, // why string?
}
```
### Functional Approach
Parse/transform functions are **pure**: take input, return `Result<T, IdxError>`, no side effects.
```rust
// ✅ Good: pure function, testable in isolation
pub fn parse_id_number(s: &str) -> Result<i64, IdxError> { ... }
pub fn normalize_name(raw: &str) -> String { ... }
pub fn parse_quote_from_str(symbol: &str, raw: &str) -> Result<Quote, IdxError> { ... }
// ❌ Bad: function with side effects, hard to test
pub fn fetch_and_save_quote(symbol: &str) -> Result<(), IdxError> { ... }
```
### Types Over Primitives
Use newtypes, enums, and rich structs. Avoid `String` where a domain type exists.
```rust
// ✅ Good
pub struct InvestorTypeCode(pub String);
pub enum Locality { Local, Foreign }
pub enum FlowSignal { Holder, Buyer, Seller, NewPosition, Exited }
// ❌ Bad
pub type InvestorType = String;
pub type Locality = String;
```
## Naming
### Files
- `types.rs` — domain data types for a module
- `mod.rs` — module declarations and re-exports
- `client.rs` — HTTP client code
- `map.rs` / `parse.rs` — response mapping / parsing functions
- `raw_types.rs` — raw API response shapes (before normalization)
### Functions
- `parse_*` — deserialize raw data into domain types
- `normalize_*` — clean/transform data (names, numbers, dates)
- `resolve_*` — lookup/match entities
- `query_*` — read from database
- `fetch_*` — HTTP requests to external APIs
- `render_*` — output formatting (tables, JSON)
- `handle` — CLI command dispatch entry point
### Types
- `*Raw` / `*RawRow` — pre-normalization data (strings from API/PDF)
- `*Holding` — ownership fact row
- `*Metrics` — computed analytics
- `*Row` — display-ready composite type
- `*Args` — clap command arguments
## Patterns
### Provider Trait Pattern
```rust
pub trait QuoteProvider {
fn quote(&self, symbol: &str) -> Result<Quote, IdxError>;
}
// Factory function, not direct construction
pub fn default_provider(kind: ProviderKind) -> Box<dyn MarketDataProvider> { ... }
```
### Parse Pipeline Pattern
```rust
// Raw API response → domain type, always via parse function
let raw: &str = &response_body;
let quote = yahoo::parse_quote_from_str("BBCA.JK", raw)?;
```
### DB Function Pattern
```rust
// Take &Connection, caller manages lifetime. Use transactions for bulk.
pub fn insert_ksei_holdings(conn: &Connection, holdings: &[KseiHolding]) -> Result<usize, IdxError> { ... }
pub fn query_ticker_holdings(conn: &Connection, code: &str) -> Result<TickerOwnership, IdxError> { ... }
```
### Error Propagation
```rust
// Map external errors to IdxError variants
let conn = Connection::open(path)
.map_err(|e| IdxError::DatabaseError(e.to_string()))?;
```
## Integer Precision
- **Percentages:** basis points (i64). `41.10%` → `4110`
- **Shares:** absolute count (i64). No floats.
- **Prices:** whole IDR (i64). Rounded from float at parse boundary.
- **Money (USD):** whole dollars (i64) for Bing data.
## Testing
- **Unit tests:** pure functions (parsers, normalizers, signals)
- **Integration tests:** in-memory SQLite (`Connection::open_in_memory()`), mock providers
- **Fixtures:** `tests/fixtures/*.json` — real API responses, sanitized
- **No live API calls in CI** — `IDX_USE_MOCK_PROVIDER=1`
- **Test naming:** `test_<function>_<scenario>` (e.g., `test_parse_id_number_with_dots`)
## Git / VCS
- **jj (Jujutsu)** as local workflow, colocated with git
- Push via `nix develop --command git push` (for prek hooks)
- Branch naming: `feat/<name>`, `fix/<name>`
- Commit messages: conventional commits (`feat:`, `fix:`, `refactor:`, `docs:`)

1
flake.nix
View file

@ -29,6 +29,7 @@
cargo-nextest cargo-nextest
prek prek
curl-impersonate # required for Yahoo Finance auth (curl_chrome* binaries for TLS fingerprinting) curl-impersonate # required for Yahoo Finance auth (curl_chrome* binaries for TLS fingerprinting)
mupdf # mutool for KSEI PDF parsing (ownership module)
]; ];
env = { env = {

406
src/api/msn/bing.rs Normal file
View file

@ -0,0 +1,406 @@
//! Bing Finance institutional ownership HTTP client.
//!
//! Provides access to 5 ownership signal endpoints from the Bing hedge fund data
//! provider API. Each endpoint returns a list of institutional holders grouped by
//! signal type (holders, buyers, sellers, new positions, exits).
//!
//! # Usage
//! ```rust,ignore
//! use crate::api::msn::bing::{BingEndpoint, fetch_all_ownership};
//!
//! // Fetch all 5 signals for BBCA (instrument ID bn91jc)
//! let results = fetch_all_ownership("bn91jc", false)?;
//! for (signal, holders) in results {
//! println!("{:?}: {} rows", signal, holders.len());
//! }
//! ```
use std::time::Duration;
use crate::error::IdxError;
use crate::ownership::types::{BingHolderRaw, FlowSignal};
/// Base URL for the Bing hedge fund data provider API.
const BING_OWNERSHIP_BASE: &str =
"https://services.bingapis.com/contentservices-finance.hedgefunddataprovider/api/v1";
/// Browser-like User-Agent matching existing MSN client pattern.
const USER_AGENT: &str = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 \
(KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36";
// ── Endpoint enum ────────────────────────────────────────────────────────────
/// Bing ownership API endpoint variants.
///
/// Each variant maps to one Bing hedge-fund endpoint path and the
/// [`FlowSignal`] it represents in the domain model.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum BingEndpoint {
/// Existing top institutional holders.
TopShareHolders,
/// Net buyers over the most recent reporting period.
TopBuyers,
/// Net sellers over the most recent reporting period.
TopSellers,
/// Institutions that opened a new position this period.
TopNewShareHolders,
/// Institutions that fully exited their position this period.
TopExitedShareHolders,
}
impl BingEndpoint {
/// URL path segment used in the Bing API request.
///
/// The full URL is constructed as:
/// `{BING_OWNERSHIP_BASE}/{path}/{instrument_id}`
pub fn path(&self) -> &'static str {
match self {
Self::TopShareHolders => "GetSecurityTopShareHolders",
Self::TopBuyers => "GetSecurityTopBuyers",
Self::TopSellers => "GetSecurityTopSellers",
Self::TopNewShareHolders => "GetSecurityTopNewShareHolders",
Self::TopExitedShareHolders => "GetSecurityTopExitedShareHolders",
}
}
/// The [`FlowSignal`] this endpoint represents.
pub fn signal(&self) -> FlowSignal {
match self {
Self::TopShareHolders => FlowSignal::Holder,
Self::TopBuyers => FlowSignal::Buyer,
Self::TopSellers => FlowSignal::Seller,
Self::TopNewShareHolders => FlowSignal::NewPosition,
Self::TopExitedShareHolders => FlowSignal::Exited,
}
}
/// All endpoint variants in a canonical order.
pub fn all() -> &'static [BingEndpoint] {
&[
Self::TopShareHolders,
Self::TopBuyers,
Self::TopSellers,
Self::TopNewShareHolders,
Self::TopExitedShareHolders,
]
}
}
// ── HTTP helpers ─────────────────────────────────────────────────────────────
/// Build a `ureq::Agent` with timeouts matching existing MSN client conventions.
fn build_agent() -> ureq::Agent {
ureq::Agent::config_builder()
.timeout_connect(Some(Duration::from_secs(5)))
.timeout_recv_body(Some(Duration::from_secs(10)))
.build()
.into()
}
/// Wrapper type used to handle both bare-array and wrapped-object responses
/// from the Bing API. Some endpoints may return `[...]` directly, others may
/// wrap in `{ "value": [...] }` or similar. We first try bare Vec, then
/// fall back to the wrapper.
#[derive(Debug, serde::Deserialize)]
#[serde(untagged)]
enum BingResponse {
/// Direct JSON array of holder objects.
Array(Vec<BingHolderRaw>),
/// Object wrapper with a nested array under a common key.
Wrapped(BingWrappedResponse),
}
#[derive(Debug, serde::Deserialize)]
struct BingWrappedResponse {
/// `value` key used by some Bing OData-style responses.
#[serde(alias = "value", alias = "Value", alias = "data", alias = "Data")]
items: Option<Vec<BingHolderRaw>>,
}
impl BingResponse {
fn into_holders(self) -> Vec<BingHolderRaw> {
match self {
BingResponse::Array(v) => v,
BingResponse::Wrapped(w) => w.items.unwrap_or_default(),
}
}
}
// ── Public API ────────────────────────────────────────────────────────────────
/// Fetch ownership data from a single Bing endpoint.
///
/// `instrument_id` is the MSN instrument identifier (for example `bn91jc` for BBCA).
/// Use [`crate::api::msn::symbols::resolve_msn_id`] to obtain this from a ticker code.
///
/// Empty responses (valid HTTP 200 with empty array) are returned as `Ok(vec![])`.
/// This is expected for some IDX stocks that Bing does not cover.
///
/// # Errors
/// Returns [`IdxError::Http`] on network or non-retriable HTTP errors.
/// Returns [`IdxError::RateLimited`] after exhausting retries on 429.
/// Returns [`IdxError::ParseError`] if the response body cannot be deserialized.
pub fn fetch_holders(
instrument_id: &str,
endpoint: &BingEndpoint,
verbose: bool,
) -> Result<Vec<BingHolderRaw>, IdxError> {
let url = format!("{BING_OWNERSHIP_BASE}/{}/{instrument_id}", endpoint.path());
if verbose {
eprintln!("[bing] GET {url}");
}
let agent = build_agent();
let endpoint_name = endpoint.path();
let mut wait = Duration::from_millis(500);
for attempt in 0..3 {
let response = agent
.get(&url)
.header("User-Agent", USER_AGENT)
.header("Accept", "application/json")
.header("Accept-Language", "en-US,en;q=0.9")
.header("Origin", "https://www.bing.com")
.header("Referer", "https://www.bing.com/")
.call();
match response {
Ok(ok) => {
let body = ok
.into_body()
.read_to_string()
.map_err(|e| IdxError::Http(format!("bing {endpoint_name}: read body: {e}")))?;
if body.trim().is_empty() || body.trim() == "null" {
return Ok(vec![]);
}
let parsed: BingResponse = serde_json::from_str(&body)
.map_err(|e| IdxError::ParseError(format!("bing {endpoint_name}: {e}")))?;
return Ok(parsed.into_holders());
}
Err(ureq::Error::StatusCode(404)) => {
// Instrument not found on Bing — return empty, not an error.
return Ok(vec![]);
}
Err(ureq::Error::StatusCode(429)) => {
if attempt < 2 {
std::thread::sleep(wait);
wait *= 2;
continue;
}
return Err(IdxError::RateLimited);
}
Err(ureq::Error::StatusCode(code)) if code >= 500 => {
if attempt < 2 {
std::thread::sleep(wait);
wait *= 2;
continue;
}
return Err(IdxError::Http(format!(
"bing {endpoint_name}: status {code}"
)));
}
Err(err) => {
return Err(IdxError::Http(format!("bing {endpoint_name}: {err}")));
}
}
}
Err(IdxError::RateLimited)
}
/// Fetch all 5 ownership endpoints for a given MSN instrument ID.
///
/// Returns results grouped by [`FlowSignal`] in canonical endpoint order:
/// `Holder`, `Buyer`, `Seller`, `NewPosition`, `Exited`.
///
/// Endpoints that return empty data are still included as `(signal, vec![])`.
///
/// # Errors
/// Propagates any non-empty error from [`fetch_holders`].
pub fn fetch_all_ownership(
instrument_id: &str,
verbose: bool,
) -> Result<Vec<(FlowSignal, Vec<BingHolderRaw>)>, IdxError> {
let mut results = Vec::with_capacity(5);
for endpoint in BingEndpoint::all() {
let holders = fetch_holders(instrument_id, endpoint, verbose)?;
results.push((endpoint.signal(), holders));
}
Ok(results)
}
// ── Tests ─────────────────────────────────────────────────────────────────────
#[cfg(test)]
mod tests {
use super::*;
// ── BingEndpoint::path() ──────────────────────────────────────────────
#[test]
fn test_endpoint_path_top_share_holders() {
assert_eq!(
BingEndpoint::TopShareHolders.path(),
"GetSecurityTopShareHolders"
);
}
#[test]
fn test_endpoint_path_top_buyers() {
assert_eq!(BingEndpoint::TopBuyers.path(), "GetSecurityTopBuyers");
}
#[test]
fn test_endpoint_path_top_sellers() {
assert_eq!(BingEndpoint::TopSellers.path(), "GetSecurityTopSellers");
}
#[test]
fn test_endpoint_path_top_new_share_holders() {
assert_eq!(
BingEndpoint::TopNewShareHolders.path(),
"GetSecurityTopNewShareHolders"
);
}
#[test]
fn test_endpoint_path_top_exited_share_holders() {
assert_eq!(
BingEndpoint::TopExitedShareHolders.path(),
"GetSecurityTopExitedShareHolders"
);
}
// ── BingEndpoint::signal() ────────────────────────────────────────────
#[test]
fn test_endpoint_signal_mapping() {
assert_eq!(BingEndpoint::TopShareHolders.signal(), FlowSignal::Holder);
assert_eq!(BingEndpoint::TopBuyers.signal(), FlowSignal::Buyer);
assert_eq!(BingEndpoint::TopSellers.signal(), FlowSignal::Seller);
assert_eq!(
BingEndpoint::TopNewShareHolders.signal(),
FlowSignal::NewPosition
);
assert_eq!(
BingEndpoint::TopExitedShareHolders.signal(),
FlowSignal::Exited
);
}
// ── BingEndpoint::all() ───────────────────────────────────────────────
#[test]
fn test_endpoint_all_has_five_variants() {
assert_eq!(BingEndpoint::all().len(), 5);
}
#[test]
fn test_endpoint_all_covers_all_signals() {
use std::collections::HashSet;
let signals: HashSet<String> = BingEndpoint::all()
.iter()
.map(|e| format!("{:?}", e.signal()))
.collect();
assert!(signals.contains("Holder"));
assert!(signals.contains("Buyer"));
assert!(signals.contains("Seller"));
assert!(signals.contains("NewPosition"));
assert!(signals.contains("Exited"));
}
// ── Fixture deserialization ───────────────────────────────────────────
#[test]
fn test_deserialize_holders_fixture() {
let json = include_str!("../../../tests/fixtures/bing_holders_bbca.json");
let holders: Vec<BingHolderRaw> = serde_json::from_str(json)
.expect("bing_holders_bbca.json should deserialize into Vec<BingHolderRaw>");
assert!(
!holders.is_empty(),
"fixture should have at least one holder"
);
// First holder should have a name
assert!(
holders[0].investor_name.is_some(),
"first holder should have investor_name"
);
}
#[test]
fn test_deserialize_buyers_fixture() {
let json = include_str!("../../../tests/fixtures/bing_buyers_bbca.json");
let holders: Vec<BingHolderRaw> = serde_json::from_str(json)
.expect("bing_buyers_bbca.json should deserialize into Vec<BingHolderRaw>");
assert!(
!holders.is_empty(),
"fixture should have at least one buyer"
);
}
// ── Empty response handling ───────────────────────────────────────────
#[test]
fn test_deserialize_empty_array_is_ok() {
let json = "[]";
let holders: Vec<BingHolderRaw> = serde_json::from_str(json).expect("empty array is valid");
assert!(holders.is_empty());
}
#[test]
fn test_bing_response_bare_array() {
let json = r#"[
{
"investorName": "Vanguard Group",
"investorType": "Institutional",
"sharesHeld": 1234567.0,
"reportDate": "2024-12-31"
}
]"#;
let resp: BingResponse = serde_json::from_str(json).unwrap();
let holders = resp.into_holders();
assert_eq!(holders.len(), 1);
assert_eq!(holders[0].investor_name.as_deref(), Some("Vanguard Group"));
}
#[test]
fn test_bing_response_wrapped_value_key() {
let json = r#"{
"value": [
{
"InvestorName": "BlackRock",
"InvestorType": "Institutional",
"SharesHeld": 9876543.0
}
]
}"#;
let resp: BingResponse = serde_json::from_str(json).unwrap();
let holders = resp.into_holders();
assert_eq!(holders.len(), 1);
assert_eq!(holders[0].investor_name.as_deref(), Some("BlackRock"));
}
#[test]
fn test_bing_response_wrapped_empty_value() {
let json = r#"{ "value": [] }"#;
let resp: BingResponse = serde_json::from_str(json).unwrap();
let holders = resp.into_holders();
assert!(holders.is_empty());
}
#[test]
fn test_holder_optional_fields_are_none() {
let json = r#"[{"investorName": "Some Fund"}]"#;
let holders: Vec<BingHolderRaw> = serde_json::from_str(json).unwrap();
assert_eq!(holders.len(), 1);
assert!(holders[0].shares_held.is_none());
assert!(holders[0].shares_changed.is_none());
assert!(holders[0].pct_outstanding.is_none());
assert!(holders[0].value.is_none());
assert!(holders[0].report_date.is_none());
}
}

2
src/api/msn/mod.rs
View file

@ -1,3 +1,5 @@
#[allow(dead_code, clippy::enum_variant_names)]
pub mod bing;
mod client; mod client;
mod map; mod map;
mod parse; mod parse;

5
src/cli/mod.rs
View file

@ -1,5 +1,7 @@
pub mod cache; pub mod cache;
pub mod config; pub mod config;
#[cfg(feature = "ownership")]
pub mod ownership;
pub mod stocks; pub mod stocks;
use clap::{Parser, Subcommand, ValueEnum}; use clap::{Parser, Subcommand, ValueEnum};
@ -45,6 +47,9 @@ pub enum Commands {
Config(config::ConfigCmd), Config(config::ConfigCmd),
#[command(about = "Manage local cache")] #[command(about = "Manage local cache")]
Cache(cache::CacheCmd), Cache(cache::CacheCmd),
#[cfg(feature = "ownership")]
#[command(about = "Ownership intelligence (KSEI + Bing)")]
Ownership(ownership::OwnershipCmd),
#[command(about = "Generate shell completions")] #[command(about = "Generate shell completions")]
Completions { shell: Shell }, Completions { shell: Shell },
#[command(about = "Show idx-cli version")] #[command(about = "Show idx-cli version")]

888
src/cli/ownership.rs Normal file
View file

@ -0,0 +1,888 @@
use std::collections::HashSet;
use std::fs;
use std::path::{Path, PathBuf};
use chrono::Utc;
use clap::{Args, Subcommand};
use comfy_table::{Cell, ContentArrangement, Table, presets::UTF8_FULL};
use directories::ProjectDirs;
use owo_colors::OwoColorize;
use serde::Serialize;
use sha2::{Digest, Sha256};
use crate::config::IdxConfig;
use crate::error::IdxError;
use crate::output::OutputFormat;
use crate::output::json;
use crate::output::table::format_idr;
use crate::ownership::types::{
ChangeType, FlowSignal, HolderRow, KseiHolding, OwnershipRelease, OwnershipSource,
};
use crate::ownership::{db, entities, graph, parser, search};
#[derive(Debug, Args)]
pub struct OwnershipCmd {
#[command(subcommand)]
pub command: OwnershipCommand,
}
#[derive(Debug, Subcommand)]
pub enum OwnershipCommand {
/// Import ownership data from KSEI PDF or Bing API.
Import(ImportArgs),
/// Show all holders for a ticker (KSEI + Bing combined).
Ticker(TickerArgs),
/// Show all holdings for an entity across tickers.
Entity(EntityArgs),
/// Search entities by name.
Search(SearchArgs),
/// Rank entities by cross-ownership breadth.
CrossHolders(CrossHolderArgs),
/// Rank tickers by ownership concentration.
Concentration(ConcentrationArgs),
/// Show Bing institutional flow for a ticker.
Flow(FlowArgs),
/// Diff two KSEI releases.
Changes(ChangesArgs),
/// Ownership network graph.
Graph(GraphArgs),
/// Manual entity resolution workflow.
Resolve(ResolveArgs),
/// List imported KSEI releases.
Releases,
}
#[derive(Debug, Args)]
pub struct ImportArgs {
/// URL to KSEI ownership PDF.
#[arg(long)]
pub url: Option<String>,
/// Path to local KSEI PDF file.
#[arg(long)]
pub file: Option<PathBuf>,
/// Fetch Bing institutional data for these symbols.
#[arg(long, value_delimiter = ',')]
pub fetch_bing: Option<Vec<String>>,
/// Re-import even if already imported.
#[arg(long)]
pub force: bool,
}
#[derive(Debug, Args)]
pub struct TickerArgs {
pub symbol: String,
#[arg(long, default_value = "all")]
pub source: String,
}
#[derive(Debug, Args)]
pub struct EntityArgs {
pub name: String,
}
#[derive(Debug, Args)]
pub struct SearchArgs {
pub query: String,
#[arg(long, default_value_t = 20)]
pub limit: usize,
}
#[derive(Debug, Args)]
pub struct CrossHolderArgs {
#[arg(long, default_value_t = 20)]
pub top: usize,
#[arg(long, default_value_t = 2)]
pub min_tickers: usize,
}
#[derive(Debug, Args)]
pub struct ConcentrationArgs {
#[arg(long, default_value = "hhi")]
pub by: String,
#[arg(long, default_value_t = 20)]
pub top: usize,
}
#[derive(Debug, Args)]
pub struct FlowArgs {
pub symbol: String,
}
#[derive(Debug, Args)]
pub struct ChangesArgs {
#[arg(long)]
pub from: String,
#[arg(long)]
pub to: String,
}
#[derive(Debug, Args)]
pub struct GraphArgs {
/// Ticker code or entity name to start from.
pub root: String,
/// Traversal depth (default 2).
#[arg(long, default_value_t = 2)]
pub depth: usize,
/// Output format: table or dot (Graphviz).
#[arg(long, default_value = "table")]
pub format: String,
}
#[derive(Debug, Args)]
pub struct ResolveArgs {
#[command(subcommand)]
pub command: ResolveCommand,
}
#[derive(Debug, Subcommand)]
pub enum ResolveCommand {
/// List unresolved or low-confidence entity aliases.
Unresolved {
#[arg(long, default_value_t = 50)]
limit: usize,
},
/// Manually map a raw investor name to a canonical entity name.
Map {
/// The raw name as it appears in the data.
alias: String,
/// The canonical entity name to map to.
entity: String,
},
/// Merge two entities (keep first, merge second into it).
Merge {
/// Entity ID to keep.
keep: i64,
/// Entity ID to merge and delete.
merge: i64,
},
}
pub fn handle(cmd: &OwnershipCommand, config: &IdxConfig) -> Result<(), IdxError> {
match cmd {
OwnershipCommand::Import(args) => handle_import(args, config),
OwnershipCommand::Ticker(args) => handle_ticker(args, config),
OwnershipCommand::Entity(args) => handle_entity(args, config),
OwnershipCommand::Search(args) => handle_search(args, config),
OwnershipCommand::CrossHolders(args) => handle_cross_holders(args, config),
OwnershipCommand::Concentration(args) => handle_concentration(args, config),
OwnershipCommand::Flow(args) => handle_flow(args, config),
OwnershipCommand::Changes(args) => handle_changes(args, config),
OwnershipCommand::Graph(args) => handle_graph(args, config),
OwnershipCommand::Resolve(args) => handle_resolve(args, config),
OwnershipCommand::Releases => handle_releases(config),
}
}
fn handle_ticker(args: &TickerArgs, config: &IdxConfig) -> Result<(), IdxError> {
let conn = db::open_db(config)?;
let symbol = args.symbol.trim().to_uppercase();
let source = args.source.trim().to_lowercase();
if !["all", "ksei", "bing"].contains(&source.as_str()) {
return Err(IdxError::ParseError(
"invalid --source, expected: all|ksei|bing".to_string(),
));
}
let mut data = db::query_ticker_holdings(&conn, &symbol)?;
if data.holders.is_empty() {
return Err(IdxError::ParseError(format!(
"no ownership data found for symbol {symbol}"
)));
}
if source != "all" {
data.holders.retain(|h| match source.as_str() {
"ksei" => matches!(h.source, OwnershipSource::Ksei),
"bing" => matches!(h.source, OwnershipSource::Bing),
_ => true,
});
for (i, row) in data.holders.iter_mut().enumerate() {
row.rank = i + 1;
}
let percentages: Vec<i64> = data.holders.iter().map(|h| h.percentage_bps).collect();
data.concentration = db::compute_concentration(&percentages);
}
if data.holders.is_empty() {
return Err(IdxError::ParseError(format!(
"no {source} ownership data found for symbol {symbol}"
)));
}
if matches!(config.output, OutputFormat::Json) {
return json::print_json(&data);
}
println!(
"{} {} KSEI={} Bing={}",
"Ownership:".bold(),
data.ticker.code.bold(),
data.ksei_as_of
.map(|d| d.format("%Y-%m-%d").to_string())
.unwrap_or_else(|| "-".to_string()),
data.bing_as_of.unwrap_or_else(|| "-".to_string())
);
let mut table = Table::new();
table
.load_preset(UTF8_FULL)
.set_content_arrangement(ContentArrangement::Dynamic)
.set_header(vec![
"RANK", "SOURCE", "INVESTOR", "TYPE", "L/F", "SHARES", "%", "SIGNAL",
]);
for h in &data.holders {
table.add_row(vec![
Cell::new(h.rank),
Cell::new(match h.source {
OwnershipSource::Ksei => "KSEI",
OwnershipSource::Bing => "BING",
}),
Cell::new(&h.name),
Cell::new(h.investor_type.clone().unwrap_or_else(|| "-".to_string())),
Cell::new(match h.locality {
Some(crate::ownership::types::Locality::Local) => "L",
Some(crate::ownership::types::Locality::Foreign) => "F",
None => "-",
}),
Cell::new(format_idr(h.shares)),
Cell::new(format_bps(h.percentage_bps)),
Cell::new(format_signal(h.signal)),
]);
}
println!("{table}");
println!(
"{} top1={} top3={} hhi={} free_float={} holders={}",
"Concentration:".bold(),
format_bps(data.concentration.top1_bps),
format_bps(data.concentration.top3_bps),
data.concentration.hhi,
format_bps(data.concentration.free_float_bps),
data.concentration.holder_count
);
Ok(())
}
fn handle_entity(args: &EntityArgs, config: &IdxConfig) -> Result<(), IdxError> {
let conn = db::open_db(config)?;
let q = format!("%{}%", args.name.trim());
let entity = conn
.query_row(
"SELECT id, canonical_name FROM entities WHERE canonical_name LIKE ?1 COLLATE NOCASE ORDER BY LENGTH(canonical_name) ASC LIMIT 1",
rusqlite::params![q],
|row| Ok((row.get::<_, i64>(0)?, row.get::<_, String>(1)?)),
)
.map_err(|_| IdxError::ParseError(format!("entity not found: {}", args.name)))?;
let data = db::query_entity_holdings(&conn, entity.0)?;
if matches!(config.output, OutputFormat::Json) {
return json::print_json(&data);
}
println!("{} {}", "Entity:".bold(), entity.1.bold());
let mut table = Table::new();
table
.load_preset(UTF8_FULL)
.set_content_arrangement(ContentArrangement::Dynamic)
.set_header(vec!["TICKER", "SOURCE", "SHARES", "%", "REPORT_DATE"]);
for row in &data.holdings {
table.add_row(vec![
Cell::new(&row.ticker.code),
Cell::new(match row.source {
OwnershipSource::Ksei => "KSEI",
OwnershipSource::Bing => "BING",
}),
Cell::new(format_idr(row.shares)),
Cell::new(format_bps(row.percentage_bps)),
Cell::new(row.report_date.format("%Y-%m-%d").to_string()),
]);
}
println!("{table}");
println!("{} {}", "Total tickers:".bold(), data.ticker_count);
Ok(())
}
fn handle_search(args: &SearchArgs, config: &IdxConfig) -> Result<(), IdxError> {
let conn = db::open_db(config)?;
let entities = search::fts_search(&conn, &args.query, args.limit)?;
if matches!(config.output, OutputFormat::Json) {
return json::print_json(&entities);
}
if entities.is_empty() {
println!("No entities found for query: {}", args.query);
return Ok(());
}
let mut table = Table::new();
table
.load_preset(UTF8_FULL)
.set_content_arrangement(ContentArrangement::Dynamic)
.set_header(vec!["ENTITY", "TYPE", "COUNTRY"]);
for entity in &entities {
table.add_row(vec![
Cell::new(&entity.canonical_name),
Cell::new(
entity
.entity_type
.clone()
.unwrap_or_else(|| "-".to_string()),
),
Cell::new(entity.country.clone().unwrap_or_else(|| "-".to_string())),
]);
}
println!("{table}");
Ok(())
}
fn handle_cross_holders(args: &CrossHolderArgs, config: &IdxConfig) -> Result<(), IdxError> {
let conn = db::open_db(config)?;
let rows = db::query_cross_holders(&conn, args.min_tickers, args.top)?;
if matches!(config.output, OutputFormat::Json) {
return json::print_json(&rows);
}
let mut table = Table::new();
table
.load_preset(UTF8_FULL)
.set_content_arrangement(ContentArrangement::Dynamic)
.set_header(vec!["RANK", "ENTITY", "TICKERS", "TOTAL_BPS"]);
for (i, row) in rows.iter().enumerate() {
table.add_row(vec![
Cell::new(i + 1),
Cell::new(&row.entity.canonical_name),
Cell::new(row.ticker_count),
Cell::new(format_bps(row.total_bps)),
]);
}
println!("{table}");
Ok(())
}
fn handle_concentration(args: &ConcentrationArgs, config: &IdxConfig) -> Result<(), IdxError> {
let conn = db::open_db(config)?;
let by = args.by.trim().to_lowercase();
if !["top1", "top3", "hhi"].contains(&by.as_str()) {
return Err(IdxError::ParseError(
"invalid --by, expected: top1|top3|hhi".to_string(),
));
}
let rows = db::query_concentration(&conn, &by, args.top)?;
if matches!(config.output, OutputFormat::Json) {
return json::print_json(&rows);
}
let mut table = Table::new();
table
.load_preset(UTF8_FULL)
.set_content_arrangement(ContentArrangement::Dynamic)
.set_header(vec![
"RANK",
"TICKER",
"TOP1%",
"TOP3%",
"HHI",
"FREE_FLOAT%",
"HOLDERS",
]);
for (idx, (ticker, m)) in rows.iter().enumerate() {
table.add_row(vec![
Cell::new(idx + 1),
Cell::new(ticker),
Cell::new(format_bps(m.top1_bps)),
Cell::new(format_bps(m.top3_bps)),
Cell::new(m.hhi),
Cell::new(format_bps(m.free_float_bps)),
Cell::new(m.holder_count),
]);
}
println!("{table}");
Ok(())
}
fn handle_flow(args: &FlowArgs, config: &IdxConfig) -> Result<(), IdxError> {
let conn = db::open_db(config)?;
let symbol = args.symbol.trim().to_uppercase();
let ticker_id = db::get_ticker_id(&conn, &symbol)?
.ok_or_else(|| IdxError::SymbolNotFound(symbol.clone()))?;
let flow = db::query_bing_flow(&conn, ticker_id)?;
let Some(flow) = flow else {
println!(
"No institutional flow data. Run: idx ownership import --fetch-bing {}",
symbol
);
return Ok(());
};
if matches!(config.output, OutputFormat::Json) {
return json::print_json(&flow);
}
println!("{} {} ({})", "Flow:".bold(), symbol.bold(), flow.period);
print_flow_section("TOP BUYERS", &flow.top_buyers);
print_flow_section("TOP SELLERS", &flow.top_sellers);
print_flow_section("NEW POSITIONS", &flow.new_positions);
print_flow_section("EXITED", &flow.exited);
Ok(())
}
fn print_flow_section(title: &str, rows: &[HolderRow]) {
println!("\n{}", title.bold());
if rows.is_empty() {
println!("-");
return;
}
let mut table = Table::new();
table
.load_preset(UTF8_FULL)
.set_content_arrangement(ContentArrangement::Dynamic)
.set_header(vec!["RANK", "INVESTOR", "SHARES", "%", "SIGNAL"]);
for (i, row) in rows.iter().enumerate() {
table.add_row(vec![
Cell::new(i + 1),
Cell::new(&row.name),
Cell::new(format_idr(row.shares)),
Cell::new(format_bps(row.percentage_bps)),
Cell::new(format_signal(row.signal)),
]);
}
println!("{table}");
}
fn handle_graph(args: &GraphArgs, config: &IdxConfig) -> Result<(), IdxError> {
let conn = db::open_db(config)?;
let (nodes, edges) = graph::query_ownership_graph(&conn, &args.root, args.depth)?;
if matches!(config.output, OutputFormat::Json) {
#[derive(Serialize)]
struct GraphOutput<'a> {
nodes: &'a [crate::ownership::types::GraphNode],
edges: &'a [crate::ownership::types::GraphEdge],
}
return json::print_json(&GraphOutput {
nodes: &nodes,
edges: &edges,
});
}
match args.format.trim().to_lowercase().as_str() {
"table" => {
print!("{}", graph::format_graph_text(&nodes, &edges));
Ok(())
}
"dot" => {
print!("{}", graph::format_graph_dot(&nodes, &edges));
Ok(())
}
other => Err(IdxError::ParseError(format!(
"invalid --format '{other}', expected table|dot"
))),
}
}
fn handle_changes(args: &ChangesArgs, config: &IdxConfig) -> Result<(), IdxError> {
let conn = db::open_db(config)?;
let changes = db::query_changes(&conn, &args.from, &args.to)?;
if matches!(config.output, OutputFormat::Json) {
return json::print_json(&changes);
}
if changes.is_empty() {
println!(
"No changes found between {} and {}.",
args.from.trim(),
args.to.trim()
);
return Ok(());
}
let mut table = Table::new();
table
.load_preset(UTF8_FULL)
.set_content_arrangement(ContentArrangement::Dynamic)
.set_header(vec!["TICKER", "ENTITY", "CHANGE", "OLD%", "NEW%", "DELTA%"]);
for row in changes {
table.add_row(vec![
Cell::new(row.ticker_code),
Cell::new(row.entity_name),
Cell::new(format_change_type(row.change_type)),
Cell::new(
row.old_bps
.map(format_bps)
.unwrap_or_else(|| "-".to_string()),
),
Cell::new(
row.new_bps
.map(format_bps)
.unwrap_or_else(|| "-".to_string()),
),
Cell::new(format_signed_bps(row.delta_bps)),
]);
}
println!("{table}");
Ok(())
}
fn handle_resolve(args: &ResolveArgs, config: &IdxConfig) -> Result<(), IdxError> {
let conn = db::open_db(config)?;
match &args.command {
ResolveCommand::Unresolved { limit } => {
let rows = db::list_unresolved(&conn, *limit)?;
if matches!(config.output, OutputFormat::Json) {
return json::print_json(&rows);
}
if rows.is_empty() {
println!("No unresolved or low-confidence aliases found.");
return Ok(());
}
let mut table = Table::new();
table
.load_preset(UTF8_FULL)
.set_content_arrangement(ContentArrangement::Dynamic)
.set_header(vec![
"RAW_NAME",
"SOURCE",
"TICKER",
"CURRENT_ENTITY",
"CONFIDENCE",
]);
for row in rows {
table.add_row(vec![
Cell::new(row.raw_name),
Cell::new(row.source),
Cell::new(row.ticker_code),
Cell::new(row.current_entity.unwrap_or_else(|| "-".to_string())),
Cell::new(
row.confidence
.map(|v| format!("{v:.2}"))
.unwrap_or_else(|| "-".to_string()),
),
]);
}
println!("{table}");
Ok(())
}
ResolveCommand::Map { alias, entity } => {
db::manual_map(&conn, alias, entity)?;
println!("Mapped alias '{}' -> '{}'.", alias, entity);
Ok(())
}
ResolveCommand::Merge { keep, merge } => {
let alias_updates: i64 = conn
.query_row(
"SELECT COUNT(*) FROM entity_aliases WHERE entity_id = ?1",
rusqlite::params![merge],
|row| row.get(0),
)
.map_err(|e| IdxError::DatabaseError(e.to_string()))?;
let ksei_updates: i64 = conn
.query_row(
"SELECT COUNT(*) FROM ksei_holdings WHERE entity_id = ?1",
rusqlite::params![merge],
|row| row.get(0),
)
.map_err(|e| IdxError::DatabaseError(e.to_string()))?;
let bing_updates: i64 = conn
.query_row(
"SELECT COUNT(*) FROM bing_holdings WHERE entity_id = ?1",
rusqlite::params![merge],
|row| row.get(0),
)
.map_err(|e| IdxError::DatabaseError(e.to_string()))?;
db::merge_entities(&conn, *keep, *merge)?;
println!(
"Merged entity {} into {} (aliases: {}, ksei_holdings: {}, bing_holdings: {}).",
merge, keep, alias_updates, ksei_updates, bing_updates
);
Ok(())
}
}
}
fn handle_import(args: &ImportArgs, config: &IdxConfig) -> Result<(), IdxError> {
if args.file.is_none() && args.url.is_none() && args.fetch_bing.is_none() {
return Err(IdxError::ParseError(
"provide one of: --file, --url, or --fetch-bing".to_string(),
));
}
if args.file.is_some() && args.url.is_some() {
return Err(IdxError::ParseError(
"--file and --url are mutually exclusive".to_string(),
));
}
if let Some(symbols) = &args.fetch_bing {
let clean: Vec<String> = symbols
.iter()
.flat_map(|v| v.split(','))
.map(str::trim)
.filter(|s| !s.is_empty())
.map(|s| s.to_uppercase())
.collect();
if !clean.is_empty() {
eprintln!(
"info: --fetch-bing requested for {} symbol(s), implementation deferred for Sprint 6",
clean.len()
);
for symbol in &clean {
eprintln!(" - {symbol}");
}
}
}
let Some(pdf_path) = resolve_pdf_input(args)? else {
return Ok(());
};
let conn = db::open_db(config)?;
let sha256 = sha256_file(&pdf_path)?;
if !args.force && db::release_exists(&conn, &sha256)? {
println!("Release already imported (sha256: {sha256}). Use --force to re-import.");
return Ok(());
}
let raw_rows = parser::parse_ksei_pdf(&pdf_path)?;
if raw_rows.is_empty() {
return Err(IdxError::ParseError(
"no KSEI rows parsed from PDF".to_string(),
));
}
let mut holdings = Vec::with_capacity(raw_rows.len());
let mut ticker_ids = HashSet::new();
for raw in &raw_rows {
let draft = entities::normalize_ksei_row(raw)?;
let ticker_id = db::upsert_ticker(&conn, &draft.ticker_code, draft.issuer_name.as_deref())?;
let entity_id =
entities::resolve_entity(&conn, &draft.raw_investor_name, OwnershipSource::Ksei)?;
ticker_ids.insert(ticker_id);
holdings.push(KseiHolding {
id: 0,
ticker_id,
entity_id: Some(entity_id),
raw_investor_name: draft.raw_investor_name,
investor_type: draft.investor_type,
locality: draft.locality,
nationality: draft.nationality,
domicile: draft.domicile,
holdings_scripless: draft.holdings_scripless,
holdings_scrip: draft.holdings_scrip,
total_shares: draft.total_shares,
percentage_bps: draft.percentage_bps,
report_date: draft.report_date,
release_sha256: sha256.clone(),
});
}
let inserted_rows = db::insert_ksei_holdings(&conn, &holdings)?;
let as_of_date = holdings
.iter()
.map(|h| h.report_date)
.max()
.ok_or_else(|| {
IdxError::ParseError("missing report_date in parsed holdings".to_string())
})?;
let release = OwnershipRelease {
id: 0,
source_url: args.url.clone(),
sha256,
as_of_date,
row_count: inserted_rows,
imported_at: Utc::now().timestamp(),
};
let _ = db::insert_release(&conn, &release)?;
println!(
"Imported {} rows for {} tickers (as of {}).",
inserted_rows,
ticker_ids.len(),
as_of_date.format("%Y-%m-%d")
);
Ok(())
}
fn handle_releases(config: &IdxConfig) -> Result<(), IdxError> {
let conn = db::open_db(config)?;
let releases = db::query_releases(&conn)?;
if matches!(config.output, OutputFormat::Json) {
return json::print_json(&releases);
}
if releases.is_empty() {
println!("No ownership releases imported yet.");
return Ok(());
}
let mut table = Table::new();
table
.load_preset(UTF8_FULL)
.set_content_arrangement(ContentArrangement::Dynamic)
.set_header(vec!["AS_OF", "ROWS", "SHA256", "SOURCE", "IMPORTED_AT"]);
for r in releases {
table.add_row(vec![
Cell::new(r.as_of_date.format("%Y-%m-%d").to_string()),
Cell::new(r.row_count),
Cell::new(r.sha256),
Cell::new(r.source_url.unwrap_or_else(|| "-".to_string())),
Cell::new(r.imported_at),
]);
}
println!("{table}");
Ok(())
}
fn resolve_pdf_input(args: &ImportArgs) -> Result<Option<PathBuf>, IdxError> {
if let Some(path) = &args.file {
if !path.exists() {
return Err(IdxError::Io(format!(
"input PDF not found: {}",
path.display()
)));
}
return Ok(Some(path.clone()));
}
if let Some(url) = &args.url {
let target = cache_pdf_path(url)?;
download_pdf(url, &target)?;
return Ok(Some(target));
}
Ok(None)
}
fn cache_pdf_path(url: &str) -> Result<PathBuf, IdxError> {
let dirs = ProjectDirs::from("", "", "idx")
.ok_or_else(|| IdxError::Io("unable to resolve cache directory".to_string()))?;
let raw_dir = dirs.cache_dir().join("ownership").join("raw");
fs::create_dir_all(&raw_dir).map_err(|e| IdxError::Io(e.to_string()))?;
let mut file_name = url
.rsplit('/')
.next()
.unwrap_or("ksei-ownership.pdf")
.split('?')
.next()
.unwrap_or("ksei-ownership.pdf")
.to_string();
if file_name.is_empty() || file_name == "/" {
file_name = format!("ksei-{}.pdf", Utc::now().timestamp());
}
if !file_name.to_ascii_lowercase().ends_with(".pdf") {
file_name.push_str(".pdf");
}
Ok(raw_dir.join(file_name))
}
fn download_pdf(url: &str, target: &Path) -> Result<(), IdxError> {
let response = ureq::get(url)
.header(
"User-Agent",
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36",
)
.header("Accept", "application/pdf,application/octet-stream,*/*;q=0.8")
.header("Accept-Language", "en-US,en;q=0.9")
.header("Referer", "https://www.idx.co.id/")
.call()
.map_err(|e| IdxError::Http(format!("failed to download PDF: {e}")))?;
let mut body = response.into_body();
let bytes = body
.read_to_vec()
.map_err(|e| IdxError::Http(format!("failed reading PDF body: {e}")))?;
fs::write(target, &bytes).map_err(|e| {
IdxError::Io(format!(
"failed writing cached PDF {}: {e}",
target.display()
))
})?;
Ok(())
}
fn sha256_file(path: &Path) -> Result<String, IdxError> {
let bytes = fs::read(path).map_err(|e| {
IdxError::Io(format!(
"failed to read file for sha256 {}: {e}",
path.display()
))
})?;
let mut hasher = Sha256::new();
hasher.update(bytes);
let digest = hasher.finalize();
Ok(digest.iter().map(|b| format!("{b:02x}")).collect())
}
fn format_bps(bps: i64) -> String {
format!("{:.2}%", bps as f64 / 100.0)
}
fn format_signal(signal: Option<FlowSignal>) -> String {
match signal {
Some(FlowSignal::Buyer) => "BUYER".green().to_string(),
Some(FlowSignal::Seller) => "SELLER".red().to_string(),
Some(FlowSignal::NewPosition) => "NEW".blue().to_string(),
Some(FlowSignal::Exited) => "EXITED".yellow().to_string(),
Some(FlowSignal::Holder) => "HOLDER".to_string(),
None => "-".to_string(),
}
}
fn format_change_type(change_type: ChangeType) -> String {
match change_type {
ChangeType::New => "NEW".green().to_string(),
ChangeType::Exited => "EXITED".red().to_string(),
ChangeType::Increased => "INCREASED".green().to_string(),
ChangeType::Decreased => "DECREASED".red().to_string(),
}
}
fn format_signed_bps(bps: i64) -> String {
let pct = bps as f64 / 100.0;
if bps > 0 {
format!("+{pct:.2}%")
} else {
format!("{pct:.2}%")
}
}

8
src/error.rs
View file

@ -26,6 +26,10 @@ pub enum IdxError {
Http(String), Http(String),
#[error("auth error: {0}")] #[error("auth error: {0}")]
AuthError(String), AuthError(String),
#[error("database error: {0}")]
DatabaseError(String),
#[error("PDF parse error: {0}")]
PdfParseError(String),
} }
#[derive(Debug, Clone, Copy, Serialize, PartialEq, Eq)] #[derive(Debug, Clone, Copy, Serialize, PartialEq, Eq)]
@ -41,6 +45,8 @@ pub enum ErrorCode {
Io, Io,
Http, Http,
AuthError, AuthError,
DatabaseError,
PdfParseError,
} }
impl IdxError { impl IdxError {
@ -57,6 +63,8 @@ impl IdxError {
Self::Io(_) => ErrorCode::Io, Self::Io(_) => ErrorCode::Io,
Self::Http(_) => ErrorCode::Http, Self::Http(_) => ErrorCode::Http,
Self::AuthError(_) => ErrorCode::AuthError, Self::AuthError(_) => ErrorCode::AuthError,
Self::DatabaseError(_) => ErrorCode::DatabaseError,
Self::PdfParseError(_) => ErrorCode::PdfParseError,
} }
} }

9
src/main.rs
View file

@ -5,6 +5,8 @@ mod cli;
mod config; mod config;
mod error; mod error;
mod output; mod output;
#[cfg(feature = "ownership")]
pub mod ownership;
use clap::CommandFactory; use clap::CommandFactory;
use clap::Parser; use clap::Parser;
@ -70,6 +72,13 @@ fn run() -> Result<(), IdxError> {
return Err(err); return Err(err);
} }
} }
#[cfg(feature = "ownership")]
Commands::Ownership(cmd) => {
if let Err(err) = cli::ownership::handle(&cmd.command, &config) {
emit_error(&err, &config.output);
return Err(err);
}
}
} }
Ok(()) Ok(())

1447
src/ownership/db.rs Normal file
View file

File diff suppressed because it is too large Load diff

404
src/ownership/entities.rs Normal file
View file

@ -0,0 +1,404 @@
use chrono::NaiveDate;
use rusqlite::{Connection, params};
use crate::error::IdxError;
use crate::ownership::types::{
InvestorTypeCode, KseiHoldingDraft, KseiRawRow, Locality, OwnershipSource,
};
/// Parse Indonesian locale number string to i64.
/// "1.533.682.440" → 1533682440, "0" → 0
pub fn parse_id_number(s: &str) -> Result<i64, IdxError> {
let trimmed = s.trim();
if trimmed.is_empty() {
return Err(IdxError::ParseError(
"empty Indonesian number string".to_string(),
));
}
let normalized = trimmed.replace(['.', ' '], "");
if normalized.is_empty() || !normalized.chars().all(|c| c.is_ascii_digit()) {
return Err(IdxError::ParseError(format!(
"invalid Indonesian number format: {s}"
)));
}
normalized
.parse::<i64>()
.map_err(|e| IdxError::ParseError(format!("failed to parse Indonesian number '{s}': {e}")))
}
/// Parse Indonesian locale percentage to basis points (i64).
/// "54,94" → 5494, "0,00" → 0, "100,00" → 10000
pub fn parse_id_percentage(s: &str) -> Result<i64, IdxError> {
let trimmed = s.trim();
if trimmed.is_empty() {
return Err(IdxError::ParseError(
"empty Indonesian percentage string".to_string(),
));
}
let mut parts = trimmed.split(',');
let whole = parts.next().unwrap_or_default();
let frac = parts.next().unwrap_or("00");
if parts.next().is_some() {
return Err(IdxError::ParseError(format!(
"invalid Indonesian percentage format: {s}"
)));
}
if whole.is_empty() || !whole.chars().all(|c| c.is_ascii_digit()) {
return Err(IdxError::ParseError(format!(
"invalid Indonesian percentage whole part: {s}"
)));
}
if !frac.chars().all(|c| c.is_ascii_digit()) || frac.len() > 2 {
return Err(IdxError::ParseError(format!(
"invalid Indonesian percentage fractional part: {s}"
)));
}
let whole_i = whole.parse::<i64>().map_err(|e| {
IdxError::ParseError(format!("failed to parse Indonesian percentage '{s}': {e}"))
})?;
let frac_padded = if frac.len() == 1 {
format!("{frac}0")
} else {
frac.to_string()
};
let frac_i = frac_padded.parse::<i64>().map_err(|e| {
IdxError::ParseError(format!("failed to parse Indonesian percentage '{s}': {e}"))
})?;
Ok((whole_i * 100) + frac_i)
}
/// Parse KSEI date format to NaiveDate.
/// "27-Feb-2026" → NaiveDate(2026, 2, 27)
pub fn parse_ksei_date(s: &str) -> Result<NaiveDate, IdxError> {
let trimmed = s.trim();
if trimmed.is_empty() {
return Err(IdxError::ParseError("empty KSEI date string".to_string()));
}
NaiveDate::parse_from_str(trimmed, "%d-%b-%Y")
.map_err(|e| IdxError::ParseError(format!("invalid KSEI date '{s}': {e}")))
}
/// Normalize an investor name for entity matching.
/// Strips: "PT.", "PT ", "TBK", "Tbk", "(PERSERO)", "LIMITED", "PTE", "LTD"
/// Collapses whitespace, trims, uppercases.
pub fn normalize_name(raw: &str) -> String {
let upper = collapse_whitespace(raw).to_uppercase();
if upper.is_empty() {
return upper;
}
let mut tokens: Vec<&str> = upper.split_whitespace().collect();
loop {
let mut changed = false;
if let Some(first) = tokens.first().copied()
&& matches!(first, "PT" | "PT.")
{
tokens.remove(0);
changed = true;
}
if let Some(last) = tokens.last().copied()
&& matches!(
last,
"TBK" | "Tbk" | "(PERSERO)" | "LIMITED" | "PTE" | "LTD"
)
{
let _ = tokens.pop();
changed = true;
}
if !changed {
break;
}
}
collapse_whitespace(&tokens.join(" "))
}
/// Convert a KseiRawRow into a normalized KseiHoldingDraft.
/// Applies all parsing functions above.
pub fn normalize_ksei_row(raw: &KseiRawRow) -> Result<KseiHoldingDraft, IdxError> {
let investor_type = normalize_investor_type(&raw.investor_type);
let locality = normalize_locality(&raw.local_foreign);
Ok(KseiHoldingDraft {
ticker_code: raw.share_code.trim().to_string(),
issuer_name: optional_string(&raw.issuer_name),
raw_investor_name: raw.investor_name.trim().to_string(),
investor_type,
locality,
nationality: optional_string(&raw.nationality),
domicile: optional_string(&raw.domicile),
holdings_scripless: parse_id_number(&raw.holdings_scripless)?,
holdings_scrip: parse_id_number(&raw.holdings_scrip)?,
total_shares: parse_id_number(&raw.total_holding_shares)?,
percentage_bps: parse_id_percentage(&raw.percentage)?,
report_date: parse_ksei_date(&raw.date)?,
})
}
/// Find or create a canonical entity for a raw investor name.
/// Strategy: exact match on normalized name → rule-based normalization → create new.
/// Returns entity_id.
pub fn resolve_entity(
conn: &Connection,
raw_name: &str,
source: OwnershipSource,
) -> Result<i64, IdxError> {
let source_db = source_to_db(source);
let raw_trimmed = raw_name.trim();
if raw_trimmed.is_empty() {
return Err(IdxError::ParseError("empty raw entity name".to_string()));
}
let exact_normalized = collapse_whitespace(raw_trimmed).to_uppercase();
let rule_normalized = normalize_name(raw_trimmed);
if let Some(entity_id) = find_entity_by_alias(conn, raw_trimmed, source_db)? {
return Ok(entity_id);
}
if let Some(entity_id) = find_entity_by_canonical(conn, &exact_normalized)? {
insert_alias(conn, entity_id, raw_trimmed, source_db, "exact")?;
return Ok(entity_id);
}
if rule_normalized != exact_normalized
&& let Some(entity_id) = find_entity_by_canonical(conn, &rule_normalized)?
{
insert_alias(conn, entity_id, raw_trimmed, source_db, "rule")?;
return Ok(entity_id);
}
let now: i64 = chrono::Utc::now().timestamp();
conn.execute(
"INSERT INTO entities (canonical_name, entity_type, country, created_at, updated_at)
VALUES (?1, NULL, NULL, ?2, ?2)",
params![
if rule_normalized.is_empty() {
&exact_normalized
} else {
&rule_normalized
},
now
],
)
.map_err(|e| IdxError::DatabaseError(format!("insert entity failed: {e}")))?;
let entity_id = conn.last_insert_rowid();
insert_alias(conn, entity_id, raw_trimmed, source_db, "exact")?;
Ok(entity_id)
}
fn normalize_investor_type(raw: &str) -> Option<InvestorTypeCode> {
let value = raw.trim();
if value.is_empty() {
None
} else {
Some(InvestorTypeCode(value.to_uppercase()))
}
}
fn normalize_locality(raw: &str) -> Option<Locality> {
match raw.trim().to_uppercase().as_str() {
"L" => Some(Locality::Local),
"F" | "A" => Some(Locality::Foreign),
_ => None,
}
}
fn optional_string(raw: &str) -> Option<String> {
let value = raw.trim();
if value.is_empty() {
None
} else {
Some(value.to_string())
}
}
fn collapse_whitespace(input: &str) -> String {
input.split_whitespace().collect::<Vec<_>>().join(" ")
}
fn source_to_db(source: OwnershipSource) -> &'static str {
match source {
OwnershipSource::Ksei => "ksei",
OwnershipSource::Bing => "bing",
}
}
fn find_entity_by_alias(
conn: &Connection,
raw_name: &str,
source: &str,
) -> Result<Option<i64>, IdxError> {
conn.query_row(
"SELECT entity_id FROM entity_aliases WHERE raw_name = ?1 AND source = ?2",
params![raw_name, source],
|row| row.get(0),
)
.map(Some)
.or_else(|e| match e {
rusqlite::Error::QueryReturnedNoRows => Ok(None),
_ => Err(IdxError::DatabaseError(e.to_string())),
})
}
fn find_entity_by_canonical(
conn: &Connection,
canonical_name: &str,
) -> Result<Option<i64>, IdxError> {
conn.query_row(
"SELECT id FROM entities WHERE canonical_name = ?1",
params![canonical_name],
|row| row.get(0),
)
.map(Some)
.or_else(|e| match e {
rusqlite::Error::QueryReturnedNoRows => Ok(None),
_ => Err(IdxError::DatabaseError(e.to_string())),
})
}
fn insert_alias(
conn: &Connection,
entity_id: i64,
raw_name: &str,
source: &str,
method: &str,
) -> Result<(), IdxError> {
conn.execute(
"INSERT OR IGNORE INTO entity_aliases (entity_id, raw_name, source, confidence, method)
VALUES (?1, ?2, ?3, 1.0, ?4)",
params![entity_id, raw_name, source, method],
)
.map_err(|e| IdxError::DatabaseError(format!("insert alias failed: {e}")))?;
Ok(())
}
#[cfg(test)]
mod tests {
use chrono::NaiveDate;
use rusqlite::Connection;
use crate::ownership::entities::{
normalize_name, parse_id_number, parse_id_percentage, parse_ksei_date, resolve_entity,
};
use crate::ownership::types::OwnershipSource;
#[test]
fn test_parse_id_number() {
assert_eq!(parse_id_number("1.533.682.440").unwrap(), 1_533_682_440);
assert_eq!(parse_id_number("0").unwrap(), 0);
assert_eq!(parse_id_number("3.200.142.830").unwrap(), 3_200_142_830);
assert!(parse_id_number("").is_err());
}
#[test]
fn test_parse_id_percentage() {
assert_eq!(parse_id_percentage("54,94").unwrap(), 5494);
assert_eq!(parse_id_percentage("0,00").unwrap(), 0);
assert_eq!(parse_id_percentage("100,00").unwrap(), 10000);
assert_eq!(parse_id_percentage("41,10").unwrap(), 4110);
}
#[test]
fn test_parse_ksei_date_various_months() {
let samples = [
("01-Jan-2026", NaiveDate::from_ymd_opt(2026, 1, 1).unwrap()),
("01-Feb-2026", NaiveDate::from_ymd_opt(2026, 2, 1).unwrap()),
("01-Mar-2026", NaiveDate::from_ymd_opt(2026, 3, 1).unwrap()),
("01-Apr-2026", NaiveDate::from_ymd_opt(2026, 4, 1).unwrap()),
("01-May-2026", NaiveDate::from_ymd_opt(2026, 5, 1).unwrap()),
("01-Jun-2026", NaiveDate::from_ymd_opt(2026, 6, 1).unwrap()),
("01-Jul-2026", NaiveDate::from_ymd_opt(2026, 7, 1).unwrap()),
("01-Aug-2026", NaiveDate::from_ymd_opt(2026, 8, 1).unwrap()),
("01-Sep-2026", NaiveDate::from_ymd_opt(2026, 9, 1).unwrap()),
("01-Oct-2026", NaiveDate::from_ymd_opt(2026, 10, 1).unwrap()),
("01-Nov-2026", NaiveDate::from_ymd_opt(2026, 11, 1).unwrap()),
("01-Dec-2026", NaiveDate::from_ymd_opt(2026, 12, 1).unwrap()),
];
for (input, expected) in samples {
assert_eq!(parse_ksei_date(input).unwrap(), expected);
}
assert_eq!(
parse_ksei_date("27-Feb-2026").unwrap(),
NaiveDate::from_ymd_opt(2026, 2, 27).unwrap()
);
}
#[test]
fn test_normalize_name() {
assert_eq!(
normalize_name("PT. ASTRA INTERNATIONAL TBK"),
"ASTRA INTERNATIONAL"
);
assert_eq!(
normalize_name("PT BANK CENTRAL ASIA Tbk"),
"BANK CENTRAL ASIA"
);
assert_eq!(
normalize_name("UOB KAY HIAN PRIVATE LIMITED"),
"UOB KAY HIAN PRIVATE"
);
assert_eq!(
normalize_name("DJS Ketenagakerjaan (JHT)"),
"DJS KETENAGAKERJAAN (JHT)"
);
assert_eq!(
normalize_name("BPJS KETENAGAKERJAAN"),
"BPJS KETENAGAKERJAAN"
);
}
#[test]
fn test_resolve_entity_create_then_reuse() {
let conn = Connection::open_in_memory().unwrap();
conn.execute_batch(
r#"
CREATE TABLE entities (
id INTEGER PRIMARY KEY,
canonical_name TEXT NOT NULL,
entity_type TEXT,
country TEXT,
created_at INTEGER NOT NULL,
updated_at INTEGER NOT NULL
);
CREATE TABLE entity_aliases (
id INTEGER PRIMARY KEY,
entity_id INTEGER NOT NULL,
raw_name TEXT NOT NULL,
source TEXT NOT NULL,
confidence REAL NOT NULL DEFAULT 1.0,
method TEXT NOT NULL,
UNIQUE(raw_name, source)
);
"#,
)
.unwrap();
let id1 =
resolve_entity(&conn, "PT. ASTRA INTERNATIONAL TBK", OwnershipSource::Ksei).unwrap();
let id2 =
resolve_entity(&conn, "PT. ASTRA INTERNATIONAL TBK", OwnershipSource::Ksei).unwrap();
assert_eq!(id1, id2);
}
}

332
src/ownership/graph.rs Normal file
View file

@ -0,0 +1,332 @@
use std::collections::{BTreeMap, BTreeSet, HashMap, HashSet};
use rusqlite::{Connection, params};
use crate::error::IdxError;
use crate::ownership::types::{GraphEdge, GraphNode, GraphNodeType, OwnershipSource};
/// Query ownership graph starting from a ticker or entity, traversing N hops.
/// Root can be ticker code (`BBCA`) or entity name — auto-detect.
pub fn query_ownership_graph(
conn: &Connection,
root: &str,
depth: usize,
) -> Result<(Vec<GraphNode>, Vec<GraphEdge>), IdxError> {
let root = root.trim();
if root.is_empty() {
return Err(IdxError::ParseError(
"graph root cannot be empty".to_string(),
));
}
let root_node_id = detect_root_node(conn, root)?;
let mut visited: BTreeSet<String> = BTreeSet::new();
{
let mut stmt = conn
.prepare(
"WITH RECURSIVE
all_edges AS (
SELECT
'entity:' || k.entity_id AS from_id,
'ticker:' || t.code AS to_id
FROM ksei_holdings k
JOIN tickers t ON t.id = k.ticker_id
WHERE k.entity_id IS NOT NULL
UNION
SELECT
'entity:' || b.entity_id AS from_id,
'ticker:' || t.code AS to_id
FROM bing_holdings b
JOIN tickers t ON t.id = b.ticker_id
WHERE b.entity_id IS NOT NULL
),
neighbors AS (
SELECT from_id AS a, to_id AS b FROM all_edges
UNION
SELECT to_id AS a, from_id AS b FROM all_edges
),
walk(node_id, depth, path) AS (
SELECT ?1 AS node_id, 0 AS depth, ?1 AS path
UNION ALL
SELECT n.b, w.depth + 1, w.path || '>' || n.b
FROM walk w
JOIN neighbors n ON n.a = w.node_id
WHERE w.depth < ?2
AND instr(w.path, n.b) = 0
)
SELECT DISTINCT node_id
FROM walk",
)
.map_err(|e| IdxError::DatabaseError(e.to_string()))?;
let rows = stmt
.query_map(params![root_node_id, depth as i64], |row| {
row.get::<_, String>(0)
})
.map_err(|e| IdxError::DatabaseError(e.to_string()))?;
for row in rows {
visited.insert(row.map_err(|e| IdxError::DatabaseError(e.to_string()))?);
}
}
let mut edges = query_all_edges(conn)?;
edges.retain(|edge| visited.contains(&edge.from) && visited.contains(&edge.to));
let nodes = build_nodes(conn, &visited)?;
Ok((nodes, edges))
}
/// Format graph as ASCII tree for terminal display.
pub fn format_graph_text(nodes: &[GraphNode], edges: &[GraphEdge]) -> String {
if nodes.is_empty() {
return "(empty graph)".to_string();
}
let labels: HashMap<&str, (&str, GraphNodeType)> = nodes
.iter()
.map(|n| (n.id.as_str(), (n.label.as_str(), n.node_type)))
.collect();
let mut ticker_to_entities: BTreeMap<&str, Vec<&GraphEdge>> = BTreeMap::new();
for edge in edges {
if let Some((_, GraphNodeType::Ticker)) = labels.get(edge.to.as_str()) {
ticker_to_entities
.entry(edge.to.as_str())
.or_default()
.push(edge);
}
}
let mut out = String::new();
out.push_str(&format!("nodes: {} edges: {}\n", nodes.len(), edges.len()));
for (ticker_id, mut rels) in ticker_to_entities {
rels.sort_by(|a, b| b.percentage_bps.cmp(&a.percentage_bps));
let ticker_label = labels.get(ticker_id).map(|v| v.0).unwrap_or(ticker_id);
out.push_str(&format!("\n{ticker_label} [TICKER]\n"));
for (idx, edge) in rels.iter().enumerate() {
let holder_label = labels.get(edge.from.as_str()).map(|v| v.0).unwrap_or("?");
let branch = if idx + 1 == rels.len() {
"└─"
} else {
"├─"
};
out.push_str(&format!(
"{branch} {holder_label} ({:.2}%, {})\n",
edge.percentage_bps as f64 / 100.0,
source_label(edge.source)
));
}
}
if edges.is_empty() {
out.push_str("\n(no ownership edges found)\n");
}
out
}
/// Format graph as Graphviz DOT for export.
pub fn format_graph_dot(nodes: &[GraphNode], edges: &[GraphEdge]) -> String {
let mut out = String::new();
out.push_str("digraph ownership {\n");
out.push_str(" rankdir=LR;\n");
out.push_str(" graph [fontname=\"Helvetica\"];\n");
out.push_str(" node [fontname=\"Helvetica\"];\n");
out.push_str(" edge [fontname=\"Helvetica\"];\n\n");
for node in nodes {
let shape = match node.node_type {
GraphNodeType::Entity => "ellipse",
GraphNodeType::Ticker => "box",
};
out.push_str(&format!(
" \"{}\" [label=\"{}\", shape={}];\n",
escape_dot(&node.id),
escape_dot(&node.label),
shape
));
}
out.push('\n');
for edge in edges {
out.push_str(&format!(
" \"{}\" -> \"{}\" [label=\"{:.2}% ({})\"];\n",
escape_dot(&edge.from),
escape_dot(&edge.to),
edge.percentage_bps as f64 / 100.0,
source_label(edge.source)
));
}
out.push_str("}\n");
out
}
fn detect_root_node(conn: &Connection, root: &str) -> Result<String, IdxError> {
let maybe_ticker = conn
.query_row(
"SELECT code FROM tickers WHERE UPPER(code) = UPPER(?1) LIMIT 1",
params![root],
|row| row.get::<_, String>(0),
)
.ok();
if let Some(code) = maybe_ticker {
return Ok(format!("ticker:{code}"));
}
let q = format!("%{}%", root);
let maybe_entity_id = conn
.query_row(
"SELECT id
FROM entities
WHERE canonical_name LIKE ?1 COLLATE NOCASE
ORDER BY LENGTH(canonical_name) ASC, canonical_name ASC
LIMIT 1",
params![q],
|row| row.get::<_, i64>(0),
)
.ok();
if let Some(entity_id) = maybe_entity_id {
return Ok(format!("entity:{entity_id}"));
}
Err(IdxError::ParseError(format!(
"graph root not found as ticker or entity: {root}"
)))
}
fn query_all_edges(conn: &Connection) -> Result<Vec<GraphEdge>, IdxError> {
let mut out = Vec::new();
{
let mut stmt = conn
.prepare(
"SELECT 'entity:' || k.entity_id,
'ticker:' || t.code,
k.percentage_bps
FROM ksei_holdings k
JOIN tickers t ON t.id = k.ticker_id
WHERE k.entity_id IS NOT NULL",
)
.map_err(|e| IdxError::DatabaseError(e.to_string()))?;
let rows = stmt
.query_map([], |row| {
Ok(GraphEdge {
from: row.get(0)?,
to: row.get(1)?,
percentage_bps: row.get(2)?,
source: OwnershipSource::Ksei,
})
})
.map_err(|e| IdxError::DatabaseError(e.to_string()))?;
for row in rows {
out.push(row.map_err(|e| IdxError::DatabaseError(e.to_string()))?);
}
}
{
let mut stmt = conn
.prepare(
"SELECT 'entity:' || b.entity_id,
'ticker:' || t.code,
COALESCE(b.pct_ownership_bps, 0)
FROM bing_holdings b
JOIN tickers t ON t.id = b.ticker_id
WHERE b.entity_id IS NOT NULL",
)
.map_err(|e| IdxError::DatabaseError(e.to_string()))?;
let rows = stmt
.query_map([], |row| {
Ok(GraphEdge {
from: row.get(0)?,
to: row.get(1)?,
percentage_bps: row.get(2)?,
source: OwnershipSource::Bing,
})
})
.map_err(|e| IdxError::DatabaseError(e.to_string()))?;
for row in rows {
out.push(row.map_err(|e| IdxError::DatabaseError(e.to_string()))?);
}
}
dedup_edges(out)
}
fn dedup_edges(edges: Vec<GraphEdge>) -> Result<Vec<GraphEdge>, IdxError> {
let mut seen: HashSet<(String, String, i64, &'static str)> = HashSet::new();
let mut out = Vec::new();
for edge in edges {
let key = (
edge.from.clone(),
edge.to.clone(),
edge.percentage_bps,
source_label(edge.source),
);
if seen.insert(key) {
out.push(edge);
}
}
Ok(out)
}
fn build_nodes(conn: &Connection, node_ids: &BTreeSet<String>) -> Result<Vec<GraphNode>, IdxError> {
let mut nodes = Vec::new();
for node_id in node_ids {
if let Some(entity_id) = node_id.strip_prefix("entity:") {
let entity_id_num = entity_id.parse::<i64>().map_err(|e| {
IdxError::ParseError(format!("invalid entity node id '{node_id}': {e}"))
})?;
let label = conn
.query_row(
"SELECT canonical_name FROM entities WHERE id = ?1",
params![entity_id_num],
|row| row.get::<_, String>(0),
)
.unwrap_or_else(|_| format!("entity:{entity_id_num}"));
nodes.push(GraphNode {
id: node_id.clone(),
label,
node_type: GraphNodeType::Entity,
});
continue;
}
if let Some(code) = node_id.strip_prefix("ticker:") {
nodes.push(GraphNode {
id: node_id.clone(),
label: code.to_string(),
node_type: GraphNodeType::Ticker,
});
}
}
Ok(nodes)
}
fn source_label(source: OwnershipSource) -> &'static str {
match source {
OwnershipSource::Ksei => "ksei",
OwnershipSource::Bing => "bing",
}
}
fn escape_dot(value: &str) -> String {
value.replace('"', "\\\"")
}

6
src/ownership/mod.rs Normal file
View file

@ -0,0 +1,6 @@
pub mod db;
pub mod entities;
pub mod graph;
pub mod parser;
pub mod search;
pub mod types;

355
src/ownership/parser.rs Normal file
View file

@ -0,0 +1,355 @@
use std::collections::HashMap;
use std::path::Path;
use std::process::Command;
use quick_xml::Reader;
use quick_xml::events::Event;
use crate::error::IdxError;
use crate::ownership::types::KseiRawRow;
/// Character grid for a single PDF page: y-coord → column-index → sorted (x, char) pairs.
type PageGrid = HashMap<i32, HashMap<usize, Vec<(i32, char)>>>;
const Y_TOLERANCE: f32 = 0.8;
/// Inclusive-left, exclusive-right X ranges for each KSEI data column.
const COLUMN_BOUNDS: [(f32, f32); 12] = [
(15.0, 52.0), // date
(52.0, 70.0), // share_code
(70.0, 167.0), // issuer_name
(167.0, 432.0), // investor_name
(432.0, 463.0), // investor_type
(463.0, 497.0), // local_foreign
(497.0, 558.0), // nationality
(558.0, 615.0), // domicile
(615.0, 653.0), // holdings_scripless
(653.0, 692.0), // holdings_scrip
(692.0, 745.0), // total_holding_shares
(745.0, 800.0), // percentage
];
/// Parse a KSEI ownership PDF into raw rows.
/// Shells out to `mutool` for XML extraction, then parses with quick-xml.
pub fn parse_ksei_pdf(path: &Path) -> Result<Vec<KseiRawRow>, IdxError> {
check_mutool()?;
let output = Command::new("mutool")
.arg("convert")
.arg("-F")
.arg("stext")
.arg("-o")
.arg("-")
.arg(path)
.output()
.map_err(|e| IdxError::PdfParseError(format!("failed to run mutool: {e}")))?;
if !output.status.success() {
let stderr = String::from_utf8_lossy(&output.stderr);
return Err(IdxError::PdfParseError(format!(
"mutool convert failed: {}",
stderr.trim()
)));
}
let xml = String::from_utf8(output.stdout)
.map_err(|e| IdxError::PdfParseError(format!("invalid utf-8 stext output: {e}")))?;
parse_stext_xml(&xml)
}
/// Parse mutool stext XML output into raw rows.
/// Pure function — takes XML string, returns parsed rows.
pub fn parse_stext_xml(xml: &str) -> Result<Vec<KseiRawRow>, IdxError> {
let mut reader = Reader::from_str(xml);
reader.config_mut().trim_text(false);
let mut rows: Vec<KseiRawRow> = Vec::new();
let mut current_page: Option<PageGrid> = None;
let mut buf = Vec::new();
loop {
match reader.read_event_into(&mut buf) {
Ok(Event::Start(e)) if e.name().as_ref() == b"page" => {
current_page = Some(HashMap::new());
}
Ok(Event::Empty(e)) if e.name().as_ref() == b"char" => {
if let Some(page) = current_page.as_mut() {
let mut x: Option<f32> = None;
let mut y: Option<f32> = None;
let mut c: Option<char> = None;
for attr_result in e.attributes().with_checks(false) {
let attr = attr_result.map_err(|err| {
IdxError::PdfParseError(format!("invalid XML attribute: {err}"))
})?;
match attr.key.as_ref() {
b"x" => {
let s = attr.decode_and_unescape_value(reader.decoder()).map_err(
|err| {
IdxError::PdfParseError(format!(
"invalid XML x attribute: {err}"
))
},
)?;
x = s.parse::<f32>().ok();
}
b"y" => {
let s = attr.decode_and_unescape_value(reader.decoder()).map_err(
|err| {
IdxError::PdfParseError(format!(
"invalid XML y attribute: {err}"
))
},
)?;
y = s.parse::<f32>().ok();
}
b"c" => {
let s = attr.decode_and_unescape_value(reader.decoder()).map_err(
|err| {
IdxError::PdfParseError(format!(
"invalid XML char attribute: {err}"
))
},
)?;
c = s.chars().next();
}
_ => {}
}
}
if let (Some(x_val), Some(y_val), Some(ch)) = (x, y, c)
&& let Some(col_idx) = x_to_column(x_val)
{
let yb = y_bucket(y_val);
let xi = (x_val * 100.0).round() as i32;
page.entry(yb)
.or_default()
.entry(col_idx)
.or_default()
.push((xi, ch));
}
}
}
Ok(Event::End(e)) if e.name().as_ref() == b"page" => {
if let Some(page) = current_page.take() {
rows.extend(extract_rows_from_page(page));
}
}
Ok(Event::Eof) => break,
Err(err) => {
return Err(IdxError::PdfParseError(format!(
"failed to parse stext XML: {err}"
)));
}
_ => {}
}
buf.clear();
}
Ok(rows)
}
/// Check if mutool is available in PATH.
pub fn check_mutool() -> Result<(), IdxError> {
// mutool with no args prints usage to stderr and exits non-zero,
// so we just check that the binary is found and executable.
Command::new("mutool")
.arg("--help")
.stdout(std::process::Stdio::null())
.stderr(std::process::Stdio::null())
.status()
.map_err(|e| IdxError::PdfParseError(format!("mutool not found in PATH: {e}")))?;
Ok(())
}
fn extract_rows_from_page(page: PageGrid) -> Vec<KseiRawRow> {
let mut page_rows = Vec::new();
let mut y_keys: Vec<i32> = page.keys().copied().collect();
y_keys.sort_unstable();
for y in y_keys {
let mut row = KseiRawRow {
date: String::new(),
share_code: String::new(),
issuer_name: String::new(),
investor_name: String::new(),
investor_type: String::new(),
local_foreign: String::new(),
nationality: String::new(),
domicile: String::new(),
holdings_scripless: String::new(),
holdings_scrip: String::new(),
total_holding_shares: String::new(),
percentage: String::new(),
};
if let Some(col_map) = page.get(&y) {
for (col_idx, chars) in col_map {
let mut sorted = chars.clone();
sorted.sort_by_key(|(x, _)| *x);
let text = normalize_spaces(&sorted.iter().map(|(_, c)| c).collect::<String>());
assign_column(&mut row, *col_idx, text);
}
}
if is_data_row(&row) {
page_rows.push(row);
}
}
page_rows
}
fn assign_column(row: &mut KseiRawRow, col_idx: usize, value: String) {
match col_idx {
0 => row.date = value,
1 => row.share_code = value,
2 => row.issuer_name = value,
3 => row.investor_name = value,
4 => row.investor_type = value,
5 => row.local_foreign = value,
6 => row.nationality = value,
7 => row.domicile = value,
8 => row.holdings_scripless = value,
9 => row.holdings_scrip = value,
10 => row.total_holding_shares = value,
11 => row.percentage = value,
_ => {}
}
}
fn x_to_column(x: f32) -> Option<usize> {
COLUMN_BOUNDS
.iter()
.position(|(left, right)| x >= *left && x < *right)
}
fn y_bucket(y: f32) -> i32 {
(y / Y_TOLERANCE).round() as i32
}
fn normalize_spaces(input: &str) -> String {
let mut out = String::with_capacity(input.len());
let mut prev_space = false;
for ch in input.chars() {
if ch == ' ' {
if !prev_space {
out.push(' ');
}
prev_space = true;
} else {
out.push(ch);
prev_space = false;
}
}
out.trim().to_string()
}
fn is_data_row(row: &KseiRawRow) -> bool {
is_ksei_date(&row.date)
&& is_percentage_like(&row.percentage)
&& !row.share_code.trim().is_empty()
&& !row.investor_name.trim().is_empty()
}
fn is_ksei_date(s: &str) -> bool {
if s.len() != 11 {
return false;
}
let mut parts = s.split('-');
let day = parts.next();
let mon = parts.next();
let year = parts.next();
if parts.next().is_some() {
return false;
}
match (day, mon, year) {
(Some(d), Some(m), Some(y)) => {
d.len() == 2
&& d.chars().all(|c| c.is_ascii_digit())
&& m.len() == 3
&& m.chars().all(|c| c.is_ascii_alphabetic())
&& y.len() == 4
&& y.chars().all(|c| c.is_ascii_digit())
}
_ => false,
}
}
fn is_percentage_like(s: &str) -> bool {
let cleaned = s.trim();
if cleaned.is_empty() {
return false;
}
let mut parts = cleaned.split(',');
let left = parts.next();
let right = parts.next();
if parts.next().is_some() {
return false;
}
match (left, right) {
(Some(l), Some(r)) => {
!l.is_empty()
&& l.chars().all(|c| c.is_ascii_digit())
&& !r.is_empty()
&& r.chars().all(|c| c.is_ascii_digit())
}
_ => false,
}
}
#[cfg(test)]
mod tests {
use std::fs;
use std::path::Path;
use super::{check_mutool, parse_ksei_pdf, parse_stext_xml};
#[test]
fn test_parse_stext_xml_fixture_extracts_rows() {
let fixture_path = Path::new("tests/fixtures/ksei_stext_sample.xml");
let xml = fs::read_to_string(fixture_path).expect("failed to read stext fixture");
let rows = parse_stext_xml(&xml).expect("failed to parse fixture XML");
assert_eq!(rows.len(), 3);
let first = &rows[0];
assert_eq!(first.date, "27-Feb-2026");
assert_eq!(first.share_code, "BBCA");
assert_eq!(first.investor_name, "PT DWIMURIA INVESTAMA ANDALAN");
assert_eq!(first.percentage, "54,94");
}
#[test]
fn test_parse_ksei_pdf_real_file_row_count() {
if check_mutool().is_err() {
eprintln!("skipping mutool-dependent test: mutool not available");
return;
}
let pdf_path =
Path::new("/var/lib/openclaw/projects/idx-cli/research/ownership_202603.pdf");
if !pdf_path.exists() {
eprintln!("skipping mutool-dependent test: sample PDF not found");
return;
}
let rows = parse_ksei_pdf(pdf_path).expect("failed to parse real KSEI PDF");
assert!(
rows.len() >= 7_200,
"expected at least 7200 rows, got {}",
rows.len()
);
}
}

111
src/ownership/search.rs Normal file
View file

@ -0,0 +1,111 @@
use rusqlite::{Connection, params};
use crate::error::IdxError;
use crate::ownership::types::Entity;
pub fn fts_search(conn: &Connection, query: &str, limit: usize) -> Result<Vec<Entity>, IdxError> {
let q = query.trim();
if q.is_empty() {
return Ok(Vec::new());
}
let limit_i64 =
i64::try_from(limit).map_err(|e| IdxError::DatabaseError(format!("invalid limit: {e}")))?;
// Ensure entity_fts has content. For external-content FTS table, populate manually.
let fts_count: i64 = conn
.query_row("SELECT COUNT(*) FROM entity_fts", [], |row| row.get(0))
.unwrap_or(0);
if fts_count == 0 {
let _ = rebuild_fts(conn);
}
if let Ok(rows) = fts_query(conn, q, limit_i64)
&& !rows.is_empty()
{
return Ok(rows);
}
like_query(conn, q, limit_i64)
}
fn fts_query(conn: &Connection, query: &str, limit: i64) -> Result<Vec<Entity>, IdxError> {
let mut stmt = conn
.prepare(
"SELECT e.id, e.canonical_name, e.entity_type, e.country
FROM entity_fts f
JOIN entities e ON e.id = f.rowid
WHERE entity_fts MATCH ?1
ORDER BY rank
LIMIT ?2",
)
.map_err(|e| IdxError::DatabaseError(e.to_string()))?;
let rows = stmt
.query_map(params![query, limit], |row| {
Ok(Entity {
id: row.get(0)?,
canonical_name: row.get(1)?,
entity_type: row.get(2)?,
country: row.get(3)?,
})
})
.map_err(|e| IdxError::DatabaseError(e.to_string()))?;
let mut out = Vec::new();
for row in rows {
out.push(row.map_err(|e| IdxError::DatabaseError(e.to_string()))?);
}
Ok(out)
}
fn like_query(conn: &Connection, query: &str, limit: i64) -> Result<Vec<Entity>, IdxError> {
let mut stmt = conn
.prepare(
"SELECT id, canonical_name, entity_type, country
FROM entities
WHERE canonical_name LIKE ?1 COLLATE NOCASE
ORDER BY canonical_name ASC
LIMIT ?2",
)
.map_err(|e| IdxError::DatabaseError(e.to_string()))?;
let like = format!("%{query}%");
let rows = stmt
.query_map(params![like, limit], |row| {
Ok(Entity {
id: row.get(0)?,
canonical_name: row.get(1)?,
entity_type: row.get(2)?,
country: row.get(3)?,
})
})
.map_err(|e| IdxError::DatabaseError(e.to_string()))?;
let mut out = Vec::new();
for row in rows {
out.push(row.map_err(|e| IdxError::DatabaseError(e.to_string()))?);
}
Ok(out)
}
pub fn rebuild_fts(conn: &Connection) -> Result<(), IdxError> {
conn.execute("DELETE FROM entity_fts", [])
.map_err(|e| IdxError::DatabaseError(e.to_string()))?;
conn.execute(
"INSERT INTO entity_fts(rowid, canonical_name, aliases)
SELECT e.id, e.canonical_name, COALESCE(a.aliases, '')
FROM entities e
LEFT JOIN (
SELECT entity_id, GROUP_CONCAT(raw_name, ' ') AS aliases
FROM entity_aliases
GROUP BY entity_id
) a ON a.entity_id = e.id",
[],
)
.map_err(|e| IdxError::DatabaseError(e.to_string()))?;
Ok(())
}

451
src/ownership/types.rs Normal file
View file

@ -0,0 +1,451 @@
use chrono::NaiveDate;
use serde::{Deserialize, Serialize};
/// Which data source a holding originates from.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
#[serde(rename_all = "lowercase")]
pub enum OwnershipSource {
/// KSEI shareholder registry source.
Ksei,
/// Bing institutional ownership source.
Bing,
}
/// Investor locality classification from KSEI.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
pub enum Locality {
/// Local Indonesian investor.
Local,
/// Foreign investor.
Foreign,
}
/// Bing institutional flow signal.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
#[serde(rename_all = "snake_case")]
pub enum FlowSignal {
/// Existing top holder.
Holder,
/// Net buyer for the period.
Buyer,
/// Net seller for the period.
Seller,
/// New position opened this period.
NewPosition,
/// Fully exited this period.
Exited,
}
/// Method used to resolve an entity alias to a canonical entity.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
#[serde(rename_all = "lowercase")]
pub enum ResolutionMethod {
/// Exact string match after normalization.
Exact,
/// Rule-based normalization match.
Rule,
/// Fuzzy or similarity-based match.
Fuzzy,
/// Human-curated manual mapping.
Manual,
}
/// Graph node category used in ownership network output.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
#[serde(rename_all = "lowercase")]
pub enum GraphNodeType {
/// Canonical investor entity node.
Entity,
/// Listed issuer ticker node.
Ticker,
}
/// KSEI investor type code (for example: `CP`, `ID`, `IB`, `MF`, `SC`, `IS`, `OT`).
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
pub struct InvestorTypeCode(pub String);
/// Canonical resolved entity (investor or shareholder).
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct Entity {
/// Internal entity identifier.
pub id: i64,
/// Canonical normalized display name.
pub canonical_name: String,
/// Optional coarse entity type (`fund`, `bank`, `conglomerate`, `govt`, `individual`).
pub entity_type: Option<String>,
/// Optional ISO-like country tag.
pub country: Option<String>,
}
/// A raw name variant mapped to a canonical entity.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct EntityAlias {
/// Internal alias identifier.
pub id: i64,
/// Referenced canonical entity id.
pub entity_id: i64,
/// Raw alias text from source.
pub raw_name: String,
/// Source that produced this alias.
pub source: OwnershipSource,
/// Resolution confidence score in `0.0..=1.0`.
pub confidence: f64,
/// Matching method used for resolution.
pub method: ResolutionMethod,
}
/// Ticker (issuer) reference metadata.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct Ticker {
/// Internal ticker identifier.
pub id: i64,
/// Exchange ticker code, for example `BBCA`.
pub code: String,
/// Optional long issuer name.
pub name: Option<String>,
/// Optional sector classification.
pub sector: Option<String>,
}
/// Single KSEI ownership row for one holder in one ticker and release.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct KseiHolding {
/// Internal holding row identifier.
pub id: i64,
/// Referenced ticker id.
pub ticker_id: i64,
/// Referenced entity id, unresolved when `None`.
pub entity_id: Option<i64>,
/// Raw investor name exactly as imported.
pub raw_investor_name: String,
/// Optional KSEI investor type code.
pub investor_type: Option<InvestorTypeCode>,
/// Optional local/foreign classification.
pub locality: Option<Locality>,
/// Optional nationality text.
pub nationality: Option<String>,
/// Optional domicile text.
pub domicile: Option<String>,
/// Scripless holdings in shares.
pub holdings_scripless: i64,
/// Script holdings in shares.
pub holdings_scrip: i64,
/// Total shares held.
pub total_shares: i64,
/// Ownership percentage in basis points (`41.10% -> 4110`).
pub percentage_bps: i64,
/// Snapshot as-of date.
pub report_date: NaiveDate,
/// SHA-256 hash of source release for deduplication.
pub release_sha256: String,
}
/// Draft holding before entity resolution (without persisted row id and entity id).
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct KseiHoldingDraft {
/// Exchange ticker code, for example `BBCA`.
pub ticker_code: String,
/// Optional issuer name as provided by source row.
pub issuer_name: Option<String>,
/// Raw investor name exactly as imported.
pub raw_investor_name: String,
/// Optional KSEI investor type code.
pub investor_type: Option<InvestorTypeCode>,
/// Optional local/foreign classification.
pub locality: Option<Locality>,
/// Optional nationality text.
pub nationality: Option<String>,
/// Optional domicile text.
pub domicile: Option<String>,
/// Scripless holdings in shares.
pub holdings_scripless: i64,
/// Script holdings in shares.
pub holdings_scrip: i64,
/// Total shares held.
pub total_shares: i64,
/// Ownership percentage in basis points (`41.10% -> 4110`).
pub percentage_bps: i64,
/// Snapshot as-of date.
pub report_date: NaiveDate,
}
/// Single Bing institutional ownership row.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct BingHolding {
/// Internal holding row identifier.
pub id: i64,
/// Referenced ticker id.
pub ticker_id: i64,
/// Referenced entity id, unresolved when `None`.
pub entity_id: Option<i64>,
/// Raw investor name exactly as imported.
pub raw_investor_name: String,
/// Optional Bing investor type text.
pub investor_type: Option<String>,
/// Shares currently held.
pub shares_held: Option<i64>,
/// Share delta over period (`+buy`, `-sell`).
pub shares_changed: Option<i64>,
/// Ownership percentage in basis points.
pub pct_ownership_bps: Option<i64>,
/// Position value in USD.
pub value_usd: Option<i64>,
/// Source report date.
pub report_date: NaiveDate,
/// Institutional flow signal for the row.
pub signal: FlowSignal,
/// Import timestamp (unix epoch seconds).
pub fetched_at: i64,
}
/// Raw row extracted from KSEI PDF before normalization.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct KseiRawRow {
/// Date string as appears in PDF (for example `27-Feb-2026`).
pub date: String,
/// Share code string (for example `BBCA`).
pub share_code: String,
/// Issuer name as appears in source.
pub issuer_name: String,
/// Investor name as appears in source.
pub investor_name: String,
/// Investor type code string.
pub investor_type: String,
/// Local or foreign marker (`L` or `F`).
pub local_foreign: String,
/// Nationality text.
pub nationality: String,
/// Domicile text.
pub domicile: String,
/// Scripless holdings string in Indonesian locale format.
pub holdings_scripless: String,
/// Script holdings string in Indonesian locale format.
pub holdings_scrip: String,
/// Total holdings string in Indonesian locale format.
pub total_holding_shares: String,
/// Percentage string in Indonesian locale decimal format.
pub percentage: String,
}
/// Raw holder object returned by Bing ownership endpoints.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct BingHolderRaw {
/// Investor name field from Bing payload.
#[serde(alias = "investorName", alias = "InvestorName")]
pub investor_name: Option<String>,
/// Investor type field from Bing payload.
#[serde(alias = "investorType", alias = "InvestorType")]
pub investor_type: Option<String>,
/// Shares held field from Bing payload.
#[serde(alias = "sharesHeld", alias = "SharesHeld")]
pub shares_held: Option<f64>,
/// Shares changed field from Bing payload.
#[serde(alias = "sharesChanged", alias = "SharesChanged")]
pub shares_changed: Option<f64>,
/// Percentage of shares outstanding field from Bing payload.
#[serde(
alias = "percentageOfSharesOutstanding",
alias = "PercentageOfSharesOutstanding"
)]
pub pct_outstanding: Option<f64>,
/// Position value field from Bing payload.
#[serde(alias = "value", alias = "Value")]
pub value: Option<f64>,
/// Report date field from Bing payload.
#[serde(alias = "reportDate", alias = "ReportDate")]
pub report_date: Option<String>,
}
/// Combined ownership view for a ticker (KSEI and Bing merged).
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct TickerOwnership {
/// Ticker metadata.
pub ticker: Ticker,
/// Latest KSEI as-of date.
pub ksei_as_of: Option<NaiveDate>,
/// Latest Bing reporting period label.
pub bing_as_of: Option<String>,
/// Combined holder rows for display.
pub holders: Vec<HolderRow>,
/// Concentration metrics calculated for the ticker.
pub concentration: ConcentrationMetrics,
/// Optional institutional flow breakdown.
pub flow: Option<InstitutionalFlow>,
}
/// Single row in a combined holders table.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct HolderRow {
/// Ranking position in result set.
pub rank: usize,
/// Source dataset for the row.
pub source: OwnershipSource,
/// Canonical holder name or unresolved raw name.
pub name: String,
/// Optional referenced canonical entity id.
pub entity_id: Option<i64>,
/// Optional investor type label.
pub investor_type: Option<String>,
/// Optional local/foreign marker.
pub locality: Option<Locality>,
/// Held shares quantity.
pub shares: i64,
/// Ownership percentage in basis points.
pub percentage_bps: i64,
/// Optional flow signal (Bing rows only).
pub signal: Option<FlowSignal>,
}
/// Concentration metrics for ownership distribution of a ticker.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct ConcentrationMetrics {
/// Largest single holder percentage in basis points.
pub top1_bps: i64,
/// Sum of top 3 holder percentages in basis points.
pub top3_bps: i64,
/// Herfindahl-Hirschman index value.
pub hhi: i64,
/// Estimated free float in basis points (`10000 - known holders`).
pub free_float_bps: i64,
/// Number of KSEI holders above or equal to 1%.
pub holder_count: usize,
}
/// Institutional flow summary from Bing for one ticker and period.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct InstitutionalFlow {
/// Reporting period label.
pub period: String,
/// Top buyer rows.
pub top_buyers: Vec<HolderRow>,
/// Top seller rows.
pub top_sellers: Vec<HolderRow>,
/// New position rows.
pub new_positions: Vec<HolderRow>,
/// Exited position rows.
pub exited: Vec<HolderRow>,
}
/// Cross-holding summary for a single entity.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct EntityHoldings {
/// Canonical entity metadata.
pub entity: Entity,
/// Number of distinct tickers held.
pub ticker_count: usize,
/// Per-ticker ownership rows.
pub holdings: Vec<EntityTickerRow>,
}
/// One ticker row within an entity holdings summary.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct EntityTickerRow {
/// Ticker metadata.
pub ticker: Ticker,
/// Source dataset for the holding.
pub source: OwnershipSource,
/// Shares held quantity.
pub shares: i64,
/// Ownership percentage in basis points.
pub percentage_bps: i64,
/// Report as-of date.
pub report_date: NaiveDate,
}
/// Cross-holder leaderboard row across multiple tickers.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct CrossHolderRow {
/// Canonical entity metadata.
pub entity: Entity,
/// Number of distinct tickers held.
pub ticker_count: usize,
/// Summed ownership basis points across tickers.
pub total_bps: i64,
}
/// Graph node used for ownership network visualization.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct GraphNode {
/// Stable graph node id (`entity:42` or `ticker:BBCA`).
pub id: String,
/// Display label.
pub label: String,
/// Node category.
pub node_type: GraphNodeType,
}
/// Graph edge used for ownership network visualization.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct GraphEdge {
/// Source node id.
pub from: String,
/// Target node id.
pub to: String,
/// Edge weight in basis points.
pub percentage_bps: i64,
/// Source dataset of the edge.
pub source: OwnershipSource,
}
/// Imported ownership release metadata.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct OwnershipRelease {
/// Internal release id.
pub id: i64,
/// Optional source URL where release was obtained.
pub source_url: Option<String>,
/// SHA-256 hash for release-level deduplication.
pub sha256: String,
/// Release as-of date.
pub as_of_date: NaiveDate,
/// Parsed row count imported.
pub row_count: usize,
/// Import timestamp (unix epoch seconds).
pub imported_at: i64,
}
/// Type of ownership change between two snapshots.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
#[serde(rename_all = "lowercase")]
pub enum ChangeType {
/// Holder appears in `to` snapshot but not in `from`.
New,
/// Holder appears in `from` snapshot but not in `to`.
Exited,
/// Holder exists in both snapshots and percentage increased.
Increased,
/// Holder exists in both snapshots and percentage decreased.
Decreased,
}
/// One ownership change row for a ticker-holder pair.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct ChangeRow {
/// Exchange ticker code.
pub ticker_code: String,
/// Canonical or raw holder name.
pub entity_name: String,
/// Change classification.
pub change_type: ChangeType,
/// Old percentage in basis points (from snapshot).
pub old_bps: Option<i64>,
/// New percentage in basis points (to snapshot).
pub new_bps: Option<i64>,
/// Delta in basis points (`new - old`, missing treated as 0).
pub delta_bps: i64,
}
/// Row describing unresolved or low-confidence alias mappings.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct UnresolvedRow {
/// Raw investor name captured from source data.
pub raw_name: String,
/// Source marker (`ksei` or `bing`).
pub source: String,
/// Ticker code where the unresolved alias appears.
pub ticker_code: String,
/// Current canonical entity name when available.
pub current_entity: Option<String>,
/// Resolution confidence score.
pub confidence: Option<f64>,
}

47
tests/fixtures/bing_buyers_bbca.json vendored Normal file
View file

@ -0,0 +1,47 @@
[
{
"investorName": "Norges Bank Investment Management",
"investorType": "Government / Sovereign Wealth",
"sharesHeld": 278330000.0,
"sharesChanged": 3200000.0,
"percentageOfSharesOutstanding": 1.13,
"value": 275196270000.0,
"reportDate": "2024-12-31"
},
{
"investorName": "Dimensional Fund Advisors LP",
"investorType": "Institutional",
"sharesHeld": 156920000.0,
"sharesChanged": 14750000.0,
"percentageOfSharesOutstanding": 0.64,
"value": 155110680000.0,
"reportDate": "2024-12-31"
},
{
"investorName": "Baillie Gifford & Co",
"investorType": "Institutional",
"sharesHeld": 132450000.0,
"sharesChanged": 11200000.0,
"percentageOfSharesOutstanding": 0.54,
"value": 130965450000.0,
"reportDate": "2024-12-31"
},
{
"investorName": "BlackRock Inc",
"investorType": "Institutional",
"sharesHeld": 498112750.0,
"sharesChanged": 5600000.0,
"percentageOfSharesOutstanding": 2.02,
"value": 492523521250.0,
"reportDate": "2024-12-31"
},
{
"investorName": "Fidelity Management & Research",
"investorType": "Institutional",
"sharesHeld": 89340000.0,
"sharesChanged": 4100000.0,
"percentageOfSharesOutstanding": 0.36,
"value": 88346060000.0,
"reportDate": "2024-12-31"
}
]

56
tests/fixtures/bing_holders_bbca.json vendored Normal file
View file

@ -0,0 +1,56 @@
[
{
"investorName": "PT Dwimuria Investama Andalan",
"investorType": "Corporate",
"sharesHeld": 13533682440.0,
"sharesChanged": 0.0,
"percentageOfSharesOutstanding": 54.94,
"value": 13380524934600.0,
"reportDate": "2024-12-31"
},
{
"investorName": "Vanguard Group Inc",
"investorType": "Institutional",
"sharesHeld": 621438200.0,
"sharesChanged": -12000000.0,
"percentageOfSharesOutstanding": 2.52,
"value": 614321778000.0,
"reportDate": "2024-12-31"
},
{
"investorName": "BlackRock Inc",
"investorType": "Institutional",
"sharesHeld": 498112750.0,
"sharesChanged": 5600000.0,
"percentageOfSharesOutstanding": 2.02,
"value": 492523521250.0,
"reportDate": "2024-12-31"
},
{
"investorName": "Government Pension Investment Fund (Japan)",
"investorType": "Government / Sovereign Wealth",
"sharesHeld": 312045000.0,
"sharesChanged": 0.0,
"percentageOfSharesOutstanding": 1.27,
"value": 308564505000.0,
"reportDate": "2024-12-31"
},
{
"investorName": "Norges Bank Investment Management",
"investorType": "Government / Sovereign Wealth",
"sharesHeld": 278330000.0,
"sharesChanged": 3200000.0,
"percentageOfSharesOutstanding": 1.13,
"value": 275196270000.0,
"reportDate": "2024-12-31"
},
{
"investorName": "Capital Research and Management",
"investorType": "Institutional",
"sharesHeld": 248760000.0,
"sharesChanged": -8000000.0,
"percentageOfSharesOutstanding": 1.01,
"value": 245971800000.0,
"reportDate": "2024-12-31"
}
]

328
tests/fixtures/ksei_stext_sample.xml vendored Normal file
View file

@ -0,0 +1,328 @@
<document>
<page id="page1" width="792" height="612">
<char x="40.0" y="40.0" c="P"/>
<char x="41.2" y="40.0" c="E"/>
<char x="42.4" y="40.0" c="N"/>
<char x="43.6" y="40.0" c="G"/>
</page>
<page id="page2" width="792" height="612">
<char x="21.7" y="100.0" c="2"/>
<char x="22.8" y="100.0" c="7"/>
<char x="23.9" y="100.0" c="-"/>
<char x="25.0" y="100.0" c="F"/>
<char x="26.1" y="100.0" c="e"/>
<char x="27.2" y="100.0" c="b"/>
<char x="28.3" y="100.0" c="-"/>
<char x="29.4" y="100.0" c="2"/>
<char x="30.5" y="100.0" c="0"/>
<char x="31.6" y="100.0" c="2"/>
<char x="32.7" y="100.0" c="6"/>
<char x="55.3" y="100.0" c="B"/>
<char x="56.4" y="100.0" c="B"/>
<char x="57.5" y="100.0" c="C"/>
<char x="58.6" y="100.0" c="A"/>
<char x="75.0" y="100.0" c="B"/>
<char x="76.1" y="100.0" c="A"/>
<char x="77.2" y="100.0" c="N"/>
<char x="78.3" y="100.0" c="K"/>
<char x="79.4" y="100.0" c=" "/>
<char x="80.8" y="100.0" c="C"/>
<char x="81.9" y="100.0" c="E"/>
<char x="83.0" y="100.0" c="N"/>
<char x="84.1" y="100.0" c="T"/>
<char x="85.2" y="100.0" c="R"/>
<char x="86.3" y="100.0" c="A"/>
<char x="87.4" y="100.0" c="L"/>
<char x="88.5" y="100.0" c=" "/>
<char x="89.9" y="100.0" c="A"/>
<char x="91.0" y="100.0" c="S"/>
<char x="92.1" y="100.0" c="I"/>
<char x="93.2" y="100.0" c="A"/>
<char x="94.3" y="100.0" c=" "/>
<char x="95.7" y="100.0" c="T"/>
<char x="96.8" y="100.0" c="b"/>
<char x="97.9" y="100.0" c="k"/>
<char x="167.9" y="100.0" c="P"/>
<char x="169.0" y="100.0" c="T"/>
<char x="170.1" y="100.0" c=" "/>
<char x="171.5" y="100.0" c="D"/>
<char x="172.6" y="100.0" c="W"/>
<char x="173.7" y="100.0" c="I"/>
<char x="174.8" y="100.0" c="M"/>
<char x="175.9" y="100.0" c="U"/>
<char x="177.0" y="100.0" c="R"/>
<char x="178.1" y="100.0" c="I"/>
<char x="179.2" y="100.0" c="A"/>
<char x="180.3" y="100.0" c=" "/>
<char x="181.7" y="100.0" c="I"/>
<char x="182.8" y="100.0" c="N"/>
<char x="183.9" y="100.0" c="V"/>
<char x="185.0" y="100.0" c="E"/>
<char x="186.1" y="100.0" c="S"/>
<char x="187.2" y="100.0" c="T"/>
<char x="188.3" y="100.0" c="A"/>
<char x="189.4" y="100.0" c="M"/>
<char x="190.5" y="100.0" c="A"/>
<char x="191.6" y="100.0" c=" "/>
<char x="193.0" y="100.0" c="A"/>
<char x="194.1" y="100.0" c="N"/>
<char x="195.2" y="100.0" c="D"/>
<char x="196.3" y="100.0" c="A"/>
<char x="197.4" y="100.0" c="L"/>
<char x="198.5" y="100.0" c="A"/>
<char x="199.6" y="100.0" c="N"/>
<char x="435.0" y="100.0" c="C"/>
<char x="436.1" y="100.0" c="P"/>
<char x="468.0" y="100.0" c="L"/>
<char x="500.0" y="100.0" c="I"/>
<char x="501.1" y="100.0" c="D"/>
<char x="560.0" y="100.0" c="I"/>
<char x="561.1" y="100.0" c="D"/>
<char x="620.0" y="100.0" c="1"/>
<char x="621.1" y="100.0" c="3"/>
<char x="622.2" y="100.0" c="."/>
<char x="623.3" y="100.0" c="5"/>
<char x="624.4" y="100.0" c="3"/>
<char x="625.5" y="100.0" c="3"/>
<char x="626.6" y="100.0" c="."/>
<char x="627.7" y="100.0" c="6"/>
<char x="628.8" y="100.0" c="8"/>
<char x="629.9" y="100.0" c="2"/>
<char x="631.0" y="100.0" c="."/>
<char x="632.1" y="100.0" c="4"/>
<char x="633.2" y="100.0" c="4"/>
<char x="634.3" y="100.0" c="0"/>
<char x="656.0" y="100.0" c="0"/>
<char x="696.0" y="100.0" c="1"/>
<char x="697.1" y="100.0" c="3"/>
<char x="698.2" y="100.0" c="."/>
<char x="699.3" y="100.0" c="5"/>
<char x="700.4" y="100.0" c="3"/>
<char x="701.5" y="100.0" c="3"/>
<char x="702.6" y="100.0" c="."/>
<char x="703.7" y="100.0" c="6"/>
<char x="704.8" y="100.0" c="8"/>
<char x="705.9" y="100.0" c="2"/>
<char x="707.0" y="100.0" c="."/>
<char x="708.1" y="100.0" c="4"/>
<char x="709.2" y="100.0" c="4"/>
<char x="710.3" y="100.0" c="0"/>
<char x="748.0" y="100.0" c="5"/>
<char x="749.1" y="100.0" c="4"/>
<char x="750.2" y="100.0" c=","/>
<char x="751.3" y="100.0" c="9"/>
<char x="752.4" y="100.0" c="4"/>
<char x="21.7" y="101.9" c="2"/>
<char x="22.8" y="101.9" c="7"/>
<char x="23.9" y="101.9" c="-"/>
<char x="25.0" y="101.9" c="F"/>
<char x="26.1" y="101.9" c="e"/>
<char x="27.2" y="101.9" c="b"/>
<char x="28.3" y="101.9" c="-"/>
<char x="29.4" y="101.9" c="2"/>
<char x="30.5" y="101.9" c="0"/>
<char x="31.6" y="101.9" c="2"/>
<char x="32.7" y="101.9" c="6"/>
<char x="55.3" y="101.9" c="B"/>
<char x="56.4" y="101.9" c="B"/>
<char x="57.5" y="101.9" c="C"/>
<char x="58.6" y="101.9" c="A"/>
<char x="75.0" y="101.9" c="B"/>
<char x="76.1" y="101.9" c="A"/>
<char x="77.2" y="101.9" c="N"/>
<char x="78.3" y="101.9" c="K"/>
<char x="79.4" y="101.9" c=" "/>
<char x="80.8" y="101.9" c="C"/>
<char x="81.9" y="101.9" c="E"/>
<char x="83.0" y="101.9" c="N"/>
<char x="84.1" y="101.9" c="T"/>
<char x="85.2" y="101.9" c="R"/>
<char x="86.3" y="101.9" c="A"/>
<char x="87.4" y="101.9" c="L"/>
<char x="88.5" y="101.9" c=" "/>
<char x="89.9" y="101.9" c="A"/>
<char x="91.0" y="101.9" c="S"/>
<char x="92.1" y="101.9" c="I"/>
<char x="93.2" y="101.9" c="A"/>
<char x="94.3" y="101.9" c=" "/>
<char x="95.7" y="101.9" c="T"/>
<char x="96.8" y="101.9" c="b"/>
<char x="97.9" y="101.9" c="k"/>
<char x="167.9" y="101.9" c="P"/>
<char x="169.0" y="101.9" c="T"/>
<char x="170.1" y="101.9" c=" "/>
<char x="171.5" y="101.9" c="G"/>
<char x="172.6" y="101.9" c="I"/>
<char x="173.7" y="101.9" c="T"/>
<char x="174.8" y="101.9" c="A"/>
<char x="175.9" y="101.9" c=" "/>
<char x="177.3" y="101.9" c="S"/>
<char x="178.4" y="101.9" c="U"/>
<char x="179.5" y="101.9" c="R"/>
<char x="180.6" y="101.9" c="Y"/>
<char x="181.7" y="101.9" c="A"/>
<char x="182.8" y="101.9" c=" "/>
<char x="184.2" y="101.9" c="H"/>
<char x="185.3" y="101.9" c="A"/>
<char x="186.4" y="101.9" c="S"/>
<char x="187.5" y="101.9" c="T"/>
<char x="188.6" y="101.9" c="A"/>
<char x="435.0" y="101.9" c="C"/>
<char x="436.1" y="101.9" c="P"/>
<char x="468.0" y="101.9" c="L"/>
<char x="500.0" y="101.9" c="I"/>
<char x="501.1" y="101.9" c="D"/>
<char x="560.0" y="101.9" c="I"/>
<char x="561.1" y="101.9" c="D"/>
<char x="620.0" y="101.9" c="5"/>
<char x="621.1" y="101.9" c="."/>
<char x="622.2" y="101.9" c="2"/>
<char x="623.3" y="101.9" c="0"/>
<char x="624.4" y="101.9" c="0"/>
<char x="625.5" y="101.9" c="."/>
<char x="626.6" y="101.9" c="0"/>
<char x="627.7" y="101.9" c="0"/>
<char x="628.8" y="101.9" c="0"/>
<char x="629.9" y="101.9" c="."/>
<char x="631.0" y="101.9" c="0"/>
<char x="632.1" y="101.9" c="0"/>
<char x="633.2" y="101.9" c="0"/>
<char x="656.0" y="101.9" c="0"/>
<char x="696.0" y="101.9" c="5"/>
<char x="697.1" y="101.9" c="."/>
<char x="698.2" y="101.9" c="2"/>
<char x="699.3" y="101.9" c="0"/>
<char x="700.4" y="101.9" c="0"/>
<char x="701.5" y="101.9" c="."/>
<char x="702.6" y="101.9" c="0"/>
<char x="703.7" y="101.9" c="0"/>
<char x="704.8" y="101.9" c="0"/>
<char x="705.9" y="101.9" c="."/>
<char x="707.0" y="101.9" c="0"/>
<char x="708.1" y="101.9" c="0"/>
<char x="709.2" y="101.9" c="0"/>
<char x="748.0" y="101.9" c="2"/>
<char x="749.1" y="101.9" c="1"/>
<char x="750.2" y="101.9" c=","/>
<char x="751.3" y="101.9" c="1"/>
<char x="752.4" y="101.9" c="1"/>
</page>
<page id="page3" width="792" height="612">
<char x="21.7" y="120.0" c="2"/>
<char x="22.8" y="120.0" c="7"/>
<char x="23.9" y="120.0" c="-"/>
<char x="25.0" y="120.0" c="F"/>
<char x="26.1" y="120.0" c="e"/>
<char x="27.2" y="120.0" c="b"/>
<char x="28.3" y="120.0" c="-"/>
<char x="29.4" y="120.0" c="2"/>
<char x="30.5" y="120.0" c="0"/>
<char x="31.6" y="120.0" c="2"/>
<char x="32.7" y="120.0" c="6"/>
<char x="55.3" y="120.0" c="B"/>
<char x="56.4" y="120.0" c="B"/>
<char x="57.5" y="120.0" c="R"/>
<char x="58.6" y="120.0" c="I"/>
<char x="75.0" y="120.0" c="B"/>
<char x="76.1" y="120.0" c="A"/>
<char x="77.2" y="120.0" c="N"/>
<char x="78.3" y="120.0" c="K"/>
<char x="79.4" y="120.0" c=" "/>
<char x="80.8" y="120.0" c="R"/>
<char x="81.9" y="120.0" c="A"/>
<char x="83.0" y="120.0" c="K"/>
<char x="84.1" y="120.0" c="Y"/>
<char x="85.2" y="120.0" c="A"/>
<char x="86.3" y="120.0" c="T"/>
<char x="87.4" y="120.0" c=" "/>
<char x="88.8" y="120.0" c="I"/>
<char x="89.9" y="120.0" c="N"/>
<char x="91.0" y="120.0" c="D"/>
<char x="92.1" y="120.0" c="O"/>
<char x="93.2" y="120.0" c="N"/>
<char x="94.3" y="120.0" c="E"/>
<char x="95.4" y="120.0" c="S"/>
<char x="96.5" y="120.0" c="I"/>
<char x="97.6" y="120.0" c="A"/>
<char x="98.7" y="120.0" c=" "/>
<char x="100.1" y="120.0" c="("/>
<char x="101.2" y="120.0" c="P"/>
<char x="102.3" y="120.0" c="E"/>
<char x="103.4" y="120.0" c="R"/>
<char x="104.5" y="120.0" c="S"/>
<char x="105.6" y="120.0" c="E"/>
<char x="106.7" y="120.0" c="R"/>
<char x="107.8" y="120.0" c="O"/>
<char x="108.9" y="120.0" c=")"/>
<char x="110.0" y="120.0" c=" "/>
<char x="111.4" y="120.0" c="T"/>
<char x="112.5" y="120.0" c="b"/>
<char x="113.6" y="120.0" c="k"/>
<char x="167.9" y="120.0" c="N"/>
<char x="169.0" y="120.0" c="E"/>
<char x="170.1" y="120.0" c="G"/>
<char x="171.2" y="120.0" c="A"/>
<char x="172.3" y="120.0" c="R"/>
<char x="173.4" y="120.0" c="A"/>
<char x="174.5" y="120.0" c=" "/>
<char x="175.9" y="120.0" c="R"/>
<char x="177.0" y="120.0" c="E"/>
<char x="178.1" y="120.0" c="P"/>
<char x="179.2" y="120.0" c="U"/>
<char x="180.3" y="120.0" c="B"/>
<char x="181.4" y="120.0" c="L"/>
<char x="182.5" y="120.0" c="I"/>
<char x="183.6" y="120.0" c="K"/>
<char x="184.7" y="120.0" c=" "/>
<char x="186.1" y="120.0" c="I"/>
<char x="187.2" y="120.0" c="N"/>
<char x="188.3" y="120.0" c="D"/>
<char x="189.4" y="120.0" c="O"/>
<char x="190.5" y="120.0" c="N"/>
<char x="191.6" y="120.0" c="E"/>
<char x="192.7" y="120.0" c="S"/>
<char x="193.8" y="120.0" c="I"/>
<char x="194.9" y="120.0" c="A"/>
<char x="435.0" y="120.0" c="C"/>
<char x="436.1" y="120.0" c="P"/>
<char x="468.0" y="120.0" c="L"/>
<char x="500.0" y="120.0" c="I"/>
<char x="501.1" y="120.0" c="D"/>
<char x="560.0" y="120.0" c="I"/>
<char x="561.1" y="120.0" c="D"/>
<char x="620.0" y="120.0" c="6"/>
<char x="621.1" y="120.0" c="7"/>
<char x="622.2" y="120.0" c="."/>
<char x="623.3" y="120.0" c="0"/>
<char x="624.4" y="120.0" c="0"/>
<char x="625.5" y="120.0" c="0"/>
<char x="626.6" y="120.0" c="."/>
<char x="627.7" y="120.0" c="0"/>
<char x="628.8" y="120.0" c="0"/>
<char x="629.9" y="120.0" c="0"/>
<char x="631.0" y="120.0" c="."/>
<char x="632.1" y="120.0" c="0"/>
<char x="633.2" y="120.0" c="0"/>
<char x="634.3" y="120.0" c="0"/>
<char x="656.0" y="120.0" c="0"/>
<char x="696.0" y="120.0" c="6"/>
<char x="697.1" y="120.0" c="7"/>
<char x="698.2" y="120.0" c="."/>
<char x="699.3" y="120.0" c="0"/>
<char x="700.4" y="120.0" c="0"/>
<char x="701.5" y="120.0" c="0"/>
<char x="702.6" y="120.0" c="."/>
<char x="703.7" y="120.0" c="0"/>
<char x="704.8" y="120.0" c="0"/>
<char x="705.9" y="120.0" c="0"/>
<char x="707.0" y="120.0" c="."/>
<char x="708.1" y="120.0" c="0"/>
<char x="709.2" y="120.0" c="0"/>
<char x="710.3" y="120.0" c="0"/>
<char x="748.0" y="120.0" c="5"/>
<char x="749.1" y="120.0" c="3"/>
<char x="750.2" y="120.0" c=","/>
<char x="751.3" y="120.0" c="1"/>
<char x="752.4" y="120.0" c="9"/>
</page>
</document>