feat(ownership): complete ownership intelligence module

Sprints 0-8: Types, schema, KSEI parser, entity resolution, DB CRUD,
Bing API client, import pipeline, query commands, graph traversal,
changes diff, entity resolution CLI, FTS5 search.

- KSEI PDF parser (mutool stext + quick-xml, 99.2% accuracy)
- 7 query commands: ticker, entity, search, cross-holders, concentration, flow, releases
- Ownership graph with recursive CTE (ASCII tree + Graphviz DOT)
- Release diff/changes between KSEI snapshots
- Entity resolution CLI: unresolved, map, merge
- FTS5 trigram search on entity names
- Feature-gated under 'ownership' (default-on)
- 82 tests passing
This commit is contained in:
Ciphercat 2026-03-07 17:31:04 +00:00
commit 6671e22976
23 changed files with 5351 additions and 47 deletions

94
docs/ARCHITECTURE.md Normal file
View file

@ -0,0 +1,94 @@
# Architecture
## Domain Map
```
┌─────────────────────────────────────────────────────────┐
│ CLI Layer │
│ main.rs → cli/stocks.rs, cli/ownership.rs, cli/... │
│ Clap derive structs, command dispatch, arg validation │
└──────────────────────┬──────────────────────────────────┘
┌──────────────┼──────────────┐
│ │ │
┌───────▼───────┐ ┌────▼────┐ ┌──────▼──────┐
│ API Layer │ │ Analysis│ │ Ownership │
│ (providers) │ │ Module │ │ Module │
│ │ │ │ │ │
│ MarketData │ │ techni- │ │ SQLite DB │
│ Provider trait│ │ cal.rs │ │ KSEI parser │
│ │ │ signals │ │ Bing client │
│ Yahoo (OHLCV) │ │ fund.rs │ │ entity res. │
│ MSN (rich) │ │ │ │ FTS5 search │
└───────┬───────┘ └────┬────┘ └──────┬──────┘
│ │ │
┌───────▼──────────────▼──────────────▼──────┐
│ Output Layer │
│ table.rs (comfy-table) │ json.rs (serde) │
└─────────────────────────────────────────────┘
```
## Provider Architecture
### Dual Provider Model
- **MSN Finance** — default provider. Rich data: quotes, fundamentals, profile, earnings, financials, sentiment, insights, news, screener. No history for IDX stocks.
- **Yahoo Finance** — history fallback. Reliable OHLCV data via `/v8/finance/chart/`.
### Hybrid History Strategy
When `history_provider = auto` (default):
1. Check if current provider supports `HistoryProvider` trait
2. MSN doesn't → transparently fallback to Yahoo
3. Log info message: `"history provider fallback active (msn -> yahoo)"`
### Capability Gating
```
MarketDataProvider = QuoteProvider + FundamentalsProvider
HistoryProvider = separate trait, not all providers implement
Factory functions:
default_provider(kind) → Box<dyn MarketDataProvider>
history_provider(kind, mode, verbose) → Result<(ProviderKind, Box<dyn HistoryProvider>)>
```
## Ownership Module (SQLite-backed)
Unlike the `stocks` module (live-fetch), ownership is **import-then-query**:
1. `idx ownership import` — ETL pipeline: fetch PDF/API → parse → normalize → load SQLite
2. All query commands read from local `~/.local/share/idx/ownership.db`
3. Fully offline after import
### Data Sources
- **KSEI** — official ≥1% shareholder registry (monthly PDF from IDX)
- **Bing Finance** — global institutional ownership (REST API, quarterly)
### Parser Pipeline
```
KSEI PDF → mutool stext (XML with coordinates) → quick-xml parse → KseiRawRow
→ normalize (ID locale numbers, dates, entity names) → KseiHolding
→ SQLite INSERT (within transaction)
```
## Data Flow Patterns
### Live Query (stocks module)
```
CLI args → resolve symbol → provider.quote/history/fundamentals → render table/json
```
### Import-Query (ownership module)
```
Import: PDF/API → parse → normalize → resolve entities → SQLite INSERT
Query: CLI args → SQLite SELECT → render table/json (no network)
```
## Configuration Precedence
```
CLI flags > environment variables > config file > defaults
```
## Error Strategy
- `IdxError` enum (thiserror) with structured error codes
- Table mode: human-readable error on stderr
- JSON mode: `{"error": true, "code": "...", "message": "..."}`
- Exit code 0 on success, non-zero on failure

127
docs/CONVENTIONS.md Normal file
View file

@ -0,0 +1,127 @@
# Conventions
## Design Philosophy
### Schema-Driven Development
Define data types FIRST, then build logic around them. Types are the spec.
```rust
// ✅ Good: rich type with documented fields, integer precision
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct KseiHolding {
/// Ownership percentage in basis points: 41.10% → 4110.
pub percentage_bps: i64,
/// Total shares held (absolute count).
pub total_shares: i64,
}
// ❌ Bad: stringly-typed, float precision
pub struct Holding {
pub percentage: f64, // what unit? what precision?
pub shares: String, // why string?
}
```
### Functional Approach
Parse/transform functions are **pure**: take input, return `Result<T, IdxError>`, no side effects.
```rust
// ✅ Good: pure function, testable in isolation
pub fn parse_id_number(s: &str) -> Result<i64, IdxError> { ... }
pub fn normalize_name(raw: &str) -> String { ... }
pub fn parse_quote_from_str(symbol: &str, raw: &str) -> Result<Quote, IdxError> { ... }
// ❌ Bad: function with side effects, hard to test
pub fn fetch_and_save_quote(symbol: &str) -> Result<(), IdxError> { ... }
```
### Types Over Primitives
Use newtypes, enums, and rich structs. Avoid `String` where a domain type exists.
```rust
// ✅ Good
pub struct InvestorTypeCode(pub String);
pub enum Locality { Local, Foreign }
pub enum FlowSignal { Holder, Buyer, Seller, NewPosition, Exited }
// ❌ Bad
pub type InvestorType = String;
pub type Locality = String;
```
## Naming
### Files
- `types.rs` — domain data types for a module
- `mod.rs` — module declarations and re-exports
- `client.rs` — HTTP client code
- `map.rs` / `parse.rs` — response mapping / parsing functions
- `raw_types.rs` — raw API response shapes (before normalization)
### Functions
- `parse_*` — deserialize raw data into domain types
- `normalize_*` — clean/transform data (names, numbers, dates)
- `resolve_*` — lookup/match entities
- `query_*` — read from database
- `fetch_*` — HTTP requests to external APIs
- `render_*` — output formatting (tables, JSON)
- `handle` — CLI command dispatch entry point
### Types
- `*Raw` / `*RawRow` — pre-normalization data (strings from API/PDF)
- `*Holding` — ownership fact row
- `*Metrics` — computed analytics
- `*Row` — display-ready composite type
- `*Args` — clap command arguments
## Patterns
### Provider Trait Pattern
```rust
pub trait QuoteProvider {
fn quote(&self, symbol: &str) -> Result<Quote, IdxError>;
}
// Factory function, not direct construction
pub fn default_provider(kind: ProviderKind) -> Box<dyn MarketDataProvider> { ... }
```
### Parse Pipeline Pattern
```rust
// Raw API response → domain type, always via parse function
let raw: &str = &response_body;
let quote = yahoo::parse_quote_from_str("BBCA.JK", raw)?;
```
### DB Function Pattern
```rust
// Take &Connection, caller manages lifetime. Use transactions for bulk.
pub fn insert_ksei_holdings(conn: &Connection, holdings: &[KseiHolding]) -> Result<usize, IdxError> { ... }
pub fn query_ticker_holdings(conn: &Connection, code: &str) -> Result<TickerOwnership, IdxError> { ... }
```
### Error Propagation
```rust
// Map external errors to IdxError variants
let conn = Connection::open(path)
.map_err(|e| IdxError::DatabaseError(e.to_string()))?;
```
## Integer Precision
- **Percentages:** basis points (i64). `41.10%``4110`
- **Shares:** absolute count (i64). No floats.
- **Prices:** whole IDR (i64). Rounded from float at parse boundary.
- **Money (USD):** whole dollars (i64) for Bing data.
## Testing
- **Unit tests:** pure functions (parsers, normalizers, signals)
- **Integration tests:** in-memory SQLite (`Connection::open_in_memory()`), mock providers
- **Fixtures:** `tests/fixtures/*.json` — real API responses, sanitized
- **No live API calls in CI**`IDX_USE_MOCK_PROVIDER=1`
- **Test naming:** `test_<function>_<scenario>` (e.g., `test_parse_id_number_with_dots`)
## Git / VCS
- **jj (Jujutsu)** as local workflow, colocated with git
- Push via `nix develop --command git push` (for prek hooks)
- Branch naming: `feat/<name>`, `fix/<name>`
- Commit messages: conventional commits (`feat:`, `fix:`, `refactor:`, `docs:`)