0xrsydn/idx-cli
1
mirror of https://github.com/0xrsydn/idx-cli.git synced 2026-08-07 01:33:52 +00:00
Code Issues Projects Releases 1 Packages Wiki Activity

Complete batch 2 ownership import hardening

Browse source
This commit is contained in:
Rasyidan Akbar F. 2026-03-30 16:40:32 +07:00
commit a52712b063
11 changed files with 698 additions and 62 deletions
Download patch file Download diff file Expand all files Collapse all files

20
FEATURE_SPEC.md
View file

@ -26,15 +26,16 @@ Use `TODO.md` as the execution log and smoke-history record.
## Verified Current State ## Verified Current State
- Current automated coverage is `153` tests: `102` unit and `51` integration. - Current automated coverage is `168` tests: `110` unit and `58` integration.
- Reusable smoke coverage exists via `scripts/live-smoke.sh`; command groups are documented in `docs/SMOKE.md`. - Reusable smoke coverage exists via `scripts/live-smoke.sh`; command groups are documented in `docs/SMOKE.md`.
- The latest smoke notes in `TODO.md` report passing live table and JSON checks for all shipped `stocks` commands. - The latest smoke notes in `TODO.md` report passing live table and JSON checks for all shipped `stocks` commands.
- Cache/offline parity, JSON startup-error handling, screener input validation, and the recent MSN output cleanups have already been completed. - Cache/offline parity, JSON startup-error handling, screener input validation, and the recent MSN output cleanups have already been completed.
- KSEI ownership import/query is now verified end-to-end from a local March 2026 PDF file into SQLite, with direct CLI reads through `ownership releases` and `ownership ticker`. - KSEI ownership import/query is now verified end-to-end from a local March 2026 PDF file into SQLite, with direct CLI reads through `ownership releases` and `ownership ticker`.
- Remote ownership ingestion should now be treated as an IDX PDF discovery-and-fetch problem first, not a hardcoded KSEI PDF URL problem: direct IDX announcement PDFs exist, but the hashed asset URL must be discovered from an IDX listing/announcement surface and fetched with browser-like behavior. - Remote ownership ingestion should now be treated as an IDX PDF discovery-and-fetch problem first, not a hardcoded KSEI PDF URL problem: direct IDX announcement PDFs exist, but the hashed asset URL must be discovered from an IDX listing/announcement surface and fetched with browser-like behavior.
- The CLI now has an explicit `idx ownership discover` surface that enumerates the latest hashed BEI ownership report URLs. - The CLI now has an explicit `idx ownership discover` surface that enumerates the latest hashed BEI ownership report URLs and defaults to the supported `above1` family.
- Live verification on `2026-03-29` confirmed that the parser-compatible discovered source is currently the `Pemegang Saham di atas 1% (KSEI)` `lamp1` attachment, and `idx ownership import --url` now works end to end for that discovered BEI PDF. - Live verification on `2026-03-29` confirmed that the parser-compatible discovered source is currently the `Pemegang Saham di atas 1% (KSEI)` `lamp1` attachment, and `idx ownership import --url` now works end to end for that discovered BEI PDF.
- The currently discoverable `above 5%` and `investor-type` BEI families remain different schemas and should now be treated as legacy / unsupported input outside the current holder-register parser contract. - The currently discoverable `above 5%` and `investor-type` BEI families remain different schemas and are now classified and rejected explicitly during import instead of falling through to a generic zero-row parse failure.
- Ownership smoke coverage now includes a dedicated `ownership-import` group that verifies live supported import plus expected unsupported-family failures.
This means the main gap is no longer endpoint coverage. This means the main gap is no longer endpoint coverage.
The remaining work is architecture cleanup, a few correctness edge cases, and selective UX expansion on top of already-shipped commands. The remaining work is architecture cleanup, a few correctness edge cases, and selective UX expansion on top of already-shipped commands.
@ -61,7 +62,7 @@ The remaining work is architecture cleanup, a few correctness edge cases, and se
| News | `idx stocks news` | Implemented | Fixture-backed CLI coverage exists | | News | `idx stocks news` | Implemented | Fixture-backed CLI coverage exists |
| Screener | `idx stocks screen` | Implemented with gaps | Validation landed; expression/preset workflow is still future work | | Screener | `idx stocks screen` | Implemented with gaps | Validation landed; expression/preset workflow is still future work |
| MSN charts | `idx stocks history --history-provider msn` | Missing | Explicit MSN history still returns unsupported for IDX | | MSN charts | `idx stocks history --history-provider msn` | Missing | Explicit MSN history still returns unsupported for IDX |
| KSEI ownership import/query | `idx ownership import --file`, `idx ownership import --url`, `idx ownership releases`, `idx ownership ticker` | Implemented with gaps | Local PDF import and SQLite-backed query flow are verified against the March 2026 KSEI release; remote IDX import now works for the discovered `above 1%` `lamp1` BEI attachment, while legacy `above 5%` and `investor-type` BEI report families still need explicit unsupported-input handling | | KSEI ownership import/query | `idx ownership import --file`, `idx ownership import --url`, `idx ownership releases`, `idx ownership ticker` | Implemented | Local PDF import and SQLite-backed query flow are verified against the March 2026 KSEI release; remote IDX import now works for the discovered `above 1%` `lamp1` BEI attachment, and legacy `above 5%` / `investor-type` BEI report families are rejected explicitly |
| Bing ownership CLI | `idx ownership import --fetch-bing` | Not implemented | Client groundwork exists, CLI import path is still deferred | | Bing ownership CLI | `idx ownership import --fetch-bing` | Not implemented | Client groundwork exists, CLI import path is still deferred |
--- ---
@ -82,6 +83,7 @@ The following items should no longer be treated as active backlog in this spec:
- Baseline parser and CLI regression coverage for the shipped MSN-only command set - Baseline parser and CLI regression coverage for the shipped MSN-only command set
- KSEI ownership parser hardening for the March 2026 live PDF layout, including the merged `DATE + SHARE_CODE` segment and `D`/`A` locality markers - KSEI ownership parser hardening for the March 2026 live PDF layout, including the merged `DATE + SHARE_CODE` segment and `D`/`A` locality markers
- Real KSEI ownership CLI verification from local file import into SQLite (`7261` rows across `955` tickers on `2026-03-28`) - Real KSEI ownership CLI verification from local file import into SQLite (`7261` rows across `955` tickers on `2026-03-28`)
- Ownership remote-import hardening for the `above1` contract, including direct-PDF-only `--url` input, discovery status output, explicit legacy-schema rejection, and live ownership-import smoke coverage
If any of the above regress, capture that in `TODO.md` as a new finding rather than reopening the old section here wholesale. If any of the above regress, capture that in `TODO.md` as a new finding rather than reopening the old section here wholesale.
@ -158,7 +160,7 @@ Current state:
Why it matters: Why it matters:
- The ownership feature now parses and stores live ownership data correctly from both local files and the currently discoverable `above 1%` BEI `lamp1` attachment. - The ownership feature now parses and stores live ownership data correctly from both local files and the currently discoverable `above 1%` BEI `lamp1` attachment.
- The remaining gap is not basic remote import anymore; it is hardening the supported `above 1%` path and clearly rejecting other discovered BEI schemas that do not match the holder-register contract. - Batch 2 hardening is now complete; the remaining ownership roadmap is snapshot publishing / `ownership sync` plus optional fallback ingest and cross-check work.
Done when: Done when:
- The CLI can discover the latest parser-compatible IDX ownership PDF URL from an IDX listing/announcement surface without hardcoded monthly paths. - The CLI can discover the latest parser-compatible IDX ownership PDF URL from an IDX listing/announcement surface without hardcoded monthly paths.
@ -167,12 +169,8 @@ Done when:
- Unsupported BEI report families are classified and rejected explicitly instead of failing later with a generic zero-row parse error. - Unsupported BEI report families are classified and rejected explicitly instead of failing later with a generic zero-row parse error.
Roadmap: Roadmap:
1. Keep discovery and import flows centered on the parser-compatible `above 1%` family and its `lamp1` attachment. 1. Publish maintained SQLite snapshot artifacts and add `idx ownership sync` after remote IDX import is stable.
2. Reuse the existing `curl-impersonate` pattern for browser-like PDF fetches. 2. Optionally add KSEI ZIP/TXT ingest later as fallback or cross-check input.
3. Add schema classification / unsupported-input UX for the legacy `above 5%` and `investor-type` BEI PDFs.
4. Decide whether the default discover output should be `above1`-first, with legacy families retained only for diagnostic use.
5. Publish maintained SQLite snapshot artifacts and add `idx ownership sync` after remote IDX import is stable.
6. Optionally add KSEI ZIP/TXT ingest later as fallback or cross-check input.
### P1 - UX and output contract cleanup ### P1 - UX and output contract cleanup

27
TODO.md
View file

@ -78,17 +78,17 @@
- the `above 5%` and `investor-type` BEI families remain different schemas; as of `2026-03-30`, they should be treated as legacy / unsupported input rather than new parser targets - the `above 5%` and `investor-type` BEI families remain different schemas; as of `2026-03-30`, they should be treated as legacy / unsupported input rather than new parser targets
### Batch 2 — Above-1 hardening and unsupported-input UX ### Batch 2 — Above-1 hardening and unsupported-input UX
- [ ] Standardize the supported remote-import contract on the `Pemegang Saham di atas 1% (KSEI)` holder-register layout and its `lamp1` attachment shape - [x] Standardize the supported remote-import contract on the `Pemegang Saham di atas 1% (KSEI)` holder-register layout and its `lamp1` attachment shape
- [ ] Add BEI PDF schema classification before parse/import so `ownership import --url` can reject non-holder-register PDFs before the parser runs - [x] Add BEI PDF schema classification before parse/import so `ownership import --url` can reject non-holder-register PDFs before the parser runs
- [ ] Improve CLI error messages and fallback behavior for discovery failure, fetch failure, invalid remote content, and known-but-unsupported legacy BEI schema variants - [x] Improve CLI error messages and fallback behavior for discovery failure, fetch failure, invalid remote content, and known-but-unsupported legacy BEI schema variants
- [ ] Capture live-like fixtures for the current discoverable `investor-type` and `above 5%` BEI attachments (or their `mutool` `stext` extracts) so unsupported-input detection and failure UX are regression-tested - [x] Capture live-like fixtures for the current discoverable `investor-type` and `above 5%` BEI attachments (or their `mutool` `stext` extracts) so unsupported-input detection and failure UX are regression-tested
- [ ] Decide and document whether `ownership discover` should default to `above1` output while keeping legacy families available only for diagnostic use - [x] Decide and document whether `ownership discover` should default to `above1` output while keeping legacy families available only for diagnostic use
- [ ] Decide and document whether `ownership import --url` accepts only direct PDF URLs or can also accept an IDX listing page as input - [x] Decide and document whether `ownership import --url` accepts only direct PDF URLs or can also accept an IDX listing page as input
- [ ] Add regression coverage for Cloudflare/HTML responses, missing announcement links, duplicate release imports, and unsupported BEI schema detections - [x] Add regression coverage for Cloudflare/HTML responses, missing announcement links, duplicate release imports, and unsupported BEI schema detections
- [ ] Batch 2 verification: `cargo build` - [x] Batch 2 verification: `cargo build`
- [ ] Batch 2 verification: `cargo clippy -- -D warnings` - [x] Batch 2 verification: `cargo clippy -- -D warnings`
- [ ] Batch 2 verification: `cargo test` - [x] Batch 2 verification: `cargo test`
- [ ] Batch 2 verification: ownership-focused smoke checks cover successful remote import plus expected failure UX - [x] Batch 2 verification: ownership-focused smoke checks cover successful remote import plus expected failure UX
### Batch 3 — Snapshot publishing + sync ### Batch 3 — Snapshot publishing + sync
- [ ] Design maintained SQLite snapshot publishing after remote IDX import is stable - [ ] Design maintained SQLite snapshot publishing after remote IDX import is stable
@ -157,6 +157,11 @@
- [x] Live `ownership import --url` checks against the currently discoverable `above 5%` and `investor-type` BEI PDFs still fail with `no KSEI rows parsed from PDF`, which is expected until unsupported-family detection / rejection UX lands - [x] Live `ownership import --url` checks against the currently discoverable `above 5%` and `investor-type` BEI PDFs still fail with `no KSEI rows parsed from PDF`, which is expected until unsupported-family detection / rejection UX lands
- [x] Live `mutool` inspection of the current discoverable `investor-type` BEI attachment shows a stock-level aggregate matrix (`DATE`, `STOCK_CODE`, `NUMBER_OF_SHARES`, investor-type columns, holder-size buckets), not the holder-level KSEI register schema the current parser imports - [x] Live `mutool` inspection of the current discoverable `investor-type` BEI attachment shows a stock-level aggregate matrix (`DATE`, `STOCK_CODE`, `NUMBER_OF_SHARES`, investor-type columns, holder-size buckets), not the holder-level KSEI register schema the current parser imports
- [x] Live `mutool` inspection of the current discoverable `above 5%` BEI attachment shows a member/tampungan report (`INVS`, member names, `KSEI UNTUK CLOSED MEMBER-...` labels), not the raw KSEI holder-register layout - [x] Live `mutool` inspection of the current discoverable `above 5%` BEI attachment shows a member/tampungan report (`INVS`, member names, `KSEI UNTUK CLOSED MEMBER-...` labels), not the raw KSEI holder-register layout
- [x] `ownership discover` now defaults to the supported `above1` family, surfaces per-URL importability status, and orders the importable `lamp1` attachment first so `--limit 1` yields the current supported PDF URL
- [x] `ownership import --url` now accepts only direct PDF URLs; IDX listing/announcement page URLs fail fast with guidance to run `ownership discover`
- [x] Valid-but-unsupported BEI PDFs now fail before row parsing with explicit schema-aware errors (`announcement_wrapper`, legacy `above5`, legacy `investor-type`) instead of the old generic `no KSEI rows parsed from PDF` path
- [x] Regression coverage now covers default `ownership discover` behavior, status visibility, listing-page rejection, duplicate SHA imports, and explicit unsupported-schema detection with compact `stext` fixtures plus fake-`mutool` CLI tests
- [x] New `ownership-import` smoke coverage now discovers the current live `above1`/`above5`/`investor-type` URLs, imports the supported `above1` attachment successfully, and confirms the legacy families fail with explicit unsupported-schema UX (`tmp/live-smoke/20260330-160201`)
## 🐛 Known Issues ## 🐛 Known Issues
- [ ] Yahoo Finance returns 429 from datacenter IPs occasionally - [ ] Yahoo Finance returns 429 from datacenter IPs occasionally

4
docs/SMOKE.md
View file

@ -23,7 +23,7 @@ scripts/live-smoke.sh --dry-run --mode full
- `live` runs the default real-network baseline: `general`, `live-table`, and `ownership` - `live` runs the default real-network baseline: `general`, `live-table`, and `ownership`
- `mock` runs the deterministic baseline: `general`, `mock`, `cache`, `routing`, `errors`, and `ownership` - `mock` runs the deterministic baseline: `general`, `mock`, `cache`, `routing`, `errors`, and `ownership`
- `full` runs every group, including live JSON output checks - `full` runs every group, including live JSON output checks and ownership-import verification
## Groups ## Groups
@ -35,10 +35,12 @@ scripts/live-smoke.sh --dry-run --mode full
- `routing`: Yahoo/MSN provider routing plus explicit MSN history unsupported behavior - `routing`: Yahoo/MSN provider routing plus explicit MSN history unsupported behavior
- `errors`: JSON error contract and invalid flag/input checks - `errors`: JSON error contract and invalid flag/input checks
- `ownership`: safe ownership smoke checks that do not require imported ownership data - `ownership`: safe ownership smoke checks that do not require imported ownership data
- `ownership-import`: live discovery/import hardening checks for supported `above1` import plus expected unsupported legacy-family failures
## Notes ## Notes
- The runner forces `IDX_OUTPUT=table` as its default environment so table cases stay stable; JSON checks use `-o json` explicitly. - The runner forces `IDX_OUTPUT=table` as its default environment so table cases stay stable; JSON checks use `-o json` explicitly.
- Cache-group warm cases clear the smoke cache before they run so each warm/offline/stale sequence starts clean and stale-cache assertions are not masked by earlier groups. - Cache-group warm cases clear the smoke cache before they run so each warm/offline/stale sequence starts clean and stale-cache assertions are not masked by earlier groups.
- Ownership commands that need imported data are intentionally not part of the baseline runner yet. The current baseline only covers `ownership releases` and the known unsupported `ownership import --fetch-bing`. - Ownership commands that need imported data are intentionally not part of the baseline runner yet. The current baseline only covers `ownership releases` and the known unsupported `ownership import --fetch-bing`.
- The new `ownership-import` group is intentionally opt-in for explicit `--group ownership-import` runs or `--mode full`; it discovers live URLs first, imports the supported `above1` attachment into the temp DB, then asserts the current `above5` and `investor-type` URLs fail with explicit unsupported-schema UX.
- When a case fails, inspect the per-case log in `tmp/live-smoke/.../logs/` before updating `TODO.md` or `FEATURE_SPEC.md`. - When a case fails, inspect the per-case log in `tmp/live-smoke/.../logs/` before updating `TODO.md` or `FEATURE_SPEC.md`.

70
scripts/live-smoke.sh
View file

@ -28,6 +28,9 @@ CACHE_HOME=""
DATA_HOME="" DATA_HOME=""
OWNERSHIP_DB="" OWNERSHIP_DB=""
LOG_DIR="" LOG_DIR=""
OWNERSHIP_ABOVE1_URL=""
OWNERSHIP_ABOVE5_URL=""
OWNERSHIP_INVESTOR_TYPE_URL=""
declare -a BASE_ENV=() declare -a BASE_ENV=()
@ -59,6 +62,7 @@ Groups:
routing provider routing and explicit unsupported checks routing provider routing and explicit unsupported checks
errors JSON error contract and invalid-flag checks errors JSON error contract and invalid-flag checks
ownership ownership commands that are safe without imported data ownership ownership commands that are safe without imported data
ownership-import live ownership discovery/import hardening checks
Examples: Examples:
scripts/live-smoke.sh scripts/live-smoke.sh
@ -124,7 +128,12 @@ group_requested() {
should_run_group() { should_run_group() {
local group="$1" local group="$1"
mode_allows_group "$group" && group_requested "$group" if [[ ${#GROUP_FILTERS[@]} -gt 0 ]]; then
group_requested "$group"
return
fi
mode_allows_group "$group"
} }
add_case() { add_case() {
@ -242,6 +251,12 @@ register_cases() {
add_case "ownership" "releases-empty" "0" "" "ownership releases" "No ownership releases imported yet." add_case "ownership" "releases-empty" "0" "" "ownership releases" "No ownership releases imported yet."
add_case "ownership" "fetch-bing-unsupported" "1" "" "ownership import --fetch-bing $live_symbol" "--fetch-bing import is not implemented yet" add_case "ownership" "fetch-bing-unsupported" "1" "" "ownership import --fetch-bing $live_symbol" "--fetch-bing import is not implemented yet"
add_case "ownership-import" "discover-default-above1" "0" "" "ownership discover --limit 1" "supported"
add_case "ownership-import" "import-supported-above1" "0" "" "ownership import --url $OWNERSHIP_ABOVE1_URL" "Imported "
add_case "ownership-import" "releases-after-import" "0" "" "ownership releases" "$OWNERSHIP_ABOVE1_URL"
add_case "ownership-import" "import-legacy-above5" "1" "" "ownership import --url $OWNERSHIP_ABOVE5_URL" "legacy IDX \`above5\` ownership PDFs are not supported for import"
add_case "ownership-import" "import-legacy-investor-type" "1" "" "ownership import --url $OWNERSHIP_INVESTOR_TYPE_URL" "legacy IDX \`investor-type\` ownership PDFs are not supported for import"
} }
parse_args() { parse_args() {
@ -400,6 +415,50 @@ prepare_environment() {
bootstrap_case "ownership-db" config set ownership.db_path "$OWNERSHIP_DB" bootstrap_case "ownership-db" config set ownership.db_path "$OWNERSHIP_DB"
} }
discover_ownership_import_url() {
local family="$1"
local log_file="$LOG_DIR/bootstrap-discover-${family}.log"
local url
if (( DRY_RUN )); then
case "$family" in
above1) printf 'https://example.invalid/above1-lamp1.pdf' ;;
above5) printf 'https://example.invalid/above5-lamp1.pdf' ;;
investor-type) printf 'https://example.invalid/investor-type-lamp1.pdf' ;;
*) return 1 ;;
esac
return 0
fi
(
cd "$ROOT_DIR" || exit 1
env "${BASE_ENV[@]}" "$BIN_PATH" -o json ownership discover --family "$family" --limit 1
) >"$log_file" 2>&1
if [[ $? -ne 0 ]]; then
echo "bootstrap failed for ownership-discover-$family; see $log_file" >&2
exit 1
fi
url="$(grep -m1 '"pdf_url"' "$log_file" | sed -E 's/.*"pdf_url": "([^"]+)".*/\1/')"
if [[ -z "$url" ]]; then
echo "bootstrap failed for ownership-discover-$family: could not parse pdf_url from $log_file" >&2
exit 1
fi
printf '%s' "$url"
}
prepare_ownership_import_inputs() {
if ! should_run_group "ownership-import"; then
return 0
fi
OWNERSHIP_ABOVE1_URL="$(discover_ownership_import_url above1)"
OWNERSHIP_ABOVE5_URL="$(discover_ownership_import_url above5)"
OWNERSHIP_INVESTOR_TYPE_URL="$(discover_ownership_import_url investor-type)"
}
cache_case_starts_fresh() { cache_case_starts_fresh() {
local group="$1" local group="$1"
local label="$2" local label="$2"
@ -494,7 +553,6 @@ main() {
parse_args "$@" parse_args "$@"
prepare_paths prepare_paths
register_cases
printf 'mode: %s\n' "$MODE" printf 'mode: %s\n' "$MODE"
printf 'workdir: %s\n' "$WORKDIR" printf 'workdir: %s\n' "$WORKDIR"
@ -506,6 +564,14 @@ main() {
build_binary build_binary
prepare_environment prepare_environment
prepare_ownership_import_inputs
register_cases
if should_run_group "ownership-import"; then
printf 'ownership above1 url: %s\n' "$OWNERSHIP_ABOVE1_URL"
printf 'ownership above5 url: %s\n' "$OWNERSHIP_ABOVE5_URL"
printf 'ownership investor-type url: %s\n' "$OWNERSHIP_INVESTOR_TYPE_URL"
fi
for case_line in "${CASES[@]}"; do for case_line in "${CASES[@]}"; do
run_case_line "$case_line" run_case_line "$case_line"

39
src/cli/ownership.rs
View file

@ -56,8 +56,8 @@ pub enum OwnershipCommand {
#[derive(Debug, Args)] #[derive(Debug, Args)]
pub struct DiscoverArgs { pub struct DiscoverArgs {
/// Report family to discover: all, above1, above5, or investor-type. /// Report family to discover: above1 (default), all, above5, or investor-type.
#[arg(long, default_value = "all")] #[arg(long, default_value = "above1")]
pub family: String, pub family: String,
/// Maximum number of discovered report URLs to print. /// Maximum number of discovered report URLs to print.
#[arg(long, default_value_t = 6)] #[arg(long, default_value_t = 6)]
@ -204,12 +204,15 @@ fn handle_discover(args: &DiscoverArgs, config: &IdxConfig) -> Result<(), IdxErr
table table
.load_preset(UTF8_FULL) .load_preset(UTF8_FULL)
.set_content_arrangement(ContentArrangement::Dynamic) .set_content_arrangement(ContentArrangement::Dynamic)
.set_header(vec!["DATE", "FAMILY", "KIND", "FILE", "TITLE", "URL"]); .set_header(vec![
"DATE", "FAMILY", "STATUS", "KIND", "FILE", "TITLE", "URL",
]);
for report in reports { for report in reports {
table.add_row(vec![ table.add_row(vec![
Cell::new(report.publish_date.split('T').next().unwrap_or("-")), Cell::new(report.publish_date.split('T').next().unwrap_or("-")),
Cell::new(report.family.label()), Cell::new(report.family.label()),
Cell::new(report.status.label()),
Cell::new(if report.is_attachment { Cell::new(if report.is_attachment {
"attachment" "attachment"
} else { } else {
@ -859,11 +862,13 @@ fn resolve_pdf_input(args: &ImportArgs) -> Result<Option<ResolvedPdfInput>, IdxE
} }
if let Some(url) = &args.url { if let Some(url) = &args.url {
let target = cache_pdf_path(url)?; let trimmed = url.trim();
download_pdf(url, &target)?; validate_import_url(trimmed)?;
let target = cache_pdf_path(trimmed)?;
download_pdf(trimmed, &target)?;
return Ok(Some(ResolvedPdfInput { return Ok(Some(ResolvedPdfInput {
pdf_path: target, pdf_path: target,
source_url: Some(url.clone()), source_url: Some(trimmed.to_string()),
})); }));
} }
@ -895,6 +900,28 @@ fn cache_pdf_path(url: &str) -> Result<PathBuf, IdxError> {
Ok(raw_dir.join(file_name)) Ok(raw_dir.join(file_name))
} }
fn validate_import_url(url: &str) -> Result<(), IdxError> {
let trimmed = url.trim();
if trimmed.is_empty() {
return Err(IdxError::InvalidInput(
"ownership import --url accepts direct PDF URLs only".to_string(),
));
}
let normalized = trimmed.to_ascii_lowercase();
let normalized = normalized
.split(['?', '#'])
.next()
.unwrap_or(normalized.as_str());
if normalized.ends_with(".pdf") {
return Ok(());
}
Err(IdxError::InvalidInput(
"ownership import --url accepts direct PDF URLs only; run `idx ownership discover` first to find the current supported attachment".to_string(),
))
}
fn download_pdf(url: &str, target: &Path) -> Result<(), IdxError> { fn download_pdf(url: &str, target: &Path) -> Result<(), IdxError> {
if is_idx_url(url) { if is_idx_url(url) {
return remote::download_idx_pdf(url, target); return remote::download_idx_pdf(url, target);

139
src/ownership/parser.rs
View file

@ -26,6 +26,49 @@ const HEADER_LABELS: &[&str] = &[
"PERCENTAGE", "PERCENTAGE",
]; ];
const HOLDER_REGISTER_SCHEMA_MARKERS: &[&str] = &[
"TEXT=\"DATE\"",
"TEXT=\"SHARE_CODE\"",
"TEXT=\"INVESTOR_NAME\"",
"TEXT=\"INVESTOR_TYPE\"",
"TEXT=\"LOCAL_FOREIGN\"",
"TEXT=\"TOTAL_HOLDING_SHARES\"",
"TEXT=\"PERCENTAGE\"",
];
const ANNOUNCEMENT_WRAPPER_SCHEMA_MARKERS: &[&str] =
&["TEXT=\"PENGUMUMAN\"", "PT BURSA EFEK INDONESIA (BEI)"];
const ABOVE_FIVE_SCHEMA_MARKERS: &[&str] = &[
"TEXT=\"INVS\"",
"REKENING TAMPUNGAN KSEI",
"CLOSED MEMBER-",
];
const INVESTOR_TYPE_SCHEMA_MARKERS: &[&str] = &[
"TEXT=\"STOCK_CODE\"",
"TEXT=\"NUMBER_OF_SHARES\"",
"TEXT=\"FOREIGN\"",
];
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum OwnershipPdfSchema {
HolderRegister,
AnnouncementWrapper,
LegacyAboveFivePercent,
LegacyInvestorType,
Unknown,
}
impl OwnershipPdfSchema {
pub fn label(self) -> &'static str {
match self {
Self::HolderRegister => "holder_register",
Self::AnnouncementWrapper => "announcement_wrapper",
Self::LegacyAboveFivePercent => "legacy_above5",
Self::LegacyInvestorType => "legacy_investor_type",
Self::Unknown => "unknown",
}
}
}
#[derive(Debug, Clone)] #[derive(Debug, Clone)]
struct PageLine { struct PageLine {
x: f32, x: f32,
@ -33,9 +76,8 @@ struct PageLine {
text: String, text: String,
} }
/// Parse a KSEI ownership PDF into raw rows. /// Extract mutool stext XML from a PDF file.
/// Shells out to `mutool` for XML extraction, then parses with quick-xml. pub fn extract_pdf_stext(path: &Path) -> Result<String, IdxError> {
pub fn parse_ksei_pdf(path: &Path) -> Result<Vec<KseiRawRow>, IdxError> {
check_mutool()?; check_mutool()?;
let output = Command::new("mutool") let output = Command::new("mutool")
@ -59,6 +101,35 @@ pub fn parse_ksei_pdf(path: &Path) -> Result<Vec<KseiRawRow>, IdxError> {
let xml = String::from_utf8(output.stdout) let xml = String::from_utf8(output.stdout)
.map_err(|e| IdxError::PdfParseError(format!("invalid utf-8 stext output: {e}")))?; .map_err(|e| IdxError::PdfParseError(format!("invalid utf-8 stext output: {e}")))?;
Ok(xml)
}
/// Classify a PDF schema from mutool stext XML before the row parser runs.
pub fn classify_stext_xml(xml: &str) -> OwnershipPdfSchema {
let normalized = xml.to_ascii_uppercase();
if count_schema_markers(&normalized, HOLDER_REGISTER_SCHEMA_MARKERS) >= 5 {
return OwnershipPdfSchema::HolderRegister;
}
if count_schema_markers(&normalized, ABOVE_FIVE_SCHEMA_MARKERS) >= 2 {
return OwnershipPdfSchema::LegacyAboveFivePercent;
}
if count_schema_markers(&normalized, INVESTOR_TYPE_SCHEMA_MARKERS) >= 3 {
return OwnershipPdfSchema::LegacyInvestorType;
}
if count_schema_markers(&normalized, ANNOUNCEMENT_WRAPPER_SCHEMA_MARKERS) >= 2 {
return OwnershipPdfSchema::AnnouncementWrapper;
}
OwnershipPdfSchema::Unknown
}
/// Parse a KSEI ownership PDF into raw rows.
/// Shells out to `mutool` for XML extraction, classifies the schema,
/// and only parses the supported holder-register layout.
pub fn parse_ksei_pdf(path: &Path) -> Result<Vec<KseiRawRow>, IdxError> {
let xml = extract_pdf_stext(path)?;
ensure_supported_schema(classify_stext_xml(&xml))?;
parse_stext_xml(&xml) parse_stext_xml(&xml)
} }
@ -116,6 +187,31 @@ pub fn check_mutool() -> Result<(), IdxError> {
Ok(()) Ok(())
} }
fn ensure_supported_schema(schema: OwnershipPdfSchema) -> Result<(), IdxError> {
match schema {
OwnershipPdfSchema::HolderRegister => Ok(()),
OwnershipPdfSchema::AnnouncementWrapper => Err(IdxError::Unsupported(
"IDX announcement wrapper PDFs are not importable; run `idx ownership discover` and use the `lamp1` attachment URL".to_string(),
)),
OwnershipPdfSchema::LegacyAboveFivePercent => Err(IdxError::Unsupported(
"legacy IDX `above5` ownership PDFs are not supported for import; only the `above1` holder-register `lamp1` attachment is supported".to_string(),
)),
OwnershipPdfSchema::LegacyInvestorType => Err(IdxError::Unsupported(
"legacy IDX `investor-type` ownership PDFs are not supported for import; only the `above1` holder-register `lamp1` attachment is supported".to_string(),
)),
OwnershipPdfSchema::Unknown => Err(IdxError::ParseError(
"PDF did not match the supported KSEI holder-register layout".to_string(),
)),
}
}
fn count_schema_markers(haystack: &str, markers: &[&str]) -> usize {
markers
.iter()
.filter(|marker| haystack.contains(**marker))
.count()
}
fn parse_line_attrs( fn parse_line_attrs(
reader: &Reader<&[u8]>, reader: &Reader<&[u8]>,
event: &BytesStart<'_>, event: &BytesStart<'_>,
@ -461,7 +557,9 @@ fn is_percentage_like(s: &str) -> bool {
mod tests { mod tests {
use std::path::Path; use std::path::Path;
use super::{check_mutool, parse_ksei_pdf, parse_stext_xml}; use super::{
OwnershipPdfSchema, check_mutool, classify_stext_xml, parse_ksei_pdf, parse_stext_xml,
};
#[test] #[test]
fn test_parse_stext_xml_live_like_lines_extract_rows() { fn test_parse_stext_xml_live_like_lines_extract_rows() {
@ -513,6 +611,39 @@ mod tests {
assert_eq!(row.percentage, "41,10"); assert_eq!(row.percentage, "41,10");
} }
#[test]
fn classify_stext_xml_detects_supported_holder_register_schema() {
let xml = include_str!("../../tests/fixtures/ksei_above1_stext_excerpt.xml");
assert_eq!(classify_stext_xml(xml), OwnershipPdfSchema::HolderRegister);
}
#[test]
fn classify_stext_xml_detects_announcement_wrapper_schema() {
let xml = include_str!("../../tests/fixtures/ksei_announcement_wrapper_stext_excerpt.xml");
assert_eq!(
classify_stext_xml(xml),
OwnershipPdfSchema::AnnouncementWrapper
);
}
#[test]
fn classify_stext_xml_detects_legacy_above5_schema() {
let xml = include_str!("../../tests/fixtures/ksei_above5_stext_excerpt.xml");
assert_eq!(
classify_stext_xml(xml),
OwnershipPdfSchema::LegacyAboveFivePercent
);
}
#[test]
fn classify_stext_xml_detects_legacy_investor_type_schema() {
let xml = include_str!("../../tests/fixtures/ksei_investor_type_stext_excerpt.xml");
assert_eq!(
classify_stext_xml(xml),
OwnershipPdfSchema::LegacyInvestorType
);
}
#[test] #[test]
fn test_parse_ksei_pdf_real_file_row_count() { fn test_parse_ksei_pdf_real_file_row_count() {
if check_mutool().is_err() { if check_mutool().is_err() {

161
src/ownership/remote.rs
View file

@ -78,9 +78,36 @@ impl OwnershipReportFamily {
} }
} }
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize)]
#[serde(rename_all = "snake_case")]
pub enum OwnershipReportStatus {
Supported,
AnnouncementOnly,
Unsupported,
}
impl OwnershipReportStatus {
pub fn label(self) -> &'static str {
match self {
Self::Supported => "supported",
Self::AnnouncementOnly => "announcement_only",
Self::Unsupported => "unsupported",
}
}
fn sort_rank(self) -> u8 {
match self {
Self::Supported => 0,
Self::AnnouncementOnly => 1,
Self::Unsupported => 2,
}
}
}
#[derive(Debug, Clone, PartialEq, Eq, Serialize)] #[derive(Debug, Clone, PartialEq, Eq, Serialize)]
pub struct DiscoveredOwnershipPdf { pub struct DiscoveredOwnershipPdf {
pub family: OwnershipReportFamily, pub family: OwnershipReportFamily,
pub status: OwnershipReportStatus,
pub listing_page_url: String, pub listing_page_url: String,
pub query_url: String, pub query_url: String,
pub pdf_url: String, pub pdf_url: String,
@ -148,13 +175,13 @@ pub fn parse_announcement_page(raw: &str) -> Result<AnnouncementPage, IdxError>
let trimmed = raw.trim(); let trimmed = raw.trim();
if trimmed.is_empty() { if trimmed.is_empty() {
return Err(IdxError::Http( return Err(IdxError::Http(
"IDX ownership discovery returned an empty announcement payload".to_string(), "IDX ownership discovery returned an empty announcement payload; the IDX announcement API did not return any JSON items".to_string(),
)); ));
} }
let normalized = trimmed.to_ascii_lowercase(); let normalized = trimmed.to_ascii_lowercase();
if normalized.starts_with("<!doctype html") || normalized.starts_with("<html") { if normalized.starts_with("<!doctype html") || normalized.starts_with("<html") {
return Err(IdxError::Http( return Err(IdxError::Http(
"IDX ownership discovery returned HTML instead of announcement JSON".to_string(), "IDX ownership discovery returned HTML instead of announcement JSON; verify the announcement endpoint or try again later".to_string(),
)); ));
} }
@ -172,7 +199,8 @@ pub fn select_latest_ownership_reports(
.find(|query| query.family == family) .find(|query| query.family == family)
else { else {
return Err(IdxError::Http(format!( return Err(IdxError::Http(format!(
"failed to discover IDX ownership reports from {query_url}: unknown report family" "failed to discover {} IDX ownership reports from {query_url}: unknown report family",
family.cli_name()
))); )));
}; };
@ -185,7 +213,8 @@ pub fn select_latest_ownership_reports(
let Some(item) = matches.into_iter().next() else { let Some(item) = matches.into_iter().next() else {
return Err(IdxError::Http(format!( return Err(IdxError::Http(format!(
"failed to discover IDX ownership reports from {query_url}: no matching announcement found" "failed to discover {} IDX ownership reports from {query_url}: no matching announcement found",
family.cli_name()
))); )));
}; };
@ -198,9 +227,11 @@ pub fn select_latest_ownership_reports(
.clone() .clone()
.or(attachment.pdf_filename.clone()) .or(attachment.pdf_filename.clone())
.map(|value| value.trim().to_string()); .map(|value| value.trim().to_string());
let status = classify_report_status(family, is_attachment);
DiscoveredOwnershipPdf { DiscoveredOwnershipPdf {
family, family,
status,
listing_page_url: announcement_listing_url(), listing_page_url: announcement_listing_url(),
query_url: query_url.to_string(), query_url: query_url.to_string(),
pdf_url: attachment.full_save_path, pdf_url: attachment.full_save_path,
@ -214,14 +245,17 @@ pub fn select_latest_ownership_reports(
.collect::<Vec<_>>(); .collect::<Vec<_>>();
attachments.sort_by(|left, right| { attachments.sort_by(|left, right| {
left.is_attachment left.status
.cmp(&right.is_attachment) .sort_rank()
.cmp(&right.status.sort_rank())
.then_with(|| right.is_attachment.cmp(&left.is_attachment))
.then_with(|| left.original_filename.cmp(&right.original_filename)) .then_with(|| left.original_filename.cmp(&right.original_filename))
}); });
if attachments.is_empty() { if attachments.is_empty() {
return Err(IdxError::Http(format!( return Err(IdxError::Http(format!(
"failed to discover IDX ownership reports from {query_url}: matching announcement had no PDF attachments" "failed to discover {} IDX ownership reports from {query_url}: matching announcement had no PDF attachments",
family.cli_name()
))); )));
} }
@ -257,7 +291,8 @@ pub fn discover_idx_ownership_reports(
right right
.publish_date .publish_date
.cmp(&left.publish_date) .cmp(&left.publish_date)
.then_with(|| left.is_attachment.cmp(&right.is_attachment)) .then_with(|| left.status.sort_rank().cmp(&right.status.sort_rank()))
.then_with(|| right.is_attachment.cmp(&left.is_attachment))
.then_with(|| left.original_filename.cmp(&right.original_filename)) .then_with(|| left.original_filename.cmp(&right.original_filename))
}); });
@ -313,12 +348,12 @@ pub fn validate_pdf_payload(bytes: &[u8]) -> Result<(), IdxError> {
let preview = String::from_utf8_lossy(&trimmed[..trimmed.len().min(256)]).to_ascii_lowercase(); let preview = String::from_utf8_lossy(&trimmed[..trimmed.len().min(256)]).to_ascii_lowercase();
if preview.contains("<!doctype html") || preview.contains("<html") { if preview.contains("<!doctype html") || preview.contains("<html") {
return Err(IdxError::Http( return Err(IdxError::Http(
"IDX ownership download returned HTML instead of a PDF".to_string(), "IDX ownership download returned HTML instead of a PDF/direct attachment".to_string(),
)); ));
} }
Err(IdxError::Http( Err(IdxError::Http(
"IDX ownership download did not look like a PDF".to_string(), "IDX ownership download did not look like a PDF/direct attachment".to_string(),
)) ))
} }
@ -415,6 +450,17 @@ fn attachment_label(attachment: &AnnouncementAttachment) -> String {
.to_ascii_lowercase() .to_ascii_lowercase()
} }
fn classify_report_status(
family: OwnershipReportFamily,
is_attachment: bool,
) -> OwnershipReportStatus {
match (family, is_attachment) {
(OwnershipReportFamily::AboveOnePercent, true) => OwnershipReportStatus::Supported,
(OwnershipReportFamily::AboveOnePercent, false) => OwnershipReportStatus::AnnouncementOnly,
_ => OwnershipReportStatus::Unsupported,
}
}
fn parse_pdf_path(raw: &str) -> Option<Vec<AnnouncementAttachment>> { fn parse_pdf_path(raw: &str) -> Option<Vec<AnnouncementAttachment>> {
serde_json::from_str::<Vec<AnnouncementAttachment>>(raw).ok() serde_json::from_str::<Vec<AnnouncementAttachment>>(raw).ok()
} }
@ -455,8 +501,8 @@ fn percent_encode(value: &str) -> String {
mod tests { mod tests {
use super::{ use super::{
AnnouncementPage, IDX_ANNOUNCEMENT_LISTING_URL, OwnershipReportFamily, AnnouncementPage, IDX_ANNOUNCEMENT_LISTING_URL, OwnershipReportFamily,
build_announcement_query_url, parse_announcement_page, select_latest_ownership_reports, OwnershipReportStatus, build_announcement_query_url, parse_announcement_page,
validate_pdf_payload, select_latest_ownership_reports, validate_pdf_payload,
}; };
use crate::error::IdxError; use crate::error::IdxError;
@ -489,22 +535,23 @@ mod tests {
assert_eq!(discovered[0].publish_date, "2026-03-27T16:34:20"); assert_eq!(discovered[0].publish_date, "2026-03-27T16:34:20");
assert_eq!( assert_eq!(
discovered[0].pdf_url, discovered[0].pdf_url,
"https://www.idx.co.id/StaticData/NewsAndAnnouncement/ANNOUNCEMENTSTOCK/From_EREP/202603/5d31bb6f49_announcement.pdf"
);
assert_eq!(
discovered[0].original_filename.as_deref(),
Some("20260327_Semua Emiten Saham_Pengumuman Bursa_32055594.pdf")
);
assert!(!discovered[0].is_attachment);
assert_eq!(
discovered[1].pdf_url,
"https://www.idx.co.id/StaticData/NewsAndAnnouncement/ANNOUNCEMENTSTOCK/From_EREP/202603/4f5c4efc6f_bf70f249ac_lamp1.pdf" "https://www.idx.co.id/StaticData/NewsAndAnnouncement/ANNOUNCEMENTSTOCK/From_EREP/202603/4f5c4efc6f_bf70f249ac_lamp1.pdf"
); );
assert_eq!( assert_eq!(
discovered[1].original_filename.as_deref(), discovered[0].original_filename.as_deref(),
Some("20260327_Semua Emiten Saham_Pengumuman Bursa_32055594_lamp1.pdf") Some("20260327_Semua Emiten Saham_Pengumuman Bursa_32055594_lamp1.pdf")
); );
assert!(discovered[1].is_attachment); assert_eq!(discovered[0].status, OwnershipReportStatus::Unsupported);
assert!(discovered[0].is_attachment);
assert_eq!(
discovered[1].pdf_url,
"https://www.idx.co.id/StaticData/NewsAndAnnouncement/ANNOUNCEMENTSTOCK/From_EREP/202603/5d31bb6f49_announcement.pdf"
);
assert_eq!(
discovered[1].original_filename.as_deref(),
Some("20260327_Semua Emiten Saham_Pengumuman Bursa_32055594.pdf")
);
assert!(!discovered[1].is_attachment);
} }
#[test] #[test]
@ -547,17 +594,18 @@ mod tests {
assert_eq!(discovered.len(), 2); assert_eq!(discovered.len(), 2);
assert_eq!(discovered[0].family, OwnershipReportFamily::AboveOnePercent); assert_eq!(discovered[0].family, OwnershipReportFamily::AboveOnePercent);
assert_eq!(discovered[0].status, OwnershipReportStatus::Supported);
assert_eq!(discovered[0].code.as_deref(), Some("Semua Emiten Saham")); assert_eq!(discovered[0].code.as_deref(), Some("Semua Emiten Saham"));
assert_eq!( assert_eq!(
discovered[0].original_filename.as_deref(), discovered[0].original_filename.as_deref(),
Some("20260310_Semua Emiten Saham_Pengumuman Bursa_32052554.pdf") Some("20260310_Semua Emiten Saham_Pengumuman Bursa_32052554_lamp1.pdf")
); );
assert!(!discovered[0].is_attachment); assert!(discovered[0].is_attachment);
assert_eq!( assert_eq!(
discovered[1].pdf_url, discovered[1].status,
"https://www.idx.co.id/StaticData/NewsAndAnnouncement/ANNOUNCEMENTSTOCK/From_EREP/202603/b9b638e5a8_8928aca255.pdf" OwnershipReportStatus::AnnouncementOnly
); );
assert!(discovered[1].is_attachment); assert!(!discovered[1].is_attachment);
} }
#[test] #[test]
@ -601,11 +649,12 @@ mod tests {
assert_eq!(discovered.len(), 2); assert_eq!(discovered.len(), 2);
assert_eq!( assert_eq!(
discovered[0].original_filename.as_deref(), discovered[0].original_filename.as_deref(),
Some("20260302_Pengumuman Bursa_32040089.pdf") Some("20260302_Pengumuman Bursa_32040089_lamp1.pdf")
); );
assert_eq!(discovered[0].status, OwnershipReportStatus::Unsupported);
assert_eq!( assert_eq!(
discovered[1].original_filename.as_deref(), discovered[1].original_filename.as_deref(),
Some("20260302_Pengumuman Bursa_32040089_lamp1.pdf") Some("20260302_Pengumuman Bursa_32040089.pdf")
); );
} }
@ -650,6 +699,57 @@ mod tests {
assert!(matches!(err, IdxError::Http(_))); assert!(matches!(err, IdxError::Http(_)));
} }
#[test]
fn rejects_empty_announcement_payload() {
let err = parse_announcement_page(" ").expect_err("empty payload must fail");
assert!(matches!(err, IdxError::Http(_)));
}
#[test]
fn select_latest_reports_errors_when_family_is_missing() {
let raw = r#"{"Items":[],"ItemCount":0,"PageCount":0}"#;
let page = parse_announcement_page(raw).expect("empty page parses");
let err = select_latest_ownership_reports(
&page,
"https://www.idx.co.id/primary/NewsAnnouncement/GetAllAnnouncement?keywords=pemegang%20saham%20di%20atas%201&pageNumber=1&pageSize=10&lang=id",
OwnershipReportFamily::AboveOnePercent,
)
.expect_err("missing announcement should fail");
assert!(matches!(err, IdxError::Http(_)));
assert!(err.to_string().contains("no matching announcement found"));
}
#[test]
fn select_latest_reports_errors_when_matching_item_has_no_pdf() {
let raw = r#"{
"Items": [
{
"PublishDate": "2026-03-10T12:09:09",
"Title": "Pemegang Saham di atas 1% (KSEI)",
"AnnouncementType": "",
"Code": "Semua Emiten Saham",
"Attachments": [],
"PdfPath": ""
}
],
"ItemCount": 1,
"PageCount": 1
}"#;
let page = parse_announcement_page(raw).expect("page parses");
let err = select_latest_ownership_reports(
&page,
"https://www.idx.co.id/primary/NewsAnnouncement/GetAllAnnouncement?keywords=pemegang%20saham%20di%20atas%201&pageNumber=1&pageSize=10&lang=id",
OwnershipReportFamily::AboveOnePercent,
)
.expect_err("missing pdf attachments should fail");
assert!(matches!(err, IdxError::Http(_)));
assert!(err.to_string().contains("no PDF attachments"));
}
#[test] #[test]
fn accepts_pdf_header() { fn accepts_pdf_header() {
validate_pdf_payload(b"%PDF-1.7\n1 0 obj\n").expect("pdf header should pass"); validate_pdf_payload(b"%PDF-1.7\n1 0 obj\n").expect("pdf header should pass");
@ -660,5 +760,6 @@ mod tests {
let err = validate_pdf_payload(b"<!doctype html><html><body>blocked</body></html>") let err = validate_pdf_payload(b"<!doctype html><html><body>blocked</body></html>")
.expect_err("html body must fail"); .expect_err("html body must fail");
assert!(matches!(err, IdxError::Http(_))); assert!(matches!(err, IdxError::Http(_)));
assert!(err.to_string().contains("PDF/direct attachment"));
} }
} }

284
tests/cli.rs
View file

@ -1,6 +1,8 @@
use std::fs; use std::fs;
use std::io::{Read, Write}; use std::io::{Read, Write};
use std::net::TcpListener; use std::net::TcpListener;
#[cfg(unix)]
use std::os::unix::fs::PermissionsExt;
use std::path::{Path, PathBuf}; use std::path::{Path, PathBuf};
use std::thread; use std::thread;
@ -65,6 +67,51 @@ fn spawn_single_response_server(content_type: &str, body: impl Into<Vec<u8>>) ->
format!("http://{addr}") format!("http://{addr}")
} }
fn fake_pdf_bytes() -> Vec<u8> {
b"%PDF-1.7\n% idx-cli test fixture\n".to_vec()
}
fn install_fake_mutool(root: &Path, xml: &str) -> PathBuf {
let bin_dir = root.join("fake-bin");
fs::create_dir_all(&bin_dir).expect("create fake bin dir");
let mutool_path = bin_dir.join("mutool");
let script = format!(
"#!/bin/sh\n\
if [ \"$1\" = \"--help\" ]; then\n\
exit 0\n\
fi\n\
if [ \"$1\" = \"convert\" ]; then\n\
cat <<'__IDX_XML__'\n\
{xml}\n\
__IDX_XML__\n\
exit 0\n\
fi\n\
echo \"unexpected mutool args: $@\" >&2\n\
exit 1\n"
);
fs::write(&mutool_path, script).expect("write fake mutool");
#[cfg(unix)]
{
let mut perms = fs::metadata(&mutool_path)
.expect("fake mutool metadata")
.permissions();
perms.set_mode(0o755);
fs::set_permissions(&mutool_path, perms).expect("set fake mutool perms");
}
bin_dir
}
fn prepend_path(dir: &Path) -> String {
match std::env::var("PATH") {
Ok(current) if !current.is_empty() => format!("{}:{current}", dir.display()),
_ => dir.display().to_string(),
}
}
fn pdf_url(base: &str, name: &str) -> String {
format!("{base}/{name}.pdf")
}
#[test] #[test]
fn help_works() { fn help_works() {
test_bin("help").arg("--help").assert().success(); test_bin("help").arg("--help").assert().success();
@ -695,18 +742,251 @@ fn ownership_discover_supports_above1_family() {
)); ));
} }
#[test]
fn ownership_discover_defaults_to_above1_and_prefers_supported_attachment() {
let body = r#"{
"Items": [
{
"PublishDate": "2026-03-10T12:09:09",
"Title": "Pemegang Saham di atas 1% (KSEI)",
"AnnouncementType": "",
"Code": "Semua Emiten Saham",
"Attachments": [
{
"PDFFilename": "d67ebf37e6_10d4080288.pdf",
"FullSavePath": "https://www.idx.co.id/StaticData/NewsAndAnnouncement/ANNOUNCEMENTSTOCK/From_EREP/202603/d67ebf37e6_10d4080288.pdf",
"IsAttachment": 0,
"OriginalFilename": "20260310_Semua Emiten Saham_Pengumuman Bursa_32052554.pdf"
},
{
"PDFFilename": "b9b638e5a8_8928aca255.pdf",
"FullSavePath": "https://www.idx.co.id/StaticData/NewsAndAnnouncement/ANNOUNCEMENTSTOCK/From_EREP/202603/b9b638e5a8_8928aca255.pdf",
"IsAttachment": 1,
"OriginalFilename": "20260310_Semua Emiten Saham_Pengumuman Bursa_32052554_lamp1.pdf"
}
],
"PdfPath": ""
}
],
"ItemCount": 1,
"PageSize": 10,
"PageNumber": 1,
"PageCount": 1
}"#;
let json_url = spawn_single_response_server("application/json", body.to_string());
let output = test_bin("ownership-discover-default")
.env("IDX_CURL_IMPERSONATE_BIN", "curl")
.env("IDX_OWNERSHIP_ANNOUNCEMENT_API_URL", &json_url)
.env(
"IDX_OWNERSHIP_ANNOUNCEMENT_PAGE_URL",
"http://127.0.0.1/pengumuman",
)
.args(["ownership", "discover", "--limit", "1"])
.assert()
.success()
.get_output()
.stdout
.clone();
let stdout = String::from_utf8(output).expect("utf8 stdout");
assert!(stdout.contains("Above 1%"));
assert!(stdout.contains("supported"));
assert!(stdout.contains("20260310_Semua Emiten Saham_Pengumuman Bursa_32052554_lamp1.pdf"));
assert!(!stdout.contains("20260310_Semua Emiten Saham_Pengumuman Bursa_32052554.pdf\n"));
}
#[test]
fn ownership_discover_json_includes_status() {
let body = r#"{
"Items": [
{
"PublishDate": "2026-03-10T12:09:09",
"Title": "Pemegang Saham di atas 1% (KSEI)",
"AnnouncementType": "",
"Code": "Semua Emiten Saham",
"Attachments": [
{
"PDFFilename": "d67ebf37e6_10d4080288.pdf",
"FullSavePath": "https://www.idx.co.id/StaticData/NewsAndAnnouncement/ANNOUNCEMENTSTOCK/From_EREP/202603/d67ebf37e6_10d4080288.pdf",
"IsAttachment": 0,
"OriginalFilename": "20260310_Semua Emiten Saham_Pengumuman Bursa_32052554.pdf"
},
{
"PDFFilename": "b9b638e5a8_8928aca255.pdf",
"FullSavePath": "https://www.idx.co.id/StaticData/NewsAndAnnouncement/ANNOUNCEMENTSTOCK/From_EREP/202603/b9b638e5a8_8928aca255.pdf",
"IsAttachment": 1,
"OriginalFilename": "20260310_Semua Emiten Saham_Pengumuman Bursa_32052554_lamp1.pdf"
}
],
"PdfPath": ""
}
],
"ItemCount": 1,
"PageSize": 10,
"PageNumber": 1,
"PageCount": 1
}"#;
let json_url = spawn_single_response_server("application/json", body.to_string());
test_bin("ownership-discover-json-status")
.env("IDX_CURL_IMPERSONATE_BIN", "curl")
.env("IDX_OWNERSHIP_ANNOUNCEMENT_API_URL", &json_url)
.env(
"IDX_OWNERSHIP_ANNOUNCEMENT_PAGE_URL",
"http://127.0.0.1/pengumuman",
)
.args(["-o", "json", "ownership", "discover", "--limit", "1"])
.assert()
.success()
.stdout(predicate::str::contains("\"status\": \"supported\""));
}
#[test] #[test]
fn ownership_import_url_rejects_html_response_before_pdf_parse() { fn ownership_import_url_rejects_html_response_before_pdf_parse() {
let html_url = spawn_single_response_server( let html_base = spawn_single_response_server(
"text/html; charset=utf-8", "text/html; charset=utf-8",
"<!doctype html><html><body>blocked</body></html>", "<!doctype html><html><body>blocked</body></html>",
); );
let html_url = pdf_url(&html_base, "blocked");
test_bin("ownership-import-url-html") test_bin("ownership-import-url-html")
.args(["ownership", "import", "--url", &html_url]) .args(["ownership", "import", "--url", &html_url])
.assert() .assert()
.failure() .failure()
.stderr(predicate::str::contains("returned HTML instead of a PDF")); .stderr(predicate::str::contains(
"returned HTML instead of a PDF/direct attachment",
));
}
#[test]
fn ownership_import_url_rejects_listing_page_inputs() {
test_bin("ownership-import-url-listing-page")
.args([
"ownership",
"import",
"--url",
"https://www.idx.co.id/id/berita/pengumuman/",
])
.assert()
.failure()
.stderr(predicate::str::contains(
"ownership import --url accepts direct PDF URLs only",
))
.stderr(predicate::str::contains("ownership discover"));
}
#[test]
fn ownership_import_url_supported_pdf_succeeds_with_fake_mutool() {
let root = test_env_dir("ownership-import-supported-remote");
let db_path = root.join("ownership.db");
let fake_mutool_dir = install_fake_mutool(
&root,
include_str!("fixtures/ksei_above1_stext_excerpt.xml"),
);
let pdf_base = spawn_single_response_server("application/pdf", fake_pdf_bytes());
let pdf_url = pdf_url(&pdf_base, "supported");
bin_with_root(&root)
.args([
"config",
"set",
"ownership.db_path",
db_path.to_str().unwrap(),
])
.assert()
.success();
bin_with_root(&root)
.env("PATH", prepend_path(&fake_mutool_dir))
.args(["ownership", "import", "--url", &pdf_url])
.assert()
.success()
.stdout(predicate::str::contains("Imported 1 rows for 1 tickers"));
bin_with_root(&root)
.args(["ownership", "releases"])
.assert()
.success()
.stdout(predicate::str::contains("2026-02-27"))
.stdout(predicate::str::contains(&pdf_url));
}
#[test]
fn ownership_import_url_duplicate_release_is_skipped() {
let root = test_env_dir("ownership-import-duplicate-release");
let db_path = root.join("ownership.db");
let fake_mutool_dir = install_fake_mutool(
&root,
include_str!("fixtures/ksei_above1_stext_excerpt.xml"),
);
let first_base = spawn_single_response_server("application/pdf", fake_pdf_bytes());
let second_base = spawn_single_response_server("application/pdf", fake_pdf_bytes());
let first_url = pdf_url(&first_base, "supported-first");
let second_url = pdf_url(&second_base, "supported-second");
bin_with_root(&root)
.args([
"config",
"set",
"ownership.db_path",
db_path.to_str().unwrap(),
])
.assert()
.success();
bin_with_root(&root)
.env("PATH", prepend_path(&fake_mutool_dir))
.args(["ownership", "import", "--url", &first_url])
.assert()
.success();
bin_with_root(&root)
.env("PATH", prepend_path(&fake_mutool_dir))
.args(["ownership", "import", "--url", &second_url])
.assert()
.success()
.stdout(predicate::str::contains("Release already imported"));
}
#[test]
fn ownership_import_url_rejects_legacy_above5_pdf_schema() {
let root = test_env_dir("ownership-import-above5-unsupported");
let fake_mutool_dir = install_fake_mutool(
&root,
include_str!("fixtures/ksei_above5_stext_excerpt.xml"),
);
let pdf_base = spawn_single_response_server("application/pdf", fake_pdf_bytes());
let pdf_url = pdf_url(&pdf_base, "legacy-above5");
bin_with_root(&root)
.env("PATH", prepend_path(&fake_mutool_dir))
.args(["ownership", "import", "--url", &pdf_url])
.assert()
.failure()
.stderr(predicate::str::contains(
"legacy IDX `above5` ownership PDFs are not supported for import",
));
}
#[test]
fn ownership_import_url_rejects_legacy_investor_type_pdf_schema() {
let root = test_env_dir("ownership-import-investor-type-unsupported");
let fake_mutool_dir = install_fake_mutool(
&root,
include_str!("fixtures/ksei_investor_type_stext_excerpt.xml"),
);
let pdf_base = spawn_single_response_server("application/pdf", fake_pdf_bytes());
let pdf_url = pdf_url(&pdf_base, "legacy-investor-type");
bin_with_root(&root)
.env("PATH", prepend_path(&fake_mutool_dir))
.args(["ownership", "import", "--url", &pdf_url])
.assert()
.failure()
.stderr(predicate::str::contains(
"legacy IDX `investor-type` ownership PDFs are not supported for import",
));
} }
#[test] #[test]

8
tests/fixtures/ksei_above5_stext_excerpt.xml vendored Normal file
View file

@ -0,0 +1,8 @@
<?xml version="1.0"?>
<document>
<page id="page1" width="595" height="842">
<line bbox="0 0 10 10" text="Pemegang Saham di atas 5% (KSEI)"></line>
<line bbox="0 12 10 22" text="INVS"></line>
<line bbox="0 24 10 34" text="PT MAGNUS Capital (REKENING TAMPUNGAN KSEI UNTUK CLOSED MEMBER-ABC)"></line>
</page>
</document>

8
tests/fixtures/ksei_announcement_wrapper_stext_excerpt.xml vendored Normal file
View file

@ -0,0 +1,8 @@
<?xml version="1.0"?>
<document>
<page id="page1" width="595" height="842">
<line bbox="0 0 10 10" text="PENGUMUMAN"></line>
<line bbox="0 12 10 22" text="PT Bursa Efek Indonesia (BEI)"></line>
<line bbox="0 24 10 34" text="Pemegang Saham di atas 1% (KSEI)"></line>
</page>
</document>

10
tests/fixtures/ksei_investor_type_stext_excerpt.xml vendored Normal file
View file

@ -0,0 +1,10 @@
<?xml version="1.0"?>
<document>
<page id="page1" width="595" height="842">
<line bbox="0 0 10 10" text="DATE"></line>
<line bbox="0 12 10 22" text="STOCK_CODE"></line>
<line bbox="0 24 10 34" text="NUMBER_OF_SHARES"></line>
<line bbox="0 36 10 46" text="FOREIGN"></line>
<line bbox="0 48 10 58" text="LOCAL INDIVIDUAL"></line>
</page>
</document>