diff --git a/.github/workflows/publish-ownership-snapshot.yml b/.github/workflows/publish-ownership-snapshot.yml index cb1baa4..e392355 100644 --- a/.github/workflows/publish-ownership-snapshot.yml +++ b/.github/workflows/publish-ownership-snapshot.yml @@ -7,6 +7,10 @@ on: description: Stable GitHub release tag to publish the snapshot assets to required: true default: ownership-snapshot-current + history: + description: Number of earlier monthly above-1% reports to include for `ownership changes` + required: false + default: "5" permissions: contents: write @@ -36,7 +40,8 @@ jobs: --idx-bin ./target/debug/idx \ --output-dir dist/ownership-snapshot \ --repo "${{ github.repository }}" \ - --release-tag "${{ inputs.release_tag }}" + --release-tag "${{ inputs.release_tag }}" \ + --history "${{ inputs.history }}" - name: Upload workflow artifact copy uses: actions/upload-artifact@v4 diff --git a/AGENTS.md b/AGENTS.md index e8bb38a..3a4a034 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -57,13 +57,13 @@ Ownership is local-first after bootstrap. Preferred bootstrap/update path: 1. `idx ownership sync` -2. if no snapshot manifest is available: `idx ownership discover` then `idx ownership import --url ` +2. if no snapshot manifest is available: `idx ownership discover` then `idx ownership import --url ` 3. local `--file` imports remain available for manual/fallback use Ownership input paths: -- primary remote source: discoverable IDX `above1` holder-register PDF +- primary remote source: IDX `above1` holder register — monthly XLSX on the Data Kepemilikan Saham page (since the 2026-05-29 report), legacy PDF announcements before that; `ownership discover` checks both - maintained snapshot path: `ownership sync` -- local fallback path: PDF, plus local archive `.zip` / `.txt` +- local file imports: IDX above-1% `.xlsx` (current format since June 2026), legacy `.pdf`, plus local archive `.zip` / `.txt` Important scope note: - archive ZIP/TXT ingest is a fallback/backstop path, not the primary product ingest surface diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index 1aecd3c..446413d 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -63,12 +63,32 @@ Unlike the `stocks` module (live-fetch), ownership is **import-then-query**: 4. Fully offline after import/sync ### Data Sources -- **KSEI** — official ≥1% shareholder registry (monthly PDF from IDX) +- **KSEI** — official ≥1% shareholder registry, published monthly by IDX. Through May 2026 it was a PDF announcement; from the 2026-05-29 report onward it is an XLSX workbook on the IDX "Data Kepemilikan Saham" page - **KSEI archive** — monthly ZIP/TXT balance-position matrix, used as a local fallback/backstop import path - **Bing Finance** — global institutional ownership (REST API, quarterly) +### Discovery +`idx ownership discover` merges two IDX sources: +- the announcement API (`GetAllAnnouncement`) for the legacy PDF announcements +- the Data Kepemilikan Saham page (`/id/perusahaan-tercatat/data-kepemilikan-saham/`), + whose server-rendered Nuxt payload lists every XLSX with a description such as + `Pemegang Saham di Atas 1% per 31 Agustus 2026`; the as-of date comes from that text + +Only the above-1% XLSX is `supported`; the daily above-5% and investor-type workbooks +are listed as `unsupported`. Both URLs can be overridden for tests with +`IDX_OWNERSHIP_ANNOUNCEMENT_API_URL` and `IDX_OWNERSHIP_DATA_PAGE_URL`. + +IDX sits behind Cloudflare, which accepts some curl-impersonate TLS profiles and +rejects others per path (for example `curl_chrome142` gets 403 on the data page). +IDX fetches therefore try several profiles until the response validates (JSON, page +payload, `%PDF`, or `PK` zip magic). `IDX_CURL_IMPERSONATE_BIN` pins a single profile. + ### Parser Pipeline ``` +KSEI XLSX → zip + quick-xml (sharedStrings + sheet1) → exact 12-column header check + → KseiHoldingDraft (Excel serial dates, numeric shares, percent → bps) + → SQLite INSERT (within transaction) + KSEI PDF → mutool stext (XML with coordinates) → quick-xml parse → KseiRawRow → normalize (ID locale numbers, dates, entity names) → KseiHolding → SQLite INSERT (within transaction) diff --git a/docs/OWNERSHIP_PUBLISH.md b/docs/OWNERSHIP_PUBLISH.md index 278a16e..05821fe 100644 --- a/docs/OWNERSHIP_PUBLISH.md +++ b/docs/OWNERSHIP_PUBLISH.md @@ -45,9 +45,11 @@ nix develop --command scripts/build-latest-ownership-snapshot.sh \ The script performs these steps: -1. `idx -o json ownership discover --family above1 --limit 1` +1. `idx -o json ownership discover --family above1 --limit 50` (newest supported + report first; XLSX from the Data Kepemilikan Saham page since June 2026) 2. verifies the discovered report is the current supported import path -3. imports that PDF into an isolated temp ownership DB +3. imports that report (plus `--history ` earlier months, oldest first) into + an isolated temp ownership DB 4. checks that the imported release metadata is non-empty and tied to the same source URL 5. runs `scripts/build-ownership-snapshot.sh` to emit the SQLite artifact and diff --git a/docs/OWNERSHIP_SELF_HOSTED.md b/docs/OWNERSHIP_SELF_HOSTED.md index 7457fbb..b51962a 100644 --- a/docs/OWNERSHIP_SELF_HOSTED.md +++ b/docs/OWNERSHIP_SELF_HOSTED.md @@ -38,7 +38,9 @@ nix develop --command scripts/publish-ownership-snapshot.sh \ That helper: 1. optionally builds `idx` -2. discovers the latest supported IDX/KSEI PDF +2. discovers the latest supported IDX/KSEI above-1% report (XLSX on the IDX + Data Kepemilikan Saham page since the 2026-05-29 report; PDF announcements + before that) 3. imports it into an isolated temp DB 4. emits the SQLite snapshot and manifest 5. ensures the stable GitHub release exists @@ -66,8 +68,14 @@ The sample timer uses: - `RandomizedDelaySec=30m` That is intentionally conservative. The ownership source is monthly, but the -exact publish day can drift. Start with an early-month schedule and adjust after -observing a few real runs. +exact publish day can drift. Observed XLSX upload times (HTTP Last-Modified): +2026-06-03, 2026-07-02 03:55Z, 2026-08-02 01:50Z, 2026-09-02 03:51Z, so a run +on the 2nd can occasionally land before the file exists; the 3rd is safer. +Start with an early-month schedule and adjust after observing a few real runs. + +Pass `--history ` (for example via `services.idxOwnershipPublish.extraArgs = +[ "--history" "5" ];` in `clan-private`) to include earlier months so +`idx ownership changes` works straight from the synced snapshot. ## Clan Integration Later diff --git a/scripts/build-latest-ownership-snapshot.sh b/scripts/build-latest-ownership-snapshot.sh index d69b54f..9b787b4 100755 --- a/scripts/build-latest-ownership-snapshot.sh +++ b/scripts/build-latest-ownership-snapshot.sh @@ -6,7 +6,8 @@ usage() { cat <<'EOF' Usage: scripts/build-latest-ownership-snapshot.sh --output-dir [options] -Discover the latest supported IDX/KSEI ownership PDF, import it into an isolated +Discover the latest supported IDX/KSEI above-1% report (XLSX from the IDX Data +Kepemilikan Saham page, or a legacy PDF announcement), import it into an isolated ownership database, and emit GitHub-release-ready snapshot artifacts plus manifest. Options: @@ -17,6 +18,9 @@ Options: (default: 0xrsydn/idx-cli) --release-tag Stable GitHub release tag used for the default base URL (default: ownership-snapshot-current) + --history Also import the previous monthly above-1% XLSX + reports (oldest first) so `ownership changes` works + from the snapshot (default: 0) --keep-workdir Keep the temp workdir instead of deleting it --help Show this help EOF @@ -28,6 +32,7 @@ BASE_URL="" REPO_FULL_NAME="0xrsydn/idx-cli" RELEASE_TAG="ownership-snapshot-current" KEEP_WORKDIR="0" +HISTORY="0" while [[ $# -gt 0 ]]; do case "$1" in @@ -51,6 +56,10 @@ while [[ $# -gt 0 ]]; do RELEASE_TAG="${2:-}" shift 2 ;; + --history) + HISTORY="${2:-}" + shift 2 + ;; --keep-workdir) KEEP_WORKDIR="1" shift @@ -73,6 +82,13 @@ if [[ -z "$OUTPUT_DIR" ]]; then exit 2 fi +if ! [[ "$HISTORY" =~ ^[0-9]+$ ]]; then + echo "--history must be a non-negative integer" >&2 + exit 2 +fi +# Canonicalize so "08" is 8, not an invalid octal literal in (( )). +HISTORY=$((10#$HISTORY)) + if [[ -z "$BASE_URL" ]]; then BASE_URL="https://github.com/${REPO_FULL_NAME}/releases/download/${RELEASE_TAG}" fi @@ -115,8 +131,8 @@ RELEASES_JSON="$WORKDIR/releases.json" MANIFEST_PATH="$OUTPUT_DIR/ownership-snapshot-manifest.json" DB_PATH="$XDG_DATA_HOME/idx/ownership.db" -printf 'Discovering latest supported IDX/KSEI ownership PDF...\n' -"$IDX_BIN" -o json ownership discover --family above1 --limit 1 > "$DISCOVERY_JSON" +printf 'Discovering latest supported IDX/KSEI above-1%% report...\n' +"$IDX_BIN" -o json ownership discover --family above1 --limit 50 > "$DISCOVERY_JSON" discovery_payload="$( jq -r ' @@ -151,7 +167,33 @@ DISCOVERED_TITLE="${discovery_fields[4]:-}" DISCOVERED_PUBLISH_DATE="${discovery_fields[5]:-}" DISCOVERED_ORIGINAL_FILENAME="${discovery_fields[6]:-}" -printf 'Importing discovered PDF into isolated DB...\n' +if (( HISTORY > 0 )); then + # Earlier supported XLSX months, one per as-of date, newest first; import oldest first. + # Run jq in a plain command substitution (not <(...)) so a jq failure + # stops the script under set -e instead of looking like zero months. + history_payload="$( + jq -r --arg latest "$DISCOVERED_PDF_URL" --argjson n "$HISTORY" ' + (map(select(.pdf_url == $latest)) | .[0].as_of_date // "") as $latest_as_of + | [ .[] + | select(.status == "supported" and .format == "xlsx" and .as_of_date != null) + | select($latest_as_of == "" or .as_of_date < $latest_as_of) ] + | unique_by(.as_of_date) + | sort_by(.as_of_date) | reverse + | .[:$n] | reverse + | .[].pdf_url + ' "$DISCOVERY_JSON" + )" + history_urls=() + if [[ -n "$history_payload" ]]; then + mapfile -t history_urls <<< "$history_payload" + fi + printf 'Importing %d earlier monthly report(s) for history...\n' "${#history_urls[@]}" + for url in "${history_urls[@]}"; do + "$IDX_BIN" ownership import --url "$url" + done +fi + +printf 'Importing discovered report into isolated DB...\n' "$IDX_BIN" ownership import --url "$DISCOVERED_PDF_URL" printf 'Inspecting imported release metadata...\n' diff --git a/scripts/publish-ownership-snapshot.sh b/scripts/publish-ownership-snapshot.sh index 22a4d36..6bbc60d 100755 --- a/scripts/publish-ownership-snapshot.sh +++ b/scripts/publish-ownership-snapshot.sh @@ -16,6 +16,8 @@ Options: (default: 0xrsydn/idx-cli) --release-tag Stable GitHub release tag used for snapshot assets (default: ownership-snapshot-current) + --history Also include the previous monthly above-1% reports + in the snapshot (passed to the builder; default: 0) --build Run `cargo build` before publishing --keep-workdir Keep the temp workdir created by the builder helper --help Show this help @@ -28,6 +30,7 @@ REPO_FULL_NAME="0xrsydn/idx-cli" RELEASE_TAG="ownership-snapshot-current" BUILD_FIRST="0" KEEP_WORKDIR="0" +HISTORY="0" PUBLISH_WORKDIR="" while [[ $# -gt 0 ]]; do @@ -48,6 +51,10 @@ while [[ $# -gt 0 ]]; do RELEASE_TAG="${2:-}" shift 2 ;; + --history) + HISTORY="${2:-}" + shift 2 + ;; --build) BUILD_FIRST="1" shift @@ -120,6 +127,7 @@ build_args=( --output-dir "$PUBLISH_WORKDIR" --repo "$REPO_FULL_NAME" --release-tag "$RELEASE_TAG" + --history "$HISTORY" ) if [[ "$KEEP_WORKDIR" == "1" ]]; then diff --git a/src/cli/ownership.rs b/src/cli/ownership.rs index 4073649..f8b4bde 100644 --- a/src/cli/ownership.rs +++ b/src/cli/ownership.rs @@ -17,13 +17,13 @@ use crate::output::table::format_idr; use crate::ownership::types::{ ChangeType, FlowSignal, HolderRow, KseiHolding, OwnershipRelease, OwnershipSource, }; -use crate::ownership::{archive, db, entities, graph, parser, remote, search, snapshot}; +use crate::ownership::{archive, db, entities, graph, parser, remote, search, snapshot, xlsx}; use crate::runtime; #[derive(Debug, Args)] #[command( about = "Ownership intelligence (KSEI + Bing)", - long_about = "Ownership intelligence (KSEI + Bing).\n\nPreferred bootstrap path:\n 1. Run `idx ownership sync` to install or refresh a maintained SQLite snapshot.\n 2. If no snapshot manifest is available, run `idx ownership discover` and then `idx ownership import --url `.\n 3. Local `.pdf`, `.zip`, and `.txt` imports remain available for manual or fallback workflows.", + long_about = "Ownership intelligence (KSEI + Bing).\n\nPreferred bootstrap path:\n 1. Run `idx ownership sync` to install or refresh a maintained SQLite snapshot.\n 2. If no snapshot manifest is available, run `idx ownership discover` and then `idx ownership import --url `.\n 3. Local `.xlsx`, `.pdf`, `.zip`, and `.txt` imports remain available for manual or fallback workflows.", after_help = "Examples:\n idx ownership sync\n idx ownership sync --manifest /path/to/ownership-snapshot-manifest.json\n idx ownership discover --limit 1\n idx ownership import --url \n idx ownership releases" )] pub struct OwnershipCmd { @@ -35,13 +35,13 @@ pub struct OwnershipCmd { pub enum OwnershipCommand { #[command( about = "Discover the latest IDX-hosted ownership report URLs", - after_help = "Examples:\n idx ownership discover --limit 1\n idx ownership discover --family all --limit 6\n\nUse this when no snapshot manifest is available and you need a direct PDF URL for `idx ownership import --url`." + after_help = "Examples:\n idx ownership discover --limit 1\n idx ownership discover --family all --limit 6\n\nUse this when no snapshot manifest is available and you need a direct XLSX or PDF URL for `idx ownership import --url`." )] Discover(DiscoverArgs), #[command( about = "Import ownership data directly from source files", long_about = "Import ownership data directly from source files.\n\nPrefer `idx ownership sync` for normal bootstrap/update flows. Use `import` when you need a direct PDF import from IDX discovery output, a local PDF, or a local KSEI archive fallback file.", - after_help = "Examples:\n idx ownership import --url \n idx ownership import --file /path/to/ksei.pdf\n idx ownership import --file /path/to/BalanceposEfek20260227.zip" + after_help = "Examples:\n idx ownership import --url \n idx ownership import --file /path/to/peng-2026-08-00017-satu-persen.xlsx\n idx ownership import --file /path/to/ksei.pdf\n idx ownership import --file /path/to/BalanceposEfek20260227.zip" )] Import(ImportArgs), #[command( @@ -84,10 +84,10 @@ pub struct DiscoverArgs { #[derive(Debug, Args)] pub struct ImportArgs { - /// Direct URL to a remote ownership PDF, typically from `ownership discover`. + /// Direct URL to a remote above-1% XLSX or PDF, typically from `ownership discover`. #[arg(long)] pub url: Option, - /// Path to a local ownership file: PDF (primary), ZIP/TXT archive (fallback). + /// Path to a local ownership file: above-1% XLSX or PDF (primary), ZIP/TXT archive (fallback). #[arg(long)] pub file: Option, /// Fetch Bing institutional data for these symbols. @@ -836,6 +836,7 @@ fn handle_import(args: &ImportArgs, config: &IdxConfig) -> Result<(), IdxError> drafts } ImportInputFormat::Archive => archive::parse_balancepos_file(&import_input.import_path)?, + ImportInputFormat::Xlsx => xlsx::parse_above1_xlsx_file(&import_input.import_path)?, }; let mut holdings = Vec::with_capacity(drafts.len()); @@ -977,6 +978,7 @@ struct OwnershipImportResult { enum ImportInputFormat { Pdf, Archive, + Xlsx, } fn resolve_import_input(args: &ImportArgs) -> Result, IdxError> { @@ -997,13 +999,16 @@ fn resolve_import_input(args: &ImportArgs) -> Result if let Some(url) = &args.url { let trimmed = url.trim(); - validate_import_url(trimmed)?; - let target = cache_pdf_path(trimmed)?; - download_pdf(trimmed, &target)?; + let format = validate_import_url(trimmed)?; + let target = cache_download_path(trimmed, format)?; + match format { + ImportInputFormat::Xlsx => download_xlsx(trimmed, &target)?, + _ => download_pdf(trimmed, &target)?, + } return Ok(Some(ResolvedImportInput { import_path: target, source_url: Some(trimmed.to_string()), - format: ImportInputFormat::Pdf, + format, })); } @@ -1018,17 +1023,22 @@ fn detect_local_import_format(path: &Path) -> Result Ok(ImportInputFormat::Pdf), + Some("xlsx") if xlsx::supports_xlsx_file(path) => Ok(ImportInputFormat::Xlsx), Some("zip") | Some("txt") if archive::supports_local_archive_file(path) => { Ok(ImportInputFormat::Archive) } _ => Err(IdxError::InvalidInput(format!( - "unsupported local ownership file {}; expected a .pdf, .zip, or .txt input", + "unsupported local ownership file {}; expected a .xlsx, .pdf, .zip, or .txt input", path.display() ))), } } -fn cache_pdf_path(url: &str) -> Result { +fn cache_download_path(url: &str, format: ImportInputFormat) -> Result { + let extension = match format { + ImportInputFormat::Xlsx => ".xlsx", + _ => ".pdf", + }; let raw_dir = crate::cache::cache_dir()?.join("ownership").join("raw"); fs::create_dir_all(&raw_dir).map_err(|e| IdxError::Io(e.to_string()))?; @@ -1042,20 +1052,20 @@ fn cache_pdf_path(url: &str) -> Result { .to_string(); if file_name.is_empty() || file_name == "/" { - file_name = format!("ksei-{}.pdf", Utc::now().timestamp()); + file_name = format!("ksei-{}{extension}", Utc::now().timestamp()); } - if !file_name.to_ascii_lowercase().ends_with(".pdf") { - file_name.push_str(".pdf"); + if !file_name.to_ascii_lowercase().ends_with(extension) { + file_name.push_str(extension); } Ok(raw_dir.join(file_name)) } -fn validate_import_url(url: &str) -> Result<(), IdxError> { +fn validate_import_url(url: &str) -> Result { let trimmed = url.trim(); if trimmed.is_empty() { return Err(IdxError::InvalidInput( - "ownership import --url accepts direct PDF URLs only".to_string(), + "ownership import --url accepts direct XLSX or PDF URLs only".to_string(), )); } @@ -1065,11 +1075,14 @@ fn validate_import_url(url: &str) -> Result<(), IdxError> { .next() .unwrap_or(normalized.as_str()); if normalized.ends_with(".pdf") { - return Ok(()); + return Ok(ImportInputFormat::Pdf); + } + if normalized.ends_with(".xlsx") { + return Ok(ImportInputFormat::Xlsx); } Err(IdxError::InvalidInput( - "ownership import --url accepts direct PDF URLs only; run `idx ownership discover` first to find the current supported attachment".to_string(), + "ownership import --url accepts direct XLSX or PDF URLs only; run `idx ownership discover` first to find the current supported file".to_string(), )) } @@ -1105,6 +1118,32 @@ fn download_pdf(url: &str, target: &Path) -> Result<(), IdxError> { Ok(()) } +fn download_xlsx(url: &str, target: &Path) -> Result<(), IdxError> { + if is_idx_url(url) { + return remote::download_idx_xlsx(url, target); + } + + let response = ureq::get(url) + .header( + "Accept", + "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet,application/octet-stream,*/*;q=0.8", + ) + .call() + .map_err(|e| IdxError::Http(format!("failed to download XLSX: {e}")))?; + let bytes = response + .into_body() + .read_to_vec() + .map_err(|e| IdxError::Http(format!("failed reading XLSX body: {e}")))?; + remote::validate_xlsx_payload(&bytes)?; + + fs::write(target, &bytes).map_err(|e| { + IdxError::Io(format!( + "failed writing cached XLSX {}: {e}", + target.display() + )) + }) +} + fn is_idx_url(url: &str) -> bool { let normalized = url.trim().to_ascii_lowercase(); normalized.starts_with("https://www.idx.co.id/") diff --git a/src/curl_impersonate.rs b/src/curl_impersonate.rs index 3735feb..dbabb8c 100644 --- a/src/curl_impersonate.rs +++ b/src/curl_impersonate.rs @@ -40,7 +40,65 @@ pub fn run(stage: &str, args: &[&str]) -> Result { pub fn run_owned(stage: &str, args: &[String]) -> Result { let binary = chrome_curl_binary()?; - let output = Command::new(&binary).args(args).output().map_err(|e| { + run_binary(stage, &binary, args) +} + +/// Browser profiles tried, in order, when a site's bot protection rejects +/// some TLS fingerprints but not others (IDX's Cloudflare rules block +/// `curl_chrome142` on some pages while Firefox/Safari profiles pass). +const FALLBACK_PROFILES: &[&str] = &[ + "curl_chrome142", + "curl_firefox144", + "curl_chrome136", + "curl_firefox135", + "curl_safari184", + "curl_chrome131", + "curl_chrome124", + "curl_chrome120", + "curl_chrome116", +]; +const FALLBACK_ROUNDS: usize = 2; + +/// Run curl with each available impersonation profile until `accept` approves +/// the response body. `IDX_CURL_IMPERSONATE_BIN`, when set, is the only +/// profile used. Returns the first accepted output, or the last error. +pub fn run_owned_with_fallback( + stage: &str, + args: &[String], + accept: &dyn Fn(&[u8]) -> Result<(), IdxError>, +) -> Result { + let profiles: Vec = match std::env::var(OVERRIDE_ENV) { + Ok(value) if !value.trim().is_empty() => vec![value.trim().to_string()], + _ => FALLBACK_PROFILES + .iter() + .copied() + .filter(|bin| Command::new(bin).arg("--version").output().is_ok()) + .map(str::to_string) + .collect(), + }; + if profiles.is_empty() { + return Err(IdxError::Http(format!( + "no curl-impersonate binary found; set {OVERRIDE_ENV} or install nixpkgs#curl-impersonate" + ))); + } + + let mut last_error = None; + for _ in 0..FALLBACK_ROUNDS { + for binary in &profiles { + match run_binary(stage, binary, args) { + Ok(output) => match accept(&output.stdout) { + Ok(()) => return Ok(output), + Err(err) => last_error = Some(err), + }, + Err(err) => last_error = Some(err), + } + } + } + Err(last_error.expect("at least one profile was tried")) +} + +fn run_binary(stage: &str, binary: &str, args: &[String]) -> Result { + let output = Command::new(binary).args(args).output().map_err(|e| { if e.kind() == std::io::ErrorKind::NotFound { return IdxError::Http(format!( "curl-impersonate binary '{binary}' not found; set {OVERRIDE_ENV} or install nixpkgs#curl-impersonate-chrome" diff --git a/src/ownership/entities.rs b/src/ownership/entities.rs index 9de47af..21c94a0 100644 --- a/src/ownership/entities.rs +++ b/src/ownership/entities.rs @@ -212,7 +212,7 @@ pub fn resolve_entity( Ok(entity_id) } -fn normalize_investor_type(raw: &str) -> Option { +pub(crate) fn normalize_investor_type(raw: &str) -> Option { let value = raw.trim(); if value.is_empty() { None @@ -221,7 +221,7 @@ fn normalize_investor_type(raw: &str) -> Option { } } -fn normalize_locality(raw: &str) -> Option { +pub(crate) fn normalize_locality(raw: &str) -> Option { match raw.trim().to_uppercase().as_str() { "L" | "D" => Some(Locality::Local), "F" | "A" => Some(Locality::Foreign), diff --git a/src/ownership/mod.rs b/src/ownership/mod.rs index 3735ff5..6635dae 100644 --- a/src/ownership/mod.rs +++ b/src/ownership/mod.rs @@ -7,3 +7,4 @@ pub mod remote; pub mod search; pub mod snapshot; pub mod types; +pub mod xlsx; diff --git a/src/ownership/remote.rs b/src/ownership/remote.rs index c8b18f6..8960dde 100644 --- a/src/ownership/remote.rs +++ b/src/ownership/remote.rs @@ -12,6 +12,10 @@ const DEFAULT_IDX_ANNOUNCEMENT_API_URL: &str = const IDX_ANNOUNCEMENT_API_ENV: &str = "IDX_OWNERSHIP_ANNOUNCEMENT_API_URL"; const IDX_ANNOUNCEMENT_LISTING_ENV: &str = "IDX_OWNERSHIP_ANNOUNCEMENT_PAGE_URL"; const IDX_ANNOUNCEMENT_PAGE_SIZE: usize = 10; +/// Since June 2026 the monthly above-1% list (XLSX) is published only here. +pub const IDX_SHARE_OWNERSHIP_PAGE_URL: &str = + "https://www.idx.co.id/id/perusahaan-tercatat/data-kepemilikan-saham/"; +const IDX_SHARE_OWNERSHIP_PAGE_ENV: &str = "IDX_OWNERSHIP_DATA_PAGE_URL"; const USER_AGENT: &str = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"; #[derive(Debug, Clone, Deserialize)] @@ -116,6 +120,12 @@ pub struct DiscoveredOwnershipPdf { pub code: Option, pub original_filename: Option, pub is_attachment: bool, + /// File format of `pdf_url`: `pdf` (announcement attachment) or `xlsx` + /// (Data Kepemilikan Saham page). + pub format: String, + /// Report as-of date (`YYYY-MM-DD`) when the source states it. The data + /// page shows no publish date, so its entries use this as `publish_date`. + pub as_of_date: Option, } struct DiscoveryQuery { @@ -152,6 +162,13 @@ pub fn announcement_listing_url() -> String { .unwrap_or_else(|| IDX_ANNOUNCEMENT_LISTING_URL.to_string()) } +pub fn share_ownership_page_url() -> String { + std::env::var(IDX_SHARE_OWNERSHIP_PAGE_ENV) + .ok() + .filter(|value| !value.trim().is_empty()) + .unwrap_or_else(|| IDX_SHARE_OWNERSHIP_PAGE_URL.to_string()) +} + pub fn announcement_api_url() -> String { std::env::var(IDX_ANNOUNCEMENT_API_ENV) .ok() @@ -240,6 +257,8 @@ pub fn select_latest_ownership_reports( code: clean_option(item.code.as_deref()), original_filename, is_attachment, + format: "pdf".to_string(), + as_of_date: None, } }) .collect::>(); @@ -278,6 +297,7 @@ pub fn discover_idx_ownership_reports( "IDX ownership announcement discovery", &query_url, &json_headers(), + &validate_json_payload, ) .and_then(|raw| parse_announcement_page(&raw)) .and_then(|page| select_latest_ownership_reports(&page, &query_url, query.family)) @@ -287,6 +307,45 @@ pub fn discover_idx_ownership_reports( } } + let mut page_error = None; + match fetch_text( + "IDX share ownership data page discovery", + &share_ownership_page_url(), + &page_headers(), + &validate_share_ownership_page, + ) { + Ok(html) => { + let mut page_reports = parse_share_ownership_page(&html, &share_ownership_page_url()); + if page_reports.is_empty() { + // The page loaded (it has its file list) but nothing parsed: + // the payload shape changed. Say so instead of silently + // falling back to announcement-only results. + page_error = Some(format!( + "IDX share ownership data page {} loaded but no report entries could be parsed; the page layout may have changed", + share_ownership_page_url() + )); + } + page_reports + .retain(|report| family_filter.is_none_or(|family| family == report.family)); + // Keep the full above-1% history (snapshot backfill needs it), but + // only the newest entry of the unsupported daily/breakdown series. + page_reports.sort_by(|left, right| right.publish_date.cmp(&left.publish_date)); + let mut seen_unsupported = Vec::new(); + page_reports.retain(|report| { + if report.family == OwnershipReportFamily::AboveOnePercent { + return true; + } + if seen_unsupported.contains(&report.family) { + return false; + } + seen_unsupported.push(report.family); + true + }); + discovered.extend(page_reports); + } + Err(err) => page_error = Some(err.to_string()), + } + discovered.sort_by(|left, right| { right .publish_date @@ -300,6 +359,15 @@ pub fn discover_idx_ownership_reports( discovered.truncate(limit); } + if !discovered.is_empty() + && let Some(err) = &page_error + { + // Announcement results still exist, but since June 2026 the newest + // above-1% reports are only on the data page: don't hide its failure. + crate::runtime::warn(err); + } + errors.extend(page_error); + if discovered.is_empty() { let detail = if errors.is_empty() { "no matching ownership reports found".to_string() @@ -307,8 +375,9 @@ pub fn discover_idx_ownership_reports( errors.join("; ") }; return Err(IdxError::Http(format!( - "failed to discover IDX ownership reports from {}: {detail}", - announcement_listing_url() + "failed to discover IDX ownership reports from {} or {}: {detail}", + announcement_listing_url(), + share_ownership_page_url() ))); } @@ -316,8 +385,12 @@ pub fn discover_idx_ownership_reports( } pub fn download_idx_pdf(url: &str, target: &Path) -> Result<(), IdxError> { - let bytes = fetch_bytes("IDX ownership PDF download", url, &pdf_headers())?; - validate_pdf_payload(&bytes)?; + let bytes = fetch_bytes( + "IDX ownership PDF download", + url, + &pdf_headers(), + &validate_pdf_payload, + )?; fs::write(target, &bytes).map_err(|e| { IdxError::Io(format!( @@ -329,6 +402,239 @@ pub fn download_idx_pdf(url: &str, target: &Path) -> Result<(), IdxError> { Ok(()) } +pub fn download_idx_xlsx(url: &str, target: &Path) -> Result<(), IdxError> { + let bytes = fetch_bytes( + "IDX ownership XLSX download", + url, + &xlsx_headers(), + &validate_xlsx_payload, + )?; + + fs::write(target, &bytes).map_err(|e| { + IdxError::Io(format!( + "failed writing cached XLSX {}: {e}", + target.display() + )) + })?; + + Ok(()) +} + +/// XLSX is a ZIP container; Cloudflare challenges come back as HTML. +pub fn validate_xlsx_payload(bytes: &[u8]) -> Result<(), IdxError> { + if bytes.starts_with(b"PK\x03\x04") { + return Ok(()); + } + if looks_like_html(bytes) { + return Err(IdxError::Http( + "IDX ownership download returned HTML (likely a Cloudflare challenge) instead of an XLSX workbook".to_string(), + )); + } + Err(IdxError::Http( + "IDX ownership download did not look like an XLSX workbook".to_string(), + )) +} + +fn validate_json_payload(bytes: &[u8]) -> Result<(), IdxError> { + if looks_like_html(bytes) { + return Err(IdxError::Http( + "IDX ownership discovery returned HTML (likely a Cloudflare challenge) instead of announcement JSON".to_string(), + )); + } + Ok(()) +} + +fn validate_share_ownership_page(bytes: &[u8]) -> Result<(), IdxError> { + if contains_bytes(bytes, b"Prospectus") { + return Ok(()); + } + Err(IdxError::Http( + "IDX share ownership data page did not contain its file list (likely a Cloudflare challenge)".to_string(), + )) +} + +fn looks_like_html(bytes: &[u8]) -> bool { + let head: Vec = bytes + .iter() + .skip_while(|byte| byte.is_ascii_whitespace()) + .take(256) + .map(u8::to_ascii_lowercase) + .collect(); + head.starts_with(b" bool { + haystack + .windows(needle.len()) + .any(|window| window == needle) +} + +/// Parse the Data Kepemilikan Saham page. The page is server-rendered by +/// Nuxt and embeds its file list as JS object literals such as +/// `{Description:"Pemegang Saham di Atas 1% per 31 Agustus 2026", +/// Prospectus:"https:\u002F\u002Fwww.idx.co.id\u002FMedia\u002F...xlsx",...}`. +pub fn parse_share_ownership_page(html: &str, page_url: &str) -> Vec { + let mut reports = Vec::new(); + let mut rest = html; + + while let Some(start) = find_js_key(rest, "Description") { + let (description, after_description) = match read_js_string(&rest[start..]) { + Some(value) => value, + None => { + rest = &rest[start..]; + continue; + } + }; + let object_tail = &rest[start + after_description..]; + // The URL belongs to the same object literal, so it must come before + // the next Description key. + let window_end = find_js_key(object_tail, "Description").unwrap_or(object_tail.len()); + let window = &object_tail[..window_end]; + rest = object_tail; + + let Some(url_start) = find_js_key(window, "Prospectus") else { + continue; + }; + let Some((url, _)) = read_js_string(&window[url_start..]) else { + continue; + }; + if let Some(report) = classify_page_entry(description.trim(), url.trim(), page_url) { + reports.push(report); + } + } + + reports +} + +/// Offset just past `Key:` / `"Key":` (the opening quote of the value). +fn find_js_key(haystack: &str, key: &str) -> Option { + let bare = format!("{key}:\""); + let quoted = format!("\"{key}\":\""); + [bare, quoted] + .iter() + .filter_map(|needle| { + haystack + .find(needle.as_str()) + .map(|index| index + needle.len() - 1) + }) + .min() +} + +/// Read a JS/JSON string literal starting at the opening quote. Returns the +/// decoded value and the byte offset just past the closing quote. +fn read_js_string(input: &str) -> Option<(String, usize)> { + let mut chars = input.char_indices(); + if chars.next()?.1 != '"' { + return None; + } + let mut value = String::new(); + while let Some((index, ch)) = chars.next() { + match ch { + '"' => return Some((value, index + 1)), + '\\' => match chars.next()?.1 { + 'u' => { + let hex: String = (0..4) + .filter_map(|_| chars.next().map(|(_, c)| c)) + .collect(); + value.push(char::from_u32(u32::from_str_radix(&hex, 16).ok()?)?); + } + 'n' => value.push('\n'), + 't' => value.push('\t'), + other => value.push(other), + }, + _ => value.push(ch), + } + } + None +} + +fn classify_page_entry( + description: &str, + url: &str, + page_url: &str, +) -> Option { + let upper = description.to_uppercase(); + let family = if upper.contains("DI ATAS 1%") || upper.contains("ABOVE 1%") { + OwnershipReportFamily::AboveOnePercent + } else if upper.contains("DI ATAS 5%") || upper.contains("ABOVE 5%") { + OwnershipReportFamily::AboveFivePercent + } else if upper.contains("TIPE INVESTOR") + || upper.contains("KLASIFIKASI") + || upper.contains("INVESTOR TYPE") + || upper.contains("CLASSIFICATION") + { + OwnershipReportFamily::InvestorTypeBreakdown + } else { + return None; + }; + + let path = url.split(['?', '#']).next().unwrap_or_default(); + let file_name = path.rsplit('/').next().unwrap_or_default().to_string(); + let lower_name = file_name.to_ascii_lowercase(); + let format = if lower_name.ends_with(".xlsx") { + "xlsx" + } else if lower_name.ends_with(".pdf") { + "pdf" + } else { + return None; + }; + let status = if family == OwnershipReportFamily::AboveOnePercent && format == "xlsx" { + OwnershipReportStatus::Supported + } else { + OwnershipReportStatus::Unsupported + }; + // Without an as-of date the entry cannot be ordered against other + // months, and the snapshot manifest needs a date, so skip it. + let as_of_date = parse_indonesian_as_of_date(description)?.to_string(); + + Some(DiscoveredOwnershipPdf { + family, + status, + listing_page_url: page_url.to_string(), + query_url: page_url.to_string(), + pdf_url: url.to_string(), + title: description.to_string(), + publish_date: as_of_date.clone(), + code: None, + original_filename: (!file_name.is_empty()).then_some(file_name), + is_attachment: true, + format: format.to_string(), + as_of_date: Some(as_of_date), + }) +} + +/// `"... per 31 Agustus 2026"` -> 2026-08-31 (Indonesian or English months). +fn parse_indonesian_as_of_date(description: &str) -> Option { + let tokens: Vec = description + .split_whitespace() + .map(|token| { + token + .trim_matches(|c: char| !c.is_alphanumeric()) + .to_lowercase() + }) + .collect(); + tokens.windows(3).rev().find_map(|window| { + let day = window[0].parse::().ok()?; + let month = match window[1].as_str() { + "januari" | "january" | "jan" => 1, + "februari" | "february" | "feb" => 2, + "maret" | "march" | "mar" => 3, + "april" | "apr" => 4, + "mei" | "may" => 5, + "juni" | "june" | "jun" => 6, + "juli" | "july" | "jul" => 7, + "agustus" | "august" | "agu" | "aug" => 8, + "september" | "sep" => 9, + "oktober" | "october" | "okt" | "oct" => 10, + "november" | "nov" => 11, + "desember" | "december" | "des" | "dec" => 12, + _ => return None, + }; + let year = window[2].parse::().ok()?; + chrono::NaiveDate::from_ymd_opt(year, month, day) + }) +} + pub fn validate_pdf_payload(bytes: &[u8]) -> Result<(), IdxError> { if bytes.is_empty() { return Err(IdxError::Http( @@ -357,13 +663,23 @@ pub fn validate_pdf_payload(bytes: &[u8]) -> Result<(), IdxError> { )) } -fn fetch_text(stage: &str, url: &str, headers: &[(String, String)]) -> Result { - let bytes = fetch_bytes(stage, url, headers)?; +fn fetch_text( + stage: &str, + url: &str, + headers: &[(String, String)], + accept: &dyn Fn(&[u8]) -> Result<(), IdxError>, +) -> Result { + let bytes = fetch_bytes(stage, url, headers, accept)?; String::from_utf8(bytes) .map_err(|e| IdxError::Http(format!("failed to decode {stage} response as utf-8: {e}"))) } -fn fetch_bytes(stage: &str, url: &str, headers: &[(String, String)]) -> Result, IdxError> { +fn fetch_bytes( + stage: &str, + url: &str, + headers: &[(String, String)], + accept: &dyn Fn(&[u8]) -> Result<(), IdxError>, +) -> Result, IdxError> { let mut args = vec![ "--silent".to_string(), "--show-error".to_string(), @@ -377,7 +693,7 @@ fn fetch_bytes(stage: &str, url: &str, headers: &[(String, String)]) -> Result Vec<(String, String)> { ] } +fn page_headers() -> Vec<(String, String)> { + vec![ + ("User-Agent".to_string(), USER_AGENT.to_string()), + ( + "Accept".to_string(), + "text/html,application/xhtml+xml,*/*;q=0.8".to_string(), + ), + ( + "Accept-Language".to_string(), + "id-ID,id;q=0.9,en-US;q=0.8,en;q=0.7".to_string(), + ), + ] +} + +fn xlsx_headers() -> Vec<(String, String)> { + vec![ + ("User-Agent".to_string(), USER_AGENT.to_string()), + ( + "Accept".to_string(), + "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet,application/octet-stream,*/*;q=0.8".to_string(), + ), + ( + "Accept-Language".to_string(), + "id-ID,id;q=0.9,en-US;q=0.8,en;q=0.7".to_string(), + ), + ("Referer".to_string(), share_ownership_page_url()), + ] +} + fn pdf_headers() -> Vec<(String, String)> { vec![ ("User-Agent".to_string(), USER_AGENT.to_string()), @@ -502,7 +847,8 @@ mod tests { use super::{ AnnouncementPage, IDX_ANNOUNCEMENT_LISTING_URL, OwnershipReportFamily, OwnershipReportStatus, build_announcement_query_url, parse_announcement_page, - select_latest_ownership_reports, validate_pdf_payload, + parse_indonesian_as_of_date, parse_share_ownership_page, select_latest_ownership_reports, + validate_pdf_payload, validate_share_ownership_page, validate_xlsx_payload, }; use crate::error::IdxError; @@ -762,4 +1108,107 @@ mod tests { assert!(matches!(err, IdxError::Http(_))); assert!(err.to_string().contains("PDF/direct attachment")); } + + #[test] + fn parses_share_ownership_page_entries() { + let page_url = "https://www.idx.co.id/id/perusahaan-tercatat/data-kepemilikan-saham/"; + let reports = parse_share_ownership_page( + include_str!("../../tests/fixtures/idx_share_ownership_page_excerpt.html"), + page_url, + ); + assert_eq!(reports.len(), 7); + + let above1: Vec<_> = reports + .iter() + .filter(|r| r.family == OwnershipReportFamily::AboveOnePercent) + .collect(); + assert_eq!(above1.len(), 3); + assert!( + above1 + .iter() + .all(|r| r.status == OwnershipReportStatus::Supported) + ); + assert!(above1.iter().all(|r| r.format == "xlsx")); + + let august = above1 + .iter() + .find(|r| r.as_of_date.as_deref() == Some("2026-08-31")) + .expect("August above-1% entry"); + assert_eq!( + august.pdf_url, + "https://www.idx.co.id/Media/fahlw1o2/peng-2026-08-00017-satu-persen.xlsx" + ); + assert_eq!(august.publish_date, "2026-08-31"); + assert_eq!(august.listing_page_url, page_url); + assert_eq!( + august.original_filename.as_deref(), + Some("peng-2026-08-00017-satu-persen.xlsx") + ); + + let above5 = reports + .iter() + .filter(|r| r.family == OwnershipReportFamily::AboveFivePercent) + .count(); + let breakdown = reports + .iter() + .filter(|r| r.family == OwnershipReportFamily::InvestorTypeBreakdown) + .count(); + assert_eq!((above5, breakdown), (2, 2)); + assert!( + reports + .iter() + .filter(|r| r.family != OwnershipReportFamily::AboveOnePercent) + .all(|r| r.status == OwnershipReportStatus::Unsupported) + ); + } + + #[test] + fn share_ownership_page_handles_query_strings_and_dateless_entries() { + let html = r#"{Description:"Pemegang Saham di Atas 1% per 31 Agustus 2026",Prospectus:"https://www.idx.co.id/Media/x/peng-satu-persen.xlsx?download=1",ListingDate:r},{Description:"Pemegang Saham di Atas 1%",Prospectus:"https://www.idx.co.id/Media/y/no-date.xlsx",ListingDate:r}"#; + let reports = parse_share_ownership_page(html, "p"); + assert_eq!( + reports.len(), + 1, + "dateless entry skipped, query-string entry kept" + ); + assert_eq!(reports[0].format, "xlsx"); + assert_eq!(reports[0].status, OwnershipReportStatus::Supported); + assert_eq!( + reports[0].original_filename.as_deref(), + Some("peng-satu-persen.xlsx") + ); + assert_eq!(reports[0].publish_date, "2026-08-31"); + } + + #[test] + fn share_ownership_page_without_payload_yields_nothing() { + assert!(parse_share_ownership_page("Just a moment...", "p").is_empty()); + assert!(validate_share_ownership_page(b"Just a moment...").is_err()); + } + + #[test] + fn parses_indonesian_and_english_as_of_dates() { + let date = |y, m, d| chrono::NaiveDate::from_ymd_opt(y, m, d); + assert_eq!( + parse_indonesian_as_of_date("Pemegang Saham di Atas 1% per 31 Agustus 2026"), + date(2026, 8, 31) + ); + assert_eq!( + parse_indonesian_as_of_date("Shareholders above 1% as of 29 May 2026"), + date(2026, 5, 29) + ); + assert_eq!( + parse_indonesian_as_of_date("Pemegang Saham di Atas 1%"), + None + ); + } + + #[test] + fn validates_xlsx_payloads() { + assert!(validate_xlsx_payload(b"PK\x03\x04rest-of-zip").is_ok()); + let err = validate_xlsx_payload(b"Just a moment...") + .expect_err("html challenge rejected"); + assert!(err.to_string().contains("Cloudflare"), "{err}"); + assert!(validate_xlsx_payload(b"%PDF-1.7").is_err()); + } } diff --git a/src/ownership/xlsx.rs b/src/ownership/xlsx.rs new file mode 100644 index 0000000..d12be20 --- /dev/null +++ b/src/ownership/xlsx.rs @@ -0,0 +1,554 @@ +//! Parser for the IDX "Pemegang Saham di Atas 1%" XLSX workbook. +//! +//! Since June 2026 IDX publishes the monthly above-1% holder register as an +//! XLSX file on the "Data Kepemilikan Saham" page instead of a PDF +//! announcement. The workbook has one sheet: a disclaimer block, a `[PUBLIC]` +//! marker, then a fixed 12-column table. Only that exact header is accepted, +//! so the daily above-5% workbook and the investor-type / classification +//! workbooks published on the same page are rejected instead of mis-imported. + +use std::collections::HashMap; +use std::fs; +use std::io::{Cursor, Read}; +use std::path::Path; + +use chrono::{Duration, NaiveDate}; +use quick_xml::Reader; +use quick_xml::events::{BytesStart, Event}; + +use crate::error::IdxError; +use crate::ownership::entities::{normalize_investor_type, normalize_locality, parse_ksei_date}; +use crate::ownership::types::KseiHoldingDraft; + +const EXPECTED_HEADER: [&str; 12] = [ + "DATE", + "SHARE_CODE", + "ISSUER_NAME", + "INVESTOR_NAME", + "INVESTOR_CLASSIFICATION", + "LOCAL_FOREIGN", + "NATIONALITY", + "DOMICILE", + "HOLDINGS_SCRIPLESS", + "HOLDINGS_SCRIP", + "TOTAL_HOLDING_SHARES", + "PERCENTAGE", +]; + +/// Rows above this line are disclaimer text; the header must appear before it. +const MAX_HEADER_SEARCH_ROWS: usize = 50; + +pub fn supports_xlsx_file(path: &Path) -> bool { + path.extension() + .and_then(|value| value.to_str()) + .is_some_and(|value| value.trim().eq_ignore_ascii_case("xlsx")) +} + +pub fn parse_above1_xlsx_file(path: &Path) -> Result, IdxError> { + let bytes = fs::read(path).map_err(|e| { + IdxError::Io(format!( + "failed to read ownership XLSX {}: {e}", + path.display() + )) + })?; + parse_above1_xlsx_bytes(&bytes) +} + +pub fn parse_above1_xlsx_bytes(bytes: &[u8]) -> Result, IdxError> { + let mut zip = zip::ZipArchive::new(Cursor::new(bytes)) + .map_err(|e| IdxError::ParseError(format!("failed to open ownership XLSX: {e}")))?; + + let shared_strings = match read_zip_entry(&mut zip, "xl/sharedStrings.xml")? { + Some(xml) => parse_shared_strings(&xml)?, + None => Vec::new(), + }; + let sheet_name = first_worksheet_name(&mut zip)?; + let sheet_xml = read_zip_entry(&mut zip, &sheet_name)? + .ok_or_else(|| IdxError::ParseError(format!("ownership XLSX is missing {sheet_name}")))?; + + let rows = parse_sheet_rows(&sheet_xml, &shared_strings)?; + drafts_from_rows(&rows) +} + +fn read_zip_entry( + zip: &mut zip::ZipArchive>, + name: &str, +) -> Result, IdxError> { + let mut file = match zip.by_name(name) { + Ok(file) => file, + Err(zip::result::ZipError::FileNotFound) => return Ok(None), + Err(e) => { + return Err(IdxError::ParseError(format!( + "failed to read ownership XLSX entry {name}: {e}" + ))); + } + }; + let mut output = String::new(); + file.read_to_string(&mut output).map_err(|e| { + IdxError::ParseError(format!("failed to decode ownership XLSX entry {name}: {e}")) + })?; + Ok(Some(output)) +} + +fn first_worksheet_name(zip: &mut zip::ZipArchive>) -> Result { + let mut sheets: Vec = zip + .file_names() + .filter(|name| name.starts_with("xl/worksheets/") && name.ends_with(".xml")) + .filter(|name| !name["xl/worksheets/".len()..].contains('/')) + .map(str::to_string) + .collect(); + // sheet1.xml, sheet2.xml, ... sort by their numeric suffix. + sheets.sort_by_key(|name| { + name.trim_start_matches("xl/worksheets/sheet") + .trim_end_matches(".xml") + .parse::() + .unwrap_or(u32::MAX) + }); + sheets + .into_iter() + .next() + .ok_or_else(|| IdxError::ParseError("ownership XLSX has no worksheet".to_string())) +} + +fn xml_error(context: &str, err: impl std::fmt::Display) -> IdxError { + IdxError::ParseError(format!("failed to parse ownership XLSX {context}: {err}")) +} + +/// Shared strings table: each `` is one string, made of one `` or +/// several rich-text runs ``. Phonetic hints (``) are skipped. +fn parse_shared_strings(xml: &str) -> Result, IdxError> { + let mut reader = Reader::from_str(xml); + let mut strings = Vec::new(); + let mut current: Option = None; + let mut in_text = false; + let mut in_phonetic = false; + + loop { + match reader.read_event() { + Ok(Event::Start(e)) => match e.local_name().as_ref() { + b"si" => current = Some(String::new()), + b"t" => in_text = true, + b"rPh" => in_phonetic = true, + _ => {} + }, + Ok(Event::Empty(e)) if e.local_name().as_ref() == b"si" => strings.push(String::new()), + Ok(Event::Text(text)) if in_text && !in_phonetic => { + if let Some(value) = current.as_mut() { + let unescaped = text + .unescape() + .map_err(|e| xml_error("shared strings", e))?; + value.push_str(&unescaped); + } + } + Ok(Event::End(e)) => match e.local_name().as_ref() { + b"si" => strings.push(current.take().unwrap_or_default()), + b"t" => in_text = false, + b"rPh" => in_phonetic = false, + _ => {} + }, + Ok(Event::Eof) => break, + Err(e) => return Err(xml_error("shared strings", e)), + _ => {} + } + } + + Ok(strings) +} + +/// One sheet row: its 1-based Excel row number and column letters +/// (`A`, `B`, ...) mapped to cell text. Sheet XML omits blank rows, so the +/// number comes from the row's `r` attribute, not its position. +#[derive(Debug, Default)] +struct SheetRow { + number: usize, + cells: HashMap, +} + +fn row_number(element: &BytesStart<'_>, fallback: usize) -> Result { + for attribute in element.attributes() { + let attribute = attribute.map_err(|e| xml_error("worksheet", e))?; + if attribute.key.local_name().as_ref() == b"r" { + if let Ok(number) = String::from_utf8_lossy(&attribute.value).trim().parse() { + return Ok(number); + } + } + } + Ok(fallback) +} + +fn parse_sheet_rows(xml: &str, shared_strings: &[String]) -> Result, IdxError> { + let mut reader = Reader::from_str(xml); + let mut rows = Vec::new(); + let mut row: Option = None; + let mut cell: Option<(String, String)> = None; + let mut value = String::new(); + let mut in_value = false; + + loop { + match reader.read_event() { + Ok(Event::Start(e)) => match e.local_name().as_ref() { + b"row" => { + row = Some(SheetRow { + number: row_number(&e, rows.len() + 1)?, + cells: HashMap::new(), + }) + } + b"c" => { + cell = Some(cell_ref_and_type(&e)?); + value.clear(); + } + b"v" | b"t" if cell.is_some() => in_value = true, + _ => {} + }, + Ok(Event::Empty(e)) if e.local_name().as_ref() == b"row" => rows.push(SheetRow { + number: row_number(&e, rows.len() + 1)?, + cells: HashMap::new(), + }), + Ok(Event::Text(text)) if in_value => { + let unescaped = text.unescape().map_err(|e| xml_error("worksheet", e))?; + value.push_str(&unescaped); + } + Ok(Event::End(e)) => match e.local_name().as_ref() { + b"v" | b"t" => in_value = false, + b"c" => { + if let (Some((column, cell_type)), Some(current_row)) = + (cell.take(), row.as_mut()) + { + let text = if cell_type == "s" { + let index = value.trim().parse::().map_err(|e| { + xml_error("worksheet", format!("bad shared string index: {e}")) + })?; + shared_strings.get(index).cloned().ok_or_else(|| { + xml_error( + "worksheet", + format!("shared string {index} out of range"), + ) + })? + } else { + value.clone() + }; + current_row.cells.insert(column, text); + } + } + b"row" => { + if let Some(current_row) = row.take() { + rows.push(current_row); + } + } + _ => {} + }, + Ok(Event::Eof) => break, + Err(e) => return Err(xml_error("worksheet", e)), + _ => {} + } + } + + Ok(rows) +} + +fn cell_ref_and_type(element: &BytesStart<'_>) -> Result<(String, String), IdxError> { + let mut column = String::new(); + let mut cell_type = String::new(); + for attribute in element.attributes() { + let attribute = attribute.map_err(|e| xml_error("worksheet", e))?; + let raw = String::from_utf8_lossy(&attribute.value).to_string(); + match attribute.key.local_name().as_ref() { + b"r" => column = raw.chars().take_while(char::is_ascii_alphabetic).collect(), + b"t" => cell_type = raw, + _ => {} + } + } + if column.is_empty() { + return Err(xml_error("worksheet", "cell without a reference")); + } + Ok((column, cell_type)) +} + +fn drafts_from_rows(rows: &[SheetRow]) -> Result, IdxError> { + let (header_index, columns) = rows + .iter() + .take(MAX_HEADER_SEARCH_ROWS) + .enumerate() + .find_map(|(index, row)| header_columns(row).map(|columns| (index, columns))) + .ok_or_else(|| { + IdxError::ParseError( + "ownership XLSX header did not match the above-1% holder register layout \ + (DATE, SHARE_CODE, ... PERCENTAGE); the above-5% and investor-type \ + workbooks are not supported" + .to_string(), + ) + })?; + + let mut drafts = Vec::new(); + for row in &rows[header_index + 1..] { + let line = row.number; + let field = |name: &str| -> &str { + columns + .get(name) + .and_then(|column| row.cells.get(column)) + .map(|value| value.trim()) + .unwrap_or("") + }; + + let ticker_code = field("SHARE_CODE"); + let investor_name = field("INVESTOR_NAME"); + if ticker_code.is_empty() && investor_name.is_empty() { + continue; + } + if ticker_code.is_empty() || investor_name.is_empty() { + return Err(IdxError::ParseError(format!( + "ownership XLSX row {line} is missing SHARE_CODE or INVESTOR_NAME" + ))); + } + + drafts.push(KseiHoldingDraft { + ticker_code: ticker_code.to_string(), + issuer_name: optional(field("ISSUER_NAME")), + raw_investor_name: investor_name.to_string(), + investor_type: normalize_investor_type(field("INVESTOR_CLASSIFICATION")), + locality: normalize_locality(field("LOCAL_FOREIGN")), + nationality: optional(field("NATIONALITY")), + domicile: optional(field("DOMICILE")), + holdings_scripless: parse_shares(field("HOLDINGS_SCRIPLESS"), line, false)?, + holdings_scrip: parse_shares(field("HOLDINGS_SCRIP"), line, false)?, + total_shares: parse_shares(field("TOTAL_HOLDING_SHARES"), line, true)?, + percentage_bps: parse_percentage_bps(field("PERCENTAGE"), line)?, + report_date: parse_report_date(field("DATE"), line)?, + }); + } + + if drafts.is_empty() { + return Err(IdxError::ParseError( + "ownership XLSX contained a header but no holder rows".to_string(), + )); + } + + Ok(drafts) +} + +/// Map header labels to column letters if `row` is exactly the expected header. +fn header_columns(row: &SheetRow) -> Option> { + let mut found: HashMap<&'static str, String> = HashMap::new(); + for (column, text) in &row.cells { + let mut label = text.trim().to_ascii_uppercase().replace(' ', "_"); + if label.is_empty() { + continue; + } + // Feb/Mar 2026 workbooks name the column INVESTOR_TYPE and hold codes + // (CP, ID, ...); later ones use INVESTOR_CLASSIFICATION with words. + if label == "INVESTOR_TYPE" { + label = "INVESTOR_CLASSIFICATION".to_string(); + } + let expected = EXPECTED_HEADER.iter().find(|name| **name == label)?; + found.insert(expected, column.clone()); + } + (found.len() == EXPECTED_HEADER.len()).then_some(found) +} + +fn optional(value: &str) -> Option { + (!value.is_empty()).then(|| value.to_string()) +} + +fn parse_number(raw: &str, line: usize, name: &str) -> Result { + raw.replace(',', "") + .parse::() + .ok() + .filter(|value| value.is_finite()) + .ok_or_else(|| { + IdxError::ParseError(format!("ownership XLSX row {line}: invalid {name} '{raw}'")) + }) +} + +fn parse_shares(raw: &str, line: usize, required: bool) -> Result { + if raw.is_empty() { + return if required { + Err(IdxError::ParseError(format!( + "ownership XLSX row {line}: missing TOTAL_HOLDING_SHARES" + ))) + } else { + Ok(0) + }; + } + let value = parse_number(raw, line, "share count")?; + if value < 0.0 || value.fract() != 0.0 || value > i64::MAX as f64 { + return Err(IdxError::ParseError(format!( + "ownership XLSX row {line}: share count '{raw}' is not a whole number" + ))); + } + Ok(value as i64) +} + +/// `54.94` (percent) -> 5494 basis points. Excel stores values such as +/// `1.1499999999999999`, so round to the nearest basis point. +fn parse_percentage_bps(raw: &str, line: usize) -> Result { + let value = parse_number(raw, line, "PERCENTAGE")?; + if !(0.0..=100.0).contains(&value) { + return Err(IdxError::ParseError(format!( + "ownership XLSX row {line}: PERCENTAGE '{raw}' is out of range" + ))); + } + Ok((value * 100.0).round() as i64) +} + +/// DATE is an Excel serial day number (`46265` = 2026-08-31); accept the +/// KSEI text form (`31-Aug-2026`) as well in case the cell is stored as text. +fn parse_report_date(raw: &str, line: usize) -> Result { + if let Ok(serial) = raw.parse::() { + let days = serial.trunc() as i64; + let epoch = NaiveDate::from_ymd_opt(1899, 12, 30).expect("valid Excel epoch"); + if (1..=2_958_465).contains(&days) { + return Ok(epoch + Duration::days(days)); + } + } + parse_ksei_date(raw).map_err(|_| { + IdxError::ParseError(format!("ownership XLSX row {line}: invalid DATE '{raw}'")) + }) +} + +#[cfg(test)] +mod tests { + use super::*; + + fn fixture() -> Vec { + parse_above1_xlsx_bytes(include_bytes!( + "../../tests/fixtures/ksei_above1_20260831_excerpt.xlsx" + )) + .expect("above-1% XLSX excerpt parses") + } + + #[test] + fn parses_above1_xlsx_excerpt() { + let drafts = fixture(); + // Pinned so a regression that silently drops rows fails loudly. + assert_eq!(drafts.len(), 17); + assert!( + drafts + .iter() + .all(|d| d.report_date == NaiveDate::from_ymd_opt(2026, 8, 31).unwrap()) + ); + + let dwimuria = drafts + .iter() + .find(|d| { + d.ticker_code == "BBCA" && d.raw_investor_name == "PT DWIMURIA INVESTAMA ANDALAN" + }) + .expect("BBCA controlling holder present"); + assert_eq!(dwimuria.percentage_bps, 5494); + assert_eq!( + dwimuria.investor_type.as_ref().map(|t| t.0.as_str()), + Some("CORPORATE") + ); + assert_eq!( + dwimuria.total_shares, + dwimuria.holdings_scripless + dwimuria.holdings_scrip + ); + } + + fn header_row(labels: &[&str]) -> SheetRow { + sheet_row(1, labels) + } + + fn sheet_row(number: usize, values: &[&str]) -> SheetRow { + SheetRow { + number, + cells: values + .iter() + .enumerate() + .filter(|(_, value)| !value.is_empty()) + .map(|(index, value)| { + ( + ((b'A' + index as u8) as char).to_string(), + value.to_string(), + ) + }) + .collect(), + } + } + + #[test] + fn accepts_legacy_investor_type_header() { + let mut labels = EXPECTED_HEADER.to_vec(); + labels[4] = "INVESTOR_TYPE"; + let columns = header_columns(&header_row(&labels)).expect("legacy header accepted"); + assert_eq!(columns["INVESTOR_CLASSIFICATION"], "E"); + + let row = sheet_row( + 2, + &[ + "46112", + "AADI", + "ADARO ANDALAN INDONESIA Tbk", + "GARIBALDI THOHIR", + "ID", + "L ", + "INDONESIAN", + "INDONESIA", + "454011607", + "0", + "454011607", + "5.83", + ], + ); + let drafts = drafts_from_rows(&[header_row(&labels), row]).expect("legacy rows parse"); + assert_eq!( + drafts[0].investor_type.as_ref().map(|t| t.0.as_str()), + Some("ID") + ); + assert_eq!( + drafts[0].report_date, + NaiveDate::from_ymd_opt(2026, 3, 31).unwrap() + ); + assert!(drafts[0].locality.is_some()); + } + + #[test] + fn row_errors_report_excel_row_numbers() { + let missing_investor = sheet_row( + 709, + &["46265", "BBCA", "BANK CENTRAL ASIA Tbk", "", "Corporate"], + ); + let err = drafts_from_rows(&[header_row(&EXPECTED_HEADER), missing_investor]) + .expect_err("row without investor rejected"); + assert!(err.to_string().contains("row 709"), "{err}"); + } + + #[test] + fn rejects_header_with_unknown_or_missing_columns() { + let mut labels = EXPECTED_HEADER.to_vec(); + labels[11] = "PERUBAHAN"; + assert!(header_columns(&header_row(&labels)).is_none()); + assert!(header_columns(&header_row(&EXPECTED_HEADER[..11])).is_none()); + } + + #[test] + fn rounds_float_noise_in_percentages() { + assert_eq!(parse_percentage_bps("1.1499999999999999", 1).unwrap(), 115); + assert_eq!(parse_percentage_bps("41.1", 1).unwrap(), 4110); + assert!(parse_percentage_bps("100.5", 1).is_err()); + } + + #[test] + fn parses_excel_serial_and_text_dates() { + let expected = NaiveDate::from_ymd_opt(2026, 8, 31).unwrap(); + assert_eq!(parse_report_date("46265", 1).unwrap(), expected); + assert_eq!(parse_report_date("31-Aug-2026", 1).unwrap(), expected); + assert!(parse_report_date("not a date", 1).is_err()); + } + + #[test] + fn rejects_above5_workbook() { + let err = parse_above1_xlsx_bytes(include_bytes!( + "../../tests/fixtures/ksei_above5_20260923_excerpt.xlsx" + )) + .expect_err("above-5% workbook must be rejected"); + assert!( + err.to_string().contains("above-1% holder register layout"), + "{err}" + ); + } + + #[test] + fn rejects_non_xlsx_bytes() { + assert!(parse_above1_xlsx_bytes(b"%PDF-1.7 not a workbook").is_err()); + } +} diff --git a/tests/cli.rs b/tests/cli.rs index 2451d77..89a58a7 100644 --- a/tests/cli.rs +++ b/tests/cli.rs @@ -407,7 +407,7 @@ fn ownership_import_help_mentions_sync_preference_and_fallback_files() { .success() .stdout(predicate::str::contains("Prefer `idx ownership sync`")) .stdout(predicate::str::contains( - "PDF (primary), ZIP/TXT archive (fallback)", + "above-1% XLSX or PDF (primary), ZIP/TXT archive (fallback)", )) .stdout(predicate::str::contains( "idx ownership import --file /path/to/BalanceposEfek20260227.zip", @@ -1189,6 +1189,43 @@ fn ownership_releases_uses_xdg_data_home_default_db_path() { ); } +/// Keeps discovery tests off the live IDX data page (nothing listens on port 9). +const UNREACHABLE_DATA_PAGE_URL: &str = "http://127.0.0.1:9/data-kepemilikan-saham/"; + +#[test] +fn ownership_discover_finds_xlsx_on_share_ownership_page() { + let page = fs::read_to_string("tests/fixtures/idx_share_ownership_page_excerpt.html") + .expect("read share ownership page fixture"); + let page_url = spawn_single_response_server("text/html", page); + + let output = test_bin("ownership-discover-data-page") + .env("IDX_CURL_IMPERSONATE_BIN", "curl") + .env( + "IDX_OWNERSHIP_ANNOUNCEMENT_API_URL", + "http://127.0.0.1:9/announcements", + ) + .env("IDX_OWNERSHIP_DATA_PAGE_URL", &page_url) + .args(["-o", "json", "ownership", "discover", "--limit", "1"]) + .output() + .expect("ownership discover json output"); + assert!( + output.status.success(), + "{}", + String::from_utf8_lossy(&output.stderr) + ); + + let reports: Value = serde_json::from_slice(&output.stdout).expect("parse discover json"); + let latest = &reports[0]; + assert_eq!(latest["family"], "above_one_percent"); + assert_eq!(latest["status"], "supported"); + assert_eq!(latest["format"], "xlsx"); + assert_eq!(latest["as_of_date"], "2026-08-31"); + assert_eq!( + latest["pdf_url"], + "https://www.idx.co.id/Media/fahlw1o2/peng-2026-08-00017-satu-persen.xlsx" + ); +} + #[test] fn ownership_discover_lists_fixture_candidates() { let body = fs::read_to_string("tests/fixtures/idx_announcement_kepemilikan.json") @@ -1198,6 +1235,7 @@ fn ownership_discover_lists_fixture_candidates() { test_bin("ownership-discover") .env("IDX_CURL_IMPERSONATE_BIN", "curl") .env("IDX_OWNERSHIP_ANNOUNCEMENT_API_URL", &json_url) + .env("IDX_OWNERSHIP_DATA_PAGE_URL", UNREACHABLE_DATA_PAGE_URL) .env( "IDX_OWNERSHIP_ANNOUNCEMENT_PAGE_URL", "http://127.0.0.1/pengumuman", @@ -1257,6 +1295,7 @@ fn ownership_discover_supports_above1_family() { test_bin("ownership-discover-above1") .env("IDX_CURL_IMPERSONATE_BIN", "curl") .env("IDX_OWNERSHIP_ANNOUNCEMENT_API_URL", &json_url) + .env("IDX_OWNERSHIP_DATA_PAGE_URL", UNREACHABLE_DATA_PAGE_URL) .env( "IDX_OWNERSHIP_ANNOUNCEMENT_PAGE_URL", "http://127.0.0.1/pengumuman", @@ -1274,6 +1313,10 @@ fn ownership_discover_supports_above1_family() { .stdout(predicate::str::contains("Above 1%")) .stdout(predicate::str::contains( "20260310_Semua Emiten Saham_Pengumuman Bursa_32052554_lamp1.pdf", + )) + .stderr(predicate::str::contains("warning:")) + .stderr(predicate::str::contains( + "IDX share ownership data page discovery", )); } @@ -1313,6 +1356,7 @@ fn ownership_discover_defaults_to_above1_and_prefers_supported_attachment() { let output = test_bin("ownership-discover-default") .env("IDX_CURL_IMPERSONATE_BIN", "curl") .env("IDX_OWNERSHIP_ANNOUNCEMENT_API_URL", &json_url) + .env("IDX_OWNERSHIP_DATA_PAGE_URL", UNREACHABLE_DATA_PAGE_URL) .env( "IDX_OWNERSHIP_ANNOUNCEMENT_PAGE_URL", "http://127.0.0.1/pengumuman", @@ -1367,6 +1411,7 @@ fn ownership_discover_json_includes_status() { test_bin("ownership-discover-json-status") .env("IDX_CURL_IMPERSONATE_BIN", "curl") .env("IDX_OWNERSHIP_ANNOUNCEMENT_API_URL", &json_url) + .env("IDX_OWNERSHIP_DATA_PAGE_URL", UNREACHABLE_DATA_PAGE_URL) .env( "IDX_OWNERSHIP_ANNOUNCEMENT_PAGE_URL", "http://127.0.0.1/pengumuman", @@ -1406,7 +1451,7 @@ fn ownership_import_url_rejects_listing_page_inputs() { .assert() .failure() .stderr(predicate::str::contains( - "ownership import --url accepts direct PDF URLs only", + "ownership import --url accepts direct XLSX or PDF URLs only", )) .stderr(predicate::str::contains("ownership discover")); } @@ -1524,6 +1569,77 @@ fn ownership_import_url_duplicate_release_is_skipped() { .stdout(predicate::str::contains("Release already imported")); } +#[test] +fn ownership_import_file_xlsx_supports_ticker_and_releases() { + let root = test_env_dir("ownership-import-xlsx"); + let db_path = root.join("ownership.db"); + let xlsx_path = root.join("peng-2026-08-00017-satu-persen.xlsx"); + fs::write( + &xlsx_path, + include_bytes!("fixtures/ksei_above1_20260831_excerpt.xlsx"), + ) + .expect("write xlsx fixture"); + + bin_with_root(&root) + .args([ + "config", + "set", + "ownership.db_path", + db_path.to_str().unwrap(), + ]) + .assert() + .success(); + + bin_with_root(&root) + .args(["ownership", "import", "--file", xlsx_path.to_str().unwrap()]) + .assert() + .success() + .stdout(predicate::str::contains("(as of 2026-08-31)")); + + let output = bin_with_root(&root) + .args(["-o", "json", "ownership", "ticker", "BBCA"]) + .output() + .expect("ownership ticker json output"); + assert!(output.status.success()); + let body = String::from_utf8_lossy(&output.stdout); + assert!(body.contains("DWIMURIA INVESTAMA ANDALAN"), "{body}"); + + let output = bin_with_root(&root) + .args(["-o", "json", "ownership", "releases"]) + .output() + .expect("ownership releases json output"); + let releases: Value = serde_json::from_slice(&output.stdout).expect("parse releases json"); + assert_eq!(releases[0]["as_of_date"], "2026-08-31"); +} + +#[test] +fn ownership_import_file_rejects_above5_xlsx() { + let root = test_env_dir("ownership-import-xlsx-above5"); + let db_path = root.join("ownership.db"); + let xlsx_path = root.join("peng-2026-09-23-00080-lima-persen.xlsx"); + fs::write( + &xlsx_path, + include_bytes!("fixtures/ksei_above5_20260923_excerpt.xlsx"), + ) + .expect("write xlsx fixture"); + + bin_with_root(&root) + .args([ + "config", + "set", + "ownership.db_path", + db_path.to_str().unwrap(), + ]) + .assert() + .success(); + + bin_with_root(&root) + .args(["ownership", "import", "--file", xlsx_path.to_str().unwrap()]) + .assert() + .failure() + .stderr(predicate::str::contains("above-1% holder register layout")); +} + #[test] fn ownership_import_force_reimports_existing_release() { let root = test_env_dir("ownership-import-force"); diff --git a/tests/fixtures/idx_share_ownership_page_excerpt.html b/tests/fixtures/idx_share_ownership_page_excerpt.html new file mode 100644 index 0000000..7962b06 --- /dev/null +++ b/tests/fixtures/idx_share_ownership_page_excerpt.html @@ -0,0 +1,5 @@ + +Data Kepemilikan Saham - Bursa Efek Indonesia +
+ + diff --git a/tests/fixtures/ksei_above1_20260831_excerpt.xlsx b/tests/fixtures/ksei_above1_20260831_excerpt.xlsx new file mode 100644 index 0000000..46bc8a1 Binary files /dev/null and b/tests/fixtures/ksei_above1_20260831_excerpt.xlsx differ diff --git a/tests/fixtures/ksei_above5_20260923_excerpt.xlsx b/tests/fixtures/ksei_above5_20260923_excerpt.xlsx new file mode 100644 index 0000000..2978145 Binary files /dev/null and b/tests/fixtures/ksei_above5_20260923_excerpt.xlsx differ