0xrsydn/idx-cli
1
mirror of https://github.com/0xrsydn/idx-cli.git synced 2026-11-07 09:45:36 +00:00
Code Issues Projects Releases 1 Packages Wiki Activity

Merge pull request #27 from 0xrsydn/feat/ownership-xlsx-snapshot

Browse source 
feat(ownership): build snapshots from XLSX with monthly history [3/3]
This commit is contained in:
brutaljokerz 2026-09-25 04:22:34 +07:00 • committed by GitHub
commit d4ec5015b9
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
17 changed files with 1354 additions and 47 deletions
Download patch file Download diff file Expand all files Collapse all files

7
.github/workflows/publish-ownership-snapshot.yml vendored
View file

@ -7,6 +7,10 @@ on:
description: Stable GitHub release tag to publish the snapshot assets to description: Stable GitHub release tag to publish the snapshot assets to
required: true required: true
default: ownership-snapshot-current default: ownership-snapshot-current
history:
description: Number of earlier monthly above-1% reports to include for `ownership changes`
required: false
default: "5"
permissions: permissions:
contents: write contents: write
@ -36,7 +40,8 @@ jobs:
--idx-bin ./target/debug/idx \ --idx-bin ./target/debug/idx \
--output-dir dist/ownership-snapshot \ --output-dir dist/ownership-snapshot \
--repo "${{ github.repository }}" \ --repo "${{ github.repository }}" \
--release-tag "${{ inputs.release_tag }}" --release-tag "${{ inputs.release_tag }}" \
--history "${{ inputs.history }}"
- name: Upload workflow artifact copy - name: Upload workflow artifact copy
uses: actions/upload-artifact@v4 uses: actions/upload-artifact@v4

6
AGENTS.md
View file

@ -57,13 +57,13 @@ Ownership is local-first after bootstrap.
Preferred bootstrap/update path: Preferred bootstrap/update path:
1. `idx ownership sync` 1. `idx ownership sync`
2. if no snapshot manifest is available: `idx ownership discover` then `idx ownership import --url <pdf-url>` 2. if no snapshot manifest is available: `idx ownership discover` then `idx ownership import --url <xlsx-or-pdf-url>`
3. local `--file` imports remain available for manual/fallback use 3. local `--file` imports remain available for manual/fallback use
Ownership input paths: Ownership input paths:
- primary remote source: discoverable IDX `above1` holder-register PDF - primary remote source: IDX `above1` holder register — monthly XLSX on the Data Kepemilikan Saham page (since the 2026-05-29 report), legacy PDF announcements before that; `ownership discover` checks both
- maintained snapshot path: `ownership sync` - maintained snapshot path: `ownership sync`
- local fallback path: PDF, plus local archive `.zip` / `.txt` - local file imports: IDX above-1% `.xlsx` (current format since June 2026), legacy `.pdf`, plus local archive `.zip` / `.txt`
Important scope note: Important scope note:
- archive ZIP/TXT ingest is a fallback/backstop path, not the primary product ingest surface - archive ZIP/TXT ingest is a fallback/backstop path, not the primary product ingest surface

22
docs/ARCHITECTURE.md
View file

@ -63,12 +63,32 @@ Unlike the `stocks` module (live-fetch), ownership is **import-then-query**:
4. Fully offline after import/sync 4. Fully offline after import/sync
### Data Sources ### Data Sources
- **KSEI** — official ≥1% shareholder registry (monthly PDF from IDX) - **KSEI** — official ≥1% shareholder registry, published monthly by IDX. Through May 2026 it was a PDF announcement; from the 2026-05-29 report onward it is an XLSX workbook on the IDX "Data Kepemilikan Saham" page
- **KSEI archive** — monthly ZIP/TXT balance-position matrix, used as a local fallback/backstop import path - **KSEI archive** — monthly ZIP/TXT balance-position matrix, used as a local fallback/backstop import path
- **Bing Finance** — global institutional ownership (REST API, quarterly) - **Bing Finance** — global institutional ownership (REST API, quarterly)
### Discovery
`idx ownership discover` merges two IDX sources:
- the announcement API (`GetAllAnnouncement`) for the legacy PDF announcements
- the Data Kepemilikan Saham page (`/id/perusahaan-tercatat/data-kepemilikan-saham/`),
whose server-rendered Nuxt payload lists every XLSX with a description such as
`Pemegang Saham di Atas 1% per 31 Agustus 2026`; the as-of date comes from that text
Only the above-1% XLSX is `supported`; the daily above-5% and investor-type workbooks
are listed as `unsupported`. Both URLs can be overridden for tests with
`IDX_OWNERSHIP_ANNOUNCEMENT_API_URL` and `IDX_OWNERSHIP_DATA_PAGE_URL`.
IDX sits behind Cloudflare, which accepts some curl-impersonate TLS profiles and
rejects others per path (for example `curl_chrome142` gets 403 on the data page).
IDX fetches therefore try several profiles until the response validates (JSON, page
payload, `%PDF`, or `PK` zip magic). `IDX_CURL_IMPERSONATE_BIN` pins a single profile.
### Parser Pipeline ### Parser Pipeline
``` ```
KSEI XLSX → zip + quick-xml (sharedStrings + sheet1) → exact 12-column header check
→ KseiHoldingDraft (Excel serial dates, numeric shares, percent → bps)
→ SQLite INSERT (within transaction)
KSEI PDF → mutool stext (XML with coordinates) → quick-xml parse → KseiRawRow KSEI PDF → mutool stext (XML with coordinates) → quick-xml parse → KseiRawRow
→ normalize (ID locale numbers, dates, entity names) → KseiHolding → normalize (ID locale numbers, dates, entity names) → KseiHolding
→ SQLite INSERT (within transaction) → SQLite INSERT (within transaction)

6
docs/OWNERSHIP_PUBLISH.md
View file

@ -45,9 +45,11 @@ nix develop --command scripts/build-latest-ownership-snapshot.sh \
The script performs these steps: The script performs these steps:
1. `idx -o json ownership discover --family above1 --limit 1` 1. `idx -o json ownership discover --family above1 --limit 50` (newest supported
report first; XLSX from the Data Kepemilikan Saham page since June 2026)
2. verifies the discovered report is the current supported import path 2. verifies the discovered report is the current supported import path
3. imports that PDF into an isolated temp ownership DB 3. imports that report (plus `--history <n>` earlier months, oldest first) into
an isolated temp ownership DB
4. checks that the imported release metadata is non-empty and tied to the same 4. checks that the imported release metadata is non-empty and tied to the same
source URL source URL
5. runs `scripts/build-ownership-snapshot.sh` to emit the SQLite artifact and 5. runs `scripts/build-ownership-snapshot.sh` to emit the SQLite artifact and

14
docs/OWNERSHIP_SELF_HOSTED.md
View file

@ -38,7 +38,9 @@ nix develop --command scripts/publish-ownership-snapshot.sh \
That helper: That helper:
1. optionally builds `idx` 1. optionally builds `idx`
2. discovers the latest supported IDX/KSEI PDF 2. discovers the latest supported IDX/KSEI above-1% report (XLSX on the IDX
Data Kepemilikan Saham page since the 2026-05-29 report; PDF announcements
before that)
3. imports it into an isolated temp DB 3. imports it into an isolated temp DB
4. emits the SQLite snapshot and manifest 4. emits the SQLite snapshot and manifest
5. ensures the stable GitHub release exists 5. ensures the stable GitHub release exists
@ -66,8 +68,14 @@ The sample timer uses:
- `RandomizedDelaySec=30m` - `RandomizedDelaySec=30m`
That is intentionally conservative. The ownership source is monthly, but the That is intentionally conservative. The ownership source is monthly, but the
exact publish day can drift. Start with an early-month schedule and adjust after exact publish day can drift. Observed XLSX upload times (HTTP Last-Modified):
observing a few real runs. 2026-06-03, 2026-07-02 03:55Z, 2026-08-02 01:50Z, 2026-09-02 03:51Z, so a run
on the 2nd can occasionally land before the file exists; the 3rd is safer.
Start with an early-month schedule and adjust after observing a few real runs.
Pass `--history <n>` (for example via `services.idxOwnershipPublish.extraArgs =
[ "--history" "5" ];` in `clan-private`) to include earlier months so
`idx ownership changes` works straight from the synced snapshot.
## Clan Integration Later ## Clan Integration Later

50
scripts/build-latest-ownership-snapshot.sh
View file

@ -6,7 +6,8 @@ usage() {
cat <<'EOF' cat <<'EOF'
Usage: scripts/build-latest-ownership-snapshot.sh --output-dir <dir> [options] Usage: scripts/build-latest-ownership-snapshot.sh --output-dir <dir> [options]
Discover the latest supported IDX/KSEI ownership PDF, import it into an isolated Discover the latest supported IDX/KSEI above-1% report (XLSX from the IDX Data
Kepemilikan Saham page, or a legacy PDF announcement), import it into an isolated
ownership database, and emit GitHub-release-ready snapshot artifacts plus manifest. ownership database, and emit GitHub-release-ready snapshot artifacts plus manifest.
Options: Options:
@ -17,6 +18,9 @@ Options:
(default: 0xrsydn/idx-cli) (default: 0xrsydn/idx-cli)
--release-tag <tag> Stable GitHub release tag used for the default base URL --release-tag <tag> Stable GitHub release tag used for the default base URL
(default: ownership-snapshot-current) (default: ownership-snapshot-current)
--history <n> Also import the <n> previous monthly above-1% XLSX
reports (oldest first) so `ownership changes` works
from the snapshot (default: 0)
--keep-workdir Keep the temp workdir instead of deleting it --keep-workdir Keep the temp workdir instead of deleting it
--help Show this help --help Show this help
EOF EOF
@ -28,6 +32,7 @@ BASE_URL=""
REPO_FULL_NAME="0xrsydn/idx-cli" REPO_FULL_NAME="0xrsydn/idx-cli"
RELEASE_TAG="ownership-snapshot-current" RELEASE_TAG="ownership-snapshot-current"
KEEP_WORKDIR="0" KEEP_WORKDIR="0"
HISTORY="0"
while [[ $# -gt 0 ]]; do while [[ $# -gt 0 ]]; do
case "$1" in case "$1" in
@ -51,6 +56,10 @@ while [[ $# -gt 0 ]]; do
RELEASE_TAG="${2:-}" RELEASE_TAG="${2:-}"
shift 2 shift 2
;; ;;
--history)
HISTORY="${2:-}"
shift 2
;;
--keep-workdir) --keep-workdir)
KEEP_WORKDIR="1" KEEP_WORKDIR="1"
shift shift
@ -73,6 +82,13 @@ if [[ -z "$OUTPUT_DIR" ]]; then
exit 2 exit 2
fi fi
if ! [[ "$HISTORY" =~ ^[0-9]+$ ]]; then
echo "--history must be a non-negative integer" >&2
exit 2
fi
# Canonicalize so "08" is 8, not an invalid octal literal in (( )).
HISTORY=$((10#$HISTORY))
if [[ -z "$BASE_URL" ]]; then if [[ -z "$BASE_URL" ]]; then
BASE_URL="https://github.com/${REPO_FULL_NAME}/releases/download/${RELEASE_TAG}" BASE_URL="https://github.com/${REPO_FULL_NAME}/releases/download/${RELEASE_TAG}"
fi fi
@ -115,8 +131,8 @@ RELEASES_JSON="$WORKDIR/releases.json"
MANIFEST_PATH="$OUTPUT_DIR/ownership-snapshot-manifest.json" MANIFEST_PATH="$OUTPUT_DIR/ownership-snapshot-manifest.json"
DB_PATH="$XDG_DATA_HOME/idx/ownership.db" DB_PATH="$XDG_DATA_HOME/idx/ownership.db"
printf 'Discovering latest supported IDX/KSEI ownership PDF...\n' printf 'Discovering latest supported IDX/KSEI above-1%% report...\n'
"$IDX_BIN" -o json ownership discover --family above1 --limit 1 > "$DISCOVERY_JSON" "$IDX_BIN" -o json ownership discover --family above1 --limit 50 > "$DISCOVERY_JSON"
discovery_payload="$( discovery_payload="$(
jq -r ' jq -r '
@ -151,7 +167,33 @@ DISCOVERED_TITLE="${discovery_fields[4]:-}"
DISCOVERED_PUBLISH_DATE="${discovery_fields[5]:-}" DISCOVERED_PUBLISH_DATE="${discovery_fields[5]:-}"
DISCOVERED_ORIGINAL_FILENAME="${discovery_fields[6]:-}" DISCOVERED_ORIGINAL_FILENAME="${discovery_fields[6]:-}"
printf 'Importing discovered PDF into isolated DB...\n' if (( HISTORY > 0 )); then
# Earlier supported XLSX months, one per as-of date, newest first; import oldest first.
# Run jq in a plain command substitution (not <(...)) so a jq failure
# stops the script under set -e instead of looking like zero months.
history_payload="$(
jq -r --arg latest "$DISCOVERED_PDF_URL" --argjson n "$HISTORY" '
(map(select(.pdf_url == $latest)) | .[0].as_of_date // "") as $latest_as_of
| [ .[]
| select(.status == "supported" and .format == "xlsx" and .as_of_date != null)
| select($latest_as_of == "" or .as_of_date < $latest_as_of) ]
| unique_by(.as_of_date)
| sort_by(.as_of_date) | reverse
| .[:$n] | reverse
| .[].pdf_url
' "$DISCOVERY_JSON"
)"
history_urls=()
if [[ -n "$history_payload" ]]; then
mapfile -t history_urls <<< "$history_payload"
fi
printf 'Importing %d earlier monthly report(s) for history...\n' "${#history_urls[@]}"
for url in "${history_urls[@]}"; do
"$IDX_BIN" ownership import --url "$url"
done
fi
printf 'Importing discovered report into isolated DB...\n'
"$IDX_BIN" ownership import --url "$DISCOVERED_PDF_URL" "$IDX_BIN" ownership import --url "$DISCOVERED_PDF_URL"
printf 'Inspecting imported release metadata...\n' printf 'Inspecting imported release metadata...\n'

8
scripts/publish-ownership-snapshot.sh
View file

@ -16,6 +16,8 @@ Options:
(default: 0xrsydn/idx-cli) (default: 0xrsydn/idx-cli)
--release-tag <tag> Stable GitHub release tag used for snapshot assets --release-tag <tag> Stable GitHub release tag used for snapshot assets
(default: ownership-snapshot-current) (default: ownership-snapshot-current)
--history <n> Also include the <n> previous monthly above-1% reports
in the snapshot (passed to the builder; default: 0)
--build Run `cargo build` before publishing --build Run `cargo build` before publishing
--keep-workdir Keep the temp workdir created by the builder helper --keep-workdir Keep the temp workdir created by the builder helper
--help Show this help --help Show this help
@ -28,6 +30,7 @@ REPO_FULL_NAME="0xrsydn/idx-cli"
RELEASE_TAG="ownership-snapshot-current" RELEASE_TAG="ownership-snapshot-current"
BUILD_FIRST="0" BUILD_FIRST="0"
KEEP_WORKDIR="0" KEEP_WORKDIR="0"
HISTORY="0"
PUBLISH_WORKDIR="" PUBLISH_WORKDIR=""
while [[ $# -gt 0 ]]; do while [[ $# -gt 0 ]]; do
@ -48,6 +51,10 @@ while [[ $# -gt 0 ]]; do
RELEASE_TAG="${2:-}" RELEASE_TAG="${2:-}"
shift 2 shift 2
;; ;;
--history)
HISTORY="${2:-}"
shift 2
;;
--build) --build)
BUILD_FIRST="1" BUILD_FIRST="1"
shift shift
@ -120,6 +127,7 @@ build_args=(
--output-dir "$PUBLISH_WORKDIR" --output-dir "$PUBLISH_WORKDIR"
--repo "$REPO_FULL_NAME" --repo "$REPO_FULL_NAME"
--release-tag "$RELEASE_TAG" --release-tag "$RELEASE_TAG"
--history "$HISTORY"
) )
if [[ "$KEEP_WORKDIR" == "1" ]]; then if [[ "$KEEP_WORKDIR" == "1" ]]; then

77
src/cli/ownership.rs
View file

@ -17,13 +17,13 @@ use crate::output::table::format_idr;
use crate::ownership::types::{ use crate::ownership::types::{
ChangeType, FlowSignal, HolderRow, KseiHolding, OwnershipRelease, OwnershipSource, ChangeType, FlowSignal, HolderRow, KseiHolding, OwnershipRelease, OwnershipSource,
}; };
use crate::ownership::{archive, db, entities, graph, parser, remote, search, snapshot}; use crate::ownership::{archive, db, entities, graph, parser, remote, search, snapshot, xlsx};
use crate::runtime; use crate::runtime;
#[derive(Debug, Args)] #[derive(Debug, Args)]
#[command( #[command(
about = "Ownership intelligence (KSEI + Bing)", about = "Ownership intelligence (KSEI + Bing)",
long_about = "Ownership intelligence (KSEI + Bing).\n\nPreferred bootstrap path:\n 1. Run `idx ownership sync` to install or refresh a maintained SQLite snapshot.\n 2. If no snapshot manifest is available, run `idx ownership discover` and then `idx ownership import --url <pdf-url>`.\n 3. Local `.pdf`, `.zip`, and `.txt` imports remain available for manual or fallback workflows.", long_about = "Ownership intelligence (KSEI + Bing).\n\nPreferred bootstrap path:\n 1. Run `idx ownership sync` to install or refresh a maintained SQLite snapshot.\n 2. If no snapshot manifest is available, run `idx ownership discover` and then `idx ownership import --url <pdf-url>`.\n 3. Local `.xlsx`, `.pdf`, `.zip`, and `.txt` imports remain available for manual or fallback workflows.",
after_help = "Examples:\n idx ownership sync\n idx ownership sync --manifest /path/to/ownership-snapshot-manifest.json\n idx ownership discover --limit 1\n idx ownership import --url <pdf-url>\n idx ownership releases" after_help = "Examples:\n idx ownership sync\n idx ownership sync --manifest /path/to/ownership-snapshot-manifest.json\n idx ownership discover --limit 1\n idx ownership import --url <pdf-url>\n idx ownership releases"
)] )]
pub struct OwnershipCmd { pub struct OwnershipCmd {
@ -35,13 +35,13 @@ pub struct OwnershipCmd {
pub enum OwnershipCommand { pub enum OwnershipCommand {
#[command( #[command(
about = "Discover the latest IDX-hosted ownership report URLs", about = "Discover the latest IDX-hosted ownership report URLs",
after_help = "Examples:\n idx ownership discover --limit 1\n idx ownership discover --family all --limit 6\n\nUse this when no snapshot manifest is available and you need a direct PDF URL for `idx ownership import --url`." after_help = "Examples:\n idx ownership discover --limit 1\n idx ownership discover --family all --limit 6\n\nUse this when no snapshot manifest is available and you need a direct XLSX or PDF URL for `idx ownership import --url`."
)] )]
Discover(DiscoverArgs), Discover(DiscoverArgs),
#[command( #[command(
about = "Import ownership data directly from source files", about = "Import ownership data directly from source files",
long_about = "Import ownership data directly from source files.\n\nPrefer `idx ownership sync` for normal bootstrap/update flows. Use `import` when you need a direct PDF import from IDX discovery output, a local PDF, or a local KSEI archive fallback file.", long_about = "Import ownership data directly from source files.\n\nPrefer `idx ownership sync` for normal bootstrap/update flows. Use `import` when you need a direct PDF import from IDX discovery output, a local PDF, or a local KSEI archive fallback file.",
after_help = "Examples:\n idx ownership import --url <pdf-url-from-discover>\n idx ownership import --file /path/to/ksei.pdf\n idx ownership import --file /path/to/BalanceposEfek20260227.zip" after_help = "Examples:\n idx ownership import --url <pdf-url-from-discover>\n idx ownership import --file /path/to/peng-2026-08-00017-satu-persen.xlsx\n idx ownership import --file /path/to/ksei.pdf\n idx ownership import --file /path/to/BalanceposEfek20260227.zip"
)] )]
Import(ImportArgs), Import(ImportArgs),
#[command( #[command(
@ -84,10 +84,10 @@ pub struct DiscoverArgs {
#[derive(Debug, Args)] #[derive(Debug, Args)]
pub struct ImportArgs { pub struct ImportArgs {
/// Direct URL to a remote ownership PDF, typically from `ownership discover`. /// Direct URL to a remote above-1% XLSX or PDF, typically from `ownership discover`.
#[arg(long)] #[arg(long)]
pub url: Option<String>, pub url: Option<String>,
/// Path to a local ownership file: PDF (primary), ZIP/TXT archive (fallback). /// Path to a local ownership file: above-1% XLSX or PDF (primary), ZIP/TXT archive (fallback).
#[arg(long)] #[arg(long)]
pub file: Option<PathBuf>, pub file: Option<PathBuf>,
/// Fetch Bing institutional data for these symbols. /// Fetch Bing institutional data for these symbols.
@ -836,6 +836,7 @@ fn handle_import(args: &ImportArgs, config: &IdxConfig) -> Result<(), IdxError>
drafts drafts
} }
ImportInputFormat::Archive => archive::parse_balancepos_file(&import_input.import_path)?, ImportInputFormat::Archive => archive::parse_balancepos_file(&import_input.import_path)?,
ImportInputFormat::Xlsx => xlsx::parse_above1_xlsx_file(&import_input.import_path)?,
}; };
let mut holdings = Vec::with_capacity(drafts.len()); let mut holdings = Vec::with_capacity(drafts.len());
@ -977,6 +978,7 @@ struct OwnershipImportResult {
enum ImportInputFormat { enum ImportInputFormat {
Pdf, Pdf,
Archive, Archive,
Xlsx,
} }
fn resolve_import_input(args: &ImportArgs) -> Result<Option<ResolvedImportInput>, IdxError> { fn resolve_import_input(args: &ImportArgs) -> Result<Option<ResolvedImportInput>, IdxError> {
@ -997,13 +999,16 @@ fn resolve_import_input(args: &ImportArgs) -> Result<Option<ResolvedImportInput>
if let Some(url) = &args.url { if let Some(url) = &args.url {
let trimmed = url.trim(); let trimmed = url.trim();
validate_import_url(trimmed)?; let format = validate_import_url(trimmed)?;
let target = cache_pdf_path(trimmed)?; let target = cache_download_path(trimmed, format)?;
download_pdf(trimmed, &target)?; match format {
ImportInputFormat::Xlsx => download_xlsx(trimmed, &target)?,
_ => download_pdf(trimmed, &target)?,
}
return Ok(Some(ResolvedImportInput { return Ok(Some(ResolvedImportInput {
import_path: target, import_path: target,
source_url: Some(trimmed.to_string()), source_url: Some(trimmed.to_string()),
format: ImportInputFormat::Pdf, format,
})); }));
} }
@ -1018,17 +1023,22 @@ fn detect_local_import_format(path: &Path) -> Result<ImportInputFormat, IdxError
.as_deref() .as_deref()
{ {
Some("pdf") => Ok(ImportInputFormat::Pdf), Some("pdf") => Ok(ImportInputFormat::Pdf),
Some("xlsx") if xlsx::supports_xlsx_file(path) => Ok(ImportInputFormat::Xlsx),
Some("zip") | Some("txt") if archive::supports_local_archive_file(path) => { Some("zip") | Some("txt") if archive::supports_local_archive_file(path) => {
Ok(ImportInputFormat::Archive) Ok(ImportInputFormat::Archive)
} }
_ => Err(IdxError::InvalidInput(format!( _ => Err(IdxError::InvalidInput(format!(
"unsupported local ownership file {}; expected a .pdf, .zip, or .txt input", "unsupported local ownership file {}; expected a .xlsx, .pdf, .zip, or .txt input",
path.display() path.display()
))), ))),
} }
} }
fn cache_pdf_path(url: &str) -> Result<PathBuf, IdxError> { fn cache_download_path(url: &str, format: ImportInputFormat) -> Result<PathBuf, IdxError> {
let extension = match format {
ImportInputFormat::Xlsx => ".xlsx",
_ => ".pdf",
};
let raw_dir = crate::cache::cache_dir()?.join("ownership").join("raw"); let raw_dir = crate::cache::cache_dir()?.join("ownership").join("raw");
fs::create_dir_all(&raw_dir).map_err(|e| IdxError::Io(e.to_string()))?; fs::create_dir_all(&raw_dir).map_err(|e| IdxError::Io(e.to_string()))?;
@ -1042,20 +1052,20 @@ fn cache_pdf_path(url: &str) -> Result<PathBuf, IdxError> {
.to_string(); .to_string();
if file_name.is_empty() || file_name == "/" { if file_name.is_empty() || file_name == "/" {
file_name = format!("ksei-{}.pdf", Utc::now().timestamp()); file_name = format!("ksei-{}{extension}", Utc::now().timestamp());
} }
if !file_name.to_ascii_lowercase().ends_with(".pdf") { if !file_name.to_ascii_lowercase().ends_with(extension) {
file_name.push_str(".pdf"); file_name.push_str(extension);
} }
Ok(raw_dir.join(file_name)) Ok(raw_dir.join(file_name))
} }
fn validate_import_url(url: &str) -> Result<(), IdxError> { fn validate_import_url(url: &str) -> Result<ImportInputFormat, IdxError> {
let trimmed = url.trim(); let trimmed = url.trim();
if trimmed.is_empty() { if trimmed.is_empty() {
return Err(IdxError::InvalidInput( return Err(IdxError::InvalidInput(
"ownership import --url accepts direct PDF URLs only".to_string(), "ownership import --url accepts direct XLSX or PDF URLs only".to_string(),
)); ));
} }
@ -1065,11 +1075,14 @@ fn validate_import_url(url: &str) -> Result<(), IdxError> {
.next() .next()
.unwrap_or(normalized.as_str()); .unwrap_or(normalized.as_str());
if normalized.ends_with(".pdf") { if normalized.ends_with(".pdf") {
return Ok(()); return Ok(ImportInputFormat::Pdf);
}
if normalized.ends_with(".xlsx") {
return Ok(ImportInputFormat::Xlsx);
} }
Err(IdxError::InvalidInput( Err(IdxError::InvalidInput(
"ownership import --url accepts direct PDF URLs only; run `idx ownership discover` first to find the current supported attachment".to_string(), "ownership import --url accepts direct XLSX or PDF URLs only; run `idx ownership discover` first to find the current supported file".to_string(),
)) ))
} }
@ -1105,6 +1118,32 @@ fn download_pdf(url: &str, target: &Path) -> Result<(), IdxError> {
Ok(()) Ok(())
} }
fn download_xlsx(url: &str, target: &Path) -> Result<(), IdxError> {
if is_idx_url(url) {
return remote::download_idx_xlsx(url, target);
}
let response = ureq::get(url)
.header(
"Accept",
"application/vnd.openxmlformats-officedocument.spreadsheetml.sheet,application/octet-stream,*/*;q=0.8",
)
.call()
.map_err(|e| IdxError::Http(format!("failed to download XLSX: {e}")))?;
let bytes = response
.into_body()
.read_to_vec()
.map_err(|e| IdxError::Http(format!("failed reading XLSX body: {e}")))?;
remote::validate_xlsx_payload(&bytes)?;
fs::write(target, &bytes).map_err(|e| {
IdxError::Io(format!(
"failed writing cached XLSX {}: {e}",
target.display()
))
})
}
fn is_idx_url(url: &str) -> bool { fn is_idx_url(url: &str) -> bool {
let normalized = url.trim().to_ascii_lowercase(); let normalized = url.trim().to_ascii_lowercase();
normalized.starts_with("https://www.idx.co.id/") normalized.starts_with("https://www.idx.co.id/")

60
src/curl_impersonate.rs
View file

@ -40,7 +40,65 @@ pub fn run(stage: &str, args: &[&str]) -> Result<Output, IdxError> {
pub fn run_owned(stage: &str, args: &[String]) -> Result<Output, IdxError> { pub fn run_owned(stage: &str, args: &[String]) -> Result<Output, IdxError> {
let binary = chrome_curl_binary()?; let binary = chrome_curl_binary()?;
let output = Command::new(&binary).args(args).output().map_err(|e| { run_binary(stage, &binary, args)
}
/// Browser profiles tried, in order, when a site's bot protection rejects
/// some TLS fingerprints but not others (IDX's Cloudflare rules block
/// `curl_chrome142` on some pages while Firefox/Safari profiles pass).
const FALLBACK_PROFILES: &[&str] = &[
"curl_chrome142",
"curl_firefox144",
"curl_chrome136",
"curl_firefox135",
"curl_safari184",
"curl_chrome131",
"curl_chrome124",
"curl_chrome120",
"curl_chrome116",
];
const FALLBACK_ROUNDS: usize = 2;
/// Run curl with each available impersonation profile until `accept` approves
/// the response body. `IDX_CURL_IMPERSONATE_BIN`, when set, is the only
/// profile used. Returns the first accepted output, or the last error.
pub fn run_owned_with_fallback(
stage: &str,
args: &[String],
accept: &dyn Fn(&[u8]) -> Result<(), IdxError>,
) -> Result<Output, IdxError> {
let profiles: Vec<String> = match std::env::var(OVERRIDE_ENV) {
Ok(value) if !value.trim().is_empty() => vec![value.trim().to_string()],
_ => FALLBACK_PROFILES
.iter()
.copied()
.filter(|bin| Command::new(bin).arg("--version").output().is_ok())
.map(str::to_string)
.collect(),
};
if profiles.is_empty() {
return Err(IdxError::Http(format!(
"no curl-impersonate binary found; set {OVERRIDE_ENV} or install nixpkgs#curl-impersonate"
)));
}
let mut last_error = None;
for _ in 0..FALLBACK_ROUNDS {
for binary in &profiles {
match run_binary(stage, binary, args) {
Ok(output) => match accept(&output.stdout) {
Ok(()) => return Ok(output),
Err(err) => last_error = Some(err),
},
Err(err) => last_error = Some(err),
}
}
}
Err(last_error.expect("at least one profile was tried"))
}
fn run_binary(stage: &str, binary: &str, args: &[String]) -> Result<Output, IdxError> {
let output = Command::new(binary).args(args).output().map_err(|e| {
if e.kind() == std::io::ErrorKind::NotFound { if e.kind() == std::io::ErrorKind::NotFound {
return IdxError::Http(format!( return IdxError::Http(format!(
"curl-impersonate binary '{binary}' not found; set {OVERRIDE_ENV} or install nixpkgs#curl-impersonate-chrome" "curl-impersonate binary '{binary}' not found; set {OVERRIDE_ENV} or install nixpkgs#curl-impersonate-chrome"

4
src/ownership/entities.rs
View file

@ -212,7 +212,7 @@ pub fn resolve_entity(
Ok(entity_id) Ok(entity_id)
} }
fn normalize_investor_type(raw: &str) -> Option<InvestorTypeCode> { pub(crate) fn normalize_investor_type(raw: &str) -> Option<InvestorTypeCode> {
let value = raw.trim(); let value = raw.trim();
if value.is_empty() { if value.is_empty() {
None None
@ -221,7 +221,7 @@ fn normalize_investor_type(raw: &str) -> Option<InvestorTypeCode> {
} }
} }
fn normalize_locality(raw: &str) -> Option<Locality> { pub(crate) fn normalize_locality(raw: &str) -> Option<Locality> {
match raw.trim().to_uppercase().as_str() { match raw.trim().to_uppercase().as_str() {
"L" | "D" => Some(Locality::Local), "L" | "D" => Some(Locality::Local),
"F" | "A" => Some(Locality::Foreign), "F" | "A" => Some(Locality::Foreign),

1
src/ownership/mod.rs
View file

@ -7,3 +7,4 @@ pub mod remote;
pub mod search; pub mod search;
pub mod snapshot; pub mod snapshot;
pub mod types; pub mod types;
pub mod xlsx;

467
src/ownership/remote.rs
View file

@ -12,6 +12,10 @@ const DEFAULT_IDX_ANNOUNCEMENT_API_URL: &str =
const IDX_ANNOUNCEMENT_API_ENV: &str = "IDX_OWNERSHIP_ANNOUNCEMENT_API_URL"; const IDX_ANNOUNCEMENT_API_ENV: &str = "IDX_OWNERSHIP_ANNOUNCEMENT_API_URL";
const IDX_ANNOUNCEMENT_LISTING_ENV: &str = "IDX_OWNERSHIP_ANNOUNCEMENT_PAGE_URL"; const IDX_ANNOUNCEMENT_LISTING_ENV: &str = "IDX_OWNERSHIP_ANNOUNCEMENT_PAGE_URL";
const IDX_ANNOUNCEMENT_PAGE_SIZE: usize = 10; const IDX_ANNOUNCEMENT_PAGE_SIZE: usize = 10;
/// Since June 2026 the monthly above-1% list (XLSX) is published only here.
pub const IDX_SHARE_OWNERSHIP_PAGE_URL: &str =
"https://www.idx.co.id/id/perusahaan-tercatat/data-kepemilikan-saham/";
const IDX_SHARE_OWNERSHIP_PAGE_ENV: &str = "IDX_OWNERSHIP_DATA_PAGE_URL";
const USER_AGENT: &str = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"; const USER_AGENT: &str = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36";
#[derive(Debug, Clone, Deserialize)] #[derive(Debug, Clone, Deserialize)]
@ -116,6 +120,12 @@ pub struct DiscoveredOwnershipPdf {
pub code: Option<String>, pub code: Option<String>,
pub original_filename: Option<String>, pub original_filename: Option<String>,
pub is_attachment: bool, pub is_attachment: bool,
/// File format of `pdf_url`: `pdf` (announcement attachment) or `xlsx`
/// (Data Kepemilikan Saham page).
pub format: String,
/// Report as-of date (`YYYY-MM-DD`) when the source states it. The data
/// page shows no publish date, so its entries use this as `publish_date`.
pub as_of_date: Option<String>,
} }
struct DiscoveryQuery { struct DiscoveryQuery {
@ -152,6 +162,13 @@ pub fn announcement_listing_url() -> String {
.unwrap_or_else(|| IDX_ANNOUNCEMENT_LISTING_URL.to_string()) .unwrap_or_else(|| IDX_ANNOUNCEMENT_LISTING_URL.to_string())
} }
pub fn share_ownership_page_url() -> String {
std::env::var(IDX_SHARE_OWNERSHIP_PAGE_ENV)
.ok()
.filter(|value| !value.trim().is_empty())
.unwrap_or_else(|| IDX_SHARE_OWNERSHIP_PAGE_URL.to_string())
}
pub fn announcement_api_url() -> String { pub fn announcement_api_url() -> String {
std::env::var(IDX_ANNOUNCEMENT_API_ENV) std::env::var(IDX_ANNOUNCEMENT_API_ENV)
.ok() .ok()
@ -240,6 +257,8 @@ pub fn select_latest_ownership_reports(
code: clean_option(item.code.as_deref()), code: clean_option(item.code.as_deref()),
original_filename, original_filename,
is_attachment, is_attachment,
format: "pdf".to_string(),
as_of_date: None,
} }
}) })
.collect::<Vec<_>>(); .collect::<Vec<_>>();
@ -278,6 +297,7 @@ pub fn discover_idx_ownership_reports(
"IDX ownership announcement discovery", "IDX ownership announcement discovery",
&query_url, &query_url,
&json_headers(), &json_headers(),
&validate_json_payload,
) )
.and_then(|raw| parse_announcement_page(&raw)) .and_then(|raw| parse_announcement_page(&raw))
.and_then(|page| select_latest_ownership_reports(&page, &query_url, query.family)) .and_then(|page| select_latest_ownership_reports(&page, &query_url, query.family))
@ -287,6 +307,45 @@ pub fn discover_idx_ownership_reports(
} }
} }
let mut page_error = None;
match fetch_text(
"IDX share ownership data page discovery",
&share_ownership_page_url(),
&page_headers(),
&validate_share_ownership_page,
) {
Ok(html) => {
let mut page_reports = parse_share_ownership_page(&html, &share_ownership_page_url());
if page_reports.is_empty() {
// The page loaded (it has its file list) but nothing parsed:
// the payload shape changed. Say so instead of silently
// falling back to announcement-only results.
page_error = Some(format!(
"IDX share ownership data page {} loaded but no report entries could be parsed; the page layout may have changed",
share_ownership_page_url()
));
}
page_reports
.retain(|report| family_filter.is_none_or(|family| family == report.family));
// Keep the full above-1% history (snapshot backfill needs it), but
// only the newest entry of the unsupported daily/breakdown series.
page_reports.sort_by(|left, right| right.publish_date.cmp(&left.publish_date));
let mut seen_unsupported = Vec::new();
page_reports.retain(|report| {
if report.family == OwnershipReportFamily::AboveOnePercent {
return true;
}
if seen_unsupported.contains(&report.family) {
return false;
}
seen_unsupported.push(report.family);
true
});
discovered.extend(page_reports);
}
Err(err) => page_error = Some(err.to_string()),
}
discovered.sort_by(|left, right| { discovered.sort_by(|left, right| {
right right
.publish_date .publish_date
@ -300,6 +359,15 @@ pub fn discover_idx_ownership_reports(
discovered.truncate(limit); discovered.truncate(limit);
} }
if !discovered.is_empty()
&& let Some(err) = &page_error
{
// Announcement results still exist, but since June 2026 the newest
// above-1% reports are only on the data page: don't hide its failure.
crate::runtime::warn(err);
}
errors.extend(page_error);
if discovered.is_empty() { if discovered.is_empty() {
let detail = if errors.is_empty() { let detail = if errors.is_empty() {
"no matching ownership reports found".to_string() "no matching ownership reports found".to_string()
@ -307,8 +375,9 @@ pub fn discover_idx_ownership_reports(
errors.join("; ") errors.join("; ")
}; };
return Err(IdxError::Http(format!( return Err(IdxError::Http(format!(
"failed to discover IDX ownership reports from {}: {detail}", "failed to discover IDX ownership reports from {} or {}: {detail}",
announcement_listing_url() announcement_listing_url(),
share_ownership_page_url()
))); )));
} }
@ -316,8 +385,12 @@ pub fn discover_idx_ownership_reports(
} }
pub fn download_idx_pdf(url: &str, target: &Path) -> Result<(), IdxError> { pub fn download_idx_pdf(url: &str, target: &Path) -> Result<(), IdxError> {
let bytes = fetch_bytes("IDX ownership PDF download", url, &pdf_headers())?; let bytes = fetch_bytes(
validate_pdf_payload(&bytes)?; "IDX ownership PDF download",
url,
&pdf_headers(),
&validate_pdf_payload,
)?;
fs::write(target, &bytes).map_err(|e| { fs::write(target, &bytes).map_err(|e| {
IdxError::Io(format!( IdxError::Io(format!(
@ -329,6 +402,239 @@ pub fn download_idx_pdf(url: &str, target: &Path) -> Result<(), IdxError> {
Ok(()) Ok(())
} }
pub fn download_idx_xlsx(url: &str, target: &Path) -> Result<(), IdxError> {
let bytes = fetch_bytes(
"IDX ownership XLSX download",
url,
&xlsx_headers(),
&validate_xlsx_payload,
)?;
fs::write(target, &bytes).map_err(|e| {
IdxError::Io(format!(
"failed writing cached XLSX {}: {e}",
target.display()
))
})?;
Ok(())
}
/// XLSX is a ZIP container; Cloudflare challenges come back as HTML.
pub fn validate_xlsx_payload(bytes: &[u8]) -> Result<(), IdxError> {
if bytes.starts_with(b"PK\x03\x04") {
return Ok(());
}
if looks_like_html(bytes) {
return Err(IdxError::Http(
"IDX ownership download returned HTML (likely a Cloudflare challenge) instead of an XLSX workbook".to_string(),
));
}
Err(IdxError::Http(
"IDX ownership download did not look like an XLSX workbook".to_string(),
))
}
fn validate_json_payload(bytes: &[u8]) -> Result<(), IdxError> {
if looks_like_html(bytes) {
return Err(IdxError::Http(
"IDX ownership discovery returned HTML (likely a Cloudflare challenge) instead of announcement JSON".to_string(),
));
}
Ok(())
}
fn validate_share_ownership_page(bytes: &[u8]) -> Result<(), IdxError> {
if contains_bytes(bytes, b"Prospectus") {
return Ok(());
}
Err(IdxError::Http(
"IDX share ownership data page did not contain its file list (likely a Cloudflare challenge)".to_string(),
))
}
fn looks_like_html(bytes: &[u8]) -> bool {
let head: Vec<u8> = bytes
.iter()
.skip_while(|byte| byte.is_ascii_whitespace())
.take(256)
.map(u8::to_ascii_lowercase)
.collect();
head.starts_with(b"<!doctype html") || head.starts_with(b"<html")
}
fn contains_bytes(haystack: &[u8], needle: &[u8]) -> bool {
haystack
.windows(needle.len())
.any(|window| window == needle)
}
/// Parse the Data Kepemilikan Saham page. The page is server-rendered by
/// Nuxt and embeds its file list as JS object literals such as
/// `{Description:"Pemegang Saham di Atas 1% per 31 Agustus 2026",
/// Prospectus:"https:\u002F\u002Fwww.idx.co.id\u002FMedia\u002F...xlsx",...}`.
pub fn parse_share_ownership_page(html: &str, page_url: &str) -> Vec<DiscoveredOwnershipPdf> {
let mut reports = Vec::new();
let mut rest = html;
while let Some(start) = find_js_key(rest, "Description") {
let (description, after_description) = match read_js_string(&rest[start..]) {
Some(value) => value,
None => {
rest = &rest[start..];
continue;
}
};
let object_tail = &rest[start + after_description..];
// The URL belongs to the same object literal, so it must come before
// the next Description key.
let window_end = find_js_key(object_tail, "Description").unwrap_or(object_tail.len());
let window = &object_tail[..window_end];
rest = object_tail;
let Some(url_start) = find_js_key(window, "Prospectus") else {
continue;
};
let Some((url, _)) = read_js_string(&window[url_start..]) else {
continue;
};
if let Some(report) = classify_page_entry(description.trim(), url.trim(), page_url) {
reports.push(report);
}
}
reports
}
/// Offset just past `Key:` / `"Key":` (the opening quote of the value).
fn find_js_key(haystack: &str, key: &str) -> Option<usize> {
let bare = format!("{key}:\"");
let quoted = format!("\"{key}\":\"");
[bare, quoted]
.iter()
.filter_map(|needle| {
haystack
.find(needle.as_str())
.map(|index| index + needle.len() - 1)
})
.min()
}
/// Read a JS/JSON string literal starting at the opening quote. Returns the
/// decoded value and the byte offset just past the closing quote.
fn read_js_string(input: &str) -> Option<(String, usize)> {
let mut chars = input.char_indices();
if chars.next()?.1 != '"' {
return None;
}
let mut value = String::new();
while let Some((index, ch)) = chars.next() {
match ch {
'"' => return Some((value, index + 1)),
'\\' => match chars.next()?.1 {
'u' => {
let hex: String = (0..4)
.filter_map(|_| chars.next().map(|(_, c)| c))
.collect();
value.push(char::from_u32(u32::from_str_radix(&hex, 16).ok()?)?);
}
'n' => value.push('\n'),
't' => value.push('\t'),
other => value.push(other),
},
_ => value.push(ch),
}
}
None
}
fn classify_page_entry(
description: &str,
url: &str,
page_url: &str,
) -> Option<DiscoveredOwnershipPdf> {
let upper = description.to_uppercase();
let family = if upper.contains("DI ATAS 1%") || upper.contains("ABOVE 1%") {
OwnershipReportFamily::AboveOnePercent
} else if upper.contains("DI ATAS 5%") || upper.contains("ABOVE 5%") {
OwnershipReportFamily::AboveFivePercent
} else if upper.contains("TIPE INVESTOR")
|| upper.contains("KLASIFIKASI")
|| upper.contains("INVESTOR TYPE")
|| upper.contains("CLASSIFICATION")
{
OwnershipReportFamily::InvestorTypeBreakdown
} else {
return None;
};
let path = url.split(['?', '#']).next().unwrap_or_default();
let file_name = path.rsplit('/').next().unwrap_or_default().to_string();
let lower_name = file_name.to_ascii_lowercase();
let format = if lower_name.ends_with(".xlsx") {
"xlsx"
} else if lower_name.ends_with(".pdf") {
"pdf"
} else {
return None;
};
let status = if family == OwnershipReportFamily::AboveOnePercent && format == "xlsx" {
OwnershipReportStatus::Supported
} else {
OwnershipReportStatus::Unsupported
};
// Without an as-of date the entry cannot be ordered against other
// months, and the snapshot manifest needs a date, so skip it.
let as_of_date = parse_indonesian_as_of_date(description)?.to_string();
Some(DiscoveredOwnershipPdf {
family,
status,
listing_page_url: page_url.to_string(),
query_url: page_url.to_string(),
pdf_url: url.to_string(),
title: description.to_string(),
publish_date: as_of_date.clone(),
code: None,
original_filename: (!file_name.is_empty()).then_some(file_name),
is_attachment: true,
format: format.to_string(),
as_of_date: Some(as_of_date),
})
}
/// `"... per 31 Agustus 2026"` -> 2026-08-31 (Indonesian or English months).
fn parse_indonesian_as_of_date(description: &str) -> Option<chrono::NaiveDate> {
let tokens: Vec<String> = description
.split_whitespace()
.map(|token| {
token
.trim_matches(|c: char| !c.is_alphanumeric())
.to_lowercase()
})
.collect();
tokens.windows(3).rev().find_map(|window| {
let day = window[0].parse::<u32>().ok()?;
let month = match window[1].as_str() {
"januari" | "january" | "jan" => 1,
"februari" | "february" | "feb" => 2,
"maret" | "march" | "mar" => 3,
"april" | "apr" => 4,
"mei" | "may" => 5,
"juni" | "june" | "jun" => 6,
"juli" | "july" | "jul" => 7,
"agustus" | "august" | "agu" | "aug" => 8,
"september" | "sep" => 9,
"oktober" | "october" | "okt" | "oct" => 10,
"november" | "nov" => 11,
"desember" | "december" | "des" | "dec" => 12,
_ => return None,
};
let year = window[2].parse::<i32>().ok()?;
chrono::NaiveDate::from_ymd_opt(year, month, day)
})
}
pub fn validate_pdf_payload(bytes: &[u8]) -> Result<(), IdxError> { pub fn validate_pdf_payload(bytes: &[u8]) -> Result<(), IdxError> {
if bytes.is_empty() { if bytes.is_empty() {
return Err(IdxError::Http( return Err(IdxError::Http(
@ -357,13 +663,23 @@ pub fn validate_pdf_payload(bytes: &[u8]) -> Result<(), IdxError> {
)) ))
} }
fn fetch_text(stage: &str, url: &str, headers: &[(String, String)]) -> Result<String, IdxError> { fn fetch_text(
let bytes = fetch_bytes(stage, url, headers)?; stage: &str,
url: &str,
headers: &[(String, String)],
accept: &dyn Fn(&[u8]) -> Result<(), IdxError>,
) -> Result<String, IdxError> {
let bytes = fetch_bytes(stage, url, headers, accept)?;
String::from_utf8(bytes) String::from_utf8(bytes)
.map_err(|e| IdxError::Http(format!("failed to decode {stage} response as utf-8: {e}"))) .map_err(|e| IdxError::Http(format!("failed to decode {stage} response as utf-8: {e}")))
} }
fn fetch_bytes(stage: &str, url: &str, headers: &[(String, String)]) -> Result<Vec<u8>, IdxError> { fn fetch_bytes(
stage: &str,
url: &str,
headers: &[(String, String)],
accept: &dyn Fn(&[u8]) -> Result<(), IdxError>,
) -> Result<Vec<u8>, IdxError> {
let mut args = vec![ let mut args = vec![
"--silent".to_string(), "--silent".to_string(),
"--show-error".to_string(), "--show-error".to_string(),
@ -377,7 +693,7 @@ fn fetch_bytes(stage: &str, url: &str, headers: &[(String, String)]) -> Result<V
} }
args.push(url.to_string()); args.push(url.to_string());
let output = curl_impersonate::run_owned(stage, &args)?; let output = curl_impersonate::run_owned_with_fallback(stage, &args, accept)?;
Ok(output.stdout) Ok(output.stdout)
} }
@ -396,6 +712,35 @@ fn json_headers() -> Vec<(String, String)> {
] ]
} }
fn page_headers() -> Vec<(String, String)> {
vec![
("User-Agent".to_string(), USER_AGENT.to_string()),
(
"Accept".to_string(),
"text/html,application/xhtml+xml,*/*;q=0.8".to_string(),
),
(
"Accept-Language".to_string(),
"id-ID,id;q=0.9,en-US;q=0.8,en;q=0.7".to_string(),
),
]
}
fn xlsx_headers() -> Vec<(String, String)> {
vec![
("User-Agent".to_string(), USER_AGENT.to_string()),
(
"Accept".to_string(),
"application/vnd.openxmlformats-officedocument.spreadsheetml.sheet,application/octet-stream,*/*;q=0.8".to_string(),
),
(
"Accept-Language".to_string(),
"id-ID,id;q=0.9,en-US;q=0.8,en;q=0.7".to_string(),
),
("Referer".to_string(), share_ownership_page_url()),
]
}
fn pdf_headers() -> Vec<(String, String)> { fn pdf_headers() -> Vec<(String, String)> {
vec![ vec![
("User-Agent".to_string(), USER_AGENT.to_string()), ("User-Agent".to_string(), USER_AGENT.to_string()),
@ -502,7 +847,8 @@ mod tests {
use super::{ use super::{
AnnouncementPage, IDX_ANNOUNCEMENT_LISTING_URL, OwnershipReportFamily, AnnouncementPage, IDX_ANNOUNCEMENT_LISTING_URL, OwnershipReportFamily,
OwnershipReportStatus, build_announcement_query_url, parse_announcement_page, OwnershipReportStatus, build_announcement_query_url, parse_announcement_page,
select_latest_ownership_reports, validate_pdf_payload, parse_indonesian_as_of_date, parse_share_ownership_page, select_latest_ownership_reports,
validate_pdf_payload, validate_share_ownership_page, validate_xlsx_payload,
}; };
use crate::error::IdxError; use crate::error::IdxError;
@ -762,4 +1108,107 @@ mod tests {
assert!(matches!(err, IdxError::Http(_))); assert!(matches!(err, IdxError::Http(_)));
assert!(err.to_string().contains("PDF/direct attachment")); assert!(err.to_string().contains("PDF/direct attachment"));
} }
#[test]
fn parses_share_ownership_page_entries() {
let page_url = "https://www.idx.co.id/id/perusahaan-tercatat/data-kepemilikan-saham/";
let reports = parse_share_ownership_page(
include_str!("../../tests/fixtures/idx_share_ownership_page_excerpt.html"),
page_url,
);
assert_eq!(reports.len(), 7);
let above1: Vec<_> = reports
.iter()
.filter(|r| r.family == OwnershipReportFamily::AboveOnePercent)
.collect();
assert_eq!(above1.len(), 3);
assert!(
above1
.iter()
.all(|r| r.status == OwnershipReportStatus::Supported)
);
assert!(above1.iter().all(|r| r.format == "xlsx"));
let august = above1
.iter()
.find(|r| r.as_of_date.as_deref() == Some("2026-08-31"))
.expect("August above-1% entry");
assert_eq!(
august.pdf_url,
"https://www.idx.co.id/Media/fahlw1o2/peng-2026-08-00017-satu-persen.xlsx"
);
assert_eq!(august.publish_date, "2026-08-31");
assert_eq!(august.listing_page_url, page_url);
assert_eq!(
august.original_filename.as_deref(),
Some("peng-2026-08-00017-satu-persen.xlsx")
);
let above5 = reports
.iter()
.filter(|r| r.family == OwnershipReportFamily::AboveFivePercent)
.count();
let breakdown = reports
.iter()
.filter(|r| r.family == OwnershipReportFamily::InvestorTypeBreakdown)
.count();
assert_eq!((above5, breakdown), (2, 2));
assert!(
reports
.iter()
.filter(|r| r.family != OwnershipReportFamily::AboveOnePercent)
.all(|r| r.status == OwnershipReportStatus::Unsupported)
);
}
#[test]
fn share_ownership_page_handles_query_strings_and_dateless_entries() {
let html = r#"{Description:"Pemegang Saham di Atas 1% per 31 Agustus 2026",Prospectus:"https://www.idx.co.id/Media/x/peng-satu-persen.xlsx?download=1",ListingDate:r},{Description:"Pemegang Saham di Atas 1%",Prospectus:"https://www.idx.co.id/Media/y/no-date.xlsx",ListingDate:r}"#;
let reports = parse_share_ownership_page(html, "p");
assert_eq!(
reports.len(),
1,
"dateless entry skipped, query-string entry kept"
);
assert_eq!(reports[0].format, "xlsx");
assert_eq!(reports[0].status, OwnershipReportStatus::Supported);
assert_eq!(
reports[0].original_filename.as_deref(),
Some("peng-satu-persen.xlsx")
);
assert_eq!(reports[0].publish_date, "2026-08-31");
}
#[test]
fn share_ownership_page_without_payload_yields_nothing() {
assert!(parse_share_ownership_page("<html>Just a moment...</html>", "p").is_empty());
assert!(validate_share_ownership_page(b"<html>Just a moment...</html>").is_err());
}
#[test]
fn parses_indonesian_and_english_as_of_dates() {
let date = |y, m, d| chrono::NaiveDate::from_ymd_opt(y, m, d);
assert_eq!(
parse_indonesian_as_of_date("Pemegang Saham di Atas 1% per 31 Agustus 2026"),
date(2026, 8, 31)
);
assert_eq!(
parse_indonesian_as_of_date("Shareholders above 1% as of 29 May 2026"),
date(2026, 5, 29)
);
assert_eq!(
parse_indonesian_as_of_date("Pemegang Saham di Atas 1%"),
None
);
}
#[test]
fn validates_xlsx_payloads() {
assert!(validate_xlsx_payload(b"PK\x03\x04rest-of-zip").is_ok());
let err = validate_xlsx_payload(b"<!DOCTYPE html><title>Just a moment...</title>")
.expect_err("html challenge rejected");
assert!(err.to_string().contains("Cloudflare"), "{err}");
assert!(validate_xlsx_payload(b"%PDF-1.7").is_err());
}
} }

554
src/ownership/xlsx.rs Normal file
View file

@ -0,0 +1,554 @@
//! Parser for the IDX "Pemegang Saham di Atas 1%" XLSX workbook.
//!
//! Since June 2026 IDX publishes the monthly above-1% holder register as an
//! XLSX file on the "Data Kepemilikan Saham" page instead of a PDF
//! announcement. The workbook has one sheet: a disclaimer block, a `[PUBLIC]`
//! marker, then a fixed 12-column table. Only that exact header is accepted,
//! so the daily above-5% workbook and the investor-type / classification
//! workbooks published on the same page are rejected instead of mis-imported.
use std::collections::HashMap;
use std::fs;
use std::io::{Cursor, Read};
use std::path::Path;
use chrono::{Duration, NaiveDate};
use quick_xml::Reader;
use quick_xml::events::{BytesStart, Event};
use crate::error::IdxError;
use crate::ownership::entities::{normalize_investor_type, normalize_locality, parse_ksei_date};
use crate::ownership::types::KseiHoldingDraft;
const EXPECTED_HEADER: [&str; 12] = [
"DATE",
"SHARE_CODE",
"ISSUER_NAME",
"INVESTOR_NAME",
"INVESTOR_CLASSIFICATION",
"LOCAL_FOREIGN",
"NATIONALITY",
"DOMICILE",
"HOLDINGS_SCRIPLESS",
"HOLDINGS_SCRIP",
"TOTAL_HOLDING_SHARES",
"PERCENTAGE",
];
/// Rows above this line are disclaimer text; the header must appear before it.
const MAX_HEADER_SEARCH_ROWS: usize = 50;
pub fn supports_xlsx_file(path: &Path) -> bool {
path.extension()
.and_then(|value| value.to_str())
.is_some_and(|value| value.trim().eq_ignore_ascii_case("xlsx"))
}
pub fn parse_above1_xlsx_file(path: &Path) -> Result<Vec<KseiHoldingDraft>, IdxError> {
let bytes = fs::read(path).map_err(|e| {
IdxError::Io(format!(
"failed to read ownership XLSX {}: {e}",
path.display()
))
})?;
parse_above1_xlsx_bytes(&bytes)
}
pub fn parse_above1_xlsx_bytes(bytes: &[u8]) -> Result<Vec<KseiHoldingDraft>, IdxError> {
let mut zip = zip::ZipArchive::new(Cursor::new(bytes))
.map_err(|e| IdxError::ParseError(format!("failed to open ownership XLSX: {e}")))?;
let shared_strings = match read_zip_entry(&mut zip, "xl/sharedStrings.xml")? {
Some(xml) => parse_shared_strings(&xml)?,
None => Vec::new(),
};
let sheet_name = first_worksheet_name(&mut zip)?;
let sheet_xml = read_zip_entry(&mut zip, &sheet_name)?
.ok_or_else(|| IdxError::ParseError(format!("ownership XLSX is missing {sheet_name}")))?;
let rows = parse_sheet_rows(&sheet_xml, &shared_strings)?;
drafts_from_rows(&rows)
}
fn read_zip_entry(
zip: &mut zip::ZipArchive<Cursor<&[u8]>>,
name: &str,
) -> Result<Option<String>, IdxError> {
let mut file = match zip.by_name(name) {
Ok(file) => file,
Err(zip::result::ZipError::FileNotFound) => return Ok(None),
Err(e) => {
return Err(IdxError::ParseError(format!(
"failed to read ownership XLSX entry {name}: {e}"
)));
}
};
let mut output = String::new();
file.read_to_string(&mut output).map_err(|e| {
IdxError::ParseError(format!("failed to decode ownership XLSX entry {name}: {e}"))
})?;
Ok(Some(output))
}
fn first_worksheet_name(zip: &mut zip::ZipArchive<Cursor<&[u8]>>) -> Result<String, IdxError> {
let mut sheets: Vec<String> = zip
.file_names()
.filter(|name| name.starts_with("xl/worksheets/") && name.ends_with(".xml"))
.filter(|name| !name["xl/worksheets/".len()..].contains('/'))
.map(str::to_string)
.collect();
// sheet1.xml, sheet2.xml, ... sort by their numeric suffix.
sheets.sort_by_key(|name| {
name.trim_start_matches("xl/worksheets/sheet")
.trim_end_matches(".xml")
.parse::<u32>()
.unwrap_or(u32::MAX)
});
sheets
.into_iter()
.next()
.ok_or_else(|| IdxError::ParseError("ownership XLSX has no worksheet".to_string()))
}
fn xml_error(context: &str, err: impl std::fmt::Display) -> IdxError {
IdxError::ParseError(format!("failed to parse ownership XLSX {context}: {err}"))
}
/// Shared strings table: each `<si>` is one string, made of one `<t>` or
/// several rich-text runs `<r><t>`. Phonetic hints (`<rPh>`) are skipped.
fn parse_shared_strings(xml: &str) -> Result<Vec<String>, IdxError> {
let mut reader = Reader::from_str(xml);
let mut strings = Vec::new();
let mut current: Option<String> = None;
let mut in_text = false;
let mut in_phonetic = false;
loop {
match reader.read_event() {
Ok(Event::Start(e)) => match e.local_name().as_ref() {
b"si" => current = Some(String::new()),
b"t" => in_text = true,
b"rPh" => in_phonetic = true,
_ => {}
},
Ok(Event::Empty(e)) if e.local_name().as_ref() == b"si" => strings.push(String::new()),
Ok(Event::Text(text)) if in_text && !in_phonetic => {
if let Some(value) = current.as_mut() {
let unescaped = text
.unescape()
.map_err(|e| xml_error("shared strings", e))?;
value.push_str(&unescaped);
}
}
Ok(Event::End(e)) => match e.local_name().as_ref() {
b"si" => strings.push(current.take().unwrap_or_default()),
b"t" => in_text = false,
b"rPh" => in_phonetic = false,
_ => {}
},
Ok(Event::Eof) => break,
Err(e) => return Err(xml_error("shared strings", e)),
_ => {}
}
}
Ok(strings)
}
/// One sheet row: its 1-based Excel row number and column letters
/// (`A`, `B`, ...) mapped to cell text. Sheet XML omits blank rows, so the
/// number comes from the row's `r` attribute, not its position.
#[derive(Debug, Default)]
struct SheetRow {
number: usize,
cells: HashMap<String, String>,
}
fn row_number(element: &BytesStart<'_>, fallback: usize) -> Result<usize, IdxError> {
for attribute in element.attributes() {
let attribute = attribute.map_err(|e| xml_error("worksheet", e))?;
if attribute.key.local_name().as_ref() == b"r" {
if let Ok(number) = String::from_utf8_lossy(&attribute.value).trim().parse() {
return Ok(number);
}
}
}
Ok(fallback)
}
fn parse_sheet_rows(xml: &str, shared_strings: &[String]) -> Result<Vec<SheetRow>, IdxError> {
let mut reader = Reader::from_str(xml);
let mut rows = Vec::new();
let mut row: Option<SheetRow> = None;
let mut cell: Option<(String, String)> = None;
let mut value = String::new();
let mut in_value = false;
loop {
match reader.read_event() {
Ok(Event::Start(e)) => match e.local_name().as_ref() {
b"row" => {
row = Some(SheetRow {
number: row_number(&e, rows.len() + 1)?,
cells: HashMap::new(),
})
}
b"c" => {
cell = Some(cell_ref_and_type(&e)?);
value.clear();
}
b"v" | b"t" if cell.is_some() => in_value = true,
_ => {}
},
Ok(Event::Empty(e)) if e.local_name().as_ref() == b"row" => rows.push(SheetRow {
number: row_number(&e, rows.len() + 1)?,
cells: HashMap::new(),
}),
Ok(Event::Text(text)) if in_value => {
let unescaped = text.unescape().map_err(|e| xml_error("worksheet", e))?;
value.push_str(&unescaped);
}
Ok(Event::End(e)) => match e.local_name().as_ref() {
b"v" | b"t" => in_value = false,
b"c" => {
if let (Some((column, cell_type)), Some(current_row)) =
(cell.take(), row.as_mut())
{
let text = if cell_type == "s" {
let index = value.trim().parse::<usize>().map_err(|e| {
xml_error("worksheet", format!("bad shared string index: {e}"))
})?;
shared_strings.get(index).cloned().ok_or_else(|| {
xml_error(
"worksheet",
format!("shared string {index} out of range"),
)
})?
} else {
value.clone()
};
current_row.cells.insert(column, text);
}
}
b"row" => {
if let Some(current_row) = row.take() {
rows.push(current_row);
}
}
_ => {}
},
Ok(Event::Eof) => break,
Err(e) => return Err(xml_error("worksheet", e)),
_ => {}
}
}
Ok(rows)
}
fn cell_ref_and_type(element: &BytesStart<'_>) -> Result<(String, String), IdxError> {
let mut column = String::new();
let mut cell_type = String::new();
for attribute in element.attributes() {
let attribute = attribute.map_err(|e| xml_error("worksheet", e))?;
let raw = String::from_utf8_lossy(&attribute.value).to_string();
match attribute.key.local_name().as_ref() {
b"r" => column = raw.chars().take_while(char::is_ascii_alphabetic).collect(),
b"t" => cell_type = raw,
_ => {}
}
}
if column.is_empty() {
return Err(xml_error("worksheet", "cell without a reference"));
}
Ok((column, cell_type))
}
fn drafts_from_rows(rows: &[SheetRow]) -> Result<Vec<KseiHoldingDraft>, IdxError> {
let (header_index, columns) = rows
.iter()
.take(MAX_HEADER_SEARCH_ROWS)
.enumerate()
.find_map(|(index, row)| header_columns(row).map(|columns| (index, columns)))
.ok_or_else(|| {
IdxError::ParseError(
"ownership XLSX header did not match the above-1% holder register layout \
(DATE, SHARE_CODE, ... PERCENTAGE); the above-5% and investor-type \
workbooks are not supported"
.to_string(),
)
})?;
let mut drafts = Vec::new();
for row in &rows[header_index + 1..] {
let line = row.number;
let field = |name: &str| -> &str {
columns
.get(name)
.and_then(|column| row.cells.get(column))
.map(|value| value.trim())
.unwrap_or("")
};
let ticker_code = field("SHARE_CODE");
let investor_name = field("INVESTOR_NAME");
if ticker_code.is_empty() && investor_name.is_empty() {
continue;
}
if ticker_code.is_empty() || investor_name.is_empty() {
return Err(IdxError::ParseError(format!(
"ownership XLSX row {line} is missing SHARE_CODE or INVESTOR_NAME"
)));
}
drafts.push(KseiHoldingDraft {
ticker_code: ticker_code.to_string(),
issuer_name: optional(field("ISSUER_NAME")),
raw_investor_name: investor_name.to_string(),
investor_type: normalize_investor_type(field("INVESTOR_CLASSIFICATION")),
locality: normalize_locality(field("LOCAL_FOREIGN")),
nationality: optional(field("NATIONALITY")),
domicile: optional(field("DOMICILE")),
holdings_scripless: parse_shares(field("HOLDINGS_SCRIPLESS"), line, false)?,
holdings_scrip: parse_shares(field("HOLDINGS_SCRIP"), line, false)?,
total_shares: parse_shares(field("TOTAL_HOLDING_SHARES"), line, true)?,
percentage_bps: parse_percentage_bps(field("PERCENTAGE"), line)?,
report_date: parse_report_date(field("DATE"), line)?,
});
}
if drafts.is_empty() {
return Err(IdxError::ParseError(
"ownership XLSX contained a header but no holder rows".to_string(),
));
}
Ok(drafts)
}
/// Map header labels to column letters if `row` is exactly the expected header.
fn header_columns(row: &SheetRow) -> Option<HashMap<&'static str, String>> {
let mut found: HashMap<&'static str, String> = HashMap::new();
for (column, text) in &row.cells {
let mut label = text.trim().to_ascii_uppercase().replace(' ', "_");
if label.is_empty() {
continue;
}
// Feb/Mar 2026 workbooks name the column INVESTOR_TYPE and hold codes
// (CP, ID, ...); later ones use INVESTOR_CLASSIFICATION with words.
if label == "INVESTOR_TYPE" {
label = "INVESTOR_CLASSIFICATION".to_string();
}
let expected = EXPECTED_HEADER.iter().find(|name| **name == label)?;
found.insert(expected, column.clone());
}
(found.len() == EXPECTED_HEADER.len()).then_some(found)
}
fn optional(value: &str) -> Option<String> {
(!value.is_empty()).then(|| value.to_string())
}
fn parse_number(raw: &str, line: usize, name: &str) -> Result<f64, IdxError> {
raw.replace(',', "")
.parse::<f64>()
.ok()
.filter(|value| value.is_finite())
.ok_or_else(|| {
IdxError::ParseError(format!("ownership XLSX row {line}: invalid {name} '{raw}'"))
})
}
fn parse_shares(raw: &str, line: usize, required: bool) -> Result<i64, IdxError> {
if raw.is_empty() {
return if required {
Err(IdxError::ParseError(format!(
"ownership XLSX row {line}: missing TOTAL_HOLDING_SHARES"
)))
} else {
Ok(0)
};
}
let value = parse_number(raw, line, "share count")?;
if value < 0.0 || value.fract() != 0.0 || value > i64::MAX as f64 {
return Err(IdxError::ParseError(format!(
"ownership XLSX row {line}: share count '{raw}' is not a whole number"
)));
}
Ok(value as i64)
}
/// `54.94` (percent) -> 5494 basis points. Excel stores values such as
/// `1.1499999999999999`, so round to the nearest basis point.
fn parse_percentage_bps(raw: &str, line: usize) -> Result<i64, IdxError> {
let value = parse_number(raw, line, "PERCENTAGE")?;
if !(0.0..=100.0).contains(&value) {
return Err(IdxError::ParseError(format!(
"ownership XLSX row {line}: PERCENTAGE '{raw}' is out of range"
)));
}
Ok((value * 100.0).round() as i64)
}
/// DATE is an Excel serial day number (`46265` = 2026-08-31); accept the
/// KSEI text form (`31-Aug-2026`) as well in case the cell is stored as text.
fn parse_report_date(raw: &str, line: usize) -> Result<NaiveDate, IdxError> {
if let Ok(serial) = raw.parse::<f64>() {
let days = serial.trunc() as i64;
let epoch = NaiveDate::from_ymd_opt(1899, 12, 30).expect("valid Excel epoch");
if (1..=2_958_465).contains(&days) {
return Ok(epoch + Duration::days(days));
}
}
parse_ksei_date(raw).map_err(|_| {
IdxError::ParseError(format!("ownership XLSX row {line}: invalid DATE '{raw}'"))
})
}
#[cfg(test)]
mod tests {
use super::*;
fn fixture() -> Vec<KseiHoldingDraft> {
parse_above1_xlsx_bytes(include_bytes!(
"../../tests/fixtures/ksei_above1_20260831_excerpt.xlsx"
))
.expect("above-1% XLSX excerpt parses")
}
#[test]
fn parses_above1_xlsx_excerpt() {
let drafts = fixture();
// Pinned so a regression that silently drops rows fails loudly.
assert_eq!(drafts.len(), 17);
assert!(
drafts
.iter()
.all(|d| d.report_date == NaiveDate::from_ymd_opt(2026, 8, 31).unwrap())
);
let dwimuria = drafts
.iter()
.find(|d| {
d.ticker_code == "BBCA" && d.raw_investor_name == "PT DWIMURIA INVESTAMA ANDALAN"
})
.expect("BBCA controlling holder present");
assert_eq!(dwimuria.percentage_bps, 5494);
assert_eq!(
dwimuria.investor_type.as_ref().map(|t| t.0.as_str()),
Some("CORPORATE")
);
assert_eq!(
dwimuria.total_shares,
dwimuria.holdings_scripless + dwimuria.holdings_scrip
);
}
fn header_row(labels: &[&str]) -> SheetRow {
sheet_row(1, labels)
}
fn sheet_row(number: usize, values: &[&str]) -> SheetRow {
SheetRow {
number,
cells: values
.iter()
.enumerate()
.filter(|(_, value)| !value.is_empty())
.map(|(index, value)| {
(
((b'A' + index as u8) as char).to_string(),
value.to_string(),
)
})
.collect(),
}
}
#[test]
fn accepts_legacy_investor_type_header() {
let mut labels = EXPECTED_HEADER.to_vec();
labels[4] = "INVESTOR_TYPE";
let columns = header_columns(&header_row(&labels)).expect("legacy header accepted");
assert_eq!(columns["INVESTOR_CLASSIFICATION"], "E");
let row = sheet_row(
2,
&[
"46112",
"AADI",
"ADARO ANDALAN INDONESIA Tbk",
"GARIBALDI THOHIR",
"ID",
"L ",
"INDONESIAN",
"INDONESIA",
"454011607",
"0",
"454011607",
"5.83",
],
);
let drafts = drafts_from_rows(&[header_row(&labels), row]).expect("legacy rows parse");
assert_eq!(
drafts[0].investor_type.as_ref().map(|t| t.0.as_str()),
Some("ID")
);
assert_eq!(
drafts[0].report_date,
NaiveDate::from_ymd_opt(2026, 3, 31).unwrap()
);
assert!(drafts[0].locality.is_some());
}
#[test]
fn row_errors_report_excel_row_numbers() {
let missing_investor = sheet_row(
709,
&["46265", "BBCA", "BANK CENTRAL ASIA Tbk", "", "Corporate"],
);
let err = drafts_from_rows(&[header_row(&EXPECTED_HEADER), missing_investor])
.expect_err("row without investor rejected");
assert!(err.to_string().contains("row 709"), "{err}");
}
#[test]
fn rejects_header_with_unknown_or_missing_columns() {
let mut labels = EXPECTED_HEADER.to_vec();
labels[11] = "PERUBAHAN";
assert!(header_columns(&header_row(&labels)).is_none());
assert!(header_columns(&header_row(&EXPECTED_HEADER[..11])).is_none());
}
#[test]
fn rounds_float_noise_in_percentages() {
assert_eq!(parse_percentage_bps("1.1499999999999999", 1).unwrap(), 115);
assert_eq!(parse_percentage_bps("41.1", 1).unwrap(), 4110);
assert!(parse_percentage_bps("100.5", 1).is_err());
}
#[test]
fn parses_excel_serial_and_text_dates() {
let expected = NaiveDate::from_ymd_opt(2026, 8, 31).unwrap();
assert_eq!(parse_report_date("46265", 1).unwrap(), expected);
assert_eq!(parse_report_date("31-Aug-2026", 1).unwrap(), expected);
assert!(parse_report_date("not a date", 1).is_err());
}
#[test]
fn rejects_above5_workbook() {
let err = parse_above1_xlsx_bytes(include_bytes!(
"../../tests/fixtures/ksei_above5_20260923_excerpt.xlsx"
))
.expect_err("above-5% workbook must be rejected");
assert!(
err.to_string().contains("above-1% holder register layout"),
"{err}"
);
}
#[test]
fn rejects_non_xlsx_bytes() {
assert!(parse_above1_xlsx_bytes(b"%PDF-1.7 not a workbook").is_err());
}
}

120
tests/cli.rs
View file

@ -407,7 +407,7 @@ fn ownership_import_help_mentions_sync_preference_and_fallback_files() {
.success() .success()
.stdout(predicate::str::contains("Prefer `idx ownership sync`")) .stdout(predicate::str::contains("Prefer `idx ownership sync`"))
.stdout(predicate::str::contains( .stdout(predicate::str::contains(
"PDF (primary), ZIP/TXT archive (fallback)", "above-1% XLSX or PDF (primary), ZIP/TXT archive (fallback)",
)) ))
.stdout(predicate::str::contains( .stdout(predicate::str::contains(
"idx ownership import --file /path/to/BalanceposEfek20260227.zip", "idx ownership import --file /path/to/BalanceposEfek20260227.zip",
@ -1189,6 +1189,43 @@ fn ownership_releases_uses_xdg_data_home_default_db_path() {
); );
} }
/// Keeps discovery tests off the live IDX data page (nothing listens on port 9).
const UNREACHABLE_DATA_PAGE_URL: &str = "http://127.0.0.1:9/data-kepemilikan-saham/";
#[test]
fn ownership_discover_finds_xlsx_on_share_ownership_page() {
let page = fs::read_to_string("tests/fixtures/idx_share_ownership_page_excerpt.html")
.expect("read share ownership page fixture");
let page_url = spawn_single_response_server("text/html", page);
let output = test_bin("ownership-discover-data-page")
.env("IDX_CURL_IMPERSONATE_BIN", "curl")
.env(
"IDX_OWNERSHIP_ANNOUNCEMENT_API_URL",
"http://127.0.0.1:9/announcements",
)
.env("IDX_OWNERSHIP_DATA_PAGE_URL", &page_url)
.args(["-o", "json", "ownership", "discover", "--limit", "1"])
.output()
.expect("ownership discover json output");
assert!(
output.status.success(),
"{}",
String::from_utf8_lossy(&output.stderr)
);
let reports: Value = serde_json::from_slice(&output.stdout).expect("parse discover json");
let latest = &reports[0];
assert_eq!(latest["family"], "above_one_percent");
assert_eq!(latest["status"], "supported");
assert_eq!(latest["format"], "xlsx");
assert_eq!(latest["as_of_date"], "2026-08-31");
assert_eq!(
latest["pdf_url"],
"https://www.idx.co.id/Media/fahlw1o2/peng-2026-08-00017-satu-persen.xlsx"
);
}
#[test] #[test]
fn ownership_discover_lists_fixture_candidates() { fn ownership_discover_lists_fixture_candidates() {
let body = fs::read_to_string("tests/fixtures/idx_announcement_kepemilikan.json") let body = fs::read_to_string("tests/fixtures/idx_announcement_kepemilikan.json")
@ -1198,6 +1235,7 @@ fn ownership_discover_lists_fixture_candidates() {
test_bin("ownership-discover") test_bin("ownership-discover")
.env("IDX_CURL_IMPERSONATE_BIN", "curl") .env("IDX_CURL_IMPERSONATE_BIN", "curl")
.env("IDX_OWNERSHIP_ANNOUNCEMENT_API_URL", &json_url) .env("IDX_OWNERSHIP_ANNOUNCEMENT_API_URL", &json_url)
.env("IDX_OWNERSHIP_DATA_PAGE_URL", UNREACHABLE_DATA_PAGE_URL)
.env( .env(
"IDX_OWNERSHIP_ANNOUNCEMENT_PAGE_URL", "IDX_OWNERSHIP_ANNOUNCEMENT_PAGE_URL",
"http://127.0.0.1/pengumuman", "http://127.0.0.1/pengumuman",
@ -1257,6 +1295,7 @@ fn ownership_discover_supports_above1_family() {
test_bin("ownership-discover-above1") test_bin("ownership-discover-above1")
.env("IDX_CURL_IMPERSONATE_BIN", "curl") .env("IDX_CURL_IMPERSONATE_BIN", "curl")
.env("IDX_OWNERSHIP_ANNOUNCEMENT_API_URL", &json_url) .env("IDX_OWNERSHIP_ANNOUNCEMENT_API_URL", &json_url)
.env("IDX_OWNERSHIP_DATA_PAGE_URL", UNREACHABLE_DATA_PAGE_URL)
.env( .env(
"IDX_OWNERSHIP_ANNOUNCEMENT_PAGE_URL", "IDX_OWNERSHIP_ANNOUNCEMENT_PAGE_URL",
"http://127.0.0.1/pengumuman", "http://127.0.0.1/pengumuman",
@ -1274,6 +1313,10 @@ fn ownership_discover_supports_above1_family() {
.stdout(predicate::str::contains("Above 1%")) .stdout(predicate::str::contains("Above 1%"))
.stdout(predicate::str::contains( .stdout(predicate::str::contains(
"20260310_Semua Emiten Saham_Pengumuman Bursa_32052554_lamp1.pdf", "20260310_Semua Emiten Saham_Pengumuman Bursa_32052554_lamp1.pdf",
))
.stderr(predicate::str::contains("warning:"))
.stderr(predicate::str::contains(
"IDX share ownership data page discovery",
)); ));
} }
@ -1313,6 +1356,7 @@ fn ownership_discover_defaults_to_above1_and_prefers_supported_attachment() {
let output = test_bin("ownership-discover-default") let output = test_bin("ownership-discover-default")
.env("IDX_CURL_IMPERSONATE_BIN", "curl") .env("IDX_CURL_IMPERSONATE_BIN", "curl")
.env("IDX_OWNERSHIP_ANNOUNCEMENT_API_URL", &json_url) .env("IDX_OWNERSHIP_ANNOUNCEMENT_API_URL", &json_url)
.env("IDX_OWNERSHIP_DATA_PAGE_URL", UNREACHABLE_DATA_PAGE_URL)
.env( .env(
"IDX_OWNERSHIP_ANNOUNCEMENT_PAGE_URL", "IDX_OWNERSHIP_ANNOUNCEMENT_PAGE_URL",
"http://127.0.0.1/pengumuman", "http://127.0.0.1/pengumuman",
@ -1367,6 +1411,7 @@ fn ownership_discover_json_includes_status() {
test_bin("ownership-discover-json-status") test_bin("ownership-discover-json-status")
.env("IDX_CURL_IMPERSONATE_BIN", "curl") .env("IDX_CURL_IMPERSONATE_BIN", "curl")
.env("IDX_OWNERSHIP_ANNOUNCEMENT_API_URL", &json_url) .env("IDX_OWNERSHIP_ANNOUNCEMENT_API_URL", &json_url)
.env("IDX_OWNERSHIP_DATA_PAGE_URL", UNREACHABLE_DATA_PAGE_URL)
.env( .env(
"IDX_OWNERSHIP_ANNOUNCEMENT_PAGE_URL", "IDX_OWNERSHIP_ANNOUNCEMENT_PAGE_URL",
"http://127.0.0.1/pengumuman", "http://127.0.0.1/pengumuman",
@ -1406,7 +1451,7 @@ fn ownership_import_url_rejects_listing_page_inputs() {
.assert() .assert()
.failure() .failure()
.stderr(predicate::str::contains( .stderr(predicate::str::contains(
"ownership import --url accepts direct PDF URLs only", "ownership import --url accepts direct XLSX or PDF URLs only",
)) ))
.stderr(predicate::str::contains("ownership discover")); .stderr(predicate::str::contains("ownership discover"));
} }
@ -1524,6 +1569,77 @@ fn ownership_import_url_duplicate_release_is_skipped() {
.stdout(predicate::str::contains("Release already imported")); .stdout(predicate::str::contains("Release already imported"));
} }
#[test]
fn ownership_import_file_xlsx_supports_ticker_and_releases() {
let root = test_env_dir("ownership-import-xlsx");
let db_path = root.join("ownership.db");
let xlsx_path = root.join("peng-2026-08-00017-satu-persen.xlsx");
fs::write(
&xlsx_path,
include_bytes!("fixtures/ksei_above1_20260831_excerpt.xlsx"),
)
.expect("write xlsx fixture");
bin_with_root(&root)
.args([
"config",
"set",
"ownership.db_path",
db_path.to_str().unwrap(),
])
.assert()
.success();
bin_with_root(&root)
.args(["ownership", "import", "--file", xlsx_path.to_str().unwrap()])
.assert()
.success()
.stdout(predicate::str::contains("(as of 2026-08-31)"));
let output = bin_with_root(&root)
.args(["-o", "json", "ownership", "ticker", "BBCA"])
.output()
.expect("ownership ticker json output");
assert!(output.status.success());
let body = String::from_utf8_lossy(&output.stdout);
assert!(body.contains("DWIMURIA INVESTAMA ANDALAN"), "{body}");
let output = bin_with_root(&root)
.args(["-o", "json", "ownership", "releases"])
.output()
.expect("ownership releases json output");
let releases: Value = serde_json::from_slice(&output.stdout).expect("parse releases json");
assert_eq!(releases[0]["as_of_date"], "2026-08-31");
}
#[test]
fn ownership_import_file_rejects_above5_xlsx() {
let root = test_env_dir("ownership-import-xlsx-above5");
let db_path = root.join("ownership.db");
let xlsx_path = root.join("peng-2026-09-23-00080-lima-persen.xlsx");
fs::write(
&xlsx_path,
include_bytes!("fixtures/ksei_above5_20260923_excerpt.xlsx"),
)
.expect("write xlsx fixture");
bin_with_root(&root)
.args([
"config",
"set",
"ownership.db_path",
db_path.to_str().unwrap(),
])
.assert()
.success();
bin_with_root(&root)
.args(["ownership", "import", "--file", xlsx_path.to_str().unwrap()])
.assert()
.failure()
.stderr(predicate::str::contains("above-1% holder register layout"));
}
#[test] #[test]
fn ownership_import_force_reimports_existing_release() { fn ownership_import_force_reimports_existing_release() {
let root = test_env_dir("ownership-import-force"); let root = test_env_dir("ownership-import-force");

5
tests/fixtures/idx_share_ownership_page_excerpt.html vendored Normal file
View file

@ -0,0 +1,5 @@
<!doctype html>
<html lang="id"><head><title>Data Kepemilikan Saham - Bursa Efek Indonesia</title></head>
<body><div id="__nuxt"><!-- trimmed excerpt of https://www.idx.co.id/id/perusahaan-tercatat/data-kepemilikan-saham/ (2026-09-24) --></div>
<script>window.__NUXT__=(function(a,r){return {fetch:{"data-v-8c8e470a:0":{tableData:[{Description:"Pemegang Saham di Atas 5% per 23 September 2026",Prospectus:"https:\u002F\u002Fwww.idx.co.id\u002FMedia\u002Fsucngcwy\u002Fpeng-2026-09-23-00080-lima-persen.xlsx",ListingDate:"2026-09-23T12:00:00"},{Description:"Pemegang Saham di Atas 5% per 31 Agustus 2026",Prospectus:"https:\u002F\u002Fwww.idx.co.id\u002FMedia\u002Fumrls4il\u002Fpeng-2026-08-31-00063-lima-persen.xlsx",ListingDate:r},{Description:"Data Kepemilikan Saham Perusahaan Tercatat Berdasarkan Klasifikasi Investor per 31 Agustus 2026",Prospectus:"https:\u002F\u002Fwww.idx.co.id\u002FMedia\u002Fy0vjakdz\u002Fpeng-2026-08-00018-klasifikasi.xlsx",ListingDate:r},{Description:"Data Kepemilikan Saham Perusahaan Tercatat per Tipe Investor per 31 Agustus 2026",Prospectus:"https:\u002F\u002Fwww.idx.co.id\u002FMedia\u002Fwkjfudt3\u002Fpeng-2026-08-00019-tipe.xlsx",ListingDate:r},{Description:"Pemegang Saham di Atas 1% per 31 Agustus 2026",Prospectus:"https:\u002F\u002Fwww.idx.co.id\u002FMedia\u002Ffahlw1o2\u002Fpeng-2026-08-00017-satu-persen.xlsx",ListingDate:r},{Description:"Pemegang Saham di Atas 1% per 31 Juli 2026",Prospectus:"https:\u002F\u002Fwww.idx.co.id\u002FMedia\u002Ffjelt0en\u002Fpeng-2026-07-00016-satu-persen.xlsx",ListingDate:s},{Description:"Pemegang Saham di Atas 1% per 27 Februari 2026",Prospectus:"https:\u002F\u002Fwww.idx.co.id\u002FMedia\u002Fr2kjzpk5\u002Fpeng-02-00011-satu-persen.xlsx",ListingDate:K}]}}}}(null,"2026-09-01T12:00:00"));</script>
</body></html>

BIN
tests/fixtures/ksei_above1_20260831_excerpt.xlsx vendored Normal file
View file

Binary file not shown.

BIN
tests/fixtures/ksei_above5_20260923_excerpt.xlsx vendored Normal file
View file

Binary file not shown.