mirror of
https://github.com/0xrsydn/idx-cli.git
synced 2026-11-07 01:35:36 +00:00
Merge pull request #27 from 0xrsydn/feat/ownership-xlsx-snapshot
feat(ownership): build snapshots from XLSX with monthly history [3/3]
This commit is contained in:
brutaljokerz
GitHub
commit
d4ec5015b9
17 changed files with 1354 additions and 47 deletions
|
|
@ -7,6 +7,10 @@ on:
|
|||
description: Stable GitHub release tag to publish the snapshot assets to
|
||||
required: true
|
||||
default: ownership-snapshot-current
|
||||
history:
|
||||
description: Number of earlier monthly above-1% reports to include for `ownership changes`
|
||||
required: false
|
||||
default: "5"
|
||||
|
||||
permissions:
|
||||
contents: write
|
||||
|
|
@ -36,7 +40,8 @@ jobs:
|
|||
--idx-bin ./target/debug/idx \
|
||||
--output-dir dist/ownership-snapshot \
|
||||
--repo "${{ github.repository }}" \
|
||||
--release-tag "${{ inputs.release_tag }}"
|
||||
--release-tag "${{ inputs.release_tag }}" \
|
||||
--history "${{ inputs.history }}"
|
||||
|
||||
- name: Upload workflow artifact copy
|
||||
uses: actions/upload-artifact@v4
|
||||
|
|
|
|||
|
|
@ -57,13 +57,13 @@ Ownership is local-first after bootstrap.
|
|||
|
||||
Preferred bootstrap/update path:
|
||||
1. `idx ownership sync`
|
||||
2. if no snapshot manifest is available: `idx ownership discover` then `idx ownership import --url <pdf-url>`
|
||||
2. if no snapshot manifest is available: `idx ownership discover` then `idx ownership import --url <xlsx-or-pdf-url>`
|
||||
3. local `--file` imports remain available for manual/fallback use
|
||||
|
||||
Ownership input paths:
|
||||
- primary remote source: discoverable IDX `above1` holder-register PDF
|
||||
- primary remote source: IDX `above1` holder register — monthly XLSX on the Data Kepemilikan Saham page (since the 2026-05-29 report), legacy PDF announcements before that; `ownership discover` checks both
|
||||
- maintained snapshot path: `ownership sync`
|
||||
- local fallback path: PDF, plus local archive `.zip` / `.txt`
|
||||
- local file imports: IDX above-1% `.xlsx` (current format since June 2026), legacy `.pdf`, plus local archive `.zip` / `.txt`
|
||||
|
||||
Important scope note:
|
||||
- archive ZIP/TXT ingest is a fallback/backstop path, not the primary product ingest surface
|
||||
|
|
|
|||
|
|
@ -63,12 +63,32 @@ Unlike the `stocks` module (live-fetch), ownership is **import-then-query**:
|
|||
4. Fully offline after import/sync
|
||||
|
||||
### Data Sources
|
||||
- **KSEI** — official ≥1% shareholder registry (monthly PDF from IDX)
|
||||
- **KSEI** — official ≥1% shareholder registry, published monthly by IDX. Through May 2026 it was a PDF announcement; from the 2026-05-29 report onward it is an XLSX workbook on the IDX "Data Kepemilikan Saham" page
|
||||
- **KSEI archive** — monthly ZIP/TXT balance-position matrix, used as a local fallback/backstop import path
|
||||
- **Bing Finance** — global institutional ownership (REST API, quarterly)
|
||||
|
||||
### Discovery
|
||||
`idx ownership discover` merges two IDX sources:
|
||||
- the announcement API (`GetAllAnnouncement`) for the legacy PDF announcements
|
||||
- the Data Kepemilikan Saham page (`/id/perusahaan-tercatat/data-kepemilikan-saham/`),
|
||||
whose server-rendered Nuxt payload lists every XLSX with a description such as
|
||||
`Pemegang Saham di Atas 1% per 31 Agustus 2026`; the as-of date comes from that text
|
||||
|
||||
Only the above-1% XLSX is `supported`; the daily above-5% and investor-type workbooks
|
||||
are listed as `unsupported`. Both URLs can be overridden for tests with
|
||||
`IDX_OWNERSHIP_ANNOUNCEMENT_API_URL` and `IDX_OWNERSHIP_DATA_PAGE_URL`.
|
||||
|
||||
IDX sits behind Cloudflare, which accepts some curl-impersonate TLS profiles and
|
||||
rejects others per path (for example `curl_chrome142` gets 403 on the data page).
|
||||
IDX fetches therefore try several profiles until the response validates (JSON, page
|
||||
payload, `%PDF`, or `PK` zip magic). `IDX_CURL_IMPERSONATE_BIN` pins a single profile.
|
||||
|
||||
### Parser Pipeline
|
||||
```
|
||||
KSEI XLSX → zip + quick-xml (sharedStrings + sheet1) → exact 12-column header check
|
||||
→ KseiHoldingDraft (Excel serial dates, numeric shares, percent → bps)
|
||||
→ SQLite INSERT (within transaction)
|
||||
|
||||
KSEI PDF → mutool stext (XML with coordinates) → quick-xml parse → KseiRawRow
|
||||
→ normalize (ID locale numbers, dates, entity names) → KseiHolding
|
||||
→ SQLite INSERT (within transaction)
|
||||
|
|
|
|||
|
|
@ -45,9 +45,11 @@ nix develop --command scripts/build-latest-ownership-snapshot.sh \
|
|||
|
||||
The script performs these steps:
|
||||
|
||||
1. `idx -o json ownership discover --family above1 --limit 1`
|
||||
1. `idx -o json ownership discover --family above1 --limit 50` (newest supported
|
||||
report first; XLSX from the Data Kepemilikan Saham page since June 2026)
|
||||
2. verifies the discovered report is the current supported import path
|
||||
3. imports that PDF into an isolated temp ownership DB
|
||||
3. imports that report (plus `--history <n>` earlier months, oldest first) into
|
||||
an isolated temp ownership DB
|
||||
4. checks that the imported release metadata is non-empty and tied to the same
|
||||
source URL
|
||||
5. runs `scripts/build-ownership-snapshot.sh` to emit the SQLite artifact and
|
||||
|
|
|
|||
|
|
@ -38,7 +38,9 @@ nix develop --command scripts/publish-ownership-snapshot.sh \
|
|||
That helper:
|
||||
|
||||
1. optionally builds `idx`
|
||||
2. discovers the latest supported IDX/KSEI PDF
|
||||
2. discovers the latest supported IDX/KSEI above-1% report (XLSX on the IDX
|
||||
Data Kepemilikan Saham page since the 2026-05-29 report; PDF announcements
|
||||
before that)
|
||||
3. imports it into an isolated temp DB
|
||||
4. emits the SQLite snapshot and manifest
|
||||
5. ensures the stable GitHub release exists
|
||||
|
|
@ -66,8 +68,14 @@ The sample timer uses:
|
|||
- `RandomizedDelaySec=30m`
|
||||
|
||||
That is intentionally conservative. The ownership source is monthly, but the
|
||||
exact publish day can drift. Start with an early-month schedule and adjust after
|
||||
observing a few real runs.
|
||||
exact publish day can drift. Observed XLSX upload times (HTTP Last-Modified):
|
||||
2026-06-03, 2026-07-02 03:55Z, 2026-08-02 01:50Z, 2026-09-02 03:51Z, so a run
|
||||
on the 2nd can occasionally land before the file exists; the 3rd is safer.
|
||||
Start with an early-month schedule and adjust after observing a few real runs.
|
||||
|
||||
Pass `--history <n>` (for example via `services.idxOwnershipPublish.extraArgs =
|
||||
[ "--history" "5" ];` in `clan-private`) to include earlier months so
|
||||
`idx ownership changes` works straight from the synced snapshot.
|
||||
|
||||
## Clan Integration Later
|
||||
|
||||
|
|
|
|||
|
|
@ -6,7 +6,8 @@ usage() {
|
|||
cat <<'EOF'
|
||||
Usage: scripts/build-latest-ownership-snapshot.sh --output-dir <dir> [options]
|
||||
|
||||
Discover the latest supported IDX/KSEI ownership PDF, import it into an isolated
|
||||
Discover the latest supported IDX/KSEI above-1% report (XLSX from the IDX Data
|
||||
Kepemilikan Saham page, or a legacy PDF announcement), import it into an isolated
|
||||
ownership database, and emit GitHub-release-ready snapshot artifacts plus manifest.
|
||||
|
||||
Options:
|
||||
|
|
@ -17,6 +18,9 @@ Options:
|
|||
(default: 0xrsydn/idx-cli)
|
||||
--release-tag <tag> Stable GitHub release tag used for the default base URL
|
||||
(default: ownership-snapshot-current)
|
||||
--history <n> Also import the <n> previous monthly above-1% XLSX
|
||||
reports (oldest first) so `ownership changes` works
|
||||
from the snapshot (default: 0)
|
||||
--keep-workdir Keep the temp workdir instead of deleting it
|
||||
--help Show this help
|
||||
EOF
|
||||
|
|
@ -28,6 +32,7 @@ BASE_URL=""
|
|||
REPO_FULL_NAME="0xrsydn/idx-cli"
|
||||
RELEASE_TAG="ownership-snapshot-current"
|
||||
KEEP_WORKDIR="0"
|
||||
HISTORY="0"
|
||||
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case "$1" in
|
||||
|
|
@ -51,6 +56,10 @@ while [[ $# -gt 0 ]]; do
|
|||
RELEASE_TAG="${2:-}"
|
||||
shift 2
|
||||
;;
|
||||
--history)
|
||||
HISTORY="${2:-}"
|
||||
shift 2
|
||||
;;
|
||||
--keep-workdir)
|
||||
KEEP_WORKDIR="1"
|
||||
shift
|
||||
|
|
@ -73,6 +82,13 @@ if [[ -z "$OUTPUT_DIR" ]]; then
|
|||
exit 2
|
||||
fi
|
||||
|
||||
if ! [[ "$HISTORY" =~ ^[0-9]+$ ]]; then
|
||||
echo "--history must be a non-negative integer" >&2
|
||||
exit 2
|
||||
fi
|
||||
# Canonicalize so "08" is 8, not an invalid octal literal in (( )).
|
||||
HISTORY=$((10#$HISTORY))
|
||||
|
||||
if [[ -z "$BASE_URL" ]]; then
|
||||
BASE_URL="https://github.com/${REPO_FULL_NAME}/releases/download/${RELEASE_TAG}"
|
||||
fi
|
||||
|
|
@ -115,8 +131,8 @@ RELEASES_JSON="$WORKDIR/releases.json"
|
|||
MANIFEST_PATH="$OUTPUT_DIR/ownership-snapshot-manifest.json"
|
||||
DB_PATH="$XDG_DATA_HOME/idx/ownership.db"
|
||||
|
||||
printf 'Discovering latest supported IDX/KSEI ownership PDF...\n'
|
||||
"$IDX_BIN" -o json ownership discover --family above1 --limit 1 > "$DISCOVERY_JSON"
|
||||
printf 'Discovering latest supported IDX/KSEI above-1%% report...\n'
|
||||
"$IDX_BIN" -o json ownership discover --family above1 --limit 50 > "$DISCOVERY_JSON"
|
||||
|
||||
discovery_payload="$(
|
||||
jq -r '
|
||||
|
|
@ -151,7 +167,33 @@ DISCOVERED_TITLE="${discovery_fields[4]:-}"
|
|||
DISCOVERED_PUBLISH_DATE="${discovery_fields[5]:-}"
|
||||
DISCOVERED_ORIGINAL_FILENAME="${discovery_fields[6]:-}"
|
||||
|
||||
printf 'Importing discovered PDF into isolated DB...\n'
|
||||
if (( HISTORY > 0 )); then
|
||||
# Earlier supported XLSX months, one per as-of date, newest first; import oldest first.
|
||||
# Run jq in a plain command substitution (not <(...)) so a jq failure
|
||||
# stops the script under set -e instead of looking like zero months.
|
||||
history_payload="$(
|
||||
jq -r --arg latest "$DISCOVERED_PDF_URL" --argjson n "$HISTORY" '
|
||||
(map(select(.pdf_url == $latest)) | .[0].as_of_date // "") as $latest_as_of
|
||||
| [ .[]
|
||||
| select(.status == "supported" and .format == "xlsx" and .as_of_date != null)
|
||||
| select($latest_as_of == "" or .as_of_date < $latest_as_of) ]
|
||||
| unique_by(.as_of_date)
|
||||
| sort_by(.as_of_date) | reverse
|
||||
| .[:$n] | reverse
|
||||
| .[].pdf_url
|
||||
' "$DISCOVERY_JSON"
|
||||
)"
|
||||
history_urls=()
|
||||
if [[ -n "$history_payload" ]]; then
|
||||
mapfile -t history_urls <<< "$history_payload"
|
||||
fi
|
||||
printf 'Importing %d earlier monthly report(s) for history...\n' "${#history_urls[@]}"
|
||||
for url in "${history_urls[@]}"; do
|
||||
"$IDX_BIN" ownership import --url "$url"
|
||||
done
|
||||
fi
|
||||
|
||||
printf 'Importing discovered report into isolated DB...\n'
|
||||
"$IDX_BIN" ownership import --url "$DISCOVERED_PDF_URL"
|
||||
|
||||
printf 'Inspecting imported release metadata...\n'
|
||||
|
|
|
|||
|
|
@ -16,6 +16,8 @@ Options:
|
|||
(default: 0xrsydn/idx-cli)
|
||||
--release-tag <tag> Stable GitHub release tag used for snapshot assets
|
||||
(default: ownership-snapshot-current)
|
||||
--history <n> Also include the <n> previous monthly above-1% reports
|
||||
in the snapshot (passed to the builder; default: 0)
|
||||
--build Run `cargo build` before publishing
|
||||
--keep-workdir Keep the temp workdir created by the builder helper
|
||||
--help Show this help
|
||||
|
|
@ -28,6 +30,7 @@ REPO_FULL_NAME="0xrsydn/idx-cli"
|
|||
RELEASE_TAG="ownership-snapshot-current"
|
||||
BUILD_FIRST="0"
|
||||
KEEP_WORKDIR="0"
|
||||
HISTORY="0"
|
||||
PUBLISH_WORKDIR=""
|
||||
|
||||
while [[ $# -gt 0 ]]; do
|
||||
|
|
@ -48,6 +51,10 @@ while [[ $# -gt 0 ]]; do
|
|||
RELEASE_TAG="${2:-}"
|
||||
shift 2
|
||||
;;
|
||||
--history)
|
||||
HISTORY="${2:-}"
|
||||
shift 2
|
||||
;;
|
||||
--build)
|
||||
BUILD_FIRST="1"
|
||||
shift
|
||||
|
|
@ -120,6 +127,7 @@ build_args=(
|
|||
--output-dir "$PUBLISH_WORKDIR"
|
||||
--repo "$REPO_FULL_NAME"
|
||||
--release-tag "$RELEASE_TAG"
|
||||
--history "$HISTORY"
|
||||
)
|
||||
|
||||
if [[ "$KEEP_WORKDIR" == "1" ]]; then
|
||||
|
|
|
|||
|
|
@ -17,13 +17,13 @@ use crate::output::table::format_idr;
|
|||
use crate::ownership::types::{
|
||||
ChangeType, FlowSignal, HolderRow, KseiHolding, OwnershipRelease, OwnershipSource,
|
||||
};
|
||||
use crate::ownership::{archive, db, entities, graph, parser, remote, search, snapshot};
|
||||
use crate::ownership::{archive, db, entities, graph, parser, remote, search, snapshot, xlsx};
|
||||
use crate::runtime;
|
||||
|
||||
#[derive(Debug, Args)]
|
||||
#[command(
|
||||
about = "Ownership intelligence (KSEI + Bing)",
|
||||
long_about = "Ownership intelligence (KSEI + Bing).\n\nPreferred bootstrap path:\n 1. Run `idx ownership sync` to install or refresh a maintained SQLite snapshot.\n 2. If no snapshot manifest is available, run `idx ownership discover` and then `idx ownership import --url <pdf-url>`.\n 3. Local `.pdf`, `.zip`, and `.txt` imports remain available for manual or fallback workflows.",
|
||||
long_about = "Ownership intelligence (KSEI + Bing).\n\nPreferred bootstrap path:\n 1. Run `idx ownership sync` to install or refresh a maintained SQLite snapshot.\n 2. If no snapshot manifest is available, run `idx ownership discover` and then `idx ownership import --url <pdf-url>`.\n 3. Local `.xlsx`, `.pdf`, `.zip`, and `.txt` imports remain available for manual or fallback workflows.",
|
||||
after_help = "Examples:\n idx ownership sync\n idx ownership sync --manifest /path/to/ownership-snapshot-manifest.json\n idx ownership discover --limit 1\n idx ownership import --url <pdf-url>\n idx ownership releases"
|
||||
)]
|
||||
pub struct OwnershipCmd {
|
||||
|
|
@ -35,13 +35,13 @@ pub struct OwnershipCmd {
|
|||
pub enum OwnershipCommand {
|
||||
#[command(
|
||||
about = "Discover the latest IDX-hosted ownership report URLs",
|
||||
after_help = "Examples:\n idx ownership discover --limit 1\n idx ownership discover --family all --limit 6\n\nUse this when no snapshot manifest is available and you need a direct PDF URL for `idx ownership import --url`."
|
||||
after_help = "Examples:\n idx ownership discover --limit 1\n idx ownership discover --family all --limit 6\n\nUse this when no snapshot manifest is available and you need a direct XLSX or PDF URL for `idx ownership import --url`."
|
||||
)]
|
||||
Discover(DiscoverArgs),
|
||||
#[command(
|
||||
about = "Import ownership data directly from source files",
|
||||
long_about = "Import ownership data directly from source files.\n\nPrefer `idx ownership sync` for normal bootstrap/update flows. Use `import` when you need a direct PDF import from IDX discovery output, a local PDF, or a local KSEI archive fallback file.",
|
||||
after_help = "Examples:\n idx ownership import --url <pdf-url-from-discover>\n idx ownership import --file /path/to/ksei.pdf\n idx ownership import --file /path/to/BalanceposEfek20260227.zip"
|
||||
after_help = "Examples:\n idx ownership import --url <pdf-url-from-discover>\n idx ownership import --file /path/to/peng-2026-08-00017-satu-persen.xlsx\n idx ownership import --file /path/to/ksei.pdf\n idx ownership import --file /path/to/BalanceposEfek20260227.zip"
|
||||
)]
|
||||
Import(ImportArgs),
|
||||
#[command(
|
||||
|
|
@ -84,10 +84,10 @@ pub struct DiscoverArgs {
|
|||
|
||||
#[derive(Debug, Args)]
|
||||
pub struct ImportArgs {
|
||||
/// Direct URL to a remote ownership PDF, typically from `ownership discover`.
|
||||
/// Direct URL to a remote above-1% XLSX or PDF, typically from `ownership discover`.
|
||||
#[arg(long)]
|
||||
pub url: Option<String>,
|
||||
/// Path to a local ownership file: PDF (primary), ZIP/TXT archive (fallback).
|
||||
/// Path to a local ownership file: above-1% XLSX or PDF (primary), ZIP/TXT archive (fallback).
|
||||
#[arg(long)]
|
||||
pub file: Option<PathBuf>,
|
||||
/// Fetch Bing institutional data for these symbols.
|
||||
|
|
@ -836,6 +836,7 @@ fn handle_import(args: &ImportArgs, config: &IdxConfig) -> Result<(), IdxError>
|
|||
drafts
|
||||
}
|
||||
ImportInputFormat::Archive => archive::parse_balancepos_file(&import_input.import_path)?,
|
||||
ImportInputFormat::Xlsx => xlsx::parse_above1_xlsx_file(&import_input.import_path)?,
|
||||
};
|
||||
|
||||
let mut holdings = Vec::with_capacity(drafts.len());
|
||||
|
|
@ -977,6 +978,7 @@ struct OwnershipImportResult {
|
|||
enum ImportInputFormat {
|
||||
Pdf,
|
||||
Archive,
|
||||
Xlsx,
|
||||
}
|
||||
|
||||
fn resolve_import_input(args: &ImportArgs) -> Result<Option<ResolvedImportInput>, IdxError> {
|
||||
|
|
@ -997,13 +999,16 @@ fn resolve_import_input(args: &ImportArgs) -> Result<Option<ResolvedImportInput>
|
|||
|
||||
if let Some(url) = &args.url {
|
||||
let trimmed = url.trim();
|
||||
validate_import_url(trimmed)?;
|
||||
let target = cache_pdf_path(trimmed)?;
|
||||
download_pdf(trimmed, &target)?;
|
||||
let format = validate_import_url(trimmed)?;
|
||||
let target = cache_download_path(trimmed, format)?;
|
||||
match format {
|
||||
ImportInputFormat::Xlsx => download_xlsx(trimmed, &target)?,
|
||||
_ => download_pdf(trimmed, &target)?,
|
||||
}
|
||||
return Ok(Some(ResolvedImportInput {
|
||||
import_path: target,
|
||||
source_url: Some(trimmed.to_string()),
|
||||
format: ImportInputFormat::Pdf,
|
||||
format,
|
||||
}));
|
||||
}
|
||||
|
||||
|
|
@ -1018,17 +1023,22 @@ fn detect_local_import_format(path: &Path) -> Result<ImportInputFormat, IdxError
|
|||
.as_deref()
|
||||
{
|
||||
Some("pdf") => Ok(ImportInputFormat::Pdf),
|
||||
Some("xlsx") if xlsx::supports_xlsx_file(path) => Ok(ImportInputFormat::Xlsx),
|
||||
Some("zip") | Some("txt") if archive::supports_local_archive_file(path) => {
|
||||
Ok(ImportInputFormat::Archive)
|
||||
}
|
||||
_ => Err(IdxError::InvalidInput(format!(
|
||||
"unsupported local ownership file {}; expected a .pdf, .zip, or .txt input",
|
||||
"unsupported local ownership file {}; expected a .xlsx, .pdf, .zip, or .txt input",
|
||||
path.display()
|
||||
))),
|
||||
}
|
||||
}
|
||||
|
||||
fn cache_pdf_path(url: &str) -> Result<PathBuf, IdxError> {
|
||||
fn cache_download_path(url: &str, format: ImportInputFormat) -> Result<PathBuf, IdxError> {
|
||||
let extension = match format {
|
||||
ImportInputFormat::Xlsx => ".xlsx",
|
||||
_ => ".pdf",
|
||||
};
|
||||
let raw_dir = crate::cache::cache_dir()?.join("ownership").join("raw");
|
||||
fs::create_dir_all(&raw_dir).map_err(|e| IdxError::Io(e.to_string()))?;
|
||||
|
||||
|
|
@ -1042,20 +1052,20 @@ fn cache_pdf_path(url: &str) -> Result<PathBuf, IdxError> {
|
|||
.to_string();
|
||||
|
||||
if file_name.is_empty() || file_name == "/" {
|
||||
file_name = format!("ksei-{}.pdf", Utc::now().timestamp());
|
||||
file_name = format!("ksei-{}{extension}", Utc::now().timestamp());
|
||||
}
|
||||
if !file_name.to_ascii_lowercase().ends_with(".pdf") {
|
||||
file_name.push_str(".pdf");
|
||||
if !file_name.to_ascii_lowercase().ends_with(extension) {
|
||||
file_name.push_str(extension);
|
||||
}
|
||||
|
||||
Ok(raw_dir.join(file_name))
|
||||
}
|
||||
|
||||
fn validate_import_url(url: &str) -> Result<(), IdxError> {
|
||||
fn validate_import_url(url: &str) -> Result<ImportInputFormat, IdxError> {
|
||||
let trimmed = url.trim();
|
||||
if trimmed.is_empty() {
|
||||
return Err(IdxError::InvalidInput(
|
||||
"ownership import --url accepts direct PDF URLs only".to_string(),
|
||||
"ownership import --url accepts direct XLSX or PDF URLs only".to_string(),
|
||||
));
|
||||
}
|
||||
|
||||
|
|
@ -1065,11 +1075,14 @@ fn validate_import_url(url: &str) -> Result<(), IdxError> {
|
|||
.next()
|
||||
.unwrap_or(normalized.as_str());
|
||||
if normalized.ends_with(".pdf") {
|
||||
return Ok(());
|
||||
return Ok(ImportInputFormat::Pdf);
|
||||
}
|
||||
if normalized.ends_with(".xlsx") {
|
||||
return Ok(ImportInputFormat::Xlsx);
|
||||
}
|
||||
|
||||
Err(IdxError::InvalidInput(
|
||||
"ownership import --url accepts direct PDF URLs only; run `idx ownership discover` first to find the current supported attachment".to_string(),
|
||||
"ownership import --url accepts direct XLSX or PDF URLs only; run `idx ownership discover` first to find the current supported file".to_string(),
|
||||
))
|
||||
}
|
||||
|
||||
|
|
@ -1105,6 +1118,32 @@ fn download_pdf(url: &str, target: &Path) -> Result<(), IdxError> {
|
|||
Ok(())
|
||||
}
|
||||
|
||||
fn download_xlsx(url: &str, target: &Path) -> Result<(), IdxError> {
|
||||
if is_idx_url(url) {
|
||||
return remote::download_idx_xlsx(url, target);
|
||||
}
|
||||
|
||||
let response = ureq::get(url)
|
||||
.header(
|
||||
"Accept",
|
||||
"application/vnd.openxmlformats-officedocument.spreadsheetml.sheet,application/octet-stream,*/*;q=0.8",
|
||||
)
|
||||
.call()
|
||||
.map_err(|e| IdxError::Http(format!("failed to download XLSX: {e}")))?;
|
||||
let bytes = response
|
||||
.into_body()
|
||||
.read_to_vec()
|
||||
.map_err(|e| IdxError::Http(format!("failed reading XLSX body: {e}")))?;
|
||||
remote::validate_xlsx_payload(&bytes)?;
|
||||
|
||||
fs::write(target, &bytes).map_err(|e| {
|
||||
IdxError::Io(format!(
|
||||
"failed writing cached XLSX {}: {e}",
|
||||
target.display()
|
||||
))
|
||||
})
|
||||
}
|
||||
|
||||
fn is_idx_url(url: &str) -> bool {
|
||||
let normalized = url.trim().to_ascii_lowercase();
|
||||
normalized.starts_with("https://www.idx.co.id/")
|
||||
|
|
|
|||
|
|
@ -40,7 +40,65 @@ pub fn run(stage: &str, args: &[&str]) -> Result<Output, IdxError> {
|
|||
|
||||
pub fn run_owned(stage: &str, args: &[String]) -> Result<Output, IdxError> {
|
||||
let binary = chrome_curl_binary()?;
|
||||
let output = Command::new(&binary).args(args).output().map_err(|e| {
|
||||
run_binary(stage, &binary, args)
|
||||
}
|
||||
|
||||
/// Browser profiles tried, in order, when a site's bot protection rejects
|
||||
/// some TLS fingerprints but not others (IDX's Cloudflare rules block
|
||||
/// `curl_chrome142` on some pages while Firefox/Safari profiles pass).
|
||||
const FALLBACK_PROFILES: &[&str] = &[
|
||||
"curl_chrome142",
|
||||
"curl_firefox144",
|
||||
"curl_chrome136",
|
||||
"curl_firefox135",
|
||||
"curl_safari184",
|
||||
"curl_chrome131",
|
||||
"curl_chrome124",
|
||||
"curl_chrome120",
|
||||
"curl_chrome116",
|
||||
];
|
||||
const FALLBACK_ROUNDS: usize = 2;
|
||||
|
||||
/// Run curl with each available impersonation profile until `accept` approves
|
||||
/// the response body. `IDX_CURL_IMPERSONATE_BIN`, when set, is the only
|
||||
/// profile used. Returns the first accepted output, or the last error.
|
||||
pub fn run_owned_with_fallback(
|
||||
stage: &str,
|
||||
args: &[String],
|
||||
accept: &dyn Fn(&[u8]) -> Result<(), IdxError>,
|
||||
) -> Result<Output, IdxError> {
|
||||
let profiles: Vec<String> = match std::env::var(OVERRIDE_ENV) {
|
||||
Ok(value) if !value.trim().is_empty() => vec![value.trim().to_string()],
|
||||
_ => FALLBACK_PROFILES
|
||||
.iter()
|
||||
.copied()
|
||||
.filter(|bin| Command::new(bin).arg("--version").output().is_ok())
|
||||
.map(str::to_string)
|
||||
.collect(),
|
||||
};
|
||||
if profiles.is_empty() {
|
||||
return Err(IdxError::Http(format!(
|
||||
"no curl-impersonate binary found; set {OVERRIDE_ENV} or install nixpkgs#curl-impersonate"
|
||||
)));
|
||||
}
|
||||
|
||||
let mut last_error = None;
|
||||
for _ in 0..FALLBACK_ROUNDS {
|
||||
for binary in &profiles {
|
||||
match run_binary(stage, binary, args) {
|
||||
Ok(output) => match accept(&output.stdout) {
|
||||
Ok(()) => return Ok(output),
|
||||
Err(err) => last_error = Some(err),
|
||||
},
|
||||
Err(err) => last_error = Some(err),
|
||||
}
|
||||
}
|
||||
}
|
||||
Err(last_error.expect("at least one profile was tried"))
|
||||
}
|
||||
|
||||
fn run_binary(stage: &str, binary: &str, args: &[String]) -> Result<Output, IdxError> {
|
||||
let output = Command::new(binary).args(args).output().map_err(|e| {
|
||||
if e.kind() == std::io::ErrorKind::NotFound {
|
||||
return IdxError::Http(format!(
|
||||
"curl-impersonate binary '{binary}' not found; set {OVERRIDE_ENV} or install nixpkgs#curl-impersonate-chrome"
|
||||
|
|
|
|||
|
|
@ -212,7 +212,7 @@ pub fn resolve_entity(
|
|||
Ok(entity_id)
|
||||
}
|
||||
|
||||
fn normalize_investor_type(raw: &str) -> Option<InvestorTypeCode> {
|
||||
pub(crate) fn normalize_investor_type(raw: &str) -> Option<InvestorTypeCode> {
|
||||
let value = raw.trim();
|
||||
if value.is_empty() {
|
||||
None
|
||||
|
|
@ -221,7 +221,7 @@ fn normalize_investor_type(raw: &str) -> Option<InvestorTypeCode> {
|
|||
}
|
||||
}
|
||||
|
||||
fn normalize_locality(raw: &str) -> Option<Locality> {
|
||||
pub(crate) fn normalize_locality(raw: &str) -> Option<Locality> {
|
||||
match raw.trim().to_uppercase().as_str() {
|
||||
"L" | "D" => Some(Locality::Local),
|
||||
"F" | "A" => Some(Locality::Foreign),
|
||||
|
|
|
|||
|
|
@ -7,3 +7,4 @@ pub mod remote;
|
|||
pub mod search;
|
||||
pub mod snapshot;
|
||||
pub mod types;
|
||||
pub mod xlsx;
|
||||
|
|
|
|||
|
|
@ -12,6 +12,10 @@ const DEFAULT_IDX_ANNOUNCEMENT_API_URL: &str =
|
|||
const IDX_ANNOUNCEMENT_API_ENV: &str = "IDX_OWNERSHIP_ANNOUNCEMENT_API_URL";
|
||||
const IDX_ANNOUNCEMENT_LISTING_ENV: &str = "IDX_OWNERSHIP_ANNOUNCEMENT_PAGE_URL";
|
||||
const IDX_ANNOUNCEMENT_PAGE_SIZE: usize = 10;
|
||||
/// Since June 2026 the monthly above-1% list (XLSX) is published only here.
|
||||
pub const IDX_SHARE_OWNERSHIP_PAGE_URL: &str =
|
||||
"https://www.idx.co.id/id/perusahaan-tercatat/data-kepemilikan-saham/";
|
||||
const IDX_SHARE_OWNERSHIP_PAGE_ENV: &str = "IDX_OWNERSHIP_DATA_PAGE_URL";
|
||||
const USER_AGENT: &str = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36";
|
||||
|
||||
#[derive(Debug, Clone, Deserialize)]
|
||||
|
|
@ -116,6 +120,12 @@ pub struct DiscoveredOwnershipPdf {
|
|||
pub code: Option<String>,
|
||||
pub original_filename: Option<String>,
|
||||
pub is_attachment: bool,
|
||||
/// File format of `pdf_url`: `pdf` (announcement attachment) or `xlsx`
|
||||
/// (Data Kepemilikan Saham page).
|
||||
pub format: String,
|
||||
/// Report as-of date (`YYYY-MM-DD`) when the source states it. The data
|
||||
/// page shows no publish date, so its entries use this as `publish_date`.
|
||||
pub as_of_date: Option<String>,
|
||||
}
|
||||
|
||||
struct DiscoveryQuery {
|
||||
|
|
@ -152,6 +162,13 @@ pub fn announcement_listing_url() -> String {
|
|||
.unwrap_or_else(|| IDX_ANNOUNCEMENT_LISTING_URL.to_string())
|
||||
}
|
||||
|
||||
pub fn share_ownership_page_url() -> String {
|
||||
std::env::var(IDX_SHARE_OWNERSHIP_PAGE_ENV)
|
||||
.ok()
|
||||
.filter(|value| !value.trim().is_empty())
|
||||
.unwrap_or_else(|| IDX_SHARE_OWNERSHIP_PAGE_URL.to_string())
|
||||
}
|
||||
|
||||
pub fn announcement_api_url() -> String {
|
||||
std::env::var(IDX_ANNOUNCEMENT_API_ENV)
|
||||
.ok()
|
||||
|
|
@ -240,6 +257,8 @@ pub fn select_latest_ownership_reports(
|
|||
code: clean_option(item.code.as_deref()),
|
||||
original_filename,
|
||||
is_attachment,
|
||||
format: "pdf".to_string(),
|
||||
as_of_date: None,
|
||||
}
|
||||
})
|
||||
.collect::<Vec<_>>();
|
||||
|
|
@ -278,6 +297,7 @@ pub fn discover_idx_ownership_reports(
|
|||
"IDX ownership announcement discovery",
|
||||
&query_url,
|
||||
&json_headers(),
|
||||
&validate_json_payload,
|
||||
)
|
||||
.and_then(|raw| parse_announcement_page(&raw))
|
||||
.and_then(|page| select_latest_ownership_reports(&page, &query_url, query.family))
|
||||
|
|
@ -287,6 +307,45 @@ pub fn discover_idx_ownership_reports(
|
|||
}
|
||||
}
|
||||
|
||||
let mut page_error = None;
|
||||
match fetch_text(
|
||||
"IDX share ownership data page discovery",
|
||||
&share_ownership_page_url(),
|
||||
&page_headers(),
|
||||
&validate_share_ownership_page,
|
||||
) {
|
||||
Ok(html) => {
|
||||
let mut page_reports = parse_share_ownership_page(&html, &share_ownership_page_url());
|
||||
if page_reports.is_empty() {
|
||||
// The page loaded (it has its file list) but nothing parsed:
|
||||
// the payload shape changed. Say so instead of silently
|
||||
// falling back to announcement-only results.
|
||||
page_error = Some(format!(
|
||||
"IDX share ownership data page {} loaded but no report entries could be parsed; the page layout may have changed",
|
||||
share_ownership_page_url()
|
||||
));
|
||||
}
|
||||
page_reports
|
||||
.retain(|report| family_filter.is_none_or(|family| family == report.family));
|
||||
// Keep the full above-1% history (snapshot backfill needs it), but
|
||||
// only the newest entry of the unsupported daily/breakdown series.
|
||||
page_reports.sort_by(|left, right| right.publish_date.cmp(&left.publish_date));
|
||||
let mut seen_unsupported = Vec::new();
|
||||
page_reports.retain(|report| {
|
||||
if report.family == OwnershipReportFamily::AboveOnePercent {
|
||||
return true;
|
||||
}
|
||||
if seen_unsupported.contains(&report.family) {
|
||||
return false;
|
||||
}
|
||||
seen_unsupported.push(report.family);
|
||||
true
|
||||
});
|
||||
discovered.extend(page_reports);
|
||||
}
|
||||
Err(err) => page_error = Some(err.to_string()),
|
||||
}
|
||||
|
||||
discovered.sort_by(|left, right| {
|
||||
right
|
||||
.publish_date
|
||||
|
|
@ -300,6 +359,15 @@ pub fn discover_idx_ownership_reports(
|
|||
discovered.truncate(limit);
|
||||
}
|
||||
|
||||
if !discovered.is_empty()
|
||||
&& let Some(err) = &page_error
|
||||
{
|
||||
// Announcement results still exist, but since June 2026 the newest
|
||||
// above-1% reports are only on the data page: don't hide its failure.
|
||||
crate::runtime::warn(err);
|
||||
}
|
||||
errors.extend(page_error);
|
||||
|
||||
if discovered.is_empty() {
|
||||
let detail = if errors.is_empty() {
|
||||
"no matching ownership reports found".to_string()
|
||||
|
|
@ -307,8 +375,9 @@ pub fn discover_idx_ownership_reports(
|
|||
errors.join("; ")
|
||||
};
|
||||
return Err(IdxError::Http(format!(
|
||||
"failed to discover IDX ownership reports from {}: {detail}",
|
||||
announcement_listing_url()
|
||||
"failed to discover IDX ownership reports from {} or {}: {detail}",
|
||||
announcement_listing_url(),
|
||||
share_ownership_page_url()
|
||||
)));
|
||||
}
|
||||
|
||||
|
|
@ -316,8 +385,12 @@ pub fn discover_idx_ownership_reports(
|
|||
}
|
||||
|
||||
pub fn download_idx_pdf(url: &str, target: &Path) -> Result<(), IdxError> {
|
||||
let bytes = fetch_bytes("IDX ownership PDF download", url, &pdf_headers())?;
|
||||
validate_pdf_payload(&bytes)?;
|
||||
let bytes = fetch_bytes(
|
||||
"IDX ownership PDF download",
|
||||
url,
|
||||
&pdf_headers(),
|
||||
&validate_pdf_payload,
|
||||
)?;
|
||||
|
||||
fs::write(target, &bytes).map_err(|e| {
|
||||
IdxError::Io(format!(
|
||||
|
|
@ -329,6 +402,239 @@ pub fn download_idx_pdf(url: &str, target: &Path) -> Result<(), IdxError> {
|
|||
Ok(())
|
||||
}
|
||||
|
||||
pub fn download_idx_xlsx(url: &str, target: &Path) -> Result<(), IdxError> {
|
||||
let bytes = fetch_bytes(
|
||||
"IDX ownership XLSX download",
|
||||
url,
|
||||
&xlsx_headers(),
|
||||
&validate_xlsx_payload,
|
||||
)?;
|
||||
|
||||
fs::write(target, &bytes).map_err(|e| {
|
||||
IdxError::Io(format!(
|
||||
"failed writing cached XLSX {}: {e}",
|
||||
target.display()
|
||||
))
|
||||
})?;
|
||||
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// XLSX is a ZIP container; Cloudflare challenges come back as HTML.
|
||||
pub fn validate_xlsx_payload(bytes: &[u8]) -> Result<(), IdxError> {
|
||||
if bytes.starts_with(b"PK\x03\x04") {
|
||||
return Ok(());
|
||||
}
|
||||
if looks_like_html(bytes) {
|
||||
return Err(IdxError::Http(
|
||||
"IDX ownership download returned HTML (likely a Cloudflare challenge) instead of an XLSX workbook".to_string(),
|
||||
));
|
||||
}
|
||||
Err(IdxError::Http(
|
||||
"IDX ownership download did not look like an XLSX workbook".to_string(),
|
||||
))
|
||||
}
|
||||
|
||||
fn validate_json_payload(bytes: &[u8]) -> Result<(), IdxError> {
|
||||
if looks_like_html(bytes) {
|
||||
return Err(IdxError::Http(
|
||||
"IDX ownership discovery returned HTML (likely a Cloudflare challenge) instead of announcement JSON".to_string(),
|
||||
));
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn validate_share_ownership_page(bytes: &[u8]) -> Result<(), IdxError> {
|
||||
if contains_bytes(bytes, b"Prospectus") {
|
||||
return Ok(());
|
||||
}
|
||||
Err(IdxError::Http(
|
||||
"IDX share ownership data page did not contain its file list (likely a Cloudflare challenge)".to_string(),
|
||||
))
|
||||
}
|
||||
|
||||
fn looks_like_html(bytes: &[u8]) -> bool {
|
||||
let head: Vec<u8> = bytes
|
||||
.iter()
|
||||
.skip_while(|byte| byte.is_ascii_whitespace())
|
||||
.take(256)
|
||||
.map(u8::to_ascii_lowercase)
|
||||
.collect();
|
||||
head.starts_with(b"<!doctype html") || head.starts_with(b"<html")
|
||||
}
|
||||
|
||||
fn contains_bytes(haystack: &[u8], needle: &[u8]) -> bool {
|
||||
haystack
|
||||
.windows(needle.len())
|
||||
.any(|window| window == needle)
|
||||
}
|
||||
|
||||
/// Parse the Data Kepemilikan Saham page. The page is server-rendered by
|
||||
/// Nuxt and embeds its file list as JS object literals such as
|
||||
/// `{Description:"Pemegang Saham di Atas 1% per 31 Agustus 2026",
|
||||
/// Prospectus:"https:\u002F\u002Fwww.idx.co.id\u002FMedia\u002F...xlsx",...}`.
|
||||
pub fn parse_share_ownership_page(html: &str, page_url: &str) -> Vec<DiscoveredOwnershipPdf> {
|
||||
let mut reports = Vec::new();
|
||||
let mut rest = html;
|
||||
|
||||
while let Some(start) = find_js_key(rest, "Description") {
|
||||
let (description, after_description) = match read_js_string(&rest[start..]) {
|
||||
Some(value) => value,
|
||||
None => {
|
||||
rest = &rest[start..];
|
||||
continue;
|
||||
}
|
||||
};
|
||||
let object_tail = &rest[start + after_description..];
|
||||
// The URL belongs to the same object literal, so it must come before
|
||||
// the next Description key.
|
||||
let window_end = find_js_key(object_tail, "Description").unwrap_or(object_tail.len());
|
||||
let window = &object_tail[..window_end];
|
||||
rest = object_tail;
|
||||
|
||||
let Some(url_start) = find_js_key(window, "Prospectus") else {
|
||||
continue;
|
||||
};
|
||||
let Some((url, _)) = read_js_string(&window[url_start..]) else {
|
||||
continue;
|
||||
};
|
||||
if let Some(report) = classify_page_entry(description.trim(), url.trim(), page_url) {
|
||||
reports.push(report);
|
||||
}
|
||||
}
|
||||
|
||||
reports
|
||||
}
|
||||
|
||||
/// Offset just past `Key:` / `"Key":` (the opening quote of the value).
|
||||
fn find_js_key(haystack: &str, key: &str) -> Option<usize> {
|
||||
let bare = format!("{key}:\"");
|
||||
let quoted = format!("\"{key}\":\"");
|
||||
[bare, quoted]
|
||||
.iter()
|
||||
.filter_map(|needle| {
|
||||
haystack
|
||||
.find(needle.as_str())
|
||||
.map(|index| index + needle.len() - 1)
|
||||
})
|
||||
.min()
|
||||
}
|
||||
|
||||
/// Read a JS/JSON string literal starting at the opening quote. Returns the
|
||||
/// decoded value and the byte offset just past the closing quote.
|
||||
fn read_js_string(input: &str) -> Option<(String, usize)> {
|
||||
let mut chars = input.char_indices();
|
||||
if chars.next()?.1 != '"' {
|
||||
return None;
|
||||
}
|
||||
let mut value = String::new();
|
||||
while let Some((index, ch)) = chars.next() {
|
||||
match ch {
|
||||
'"' => return Some((value, index + 1)),
|
||||
'\\' => match chars.next()?.1 {
|
||||
'u' => {
|
||||
let hex: String = (0..4)
|
||||
.filter_map(|_| chars.next().map(|(_, c)| c))
|
||||
.collect();
|
||||
value.push(char::from_u32(u32::from_str_radix(&hex, 16).ok()?)?);
|
||||
}
|
||||
'n' => value.push('\n'),
|
||||
't' => value.push('\t'),
|
||||
other => value.push(other),
|
||||
},
|
||||
_ => value.push(ch),
|
||||
}
|
||||
}
|
||||
None
|
||||
}
|
||||
|
||||
fn classify_page_entry(
|
||||
description: &str,
|
||||
url: &str,
|
||||
page_url: &str,
|
||||
) -> Option<DiscoveredOwnershipPdf> {
|
||||
let upper = description.to_uppercase();
|
||||
let family = if upper.contains("DI ATAS 1%") || upper.contains("ABOVE 1%") {
|
||||
OwnershipReportFamily::AboveOnePercent
|
||||
} else if upper.contains("DI ATAS 5%") || upper.contains("ABOVE 5%") {
|
||||
OwnershipReportFamily::AboveFivePercent
|
||||
} else if upper.contains("TIPE INVESTOR")
|
||||
|| upper.contains("KLASIFIKASI")
|
||||
|| upper.contains("INVESTOR TYPE")
|
||||
|| upper.contains("CLASSIFICATION")
|
||||
{
|
||||
OwnershipReportFamily::InvestorTypeBreakdown
|
||||
} else {
|
||||
return None;
|
||||
};
|
||||
|
||||
let path = url.split(['?', '#']).next().unwrap_or_default();
|
||||
let file_name = path.rsplit('/').next().unwrap_or_default().to_string();
|
||||
let lower_name = file_name.to_ascii_lowercase();
|
||||
let format = if lower_name.ends_with(".xlsx") {
|
||||
"xlsx"
|
||||
} else if lower_name.ends_with(".pdf") {
|
||||
"pdf"
|
||||
} else {
|
||||
return None;
|
||||
};
|
||||
let status = if family == OwnershipReportFamily::AboveOnePercent && format == "xlsx" {
|
||||
OwnershipReportStatus::Supported
|
||||
} else {
|
||||
OwnershipReportStatus::Unsupported
|
||||
};
|
||||
// Without an as-of date the entry cannot be ordered against other
|
||||
// months, and the snapshot manifest needs a date, so skip it.
|
||||
let as_of_date = parse_indonesian_as_of_date(description)?.to_string();
|
||||
|
||||
Some(DiscoveredOwnershipPdf {
|
||||
family,
|
||||
status,
|
||||
listing_page_url: page_url.to_string(),
|
||||
query_url: page_url.to_string(),
|
||||
pdf_url: url.to_string(),
|
||||
title: description.to_string(),
|
||||
publish_date: as_of_date.clone(),
|
||||
code: None,
|
||||
original_filename: (!file_name.is_empty()).then_some(file_name),
|
||||
is_attachment: true,
|
||||
format: format.to_string(),
|
||||
as_of_date: Some(as_of_date),
|
||||
})
|
||||
}
|
||||
|
||||
/// `"... per 31 Agustus 2026"` -> 2026-08-31 (Indonesian or English months).
|
||||
fn parse_indonesian_as_of_date(description: &str) -> Option<chrono::NaiveDate> {
|
||||
let tokens: Vec<String> = description
|
||||
.split_whitespace()
|
||||
.map(|token| {
|
||||
token
|
||||
.trim_matches(|c: char| !c.is_alphanumeric())
|
||||
.to_lowercase()
|
||||
})
|
||||
.collect();
|
||||
tokens.windows(3).rev().find_map(|window| {
|
||||
let day = window[0].parse::<u32>().ok()?;
|
||||
let month = match window[1].as_str() {
|
||||
"januari" | "january" | "jan" => 1,
|
||||
"februari" | "february" | "feb" => 2,
|
||||
"maret" | "march" | "mar" => 3,
|
||||
"april" | "apr" => 4,
|
||||
"mei" | "may" => 5,
|
||||
"juni" | "june" | "jun" => 6,
|
||||
"juli" | "july" | "jul" => 7,
|
||||
"agustus" | "august" | "agu" | "aug" => 8,
|
||||
"september" | "sep" => 9,
|
||||
"oktober" | "october" | "okt" | "oct" => 10,
|
||||
"november" | "nov" => 11,
|
||||
"desember" | "december" | "des" | "dec" => 12,
|
||||
_ => return None,
|
||||
};
|
||||
let year = window[2].parse::<i32>().ok()?;
|
||||
chrono::NaiveDate::from_ymd_opt(year, month, day)
|
||||
})
|
||||
}
|
||||
|
||||
pub fn validate_pdf_payload(bytes: &[u8]) -> Result<(), IdxError> {
|
||||
if bytes.is_empty() {
|
||||
return Err(IdxError::Http(
|
||||
|
|
@ -357,13 +663,23 @@ pub fn validate_pdf_payload(bytes: &[u8]) -> Result<(), IdxError> {
|
|||
))
|
||||
}
|
||||
|
||||
fn fetch_text(stage: &str, url: &str, headers: &[(String, String)]) -> Result<String, IdxError> {
|
||||
let bytes = fetch_bytes(stage, url, headers)?;
|
||||
fn fetch_text(
|
||||
stage: &str,
|
||||
url: &str,
|
||||
headers: &[(String, String)],
|
||||
accept: &dyn Fn(&[u8]) -> Result<(), IdxError>,
|
||||
) -> Result<String, IdxError> {
|
||||
let bytes = fetch_bytes(stage, url, headers, accept)?;
|
||||
String::from_utf8(bytes)
|
||||
.map_err(|e| IdxError::Http(format!("failed to decode {stage} response as utf-8: {e}")))
|
||||
}
|
||||
|
||||
fn fetch_bytes(stage: &str, url: &str, headers: &[(String, String)]) -> Result<Vec<u8>, IdxError> {
|
||||
fn fetch_bytes(
|
||||
stage: &str,
|
||||
url: &str,
|
||||
headers: &[(String, String)],
|
||||
accept: &dyn Fn(&[u8]) -> Result<(), IdxError>,
|
||||
) -> Result<Vec<u8>, IdxError> {
|
||||
let mut args = vec![
|
||||
"--silent".to_string(),
|
||||
"--show-error".to_string(),
|
||||
|
|
@ -377,7 +693,7 @@ fn fetch_bytes(stage: &str, url: &str, headers: &[(String, String)]) -> Result<V
|
|||
}
|
||||
args.push(url.to_string());
|
||||
|
||||
let output = curl_impersonate::run_owned(stage, &args)?;
|
||||
let output = curl_impersonate::run_owned_with_fallback(stage, &args, accept)?;
|
||||
Ok(output.stdout)
|
||||
}
|
||||
|
||||
|
|
@ -396,6 +712,35 @@ fn json_headers() -> Vec<(String, String)> {
|
|||
]
|
||||
}
|
||||
|
||||
fn page_headers() -> Vec<(String, String)> {
|
||||
vec![
|
||||
("User-Agent".to_string(), USER_AGENT.to_string()),
|
||||
(
|
||||
"Accept".to_string(),
|
||||
"text/html,application/xhtml+xml,*/*;q=0.8".to_string(),
|
||||
),
|
||||
(
|
||||
"Accept-Language".to_string(),
|
||||
"id-ID,id;q=0.9,en-US;q=0.8,en;q=0.7".to_string(),
|
||||
),
|
||||
]
|
||||
}
|
||||
|
||||
fn xlsx_headers() -> Vec<(String, String)> {
|
||||
vec![
|
||||
("User-Agent".to_string(), USER_AGENT.to_string()),
|
||||
(
|
||||
"Accept".to_string(),
|
||||
"application/vnd.openxmlformats-officedocument.spreadsheetml.sheet,application/octet-stream,*/*;q=0.8".to_string(),
|
||||
),
|
||||
(
|
||||
"Accept-Language".to_string(),
|
||||
"id-ID,id;q=0.9,en-US;q=0.8,en;q=0.7".to_string(),
|
||||
),
|
||||
("Referer".to_string(), share_ownership_page_url()),
|
||||
]
|
||||
}
|
||||
|
||||
fn pdf_headers() -> Vec<(String, String)> {
|
||||
vec![
|
||||
("User-Agent".to_string(), USER_AGENT.to_string()),
|
||||
|
|
@ -502,7 +847,8 @@ mod tests {
|
|||
use super::{
|
||||
AnnouncementPage, IDX_ANNOUNCEMENT_LISTING_URL, OwnershipReportFamily,
|
||||
OwnershipReportStatus, build_announcement_query_url, parse_announcement_page,
|
||||
select_latest_ownership_reports, validate_pdf_payload,
|
||||
parse_indonesian_as_of_date, parse_share_ownership_page, select_latest_ownership_reports,
|
||||
validate_pdf_payload, validate_share_ownership_page, validate_xlsx_payload,
|
||||
};
|
||||
use crate::error::IdxError;
|
||||
|
||||
|
|
@ -762,4 +1108,107 @@ mod tests {
|
|||
assert!(matches!(err, IdxError::Http(_)));
|
||||
assert!(err.to_string().contains("PDF/direct attachment"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn parses_share_ownership_page_entries() {
|
||||
let page_url = "https://www.idx.co.id/id/perusahaan-tercatat/data-kepemilikan-saham/";
|
||||
let reports = parse_share_ownership_page(
|
||||
include_str!("../../tests/fixtures/idx_share_ownership_page_excerpt.html"),
|
||||
page_url,
|
||||
);
|
||||
assert_eq!(reports.len(), 7);
|
||||
|
||||
let above1: Vec<_> = reports
|
||||
.iter()
|
||||
.filter(|r| r.family == OwnershipReportFamily::AboveOnePercent)
|
||||
.collect();
|
||||
assert_eq!(above1.len(), 3);
|
||||
assert!(
|
||||
above1
|
||||
.iter()
|
||||
.all(|r| r.status == OwnershipReportStatus::Supported)
|
||||
);
|
||||
assert!(above1.iter().all(|r| r.format == "xlsx"));
|
||||
|
||||
let august = above1
|
||||
.iter()
|
||||
.find(|r| r.as_of_date.as_deref() == Some("2026-08-31"))
|
||||
.expect("August above-1% entry");
|
||||
assert_eq!(
|
||||
august.pdf_url,
|
||||
"https://www.idx.co.id/Media/fahlw1o2/peng-2026-08-00017-satu-persen.xlsx"
|
||||
);
|
||||
assert_eq!(august.publish_date, "2026-08-31");
|
||||
assert_eq!(august.listing_page_url, page_url);
|
||||
assert_eq!(
|
||||
august.original_filename.as_deref(),
|
||||
Some("peng-2026-08-00017-satu-persen.xlsx")
|
||||
);
|
||||
|
||||
let above5 = reports
|
||||
.iter()
|
||||
.filter(|r| r.family == OwnershipReportFamily::AboveFivePercent)
|
||||
.count();
|
||||
let breakdown = reports
|
||||
.iter()
|
||||
.filter(|r| r.family == OwnershipReportFamily::InvestorTypeBreakdown)
|
||||
.count();
|
||||
assert_eq!((above5, breakdown), (2, 2));
|
||||
assert!(
|
||||
reports
|
||||
.iter()
|
||||
.filter(|r| r.family != OwnershipReportFamily::AboveOnePercent)
|
||||
.all(|r| r.status == OwnershipReportStatus::Unsupported)
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn share_ownership_page_handles_query_strings_and_dateless_entries() {
|
||||
let html = r#"{Description:"Pemegang Saham di Atas 1% per 31 Agustus 2026",Prospectus:"https://www.idx.co.id/Media/x/peng-satu-persen.xlsx?download=1",ListingDate:r},{Description:"Pemegang Saham di Atas 1%",Prospectus:"https://www.idx.co.id/Media/y/no-date.xlsx",ListingDate:r}"#;
|
||||
let reports = parse_share_ownership_page(html, "p");
|
||||
assert_eq!(
|
||||
reports.len(),
|
||||
1,
|
||||
"dateless entry skipped, query-string entry kept"
|
||||
);
|
||||
assert_eq!(reports[0].format, "xlsx");
|
||||
assert_eq!(reports[0].status, OwnershipReportStatus::Supported);
|
||||
assert_eq!(
|
||||
reports[0].original_filename.as_deref(),
|
||||
Some("peng-satu-persen.xlsx")
|
||||
);
|
||||
assert_eq!(reports[0].publish_date, "2026-08-31");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn share_ownership_page_without_payload_yields_nothing() {
|
||||
assert!(parse_share_ownership_page("<html>Just a moment...</html>", "p").is_empty());
|
||||
assert!(validate_share_ownership_page(b"<html>Just a moment...</html>").is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn parses_indonesian_and_english_as_of_dates() {
|
||||
let date = |y, m, d| chrono::NaiveDate::from_ymd_opt(y, m, d);
|
||||
assert_eq!(
|
||||
parse_indonesian_as_of_date("Pemegang Saham di Atas 1% per 31 Agustus 2026"),
|
||||
date(2026, 8, 31)
|
||||
);
|
||||
assert_eq!(
|
||||
parse_indonesian_as_of_date("Shareholders above 1% as of 29 May 2026"),
|
||||
date(2026, 5, 29)
|
||||
);
|
||||
assert_eq!(
|
||||
parse_indonesian_as_of_date("Pemegang Saham di Atas 1%"),
|
||||
None
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn validates_xlsx_payloads() {
|
||||
assert!(validate_xlsx_payload(b"PK\x03\x04rest-of-zip").is_ok());
|
||||
let err = validate_xlsx_payload(b"<!DOCTYPE html><title>Just a moment...</title>")
|
||||
.expect_err("html challenge rejected");
|
||||
assert!(err.to_string().contains("Cloudflare"), "{err}");
|
||||
assert!(validate_xlsx_payload(b"%PDF-1.7").is_err());
|
||||
}
|
||||
}
|
||||
|
|
|
|||
554
src/ownership/xlsx.rs
Normal file
554
src/ownership/xlsx.rs
Normal file
|
|
@ -0,0 +1,554 @@
|
|||
//! Parser for the IDX "Pemegang Saham di Atas 1%" XLSX workbook.
|
||||
//!
|
||||
//! Since June 2026 IDX publishes the monthly above-1% holder register as an
|
||||
//! XLSX file on the "Data Kepemilikan Saham" page instead of a PDF
|
||||
//! announcement. The workbook has one sheet: a disclaimer block, a `[PUBLIC]`
|
||||
//! marker, then a fixed 12-column table. Only that exact header is accepted,
|
||||
//! so the daily above-5% workbook and the investor-type / classification
|
||||
//! workbooks published on the same page are rejected instead of mis-imported.
|
||||
|
||||
use std::collections::HashMap;
|
||||
use std::fs;
|
||||
use std::io::{Cursor, Read};
|
||||
use std::path::Path;
|
||||
|
||||
use chrono::{Duration, NaiveDate};
|
||||
use quick_xml::Reader;
|
||||
use quick_xml::events::{BytesStart, Event};
|
||||
|
||||
use crate::error::IdxError;
|
||||
use crate::ownership::entities::{normalize_investor_type, normalize_locality, parse_ksei_date};
|
||||
use crate::ownership::types::KseiHoldingDraft;
|
||||
|
||||
const EXPECTED_HEADER: [&str; 12] = [
|
||||
"DATE",
|
||||
"SHARE_CODE",
|
||||
"ISSUER_NAME",
|
||||
"INVESTOR_NAME",
|
||||
"INVESTOR_CLASSIFICATION",
|
||||
"LOCAL_FOREIGN",
|
||||
"NATIONALITY",
|
||||
"DOMICILE",
|
||||
"HOLDINGS_SCRIPLESS",
|
||||
"HOLDINGS_SCRIP",
|
||||
"TOTAL_HOLDING_SHARES",
|
||||
"PERCENTAGE",
|
||||
];
|
||||
|
||||
/// Rows above this line are disclaimer text; the header must appear before it.
|
||||
const MAX_HEADER_SEARCH_ROWS: usize = 50;
|
||||
|
||||
pub fn supports_xlsx_file(path: &Path) -> bool {
|
||||
path.extension()
|
||||
.and_then(|value| value.to_str())
|
||||
.is_some_and(|value| value.trim().eq_ignore_ascii_case("xlsx"))
|
||||
}
|
||||
|
||||
pub fn parse_above1_xlsx_file(path: &Path) -> Result<Vec<KseiHoldingDraft>, IdxError> {
|
||||
let bytes = fs::read(path).map_err(|e| {
|
||||
IdxError::Io(format!(
|
||||
"failed to read ownership XLSX {}: {e}",
|
||||
path.display()
|
||||
))
|
||||
})?;
|
||||
parse_above1_xlsx_bytes(&bytes)
|
||||
}
|
||||
|
||||
pub fn parse_above1_xlsx_bytes(bytes: &[u8]) -> Result<Vec<KseiHoldingDraft>, IdxError> {
|
||||
let mut zip = zip::ZipArchive::new(Cursor::new(bytes))
|
||||
.map_err(|e| IdxError::ParseError(format!("failed to open ownership XLSX: {e}")))?;
|
||||
|
||||
let shared_strings = match read_zip_entry(&mut zip, "xl/sharedStrings.xml")? {
|
||||
Some(xml) => parse_shared_strings(&xml)?,
|
||||
None => Vec::new(),
|
||||
};
|
||||
let sheet_name = first_worksheet_name(&mut zip)?;
|
||||
let sheet_xml = read_zip_entry(&mut zip, &sheet_name)?
|
||||
.ok_or_else(|| IdxError::ParseError(format!("ownership XLSX is missing {sheet_name}")))?;
|
||||
|
||||
let rows = parse_sheet_rows(&sheet_xml, &shared_strings)?;
|
||||
drafts_from_rows(&rows)
|
||||
}
|
||||
|
||||
fn read_zip_entry(
|
||||
zip: &mut zip::ZipArchive<Cursor<&[u8]>>,
|
||||
name: &str,
|
||||
) -> Result<Option<String>, IdxError> {
|
||||
let mut file = match zip.by_name(name) {
|
||||
Ok(file) => file,
|
||||
Err(zip::result::ZipError::FileNotFound) => return Ok(None),
|
||||
Err(e) => {
|
||||
return Err(IdxError::ParseError(format!(
|
||||
"failed to read ownership XLSX entry {name}: {e}"
|
||||
)));
|
||||
}
|
||||
};
|
||||
let mut output = String::new();
|
||||
file.read_to_string(&mut output).map_err(|e| {
|
||||
IdxError::ParseError(format!("failed to decode ownership XLSX entry {name}: {e}"))
|
||||
})?;
|
||||
Ok(Some(output))
|
||||
}
|
||||
|
||||
fn first_worksheet_name(zip: &mut zip::ZipArchive<Cursor<&[u8]>>) -> Result<String, IdxError> {
|
||||
let mut sheets: Vec<String> = zip
|
||||
.file_names()
|
||||
.filter(|name| name.starts_with("xl/worksheets/") && name.ends_with(".xml"))
|
||||
.filter(|name| !name["xl/worksheets/".len()..].contains('/'))
|
||||
.map(str::to_string)
|
||||
.collect();
|
||||
// sheet1.xml, sheet2.xml, ... sort by their numeric suffix.
|
||||
sheets.sort_by_key(|name| {
|
||||
name.trim_start_matches("xl/worksheets/sheet")
|
||||
.trim_end_matches(".xml")
|
||||
.parse::<u32>()
|
||||
.unwrap_or(u32::MAX)
|
||||
});
|
||||
sheets
|
||||
.into_iter()
|
||||
.next()
|
||||
.ok_or_else(|| IdxError::ParseError("ownership XLSX has no worksheet".to_string()))
|
||||
}
|
||||
|
||||
fn xml_error(context: &str, err: impl std::fmt::Display) -> IdxError {
|
||||
IdxError::ParseError(format!("failed to parse ownership XLSX {context}: {err}"))
|
||||
}
|
||||
|
||||
/// Shared strings table: each `<si>` is one string, made of one `<t>` or
|
||||
/// several rich-text runs `<r><t>`. Phonetic hints (`<rPh>`) are skipped.
|
||||
fn parse_shared_strings(xml: &str) -> Result<Vec<String>, IdxError> {
|
||||
let mut reader = Reader::from_str(xml);
|
||||
let mut strings = Vec::new();
|
||||
let mut current: Option<String> = None;
|
||||
let mut in_text = false;
|
||||
let mut in_phonetic = false;
|
||||
|
||||
loop {
|
||||
match reader.read_event() {
|
||||
Ok(Event::Start(e)) => match e.local_name().as_ref() {
|
||||
b"si" => current = Some(String::new()),
|
||||
b"t" => in_text = true,
|
||||
b"rPh" => in_phonetic = true,
|
||||
_ => {}
|
||||
},
|
||||
Ok(Event::Empty(e)) if e.local_name().as_ref() == b"si" => strings.push(String::new()),
|
||||
Ok(Event::Text(text)) if in_text && !in_phonetic => {
|
||||
if let Some(value) = current.as_mut() {
|
||||
let unescaped = text
|
||||
.unescape()
|
||||
.map_err(|e| xml_error("shared strings", e))?;
|
||||
value.push_str(&unescaped);
|
||||
}
|
||||
}
|
||||
Ok(Event::End(e)) => match e.local_name().as_ref() {
|
||||
b"si" => strings.push(current.take().unwrap_or_default()),
|
||||
b"t" => in_text = false,
|
||||
b"rPh" => in_phonetic = false,
|
||||
_ => {}
|
||||
},
|
||||
Ok(Event::Eof) => break,
|
||||
Err(e) => return Err(xml_error("shared strings", e)),
|
||||
_ => {}
|
||||
}
|
||||
}
|
||||
|
||||
Ok(strings)
|
||||
}
|
||||
|
||||
/// One sheet row: its 1-based Excel row number and column letters
|
||||
/// (`A`, `B`, ...) mapped to cell text. Sheet XML omits blank rows, so the
|
||||
/// number comes from the row's `r` attribute, not its position.
|
||||
#[derive(Debug, Default)]
|
||||
struct SheetRow {
|
||||
number: usize,
|
||||
cells: HashMap<String, String>,
|
||||
}
|
||||
|
||||
fn row_number(element: &BytesStart<'_>, fallback: usize) -> Result<usize, IdxError> {
|
||||
for attribute in element.attributes() {
|
||||
let attribute = attribute.map_err(|e| xml_error("worksheet", e))?;
|
||||
if attribute.key.local_name().as_ref() == b"r" {
|
||||
if let Ok(number) = String::from_utf8_lossy(&attribute.value).trim().parse() {
|
||||
return Ok(number);
|
||||
}
|
||||
}
|
||||
}
|
||||
Ok(fallback)
|
||||
}
|
||||
|
||||
fn parse_sheet_rows(xml: &str, shared_strings: &[String]) -> Result<Vec<SheetRow>, IdxError> {
|
||||
let mut reader = Reader::from_str(xml);
|
||||
let mut rows = Vec::new();
|
||||
let mut row: Option<SheetRow> = None;
|
||||
let mut cell: Option<(String, String)> = None;
|
||||
let mut value = String::new();
|
||||
let mut in_value = false;
|
||||
|
||||
loop {
|
||||
match reader.read_event() {
|
||||
Ok(Event::Start(e)) => match e.local_name().as_ref() {
|
||||
b"row" => {
|
||||
row = Some(SheetRow {
|
||||
number: row_number(&e, rows.len() + 1)?,
|
||||
cells: HashMap::new(),
|
||||
})
|
||||
}
|
||||
b"c" => {
|
||||
cell = Some(cell_ref_and_type(&e)?);
|
||||
value.clear();
|
||||
}
|
||||
b"v" | b"t" if cell.is_some() => in_value = true,
|
||||
_ => {}
|
||||
},
|
||||
Ok(Event::Empty(e)) if e.local_name().as_ref() == b"row" => rows.push(SheetRow {
|
||||
number: row_number(&e, rows.len() + 1)?,
|
||||
cells: HashMap::new(),
|
||||
}),
|
||||
Ok(Event::Text(text)) if in_value => {
|
||||
let unescaped = text.unescape().map_err(|e| xml_error("worksheet", e))?;
|
||||
value.push_str(&unescaped);
|
||||
}
|
||||
Ok(Event::End(e)) => match e.local_name().as_ref() {
|
||||
b"v" | b"t" => in_value = false,
|
||||
b"c" => {
|
||||
if let (Some((column, cell_type)), Some(current_row)) =
|
||||
(cell.take(), row.as_mut())
|
||||
{
|
||||
let text = if cell_type == "s" {
|
||||
let index = value.trim().parse::<usize>().map_err(|e| {
|
||||
xml_error("worksheet", format!("bad shared string index: {e}"))
|
||||
})?;
|
||||
shared_strings.get(index).cloned().ok_or_else(|| {
|
||||
xml_error(
|
||||
"worksheet",
|
||||
format!("shared string {index} out of range"),
|
||||
)
|
||||
})?
|
||||
} else {
|
||||
value.clone()
|
||||
};
|
||||
current_row.cells.insert(column, text);
|
||||
}
|
||||
}
|
||||
b"row" => {
|
||||
if let Some(current_row) = row.take() {
|
||||
rows.push(current_row);
|
||||
}
|
||||
}
|
||||
_ => {}
|
||||
},
|
||||
Ok(Event::Eof) => break,
|
||||
Err(e) => return Err(xml_error("worksheet", e)),
|
||||
_ => {}
|
||||
}
|
||||
}
|
||||
|
||||
Ok(rows)
|
||||
}
|
||||
|
||||
fn cell_ref_and_type(element: &BytesStart<'_>) -> Result<(String, String), IdxError> {
|
||||
let mut column = String::new();
|
||||
let mut cell_type = String::new();
|
||||
for attribute in element.attributes() {
|
||||
let attribute = attribute.map_err(|e| xml_error("worksheet", e))?;
|
||||
let raw = String::from_utf8_lossy(&attribute.value).to_string();
|
||||
match attribute.key.local_name().as_ref() {
|
||||
b"r" => column = raw.chars().take_while(char::is_ascii_alphabetic).collect(),
|
||||
b"t" => cell_type = raw,
|
||||
_ => {}
|
||||
}
|
||||
}
|
||||
if column.is_empty() {
|
||||
return Err(xml_error("worksheet", "cell without a reference"));
|
||||
}
|
||||
Ok((column, cell_type))
|
||||
}
|
||||
|
||||
fn drafts_from_rows(rows: &[SheetRow]) -> Result<Vec<KseiHoldingDraft>, IdxError> {
|
||||
let (header_index, columns) = rows
|
||||
.iter()
|
||||
.take(MAX_HEADER_SEARCH_ROWS)
|
||||
.enumerate()
|
||||
.find_map(|(index, row)| header_columns(row).map(|columns| (index, columns)))
|
||||
.ok_or_else(|| {
|
||||
IdxError::ParseError(
|
||||
"ownership XLSX header did not match the above-1% holder register layout \
|
||||
(DATE, SHARE_CODE, ... PERCENTAGE); the above-5% and investor-type \
|
||||
workbooks are not supported"
|
||||
.to_string(),
|
||||
)
|
||||
})?;
|
||||
|
||||
let mut drafts = Vec::new();
|
||||
for row in &rows[header_index + 1..] {
|
||||
let line = row.number;
|
||||
let field = |name: &str| -> &str {
|
||||
columns
|
||||
.get(name)
|
||||
.and_then(|column| row.cells.get(column))
|
||||
.map(|value| value.trim())
|
||||
.unwrap_or("")
|
||||
};
|
||||
|
||||
let ticker_code = field("SHARE_CODE");
|
||||
let investor_name = field("INVESTOR_NAME");
|
||||
if ticker_code.is_empty() && investor_name.is_empty() {
|
||||
continue;
|
||||
}
|
||||
if ticker_code.is_empty() || investor_name.is_empty() {
|
||||
return Err(IdxError::ParseError(format!(
|
||||
"ownership XLSX row {line} is missing SHARE_CODE or INVESTOR_NAME"
|
||||
)));
|
||||
}
|
||||
|
||||
drafts.push(KseiHoldingDraft {
|
||||
ticker_code: ticker_code.to_string(),
|
||||
issuer_name: optional(field("ISSUER_NAME")),
|
||||
raw_investor_name: investor_name.to_string(),
|
||||
investor_type: normalize_investor_type(field("INVESTOR_CLASSIFICATION")),
|
||||
locality: normalize_locality(field("LOCAL_FOREIGN")),
|
||||
nationality: optional(field("NATIONALITY")),
|
||||
domicile: optional(field("DOMICILE")),
|
||||
holdings_scripless: parse_shares(field("HOLDINGS_SCRIPLESS"), line, false)?,
|
||||
holdings_scrip: parse_shares(field("HOLDINGS_SCRIP"), line, false)?,
|
||||
total_shares: parse_shares(field("TOTAL_HOLDING_SHARES"), line, true)?,
|
||||
percentage_bps: parse_percentage_bps(field("PERCENTAGE"), line)?,
|
||||
report_date: parse_report_date(field("DATE"), line)?,
|
||||
});
|
||||
}
|
||||
|
||||
if drafts.is_empty() {
|
||||
return Err(IdxError::ParseError(
|
||||
"ownership XLSX contained a header but no holder rows".to_string(),
|
||||
));
|
||||
}
|
||||
|
||||
Ok(drafts)
|
||||
}
|
||||
|
||||
/// Map header labels to column letters if `row` is exactly the expected header.
|
||||
fn header_columns(row: &SheetRow) -> Option<HashMap<&'static str, String>> {
|
||||
let mut found: HashMap<&'static str, String> = HashMap::new();
|
||||
for (column, text) in &row.cells {
|
||||
let mut label = text.trim().to_ascii_uppercase().replace(' ', "_");
|
||||
if label.is_empty() {
|
||||
continue;
|
||||
}
|
||||
// Feb/Mar 2026 workbooks name the column INVESTOR_TYPE and hold codes
|
||||
// (CP, ID, ...); later ones use INVESTOR_CLASSIFICATION with words.
|
||||
if label == "INVESTOR_TYPE" {
|
||||
label = "INVESTOR_CLASSIFICATION".to_string();
|
||||
}
|
||||
let expected = EXPECTED_HEADER.iter().find(|name| **name == label)?;
|
||||
found.insert(expected, column.clone());
|
||||
}
|
||||
(found.len() == EXPECTED_HEADER.len()).then_some(found)
|
||||
}
|
||||
|
||||
fn optional(value: &str) -> Option<String> {
|
||||
(!value.is_empty()).then(|| value.to_string())
|
||||
}
|
||||
|
||||
fn parse_number(raw: &str, line: usize, name: &str) -> Result<f64, IdxError> {
|
||||
raw.replace(',', "")
|
||||
.parse::<f64>()
|
||||
.ok()
|
||||
.filter(|value| value.is_finite())
|
||||
.ok_or_else(|| {
|
||||
IdxError::ParseError(format!("ownership XLSX row {line}: invalid {name} '{raw}'"))
|
||||
})
|
||||
}
|
||||
|
||||
fn parse_shares(raw: &str, line: usize, required: bool) -> Result<i64, IdxError> {
|
||||
if raw.is_empty() {
|
||||
return if required {
|
||||
Err(IdxError::ParseError(format!(
|
||||
"ownership XLSX row {line}: missing TOTAL_HOLDING_SHARES"
|
||||
)))
|
||||
} else {
|
||||
Ok(0)
|
||||
};
|
||||
}
|
||||
let value = parse_number(raw, line, "share count")?;
|
||||
if value < 0.0 || value.fract() != 0.0 || value > i64::MAX as f64 {
|
||||
return Err(IdxError::ParseError(format!(
|
||||
"ownership XLSX row {line}: share count '{raw}' is not a whole number"
|
||||
)));
|
||||
}
|
||||
Ok(value as i64)
|
||||
}
|
||||
|
||||
/// `54.94` (percent) -> 5494 basis points. Excel stores values such as
|
||||
/// `1.1499999999999999`, so round to the nearest basis point.
|
||||
fn parse_percentage_bps(raw: &str, line: usize) -> Result<i64, IdxError> {
|
||||
let value = parse_number(raw, line, "PERCENTAGE")?;
|
||||
if !(0.0..=100.0).contains(&value) {
|
||||
return Err(IdxError::ParseError(format!(
|
||||
"ownership XLSX row {line}: PERCENTAGE '{raw}' is out of range"
|
||||
)));
|
||||
}
|
||||
Ok((value * 100.0).round() as i64)
|
||||
}
|
||||
|
||||
/// DATE is an Excel serial day number (`46265` = 2026-08-31); accept the
|
||||
/// KSEI text form (`31-Aug-2026`) as well in case the cell is stored as text.
|
||||
fn parse_report_date(raw: &str, line: usize) -> Result<NaiveDate, IdxError> {
|
||||
if let Ok(serial) = raw.parse::<f64>() {
|
||||
let days = serial.trunc() as i64;
|
||||
let epoch = NaiveDate::from_ymd_opt(1899, 12, 30).expect("valid Excel epoch");
|
||||
if (1..=2_958_465).contains(&days) {
|
||||
return Ok(epoch + Duration::days(days));
|
||||
}
|
||||
}
|
||||
parse_ksei_date(raw).map_err(|_| {
|
||||
IdxError::ParseError(format!("ownership XLSX row {line}: invalid DATE '{raw}'"))
|
||||
})
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
fn fixture() -> Vec<KseiHoldingDraft> {
|
||||
parse_above1_xlsx_bytes(include_bytes!(
|
||||
"../../tests/fixtures/ksei_above1_20260831_excerpt.xlsx"
|
||||
))
|
||||
.expect("above-1% XLSX excerpt parses")
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn parses_above1_xlsx_excerpt() {
|
||||
let drafts = fixture();
|
||||
// Pinned so a regression that silently drops rows fails loudly.
|
||||
assert_eq!(drafts.len(), 17);
|
||||
assert!(
|
||||
drafts
|
||||
.iter()
|
||||
.all(|d| d.report_date == NaiveDate::from_ymd_opt(2026, 8, 31).unwrap())
|
||||
);
|
||||
|
||||
let dwimuria = drafts
|
||||
.iter()
|
||||
.find(|d| {
|
||||
d.ticker_code == "BBCA" && d.raw_investor_name == "PT DWIMURIA INVESTAMA ANDALAN"
|
||||
})
|
||||
.expect("BBCA controlling holder present");
|
||||
assert_eq!(dwimuria.percentage_bps, 5494);
|
||||
assert_eq!(
|
||||
dwimuria.investor_type.as_ref().map(|t| t.0.as_str()),
|
||||
Some("CORPORATE")
|
||||
);
|
||||
assert_eq!(
|
||||
dwimuria.total_shares,
|
||||
dwimuria.holdings_scripless + dwimuria.holdings_scrip
|
||||
);
|
||||
}
|
||||
|
||||
fn header_row(labels: &[&str]) -> SheetRow {
|
||||
sheet_row(1, labels)
|
||||
}
|
||||
|
||||
fn sheet_row(number: usize, values: &[&str]) -> SheetRow {
|
||||
SheetRow {
|
||||
number,
|
||||
cells: values
|
||||
.iter()
|
||||
.enumerate()
|
||||
.filter(|(_, value)| !value.is_empty())
|
||||
.map(|(index, value)| {
|
||||
(
|
||||
((b'A' + index as u8) as char).to_string(),
|
||||
value.to_string(),
|
||||
)
|
||||
})
|
||||
.collect(),
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn accepts_legacy_investor_type_header() {
|
||||
let mut labels = EXPECTED_HEADER.to_vec();
|
||||
labels[4] = "INVESTOR_TYPE";
|
||||
let columns = header_columns(&header_row(&labels)).expect("legacy header accepted");
|
||||
assert_eq!(columns["INVESTOR_CLASSIFICATION"], "E");
|
||||
|
||||
let row = sheet_row(
|
||||
2,
|
||||
&[
|
||||
"46112",
|
||||
"AADI",
|
||||
"ADARO ANDALAN INDONESIA Tbk",
|
||||
"GARIBALDI THOHIR",
|
||||
"ID",
|
||||
"L ",
|
||||
"INDONESIAN",
|
||||
"INDONESIA",
|
||||
"454011607",
|
||||
"0",
|
||||
"454011607",
|
||||
"5.83",
|
||||
],
|
||||
);
|
||||
let drafts = drafts_from_rows(&[header_row(&labels), row]).expect("legacy rows parse");
|
||||
assert_eq!(
|
||||
drafts[0].investor_type.as_ref().map(|t| t.0.as_str()),
|
||||
Some("ID")
|
||||
);
|
||||
assert_eq!(
|
||||
drafts[0].report_date,
|
||||
NaiveDate::from_ymd_opt(2026, 3, 31).unwrap()
|
||||
);
|
||||
assert!(drafts[0].locality.is_some());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn row_errors_report_excel_row_numbers() {
|
||||
let missing_investor = sheet_row(
|
||||
709,
|
||||
&["46265", "BBCA", "BANK CENTRAL ASIA Tbk", "", "Corporate"],
|
||||
);
|
||||
let err = drafts_from_rows(&[header_row(&EXPECTED_HEADER), missing_investor])
|
||||
.expect_err("row without investor rejected");
|
||||
assert!(err.to_string().contains("row 709"), "{err}");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn rejects_header_with_unknown_or_missing_columns() {
|
||||
let mut labels = EXPECTED_HEADER.to_vec();
|
||||
labels[11] = "PERUBAHAN";
|
||||
assert!(header_columns(&header_row(&labels)).is_none());
|
||||
assert!(header_columns(&header_row(&EXPECTED_HEADER[..11])).is_none());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn rounds_float_noise_in_percentages() {
|
||||
assert_eq!(parse_percentage_bps("1.1499999999999999", 1).unwrap(), 115);
|
||||
assert_eq!(parse_percentage_bps("41.1", 1).unwrap(), 4110);
|
||||
assert!(parse_percentage_bps("100.5", 1).is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn parses_excel_serial_and_text_dates() {
|
||||
let expected = NaiveDate::from_ymd_opt(2026, 8, 31).unwrap();
|
||||
assert_eq!(parse_report_date("46265", 1).unwrap(), expected);
|
||||
assert_eq!(parse_report_date("31-Aug-2026", 1).unwrap(), expected);
|
||||
assert!(parse_report_date("not a date", 1).is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn rejects_above5_workbook() {
|
||||
let err = parse_above1_xlsx_bytes(include_bytes!(
|
||||
"../../tests/fixtures/ksei_above5_20260923_excerpt.xlsx"
|
||||
))
|
||||
.expect_err("above-5% workbook must be rejected");
|
||||
assert!(
|
||||
err.to_string().contains("above-1% holder register layout"),
|
||||
"{err}"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn rejects_non_xlsx_bytes() {
|
||||
assert!(parse_above1_xlsx_bytes(b"%PDF-1.7 not a workbook").is_err());
|
||||
}
|
||||
}
|
||||
120
tests/cli.rs
120
tests/cli.rs
|
|
@ -407,7 +407,7 @@ fn ownership_import_help_mentions_sync_preference_and_fallback_files() {
|
|||
.success()
|
||||
.stdout(predicate::str::contains("Prefer `idx ownership sync`"))
|
||||
.stdout(predicate::str::contains(
|
||||
"PDF (primary), ZIP/TXT archive (fallback)",
|
||||
"above-1% XLSX or PDF (primary), ZIP/TXT archive (fallback)",
|
||||
))
|
||||
.stdout(predicate::str::contains(
|
||||
"idx ownership import --file /path/to/BalanceposEfek20260227.zip",
|
||||
|
|
@ -1189,6 +1189,43 @@ fn ownership_releases_uses_xdg_data_home_default_db_path() {
|
|||
);
|
||||
}
|
||||
|
||||
/// Keeps discovery tests off the live IDX data page (nothing listens on port 9).
|
||||
const UNREACHABLE_DATA_PAGE_URL: &str = "http://127.0.0.1:9/data-kepemilikan-saham/";
|
||||
|
||||
#[test]
|
||||
fn ownership_discover_finds_xlsx_on_share_ownership_page() {
|
||||
let page = fs::read_to_string("tests/fixtures/idx_share_ownership_page_excerpt.html")
|
||||
.expect("read share ownership page fixture");
|
||||
let page_url = spawn_single_response_server("text/html", page);
|
||||
|
||||
let output = test_bin("ownership-discover-data-page")
|
||||
.env("IDX_CURL_IMPERSONATE_BIN", "curl")
|
||||
.env(
|
||||
"IDX_OWNERSHIP_ANNOUNCEMENT_API_URL",
|
||||
"http://127.0.0.1:9/announcements",
|
||||
)
|
||||
.env("IDX_OWNERSHIP_DATA_PAGE_URL", &page_url)
|
||||
.args(["-o", "json", "ownership", "discover", "--limit", "1"])
|
||||
.output()
|
||||
.expect("ownership discover json output");
|
||||
assert!(
|
||||
output.status.success(),
|
||||
"{}",
|
||||
String::from_utf8_lossy(&output.stderr)
|
||||
);
|
||||
|
||||
let reports: Value = serde_json::from_slice(&output.stdout).expect("parse discover json");
|
||||
let latest = &reports[0];
|
||||
assert_eq!(latest["family"], "above_one_percent");
|
||||
assert_eq!(latest["status"], "supported");
|
||||
assert_eq!(latest["format"], "xlsx");
|
||||
assert_eq!(latest["as_of_date"], "2026-08-31");
|
||||
assert_eq!(
|
||||
latest["pdf_url"],
|
||||
"https://www.idx.co.id/Media/fahlw1o2/peng-2026-08-00017-satu-persen.xlsx"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn ownership_discover_lists_fixture_candidates() {
|
||||
let body = fs::read_to_string("tests/fixtures/idx_announcement_kepemilikan.json")
|
||||
|
|
@ -1198,6 +1235,7 @@ fn ownership_discover_lists_fixture_candidates() {
|
|||
test_bin("ownership-discover")
|
||||
.env("IDX_CURL_IMPERSONATE_BIN", "curl")
|
||||
.env("IDX_OWNERSHIP_ANNOUNCEMENT_API_URL", &json_url)
|
||||
.env("IDX_OWNERSHIP_DATA_PAGE_URL", UNREACHABLE_DATA_PAGE_URL)
|
||||
.env(
|
||||
"IDX_OWNERSHIP_ANNOUNCEMENT_PAGE_URL",
|
||||
"http://127.0.0.1/pengumuman",
|
||||
|
|
@ -1257,6 +1295,7 @@ fn ownership_discover_supports_above1_family() {
|
|||
test_bin("ownership-discover-above1")
|
||||
.env("IDX_CURL_IMPERSONATE_BIN", "curl")
|
||||
.env("IDX_OWNERSHIP_ANNOUNCEMENT_API_URL", &json_url)
|
||||
.env("IDX_OWNERSHIP_DATA_PAGE_URL", UNREACHABLE_DATA_PAGE_URL)
|
||||
.env(
|
||||
"IDX_OWNERSHIP_ANNOUNCEMENT_PAGE_URL",
|
||||
"http://127.0.0.1/pengumuman",
|
||||
|
|
@ -1274,6 +1313,10 @@ fn ownership_discover_supports_above1_family() {
|
|||
.stdout(predicate::str::contains("Above 1%"))
|
||||
.stdout(predicate::str::contains(
|
||||
"20260310_Semua Emiten Saham_Pengumuman Bursa_32052554_lamp1.pdf",
|
||||
))
|
||||
.stderr(predicate::str::contains("warning:"))
|
||||
.stderr(predicate::str::contains(
|
||||
"IDX share ownership data page discovery",
|
||||
));
|
||||
}
|
||||
|
||||
|
|
@ -1313,6 +1356,7 @@ fn ownership_discover_defaults_to_above1_and_prefers_supported_attachment() {
|
|||
let output = test_bin("ownership-discover-default")
|
||||
.env("IDX_CURL_IMPERSONATE_BIN", "curl")
|
||||
.env("IDX_OWNERSHIP_ANNOUNCEMENT_API_URL", &json_url)
|
||||
.env("IDX_OWNERSHIP_DATA_PAGE_URL", UNREACHABLE_DATA_PAGE_URL)
|
||||
.env(
|
||||
"IDX_OWNERSHIP_ANNOUNCEMENT_PAGE_URL",
|
||||
"http://127.0.0.1/pengumuman",
|
||||
|
|
@ -1367,6 +1411,7 @@ fn ownership_discover_json_includes_status() {
|
|||
test_bin("ownership-discover-json-status")
|
||||
.env("IDX_CURL_IMPERSONATE_BIN", "curl")
|
||||
.env("IDX_OWNERSHIP_ANNOUNCEMENT_API_URL", &json_url)
|
||||
.env("IDX_OWNERSHIP_DATA_PAGE_URL", UNREACHABLE_DATA_PAGE_URL)
|
||||
.env(
|
||||
"IDX_OWNERSHIP_ANNOUNCEMENT_PAGE_URL",
|
||||
"http://127.0.0.1/pengumuman",
|
||||
|
|
@ -1406,7 +1451,7 @@ fn ownership_import_url_rejects_listing_page_inputs() {
|
|||
.assert()
|
||||
.failure()
|
||||
.stderr(predicate::str::contains(
|
||||
"ownership import --url accepts direct PDF URLs only",
|
||||
"ownership import --url accepts direct XLSX or PDF URLs only",
|
||||
))
|
||||
.stderr(predicate::str::contains("ownership discover"));
|
||||
}
|
||||
|
|
@ -1524,6 +1569,77 @@ fn ownership_import_url_duplicate_release_is_skipped() {
|
|||
.stdout(predicate::str::contains("Release already imported"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn ownership_import_file_xlsx_supports_ticker_and_releases() {
|
||||
let root = test_env_dir("ownership-import-xlsx");
|
||||
let db_path = root.join("ownership.db");
|
||||
let xlsx_path = root.join("peng-2026-08-00017-satu-persen.xlsx");
|
||||
fs::write(
|
||||
&xlsx_path,
|
||||
include_bytes!("fixtures/ksei_above1_20260831_excerpt.xlsx"),
|
||||
)
|
||||
.expect("write xlsx fixture");
|
||||
|
||||
bin_with_root(&root)
|
||||
.args([
|
||||
"config",
|
||||
"set",
|
||||
"ownership.db_path",
|
||||
db_path.to_str().unwrap(),
|
||||
])
|
||||
.assert()
|
||||
.success();
|
||||
|
||||
bin_with_root(&root)
|
||||
.args(["ownership", "import", "--file", xlsx_path.to_str().unwrap()])
|
||||
.assert()
|
||||
.success()
|
||||
.stdout(predicate::str::contains("(as of 2026-08-31)"));
|
||||
|
||||
let output = bin_with_root(&root)
|
||||
.args(["-o", "json", "ownership", "ticker", "BBCA"])
|
||||
.output()
|
||||
.expect("ownership ticker json output");
|
||||
assert!(output.status.success());
|
||||
let body = String::from_utf8_lossy(&output.stdout);
|
||||
assert!(body.contains("DWIMURIA INVESTAMA ANDALAN"), "{body}");
|
||||
|
||||
let output = bin_with_root(&root)
|
||||
.args(["-o", "json", "ownership", "releases"])
|
||||
.output()
|
||||
.expect("ownership releases json output");
|
||||
let releases: Value = serde_json::from_slice(&output.stdout).expect("parse releases json");
|
||||
assert_eq!(releases[0]["as_of_date"], "2026-08-31");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn ownership_import_file_rejects_above5_xlsx() {
|
||||
let root = test_env_dir("ownership-import-xlsx-above5");
|
||||
let db_path = root.join("ownership.db");
|
||||
let xlsx_path = root.join("peng-2026-09-23-00080-lima-persen.xlsx");
|
||||
fs::write(
|
||||
&xlsx_path,
|
||||
include_bytes!("fixtures/ksei_above5_20260923_excerpt.xlsx"),
|
||||
)
|
||||
.expect("write xlsx fixture");
|
||||
|
||||
bin_with_root(&root)
|
||||
.args([
|
||||
"config",
|
||||
"set",
|
||||
"ownership.db_path",
|
||||
db_path.to_str().unwrap(),
|
||||
])
|
||||
.assert()
|
||||
.success();
|
||||
|
||||
bin_with_root(&root)
|
||||
.args(["ownership", "import", "--file", xlsx_path.to_str().unwrap()])
|
||||
.assert()
|
||||
.failure()
|
||||
.stderr(predicate::str::contains("above-1% holder register layout"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn ownership_import_force_reimports_existing_release() {
|
||||
let root = test_env_dir("ownership-import-force");
|
||||
|
|
|
|||
5
tests/fixtures/idx_share_ownership_page_excerpt.html
vendored
Normal file
5
tests/fixtures/idx_share_ownership_page_excerpt.html
vendored
Normal file
|
|
@ -0,0 +1,5 @@
|
|||
<!doctype html>
|
||||
<html lang="id"><head><title>Data Kepemilikan Saham - Bursa Efek Indonesia</title></head>
|
||||
<body><div id="__nuxt"><!-- trimmed excerpt of https://www.idx.co.id/id/perusahaan-tercatat/data-kepemilikan-saham/ (2026-09-24) --></div>
|
||||
<script>window.__NUXT__=(function(a,r){return {fetch:{"data-v-8c8e470a:0":{tableData:[{Description:"Pemegang Saham di Atas 5% per 23 September 2026",Prospectus:"https:\u002F\u002Fwww.idx.co.id\u002FMedia\u002Fsucngcwy\u002Fpeng-2026-09-23-00080-lima-persen.xlsx",ListingDate:"2026-09-23T12:00:00"},{Description:"Pemegang Saham di Atas 5% per 31 Agustus 2026",Prospectus:"https:\u002F\u002Fwww.idx.co.id\u002FMedia\u002Fumrls4il\u002Fpeng-2026-08-31-00063-lima-persen.xlsx",ListingDate:r},{Description:"Data Kepemilikan Saham Perusahaan Tercatat Berdasarkan Klasifikasi Investor per 31 Agustus 2026",Prospectus:"https:\u002F\u002Fwww.idx.co.id\u002FMedia\u002Fy0vjakdz\u002Fpeng-2026-08-00018-klasifikasi.xlsx",ListingDate:r},{Description:"Data Kepemilikan Saham Perusahaan Tercatat per Tipe Investor per 31 Agustus 2026",Prospectus:"https:\u002F\u002Fwww.idx.co.id\u002FMedia\u002Fwkjfudt3\u002Fpeng-2026-08-00019-tipe.xlsx",ListingDate:r},{Description:"Pemegang Saham di Atas 1% per 31 Agustus 2026",Prospectus:"https:\u002F\u002Fwww.idx.co.id\u002FMedia\u002Ffahlw1o2\u002Fpeng-2026-08-00017-satu-persen.xlsx",ListingDate:r},{Description:"Pemegang Saham di Atas 1% per 31 Juli 2026",Prospectus:"https:\u002F\u002Fwww.idx.co.id\u002FMedia\u002Ffjelt0en\u002Fpeng-2026-07-00016-satu-persen.xlsx",ListingDate:s},{Description:"Pemegang Saham di Atas 1% per 27 Februari 2026",Prospectus:"https:\u002F\u002Fwww.idx.co.id\u002FMedia\u002Fr2kjzpk5\u002Fpeng-02-00011-satu-persen.xlsx",ListingDate:K}]}}}}(null,"2026-09-01T12:00:00"));</script>
|
||||
</body></html>
|
||||
BIN
tests/fixtures/ksei_above1_20260831_excerpt.xlsx
vendored
Normal file
BIN
tests/fixtures/ksei_above1_20260831_excerpt.xlsx
vendored
Normal file
Binary file not shown.
BIN
tests/fixtures/ksei_above5_20260923_excerpt.xlsx
vendored
Normal file
BIN
tests/fixtures/ksei_above5_20260923_excerpt.xlsx
vendored
Normal file
Binary file not shown.
Loading…
Add table
Add a link
Reference in a new issue