MSN stamps each IDX trading day at local midnight (17:00Z the previous
day), so every MSN candle was labelled one day early. Yahoo weekly and
monthly bars have the same stamp and were labelled Sunday / last day of
the previous month. Take the date in the exchange offset instead: Yahoo's
meta.gmtoffset when present, else WIB (UTC+7).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ld3fDSDiBzAaQvgZJX5NKM
Move fmt, clippy, tests, cargo package verification and the offline mock
smoke into flake checks built with crane, so the crate dependency build is
its own derivation that is reused until Cargo.lock changes. CI runs a single
nix flake check and restores /nix/store from the GitHub Actions cache via
nix-community/cache-nix-action, replacing the rustup + cargo test job.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013cfWL2GVg97oLfkNvPLaci
--no-default-features failed to compile because bing.rs imports
crate::ownership types.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Q2PufCUdvVZoVW1Dy53js
- --offline now warns when it serves a cache entry past its TTL; before,
arbitrarily old data was returned with no signal at all
- version, cache info and cache clear honour -o json
- ownership resolve map/merge honour -o json
- stocks quote/compare require at least one symbol instead of printing []
- cache schema-mismatch notice goes through runtime::info so --quiet
silences it, instead of a raw debug eprintln
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Q2PufCUdvVZoVW1Dy53js
Addresses review on #33: Yahoo auto-discovers only curl_chrome*; the
ownership fallback also tries Firefox and Safari profiles.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017a5Cv5Z1NzdYKgKnTiAF4E
- Install: add npm/npx (with the pnpm/bun opt-in), platform coverage,
pin example v0.2.4; Cargo builds from source.
- Runtime helpers as a table, corrected: curl-impersonate is needed for
IDX ownership discover/import and Yahoo fundamentals
(IDX_PROVIDER=yahoo), not for stocks history; mutool only for legacy
PDF reports. Everyday use needs neither.
- Quick start and Features now cover ownership (sync, ticker, entity,
search, changes, concentration, cross-holders, graph; XLSX source on
the IDX Data Kepemilikan Saham page) and the full stocks command set
and providers. Every example was run against v0.2.4.
- Documentation index; script test suites.
- docs: example versions -> v0.2.4; npm publish needs 2FA and a fresh
version number.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017a5Cv5Z1NzdYKgKnTiAF4E
Addresses review on #31:
- the snapshot download limit is the manifest's size_bytes (+1, since
ureq fails once a body reaches its limit), capped at 1 GiB; a body
larger than advertised fails during download. New CLI test; the
>10 MiB test also caught the off-by-one.
- a refused (oversized) standalone build removes its sqlite and
manifest from --output-dir.
- docs: downloads are buffered, not streamed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017a5Cv5Z1NzdYKgKnTiAF4E
On 2026-09-25 a snapshot built with --history 5 (15.3 MB) was
published, and `idx ownership sync` failed for every client:
"the response body is larger than request limit: 10485760" (ureq's
default body cap). It was republished by hand with --history 2 (8.2 MB).
- snapshot download: raise the ureq body limit to 1 GiB; the file is
still verified against the manifest's size and SHA-256. The PDF/XLSX
report downloads get a 256 MiB limit for the same reason.
- CLI test: sync a >10 MiB snapshot over HTTP (fails without the fix
with the exact production error).
- build-latest-ownership-snapshot.sh: --max-snapshot-bytes (default
10485760, the limit of released clients <= v0.2.3); an oversized
snapshot fails the build, so the published one stays untouched.
- publish-ownership-snapshot.sh passes --max-snapshot-bytes through;
publisher tests cover the failure and that the release is unchanged.
- docs: size table and guidance.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017a5Cv5Z1NzdYKgKnTiAF4E
Addresses review on #29:
- drop errtrace: a failure inside $(...) printed two RESULT lines
- explicit failures go through fail(), which always prints
RESULT: FAILED stage=...; idx check gets its own `preflight` stage;
the never-failing published-manifest read is no longer a stage
- validate the built latest_as_of_date (`// empty` + date regex) so a
corrupt manifest cannot compare as "newer"
- freshness: validate --now (exit 2) and latest_as_of_date (exit 3)
before calling date
- flake: package an explicit script list instead of scripts/*.sh
- docs: GH_TOKEN prerequisite for the packaged publisher
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017a5Cv5Z1NzdYKgKnTiAF4E
The self-hosted publish timer has not produced an upload since it was
enabled (manifest last updated 2026-07-14, a manual run), and failures
were invisible. Make the job safe to run daily and easy to deploy:
- publish-ownership-snapshot.sh: skip the upload when the published
manifest already has the latest as-of date (pre-check via discover
for XLSX sources, post-build check for legacy PDF); --force to
override. Always ends with one line: RESULT: published|up-to-date
<as-of> or RESULT: FAILED stage=<stage> (exit N). gh release create
no longer requires a git checkout.
- check-ownership-snapshot-freshness.sh: exit 1 when the published
latest_as_of_date is older than --max-age-days (default 40).
- flake: packages.ownership-publisher bundles the scripts with a
pinned idx (curl-impersonate + mupdf) and jq/gh/sqlite/curl as
bin/idx-ownership-publish and bin/idx-ownership-freshness, so hosts
need no checkout, nix develop or cargo build at run time. Built in CI.
- contrib/systemd + docs: daily 06:00 UTC timer, run semantics table,
freshness alarm, packaged usage.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017a5Cv5Z1NzdYKgKnTiAF4E
Addresses review on #27: canonicalize --history ("08" no longer hits
bash octal parsing), run the history jq in a plain command
substitution so a jq failure aborts under set -e, and restore the
schedule-guidance paragraph in OWNERSHIP_SELF_HOSTED.md.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017a5Cv5Z1NzdYKgKnTiAF4E
- build-latest-ownership-snapshot.sh: discover with a wider limit and
add --history <n> to import the n previous supported above-1% XLSX
months (one per as-of date, oldest first) before the latest, so
`idx ownership changes` works straight from a synced snapshot.
- publish-ownership-snapshot.sh: pass --history through (for the
clan-private systemd timer via extraArgs).
- publish-ownership-snapshot.yml: `history` input (default 5).
- Docs: publish/self-hosted flow now describes the XLSX source,
observed upload days (2nd-3rd of the month), and --history.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017a5Cv5Z1NzdYKgKnTiAF4E
Addresses review on #26:
- strip ?query/#fragment before the .xlsx/.pdf extension check
- skip page entries without a parseable as-of date instead of emitting
an empty publish_date
- warn on stderr when the data page fails or parses to zero entries
while announcement results still exist, instead of silently
degrading to announcement-only results
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017a5Cv5Z1NzdYKgKnTiAF4E
IDX stopped posting the monthly above-1% list as an announcement after
the 2026-05-29 report; June/July/August are only on the Data
Kepemilikan Saham page. `ownership discover` could not see them, and
IDX's Cloudflare now rejects curl_chrome142 on that page and on the
/Media/ XLSX files.
- remote: parse the page's Nuxt payload (Description/Prospectus pairs,
/ escapes), classify above-1% XLSX as supported and above-5% /
investor-type workbooks as unsupported, derive as_of_date from the
Indonesian description. Keep all above-1% entries, only the newest
of the other series. Merge with the announcement results.
- discover JSON gains `format` (xlsx|pdf) and `as_of_date`; page
entries use the as-of date as publish_date (the page shows none).
- import --url accepts .xlsx and downloads via the XLSX validator.
- curl_impersonate: run_owned_with_fallback tries several browser
profiles until the body validates (JSON / page payload / %PDF / PK);
IDX_CURL_IMPERSONATE_BIN still pins one.
- Tests: page fixture parsing, date parsing, payload validation, CLI
discover via a local page server; existing discover tests now point
IDX_OWNERSHIP_DATA_PAGE_URL at a closed port so no test hits IDX.
- Docs: ARCHITECTURE discovery section, AGENTS ownership inputs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017a5Cv5Z1NzdYKgKnTiAF4E
Addresses review on #25: sheet XML omits blank rows, so error messages
now use each <row r=...> number instead of the position; the excerpt
test pins the exact row count so dropped rows fail loudly.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017a5Cv5Z1NzdYKgKnTiAF4E
The February and March 2026 above-1% workbooks name the investor
column INVESTOR_TYPE (codes such as CP, ID); April onward uses
INVESTOR_CLASSIFICATION (words). Treat both as the same field; every
other column stays strict.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017a5Cv5Z1NzdYKgKnTiAF4E
Since June 2026 IDX publishes the monthly "Pemegang Saham di Atas 1%"
list as an XLSX workbook on the Data Kepemilikan Saham page instead of
a PDF announcement. Add a parser for it and accept `.xlsx` in
`ownership import --file`.
- src/ownership/xlsx.rs: zip + quick-xml reader (shared strings incl.
rich text, sheet cells), exact 12-column header match so the daily
above-5% and investor-type/classification workbooks are rejected,
Excel serial dates, numeric shares, percent -> bps with float-noise
rounding. Builds KseiHoldingDraft directly and reuses the entity
normalizers, so investor types stay uppercase like the PDF path.
- Fixtures: trimmed excerpts of the real 2026-08-31 above-1% and
2026-09-23 above-5% workbooks.
- Unit + CLI tests: import, ticker, releases, above-5% rejection.
- Docs: ARCHITECTURE parser pipeline, AGENTS ownership inputs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017a5Cv5Z1NzdYKgKnTiAF4E
- install.sh: POSIX sh installer that detects OS/arch (incl. Rosetta),
resolves the newest vX.Y.Z release via the GitHub API (skipping
ownership-snapshot-current), verifies SHA256SUMS, sanity-runs the
binary, and installs atomically into ~/.local/bin without sudo.
Supports --version/--dir and IDX_* env overrides; prints a PATH
hint, a NixOS flake suggestion, and helper-tool install hints
instead of calling package managers.
- scripts/install-sh-test.sh: fixture tests against a local fake
release server (latest, pinned, piped, checksum mismatch, missing
release, unsupported OS, PATH hint, reinstall).
- ci.yml: install-script job (ShellCheck, sh + dash on Linux, sh on
macOS).
- release.yml: attach install.sh to each release.
- README + docs/INSTALL.md.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017a5Cv5Z1NzdYKgKnTiAF4E
- release.yml: build static musl Linux binaries (no host glibc
requirement), add darwin-x64, publish SHA256SUMS, allow
workflow_dispatch for an existing tag (v0.2.3 has no binaries yet),
and run build-only on PRs that touch the workflow or installer
- install.js: verify downloads against the release SHA256SUMS
(IDX_BINARY_SHA256 for overrides), add a 60s request timeout, use
stream.pipeline so truncated downloads fail, and skip the download
inside the source checkout
- idx.js: forward SIGTERM/SIGHUP to the binary and keep the signal
exit status
- package.json: drop the prepare script (no Rust toolchain needed to
pack/publish)
- npm-smoke.sh: read the tarball name from npm pack, use grep over rg
- .gitignore: ignore /bin/ and *.tgz
- docs: musl targets, checksums, pnpm/bun opt-in, manual release flow
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017a5Cv5Z1NzdYKgKnTiAF4E
`.agents/setup` now pre-loads the CLI's local data after the build: it
installs the published ownership snapshot (`idx ownership sync`, the
repo's preferred bootstrap path) and warms the file cache for the shipped
`stocks` read commands, so a fresh orb can query ownership data and serve
`--offline` reads immediately.
Setup also adopts the dev shell environment for its remaining steps via
`nix print-dev-env` (the same mechanism the login-shell hook uses), and
the orb guidance block is replaced in place so text updates reach orbs
that already have one without accumulating blank lines.
Amp-Thread-ID: https://ampcode.com/threads/T-01a0ba76-221d-728b-9d68-45ab48750d3d
Co-authored-by: The Diligence Dev <thediligencedev@gmail.com>
Fresh Amp orbs start without a Rust toolchain or Nix, so every agent would
repeat the same install. `.agents/setup` installs Nix (single-user),
realizes the flake dev shell, activates it for the login shells Amp and orb
services start, and warms target/ so the first agent command is incremental.
`.agents/resume` re-checks the toolchain after a pause.
Also ignore .amp/portals/ so orb runtime files stay out of git status.
Amp-Thread-ID: https://ampcode.com/threads/T-01a0ba76-221d-728b-9d68-45ab48750d3d
Co-authored-by: The Diligence Dev <thediligencedev@gmail.com>
Add an npm wrapper (package.json + postinstall downloader) so the Rust
binary can be installed via npm/npx. Include a local smoke test, a GitHub
Release cross-compile workflow, and nodejs in the dev shell.
- package.json: idx-cli v0.2.3, bin idx, engines node>=16
- scripts/install.js: dependency-free postinstall that downloads the
platform-specific GitHub Release asset (or IDX_BINARY_URL override)
- scripts/idx.js: binary launcher preserving exit codes/signals
- scripts/npm-smoke.sh: npm pack -> install tarball -> run idx --help
and idx config --help, locale, no publish required
- .github/workflows/release.yml: cross-compile linux-x64/arm64 and
darwin-arm64 releases and attach binaries to the version tag
- flake.nix: add nodejs_22 to the dev shell
- docs/NPM_DISTRIBUTION.md: usage, local test, and release flow
- sanitize_current_ratio: and_then if/else -> filter
- screener/graph sorts: sort_by with reversed cmp -> sort_by_key(Reverse)
- insert_ksei_holdings_rows: manual Result match -> ?
Fixes the Clippy step on ubuntu-latest (rust 1.97) that the nix
toolchain (1.93.1) did not flag.
Newer KSEI above1 PDFs use spaces in column headers (SHARE CODE,
INVESTOR NAME, INVESTOR CLASSIFICATION) instead of the underscored
form (SHARE_CODE, INVESTOR_NAME, INVESTOR_TYPE) that the parser
schema markers expected. This caused valid holder-register PDFs to
be misclassified as LegacyAboveFivePercent (false positive on
REKENING TAMPUNGAN KSEI text in the data), blocking import.
Changes:
- Add normalize_stext_for_classification() that replaces spaces with
underscores inside text="..." attributes before schema marker
matching, so both old and new header layouts are recognized
- Add INVESTOR_CLASSIFICATION to HOLDER_REGISTER_SCHEMA_MARKERS and
HEADER_LABELS to match the renamed column
- Update ANNOUNCEMENT_WRAPPER and ABOVE_FIVE marker constants to use
underscores (matching the normalized text)
- Add test fixture and test case for the spaced-header layout
- Bump version to 0.2.3
Tested against the live June 2026 KSEI PDF (as_of 2026-05-29):
idx ownership import --url <latest-pdf-url>
-> Imported 7124 rows for 956 tickers (as of 2026-05-29)
