From the full-repo Codex audit:
- MSN and Yahoo clients set connect and body timeouts but none for the
response headers, so a server that stalled before answering hung the
call; four such calls take every MCP call slot. Add a 30 s overall
deadline. Every curl-impersonate run gets --max-time 600 and a 256 MiB
--max-filesize, and output above that is rejected (curl cannot cap a
response without a length); the Yahoo cookie fetch gets --max-time.
- Ownership commands blocked on the database lock without limit, so a
long sync or import, or a stuck process, held MCP call slots
indefinitely. Wait up to 30 s, then fail with a "busy, try again" error.
- A price of 1e20 saturated `as i64` and the change subtraction then
overflowed. Prices that are not finite, negative, or above 1e12 IDR are
treated as missing, and a quote's own price as an error.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V3syn2fN2LNo5pvTSqc6eh
A pre-0.3.3 install interrupted between its renames left only a backup.
Restoring it inside the install step came after the decision, so an
older snapshot replaced a newer database without --force, and a failed
download left no database (Pullfrog review on #54).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V3syn2fN2LNo5pvTSqc6eh
From the Codex re-gate of the 0.3.3 fixes:
- Imports recorded row_count as the rows parsed, but INSERT OR IGNORE
drops a file's repeated ticker/investor rows, so the count could exceed
what is stored, and the new stored-row validation would then reject a
snapshot the publisher built from such a file. Record the rows stored.
- A local manifest over the 1 MiB limit was read truncated rather than
rejected; read one byte more and fail when it does not fit.
- The backup-recovery test also passed on the old code; it now fails an
install after the leftover backup was found and checks the old data is
back.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V3syn2fN2LNo5pvTSqc6eh
From the full-repo Codex audit:
- Install moved the database to a backup, then renamed the snapshot in;
a crash between the two left no database, and the next sync deleted the
backup first. Replace it with a single rename (old file or new, never
neither), and restore a backup an older version left behind.
- Validation trusted the release's row_count metadata, so a snapshot with
an empty or missing holdings table passed and replaced good data. Also
count the rows stored for the latest release.
- A manifest fetched over HTTP(S) could name a local download_url such as
/dev/zero, read without a size limit. Remote manifests must give an
HTTP(S) artifact; local reads take regular files only, within the size
limit, and manifests are capped at 1 MiB.
- A same-date build with a different row count replaced the local file
even when it did not list it in replaces_sqlite_sha256, so a replayed
older build could undo a corrected one. Same-date rebuilds now go only
through replaces_sqlite_sha256, which the publisher fills.
Verified against the published snapshot: fresh install, no-op re-sync,
and forced reinstall all succeed with no leftover files.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V3syn2fN2LNo5pvTSqc6eh
A KSEI file can carry rows from several report dates; its release is
dated by the latest. Filtering on the release alone pulled January rows
into a February comparison (Pullfrog review on #53).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V3syn2fN2LNo5pvTSqc6eh
From the Codex re-gate of the 0.3.3 fixes:
- `changes` broke ties between same-date releases by row id, but the
ticker, entity, and graph queries and the snapshot builder did not, so
two files imported in the same second could resolve differently per
query. Order by as_of_date, imported_at, then id everywhere.
- search::fts_search and search::rebuild_fts were public in a published
crate; keep them as deprecated wrappers.
- Invalid --source and --by are INVALIDINPUT, like other argument errors.
- The prefix-ranking test could not tell prefix from exact-alias ranking;
add one where only prefix vs substring differs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V3syn2fN2LNo5pvTSqc6eh
From a full-repo Codex audit, each reproduced on the synced ownership DB:
- `changes` paired an investor's accounts across releases instead of
comparing totals: KREN's KRESNA PRIMA INVEST went 9.44% -> 13.45% (a
second 4.01% account) but was reported as decreased 944 -> 401 bps.
Sum a holder's accounts per ticker, and read one release per date (the
latest imported, as other queries do), so a corrected file supersedes
the original instead of mixing with it.
- `changes` / get_ownership_changes accepted any dates: `from: ""`
reported all 7,154 holders as new, unknown dates returned no changes.
Dates must be YYYY-MM-DD imported releases with `from` before `to`.
- get_investor_holdings / `entity` with an empty name matched every
investor and returned an arbitrary one ("AMY"); `%` and `_` acted as
LIKE wildcards. Reject empty names; match wildcards literally.
- Alias search never worked: the FTS table declared an `aliases` column
its content table lacks (`no such column: T.aliases`), so searches
fell back to canonical names and a raw KSEI name such as
"PT ASTRA INTERNATIONAL TBK" found nothing. Search names and aliases
directly, ranked exact name, exact alias, prefix, then shorter names.
- `graph` enumerated every path (exponential in depth) and its cycle
check matched substrings, dropping entity:1 when entity:10 was on the
path. Traverse breadth-first with an exact visited set.
- Name normalization removed leading PT tokens from the front of a Vec,
quadratic in hostile names; narrow a slice instead.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V3syn2fN2LNo5pvTSqc6eh
Every zip 2.x release from 2.4.2 on is yanked; crates.io warned on each
0.3.2 publish. zip 9 returns entry names as a Result (it validates their
encoding): skip undecodable names, which can be neither the archive's TXT
payload nor an XLSX worksheet. zip 9 needs Rust 1.88, within our 1.89.
Closes#51.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V3syn2fN2LNo5pvTSqc6eh
String order put 10:00:00Z before the newer 10:00:00.500Z and mixed
time-zone offsets; with a small --limit the newest item could drop out.
Compare parsed RFC 3339 instants; missing or unparseable dates sort last.
From a Codex review.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V3syn2fN2LNo5pvTSqc6eh
MSN's feed for a stock is often mostly general market news: of 10 items,
ADRO, BREN and CUAN had none about the company, BBRI one, yet they were
presented as the company's news. Fetch at least 50 items, label each
`relevance: company` (tagged by MSN with the stock's instrument id, or
naming the ticker or the company) or `market`, and return company items
first, newest first, trimmed to --limit.
Company names come from the quote's short and display names, best
effort: MSN tags Telkom articles with its NYSE ADR, not TLKM, so names
catch what tags miss. Classification stays a pure function in parse_news.
Output change, intentional: news JSON gains `relevance`, the table an
ABOUT column (goldens regenerated), and the get_company tool description
explains the field (tools.json and the website tools page regenerated).
News cache keys change so entries without the field are not reused.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V3syn2fN2LNo5pvTSqc6eh
Final Codex (gpt-6.1-sol, xhigh) pass over c65982f confirmed every
earlier item fixed and found:
- high: readers skipped the lock on any error, so a lock file owned by
another user (who can still sync) let a reader run unlocked. Readers
now open an existing lock file read-only for the shared lock, and read
unlocked only when no lock file exists and none can be created.
- low: sync resolved the symlinked path for the install target and the
lock separately; it now resolves once and passes that path to the
lock callback, so both always name the same database.
Verified live: a reader with a read-only lock file waits for an
exclusive holder; install waits for readers; a symlinked path keeps its
link. CLI goldens unchanged; live A/B vs 0.2.4: 15 passed, screen hit a
transient MSN 408 on the old build only (retries pass), discover hits
the IDX 403 on both (exit 4 vs 1, intended).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Fourth Codex (gpt-6.1-sol, xhigh) pass over 5e0831d, each item verified
before fixing:
- sync inspects the local database under the shared lock, so another
sync cannot swap it under that connection.
- symlinks are resolved by reading the link (db::resolve_db_path), not
canonicalize, so a symlinked path finds the same lock and install
target even while the real file is missing (fresh install, mid-swap)
or the link dangles.
- readers go ahead unlocked when no lock file can be created; nothing
can replace the database there. (A read-only data directory still
fails in SQLite's WAL setup, as in 0.2.4.)
- XLSX shared strings are capped at 2,000,000 entries.
Verified live: an install waits for a reader's shared lock; a symlinked
db_path waits on the real path's lock and stays a symlink. CLI goldens
unchanged; live A/B vs 0.2.4: 16 passed, discover hits the IDX 403 on
both builds (exit 4 vs 1, intended).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Third Codex (gpt-6.1-sol, xhigh) pass over 1e55138, each item verified
before fixing:
- readers now hold the ownership lock shared for the life of their
connection, so a snapshot swap (exclusive) never happens under an
open SQLite connection whose WAL cleanup could hit a later import.
- the lock file name uses the resolved database path, so a symlinked
path shares the real database's lock; sync also installs at the
resolved path, keeping a symlinked path a symlink (it used to be
overwritten with a regular file).
- sync downloads without the lock and takes it only to recheck and
install; snapshot downloads have connect (15 s) and total (10 min)
timeouts. idx-mcp drops its in-process RwLock: idx-core's lock covers
it across processes.
- XLSX parsing caps rows at 1,000,000 and checks a shared string's size
before copying it.
- the temp snapshot's -wal/-shm files are removed instead of left in the
data directory.
Remaining, by design: `changes` loads all rows of two releases from the
local database before filtering; its size is bounded by the snapshot.
CLI goldens unchanged. Live A/B vs 0.2.4: 16 passed; discover hits the
IDX 403 on both builds (exit 4 vs 1, intended).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Second Codex (gpt-6.1-sol, xhigh) pass over 60634d3, each item verified
before fixing:
- new: the sync recheck hashed an existing empty database before schema
setup, so the first sync falsely reported an external change. Hash
after setup; a published snapshot already has the schema and keeps
its bytes.
- sync, import, and entity edits now hold a cross-process lock
(<db>.lock, std File::lock), so a sync cannot replace the database
under an import running in another process. Import validates and
downloads before taking the lock, keeping its error order. MSRV
1.85 -> 1.89 for File::lock.
- the row-count rebuild branch also never downgrades history.
- cache reads no longer delete corrupt entries (a concurrent writer may
have just replaced them); the next put overwrites atomically.
- idx-mcp validates the changes limit before querying and runs each
requested section or analysis once.
- XLSX parsing bounds total cell text, so one large shared string
referenced from many cells cannot exhaust memory.
- PDF/archive percentages must be 0-100%; HHI uses saturating math.
CLI goldens unchanged. Live A/B vs 0.2.4: 16 passed, discover hits the
IDX 403 on both builds (exit 4 vs 1, intended).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Findings from a Codex (gpt-6.1-sol, xhigh) review of v0.2.4..HEAD, each
verified in the code before fixing:
- high: ticker symbols reached cache file paths unchecked, so a symbol
like `../../REPORT` read (and deleted, if unparseable) .json files
outside the cache, reachable through MCP get_quotes. resolve_symbol now
accepts only ticker characters, and cache keys are encoded as single
safe path components.
- high: sync decided before downloading and replaced the database
without rechecking; an import that landed meanwhile was erased. It now
rechecks the local state before installing, and idx-mcp runs
ownership_sync exclusively with ownership reads.
- replaces_sqlite_sha256 never downgrades history: the client requires
the rebuild to keep at least the local release count, and the
publisher lists a replaced build only then.
- cache writes are atomic (temp file + rename), so concurrent readers
never see, and delete, a partial entry.
- idx-mcp caps symbols (20, deduplicated) and limits, and runs at most 4
tool calls at once.
- each Yahoo client gets its own cookie jar, paired with its crumb.
- ZIP entries (XLSX, KSEI archives) are read with a 256 MiB bound;
non-ASCII archive dates and huge percentages are parse errors instead
of panics or overflow.
- agent-smoke --keep deletes every copied agent config, not only auth
files.
CLI goldens unchanged; the MCP tool golden changes only to state the
new caps.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Sync hashed only the main database file, but local edits can sit in
the -wal sidecar, so an edited database could match an earlier build
and be replaced. Checkpoint the WAL before hashing; if changes remain,
never take the rebuilt-snapshot path.
- MCP tools rejected nothing for empty symbols/sections/kinds and
returned an empty success; they now fail with INVALIDINPUT, as the CLI
requires input.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A snapshot republished for the same as-of date (the TRUE ticker fix)
kept the date, latest release, and release count, so sync saw nothing
to do and existing installs kept the bad data without --force.
The manifest gains an optional snapshot.replaces_sqlite_sha256 listing
earlier builds for that date. Queries never write to a synced file, so
a local database byte-identical to a listed build is replaced; one that
matches nothing (local imports or edits) still needs --force. The
publisher fills the list on a same-date rebuild. Older clients ignore
the field.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Phase 4 of docs/DESIGN.md. Eleven grouped tools (quotes, history, screen,
company, analysis, compare, shareholders, investor holdings, search,
ownership changes, status); ownership_sync is listed only with
--allow-sync. Live results keep the Fetched envelope, errors use the CLI's
{error, code, message} body, and investor tools on an empty database
point at the sync path.
idx-core gains Period/Interval FromStr, Investor::status, a ticker
filter on changes, and Display for ErrorCode (now used by the CLI too).
CLI output and goldens are unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The IDX above-1% workbook stores the share code TRUE (TRINITI DINAMIK)
as a boolean cell holding 1, so its holders were imported under ticker
"1". Map t="b" cells to what Excel displays.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Phase 3c of docs/DESIGN.md. `ownership` handlers are now argument
parsing, one idx-core call, and rendering.
- investor.rs: Investor (also Idx::investor()) with releases, ticker,
entity, search, cross_holders, concentration, flow, changes, graph,
unresolved, map_alias, merge_entities, discover, sync, and import.
Import orchestration (input resolution, download, SHA-256 dedup,
PDF/XLSX/archive parsing, entity resolution, release write) moves
from the CLI unchanged.
- Typed inputs: ImportSource, HolderSource, ConcentrationRank, and
parse_report_family, with the same error messages.
- CoreConfig gains an ownership_db override; db::open_db_at opens an
explicit path. idx-cli no longer depends on rusqlite, sha2, or ureq
at runtime.
- Behavior note: an invalid `ticker --source` or `concentration --by`
is now rejected before the database file is opened (and created);
the error and exit code are unchanged.
- Fix: building idx-cli without the ownership feature no longer warns
about the ownership-only `blue` style.
All 143 goldens unchanged; 170 core + 7 CLI unit tests and 62 CLI tests
pass; clippy clean for default, release, testing, and no-default
feature sets. Live A/B against main: 16/16 stocks identical; ownership
discover got IDX 403 on both (exit 4 vs 1 is the phase 1 change).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Phase 3b of docs/DESIGN.md. `stocks` handlers are now argument parsing,
one idx-core call, and rendering.
- Idx (client.rs) is the entry point: Idx::new(CoreConfig) with
market(), financial(), and analysis(). Live lookups return
Fetched<T> { data, source, fetched_at, stale } and take a CachePolicy
(Default | Offline | Refresh, from the --offline/--no-cache flags).
- One cached_fetch replaces the CLI's three cache flows; they differed
only in whether a fresh entry is read via get or get_offline when
offline, which yields the same data and diagnostics. Report-building
errors (technical) still skip the stale-cache fallback.
- market.rs: quotes, history (history_source is resolved first so the
CLI's fallback note keeps its position), and a typed screener
(ScreenFilter/ScreenRegion with the same error messages).
- financial.rs: profile, statements, earnings, sentiment, insights,
news, with StatementFilter/EarningsFilter.
- analysis: TechnicalReport moves to analysis::report; Analysis offers
technical, growth, valuation, risk, fundamental, and compare.
- CoreConfig::load() owns provider/exchange/TTL settings from
config.toml and IDX_*; the CLI's IdxConfig adds output and color.
With two different invalid settings at once, the reported one may
differ from before; every single-error message is unchanged.
- Cache gains fresh/any/offline accessors returning fetch metadata.
Unit tests moved with their code (core 157 -> 168, CLI 17 -> 7). All
143 goldens unchanged; live A/B against main: 16/16 stocks identical.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Phase 3a of docs/DESIGN.md. MockProvider moves to api/mock.rs, and it,
the embedded MSN fixture bodies, and the IDX_MOCK_* hooks compile only
with idx-core's new `testing` feature. idx-cli enables it as a
dev-dependency, so `cargo test` keeps the fixture-backed provider while
release binaries no longer embed fixture data or honor
IDX_USE_MOCK_PROVIDER. Parser helpers used only by the mock are gated
the same way.
The packaged-cli flake check no longer relies on the mock; it checks
an offline cache miss (JSON CACHEMISS, exit 3) instead.
Goldens unchanged; clippy is clean with and without `testing`.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Phase 2 of docs/DESIGN.md, no behavior change:
- crates/idx-core (library): api, analysis, cache, config file handling
and provider selection, curl_impersonate, error, ownership, runtime,
plus the shared test fixtures. clap derives are behind an optional
`clap` feature so the library does not require clap.
- crates/idx-cli (binary `idx`): cli/, output/, and IdxConfig (output
format and color resolution). Package and binary names are unchanged.
- config.rs is split: file read/write and validation move to core; a
CLI test keeps core's OUTPUT_VALUES in sync with OutputSetting.
- ownership::db::open_db/db_path drop their unused IdxConfig argument.
- One workspace version; shared dependency versions; no third-party
version changes in Cargo.lock. LICENSE is symlinked into each crate.
- flake: workspace version, new source paths, and a per-crate package
check (idx-cli is built from its tarball against packaged idx-core,
since idx-core is not on crates.io yet). CI cache key, release path
filter, and the golden whitespace-hook exclusion follow the new layout.
Parity: all 143 golden outputs match byte for byte without
regeneration; 157 core + 17 CLI unit tests (173 before plus the new
OUTPUT_VALUES sync test), 62 CLI tests pass. Live A/B against main:
16/16 stocks cases identical; ownership discover got IDX 403 on both
(exit 4 vs 1 is the phase 1 exit-code change).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
