Final Codex (gpt-6.1-sol, xhigh) pass over c65982f confirmed every
earlier item fixed and found:
- high: readers skipped the lock on any error, so a lock file owned by
another user (who can still sync) let a reader run unlocked. Readers
now open an existing lock file read-only for the shared lock, and read
unlocked only when no lock file exists and none can be created.
- low: sync resolved the symlinked path for the install target and the
lock separately; it now resolves once and passes that path to the
lock callback, so both always name the same database.
Verified live: a reader with a read-only lock file waits for an
exclusive holder; install waits for readers; a symlinked path keeps its
link. CLI goldens unchanged; live A/B vs 0.2.4: 15 passed, screen hit a
transient MSN 408 on the old build only (retries pass), discover hits
the IDX 403 on both (exit 4 vs 1, intended).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Fourth Codex (gpt-6.1-sol, xhigh) pass over 5e0831d, each item verified
before fixing:
- sync inspects the local database under the shared lock, so another
sync cannot swap it under that connection.
- symlinks are resolved by reading the link (db::resolve_db_path), not
canonicalize, so a symlinked path finds the same lock and install
target even while the real file is missing (fresh install, mid-swap)
or the link dangles.
- readers go ahead unlocked when no lock file can be created; nothing
can replace the database there. (A read-only data directory still
fails in SQLite's WAL setup, as in 0.2.4.)
- XLSX shared strings are capped at 2,000,000 entries.
Verified live: an install waits for a reader's shared lock; a symlinked
db_path waits on the real path's lock and stays a symlink. CLI goldens
unchanged; live A/B vs 0.2.4: 16 passed, discover hits the IDX 403 on
both builds (exit 4 vs 1, intended).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Third Codex (gpt-6.1-sol, xhigh) pass over 1e55138, each item verified
before fixing:
- readers now hold the ownership lock shared for the life of their
connection, so a snapshot swap (exclusive) never happens under an
open SQLite connection whose WAL cleanup could hit a later import.
- the lock file name uses the resolved database path, so a symlinked
path shares the real database's lock; sync also installs at the
resolved path, keeping a symlinked path a symlink (it used to be
overwritten with a regular file).
- sync downloads without the lock and takes it only to recheck and
install; snapshot downloads have connect (15 s) and total (10 min)
timeouts. idx-mcp drops its in-process RwLock: idx-core's lock covers
it across processes.
- XLSX parsing caps rows at 1,000,000 and checks a shared string's size
before copying it.
- the temp snapshot's -wal/-shm files are removed instead of left in the
data directory.
Remaining, by design: `changes` loads all rows of two releases from the
local database before filtering; its size is bounded by the snapshot.
CLI goldens unchanged. Live A/B vs 0.2.4: 16 passed; discover hits the
IDX 403 on both builds (exit 4 vs 1, intended).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Second Codex (gpt-6.1-sol, xhigh) pass over 60634d3, each item verified
before fixing:
- new: the sync recheck hashed an existing empty database before schema
setup, so the first sync falsely reported an external change. Hash
after setup; a published snapshot already has the schema and keeps
its bytes.
- sync, import, and entity edits now hold a cross-process lock
(<db>.lock, std File::lock), so a sync cannot replace the database
under an import running in another process. Import validates and
downloads before taking the lock, keeping its error order. MSRV
1.85 -> 1.89 for File::lock.
- the row-count rebuild branch also never downgrades history.
- cache reads no longer delete corrupt entries (a concurrent writer may
have just replaced them); the next put overwrites atomically.
- idx-mcp validates the changes limit before querying and runs each
requested section or analysis once.
- XLSX parsing bounds total cell text, so one large shared string
referenced from many cells cannot exhaust memory.
- PDF/archive percentages must be 0-100%; HHI uses saturating math.
CLI goldens unchanged. Live A/B vs 0.2.4: 16 passed, discover hits the
IDX 403 on both builds (exit 4 vs 1, intended).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Findings from a Codex (gpt-6.1-sol, xhigh) review of v0.2.4..HEAD, each
verified in the code before fixing:
- high: ticker symbols reached cache file paths unchecked, so a symbol
like `../../REPORT` read (and deleted, if unparseable) .json files
outside the cache, reachable through MCP get_quotes. resolve_symbol now
accepts only ticker characters, and cache keys are encoded as single
safe path components.
- high: sync decided before downloading and replaced the database
without rechecking; an import that landed meanwhile was erased. It now
rechecks the local state before installing, and idx-mcp runs
ownership_sync exclusively with ownership reads.
- replaces_sqlite_sha256 never downgrades history: the client requires
the rebuild to keep at least the local release count, and the
publisher lists a replaced build only then.
- cache writes are atomic (temp file + rename), so concurrent readers
never see, and delete, a partial entry.
- idx-mcp caps symbols (20, deduplicated) and limits, and runs at most 4
tool calls at once.
- each Yahoo client gets its own cookie jar, paired with its crumb.
- ZIP entries (XLSX, KSEI archives) are read with a 256 MiB bound;
non-ASCII archive dates and huge percentages are parse errors instead
of panics or overflow.
- agent-smoke --keep deletes every copied agent config, not only auth
files.
CLI goldens unchanged; the MCP tool golden changes only to state the
new caps.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Require bash 4.4+ up front, and match answer facts with jq regexes
(Oniguruma, (?i) everywhere) instead of GNU-only grep -P.
- Unparseable agent output fails that run instead of aborting the
matrix; check_run now also reports unparseable tool calls, which
previously passed the tool check silently.
- Copy the Hermes model block with awk up to the next top-level key, so
a final or differently ordered block is handled.
- --keep keeps run output but deletes the copied agent credentials.
- docs: full path to the tool golden.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- tests/agents/agent-smoke.sh runs tests/agents/cases.json through
Claude Code, Codex, pi, Hermes Agent, and opencode headless against a
local idx-mcp build, each with an isolated config, and checks the
tools called, their key arguments, and (mock data) facts in the
answer. Opt-in; never part of the gate.
- crates/idx-mcp/tests/golden/tools.json pins tool names, descriptions,
and input schemas, so changes to what agents see are reviewed.
- nix develop .#agents adds Claude Code, Codex, and opencode from
nixpkgs after the host PATH.
First run: 30/30 (5 agents x 6 cases, mock data).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Sync hashed only the main database file, but local edits can sit in
the -wal sidecar, so an edited database could match an earlier build
and be replaced. Checkpoint the WAL before hashing; if changes remain,
never take the rebuilt-snapshot path.
- MCP tools rejected nothing for empty symbols/sections/kinds and
returned an empty success; they now fail with INVALIDINPUT, as the CLI
requires input.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Bump the workspace and npm packages to 0.3.0. The release workflow
builds and attaches idx-mcp binaries next to idx, the npm launcher and
installer pick their binary from package.json so npm/idx-mcp reuses
them, and the npm smoke test covers both packages.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A snapshot republished for the same as-of date (the TRUE ticker fix)
kept the date, latest release, and release count, so sync saw nothing
to do and existing installs kept the bad data without --force.
The manifest gains an optional snapshot.replaces_sqlite_sha256 listing
earlier builds for that date. Queries never write to a synced file, so
a local database byte-identical to a listed build is replaced; one that
matches nothing (local imports or edits) still needs --force. The
publisher fills the list on a same-date rebuild. Older clients ignore
the field.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Phase 4 of docs/DESIGN.md. Eleven grouped tools (quotes, history, screen,
company, analysis, compare, shareholders, investor holdings, search,
ownership changes, status); ownership_sync is listed only with
--allow-sync. Live results keep the Fetched envelope, errors use the CLI's
{error, code, message} body, and investor tools on an empty database
point at the sync path.
idx-core gains Period/Interval FromStr, Investor::status, a ticker
filter on changes, and Display for ErrorCode (now used by the CLI too).
CLI output and goldens are unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
comfy-table counted ANSI escapes as cell width, so colored signal and
change cells pushed the right border out of line on a terminal. Its
custom_styling feature measures the visible width instead. Piped output
has no color and is unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The IDX above-1% workbook stores the share code TRUE (TRINITI DINAMIK)
as a boolean cell holding 1, so its holders were imported under ticker
"1". Map t="b" cells to what Excel displays.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The IDX above-1% workbook stores the share code TRUE (TRINITI DINAMIK)
as a boolean cell holding 1, so its holders were imported under ticker
"1". Map t="b" cells to what Excel displays.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Phase 3c of docs/DESIGN.md. `ownership` handlers are now argument
parsing, one idx-core call, and rendering.
- investor.rs: Investor (also Idx::investor()) with releases, ticker,
entity, search, cross_holders, concentration, flow, changes, graph,
unresolved, map_alias, merge_entities, discover, sync, and import.
Import orchestration (input resolution, download, SHA-256 dedup,
PDF/XLSX/archive parsing, entity resolution, release write) moves
from the CLI unchanged.
- Typed inputs: ImportSource, HolderSource, ConcentrationRank, and
parse_report_family, with the same error messages.
- CoreConfig gains an ownership_db override; db::open_db_at opens an
explicit path. idx-cli no longer depends on rusqlite, sha2, or ureq
at runtime.
- Behavior note: an invalid `ticker --source` or `concentration --by`
is now rejected before the database file is opened (and created);
the error and exit code are unchanged.
- Fix: building idx-cli without the ownership feature no longer warns
about the ownership-only `blue` style.
All 143 goldens unchanged; 170 core + 7 CLI unit tests and 62 CLI tests
pass; clippy clean for default, release, testing, and no-default
feature sets. Live A/B against main: 16/16 stocks identical; ownership
discover got IDX 403 on both (exit 4 vs 1 is the phase 1 change).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Phase 3b of docs/DESIGN.md. `stocks` handlers are now argument parsing,
one idx-core call, and rendering.
- Idx (client.rs) is the entry point: Idx::new(CoreConfig) with
market(), financial(), and analysis(). Live lookups return
Fetched<T> { data, source, fetched_at, stale } and take a CachePolicy
(Default | Offline | Refresh, from the --offline/--no-cache flags).
- One cached_fetch replaces the CLI's three cache flows; they differed
only in whether a fresh entry is read via get or get_offline when
offline, which yields the same data and diagnostics. Report-building
errors (technical) still skip the stale-cache fallback.
- market.rs: quotes, history (history_source is resolved first so the
CLI's fallback note keeps its position), and a typed screener
(ScreenFilter/ScreenRegion with the same error messages).
- financial.rs: profile, statements, earnings, sentiment, insights,
news, with StatementFilter/EarningsFilter.
- analysis: TechnicalReport moves to analysis::report; Analysis offers
technical, growth, valuation, risk, fundamental, and compare.
- CoreConfig::load() owns provider/exchange/TTL settings from
config.toml and IDX_*; the CLI's IdxConfig adds output and color.
With two different invalid settings at once, the reported one may
differ from before; every single-error message is unchanged.
- Cache gains fresh/any/offline accessors returning fetch metadata.
Unit tests moved with their code (core 157 -> 168, CLI 17 -> 7). All
143 goldens unchanged; live A/B against main: 16/16 stocks identical.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Phase 3a of docs/DESIGN.md. MockProvider moves to api/mock.rs, and it,
the embedded MSN fixture bodies, and the IDX_MOCK_* hooks compile only
with idx-core's new `testing` feature. idx-cli enables it as a
dev-dependency, so `cargo test` keeps the fixture-backed provider while
release binaries no longer embed fixture data or honor
IDX_USE_MOCK_PROVIDER. Parser helpers used only by the mock are gated
the same way.
The packaged-cli flake check no longer relies on the mock; it checks
an offline cache miss (JSON CACHEMISS, exit 3) instead.
Goldens unchanged; clippy is clean with and without `testing`.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Phase 2 of docs/DESIGN.md, no behavior change:
- crates/idx-core (library): api, analysis, cache, config file handling
and provider selection, curl_impersonate, error, ownership, runtime,
plus the shared test fixtures. clap derives are behind an optional
`clap` feature so the library does not require clap.
- crates/idx-cli (binary `idx`): cli/, output/, and IdxConfig (output
format and color resolution). Package and binary names are unchanged.
- config.rs is split: file read/write and validation move to core; a
CLI test keeps core's OUTPUT_VALUES in sync with OutputSetting.
- ownership::db::open_db/db_path drop their unused IdxConfig argument.
- One workspace version; shared dependency versions; no third-party
version changes in Cargo.lock. LICENSE is symlinked into each crate.
- flake: workspace version, new source paths, and a per-crate package
check (idx-cli is built from its tarball against packaged idx-core,
since idx-core is not on crates.io yet). CI cache key, release path
filter, and the golden whitespace-hook exclusion follow the new layout.
Parity: all 143 golden outputs match byte for byte without
regeneration; 157 core + 17 CLI unit tests (173 before plus the new
OUTPUT_VALUES sync test), 62 CLI tests pass. Live A/B against main:
16/16 stocks cases identical; ownership discover got IDX 403 on both
(exit 4 vs 1 is the phase 1 exit-code change).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Phase 1 of docs/DESIGN.md:
- Output format resolves -o > IDX_OUTPUT > general.output > auto; auto
is a table on a terminal and JSON when stdout is piped or captured.
`config init` now writes output = "auto".
- JSON is pretty only on a terminal and compact otherwise. New `-o
jsonl` prints one array item per line with the same key order.
- No ANSI color when stdout is not a terminal (direct owo-colors calls
now go through output::style).
- `stocks quote -` and `stocks compare -` read symbols from stdin.
- Exit codes by class: 2 invalid input/config, 3 not found or cache
miss, 4 upstream unavailable, 5 unsupported, 1 other.
Golden changes are limited to those documented kinds and were verified
mechanically: JSON goldens are semantically identical, tables differ
only by removed ANSI codes and the column padding comfy-table had
reserved for them (cell contents identical), exit codes match each
error's class, and help/completions only gain the new --output values.
17 new golden cases cover auto, jsonl, stdin, and the new errors.
tests/cli.rs now pins IDX_OUTPUT=table in its harness (its tests assert
table text and run piped) and parses discover JSON instead of matching
pretty-printed whitespace.
Live A/B against main: 16/16 stocks cases identical; ownership discover
got IDX 403 on both binaries, now exiting 4 instead of 1 as intended.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Phase 0 of docs/DESIGN.md, before any refactor code:
- tests/golden.rs runs every case in tests/golden/cases.toml (126
outputs across help, completions, config, cache, all stocks commands
on MSN and Yahoo mocks, error paths, and ownership queries on a seeded
fixture DB) and compares exit code, stdout, and stderr byte for byte.
Regenerate only with UPDATE_GOLDEN=1 for a reviewed output change.
- scripts/live-smoke.sh --compare <old-idx> runs each live case with
both binaries and fails on differing exit codes, added/missing/retyped
JSON keys, or changed values for slow-moving data.
- Exclude golden files from the whitespace/EOF pre-commit fixers, which
would otherwise rewrite pinned help output.
Both gates were fault-injected: a one-word table change fails exactly
the three history goldens, and a binary dropping profile.long_name fails
only the profile live comparison.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
StatementSection.values was a HashMap, so `stocks financials -o json`
printed line items in a different order on every run. Table output
already sorted them alphabetically; JSON now matches via BTreeMap.
Values are unchanged, and older cache entries deserialize as before.
Found while capturing golden output, which needs deterministic bytes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Accepted design for extracting idx-core as the shared data layer, the
agent-native CLI output contract, idx-mcp tools, packaging, and the
parity gates (golden output tests plus live A/B check) that the
refactor must pass before merging to main.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The 43-case mock smoke duplicated tests/cli.rs, which already runs the
real binary against fixture providers. Replace the flake's smoke-mock
check with a small packaged-cli check (version JSON, bash completions,
one mock quote), and cut scripts/live-smoke.sh from 672 to ~170 lines:
it now only runs shipped commands against live MSN/Yahoo/IDX and
validates output shape. Rewrite docs/SMOKE.md around the Nix gate and
the drift check.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Remove FEATURE_SPEC.md, TODO.md, and docs/AGENT_INTEGRATION.md; their
core gaps have shipped. Open work moves to a short ROADMAP.md.
- Remove the CLAUDE.md symlink and rewrite AGENTS.md as a quick guide:
navigation, rules, and the nix flake check gate.
- Merge OWNERSHIP_SELF_HOSTED.md into OWNERSHIP_PUBLISH.md and drop
private host integration notes.
- ARCHITECTURE: add MSN endpoints, clarify Bing ingest is unsupported.
- OWNERSHIP_SYNC: note source.pdf_url also carries XLSX URLs.
- systemd example: --history 2 so snapshots stay under the 10 MiB
client cap.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Describe canonical local and CI verification, cache reuse, output assertions, workflow boundaries, and observed regression results without treating test counts as coverage targets.
Run deterministic installer and publisher checks in Nix, align dev and release dependency artifacts, narrow derivation inputs, and use content-aware CI cache keys. Replace duplicate Cargo gates with a Nix pre-push gate; validate JSON streams and cache data in representative packaged-binary smoke cases.
MSN orders screener lists by 1-year return, but top/worst-performers
re-sort by day change after truncating to --limit. So `--limit 3` sorted
an arbitrary 3 stocks (worst-performers returned -1.33%, +0.25%, +0.95%)
and gave different stocks than `--limit 25`. Fetch the whole list for
these filters, as high-volume and large-cap already do.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ld3fDSDiBzAaQvgZJX5NKM
change, change_pct and volume were `unwrap_or(0)` when the provider
omitted them (MSN quotes and screener rows; Yahoo with no previous
close), so "unknown" read as "unchanged, zero volume". They are now
Option fields that serialize as null when missing. The table shows "-",
and the performers screens sort unknown changes last in both
directions. Present values are unchanged and cached entries still load.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ld3fDSDiBzAaQvgZJX5NKM
Quotes had no timestamp, so a suspended stock's last price looked
current: Yahoo returns WIKA at 204 with regularMarketTime 2025-02-17.
Add `as_of` (RFC 3339 in WIB) from MSN `timeLastTraded` / Yahoo
`regularMarketTime`, also on screener rows. The table prints a note
under any quote whose last trade is over a week old. The field is
additive and defaults to null, so existing cache entries still load.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ld3fDSDiBzAaQvgZJX5NKM
MSN returns suspended names (21 of 896 on 2026-10-02, e.g. WIKA, SUPR)
with symbol, name and market cap but no price. These were reported as
SYMBOLNOTFOUND, which tells an agent the ticker does not exist. Add a
NoMarketData error for this case; unknown tickers stay SYMBOLNOTFOUND.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ld3fDSDiBzAaQvgZJX5NKM
MSN SentimentBrowser returns `[]` for every IDX ticker checked live
(BBCA, BBRI, TLKM), so `stocks sentiment` always failed with
PARSEERROR. Return empty statistics instead; the table prints a
one-line notice and JSON gets `"statistics": []` with exit 0.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ld3fDSDiBzAaQvgZJX5NKM
MSN stamps each IDX trading day at local midnight (17:00Z the previous
day), so every MSN candle was labelled one day early. Yahoo weekly and
monthly bars have the same stamp and were labelled Sunday / last day of
the previous month. Take the date in the exchange offset instead: Yahoo's
meta.gmtoffset when present, else WIB (UTC+7).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ld3fDSDiBzAaQvgZJX5NKM
Move fmt, clippy, tests, cargo package verification and the offline mock
smoke into flake checks built with crane, so the crate dependency build is
its own derivation that is reused until Cargo.lock changes. CI runs a single
nix flake check and restores /nix/store from the GitHub Actions cache via
nix-community/cache-nix-action, replacing the rustup + cargo test job.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013cfWL2GVg97oLfkNvPLaci
--no-default-features failed to compile because bing.rs imports
crate::ownership types.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Q2PufCUdvVZoVW1Dy53js
- --offline now warns when it serves a cache entry past its TTL; before,
arbitrarily old data was returned with no signal at all
- version, cache info and cache clear honour -o json
- ownership resolve map/merge honour -o json
- stocks quote/compare require at least one symbol instead of printing []
- cache schema-mismatch notice goes through runtime::info so --quiet
silences it, instead of a raw debug eprintln
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Q2PufCUdvVZoVW1Dy53js
Addresses review on #33: Yahoo auto-discovers only curl_chrome*; the
ownership fallback also tries Firefox and Safari profiles.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017a5Cv5Z1NzdYKgKnTiAF4E
- Install: add npm/npx (with the pnpm/bun opt-in), platform coverage,
pin example v0.2.4; Cargo builds from source.
- Runtime helpers as a table, corrected: curl-impersonate is needed for
IDX ownership discover/import and Yahoo fundamentals
(IDX_PROVIDER=yahoo), not for stocks history; mutool only for legacy
PDF reports. Everyday use needs neither.
- Quick start and Features now cover ownership (sync, ticker, entity,
search, changes, concentration, cross-holders, graph; XLSX source on
the IDX Data Kepemilikan Saham page) and the full stocks command set
and providers. Every example was run against v0.2.4.
- Documentation index; script test suites.
- docs: example versions -> v0.2.4; npm publish needs 2FA and a fresh
version number.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017a5Cv5Z1NzdYKgKnTiAF4E
