fix: handle non-finite msn key ratios

This commit is contained in:
Rasyidan Akbar F. 2026-04-13 23:34:36 +07:00
commit 19748d397e
9 changed files with 484 additions and 15 deletions

View file

@ -205,6 +205,9 @@
- [x] Product decision on `2026-04-02`: keep `screen` under `stocks` for now; revisit a dedicated surface only when `screen query` / `screen presets` graduate from backlog into a richer workflow
- [x] Fixture-backed parser and CLI JSON regression coverage now covers the remaining MSN-only `sentiment`, `news`, and `screen` commands
- [x] Fresh post-coverage live MSN smoke rerun still passes for table and JSON surfaces: 30/30 (`tmp/live-smoke/20260327-163403`)
- [x] MSN key-ratios parsing now tolerates stringified non-finite numeric sentinels (`Infinity`, `-Infinity`, `NaN`) as missing data instead of failing the entire fundamentals payload
- [x] Live direct CLI repros on `2026-04-13` confirmed the original failure class on `BUMI`, `ADRO`, and `AIMS`; an opt-in `scripts/live-smoke.sh --group live-nonfinite` group now covers those real ticker paths
- [x] Added `scripts/audit-msn-fundamentals.sh` for a full CLI valuation sweep across the IDX MSN symbol map; keep it as a provider-health audit, not part of the default smoke baseline
- [ ] `ownership import --fetch-bing` is still deferred and returns unsupported
- [x] Real KSEI PDF import from local file now works again: `ownership import --file /Users/rasyidanakbar/Downloads/ksei_raw_data.pdf` imported `7261` rows for `955` tickers on `2026-03-28`, replacing the previous `6`-row/`1`-ticker failure mode
- [x] KSEI parser no longer depends on the old hardcoded column bounds fixture layout; it now reconstructs rows from `mutool` line output and handles the live `DATE + SHARE_CODE` merged segment plus `D`/`A` locality markers

View file

@ -15,9 +15,11 @@ scripts/live-smoke.sh
scripts/live-smoke.sh --mode full
scripts/live-smoke.sh --mode mock
scripts/live-smoke.sh --group live-table --group live-json
scripts/live-smoke.sh --group live-nonfinite
scripts/live-smoke.sh --group cache --symbol BBRI
scripts/live-smoke.sh --dry-run --mode full
scripts/live-smoke.sh --bin ./tmp/release-install/bin/idx --no-build --mode mock
scripts/audit-msn-fundamentals.sh --tickers BUMI,ADRO,AIMS
```
## Modes
@ -35,6 +37,7 @@ scripts/live-smoke.sh --bin ./tmp/release-install/bin/idx --no-build --mode mock
- `cache`: cache warm, `--offline`, and stale-cache fallback checks for quote, technical, and MSN `profile`
- `routing`: Yahoo/MSN provider routing plus explicit MSN history unsupported behavior
- `errors`: JSON error contract and invalid flag/input checks
- `live-nonfinite`: opt-in live MSN fundamentals checks for known non-finite ticker payloads (`BUMI`, `ADRO`, `AIMS`)
- `ownership`: safe ownership smoke checks that do not require imported ownership data
- `ownership-import`: live discovery/import hardening checks for supported `above1` import plus expected unsupported legacy-family failures
@ -43,7 +46,9 @@ scripts/live-smoke.sh --bin ./tmp/release-install/bin/idx --no-build --mode mock
- The runner forces `IDX_OUTPUT=table` as its default environment so table cases stay stable; JSON checks use `-o json` explicitly.
- Cache-group warm cases clear the smoke cache before they run so each warm/offline/stale sequence starts clean and stale-cache assertions are not masked by earlier groups.
- Use `--bin <path> --no-build` when you want to validate an installed binary instead of the workspace `target/debug/idx` build.
- Use `scripts/audit-msn-fundamentals.sh` for a full CLI valuation sweep across the IDX MSN symbol map. It is intentionally separate from the reusable smoke runner because it is a heavier provider-health audit, not a stable baseline check.
- Ownership commands that need imported data are intentionally not part of the baseline runner yet. The current baseline only covers `ownership releases` and the known unsupported `ownership import --fetch-bing`.
- `ownership sync` is still primarily covered by fixture-backed CLI tests rather than the reusable smoke runner.
- The new `ownership-import` group is intentionally opt-in for explicit `--group ownership-import` runs or `--mode full`; it discovers live URLs first, imports the supported `above1` attachment into the temp DB, then asserts the current `above5` and `investor-type` URLs fail with explicit unsupported-schema UX.
- The new `live-nonfinite` group is intentionally opt-in only. As of `2026-04-13`, the known real repro tickers are `BUMI`, `ADRO`, and `AIMS`.
- When a case fails, inspect the per-case log in `tmp/live-smoke/.../logs/` before updating `TODO.md` or `FEATURE_SPEC.md`.

186
scripts/audit-msn-fundamentals.sh Executable file
View file

@ -0,0 +1,186 @@
#!/usr/bin/env bash
set -u
set -o pipefail
ROOT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
BIN_PATH=""
WORKDIR=""
TICKERS=""
BUILD=1
CONFIG_HOME=""
CACHE_HOME=""
DATA_HOME=""
LOG_DIR=""
SUMMARY_FILE=""
declare -a BASE_ENV=()
declare -a TICKER_LIST=()
usage() {
cat <<'EOF'
Usage: scripts/audit-msn-fundamentals.sh [options]
Run `idx -o json stocks valuation <ticker>` across the IDX MSN symbol map.
This is a heavier provider audit, not a default smoke check.
Options:
--bin <path> Use an existing idx binary instead of target/debug/idx
--no-build Skip cargo build
--workdir <path> Artifact root. Default: tmp/msn-fundamentals-audit/<timestamp>
--tickers <list> Comma-separated ticker override instead of the full symbol map
--help Show this help
Examples:
scripts/audit-msn-fundamentals.sh
scripts/audit-msn-fundamentals.sh --tickers BUMI,ADRO,AIMS
scripts/audit-msn-fundamentals.sh --bin ./target/debug/idx --no-build
EOF
}
parse_args() {
while [[ $# -gt 0 ]]; do
case "$1" in
--bin)
BIN_PATH="${2:-}"
BUILD=0
shift 2
;;
--no-build)
BUILD=0
shift
;;
--workdir)
WORKDIR="${2:-}"
shift 2
;;
--tickers)
TICKERS="${2:-}"
shift 2
;;
--help|-h)
usage
exit 0
;;
*)
echo "unknown argument: $1" >&2
usage >&2
exit 2
;;
esac
done
}
prepare_paths() {
if [[ -z "$WORKDIR" ]]; then
WORKDIR="$ROOT_DIR/tmp/msn-fundamentals-audit/$(date +%Y%m%d-%H%M%S)"
fi
CONFIG_HOME="$WORKDIR/config"
CACHE_HOME="$WORKDIR/cache"
DATA_HOME="$WORKDIR/data"
LOG_DIR="$WORKDIR/logs"
SUMMARY_FILE="$WORKDIR/summary.tsv"
mkdir -p "$CONFIG_HOME" "$CACHE_HOME" "$DATA_HOME" "$LOG_DIR"
BASE_ENV=(
"XDG_CONFIG_HOME=$CONFIG_HOME"
"XDG_CACHE_HOME=$CACHE_HOME"
"XDG_DATA_HOME=$DATA_HOME"
"IDX_PROVIDER=msn"
"IDX_HISTORY_PROVIDER=auto"
)
}
build_binary() {
if [[ -z "$BIN_PATH" ]]; then
BIN_PATH="$ROOT_DIR/target/debug/idx"
fi
if (( BUILD )); then
(
cd "$ROOT_DIR" || exit 1
cargo build --quiet --bin idx
) || {
echo "build failed" >&2
exit 1
}
fi
if [[ ! -x "$BIN_PATH" ]]; then
echo "idx binary not found or not executable: $BIN_PATH" >&2
exit 1
fi
}
resolve_tickers() {
local raw_ticker
local old_ifs
if [[ -n "$TICKERS" ]]; then
old_ifs="$IFS"
IFS=','
for raw_ticker in $TICKERS; do
raw_ticker="${raw_ticker//[[:space:]]/}"
if [[ -n "$raw_ticker" ]]; then
TICKER_LIST+=("$raw_ticker")
fi
done
IFS="$old_ifs"
else
mapfile -t TICKER_LIST < <(cut -f1 "$ROOT_DIR/src/api/msn/symbol_ids.tsv")
fi
if [[ ${#TICKER_LIST[@]} -eq 0 ]]; then
echo "no tickers selected" >&2
exit 1
fi
}
run_audit() {
local total="${#TICKER_LIST[@]}"
local index=0
local passed=0
local failed=0
local ticker
local log_file
local message
printf 'ticker\tstatus\tmessage\tlog\n' >"$SUMMARY_FILE"
for ticker in "${TICKER_LIST[@]}"; do
index=$((index + 1))
log_file="$LOG_DIR/${ticker}.log"
printf '[%04d/%04d] %s\n' "$index" "$total" "$ticker"
if env "${BASE_ENV[@]}" "$BIN_PATH" -q -o json stocks valuation "$ticker" >"$log_file" 2>&1 \
&& grep -q '"overall_signal"' "$log_file"; then
passed=$((passed + 1))
printf '%s\tok\t-\t%s\n' "$ticker" "$log_file" >>"$SUMMARY_FILE"
else
failed=$((failed + 1))
message="$(tr '\n' ' ' <"$log_file" | tr '\t' ' ' | cut -c1-240)"
printf '%s\tfailed\t%s\t%s\n' "$ticker" "$message" "$log_file" >>"$SUMMARY_FILE"
fi
done
printf '\nresults: passed=%d failed=%d total=%d\n' "$passed" "$failed" "$total"
printf 'summary: %s\n' "$SUMMARY_FILE"
if (( failed > 0 )); then
exit 1
fi
}
main() {
parse_args "$@"
prepare_paths
build_binary
resolve_tickers
run_audit
}
main "$@"

View file

@ -61,6 +61,7 @@ Groups:
cache deterministic cache/offline/stale-cache checks
routing provider routing and explicit unsupported checks
errors JSON error contract and invalid-flag checks
live-nonfinite opt-in live MSN fundamentals checks for known non-finite tickers
ownership ownership commands that are safe without imported data
ownership-import live ownership discovery/import hardening checks
@ -69,6 +70,7 @@ Examples:
scripts/live-smoke.sh --mode full
scripts/live-smoke.sh --mode mock --group cache
scripts/live-smoke.sh --group live-table --group live-json --symbol BBRI
scripts/live-smoke.sh --group live-nonfinite
EOF
}
@ -198,6 +200,11 @@ register_cases() {
add_case "live-json" "news" "0" "$live_env" "-o json $news_args" ""
add_case "live-json" "screen" "0" "$live_env" "-o json $screen_args" ""
add_case "live-nonfinite" "valuation-bumi" "0" "$live_env" "-o json stocks valuation BUMI" "\"overall_signal\""
add_case "live-nonfinite" "valuation-adro" "0" "$live_env" "-o json stocks valuation ADRO" "\"overall_signal\""
add_case "live-nonfinite" "valuation-aims" "0" "$live_env" "-o json stocks valuation AIMS" "\"overall_signal\""
add_case "live-nonfinite" "compare-bad-tickers" "0" "$live_env" "-o json stocks compare BUMI,ADRO,AIMS" "\"symbol\": \"BUMI.JK\""
add_case "mock" "quote-table" "0" "$mock_env" "stocks quote $live_symbol" ""
add_case "mock" "quote-json" "0" "$mock_env" "-o json stocks quote $live_symbol" ""
add_case "mock" "history-table" "0" "$mock_env" "$history_args" ""

View file

@ -31,6 +31,10 @@ pub(crate) fn parse_fundamentals_from_str(
mod tests {
use super::{parse_fundamentals_from_str, parse_quote_from_str};
fn minimal_quote_raw() -> &'static str {
r#"[{"symbol":"BBCA","marketCap":1215200000000000}]"#
}
#[test]
fn parses_quote_fixture_json() {
let raw = std::fs::read_to_string("tests/fixtures/msn_quote_bbca.json")
@ -59,4 +63,84 @@ mod tests {
assert_eq!(fundamentals.earnings_growth, Some(0.121));
assert_eq!(fundamentals.market_cap, Some(1_215_200_000_000_000));
}
#[test]
fn parses_fundamentals_with_infinity_string_as_missing_data() {
let raw = r#"[
{
"companyMetrics": [
{
"year": "2025",
"fiscalPeriodType": "TTM",
"priceToEarningsRatio": "Infinity",
"priceToBookRatio": 4.6,
"roe": 19.8,
"profitMargin": 44.2,
"debtToEquityRatio": 0.75,
"currentRatio": 1.4
}
]
}
]"#;
let fundamentals = parse_fundamentals_from_str(raw, Some(minimal_quote_raw()))
.expect("fundamentals parsed");
assert_eq!(fundamentals.trailing_pe, None);
assert_eq!(fundamentals.price_to_book, Some(4.6));
assert_eq!(fundamentals.return_on_equity, Some(0.198));
}
#[test]
fn parses_fundamentals_with_negative_infinity_string_as_missing_data() {
let raw = r#"[
{
"companyMetrics": [
{
"year": "2025",
"fiscalPeriodType": "TTM",
"priceToEarningsRatio": 12.5,
"debtToEquityRatio": "-Infinity",
"roe": 19.8,
"profitMargin": 44.2,
"currentRatio": 1.4
}
]
}
]"#;
let fundamentals = parse_fundamentals_from_str(raw, Some(minimal_quote_raw()))
.expect("fundamentals parsed");
assert_eq!(fundamentals.trailing_pe, Some(12.5));
assert_eq!(fundamentals.debt_to_equity, None);
assert_eq!(fundamentals.return_on_equity, Some(0.198));
}
#[test]
fn parses_fundamentals_with_nan_string_as_missing_data() {
let raw = r#"[
{
"companyMetrics": [
{
"year": "2025",
"fiscalPeriodType": "TTM",
"revenueGrowthRate": "NaN",
"earningsGrowthRate": 12.1,
"roe": 19.8,
"profitMargin": 44.2,
"debtToEquityRatio": 0.75,
"currentRatio": 1.4
}
]
}
]"#;
let fundamentals = parse_fundamentals_from_str(raw, Some(minimal_quote_raw()))
.expect("fundamentals parsed");
assert_eq!(fundamentals.revenue_growth, None);
assert_eq!(fundamentals.earnings_growth, Some(0.121));
assert_eq!(fundamentals.return_on_equity, Some(0.198));
}
}

View file

@ -47,33 +47,45 @@ pub(crate) struct KeyRatios {
pub(crate) struct IndustryMetric {
pub(crate) year: Option<String>,
pub(crate) fiscal_period_type: Option<String>,
#[serde(default)]
#[serde(default, deserialize_with = "de_opt_f64_lenient")]
pub(crate) revenue_growth_rate: Option<f64>,
#[serde(default)]
#[serde(default, deserialize_with = "de_opt_f64_lenient")]
pub(crate) earnings_growth_rate: Option<f64>,
#[serde(default, rename = "netIncomeYTDYTDGrowthRate")]
#[serde(
default,
rename = "netIncomeYTDYTDGrowthRate",
deserialize_with = "de_opt_f64_lenient"
)]
pub(crate) net_income_ytd_ytd_growth_rate: Option<f64>,
#[serde(default, rename = "revenueYTDYTD")]
#[serde(
default,
rename = "revenueYTDYTD",
deserialize_with = "de_opt_f64_lenient"
)]
pub(crate) revenue_ytd_ytd: Option<f64>,
#[serde(default)]
#[serde(default, deserialize_with = "de_opt_f64_lenient")]
pub(crate) net_margin: Option<f64>,
#[serde(default)]
#[serde(default, deserialize_with = "de_opt_f64_lenient")]
pub(crate) profit_margin: Option<f64>,
#[serde(default)]
#[serde(default, deserialize_with = "de_opt_f64_lenient")]
pub(crate) roe: Option<f64>,
#[serde(default, rename = "roaTTM")]
#[serde(default, rename = "roaTTM", deserialize_with = "de_opt_f64_lenient")]
pub(crate) roa_ttm: Option<f64>,
#[serde(default)]
#[serde(default, deserialize_with = "de_opt_f64_lenient")]
pub(crate) return_on_asset_current: Option<f64>,
#[serde(default)]
#[serde(default, deserialize_with = "de_opt_f64_lenient")]
pub(crate) debt_to_equity_ratio: Option<f64>,
#[serde(default, deserialize_with = "de_opt_f64_lenient")]
pub(crate) current_ratio: Option<f64>,
#[serde(default)]
#[serde(default, deserialize_with = "de_opt_f64_lenient")]
pub(crate) price_to_earnings_ratio: Option<f64>,
#[serde(default, rename = "forwardPriceToEPS")]
#[serde(
default,
rename = "forwardPriceToEPS",
deserialize_with = "de_opt_f64_lenient"
)]
pub(crate) forward_price_to_eps: Option<f64>,
#[serde(default)]
#[serde(default, deserialize_with = "de_opt_f64_lenient")]
pub(crate) price_to_book_ratio: Option<f64>,
}
@ -324,10 +336,23 @@ where
Some(NumberLike::F64(_)) => Ok(None),
Some(NumberLike::String(raw)) => {
let trimmed = raw.trim();
if trimmed.is_empty() || trimmed.eq_ignore_ascii_case("nan") {
if trimmed.is_empty()
|| trimmed.eq_ignore_ascii_case("nan")
|| trimmed.eq_ignore_ascii_case("infinity")
|| trimmed.eq_ignore_ascii_case("-infinity")
{
Ok(None)
} else {
trimmed.parse::<f64>().map(Some).map_err(D::Error::custom)
trimmed
.parse::<f64>()
.map_err(D::Error::custom)
.map(|number| {
if number.is_finite() {
Some(number)
} else {
None
}
})
}
}
None => Ok(None),

View file

@ -128,6 +128,17 @@ fn fixture_path(name: &str) -> PathBuf {
.join(name)
}
fn run_success_stdout(cmd: &mut Command) -> String {
let output = cmd.output().expect("run command");
assert!(
output.status.success(),
"command failed\nstdout:\n{}\nstderr:\n{}",
String::from_utf8_lossy(&output.stdout),
String::from_utf8_lossy(&output.stderr),
);
String::from_utf8(output.stdout).expect("utf8 stdout")
}
fn spawn_single_response_server(content_type: &str, body: impl Into<Vec<u8>>) -> String {
let listener = TcpListener::bind("127.0.0.1:0").expect("bind local test server");
let addr = listener.local_addr().expect("local addr");
@ -900,6 +911,112 @@ fn compare_with_mock_provider_table_contains_resolved_symbol() {
.stdout(predicate::str::contains("BBCA.JK"));
}
#[test]
fn infinity_msn_mock_fundamentals_succeed_for_analysis_commands() {
let fixture = fixture_path("msn_keyratios_infinity.json");
let fixture_str = fixture
.to_str()
.expect("fixture path should be valid unicode")
.to_string();
let cases = [
(
"growth-infinity-json",
vec!["-o", "json", "stocks", "growth", "BBCA"],
vec!["\"revenue_growth\"", "\"overall_signal\""],
),
(
"valuation-infinity-json",
vec!["-o", "json", "stocks", "valuation", "BBCA"],
vec!["\"pe_trailing\": null", "\"overall_signal\""],
),
(
"risk-infinity-json",
vec!["-o", "json", "stocks", "risk", "BBCA"],
vec!["\"debt_to_equity\"", "\"overall_signal\""],
),
(
"fundamental-infinity-json",
vec!["-o", "json", "stocks", "fundamental", "BBCA"],
vec!["\"symbol\": \"BBCA.JK\"", "\"overall_signal\""],
),
(
"compare-infinity-json",
vec!["-o", "json", "stocks", "compare", "BBCA,BBRI"],
vec!["\"symbol\": \"BBCA.JK\"", "\"symbol\": \"BBRI.JK\""],
),
];
for (name, args, needles) in cases {
let stdout = run_success_stdout(
test_bin(name)
.env("IDX_PROVIDER", "msn")
.env("IDX_USE_MOCK_PROVIDER", "1")
.env("IDX_MOCK_MSN_KEYRATIOS_FIXTURE", &fixture_str)
.args(args),
);
for needle in needles {
assert!(
stdout.contains(needle),
"missing `{needle}` in output: {stdout}"
);
}
}
}
#[test]
fn negative_infinity_msn_mock_fundamentals_succeed_for_analysis_commands() {
let fixture = fixture_path("msn_keyratios_negative_infinity.json");
let fixture_str = fixture
.to_str()
.expect("fixture path should be valid unicode")
.to_string();
let cases = [
(
"growth-negative-infinity-json",
vec!["-o", "json", "stocks", "growth", "BBCA"],
vec!["\"revenue_growth\"", "\"overall_signal\""],
),
(
"valuation-negative-infinity-json",
vec!["-o", "json", "stocks", "valuation", "BBCA"],
vec!["\"pe_trailing\": null", "\"overall_signal\""],
),
(
"risk-negative-infinity-json",
vec!["-o", "json", "stocks", "risk", "BBCA"],
vec!["\"debt_to_equity\": null", "\"overall_signal\""],
),
(
"fundamental-negative-infinity-json",
vec!["-o", "json", "stocks", "fundamental", "BBCA"],
vec!["\"symbol\": \"BBCA.JK\"", "\"overall_signal\""],
),
(
"compare-negative-infinity-json",
vec!["-o", "json", "stocks", "compare", "BBCA,BBRI"],
vec!["\"symbol\": \"BBCA.JK\"", "\"symbol\": \"BBRI.JK\""],
),
];
for (name, args, needles) in cases {
let stdout = run_success_stdout(
test_bin(name)
.env("IDX_PROVIDER", "msn")
.env("IDX_USE_MOCK_PROVIDER", "1")
.env("IDX_MOCK_MSN_KEYRATIOS_FIXTURE", &fixture_str)
.args(args),
);
for needle in needles {
assert!(
stdout.contains(needle),
"missing `{needle}` in output: {stdout}"
);
}
}
}
#[test]
fn config_path_prints_path() {
test_bin("config-path")
@ -1460,6 +1577,7 @@ fn ownership_import_force_reimports_existing_release() {
#[test]
fn ownership_import_url_rejects_legacy_above5_pdf_schema() {
let root = test_env_dir("ownership-import-above5-unsupported");
let data_home = root.join("data");
let fake_mutool_dir = install_fake_mutool(
&root,
include_str!("fixtures/ksei_above5_stext_excerpt.xml"),
@ -1469,6 +1587,7 @@ fn ownership_import_url_rejects_legacy_above5_pdf_schema() {
bin_with_root(&root)
.env("PATH", prepend_path(&fake_mutool_dir))
.env("XDG_DATA_HOME", &data_home)
.args(["ownership", "import", "--url", &pdf_url])
.assert()
.failure()
@ -1480,6 +1599,7 @@ fn ownership_import_url_rejects_legacy_above5_pdf_schema() {
#[test]
fn ownership_import_url_rejects_legacy_investor_type_pdf_schema() {
let root = test_env_dir("ownership-import-investor-type-unsupported");
let data_home = root.join("data");
let fake_mutool_dir = install_fake_mutool(
&root,
include_str!("fixtures/ksei_investor_type_stext_excerpt.xml"),
@ -1489,6 +1609,7 @@ fn ownership_import_url_rejects_legacy_investor_type_pdf_schema() {
bin_with_root(&root)
.env("PATH", prepend_path(&fake_mutool_dir))
.env("XDG_DATA_HOME", &data_home)
.args(["ownership", "import", "--url", &pdf_url])
.assert()
.failure()

View file

@ -0,0 +1,19 @@
[
{
"companyMetrics": [
{
"year": "2025",
"fiscalPeriodType": "TTM",
"revenueGrowthRate": 8.1,
"earningsGrowthRate": 12.1,
"priceToEarningsRatio": "Infinity",
"priceToBookRatio": 4.6,
"roe": 19.8,
"profitMargin": 44.2,
"debtToEquityRatio": 0.75,
"currentRatio": 1.4,
"roaTTM": 3.79
}
]
}
]

View file

@ -0,0 +1,19 @@
[
{
"companyMetrics": [
{
"year": "2025",
"fiscalPeriodType": "TTM",
"revenueGrowthRate": 8.1,
"earningsGrowthRate": 12.1,
"priceToEarningsRatio": "-Infinity",
"priceToBookRatio": 4.6,
"roe": 19.8,
"profitMargin": 44.2,
"debtToEquityRatio": "-Infinity",
"currentRatio": 1.4,
"roaTTM": 3.79
}
]
}
]