diff --git a/docs/research/05-failure-modes.md b/docs/research/05-failure-modes.md index c5f2a8d..e99daa3 100644 --- a/docs/research/05-failure-modes.md +++ b/docs/research/05-failure-modes.md @@ -540,3 +540,110 @@ instead of waiting. Cost is one wasted action. No revalidation pass is needed. - `hand_select` confirms for `upgrade_select` - character select alternates select/embark and resets correctly - a `StubClient` makes the model paths testable offline and deterministically + +--- + +## 35. Combat had no defense policy at all (the biggest one) + +Found by mining the 37 run files, not by reading the code. + +**Symptom.** 81% of runs (30/37) died in Act 1. `THE_KIN_BOSS` alone killed 9. +Every one of those fights ran **5-10 turns** and cost **44-80 HP**, i.e. ~10-13 a +turn, with block cards in hand the entire time. + +``` +damage taken: [44, 50, 50, 53, 63, 64, 70, 75, 80] +turns: [6, 9, 10, 6, 6, 10, 9, 7, 5] +``` + +**Cause A.** `facts.py` classes a hit of `<= 15% of max HP` as `THREAT_CHIP`. +At 80 max HP that is **12**. `_fallback_combat` then required + +```python +must_respect = threat in (HEAVY, SEVERE, LETHAL) or hp_bucket in (WOUNDED, CRITICAL) +``` + +so at 74/80 HP (HEALTHY) a 12-damage hit was **ignored** and the bot attacked. +The boss's main attack sits exactly on that boundary. Measured: + +``` + hp incoming threat hp_bucket old play + 74 12 chip healthy Bash (dmg) + 60 12 chip healthy Bash (dmg) + 45 12 chip wounded Defend <- only now, 29 HP already gone +``` + +**Cause B.** `_jev_combat` asked a `should_defend` Noul on **every combat turn +and never read it**. `grep -rn should_defend *.py` returned exactly one line — +the one that created it. Pure latency cost. + +**Cause C.** With that question dead, defense fell entirely to +`choice("Which single play best advances winning this fight?")` — a phrasing +biased to damage, in the `Choice` shape already measured as diluting with +option count. Measured against the real Kin state, Jev answered **Bash at 0.42 +confidence**. 0.42 is below the 0.45 gate, so it fell through to the fallback, +which also chose damage. **Both paths agreed on the wrong answer.** + +**Fix.** Blocking is arithmetic, so it is decided in code before Jev is asked: + +- `facts.CombatFacts.turns_to_kill` — remaining fight length from this turn's + reachable damage. +- `facts.CombatFacts.projected_incoming` — `turns_to_kill * incoming_damage`. +- `facts.CombatFacts.affordable_loss` — `hp - 30% of max_hp`. +- `facts.CombatFacts.must_block` — `projected_incoming > affordable_loss`. +- `combat_decision` now forces a block when `block_urgent` and a blocker is in + hand, **after** the lethal check, **before** Jev. +- `should_defend` deleted. + +**Verification.** Replayed all 600 real combat captures. The rule changes +**8 of 68** in-play turns (11.8%), and stays silent on short fights and when +nothing is incoming: + +``` +live_162_combat.json 80/80 12 in 6 turns proj 72 > afford 56 -> Defend +live_238_combat.json 78/80 17 in 20 turns proj 340 > afford 54 -> Rage +trivial (15 HP enemy) -> still attacks +``` + +Tests: 50 in `test_facts.py`, 131 in `test_brain.py` (the two lethal-search +regressions in [09](09-typesafe-best-practice.md) §5 are included). + +**Lesson.** The bug was invisible from the code and obvious from the data. Nine +runs died the same way, and the run files recorded the damage, the turn count +and the potions spent. Mine the history before theorising about the meta. + +## 36. `relic_select` asked one question id and read another + +`relic_select_decision` builds its ranking questions keyed `good_relicN`: + +```python +keys = [f"relic{r.get('index', 0)}" for r in relics] +questions = {f"good_{k}": noul(...) for k in keys} +``` + +but then read the answers under the unprefixed ids: + +```python +best_key, best_noul = best_by_noul(response, keys, CARD_PICK_THRESHOLD) +``` + +`best_by_noul` looks each key up with `response.get(key)`, so every lookup +returned `None`, `ranked` was empty, and the function returned `(None, 0.0)` +**for every state**. The relic path could therefore only ever take the rarest +relic — Jev's answer was silently discarded on every boss and elite relic +offer. The same shape is correct in `card_reward_decision` and +`card_select_decision`, which keep the `good_` prefix end to end; relic_select +was the one that did not. + +Nothing in the trace caught it, because no session in `decisions.jsonl` ever +reached a `relic_select` state. Found by auditing every `best_by_noul` call +site against the ids it asked for, not by a run. + +Fix: rank `[f"good_{k}" for k in keys]` and map the winner back with the same +prefix. Regression test in `test_brain.py`: the highest-rated relic is +deliberately the **Common** one, so the rarity fallback cannot produce the +expected answer by accident — the test fails before the fix and passes after. + +**Lesson.** A question id that does not match its answer id fails silently and +looks exactly like "the model had nothing to say". Every `best_by_noul` call +site must be checked against the ids it actually asked for. diff --git a/docs/research/07-run-log.md b/docs/research/07-run-log.md index 9433b21..3e69a03 100644 --- a/docs/research/07-run-log.md +++ b/docs/research/07-run-log.md @@ -204,3 +204,47 @@ went a full act further. The Timeline epoch reveal blocked the menu again after run 004, this time with `IRONCLAD2_EPOCH`. The mod refuses to automate it. A human must reveal it before the next run can start. Expect this after most runs. + +## Comparing two arms requires ONE revision + +A run is only comparable with another run collected from the **same revision of +the decision layer**. `brain.py`, `facts.py`, `jev.py` and `run.py` decide how +*both* arms behave, so a change to any of them — the fight-length projection, +the lethal search, the state handed to Jev, the relic fix — invalidates every +run collected before it. The old runs stay as history; they are not a baseline. + +`ab_card_skip.sh` now enforces this instead of relying on discipline: + +- it hashes the decision layer (`brain.py facts.py jev.py run.py sts2.py`) into + a 12-character content hash, printed as `code revision: `; +- the hash is **recomputed before every session**, and the session line records + it: `--- jev 1/5 (attempt 1) rev=978485b6639a ---`. Taking it once at startup + would be worse than useless — `$RESULTS` is truncated when the run starts, so + every row would carry the startup hash by construction and the mixed-revision + warning could never fire; +- a session whose hash differs from the startup hash **aborts the experiment** + (`!! ABORT: the decision layer changed mid-experiment`), skips the remaining + arm, and still prints the report for whatever was collected; +- each attributed run is written as `policy \t run \t hash` in + `/tmp/ab_results.tsv`, stamped with the revision that actually produced it; +- the report scores **one** revision — the one this invocation collected from — + and prints a `!! MIXED REVISIONS` block when the file holds more than one, + with a `!! THIN SAMPLE` warning under five runs per arm. It reads the same + history directory the shell snapshotted, instead of a second hardcoded copy of + that path (which silently reported "no runs recorded" for runs that existed). + +A content hash, not a commit id: the decision layer is normally edited in place +and uncommitted, so a commit id would not tell two revisions apart. + +Verified in a sandbox with a stub `run.py` that edits `brain.py` between +sessions: the abort fires, the remaining arm is skipped, and the collected rows +keep the revision that produced them. A clean two-arm run stamps all four rows +with one hash. + +**Rule:** before interpreting any A/B number, re-run BOTH arms from the same +revision. Runs already in `/tmp/ab_results.tsv` from before 2026-09-22 03:2x +are pre-change and must be re-collected. + +Note also that `--steps` must stay generous. A session that hits the step cap +while the run is still going produces **no run record at all**, so a small step +budget silently fills the sample with runs that died early.