docs(research): record the defence failure mode and refresh the run log

Failure #35 is the largest defect found so far and it was invisible from the
code: 9 runs lost to one Act 1 boss, all with the same cause, all recorded in
the run files as damage taken, turns elapsed and potions spent. Recorded with
the reproduction, the fix, and the replay that verifies it.

Also adds a rule to the docs index: re-measure numbers before copying them.
Test counts, latencies and run totals in these notes have gone stale more than
once -- a hand-off summary recorded 29 and 118 assertions where the suites
actually printed 50 and 131.
This commit is contained in:
0xrsydn 2026-09-22 06:06:21 +07:00
commit 239a42c317
2 changed files with 151 additions and 0 deletions

View file

@ -540,3 +540,110 @@ instead of waiting. Cost is one wasted action. No revalidation pass is needed.
- `hand_select` confirms for `upgrade_select`
- character select alternates select/embark and resets correctly
- a `StubClient` makes the model paths testable offline and deterministically
---
## 35. Combat had no defense policy at all (the biggest one)
Found by mining the 37 run files, not by reading the code.
**Symptom.** 81% of runs (30/37) died in Act 1. `THE_KIN_BOSS` alone killed 9.
Every one of those fights ran **5-10 turns** and cost **44-80 HP**, i.e. ~10-13 a
turn, with block cards in hand the entire time.
```
damage taken: [44, 50, 50, 53, 63, 64, 70, 75, 80]
turns: [6, 9, 10, 6, 6, 10, 9, 7, 5]
```
**Cause A.** `facts.py` classes a hit of `<= 15% of max HP` as `THREAT_CHIP`.
At 80 max HP that is **12**. `_fallback_combat` then required
```python
must_respect = threat in (HEAVY, SEVERE, LETHAL) or hp_bucket in (WOUNDED, CRITICAL)
```
so at 74/80 HP (HEALTHY) a 12-damage hit was **ignored** and the bot attacked.
The boss's main attack sits exactly on that boundary. Measured:
```
hp incoming threat hp_bucket old play
74 12 chip healthy Bash (dmg)
60 12 chip healthy Bash (dmg)
45 12 chip wounded Defend <- only now, 29 HP already gone
```
**Cause B.** `_jev_combat` asked a `should_defend` Noul on **every combat turn
and never read it**. `grep -rn should_defend *.py` returned exactly one line —
the one that created it. Pure latency cost.
**Cause C.** With that question dead, defense fell entirely to
`choice("Which single play best advances winning this fight?")` — a phrasing
biased to damage, in the `Choice` shape already measured as diluting with
option count. Measured against the real Kin state, Jev answered **Bash at 0.42
confidence**. 0.42 is below the 0.45 gate, so it fell through to the fallback,
which also chose damage. **Both paths agreed on the wrong answer.**
**Fix.** Blocking is arithmetic, so it is decided in code before Jev is asked:
- `facts.CombatFacts.turns_to_kill` — remaining fight length from this turn's
reachable damage.
- `facts.CombatFacts.projected_incoming``turns_to_kill * incoming_damage`.
- `facts.CombatFacts.affordable_loss``hp - 30% of max_hp`.
- `facts.CombatFacts.must_block``projected_incoming > affordable_loss`.
- `combat_decision` now forces a block when `block_urgent` and a blocker is in
hand, **after** the lethal check, **before** Jev.
- `should_defend` deleted.
**Verification.** Replayed all 600 real combat captures. The rule changes
**8 of 68** in-play turns (11.8%), and stays silent on short fights and when
nothing is incoming:
```
live_162_combat.json 80/80 12 in 6 turns proj 72 > afford 56 -> Defend
live_238_combat.json 78/80 17 in 20 turns proj 340 > afford 54 -> Rage
trivial (15 HP enemy) -> still attacks
```
Tests: 50 in `test_facts.py`, 131 in `test_brain.py` (the two lethal-search
regressions in [09](09-typesafe-best-practice.md) §5 are included).
**Lesson.** The bug was invisible from the code and obvious from the data. Nine
runs died the same way, and the run files recorded the damage, the turn count
and the potions spent. Mine the history before theorising about the meta.
## 36. `relic_select` asked one question id and read another
`relic_select_decision` builds its ranking questions keyed `good_relicN`:
```python
keys = [f"relic{r.get('index', 0)}" for r in relics]
questions = {f"good_{k}": noul(...) for k in keys}
```
but then read the answers under the unprefixed ids:
```python
best_key, best_noul = best_by_noul(response, keys, CARD_PICK_THRESHOLD)
```
`best_by_noul` looks each key up with `response.get(key)`, so every lookup
returned `None`, `ranked` was empty, and the function returned `(None, 0.0)`
**for every state**. The relic path could therefore only ever take the rarest
relic — Jev's answer was silently discarded on every boss and elite relic
offer. The same shape is correct in `card_reward_decision` and
`card_select_decision`, which keep the `good_` prefix end to end; relic_select
was the one that did not.
Nothing in the trace caught it, because no session in `decisions.jsonl` ever
reached a `relic_select` state. Found by auditing every `best_by_noul` call
site against the ids it asked for, not by a run.
Fix: rank `[f"good_{k}" for k in keys]` and map the winner back with the same
prefix. Regression test in `test_brain.py`: the highest-rated relic is
deliberately the **Common** one, so the rarity fallback cannot produce the
expected answer by accident — the test fails before the fix and passes after.
**Lesson.** A question id that does not match its answer id fails silently and
looks exactly like "the model had nothing to say". Every `best_by_noul` call
site must be checked against the ids it actually asked for.