docs(research): record the defence failure mode and refresh the run log

Failure #35 is the largest defect found so far and it was invisible from the
code: 9 runs lost to one Act 1 boss, all with the same cause, all recorded in
the run files as damage taken, turns elapsed and potions spent. Recorded with
the reproduction, the fix, and the replay that verifies it.

Also adds a rule to the docs index: re-measure numbers before copying them.
Test counts, latencies and run totals in these notes have gone stale more than
once -- a hand-off summary recorded 29 and 118 assertions where the suites
actually printed 50 and 131.
This commit is contained in:
0xrsydn 2026-09-22 06:06:21 +07:00
commit 239a42c317
2 changed files with 151 additions and 0 deletions

View file

@ -204,3 +204,47 @@ went a full act further.
The Timeline epoch reveal blocked the menu again after run 004, this time with
`IRONCLAD2_EPOCH`. The mod refuses to automate it. A human must reveal it
before the next run can start. Expect this after most runs.
## Comparing two arms requires ONE revision
A run is only comparable with another run collected from the **same revision of
the decision layer**. `brain.py`, `facts.py`, `jev.py` and `run.py` decide how
*both* arms behave, so a change to any of them — the fight-length projection,
the lethal search, the state handed to Jev, the relic fix — invalidates every
run collected before it. The old runs stay as history; they are not a baseline.
`ab_card_skip.sh` now enforces this instead of relying on discipline:
- it hashes the decision layer (`brain.py facts.py jev.py run.py sts2.py`) into
a 12-character content hash, printed as `code revision: <hash>`;
- the hash is **recomputed before every session**, and the session line records
it: `--- jev 1/5 (attempt 1) rev=978485b6639a ---`. Taking it once at startup
would be worse than useless — `$RESULTS` is truncated when the run starts, so
every row would carry the startup hash by construction and the mixed-revision
warning could never fire;
- a session whose hash differs from the startup hash **aborts the experiment**
(`!! ABORT: the decision layer changed mid-experiment`), skips the remaining
arm, and still prints the report for whatever was collected;
- each attributed run is written as `policy \t run \t hash` in
`/tmp/ab_results.tsv`, stamped with the revision that actually produced it;
- the report scores **one** revision — the one this invocation collected from —
and prints a `!! MIXED REVISIONS` block when the file holds more than one,
with a `!! THIN SAMPLE` warning under five runs per arm. It reads the same
history directory the shell snapshotted, instead of a second hardcoded copy of
that path (which silently reported "no runs recorded" for runs that existed).
A content hash, not a commit id: the decision layer is normally edited in place
and uncommitted, so a commit id would not tell two revisions apart.
Verified in a sandbox with a stub `run.py` that edits `brain.py` between
sessions: the abort fires, the remaining arm is skipped, and the collected rows
keep the revision that produced them. A clean two-arm run stamps all four rows
with one hash.
**Rule:** before interpreting any A/B number, re-run BOTH arms from the same
revision. Runs already in `/tmp/ab_results.tsv` from before 2026-09-22 03:2x
are pre-change and must be re-collected.
Note also that `--steps` must stay generous. A session that hits the step cap
while the run is still going produces **no run record at all**, so a small step
budget silently fills the sample with runs that died early.