239a42c317
Failure #35 is the largest defect found so far and it was invisible from the code: 9 runs lost to one Act 1 boss, all with the same cause, all recorded in the run files as damage taken, turns elapsed and potions spent. Recorded with the reproduction, the fix, and the replay that verifies it. Also adds a rule to the docs index: re-measure numbers before copying them. Test counts, latencies and run totals in these notes have gone stale more than once -- a hand-off summary recorded 29 and 118 assertions where the suites actually printed 50 and 131.
250 lines
9.1 KiB
Markdown
250 lines
9.1 KiB
Markdown
# 07 — Run log
|
||
|
||
Results per run. Source of truth is the game's own history file:
|
||
|
||
```
|
||
~/Library/Application Support/SlayTheSpire2/steam/<steamid>/modded/profile1/saves/history/<ts>.run
|
||
```
|
||
|
||
That file records `win`, `killed_by_encounter`, `seed`, `run_time`, `build_id`,
|
||
`ascension`, `was_abandoned`, and the full deck. **It is the evaluation
|
||
harness.** Read it rather than instrumenting the bot.
|
||
|
||
---
|
||
|
||
## Run 001 — Ironclad, standard, ascension 0
|
||
|
||
| Field | Value |
|
||
|---|---|
|
||
| Outcome | **Loss** |
|
||
| Killed by | `ENCOUNTER.VANTOM_BOSS` (Act 1 boss) |
|
||
| Floor reached | 16 |
|
||
| Seed | `JUS59Z32HF` |
|
||
| Run time | 2017 s (33 min) |
|
||
| Game build | `v0.107.1` |
|
||
| Abandoned | no |
|
||
|
||
### Progress
|
||
|
||
- Cleared 15 floors.
|
||
- Beat an elite: **Byrdonis**, 81 HP, 17-damage attack, `Territorial` (+1
|
||
Strength each turn).
|
||
- Fought Nibbits, Twig Slimes, Leaf Slimes, Shrinker Beetle, Inklings, Mawler,
|
||
Ruby Raiders (3-enemy fight), Vantom.
|
||
- Healed at rest sites at 43% and 24% HP.
|
||
|
||
### Final deck (19 cards, 1 upgraded)
|
||
|
||
`5× Strike` (one at upgrade level 1), `4× Defend`, `Bash`, `Setup Strike`,
|
||
`Inflame`, `Iron Wave`, `Rage`, `Stomp`, `Bludgeon`, `Battle Trance`,
|
||
`Second Wind`, `Bloodletting`.
|
||
|
||
### Relics (3)
|
||
|
||
`Burning Blood` (floor 1), `Gorget` (floor 10), `Vajra` (floor 11).
|
||
|
||
### Potions held at death — all three
|
||
|
||
```
|
||
POTION.EXPLOSIVE_AMPOULE slot 0
|
||
POTION.STRENGTH_POTION slot 1
|
||
POTION.ENERGY_POTION slot 2
|
||
```
|
||
|
||
**This is the headline finding.** The combat brain had no potion logic, so the
|
||
bot died holding an Explosive Ampoule, a Strength Potion, and an Energy Potion.
|
||
The STS2MCP strategy notes state the principle directly: *"Don't hoard potions.
|
||
Dying with full potions is the worst outcome."*
|
||
|
||
The run was lost on tactics, not on deck quality. The deck was reasonable for
|
||
floor 16 and the relic set was functional.
|
||
|
||
### Decision sources
|
||
|
||
| Source | Count |
|
||
|---|---|
|
||
| `code` | 141 |
|
||
| `jev` | 70 |
|
||
| `fallback` | 35 |
|
||
| Lethal lines executed by code | 33 |
|
||
|
||
Aggregate across 759 loop steps in 5 sessions.
|
||
|
||
### Notable Jev calls
|
||
|
||
| Situation | Jev chose | Confidence |
|
||
|---|---|---|
|
||
| Combat, 5 cards | Bash over Defend | 0.85 |
|
||
| Card reward | Inflame | 0.28 |
|
||
| Card reward | Rage | 0.40 |
|
||
| Card reward | Setup Strike | 0.33 |
|
||
| Map, 2 options | Shop | 0.97 |
|
||
| Map, 2 options | RestSite | 0.88 |
|
||
| Boss fight | Battle Trance, Bloodletting, Second Wind | 0.62–1.00 |
|
||
|
||
Several card-reward margins were thin enough that the gate fell back to "take
|
||
the first". Card reward is the weakest Jev decision in the current design.
|
||
|
||
### Bugs this run exposed
|
||
|
||
Eight infinite loops or stalls, all now fixed. See
|
||
[05-failure-modes.md](05-failure-modes.md). The most expensive were the
|
||
right-to-left reward indexing and the silently-dropped potion reward.
|
||
|
||
---
|
||
|
||
## Baseline to beat
|
||
|
||
| Metric | Run 001 |
|
||
|---|---|
|
||
| Win | no |
|
||
| Floor | 16 |
|
||
| Act 1 boss reached | yes |
|
||
| Act 1 boss killed | no |
|
||
|
||
Run 002 should be measured against this. The first question is not "does it
|
||
win" but "does it get past Vantom, and does it die holding potions again".
|
||
|
||
## Characters unlocked
|
||
|
||
| Character | Status |
|
||
|---|---|
|
||
| The Ironclad | unlocked |
|
||
| The Silent | unlocked (after the `NEOW_EPOCH` reveal) |
|
||
| The Regent | locked |
|
||
| The Necrobinder | locked |
|
||
| The Defect | locked |
|
||
|
||
`progress.save` recorded `pending_character_unlock: "CHARACTER.SILENT"` before
|
||
the reveal, then Silent became selectable.
|
||
|
||
## Seeded runs
|
||
|
||
Standard singleplayer exposes **no seed**, and `run` state carries no seed
|
||
field. The seed is visible only in the history file, after the run.
|
||
|
||
Seeded modes exist but were not available until the first epoch unlock:
|
||
|
||
| Mode | Seeded |
|
||
|---|---|
|
||
| `standard` | no |
|
||
| `daily` | yes (fixed per day) |
|
||
| `custom` | yes |
|
||
|
||
For a reproducible evaluation harness, `custom` is the likely path. This needs
|
||
verification — the mode screen appeared once and was not captured.
|
||
|
||
---
|
||
|
||
# Session 2 — potions, shops, treasure, events
|
||
|
||
## Scoreboard
|
||
|
||
| # | Win | Killed by | Seed | Time | Potions at end | Act reached |
|
||
|---|---|---|---|---|---|---|
|
||
| 001 | no | `VANTOM_BOSS` | `JUS59Z32HF` | 2017 s | **3** | 1 boss |
|
||
| 002 | no | `BYRDONIS_ELITE` | `MJ4J25A3BH` | 688 s | 0 | 1 |
|
||
| 003 | no | `SNAPPING_JAXFRUIT_NORMAL` | `HKLFARJJJQ` | 186 s | 0 | 1 |
|
||
| 004 | no | **`THE_INSATIABLE_BOSS`** | `MLQ4KBZQWV` | 2235 s | 2 | **2 boss** |
|
||
|
||
Run 004 reached the Act 2 boss with 11 relics and a 29-card deck. That is the
|
||
best result so far and the first time the bot cleared an act.
|
||
|
||
## Run 003 — the catastrophic event
|
||
|
||
Died to a **normal** encounter at 186 s. The cause was not combat:
|
||
|
||
```
|
||
[102] jev chose Keep Deciphering conf=0.28
|
||
[104] jev chose Lose Everything conf=0.49
|
||
```
|
||
|
||
The event offered "Keep Deciphering" and "Lose Everything". Jev picked
|
||
"Lose Everything", which **set the player's max HP to 1**. Every subsequent
|
||
state read then showed:
|
||
|
||
```
|
||
hp=1/1 (healthy)
|
||
```
|
||
|
||
The bot believed it was at full health, walked into a normal fight at 1 HP, and
|
||
died. Two separate defects, both now fixed:
|
||
|
||
1. `_hp_bucket` bucketed by percentage only, so `1/1` was 100% = "healthy".
|
||
Fixed: absolute HP <= 5 is always `critical`.
|
||
2. The event gate accepted a run-ending option at 0.49. Fixed with a stricter
|
||
event gate plus a deterministic keyword safety net. See
|
||
[05](05-failure-modes.md) and [02](02-system-one-jev.md).
|
||
|
||
## Run 004 — what the new decision loops did
|
||
|
||
| Decision | Count | Notes |
|
||
|---|---|---|
|
||
| `shop_purchase` | 6 in one shop | Lantern, Feel No Pain, Evil Eye, Salvo, Equilibrium, Headbutt |
|
||
| `use_potion` | 2+ | Fire Potion chosen by Jev at conf 1.00; two spent on lethal hits |
|
||
| `claim_treasure_relic` | 2 | Lucky Fysh, plus earlier Orichalcum and Bronze Scales |
|
||
| `combat_confirm_selection` | 4 | Hand-select now confirms instead of failing |
|
||
| event safety net | 1 | "uncertain (0.44); took safest option" |
|
||
|
||
Gold went from 528 to 130 across Act 2, so the "never buy anything" gap is
|
||
closed. Potions at death dropped from 3 (run 001) to 2 (run 004) while the run
|
||
went a full act further.
|
||
|
||
## Still open
|
||
|
||
- The bot still died holding 2 potions. The hard-need path only fires when the
|
||
incoming hit is *lethal*; it does not yet spend potions on a losing
|
||
attrition fight where HP is dropping every turn.
|
||
- Buying 6 items in one shop produced a 29-card deck. `SHOP_BUY_THRESHOLD`
|
||
may be too permissive; deck dilution is a real cost.
|
||
- Card reward still falls back to "take the first" fairly often.
|
||
|
||
## Recurring manual blocker
|
||
|
||
The Timeline epoch reveal blocked the menu again after run 004, this time with
|
||
`IRONCLAD2_EPOCH`. The mod refuses to automate it. A human must reveal it
|
||
before the next run can start. Expect this after most runs.
|
||
|
||
## Comparing two arms requires ONE revision
|
||
|
||
A run is only comparable with another run collected from the **same revision of
|
||
the decision layer**. `brain.py`, `facts.py`, `jev.py` and `run.py` decide how
|
||
*both* arms behave, so a change to any of them — the fight-length projection,
|
||
the lethal search, the state handed to Jev, the relic fix — invalidates every
|
||
run collected before it. The old runs stay as history; they are not a baseline.
|
||
|
||
`ab_card_skip.sh` now enforces this instead of relying on discipline:
|
||
|
||
- it hashes the decision layer (`brain.py facts.py jev.py run.py sts2.py`) into
|
||
a 12-character content hash, printed as `code revision: <hash>`;
|
||
- the hash is **recomputed before every session**, and the session line records
|
||
it: `--- jev 1/5 (attempt 1) rev=978485b6639a ---`. Taking it once at startup
|
||
would be worse than useless — `$RESULTS` is truncated when the run starts, so
|
||
every row would carry the startup hash by construction and the mixed-revision
|
||
warning could never fire;
|
||
- a session whose hash differs from the startup hash **aborts the experiment**
|
||
(`!! ABORT: the decision layer changed mid-experiment`), skips the remaining
|
||
arm, and still prints the report for whatever was collected;
|
||
- each attributed run is written as `policy \t run \t hash` in
|
||
`/tmp/ab_results.tsv`, stamped with the revision that actually produced it;
|
||
- the report scores **one** revision — the one this invocation collected from —
|
||
and prints a `!! MIXED REVISIONS` block when the file holds more than one,
|
||
with a `!! THIN SAMPLE` warning under five runs per arm. It reads the same
|
||
history directory the shell snapshotted, instead of a second hardcoded copy of
|
||
that path (which silently reported "no runs recorded" for runs that existed).
|
||
|
||
A content hash, not a commit id: the decision layer is normally edited in place
|
||
and uncommitted, so a commit id would not tell two revisions apart.
|
||
|
||
Verified in a sandbox with a stub `run.py` that edits `brain.py` between
|
||
sessions: the abort fires, the remaining arm is skipped, and the collected rows
|
||
keep the revision that produced them. A clean two-arm run stamps all four rows
|
||
with one hash.
|
||
|
||
**Rule:** before interpreting any A/B number, re-run BOTH arms from the same
|
||
revision. Runs already in `/tmp/ab_results.tsv` from before 2026-09-22 03:2x
|
||
are pre-change and must be re-collected.
|
||
|
||
Note also that `--steps` must stay generous. A session that hits the step cap
|
||
while the run is still going produces **no run record at all**, so a small step
|
||
budget silently fills the sample with runs that died early.
|