diff --git a/docs/research/08-what-actually-wins.md b/docs/research/08-what-actually-wins.md new file mode 100644 index 0000000..975740a --- /dev/null +++ b/docs/research/08-what-actually-wins.md @@ -0,0 +1,244 @@ +# 08 — What actually wins (and why the web meta is not usable) + +Status: **current**. Supersedes the assumption in `06-decision-architecture.md` +that deck composition is the primary lever. + +## Question + +Before tuning decision quality, establish what actually decides a run in STS2: +deck archetype, deck size ("lean weight"), card rarity, or something else. + +## Finding 1 — the public STS2 "meta" is not usable as evidence + +Searched for deck archetypes, tier lists and Act 1 advice. The sources +contradict each other badly, and at least one is a copy of a Slay the Spire 1 +list relabelled. + +Deck size target, by source: + +| Source | Deck size target | +|---|---| +| metabot.gg | 12–18 | +| gamestrategyhub | 15–20 | +| games.gg | ~22 | +| slashskill | 25–30 | +| sts2guides | 25–35 | + +That is a 3x range. A real meta does not disagree with itself by 3x. + +Ironclad S-tier, by source: + +| Source | S-tier | +|---|---| +| slaythespire2.space | Demon Form, Barricade, Dark Embrace, Limit Break, Body Slam, Whirlwind | +| spireforge | Barricade, Cruelty, Dark Embrace | +| stratgg | Offering, Pyre, Battle Trance | +| metabot.gg | Apotheosis, Secret Weapon, Mad Science, Folly, Tank | +| nat1gaming | Offering, Break, Feed | + +**`slaythespire2.space` lists Demon Form, Barricade, Limit Break, Body Slam and +Whirlwind as STS2 Ironclad cards. Those are Slay the Spire 1 cards.** That page +is an STS1 tier list with the title changed. + +Only two cards — **Offering** and **Break** — appear on more than one list. + +Conclusion: these are AI-generated SEO pages. Do not build a scoring model on +them. The one idea they agree on is qualitative and obvious ("commit to one +archetype, keep the deck lean"), which we cannot operationalise without numbers. + +## Finding 2 — our own run history is far better evidence + +37 runs on disk, all reachable and machine-readable: + +``` +~/Library/Application Support/SlayTheSpire2/steam//modded/profile1/saves/history/*.run +``` + +Each file records `killed_by_encounter`, `win`, the full deck, relics, and +`map_point_history` — a per-map-point log that includes **every card offered +with a `was_picked` flag**, plus per-fight `damage_taken`, `potion_used`, +`rest_site_choices` and `upgraded_cards`. + +That is a labelled dataset of our own drafting and combat outcomes. + +### Outcome distribution + +``` +runs=37 wins=0 max_pts=33 +act1 deaths: 30/37 (81%) +act2 deaths: 4/37 +act3 deaths: 3/37 +``` + +| Killed by | Count | +|---|---| +| `THE_KIN_BOSS` | **9** | +| `VANTOM_BOSS` | 5 | +| `CEREMONIAL_BEAST_BOSS` | 2 | +| Act 1 normals/elites | 21 | + +**43% of runs die to an Act 1 boss. `THE_KIN_BOSS` alone kills 9 of 37.** That +is the single largest lever in the dataset. + +### The bot is not bloating + +``` +total offers=1087 total picks=249 overall pick rate=23% +rewards=323 rewards where a card was taken=249 -> 77% taken, 23% skipped +``` + +**Descriptive, not a benchmark.** The per-card rate (23%) is mechanically +diluted by offer size — 280 of 323 rewards offered 3 cards, some offered up to 7 +— so it cannot diagnose skip policy on its own. The metric that can is the share +of rewards taken (77%) and, below it, cards added per map point. We have no +reference for what a good skip rate is in STS2, so this says what the bot does, +not whether it is right. + +The "deck bloat" hypothesis is **not supported**: 23% of rewards were skipped, +one card was taken per reward at most, and 81% of runs never reach the point +where deck composition matters. + +### Drafting profile (offers / picks) + +| Card | Seen | Took | Pick rate | +|---|---|---|---| +| `PERFECTED_STRIKE` | 17 | 17 | 100% | +| `EVIL_EYE` | 19 | 12 | 63% | +| `POMMEL_STRIKE` | 26 | 14 | 54% | +| `SHRUG_IT_OFF` | 30 | 16 | 53% | +| `ARMAMENTS` | 39 | 19 | 49% | +| `TREMBLE` | 45 | **0** | **0%** | +| `RUPTURE` | 18 | 0 | 0% | +| `FIGHT_ME` | 17 | 0 | 0% | +| `SWORD_BOOMERANG` | 16 | 0 | 0% | + +`TREMBLE` is the most-offered card in the game and the bot has never once taken +it. Worth auditing, but it is not the binding constraint. + +## Finding 3 — the binding constraint is combat defense, not deck building + +Every `THE_KIN_BOSS` fight, from the run files: + +``` +damage taken: [44, 50, 50, 53, 63, 64, 70, 75, 80] +potions used: [3, 3, 2, 1, 3, 3, 2, 3, 2] +turns: [6, 9, 10, 6, 6, 10, 9, 7, 5] +deck size: [17, 18, 19, 19, 20, 20, 21, 22, 23] +``` + +The bot enters at ~74–80 HP and dies. Fights last **5–10 turns** and it takes +**44–80 damage**, i.e. **~10–13 per turn, almost entirely unblocked**. It spends +1–3 potions and still loses. + +Every one of those decks contained block cards: + +``` +DEFEND_IRONCLAD 9/9 +BASH 9/9 +STRIKE_IRONCLAD 9/9 +``` + +So the cards are there. The bot does not play them. + +### Root cause A — the fallback treats a boss's main attack as "chip damage" + +`facts.py`: + +```python +THREAT_CHIP = "chip" # <= 15% of max HP +THREAT_HEAVY = "heavy" # <= 40% of max HP +HP_HEALTHY = "healthy" # > 60% +``` + +At `max_hp = 80` that gives `CHIP <= 12`, and `HEALTHY` for `hp > 48`. + +`brain.py::_fallback_combat`: + +```python +must_respect = ( + f.threat in (F.THREAT_HEAVY, F.THREAT_SEVERE, F.THREAT_LETHAL) + or f.hp_bucket in (F.HP_WOUNDED, F.HP_CRITICAL) +) +``` + +So at 74/80 HP, any hit of **12 or less is ignored**. Measured, reproduced: + +``` + hp incoming threat hp_bucket fallback play + 74 12 chip healthy Bash (dmg) <- takes the full 12 + 74 13 heavy healthy Defend (block) + 60 12 chip healthy Bash (dmg) <- still attacking + 45 12 chip wounded Defend (block) <- too late, 29 HP gone +``` + +The boss's main attack lands exactly on the `CHIP`/`HEAVY` boundary, so the bot +attacks and eats it. It only starts blocking once HP drops below 48 — after +donating ~29 HP. + +The model is **myopic**: it asks "is this turn's hit big?" when the correct +question is "will the total damage over the remaining fight kill me?". + +### Root cause B — `should_defend` is asked and then discarded + +`brain.py::_jev_combat` builds this question: + +```python +questions["should_defend"] = noul( + "Given `combat.incoming_threat` and `combat.your_health`, " + "is preventing damage more valuable than dealing damage this turn?" +) +``` + +It appears **exactly once in the entire codebase** — at the line where it is +created. It is never read. + +``` +$ grep -rn 'should_defend' *.py +brain.py:278: questions["should_defend"] = noul( +``` + +So it costs latency in every combat turn and does nothing. + +### Root cause C — the surviving question is damage-biased + +Defense is therefore delegated entirely to: + +```python +questions["best_play"] = choice( + "Which single play best advances winning this fight?", options) +``` + +"Advances winning" biases toward damage, and a `Choice` over 5–8 cards is the +question shape measured in `02-system-one-jev.md` as **diluting with option +count**. + +Measured directly against the Kin state (74/80 HP, 12 incoming, turn 6): + +``` +$ python3 -c "... brain._jev_combat(f, jev.JevClient()) ..." +play_card {'card_index': 4, 'target': 'kin'} | jev chose Bash | conf: 0.42 + -> Bash DAMAGE +``` + +Jev picks **Bash** at confidence **0.42**. That is below the 0.45 gate, so +`_jev_combat` bails to `_fallback_combat`, which also picks damage. **Both +paths agree on the wrong answer.** + +## Conclusion + +The bot has no defense policy. In boss fights it attacks until it dies, and the +data shows exactly that: 5–10 turn fights, 10–13 unblocked damage per turn, +44–80 damage taken, 9 runs ended by one boss. + +Priority order for the next work: + +1. **Make defense a first-class, code-computed decision.** Not a Jev preference. + The quantity that matters is total incoming damage over the expected + remaining fight, not this turn's hit. +2. **Fix or delete `should_defend`.** A dead question that costs latency is + worse than no question. +3. **Only then** revisit deck composition. 81% of runs never reach the point + where it matters. + +Deck size, rarity and archetype are real concerns in Slay the Spire, but they +are not what is killing this bot right now.