Read the vendor documentation against what the bot actually does, and separate
what is measured from what is merely by construction.
Adopt structured criteria where disambiguation costs us -- measured on card
play, the structured shape picked the same card 6/6 with margin 0.425 -> 0.473,
so it is a small effect worth having at irreversible decisions, not a blanket
rewrite.
What the measurements KILLED, recorded so it is not retried:
* A fight-level plan asked as a Score was unusable on 4 of 6 combat states,
confidence as low as 0.01. Fight-level planning stays in code.
* "A bigger margin means a better play" is not supported: the same question
repeated on the same state returned 0.04 -> 0.24 and 0.36 -> 0.02. We have
no optimal-action label, so a higher margin is evidence of noise, not skill.
Also records three correctness fixes that are independent of any model question:
enemy block counted twice in the lethal search, the executor ignoring player
statuses, and `relic_select` asking `good_relicN` while reading `relicN`.
Two findings, both from measurement rather than the public guides.
1. The public STS2 meta is not usable as evidence. Deck-size targets disagree by
3x across five sources (12-18 to 25-35), and one tier list names Demon Form,
Barricade, Limit Break and Body Slam as STS2 Ironclad cards -- those are Slay the
Spire 1 cards, so that page is an STS1 list retitled. Only Offering and Break
appear on more than one list.
2. Our own run files are far better evidence, and they contain 1052 labelled
decisions. They show the bot is NOT bloating (249 picks from 1087 offers = 23%
per card, ~69% per reward), that 81% of runs die in Act 1, and that 43% die to an
Act 1 boss with `THE_KIN_BOSS` alone killing 9.
The binding constraint is combat defence, not deck composition: those Kin fights
ran 5-10 turns and cost 44-80 HP with block cards in hand the whole time.
Two defects made the earlier A/B numbers meaningless:
* Without `--stop-on-run-end` a session contained several runs, so runs could
not be attributed to an arm.
* The script then read "the last 2N runs", which silently compared runs from
earlier experiments rather than the two arms.
Pass `--stop-on-run-end`, and attribute each run record to its arm by diffing
the history directory around each session.
Also raise the step budget and warn on a thin sample: measured, a run still
going at step 600 produces NO run record, so a short budget biases the sample
toward runs that died early -- exactly the wrong bias for this question.
Mining the 37 run files showed 81% of runs (30/37) die in Act 1, and 43% to an
Act 1 boss. THE_KIN_BOSS alone killed 9. Every one of those fights ran 5-10
turns and cost 44-80 HP -- about 10-13 a turn, with block cards in hand the
whole time. Four defects, fixed here together because they were found and
verified as one combat-correctness pass.
1. NO DEFENCE POLICY (the big one, found by mining the data)
facts classes a hit of <=15% of max HP as THREAT_CHIP. At 80 max HP that is
12, exactly what the boss deals, and _fallback_combat only blocked for HEAVY
or worse while HP was HEALTHY (>60%). So at 74/80 HP the bot attacked through
the boss's main attack and only started blocking below 48 HP.
_jev_combat also asked a should_defend Noul on every combat turn and never
read it -- grep -rn should_defend returned one line, the one creating it.
Defence therefore fell to choice("Which single play best advances winning
this fight?"), which is damage-biased: on the real Kin state Jev answered
Bash at 0.42 confidence, below the 0.45 gate, so it fell through to the
fallback, which also chose damage. Both paths agreed on the wrong answer.
Blocking is arithmetic, so it is now decided in code before Jev is asked,
using turns_to_kill, projected_incoming, affordable_loss and must_block /
block_urgent. Measured against all 600 real combat captures, the rule changes
8 of 68 in-play turns (11.8%) and stays silent on short fights and when
nothing is incoming.
2. ENEMY BLOCK COUNTED TWICE IN THE LETHAL SEARCH
`total - max(0, enemy.block) >= enemy.effective_hp` subtracts block a second
time, because effective_hp is already hp + block. A 10 hp / 5 block enemy
against 18 raw damage read as "not lethal" and real kills were discarded.
3. THE LETHAL EXECUTOR IGNORED PLAYER STATUSES
`_lethal_line(f.playable, f.energy, [], enemy)` passed an empty status list,
so facts reported lethal_available: true while the code meant to execute the
kill found nothing. CombatFacts.player_status is now passed through.
4. RELIC_SELECT ASKED good_relicN AND READ relicN
Every answer missed, so best_by_noul returned (None, 0.0) for every state and
the path could only ever take the rarest relic.
Tests: 50 in test_facts.py and 131 in test_brain.py, with a regression case for
each -- a blocked enemy, a Strength-carrying player, a relic offer whose
highest-rated relic is deliberately the common one so the rarity fallback cannot
pass by accident, and the Kin turn itself.
A session was `--steps 600`, and dying did not stop it: the bot returned to the
menu and started a fresh run inside the same session. One session therefore
produced several run records, and nothing could be attributed to an experimental
arm. The A/B harness then compared "the last 2N runs", which silently included
runs from earlier experiments.
Add `--stop-on-run-end`, so one session is one run.
Two deadlocks found while making it work, both measured:
* Breaking on `game_over` BEFORE dismissing the screen left the game parked
there. Every later session then saw `game_over` at step 1 and stopped
instantly -- `takes=0 skips=0` across a whole batch.
* `preflight` could not recover from that parked screen, so it now dismisses
it. The dismissal is not instant, hence the wait: without it the loop
re-reads the state, still sees `game_over`, and still stops at step 1.
`preflight` also fails fast on a pending Timeline epoch, which the mod refuses
to automate. A whole A/B arm once ran with 0 decisions because of it.
`JEV_TRACE` alone cannot be joined to a decision: it records a time to the
second and the answers, with no run, step or action, so nothing in it can be
traced back to a run.
Wrap the client in `RecordingClient` so the last call's questions and answers
travel with the decision row they produced, and give each process a
`SESSION_ID` so rows from two sessions in the same second stay apart. Diff the
history directory before and after a session to attribute the run records it
produced.
What this supports is arm-level analysis: this decision belongs to this session,
and the session's outcome is the run file. It does NOT say whether an individual
answer was correct -- one run result attached to one step cannot label that
step. Per-decision accuracy needs replay, expert judgement, or ground truth the
code can verify on its own (lethal, legality, affordability).
A decision that asked nothing must carry no answers, so `client.last` is
cleared before each decide(); otherwise rows silently inherit the previous
step's answers.
The documented API accepts an object or array wherever a string is accepted:
`instructions`, every Choice option description, every Score level, and the
Noul `criteria.true` / `criteria.false` entries. The client sent bare strings
everywhere and never used `criteria` on a Noul at all.
Add the builders for those shapes:
* `ask()` structured instructions: question, focus, inspect, compare
* `entry()` a description with what it covers, what it is not for, examples
* `noul_criteria()` contrastive true/false criteria
* `score()` the ordered-level primitive, documented but not implemented
* `gate_choice()` an explicit floor per Choice call site
A threshold tuned on a Noul is never reused on a Choice: the two answer different
questions and are not on a comparable scale, so each gate states its own pair
(`CHOICE_TOP_MIN`, `CHOICE_MARGIN_MIN`).
Also add `answer_record()` and the `JEV_TRACE` writer, so a decision can carry
the answers that produced it. Without a trace there are no labels for any model
decision, which blocks every question about decision quality.
Hand-captured state snapshots (menu through hand_select) used to
build the facts layer and decision handlers. Live run dumps are
gitignored, these stay for reference.
eval_batch.sh: N back-to-back sessions with per-session summaries.
ab_card_skip.sh: A/B the jev vs combined card-reward skip policy and
compare deck size and progress from the game's run history.
observe -> decide -> act loop: Timeline preflight, stuck detection,
duplicate-action suppression, deck snapshot persistence. Appends one
JSON line per decision and per rejected action to
capture/decisions.jsonl for tail -f.
Batched question API (noul/choice/score), retrying transport, and a
margin-based confidence gate. Every call appends one JSON line of
questions and parsed answers to $JEV_TRACE when set; tracing never
raises, so it cannot break a run.
Pure computation: lethal damage lines, block values, threat model,
deck counts, affordability. Jev never does arithmetic; facts.py
computes the numbers and passes conclusions in.
sts2.py: thin client for the mod's localhost HTTP API, maps
state_type -> legal actions (the action space), strict
observe -> act once -> observe-again loop.
capture.py: manual state snapshot tool for building facts.py.
Full C# source of the kunology STS2MCP fork (McpMod HTTP bridge for
game state and actions) plus the prebuilt 194,560-byte release DLL and
its manifest. The rebuilt fork DLL from docs/research/03 is not saved;
rebuild from this source if the pinned 0.4.0 breaks on a game update.
DESIGN.md covers the three-layer architecture (facts in code, Jev for
tactics, gated escalation for macro). research/ documents the engine and
mod surface, the Jev classifier's measured behavior, the STS2MCP HTTP
interface, state shapes, failure modes, decision architecture, and a
run log of the first four sessions.
