Add design doc and research notes
DESIGN.md covers the three-layer architecture (facts in code, Jev for tactics, gated escalation for macro). research/ documents the engine and mod surface, the Jev classifier's measured behavior, the STS2MCP HTTP interface, state shapes, failure modes, decision architecture, and a run log of the first four sessions.
This commit is contained in:
0xrsydn
commit
fb32822468
9 changed files with 2341 additions and 0 deletions
374
docs/DESIGN.md
Normal file
374
docs/DESIGN.md
Normal file
|
|
@ -0,0 +1,374 @@
|
|||
# STS2 Bot — Design (verified 2026)
|
||||
|
||||
Jev (TypeSafe System One) as the tactics policy. Confidence-gated LLM escalation for macro.
|
||||
|
||||
## Verified environment
|
||||
|
||||
| Item | Value |
|
||||
|---|---|
|
||||
| Game build | v0.107.1, commit 59260271 |
|
||||
| Engine | Godot 4.5.1 (.NET), .NET 9.0.7, osx-arm64 |
|
||||
| Main assembly | `sts2.dll` + `sts2.xml` (19,635 documented members) |
|
||||
| Mods dir | `SlayTheSpire2.app/Contents/MacOS/mods/` (absent) |
|
||||
| Workshop dir | `Steam/steamapps/workshop/content/2868840/` (absent) |
|
||||
| TypeSafe API | live; needs `TYPESAFE_API_KEY` (unset) |
|
||||
| STS2MCP | v0.4.0, tested against game v0.103.2 |
|
||||
| Python SDKs | none installed; Node available |
|
||||
|
||||
## Manifest schema (JSON, from STS2_MCP.json)
|
||||
|
||||
{"id","name","author","description","version","has_pck","has_dll","affects_gameplay"}
|
||||
|
||||
Note: snake_case in JSON, not the C# property names.
|
||||
|
||||
## Actuator: STS2MCP
|
||||
|
||||
Base URL `http://localhost:15526`. No auth.
|
||||
|
||||
- `GET /api/v1/singleplayer?format=json|markdown` — read state
|
||||
- `POST /api/v1/singleplayer` — `{"action": ..., ...}`
|
||||
- `GET /api/v1/compendium`, `/profile`, `/wiki?query=`, `/profiles`
|
||||
|
||||
Response always has `state_type`, `run{act,floor,ascension}`, `player`.
|
||||
|
||||
### state_type -> legal actions (this IS the action enumerator)
|
||||
|
||||
menu, game_over -> menu_select
|
||||
monster/elite/boss -> play_card, use_potion, end_turn
|
||||
hand_select -> combat_select_card, combat_confirm_selection
|
||||
rewards -> claim_reward, proceed
|
||||
card_reward -> select_card_reward, skip_card_reward
|
||||
map -> choose_map_node
|
||||
event -> choose_event_option, advance_dialogue
|
||||
rest_site -> choose_rest_option, proceed
|
||||
shop/fake_merchant -> shop_purchase, proceed
|
||||
treasure -> claim_treasure_relic, proceed
|
||||
card_select -> select_card, confirm_selection, cancel_selection
|
||||
bundle_select -> select_bundle, confirm_bundle_selection, cancel_bundle_selection
|
||||
relic_select -> select_relic, skip_relic_selection
|
||||
crystal_sphere -> crystal_sphere_set_tool/click_cell/proceed
|
||||
overlay, unknown -> none (manual)
|
||||
|
||||
`use_potion` / `discard_potion` work in any state where potions are reachable.
|
||||
|
||||
Card rules text ships in game state when cards are visible.
|
||||
|
||||
## Decision loop
|
||||
|
||||
Card indices shift on every play. The loop is strictly closed:
|
||||
|
||||
observe -> compute facts (pure code) -> ask Jev (1 batched call)
|
||||
-> gate on confidence -> execute ONE action -> re-observe
|
||||
|
||||
Never precompute an action list.
|
||||
|
||||
## Three layers
|
||||
|
||||
| Layer | Engine | Responsibility |
|
||||
|---|---|---|
|
||||
| Facts | pure Python | arithmetic, legality, deck stats, threat model |
|
||||
| Tactics | Jev | per-turn card play, targeting, potions, event options |
|
||||
| Macro | scripted -> LLM | archetype, card rewards, routing, boss prep |
|
||||
| Gate | confidence threshold | when to escalate to Claude/GPT |
|
||||
|
||||
Critical: Jev cannot count or do arithmetic (documented jaggedness).
|
||||
Compute numbers in code. Send conclusions, not raw values.
|
||||
|
||||
## Jev primitives
|
||||
|
||||
- Choice: pick 1 of <=255 options -> choice, probabilities, confidence
|
||||
- Score: position on ordered levels -> score, legend, probabilities, confidence
|
||||
- Noul: yes/no -> noul (probability only, no confidence)
|
||||
|
||||
Batch all questions for one state into ONE call. Parallel, ~100ms,
|
||||
near-free latency, output tokens free. 12.2x cheaper than separate calls.
|
||||
|
||||
Limits: 64k context total; 32k state + longest question. Text only.
|
||||
Price $0.042/Mtok input. Rate 250k tok/s, 1200 req/min.
|
||||
|
||||
## Combat question template (one batched call)
|
||||
|
||||
lethal_this_turn Noul any line in hand kills all enemies
|
||||
must_block Noul preventing damage beats dealing damage
|
||||
best_play Choice one option per legal play, described in words
|
||||
target Choice which enemy to hit
|
||||
|
||||
Precedence in code: lethal -> must_block -> best_play.
|
||||
Confidence < threshold -> fall back to AGENTS.md heuristics.
|
||||
|
||||
## Cost projection
|
||||
|
||||
| Metric | Estimate |
|
||||
|---|---|
|
||||
| Tokens/run | ~2M filtered |
|
||||
| Cost/run | ~$0.08-0.35 |
|
||||
| Model time/run | ~100-150 s |
|
||||
| Wall clock/run | ~5-15 min |
|
||||
| Frontier LLM agent (reported) | ~8M tok, ~$20-40/run |
|
||||
|
||||
## Risks
|
||||
|
||||
1. Version drift: STS2MCP tested on v0.103.2, game is v0.107.1.
|
||||
2. Jev arithmetic/counting failure -> mitigate by computing in code.
|
||||
3. Context rot -> send filtered state, never a raw dump.
|
||||
4. Index shift -> strictly closed loop, re-read after each action.
|
||||
5. Macro plateau -> confidence-gated escalation to a reasoning LLM.
|
||||
6. Doc drift: repo AGENTS.md uses MCP tool names (`combat_end_turn`,
|
||||
`rewards_pick_card`); HTTP uses (`end_turn`, `select_card_reward`).
|
||||
Trust raw-simplified.md for direct HTTP use.
|
||||
|
||||
## MEASURED RESULTS (validated against live API)
|
||||
|
||||
### Credential
|
||||
Key found at `~/.config/secrets/global-env/TYPESAFEAI_API_KEY`
|
||||
(sops-nix symlink -> `~/.config/sops-nix/secrets/`). NOT in the shell
|
||||
environment. Read from that path at runtime; never log the value.
|
||||
|
||||
### Validation
|
||||
First call: HTTP 200, `model: jev-1.13.0`. Game-correct answers on a
|
||||
combat scenario (Cleave into the 12 HP attacker, confidence 0.85).
|
||||
|
||||
### Latency (measured, NOT the documented 100 ms)
|
||||
| Request | Time |
|
||||
|---|---|
|
||||
| Minimal (1 short noul) | 0.73 s |
|
||||
| Minimal, 5 repeats | 0.73 / 0.73 / 0.74 / 0.80 / 0.76 s |
|
||||
| Full combat state, 3 questions | 0.90 s |
|
||||
| GET /v1/models ttfb | 0.68 s (connect 0.22 s) |
|
||||
|
||||
Floor is ~0.73 s, dominated by network + server TTFB.
|
||||
State size and question count are nearly free -> fan-out is confirmed.
|
||||
|
||||
Revised budget: ~0.75 s per action call. ~500-600 actions per run
|
||||
-> ~7-8 min pure API time. Wall clock per run: ~20-45 min.
|
||||
|
||||
### Arithmetic failure CONFIRMED (critical)
|
||||
Ground truth: energy 3, hand Strike(1c/6dmg) x3 + Bash(2c/8dmg),
|
||||
target 19 HP. Max reachable damage = 18. Lethal = NO.
|
||||
|
||||
max_damage_bucket -> "18_to_23" (0.73) CORRECT
|
||||
lethal_available -> 0.79 WRONG (true answer: no)
|
||||
|
||||
The model bucketed the magnitude correctly but failed the threshold
|
||||
comparison (18 vs 19).
|
||||
|
||||
CRITICAL IMPLICATION: confidence was 0.79 on a wrong answer.
|
||||
Confidence gating does NOT protect against arithmetic errors.
|
||||
All arithmetic (lethal, block deficit, damage totals) MUST be computed
|
||||
in code and passed in as conclusions. Never ask Jev to compare numbers.
|
||||
|
||||
### Bundle signing (macOS)
|
||||
App is adhoc-signed; `spctl -a` already returns "rejected". No quarantine
|
||||
xattr (only com.apple.provenance). Adding mod files breaks the seal but
|
||||
there is no enforced signature. Rollback: `rm -rf <bundle>/Contents/MacOS/mods`.
|
||||
|
||||
### Installed
|
||||
STS2MCP v0.4.0 -> `SlayTheSpire2.app/Contents/MacOS/mods/`
|
||||
(staged copy kept in `~/sts2-bot/vendor/`)
|
||||
Mods load at startup, so the game must be restarted.
|
||||
|
||||
## VERSION DRIFT: STS2MCP v0.4.0 is broken on game v0.107.1
|
||||
|
||||
### Symptom
|
||||
GET /api/v1/singleplayer returns HTTP 500 once combat starts:
|
||||
|
||||
"error": "Failed to read game state: Method not found:
|
||||
'Boolean MegaCrit.Sts2.Core.Combat.CombatManager.get_IsPlayPhase()'."
|
||||
"exception_type": "System.MissingMethodException"
|
||||
at STS2_MCP.McpMod.BuildBattleState(RunState, CombatRoom)
|
||||
|
||||
Menu, map, and character-select states work. COMBAT state fails.
|
||||
|
||||
### Root cause
|
||||
`CombatManager.IsPlayPhase` was removed in game v0.107.1.
|
||||
Verified: grep -c IsPlayPhase sts2.xml -> 0.
|
||||
|
||||
Replacement in v0.107.1 (confirmed in sts2.xml):
|
||||
* `MegaCrit.Sts2.Core.Combat.PlayerTurnPhase.Play` (per-player)
|
||||
* `PlayerCombatState.Phase`
|
||||
* `ActionSynchronizerCombatState.PlayPhase` / `.NotPlayPhase`
|
||||
* `CombatManager.DebugOnlyGetState().CurrentSide == CombatSide.Player`
|
||||
|
||||
### Timeline
|
||||
* release 0.4.0 2026-05-05 (broken, pre-fix)
|
||||
* fix commit 55e06485 2026-07-29 PR #123 "Fix game API compatibility with STS2 v0.107"
|
||||
* no tag after 0.4.0, no CI workflow, no fork ships a prebuilt DLL
|
||||
|
||||
### Resolution: build from source
|
||||
Installed .NET 9 SDK ephemerally via nix (no system changes):
|
||||
|
||||
nix eval --raw nixpkgs#dotnet-sdk_9.name -> dotnet-sdk-wrapped-9.0.310
|
||||
|
||||
Build (macOS):
|
||||
|
||||
cd ~/sts2-bot/vendor/STS2MCP
|
||||
nix shell nixpkgs#dotnet-sdk_9 --command bash -c '
|
||||
dotnet build STS2_MCP.csproj -c Release -o out/STS2_MCP \
|
||||
-p:STS2GameDir="$HOME/Library/Application Support/Steam/steamapps/common/Slay the Spire 2"'
|
||||
|
||||
Result: Build succeeded, 0 warnings, 0 errors.
|
||||
Installed 236,544 B dll (release was 194,560 B).
|
||||
Old release dll kept at vendor/STS2_MCP.dll.release-0.4.0.
|
||||
|
||||
### Notes
|
||||
* Upstream open issues #131/#132 track v0.111 compatibility. Re-check before
|
||||
upgrading the game, since the same class of break will recur.
|
||||
* The csproj resolves the data dir on macOS automatically to
|
||||
SlayTheSpire2.app/Contents/Resources/data_sts2_macos_arm64 and references
|
||||
sts2.dll, GodotSharp.dll, 0Harmony.dll with Private=false.
|
||||
|
||||
## Confirmed save layout (modded)
|
||||
|
||||
~/Library/Application Support/SlayTheSpire2/steam/<steamid>/
|
||||
settings.save {"mod_settings": {"mods_enabled": true}}
|
||||
profile.save
|
||||
profile1/saves/... vanilla progress
|
||||
modded/profile1/saves/ MODDED progress (isolated)
|
||||
prefs.save, progress.save, history/
|
||||
|
||||
Modded runs never touch vanilla saves.
|
||||
|
||||
## Confirmed state shapes
|
||||
|
||||
menu (main):
|
||||
{state_type, menu_screen, message, options[]}
|
||||
character_select:
|
||||
{state_type, menu_screen, message, characters[]{name,id,locked,hp,gold,energy,
|
||||
description,starting_relics[],starting_deck[],total_cards,total_relics,total_potions},
|
||||
options[]{name,enabled}}
|
||||
Only IRONCLAD unlocked on a fresh profile. No seed option in standard SP
|
||||
(docs: supplying seed here errors).
|
||||
tutorial_prompt:
|
||||
{state_type, menu_screen, message, options[]{name,enabled}} -> choose 'no'
|
||||
map:
|
||||
{state_type, map{visited,next_options[]{index,col,row,type,leads_to},nodes[]},
|
||||
run{act,floor,ascension},
|
||||
player{character,hp,max_hp,block,gold,status[],relics[],potions[],max_potion_slots}}
|
||||
No seed field is exposed in run state.
|
||||
combat: BLOCKED until the rebuilt dll loads.
|
||||
|
||||
# MILESTONE: bot reached the Act 1 boss
|
||||
|
||||
## First full run result (run history file)
|
||||
|
||||
~/Library/Application Support/SlayTheSpire2/steam/<id>/modded/profile1/saves/history/<ts>.run
|
||||
|
||||
win : false
|
||||
killed_by_encounter : "ENCOUNTER.VANTOM_BOSS"
|
||||
seed : "JUS59Z32HF"
|
||||
run_time : 2017 s
|
||||
build_id : "v0.107.1"
|
||||
ascension : 0
|
||||
game_mode : "standard"
|
||||
was_abandoned : false
|
||||
acts : 3
|
||||
schema_version : 9
|
||||
|
||||
The bot cleared 15 floors, beat an elite (Byrdonis, 81 HP), healed at rest
|
||||
sites, and died to the Act 1 boss Vantom on floor 16.
|
||||
|
||||
## Evaluation harness: SOLVED
|
||||
|
||||
The .run history file IS the eval record. It exposes win/loss, killed_by,
|
||||
seed, run_time, and build_id. Stage 5 (win rate over seeds) reads this file.
|
||||
The seed is only visible HERE -- it is not in live state and cannot be set
|
||||
for standard singleplayer.
|
||||
|
||||
## Decision sources (759 loop steps, 5 sessions)
|
||||
|
||||
code 141 deterministic: lethal lines, procedure, arithmetic
|
||||
jev 70 preference: which card, which node, which event option
|
||||
fallback 35 heuristic: low-confidence recovery, no-Jev mode
|
||||
lethal lines executed by code: 33
|
||||
|
||||
## State shapes discovered (all verified live)
|
||||
|
||||
menu {state_type, menu_screen, message, options[],
|
||||
blocked_options[]{name,enabled,reason,pending_epoch_ids}}
|
||||
character_select {characters[]{name,id,locked,hp,gold,energy,description,
|
||||
starting_relics[],starting_deck[],total_cards,...}, options[]}
|
||||
map {map{visited,next_options[]{index,col,row,type,leads_to},nodes[],
|
||||
boss{id,name}}, run, player}
|
||||
monster/elite/boss
|
||||
{battle{round,turn,is_play_phase,enemies[]{entity_id,combat_id,
|
||||
name,hp,max_hp,block,status[]{id,name,amount,type,description},
|
||||
intents[]{type,label,title,description}}},
|
||||
run{act,floor,ascension},
|
||||
player{character,hp,max_hp,block,energy,max_energy,hand[],
|
||||
draw_pile[],draw_pile_count,discard_pile[],exhaust_pile[],
|
||||
gold,status[],relics[],potions[],max_potion_slots}}
|
||||
card : {id,name,type,cost(str),star_cost,description,rarity,is_upgraded,
|
||||
keywords[],index,target_type,can_play,unplayable_reason}
|
||||
pile card: {name,cost,star_cost,description} <-- NO id/type/index
|
||||
intent label: damage as a STRING, e.g. "12"; may be "12x2"
|
||||
rewards {rewards{items[]{index,type,description,[gold_amount],
|
||||
[potion_id,potion_name,potion_description]}, can_proceed}}
|
||||
items[] contains ONLY enabled rewards, re-indexed per claim
|
||||
card_reward {card_reward{cards[], can_skip}} <-- NO deck composition
|
||||
card_select {card_select{screen_type,prompt,cards[],preview_showing,
|
||||
preview_cards,can_cancel,can_confirm}}
|
||||
event {event{in_dialogue,body,options[]{index,title,description,
|
||||
is_locked,is_proceed,was_chosen,[relic_name,relic_description]}}}
|
||||
rest_site {rest_site{options[]{index,id,name,description,is_enabled},
|
||||
can_proceed}} <-- `name`, NOT `title`
|
||||
treasure {treasure{relics[]{index,id,name,description,rarity},
|
||||
can_proceed,message}} <-- relics ABSENT while chest opens
|
||||
shop {shop{items[],can_proceed}}
|
||||
|
||||
## Bugs found live and fixed (all were real infinite loops)
|
||||
|
||||
1. claim_reward(index=0) forever.
|
||||
items[] re-indexes on every claim. Must claim RIGHT-TO-LEFT.
|
||||
2. claim_reward on a potion with all slots full.
|
||||
Returns "ok" and is SILENTLY DROPPED, so the item never leaves the list.
|
||||
Fix: discard the weakest potion first (POTION_VALUE table in brain.py).
|
||||
3. select_card twice on one index.
|
||||
It TOGGLES on grid screens; the second call deselects and freezes the
|
||||
screen. Fix: if preview_showing and can_confirm -> confirm_selection.
|
||||
4. Stale preview: confirm_selection returns ok and changes nothing.
|
||||
Fix: if the identical card_select state repeats, cancel_selection to reset.
|
||||
5. rest_site read options[].title, but the field is `name` (+ is_enabled).
|
||||
Fix applied.
|
||||
6. treasure claimed a relic before the chest auto-opened.
|
||||
relics[] is absent during opening; only can_proceed is set.
|
||||
Fix: wait while relics absent.
|
||||
7. Acting during transitions.
|
||||
choose_map_node fired 3x during travel; end_turn during the enemy turn.
|
||||
Fix: __wait__ on not-in-play-phase, plus an unchanged-state guard in run.py
|
||||
that waits when the last action LANDED, and re-decides when it was REJECTED.
|
||||
8. state_type 'unknown'/'overlay' treated as a dead end.
|
||||
These are transitions. Fix: wait instead of stopping.
|
||||
|
||||
## Confidence gate bug (found by measurement)
|
||||
|
||||
A fixed confidence floor is wrong when option count varies. Measured:
|
||||
5 cards, Jev picks Bash at 0.61 with Defend at 0.29 -> confidence 0.50
|
||||
because confidence measures peakedness: (5*0.61-1)/4 = 0.50.
|
||||
A 0.55 floor rejected a clear plurality.
|
||||
|
||||
Fix: gate a Choice on (top >= 0.45 AND top - runner_up >= 0.20).
|
||||
Noul has no confidence; gate on |noul-0.5| >= 0.15.
|
||||
|
||||
## BLOCKER: Timeline epoch reveal needs a human
|
||||
|
||||
After the run ended, the main menu offers only `settings` and `quit`.
|
||||
blocked_options reports:
|
||||
|
||||
{"name":"timeline","enabled":false,
|
||||
"reason":"manual_epoch_reveal_required",
|
||||
"pending_epoch_ids":["NEOW_EPOCH"]}
|
||||
|
||||
menu_select("timeline") refuses:
|
||||
|
||||
"Timeline has obtained epochs that still need to be revealed manually;
|
||||
not opening Timeline because this game state logs invalid unlock-state
|
||||
errors when entered through automation"
|
||||
|
||||
This is a deliberate guard in the mod, not a bug. The user must open the
|
||||
Timeline in-game and reveal NEOW_EPOCH by hand. Until then the bot cannot
|
||||
start a new run.
|
||||
|
||||
## Test suite
|
||||
|
||||
test_facts.py: 29 assertions, all passing, including the exact case where
|
||||
jev-1.13.0 answered "lethal available" 0.79 YES when the true answer was NO.
|
||||
Loading…
Add table
Add a link
Reference in a new issue