feat(dataset): migrate game history and captures into a trainable corpus

Browse source

Turn the data we already have into an open, educational dataset, so the trace we
are about to start collecting has somewhere to go.

`migrate.py` produces:
  * runs.jsonl          37 runs with outcome, killer, seed and final deck
  * decisions.jsonl     1052 decisions, each with its own outcome attached
  * states_index.jsonl  346 unique observations
  * states/             content-addressed gzipped blobs

Content addressing matters: measured, only 56% of captures are unique, so 44% of
storage is duplicates. 3.28 MB raw -> 0.42 MB stored.

The card-reward rows keep the REJECTED options, so this is a ranking dataset
rather than a classification one, and the per-fight `damage_taken` /
`turns_taken` pair is the dense reward signal a combat policy is judged on.

What it deliberately does NOT do: reconstruct per-step combat state/action pairs.
The session logs record the action but not the observation, and captures exist
only for combat, so a step has a state with no action or an action with no state
-- never both. Inventing them would poison the corpus. The gap is declared in
manifest.json instead, and collect.py will close it going forward.

Integrity checking is a separate entry point (`--check-only`) because `--verify`
alone rebuilds first and so can only ever see data that is correct by
construction -- a smoke test pretending to be a check. All six invariants were
verified by deliberately breaking the dataset and confirming the checker fails.

...

This commit is contained in:

0xrsydn 2026-09-22 06:06:21 +07:00

parent 239a42c317

commit 8fae007e50

352 changed files with 2164 additions and 0 deletions

Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL

Download patch file Download diff file Expand all files Collapse all files

BIN

dataset/states/00/001ec3fa6ec0110563c995eeda471c7a88aea3a8b624b6003d3813b0e02f3526.json.gz Normal file

View file

Binary file not shown.

BIN

dataset/states/02/020bb5844a357f98b70e7ac4dcc4fdd32e4a03346a87e860a8a8564087f3cf0a.json.gz Normal file

View file

Binary file not shown.

BIN

dataset/states/02/021b78d04a551063748ccb72835ba6e79bf1b86cab6ca642e1b89448c13f7747.json.gz Normal file

View file

Binary file not shown.

BIN

dataset/states/02/0220066524cf5ac390252a36bb7b3763976ad5c0c6bbc92de2924615374bf083.json.gz Normal file

View file

Binary file not shown.

BIN

dataset/states/02/02c4af344fcb277be34f8e3b16828f81f686e8b210e2ebc37ca61e003c711fc1.json.gz Normal file

View file

Binary file not shown.

BIN

dataset/states/03/03f40f412d55524661037fcb6c9199eeb8a307266912c8f188d5627a5dac1f45.json.gz Normal file

View file

Binary file not shown.

BIN

dataset/states/04/04707526733b677a1d3afa088d8c78a600704e9f46f49bac6eff630018e08a4e.json.gz Normal file

View file

Binary file not shown.

BIN

dataset/states/04/04750fffe560ee5db9f0318db510cd8001202a204a9f540147f5ce49cfa589c5.json.gz Normal file

View file

Binary file not shown.

BIN

dataset/states/04/047e27c79549e82fd18c2a177dfab07b036938997397495b906de014b690d02e.json.gz Normal file

View file

Binary file not shown.

BIN

dataset/states/05/0564ed044c883ca7c6651bf0748fed5cfaf630e92d05979627089a31e058381e.json.gz Normal file

View file

Binary file not shown.

BIN

dataset/states/05/05c5ebb5751051037a323f649f7c73d7be99f0c654f2d1ea498ed70e9c73998b.json.gz Normal file

View file

Binary file not shown.

BIN

dataset/states/06/0618279b1a19a34c32469d654d00386fa904e76fc1cf8ee0943d47469d257a49.json.gz Normal file

View file

Binary file not shown.

BIN

dataset/states/06/06e3dea28630300b2cd5d79d8c030ce1a80c571328c90c6a7d4476e3dfe430d3.json.gz Normal file

View file

Binary file not shown.

BIN

dataset/states/07/0744793ca2ea455b3c663ddfb5e8f3d1a9df58c681b62def6fd270761b3b9c26.json.gz Normal file

View file

Binary file not shown.

BIN

dataset/states/07/079914414866badb0b2c1068f14171578a32be5df3e60acfdbb3ff758706d23d.json.gz Normal file

View file

Binary file not shown.

BIN

dataset/states/08/0841afec40ba29ed7038f45fdf588589b5e6088a1dff260087fe5022ddf7abdc.json.gz Normal file

View file

Binary file not shown.

BIN

dataset/states/08/08a93626224eaf242c22322f2bdd4b109c9543917b0413a467e5362818865149.json.gz Normal file

View file

Binary file not shown.

BIN

dataset/states/09/09037c2ef635ea0b352586735f1b7916efdc205e51e77872a49c73f8cb876a62.json.gz Normal file

View file

Binary file not shown.

BIN

dataset/states/09/090e0de8334ca270fbb01d5e5b33faef35a037e136300aa8efb09b3f4210f86f.json.gz Normal file

View file

Binary file not shown.

BIN

dataset/states/0b/0ba60536ab2c0e1d87cc637eeecebaee34ee19eeabdce4b05f46dfeb42cc4453.json.gz Normal file

View file

Binary file not shown.

BIN

dataset/states/0b/0be57966bdc7dafafb6ff2b13f705b44bd998a0001eb61a076baf53a92fa0e94.json.gz Normal file

View file

Binary file not shown.

BIN

dataset/states/0c/0c5aafea678b985ba378e7a4a8f285fb0f41eef6725b2df527cbb8db369a0b99.json.gz Normal file

View file

Binary file not shown.

BIN

dataset/states/0d/0d221d37d1fe6f5c6f13ef94e5e838b27882be183e43451be9b3dc7d3f8c45a7.json.gz Normal file

View file

Binary file not shown.

BIN

dataset/states/0e/0e8f23c1c2dd2831de34c93b3249dc5b7add5f23e6bb04ec070de6195c2eb6d2.json.gz Normal file

View file

Binary file not shown.

BIN

dataset/states/0e/0ee00d94f80cda96ae97db7cb0e1d94c3421140a6d1a606520e698e1a226dd0d.json.gz Normal file

View file

Binary file not shown.

BIN

dataset/states/0f/0facc1e7c1f90ef661c32fa2bff9222e10a8a6771154853d8bc6ed0c1397685f.json.gz Normal file

View file

Binary file not shown.

BIN

dataset/states/11/11b5b887be1005550d6b933edccaee0becc2a7ff99a21178effa598e10521fa2.json.gz Normal file

View file

Binary file not shown.

BIN

dataset/states/11/11ec0d9b985ded283430ece814c22d394ac8aedd70b2d92dcf05f87f48fa09dd.json.gz Normal file

View file

Binary file not shown.

BIN

dataset/states/12/12744263a32e9303683d6e1554b08148379fbe817f433487b4c3b0d9c9198a6b.json.gz Normal file

View file

Binary file not shown.

BIN

dataset/states/12/1283b78b1b561bd6feadcc8a76a4c8ae33a0941fb78565191543768a06cc5984.json.gz Normal file

View file

Binary file not shown.

BIN

dataset/states/12/12aed92107a9e6086aa4b1775205ce4444c5af897ebb571628c9151cd8ba8874.json.gz Normal file

View file

Binary file not shown.

BIN

dataset/states/13/13653794bb48f047bd171583fa3c54d339bca521ef9fb963fc13fcc74ca07f71.json.gz Normal file

View file

Binary file not shown.

BIN

dataset/states/15/1515ee31da69979fbdb0a52bdc5187489ca4cb112a59ef85fc7f7e67ed350851.json.gz Normal file

View file

Binary file not shown.

BIN

dataset/states/16/169bec6212986bffae3aefff178be6a7353db7b32b06fcbbc5d9ae059e54ea4f.json.gz Normal file

View file

Binary file not shown.

BIN

dataset/states/17/17508685f08e5e372629ac2c93fef8d0e7faec9c8323dd1e19c9470230a08c41.json.gz Normal file

View file

Binary file not shown.

BIN

dataset/states/19/198104530cca28553293a6d21f6ef9cac7c2cc792708258be6516895b27320d1.json.gz Normal file

View file

Binary file not shown.

BIN

dataset/states/19/19ba4d2f0dfa9593f4d38a97a4fe061e0d167701a88bb88106d3c38f85d8afdc.json.gz Normal file

View file

Binary file not shown.

BIN

dataset/states/1a/1a8cd235ca7095baeb3f1f8d784914b636ee8adc2e6408cb581c398c7d917327.json.gz Normal file

View file

Binary file not shown.

BIN

dataset/states/1b/1b1df419f1ce4bf97d2dff5a6c08a124c1dcc1fad1dfcf9cf001971abef9630a.json.gz Normal file

View file

Binary file not shown.

BIN

dataset/states/1b/1badf8302cb9c3888e3e7cde3c7d7629b97058bf4f808197134207971aa72378.json.gz Normal file

View file

Binary file not shown.

BIN

dataset/states/1b/1bb11dfc37cb7e8de86914a05a1855402e56e430fa3e11f9f8716f72992c83ab.json.gz Normal file

View file

Binary file not shown.

BIN

dataset/states/1b/1bcfc1c43ea26a8a3a54093652273a2ceea9c93db083cc292d8d6e98d12d9d18.json.gz Normal file

View file

Binary file not shown.

BIN

dataset/states/1c/1c78e6936a77a7d70fb36bf764a163339183e667acddc565fde4f86f3cd8f141.json.gz Normal file

View file

Binary file not shown.

BIN

dataset/states/1c/1cd4fc0ff1556760263896655b38de76827010b05a33676cc0ec42a1687c2d58.json.gz Normal file

View file

Binary file not shown.

BIN

dataset/states/1d/1dfa012a80eda7dc3d6a6b7d3b5a46c611f28093773b0bd86393022ccf004fbf.json.gz Normal file

View file

Binary file not shown.

BIN

dataset/states/1e/1e50f85e9b1e22380b91378c0414acc9e73504e43d1e76369c042471e8eacbae.json.gz Normal file

View file

Binary file not shown.

BIN

dataset/states/1e/1e699def06dcbf11414e01b2764db430dd94ceed4c6f751968adaad17f5c3b30.json.gz Normal file

View file

Binary file not shown.

BIN

dataset/states/1f/1fae2fe49b55a068cf992393322e5307214f0c69c8b2b3234c45d99a0e61615a.json.gz Normal file

View file

Binary file not shown.

BIN

dataset/states/20/206f0886f1df26dd12826d0e1dcc6cd1f842087a7b05db9879f6b1023bc1cad9.json.gz Normal file

View file

Binary file not shown.

BIN

dataset/states/20/20e4e65537819dc36a0a5cc9e8763dad78494a7798299eb97fa0d2a0f569c8d8.json.gz Normal file

View file

Binary file not shown.

BIN

dataset/states/21/210502abca2bceb8db85b273f1c731fe770463915cada734d645f1e318dee373.json.gz Normal file

View file

Binary file not shown.

BIN

dataset/states/21/210f31a1fecbd9336a6ea7f2dd4f6e21c1fd175219bb61d37939f1d4dbf690bd.json.gz Normal file

View file

Binary file not shown.

BIN

dataset/states/21/2197517adfc21f92c116380c2841ca86aba23b23877f60bb13eede7dc089ac15.json.gz Normal file

View file

Binary file not shown.

BIN

dataset/states/23/23d18242e5da21723268711a92de09cb51ae050b8a4239035f8c66ee441d44b9.json.gz Normal file

View file

Binary file not shown.

BIN

dataset/states/23/23d8c405654b9eadb3aeee9d1d871e790a1485088e4e5460fa58008e9cbb4268.json.gz Normal file

View file

Binary file not shown.

BIN

dataset/states/24/24ce388c30e0fcd5e6144d435d383ff9f0b396403a61bd9b5d1874ecdb7fcbd2.json.gz Normal file

View file

Binary file not shown.

BIN

dataset/states/25/2541867d3ddc7893ebece35db22d6c243aa20457cf979cbea50194805abcc066.json.gz Normal file

View file

Binary file not shown.

BIN

dataset/states/27/270e8b6b288f7d997244e957bf60e80c6f6e71e681d6929db2f946aa6bb95b93.json.gz Normal file

View file

Binary file not shown.

BIN

dataset/states/27/27f32f9969292d39a475f3ccd3f2a289cbdc1b720e63293b36aa07dfbeed7962.json.gz Normal file

View file

Binary file not shown.

BIN

dataset/states/28/283a2d0e992c66a43588ce91d89f54c40edc13d9c5333e67dbb2a5481d8803ca.json.gz Normal file

View file

Binary file not shown.

BIN

dataset/states/2a/2a2752ac81d1fd997adcf85b69957a53bcc703e69fd666f74935178a9d23a105.json.gz Normal file

View file

Binary file not shown.

BIN

dataset/states/2a/2ac4d9ecc37eb34490f4999299bec43dd7865046d0caa31f19225559503b8039.json.gz Normal file

View file

Binary file not shown.

BIN

dataset/states/2a/2adc220a36d68c945e2a35ccc3e96915830d6c72b9ed36d4d6b664634d1f848f.json.gz Normal file

View file

Binary file not shown.

BIN

dataset/states/2b/2ba0fd15721aad4e115af33adf7efb41ec54aa8a1301b623da29e03e16833818.json.gz Normal file

View file

Binary file not shown.

BIN

dataset/states/2b/2bb595c5ab4aa0ab694d4b50e7ac55ba7f4474f32907c9ea57375d29fa9404bc.json.gz Normal file

View file

Binary file not shown.

BIN

dataset/states/2b/2beec997692aef39890088d84563dd01c29f4f841889ed0137e5ee5952f3e6db.json.gz Normal file

View file

Binary file not shown.

BIN

dataset/states/2c/2c8d4bc969cce6df76bef7f2b99b5abf0234e7771af0767ec8c0bf1e94dcf84d.json.gz Normal file

View file

Binary file not shown.

BIN

dataset/states/2d/2d1372656a1638dfe5b98416443f62538a0edfe4ca5a49eee1e309e546964302.json.gz Normal file

View file

Binary file not shown.

BIN

dataset/states/2d/2da562830dbbff1e29e83db65c1cd7027aca0f6f94833226c298d50776b033fa.json.gz Normal file

View file

Binary file not shown.

BIN

dataset/states/2d/2df8292469db15b742ae1c6a87546033cad4f35a046e168a09ad3d721f1b25ad.json.gz Normal file

View file

Binary file not shown.

BIN

dataset/states/2e/2ea2fdb9acca9608fbd930241de7ec9da66cfc768de501d1fcf66a2f3ef1f873.json.gz Normal file

View file

Binary file not shown.

BIN

dataset/states/2e/2ec44b9da79079a39b5f29b784ce1430b7e43f45866a2c5f455fb31430a5d636.json.gz Normal file

View file

Binary file not shown.

BIN

dataset/states/2f/2f1a10f15915993eeca17804019ec18e448bb0391beeec5953a76d92fd0fa950.json.gz Normal file

View file

Binary file not shown.

BIN

dataset/states/2f/2f7cc7766de19c9600d776a2bce7ddada8f72549ccc65ed5079ff273560e8bcc.json.gz Normal file

View file

Binary file not shown.

BIN

dataset/states/2f/2f95393dc29f49569a42533f3c868c384fec211f3d635f13184b9eb5d9e9e025.json.gz Normal file

View file

Binary file not shown.

BIN

dataset/states/30/308b40127fbe1ebc3b7f0854fb864ac4d131a02518ed216aa0dd55891068f563.json.gz Normal file

View file

Binary file not shown.

BIN

dataset/states/32/32151abe6a1de77eae0549b9f2f4ec1ec53ce69212a960817b60f8e9dcd399f2.json.gz Normal file

View file

Binary file not shown.

BIN

dataset/states/33/3307d6df9ada4686832708ef5a3e434c23a6687314cb471e907a79634bbc10c6.json.gz Normal file

View file

Binary file not shown.

BIN

dataset/states/33/33ba9931bd79fe830883f7bc1e510df17e0faadf9e0b7307be6e349d1d8cf59c.json.gz Normal file

View file

Binary file not shown.

BIN

dataset/states/33/33ea0178152908ecefc29bbe894baaeb0c2afdcde75fa8e0080a9b427c5a4cea.json.gz Normal file

View file

Binary file not shown.

BIN

dataset/states/34/34bcf5d2feae26ebac0feadd0c6cd53fb46d1e1249eb9a3a9d7db1a33ddb16ca.json.gz Normal file

View file

Binary file not shown.

BIN

dataset/states/37/37c830cb4be5064422e3d56593f789ca9627785a8555544b76229ff12dbf310d.json.gz Normal file

View file

Binary file not shown.

BIN

dataset/states/38/384786c4249adeb55994db574d5c9a67fd429c09002cbcb45d640ffe547fcc25.json.gz Normal file

View file

Binary file not shown.

BIN

dataset/states/38/38a008d7dc635dd2b46c8b9b3e64f0679cba025cad1c383c22bbe9ceb73b3e05.json.gz Normal file

View file

Binary file not shown.

BIN

dataset/states/39/397a0108ccd4d4abfca066a4fbd07529e78157c19677a0233e045a508a22ddf7.json.gz Normal file

View file

Binary file not shown.

BIN

dataset/states/39/39d8c2eb835fc8318ac7202b96dde71ebb395cf3593db308acaebdbd3a8a5221.json.gz Normal file

View file

Binary file not shown.

BIN

dataset/states/3b/3b9ca1f379d73529798970457d2456e77f7102364d2c1fdcd8132b97179458ce.json.gz Normal file

View file

Binary file not shown.

BIN

dataset/states/3d/3d54f332ee132ea115f9d17d21b084e8606caf3b76080c36b0c754b1f4de2310.json.gz Normal file

View file

Binary file not shown.

BIN

dataset/states/3d/3ddedefb6c723ac8773daf9fb76b76a25d685a809731857b6e496b0b90b0dd57.json.gz Normal file

View file

Binary file not shown.

BIN

dataset/states/3d/3dfa6e2232178965e74f6e99c4aab2f0181c6063c0fa9dd26c07fe9daf485498.json.gz Normal file

View file

Binary file not shown.

BIN

dataset/states/3e/3e49a6f325a0d0f15402f244342e0b1b036ea3889ceec9645c9c73dbf3e51c2e.json.gz Normal file

View file

Binary file not shown.

BIN

dataset/states/3e/3e92d5f17003c90ee22751e7d530c3a388e8b7fb5bcf16d41e2bf2efd30d54d9.json.gz Normal file

View file

Binary file not shown.

BIN

dataset/states/3f/3f6a94bab07e9554f78089fda870287dc638a52ac1eadb1fb508dcf613213403.json.gz Normal file

View file

Binary file not shown.

BIN

dataset/states/3f/3f74b486d2e2bcf50182158dfe521c32b4fc3254faf194753fbfe9b004069a4a.json.gz Normal file

View file

Binary file not shown.

BIN

dataset/states/41/4110fa6fe66e6e9ee8c30bc6c83f82fb944b3ebe7c48b13f7f731aaae2fa4f31.json.gz Normal file

View file

Binary file not shown.

BIN

dataset/states/42/42a5b25269ecfbe9e572516f2345388b584c66c8b18d11f9ecbd9fbb7794738a.json.gz Normal file

View file

Binary file not shown.

BIN

dataset/states/44/440860eb0de1b5525bdc1348f0cb8d97ae59a727a8420a3d4f2b6e8b1d752812.json.gz Normal file

View file

Binary file not shown.

BIN

dataset/states/44/4418580d215bef7dddc8865d645fbdd17ff7f5adffbdf9bd92a7cbc58adf0f72.json.gz Normal file

View file

Binary file not shown.

BIN

dataset/states/44/442e03b19d5df7a19d341cef6a6a445733a80cb2c7f613c54cf8d804c83197cb.json.gz Normal file

View file

Binary file not shown.

BIN

dataset/states/45/453bb0c4ffe0599771a4933399e9e21663bf29b340d9768265091f2cddccfeb9.json.gz Normal file

View file

Binary file not shown.

Some files were not shown because too many files have changed in this diff Show more