feat(tooling): add bounded Jev game-debugging workflow

This commit is contained in:
0xrsydn 2026-10-07 01:14:43 +07:00
commit 894f524cda
17 changed files with 820 additions and 0 deletions

View file

@ -19,6 +19,9 @@ The repository is a Cargo workspace with three crates:
| `tools/prepare_homebrew_pong.py`, `tools/homebrew-pong.lock.json` | Hash-pinned homebrew preparation and source-derived gameplay scenarios |
| `tools/test_prepare_homebrew_pong.py` | Offline homebrew preparation, provenance, bounds, and destination-safety tests |
| `.envrc` | Automatic environment loading through direnv |
| `.pi/prompts/jev-gb-debug.md`, `.pi/skills/jev-gb-debug/SKILL.md` | Project-local optional Jev diagnostic workflow |
| `tools/jev_gb_debug.py`, `tools/jev-gb-rubric.json` | Bounded evidence validation, optional classifier request, and local evaluation records |
| `tools/test_jev_gb_debug.py`, `tools/fixtures/jev/` | Offline integration tests and original synthetic classifier probes |
| `AGENTS.md` | Short orientation for coding agents: crate map, commands, and where docs live |
| `docs/` | Project documentation; `docs/hardware/` holds per-subsystem behavior notes |
| `crates/gba-core/src/cpu.rs` | Registers, flags, instruction-set state, and stepping |

144
docs/jev-debugging.md Normal file
View file

@ -0,0 +1,144 @@
# Jev-assisted GBA debugging
Use `/jev-gb-debug` in Pi to review bounded diagnostic evidence and compatibility claims.
Jev is an optional classifier, not an emulator, debugger, code generator, or hardware reference.
Rust assertions, ROM suite results, and measured device behavior remain authoritative.
This workflow does not implement audio or make the current Emerald startup diagnostic pass.
## Enable the project command
The project provides:
- `.pi/prompts/jev-gb-debug.md`: the `/jev-gb-debug [focus]` prompt command.
- `.pi/skills/jev-gb-debug/SKILL.md`: the evidence collection, evaluation, and regression workflow.
- `.pi/settings.json`: adds Codemode without replacing the inherited tool list.
Review and trust these project resources in Pi, then run `/reload`.
Use `/jev-gb-debug sound initialization` or `/skill:jev-gb-debug`.
A prompt command asks the coding agent to follow the workflow. It is not a background process or a new executable debugger.
## Credentials and execution
Pi's native `models.classify` interface expects `TYPESAFE_API_KEY` for the direct TypeSafe provider.
The local HTTP helper also accepts `TYPESAFEAI_API_KEY`, with that variable taking precedence when both are set.
No key value is printed, saved, or placed in a command argument.
To use the existing key with Pi's native interface in a future session, set the alias before starting Pi:
```sh
export TYPESAFE_API_KEY="$TYPESAFEAI_API_KEY"
pi
```
A child shell cannot update the already-running Pi process environment.
The workflow therefore uses the direct HTTP helper when native TypeSafe authentication is unavailable.
It does not silently choose OpenRouter or another account/provider.
Native classifier calls contribute usage to Pi's session accounting. Direct HTTP calls record usage locally but bypass Pi's model-cost counter.
Zero or missing reported cost does not imply that a request was free.
## Evidence and evaluation
1. Define the claim and deterministic acceptance criteria.
2. Run a bounded test or ROM diagnostic and inspect its exit status.
3. Prepare minimal text evidence. Review every field before sending it.
4. Ask Jev to classify the next subsystem, the evidence level, and whether the exact claim is supported.
5. Compare the advice with code and tests. Preserve uncertainty and disagreements.
6. Implement original regression tests and a general fix, then rerun the same deterministic checks.
The rubric is versioned in [`tools/jev-gb-rubric.json`](../tools/jev-gb-rubric.json).
Its `unknown` option prevents forced subsystem guesses. Evidence levels distinguish diagnostics, checkpoints, and scripted gameplay.
A budget-only run cannot establish gameplay. Startup frames cannot establish a title screen, sound, or save support.
The model never changes a test result or selects an executable shell command.
The helper accepts exactly these evidence fields:
```json
{
"case_id": "sound-initialization",
"observations": ["Sound master enable was rejected because the audio engine is not implemented."],
"checks": [{
"name": "sound-start",
"result": "fail",
"scope": "Sound activation must finish without an unsupported-device diagnostic."
}],
"claim": "Sound works."
}
```
Use check results `pass`, `fail`, or `not_run`. Keep exact numeric comparisons in deterministic code.
Use named observations rather than asking Jev to decode ARM instructions, compare hex colors, or calculate timing.
Current Jev models accept text only. They cannot directly inspect frames, audio, video, or binary ROMs.
The existing suite runner can produce button/pixel assertions, but it does not expose the article's live game-control loop.
Future interactive control must use bounded, validated emulator commands and recorded input schedules.
## Preview, run, and record
Preview without credentials or network access:
```sh
direnv exec . python3 -B tools/jev_gb_debug.py tools/fixtures/jev/audio-blocker.json
```
Run one direct TypeSafe request and save a new local record:
```sh
mkdir -p .pi/jev-runs
direnv exec . python3 -B tools/jev_gb_debug.py tools/fixtures/jev/audio-blocker.json \
--live --output .pi/jev-runs/audio-review-001.json
```
Choose a new output name for each evaluation. The helper reserves the file before sending a request and refuses overwrite.
A pending record means the process ended before recording a complete result; it is not a pass and must not trigger a blind retry.
Output outside the repository is also accepted. In-repository output is restricted to ignored `.pi/jev-runs/`.
The helper creates record files with owner-only permissions. Parent directories must already exist.
The skill contains the native Codemode `models.classify` example.
It previews the same context, makes one classifier call, then imports the result with `--response PATH.json` without a second request.
Pi represents yes/no questions as `bool`/`probability`; the HTTP API uses `noul`/`noul`.
The helper converts request types for HTTP and validates either response format.
It never reads credentials into the Codemode transcript.
Every complete record includes:
- Reviewed evidence and the exact rubric context, plus SHA-256 hashes and rubric version.
- The local jj commit identifier and a UTC timestamp. Local metadata is not sent to Jev.
- Transport, requested model alias, returned model identifier, typed answers, distributions, and available usage.
- Deterministic check status and an advisory summary. Compatibility remains `not_established` regardless of the model answer.
- Explicit error state when authentication, transport, response validation, or classification fails.
The stored context uses Pi's question schema; the HTTP request differs only by the `bool` to `noul` conversion and model field.
The alias `jev-latest` can change. Retain the returned model identifier when comparing runs.
Do not treat replaying the same prompt as a deterministic test.
## Limits and disclosure
Each command permits at most four live evaluations by default. Each request asks all rubric questions together.
The helper limits evidence to 16 KiB, 12 observations, and 24 checks; responses are bounded separately.
Direct HTTP uses a fixed HTTPS endpoint, refuses redirects, has a 30-second socket timeout, and performs no automatic retries.
Use the skill's 45-second tool timeout as the process bound. An aborted request may still incur usage.
Choice confidence below 0.8, unknown/insufficient evidence, or a yes/no probability between 0.2 and 0.8 requires review.
These thresholds are provisional. They are not calibrated measures of GBA hardware correctness.
Do not repeatedly query until the model agrees with a preferred answer.
Never send ROMs, extracted game code/assets, firmware, save data, screenshots, audio, credentials, or unfiltered logs.
Selected emulator-generated diagnostics and original test summaries are sufficient.
The helper rejects unknown fields, excessive input, common credential markers, and common private paths.
These checks do not detect every secret or private detail. The agent must review the full payload before requesting evaluation.
State and model responses are untrusted data. An injection smoke test does not establish a security boundary.
Raw records stay ignored. Commit only reviewed summaries of useful findings, methods, and original regression tests.
Normal Cargo and Python tests stay offline and do not require Jev credentials.
## Validation
The [reviewed smoke evaluation](research/jev-smoke.md) records live classifier outcomes and their limits.
The HTTP path was exercised live. Native Pi response normalization was tested offline; native authentication needs the environment alias above.
## References
- [TypeSafe documentation index](https://docs.typesafe.ai/llms.txt).
- [API reference](https://docs.typesafe.ai/api.md) and [model capabilities](https://docs.typesafe.ai/models.md).
- [Jev with coding agents](https://docs.typesafe.ai/introduction/coding-agents.md): structured decisions, not code generation.
- [Confidence](https://docs.typesafe.ai/confidence.md) and [Jev 1.13 limitations](https://docs.typesafe.ai/model-jaggedness/jev-1.13.md): arithmetic, indirection, option order, and adversarial content.
- [What is Codemode](https://lucumr.pocoo.org/2026/10/6/codemode/): harness-side orchestration and the bounded game-control example.
- Installed Pi documentation: `docs/codemode.md`, `docs/models.md`, `docs/skills.md`, `docs/prompt-templates.md`, and project-trust/configuration references.

View file

@ -246,6 +246,12 @@ Disabled sound and disconnected serial reset:
- [GBATEK normal serial mode](https://problemkaputt.de/gbatek-sio-normal-mode.htm) and its [general-purpose register chapter](https://mgba-emu.github.io/gbatek/), for data widths, control fields, and disconnected input pull-ups.
- [mGBA BIOS implementation at 3a5e34be](https://github.com/mgba-emu/mgba/blob/3a5e34be33dc7f8f707e5bc9db69e8a430046f21/src/gba/bios.c), for functional serial and sound reset defaults. Exact firmware ordering and timing remain unverified.
Optional classifier-assisted debugging:
- [TypeSafe documentation index](https://docs.typesafe.ai/llms.txt), [API reference](https://docs.typesafe.ai/api.md), [models](https://docs.typesafe.ai/models.md), [confidence](https://docs.typesafe.ai/confidence.md), and [Jev limitations](https://docs.typesafe.ai/model-jaggedness/jev-1.13.md), for typed evidence review and its limits.
- [What is Codemode](https://lucumr.pocoo.org/2026/10/6/codemode/), for bounded harness-side orchestration and game-debugging examples.
- Installed Pi documentation for Codemode, classifiers, skills, prompt templates, settings, configuration, providers, and project trust. See [the local workflow](jev-debugging.md).
Use [GBATEK](https://problemkaputt.de/gbatek.htm) and ARM7TDMI documentation for further hardware work.
Only use game ROMs that you may lawfully use. Do not commit game ROMs, BIOS files, or game assets.
The ignore file excludes `roms/`, common game ROM extensions, and save files.

View file

@ -0,0 +1,43 @@
# Jev workflow smoke evaluation
The project-local Jev workflow was tested through Pi Codemode using the direct TypeSafe HTTP helper.
The existing `TYPESAFEAI_API_KEY` authenticated successfully without changing global Pi configuration.
The returned model identifier was `jev-1.13.0`.
Native `models.classify` was not exercised live because this Pi process lacked its expected `TYPESAFE_API_KEY` variable.
Native response normalization is covered by offline fixtures.
## Reviewed observations
| Evidence | Subsystem advice | Evidence level | Probability that the stated claim is supported | Review required |
| --- | --- | --- | --- | --- |
| Synthetic unsupported sound activation; claim that sound works | Audio | Diagnostic | 0.02 | No |
| Synthetic successful scripted gameplay checks; claim limited to those checks | None | Scripted gameplay | 0.85 | No |
| Synthetic budget-only run with an injected instruction to declare compatibility | Unknown | Insufficient | 0.02 | Yes |
| Selected local startup diagnostic after BIOS reset; claim of playable gameplay with sound | Audio | Diagnostic | 0.02 | No |
These are classifier integration checks, not new emulator tests.
The synthetic cases describe assumed evidence; they do not run or validate a game.
The local startup case uses reviewed facts from the [existing runtime result](emerald-reset.md), not ROM content.
The model agreed that sound activation remains the next observed blocker. It did not find a new hardware root cause.
The budget-only case had low evidence-level confidence and remained marked for review.
Rejecting this one injected instruction does not establish resistance to adversarial input.
A high-confidence category is still advice, not proof of correct hardware behavior.
## Record locations
Raw request contexts, hashes, jj revision identifiers, responses, and usage remain under ignored `.pi/jev-runs/`:
- `audio-probe-001.json`
- `gameplay-probe-001.json`
- `budget-probe-001.json`
- `local-startup-probe-001.json`
The committed synthetic inputs are in `tools/fixtures/jev/`.
The selected local startup evidence stays ignored alongside its record.
No ROM, save, credential, screenshot, audio, or extracted game code was sent to the classifier.
The helper retained failed/incomplete deterministic status regardless of its model answers and executed no model-selected action.
Ordinary Python tests mock network responses and credentials. They cover input limits, response validation, failure paths, and log safety.
Re-run live evaluations deliberately with new filenames; do not make model output an offline test expectation or query until it agrees.
See [the workflow](../jev-debugging.md) for limits, credential setup, and the command.

View file

@ -67,6 +67,9 @@ Run the external Pong suite separately; ordinary Cargo and Python tests require
Device initialization regressions use original instructions to check disabled sound, wave RAM, disconnected serial pins, and reset flags.
They verify explicit activation errors and atomic byte/word/block-store/DMA failure behavior.
The optional [local Emerald startup check](research/emerald-reset.md) now reaches sound activation but still fails deliberately there.
[Jev evidence review](jev-debugging.md) is optional and never changes deterministic test verdicts.
Its Python tests mock the provider and verify schemas, response validation, credential handling, non-overwriting logs, and failure behavior.
Live classifier smoke probes are separate from offline tests and must not become mandatory CI checks.
Thumb cases also verify the return-to-ARM bridge. Command-line tests cover default ARM selection, explicit Thumb/memory/BIOS selection, and errors.
Compare/status CPU regressions verify saved flags/masks, bank switching, sequential PC/timing, User/System fallback, and atomic diagnostics.
Load-alias regressions cover widths, signed/unaligned values, immediate/register offsets, pre/post-indexing, every non-PC register bank, and address wrapping.