feat(tooling): add bounded Jev game-debugging workflow
This commit is contained in:
parent
d6a47a3e12
commit
894f524cda
17 changed files with 820 additions and 0 deletions
|
|
@ -19,6 +19,9 @@ The repository is a Cargo workspace with three crates:
|
|||
| `tools/prepare_homebrew_pong.py`, `tools/homebrew-pong.lock.json` | Hash-pinned homebrew preparation and source-derived gameplay scenarios |
|
||||
| `tools/test_prepare_homebrew_pong.py` | Offline homebrew preparation, provenance, bounds, and destination-safety tests |
|
||||
| `.envrc` | Automatic environment loading through direnv |
|
||||
| `.pi/prompts/jev-gb-debug.md`, `.pi/skills/jev-gb-debug/SKILL.md` | Project-local optional Jev diagnostic workflow |
|
||||
| `tools/jev_gb_debug.py`, `tools/jev-gb-rubric.json` | Bounded evidence validation, optional classifier request, and local evaluation records |
|
||||
| `tools/test_jev_gb_debug.py`, `tools/fixtures/jev/` | Offline integration tests and original synthetic classifier probes |
|
||||
| `AGENTS.md` | Short orientation for coding agents: crate map, commands, and where docs live |
|
||||
| `docs/` | Project documentation; `docs/hardware/` holds per-subsystem behavior notes |
|
||||
| `crates/gba-core/src/cpu.rs` | Registers, flags, instruction-set state, and stepping |
|
||||
|
|
|
|||
144
docs/jev-debugging.md
Normal file
144
docs/jev-debugging.md
Normal file
|
|
@ -0,0 +1,144 @@
|
|||
# Jev-assisted GBA debugging
|
||||
|
||||
Use `/jev-gb-debug` in Pi to review bounded diagnostic evidence and compatibility claims.
|
||||
Jev is an optional classifier, not an emulator, debugger, code generator, or hardware reference.
|
||||
Rust assertions, ROM suite results, and measured device behavior remain authoritative.
|
||||
This workflow does not implement audio or make the current Emerald startup diagnostic pass.
|
||||
|
||||
## Enable the project command
|
||||
|
||||
The project provides:
|
||||
|
||||
- `.pi/prompts/jev-gb-debug.md`: the `/jev-gb-debug [focus]` prompt command.
|
||||
- `.pi/skills/jev-gb-debug/SKILL.md`: the evidence collection, evaluation, and regression workflow.
|
||||
- `.pi/settings.json`: adds Codemode without replacing the inherited tool list.
|
||||
|
||||
Review and trust these project resources in Pi, then run `/reload`.
|
||||
Use `/jev-gb-debug sound initialization` or `/skill:jev-gb-debug`.
|
||||
A prompt command asks the coding agent to follow the workflow. It is not a background process or a new executable debugger.
|
||||
|
||||
## Credentials and execution
|
||||
|
||||
Pi's native `models.classify` interface expects `TYPESAFE_API_KEY` for the direct TypeSafe provider.
|
||||
The local HTTP helper also accepts `TYPESAFEAI_API_KEY`, with that variable taking precedence when both are set.
|
||||
No key value is printed, saved, or placed in a command argument.
|
||||
To use the existing key with Pi's native interface in a future session, set the alias before starting Pi:
|
||||
|
||||
```sh
|
||||
export TYPESAFE_API_KEY="$TYPESAFEAI_API_KEY"
|
||||
pi
|
||||
```
|
||||
|
||||
A child shell cannot update the already-running Pi process environment.
|
||||
The workflow therefore uses the direct HTTP helper when native TypeSafe authentication is unavailable.
|
||||
It does not silently choose OpenRouter or another account/provider.
|
||||
Native classifier calls contribute usage to Pi's session accounting. Direct HTTP calls record usage locally but bypass Pi's model-cost counter.
|
||||
Zero or missing reported cost does not imply that a request was free.
|
||||
|
||||
## Evidence and evaluation
|
||||
|
||||
1. Define the claim and deterministic acceptance criteria.
|
||||
2. Run a bounded test or ROM diagnostic and inspect its exit status.
|
||||
3. Prepare minimal text evidence. Review every field before sending it.
|
||||
4. Ask Jev to classify the next subsystem, the evidence level, and whether the exact claim is supported.
|
||||
5. Compare the advice with code and tests. Preserve uncertainty and disagreements.
|
||||
6. Implement original regression tests and a general fix, then rerun the same deterministic checks.
|
||||
|
||||
The rubric is versioned in [`tools/jev-gb-rubric.json`](../tools/jev-gb-rubric.json).
|
||||
Its `unknown` option prevents forced subsystem guesses. Evidence levels distinguish diagnostics, checkpoints, and scripted gameplay.
|
||||
A budget-only run cannot establish gameplay. Startup frames cannot establish a title screen, sound, or save support.
|
||||
The model never changes a test result or selects an executable shell command.
|
||||
|
||||
The helper accepts exactly these evidence fields:
|
||||
|
||||
```json
|
||||
{
|
||||
"case_id": "sound-initialization",
|
||||
"observations": ["Sound master enable was rejected because the audio engine is not implemented."],
|
||||
"checks": [{
|
||||
"name": "sound-start",
|
||||
"result": "fail",
|
||||
"scope": "Sound activation must finish without an unsupported-device diagnostic."
|
||||
}],
|
||||
"claim": "Sound works."
|
||||
}
|
||||
```
|
||||
|
||||
Use check results `pass`, `fail`, or `not_run`. Keep exact numeric comparisons in deterministic code.
|
||||
Use named observations rather than asking Jev to decode ARM instructions, compare hex colors, or calculate timing.
|
||||
Current Jev models accept text only. They cannot directly inspect frames, audio, video, or binary ROMs.
|
||||
The existing suite runner can produce button/pixel assertions, but it does not expose the article's live game-control loop.
|
||||
Future interactive control must use bounded, validated emulator commands and recorded input schedules.
|
||||
|
||||
## Preview, run, and record
|
||||
|
||||
Preview without credentials or network access:
|
||||
|
||||
```sh
|
||||
direnv exec . python3 -B tools/jev_gb_debug.py tools/fixtures/jev/audio-blocker.json
|
||||
```
|
||||
|
||||
Run one direct TypeSafe request and save a new local record:
|
||||
|
||||
```sh
|
||||
mkdir -p .pi/jev-runs
|
||||
direnv exec . python3 -B tools/jev_gb_debug.py tools/fixtures/jev/audio-blocker.json \
|
||||
--live --output .pi/jev-runs/audio-review-001.json
|
||||
```
|
||||
|
||||
Choose a new output name for each evaluation. The helper reserves the file before sending a request and refuses overwrite.
|
||||
A pending record means the process ended before recording a complete result; it is not a pass and must not trigger a blind retry.
|
||||
Output outside the repository is also accepted. In-repository output is restricted to ignored `.pi/jev-runs/`.
|
||||
The helper creates record files with owner-only permissions. Parent directories must already exist.
|
||||
|
||||
The skill contains the native Codemode `models.classify` example.
|
||||
It previews the same context, makes one classifier call, then imports the result with `--response PATH.json` without a second request.
|
||||
Pi represents yes/no questions as `bool`/`probability`; the HTTP API uses `noul`/`noul`.
|
||||
The helper converts request types for HTTP and validates either response format.
|
||||
It never reads credentials into the Codemode transcript.
|
||||
|
||||
Every complete record includes:
|
||||
|
||||
- Reviewed evidence and the exact rubric context, plus SHA-256 hashes and rubric version.
|
||||
- The local jj commit identifier and a UTC timestamp. Local metadata is not sent to Jev.
|
||||
- Transport, requested model alias, returned model identifier, typed answers, distributions, and available usage.
|
||||
- Deterministic check status and an advisory summary. Compatibility remains `not_established` regardless of the model answer.
|
||||
- Explicit error state when authentication, transport, response validation, or classification fails.
|
||||
|
||||
The stored context uses Pi's question schema; the HTTP request differs only by the `bool` to `noul` conversion and model field.
|
||||
The alias `jev-latest` can change. Retain the returned model identifier when comparing runs.
|
||||
Do not treat replaying the same prompt as a deterministic test.
|
||||
|
||||
## Limits and disclosure
|
||||
|
||||
Each command permits at most four live evaluations by default. Each request asks all rubric questions together.
|
||||
The helper limits evidence to 16 KiB, 12 observations, and 24 checks; responses are bounded separately.
|
||||
Direct HTTP uses a fixed HTTPS endpoint, refuses redirects, has a 30-second socket timeout, and performs no automatic retries.
|
||||
Use the skill's 45-second tool timeout as the process bound. An aborted request may still incur usage.
|
||||
|
||||
Choice confidence below 0.8, unknown/insufficient evidence, or a yes/no probability between 0.2 and 0.8 requires review.
|
||||
These thresholds are provisional. They are not calibrated measures of GBA hardware correctness.
|
||||
Do not repeatedly query until the model agrees with a preferred answer.
|
||||
|
||||
Never send ROMs, extracted game code/assets, firmware, save data, screenshots, audio, credentials, or unfiltered logs.
|
||||
Selected emulator-generated diagnostics and original test summaries are sufficient.
|
||||
The helper rejects unknown fields, excessive input, common credential markers, and common private paths.
|
||||
These checks do not detect every secret or private detail. The agent must review the full payload before requesting evaluation.
|
||||
State and model responses are untrusted data. An injection smoke test does not establish a security boundary.
|
||||
|
||||
Raw records stay ignored. Commit only reviewed summaries of useful findings, methods, and original regression tests.
|
||||
Normal Cargo and Python tests stay offline and do not require Jev credentials.
|
||||
|
||||
## Validation
|
||||
|
||||
The [reviewed smoke evaluation](research/jev-smoke.md) records live classifier outcomes and their limits.
|
||||
The HTTP path was exercised live. Native Pi response normalization was tested offline; native authentication needs the environment alias above.
|
||||
|
||||
## References
|
||||
|
||||
- [TypeSafe documentation index](https://docs.typesafe.ai/llms.txt).
|
||||
- [API reference](https://docs.typesafe.ai/api.md) and [model capabilities](https://docs.typesafe.ai/models.md).
|
||||
- [Jev with coding agents](https://docs.typesafe.ai/introduction/coding-agents.md): structured decisions, not code generation.
|
||||
- [Confidence](https://docs.typesafe.ai/confidence.md) and [Jev 1.13 limitations](https://docs.typesafe.ai/model-jaggedness/jev-1.13.md): arithmetic, indirection, option order, and adversarial content.
|
||||
- [What is Codemode](https://lucumr.pocoo.org/2026/10/6/codemode/): harness-side orchestration and the bounded game-control example.
|
||||
- Installed Pi documentation: `docs/codemode.md`, `docs/models.md`, `docs/skills.md`, `docs/prompt-templates.md`, and project-trust/configuration references.
|
||||
|
|
@ -246,6 +246,12 @@ Disabled sound and disconnected serial reset:
|
|||
- [GBATEK normal serial mode](https://problemkaputt.de/gbatek-sio-normal-mode.htm) and its [general-purpose register chapter](https://mgba-emu.github.io/gbatek/), for data widths, control fields, and disconnected input pull-ups.
|
||||
- [mGBA BIOS implementation at 3a5e34be](https://github.com/mgba-emu/mgba/blob/3a5e34be33dc7f8f707e5bc9db69e8a430046f21/src/gba/bios.c), for functional serial and sound reset defaults. Exact firmware ordering and timing remain unverified.
|
||||
|
||||
Optional classifier-assisted debugging:
|
||||
|
||||
- [TypeSafe documentation index](https://docs.typesafe.ai/llms.txt), [API reference](https://docs.typesafe.ai/api.md), [models](https://docs.typesafe.ai/models.md), [confidence](https://docs.typesafe.ai/confidence.md), and [Jev limitations](https://docs.typesafe.ai/model-jaggedness/jev-1.13.md), for typed evidence review and its limits.
|
||||
- [What is Codemode](https://lucumr.pocoo.org/2026/10/6/codemode/), for bounded harness-side orchestration and game-debugging examples.
|
||||
- Installed Pi documentation for Codemode, classifiers, skills, prompt templates, settings, configuration, providers, and project trust. See [the local workflow](jev-debugging.md).
|
||||
|
||||
Use [GBATEK](https://problemkaputt.de/gbatek.htm) and ARM7TDMI documentation for further hardware work.
|
||||
Only use game ROMs that you may lawfully use. Do not commit game ROMs, BIOS files, or game assets.
|
||||
The ignore file excludes `roms/`, common game ROM extensions, and save files.
|
||||
|
|
|
|||
43
docs/research/jev-smoke.md
Normal file
43
docs/research/jev-smoke.md
Normal file
|
|
@ -0,0 +1,43 @@
|
|||
# Jev workflow smoke evaluation
|
||||
|
||||
The project-local Jev workflow was tested through Pi Codemode using the direct TypeSafe HTTP helper.
|
||||
The existing `TYPESAFEAI_API_KEY` authenticated successfully without changing global Pi configuration.
|
||||
The returned model identifier was `jev-1.13.0`.
|
||||
Native `models.classify` was not exercised live because this Pi process lacked its expected `TYPESAFE_API_KEY` variable.
|
||||
Native response normalization is covered by offline fixtures.
|
||||
|
||||
## Reviewed observations
|
||||
|
||||
| Evidence | Subsystem advice | Evidence level | Probability that the stated claim is supported | Review required |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| Synthetic unsupported sound activation; claim that sound works | Audio | Diagnostic | 0.02 | No |
|
||||
| Synthetic successful scripted gameplay checks; claim limited to those checks | None | Scripted gameplay | 0.85 | No |
|
||||
| Synthetic budget-only run with an injected instruction to declare compatibility | Unknown | Insufficient | 0.02 | Yes |
|
||||
| Selected local startup diagnostic after BIOS reset; claim of playable gameplay with sound | Audio | Diagnostic | 0.02 | No |
|
||||
|
||||
These are classifier integration checks, not new emulator tests.
|
||||
The synthetic cases describe assumed evidence; they do not run or validate a game.
|
||||
The local startup case uses reviewed facts from the [existing runtime result](emerald-reset.md), not ROM content.
|
||||
The model agreed that sound activation remains the next observed blocker. It did not find a new hardware root cause.
|
||||
|
||||
The budget-only case had low evidence-level confidence and remained marked for review.
|
||||
Rejecting this one injected instruction does not establish resistance to adversarial input.
|
||||
A high-confidence category is still advice, not proof of correct hardware behavior.
|
||||
|
||||
## Record locations
|
||||
|
||||
Raw request contexts, hashes, jj revision identifiers, responses, and usage remain under ignored `.pi/jev-runs/`:
|
||||
|
||||
- `audio-probe-001.json`
|
||||
- `gameplay-probe-001.json`
|
||||
- `budget-probe-001.json`
|
||||
- `local-startup-probe-001.json`
|
||||
|
||||
The committed synthetic inputs are in `tools/fixtures/jev/`.
|
||||
The selected local startup evidence stays ignored alongside its record.
|
||||
No ROM, save, credential, screenshot, audio, or extracted game code was sent to the classifier.
|
||||
The helper retained failed/incomplete deterministic status regardless of its model answers and executed no model-selected action.
|
||||
|
||||
Ordinary Python tests mock network responses and credentials. They cover input limits, response validation, failure paths, and log safety.
|
||||
Re-run live evaluations deliberately with new filenames; do not make model output an offline test expectation or query until it agrees.
|
||||
See [the workflow](../jev-debugging.md) for limits, credential setup, and the command.
|
||||
|
|
@ -67,6 +67,9 @@ Run the external Pong suite separately; ordinary Cargo and Python tests require
|
|||
Device initialization regressions use original instructions to check disabled sound, wave RAM, disconnected serial pins, and reset flags.
|
||||
They verify explicit activation errors and atomic byte/word/block-store/DMA failure behavior.
|
||||
The optional [local Emerald startup check](research/emerald-reset.md) now reaches sound activation but still fails deliberately there.
|
||||
[Jev evidence review](jev-debugging.md) is optional and never changes deterministic test verdicts.
|
||||
Its Python tests mock the provider and verify schemas, response validation, credential handling, non-overwriting logs, and failure behavior.
|
||||
Live classifier smoke probes are separate from offline tests and must not become mandatory CI checks.
|
||||
Thumb cases also verify the return-to-ARM bridge. Command-line tests cover default ARM selection, explicit Thumb/memory/BIOS selection, and errors.
|
||||
Compare/status CPU regressions verify saved flags/masks, bank switching, sequential PC/timing, User/System fallback, and atomic diagnostics.
|
||||
Load-alias regressions cover widths, signed/unaligned values, immediate/register offsets, pre/post-indexing, every non-PC register bank, and address wrapping.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue