feat(testing): add bounded headless ROM suites and JSON reports

Read strict versioned suite manifests with independent ROM cases, explicit ARM/Thumb checkpoints, register/CPSR/memory assertions, and aggregate step limits. Report every case deterministically and fail on timeouts, STOP, diagnostics, missing files, or mismatches. Keep host JSON dependencies outside the core.

Add an original ARM/Thumb/BIOS suite generator, unit and process-level regressions, and schema/usage documentation. Verified on Darwin arm64 with debug/release workspace and core suites, Clippy, rustdoc, native ROM-window tests, all ten demo smoke tests, repeatable release reports, and a deliberate assertion failure. Independent public hardware-test ROM validation remains next.

docs/architecture.md

@@ -5,6 +5,7 @@ The repository is a Cargo workspace with three crates:
 - `crates/gba-core`: the platform-independent GBA emulator. It has no dependencies and must not depend on window, audio, or file-system libraries.
 - `crates/gba-demos`: original ARM test programs and their frame runners. Depends on `gba-core`.
 - `crates/desktop`: the minifb window frontend and the `gameboy-rust` executable. Depends on both.
+  Serde and serde_json handle host-side test manifests and reports; neither dependency enters the core.
 
 `gba-core` uses `gba-demos` as a development dependency, because its integration tests run the demo programs.
 
@@ -80,6 +81,10 @@ The repository is a Cargo workspace with three crates:
 | `crates/desktop/src/desktop/raster.rs` | Raster window and scanline color-band smoke test |
 | `crates/desktop/src/main.rs` | Command-line modes, error exit status, and fixed-length CPU demo |
 | `crates/desktop/src/cartridge.rs` | Read-only ROM files, mode/limit checks, bounded terminal execution, and shared state reports |
+| `crates/desktop/src/cartridge/suite.rs` | Strict JSON manifests, bounded checkpoint execution, assertions, and structured reports |
+| `crates/desktop/src/cartridge/suite/tests.rs` | Manifest limits, checkpoint boundaries, memory/register checks, and diagnostic accounting |
+| `crates/desktop/tests/rom_suite_cli.rs` | File-backed suite execution, case isolation, output schema, determinism, and exit status |
+| `crates/desktop/examples/write_test_suite.rs` | Original ARM/Thumb/BIOS fixture generator with a local JSON suite |
 | `crates/desktop/src/cartridge/window.rs` | ROM-window session, bounded slices, capture, STOP/input handling, and progress limits |
 | `crates/desktop/src/cartridge/window/tests.rs` | Original ROM pixels, keyboard mapping, STOP wake, HALT, frame budgets, and diagnostics |
 | `crates/desktop/src/desktop/rom.rs` | Main-thread ROM window, input polling, presentation pacing, and frame-limited exit |

docs/development.md

@@ -19,6 +19,7 @@ Verified locally on macOS 15.7.3:
 - The terminal-only CPU and timer IRQ demos run successfully.
 - The bounded terminal ROM runner loads original raw programs and reports state without opening a window.
 - The native ROM-window test presents a file-backed original program and reports STOP, CPU, and video failures.
+- The headless suite runner passes generated ARM/Thumb/BIOS cases, produces repeatable JSON, and rejects deliberate assertion mismatches.
 
 These results cover the CPU core, memory, Mode 0–5 snapshots and row capture, input mapping, and desktop windows.
 They do not verify a complete GBA display controller or audio.

docs/hardware/cartridge.md

@@ -2,6 +2,7 @@
 
 The desktop executable can load raw ROM bytes for terminal diagnostics or window execution.
 This supports original test programs. It does not establish commercial-game compatibility.
+For explicit pass/fail assertions instead of diagnostic limits, use the [headless ROM suite runner](../rom-tests.md).
 
 ```sh
 direnv exec . cargo run --locked --release -- --rom path/to/original.gba --steps 100000

docs/rom-tests.md

@@ -0,0 +1,136 @@
+# Headless ROM test suites
+
+Use `--test-suite PATH.json` to run repeatable ROM checks without opening a window.
+Each case has a machine-step budget, a completion address, and explicit assertions.
+A timeout never counts as a pass. Existing `--rom ... --steps ...` remains a diagnostic mode, not a test verdict.
+
+This runner currently has regression coverage from original programs, not independent public hardware-test results.
+A passing suite establishes only its specified checkpoints and assertions. It does not establish Emerald compatibility.
+
+## Run the original smoke suite
+
+```sh
+direnv exec . cargo run --locked --example write_test_suite -- /tmp/gba-original-tests
+direnv exec . cargo run --locked --release -- --test-suite /tmp/gba-original-tests/suite.json > /tmp/gba-test-report.json
+```
+
+The generator creates original ARM, Thumb, and BIOS-division programs, plus a JSON manifest.
+The destination directory must not already exist. No Nintendo code, firmware, or game assets are included.
+Generated binaries stay outside the repository.
+The runner writes one JSON report to stdout. Cargo build messages and errors go to stderr.
+Do not redirect the report over the manifest or a ROM file.
+
+## Manifest version 1
+
+```json
+{
+  "version": 1,
+  "cases": [
+    {
+      "name": "arm-store",
+      "rom": "arm.gba",
+      "step_limit": 10000,
+      "completion": {
+        "pc": "0x0800000c",
+        "instruction_set": "arm"
+      },
+      "checks": [
+        { "kind": "register", "index": 0, "equals": 42 },
+        { "kind": "memory32", "address": "0x02000000", "equals": 42 },
+        { "kind": "cpsr", "equals": "0x1f" }
+      ]
+    }
+  ]
+}
+```
+
+This example matches the generated `arm.gba`, not arbitrary ROMs.
+The program stores 42 to external RAM, then reaches an ARM self-branch at `0x0800000c`.
+
+Rules:
+
+- All shown fields are required. Unknown fields and unsupported check kinds are rejected.
+- The manifest must be a regular UTF-8 JSON file, no larger than 1 MiB.
+- Supply 1–256 cases. Names must be unique, nonempty after trimming, and at most 128 UTF-8 bytes.
+- Each case needs 1–256 checks and a nonempty ROM path.
+- Relative ROM paths resolve from the manifest's directory, not the current working directory. Absolute paths are accepted.
+- Each step limit is a JSON integer from 1 through 100,000,000. Their sum must not exceed 100,000,000.
+- Addresses and expected values accept unsigned 32-bit JSON integers, decimal strings, or lowercase `0x`-prefixed hexadecimal strings.
+- Signed values, floating-point values, surrounding whitespace, and overflow are rejected. Encode negative bit patterns as unsigned words.
+- `instruction_set` is exactly `arm` or `thumb`. ARM completion addresses must be word-aligned; Thumb addresses must be halfword-aligned.
+- Use the actual Thumb instruction address, without the address bit used by `BX`.
+- Register indices are integers from 0 through 15. Checks read the active register bank.
+- `memory32` addresses must be word-aligned. Reads use the existing little-endian memory bus, including its mapping errors.
+- `cpsr` compares the complete current program status register. No mask is applied.
+
+The runner validates the whole manifest before opening or executing any ROM.
+It uses the same read-only [ROM loader](hardware/cartridge.md#loading-and-boot), with its 4-byte minimum and 32 MiB maximum.
+The CLI accepts no other options with `--test-suite`.
+
+## Completion and failure semantics
+
+Each case starts with fresh RAM, devices, CPU state, and the original BIOS replacement.
+The budget includes BIOS boot, instructions, DMA units, IRQ entries, and HALT idle batches.
+A failed CPU or DMA step does not consume a successful step or advance the reported state.
+The runner supplies no input and does not enable frame capture. Device clocks still advance through normal machine stepping.
+
+After successful progress, the runner checks PC and instruction set at machine-step boundaries.
+It stops before executing the instruction at the completion address.
+Reaching that address on the final allowed step succeeds if all checks pass.
+The initial reset state alone cannot pass a case. STOP always ends the case as a failure because no input source can wake it.
+HALT continues advancing devices until completion, a diagnostic, or budget exhaustion.
+
+The checkpoint does not imply that pending DMA or IRQ work has completed.
+Choose an address reached only after the tested operations finish.
+A loop address reached before those operations would define an invalid test, even if the runner reports a pass.
+The runner does not infer program completion from an infinite loop, a screen, or matching register values alone.
+
+All checks run at the checkpoint. A mismatch or unreadable memory address fails the case.
+Cases that never reach the checkpoint do not evaluate their checks.
+A failure does not skip later cases. No machine state passes from one case to another.
+
+The budget bounds machine steps, not elapsed host time. Use an external process timeout in continuous integration if needed.
+This is not a filesystem security sandbox. Run trusted suite configurations and use files you may lawfully use.
+
+## Report version 1 and exit status
+
+Top-level fields:
+
+| Field | Meaning |
+| --- | --- |
+| `format_version` | Report schema version, currently 1 |
+| `bios` | `original`; no external BIOS option |
+| `passed` | True only when every case passed |
+| `case_count`, `failed_count` | Number of executed cases and failures |
+| `cases` | Results in manifest order |
+
+Each case records its name, ROM path, debug-formatted resolved path, byte count, budget, completion condition, and result.
+`state` contains registers r0–r15, PC, CPSR, mode, instruction set, HALT/STOP flags, nominal cycles, and step counters.
+Addresses, registers, expected values, and counters in reports are JSON numbers, even when the manifest used hexadecimal strings.
+A load failure has null `state` and `rom_bytes`.
+`checks` contains the configured check, actual value, pass flag, and optional read error when the checkpoint was reached.
+Otherwise `checks` is empty. A failed read has a null actual value.
+
+Case reasons are:
+
+- `checkpoint`: completion reached and all assertions passed.
+- `assertion_failed`: completion reached but at least one assertion failed or could not read memory.
+- `step_limit`: completion was not reached within the budget.
+- `stopped`: STOP prevented further execution.
+- `emulation_error`: an instruction or DMA diagnostic stopped execution; `error` contains its message.
+- `load_error`: loading or machine creation failed; `error` contains its message.
+
+Exit status 0 means all cases passed and the report was written successfully.
+Any failed case, invalid configuration, or output failure returns a nonzero status.
+Invalid manifests produce a stderr diagnostic without a JSON report. A broken output stream can leave an incomplete report.
+Valid suites produce a report even when some ROMs fail to load.
+Reports omit host timestamps and durations so unchanged files produce repeatable output on the same host.
+They do not hash ROM contents; record source revisions and file checksums alongside external test results.
+
+## Next validation stage
+
+Select public ARM7TDMI/GBA tests and inspect their source, license, entry assumptions, and result protocol.
+Pin source revisions and build instructions before comparing emulator changes.
+Adapt suites to verified completion addresses and result locations; do not guess them or treat timeouts as passes.
+Some tests need debug-port logging, input, different firmware behavior, or rendered-image checks that this runner does not support yet.
+Keep external binaries local. Record consulted sources in [references.md](references.md).

docs/status.md

@@ -65,6 +65,7 @@ Update this file when a feature lands or a limit is removed.
 - Raw ROM-file loading with original BIOS boot, terminal diagnostics, and final CPU/step reports.
 - ROM windows with scanline capture, keyboard input, focus-loss release, bounded execution slices, and interactive keypad wake from STOP.
 - Optional ROM-window frame limits and an original input-test ROM generator.
+- Headless JSON ROM suites with per-case checkpoints, register/CPSR/memory assertions, bounded budgets, and structured failure reports.
 - Little-endian byte, halfword, and word reads/writes.
 - ARM7TDMI word-load rotation, aligned stores, and odd-address halfword-load behavior.
 - Explicit errors for unsupported instructions when their condition passes, and for unsupported memory accesses.
@@ -109,6 +110,8 @@ ROM files can run in bounded terminal mode or a window with the original BIOS re
 ROM windows have no demo-specific startup checks. Input is sampled between bounded execution slices.
 Audio, saves, and external BIOS loading remain unavailable. ROM loading does not establish commercial-game compatibility.
 See [cartridge loading](hardware/cartridge.md) for file limits, step budgets, and exit status.
+The [ROM suite runner](rom-tests.md) currently uses original regression programs, not independent public hardware-test results.
+It requires known completion addresses and has no debug-port protocol, scripted input, or rendered-image assertions.
 
 Unmapped reads return errors instead of hardware open-bus values.
 Reads beyond the supplied cartridge bytes also return errors.
@@ -118,7 +121,7 @@ Writes to cartridge addresses return errors instead of modeling cartridge hardwa
 
 ## Next steps
 
-1. Validate CPU behavior with public ARM7TDMI test programs before claiming instruction compatibility.
+1. Adapt the ROM suite runner to pinned public ARM7TDMI test programs and verified result protocols before claiming instruction compatibility.
 2. Refine nominal timing with a fetch pipeline, Game Pak prefetch, per-access device updates, and verified timer/IRQ delays.
 3. Validate keypad retrigger behavior with hardware tests and add remaining device registers; extend BIOS reset coverage as sound/serial support becomes available.
    Validate STOP entry/wake edges and add external wake sources and remaining DMA device modes.

docs/testing.md

@@ -46,6 +46,11 @@ The core's integration tests use `gba-demos` as a development dependency.
 On Apple Silicon, `file` must report a Mach-O `arm64` executable.
 Do not set a Linux cross-compilation target for this validation.
 
+The [headless ROM suite runner](rom-tests.md) provides explicit checkpoint and assertion results through `--test-suite PATH.json`.
+Its tests verify manifest limits, whole-suite validation, exact step boundaries, ARM/Thumb checkpoints, and register/CPSR/memory checks.
+Additional tests cover DMA/IRQ accounting, HALT/STOP, diagnostic state, case isolation, repeatable JSON output, and failure exit status.
+These original tests validate the runner; they are not independent public hardware-test results.
+
 ROM-runner tests use temporary files containing original instructions, without Nintendo assets or a desktop session.
 They check file-size boundaries, bounded reads, short reads, I/O errors, paths, symlinks, and read-only loading.
 CLI tests cover option order, missing/conflicting options, invalid step limits, process exit status, and reports.