feat(testing): add bounded headless ROM suites and JSON reports

Read strict versioned suite manifests with independent ROM cases, explicit ARM/Thumb checkpoints, register/CPSR/memory assertions, and aggregate step limits. Report every case deterministically and fail on timeouts, STOP, diagnostics, missing files, or mismatches. Keep host JSON dependencies outside the core.

Add an original ARM/Thumb/BIOS suite generator, unit and process-level regressions, and schema/usage documentation. Verified on Darwin arm64 with debug/release workspace and core suites, Clippy, rustdoc, native ROM-window tests, all ten demo smoke tests, repeatable release reports, and a deliberate assertion failure. Independent public hardware-test ROM validation remains next.

AGENTS.md

@@ -34,6 +34,7 @@ Window smoke tests (`--*-smoke-test`) need a logged-in desktop session. Details
 | Working on | Read |
 | --- | --- |
 | ARM/Thumb instructions, modes, exceptions, timing | [docs/hardware/cpu.md](docs/hardware/cpu.md) |
+| ROM test suites, assertions, and JSON reports | [docs/rom-tests.md](docs/rom-tests.md) |
 | ROM files, window execution, and terminal diagnostics | [docs/hardware/cartridge.md](docs/hardware/cartridge.md) |
 | BIOS services (SWI) | [docs/hardware/bios.md](docs/hardware/bios.md) |
 | Timers, IE/IF/IME, HALT/STOP | [docs/hardware/timers-irq.md](docs/hardware/timers-irq.md) |

Cargo.lock

@@ -147,6 +147,8 @@ dependencies = [
  "gba-core",
  "gba-demos",
  "minifb",
+ "serde",
+ "serde_json",
 ]
 
 [[package]]

README.md

@@ -30,6 +30,7 @@ Arrows scroll, Q rotates, W zooms, Z flips, X changes priority, Enter resets, an
 Run `direnv exec . cargo run --locked -- --help` to list all modes.
 For original ROM files, use `--rom path/to/original.gba --window`, or `--steps 100000` instead of `--window` for terminal diagnostics.
 See [cartridge loading](docs/hardware/cartridge.md) for limits and report meanings.
+Use `--test-suite suite.json` for [repeatable headless ROM assertions](docs/rom-tests.md).
 
 ## Documentation
 
@@ -39,6 +40,7 @@ See [cartridge loading](docs/hardware/cartridge.md) for limits and report meanin
 | [docs/demos.md](docs/demos.md) | Every demo, its command, and its controls |
 | [docs/development.md](docs/development.md) | Nix, direnv, and macOS build notes |
 | [docs/testing.md](docs/testing.md) | Validation commands and what the test suites cover |
+| [docs/rom-tests.md](docs/rom-tests.md) | Headless ROM suites, completion checks, assertions, and JSON reports |
 | [docs/architecture.md](docs/architecture.md) | Crate boundaries and a file-by-file map |
 | [docs/hardware/cpu.md](docs/hardware/cpu.md) | ARM/Thumb execution rules, modes, exceptions, and timing |
 | [docs/hardware/cartridge.md](docs/hardware/cartridge.md) | ROM loading, window/input controls, original ROM generator, and diagnostics |

crates/desktop/Cargo.toml

@@ -11,5 +11,7 @@ path = "src/main.rs"
 [dependencies]
 gba-core = { path = "../gba-core" }
 gba-demos = { path = "../gba-demos" }
+serde = { version = "1", features = ["derive"] }
+serde_json = "1"
 # Pinned because src/desktop.rs corrects a macOS focus bug in this version.
 minifb = { version = "=0.28.0", default-features = false, features = ["x11"] }

crates/desktop/examples/write_test_suite.rs

@@ -0,0 +1,78 @@
+//! Export original ROM instructions and a reproducible smoke-test manifest.
+//! The destination directory must not already exist. No Nintendo content is used.
+use serde_json::json;
+use std::{
+    error::Error,
+    fs::{self, OpenOptions},
+    io::{self, Write},
+    path::{Path, PathBuf},
+};
+
+fn words(code: &[u32]) -> Vec<u8> {
+    code.iter().flat_map(|word| word.to_le_bytes()).collect()
+}
+fn write_new(path: &Path, bytes: &[u8]) -> io::Result<()> {
+    let mut file = OpenOptions::new().write(true).create_new(true).open(path)?;
+    file.write_all(bytes)?;
+    file.flush()
+}
+fn main() -> Result<(), Box<dyn Error>> {
+    let invalid = || {
+        io::Error::new(
+            io::ErrorKind::InvalidInput,
+            "usage: write_test_suite NEW_DIRECTORY",
+        )
+    };
+    let mut args = std::env::args_os().skip(1);
+    let directory = PathBuf::from(
+        args.next()
+            .filter(|path| !path.is_empty())
+            .ok_or_else(invalid)?,
+    );
+    if args.next().is_some() {
+        return Err(invalid().into());
+    }
+    fs::create_dir(&directory)?;
+    write_new(
+        &directory.join("arm.gba"),
+        &words(&[
+            0xe3a0_002a, // MOV r0,#42
+            0xe3a0_1402, // MOV r1,#0x02000000
+            0xe581_0000, // STR r0,[r1]
+            0xeaff_fffe, // completion: B .
+        ]),
+    )?;
+    let mut thumb = words(&[0xe59f_1000, 0xe12f_ff11, 0x0800_000d]); // LDR r1,literal; BX r1
+    thumb.extend([0x2a, 0x20, 0xfe, 0xe7]); // Thumb MOV r0,#42; completion: B .
+    write_new(&directory.join("thumb.gba"), &thumb)?;
+    write_new(
+        &directory.join("division.gba"),
+        &words(&[
+            0xe3a0_002a, // MOV r0,#42
+            0xe3a0_1005, // MOV r1,#5
+            0xef06_0000, // SWI Div (original BIOS replacement)
+            0xeaff_fffe, // completion: B .
+        ]),
+    )?;
+    let manifest = json!({"version":1,"cases":[
+        {"name":"arm-store","rom":"arm.gba","step_limit":10000,
+         "completion":{"pc":"0x0800000c","instruction_set":"arm"},
+         "checks":[{"kind":"register","index":0,"equals":42},
+                   {"kind":"memory32","address":"0x02000000","equals":42},
+                   {"kind":"cpsr","equals":"0x1f"}]},
+        {"name":"thumb-move","rom":"thumb.gba","step_limit":10000,
+         "completion":{"pc":"0x0800000e","instruction_set":"thumb"},
+         "checks":[{"kind":"register","index":0,"equals":42}]},
+        {"name":"bios-division","rom":"division.gba","step_limit":10000,
+         "completion":{"pc":"0x0800000c","instruction_set":"arm"},
+         "checks":[{"kind":"register","index":0,"equals":8},
+                   {"kind":"register","index":1,"equals":2},
+                   {"kind":"register","index":3,"equals":8}]}
+    ]});
+    write_new(
+        &directory.join("suite.json"),
+        &serde_json::to_vec_pretty(&manifest)?,
+    )?;
+    println!("Wrote original ARM, Thumb, and BIOS test ROMs with suite.json in {directory:?}");
+    Ok(())
+}

crates/desktop/src/cartridge.rs

@@ -1,5 +1,6 @@
 //! Host-only raw ROM loading, execution modes, and diagnostics.
 
+pub mod suite;
 pub mod window;
 
 use std::{

crates/desktop/src/cartridge/suite.rs

@@ -0,0 +1,344 @@
+//! Bounded, headless ROM tests with explicit checkpoints and assertions.
+//! Passing means only that the configured checkpoint and checks succeeded.
+
+use std::{
+    collections::HashSet,
+    error::Error,
+    ffi::OsString,
+    fs::{self, File},
+    io::{self, Read, Write},
+    path::{Path, PathBuf},
+};
+
+use gba_core::{
+    bios,
+    cpu::InstructionSet,
+    machine::{Machine, StepKind},
+};
+use serde::{Deserialize, Deserializer, Serialize};
+use serde_json::{json, Value};
+
+use super::{load, Stats, MAX_STEPS};
+
+const MAX_MANIFEST_BYTES: u64 = 1024 * 1024;
+const MAX_CASES: usize = 256;
+const MAX_CHECKS: usize = 256;
+const USAGE: &str = "usage: gameboy-rust --test-suite PATH.json (no other options)";
+
+pub fn parse_args(args: &[OsString]) -> io::Result<PathBuf> {
+    match args {
+        [flag, path] if flag == "--test-suite" && !path.is_empty() => Ok(path.into()),
+        _ => Err(io::Error::new(io::ErrorKind::InvalidInput, USAGE)),
+    }
+}
+
+/// JSON integers or unsigned decimal/0x-prefixed strings, never floats or signs.
+fn word<'de, D: Deserializer<'de>>(deserializer: D) -> Result<u32, D::Error> {
+    let value = Value::deserialize(deserializer)?;
+    let parsed = match value {
+        Value::Number(number) => number.as_u64().and_then(|value| u32::try_from(value).ok()),
+        Value::String(text) => {
+            let (digits, radix) = text
+                .strip_prefix("0x")
+                .map_or((text.as_str(), 10), |digits| (digits, 16));
+            if digits.is_empty()
+                || !digits.bytes().all(|byte| {
+                    if radix == 16 {
+                        byte.is_ascii_hexdigit()
+                    } else {
+                        byte.is_ascii_digit()
+                    }
+                })
+            {
+                None
+            } else {
+                u32::from_str_radix(digits, radix).ok()
+            }
+        }
+        _ => None,
+    };
+    parsed.ok_or_else(|| {
+        serde::de::Error::custom("expected a u32 integer or unsigned decimal/0x string")
+    })
+}
+
+#[derive(Debug, Deserialize)]
+#[serde(deny_unknown_fields)]
+struct Suite {
+    version: u32,
+    cases: Vec<Case>,
+}
+
+#[derive(Debug, Deserialize)]
+#[serde(deny_unknown_fields)]
+struct Case {
+    name: String,
+    rom: PathBuf,
+    step_limit: u64,
+    completion: Completion,
+    checks: Vec<Check>,
+}
+
+#[derive(Debug, Deserialize, Serialize)]
+#[serde(deny_unknown_fields)]
+struct Completion {
+    #[serde(deserialize_with = "word")]
+    pc: u32,
+    instruction_set: State,
+}
+
+#[derive(Debug, Clone, Copy, Deserialize, Serialize)]
+#[serde(rename_all = "snake_case")]
+enum State {
+    Arm,
+    Thumb,
+}
+
+impl State {
+    fn instruction_set(self) -> InstructionSet {
+        match self {
+            Self::Arm => InstructionSet::Arm,
+            Self::Thumb => InstructionSet::Thumb,
+        }
+    }
+}
+
+#[derive(Debug, Deserialize, Serialize)]
+#[serde(tag = "kind", rename_all = "snake_case", deny_unknown_fields)]
+enum Check {
+    Register {
+        index: usize,
+        #[serde(deserialize_with = "word")]
+        equals: u32,
+    },
+    Memory32 {
+        #[serde(deserialize_with = "word")]
+        address: u32,
+        #[serde(deserialize_with = "word")]
+        equals: u32,
+    },
+    Cpsr {
+        #[serde(deserialize_with = "word")]
+        equals: u32,
+    },
+}
+
+impl Suite {
+    fn validate(&self) -> io::Result<()> {
+        let invalid = |message: &str| io::Error::new(io::ErrorKind::InvalidInput, message);
+        if self.version != 1 {
+            return Err(invalid("suite version must be 1"));
+        }
+        if self.cases.is_empty() || self.cases.len() > MAX_CASES {
+            return Err(invalid("suite must contain 1..=256 cases"));
+        }
+        let mut names = HashSet::new();
+        let mut total = 0u64;
+        for case in &self.cases {
+            if case.name.trim().is_empty() || case.name.len() > 128 || !names.insert(&case.name) {
+                return Err(invalid(
+                    "case names must be nonempty, unique, and at most 128 UTF-8 bytes",
+                ));
+            }
+            if case.rom.as_os_str().is_empty() {
+                return Err(invalid("case ROM path must not be empty"));
+            }
+            if !(1..=MAX_STEPS).contains(&case.step_limit) {
+                return Err(invalid("case step_limit must be 1..=100000000"));
+            }
+            total += case.step_limit;
+            if total > MAX_STEPS {
+                return Err(invalid("sum of case step limits must not exceed 100000000"));
+            }
+            let alignment = match case.completion.instruction_set {
+                State::Arm => 4,
+                State::Thumb => 2,
+            };
+            if case.completion.pc % alignment != 0 {
+                return Err(invalid("completion PC must be aligned for its instruction set; do not set the Thumb address bit"));
+            }
+            if case.checks.is_empty() || case.checks.len() > MAX_CHECKS {
+                return Err(invalid("each case must contain 1..=256 checks"));
+            }
+            for check in &case.checks {
+                match check {
+                    Check::Register { index, .. } if *index > 15 => {
+                        return Err(invalid("register index must be 0..=15"))
+                    }
+                    Check::Memory32 { address, .. } if address % 4 != 0 => {
+                        return Err(invalid("memory32 address must be word-aligned"))
+                    }
+                    _ => {}
+                }
+            }
+        }
+        Ok(())
+    }
+}
+
+fn read_suite(path: &Path) -> Result<Suite, Box<dyn Error>> {
+    let validate = |metadata: fs::Metadata| -> io::Result<()> {
+        if !metadata.is_file() || metadata.len() > MAX_MANIFEST_BYTES {
+            return Err(io::Error::new(
+                io::ErrorKind::InvalidInput,
+                "suite must be a regular file no larger than 1 MiB",
+            ));
+        }
+        Ok(())
+    };
+    validate(fs::metadata(path)?)?;
+    let file = File::open(path)?;
+    validate(file.metadata()?)?;
+    let mut bytes = Vec::new();
+    file.take(MAX_MANIFEST_BYTES + 1).read_to_end(&mut bytes)?;
+    if bytes.len() as u64 > MAX_MANIFEST_BYTES {
+        return Err(io::Error::new(io::ErrorKind::InvalidInput, "suite exceeds 1 MiB").into());
+    }
+    let suite: Suite = serde_json::from_slice(&bytes)?;
+    suite.validate()?;
+    Ok(suite)
+}
+
+#[derive(Debug, PartialEq, Eq)]
+enum Outcome {
+    Checkpoint,
+    StepLimit,
+    Stopped,
+    EmulationError(String),
+}
+
+fn run_to_checkpoint(machine: &mut Machine, case: &Case) -> (Stats, Outcome) {
+    let mut stats = Stats::default();
+    loop {
+        if machine.stopped() {
+            return (stats, Outcome::Stopped);
+        }
+        // Check after successful progress, before trying to execute the checkpoint.
+        // The final budgeted step can reach it. No initial/reset-state pass is allowed.
+        if stats.steps != 0
+            && machine.cpu().pc() == case.completion.pc
+            && machine.cpu().instruction_set() == case.completion.instruction_set.instruction_set()
+        {
+            return (stats, Outcome::Checkpoint);
+        }
+        if stats.steps == case.step_limit {
+            return (stats, Outcome::StepLimit);
+        }
+        match machine.step() {
+            Ok(StepKind::StopIdle) => return (stats, Outcome::Stopped),
+            Ok(kind) => stats.record(kind),
+            Err(error) => return (stats, Outcome::EmulationError(error.to_string())),
+        }
+    }
+}
+
+fn check(machine: &Machine, check: &Check) -> Value {
+    let (expected, actual) = match check {
+        Check::Register { index, equals } => (*equals, Ok(machine.cpu().registers()[*index])),
+        Check::Memory32 { address, equals } => (*equals, machine.memory().read32(*address)),
+        Check::Cpsr { equals } => (*equals, Ok(machine.cpu().cpsr())),
+    };
+    match actual {
+        Ok(actual) => {
+            json!({"check": check, "actual": actual, "passed": actual == expected, "error": null})
+        }
+        Err(error) => {
+            json!({"check": check, "actual": null, "passed": false, "error": error.to_string()})
+        }
+    }
+}
+
+fn state(machine: &Machine, stats: &Stats) -> Value {
+    json!({
+        "registers": machine.cpu().registers(), "pc": machine.cpu().pc(), "cpsr": machine.cpu().cpsr(),
+        "mode": format!("{:?}", machine.cpu().mode()),
+        "instruction_set": match machine.cpu().instruction_set() { InstructionSet::Arm => "arm", InstructionSet::Thumb => "thumb" },
+        "halted": machine.halted(), "stopped": machine.stopped(), "nominal_cycles": machine.cycles(),
+        "steps": stats.steps, "instructions": stats.instructions, "irq_entries": stats.irq_entries,
+        "dma_units": stats.dma_units, "halt_idle": stats.halt_idle,
+    })
+}
+
+fn run_case(case: &Case, base: &Path) -> Value {
+    let path = base.join(&case.rom);
+    // Debug formatting preserves non-UTF-8 host paths without lossy replacement.
+    let mut report = json!({
+        "name": case.name, "rom": case.rom, "resolved_path_debug": format!("{path:?}"),
+        "step_limit": case.step_limit, "completion": case.completion,
+        "passed": false, "reason": "load_error", "error": null,
+        "rom_bytes": null, "state": null, "checks": [],
+    });
+    let bytes = match load(&path) {
+        Ok(bytes) => bytes,
+        Err(error) => {
+            report["error"] = json!(error.to_string());
+            return report;
+        }
+    };
+    report["rom_bytes"] = json!(bytes.len());
+    let mut machine = match bios::boot(bytes) {
+        Ok(machine) => machine,
+        Err(error) => {
+            report["error"] = json!(error.to_string());
+            return report;
+        }
+    };
+    let (stats, outcome) = run_to_checkpoint(&mut machine, case);
+    report["state"] = state(&machine, &stats);
+    let reason = match outcome {
+        Outcome::Checkpoint => {
+            let checks: Vec<_> = case
+                .checks
+                .iter()
+                .map(|assertion| check(&machine, assertion))
+                .collect();
+            let passed = checks.iter().all(|check| check["passed"] == true);
+            report["passed"] = json!(passed);
+            report["checks"] = json!(checks);
+            if passed {
+                "checkpoint"
+            } else {
+                "assertion_failed"
+            }
+        }
+        Outcome::StepLimit => "step_limit",
+        Outcome::Stopped => "stopped",
+        Outcome::EmulationError(error) => {
+            report["error"] = json!(error);
+            "emulation_error"
+        }
+    };
+    report["reason"] = json!(reason);
+    report
+}
+
+fn write_json_report(writer: &mut impl Write, report: &Value) -> Result<(), Box<dyn Error>> {
+    serde_json::to_writer_pretty(&mut *writer, report)?;
+    writeln!(writer)?;
+    writer.flush()?;
+    Ok(())
+}
+
+pub fn execute(path: &Path, writer: &mut impl Write) -> Result<(), Box<dyn Error>> {
+    let suite = read_suite(path)
+        .map_err(|error| io::Error::other(format!("cannot read test suite {path:?}: {error}")))?;
+    let base = path.parent().unwrap_or_else(|| Path::new("."));
+    let cases: Vec<_> = suite
+        .cases
+        .iter()
+        .map(|case| run_case(case, base))
+        .collect();
+    let failed = cases.iter().filter(|case| case["passed"] != true).count();
+    let report = json!({
+        "format_version": 1, "bios": "original", "passed": failed == 0,
+        "case_count": cases.len(), "failed_count": failed, "cases": cases,
+    });
+    write_json_report(writer, &report)?;
+    if failed != 0 {
+        return Err(io::Error::other(format!("ROM test suite: {failed} case(s) failed")).into());
+    }
+    Ok(())
+}
+
+#[cfg(test)]
+mod tests;

crates/desktop/src/cartridge/suite/tests.rs

@@ -0,0 +1,346 @@
+use super::*;
+use gba_core::{
+    dma::DMA_BASE,
+    io::{IE, IME, TIMER_BASE},
+    memory::ROM_START,
+};
+
+fn case_value() -> Value {
+    json!({"name": "original", "rom": "original.gba", "step_limit": 100,
+        "completion": {"pc": "0x08000004", "instruction_set": "arm"},
+        "checks": [{"kind": "register", "index": 0, "equals": "0x2a"}]})
+}
+fn case() -> Case {
+    serde_json::from_value(case_value()).unwrap()
+}
+fn suite(value: Value) -> Result<Suite, Box<dyn Error>> {
+    let suite: Suite = serde_json::from_value(value)?;
+    suite.validate()?;
+    Ok(suite)
+}
+fn boot(words: &[u32]) -> Machine {
+    bios::boot(words.iter().flat_map(|word| word.to_le_bytes()).collect()).unwrap()
+}
+
+#[test]
+fn suite_requires_one_path_and_no_other_options() {
+    for args in [
+        vec![],
+        vec!["--test-suite"],
+        vec!["--test-suite", ""],
+        vec!["--test-suite", "a.json", "--help"],
+        vec!["--rom", "a.json"],
+    ] {
+        assert!(parse_args(&args.into_iter().map(OsString::from).collect::<Vec<_>>()).is_err());
+    }
+    assert_eq!(
+        parse_args(&["--test-suite".into(), "space λ.json".into()]).unwrap(),
+        PathBuf::from("space λ.json")
+    );
+}
+
+#[test]
+fn word_values_accept_u32_numbers_decimal_strings_and_hex() {
+    for value in [
+        json!(42),
+        json!("42"),
+        json!("0x2a"),
+        json!("0x2A"),
+        json!(4294967295u32),
+        json!("0xffffffff"),
+    ] {
+        let mut input = case_value();
+        input["checks"][0]["equals"] = value;
+        assert!(serde_json::from_value::<Case>(input).is_ok());
+    }
+    for value in [
+        json!(-1),
+        json!(42.0),
+        json!(4294967296u64),
+        json!(null),
+        json!(true),
+        json!(""),
+        json!("-1"),
+        json!("+1"),
+        json!(" 1"),
+        json!("1 "),
+        json!("0x"),
+        json!("0x100000000"),
+        json!("0xgg"),
+        json!("1.0"),
+    ] {
+        let mut input = case_value();
+        input["checks"][0]["equals"] = value.clone();
+        assert!(serde_json::from_value::<Case>(input).is_err(), "{value}");
+    }
+}
+
+#[test]
+fn unknown_fields_and_unknown_check_kinds_are_rejected() {
+    for pointer in ["", "/cases/0", "/cases/0/completion", "/cases/0/checks/0"] {
+        let mut value = json!({"version": 1, "cases": [case_value()]});
+        value.pointer_mut(pointer).unwrap()["typo"] = json!(1);
+        assert!(suite(value).is_err());
+    }
+    let mut value = case_value();
+    value["checks"][0]["kind"] = json!("memory8");
+    assert!(suite(json!({"version": 1, "cases": [value]})).is_err());
+    assert!(serde_json::from_str::<Suite>(r#"{"version":1,"version":1,"cases":[]}"#).is_err());
+}
+
+#[test]
+fn manifest_rejects_empty_duplicate_excessive_or_invalid_cases() {
+    assert!(suite(json!({"version": 2, "cases": [case_value()]})).is_err());
+    assert!(suite(json!({"version": 1, "cases": []})).is_err());
+    assert!(suite(json!({"version": 1, "cases": [case_value(), case_value()]})).is_err());
+    for (field, invalid) in [
+        ("name", json!(" ")),
+        ("name", json!("x".repeat(129))),
+        ("rom", json!("")),
+        ("step_limit", json!(0)),
+        ("step_limit", json!(100000001u64)),
+        ("checks", json!([])),
+    ] {
+        let mut value = case_value();
+        value[field] = invalid;
+        assert!(suite(json!({"version": 1, "cases": [value]})).is_err());
+    }
+    let cases: Vec<_> = (0..257)
+        .map(|index| {
+            let mut value = case_value();
+            value["name"] = json!(format!("test-{index}"));
+            value
+        })
+        .collect();
+    assert!(suite(json!({"version": 1, "cases": cases})).is_err());
+    let mut first = case_value();
+    first["step_limit"] = json!(MAX_STEPS);
+    assert!(suite(json!({"version": 1, "cases": [first.clone()]})).is_ok());
+    let mut second = case_value();
+    second["name"] = json!("second");
+    assert!(suite(json!({"version": 1, "cases": [first, second]})).is_err());
+}
+
+#[test]
+fn manifest_checks_alignment_register_bounds_and_check_count() {
+    for (pc, state, valid) in [
+        ("0x08000002", "arm", false),
+        ("0x08000002", "thumb", true),
+        ("0x08000003", "thumb", false),
+        ("0x08000004", "invalid", false),
+    ] {
+        let mut value = case_value();
+        value["completion"] = json!({"pc": pc, "instruction_set": state});
+        assert_eq!(
+            suite(json!({"version": 1, "cases": [value]})).is_ok(),
+            valid
+        );
+    }
+    for check in [
+        json!({"kind": "register", "index":16,"equals":0}),
+        json!({"kind":"memory32","address":"0x02000001","equals":0}),
+    ] {
+        let mut value = case_value();
+        value["checks"] = json!([check]);
+        assert!(suite(json!({"version": 1, "cases": [value]})).is_err());
+    }
+    let mut value = case_value();
+    value["checks"] = json!(vec![value["checks"][0].clone(); MAX_CHECKS + 1]);
+    assert!(suite(json!({"version": 1, "cases": [value]})).is_err());
+}
+
+#[test]
+fn json_report_propagates_write_and_flush_failures() {
+    struct Fails {
+        on_flush: bool,
+    }
+    impl Write for Fails {
+        fn write(&mut self, bytes: &[u8]) -> io::Result<usize> {
+            if self.on_flush {
+                Ok(bytes.len())
+            } else {
+                Err(io::Error::new(io::ErrorKind::BrokenPipe, "write failure"))
+            }
+        }
+        fn flush(&mut self) -> io::Result<()> {
+            Err(io::Error::other("flush failure"))
+        }
+    }
+    let report = json!({"passed":true});
+    for on_flush in [false, true] {
+        let error = write_json_report(&mut Fails { on_flush }, &report).unwrap_err();
+        assert!(error.to_string().contains(if on_flush {
+            "flush failure"
+        } else {
+            "write failure"
+        }));
+    }
+    let mut output = Vec::new();
+    write_json_report(&mut output, &report).unwrap();
+    assert_eq!(serde_json::from_slice::<Value>(&output).unwrap(), report);
+    assert_eq!(output.last(), Some(&b'\n'));
+}
+
+#[test]
+fn reset_pc_is_not_an_immediate_pass_without_execution() {
+    let mut config = case();
+    config.completion.pc = 0;
+    config.step_limit = 1;
+    let (stats, outcome) = run_to_checkpoint(&mut boot(&[0xeaff_fffe]), &config);
+    assert_eq!(stats.steps, 1);
+    assert_eq!(outcome, Outcome::StepLimit);
+}
+
+#[test]
+fn checkpoint_is_checked_before_execution_and_on_the_last_budgeted_step() {
+    let code = [0xe3a0_002a, 0xee00_0000]; // MOV r0,#42; unsupported (never fetched)
+    let mut machine = boot(&code);
+    let (stats, outcome) = run_to_checkpoint(&mut machine, &case());
+    assert_eq!(outcome, Outcome::Checkpoint);
+    assert!(stats.steps > 1); // BIOS boot is included.
+    assert_eq!(check(&machine, &case().checks[0])["passed"], true);
+    let mut exact = case();
+    exact.step_limit = stats.steps;
+    assert_eq!(
+        run_to_checkpoint(&mut boot(&code), &exact).1,
+        Outcome::Checkpoint
+    );
+    exact.step_limit -= 1;
+    assert_eq!(
+        run_to_checkpoint(&mut boot(&code), &exact).1,
+        Outcome::StepLimit
+    );
+}
+
+#[test]
+fn matching_assertions_do_not_pass_without_the_checkpoint() {
+    let mut config = case();
+    config.completion.pc = ROM_START + 8;
+    let mut machine = boot(&[0xe3a0_002a, 0xeaff_fffe]);
+    let (stats, outcome) = run_to_checkpoint(&mut machine, &config);
+    assert_eq!(outcome, Outcome::StepLimit);
+    assert_eq!(stats.steps, config.step_limit);
+    assert_eq!(check(&machine, &config.checks[0])["passed"], true);
+    config.completion.pc = ROM_START + 4;
+    config.completion.instruction_set = State::Thumb;
+    assert_eq!(
+        run_to_checkpoint(&mut boot(&[0xe3a0_002a, 0xeaff_fffe]), &config).1,
+        Outcome::StepLimit
+    );
+}
+
+#[test]
+fn thumb_checkpoint_uses_aligned_architectural_pc() {
+    let mut bytes: Vec<u8> = [0xe59f_1000u32, 0xe12f_ff11, ROM_START + 13]
+        .iter()
+        .flat_map(|word| word.to_le_bytes())
+        .collect();
+    bytes.extend([0x2a, 0x20, 0xfe, 0xe7]);
+    let mut machine = bios::boot(bytes).unwrap();
+    let mut config = case();
+    config.completion.pc = ROM_START + 14;
+    config.completion.instruction_set = State::Thumb;
+    assert_eq!(
+        run_to_checkpoint(&mut machine, &config).1,
+        Outcome::Checkpoint
+    );
+    assert_eq!(check(&machine, &config.checks[0])["passed"], true);
+}
+
+#[test]
+fn memory_and_cpsr_checks_report_actual_values_and_read_errors() {
+    let mut config = case();
+    config.completion.pc = ROM_START + 12;
+    let mut machine = boot(&[0xe3a0_002a, 0xe3a0_1402, 0xe581_0000, 0xeaff_fffe]);
+    assert_eq!(
+        run_to_checkpoint(&mut machine, &config).1,
+        Outcome::Checkpoint
+    );
+    for assertion in [
+        Check::Memory32 {
+            address: 0x0200_0000,
+            equals: 42,
+        },
+        Check::Cpsr { equals: 0x1f },
+    ] {
+        let result = check(&machine, &assertion);
+        assert_eq!(result["passed"], true);
+        assert!(result["error"].is_null());
+    }
+    let wrong = check(
+        &machine,
+        &Check::Register {
+            index: 0,
+            equals: 43,
+        },
+    );
+    assert_eq!(wrong["passed"], false);
+    assert_eq!(wrong["actual"], 42);
+    let unmapped = check(
+        &machine,
+        &Check::Memory32 {
+            address: 0x0100_0000,
+            equals: 0,
+        },
+    );
+    assert_eq!(unmapped["passed"], false);
+    assert!(unmapped["actual"].is_null());
+    assert!(unmapped["error"].is_string());
+}
+
+#[test]
+fn stop_is_failure_while_halt_consumes_a_bounded_budget() {
+    let mut config = case();
+    config.completion.pc = ROM_START + 8;
+    let (stats, outcome) = run_to_checkpoint(&mut boot(&[0xef03_0000, 0xeaff_fffe]), &config);
+    assert_eq!(outcome, Outcome::Stopped);
+    assert!(stats.steps < config.step_limit);
+    let (stats, outcome) = run_to_checkpoint(&mut boot(&[0xef02_0000, 0xeaff_fffe]), &config);
+    assert_eq!(outcome, Outcome::StepLimit);
+    assert_eq!(stats.steps, config.step_limit);
+    assert!(stats.halt_idle > 0);
+}
+
+#[test]
+fn failing_instructions_do_not_advance_the_reported_state() {
+    let mut machine = boot(&[0xee00_0000]);
+    let (stats, outcome) = run_to_checkpoint(&mut machine, &case());
+    assert!(matches!(outcome, Outcome::EmulationError(_)));
+    assert!(stats.steps < case().step_limit);
+    assert_eq!(machine.cpu().pc(), ROM_START);
+    let cpu = machine.cpu().clone();
+    let cycles = machine.cycles();
+    let (retry, outcome) = run_to_checkpoint(&mut machine, &case());
+    assert_eq!(retry.steps, 0);
+    assert!(matches!(outcome, Outcome::EmulationError(_)));
+    assert_eq!(machine.cpu(), &cpu);
+    assert_eq!(machine.cycles(), cycles);
+}
+
+#[test]
+fn dma_and_irq_each_consume_a_budget_step_and_dma_errors_fail() {
+    let mut machine = boot(&[0xe3a0_002a, 0xeaff_fffe]);
+    run_to_checkpoint(&mut machine, &case());
+    let mut config = case();
+    config.completion.pc = ROM_START + 8;
+    config.step_limit = 3;
+    let bus = machine.memory_mut();
+    bus.write32(DMA_BASE, 0x0200_0000).unwrap();
+    bus.write32(DMA_BASE + 4, 0x0300_0000).unwrap();
+    bus.write32(DMA_BASE + 8, 0x8000_0002).unwrap();
+    bus.write16(IE, 8).unwrap();
+    bus.write16(IME, 1).unwrap();
+    bus.write32(TIMER_BASE, 0x00c0_ffff).unwrap();
+    bus.advance_cycles(1);
+    let (stats, outcome) = run_to_checkpoint(&mut machine, &config);
+    assert_eq!(outcome, Outcome::StepLimit);
+    assert_eq!(stats.dma_units, 2);
+    assert_eq!(stats.irq_entries, 1);
+    assert_eq!(stats.instructions, 0);
+    let bus = machine.memory_mut();
+    bus.write32(DMA_BASE + 4, ROM_START).unwrap();
+    bus.write32(DMA_BASE + 8, 0x8000_0001).unwrap();
+    let (stats, outcome) = run_to_checkpoint(&mut machine, &config);
+    assert_eq!(stats.steps, 0);
+    assert!(matches!(outcome, Outcome::EmulationError(_)));
+}

crates/desktop/src/main.rs

@@ -1,7 +1,7 @@
 mod cartridge;
 mod desktop;
 
-use std::{error::Error, ffi::OsString, io, process::ExitCode};
+use std::{error::Error, ffi::OsString, io, path::PathBuf, process::ExitCode};
 
 use gba_core::{
     cpu::Cpu,
@@ -19,6 +19,7 @@ use gba_demos::{
 enum RunMode {
     Window,
     Rom(cartridge::Options),
+    TestSuite(PathBuf),
     SmokeTest,
     CpuDemo,
     TimerDemo,
@@ -42,6 +43,9 @@ enum RunMode {
 }
 
 fn parse_args(args: &[OsString]) -> Result<RunMode, io::Error> {
+    if args.first().is_some_and(|arg| arg == "--test-suite") {
+        return cartridge::suite::parse_args(args).map(RunMode::TestSuite);
+    }
     if args.iter().any(|arg| {
         ["--rom", "--steps", "--window", "--frames"]
             .iter()
@@ -75,7 +79,7 @@ fn parse_args(args: &[OsString]) -> Result<RunMode, io::Error> {
         [arg] if arg == "--help" || arg == "-h" => Ok(RunMode::Help),
         _ => Err(io::Error::new(
             io::ErrorKind::InvalidInput,
-            "usage: gameboy-rust [--cpu-demo | --timer-demo | --graphics-demo | --smoke-test | --graphics-smoke-test | --raster-demo | --raster-smoke-test | --mosaic-demo | --mosaic-smoke-test | --effects-demo | --effects-smoke-test | --tile-demo | --tile-smoke-test | --affine-demo | --affine-smoke-test | --affine-raster-demo | --affine-raster-smoke-test | --bitmap4-demo | --bitmap5-demo | --bitmap4-smoke-test | --bitmap5-smoke-test | --help]; or gameboy-rust --rom PATH (--steps COUNT | --window [--frames COUNT])",
+            "usage: gameboy-rust [--cpu-demo | --timer-demo | --graphics-demo | --smoke-test | --graphics-smoke-test | --raster-demo | --raster-smoke-test | --mosaic-demo | --mosaic-smoke-test | --effects-demo | --effects-smoke-test | --tile-demo | --tile-smoke-test | --affine-demo | --affine-smoke-test | --affine-raster-demo | --affine-raster-smoke-test | --bitmap4-demo | --bitmap5-demo | --bitmap4-smoke-test | --bitmap5-smoke-test | --help]; or gameboy-rust --rom PATH (--steps COUNT | --window [--frames COUNT]); or gameboy-rust --test-suite PATH.json",
         )),
     }
 }
@@ -94,6 +98,7 @@ fn run() -> Result<(), Box<dyn Error>> {
     match parse_args(&std::env::args_os().skip(1).collect::<Vec<_>>())? {
         RunMode::Window => desktop::run(None),
         RunMode::Rom(options) => cartridge::execute(options, &mut io::stdout().lock()),
+        RunMode::TestSuite(path) => cartridge::suite::execute(&path, &mut io::stdout().lock()),
         RunMode::SmokeTest => desktop::run(Some(60)),
         RunMode::CpuDemo => run_cpu_demo(),
         RunMode::TimerDemo => run_timer_demo(),
@@ -120,6 +125,7 @@ fn run() -> Result<(), Box<dyn Error>> {
                 --rom PATH --steps COUNT  Run a raw ROM in the terminal (1..=100000000 steps)\n\
                 --rom PATH --window       Open a raw ROM with keyboard input\n\
                 --frames COUNT            Exit ROM window after 1..=100000 captured frames\n\
+                --test-suite PATH.json    Run bounded ROM assertions; write a JSON report\n\
                 --cpu-demo     Run the terminal-only ARM/Thumb instruction demo\n\
                 --timer-demo   Run the timer IRQ demo with nominal cycle costs\n\
                 --graphics-demo        Open the CPU-driven Mode 3 demo\n\

crates/desktop/tests/rom_suite_cli.rs

@@ -0,0 +1,226 @@
+//! File-backed suite tests. All ROM instructions are original and temporary.
+use serde_json::{json, Value};
+use std::{
+    fs::{self, File},
+    path::{Path, PathBuf},
+    process::{Command, Output},
+    sync::atomic::{AtomicUsize, Ordering},
+};
+
+struct Fixture(PathBuf);
+impl Fixture {
+    fn new() -> Self {
+        static NEXT: AtomicUsize = AtomicUsize::new(0);
+        let path = std::env::temp_dir().join(format!(
+            "gba-suite-{}-{}",
+            std::process::id(),
+            NEXT.fetch_add(1, Ordering::Relaxed)
+        ));
+        fs::create_dir(&path).unwrap();
+        Self(path)
+    }
+    fn rom(&self, name: &str, code: &[u32]) {
+        fs::write(
+            self.0.join(name),
+            code.iter()
+                .flat_map(|word| word.to_le_bytes())
+                .collect::<Vec<_>>(),
+        )
+        .unwrap();
+    }
+    fn manifest(&self, cases: Vec<Value>) -> PathBuf {
+        let path = self.0.join("suite λ.json");
+        fs::write(
+            &path,
+            serde_json::to_vec(&json!({"version":1,"cases":cases})).unwrap(),
+        )
+        .unwrap();
+        path
+    }
+}
+impl Drop for Fixture {
+    fn drop(&mut self) {
+        let _ = fs::remove_dir_all(&self.0);
+    }
+}
+fn case(name: &str, rom: &str) -> Value {
+    json!({"name":name,"rom":rom,"step_limit":100,
+        "completion":{"pc":"0x08000004","instruction_set":"arm"},
+        "checks":[{"kind":"register","index":0,"equals":42}]})
+}
+fn command() -> Command {
+    Command::new(env!("CARGO_BIN_EXE_gameboy-rust"))
+}
+fn run(path: &Path) -> Output {
+    command()
+        .arg("--test-suite")
+        .arg(path)
+        .current_dir(std::env::temp_dir())
+        .output()
+        .unwrap()
+}
+fn stderr(output: &Output) -> String {
+    String::from_utf8(output.stderr.clone()).unwrap()
+}
+fn report(output: &Output) -> Value {
+    serde_json::from_slice(&output.stdout).unwrap_or_else(|error| {
+        panic!(
+            "invalid JSON: {error}; stdout={:?}; stderr={}",
+            output.stdout,
+            stderr(output)
+        )
+    })
+}
+
+#[test]
+fn relative_paths_resolve_from_suite_and_success_reports_are_repeatable() {
+    let fixture = Fixture::new();
+    fixture.rom("original λ.gba", &[0xe3a0_002a, 0xee00_0000]);
+    let path = fixture.manifest(vec![case("original \"ARM\"\ncase", "original λ.gba")]);
+    let first = run(&path);
+    let second = run(&path);
+    assert!(first.status.success(), "{}", stderr(&first));
+    assert!(first.stderr.is_empty());
+    assert_eq!(first.stdout, second.stdout);
+    let report = report(&first);
+    assert_eq!(report["format_version"], 1);
+    assert_eq!(report["bios"], "original");
+    assert_eq!(report["passed"], true);
+    assert_eq!(report["failed_count"], 0);
+    let result = &report["cases"][0];
+    assert_eq!(result["reason"], "checkpoint");
+    assert_eq!(result["name"], "original \"ARM\"\ncase");
+    assert_eq!(result["rom_bytes"], 8);
+    assert_eq!(result["state"]["pc"], 0x0800_0004u32);
+    assert_eq!(result["state"]["instruction_set"], "arm");
+    assert_eq!(result["state"]["registers"].as_array().unwrap().len(), 16);
+    assert_eq!(result["checks"][0]["actual"], 42);
+}
+
+#[test]
+fn failures_do_not_skip_later_cases_and_never_count_as_passes() {
+    let fixture = Fixture::new();
+    fixture.rom("valid.gba", &[0xe3a0_002a, 0xeaff_fffe]);
+    fixture.rom("invalid.gba", &[0xee00_0000]);
+    fixture.rom("stop.gba", &[0xef03_0000, 0xeaff_fffe]);
+    let mut mismatch = case("mismatch", "valid.gba");
+    mismatch["checks"][0]["equals"] = json!(43);
+    let mut timeout = case("timeout", "valid.gba");
+    timeout["completion"]["pc"] = json!("0x08000008");
+    let mut bad_read = case("read-error", "valid.gba");
+    bad_read["checks"] = json!([{"kind":"memory32","address":"0x01000000","equals":0}]);
+    let path = fixture.manifest(vec![
+        mismatch,
+        timeout,
+        case("diagnostic", "invalid.gba"),
+        case("stop", "stop.gba"),
+        case("missing", "missing.gba"),
+        bad_read,
+        case("passes-last", "valid.gba"),
+    ]);
+    let output = run(&path);
+    assert_eq!(output.status.code(), Some(1));
+    let report = report(&output);
+    assert_eq!(report["case_count"], 7);
+    assert_eq!(report["failed_count"], 6);
+    assert_eq!(report["passed"], false);
+    for (index, reason) in [
+        "assertion_failed",
+        "step_limit",
+        "emulation_error",
+        "stopped",
+        "load_error",
+        "assertion_failed",
+        "checkpoint",
+    ]
+    .iter()
+    .enumerate()
+    {
+        let result = &report["cases"][index];
+        assert_eq!(result["reason"], *reason);
+        assert_eq!(result["passed"], index == 6);
+    }
+    assert_eq!(report["cases"][0]["checks"][0]["actual"], 42);
+    assert_eq!(report["cases"][1]["state"]["steps"], 100);
+    assert_eq!(report["cases"][1]["checks"], json!([]));
+    assert_eq!(report["cases"][2]["state"]["pc"], 0x0800_0000u32);
+    assert_eq!(report["cases"][3]["state"]["stopped"], true);
+    assert!(report["cases"][4]["state"].is_null());
+    assert!(report["cases"][5]["checks"][0]["error"].is_string());
+    assert!(stderr(&output).contains("6 case(s) failed"));
+}
+
+#[test]
+fn every_case_starts_with_fresh_machine_state() {
+    let fixture = Fixture::new();
+    fixture.rom(
+        "write.gba",
+        &[0xe3a0_002a, 0xe3a0_1402, 0xe581_0000, 0xeaff_fffe],
+    );
+    fixture.rom("fresh.gba", &[0xe3a0_002a, 0xeaff_fffe]);
+    let mut writer = case("write", "write.gba");
+    writer["completion"]["pc"] = json!("0x0800000c");
+    writer["checks"] = json!([{"kind":"memory32","address":"0x02000000","equals":42}]);
+    let mut fresh = case("fresh", "fresh.gba");
+    fresh["checks"] = json!([{"kind":"memory32","address":"0x02000000","equals":0}, {"kind":"cpsr","equals":"0x1f"}]);
+    let output = run(&fixture.manifest(vec![writer, fresh]));
+    assert!(output.status.success(), "{}", stderr(&output));
+    assert_eq!(report(&output)["failed_count"], 0);
+}
+
+#[test]
+fn malformed_or_oversized_suites_fail_before_any_case_execution() {
+    let fixture = Fixture::new();
+    let path = fixture.0.join("invalid.json");
+    for bytes in [
+        b"".as_slice(),
+        b"{",
+        br#"{"version":1,"cases":[]}"#,
+        br#"{"version":2,"cases":[]}"#,
+    ] {
+        fs::write(&path, bytes).unwrap();
+        let output = run(&path);
+        assert_eq!(output.status.code(), Some(1));
+        assert!(output.stdout.is_empty());
+        assert!(stderr(&output).contains("cannot read test suite"));
+    }
+    File::create(&path)
+        .unwrap()
+        .set_len(1024 * 1024 + 1)
+        .unwrap();
+    let output = run(&path);
+    assert_eq!(output.status.code(), Some(1));
+    assert!(stderr(&output).contains("1 MiB"));
+    for path in [&fixture.0, &fixture.0.join("missing.json")] {
+        let output = run(path);
+        assert_eq!(output.status.code(), Some(1));
+        assert!(output.stdout.is_empty());
+    }
+    fixture.rom("valid.gba", &[0xe3a0_002a, 0xeaff_fffe]);
+    let mut invalid = case("invalid-later", "valid.gba");
+    invalid["checks"] = json!([]);
+    let output = run(&fixture.manifest(vec![case("would-pass", "valid.gba"), invalid]));
+    assert_eq!(output.status.code(), Some(1));
+    assert!(output.stdout.is_empty());
+}
+
+#[test]
+fn suite_mode_rejects_other_modes_and_help_advertises_it() {
+    for args in [
+        vec!["--test-suite"],
+        vec!["--test-suite", "missing.json", "--steps", "10"],
+        vec!["--test-suite", "missing.json", "--window"],
+        vec!["--test-suite", "missing.json", "--cpu-demo"],
+        vec!["--test-suite", "missing.json", "--help"],
+    ] {
+        let output = command().args(args).output().unwrap();
+        assert_eq!(output.status.code(), Some(1));
+        assert!(output.stdout.is_empty());
+        assert!(stderr(&output).contains("usage:"));
+    }
+    let output = command().arg("--help").output().unwrap();
+    assert!(output.status.success());
+    assert!(String::from_utf8(output.stdout)
+        .unwrap()
+        .contains("--test-suite PATH.json"));
+}

docs/architecture.md

@@ -5,6 +5,7 @@ The repository is a Cargo workspace with three crates:
 - `crates/gba-core`: the platform-independent GBA emulator. It has no dependencies and must not depend on window, audio, or file-system libraries.
 - `crates/gba-demos`: original ARM test programs and their frame runners. Depends on `gba-core`.
 - `crates/desktop`: the minifb window frontend and the `gameboy-rust` executable. Depends on both.
+  Serde and serde_json handle host-side test manifests and reports; neither dependency enters the core.
 
 `gba-core` uses `gba-demos` as a development dependency, because its integration tests run the demo programs.
 
@@ -80,6 +81,10 @@ The repository is a Cargo workspace with three crates:
 | `crates/desktop/src/desktop/raster.rs` | Raster window and scanline color-band smoke test |
 | `crates/desktop/src/main.rs` | Command-line modes, error exit status, and fixed-length CPU demo |
 | `crates/desktop/src/cartridge.rs` | Read-only ROM files, mode/limit checks, bounded terminal execution, and shared state reports |
+| `crates/desktop/src/cartridge/suite.rs` | Strict JSON manifests, bounded checkpoint execution, assertions, and structured reports |
+| `crates/desktop/src/cartridge/suite/tests.rs` | Manifest limits, checkpoint boundaries, memory/register checks, and diagnostic accounting |
+| `crates/desktop/tests/rom_suite_cli.rs` | File-backed suite execution, case isolation, output schema, determinism, and exit status |
+| `crates/desktop/examples/write_test_suite.rs` | Original ARM/Thumb/BIOS fixture generator with a local JSON suite |
 | `crates/desktop/src/cartridge/window.rs` | ROM-window session, bounded slices, capture, STOP/input handling, and progress limits |
 | `crates/desktop/src/cartridge/window/tests.rs` | Original ROM pixels, keyboard mapping, STOP wake, HALT, frame budgets, and diagnostics |
 | `crates/desktop/src/desktop/rom.rs` | Main-thread ROM window, input polling, presentation pacing, and frame-limited exit |

docs/development.md

@@ -19,6 +19,7 @@ Verified locally on macOS 15.7.3:
 - The terminal-only CPU and timer IRQ demos run successfully.
 - The bounded terminal ROM runner loads original raw programs and reports state without opening a window.
 - The native ROM-window test presents a file-backed original program and reports STOP, CPU, and video failures.
+- The headless suite runner passes generated ARM/Thumb/BIOS cases, produces repeatable JSON, and rejects deliberate assertion mismatches.
 
 These results cover the CPU core, memory, Mode 0–5 snapshots and row capture, input mapping, and desktop windows.
 They do not verify a complete GBA display controller or audio.

docs/hardware/cartridge.md

@@ -2,6 +2,7 @@
 
 The desktop executable can load raw ROM bytes for terminal diagnostics or window execution.
 This supports original test programs. It does not establish commercial-game compatibility.
+For explicit pass/fail assertions instead of diagnostic limits, use the [headless ROM suite runner](../rom-tests.md).
 
 ```sh
 direnv exec . cargo run --locked --release -- --rom path/to/original.gba --steps 100000

docs/rom-tests.md

@@ -0,0 +1,136 @@
+# Headless ROM test suites
+
+Use `--test-suite PATH.json` to run repeatable ROM checks without opening a window.
+Each case has a machine-step budget, a completion address, and explicit assertions.
+A timeout never counts as a pass. Existing `--rom ... --steps ...` remains a diagnostic mode, not a test verdict.
+
+This runner currently has regression coverage from original programs, not independent public hardware-test results.
+A passing suite establishes only its specified checkpoints and assertions. It does not establish Emerald compatibility.
+
+## Run the original smoke suite
+
+```sh
+direnv exec . cargo run --locked --example write_test_suite -- /tmp/gba-original-tests
+direnv exec . cargo run --locked --release -- --test-suite /tmp/gba-original-tests/suite.json > /tmp/gba-test-report.json
+```
+
+The generator creates original ARM, Thumb, and BIOS-division programs, plus a JSON manifest.
+The destination directory must not already exist. No Nintendo code, firmware, or game assets are included.
+Generated binaries stay outside the repository.
+The runner writes one JSON report to stdout. Cargo build messages and errors go to stderr.
+Do not redirect the report over the manifest or a ROM file.
+
+## Manifest version 1
+
+```json
+{
+  "version": 1,
+  "cases": [
+    {
+      "name": "arm-store",
+      "rom": "arm.gba",
+      "step_limit": 10000,
+      "completion": {
+        "pc": "0x0800000c",
+        "instruction_set": "arm"
+      },
+      "checks": [
+        { "kind": "register", "index": 0, "equals": 42 },
+        { "kind": "memory32", "address": "0x02000000", "equals": 42 },
+        { "kind": "cpsr", "equals": "0x1f" }
+      ]
+    }
+  ]
+}
+```
+
+This example matches the generated `arm.gba`, not arbitrary ROMs.
+The program stores 42 to external RAM, then reaches an ARM self-branch at `0x0800000c`.
+
+Rules:
+
+- All shown fields are required. Unknown fields and unsupported check kinds are rejected.
+- The manifest must be a regular UTF-8 JSON file, no larger than 1 MiB.
+- Supply 1–256 cases. Names must be unique, nonempty after trimming, and at most 128 UTF-8 bytes.
+- Each case needs 1–256 checks and a nonempty ROM path.
+- Relative ROM paths resolve from the manifest's directory, not the current working directory. Absolute paths are accepted.
+- Each step limit is a JSON integer from 1 through 100,000,000. Their sum must not exceed 100,000,000.
+- Addresses and expected values accept unsigned 32-bit JSON integers, decimal strings, or lowercase `0x`-prefixed hexadecimal strings.
+- Signed values, floating-point values, surrounding whitespace, and overflow are rejected. Encode negative bit patterns as unsigned words.
+- `instruction_set` is exactly `arm` or `thumb`. ARM completion addresses must be word-aligned; Thumb addresses must be halfword-aligned.
+- Use the actual Thumb instruction address, without the address bit used by `BX`.
+- Register indices are integers from 0 through 15. Checks read the active register bank.
+- `memory32` addresses must be word-aligned. Reads use the existing little-endian memory bus, including its mapping errors.
+- `cpsr` compares the complete current program status register. No mask is applied.
+
+The runner validates the whole manifest before opening or executing any ROM.
+It uses the same read-only [ROM loader](hardware/cartridge.md#loading-and-boot), with its 4-byte minimum and 32 MiB maximum.
+The CLI accepts no other options with `--test-suite`.
+
+## Completion and failure semantics
+
+Each case starts with fresh RAM, devices, CPU state, and the original BIOS replacement.
+The budget includes BIOS boot, instructions, DMA units, IRQ entries, and HALT idle batches.
+A failed CPU or DMA step does not consume a successful step or advance the reported state.
+The runner supplies no input and does not enable frame capture. Device clocks still advance through normal machine stepping.
+
+After successful progress, the runner checks PC and instruction set at machine-step boundaries.
+It stops before executing the instruction at the completion address.
+Reaching that address on the final allowed step succeeds if all checks pass.
+The initial reset state alone cannot pass a case. STOP always ends the case as a failure because no input source can wake it.
+HALT continues advancing devices until completion, a diagnostic, or budget exhaustion.
+
+The checkpoint does not imply that pending DMA or IRQ work has completed.
+Choose an address reached only after the tested operations finish.
+A loop address reached before those operations would define an invalid test, even if the runner reports a pass.
+The runner does not infer program completion from an infinite loop, a screen, or matching register values alone.
+
+All checks run at the checkpoint. A mismatch or unreadable memory address fails the case.
+Cases that never reach the checkpoint do not evaluate their checks.
+A failure does not skip later cases. No machine state passes from one case to another.
+
+The budget bounds machine steps, not elapsed host time. Use an external process timeout in continuous integration if needed.
+This is not a filesystem security sandbox. Run trusted suite configurations and use files you may lawfully use.
+
+## Report version 1 and exit status
+
+Top-level fields:
+
+| Field | Meaning |
+| --- | --- |
+| `format_version` | Report schema version, currently 1 |
+| `bios` | `original`; no external BIOS option |
+| `passed` | True only when every case passed |
+| `case_count`, `failed_count` | Number of executed cases and failures |
+| `cases` | Results in manifest order |
+
+Each case records its name, ROM path, debug-formatted resolved path, byte count, budget, completion condition, and result.
+`state` contains registers r0–r15, PC, CPSR, mode, instruction set, HALT/STOP flags, nominal cycles, and step counters.
+Addresses, registers, expected values, and counters in reports are JSON numbers, even when the manifest used hexadecimal strings.
+A load failure has null `state` and `rom_bytes`.
+`checks` contains the configured check, actual value, pass flag, and optional read error when the checkpoint was reached.
+Otherwise `checks` is empty. A failed read has a null actual value.
+
+Case reasons are:
+
+- `checkpoint`: completion reached and all assertions passed.
+- `assertion_failed`: completion reached but at least one assertion failed or could not read memory.
+- `step_limit`: completion was not reached within the budget.
+- `stopped`: STOP prevented further execution.
+- `emulation_error`: an instruction or DMA diagnostic stopped execution; `error` contains its message.
+- `load_error`: loading or machine creation failed; `error` contains its message.
+
+Exit status 0 means all cases passed and the report was written successfully.
+Any failed case, invalid configuration, or output failure returns a nonzero status.
+Invalid manifests produce a stderr diagnostic without a JSON report. A broken output stream can leave an incomplete report.
+Valid suites produce a report even when some ROMs fail to load.
+Reports omit host timestamps and durations so unchanged files produce repeatable output on the same host.
+They do not hash ROM contents; record source revisions and file checksums alongside external test results.
+
+## Next validation stage
+
+Select public ARM7TDMI/GBA tests and inspect their source, license, entry assumptions, and result protocol.
+Pin source revisions and build instructions before comparing emulator changes.
+Adapt suites to verified completion addresses and result locations; do not guess them or treat timeouts as passes.
+Some tests need debug-port logging, input, different firmware behavior, or rendered-image checks that this runner does not support yet.
+Keep external binaries local. Record consulted sources in [references.md](references.md).

docs/status.md

@@ -65,6 +65,7 @@ Update this file when a feature lands or a limit is removed.
 - Raw ROM-file loading with original BIOS boot, terminal diagnostics, and final CPU/step reports.
 - ROM windows with scanline capture, keyboard input, focus-loss release, bounded execution slices, and interactive keypad wake from STOP.
 - Optional ROM-window frame limits and an original input-test ROM generator.
+- Headless JSON ROM suites with per-case checkpoints, register/CPSR/memory assertions, bounded budgets, and structured failure reports.
 - Little-endian byte, halfword, and word reads/writes.
 - ARM7TDMI word-load rotation, aligned stores, and odd-address halfword-load behavior.
 - Explicit errors for unsupported instructions when their condition passes, and for unsupported memory accesses.
@@ -109,6 +110,8 @@ ROM files can run in bounded terminal mode or a window with the original BIOS re
 ROM windows have no demo-specific startup checks. Input is sampled between bounded execution slices.
 Audio, saves, and external BIOS loading remain unavailable. ROM loading does not establish commercial-game compatibility.
 See [cartridge loading](hardware/cartridge.md) for file limits, step budgets, and exit status.
+The [ROM suite runner](rom-tests.md) currently uses original regression programs, not independent public hardware-test results.
+It requires known completion addresses and has no debug-port protocol, scripted input, or rendered-image assertions.
 
 Unmapped reads return errors instead of hardware open-bus values.
 Reads beyond the supplied cartridge bytes also return errors.
@@ -118,7 +121,7 @@ Writes to cartridge addresses return errors instead of modeling cartridge hardwa
 
 ## Next steps
 
-1. Validate CPU behavior with public ARM7TDMI test programs before claiming instruction compatibility.
+1. Adapt the ROM suite runner to pinned public ARM7TDMI test programs and verified result protocols before claiming instruction compatibility.
 2. Refine nominal timing with a fetch pipeline, Game Pak prefetch, per-access device updates, and verified timer/IRQ delays.
 3. Validate keypad retrigger behavior with hardware tests and add remaining device registers; extend BIOS reset coverage as sound/serial support becomes available.
    Validate STOP entry/wake edges and add external wake sources and remaining DMA device modes.

docs/testing.md

@@ -46,6 +46,11 @@ The core's integration tests use `gba-demos` as a development dependency.
 On Apple Silicon, `file` must report a Mach-O `arm64` executable.
 Do not set a Linux cross-compilation target for this validation.
 
+The [headless ROM suite runner](rom-tests.md) provides explicit checkpoint and assertion results through `--test-suite PATH.json`.
+Its tests verify manifest limits, whole-suite validation, exact step boundaries, ARM/Thumb checkpoints, and register/CPSR/memory checks.
+Additional tests cover DMA/IRQ accounting, HALT/STOP, diagnostic state, case isolation, repeatable JSON output, and failure exit status.
+These original tests validate the runner; they are not independent public hardware-test results.
+
 ROM-runner tests use temporary files containing original instructions, without Nintendo assets or a desktop session.
 They check file-size boundaries, bounded reads, short reads, I/O errors, paths, symlinks, and read-only loading.
 CLI tests cover option order, missing/conflicting options, invalid step limits, process exit status, and reports.