195 lines
11 KiB
Markdown
195 lines
11 KiB
Markdown
# Headless ROM test suites
|
||
|
||
Use `--test-suite PATH.json` to run repeatable ROM checks without opening a window.
|
||
Each case has a machine-step budget, a completion condition, and explicit assertions.
|
||
A timeout never counts as a pass. Existing `--rom ... --steps ...` remains a diagnostic mode, not a test verdict.
|
||
|
||
Original programs cover the runner itself. Pinned public [ARM](public-arm-tests.md), [Thumb](public-thumb-tests.md), [memory](public-memory-tests.md), and [BIOS](public-bios-tests.md) ROMs provide independent compatibility results.
|
||
These pinned ROMs pass their checkpoints on Darwin arm64 in debug and release builds.
|
||
The memory ROM has documented assertion and mode-selection limits; a pass is not exhaustive memory validation.
|
||
Passing the runner's unit tests does not mean an external suite passes.
|
||
A passing suite establishes only its specified checkpoints and assertions. It does not establish Emerald compatibility.
|
||
|
||
## Run the original smoke suite
|
||
|
||
```sh
|
||
direnv exec . cargo run --locked --example write_test_suite -- /tmp/gba-original-tests
|
||
direnv exec . cargo run --locked --release -- --test-suite /tmp/gba-original-tests/suite.json > /tmp/gba-test-report.json
|
||
```
|
||
|
||
The generator creates original ARM, Thumb, and BIOS-division programs, plus a JSON manifest.
|
||
The destination directory must not already exist. No Nintendo code, firmware, or game assets are included.
|
||
Generated binaries stay outside the repository.
|
||
The runner writes one JSON report to stdout. Cargo build messages and errors go to stderr.
|
||
Do not redirect the report over the manifest or a ROM file.
|
||
|
||
## Manifest version 1
|
||
|
||
```json
|
||
{
|
||
"version": 1,
|
||
"cases": [
|
||
{
|
||
"name": "arm-store",
|
||
"rom": "arm.gba",
|
||
"step_limit": 10000,
|
||
"completion": {
|
||
"pc": "0x0800000c",
|
||
"instruction_set": "arm"
|
||
},
|
||
"checks": [
|
||
{ "kind": "register", "index": 0, "equals": 42 },
|
||
{ "kind": "memory32", "address": "0x02000000", "equals": 42 },
|
||
{ "kind": "cpsr", "equals": "0x1f" }
|
||
]
|
||
}
|
||
]
|
||
}
|
||
```
|
||
|
||
This example matches the generated `arm.gba`, not arbitrary ROMs.
|
||
The program stores 42 to external RAM, then reaches an ARM self-branch at `0x0800000c`.
|
||
|
||
Rules:
|
||
|
||
- All shown fields are required. Unknown fields and unsupported check kinds are rejected.
|
||
- The manifest must be a regular UTF-8 JSON file, no larger than 1 MiB.
|
||
- Supply 1–256 cases. Names must be unique, nonempty after trimming, and at most 128 UTF-8 bytes.
|
||
- Each case needs 1–256 checks and a nonempty ROM path.
|
||
- Relative ROM paths resolve from the manifest's directory, not the current working directory. Absolute paths are accepted.
|
||
- Each step limit is a JSON integer from 1 through 100,000,000. Their sum must not exceed 100,000,000.
|
||
- Addresses and expected values accept unsigned 32-bit JSON integers, decimal strings, or lowercase `0x`-prefixed hexadecimal strings.
|
||
- Signed values, floating-point values, surrounding whitespace, and overflow are rejected. Encode negative bit patterns as unsigned words.
|
||
- `instruction_set` is exactly `arm` or `thumb`. ARM completion addresses must be word-aligned; Thumb addresses must be halfword-aligned.
|
||
- Use the actual Thumb instruction address, without the address bit used by `BX`.
|
||
- Register indices are integers from 0 through 15. Checks read the active register bank.
|
||
- `memory32` addresses must be word-aligned. Reads use the host inspection path, including mapping errors.
|
||
BIOS assertions inspect raw mapped bytes, not the protected values seen by CPU code executing outside BIOS.
|
||
- `cpsr` compares the complete current program status register. No mask is applied.
|
||
|
||
The runner validates the whole manifest before opening or executing any ROM.
|
||
It uses the same read-only [ROM loader](hardware/cartridge.md#loading-and-boot), with its 4-byte minimum and 32 MiB maximum.
|
||
The CLI accepts no other options with `--test-suite`.
|
||
|
||
## Manifest version 2: gameplay checks
|
||
|
||
Version 2 accepts version 1 PC checkpoints and adds VBlank completion, button snapshots, and captured-pixel checks.
|
||
Version 1 behavior and report fields remain unchanged.
|
||
|
||
```json
|
||
{
|
||
"version": 2,
|
||
"cases": [{
|
||
"name": "original-input-release",
|
||
"rom": "input.gba",
|
||
"step_limit": 1000000,
|
||
"completion": {"vblanks": 3},
|
||
"inputs": [
|
||
{"vblank": 0, "buttons": "0x001"},
|
||
{"vblank": 1, "buttons": "0x011"},
|
||
{"vblank": 2, "buttons": 0}
|
||
],
|
||
"checks": [{"kind": "pixel", "x": 239, "y": 159, "equals": "0x0000ff"}]
|
||
}]
|
||
}
|
||
```
|
||
|
||
This example uses the original input ROM from `cargo run --example write_rom_demo -- /tmp/input.gba`.
|
||
Place the manifest alongside that ROM. The CPU writes red for A, green for Right, and blue after release.
|
||
|
||
- Use exactly one completion form: PC/state or `vblanks`. Mixed forms are invalid.
|
||
- VBlank completion requires an integer count from 1 through 100,000, measured from machine reset.
|
||
- The existing step budget still applies. A VBlank target does not permit unbounded execution.
|
||
- `inputs` is optional and only valid with VBlank completion. Supply at most 256 snapshots.
|
||
- Snapshot VBlank numbers must increase strictly and remain below the completion count. Zero applies before the first step.
|
||
- Each snapshot replaces all held buttons. Its state remains active until the next snapshot.
|
||
- Button masks use bits 0–9: A, B, Select, Start, Right, Left, Up, Down, R, L. One means pressed.
|
||
- Snapshots apply at the first machine-step boundary at or after their VBlank, after capture and before the next step.
|
||
- Pixel checks require VBlank completion, `x` from 0–239, `y` from 0–159, and packed `0xRRGGBB` expected values.
|
||
- Pixel checks read the latest completed scanline capture, not a snapshot of current video RAM.
|
||
- All VBlank cases enable capture. The first published render diagnostic fails the case, even before its target.
|
||
- A missing captured frame is an error, never an implicit black image.
|
||
|
||
Input timing is deterministic but not sub-instruction accurate. A snapshot cannot affect the frame already captured at that VBlank.
|
||
STOP still fails: emulated VBlanks cannot advance to a future scheduled wake-up event.
|
||
This is a regression runner, not a replacement for manual gameplay and host-keyboard checks.
|
||
|
||
## Completion and failure semantics
|
||
|
||
Each case starts with fresh RAM, devices, CPU state, and the original BIOS replacement.
|
||
The budget includes BIOS boot, instructions, DMA units, IRQ entries, and HALT idle batches.
|
||
A failed CPU or DMA step does not consume a successful step or advance the reported state.
|
||
PC-checkpoint cases supply no input and do not enable frame capture. Device clocks still advance through normal machine stepping.
|
||
|
||
After successful progress, the runner checks PC and instruction set at machine-step boundaries.
|
||
It stops before executing the instruction at the completion address.
|
||
Reaching that address on the final allowed step succeeds if all checks pass.
|
||
The initial reset state alone cannot pass a case. STOP always ends the case as a failure because no input source can wake it.
|
||
HALT continues advancing devices until completion, a diagnostic, or budget exhaustion.
|
||
|
||
The checkpoint does not imply that pending DMA or IRQ work has completed.
|
||
Choose an address reached only after the tested operations finish.
|
||
A loop address reached before those operations would define an invalid test, even if the runner reports a pass.
|
||
The runner does not infer program completion from an infinite loop, a screen, or matching register values alone.
|
||
|
||
All checks run at the checkpoint. A mismatch or unreadable memory address fails the case.
|
||
Cases that never reach the checkpoint do not evaluate their checks.
|
||
A failure does not skip later cases. No machine state passes from one case to another.
|
||
|
||
The budget bounds machine steps, not elapsed host time. Use an external process timeout in continuous integration if needed.
|
||
This is not a filesystem security sandbox. Run trusted suite configurations and use files you may lawfully use.
|
||
|
||
## Reports and exit status
|
||
|
||
Top-level fields:
|
||
|
||
| Field | Meaning |
|
||
| --- | --- |
|
||
| `format_version` | Manifest version: 1 or 2 |
|
||
| `bios` | `original`; no external BIOS option |
|
||
| `passed` | True only when every case passed |
|
||
| `case_count`, `failed_count` | Number of executed cases and failures |
|
||
| `cases` | Results in manifest order |
|
||
|
||
Each case records its name, ROM path, debug-formatted resolved path, byte count, budget, completion condition, and result.
|
||
`state` contains registers r0–r15, PC, CPSR, mode, instruction set, HALT/STOP flags, nominal cycles, and step counters.
|
||
Addresses, registers, expected values, and counters in reports are JSON numbers, even when the manifest used hexadecimal strings.
|
||
A load failure has null `state` and `rom_bytes`.
|
||
`checks` contains the configured check, actual value, pass flag, and optional read error when the checkpoint was reached.
|
||
Otherwise `checks` is empty. A failed read has a null actual value.
|
||
|
||
Version 2 VBlank cases also report their normalized `inputs` and state counters `vblanks` and `captured_vblank`.
|
||
These state fields are absent when loading fails. A PC case retains the version 1 case layout.
|
||
|
||
Case reasons are:
|
||
|
||
- `checkpoint`: completion reached and all assertions passed.
|
||
- `assertion_failed`: completion reached but at least one assertion failed or could not read memory.
|
||
- `step_limit`: completion was not reached within the budget.
|
||
- `stopped`: STOP prevented further execution.
|
||
- `emulation_error`: an instruction or DMA diagnostic stopped execution; `error` contains its message.
|
||
- `load_error`: loading or machine creation failed; `error` contains its message.
|
||
- `render_error`: frame capture failed or no complete frame was available at VBlank; `error` contains its message.
|
||
|
||
Exit status 0 means all cases passed and the report was written successfully.
|
||
Any failed case, invalid configuration, or output failure returns a nonzero status.
|
||
Invalid manifests produce a stderr diagnostic without a JSON report. A broken output stream can leave an incomplete report.
|
||
Valid suites produce a report even when some ROMs fail to load.
|
||
Reports omit host timestamps and durations so unchanged files produce repeatable output on the same host.
|
||
They do not hash ROM contents; record source revisions and file checksums alongside external test results.
|
||
|
||
## Next validation stage
|
||
|
||
The first public integration is [jsmolka/gba-tests ARM](public-arm-tests.md), with a verified result-register checkpoint.
|
||
The pinned ARM ROM now passes this checkpoint on Darwin arm64. Historical failures remain documented.
|
||
The [pinned Thumb ROM](public-thumb-tests.md) also passes its verified r7 checkpoint after the empty-list store correction.
|
||
The [pinned memory ROM](public-memory-tests.md) also passes, without a core change and with explicit coverage limits.
|
||
The [pinned BIOS read-protection ROM](public-bios-tests.md) also passes with the original replacement BIOS.
|
||
Its documented boundary values come from compatibility data in our firmware's exit layout, not a bus override.
|
||
The ROM, assertions, checkpoint, and budget remain unchanged. This is not a complete BIOS service test.
|
||
For further public ARM7TDMI/GBA tests, inspect their source, license, entry assumptions, and result protocol.
|
||
Pin source revisions and build instructions before comparing emulator changes.
|
||
Adapt suites to verified completion addresses and result locations; do not guess them or treat timeouts as passes.
|
||
Version 2 supports scheduled input and exact captured-pixel checks.
|
||
Debug-port logging, external firmware selection, audio assertions, and whole-image comparisons remain unsupported.
|
||
Keep external binaries local. Record consulted sources in [references.md](references.md).
|