6e736c634b
Add --suite bios with hash-verified upstream files and the ARM evaluation checkpoint at 0x08000248. Preserve the r12=0 assertion, original BIOS boot, and all test bytes. Add synthetic adapter and CLI regressions. On Darwin arm64 the public ROM fails test 1 in debug and release after 64 steps and 543 nominal cycles. Document that this tests protected reads of Nintendo firmware opcodes, not general BIOS service compatibility. Online and offline preparation agree; 39 Python tests pass. Core behavior is unchanged.
145 lines
8.6 KiB
Markdown
145 lines
8.6 KiB
Markdown
# Headless ROM test suites
|
||
|
||
Use `--test-suite PATH.json` to run repeatable ROM checks without opening a window.
|
||
Each case has a machine-step budget, a completion address, and explicit assertions.
|
||
A timeout never counts as a pass. Existing `--rom ... --steps ...` remains a diagnostic mode, not a test verdict.
|
||
|
||
Original programs cover the runner itself. Pinned public [ARM](public-arm-tests.md), [Thumb](public-thumb-tests.md), and [memory](public-memory-tests.md) ROMs provide independent compatibility results.
|
||
These pinned ROMs pass their checkpoints on Darwin arm64 in debug and release builds.
|
||
The memory ROM has documented assertion and mode-selection limits; a pass is not exhaustive memory validation.
|
||
Passing the runner's unit tests does not mean an external suite passes.
|
||
A passing suite establishes only its specified checkpoints and assertions. It does not establish Emerald compatibility.
|
||
|
||
## Run the original smoke suite
|
||
|
||
```sh
|
||
direnv exec . cargo run --locked --example write_test_suite -- /tmp/gba-original-tests
|
||
direnv exec . cargo run --locked --release -- --test-suite /tmp/gba-original-tests/suite.json > /tmp/gba-test-report.json
|
||
```
|
||
|
||
The generator creates original ARM, Thumb, and BIOS-division programs, plus a JSON manifest.
|
||
The destination directory must not already exist. No Nintendo code, firmware, or game assets are included.
|
||
Generated binaries stay outside the repository.
|
||
The runner writes one JSON report to stdout. Cargo build messages and errors go to stderr.
|
||
Do not redirect the report over the manifest or a ROM file.
|
||
|
||
## Manifest version 1
|
||
|
||
```json
|
||
{
|
||
"version": 1,
|
||
"cases": [
|
||
{
|
||
"name": "arm-store",
|
||
"rom": "arm.gba",
|
||
"step_limit": 10000,
|
||
"completion": {
|
||
"pc": "0x0800000c",
|
||
"instruction_set": "arm"
|
||
},
|
||
"checks": [
|
||
{ "kind": "register", "index": 0, "equals": 42 },
|
||
{ "kind": "memory32", "address": "0x02000000", "equals": 42 },
|
||
{ "kind": "cpsr", "equals": "0x1f" }
|
||
]
|
||
}
|
||
]
|
||
}
|
||
```
|
||
|
||
This example matches the generated `arm.gba`, not arbitrary ROMs.
|
||
The program stores 42 to external RAM, then reaches an ARM self-branch at `0x0800000c`.
|
||
|
||
Rules:
|
||
|
||
- All shown fields are required. Unknown fields and unsupported check kinds are rejected.
|
||
- The manifest must be a regular UTF-8 JSON file, no larger than 1 MiB.
|
||
- Supply 1–256 cases. Names must be unique, nonempty after trimming, and at most 128 UTF-8 bytes.
|
||
- Each case needs 1–256 checks and a nonempty ROM path.
|
||
- Relative ROM paths resolve from the manifest's directory, not the current working directory. Absolute paths are accepted.
|
||
- Each step limit is a JSON integer from 1 through 100,000,000. Their sum must not exceed 100,000,000.
|
||
- Addresses and expected values accept unsigned 32-bit JSON integers, decimal strings, or lowercase `0x`-prefixed hexadecimal strings.
|
||
- Signed values, floating-point values, surrounding whitespace, and overflow are rejected. Encode negative bit patterns as unsigned words.
|
||
- `instruction_set` is exactly `arm` or `thumb`. ARM completion addresses must be word-aligned; Thumb addresses must be halfword-aligned.
|
||
- Use the actual Thumb instruction address, without the address bit used by `BX`.
|
||
- Register indices are integers from 0 through 15. Checks read the active register bank.
|
||
- `memory32` addresses must be word-aligned. Reads use the existing little-endian memory bus, including its mapping errors.
|
||
- `cpsr` compares the complete current program status register. No mask is applied.
|
||
|
||
The runner validates the whole manifest before opening or executing any ROM.
|
||
It uses the same read-only [ROM loader](hardware/cartridge.md#loading-and-boot), with its 4-byte minimum and 32 MiB maximum.
|
||
The CLI accepts no other options with `--test-suite`.
|
||
|
||
## Completion and failure semantics
|
||
|
||
Each case starts with fresh RAM, devices, CPU state, and the original BIOS replacement.
|
||
The budget includes BIOS boot, instructions, DMA units, IRQ entries, and HALT idle batches.
|
||
A failed CPU or DMA step does not consume a successful step or advance the reported state.
|
||
The runner supplies no input and does not enable frame capture. Device clocks still advance through normal machine stepping.
|
||
|
||
After successful progress, the runner checks PC and instruction set at machine-step boundaries.
|
||
It stops before executing the instruction at the completion address.
|
||
Reaching that address on the final allowed step succeeds if all checks pass.
|
||
The initial reset state alone cannot pass a case. STOP always ends the case as a failure because no input source can wake it.
|
||
HALT continues advancing devices until completion, a diagnostic, or budget exhaustion.
|
||
|
||
The checkpoint does not imply that pending DMA or IRQ work has completed.
|
||
Choose an address reached only after the tested operations finish.
|
||
A loop address reached before those operations would define an invalid test, even if the runner reports a pass.
|
||
The runner does not infer program completion from an infinite loop, a screen, or matching register values alone.
|
||
|
||
All checks run at the checkpoint. A mismatch or unreadable memory address fails the case.
|
||
Cases that never reach the checkpoint do not evaluate their checks.
|
||
A failure does not skip later cases. No machine state passes from one case to another.
|
||
|
||
The budget bounds machine steps, not elapsed host time. Use an external process timeout in continuous integration if needed.
|
||
This is not a filesystem security sandbox. Run trusted suite configurations and use files you may lawfully use.
|
||
|
||
## Report version 1 and exit status
|
||
|
||
Top-level fields:
|
||
|
||
| Field | Meaning |
|
||
| --- | --- |
|
||
| `format_version` | Report schema version, currently 1 |
|
||
| `bios` | `original`; no external BIOS option |
|
||
| `passed` | True only when every case passed |
|
||
| `case_count`, `failed_count` | Number of executed cases and failures |
|
||
| `cases` | Results in manifest order |
|
||
|
||
Each case records its name, ROM path, debug-formatted resolved path, byte count, budget, completion condition, and result.
|
||
`state` contains registers r0–r15, PC, CPSR, mode, instruction set, HALT/STOP flags, nominal cycles, and step counters.
|
||
Addresses, registers, expected values, and counters in reports are JSON numbers, even when the manifest used hexadecimal strings.
|
||
A load failure has null `state` and `rom_bytes`.
|
||
`checks` contains the configured check, actual value, pass flag, and optional read error when the checkpoint was reached.
|
||
Otherwise `checks` is empty. A failed read has a null actual value.
|
||
|
||
Case reasons are:
|
||
|
||
- `checkpoint`: completion reached and all assertions passed.
|
||
- `assertion_failed`: completion reached but at least one assertion failed or could not read memory.
|
||
- `step_limit`: completion was not reached within the budget.
|
||
- `stopped`: STOP prevented further execution.
|
||
- `emulation_error`: an instruction or DMA diagnostic stopped execution; `error` contains its message.
|
||
- `load_error`: loading or machine creation failed; `error` contains its message.
|
||
|
||
Exit status 0 means all cases passed and the report was written successfully.
|
||
Any failed case, invalid configuration, or output failure returns a nonzero status.
|
||
Invalid manifests produce a stderr diagnostic without a JSON report. A broken output stream can leave an incomplete report.
|
||
Valid suites produce a report even when some ROMs fail to load.
|
||
Reports omit host timestamps and durations so unchanged files produce repeatable output on the same host.
|
||
They do not hash ROM contents; record source revisions and file checksums alongside external test results.
|
||
|
||
## Next validation stage
|
||
|
||
The first public integration is [jsmolka/gba-tests ARM](public-arm-tests.md), with a verified result-register checkpoint.
|
||
The pinned ARM ROM now passes this checkpoint on Darwin arm64. Historical failures remain documented.
|
||
The [pinned Thumb ROM](public-thumb-tests.md) also passes its verified r7 checkpoint after the empty-list store correction.
|
||
The [pinned memory ROM](public-memory-tests.md) also passes, without a core change and with explicit coverage limits.
|
||
The [pinned BIOS read-protection ROM](public-bios-tests.md) reaches its checkpoint but fails test 1 with the original replacement BIOS.
|
||
Its expected values depend on Nintendo firmware. Record that failure rather than changing the assertion or returning guessed constants.
|
||
For further public ARM7TDMI/GBA tests, inspect their source, license, entry assumptions, and result protocol.
|
||
Pin source revisions and build instructions before comparing emulator changes.
|
||
Adapt suites to verified completion addresses and result locations; do not guess them or treat timeouts as passes.
|
||
Some tests need debug-port logging, input, different firmware behavior, or rendered-image checks that this runner does not support yet.
|
||
Keep external binaries local. Record consulted sources in [references.md](references.md).
|