feat(testing): pin public ARM fixture and document compatibility baseline

Add a standard-library Python adapter and SHA-256/size lock for jsmolka/gba-tests ARM at a7113b67e63f83a9b321696ddd7042ccfad6c881. Retain sources and license locally, refuse tracked output paths or overwrites, verify offline inputs, and generate the source-verified eval/r12 suite checkpoint without patching the ROM.

Add Python to the pinned Nix shell, offline adapter regressions, and public-test provenance/results documentation. Verified on Darwin arm64: Rust debug/release workspace tests, adapter tests, Clippy, rustdoc, native ROM and all demo window checks. The public test remains a failing baseline at instruction 0xe5b00004, PC 0x0800132c, in test 360.
This commit is contained in:
0xrsydn 2026-10-05 20:00:16 +07:00
commit a28b830b38
13 changed files with 460 additions and 8 deletions

View file

@ -13,7 +13,9 @@ The repository is a Cargo workspace with three crates:
| Path | Purpose |
| --- | --- |
| `flake.nix`, `flake.lock` | Pinned development environment |
| `flake.nix`, `flake.lock` | Pinned Rust/Python development environment |
| `tools/prepare_gba_tests.py`, `tools/gba-tests-arm.lock.json` | Hash-verified public ARM source/ROM preparation and suite generation |
| `tools/test_prepare_gba_tests.py` | Offline adapter tests using original synthetic data |
| `.envrc` | Automatic environment loading through direnv |
| `AGENTS.md` | Short orientation for coding agents: crate map, commands, and where docs live |
| `docs/` | Project documentation; `docs/hardware/` holds per-subsystem behavior notes |
@ -24,6 +26,7 @@ The repository is a Cargo workspace with three crates:
| `crates/gba-core/src/cpu/thumb.rs` | Thumb decoding and execution |
| `crates/gba-core/src/cpu/status.rs` | Processor modes, register banks, and status transfers |
| `crates/gba-core/src/cpu/exception.rs` | Reset state, exception entry, and interrupt sampling |
| `crates/gba-core/src/cpu/compare_psr_tests.rs` | Original Rd=15 test/compare status restoration, flags, bank, and timing regressions |
| `crates/gba-core/src/cpu/status_tests.rs`, `crates/gba-core/src/cpu/exception_tests.rs` | Status, banking, exception, and return tests |
| `crates/gba-core/src/cpu/thumb_tests.rs` | Thumb instructions, state switching, and decoder checks |
| `crates/gba-core/src/cpu/transfer_tests.rs` | Stack, addressing, overlap, empty-list, and swap tests |

View file

@ -54,7 +54,8 @@ direnv exec . cargo run --locked
direnv exec . cargo test --locked
```
`.envrc` uses the Nix flake to load Rust, Cargo, rustfmt, and Clippy.
`.envrc` uses the Nix flake to load Rust, Cargo, rustfmt, Clippy, and Python 3.
Python supports the pinned public-test preparation adapter and its standard-library-only tests.
`flake.lock` pins the Nix package source. Keep it in version control.
The first run needs network access to download the development tools and Rust dependencies.

98
docs/public-arm-tests.md Normal file
View file

@ -0,0 +1,98 @@
# Pinned public ARM test
The headless runner now executes the unmodified ARM ROM from [jsmolka/gba-tests](https://github.com/jsmolka/gba-tests).
This is independent public test code, not a complete ARM7TDMI conformance suite or a new hardware measurement.
The full public ARM test still fails. The first discovered CPU error is fixed; the next unsupported instruction is recorded below.
## Prepare and run
```sh
direnv exec . python3 tools/prepare_gba_tests.py /tmp/gba-public-arm
direnv exec . cargo run --locked --release -- --test-suite /tmp/gba-public-arm/suite.json > /tmp/gba-public-arm-report.json
```
The destination must not exist. Its parent directory must exist.
Use a directory outside the repository or within the ignored `roms/` directory.
The adapter downloads pinned files over HTTPS and checks each byte count and SHA-256 digest before creating output.
It retains the upstream MIT license, README, ARM sources, shared assembly includes, and the original ROM.
It also writes `suite.json` and `source-lock.json`. The manifest references the ROM relative to its own directory.
No source rewriting, header removal, binary patching, or expected-result adjustment occurs.
Python comes from the existing pinned Nix environment. No Python packages are required.
The adapter bounds each download, sets a network timeout, and refuses existing output directories.
A disk-write failure can leave partial output; choose a new destination for another attempt.
Ordinary Rust and Python unit tests do not download or execute external ROMs.
For an offline copy of the selected upstream files:
```sh
direnv exec . python3 tools/prepare_gba_tests.py /tmp/gba-public-arm-offline --source-dir /path/to/gba-tests
```
Offline input must match every selected file in the lock. A directory from a different revision can fail verification.
The adapter does not overwrite or modify the input tree.
## Provenance and checkpoint
- Repository: `jsmolka/gba-tests`.
- Revision: `a7113b67e63f83a9b321696ddd7042ccfad6c881`.
- Upstream project license: MIT, copyright Julian Smolka. The license is retained with downloaded files.
- ROM: `arm/arm.gba`, 8,824 bytes.
- SHA-256: `77ee88662552bdc885c1080c0172ff119d54db791bd73b21808cf1ff1fe5b40e`.
- All selected file hashes and sizes: [`tools/gba-tests-arm.lock.json`](../tools/gba-tests-arm.lock.json).
This adapter uses the committed upstream binary, not a local rebuild.
Upstream documents FASMARM as its assembler. No FASMARM build or source-to-binary reproduction was performed on Darwin.
The binary's console header includes logo bytes. Do not copy the downloaded binary or header source into tracked files.
Only original adapter code and provenance metadata are committed here.
The ARM source initializes r12 to zero before testing.
Each failure path sets r12 to its test number and branches to `eval`.
Successful execution falls through to the same label with r12 still zero.
At this pinned revision, `eval` is `0x08001d4c`, in ARM state.
The ROM word there is the initial register save of the `m_vsync` macro.
The source failure exits and their encoded branch targets were inspected against that address.
The manifest stops before executing `eval` and requires `r12 == 0`.
This avoids depending on the result renderer, VBlank polling, or BIOS division used to print failure numbers.
It does not bypass the ROM's startup code or any CPU tests.
In particular, evaluation must stop before using a possibly invalid stack left by a failed mode-switch test.
The instruction at the checkpoint is checked during preparation, in addition to whole-file hash verification.
The case budget is 1,000,000 successful machine steps, including our original BIOS boot.
Timeouts, emulator diagnostics, and nonzero r12 all remain failures under the [normal suite rules](rom-tests.md).
Preparation does not mean the test passed. The current run exits nonzero and writes a failed JSON report.
## Observed results on Darwin arm64
Before the compare/status fix:
- The ROM reaches `eval` after 579 successful steps.
- The assertion reports r12 = 234, in FIQ mode.
- Running beyond `eval` produces an unmapped stack access because the CPU incorrectly remains in FIQ mode.
- Upstream test 234 uses an ARM compare encoding with unused destination bits set to R15.
The CPU previously treated that encoding as an ordinary compare and only changed arithmetic flags.
It now restores CPSR from the current mode's SPSR when one exists, without writing a result to PC.
User/System forms still use ordinary test/compare flags because those modes have no SPSR.
Original regressions cover all four test/compare operations, exception banks, flags, masks, operands, timing, conditions, and invalid saved modes.
See [CPU status behavior](hardware/cpu.md#processor-status-and-exceptions).
After the fix:
- Execution progresses past tests 234 and 235 and the PSR-transfer section.
- After 861 successful steps, the runner reports unsupported instruction `0xe5b00004` at `0x0800132c`.
- This is `LDR r0, [r0, #4]!` in upstream test 360, with base and destination equal.
- The CPU is in System/ARM state, and r12 remains zero. The final completion checkpoint has not been reached.
The load/writeback alias case is the next compatibility task. Do not weaken the assertion or label this run a pass.
No public Thumb, timing, memory, BIOS, graphics, or unsafe suite has been claimed as passing.
These results do not establish Pokémon Emerald compatibility.
## References
- [Pinned ARM entry and result flow](https://github.com/jsmolka/gba-tests/blob/a7113b67e63f83a9b321696ddd7042ccfad6c881/arm/arm.asm).
- [Pinned test/result macros](https://github.com/jsmolka/gba-tests/blob/a7113b67e63f83a9b321696ddd7042ccfad6c881/lib/macros.inc).
- [Tests 234 and 235](https://github.com/jsmolka/gba-tests/blob/a7113b67e63f83a9b321696ddd7042ccfad6c881/arm/data_processing.asm).
- [Load/writeback alias tests](https://github.com/jsmolka/gba-tests/blob/a7113b67e63f83a9b321696ddd7042ccfad6c881/arm/single_transfer.asm).
- [Pinned license](https://github.com/jsmolka/gba-tests/blob/a7113b67e63f83a9b321696ddd7042ccfad6c881/LICENSE).

View file

@ -4,7 +4,8 @@ Use `--test-suite PATH.json` to run repeatable ROM checks without opening a wind
Each case has a machine-step budget, a completion address, and explicit assertions.
A timeout never counts as a pass. Existing `--rom ... --steps ...` remains a diagnostic mode, not a test verdict.
This runner currently has regression coverage from original programs, not independent public hardware-test results.
Original programs cover the runner itself. A [pinned public ARM test](public-arm-tests.md) now provides an independent compatibility baseline.
That public ROM still fails; passing the runner's unit tests does not mean the external ARM suite passes.
A passing suite establishes only its specified checkpoints and assertions. It does not establish Emerald compatibility.
## Run the original smoke suite
@ -129,7 +130,9 @@ They do not hash ROM contents; record source revisions and file checksums alongs
## Next validation stage
Select public ARM7TDMI/GBA tests and inspect their source, license, entry assumptions, and result protocol.
The first public integration is [jsmolka/gba-tests ARM](public-arm-tests.md), with a verified result-register checkpoint and a known failing baseline.
Fix its next unsupported load/writeback alias case before claiming the full ARM ROM passes.
For further public ARM7TDMI/GBA tests, inspect their source, license, entry assumptions, and result protocol.
Pin source revisions and build instructions before comparing emulator changes.
Adapt suites to verified completion addresses and result locations; do not guess them or treat timeouts as passes.
Some tests need debug-port logging, input, different firmware behavior, or rendered-image checks that this runner does not support yet.

View file

@ -66,6 +66,8 @@ Update this file when a feature lands or a limit is removed.
- ROM windows with scanline capture, keyboard input, focus-loss release, bounded execution slices, and interactive keypad wake from STOP.
- Optional ROM-window frame limits and an original input-test ROM generator.
- Headless JSON ROM suites with per-case checkpoints, register/CPSR/memory assertions, bounded budgets, and structured failure reports.
- Hash-pinned preparation of the public `jsmolka/gba-tests` ARM ROM, with an explicit failing compatibility baseline.
- ARM test/compare Rd=15 status restoration, without a PC result write or nominal refill.
- Little-endian byte, halfword, and word reads/writes.
- ARM7TDMI word-load rotation, aligned stores, and odd-address halfword-load behavior.
- Explicit errors for unsupported instructions when their condition passes, and for unsupported memory accesses.
@ -110,7 +112,8 @@ ROM files can run in bounded terminal mode or a window with the original BIOS re
ROM windows have no demo-specific startup checks. Input is sampled between bounded execution slices.
Audio, saves, and external BIOS loading remain unavailable. ROM loading does not establish commercial-game compatibility.
See [cartridge loading](hardware/cartridge.md) for file limits, step budgets, and exit status.
The [ROM suite runner](rom-tests.md) currently uses original regression programs, not independent public hardware-test results.
The [ROM suite runner](rom-tests.md) has original regressions and a [pinned public ARM baseline](public-arm-tests.md).
The public ARM ROM now gets past its compare/status tests but stops at test 360, a load/writeback base-destination alias.
It requires known completion addresses and has no debug-port protocol, scripted input, or rendered-image assertions.
Unmapped reads return errors instead of hardware open-bus values.
@ -121,7 +124,8 @@ Writes to cartridge addresses return errors instead of modeling cartridge hardwa
## Next steps
1. Adapt the ROM suite runner to pinned public ARM7TDMI test programs and verified result protocols before claiming instruction compatibility.
1. Implement verified load/writeback alias behavior exposed by public ARM test 360; rerun the pinned suite and fix subsequent failures.
Add further public suites only with verified result protocols before claiming instruction compatibility.
2. Refine nominal timing with a fetch pipeline, Game Pak prefetch, per-access device updates, and verified timer/IRQ delays.
3. Validate keypad retrigger behavior with hardware tests and add remaining device registers; extend BIOS reset coverage as sound/serial support becomes available.
Validate STOP entry/wake edges and add external wake sources and remaining DMA device modes.

View file

@ -49,7 +49,15 @@ Do not set a Linux cross-compilation target for this validation.
The [headless ROM suite runner](rom-tests.md) provides explicit checkpoint and assertion results through `--test-suite PATH.json`.
Its tests verify manifest limits, whole-suite validation, exact step boundaries, ARM/Thumb checkpoints, and register/CPSR/memory checks.
Additional tests cover DMA/IRQ accounting, HALT/STOP, diagnostic state, case isolation, repeatable JSON output, and failure exit status.
These original tests validate the runner; they are not independent public hardware-test results.
These original tests validate the runner. The [public ARM adapter](public-arm-tests.md) is a separate, currently failing compatibility baseline.
Its preparation tests use only synthetic original bytes and no network:
```sh
direnv exec . python3 -B -m unittest discover -s tools -p 'test_*.py' -v
```
They verify pinned URLs, hashes, sizes, checkpoint validation, offline copies, destination safety, and non-overwriting output.
Compare/status CPU regressions verify saved flags/masks, bank switching, sequential PC/timing, User/System fallback, and atomic diagnostics.
ROM-runner tests use temporary files containing original instructions, without Nintendo assets or a desktop session.
They check file-size boundaries, bounded reads, short reads, I/O errors, paths, symlinks, and read-only loading.