feat(testing): pin public ARM fixture and document compatibility baseline
Add a standard-library Python adapter and SHA-256/size lock for jsmolka/gba-tests ARM at a7113b67e63f83a9b321696ddd7042ccfad6c881. Retain sources and license locally, refuse tracked output paths or overwrites, verify offline inputs, and generate the source-verified eval/r12 suite checkpoint without patching the ROM. Add Python to the pinned Nix shell, offline adapter regressions, and public-test provenance/results documentation. Verified on Darwin arm64: Rust debug/release workspace tests, adapter tests, Clippy, rustdoc, native ROM and all demo window checks. The public test remains a failing baseline at instruction 0xe5b00004, PC 0x0800132c, in test 360.
This commit is contained in:
parent
75fb98bc55
commit
a28b830b38
13 changed files with 460 additions and 8 deletions
|
|
@ -13,7 +13,9 @@ The repository is a Cargo workspace with three crates:
|
|||
|
||||
| Path | Purpose |
|
||||
| --- | --- |
|
||||
| `flake.nix`, `flake.lock` | Pinned development environment |
|
||||
| `flake.nix`, `flake.lock` | Pinned Rust/Python development environment |
|
||||
| `tools/prepare_gba_tests.py`, `tools/gba-tests-arm.lock.json` | Hash-verified public ARM source/ROM preparation and suite generation |
|
||||
| `tools/test_prepare_gba_tests.py` | Offline adapter tests using original synthetic data |
|
||||
| `.envrc` | Automatic environment loading through direnv |
|
||||
| `AGENTS.md` | Short orientation for coding agents: crate map, commands, and where docs live |
|
||||
| `docs/` | Project documentation; `docs/hardware/` holds per-subsystem behavior notes |
|
||||
|
|
@ -24,6 +26,7 @@ The repository is a Cargo workspace with three crates:
|
|||
| `crates/gba-core/src/cpu/thumb.rs` | Thumb decoding and execution |
|
||||
| `crates/gba-core/src/cpu/status.rs` | Processor modes, register banks, and status transfers |
|
||||
| `crates/gba-core/src/cpu/exception.rs` | Reset state, exception entry, and interrupt sampling |
|
||||
| `crates/gba-core/src/cpu/compare_psr_tests.rs` | Original Rd=15 test/compare status restoration, flags, bank, and timing regressions |
|
||||
| `crates/gba-core/src/cpu/status_tests.rs`, `crates/gba-core/src/cpu/exception_tests.rs` | Status, banking, exception, and return tests |
|
||||
| `crates/gba-core/src/cpu/thumb_tests.rs` | Thumb instructions, state switching, and decoder checks |
|
||||
| `crates/gba-core/src/cpu/transfer_tests.rs` | Stack, addressing, overlap, empty-list, and swap tests |
|
||||
|
|
|
|||
|
|
@ -54,7 +54,8 @@ direnv exec . cargo run --locked
|
|||
direnv exec . cargo test --locked
|
||||
```
|
||||
|
||||
`.envrc` uses the Nix flake to load Rust, Cargo, rustfmt, and Clippy.
|
||||
`.envrc` uses the Nix flake to load Rust, Cargo, rustfmt, Clippy, and Python 3.
|
||||
Python supports the pinned public-test preparation adapter and its standard-library-only tests.
|
||||
`flake.lock` pins the Nix package source. Keep it in version control.
|
||||
The first run needs network access to download the development tools and Rust dependencies.
|
||||
|
||||
|
|
|
|||
98
docs/public-arm-tests.md
Normal file
98
docs/public-arm-tests.md
Normal file
|
|
@ -0,0 +1,98 @@
|
|||
# Pinned public ARM test
|
||||
|
||||
The headless runner now executes the unmodified ARM ROM from [jsmolka/gba-tests](https://github.com/jsmolka/gba-tests).
|
||||
This is independent public test code, not a complete ARM7TDMI conformance suite or a new hardware measurement.
|
||||
The full public ARM test still fails. The first discovered CPU error is fixed; the next unsupported instruction is recorded below.
|
||||
|
||||
## Prepare and run
|
||||
|
||||
```sh
|
||||
direnv exec . python3 tools/prepare_gba_tests.py /tmp/gba-public-arm
|
||||
direnv exec . cargo run --locked --release -- --test-suite /tmp/gba-public-arm/suite.json > /tmp/gba-public-arm-report.json
|
||||
```
|
||||
|
||||
The destination must not exist. Its parent directory must exist.
|
||||
Use a directory outside the repository or within the ignored `roms/` directory.
|
||||
The adapter downloads pinned files over HTTPS and checks each byte count and SHA-256 digest before creating output.
|
||||
It retains the upstream MIT license, README, ARM sources, shared assembly includes, and the original ROM.
|
||||
It also writes `suite.json` and `source-lock.json`. The manifest references the ROM relative to its own directory.
|
||||
No source rewriting, header removal, binary patching, or expected-result adjustment occurs.
|
||||
|
||||
Python comes from the existing pinned Nix environment. No Python packages are required.
|
||||
The adapter bounds each download, sets a network timeout, and refuses existing output directories.
|
||||
A disk-write failure can leave partial output; choose a new destination for another attempt.
|
||||
Ordinary Rust and Python unit tests do not download or execute external ROMs.
|
||||
|
||||
For an offline copy of the selected upstream files:
|
||||
|
||||
```sh
|
||||
direnv exec . python3 tools/prepare_gba_tests.py /tmp/gba-public-arm-offline --source-dir /path/to/gba-tests
|
||||
```
|
||||
|
||||
Offline input must match every selected file in the lock. A directory from a different revision can fail verification.
|
||||
The adapter does not overwrite or modify the input tree.
|
||||
|
||||
## Provenance and checkpoint
|
||||
|
||||
- Repository: `jsmolka/gba-tests`.
|
||||
- Revision: `a7113b67e63f83a9b321696ddd7042ccfad6c881`.
|
||||
- Upstream project license: MIT, copyright Julian Smolka. The license is retained with downloaded files.
|
||||
- ROM: `arm/arm.gba`, 8,824 bytes.
|
||||
- SHA-256: `77ee88662552bdc885c1080c0172ff119d54db791bd73b21808cf1ff1fe5b40e`.
|
||||
- All selected file hashes and sizes: [`tools/gba-tests-arm.lock.json`](../tools/gba-tests-arm.lock.json).
|
||||
|
||||
This adapter uses the committed upstream binary, not a local rebuild.
|
||||
Upstream documents FASMARM as its assembler. No FASMARM build or source-to-binary reproduction was performed on Darwin.
|
||||
The binary's console header includes logo bytes. Do not copy the downloaded binary or header source into tracked files.
|
||||
Only original adapter code and provenance metadata are committed here.
|
||||
|
||||
The ARM source initializes r12 to zero before testing.
|
||||
Each failure path sets r12 to its test number and branches to `eval`.
|
||||
Successful execution falls through to the same label with r12 still zero.
|
||||
At this pinned revision, `eval` is `0x08001d4c`, in ARM state.
|
||||
The ROM word there is the initial register save of the `m_vsync` macro.
|
||||
The source failure exits and their encoded branch targets were inspected against that address.
|
||||
|
||||
The manifest stops before executing `eval` and requires `r12 == 0`.
|
||||
This avoids depending on the result renderer, VBlank polling, or BIOS division used to print failure numbers.
|
||||
It does not bypass the ROM's startup code or any CPU tests.
|
||||
In particular, evaluation must stop before using a possibly invalid stack left by a failed mode-switch test.
|
||||
The instruction at the checkpoint is checked during preparation, in addition to whole-file hash verification.
|
||||
|
||||
The case budget is 1,000,000 successful machine steps, including our original BIOS boot.
|
||||
Timeouts, emulator diagnostics, and nonzero r12 all remain failures under the [normal suite rules](rom-tests.md).
|
||||
Preparation does not mean the test passed. The current run exits nonzero and writes a failed JSON report.
|
||||
|
||||
## Observed results on Darwin arm64
|
||||
|
||||
Before the compare/status fix:
|
||||
|
||||
- The ROM reaches `eval` after 579 successful steps.
|
||||
- The assertion reports r12 = 234, in FIQ mode.
|
||||
- Running beyond `eval` produces an unmapped stack access because the CPU incorrectly remains in FIQ mode.
|
||||
- Upstream test 234 uses an ARM compare encoding with unused destination bits set to R15.
|
||||
|
||||
The CPU previously treated that encoding as an ordinary compare and only changed arithmetic flags.
|
||||
It now restores CPSR from the current mode's SPSR when one exists, without writing a result to PC.
|
||||
User/System forms still use ordinary test/compare flags because those modes have no SPSR.
|
||||
Original regressions cover all four test/compare operations, exception banks, flags, masks, operands, timing, conditions, and invalid saved modes.
|
||||
See [CPU status behavior](hardware/cpu.md#processor-status-and-exceptions).
|
||||
|
||||
After the fix:
|
||||
|
||||
- Execution progresses past tests 234 and 235 and the PSR-transfer section.
|
||||
- After 861 successful steps, the runner reports unsupported instruction `0xe5b00004` at `0x0800132c`.
|
||||
- This is `LDR r0, [r0, #4]!` in upstream test 360, with base and destination equal.
|
||||
- The CPU is in System/ARM state, and r12 remains zero. The final completion checkpoint has not been reached.
|
||||
|
||||
The load/writeback alias case is the next compatibility task. Do not weaken the assertion or label this run a pass.
|
||||
No public Thumb, timing, memory, BIOS, graphics, or unsafe suite has been claimed as passing.
|
||||
These results do not establish Pokémon Emerald compatibility.
|
||||
|
||||
## References
|
||||
|
||||
- [Pinned ARM entry and result flow](https://github.com/jsmolka/gba-tests/blob/a7113b67e63f83a9b321696ddd7042ccfad6c881/arm/arm.asm).
|
||||
- [Pinned test/result macros](https://github.com/jsmolka/gba-tests/blob/a7113b67e63f83a9b321696ddd7042ccfad6c881/lib/macros.inc).
|
||||
- [Tests 234 and 235](https://github.com/jsmolka/gba-tests/blob/a7113b67e63f83a9b321696ddd7042ccfad6c881/arm/data_processing.asm).
|
||||
- [Load/writeback alias tests](https://github.com/jsmolka/gba-tests/blob/a7113b67e63f83a9b321696ddd7042ccfad6c881/arm/single_transfer.asm).
|
||||
- [Pinned license](https://github.com/jsmolka/gba-tests/blob/a7113b67e63f83a9b321696ddd7042ccfad6c881/LICENSE).
|
||||
|
|
@ -4,7 +4,8 @@ Use `--test-suite PATH.json` to run repeatable ROM checks without opening a wind
|
|||
Each case has a machine-step budget, a completion address, and explicit assertions.
|
||||
A timeout never counts as a pass. Existing `--rom ... --steps ...` remains a diagnostic mode, not a test verdict.
|
||||
|
||||
This runner currently has regression coverage from original programs, not independent public hardware-test results.
|
||||
Original programs cover the runner itself. A [pinned public ARM test](public-arm-tests.md) now provides an independent compatibility baseline.
|
||||
That public ROM still fails; passing the runner's unit tests does not mean the external ARM suite passes.
|
||||
A passing suite establishes only its specified checkpoints and assertions. It does not establish Emerald compatibility.
|
||||
|
||||
## Run the original smoke suite
|
||||
|
|
@ -129,7 +130,9 @@ They do not hash ROM contents; record source revisions and file checksums alongs
|
|||
|
||||
## Next validation stage
|
||||
|
||||
Select public ARM7TDMI/GBA tests and inspect their source, license, entry assumptions, and result protocol.
|
||||
The first public integration is [jsmolka/gba-tests ARM](public-arm-tests.md), with a verified result-register checkpoint and a known failing baseline.
|
||||
Fix its next unsupported load/writeback alias case before claiming the full ARM ROM passes.
|
||||
For further public ARM7TDMI/GBA tests, inspect their source, license, entry assumptions, and result protocol.
|
||||
Pin source revisions and build instructions before comparing emulator changes.
|
||||
Adapt suites to verified completion addresses and result locations; do not guess them or treat timeouts as passes.
|
||||
Some tests need debug-port logging, input, different firmware behavior, or rendered-image checks that this runner does not support yet.
|
||||
|
|
|
|||
|
|
@ -66,6 +66,8 @@ Update this file when a feature lands or a limit is removed.
|
|||
- ROM windows with scanline capture, keyboard input, focus-loss release, bounded execution slices, and interactive keypad wake from STOP.
|
||||
- Optional ROM-window frame limits and an original input-test ROM generator.
|
||||
- Headless JSON ROM suites with per-case checkpoints, register/CPSR/memory assertions, bounded budgets, and structured failure reports.
|
||||
- Hash-pinned preparation of the public `jsmolka/gba-tests` ARM ROM, with an explicit failing compatibility baseline.
|
||||
- ARM test/compare Rd=15 status restoration, without a PC result write or nominal refill.
|
||||
- Little-endian byte, halfword, and word reads/writes.
|
||||
- ARM7TDMI word-load rotation, aligned stores, and odd-address halfword-load behavior.
|
||||
- Explicit errors for unsupported instructions when their condition passes, and for unsupported memory accesses.
|
||||
|
|
@ -110,7 +112,8 @@ ROM files can run in bounded terminal mode or a window with the original BIOS re
|
|||
ROM windows have no demo-specific startup checks. Input is sampled between bounded execution slices.
|
||||
Audio, saves, and external BIOS loading remain unavailable. ROM loading does not establish commercial-game compatibility.
|
||||
See [cartridge loading](hardware/cartridge.md) for file limits, step budgets, and exit status.
|
||||
The [ROM suite runner](rom-tests.md) currently uses original regression programs, not independent public hardware-test results.
|
||||
The [ROM suite runner](rom-tests.md) has original regressions and a [pinned public ARM baseline](public-arm-tests.md).
|
||||
The public ARM ROM now gets past its compare/status tests but stops at test 360, a load/writeback base-destination alias.
|
||||
It requires known completion addresses and has no debug-port protocol, scripted input, or rendered-image assertions.
|
||||
|
||||
Unmapped reads return errors instead of hardware open-bus values.
|
||||
|
|
@ -121,7 +124,8 @@ Writes to cartridge addresses return errors instead of modeling cartridge hardwa
|
|||
|
||||
## Next steps
|
||||
|
||||
1. Adapt the ROM suite runner to pinned public ARM7TDMI test programs and verified result protocols before claiming instruction compatibility.
|
||||
1. Implement verified load/writeback alias behavior exposed by public ARM test 360; rerun the pinned suite and fix subsequent failures.
|
||||
Add further public suites only with verified result protocols before claiming instruction compatibility.
|
||||
2. Refine nominal timing with a fetch pipeline, Game Pak prefetch, per-access device updates, and verified timer/IRQ delays.
|
||||
3. Validate keypad retrigger behavior with hardware tests and add remaining device registers; extend BIOS reset coverage as sound/serial support becomes available.
|
||||
Validate STOP entry/wake edges and add external wake sources and remaining DMA device modes.
|
||||
|
|
|
|||
|
|
@ -49,7 +49,15 @@ Do not set a Linux cross-compilation target for this validation.
|
|||
The [headless ROM suite runner](rom-tests.md) provides explicit checkpoint and assertion results through `--test-suite PATH.json`.
|
||||
Its tests verify manifest limits, whole-suite validation, exact step boundaries, ARM/Thumb checkpoints, and register/CPSR/memory checks.
|
||||
Additional tests cover DMA/IRQ accounting, HALT/STOP, diagnostic state, case isolation, repeatable JSON output, and failure exit status.
|
||||
These original tests validate the runner; they are not independent public hardware-test results.
|
||||
These original tests validate the runner. The [public ARM adapter](public-arm-tests.md) is a separate, currently failing compatibility baseline.
|
||||
Its preparation tests use only synthetic original bytes and no network:
|
||||
|
||||
```sh
|
||||
direnv exec . python3 -B -m unittest discover -s tools -p 'test_*.py' -v
|
||||
```
|
||||
|
||||
They verify pinned URLs, hashes, sizes, checkpoint validation, offline copies, destination safety, and non-overwriting output.
|
||||
Compare/status CPU regressions verify saved flags/masks, bank switching, sequential PC/timing, User/System fallback, and atomic diagnostics.
|
||||
|
||||
ROM-runner tests use temporary files containing original instructions, without Nintendo assets or a desktop session.
|
||||
They check file-size boundaries, bounded reads, short reads, I/O errors, paths, symlinks, and read-only loading.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue