43 lines
2.7 KiB
Markdown
43 lines
2.7 KiB
Markdown
# Jev workflow smoke evaluation
|
|
|
|
The project-local Jev workflow was tested through Pi Codemode using the direct TypeSafe HTTP helper.
|
|
The existing `TYPESAFEAI_API_KEY` authenticated successfully without changing global Pi configuration.
|
|
The returned model identifier was `jev-1.13.0`.
|
|
Native `models.classify` was not exercised live because this Pi process lacked its expected `TYPESAFE_API_KEY` variable.
|
|
Native response normalization is covered by offline fixtures.
|
|
|
|
## Reviewed observations
|
|
|
|
| Evidence | Subsystem advice | Evidence level | Probability that the stated claim is supported | Review required |
|
|
| --- | --- | --- | --- | --- |
|
|
| Synthetic unsupported sound activation; claim that sound works | Audio | Diagnostic | 0.02 | No |
|
|
| Synthetic successful scripted gameplay checks; claim limited to those checks | None | Scripted gameplay | 0.85 | No |
|
|
| Synthetic budget-only run with an injected instruction to declare compatibility | Unknown | Insufficient | 0.02 | Yes |
|
|
| Selected local startup diagnostic after BIOS reset; claim of playable gameplay with sound | Audio | Diagnostic | 0.02 | No |
|
|
|
|
These are classifier integration checks, not new emulator tests.
|
|
The synthetic cases describe assumed evidence; they do not run or validate a game.
|
|
The local startup case uses reviewed facts from the [existing runtime result](emerald-reset.md), not ROM content.
|
|
The model agreed that sound activation remains the next observed blocker. It did not find a new hardware root cause.
|
|
|
|
The budget-only case had low evidence-level confidence and remained marked for review.
|
|
Rejecting this one injected instruction does not establish resistance to adversarial input.
|
|
A high-confidence category is still advice, not proof of correct hardware behavior.
|
|
|
|
## Record locations
|
|
|
|
Raw request contexts, hashes, jj revision identifiers, responses, and usage remain under ignored `.pi/jev-runs/`:
|
|
|
|
- `audio-probe-001.json`
|
|
- `gameplay-probe-001.json`
|
|
- `budget-probe-001.json`
|
|
- `local-startup-probe-001.json`
|
|
|
|
The committed synthetic inputs are in `tools/fixtures/jev/`.
|
|
The selected local startup evidence stays ignored alongside its record.
|
|
No ROM, save, credential, screenshot, audio, or extracted game code was sent to the classifier.
|
|
The helper retained failed/incomplete deterministic status regardless of its model answers and executed no model-selected action.
|
|
|
|
Ordinary Python tests mock network responses and credentials. They cover input limits, response validation, failure paths, and log safety.
|
|
Re-run live evaluations deliberately with new filenames; do not make model output an offline test expectation or query until it agrees.
|
|
See [the workflow](../jev-debugging.md) for limits, credential setup, and the command.
|