feat(jev): structured question shapes, Score primitive, per-call answer record
The documented API accepts an object or array wherever a string is accepted: `instructions`, every Choice option description, every Score level, and the Noul `criteria.true` / `criteria.false` entries. The client sent bare strings everywhere and never used `criteria` on a Noul at all. Add the builders for those shapes: * `ask()` structured instructions: question, focus, inspect, compare * `entry()` a description with what it covers, what it is not for, examples * `noul_criteria()` contrastive true/false criteria * `score()` the ordered-level primitive, documented but not implemented * `gate_choice()` an explicit floor per Choice call site A threshold tuned on a Noul is never reused on a Choice: the two answer different questions and are not on a comparable scale, so each gate states its own pair (`CHOICE_TOP_MIN`, `CHOICE_MARGIN_MIN`). Also add `answer_record()` and the `JEV_TRACE` writer, so a decision can carry the answers that produced it. Without a trace there are no labels for any model decision, which blocks every question about decision quality.
This commit is contained in:
parent
fe889a3f5c
commit
b5b5485f8a
1 changed files with 137 additions and 15 deletions
152
jev.py
152
jev.py
|
|
@ -16,6 +16,27 @@ Constraints baked into this client (all measured, see DESIGN.md):
|
||||||
* Because answers to questions in one call are independent, never make a
|
* Because answers to questions in one call are independent, never make a
|
||||||
question depend on another question's answer. Use a second call.
|
question depend on another question's answer. Use a second call.
|
||||||
|
|
||||||
|
Question shapes follow the documented "advanced: structure" rules: the
|
||||||
|
`instructions` of every primitive, each Choice option description, each Score
|
||||||
|
level description and the Noul `criteria.true` / `criteria.false` entries all
|
||||||
|
accept a string, an object, or an array. Build them with `ask()` and `entry()`:
|
||||||
|
|
||||||
|
noul(ask("Would `offered.card0` make this deck stronger?",
|
||||||
|
focus="Judge deck fit, not raw power.",
|
||||||
|
inspect="`offered.card0`, `deck`"),
|
||||||
|
true=entry("It adds damage, block or scaling this deck lacks",
|
||||||
|
examples=["A 2-cost attack that scales with Strength"]),
|
||||||
|
false=entry("It is off-plan, redundant, or too slow to matter",
|
||||||
|
not_for="A card that is merely different"))
|
||||||
|
|
||||||
|
choice(ask("Which card best improves this deck, or is skipping better?"),
|
||||||
|
{f"card{i['index']}": entry(f"{i['name']}: {i['description']}",
|
||||||
|
not_for="...") for i in cards}
|
||||||
|
| {"skip": entry("Take nothing; keep the deck lean")})
|
||||||
|
|
||||||
|
A threshold tuned on a Noul is never reused on a Choice: state each gate's
|
||||||
|
floor and margin at the call site (`gate_choice`).
|
||||||
|
|
||||||
The API key is read from the sops-nix secrets path and is never logged.
|
The API key is read from the sops-nix secrets path and is never logged.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
|
|
@ -51,6 +72,13 @@ TRACE_PATH = os.environ.get("JEV_TRACE")
|
||||||
# low-stakes binary (end the turn? use a potion?) is recoverable in this game.
|
# low-stakes binary (end the turn? use a potion?) is recoverable in this game.
|
||||||
NOUL_MARGIN = 0.15
|
NOUL_MARGIN = 0.15
|
||||||
|
|
||||||
|
# Choice gates are stated per call site, because a threshold tuned on a Noul
|
||||||
|
# must not be carried over to a Choice: the two answer different questions and
|
||||||
|
# are not on a comparable scale. These are the DEFAULTS for a low-stakes Choice
|
||||||
|
# (a card play). Irreversible decisions (events) pass their own stricter pair.
|
||||||
|
CHOICE_TOP_MIN = 0.45
|
||||||
|
CHOICE_MARGIN_MIN = 0.20
|
||||||
|
|
||||||
|
|
||||||
class JevError(RuntimeError):
|
class JevError(RuntimeError):
|
||||||
"""Raised for transport, auth, and protocol failures."""
|
"""Raised for transport, auth, and protocol failures."""
|
||||||
|
|
@ -60,21 +88,91 @@ class JevError(RuntimeError):
|
||||||
# Question builders
|
# Question builders
|
||||||
# --------------------------------------------------------------------------
|
# --------------------------------------------------------------------------
|
||||||
|
|
||||||
def noul(instructions: Any, criteria: dict | None = None) -> dict:
|
def entry(what: Any = None, *, not_for: Any = None, examples: Any = None,
|
||||||
"""Yes/no question. Returns only P(yes); there is no confidence field."""
|
signals: Any = None, **more: Any) -> Any:
|
||||||
|
"""
|
||||||
|
One structured description entry, per the documented shapes.
|
||||||
|
|
||||||
|
`instructions`, Choice option descriptions, Score level descriptions and
|
||||||
|
Noul `criteria.true` / `criteria.false` all accept a string, an object, or
|
||||||
|
an array. Use the object form when a description needs several kinds of
|
||||||
|
guidance: what it covers, what belongs elsewhere, and examples.
|
||||||
|
|
||||||
|
Returns the bare string when only `what` is given, so a simple description
|
||||||
|
stays simple. The documented field names are used consistently across
|
||||||
|
entries so the model can compare them directly.
|
||||||
|
"""
|
||||||
|
fields = {"what": what, "not_for": not_for, "examples": examples,
|
||||||
|
"signals": signals, **more}
|
||||||
|
present = {k: v for k, v in fields.items() if v is not None}
|
||||||
|
if not present:
|
||||||
|
return None
|
||||||
|
if set(present) == {"what"} and isinstance(present["what"], str):
|
||||||
|
return present["what"]
|
||||||
|
return present
|
||||||
|
|
||||||
|
|
||||||
|
def ask(question: Any, *, focus: Any = None, inspect: Any = None,
|
||||||
|
compare: Any = None, **more: Any) -> Any:
|
||||||
|
"""
|
||||||
|
Structured instructions: the question plus named guidance fields.
|
||||||
|
|
||||||
|
`focus` says what to judge, `inspect` / `compare` point at the exact state
|
||||||
|
paths the answer depends on (backticked dot-and-index paths). Use it when a
|
||||||
|
bare sentence would blur several instructions together; a short,
|
||||||
|
unambiguous question stays a string.
|
||||||
|
"""
|
||||||
|
fields = {"focus": focus, "inspect": inspect, "compare": compare, **more}
|
||||||
|
present = {k: v for k, v in fields.items() if v is not None}
|
||||||
|
if not present:
|
||||||
|
return question
|
||||||
|
return {"question": question, **present}
|
||||||
|
|
||||||
|
|
||||||
|
def noul_criteria(true: Any = None, false: Any = None) -> dict | None:
|
||||||
|
"""
|
||||||
|
Contrastive Noul criteria: what a yes means, what a no means.
|
||||||
|
|
||||||
|
The instruction and the criteria must ask for the same thing — a Noul whose
|
||||||
|
`true` maps to "no" is a documented failure mode.
|
||||||
|
"""
|
||||||
|
criteria = {k: v for k, v in (("true", true), ("false", false)) if v is not None}
|
||||||
|
return criteria or None
|
||||||
|
|
||||||
|
|
||||||
|
def noul(instructions: Any, criteria: dict | None = None, *,
|
||||||
|
true: Any = None, false: Any = None) -> dict:
|
||||||
|
"""
|
||||||
|
Yes/no question. Returns only P(yes); there is no confidence field.
|
||||||
|
|
||||||
|
Pass `true=` / `false=` for the contrastive criteria, or `criteria=` for a
|
||||||
|
prebuilt mapping.
|
||||||
|
"""
|
||||||
q: dict[str, Any] = {"type": "noul", "instructions": instructions}
|
q: dict[str, Any] = {"type": "noul", "instructions": instructions}
|
||||||
if criteria is not None:
|
built = criteria if criteria is not None else noul_criteria(true, false)
|
||||||
q["criteria"] = criteria
|
if built is not None:
|
||||||
|
q["criteria"] = built
|
||||||
return q
|
return q
|
||||||
|
|
||||||
|
|
||||||
def choice(instructions: Any, criteria: dict) -> dict:
|
def choice(instructions: Any, criteria: dict) -> dict:
|
||||||
"""Pick one option from a map of option -> description (max 255)."""
|
"""
|
||||||
|
Pick one option from a map of option -> description (max 255).
|
||||||
|
|
||||||
|
Each description is an `entry()`. Give the model the full list rather than
|
||||||
|
a shortlist, and add an explicit "none of the above"-style option whenever
|
||||||
|
the set might not cover every input.
|
||||||
|
"""
|
||||||
return {"type": "choice", "instructions": instructions, "criteria": criteria}
|
return {"type": "choice", "instructions": instructions, "criteria": criteria}
|
||||||
|
|
||||||
|
|
||||||
def score(instructions: Any, criteria: list) -> dict:
|
def score(instructions: Any, criteria: list) -> dict:
|
||||||
"""Position on an ordered list of levels."""
|
"""
|
||||||
|
Position on an ordered list of levels, each an `entry()`.
|
||||||
|
|
||||||
|
Use a Score when the answer is a position on a spectrum. The returned
|
||||||
|
`score` is a fractional position on the level scale, not an index.
|
||||||
|
"""
|
||||||
return {"type": "score", "instructions": instructions, "criteria": criteria}
|
return {"type": "score", "instructions": instructions, "criteria": criteria}
|
||||||
|
|
||||||
|
|
||||||
|
|
@ -121,6 +219,15 @@ class ScoreAnswer:
|
||||||
def kind(self) -> str:
|
def kind(self) -> str:
|
||||||
return "score"
|
return "score"
|
||||||
|
|
||||||
|
@property
|
||||||
|
def level(self) -> int:
|
||||||
|
"""Nearest level index. `score` is a FRACTIONAL position on the scale."""
|
||||||
|
return int(round(self.score))
|
||||||
|
|
||||||
|
def level_entry(self) -> Any:
|
||||||
|
"""The legend entry for the nearest level, keyed by index as a string."""
|
||||||
|
return self.legend.get(str(self.level))
|
||||||
|
|
||||||
|
|
||||||
Answer = NoulAnswer | ChoiceAnswer | ScoreAnswer
|
Answer = NoulAnswer | ChoiceAnswer | ScoreAnswer
|
||||||
|
|
||||||
|
|
@ -284,7 +391,7 @@ class JevClient:
|
||||||
"input_tokens": response.input_tokens,
|
"input_tokens": response.input_tokens,
|
||||||
"output_tokens": response.output_tokens,
|
"output_tokens": response.output_tokens,
|
||||||
"questions": questions,
|
"questions": questions,
|
||||||
"answers": {qid: _answer_record(a) for qid, a in response.answers.items()},
|
"answers": {qid: answer_record(a) for qid, a in response.answers.items()},
|
||||||
})
|
})
|
||||||
return response
|
return response
|
||||||
|
|
||||||
|
|
@ -327,9 +434,14 @@ class JevClient:
|
||||||
# Trace log (one JSON line per call, path from $JEV_TRACE)
|
# Trace log (one JSON line per call, path from $JEV_TRACE)
|
||||||
# --------------------------------------------------------------------------
|
# --------------------------------------------------------------------------
|
||||||
|
|
||||||
def _answer_record(a: Answer) -> dict:
|
def answer_record(a: Answer) -> dict:
|
||||||
|
"""Machine-readable form of one answer, for a decision log.
|
||||||
|
|
||||||
|
Carries the gate outcome for every primitive, so a gate can be analysed
|
||||||
|
offline from the log alone instead of being recomputed from the value.
|
||||||
|
"""
|
||||||
if isinstance(a, NoulAnswer):
|
if isinstance(a, NoulAnswer):
|
||||||
return {"kind": "noul", "noul": a.noul, "yes": a.yes}
|
return {"kind": "noul", "noul": a.noul, "yes": a.yes, "gated": gate(a)}
|
||||||
if isinstance(a, ChoiceAnswer):
|
if isinstance(a, ChoiceAnswer):
|
||||||
return {
|
return {
|
||||||
"kind": "choice",
|
"kind": "choice",
|
||||||
|
|
@ -364,6 +476,21 @@ def _trace(record: dict) -> None:
|
||||||
# Confidence gate
|
# Confidence gate
|
||||||
# --------------------------------------------------------------------------
|
# --------------------------------------------------------------------------
|
||||||
|
|
||||||
|
def gate_choice(answer: ChoiceAnswer, top_min: float, margin_min: float) -> bool:
|
||||||
|
"""
|
||||||
|
Gate a Choice with thresholds stated for THIS decision.
|
||||||
|
|
||||||
|
A threshold tuned on a Noul must not be carried over to a Choice: the two
|
||||||
|
answer different questions and are not on a comparable scale. So each call
|
||||||
|
site names the floor and the margin that decision actually needs.
|
||||||
|
|
||||||
|
The margin is the scale-free signal; `confidence` is peakedness and falls as
|
||||||
|
the option count rises, so it is never the primary gate.
|
||||||
|
"""
|
||||||
|
top = answer.probabilities.get(answer.choice, 0.0)
|
||||||
|
return top >= top_min and margin(answer) >= margin_min
|
||||||
|
|
||||||
|
|
||||||
def gate(answer: Answer, threshold: float = 0.6) -> bool:
|
def gate(answer: Answer, threshold: float = 0.6) -> bool:
|
||||||
"""
|
"""
|
||||||
True when the answer is safe to act on automatically.
|
True when the answer is safe to act on automatically.
|
||||||
|
|
@ -381,12 +508,7 @@ def gate(answer: Answer, threshold: float = 0.6) -> bool:
|
||||||
if isinstance(answer, NoulAnswer):
|
if isinstance(answer, NoulAnswer):
|
||||||
return abs(answer.noul - 0.5) >= NOUL_MARGIN
|
return abs(answer.noul - 0.5) >= NOUL_MARGIN
|
||||||
if isinstance(answer, ChoiceAnswer):
|
if isinstance(answer, ChoiceAnswer):
|
||||||
top = answer.probabilities.get(answer.choice, 0.0)
|
return gate_choice(answer, CHOICE_TOP_MIN, CHOICE_MARGIN_MIN)
|
||||||
runner = max(
|
|
||||||
(v for k, v in answer.probabilities.items() if k != answer.choice),
|
|
||||||
default=0.0,
|
|
||||||
)
|
|
||||||
return top >= 0.45 and (top - runner) >= 0.20
|
|
||||||
return answer.confidence >= threshold
|
return answer.confidence >= threshold
|
||||||
|
|
||||||
|
|
||||||
|
|
|
||||||
Loading…
Add table
Add a link
Reference in a new issue