No description
- Python 88.4%
- C# 8.7%
- Shell 2.9%
d676bf067e
Two defects made the earlier A/B numbers meaningless:
* Without `--stop-on-run-end` a session contained several runs, so runs could
not be attributed to an arm.
* The script then read "the last 2N runs", which silently compared runs from
earlier experiments rather than the two arms.
Pass `--stop-on-run-end`, and attribute each run record to its arm by diffing
the history directory around each session.
Also raise the step budget and warn on a thin sample: measured, a run still
going at step 600 produces NO run record, so a short budget biases the sample
toward runs that died early -- exactly the wrong bias for this question.
|
||
|---|---|---|
| capture | ||
| docs | ||
| utils | ||
| vendor | ||
| .gitignore | ||
| ab_card_skip.sh | ||
| brain.py | ||
| capture.py | ||
| eval_batch.sh | ||
| facts.py | ||
| jev.py | ||
| run.py | ||
| sts2.py | ||
| test_brain.py | ||
| test_facts.py | ||
| test_run.py | ||