soup eval design — derive evals from your training data

Before v0.55 you wrote your eval suite by hand. Now Soup drafts one from your data.

soup eval design

soup eval design data.jsonl --goal "polite customer support chat" --num-dimensions 5

How it works:

1. TF-IDF salience picks up to num_dimensions (default 5) salient terms over the dataset. 10,000-row DoS-capped subsample.
2. Goal-keyword dispatch maps each dimension to a scorer: json / code / math → rlvr (verifiable reward), classify → exact_match, extract → regex, and everything else → judge.
3. Output: a frozen EvalDesign (JSON) with one EvalDimension per row.

Scorer allowlist: {exact_match, regex, judge, rlvr}.

soup eval discover — canaries

soup eval discover data.jsonl --num-clusters 5 --per-cluster 3

Three sets:

• Held-out canaries — prompts from each cluster of your own data, found by greedy farthest-first Jaccard-distance clustering (_CLUSTER_SUBSAMPLE = 10_000).
• Adjacent-skill probes — rows from the rarest clusters of your own data, used as forgetting probes. They are rarer themes inside your data, not neighbors from outside it.
• Memorization probes — 25%-prefix truncation. If the trained model can continue the rest of a training row from its prefix, it memorized.

Per-group cap: _MAX_CANARIES_PER_GROUP = 1024. The canary set is written to evals/canaries.json by default (--output changes it).

Every canary is drawn from the file you pass in. "Held-out" therefore means held out of the evaluation, not out of training: if you want them held out of training too, remove those rows from the training file yourself.

soup eval lock — pin the suite

soup eval lock my-design.json

Locks the design as a SHA-256-checksummed eval_suite artifact via canonicalise_design_bytes (canonical-JSON for stable hashes across runs), written to evals/locked.json unless you pass -o. It is not registered anywhere by default: pass --attach-to-registry <id-or-name> to attach the frozen LockedSuite (path / sha256 / dimension_count) to a v0.26 registry entry. If that attach fails the command prints a warning and still exits 0.

soup eval coverage — gap analysis

soup eval coverage my-design.json --task reasoning

Checks the locked design against the v0.54.0 TASK_CATEGORIES taxonomy and the _RECOMMENDED_SCORERS allowlist (e.g. reasoning → (rlvr, judge), format_conversion → (regex, rlvr)). Returns a CoverageReport with concrete gap recommendations. --task is required and is one of factual_lookup, style_shaping, format_conversion, reasoning, tool_use, summarization or classification; anything else exits 2.

soup eval gate-install — git regression gate

soup eval gate-install --baseline run-id-7f3a

Writes .git/hooks/pre-push (atomic, POSIX 0o755). --suite names the locked suite (default evals/locked.json), --hook-path writes somewhere else, and --force overwrites an existing hook; without it an existing file is refused, and so is a symlink at the hook path. The hook is a thin wrapper around soup eval against, and three things about it are easy to miss:

1. It does nothing unless SOUP_CANDIDATE_RUN_ID is set. With the variable unset the hook prints SOUP_CANDIDATE_RUN_ID not set; skipping regression gate. and exits 0, so the push goes through ungated. Set the variable in the environment that runs git push (a CI job, for instance) before you rely on the hook.
2. It does not run an eval. With the variable set it runs soup eval against <baseline> --candidate $SOUP_CANDIDATE_RUN_ID --suite <suite> --json-only, which compares metric series that are already recorded in the experiment tracker for the two runs. The locked suite is only checked to exist and parse; a missing or unparseable suite, or a run with no recorded series, blocks the push (the hook turns any non-zero exit into exit 1).
3. It checks one metric, task_accuracy. soup eval against takes one --metric per call and the hook passes none, so refusal_rate, format_validity and p95_latency_ms are checked only if you call soup eval against --metric ... yourself. The verdict is a paired-bootstrap interval (paired_bootstrap_ci, n_samples bounded to [100, 100_000]) compared against the built-in GateThresholds defaults, and decide_regression is direction-aware through _METRIC_DIRECTION: higher-is-better for accuracy, lower-is-better for latency.

The hook script is rendered via render_pre_push_hook with shlex.quote for every interpolated path — no shell injection.

soup eval against — the check the hook runs

The hook above is a wrapper. This is the command inside it, and you can run it by hand any time you want to compare two runs without pushing anything.

soup eval against <baseline-run-id> --candidate <candidate-run-id>

It reads the per-row metric series for both runs from the experiment tracker, runs the same decide_regression on the paired delta, and exits 0 when there is no regression and 1 when there is, so it drops into any CI step, not only a git hook.

The --suite behaviour is the part worth knowing: the generated pre-push hook passes the locked suite through, so a suite that has gone missing fails the gate instead of quietly turning it off. An unset SOUP_CANDIDATE_RUN_ID still skips the whole check, suite included; see the hook section above.

See also

• Quant-check — same idea, but for quant-induced regression
• Eval-gated training — halt training when quality drops
• Registry — where eval_suite and canaries artifacts live
• Eval depth (v0.65) — 4 deeper optional probes (behavior, capability, checklist, irt-subset) that stack on top of v0.55 dimensions
• Post-train x-rays (v0.66) — 4 mechanistic-interpretability probes (sae-diff, sleeper, interference, pack); failure-mode coverage 7 → 11