Experiment Tracking

Every soup train run is automatically tracked in a local SQLite database (~/.soup/experiments.db).

List Runs

soup runs

Shows all training runs with task, model, status, and final loss. The run database at ~/.soup/experiments.db is machine-wide, so this lists every run on the box; add --cwd-only to narrow it to runs whose output_dir sits under the current directory.

Run Details

soup runs show run_20260223_143052_a1b2

Detailed info including config, metrics, and an ASCII loss curve.

A run that is not running no longer says it is

Until v0.73.3 a run whose watcher process died was reported running forever, and the listing hardcoded that word for any row it could not otherwise classify. That trained you to ignore the one status field that guards against starting a second concurrent run.

Both soup runs and soup runs show now reconcile on read: a running row whose recorded process is gone is rewritten to terminated with an unknown exit code. Unknown, deliberately — a lost outcome is never recorded as a success — and the richer completed and failed statuses are left exactly as they are. Only runs recorded with a process id can be checked this way, so a run without one is left alone rather than guessed at.

On Windows this depended on a second fix, and it is a good example of a bug that cannot be reasoned around: the API that reports a process's exit code returns 259 for a process that is still alive, but 259 is also a perfectly legal exit code, so a child that genuinely exited with 259 was indistinguishable from one still running. Liveness is now decided by waiting on the process handle with a zero timeout, which is signalled the instant the process exits whatever code it exits with.

Compare Runs

soup runs compare run_1 run_2

Side-by-side comparison of two runs with loss curves and metrics.

Delete Runs

soup runs delete run_1

Reclaim disk without losing the run

Deleting a run is the blunt option. Checkpoints are what actually fill the disk, and most of that weight is optimizer state you will never load again.

soup runs clean run_1 --dry-run        # estimate the saving first, delete nothing
soup runs clean run_1                  # surgical: drop optimizer states, keep the weights
soup runs clean run_1 --no-keep-weights # the one way to delete whole checkpoints
soup runs clean --all                  # every historical run

If you typed --keep-weights on Click 8.1, it did the opposite and deleted whole checkpoints. The option was a bare flag with a True default, and Click 8.1 treats that as a toggle, so passing it explicitly parsed as False. Anyone who read the flag name, typed it to be safe, and had Click 8.1 installed lost their non-best checkpoints. v0.75.1 makes it the paired --keep-weights/--no-keep-weights, so the default and the explicit flag now mean the same thing on every supported Click version, and --no-keep-weights is the one way to delete. Upstream added a repo-wide check that fails any True-default flag without a negative form, so this shape cannot come back.

--keep-weights defaults to on, which is the whole design: it deletes optimizer.pt from the lesser checkpoints and leaves the model weights, so you can still evaluate or export from any of them. --dry-run estimates the saving without touching anything, and --force skips the confirmation for CI.

This matters more than it sounds at scale: a LoRA run training a fraction of a percent of the model can drag a checkpoint hundreds of times the size of the adapter it produced.

Model Evaluation

Soup includes a comprehensive evaluation platform (v0.19.0+):

pip install "soup-cli[eval]"

# Run benchmarks (mmlu, gsm8k, hellaswag, etc.)
soup eval benchmark --model ./output --benchmarks mmlu,gsm8k

# Custom eval tasks from JSONL
soup eval custom --model ./output --tasks ./eval_tasks.jsonl

# LLM-as-a-judge over saved outputs: --target is a JSONL file of
# {"prompt": ..., "response": ...} rows (soup infer writes this shape)
soup eval judge --target ./results.jsonl --model gpt-4o --provider openai

# Run the benchmarks and custom tasks listed in the eval block of soup.yaml
soup eval auto --config soup.yaml

# Compare eval results between runs
soup eval compare run_1 run_2

# Local leaderboard across models
soup eval leaderboard

# Human A/B evaluation with Elo ratings
soup eval human --model-a ./model_v1 --model-b ./model_v2 --input ./prompts.jsonl

On soup eval judge, --model names the judge model, and --target is the file of pairs to score, not the model under test. Every row needs prompt and response, or the command exits 1 naming the line.

soup eval auto takes its work from an eval block in the config, and --benchmarks and --tasks override the two lists:

eval:
  benchmarks: [mmlu, gsm8k]       # lm-evaluation-harness task names
  custom_tasks: ./eval_tasks.jsonl

It needs the output directory to exist, so run it after soup train, and it exits 1 when neither list is set. The block has two more keys that this release does not act on. eval.auto_eval is accepted by the schema, but nothing in v0.75.2 hands the eval block to the training callback, so a finished soup train does not start an evaluation by itself whatever it says. eval.judge is accepted and read by no code path. Run soup eval auto or soup eval judge yourself. soup eval auto --trust-remote-code is likewise accepted, but nothing in v0.75.2 reads it: the value is not passed on to the benchmark or custom-eval steps it runs, so it does not change how they load the model.

Aider Polyglot (soup eval aider, v0.74.0)

pip install "soup-cli[aider]"
git clone https://github.com/Aider-AI/aider.git
cd aider
./benchmark/docker_build.sh
cd ..
git clone https://github.com/Aider-AI/polyglot-benchmark.git

soup eval aider --model openai/gpt-4.1 --output ./aider-results \
  --exercises-dir ./polyglot-benchmark --run-id run_20260301_143052_a1b2

This scores a model through Aider's own benchmark harness, in Aider's Docker image. --model is an Aider / LiteLLM model identifier such as openai/gpt-4.1, not a path to a local checkpoint. The [aider] extra alone does not make the command runnable, because the aider-chat wheel does not ship the harness: you build the aider-benchmark image from an Aider checkout and clone the exercises once, as above. Docker has to be running, and Soup checks the Docker CLI, the daemon and the image before it starts. The exercises are mounted read-only (a directory outside the current one is allowed, with a warning), and provider credentials reach the container by environment-variable name, never as a value on the command line: OPENAI_API_KEY, OPENAI_API_BASE, ANTHROPIC_API_KEY, GEMINI_API_KEY, OPENROUTER_API_KEY and the three AZURE_API_* variables, whichever are set.

--output is a directory, not a report file, and it must sit under the current directory. It receives Aider's per-exercise results and a soup_result.json with the model, the score, the error counts and aggregate details. --run-id must name an existing run; the aider_polyglot score is stored under it, which is what lets soup eval compare run_before run_after compare two of them. --threads (1 to 64, default 1), --num-tests (default -1, every exercise) and --timeout (60 to 86400 seconds, default 86400) bound the run. --allow-host-services lets the container reach services on the host, for a trusted local OpenAI-compatible endpoint; it also lets the model-written code the benchmark executes reach those services, so leave it off for a remote provider. The benchmark runs model-generated code, and Soup offers no host-execution fallback.

Hyperparameter Sweep

Search for the best hyperparameters:

# Grid search
soup sweep --config soup.yaml --param lr=1e-5,2e-5,5e-5 --param lora_r=8,16,32

# Random search with max runs
soup sweep --config soup.yaml --param lr=1e-5,2e-5,5e-5 --strategy random --max-runs 5

# Preview without running
soup sweep --config soup.yaml --param lr=1e-5,2e-5 --dry-run

# Early stopping: skip remaining runs if loss exceeds 1.5x best
soup sweep --config soup.yaml --param lr=1e-5,2e-5,5e-5 --early-stop 1.5

Model Comparison

Compare outputs of two models side-by-side:

soup diff --model-a ./model_v1 --model-b ./model_v2 --prompt "Explain gravity"
soup diff --model-a ./base --model-b ./finetuned --prompts test_prompts.jsonl
soup diff --model-a ./a --model-b ./b --prompts prompts.txt --output results.jsonl

Both models must be existing local paths (exit 1 otherwise). For an adapter directory, soup diff reads the base model from its adapter_config.json; --base-a and --base-b name the base for model A and model B explicitly, and are required when that file records none (the command exits 1 with Cannot detect base model). A full (non-adapter) model is loaded from its own directory; the base is not used to load it. Custom modeling code is denied by default: a local model whose config.json has an auto_map is refused unless you pass --trust-remote-code.

Batch Inference

Run a model on a list of prompts:

soup infer --model ./output --input prompts.jsonl --output results.jsonl
soup infer --model ./output --input prompts.txt --output results.jsonl \
  --max-tokens 512 --temperature 0.3

Output is JSONL with prompt, response, and tokens_generated fields.

Training Profiler (v0.23.0+)

Estimate memory, speed, and GPU requirements before training:

soup profile --config soup.yaml            # model, task and quantization come from the config

Shows estimated GPU memory, training speed, and hardware recommendations. Add --json to print the same figures as one JSON object (memory components, total_memory_gb, tokens_per_sec, samples_per_sec, recommended_batch_size, compatible_gpus, gpu_memory_gb) for scripting.

Adapter Management (v0.22.0+)

# Scan directory for LoRA adapters
soup adapters list ./experiments

# Show adapter metadata (base model, rank, size)
soup adapters info ./output

# Compare two adapters side-by-side
soup adapters compare ./adapter_v1 ./adapter_v2

Logging Integrations

TensorBoard

soup train --config soup.yaml --tensorboard
tensorboard --logdir ./output/runs/

Weights & Biases

soup train --config soup.yaml --wandb

--tensorboard and --wandb cannot be used together.