Training Methods

Soup supports 23 training tasks via the task config key. That number is the exact cardinality of the task field in the config schema, not a marketing round-up: the full list is in All Training Tasks below. Switching method means changing one line of YAML, not changing tools.

soup edit set (ROME / MEMIT / AlphaEdit) and soup steer train (CAA / ITI / RepE) are not task: values. They are weight-surgery and inference-time commands, documented under Unlearning & knowledge editing and RAG & steering.

If the model does not fit in your card, add training.stream_layers: true and the frozen base never enters VRAM at all. That works for sft and, since v0.72.4, for dpo, orpo, simpo and kto, with DPO's reference model taken from the same stream with its adapters switched off, so it costs no extra weights. See Layer streaming and Preference losses over streaming.

Variables that move Soup's own files, and the ones that carry credentials, are listed on Environment variables and state files.

Supervised Fine-Tuning (SFT)

The most common method. Train on instruction-response pairs.

base: meta-llama/Llama-3.1-8B-Instruct
task: sft

data:
  train: ./data/train.jsonl
  format: alpaca

training:
  epochs: 3
  lr: 2e-5
  batch_size: auto
  quantization: 4bit
  lora:
    r: 64
    alpha: 16

Two things SFT refuses since v0.74.0 (#535). A row that leaves nothing to learn is refused rather than trained as an all-masked sequence. On the text path with assistant-only masking on (the default, data.train_on_responses_only: true), a row whose assistant reply is missing, or is cut off entirely by data.max_length, stops the run before training with a message like train row 3: no causal-loss target remains after tokenization/truncation at data.max_length=2048; the assistant response is absent or fully truncated. Increase data.max_length or shorten the prompt. The row is named by split and number, and the same check covers pre-tokenized datasets. Long answers under a small max_length are how most people meet it. Second, the final save is refused (refusing to save the final model) when the last logged loss, grad_norm, entropy, train_loss or eval_loss is NaN or infinite, or when any trainable parameter is, so a run that diverged does not write an adapter that merely looks finished. A metric that was non-finite earlier in the run and then recovered does not trigger it.

Direct Preference Optimization (DPO)

Train with preference pairs (chosen vs rejected).

base: meta-llama/Llama-3.1-8B-Instruct
task: dpo

data:
  train: ./data/preferences.jsonl
  format: dpo

training:
  dpo_beta: 0.1
  quantization: 4bit
  lora:
    r: 64
    alpha: 16

Group Relative Policy Optimization (GRPO)

Reasoning training (DeepSeek-R1 style) with reward functions instead of a reward model.

base: meta-llama/Llama-3.1-8B-Instruct
task: grpo

data:
  train: ./data/reasoning_train.jsonl
  format: sharegpt
  max_length: 4096

training:
  grpo_beta: 0.1
  num_generations: 4
  reward_fn: accuracy     # or 'format', or path to custom .py
  quantization: 4bit
  lora:
    r: 64
    alpha: 16

Built-in reward functions:

• accuracy — checks if the final answer matches expected (supports #### and \boxed{} formats)
• format — checks for structured <think>...</think> reasoning blocks

Custom reward functions — point to a Python file:

# my_reward.py
def reward_fn(completions, **kwargs):
    # kwargs carries your dataset's own columns, row-aligned with completions.
    golds = kwargs["answer"]
    return [1.0 if g in c[-1]["content"] else 0.0 for c, g in zip(completions, golds)]

What is in kwargs. Source columns are preserved and passed through, and the trainer additionally exposes the target as answer: an alpaca output, or the final assistant turn of a chat row. Required columns are validated before generation starts, so a missing one refuses the run rather than producing a silently useless reward: accuracy and the verifiable math domain need answer, verifiable code needs expected or answer, and json_schema needs schema.

What a reward function must return. Exactly one finite number per completion, in the order of completions. The GRPO trainer wraps every reward function (built-in, .py file or ensemble member) in a check, and stops with a message naming the function when it returns the wrong count, a non-number, nan or infinity, or a boolean: return 1.0 and 0.0, not True and False. A comma-separated reward_fn: "accuracy,format" trains an ensemble and is refused on every task except grpo; reward_fn: verifiable is refused unless verifiable_domain (math, code or json_schema) is set.

PPO / Full RLHF Pipeline

Three-step pipeline: SFT warmup -> Reward Model -> PPO alignment.

# Step 3: PPO alignment
base: meta-llama/Llama-3.1-8B-Instruct
task: ppo

data:
  train: ./data/prompts.jsonl
  format: chatml

training:
  reward_model: ./output_rm   # From step 2
  epochs: 1                   # passes over the prompt dataset
  ppo_epochs: 4               # optimization passes inside each PPO update
  ppo_clip_ratio: 0.2
  ppo_kl_penalty: 0.05
  quantization: 4bit
  lora:
    r: 64
    alpha: 16

The two epoch keys are different things. epochs (default 3) is complete passes over the training data; ppo_epochs is the optimization passes within each PPO update. Soup forwards both, plus ppo_kl_penalty, to the names the installed TRL PPOConfig uses.

All Training Tasks

All 23, exactly as the schema accepts them.

Running Training

# Start training
soup train --config soup.yaml

# Resume from checkpoint
soup train --config soup.yaml --resume auto
soup train --config soup.yaml --resume ./output/checkpoint-500

# With W&B logging
soup train --config soup.yaml --wandb

# With TensorBoard
soup train --config soup.yaml --tensorboard

# With DeepSpeed (multi-GPU)
soup train --config soup.yaml --deepspeed zero2

# With FSDP2
soup train --config soup.yaml --fsdp full_shard

# Skip confirmation
soup train --config soup.yaml --yes

# Continual learning: rehearse an old dataset so the new task
# does not erase it (v0.71.36, sft/pretrain only)
soup train --config new_task.yaml --replay old_task.jsonl --replay-ratio 0.1

--replay interleaves a seeded sample of old_task.jsonl into training; --replay-ratio is the fraction of the final mixed set. Validation stays pure new-task. See Data Moat II for the full mechanism and its measured limits.

The same three settings exist as config keys, for a soup.yaml you commit: data.replay (the path), data.replay_ratio (default 0.1; above 0 and at most 0.5, so 0.6 is refused at load) and data.replay_seed (0 to 2147483647; unset means seed 0). The ratio and the seed are refused without data.replay, data.replay is refused on any task other than sft and pretrain, and it is refused alongside training.packing or training.multipack, whose bin-packing breaks the ratio. A run that uses it prints Replay: +N old rows interleaved (P% of M).

Read the console before trusting --find-lr

soup train --config soup.yaml --find-lr sweeps the learning rate and writes a report (how it works), but the sweep is not a dry run of your training run. It loads the base model without quantization and without LoRA, puts the whole model on the device and trains every parameter with plain AdamW, one row at a time, on at most 256 tokens of each of the first rows of your data. So it needs the full-precision model to fit, and the rate it recommends is one for full fine-tuning, so it does not tell you the rate for the LoRA run you will launch. It reads each row's text, prompt or messages field only; a row in any other shape, an alpaca instruction and output row for instance, is tokenized as a blank string.

If the sweep cannot run at all (the config fails to load, the base does not fit unquantized, a dependency is missing), Soup prints a yellow line ending writing synthetic curve, writes a report from a built-in placeholder curve and exits 0. The report file carries no marker saying so, so a recommended learning rate from a run that printed that line is not a measurement.