Training Speed & Memory

Soup v0.28.0 shipped six throughput and memory features. They were SFT-only in v0.28, a gate lifted in v0.33 and finished in v0.35; tasks outside the 13-task allowlist (see Which tasks below) are still refused at config load, so a task that would ignore these keys is refused rather than allowed to ignore them (inside the allowlist, one exception: kernel_auto_compose is not read by the SFT trainer). Not every feature below does what its name suggests at v0.75.2: gradient checkpointing tiers and kernel auto-composition are described as they actually behave, and cross-document attention masking is a refused key, not a working mask.

If the model does not fit at all, none of these six is the answer. They shave a resident run; they cannot make a model smaller than your card. For that, see layer streaming: the frozen base never loads resident, so peak VRAM is bounded by one decoder layer instead of the whole model, and Llama-3.1-8B fine-tunes on a 4 GB card. Since v0.72.4 that covers preference losses too, not just supervised fine-tuning. It is BETA, and it is slower than resident training, so reach for it only when resident training is impossible.

Cut Cross-Entropy

training:
  use_cut_ce: true

Cut CE avoids materialising the full [seq_len × vocab_size] logits tensor during the cross-entropy backward pass. Biggest expected win on models with vocab ≥ 128k (Llama 3, Qwen 3), when it engages (see the log note below). Architecture detection reads config.model_type first, so a local checkpoint directory resolves as correctly as a hub id; a substring match on the last path component (model_name.rsplit("/", 1)[-1]) is only the fallback when no config can be read. That order was the v0.74.0 repair: before it, name matching came first, so a soup merge output called checkpoint-2000 matched nothing and cut-cross-entropy stayed off on a flag you had explicitly set. It also means org prefixes like deepseek-ai/...-phi-... can't trigger the wrong patcher.

Install the optional dep:

pip install "soup-cli[cce]"

Check the log: it may not engage. use_cut_ce patches the model only if the third-party cut_cross_entropy.transformers.cce_patch imports. When it cannot, or no patcher matches the architecture, the run continues without it and prints a yellow Cut Cross-Entropy: no matching architecture or cut_cross_entropy not installed. That line is also what you see when the package is installed and an import inside it fails, and nothing checks beforehand that the package is present. A success prints Cut Cross-Entropy enabled. The v0.73.0 probe record (see Measurement records) could not make the flag engage on any pip-installed stack it tried: cut-cross-entropy 25.1.1 ships no transformers submodule, and 25.9.3 failed importing symbols that transformers 4.57.6 no longer exports. That was measured on transformers 4.57.6, not on the 5.x that v0.75.2 requires, and it has not been re-measured, so whether the current cut-cross-entropy release engages on transformers 5.x is unverified. Treat the memory saving as unproven until your own log says it is enabled.

FP8 training (Hopper+)

training:
  quantization_aware: fp8
  fp8_recipe: tensorwise   # tensorwise | rowwise | rowwise_with_gw_hp

fp8_recipe picks the torchao scaling recipe and defaults to tensorwise (one scale per tensor). rowwise scales per row, and rowwise_with_gw_hp is rowwise with the weight gradient kept in high precision. The source describes tensorwise as the fastest and the two rowwise recipes as more accurate; none of that has been measured here. The key is refused unless quantization_aware is fp8, and a value outside those three is refused at config load.

Configures float8 linear layers via torchao. Pydantic accepts only true / false / "fp8" — unknown strings like "fp16" are rejected at config load. Hopper or newer (SM 9.0+) is the intended target, but Soup does not check for one at launch. The launch check for this key is the one int8 QAT uses: it refuses the unsloth backend, a quantization other than 4bit or none, and a missing torchao. The stricter FP8 validator (CUDA, Hopper+, transformers backend) exists in the source but has no caller at v0.75.2. Soup then tries to convert the linear layers; if torchao.float8 does not import or the conversion raises, it prints a yellow notice and trains on without FP8 (the SFT trainer's reads "FP8 training requested but unavailable"; the other trainers print "FP8 training: torchao.float8 unavailable or no compatible linears"). The SFT message mentions a Hopper GPU, but no code on that path tests the GPU, so look for the FP8 training enabled line in the log rather than assuming the conversion happened. (fp8_attention, a separate key, does check for a Hopper-class GPU.)

Tiered gradient checkpointing

training:
  gradient_checkpointing: auto   # or: selective | medium | full

The schema accepts false, true, selective, medium, full and auto, but at v0.75.2 every enabled tier does the same thing: it turns on Hugging Face's standard gradient checkpointing of every block (use_reentrant: False). The selective (attention only) and medium (every other block) granularities have a hook installer in the source, but nothing calls it, so the tier name changes only the label Soup prints ("medium (every other block)", "selective (attention only)", and for auto a choice made from your detected VRAM). Treat gradient_checkpointing as on or off; there is no per-tier memory or speed difference to choose between.

When stream_layers is on, Soup leaves Hugging Face's checkpointing off, because the streamed layers already recompute; see layer streaming.

Kernel auto-composition

training:
  kernel_auto_compose: true

On a CUDA host with the transformers backend, this enumerates the installed combinations of Liger Kernel, FlashAttention and Cut Cross-Entropy (plus a no-kernel baseline), times a few forward passes per candidate, and prints Kernel auto-compose picked: <name>. The SFT trainer does not call it at all, so on task: sft the key is accepted and does nothing; every other allowlisted task runs the benchmark and prints the pick (the unified preference task hands off to the DPO, SimPO, ORPO, IPO and BCO trainers, which do). Where it runs, it reports a pick; it does not apply it. Nothing sets use_liger, use_flash_attn or use_cut_ce from the result, so set those yourself. The timing loop also does not switch kernels between candidates (the source calls it a coarse "did anything install correctly" check), so treat the printed name as a hint, not as a measured winner. On CPU, unsloth and MLX there is only the baseline and Soup prints "benchmarking unavailable on this host". pick_best_kernel raises ValueError when every candidate is missing a finite time_ms, so an untimed combo is never promoted when benchmarking fails; NaN / None times are treated as +inf.

Cross-document attention masking

training:
  packing: true      # this is the whole answer

When sample packing is on, documents in one packed sequence must not attend across each other. That isolation is TRL's default bfd strategy, and you get it by setting packing: true with a FlashAttention attn_implementation. There is no second flag to set.

packing_cross_doc_attn_mask is refused at config load since v0.75.0, and it never worked before that. Soup mapped it to TRL packing_strategy="attention_free", which has never been in TRL's allowlist on any released version, so the flag was a TypeError at trainer setup rather than a working mask. It is now rejected by a @model_validator that names bfd / bfd-requeue / wrapped as the real allowlist and points at packing: true. This is a false-capability correction, not a regression: no release ever built that mask. Remove the key from any config that carries it.

Activation offloading

training:
  activation_offloading: cpu    # or: disk

Saved-tensor hooks offload activations to pinned RAM (cpu) or an on-disk scratch dir (disk). The disk variant keeps the mkstemp file descriptor open until torch.save flushes, closing the TOCTOU window between os.close(fd) and torch.save(path). torch.load(weights_only=True) prevents arbitrary Python deserialization on reload. Scratch dirs are containment-checked against cwd. Unsloth and MLX backends are rejected.

Which tasks

These four keys (use_cut_ce, quantization_aware: fp8, kernel_auto_compose, activation_offloading) were SFT-only in v0.28. Multi-trainer wiring landed in v0.33.0 and every transformer-backend trainer was wired in v0.35.0. At v0.75.2 they are accepted for 13 tasks (sft, dpo, pretrain, grpo, kto, orpo, simpo, ipo, bco, preference, ppo, reward_model, embedding) and refused at config load for the other ten (prm, tts, classifier, reranker, cross_encoder, distill, unlearn, moe_lora_routing, online_dpo, asr), and on backend: mlx. See multi-trainer speed and memory for the per-trainer wiring.

See also

• Multi-GPU mastery — ZeRO++ / FSDP2 / pipeline
• Backends — transformers / unsloth / MLX