Long Context (v0.49.0)

Five RoPE-scaling types for 128k+ context fine-tuning: linear, dynamic, yarn, longrope and llama3.

YaRN

training:
  rope_scaling_type: yarn   # linear | dynamic | yarn | longrope | llama3
data:
  max_length: 32768         # the scaling factor is derived from this and the model's native context

The RoPE keys are flat fields on training, not a nested rope_scaling block.

How the factor is chosen. Soup divides data.max_length by the model's own max_position_embeddings and applies no scaling at all unless data.max_length is larger. yarn_factor, yarn_attn_factor, yarn_beta_fast and yarn_beta_slow are accepted by the schema, but only with rope_scaling_type: yarn, and nothing in v0.75.2 reads them: no trainer passes them on, so setting them does not change the run. rope_scaling_type itself is read only by the SFT trainer's transformers text path; the DPO, GRPO and pretrain trainers and the MLX backend accept the key and ignore it.

What the SFT trainer does with it. It writes the derived factor into model.config after the model has been loaded and prints Long-context enabled. The rotary embeddings are built when the model is constructed, and in a check on a small randomly initialised Llama (transformers 5.17; yarn, linear and dynamic) that later edit changed neither the rotary frequencies nor the outputs. The check was not run on a real checkpoint, so do not rely on this key to extend a model's context at v0.75.2.

Dynamic NTK

data:
  max_length: 65536
training:
  rope_scaling_type: dynamic

Dynamic NTK has no factor key, and neither do linear, longrope or llama3. The scaling factor is derived from data.max_length against the model's native context, so you ask for a longer context by raising max_length. The yarn_* keys are refused unless rope_scaling_type: yarn.

LongLoRA S² shifted-sparse attention

training:
  use_longlora: true

Live: the forward-pass override is applied by the SFT trainer, and the group size is derived rather than configured.

Llama 3.1 NTK-aware scaling

Set rope_scaling_type: llama3 and Soup wires up the Llama 3.1-style NTK-aware schedule, with the same derived factor as the other types rather than a fixed 8. Naming the type is required: the SFT trainer only applies long-context scaling when rope_scaling_type is set, so there is no auto-detect of a Llama 3.1 base from soup train.

Gates: validate_longlora_compat + is_llama_model reject incompatible architectures at config-load. Pair with Multipack to keep variable-length samples efficient on long-context runs.