LISA (v0.71.34)

LISA (Layerwise Importance Sampled AdamW, arXiv:2403.17919) reaches for full-fine-tuning quality without a full fine-tune's memory. On the quality half it delivers, measured at 3B and 8B. On memory it beats full fine-tuning and loses to LoRA, by a margin that widens with scale, which is measured below rather than glossed. Where Spectrum picks the layers once and trains that fixed set, LISA re-samples a small random set of decoder layers every N steps and freezes the rest. The input embeddings, the LM head, and the final norm stay trainable throughout.

task: sft
backend: transformers
modality: text
training:
  quantization: none       # LISA is a full fine-tune of the active layers
  lisa_enabled: true
  lisa_num_layers: 2       # decoder layers active per interval (clamped to model depth)
  lisa_interval_steps: 20  # re-sample cadence, in global steps

Config fields

Setting any lisa_* field away from its default while lisa_enabled is false is a hard error, not a silent no-op.

lisa_train_embeddings: false is a trade, not a free win, which is why it is opt-in. The always-on group is about 70% of everything LISA trains at 8B, so it is where LISA's memory actually goes, and freezing it changes what the run can learn. Two riders travel with it: the analytical VRAM pre-flight still treats LISA as a full fine-tune regardless of the flag, because crediting the frozen-embeddings saving needs a measured constant nobody has taken on GPU hardware, so a run that would now fit can still be conservatively refused, and soup train --allow-oom-attempt is how you launch it anyway.

LISA also accepts task: pretrain as of v0.74.0. Continued pre-training is the same rotating-full-fine-tune mechanism LISA was built for, so the supervised-only gate it had inherited from Spectrum was arbitrary. The rest of the gate is unchanged: transformers, text, quantization: none, and mutually exclusive with the LoRA feature flags, freeze_layers / freeze_ratio and unfrozen_parameters.

Why it saves memory, and how much

Only a handful of layers train at any moment, and their optimizer state is cleared when they are re-frozen, so peak optimizer memory is roughly embeddings + head + lisa_num_layers, far below a full fine-tune. Unlike a static freeze, the active set keeps moving, so updates spread across the depth of the model instead of staying pinned to one slice. The set is drawn at random each interval, so that is a tendency rather than a guarantee: a short run fires few intervals and will not reach every layer.

Measured at 3B and 8B, and half the pitch did not survive

LISA is often described, here included, as full fine-tuning quality at something like LoRA's memory. On an H100 80 GB, over Alpaca with 200 steps and three interleaved repeats per arm, the quality half holds and the memory half does not.

The LoRA rows predate v0.74.0. The same measurement session found that every non-quantized model loaded in float32 (#339), the defect v0.74.0 fixed for a frozen base, and the two LoRA rows were measured before that fix, so a LoRA run today needs less memory than these rows say. A trainable base, as in LISA or full fine-tuning, still loads in fp32 on purpose, so those rows are not affected by that change. The 1.22x and 1.51x figures below are what was measured; no re-measured ratio has been published, so no updated figure exists.

LISA beat full fine-tuning at both learning rates, so the quality claim stands. But it costs 1.22x LoRA's memory at 3B and 1.51x at 8B, and the gap widens with scale, because the embeddings, the LM head and the final norm stay trainable through every interval and account for 70.7% of everything LISA trains at 8B. The arithmetic in the paragraph above is correct and still reads as a win over LoRA. It is not one.

Stated positively, which is the honest version of the pitch: LISA trains an 8B on a single 80 GB card where full fine-tuning needs roughly 120 GB and cannot run at all. Reach for it when you want closer-to-full-fine-tune behaviour and LoRA is not enough, not when you want to save memory over LoRA.

Two caveats on the table. Held-out quality here is in-distribution loss and token accuracy on an Alpaca validation split, not a downstream benchmark. And raising lisa_num_layers costs quality as well as memory on this data: at 3B the held-out loss went 1.2504 at 2 layers, 1.2673 at 8 and 1.2950 at 16, while at 8B anything above 8 exhausts an 80 GB card. lisa_interval_steps was indistinguishable anywhere from 1 to 50.

Gates

LISA is sft or pretrain (since v0.74.0) + transformers + text + quantization: none only. It is mutually exclusive with LoRA features, freeze_layers / freeze_ratio, and Spectrum's unfrozen_parameters, because each of those independently decides what trains and stacking them is a footgun, not a feature. It is also mutually exclusive with train_router_only, expand_layers, freeze_trainable_layers, moe_lora, relora_steps, loraplus_lr_ratio and stream_layers (a streamed base is frozen on the meta device, so per-step layer sampling has nothing to sample). Every reject config fails with a specific message.

batch_size: auto is not LISA-aware. The static estimate sizes the trainable set from lora.r as if a small adapter were all that trains, and the CUDA probe that refines it runs one forward and backward with no optimizer step, so it never measures the Adam state for the embeddings, head and active layers (it also runs while every layer is still trainable, so its gradient figure is not LISA's either). Do not trust the number it picks: set batch_size explicitly, or start low and raise it (what auto does).

Implementation note: the model is left fully trainable at trainer-setup time so Hugging Face's optimizer (built before the first callback fires) contains every decoder parameter; the LISA callback then toggles requires_grad per interval, and frozen parameters produce no gradient so the optimizer skips them. This ordering invariant is what makes the callback work at all.

Live on a 4 GB GPU for small models: a 4-epoch SmolLM2-135M-Instruct run completes with no callback crash (loss 0.66, token accuracy 0.80).

See also

• Spectrum targeted training — the static, SNR-ranked counterpart.
• PEFT & efficiency — LoRA, PiSSA, ReLoRA, per-pattern rank.
• Adapter algebra — the other half of the v0.71.34 release.