Fine-tune Llama 3.1 with LoRA using Soup CLI

This guide shows how to fine-tune Meta Llama 3.1 8B with LoRA adapters on a custom dataset using Soup CLI. End-to-end from install to inference on a single GPU; how long the training step takes depends on your dataset size, sequence length and card, and no timing is promised here.

Why LoRA on Llama 3.1?

LoRA (Low-Rank Adaptation) trains only a small fraction (~0.1%) of model parameters, which means:

• Train 8B parameter Llama 3.1 on a single 24GB GPU (RTX 4090, A10)
• Checkpoints are tiny (~100MB instead of 16GB)
• Faster training and easier experimentation

1. Install

pip install "soup-cli[train,fast]"

Since v0.71.0 the base soup-cli package is a light, PyTorch-free CLI. [train] adds the training stack; [fast] adds Unsloth on top. Installing [fast] alone cannot train.

The [fast] extra adds Unsloth for 2–5× training speedup on Llama-family models.

2. Prepare your dataset

Use the Alpaca format (JSON list of instruction/input/output triples):

[
  {
    "instruction": "Summarize the following text.",
    "input": "Soup CLI is a fine-tuning toolkit...",
    "output": "Soup CLI is an open-source LLM fine-tuning tool."
  }
]

Save as train.json.

3. Create the config

Save as llama31.yaml:

base: meta-llama/Llama-3.1-8B-Instruct
task: sft
backend: unsloth            # root-level, NOT under training

data:
  train: train.json
  format: alpaca
  max_length: 2048          # sequence length lives under data

training:
  epochs: 3
  lr: 2.0e-4                # the key is lr, not learning_rate
  batch_size: 2
  gradient_accumulation_steps: 8
  lora:
    r: 16                   # LoRA turns on when r > 0; there is no enabled flag
    alpha: 32
    dropout: 0.05
    target_modules: [q_proj, k_proj, v_proj, o_proj]

output: ./runs/llama31

Spelling matters, and until v0.74.0 nothing told you so. Writing learning_rate instead of lr, or nesting backend under training, used to be dropped in silence and the default used instead. Since v0.74.0 loading a config walks the whole tree and reports every key it cannot place, naming the field you probably meant, and since v0.75.0 the same config is refused: soup train exits 1 before the training stack is imported. If a setting seems to have no effect on an older wheel, read the unknown-key report and check the spelling against Configuration.

4. Train

soup train --config llama31.yaml

Soup auto-detects GPU, enables FlashAttention, and trains LoRA adapters. How long it takes depends on your sequence length, batch size and hardware; soup profile --config llama31.yaml estimates memory and speed before you start.

5. Chat with your fine-tuned model

soup chat --model ./runs/llama31

6. Export for deployment

# Merge LoRA into base model and export GGUF for Ollama
soup export --model ./runs/llama31 --format gguf --quant q4_k_m

Soup writes the training output (the adapter and tokenizer) straight into the output directory, so ./runs/llama31 above is the config's output: value, and there is no latest subdirectory. Without --output, the GGUF is written next to the model directory as ./runs/llama31.q4_k_m.gguf.

Troubleshooting

Out of memory? Enable QLoRA (a 4-bit base model). Note 4bit is already the default, so this is only worth setting explicitly if you changed it:

training:
  quantization: 4bit
  lora:
    r: 16

Slow training? Ensure backend: unsloth is set at the root of the config (not under training) and that you installed pip install "soup-cli[train,fast]".

Next steps

• Export to GGUF and Ollama
• Multi-GPU training with DeepSpeed
• Training methods reference