Export fine-tuned models to GGUF and deploy on Ollama

After training with Soup CLI, export your model to GGUF format and serve it locally with Ollama in three commands.

0. Build llama.cpp first, if you want a quantized GGUF

Soup auto-clones llama.cpp (pinned at tag b5270) into ~/.soup/llama.cpp on first use, but it does not build it. Converting to f16 or f32 works from the Python script alone. Every quantized type additionally needs the llama-quantize binary, so build it once:

cd ~/.soup/llama.cpp
cmake -B build -DGGML_NATIVE=OFF -DLLAMA_CURL=OFF -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release --target llama-quantize -j 4

Soup finds the binary in both the single-config layout (build/bin/llama-quantize, from Make or Ninja) and the multi-config one (build/bin/Release/llama-quantize.exe, from MSVC or Xcode), or on your PATH. To use a checkout you already have, pass --llama-cpp /path/to/llama.cpp or set LLAMA_CPP_PATH.

On Windows the validated toolchain is Visual Studio 2022 Build Tools with the "Desktop development with C++" workload plus CMake 3.14 or newer, CPU-only. Linux and macOS need only a C++ toolchain and CMake. CUDA builds of llama.cpp are untested.

Do not run pip install -r ~/.soup/llama.cpp/requirements.txt. It pins torch~=2.2.1 against the CPU wheel index, so it will downgrade a CUDA PyTorch and break training. Soup deliberately installs only the convert script's own extras (gguf, sentencepiece, protobuf), unpinned.

1. Export to GGUF

soup export --model ./runs/my-model \
            --format gguf \
            --quant q4_k_m \
            --output ./my-model.gguf

Quantization levels:

--quant accepts exactly these six values (q4_0, q4_k_m, q5_k_m, q8_0, f16, f32) and exits 1 on anything else, so the smaller llama.cpp rungs like q2_k and q3_k_m are not reachable through Soup.

2. Deploy to Ollama

Soup CLI ships with Ollama integration (v0.18.0+):

soup deploy ollama \
    --model ./my-model.gguf \
    --name my-model \
    --template chatml

This creates an Ollama Modelfile, imports the GGUF, and registers your model.

3. Chat

ollama run my-model

Or via the API:

curl http://localhost:11434/api/generate -d '{
  "model": "my-model",
  "prompt": "Hello!"
}'

One-liner: train → export → deploy

soup train --config soup.yaml && \
soup export --model ./output --format gguf --quant q4_k_m --output ./my-model.gguf && \
soup deploy ollama --model ./my-model.gguf --name my-model --yes

soup train writes to the config's output: directory (./output unless you set it), and there is no latest subdirectory, so point --model at that directory. Without --output, soup export names the file <model-name>.<quant>.gguf and writes it next to the model directory (./output.q4_k_m.gguf), not inside it, so pass --output when a later command has to find the file. --yes skips the confirmation prompt of soup deploy ollama.

Running the llama.cpp programs with soup llama

soup llama <subcommand> [args...] starts one of five llama.cpp programs and forwards your arguments to it unchanged:

soup llama quantize ./my-model.f16.gguf ./my-model.Q5_K_M.gguf Q5_K_M
soup llama server -m ./my-model.gguf --port 8080

The arguments are llama.cpp's own syntax, not Soup's, and they depend on the llama.cpp build you installed. A few things are Soup's:

• The program must be on your PATH. It is looked up by name only. Setting LLAMA_CPP_HOME does not make Soup find it, and neither does the ~/.soup/llama.cpp/build/bin directory from step 0 above: soup export searches that directory itself, soup llama does not, so add it to PATH first.
• When the program is missing the command prints llama-quantize not found on PATH; install llama.cpp or set LLAMA_CPP_HOME (with the name of the program you asked for) and exits 2.
• --help after the subcommand shows Soup's help, not the program's. Run the program itself for its own options.
• Arguments are limited: at most 64, each at most 1024 characters, with no NUL or line break in any of them. Otherwise it exits 2 before starting anything.
• The program runs with a reduced environment. HF_TOKEN, OPENAI_API_KEY and every other variable outside a short allowlist are dropped (the list is in the CLI reference). Its output goes straight to your terminal and its exit code becomes the exit code of soup llama.

Other export formats

Soup CLI also supports:

• ONNX — --format onnx for cross-platform inference
• TensorRT-LLM — --format tensorrt for NVIDIA optimized serving
• AWQ / GPTQ — --format awq or --format gptq plus --calibration-data <calib.jsonl> (required for both) for quantized GPU inference
• Hugging Face — not an export format. A merged checkpoint comes from soup merge --adapter <path> --output <dir>. The formats soup export accepts are gguf, gguf-ud, onnx, tensorrt, awq, gptq, bitnet, tq1_0 and torchao.

Related

• Serving with vLLM and SGLang
• Fine-tune Llama 3.1 with LoRA