Train LoRA directly on Quantized GGUFs, no PyTorch, no Parent(FP16) needed, just your GGUF and the data
retained text
I've seen this one floating around quite a bit now and since this is something that can be done without the parent and or other dependencies i thought i'd share something.
Usually when you fine-tune a model you can't:
\- train against the quantized GGUF file itself
\- do it without PyTorch, transformers or a separate Python setup
\- use the exact same GGUF for inference afterwards, without converting the model
(QLoRA does train through a 4-bit base, but in PyTorch and on its own NF4 format, not on the GGUF you serve.)
[With Runner, the CPU training forward pass is the CPU serving forward pass, so the adapter is trained on the exact weights it will run on. CUDA\/Metal serving uses the GPU's own arithmetic, within a measured tolerance but not bit-identical. There is no FP16 copy to keep, and no framework to install.](https://preview.redd.it/roepm5d3cdvh1.png?width=1280&format=png&auto=webp&s=ced357d5d53429c91103a9957d873c349a42a206)
With runner the 16-bit nor the PyTorch package is needed, this is done through "freezing" the weights of the current X-bit GGUF, only a "small" LoRA adapter is trained on top.
Quick-Guide:
# train through the quantized base you will serve
./runner -m base-Q4_K_M.gguf --train data.jsonl --train-steps 200 \
--lora-rank 8 --train-out adapter.gguf
# adapter.gguf.train.json now records base/data/adapter sha256, seed, config, binary
# serve the adapter beside the frozen base (the exact form)
./runner -m base-Q4_K_M.gguf --lora adapter.gguf --serve
# fold it into a standalone file for any GGUF runtime: 8-bit or better
./runner -m base-Q4_K_M.gguf --lora adapter.gguf \
--merge-lora merged-Q8_0.gguf --quant q8_0
# teacher-forced logprobs for evals and rewards
./runner -m base-Q4_K_M.gguf --lora adapter.gguf -f eval.txt --score
on thing to note though, when Training 4-bits, the adapter seems to disappear when merging it into the weights, runner refuses such a merge just for that reason unless overridden.
[Held-out exact tool-call score before training, with the adapter served beside the base, and after merging the adapter into three output types. One base \(Qwen3-4B Q4\_K\_M\) and one task, measured 2026-08-22, so not a promise for other models. Source: docs\/adaptation-engine.md.](https://preview.redd.it/08a72sd7cdvh1.png?width=1480&format=png&auto=webp&s=f06fb33c84a5f9541df4c33ab69b738e5cd7c9a4)
You can find the repo with the binaries here:
[https://github.com/Joakimpalm-Zen/xyntetik-runner](https://github.com/Joakimpalm-Zen/xyntetik-runner)
Happy Training!
document fetched 2026-10-15 00:46 · canonical https://reddit.com/r/LocalLLaMA/comments/1x5j48s/train_lora_directly_on_quantized_ggufs_no_pytorch