# Train LoRA directly on Quantized GGUFs, no PyTorch, no Parent(FP16) needed, just your GGUF and the data — retained record

> Retained reddit record from reddit:LocalLLaMA.

Canonical: https://highsignal.app/data/records/2cacee9f513b2c90

[← sources](https://highsignal.app/data)


source: [reddit](https://highsignal.app/data/reddit) (reddit:LocalLLaMA)  published 2026-10-14 05:34 ingested 2026-10-15 02:42

[original source ↗](https://www.reddit.com/r/LocalLLaMA/comments/1x5j48s/train_lora_directly_on_quantized_ggufs_no_pytorch/)

## retained text



```
I've seen this one floating around quite a bit now and since this is something that can be done without the parent and or other dependencies i thought i'd share something.

  
Usually when you fine-tune a model you can't:

\- train against the quantized GGUF file itself

\- do it without PyTorch, transformers or a separate Python setup

\- use the exact same GGUF for inference afterwards, without converting the model

(QLoRA does train through a 4-bit base, but in PyTorch and on its own NF4 format, not on the GGUF you serve.)



[With Runner, the CPU training forward pass is the CPU serving forward pass, so the adapter is trained on the exact weights it will run on. CUDA\/Metal serving uses the GPU's own arithmetic, within a measured tolerance but not bit-identical. There is no FP16 copy to keep, and no framework to install.](https://preview.redd.it/roepm5d3cdvh1.png?width=1280&format=png&auto=webp&s=ced357d5d53429c91103a9957d873c349a42a206)

  
With runner the 16-bit nor the PyTorch package is needed, this is done through "freezing" the weights of the current X-bit GGUF, only a "small" LoRA adapter is trained on top.

Quick-Guide:

    # train through the quantized base you will serve
    ./runner -m base-Q4_K_M.gguf --train data.jsonl --train-steps 200 \
        --lora-rank 8 --train-out adapter.gguf
    # adapter.gguf.train.json now records base/data/adapter sha256, seed, config, binary
    
    # serve the adapter beside the frozen base (the exact form)
    ./runner -m base-Q4_K_M.gguf --lora adapter.gguf --serve
    
    # fold it into a standalone file for any GGUF runtime: 8-bit or better
    ./runner -m base-Q4_K_M.gguf --lora adapter.gguf \
        --merge-lora merged-Q8_0.gguf --quant q8_0
    
    # teacher-forced logprobs for evals and rewards
    ./runner -m base-Q4_K_M.gguf --lora adapter.gguf -f eval.txt --score

on thing to note though, when Training 4-bits, the adapter seems to disappear when merging it into the weights, runner refuses such a merge just for that reason unless overridden.

[Held-out exact tool-call score before training, with the adapter served beside the base, and after merging the adapter into three output types. One base \(Qwen3-4B Q4\_K\_M\) and one task, measured 2026-08-22, so not a promise for other models. Source: docs\/adaptation-engine.md.](https://preview.redd.it/08a72sd7cdvh1.png?width=1480&format=png&auto=webp&s=f06fb33c84a5f9541df4c33ab69b738e5cd7c9a4)

  
You can find the repo with the binaries here:

[https://github.com/Joakimpalm-Zen/xyntetik-runner](https://github.com/Joakimpalm-Zen/xyntetik-runner)

  
Happy Training!






```



document fetched 2026-10-15 00:46 · canonical https://reddit.com/r/LocalLLaMA/comments/1x5j48s/train_lora_directly_on_quantized_ggufs_no_pytorch
