← sources

When fine-tuning an LLM, how do you decide which layers/modules to train before actually running the experiment?

source: reddit (reddit:deeplearning)published 2026-09-21 04:20ingested 2026-09-22 02:36

retained text

For example, how do you determine whether to fine-tune:only adapters/LoRA

* specific transformer layers
* input/projector layers
* deeper/middle layers
* or the full model?

Are there reliable diagnostics, probing methods, gradient analysis, ablations, or small-scale tests that can tell you **where the bottleneck is** before doing a full training run and only finding out afterward that the chosen layers didn’t help?

Curious how people make this decision in practice.

document fetched 2026-09-22 00:23 · canonical https://reddit.com/r/deeplearning/comments/1wm2ilj/when_finetuning_an_llm_how_do_you_decide_which