When fine-tuning an LLM, how do you decide which layers/modules to train before actually running the experiment?
retained text
For example, how do you determine whether to fine-tune:only adapters/LoRA * specific transformer layers * input/projector layers * deeper/middle layers * or the full model? Are there reliable diagnostics, probing methods, gradient analysis, ablations, or small-scale tests that can tell you **where the bottleneck is** before doing a full training run and only finding out afterward that the chosen layers didn’t help? Curious how people make this decision in practice.
document fetched 2026-09-22 00:23 · canonical https://reddit.com/r/deeplearning/comments/1wm2ilj/when_finetuning_an_llm_how_do_you_decide_which