PEFT, LoRA, and QLoRA: a theoretical comparison
My first experiment compared prompting techniques on a frozen model. This one asks a different question: once prompting isn’t enough and you actually need to fine-tune a model, what does that cost, and how do the cheaper alternatives to full fine-tuning actually differ from each other?
Try the live dashboard or read the full write-up and code.
What is PEFT, and why
A large language model is really just a huge collection of numbers, its parameters, that got tuned during training until the model could produce good answers. Fine-tuning means continuing that same training process on new data, so the model picks up something specific: a task, a style, a domain. The original way to do this, full fine-tuning, keeps every single one of those numbers free to change. That gets expensive fast, because during training the computer has to remember, for every number that’s allowed to move, which direction it should move in and a bit of its recent history, not just the number itself.
Parameter-efficient fine-tuning, PEFT, is the family of techniques built to avoid that cost. Instead of letting every number move, freeze almost all of them, and train only a small slice, sometimes a handful of the model’s existing numbers, sometimes a small number of new ones added just for this purpose. The bet underneath all of them is that adapting a model to a new task doesn’t require moving every number, just moving a small, well-chosen handful of them.
This experiment doesn’t run any training. It computes, directly from llama3.1:8b’s published design (the same model from my first experiment), exactly how many numbers each technique would need to train and how much memory that would take, for the scenario of fine-tuning it to summarize SAMSum dialogues. No GPU needed, and the numbers are exact rather than estimated.
flowchart LR
A["llama3.1:8b architecture constants"] --> B["6 techniques, each computes its own trainable/frozen split"]
B --> C["Trainable parameter count + memory footprint, per technique"]
Six techniques, four families
| Technique | Family | Idea |
|---|---|---|
| Full fine-tuning | Baseline | Every weight is trainable |
| LoRA | Reparameterization | Freeze the weights, learn a low-rank update alongside them |
| QLoRA | Reparameterization | Same as LoRA, but the frozen weights are stored in 4-bit |
| Adapters | Bottleneck module | Insert small trainable modules in series inside each layer |
| Prefix-tuning | Soft prompt | Learn virtual key/value vectors prepended at every layer’s attention |
| BitFit | Selective | Train only the model’s existing bias terms |
One more thing worth being precise about: none of these techniques “figure out” which numbers matter by studying the model. For LoRA, QLoRA, Adapters, and Prefix-tuning, where the new trainable numbers go is a decision someone made ahead of time, informed by earlier research, not something the technique works out for each model it’s used on. BitFit is the exception: it actually looks at the model’s existing numbers and picks a subset, using the simplest possible rule, train whatever counts as a “bias” number (more on that below). Either way, whatever ends up trainable is then learned the completely ordinary way: by trying it on real examples and adjusting, same as always.
What I found
Full fine-tuning’s 120GB cost is almost entirely bookkeeping, not the numbers themselves. Just holding llama3.1:8b’s numbers in memory takes about 15GB. Full fine-tuning needs about 120GB to train it, roughly 8x that, because for every one of its 8.03 billion numbers, training has to also remember which direction that number should move and a bit of its recent history, on top of the number itself (Kingma & Ba, 2014). Every PEFT technique here is, underneath the differences, an attempt to avoid paying that bookkeeping cost across the whole model.
QLoRA’s win is compression, not training fewer numbers. This surprised me a little going in. LoRA and QLoRA train the exact same 3.4 million numbers in this experiment. QLoRA’s memory footprint (3.79GB) is about a quarter of LoRA’s (15.00GB) for a completely different reason: it also shrinks the frozen part of the model down to a smaller storage format, the way a compressed photo takes up far less space than the original while still looking almost the same. Dettmers et al., 2023 built QLoRA specifically so those two questions, how much to compress the frozen model, and how many numbers to actually train, could be answered separately, and this experiment’s numbers show that separation directly.
BitFit trains zero numbers on this model, because Llama leaves out the one kind of number it looks for. BitFit’s whole idea is to unfreeze a model’s existing “bias” numbers, small nudges added at the end of a calculation, separate from the main multiplication, and train only those. Llama-family models don’t have any: their design leaves biases out entirely, unlike the older BERT-style models BitFit (Zaken et al., 2021) was originally tested against. This is a known property of the LLaMA architecture (Touvron et al., 2023), not a new finding, but it’s a clean, concrete example of why a technique’s assumptions need checking against the specific model in front of you, not just its reputation.
Adapters train about 10x more numbers than LoRA, for a cost LoRA was built specifically to avoid. Adapters came first and solve a similar problem to LoRA’s, adapting a large model without a full copy per task, but they insert a small extra step directly into the model’s path, more like a permanent stop on an assembly line than a side path you can remove later. LoRA’s side path folds back into the model once training is done and disappears completely; an Adapter’s extra step stays, adding a small delay every time the model is used afterward. Hu et al., 2021 name this directly as part of why they built LoRA instead of just using Adapters (Houlsby et al., 2019).
The full write-up has the exact memory-footprint formula, a rank-vs-parameters sweep for LoRA, per-technique diagrams, and the caveats worth knowing (these numbers are computed, not measured, and this is one reference model, so a different architecture would shift some specifics, like BitFit’s zero).
Read the details, or poke around the dashboard yourself.