How to Fine-Tune an LLM with LoRA and QLoRA on a Single GPU

Key takeaways

• You can fine-tune an LLM with LoRA on a single consumer GPU because LoRA freezes the original weights and trains only small low-rank adapter matrices. The LoRA paper reports up to 10,000 times fewer trainable parameters than full fine-tuning of GPT-3 175B.
• QLoRA goes further. It loads the frozen base model in 4-bit NF4, which is how its authors fine-tuned a 65-billion-parameter model on one 48 GB GPU.
• With Hugging Face TRL (v1.16 at the time of writing) and PEFT (v0.21), a full QLoRA run takes about 40 lines of Python: a dataset, a BitsAndBytesConfig, a LoraConfig, an SFTConfig and SFTTrainer.
• LoRA usually needs a learning rate about 10 times higher than full fine-tuning. TRL's docs suggest 2e-4 for supervised fine-tuning, and it works best when applied to all linear layers, not just attention.
• Fine-tuning teaches a model a behaviour, format or style. It is not a good way to give it fresh facts. For changing knowledge, retrieval-augmented generation (RAG) is usually the better tool.

How to fine-tune an LLM with LoRA: what you are actually training

You can fine-tune an LLM with LoRA on a single GPU because LoRA freezes the weights and trains only low-rank adapter matrices. QLoRA loads the frozen base model in 4-bit NF4; its authors used it to fine-tune a 65-billion-parameter model on one 48 GB GPU.

The paper's headline results set the tone for everything that followed. Compared with full fine-tuning of GPT-3 175B with Adam, LoRA cut trainable parameters by up to 10,000 times and GPU memory by about three times. Because the learned update can be merged back into the original weights, it adds no extra latency at inference time.

QLoRA, published in May 2023 by Tim Dettmers, Artidoro Pagnoni, Ari Holtzman and Luke Zettlemoyer, combines LoRA with a 4-bit quantized base model. It introduced three tricks:

• 4-bit NormalFloat (NF4): a data type designed for normally distributed weights.
• Double quantization: it also compresses the quantization constants.
• Paged optimizers: they absorb memory spikes during training.

With these, the authors fine-tuned a 65B model on a single 48 GB GPU in 24 hours. Their Guanaco model reached 99.3% of ChatGPT's score on the Vicuna benchmark.

This guide uses the two Hugging Face libraries built around these ideas. PEFT supplies the adapters. TRL supplies SFTTrainer, the supervised fine-tuning loop. If you have not run an open-weight model locally before, start with our guide on how to run a local LLM with Ollama. This tutorial ends by feeding your fine-tuned model back into that setup.

Fine-tuning vs RAG vs prompting: when is LoRA worth it?

Before you collect a single training example, check that fine-tuning is the right tool. A practical rule of thumb:

• Try prompting first if a clear system prompt with a few examples gets you most of the way. It costs nothing and changes in seconds. Hosted models with long context windows, such as those covered in our Gemini 4 Argon API tutorial, can take dozens of examples in a single prompt.
• Use RAG when the model needs facts that change or that it never saw: product manuals, policies, recent documents. Retrieval keeps answers tied to sources you can cite and update without retraining. Our guide to building a RAG app in Python covers that route.
• Fine-tune with LoRA when you need a consistent behaviour that prompting delivers unreliably. Examples include a fixed output format or JSON schema, a house writing style, a narrow classification or extraction task, tool-calling patterns, or squeezing a small model's quality up so you can stop paying for a large one.

These approaches combine well. A common pattern is a small fine-tuned model that has learned your format and tone, answering from documents supplied by a RAG pipeline.

How well does LoRA compare with full fine-tuning? In "LoRA Without Regret" (September 2025), John Schulman and colleagues at Thinking Machines Lab found that LoRA performs the same as full fine-tuning on small-to-medium instruction-tuning and reasoning datasets. It falls behind only when the dataset outgrows the adapter's capacity. That covers most single-GPU projects.

How much VRAM do you need for LoRA and QLoRA?

Memory depends on model size, precision, sequence length, batch size and whether gradient checkpointing is on. As a starting point, the weights alone take about 2 bytes per parameter in bfloat16 and roughly 0.5 bytes per parameter in 4-bit. On top of that come activations, the adapter's gradients and optimizer states, and framework overhead. The table combines that arithmetic with figures published by the QLoRA authors and Hugging Face.

Sources: Hu et al. (LoRA), Dettmers et al. (QLoRA), Hugging Face bitsandbytes 4-bit blog post. The weight columns are simple parameter-count arithmetic and do not include training overhead.

In practice, a 16 GB card is a comfortable home for QLoRA on 7B–8B models, and a 24 GB card gives you room for longer sequences or a 13B–14B model. Plain LoRA on a bf16 base model needs roughly four times more memory for the weights, so it suits smaller models or larger cards. One hardware note: bitsandbytes, the library behind 4-bit loading, is built mainly around NVIDIA CUDA. For background on why that ecosystem dominates, see our explainer on TorchTPU, TorchNeuron and Nvidia CUDA.

Step 1: Set up the environment

Create a fresh virtual environment with a recent Python and a CUDA-enabled PyTorch build, then install the libraries:

python -m venv .venv
source .venv/bin/activate
pip install -U torch
pip install -U "trl[peft]" bitsandbytes datasets accelerate

Check that PyTorch sees your GPU before going further:

python -c "import torch; print(torch.cuda.is_available(), torch.cuda.get_device_name(0))"

This guide uses Meta's meta-llama/Llama-3.1-8B-Instruct. It is a gated model, so accept the licence on its Hugging Face page and log in with hf auth login first. Any causal language model on the Hub works the same way. For a two-minute test run, swap in Qwen/Qwen3-0.6B.

Step 2: Prepare the dataset

Data quality matters more than any hyperparameter. A few hundred to a few thousand clean, consistent examples of exactly the behaviour you want will usually beat a large, noisy scrape. Every example should look like the output you want in production: the same format, length and tone.

TRL's SFTTrainer accepts plain text, conversational "messages", and prompt-completion formats. The conversational prompt-completion format is the most useful for chat models. TRL applies the model's chat template for you, and by default it computes the loss only on the completion. The model learns to write the answers and not to parrot the questions. Save one JSON object per line in data/train.jsonl:

{"prompt": [{"role": "system", "content": "You turn support emails into JSON tickets."}, {"role": "user", "content": "My invoice for March is missing and the app logs me out every hour."}], "completion": [{"role": "assistant", "content": "{\"issues\": [\"missing_invoice\", \"session_timeout\"], \"priority\": \"medium\"}"}]}
{"prompt": [{"role": "system", "content": "You turn support emails into JSON tickets."}, {"role": "user", "content": "I was charged twice for my subscription."}], "completion": [{"role": "assistant", "content": "{\"issues\": [\"duplicate_charge\"], \"priority\": \"high\"}"}]}

Strip personal data from real logs before training. Remove duplicates, and keep 5–10% of the examples aside for evaluation.

Step 3: Configure QLoRA and train with SFTTrainer

The script below follows the patterns in TRL's PEFT integration guide. You pass quantization_config and peft_config straight to SFTTrainer, which loads the model in 4-bit and wraps it with LoRA adapters. Save it as train.py:

import torch
from datasets import load_dataset
from peft import LoraConfig
from transformers import BitsAndBytesConfig
from trl import SFTConfig, SFTTrainer

MODEL_ID = "meta-llama/Llama-3.1-8B-Instruct"

dataset = load_dataset("json", data_files="data/train.jsonl", split="train")
splits = dataset.train_test_split(test_size=0.1, seed=42)

# 4-bit NF4 base model (QLoRA)
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
)

# LoRA adapters on every linear layer
peft_config = LoraConfig(
    r=16,
    lora_alpha=32,
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
    target_modules="all-linear",
)

args = SFTConfig(
    output_dir="llama31-8b-tickets-qlora",
    model_init_kwargs={"dtype": torch.bfloat16},
    max_length=1024,
    per_device_train_batch_size=2,
    gradient_accumulation_steps=8,
    num_train_epochs=2,
    learning_rate=2e-4,
    lr_scheduler_type="cosine",
    warmup_steps=10,
    logging_steps=10,
    eval_strategy="steps",
    eval_steps=50,
    save_steps=50,
    save_total_limit=2,
)

trainer = SFTTrainer(
    model=MODEL_ID,
    args=args,
    train_dataset=splits["train"],
    eval_dataset=splits["test"],
    quantization_config=bnb_config,
    peft_config=peft_config,
)
trainer.model.print_trainable_parameters()
trainer.train()
trainer.save_model("llama31-8b-tickets-adapter")

Run it with python train.py. Here is what the important settings do and why they are set this way:

• target_modules="all-linear" puts adapters on every linear layer, as QLoRA does. The PEFT docs note this can match full fine-tuning, and "LoRA Without Regret" found that attention-only LoRA underperforms LoRA on the MLP layers.
• r and lora_alpha set the adapter's capacity and scaling. Ranks of 8–64 are typical. TRL's docs describe alpha of about twice the rank as a common choice. Raise the rank if a large dataset seems to plateau early.
• learning_rate=2e-4 is 10 times TRL's full fine-tuning default of 2e-5, matching the roughly 10x ratio both TRL and Thinking Machines recommend for LoRA.
• Batch size 2 with 8 accumulation steps gives an effective batch of 16 while keeping memory low. Thinking Machines found LoRA is less tolerant of very large batches than full fine-tuning, so moderate effective batches are a safe default.
• model_init_kwargs={"dtype": torch.bfloat16} matters because, according to TRL's reference, a model loaded from a name string defaults to float32 unless you set a dtype.
• Gradient checkpointing is already on by default in SFTConfig. It trades some speed for a large saving in activation memory.

On older cards without bfloat16 support, such as the T4 or RTX 20-series, set bnb_4bit_compute_dtype=torch.float16, the float16 dtype in model_init_kwargs, and fp16=True in SFTConfig. If you have plenty of VRAM and want plain LoRA instead of QLoRA, simply drop the quantization_config argument.

The saved adapter is small, usually tens to a few hundred megabytes depending on rank. The multi-gigabyte base model is not copied.

Step 4: Evaluate the fine-tuned model

Watch two numbers during training. Training loss should fall steadily. Evaluation loss on your held-out split should fall and then flatten. If eval loss starts rising while training loss keeps dropping, the model is memorising. Stop earlier, use fewer epochs or add more varied data. TRL also logs mean_token_accuracy, which is useful as a rough trend line.

Loss alone will not tell you whether the model is useful, so test it the way it will be used. Load the adapter on top of the base model and generate answers for held-out prompts:

import torch
from peft import AutoPeftModelForCausalLM
from transformers import AutoTokenizer

ADAPTER = "llama31-8b-tickets-adapter"
model = AutoPeftModelForCausalLM.from_pretrained(
    ADAPTER, dtype=torch.bfloat16, device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3.1-8B-Instruct")

messages = [
    {"role": "system", "content": "You turn support emails into JSON tickets."},
    {"role": "user", "content": "The export button does nothing and I need my data today."},
]
inputs = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, return_tensors="pt", return_dict=True
).to(model.device)
output = model.generate(**inputs, max_new_tokens=128, do_sample=False)
print(tokenizer.decode(output[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Run the same held-out prompts through the untouched base model and compare the two side by side. For structured tasks, score them automatically: count how often the output parses as valid JSON, matches the expected labels or passes your schema. Also spot-check a few general questions. Heavy fine-tuning on a narrow task can erode a model's broader abilities, and you want to catch that before deployment.

Step 5: Merge the adapter and run it locally

You can serve the base model plus adapter as they are. That is handy if you keep several adapters for one base model. For simpler deployment, merge the adapter into the weights. With QLoRA, do the merge on a base model reloaded in 16-bit, not on the 4-bit training copy:

import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

BASE = "meta-llama/Llama-3.1-8B-Instruct"
base = AutoModelForCausalLM.from_pretrained(BASE, dtype=torch.bfloat16, device_map="cpu")
model = PeftModel.from_pretrained(base, "llama31-8b-tickets-adapter")
model = model.merge_and_unload()  # returns a new model; not in-place

model.save_pretrained("llama31-8b-tickets-merged")
AutoTokenizer.from_pretrained(BASE).save_pretrained("llama31-8b-tickets-merged")

The PEFT docs stress that merge_and_unload() is not an in-place operation, so always use the model it returns. Merging on the CPU avoids needing VRAM, but an 8B model in bfloat16 needs roughly 16 GB of system RAM.

To run the merged model in Ollama, convert it to GGUF with llama.cpp's convert_hf_to_gguf.py script and quantize it if you like. Then point a Modelfile at the result with FROM ./model.gguf and run ollama create. From there it works like any other local model, including as the brain behind an MCP server or a RAG app.

Common problems and fixes

• CUDA out of memory: lower max_length or per_device_train_batch_size first, and raise gradient_accumulation_steps to keep the effective batch size. Move to a smaller model if that is not enough.
• The model never stops generating: the end-of-turn token does not match the chat template. TRL's docs advise setting eos_token in SFTConfig for some base models.
• Loss drops but outputs get worse: this usually means overfitting or inconsistent training examples. Audit the data before touching hyperparameters.
• Training is slow: enable packing=True to pack short examples into full-length sequences. TRL also integrates with Unsloth, which claims up to 2 times faster training with up to 70% less VRAM.

What it means

Parameter-efficient fine-tuning has turned model customisation from a data-centre job into a weekend project. With LoRA's frozen-weights design, QLoRA's 4-bit base model and TRL's trainer, a 16 GB consumer GPU can adapt an 8B open-weight model to a specific task. The result is a small adapter file you can version, share or swap. The hard part is no longer hardware. It is deciding whether you need fine-tuning at all, and then building a clean dataset and an honest evaluation set. Use prompting for quick wins, RAG for knowledge that changes, and LoRA for behaviour you need the model to produce reliably. Treat licences as part of the job too: models such as Llama ship under their own terms, and those terms apply to your fine-tuned derivative.

Sources

• Hu et al., LoRA: Low-Rank Adaptation of Large Language Models (arXiv)
• Dettmers et al., QLoRA: Efficient Finetuning of Quantized LLMs (arXiv)
• Hugging Face PEFT documentation
• Hugging Face PEFT: LoRA developer guide
• Hugging Face TRL: SFT Trainer
• Hugging Face TRL: PEFT integration (LoRA and QLoRA)
• Hugging Face blog: Making LLMs even more accessible with bitsandbytes, 4-bit quantization and QLoRA
• Thinking Machines Lab: LoRA Without Regret
• Meta Llama 3.1 8B Instruct model card (Hugging Face)