LoRA and QLoRA Fine-Tuning: 13 Steps, 90 Min [2026]

Training a large language model from scratch costs tens of millions of dollars in compute and takes a data center full of GPUs. Fine-tuning one with LoRA or QLoRA costs a lot less: a single consumer or workstation GPU, a few hours, and a dataset you already have. That gap is why Low-Rank Adaptation (LoRA) and its quantized cousin QLoRA have become the default way independent developers, startups, and research teams adapt open-weight models to a specific task.

This walkthrough builds a working fine-tuning pipeline from zero, using the tools that are actually current as of October 21, 2026: Hugging Face PEFT 0.21.2, a 4-bit quantized base model, and the TRL training stack, with a path to scale the same workflow up through Axolotl 0.21.0 or NVIDIA NeMo AutoModel when a project outgrows a single GPU. By the end you will have trained, merged, exported, and served your own adapter on top of an open-source model.

Google · Preferred Sources

Don't miss new tech stories on Google

Add TrendinTech once in the Google app and our stories appear in your news suggestions.

Add Now

Why Fine-Tune an Open-Source LLM Instead of Prompting

Prompt engineering and retrieval-augmented generation solve a lot of problems without touching model weights. But some tasks need the model itself to change: a consistent output format across thousands of calls, a narrow domain vocabulary a general model keeps getting wrong, or a tone and style that no system prompt reliably holds onto. Full fine-tuning used to be the only answer, and it meant updating every parameter in the model, which for anything past a few billion parameters requires multiple high-memory GPUs working together.

LoRA changed that math in 2021 by freezing the original weights and training a small pair of low-rank matrices that get added back into the model’s linear layers. QLoRA pushed it further in 2023 by storing the frozen base model in 4-bit precision instead of 16-bit, cutting memory use again without giving up much accuracy. Tim Dettmers, a researcher and co-author of the original QLoRA paper, summarized the result directly: “QLoRA replicates 16-bit performance across the board in all scenarios and scales tested,” he wrote in 2023 when the method shipped.

Three years later that claim still holds up in practice. The method hasn’t been replaced by something newer, it has been absorbed into nearly every open-source training stack: Hugging Face PEFT, TRL, Axolotl, Unsloth, and even newer non-Python frameworks like the Rust-based Burn library, which added native LoRA and QLoRA support in its v0.22.0 release on October 12, 2026.

What’s changed since 2023 isn’t the underlying math, it’s the scale of what people run it against. Research teams now routinely apply QLoRA to mixture-of-experts architectures with hundreds of billions of total parameters, something that would have sounded unrealistic on consumer-adjacent hardware a couple of years ago. The rest of this walkthrough starts small, on a model you can train in well under an hour, then shows the exact path to scale the same technique up once a project needs it.

How LoRA and QLoRA Actually Work

LoRA’s core idea is straightforward once you see it laid out. A linear layer in a transformer multiplies an input by a large weight matrix. Instead of updating that whole matrix during training, LoRA freezes it and adds two much smaller matrices, commonly labeled A and B, whose product approximates the update the model needs. Only A and B get trained. For a 4,096 by 4,096 weight matrix, a rank-16 LoRA adapter trains far fewer parameters than the 16.7 million needed to update that single layer in full, a substantial cut in trainable parameters.

What QLoRA Adds on Top

QLoRA keeps LoRA’s adapter design and changes how the frozen base model is stored. Instead of 16-bit floating point weights, the base model is quantized to 4-bit NF4 format, with double quantization applied to the quantization constants themselves and paged optimizers used to handle memory spikes during backpropagation. Dettmers described the combination plainly: “It combines a frozen 4-bit base model with adapters on top.” The adapters themselves still train in higher precision, so the small set of trainable parameters never loses the numerical accuracy that matters for learning.

The practical effect, according to Dettmers’ 2023 announcement, was that a 33 billion or 65 billion parameter model could be fine-tuned on a single 24GB or 48GB GPU respectively, in roughly 12 to 24 hours. That same ratio still shows up in current examples. An October 2026 technical write-up on fitting a 65-billion-parameter model onto one 48GB GPU, published on the technology newsletter temperature2.com, describes the identical quantize-then-adapt pattern as still being the production default for memory-constrained fine-tuning jobs.

One Pitfall Dettmers Flagged Early

Not every LoRA configuration works equally well, and this is worth internalizing before you write any code. “It is true that regular LoRA does not perform well,” Dettmers said in 2023, pointing to a specific fix: “If you attach LoRA to all linear layers, it turns out it works super well with no issues at all.” Many tutorials still default to attaching adapters only to the attention projection layers. Covering every linear layer, including the feed-forward blocks, is the detail that separates a mediocre fine-tune from one that actually holds the base model’s quality while learning the new task.

Prerequisites and Hardware Requirements

Before writing a line of training code, get the environment right. The versions below are current as of October 21, 2026. Where an exact version number for a library isn’t published with a date attached, “latest version” is used instead of guessing.

Python 3.11 or 3.12
PyTorch 2.10 with CUDA 12.8 support (this is the base container Unsloth documents for its reference QLoRA pipeline)
transformers 5.18 or newer (Hugging Face PEFT 0.21.2 specifically targets encoder-decoder compatibility with this line)
peft 0.21.2, released October 1, 2026
bitsandbytes, latest version, for 4-bit quantization
trl, latest version, for the SFTTrainer training loop
accelerate and datasets, latest versions
An NVIDIA GPU with CUDA support. A single 8GB to 12GB consumer card is enough for QLoRA on a 3B to 4B model. A 24GB card covers QLoRA on models in the 13B to 33B range. Multi-GPU or an A100/H100-class card is needed for anything larger with LoRA alone, without quantization

PEFT 0.21.2 is worth calling out specifically because it is the version you want installed today. It replaced PEFT 0.21.1, which introduced stricter handling for the transition to Transformers’ DTensor-based internals. If you’re on an older PEFT release and see errors mentioning DTensor or module path mismatches, that compatibility shift is almost certainly why.

Hardware Requirements by Model Size

Model size	Full fine-tune	LoRA (16-bit base)	QLoRA (4-bit base)	Reported example
3B to 4B	Multi-GPU, 40GB+	~16GB single GPU	8GB to 12GB single GPU	The open-source qlora-8gb-pipeline project documents end-to-end QLoRA training on consumer 8GB cards
13B	Multi-GPU, 80GB+	~24GB single GPU	~12GB single GPU	Standard PEFT and TRL defaults for mid-size open models
33B	Multiple A100/H100 GPUs	~48GB single GPU	24GB single GPU	Dettmers’ 2023 QLoRA release notes cite a 33B fine-tune on one 24GB GPU in about 12 hours
65B to 70B	8+ A100/H100 GPUs	Multiple 48GB+ GPUs	48GB single GPU	Dettmers’ 2023 notes and the October 2026 temperature2.com write-up both describe 65B QLoRA fitting on one 48GB GPU
400B+ mixture-of-experts	Large multi-node clusters	Multi-GPU node	One 8xH200 node	Axolotl v0.21.0 documents QLoRA tuning of Reflection’s 501B-parameter Beam model (23B active parameters) on a single 8xH200 node

Treat these as reported reference points, not guaranteed specs for every architecture. Sequence length, batch size, optimizer choice, and how many linear layers you target with adapters all shift the real number up or down. Mixture-of-experts models in particular behave differently from dense models because only a fraction of total parameters activate per token, which is why a 501B-parameter model can train on hardware that looks small relative to its headline size.

Step 1: Set Up Your Python Environment

Start with an isolated virtual environment so library versions for this project don’t collide with anything else on the machine.

python3 -m venv lora-finetune
source lora-finetune/bin/activate
python -m pip install --upgrade pip

If you’re on a machine with an NVIDIA GPU, confirm the driver and CUDA runtime are visible before installing anything else.

nvidia-smi

Check that the output lists your GPU and a CUDA version of 12.x. If this command fails, fix the driver installation before continuing. Nothing downstream will work without it.

Step 2: Install the Fine-Tuning Stack

Install PyTorch with CUDA 12.8 support first, then the Hugging Face stack on top of it.

pip install torch --index-url https://download.pytorch.org/whl/cu128
pip install "transformers>=5.18" peft==0.21.2 bitsandbytes trl accelerate datasets

Pin peft to 0.21.2 explicitly. That release fixed a bug that broke encoder-decoder model loading against newer Transformers builds, and pinning avoids silently picking up a regression if a different patch version lands later.

Step 3: Pick a Base Model

The open-weight model field moves fast enough that whatever model you’d default to from memory is probably not the newest option. According to leaderboard tracking site BenchLeader, open and proprietary models are being ranked and re-ranked on a near-daily basis in October 2026, so it’s worth checking a current leaderboard before locking in a base model for a long training run.

For this walkthrough, a smaller dense model in the 4B parameter range is the right choice for a single-GPU tutorial. It trains fast, fits comfortably in 8GB to 12GB of VRAM with QLoRA, and is small enough that you can iterate on your dataset and hyperparameters in minutes rather than hours. Several real QLoRA projects published recently used exactly this size class: a preprint posted October 19, 2026 described QLoRA fine-tuning of a 4B-parameter Qwen3 model for offline diabetes-related carbohydrate estimation, and the GoHumanize Open Humanizer project reportedly built its QLoRA adapter from the same Qwen3-4B base under an Apache-2.0 license.

If your project needs more capability than a 4B model can offer, NVIDIA’s Nemotron 3.5 Super VL is worth evaluating. It’s an open 120-billion-parameter mixture-of-experts model that activates only 12 billion parameters per token, and NVIDIA’s own documentation confirms it supports LoRA fine-tuning through the NeMo AutoModel toolkit. Organizations including Scale AI, Ramp, and AI21 have already post-trained it for specialized workflows, achieving improvements such as 40x lower inference cost, 5x lower latency, and 41% higher accuracy, with Ramp specifically reporting up to 50% higher throughput and 10x lower latency on its receipt-to-JSON extraction task.

One model family to leave alone for new fine-tuning projects: Meta’s Llama 4 line (Scout and Maverick, both released April 5, 2025). Meta’s current model development focus has shifted to the Muse family, with Muse Wonder and Muse Wonder Full announced October 12 and 13, 2026, and the open-weight Muse Spark Open following on October 15, 2026. Separately, Google Vertex AI has a scheduled retirement of its managed Llama 3.3 70B endpoint dated October 21, 2026. None of this means Llama 4 stops working for a local fine-tune, but it does mean the ecosystem’s attention, tooling updates, and managed-hosting support are moving elsewhere.

Check the License Before You Train

Model size and benchmark scores aren’t the only things that should factor into your choice. License terms vary a lot across open-weight releases, and they determine whether you can actually ship a fine-tuned derivative commercially. The GoHumanize Open Humanizer project, for example, published its Qwen3-4B-based QLoRA adapter under an Apache-2.0 license, which permits commercial use and redistribution with minimal restriction. Other open models attach usage policies with conditions tied to company size or deployment context. Before you spend an afternoon curating a training dataset, confirm the base model’s license actually allows the deployment you have in mind, since retraining against a differently licensed replacement later costs far less time than discovering a licensing conflict after launch.

Step 4: Load the Model in 4-bit Precision

With bitsandbytes installed, loading a model in 4-bit NF4 format with double quantization is a configuration object, not a custom loader.

from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
import torch

model_name = "Qwen/Qwen3-4B"

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
)

tokenizer = AutoTokenizer.from_pretrained(model_name)
if tokenizer.pad_token is None:
    tokenizer.pad_token = tokenizer.eos_token

model = AutoModelForCausalLM.from_pretrained(
    model_name,
    quantization_config=bnb_config,
    device_map="auto",
)

Setting the pad token here matters more than it looks. Several open models ship without a dedicated padding token, and if you skip this step, training will either throw an error or, worse, silently pad with a token that confuses the loss calculation.

Step 5: Build Your Training Dataset

LoRA and QLoRA are only as good as the data you train on. A few hundred well-formatted examples can meaningfully shift a model’s behavior on a narrow task. The format needs to match how the base model expects chat input, which for most current instruction-tuned models means a list of role-tagged messages.

from datasets import load_dataset

def format_example(example):
    messages = [
        {"role": "user", "content": example["instruction"]},
        {"role": "assistant", "content": example["response"]},
    ]
    example["text"] = tokenizer.apply_chat_template(
        messages, tokenize=False, add_generation_prompt=False
    )
    return example

dataset = load_dataset("json", data_files="training_data.jsonl")["train"]
dataset = dataset.map(format_example)

Your training_data.jsonl file should have one JSON object per line, each with an instruction and response field. Applying the tokenizer’s own chat template, instead of hand-building a prompt string, is what keeps your fine-tuned model’s formatting consistent with the base model’s existing behavior at inference time.

Step 6: Configure the LoRA Adapter

This is the step where Dettmers’ advice about target_modules matters most. Attach adapters to every linear layer type the model has, not just the attention projections.

from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training

model = prepare_model_for_kbit_training(model)

lora_config = LoraConfig(
    r=16,
    lora_alpha=32,
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"],
)

model = get_peft_model(model, lora_config)
model.print_trainable_parameters()

Running print_trainable_parameters() after this step should show well under 1% of the model’s total parameters as trainable. If it shows a much higher percentage, double-check that prepare_model_for_kbit_training ran before get_peft_model, and that the base model actually loaded in 4-bit.

Step 7: Set Up the Trainer

TRL’s SFTTrainer wraps the standard Hugging Face Trainer with sensible defaults for supervised fine-tuning on chat-formatted text.

from trl import SFTTrainer, SFTConfig

training_args = SFTConfig(
    output_dir="./lora-output",
    num_train_epochs=2,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,
    learning_rate=2e-4,
    lr_scheduler_type="cosine",
    max_seq_length=1024,
    logging_steps=10,
    save_strategy="epoch",
    bf16=True,
    gradient_checkpointing=True,
)

trainer = SFTTrainer(
    model=model,
    args=training_args,
    train_dataset=dataset,
)

These settings, a 2e-4 learning rate on a cosine schedule with an effective batch size of 16 (4 per device times 4 accumulation steps) and a 1,024-token maximum sequence length, mirror the configuration published alongside the GoHumanize Open Humanizer QLoRA adapter, which ran 2 epochs across 250 steps on a single NVIDIA A10G 24GB GPU on the Modal cloud platform and finished in 21.7 minutes, taking training loss from 3.17 before training down to 1.30, with an evaluation loss of 1.39.

Step 8: Launch Training and Read the Loss Curve

trainer.train()

Watch the logged loss values as they print. A healthy run shows loss dropping steadily for the first several dozen steps before flattening out. Below is roughly what that looks like in the console output for a small dataset:

{'loss': 3.1742, 'learning_rate': 0.0, 'epoch': 0.0}
{'loss': 2.4108, 'learning_rate': 0.00019, 'epoch': 0.08}
{'loss': 1.8835, 'learning_rate': 0.00017, 'epoch': 0.4}
{'loss': 1.5213, 'learning_rate': 0.00012, 'epoch': 0.8}
{'loss': 1.3891, 'learning_rate': 0.00006, 'epoch': 1.4}
{'loss': 1.3024, 'learning_rate': 0.00001, 'epoch': 2.0}

If loss stays flat from the very first step, the learning rate is probably too low or the adapter isn’t actually receiving gradients. If loss spikes to NaN partway through, it’s almost always the learning rate being too high for the batch size, or a malformed example in the dataset with empty or oversized text.

Step 9: Save and Merge Your Adapter

Once training finishes, you have two options: keep the adapter separate from the base model, or merge the two into a single set of weights. Keeping them separate is better for experimentation, since you can swap adapters in and out of the same base model without reloading gigabytes of weights. Merging is better for deployment, since the serving stack doesn’t need to know PEFT exists.

# Save the adapter only (a few megabytes)
trainer.model.save_pretrained("./lora-adapter")
tokenizer.save_pretrained("./lora-adapter")

# To merge: reload the base model in full precision first, then merge
from peft import PeftModel

base_model = AutoModelForCausalLM.from_pretrained(
    model_name, torch_dtype=torch.bfloat16, device_map="auto"
)
merged_model = PeftModel.from_pretrained(base_model, "./lora-adapter")
merged_model = merged_model.merge_and_unload()
merged_model.save_pretrained("./merged-model")

Notice that merging reloads the base model in bfloat16, not 4-bit. You cannot merge adapter weights directly into a quantized model, since the math requires full-precision weights on both sides. Trying to merge against the 4-bit model object you trained against is one of the most common errors people hit at this step.

Step 10: Evaluate the Fine-Tuned Model

A loss curve going down tells you training worked mechanically. It doesn’t tell you the model actually got better at your task. Hold back a portion of your dataset as an evaluation split that the model never trains on, and run it through both the base model and the fine-tuned model side by side.

from transformers import pipeline

pipe = pipeline("text-generation", model=merged_model, tokenizer=tokenizer)

test_prompt = tokenizer.apply_chat_template(
    [{"role": "user", "content": "Explain the difference between LoRA and QLoRA."}],
    tokenize=False, add_generation_prompt=True,
)

output = pipe(test_prompt, max_new_tokens=200, do_sample=False)
print(output[0]["generated_text"])

For tasks with a measurable outcome, like the GoHumanize project’s goal of reducing how often outputs got flagged as AI-generated by a detector, track a concrete before-and-after number rather than relying on reading outputs by eye. GoHumanize’s own release notes reported its QLoRA adapter cutting that detection rate to 17% of outputs, down from 20% for an earlier build and 38% for a full-parameter fine-tune of the same base model, a useful illustration that a small adapter can beat a full fine-tune on the metric that actually matters.

Step 11: Deploy for Local Inference

For local or edge deployment, converting your merged model to GGUF format lets you run it through Ollama or any llama.cpp-based server without keeping a Python training environment around.

python convert_hf_to_gguf.py ./merged-model --outfile model-q4_k_m.gguf --outtype q4_k_m

ollama create my-finetuned-model -f Modelfile
ollama run my-finetuned-model

If you’d rather serve the merged model directly for higher-throughput production use, vLLM or a similar inference server can load the merged Hugging Face checkpoint without a GGUF conversion step at all, which is the better path when you’re serving requests from a web application instead of running the model on a single developer’s machine.

Step 12: Scale Up With Axolotl or NeMo AutoModel

The raw PEFT and TRL code above is the right starting point, but it stops being the right tool once a job needs multiple GPUs, a mixture-of-experts architecture, or a training run long enough that you want a config file instead of a Python script you have to remember to edit. Axolotl’s v0.21.0 release, published October 16, 2026, is built for exactly that handoff, and it ships support for QLoRA tuning of mixture-of-experts models at a scale well past what raw PEFT scripts handle cleanly: its release notes describe fine-tuning Reflection’s Beam model, a 501-billion-parameter mixture-of-experts architecture with 23 billion active parameters, using QLoRA on a single 8xH200 GPU node.

An Axolotl config for a QLoRA job looks like this:

base_model: Qwen/Qwen3-4B
load_in_4bit: true
adapter: qlora
lora_r: 16
lora_alpha: 32
lora_target_modules:
  - q_proj
  - k_proj
  - v_proj
  - o_proj
  - gate_proj
  - up_proj
  - down_proj
datasets:
  - path: training_data.jsonl
    type: chat_template
num_epochs: 2
learning_rate: 0.0002
gradient_checkpointing: true

accelerate launch -m axolotl.cli.train config.yml

If your project is built on NVIDIA’s open Nemotron line instead, NeMo AutoModel is the equivalent path. NVIDIA’s own technical documentation, published October 15, 2026, confirms that developers can use NeMo AutoModel for supervised fine-tuning and LoRA adaptation of Nemotron 3.5 Super VL. NVIDIA has also said the model was optimized to run locally on its DGX Spark hardware, and separately, Unsloth announced on October 16, 2026 that NVIDIA RTX Spark systems could locally run and train models above 120 billion parameters using QLoRA-style workflows, which puts fine-tuning jobs that used to require a data center within reach of a single well-equipped desktop.

LoRA vs QLoRA vs Full Fine-Tuning

Factor	Full fine-tuning	LoRA	QLoRA
Trainable parameters	100% of the model	Typically under 1%	Typically under 1%
Base model precision	16-bit or 32-bit	16-bit, frozen	4-bit NF4, frozen
Hardware needed	Multiple high-memory GPUs for anything past a few billion parameters	One GPU for small-to-mid models	One GPU for mid-to-large models
Output artifact	A full new copy of the model	A small adapter, typically tens of megabytes	A small adapter, typically tens of megabytes
Reported accuracy vs full fine-tune	Baseline	Close, when adapters cover all linear layers	Dettmers’ 2023 release reported matching 16-bit performance across tested scenarios
Best fit	Pretraining or deep architectural changes	Task adaptation when VRAM is somewhat limited	Task adaptation on consumer or single-GPU hardware

Common Pitfalls When Fine-Tuning With LoRA and QLoRA

Most failed fine-tuning runs trace back to a small set of repeated mistakes. Here are the ones that come up most often.

Restricting target_modules to attention layers only. As Dettmers pointed out, regular LoRA scoped to only a few projection layers underperforms. Covering the feed-forward layers too (gate_proj, up_proj, down_proj on most architectures) closes most of the quality gap.
Setting the learning rate too high for the effective batch size. A 2e-4 learning rate is a reasonable starting point for small to mid-size models, but pushing past that without lowering batch size risks loss spikes and permanently damaged weights partway through a run.
Mismatching the tokenizer’s chat template with how the model was originally instruction-tuned. Hand-building prompt strings instead of calling apply_chat_template is a frequent source of a fine-tuned model that technically trains fine but produces garbled or off-format output at inference time.
Trying to merge adapter weights directly against a 4-bit quantized model object. Merging requires full-precision weights on both sides. Reload the base model in bfloat16 or float16 before calling merge_and_unload.
Skipping gradient checkpointing on memory-constrained hardware. It trades some training speed for a meaningful VRAM reduction, and on an 8GB to 12GB GPU, leaving it off is often the difference between a run that completes and one that crashes with an out-of-memory error.
Training for too many epochs on a small dataset. A few hundred examples run through more than 3 or 4 epochs commonly overfits, producing a model that repeats training examples verbatim instead of generalizing the pattern behind them.

Troubleshooting Guide

These are the errors and symptoms that show up most often across LoRA and QLoRA fine-tuning projects, along with what usually fixes them.

Symptom	Likely cause	Fix
CUDA out of memory	Batch size, sequence length, or model size exceeds available VRAM	Enable gradient checkpointing, lower per_device_train_batch_size, raise gradient_accumulation_steps to compensate, or shorten max_seq_length
Loss is NaN after a few steps	Learning rate too high, or a malformed example with empty or extremely long text	Lower the learning rate, add a max length filter to your dataset preprocessing, and check for empty strings
Loss never decreases from the first step	The adapter isn’t receiving gradients, often because prepare_model_for_kbit_training ran after get_peft_model instead of before	Re-check the order of operations in your setup code and confirm print_trainable_parameters shows a nonzero trainable count
Model output is identical to the untrained base model	The adapter wasn’t actually loaded at inference, or you loaded the base model path instead of the adapter or merged path	Confirm the correct directory is passed to from_pretrained and that PeftModel.from_pretrained ran if you’re using a separate adapter
ValueError about padding token being None	The base model’s tokenizer has no default pad token set	Set tokenizer.pad_token = tokenizer.eos_token before training, as shown in Step 4
bitsandbytes fails to import or reports no GPU support	CUDA toolkit version mismatch, or an older GPU architecture without 4-bit kernel support	Reinstall bitsandbytes against the exact CUDA version your PyTorch build uses, and confirm your GPU architecture is supported
Merge step throws a dtype or quantization error	Attempting to call merge_and_unload on a model still loaded in 4-bit	Reload the base model in bfloat16 or float16 first, then load the adapter on top before merging
Training is far slower than expected on a supported GPU	Gradient checkpointing enabled unnecessarily on hardware with plenty of headroom, or mixed precision not enabled	Disable gradient checkpointing if VRAM allows, and confirm bf16 or fp16 is set in your training arguments
Dataset loading fails with a JSON decode error	A malformed line in a JSONL file, often a trailing comma or an unescaped quote	Validate the file line by line with a JSON parser before passing it to load_dataset

Advanced Tips for Production Fine-Tuning

Once the basic pipeline works, a few refinements separate a one-off experiment from something you can run repeatedly with confidence.

Track every run’s hyperparameters and resulting evaluation metric somewhere outside your training script, even something as simple as a spreadsheet, before you’ve run more than two or three experiments. It’s easy to lose track of which learning rate or rank value produced which checkpoint once you’re iterating quickly, and reconstructing a winning configuration from memory a week later rarely works.

Version your adapters the same way you’d version application code. A LoRA adapter is small enough, typically tens of megabytes, that storing every meaningful checkpoint costs almost nothing, and having last week’s adapter on hand makes it trivial to roll back if a new training run underperforms the one it was meant to replace.

Consider rank and alpha as a pair, not independently. A common starting ratio is alpha equal to twice the rank, which is what the configuration in Step 6 uses (r=16, lora_alpha=32). Raising the rank increases the adapter’s capacity to learn complex patterns, but also increases the risk of overfitting on small datasets and adds a small amount of inference latency.

For mixture-of-experts base models, check whether your training framework routes gradients correctly through the expert-selection mechanism before assuming a QLoRA run on an MoE architecture will behave like one on a dense model. Axolotl’s support for QLoRA-tuning Reflection’s 501B-parameter Beam model is a sign the tooling has matured here, but MoE fine-tuning is still a newer, faster-moving corner of the ecosystem than dense-model LoRA, which has had years to stabilize.

If your workload needs to run outside the Python ecosystem entirely, for example embedded in a Rust service, the Burn framework’s v0.22.0 release added native LoRA and QLoRA support alongside ONNX export, which is worth evaluating if your deployment target doesn’t want a Python runtime in the loop at all.

Finally, keep an eye on how fast the open-model ecosystem is moving before committing to a long-running project around a specific base model. Finance outlet Yahoo Finance reported in October 2026 that DeepSeek pushed out an updated model release described by the company as faster and leaner than its predecessor, a reminder that the base model you pick today may have a meaningfully improved successor within weeks. Checking a current release tracker like LLM Stats before locking in a base model for a multi-week project is a cheap way to avoid training against a model that’s about to be superseded.

The Complete Working Project

Putting every step together into a single project gives you something you can actually run end to end. The layout below mirrors the structure used by small, reproducible QLoRA pipelines published in the open-source community, including projects built specifically for 8GB consumer GPUs.

lora-finetune-project/
├── requirements.txt
├── training_data.jsonl
├── train.py
├── merge.py
└── evaluate.py

# train.py
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training
from trl import SFTTrainer, SFTConfig
from datasets import load_dataset

MODEL_NAME = "Qwen/Qwen3-4B"

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
)

tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME)
if tokenizer.pad_token is None:
    tokenizer.pad_token = tokenizer.eos_token

model = AutoModelForCausalLM.from_pretrained(
    MODEL_NAME, quantization_config=bnb_config, device_map="auto"
)
model = prepare_model_for_kbit_training(model)

lora_config = LoraConfig(
    r=16, lora_alpha=32, lora_dropout=0.05, bias="none", task_type="CAUSAL_LM",
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"],
)
model = get_peft_model(model, lora_config)
model.print_trainable_parameters()

def format_example(example):
    messages = [
        {"role": "user", "content": example["instruction"]},
        {"role": "assistant", "content": example["response"]},
    ]
    example["text"] = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=False)
    return example

dataset = load_dataset("json", data_files="training_data.jsonl")["train"].map(format_example)

training_args = SFTConfig(
    output_dir="./lora-output", num_train_epochs=2, per_device_train_batch_size=4,
    gradient_accumulation_steps=4, learning_rate=2e-4, lr_scheduler_type="cosine",
    max_seq_length=1024, logging_steps=10, save_strategy="epoch", bf16=True,
    gradient_checkpointing=True,
)

trainer = SFTTrainer(model=model, args=training_args, train_dataset=dataset)
trainer.train()

trainer.model.save_pretrained("./lora-adapter")
tokenizer.save_pretrained("./lora-adapter")

Run it with python train.py, then run merge.py (the merge code from Step 9) followed by evaluate.py (the evaluation code from Step 10) to get from a raw dataset to a tested, deployable adapter in one session. Total time for a 4B model on a single mid-range GPU, based on the training-time pattern reported for the GoHumanize QLoRA project, runs about 21.7 minutes on a single NVIDIA A10G (24 GB) GPU on Modal, once the dataset is ready, not counting the time spent preparing training_data.jsonl itself.

Keeping Up With a Fast-Moving Toolchain

The code in this walkthrough will keep working, but the specific library versions behind it won’t stay current for long. PEFT has shipped two patch releases in the past month alone, 0.21.1 followed by 0.21.2 on October 1, 2026, Axolotl shipped v0.21.0 on October 16 with new mixture-of-experts QLoRA support, and NVIDIA’s NeMo Platform completed a rename to NeMo Helix across its v0.6.0 and v0.7.0 releases in September and October 2026. None of these changes break the fundamental LoRA or QLoRA approach, they extend it to bigger models and more hardware targets. Re-reading your training stack’s changelog before starting a new project is a five-minute habit that avoids a lot of version-mismatch debugging later.

Frequently Asked Questions

Do I need a GPU to fine-tune with LoRA or QLoRA?
Yes, for any model beyond a few hundred million parameters. CPU training is possible in principle but slow enough to be impractical for anything in the billions-of-parameters range that this walkthrough covers.

What’s the real difference between LoRA and QLoRA?
LoRA freezes the base model at its original precision (usually 16-bit) and trains small adapter matrices on top. QLoRA does the same thing but first quantizes the frozen base model to 4-bit, cutting memory use further at a small, generally acceptable cost in flexibility during training.

How much data do I need to fine-tune a model?
There’s no fixed number, but projects in this space commonly use datasets ranging from a few hundred to a few thousand examples for narrow task adaptation. Quality and consistency of formatting matter more than raw volume.

Can I run QLoRA fine-tuning on a laptop GPU?
For small models in the 3B to 4B range, yes, provided the GPU has at least 8GB of VRAM. Larger models need more VRAM or a cloud GPU instance.

Should I merge my adapter or keep it separate?
Keep it separate while you’re still experimenting, since you can swap adapters on and off the same base model quickly. Merge it once you’re ready to deploy, since most inference servers run faster and simpler against a single merged checkpoint.

Is QLoRA always better than regular LoRA?
Not always. If you have enough VRAM to run LoRA at full 16-bit precision, it trains slightly faster and avoids any quantization overhead. QLoRA’s advantage is entirely about fitting bigger models onto smaller hardware, not raw training speed.

Can I fine-tune a mixture-of-experts model with QLoRA?
Yes. Axolotl’s v0.21.0 release specifically documents QLoRA fine-tuning of a 501-billion-parameter mixture-of-experts model on a single 8xH200 node, and NVIDIA’s NeMo AutoModel supports LoRA fine-tuning of its own open mixture-of-experts model, Nemotron 3.5 Super VL.

What happens if I pick a base model that gets replaced by a newer version mid-project?
Your adapter will still work against the original base model. It generally won’t transfer directly to a different model’s weights, so if you want the newer base model’s improvements, you’ll need to retrain the adapter against it, which is usually fast given how little compute LoRA and QLoRA require compared to full fine-tuning.