LoRA & QLoRA Explained

Parameter-Efficient Fine-Tuning (PEFT)

Full fine-tuning updates all billions of parameters — requiring massive GPU clusters. PEFT freezes the base model and trains only a tiny fraction of parameters.

LoRA: Low-Rank Adaptation

LoRA injects small trainable matrices into frozen model layers. Instead of updating a 4096×4096 weight matrix, you train two small matrices (e.g., 4096×16 and 16×4096) — reducing trainable parameters by 99.9%.

QLoRA: Quantized LoRA

QLoRA goes further: quantize the frozen base to 4-bit (NF4), then apply LoRA on top. This cuts memory by ~75%:

Tooling

Best Practices

• Apply LoRA to all linear layers (q, k, v, o, gate, up, down) — not just attention
• Data quality > quantity: 1000 high-quality examples often beats 100K noisy ones
• After training, merge adapters into base model for zero-latency inference
• Export merged model to GGUF for local deployment

from unsloth import FastLanguageModel
model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="unsloth/Qwen3-8B-bnb-4bit",
    max_seq_length=8192, load_in_4bit=True
)
model = FastLanguageModel.get_peft_model(model,
    r=16, target_modules=["q_proj","k_proj","v_proj","o_proj",
                          "gate_proj","up_proj","down_proj"],
    lora_alpha=16, lora_dropout=0
)