[ ABORT TO HUD ]
SEQ. 1
SEQ. 2
LoRA & QLoRA Explained
🎯 Fine-Tuning & LoRA⏱ 15 min⭐ 150 BASE XP⌨ HANDS-ON LAB
Parameter-Efficient Fine-Tuning (PEFT)
Full fine-tuning updates all billions of parameters - requiring massive GPU clusters. PEFT freezes the base model and trains only a tiny fraction of parameters.
LoRA: Low-Rank Adaptation
LoRA injects small trainable matrices into frozen model layers. Instead of updating a 4096� - 4096 weight matrix, you train two small matrices (e.g., 4096� - 16 and 16� - 4096) - reducing trainable parameters by 99.9%.
QLoRA: Quantized LoRA
QLoRA goes further: quantize the frozen base to 4-bit (NF4), then apply LoRA on top. This cuts memory by ~75%:
| Method | 70B Model VRAM | Trainable Params |
|---|---|---|
| Full Fine-Tune | ~280GB (multi-GPU) | 70B (100%) |
| LoRA (FP16) | ~140GB | ~50M (0.07%) |
| QLoRA (4-bit) | ~36GB (1� - A100) | ~50M (0.07%) |
Tooling
| Tool | Strength | Best For |
|---|---|---|
| Unsloth | 2-5x faster via hand-written Triton kernels | Speed and efficiency |
| Axolotl | YAML-driven config, multi-GPU | Reproducible, complex pipelines |
| HF trl | Official HF library for SFT + RLHF | Integration with HF ecosystem |
Best Practices
- Apply LoRA to all linear layers (q, k, v, o, gate, up, down) - not just attention
- Data quality > quantity: 1000 high-quality examples often beats 100K noisy ones
- After training, merge adapters into base model for zero-latency inference
- Export merged model to GGUF for local deployment
💡 Quick Example (Unsloth):
from unsloth import FastLanguageModel
model, tokenizer = FastLanguageModel.from_pretrained(
model_name="unsloth/Qwen3-8B-bnb-4bit",
max_seq_length=8192, load_in_4bit=True
)
model = FastLanguageModel.get_peft_model(model,
r=16, target_modules=["q_proj","k_proj","v_proj","o_proj",
"gate_proj","up_proj","down_proj"],
lora_alpha=16, lora_dropout=0
)⌨ HANDS-ON LABKick Off a LoRA Fine-Tune with Axolotl
⭐ +200 XPTheory says LoRA trains <1% of weights - prove it. Install Axolotl, pull its curated example configs, and launch a real LoRA run from one YAML file.
1Install the Axolotl fine-tuning framework.
2Fetch the maintained example configs - every architecture, LoRA and QLoRA variants.
3Launch a LoRA training run from the Llama-3 example config.
OBJECTIVE 1 / 3 — type "hint" if stuck
KNOWLEDGE CHECK
QUERY 1 // 3
What percentage of parameters does QLoRA typically train?
100%
10%
1%
Less than 0.1%