Self-Attention & Multi-Head Attention — Transformer Architecture
[ ABORT TO HUD ]
SEQ. 1
SEQ. 2
Self-Attention & Multi-Head Attention
🧠 Transformer Architecture⏱ 10 min⭐ 75 BASE XP⌨ HANDS-ON LAB
The Engine Behind Every LLM
Every modern language model is built on the Transformer architecture (Vaswani et al., 2017). At its core is the Self-Attention mechanism.
How Attention Works
For every token in a sequence, the model computes three vectors:
- Query (Q): "What information am I looking for?"
- Key (K): "What information do I contain?"
- Value (V): "What information do I provide?"
The attention score is: Attention(Q,K,V) = softmax(QK^T / √d_k) × V
This lets each token "attend" to every other token, capturing long-range dependencies.
Multi-Head Attention (MHA)
Instead of one attention computation, MHA runs multiple heads in parallel - each learning different relationship types (syntax, semantics, coreference). A typical model uses 32-128 heads.
Modern Variants
| Variant | What It Does | Used By |
|---|---|---|
| MHA | Full Q/K/V per head | Original Transformer |
| GQA | Groups share K/V heads (reduces memory) | Llama 3/4, Mistral |
| MLA | Compresses KV cache via latent projection | DeepSeek-V3/V3.2/R1/V4 |
| iRoPE / Hybrid Linear | Interleaved RoPE & Gated DeltaNet / KDA for 1M–10M context | Llama 4 Scout, Qwen 3.5/3.8, Kimi K3 |
💡 Pro Tip: GQA is the current industry default - it provides 90%+ of MHA quality with significantly less memory usage.
⌨ HANDS-ON LABCompute Multi-Head Attention Shapes in PyTorch
⭐ +200 XPVerify transformer tensor dimensions by executing scaled dot-product attention across batched queries, keys, and values.
1Run scaled dot product attention with batch size 2, 8 heads, sequence length 64, and head dimension 64.
OBJECTIVE 1 / 1 — type "hint" if stuck
KNOWLEDGE CHECK
QUERY 1 // 2
What are the three vectors computed in self-attention?
Input, Output, Hidden
Query, Key, Value
Encoder, Decoder, Attention
Weight, Bias, Gradient