Self-Attention & Multi-Head Attention

The Engine Behind Every LLM

Every modern language model is built on the Transformer architecture (Vaswani et al., 2017). At its core is the Self-Attention mechanism.

How Attention Works

For every token in a sequence, the model computes three vectors:

• Query (Q): "What information am I looking for?"
• Key (K): "What information do I contain?"
• Value (V): "What information do I provide?"

The attention score is: Attention(Q,K,V) = softmax(QK^T / √d_k) × V

This lets each token "attend" to every other token, capturing long-range dependencies.

Multi-Head Attention (MHA)

Instead of one attention computation, MHA runs multiple heads in parallel — each learning different relationship types (syntax, semantics, coreference). A typical model uses 32-128 heads.

Modern Variants