Llama 3.3 & Llama 3.1 Architecture — The Meta Llama Family
[ ABORT TO HUD ]
SEQ. 1
SEQ. 2
Llama 3.3 & Llama 3.1 Architecture
🦙 The Meta Llama Family⏱ 12 min⭐ 100 BASE XP⌨ HANDS-ON LAB
The Meta Llama 3.3 & 3.1 Family
Meta's Llama 4 (Scout & Maverick Mixture-of-Experts), Llama 3.3 (70B), and Llama 3.1 (405B, 70B, 8B) anchor global enterprise open-weight deployments. While Llama 3.1–3.3 perfected dense Transformers with Grouped-Query Attention (GQA) at 128K context, Llama 4 transitioned to sparse Mixture-of-Experts (MoE) and native early-fusion multimodality under the Llama 4 Community License.
Llama 4 MoE Herd (2025–2026)
| Model | Active / Total Params | Experts | Context Window | Key Highlights |
|---|---|---|---|---|
| Llama 4 Scout (17B-16E) | 17B active / ~109B total | 16 experts | 10M tokens (iRoPE) | Extreme full-repo & multi-doc context; fits on a single H100 80GB GPU in INT4/FP4 |
| Llama 4 Maverick (17B-128E) | 17B active / ~400B total | 128 routed + shared | 1M tokens | High-capacity multimodal reasoning, coding, and chat workhorse |
| Llama 4 Behemoth (288B-16E) | 288B active / ~2T total | 16 experts | Internal | Teacher model used to codistill Scout & Maverick (unreleased as public weights) |
For dense deployments requiring predictable per-token latency without MoE expert-routing overhead, Meta's Llama 3.3, Llama 3.2 Vision, and Llama 3.1 dense GQA models remain standard enterprise workhorses:
Model Comparison
| Model | Parameters | Context Window | Architecture | Key Highlights |
|---|---|---|---|---|
| Llama 3.3 70B | 70 Billion | 128K tokens | Dense Transformer (GQA) | Matches 405B-class benchmark performance in an efficient 70B footprint |
| Llama 3.1 405B | 405 Billion | 128K tokens | Dense Transformer (GQA) | Frontier reasoning, teacher model for distillation and synthetic data |
| Llama 3.1 70B | 70 Billion | 128K tokens | Dense Transformer (GQA) | Strong enterprise workhorse for code, reasoning, and summarization |
| Llama 3.1 8B | 8 Billion | 128K tokens | Dense Transformer (GQA) | Ultra-fast, runs on consumer hardware, ideal for edge and agents |
| Llama 3.2 11B / 90B | 11B / 90B | 128K tokens | Multimodal Vision | Native image reasoning, visual document understanding, OCR |
Architectural Foundations
- Grouped-Query Attention (GQA): Reduces KV-cache memory bandwidth by sharing key-value heads across multiple query heads.
- 128K Context Window: Powered by RoPE (Rotary Position Embeddings) scaled with a base frequency theta of 500,000.
- 128K Token Vocabulary: Byte-level BPE tokenizer yielding high encoding efficiency across coding languages and multilingual text.
Hardware Requirements
| Model | Quantization | Min VRAM | Recommended Setup |
|---|---|---|---|
| Llama 3.1 8B | Q4_K_M / FP16 | ~6GB / 16GB | Single consumer GPU (RTX 3060 / 4060 or Apple Silicon) |
| Llama 3.3 70B | Q4_K_M / FP8 | ~42GB / ~75GB | Dual RTX 3090/4090 or single A100/H100 80GB |
| Llama 3.1 405B | FP8 / FP16 | ~420GB / ~810GB | Multi-GPU node (8x A100 80GB or 8x H100) |
⌨ HANDS-ON LABInspect Llama RoPE Scaling and GQA Parameters
⭐ +200 XPInspect official Hugging Face config.json to confirm rotary position embeddings base theta and group query attention ratio.
1Fetch the config.json for Llama-3.1-8B-Instruct and extract key attention parameters.
OBJECTIVE 1 / 1 — type "hint" if stuck
KNOWLEDGE CHECK
QUERY 1 // 2
What is the primary breakthrough of Llama 3.3 70B?
It uses a 1000B parameter router
It delivers 405B-class benchmark performance within a compact 70B dense parameter footprint
It is closed source only
It only runs on CPU