← [ ABORT TO HUD ]
SEQ. 1
SEQ. 2
PagedAttention & Radix Tree Prefix Caching
Virtual Memory Paging for Attention Tensors
Traditional serving allocates a contiguous chunk of GPU memory for the maximum possible sequence length. Because sequence lengths vary wildly, up to 60% - 80% of VRAM was wasted due to internal and external memory fragmentation.
PagedAttention (introduced by Kwon et al. in vLLM) borrows the classic operating system concept of virtual memory with page tables:
- Physical KV blocks are allocated on-demand in fixed block sizes (typically 16 or 32 tokens).
- A logical block table maps the sequential tokens of a request to disparate physical memory pages scattered across VRAM.
- Physical blocks can be shared across multiple requests (Copy-on-Write), enabling instantaneous Prefix Caching for long system instructions and few-shot templates.
⌨ HANDS-ON LABSimulate Radix Prefix Cache Hits
⭐ +225 XPConfigure a shared system prompt in the Radix prefix tree and observe instantaneous prefill reuse.
1Register a 2,000-token system prompt in the Radix cache manager.
2Dispatch two incoming requests sharing this prefix and inspect cache hit rates.
OBJECTIVE 1 / 2 — type "hint" if stuck
SYNAPSE VERIFICATION
QUERY 1 // 1
How does Radix Tree Prefix Caching reduce Time To First Token (TTFT)?
It re-downloads the model weights from HuggingFace
It skips prefill computation for shared prompt prefixes by referencing pre-existing KV blocks already in GPU memory
It generates tokens before the user asks a question
It converts the prompt into an image