vLLM Architecture & Optimization
The Production Inference Standard
vLLM is the industry-standard engine for high-throughput, multi-user GPU serving. It's what you use when Ollama isn't enough.
Core Optimizations
| Feature | Problem Solved | Impact |
|---|---|---|
| PagedAttention | KV cache wastes 60-80% VRAM with pre-allocation | On-demand block allocation, 2-4x more concurrent users |
| Continuous Batching | Static batching idles GPU when requests finish | >90% GPU utilization, no idle gaps |
| Prefix Caching | Shared system prompts recomputed per request | Skip redundant computation for shared prefixes |
| FP8 Inference | FP16 wastes compute on Hopper/Blackwell GPUs | ~2x throughput on H100/B200 hardware |
Inference Optimization Deep Dive
PagedAttention applies OS-style virtual memory to the KV cache. Instead of pre-allocating contiguous memory for max sequence length, it allocates small blocks (16 tokens) on demand - like how your OS manages RAM with paging.
Prefill-Decode Disaggregation (advanced): Split compute-heavy prefill and memory-bound decoding across different hardware clusters for optimal resource usage.
Model Runner V2 (MRV2)
Introduced in vLLM v0.17+, MRV2 delivers up to 56% throughput improvement via GPU-native Triton kernels and async scheduling:
VLLM_USE_V2_MODEL_RUNNER=1 vllm serve mistralai/Mistral-Large-3
Transformers as a Native-Speed Backend
Since mid-2026, the --model-impl transformers backend in vLLM has closed the performance gap with vLLM's hand-written model implementations entirely. Benchmarked across a 4B dense model, a 32B dense model (tensor-parallel), and a 235B-parameter FP8 MoE (data + expert parallel across 8x H100), the Transformers backend now matches or beats native vLLM throughput on every case:
# Any compatible Hugging Face model, one flag - composes with the usual parallelism options vllm serve Qwen/Qwen3-4B --model-impl transformers vllm serve Qwen/Qwen3-32B --model-impl transformers --tensor-parallel-size 2 vllm serve Qwen/Qwen3-235B-A22B-FP8 --model-impl transformers --data-parallel-size 8 --enable-expert-parallel
Under the hood, vLLM uses torch.fx to statically analyze the model graph and rewrites matching patterns (for example into MergedColumnParallelLinear/QKVParallelLinear for tensor-parallel plans, or fused Expert-Parallel kernels for MoE) - while staying fully compatible with torch.compile and CUDA Graphs. The practical effect: a correct Transformers model implementation now gets optimized vLLM inference automatically, with no custom port required (linear-attention architectures aren't supported yet).
services:
vllm:
image: vllm/vllm-openai:latest
runtime: nvidia
ports: ["8000:8000"]
volumes: ["./models:/models"]
environment:
- NVIDIA_VISIBLE_DEVICES=all
- VLLM_USE_V2_MODEL_RUNNER=1
command: >
--model /models/Mistral-Large-3-AWQ
--quantization awq
--tensor-parallel-size 2
--max-model-len 32768
--gpu-memory-utilization 0.9
nginx:
image: nginx:alpine
ports: ["443:443"]
volumes: ["./nginx.conf:/etc/nginx/nginx.conf"]
⚠️ Security: Always deploy behind a reverse proxy (Nginx/Traefik) for rate limiting and auth - vLLM's built-in --api-key is insufficient for production.Production time. Install vLLM, expose a model behind an OpenAI-compatible endpoint, then verify it with a live completion request.