Multi-Tenant GPU Interference & L2 Cache Contention

The Physics of Hardware Contention in Modern GPUs

When running high-density LLM serving, hosting multiple concurrent inference requests, LoRA adapters, or multi-modal encoders on a single GPU leads to hardware interference. While modern GPUs like the NVIDIA H100 SXM5 boast 3.35 TB/s of high-bandwidth memory (HBM3), their on-chip 50MB L2 cache is a shared physical resource.

The Anatomy of L2 Cache Thrashing

When two uncoordinated requests execute concurrently:

1. Request A streams weights for a large feed-forward network (FFN) projection, flooding the L2 cache lines.
2. Request B attempts to read active KV cache blocks for an attention step, encountering an immediate L2 cache miss.
3. Request B stalls on high-latency DRAM round-trips (hundreds of clock cycles), resulting in massive tail-latency (P99) jitter that destroys interactive voice and code completion SLAs.

Mitigation Strategies in Production Harnesses

• NVIDIA Multi-Process Service (MPS) Partitioning: Set explicit thread percentage and memory limits per client to prevent noisy neighbors from starving compute pipelines.
• Priority-Aware CUDA Streams: Assign high-priority CUDA streams (cudaStreamCreateWithPriority) to interactive decode tokens while prefill workloads run on background queues.
• L2 Cache Persistence Window: Reserve dedicated L2 cache segments for KV cache index tables using CUDA 12's cudaStreamSetAttribute with cudaStreamAttrAccessPolicyWindow.