← [ ABORT TO HUD ]
SEQ. 1
Static vs Continuous (Iteration-Level) Batching
Eliminating Head-of-Line Blocking
In traditional Static Batching, requests arriving together are bundled into a batch. The entire batch must run until the longest sequence finishes generating. Shorter requests that complete in 10 tokens sit completely idle in GPU memory while a 2,000-token request finishes.
Continuous Batching (pioneered by Orca and vLLM) operates at the iteration level. At every step of the decode loop:
- Requests that completed in the previous step are evicted immediately, freeing their VRAM pages.
- New pending requests from the incoming queue are spliced into the active batch without waiting for a new batch boundary.
- GPU utilization jumps from ~20% to over 85% sustained compute saturation.
⌨ HANDS-ON LABRun Iteration-Level Scheduler
⭐ +200 XPSimulate continuous batch insertion and dynamic prefill chunking to prevent decode latency spikes.
1Launch the iteration-level continuous batching scheduler.
2Inject mixed workload (1 long prefill + 5 active decode streams).
OBJECTIVE 1 / 2 — type "hint" if stuck
SYNAPSE VERIFICATION
QUERY 1 // 1
What was the main flaw of static batching in generative AI?
It could only run on AMD GPUs
It did not support temperature sampling
Shorter requests were trapped waiting for the longest request in the batch to finish, wasting memory and throughput
It required Python 2.7