The Lifecycle of an Inference Request

Step-by-Step Anatomy of a Token Generation Loop

Every token generated in a modern LLM follows a strictly choreographed lifecycle across CPU host memory and GPU High-Bandwidth Memory (HBM).

1. Tokenization & Validation: Client UTF-8 string is converted into token IDs using Byte-Pair Encoding (BPE) or SentencePiece. Length is checked against context window limits.
2. Prefill Phase (Prompt Evaluation): All prompt tokens are processed simultaneously in parallel. Compute-bound GEMM operations compute initial Key and Value matrices across all attention heads.
3. KV Cache Allocation: Non-contiguous virtual memory pages are provisioned in GPU VRAM to store attention keys and values.
4. Decode Phase (Autoregressive Generation): Memory-bound memory fetches. Tokens are generated one by one. The model reads the accumulated KV cache and computes the next single token logit distribution.
5. Sampling & Grammar Constraint: Temperature, top-p, min-p, and grammar bitmasks filter illegal token IDs before argmax/sampling selects the final token.
6. De-tokenization & SSE Streaming: The chosen token ID is decoded back into UTF-8 text and pushed through Server-Sent Events (SSE) to the client.

// Minimalist Rust representation of an inference harness step
pub struct TokenGenerationStep {
    pub step_id: usize,
    pub input_token: u32,
    pub logits: Vec,
    pub sampled_token: u32,
    pub is_eos: bool,
}