← [ ABORT TO HUD ]
SEQ. 1

Draft-Target Speculative Decoding

🚀 Speculative Decoding & Multi-Model Harnesses⏱ 25 min⭐ 225 BASE XP⌨ HANDS-ON LAB

Breaking the Autoregressive Speed Barrier

Autoregressive decoding generates one token per forward pass. Because modern GPUs have massive parallel compute, running a single token through a 70B parameter model severely underutilizes the GPU's execution units.

The Two-Phase Speculative Mechanism

  1. Speculative Drafting: A small, ultra-fast Draft Model (e.g. Llama 3.2 1B or Qwen2.5-Coder 0.5B), a multi-head prediction head (Medusa / Hydra), or a feature-level recurrent draft engine (EAGLE-3) generates K candidate tokens in a sequence or tree structure.
  2. Target Verification: The large Target Model (e.g. Llama 3.3 70B or Qwen2.5-Coder 32B) evaluates all M = K + 1 candidate positions in a single parallel forward pass using causal or tree-attention masks.
  3. Exact Rejection Sampling Criterion: Given draft probability q(x) and target probability p(x), each proposed token x is accepted with probability α(x) = min(1, p(x) / q(x)). At the first rejection, a replacement token is sampled from the normalized residual distribution p'(x) = norm(max(0, p(x) - q(x))), guaranteeing a 2.5x to 3.8x wall-clock speedup with a mathematically identical output distribution!
Empirical Hardware Law (HARNESS Engine on 8GB RTX 5060): Dense vs. Sparse MoE Batching
  • Dense 32B Wins with Speculation: Pairing Qwen2.5-Coder-32B (Q4_K_M) with a 100% GPU-resident 0.5B drafter and OnlineDraftCalibrator reaches 3.90 tok/s pooled (4.80 tok/s peak on Rust at 96.2% draft acceptance, a 3.36x speedup over 1.16 tok/s baseline), and 15.89 tok/s pooled (19.17 tok/s peak) on 14B + 0.5B.
  • The Sparse MoE Expert-Union Law: In 128-expert Top-8 Qwen3-Coder-30B-A3B (30.5B total, 3.3B active), verifying M = 7 draft tokens simultaneously activates the union of 7 tokens' Top-8 choices (~35–45 experts per CPU layer, 4–5x DDR4 traffic = 11.59 tok/s). Pure zero-draft Hybrid MoE Partitioning (-ngl 99 --n-cpu-moe 32, streaming only 8/128 = 6.25% experts = ~0.70 GB/token) reaches 16.37 tok/s pooled (16.53 tok/s peak, 14.11x speedup)!

Next-Gen Speculative Architectures: EAGLE-3 & Medusa

ArchitectureMechanismDraft OverheadAcceptance Rate
EAGLE-3Feature-level recurrence; drafts at the second-to-last hidden layer~10% VRAM of standard draft modelUp to 82% token acceptance
Medusa HeadsMulti-head linear decoders atop target model; zero auxiliary draft model0 auxiliary model parameters (heads only)2.2x – 2.8x speedup
Independent DraftSeparate small model (e.g., Llama 1B drafting for 70B, or 0.5B for 32B)Requires dedicated GPU memory for draft KV cache65% – 96% acceptance (domain-aligned)
⌨ HANDS-ON LABConfigure Speculative Decoding Engine
⭐ +250 XP

Pair a 1B draft model with a 70B target model and measure candidate acceptance rate.

1Initialize dual-model speculative decoding harness.
2Run benchmark request and compute acceptance rate.
lab-sandbox — simulated environment
INFINITY LAB SANDBOX v2.6 — simulated shell
Type the command for the current objective. Helpers: "hint", "solution", "clear".
$
OBJECTIVE 1 / 2 — type "hint" if stuck
SYNAPSE VERIFICATION
QUERY 1 // 1
Why does speculative decoding produce the exact same mathematical probability distribution as standard sampling?
Because rejection sampling adjusts candidate logits so the joint probability strictly equals the target model's distribution
Because it uses quantum computing
Because the draft model is an exact clone of the target model
It does not; speculative decoding is always lossy