llm-inference-serving

LLM Inference Serving Operational Guide

Concise operational pointers for deep LLM-serving troubleshooting and tuning.

Assumes you already know what an LLM, KV cache, and a token are. This skill covers the operational layer — the parts models tend to gloss over: V1-engine internals, KV-cache math (MHA/GQA/MLA), continuous-batching/chunked-prefill economics, prefill/decode disaggregation, speculative decoding tradeoffs, quantization-format choice, and MoE+parallelism layout — current as of late 2025/early 2026.

When to use

Load when the question is about:

• Tuning vLLM/SGLang/TGI/TRT-LLM serving knobs in production
• Sizing GPUs / KV-cache budget for a model + concurrency target
• Picking a quantization format for the deployment target (server vs edge)
• Diagnosing throughput-vs-latency regressions (TTFT vs ITL trade)
• Choosing between TP, PP, EP, DP for an MoE or dense model
• Deciding whether disaggregated prefill/decode is worth it
• Selecting a speculative-decoding strategy and predicting acceptance rate
• Long-context tuning (RoPE scaling, sliding window, chunked prefill)
• KV-cache memory math when adding GQA/MLA models
• Prefix-caching impact on RAG / multi-turn / system-prompt-heavy workloads
• Benchmark interpretation: goodput vs raw throughput, P99 latency
• Scaling MoE serving (Mixtral, DeepSeek-V3/R1) across nodes
• Deciding when to keep llama.cpp / GGUF vs server-grade engines
• Migrating between vLLM V0 and V1 engine

Do NOT load for: training / fine-tuning, embedding or reranker serving, agent / tool-calling protocol design, hosted-API integration, plain "what's an LLM" questions. For training fundamentals see dl-training; for fine-tuning see fine-tuning-llms.

vLLM and the V1 engine

V1 is the default engine; V0 is deprecated. V1 unifies prefill and decode under one scheduler with a {request_id: num_tokens} budget. EngineCore runs in its own process; scheduler and worker 0 are decoupled (symmetric TP); Persistent Batch caches input tensors and applies diffs.

Key knobs and how they interact:

• --gpu-memory-utilization (default 0.9) — fraction of GPU memory for the executor. KV cache pool ≈ this fraction × VRAM − weights − activations − buffers.
• --max-num-seqs (default 256 in V1) — max concurrent sequences (decode parallelism cap).
• --max-num-batched-tokens (default ≈8192 in V1; was 2048 in V0) — per-iteration token budget shared by prefill + decode chunks. Lower → better ITL; higher → better TTFT/throughput. Sweep 8k–64k.
• --enable-chunked-prefill (default True in V1) — splits long prompts into chunks so a giant prefill doesn't stall queued decodes.
• --enable-prefix-caching (default True in V1) — hash-keyed shared KV blocks across requests. Massive win for RAG / system-prompt / multi-turn; near-zero loss otherwise.
• --kv-cache-dtype fp8 (E4M3 by default on Hopper+) — halves KV bytes vs FP16 with negligible quality loss for most models.
• --quantization {fp8,awq,gptq,awq_marlin,gptq_marlin,nvfp4,...} — weight scheme. Marlin/Machete kernels accelerate GPTQ/AWQ vs naïve dequant.
• --tensor-parallel-size, --pipeline-parallel-size, --data-parallel-size, --enable-expert-parallel.
• --speculative-config '{...}' (V1 unified) replaces the older --speculative-model.

KV-token capacity ≈ max-num-seqs × max-model-len. Setting max-model-len=128k with max-num-seqs=256 blows the KV budget on a single H100 — drop one. gpu-memory-utilization ≥ 0.95 leaves no headroom for activations under bursty prefill; OOM appears under load, not at boot.

KV-cache math

Per-token MHA bytes: 2 × n_layers × n_kv_heads × head_dim × bytes_per_elem.

Worked: Llama 3 70B (80 layers, 8 KV heads via GQA, head_dim 128, BF16) = 2 × 80 × 8 × 128 × 2 = 327,680 B ≈ 0.31 MB/token. At 100K tokens that's ~31 GB just for one sequence.

GQA collapses n_kv_heads from n_query_heads → n_groups (Llama 3: 64→8 = 8× KV reduction).

MLA (DeepSeek V2/V3/R1) replaces per-head K/V with a per-token shared low-rank latent (d_c ≈ 512) plus a small RoPE key (d_R ≈ 64). Bytes ≈ batch × seq × n_layers × (d_c + d_R) × bytes. DeepSeek-V3 at FP16 ≈ ~70 KB/token vs ~860 KB on a hypothetical MHA equivalent (~60× reduction, ~12× vs comparable GQA). Why DeepSeek can serve 128K cheaply and why vLLM treats MLA via a separate hybrid_kv_cache_manager.

KV quant: FP8 E4M3 cuts bytes 2×. NVFP4 KV (Blackwell) halves again with micro-block scaling (block 16 + FP8 scale + per-tensor FP32 scale).

Batching evolution

• Static: all in-batch requests run to completion together. Padding waste, head-of-line blocking.
• Dynamic (Triton-style): wait window then batch. Predictable latency, lower throughput.
• Continuous (iteration-level) (Orca 2022; vLLM/SGLang/TGI/TRT-LLM): scheduler reconsiders the batch every forward step. Finished sequences leave; new ones enter freed KV slots immediately. Large reported speedups vs static / HF Transformers baselines.
• Chunked prefill: split a 32K-prompt prefill into per-step chunks sharing max-num-batched-tokens with decodes. Eliminates head-of-line ITL spikes; smooths P99. Default-on in V1.

Prefill vs decode and disaggregated serving

• Prefill: compute-bound; full S×S attention per head per layer; saturates Tensor Cores.
• Decode: memory-bandwidth-bound; one token per step requires a full pass over weights and the entire growing KV. Tensor cores idle, HBM saturated.

Co-locating both wastes the resource the other phase needs. PD disaggregation runs prefill and decode on separate GPU pools and transfers KV across; goal is to satisfy TTFT and TPOT independently and optimize goodput (requests/s meeting SLO), not raw throughput. Specific systems (Splitwise, DistServe, Mooncake, etc.) churn fast — pick what your engine integrates with at the time.

KV transfer is hundreds of MB/request — the new bottleneck. RDMA preferred over NCCL for cross-pod transfers. When worth it: medium-to-long prompts, strict TTFT SLO, concurrency high enough that PD interference is real. Skip for short-prompt high-frequency chat — KV-transfer overhead dominates.

Speculative decoding

Verifier runs N draft tokens per step, accepts the longest matching prefix. Speedup ≈ accepted_length × target_step_cost / (target_step_cost + draft_cost).

• Draft model: small same-family LM (e.g., 1B drafting 70B). Quality drives acceptance.
• N-gram / prompt lookup: matches recent generated/prompt tokens; zero training. Acceptance 0.75–0.85 on code/structured/repetitive output, 0.5–0.65 on creative.
• Medusa: extra prediction heads on the target; trains heads, no separate draft.
• EAGLE / EAGLE-2 / EAGLE-3: predicts at the feature level (residual stream); reuses target features. EAGLE-3 was the current SOTA at last update — re-check before committing.
• MTP (multi-token prediction, native in DeepSeek-V3) treated as in-model spec decode.
• PARD / MLP / suffix decoding: lighter alternatives.

vLLM CLI: --speculative-config '{"method":"ngram","num_speculative_tokens":4,"prompt_lookup_min":2,"prompt_lookup_max":5}' or '{"method":"eagle3","model":"…","num_speculative_tokens":2}'.

Footgun: under high concurrency the batch fills naturally — speculation eats compute that would have served other requests, net throughput drops. Use spec decode for low-concurrency latency-sensitive workloads, not at saturation.

Quantization formats

Server-grade:

• FP8 W8A8 (E4M3 weights+activations): vLLM --quantization fp8 on Hopper (cc ≥ 8.9) and Ada. E4M3 (range ±448) for weights/activations; E5M2 (range ±57344) historically for grads. Per-tensor scale standard; per-channel for sensitive layers.
• NVFP4 (Blackwell-class GPUs): block-16 microscaling with FP8 (E4M3) block scales + FP32 per-tensor. Better accuracy than MXFP4 (block 32) at comparable throughput.
• AWQ (4-bit) + Marlin: weight-only INT4, activation-aware. Often highest throughput on GPU; quality close to FP16 for ≥7B.
• GPTQ + Marlin: similar 4-bit with Hessian calibration. GPTQModel emits Marlin/Machete-ready checkpoints.
• INT8 W8A8 (SmoothQuant): less common since FP8 landed on Hopper; useful pre-Hopper.
• BNB load-in-4bit / 8bit: Transformers-side, slow; prototyping only.

Edge / CPU / Mac:

• GGUF (llama.cpp). Q4_K_M is the sweet spot (mixed 4-bit with selective higher precision); Q5_K_M for headroom; Q3_K_S/Q2_K when you must squeeze. CPU, Apple Metal, Vulkan, CUDA, ROCm.

Per-channel scales > per-tensor on sensitivity-prone layers (e.g., down_proj); per-tensor cheaper, often fine elsewhere.

Frameworks compared

• vLLM (Apache 2.0): broadest model coverage, V1 engine, strong NVIDIA + growing AMD/Inferentia/TPU. Default for general-purpose serving; wins on heterogeneous fleets and unique-prompt workloads.
• SGLang (Apache 2.0): RadixAttention reuses prefix-tree KV across siblings — meaningful throughput edge over vLLM at modest sizes, large gains on prefix-heavy RAG/multi-turn, modestly lower TTFT P95. Caveat: Python-router GIL ceiling at very high concurrency. Default for DeepSeek, structured output, multi-turn agentic.
• TensorRT-LLM (NVIDIA): peak Hopper/Blackwell perf; ahead-of-time engine compile; first-class NVFP4. Best raw throughput on NVIDIA; least flexible; engine rebuild on every config change.
• TGI (HuggingFace): maintenance mode since late 2025 — bug fixes only. Has TRT-LLM/vLLM backends. Pick only for legacy compatibility.
• llama.cpp: GGUF native; CPU + Metal + Vulkan + CUDA + ROCm; tiny binaries; no continuous batching at server-grade quality. Best on Mac/edge/single-user.
• MLC-LLM (TVM-based): broad hardware (Vulkan, WebGPU, mobile); compile-once-run-anywhere; weaker server batching.

Parallelism and MoE

• TP shards weight matrices intra-layer; AllReduce per layer. Needs NVLink/NVSwitch for ≥8-way; degrades fast on PCIe.
• PP cuts the model by layers; AllReduce-free but bubble unless micro-batched.
• EP shards experts across ranks for MoE; AllToAll (not AllReduce). Pair with DP-attention for the modern wide-EP MoE pattern.
• DP replicates.

MoE serving: Mixtral-style 8-expert models are straightforward TP. Wide-MoE designs (DeepSeek-V3/R1 with hundreds of experts, MLA) are the hard case — the bottleneck is per-step expert load imbalance, addressed by hot-expert duplication / load-balancing schemes (EPLB-family). Capacity factor caps per-expert tokens-per-step; over-cap tokens drop or reroute. Specific layout/balancer choices change quarterly — re-check what your engine ships.

Long context, latency goals

• Long-context techniques: PagedAttention page tables for non-contiguous KV; sliding-window attention gets a dedicated KV-cache group in vLLM's hybrid manager so windowed and full-attention layers share blocks without waste. RoPE scaling: rope_scaling={"type":"dynamic","factor":2.0}, linear, yarn, llama3 for trained extensions. ChunkAttention: prefix-tree-aware FlashAttention.
• Latency targets: TTFT (prefill-dominated), TPOT/ITL (decode-dominated). Production: TTFT P99 < 500–1000 ms for chat; ITL P99 < 50–100 ms (≥ 10 tok/s feels live). Goodput = requests/s meeting both SLOs simultaneously — what DistServe/Mooncake actually optimize. Naive throughput hides that batching too aggressively wins tokens/s but blows TTFT P99.

Authoritative references

Engine docs:

• vLLM V1 announcement
• Inside vLLM (Sep 2025)
• vLLM optimization
• vLLM speculative decoding
• vLLM FP8 W8A8

Speculation, kernels:

• EAGLE repo (1/2/3)
• Anyscale continuous batching

Guardrails

Before recommending a non-trivial serving change (KV dtype, quantization scheme, parallelism layout, spec-decoding method, disaggregation):

1. Quote the engine flag/parameter and its current default
2. Cite the engine's docs / release blog for the version in use
3. Make the recommendation conditional on observed metrics (TTFT/ITL P99, KV-cache hit rate, expert-utilization variance) — never blanket-tune
4. Verify the engine version — defaults shift fast in this space.

Tuning without measurement is worse than defaults.