Cover illustration

Date: 2026-08-17
Suitable for readers: graduate students, beginners in LLM system, AI readers with engineering backgrounds
Search date: 2026-08-17

Abstract

The core of the Efficient Inference research is not to "make the model run faster", but to understand the cost structure of autoregressive generation: the prefill stage is usually more like a large matrix calculation, and the decode stage is used by the KV cache. Read and write, video memory bandwidth, scheduling, and BATCH morphological limitations. KV cache, PagedAttention, Continuous Batching, Speculative Decoding, Quantitative, Prefix Cache, Prefill/Decode Disggregation and Serving Runtime is all about surgery around this cost chain.

This paper gives a intensive reading route of the paper: first disassemble the inference into six accounts of TTFT, ITL, Throughput, video memory, quality and scheduling, and then press KV cache and specularative. Decoding, Quantization, Kernel/Runtime, and Serving System Five lines to read representative papers. Finally, a reproducible small experimental design is given to help graduate students determine what a reasoning optimization method is omitted, what sacrifices, and whether it can be established on its own model and load.

Contents

  1. Why is this topic important
  2. Core reading method
  3. A reading guide to the key papers
  4. Methods and experiments compared
  5. How to reproduce the results
  6. Common misconceptions
  7. Research questions for graduate students
  8. Summary and References

Why is this topic important

The training efficiency answers "how the model is made", and the inference efficiency answers "how the model is used repeatedly". The latter is also a serious research problem for graduate students: the same open source model, in different Serving Engine, Batch Policy, Cache Policy, Quantization Recipe and Decoding Strategy Under, TTFT, ITL, throughput, video memory peak and output quality may be completely different. If the paper only reports "a few times faster", but does not specify the Prompt length, output length, batch arrival process, GPU model, concurrency, sampling settings and quality check, the conclusion is difficult to reuse.

Autoregression imposes the first practical constraint on modern LLM inference. Generating 100 tokens takes 100 stateful iterations. During prefill, the model processes the prompt and stores the attention keys and values for each layer. Each decoding step adds only one or a few tokens but must read the existing KV cache. For long contexts, multi-turn conversations, RAG, and agent tasks, the cache can account for a large share of GPU memory. A 2025 TMLR survey groups KV cache optimizations by whether they operate at the token, model, or system level, reflecting how far the field has grown beyond individual techniques.

The second constraint is Workload. Offline batch processing pursues tokens/sec; interactive Q&A focuses on TTFT and ITL; code completion focuses on short delay and cancel requests; RAG focuses on prefix reuse; Agent workflow focuses on multiple small requests and structured output; MoE The model also adds Expert Routing and cross-card communication. The Efficient Inference paper must be bound to the load, and cannot directly extrapolate the speed conclusion in one scenario to another.

Core reading method

Read the theoretical essay, first draw six accounts.

The first is the Latency account. TTFT is the time from the request to the first token, which is mainly affected by prefill, queuing, scheduling and cache hits; ITL is the interval between continuous output tokens, decode, video memory bandwidth, kV cache Reading and Sampling Kernel are more critical. Only the average delay is reported, and it is best to look at P50/P95/P99.

The second is throughput account. tokens/sec to distinguish between Prompt Tokens, Generated Tokens, Tokens, and per GPU tokens/sec. The larger the batch, the higher the throughput, but the user waits for possible longer. Continuous batching, chunked prefill, request preemption, and admission control are all scheduling between throughput and delay.

The third is the Memory account. Model weights, KV cache, Activation temporary amount, CUDA graph, workspace, fragmentation, LoRA adapter, and paged blocks should be viewed separately. The contribution of PagedAttention is to change the KV cache from continuous allocation to block management similar to virtual memory, reduce fragmentation and duplication of storage, thereby allowing larger effective batches.

The fourth is the quality account. Quantization, Speculative Decoding, Cache Eviction, Prefix Reuse, Structured Decoding may all affect output distribution or task metrics. The Speculative decoding of Leviathan et al. clearly pursues the purpose of not changing the target model distribution; Medusa-2, approximated cache multiplexing or low-bit quantization require additional quality evaluation, not just speed.

The fifth is the Implementation account. The conclusions of FlashAttention, FlashInfer, TensorRT-LLM, vLLM, SGLang, TGI, Transformers cache classes all depend on CUDA, ROCm, GPU Schema, Kernel version, model structure, dtype and framework version. For the official reproducing experiment, you must write clear commit, image, driver and startup parameters.

The sixth is the workload account. Synthetic Benchmark with a fixed batch cannot represent online Poisson; 128-token prompt cannot represent long context; use greedy decoding cannot represent top-p Sampling; single-wheel QA cannot represent RAG or Agent. When reading the paper, the author's load generator should be re-engraved, otherwise the velocity number has no coordinate system.

Efficient Inference Reading Roadmap

A reading guide to the key papers

1. KV cache and PagedAttention: the basic ledger for inference video memory

The intuition of KV cache is very simple: each step of the autoregressive model needs the key/value of the past token. Hugging Face Transformers document divides cache strategy into dynamic, static, offloaded, quantized and other types, emphasizing different caches There are trade-offs between speed, video memory and compilation compatibility.

PagedAttention is a key paper in this area. Kwon et al. (SOSP 2023, arXiv 2309.06180) explain why high-throughput LLM serving depends on batching. Each request has a large KV cache whose size changes over time, and poor cache management leads to fragmentation and duplication. PagedAttention borrows the operating system idea of paging to store the KV cache in noncontiguous blocks. vLLM builds a serving system around it. To interpret the reported 2–4× throughput, check the baseline, the distribution of prompt and output lengths, and how the latency constraint was held fixed. Why might beam search or parallel sampling benefit more?

The KV cache study after 2025 continued to extend to three categories. The first category is compression and quantization, such as KIVI to study the distribution difference between key cache and value cache, and propose Tuning-Free asymmetric 2-bit KV cache quantization. The second category is selection, discarding, merging and budget allocation, trying to preserve the most useful KV in long context. The third category is systematic reuse, such as LMCache converting KV cache from a single request state to a cross-request, cross-engine, and cross-storage hierarchy-based infrastructure; SGLang's RadixAttention also serves complex programs prefix/cache reuse.

2. Speculative Decoding: Reduce the number of serial decode steps

The problem setting of Specularative decoding is: decode must be serialized by token, but the utilization rate of GPU is low in single token decode. Leviathan, Kalman, and Matias proposed to guess multiple tokens with a small drag model in ICML 2023, and then the big target model parallel verification; token. The important constraint of the paper is "not changing the output distribution", so it is not just a Heuristic Acceleration, but an algorithm with sampling correction.

To read Speculative decoding, you need to grab three points. First, the quality of Draft Model determines the Acceptance rate; Draft is too weak to verify, and Draft is too strong and may offset the speed gain. Second, the speed is affected by the ratio of batch size, sequence length, sampling tempperature, target/draft, and is not a fixed constant. Third, the system implementation is critical: how the two models are placed, how the KV cache is managed, whether the Draft and the target are shared tokenizer, and whether the communication and scheduling offset the benefits.

Medusa and EAGLE are subsequent representatives. Medusa does not maintain an independent draft model, but adds multiple decoding heads on the same backbone, and uses tree attention to verify candidates; its arXiv version is reported Medusa-1/Medusa-2 Two training settings. EAGLE transfers the draft generation to the Feature level and continues to improve it in the 2025 revision. When reading these papers, in particular, it is necessary to distinguish the boundaries of "Lossless Acceleration" "maintain the quality is almost the same" and "requires additional training".

3. Quantization: Low-precision reasoning for weights, activation and kV Cache

Quantitative papers are most likely to be misread as "the lower the bit, the better". In fact, it is necessary to distinguish between Weight-Only, Weight+Activation, KV cache Quantization, FP8/FP4, INT8/INT4, Per-Channel/Per-Token/Group-wise, Calibration Set and Kernel support.

GPTQ is a classic starting point for Post-Training Weight Quantization, using similar second-order information for one-shot quantization. SmoothQuant migrates quantization difficulties from Activation to Weights for Activation Outlier, thereby supporting W8A8 inference. AWQ observes that a small amount of salient weight can be significantly reduced, using activation-aware scaling to do low bit weight-only quantization, and in MLSys 2024 published. KIVI turns his attention to the KV cache, pointing out that the Key Cache is more suitable for Per-Channel, and the Value Cache is more suitable for Per-Token.

When reading a quantitative paper, three tables must be checked at the same time: the quality table, the speed table, and the hardware table. The quality table depends on whether the task is covered with confusion, Q&A, code, math, long context and multimodal; Differences between A100/H100/4090/Jetson/MI300X/CPU. TensorRT-LLM and vLLM documents show, FP8 KV cache, AWQ, GPTQ, FP4/NVFP4, different GPUs Backend support is constantly changing, so the version and search dates are marked when referencing.

4. Kernel and Runtime: from FlashAttention to FlashInfer

Reasoning optimization does not only happen in the paper formula. Attention Kernel, GEMM Kernel, Sampling Kernel, CUDA Graph, Torch.Compile, Paged/Ragged KV Layout, MoE Grouped GEMM and Fused operators will determine whether the paper method is really faster.

The FlashAttention series solves the IO-aware Attention, and both training and prefill benefit. FlashInfer is more direct-oriented to LLM Serving Kernel, the official repository lists prefill, decode, append, paged/ragged KV cache, MLA attention, cascade Attention, Sparse Attention, FP8/FP4, Fused MoE, Sorting-Free Sampling, and Speculative Decoding. When reading the Kernel/Runtime document, the research problem is not "Which library is the fastest", but to figure out why a workload falls to a certain kernel: batch is ragged, kv blocks is Paged, whether to mix prefill/decode, whether MoE, whether structured output, whether it needs low precision.

5. Serving System:vLLM、SGLang、TensorRT-LLM、TGI、LMCache

Serving system papers and documents should be read by "Request Life Cycle": Entry, Tokenize, Prefill, KV Block Assignment, Decode Step Scheduling, Sampling, DeTokenize, Streaming, Finish, release cache, indicators are reported. The vLLM official website emphasizes PagedAttention, Advanced Scheduling and Continuous Batching; SGLang's NeurIPS 2024 paper put the Structured Language Model programs as objects, use RadixAttention and Runtime optimization to support complex programs such as Agent, FEW-Shot, JSON Decoding, RAG, multi-round dialogue, etc. Management is independent, supports GPU/CPU/Disk/Remote Storage, non-Prefix multiplexing, prefill/decode disggregation and observation indicators; TensorRT-LLM Provides production-level functions such as quantization, Speculative Decoding, KV cache Connector/Offloading, etc.; Hugging Face TGI documents show early Optimized Inference Common feature of the engine: Continuous Batching, Flash/Paged Attention, Tensor Parallelism, Streaming, Quantization and Metrics.

This type of data is updated very quickly. Don't take the official website slogan as a paper conclusion when reading, but go back to the Release Note, Benchmark script, Commit, configuration file and reproducible experiments. In particular, "support a model", "support a certain quantization format" and "support a GPU architecture" will change with the version. The article can only write the status under the search date, and it should be re-checked before the official release.

Methods and experiments compared

branch representative data Main savings Main cost intensive reading
PagedAttention / vLLM PagedAttention, vLLM KV cache fragments, duplicate storage, valid Batch Block management and scheduling complexity prompt/output distribution, latency constraint, baseline
KV cache quantization KIVI, Transformers QuantizedCache KV cache video memory, long context Batch Quantize error, additional dequant overhead Key/value distribution, length context, quality indicators
Speculative decoding Leviathan et al., Medusa, EAGLE Serial Decode Steps Draft/heads training, acceptance rate, system complexity acceptance rate、distribution/quality、batch size
Weight/activation quantization GPTQ, SmoothQuant, AWQ, TensorRT-LLM Weight memory, bandwidth, GEMM cost Calibration, Kernel support, reduced precision W/A/KV bit width, calibration set, end-to-end throughput
Kernel runtime FlashAttention, FlashInfer HBM IO, attention/sampling/MoE kernel overhead Hardware and version dependencies kernel benchmark vs serving benchmark
Serving framework vLLM, SGLang, TGI, TensorRT-LLM Scheduling, Batch, Parallel, Streaming Complex configuration, load-sensitive TTFT, ITL, P95, cancel request, monitoring
KV reuse layer LMCache, RadixAttention Repeat prefill, long context multiplexing Cache consistency, storage and transport overhead Cache hit rate, Reuse granularity, quality recovery

How to reproduce the results

First, do a minimum serving benchmark. Select the 7B-8B model, fixed GPU, dtype, prompt length distribution and output length distribution, and compare Transformers generator, vLLM, SGLang or TGI. At least TTFT, ITL, request/sec, prompt tokens/sec, decode tokens/sec, GPU memory, GPU utilization, p50/p95 are recorded. Don't just record a single request time.

Second, the KV cache ledger is reproduced. Change MAX Context, Batch Size, Output Length, turn on/off Paged Attention, Prefix Caching, Offloading or Quantized Cache. Disassemble the video memory into Model weights, KV cache, and other Runtime Workspaces. Long context experiments require additional OOM boundaries and the presence of Cache Eviction.

Third, reproduce the Speculative decoding. Compare the two settings of greedy and sampling with a target model and a small draft model. Record Acceptance Rate, average number of tokens per step, number of target forwards, number of draft forwards, TTFT, ITL, output consistency, or task metrics. If using Medusa/EAGLE, indicate whether heads or feature predictor retrain.

Fourth, the recurring quantization. Select a Recipe in GPTQ/AWQ/SmoothQuant/FP8 to fix the same model and the same evaluation set. Report Perplexity or task metrics, end-to-end Latency, throughput, video memory, and load time. Don't equate the acceleration of a single GEMM Kernel directly from the application end-to-end acceleration.

Fifth, do Serving Workload Ablation. Generate three types of requests: short PRIMPT short output, long PRIMPT short output, long PROMPT long output. Then use low concurrency, high concurrency, and burst to arrive. Observe the changes of Continuous Batching, Chunked Prefill, Prefix Cache and P/D Disggregation in different scenarios. Negative results are reserved, because the boundary conditions of the inference system are more important than the single maximum throughput.

Common misconceptions

  1. Mix prefill and decode into one metric. TTFT and ITL reflect different bottlenecks, and the optimization direction is also different.
  2. Just look at tokens/sec. High throughput may come from a large batch, but the p95 Latency may not be acceptable.
  3. Treat KV cache as a free state. Under long context and high concurrency, KV cache often determines the upper limit of the batch more than the weight.
  4. Only refer to speed multiples, not Workload. prompt/output Length, concurrency, sampling, GPU, dtype, and framework versions are indispensable.
  5. Ignore quality. Quantization, approximate cache, structured decoding, and non-lossless specular heads all require quality or distribution checks.
  6. Use the official website to support the matrix as a permanent fact. vLLM, SGLang, TensorRT-LLM, FlashInfer, Transformers and LMCache are all fast updates, and model, GPU, quantization format, and API behavior must be checked on a search date.

Research questions for graduate students

  1. Long context KV cache policy comparison: Compare FULL CACHE, QUANTIZED CACHE, OFFLOAD, SELECT and EVICTION on LongBench or self-built long document QA.
  2. Specualative Decoding Workload Map: System researches the relationship between draft size, temperature, batch size, output length and acceptance rate.
  3. End-to-end quantization reproducing report: Compare the quality, video memory and delay of AWQ/GPTQ/FP8 under the same Serving Engine.
  4. Prefix Cache for RAG: Construct multi-document multiplexed loads to measure Prefix hit rates, TTFT and quality.
  5. P/D Disggregation Small Experiment: Simulate Prefill Worker and Decode Worker on two or more GPUs to analyze transmission bottlenecks.
  6. Serving Benchmark Certification Study: Compare the impact of synthetic benchmark, real trace and agent workload on system ranking.
  7. Structured output decoding Cost analysis: Compare the delays and failure rates of JSON Schema, Tool Call, and Grammar Decoding in different runtimes.

Summary

The correct pronunciation of Efficient Inference is to split "faster" into a reproducible cost ledger. KV cache determines long context and concurrent memory boundaries; PagedAttention and serving scheduler determine whether batch is really effective; spherede decoding tries to reduce serial decoding number of steps; quantization reduces the byte stream of weight, activation or kv; kernel/runtime determines whether the algorithm can fall to the hardware; LMCache, SGLang, vLLM, TensorRT-LLM and other systems combine these technologies into the request life cycle.

Scientific newcomers can start training abilities with a small model and a GPU: fixed workload, change an inference switch, and record TTFT, ITL, throughput, video memory, quality and failure cases. As long as the ledger is clear enough, a "no acceleration" reproduction experiment may be more research value than a beautiful speedometer.

References

Search date: 2026-08-17. The following links give priority to paper pages, official project pages, official documents, GitHub, pmr, MLSys, TMLR/ml Anthology, Stanford MAST Lab, Hugging Face and NVIDIA document. Model support, GPU architecture, quantitative format, benchmark numbers, repository commit, API behavior and document content change very quickly, please re-check before the official release; the details that have not been confirmed in the first-hand data should be marked with "to be verified by labor".

  1. Efficient Memory Management for Large Language Model Serving with PagedAttention, arXiv 2309.06180 / SOSP 2023: https://arxiv.org/abs/2309.06180
  2. Efficient Memory Management for Large Language Model Serving with PagedAttention, ACM SOSP 2023: https://doi.org/10.1145/3600006.3613165
  3. vLLM official website: https://vllm.ai/
  4. vLLM GitHub repository: https://github.com/vllm-project/vllm
  5. vLLM quantization documentation: https://docs.vllm.ai/en/latest/features/quantization/
  6. A Survey on Large Language Model Acceleration Based on KV Cache Management, TMLR 2025 / ML Anthology: https://mlanthology.org/tmlr/2025/li2025tmlr-survey/
  7. Hugging Face Transformers cache strategies documentation: https://huggingface.co/docs/transformers/main/kv_cache
  8. KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache, arXiv 2402.02750: https://arxiv.org/abs/2402.02750
  9. KIVI official GitHub repository: https://github.com/jy-yuan/KIVI
  10. Fast Inference from Transformers via Speculative Decoding, ICML 2023 / PMLR: https://proceedings.mlr.press/v202/leviathan23a.html
  11. Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads, arXiv 2401.10774: https://arxiv.org/abs/2401.10774
  12. Medusa official GitHub repository: https://github.com/FasterDecoding/Medusa
  13. EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty, arXiv 2401.15077: https://arxiv.org/abs/2401.15077
  14. GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers, ICLR 2023 / OpenReview record: https://openreview.net/forum?id=tcbBPnfwxS
  15. GPTQ publication record, ETH Research Collection: https://www.research-collection.ethz.ch/entities/publication/00736213-37b2-4e99-b015-141349b71413
  16. SmoothQuant official GitHub repository: https://github.com/mit-han-lab/smoothquant
  17. AWQ: Activation-aware Weight Quantization for On-Device LLM Compression and Acceleration, MLSys 2024: https://proceedings.mlsys.org/paper_files/paper/2024/hash/42a452cbafa9dd64e9ba4aa95cc1ef21-Abstract-Conference.html
  18. AWQ official GitHub repository: https://github.com/mit-han-lab/llm-awq
  19. TensorRT-LLM official documentation: https://nvidia.github.io/TensorRT-LLM/
  20. TensorRT-LLM quantization documentation: https://github.com/NVIDIA/TensorRT-LLM/blob/main/docs/source/features/quantization.md
  21. TensorRT-LLM speculative decoding documentation: https://github.com/NVIDIA/TensorRT-LLM/blob/main/docs/source/features/speculative-decoding.md
  22. FlashInfer official GitHub repository: https://github.com/flashinfer-ai/flashinfer
  23. SGLang: Efficient Execution of Structured Language Model Programs, NeurIPS 2024 / Stanford MAST Lab: https://mast.stanford.edu/pubs/sglang_efficient_execution_of_structured_language_model_programs/
  24. SGLang official documentation: https://docs.sglang.io/
  25. LMCache official documentation: https://docs.lmcache.ai/index.html
  26. LMCache official GitHub repository: https://github.com/LMCache/LMCache
  27. Hugging Face Text Generation Inference documentation: https://huggingface.co/docs/text-generation-inference/index