Cover illustration

Date:2026-07-04
Who this is for: Graduate student, LLM system researcher, reasoning platform and AI application engineer

Abstract

From 2025 to 2026, the main battlefield of the LLM system is shifting from "how to train a larger model" to "how to serve the model quickly and cheaply to real users". The reason is very direct: inference is not a one-time cost, but the cost that will continue to occur for each request, every token, and every agent tool call. Especially in long context, inference model, multi-round dialogue and code agent scenarios, the bottleneck is often no longer a single matrix multiplication, but KV cache memory, prefill/decode stage interference, request scheduling, low BATCH delay, quantization error, system problem of the GPU core and network transmission.

This article sorts the technical routes of Efficient LLM Inference and Serving according to the system stack: How PagedAttention and vLLM use KV cache as operating system-style paging resource management; SGLang's How RadixAttention, vLLM Automatic Prefix Caching, and Mooncake's KVCache-centric architecture turns "duplicate context" into a reusable asset; speculative Decoding, Medusa, EAGLE-3 and TensorRT-LLM project support how to rewrite serial decode into draft-and-verify; KIVI, QServe, TurboQuant and How TensorRT-LLM quantization route extends low bits from weight to kV cache; FlashAttention-3 and FlashInfer indicate that kernel layer optimization is still the base of the service system.

Contents

  1. Research background: Why reasoning has become a bottleneck in AI infrastructure
  2. Roadmap in the past year: from stand-alone acceleration to layered Serving architecture
  3. Key papers by research question
  4. Comparing the methods
  5. Technical trends
  6. Implications for deployment
  7. Limitations and open questions
  8. Questions to watch
  9. Summary
  10. References

Research background: Why reasoning has become a bottleneck in AI infrastructure

The core indicators of the Transformer training era are throughput, parallel efficiency and training stability; the core indicator of the AT Inference Time generation becomes TTFT (Time to first token), TPOT/ITL (each output Token delay), throughput, SLO compliance rate, video memory occupancy and unit token cost. What is even more troublesome is that these indicators are often restrained from each other: increasing the batch can improve the throughput, but it may increase the delay of the first token; aggressive quantization can reduce the memory, but it may introduce additional inverse quantization overhead; KV cache linear expansion.

Generated LLM inference has two phases. Prefill phase processes Prompt, which is usually computationally dense and parallel; the Decode stage is token generated by token, which is often limited by memory bandwidth, KV cache read and serial dependencies. The short question and answer service may be mainly stuck in Decode; RAG, long document quiz and code repository agent are often stuck in long Prompt prefill and duplicate context multiplexing. The difficulty of the service system is that different request lengths, output lengths, sampling policies, LoRA Adapter, Multimodal input, structured output and Agent call chain will be mixed in the same cluster.

The classic FlashAttention brings attention optimization to the IO-aware level: don't just look at FLOPs, see how much you read and write between HBM and SRAM. PagedAttention further puts the core problem of Serving into KV cache management: KV cache will grow dynamically with the request. If continuous video memory is reserved according to the maximum length, fragments and waste will directly lower the batch size. By 2025-2026, the system begins to more explicitly regard the inference platform as a "memory, scheduling, kernel, network and caching" common optimization problem.

system architecture diagram

roadmap for the past year

The first line: KV cache has changed from a by-product to a first-class resource.
PagedAttention enables KV cache to be paginated, shared and assigned on demand; vLLM's Automatic Prefix Caching multiplexes the KV cache of the shared Prefix; SGLang's RadixAttention reuses prefixes for structured prompt, multi-round dialogue and programmatic Generation; Mooncake puts the KV cache in the center of the architecture, using CPU DRAM, SSD and independent cache Layers to serve the long context and overload scenarios.

The second line: prefill/decode is gradually decoupled.
The disaggregated serving example of DistServe, Splitwise, Mooncake and TensorRT-LLM/vLLM all points to the same judgment: prefill and decode The resource portrait of the resource is different, forcing colocate will cause interference. The work after 2025 further explores the prefill/decode partitions, cross-node KV transmissions, and stage-aware scheduling inside a single GPU.

The third line: Decode serial bottleneck with speculative and multi-token prediction mitigation.
The core of Speculative decoding is "the cheap model guess first, the target model batch validation". Medusa does not maintain an independent draft model, but adds a long on the target model; the EAGLE series moves from feature-level draft to the token prediction of EAGLE-3 Fusion with multi-layered features. The TensorRT-LLM documentation has listed DraftTarget, Eagle3, NGram, MTP, PARD, DFlash, SA and other Speculative decodings by 2026 Options, which means that it has changed from the paper skill to the Serving framework standard.

The fourth line: Quantization goes from weight to end-to-end serving.
Early routes such as GPTQ, AWQ, and SmoothQuant mainly focus on weight and activation; KIVI uses 2-bit asymmetric quantization for KV cache; QServe uses W4A8KV4 Do algorithm-system collaboration; TensorRT-LLM supports FP8, FP4, AWQ/GPTQ and FP8/NVFP4 kV cache; 2025-2026 Jobs such as TurboQuant, KVTC, GPU-accelerated INT8 KV cache continue to point the bottleneck of long context reasoning to KV cache compression.

The fifth line: The kernel library has become the competitiveness of the Serving framework.
FlashAttention-3 Asynchronous, TMA, FP8 attention optimization for Hopper GPU; FlashInfer combines attention template, JIT, block-sparse KV Format and load balancing scheduling is made into LLM Serving Kernel Engine, and has been integrated by vLLM, SGLang, MLC-Engine, etc. The more the upper layer API of the inference service is like a cloud service, the more the underlying layer is dependent on the kernel details.

Key papers by research question

1. PagedAttention and KV cache memory management

The contribution of vLLM / PagedAttention is not just "fast", but redefines the Serving memory management interface. Traditional implementations tend to reserve consecutive KV cache regions for each request, and a large number of internal and external fragments are generated when the long and short requests are mixed. PagedAttention draws on virtual memory, disassembles the KV cache of each sequence into fixed-sized blocks, and uses block table to map the logical block to the physical block. In this way, requests do not need to occupy continuous physical video memory, and blocks can be added as needed during the generation process; shared Prompt, Parallel Sampling and Beam Search can also reduce duplication through block sharing and copy-on-write.

The engineering inspiration of this line is that the long context serving is first a memory system problem. Whether a model "can run full GPU" depends not only on GEMM throughput, but also on whether the scheduler can load more active requests into the memory. The Automatic Prefix Caching in the vLLM document further brings multiplexing from the paper experiment to the common scenario: the new request can skip the shared Prefix when the same long document is queried multiple times, and the multi-round chat multiplex history Prefix is used. prefill calculation. But APC also has boundaries: it mainly reduces prefill and does not shorten the decode of long answers.

2. Structured Generation and Prefix Multiplexing

The angle of SGLang is more "LLM program". When the application is no more Prompt once, it includes retrieval, branch, parallel calls, JSON constraints, multiple rounds of self-consistent, and Agent control flow, the system needs to know which contexts can be reused and which generation steps can be parallel. SGLang paper proposes front-end language and back-end runtime, and the back-end RadixAttention can be shared prefix with RADIX Tree, which reduces duplication of kV cache and duplicate calculations. It is especially suitable for shared prefix such as FEW-SHOT, RAG, multi-round dialogue and structured output.

This also explains why the 2025-2026 Serving framework is no longer just a "model loader". They are becoming more and more like a database execution engine: understand the request structure, cache intermediate states, schedule different stages, and optimize between consistency, delay and throughput.

3. Prefill / Decode Decoupling with Disggregated Serving

DistServe explicitly proposes to put prefill and decode into different GPU pools to reduce two-stage interference, and optimize TTFT and TPOT respectively. Mooncake further puts the KV cache in the center of the architecture: using CPU DRAM, SSD and Cache Scheduler outside the GPU cluster, the goal is to support long context chat services such as Kimi. The latest documents of TensorRT-LLM and vLLM also list the capabilities of Disaggregated Serving, KV cache Connector, and KV Offloading as an important engineering interface.

The key here is not "all scenarios should be split", but splitting to make resource planning more refined. The chat traffic with short prompt and long output may require a strong Decode pool; long document Q&A requires strong prefill and cache reuse; the Agent workflow will also generate bursty, interruptable, and shared context requests. Future Serving platforms are likely to dynamically select Colocated, Chunked Prefill, Disaggregated Prefill/Decode, KV Offload or Cache Transfer by Workload.

4. Speculative decoding: change the serial decode to draft and validate

Both Leviathan et al.'s Speculative Decoding and DeepMind's Speculative Sampling both use the fact that the target model validates the cost of multiple candidate tokens at a time, and is often close to validating one. token; if the cheap draft model can guess multiple tokens, the number of serial forwarding of the target model can be reduced. In theory, versions with rejected sampling can keep the sample distribution of the target model constant.

The follow-up method starts to solve "Where does Draft come from". Medusa adds multiple decoders to the model to avoid maintaining a separate small model; the EAGLE series uses the internal features of the target model or the multi-layer feature to build a high acceptance rate DRAFT; EAGLE-3 reported on chat and reasoning in 2025. The model has a stronger acceleration effect. TensorRT-LLM's documentation to 2026 has made EAGLE 3, NGram, MTP, Suffix Automaton, etc. as configurable options. This means that the engineering problem of Speculative decoding has changed from "can speed up" to "what is worth enabling under what workload, what batch size, and what acceptance rate".

5. Quantization: Compression from model file to Serving collaboration

Low-bit quantization changes more than model size. In a serving system it changes memory bandwidth requirements, cache capacity, dequantization overhead, and the kernels that do the work. QServe’s W4A8KV4 is a case in point. Its QoQ algorithm uses 4-bit weights, 8-bit activations, and a 4-bit KV cache, with a matching QServe runtime. If low-bit arithmetic makes dequantization on CUDA cores slower, the theoretical compression may yield no throughput gain.

KV cache quantization is particularly important. KIVI observes the distribution of key cache and value cache, and adopts Per-Channel and Per-Token strategies respectively, and proposes the quantization of TUNING-FREE 2-bit KV cache. TurboQuant, KVTC, eOptShrinkQ, Fast-TurboQuant and GPU-accelerated INT8 KV cache 2025-2026 The work continues to advance along the "long context memory bottleneck". It should be noted that some indicators in the new papers and manufacturer's blog in 2026 still need to be manually retested according to the specific hardware, model, context length and workload, and cannot be directly regarded as a general-purpose SOTA.

6. Kernel and compilation: FlashAttention-3, FlashInfer and hardware collaboration

The Core idea of FlashAttention is IO-aware Exact Attention: Reduce HBM read and write through Tiling, rather than just optimizing operator FLOPs. FlashAttention-3 is oriented to Hopper GPU, using the asynchronous capabilities of Tensor Core and TMA, Warp Specialization, FP8 and other hardware features, describe the Attention Kernel It will continue to be rewritten with the hardware generation. FlashInfer makes the attention kernel into a Serving-Oriented Engine, which supports a variety of KV cache formats, JIT templates and load balancing scheduling.

For engineering teams, this means that "changing a Serving framework" is not just an API migration, but also may change the Kernel backend, CUDAGraph strategy, Batch Invariance, quantify Recipe, and observable metrics. Inference performance tuning must be measured with real request distribution pressure, not just a single sample or fixed length Benchmark.

Comparing the methods

Direction Representative Work / System Mainly resolved problems When to use it Main risk
Paged KV cache PagedAttention / vLLM KV cache fragments, dynamic growth, shared prefixes Multi-tenant Serving, Long Context, Beam/Parallel Sampling Kernel and memory management complexity increase
Prefix reuse vLLM APC、SGLang RadixAttention Duplicate Prompt/Historical Context Duplicate Calculation RAG, multi-round dialogue, FEW-SHOT, Agent programs Sensitive to shared prefixes, long output benefits are limited
Prefill/decode decoupling DistServe、Mooncake、TensorRT-LLM/vLLM disaggregated serving Prefill and decode interference, SLO split Long context, mixed traffic, large-scale clusters KV transmission, scheduling and capacity planning are complex
Speculative decoding Leviathan、Speculative Sampling、Medusa、EAGLE-3 Decode has many serial steps, low batch high delay Low BATCH, short to medium output, high DRAFT acceptance rate DRAFT maintenance costs, acceptance rate fluctuations, and decrease in BATCH
KV / Weight Quantization KIVI、QServe、TensorRT-LLM FP8/NVFP4、TurboQuant Video memory and bandwidth bottlenecks Long context, high concurrency, edge or cost-sensitive deployment Precision, Inverse Quantization Overhead, Hardware Support Differences
Attention kernel FlashAttention-3、FlashInfer HBM read and write, Kernel scheduling, hardware utilization New GPUs, long contexts such as Hopper/Blackwell Relying on the integration of hardware and framework, the threshold for reproducing experiments is high

Technical trends

Trend 1: KV cache will become the "data layer" of the inference system.
In the future, Serving is not just loading weights and running forward, but managing reusable KV, compressible KV, migrating KV, and expiring KV. The more popular the long context Agent, the code repository and the multi-round workflow, the more popular the KV cache life cycle management is like a database cache and storage system.

Trend 2: The scheduler has changed from Batch Scheduler to SLO Scheduler.
Continuous batching is only the starting point. A serving system must also account for time to first token (TTFT), time per output token (TPOT), predicted output lengths, prefix cache hit rate, GPU memory occupancy, KV transfer costs, tenant priorities, and request rejection policy. Mooncake’s early rejection, DistServe’s goodput optimization, and Nexus’s intra-GPU partitioning all reflect a shift from average throughput toward tail latency and service-level objectives (SLOs).

Trend 3: Speculative decoding will be jointly designed with model training.
In the early stage, Speculative decoding can plug into small models; now EAGLE-3, MTP, Medusa and other methods suggest that the model may be multi-token prediction and draft in the training or post-training stage. head or validation path reservation structure. The reasoning system and model structure will be more coupled.

Trend 4: Low bits are not single-point algorithms, but system synergy.
Weight quantization, activation quantization, KV cache quantization, FP8/FP4 kernel, inverse quantization Fusion, and Scheduler batch size will affect the final gain. Only Perplexity or compression ratios are reported only, and end-to-end TTFT, TPOT, throughput, and cost must be reported.

Trend 5: The open source Serving framework will continue to move closer to the production stack.
Ecology, such as vLLM, SGLang, TensorRT-LLM, LLAMA.cpp, TGI, MLC, etc., no longer only fights single-player speed, but also spells OpenAI-compatible API, Kubernetes Deployment, Metrics, KV Connector, LoRA Serving, Structured Output, Reasoning Parser, and multi-hardware backends.

Implications for deployment

First, do the workload portrait first, and then choose the optimization. At least count the input length, output length, concurrency, shared prefix ratio, streaming ratio, tool call chain length and SLO. Without this data, the discussion of PagedAttention, SPEC decoding or quantization is all blind.

Second, look at TTFT and TPOT separately. Users of long-document Q&A may be the most sensitive TTFT; code generation and long answers are more sensitive to TPOT; the agent's internal calls may be more concerned with total completion time and concurrency costs. Different indicators lead to different system solutions.

Third, the KV cache budget should be independently planned. Don't just ask "how much memory the model weight accounts for". Under long context and high concurrency, KV cache may exceed the weight to become the main memory occupancy; prefix caching, kv offload, kv quantization and cache eviction need to be designed together.

Fourth, Speculative decoding requires online switches and rollbacks. It is not a common accelerator: low BATCH, DRAFT acceptance rate is high, and the target model is highly efficient at validity; large BATCH, long complex inference, and DRAFT and target distribution do not match the target distribution may degenerate. The production system should be dynamically enabled by model, request type, and batch status.

Fifth, quantization must be end-to-end pressure measurement. Weight 4-bit is not equal to double throughput, and kv 4-bit is not equal to lossless. It is necessary to simultaneously measure task quality, long context recall, structured output stability, tool call JSON legal rate, dequantization overhead and hardware compatibility.

Sixth, retain observability. At least record prefill tokens/s, decode tokens/s, kv cache hit rate, prefix cache hits, GPU memory fragments, queueing delay, spec acceptance Rate, rejected tokens, per-tenant SLO, and OOM/eviction events.

Limitations and open questions

First, many paper metrics rely on specific models, hardware and workloads. For example, the acceleration of EAGLE-3, the throughput improvement of QServe, and the compression gain of TurboQuant cannot be unconditionally migrated to private models, different GPUs or Agent traffic.

Second, the complexity of the system is rising rapidly. Debug difficulty increases significantly after Paged KV, Disaggregated Serving, KV Offload, Speculation, Quantization, and Multiple Backend Deployment is superimposed. A performance fallback may come from scheduling, network, cache hits, kernel fallback or tokenizer inconsistencies.

Third, the quality assessment is still insufficient. Low-bit and speculative methods are commonly verified by Perplexity, MMLU, LongBench or local tasks, but production applications are also concerned about format stability, abstention boundary, tool parameter correctness and multi-round consistency. These indicators are rarely fully covered in the system paper.

Fourth, the ecological version changes rapidly. Feature Matrix for vLLM, SGLang, TensorRT-LLM, support model, quantization Recipe, and Speculative Decoding implementations are all rapidly evolving. This article covers the 2026 documentation and news section, and it is recommended to review the project's latest Release Notes and on-premises Benchmark before deployment.

Questions to watch

  1. Will KV cache form a standardized interface? If vLLM, TensorRT-LLM, SGLang, LMCache, and Mooncake Connector can communicate, the reasoning platform will be more like a combinable storage system.
  2. Will the inference model natively train the Speculative Head or Verifier? This may allow the test-time compute to merge further with the Efficient Serving.
  3. How do multimodal and Agent Workload change the scheduler? Images, videos, tool calls, and code execution can lead to a more heterogeneous phase.
  4. Where is the quality boundary of the low-bit KV cache? Long context retrieval, mathematical reasoning, code completion and security Abstention may have different sensitivity.
  5. Can AI-generated Kernel enter the production closed loop? FlashInfer-Bench work likes to connect agents to write kernels, benchmarks, and replace production, which is worthy of continuous observation.

Summary

Efficient LLM Inference and Serving are no longer a single acceleration skill, but a complete set of system engineering: KV cache The management determines how many contexts and concurrency can be installed; scheduling determines whether the throughput can be converted into user-aware delays; speculative decoding challenges AutoRegressive serial bottlenecks; quantify changes to memory, bandwidth and kernel Form; core work such as FlashAttention and FlashInfer determines whether the framework can really get hardware dividends.

For researchers, the opportunity for this direction is to model the algorithm, model structure and system constraints together. For the engineering team, the most important thing is not to pursue a "fastest framework", but to establish a reasoning platform that can be measured, retractable, and layerable evolution: real traffic portrait, end-to-end metrics, caching strategy, dynamic scheduling, quantitative verification and versioning Benchmark is the chassis for LLM application cost control in 2026.

References

Search date: 2026-07-04. The following priority is given to listing first-hand materials such as papers, official documents, and project homepages; the performance figures involving new data in 2026 are recommended for manual review before deployment.

  1. Kwon et al., “Efficient Memory Management for Large Language Model Serving with PagedAttention,” arXiv, 2023. https://arxiv.org/abs/2309.06180
  2. vLLM Documentation, “Automatic Prefix Caching,” latest developer preview, accessed 2026-07-04. https://docs.vllm.ai/en/latest/features/automatic_prefix_caching/
  3. Zheng et al., “SGLang: Efficient Execution of Structured Language Model Programs,” arXiv, 2023 / NeurIPS 2024. https://arxiv.org/abs/2312.07104
  4. Zhong et al., “DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving,” arXiv, 2024. https://arxiv.org/abs/2401.09670
  5. Qin et al., “Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving,” arXiv, 2024. https://arxiv.org/abs/2407.00079
  6. Leviathan et al., “Fast Inference from Transformers via Speculative Decoding,” arXiv, 2022 / ICML 2023. https://arxiv.org/abs/2211.17192
  7. Chen et al., “Accelerating Large Language Model Decoding with Speculative Sampling,” arXiv, 2023. https://arxiv.org/abs/2302.01318
  8. Cai et al., “Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads,” arXiv, 2024. https://arxiv.org/abs/2401.10774
  9. Li et al., “EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test,” arXiv, 2025. https://arxiv.org/abs/2503.01840
  10. NVIDIA TensorRT-LLM Documentation, “Speculative Decoding,” accessed 2026-07-04. https://nvidia.github.io/TensorRT-LLM/features/speculative-decoding.html
  11. Liu et al., “KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache,” arXiv, 2024. https://arxiv.org/abs/2402.02750
  12. Lin et al., “QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Serving,” arXiv, 2024. https://arxiv.org/abs/2405.04532
  13. NVIDIA TensorRT-LLM Documentation, “Quantization,” accessed 2026-07-04. https://nvidia.github.io/TensorRT-LLM/features/quantization.html
  14. Zandieh et al., “TurboQuant: Online Vector Quantization with Near-optimal Distortion Rate,” arXiv, 2025. https://arxiv.org/abs/2504.19874
  15. Staniszewski and Łańcucki, “KV Cache Transform Coding for Compact Storage in LLM Inference,” arXiv, 2025. https://arxiv.org/abs/2511.01815
  16. Dao et al., “FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness,” arXiv, 2022. https://arxiv.org/abs/2205.14135
  17. Shah et al., “FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision,” arXiv, 2024. https://arxiv.org/abs/2407.08608
  18. Ye et al., “FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving,” arXiv, 2025. https://arxiv.org/abs/2501.01005
  19. Xing et al., “FlashInfer-Bench: Building the Virtuous Cycle for AI-driven LLM Systems,” arXiv, 2026. https://arxiv.org/abs/2601.00227
  20. Li et al., “LLM Inference Serving: Survey of Recent Advances and Opportunities,” arXiv, 2024. https://arxiv.org/abs/2407.12391
  21. Miao et al., “Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systems,” arXiv, 2023. https://arxiv.org/abs/2312.15234
  22. Shi et al., “Nexus: Proactive Intra-GPU Disaggregation of Prefill and Decode in LLM Serving,” arXiv, 2025. https://arxiv.org/abs/2507.06608
  23. Tiwari et al., “QuantSpec: Self-Speculative Decoding with Hierarchical Quantized KV Cache,” arXiv, 2025. https://arxiv.org/abs/2502.10424
  24. Su, “eOptShrinkQ: Near-Lossless KV Cache Compression Through Optimal Spectral Denoising and Quantization,” arXiv, 2026. https://arxiv.org/abs/2605.02905
  25. Pereira et al., “Fast-TurboQuant: A Multiplier-Free Online Vector Quantization Approach,” arXiv, 2026. https://arxiv.org/abs/2606.21448