
Date: 2026-08-16
Suitable for readers: graduate students, beginners in LLM training, AI readers with engineering backgrounds
Search date: 2026-08-16
Abstract
Efficient training poses a resource allocation question. With the same token budget, hardware, and evaluation, how do memory use, communication, recomputation, numerical stability, and final model capabilities trade off? LoRA and QLoRA bring large-model fine-tuning within reach of ordinary labs. ZeRO, activation checkpointing, FlashAttention, and Megatron/FSDP provide ways to manage the memory used by training state and activations. MoE aims to decouple total parameter capacity from compute per token. Work on optimizers such as GaLore, Q-GaLore, and Muon revisits optimizer state, the subspace in which updates are made, and stability at scale.
This article gives an intensive reading route of the paper: first learn to write "training resource ledger", and then press PEFT, quantify fine-tuning, Sparse experts, memory parallelism, optimizer five lines to read the representative paper, and finally design a small-scale reproduction experiment to verify what an efficiency method has omitted, what has been given, and whether the conclusion can be stopped.
Contents
- Why is this topic important
- Core reading method
- A reading guide to the key papers
- Methods and experiments compared
- How to reproduce the results
- Common misconceptions
- Research questions for graduate students
- Summary and References
Why is this topic important
The large model training bottleneck is not a single "GPU is not enough". Freezing parameters can reduce the gradient and optimizer state, but do not necessarily reduce the activation of the memory; quantify the weight can save the model memory, but may introduce numerical errors and kernels; checkpointing Replace the memory with additional forward recalculation; ZeRO/FSDP shards parameters, gradients and optimizer states, but introduces communication; MoE increases the total capacity, but brings Router, Load Balancing, Expert Parallelism and stability issues; optimizer improvements may reduce state or improve convergence, but must match the learning rate, batch, warmup, training token and hardware when comparing.
These papers are especially valuable for graduate students. First, you can make interpretable reproducing in the following model, public data and stand-alone environments. Second, it forces you to report Peak Memory, Tokens/Sec, Wall-Clock, Loss Curve, final metrics and failure rate at the same time, instead of just picking a good-looking number. Third, efficiency methods often have clear boundary conditions: some tasks are effective, some scales are invalid, and some implementations are fast but unstable. These boundaries are published research questions.
The 2025-2026 model technical report also shows that training efficiency has entered the main storyline. DeepSeek-V3, Qwen3, Kimi K2, Llama 4 and other public information are all MoE, FP8, low rank structure, expert parallelism, long context training and optimizer stability are written into the Core design; PyTorch, DeepSpeed, Megatron Core documents are also continuously updated Activation CheckPointing, ZeRO, Parallelism, and MoE training abilities. When reading a modern LLM report, if you ignore Efficient Training, it is difficult to understand how the model is trained.
Core reading method
To read the Efficient Training paper, the first step is to draw a resource ledger. Split at least into six items: Parameters, Gradients, Optimizer States, Activations, Communication, and Recompute. LoRA mainly reduces trainable parameters, gradients and optimizer states; QLoRA further compresses Frozen base weights; ZeRO/FSDP sharding Training states; activation checkpointing Reduce saved activation; FlashAttention reduces the middle amount of attention and HBM IO; MoE reduces the cost of calculating the relative total capacity per token; the GaLore class method compresses gradient statistics or optimizer state; Muon Focus on the stability of the update geometry and scaling into large-scale Training.
The second step is to distinguish the training stage. Pre-training, continued pretraining, SFT, DPO/RLHF, field fine-tuning and adapter training are not the same problem. LoRA/QLoRA is very suitable for downstream adaptation, but does not equal low-cost pre-training; GaLore attempts to retain full-parameter learning; MoE usually needs to be designed from the architecture and system level together; checkpointing and parallel systems span multiple stages.
The third step is to check the strength of the evidence. Trusted papers should try to fix the model, data, token budget, hardware, evaluation and random seeds, and only change the efficiency method. Only report trainable parameters, not activation; only Peak memory, not step time; only for final benchmark, no loss curve. The results of tokens are placed in a table, which are all signals that need to be vigilant.

A reading guide to the key papers
1. LoRA / PEFT: How Low Rank Updates Change Fine-Tuning
LoRA freezes the pretrained weights and adds low-rank updates to Transformer layers, approximating full fine-tuning by restricting adaptation to a low-rank subspace. Two questions matter here. Why might the update have low rank, and which modules should receive it? In a reproduction, vary the rank, alpha, target modules, dropout, bias settings, and whether the updates are merged into the weights. Compare with full fine-tuning under the same token budget.
The variants of DoRA, LoRA+, RSLoRA, LoftQ, and aLoRA after 2024 can be read as a second layer. DoRA splits the weights into Magnitude and Direction; LoRA+ updates two Adapter matrices with different learning rates; the PEFT document shows that these methods have become composable tools. When reading a variant paper, the focus is not "the new method must be stronger", but to find out which boundary condition of LoRA it has fixed.
2. QLoRA: Quantify the training adaptation
The key to QLoRA is the combination scheme: NF4 quantization, double quantization, paged optimizers, and training LoRA Adapter on the quantization base model. It lowers the weight of the frozen model to 4-bit storage, while preserving the trainability of the Adapter. When intensive reading, draw clearly which tensors are quantized, which tensors can be trained, how the gradient flows, and where the optimizer states exist. BitsAndBytes, Transformers, PEFT, CUDA, GPU models, Compute DType, and Gradient enabled must be logged when reappearing Checkpointing, because these implementation details directly affect stability and speed.
3. Memory / Parallelism:ZeRO、Checkpointing、FlashAttention、Megatron
The core of ZeRO is to remove redundant training states in parallel data. The DeepSpeed document disassembles it into Stage 1 Optimizer states, Stage 2 Gradients, Stage 3 Parameters, and provides CPU/NVMe Offload. When reading ZeRO, you should look at the video memory savings with all-gather, reduce-scatter, and offload bandwidth.
Activation Checkpointing Replaces the memory with recalculation. The PyTorch documentation clearly states that partial forwarding will be re-executed during back-propagation; if dropout and RNG state are involved, you should also pay attention to Determinism and performance overhead. FlashAttention reduces intermediate matrix and HBM access through IO-aware attention, which is especially critical for long sequence training. Megatron-LM/Megatron Core provides training blocks such as Tensor, Pipeline, Data, Expert, and Context Parallelism. The reading method of system papers and documents is to translate each switch into "which type of tensor is divided, which type of communication is introduced, and which type of Kernel is fused".
4. MoE: Decoupling of Total Capacity and Activation Calculation
GShard and Switch Transformer are the classic starting points for the MoE route. GShard display condition calculation and automatic sharding; Switch Transformer simplifies expert selection with Top-1 Routing, and put Load Balancing, Capacity Factor, BFloat16 Stability is at the core. Then Mixtral, DeepSeekMoE, DeepSeek-V2/V3, Qwen3, Kimi K2, Llama 4 and other reports pushed the MoE to the mainstream of the open model.
When reading a MoE report, you can't just look at Total Parameters. Activated Parameters, Experts Per Token, Shared Experts, Router, Load Balancing Loss, Expert Parallelism, Token Dropping and communication must be recorded at the same time. The model parameters, token number, GPU HOURS, BENCHMARK and repository status are all easy to change. Before the official reference, it should be returned to the official model card and technical report for re-checking.
5. Optimizer: Memory and Stability After AdamW
AdamW is a strong baseline for the new optimizer, because it decouples Weight Decay from the Adam update. The problem setting of GaLore is Optimizer State Memory: It is not freezing weights, but maintains the optimizer state in a low rank gradient projection space, trying to keep the full-parameter learning. Q-GaLore further combines quantization and layer-adaptive low-rank Gradients. The Muon and Kimi K2 report pushes the Optimizer scalability and training stability to the center.
The most misjudgmental contributions here: the effect of the optimizer is often mixed with algorithm, implementation, learning rate, batch, data and hardware. When reading a paper, you must adhere to the same model, the same data, independent parameter adjustment, complete loss curve and failure rate, otherwise it is difficult to judge whether the optimizer is better or the baseline is not adjusted.
Methods and experiments compared
| branch | representative data | Main savings | Main cost | intensive reading |
|---|---|---|---|---|
| LoRA / PEFT | LoRA, DoRA, LoRA+, PEFT | trainable params、gradients、optimizer states | May restrict update subspaces | rank, target modules, the same budget full baseline |
| QLoRA | QLoRA, PEFT quantization | Frozen base weights | Quantization error, Kernel/version limit | NF4、compute dtype、paged optimizer |
| ZeRO / FSDP | ZeRO, DeepSpeed, PyTorch FSDP | Parameters, gradient, optimizer state redundancy | Communications and Offload Costs | Stage, bucket, batch, network bandwidth |
| Checkpointing | PyTorch checkpointing | activations | Additional recalculation, RNG complexity | Granularity, Step Time, Determinism |
| FlashAttention | FlashAttention, Megatron | Attention intermediate quantity, HBM IO | Kernel and hardware dependencies | Sequence length, dtype, end-to-end throughput |
| MoE | GShard, Switch, Mixtral, DeepSeek, Qwen3, Kimi K2, Llama 4 | Calculate relative total capacity per token | routing, communication, stability | active vs total params、expert load |
| Optimizer | AdamW, GaLore, Q-GaLore, Muon | Optimizer states or convergence steps | Super Sensitive Sensitivity, Difficulty Attribution | Budget adjustment, loss curve, failure rate |
How to reproduce the results
First, reproduce LoRA / QLoRA fine-tuning . Select a 1B-8B open source model and a small instruction dataset to compare Full fine-tune, LoRA, QLoRA. Fixed training tokens, batch, review sets and SEED, report Peak Memory, tokens/sec, training time, loss curve, final metric and checkpoint size.
Second, reproduce the activation checkpointing trade-off. Open/close CheckPointing, change the CheckPoint granularity, and record the maximum video memory, STEP time, throughput and loss. If the model contains dropout, write the RNG and Determinism settings clearly.
Third, do a small experiment of ZeRO/FSDP. Compare Data Parallel, ZeRO-1/2/3 or FSDP in a multi-card environment; you can try OFFLOAD for a single card, but do not extrapolate the single-card OFFLOAD conclusion to a high-speed interconnected cluster.
Fourth, do TOY MoE. Change the FFN of the small Transformer to 4 or 8 experts, and compare the top-1/top-2 routing, shared expert, and load balancing loss. Report Expert Load Entropy, Dropped Token Rate, Active FLOPs, and Loss Curve.
Fifth, do Optimizer Ablation. Use AdamW as a strong baseline, and then add Adafactor, GaLore or Muon. Each optimizer has to independently adjust the learning rate and weight decay, and report the failure rate. Negative results should also be written, because the boundary conditions of efficiency papers are often more valuable than positive results.
Common misconceptions
- Just look at Trainable Parameters. LoRA parameters are few, but the Activation Memory, Attention, and Batch Size may still decide whether it can be trained.
- Treat Total Parameters as a training cost. MoE must report Active Parameters, Experts Per Token, Router and communication.
- Only Peak Memory is reported. It is not necessarily more effective to save video memory, but the STEP time is doubled, and the final loss is deteriorated or the training is unstable.
- Does not match Token budget and adjust the budget. Efficiency methods often change the optimal learning rate, warmup, rank, batch and precision.
- Ignored version. PEFT, BitSandbytes, PyTorch, DeepSpeed, Megatron, CUDA Kernel, official model cards and technical reports are constantly changing. For content involving the amount of parameters, GPU HOURS, BENCHMARK, and repository status, the search date should be marked before the official release;
Research questions for graduate students
- LoRA Rank Scaling: Compare rank, target modules and data scale on 0.5b, 1b, 3b models.
- QLoRA Stability Audit: Compare NF4, FP4, BF16/FP16 Compute DType, Paged Optimizer, and CheckPointing.
- Checkpointing Policy Search: Automatically select CheckPoint node under fixed video memory budget.
- Small-scale MoE routing mechanism: Study Shared Expert, Load Balancing Loss, and Capacity Factor.
- Optimizer State Compression Reproduction: Compare the differences between AdamW, Adafactor, GaLore, Q-Galorere in SFT and Continued pretraining.
- Efficiency Report Template: Rewrite video memory, throughput, communication, failure rate and reproducing boundaries for a LLM training paper.
Summary
The correct reading of the Efficient Training paper is to translate "more economical" into an auditable cost, quality and stability triangle. LoRA/QLoRA lowers the threshold of fine-tuning, ZeRO/checkpointing/FlashAttention/Megatron split the Training system into composable modules, MoE Changing the relationship between capacity and activation calculations, Optimizer continues to challenge AdamW's memory and convergence boundaries. The most important Training ability for newcomers to scientific research is to reproduce a resource ledger: under the same model, data and budget, what has been saved, what has been paid, and whether the conclusion can still be stopped.
References
Search date: 2026-08-16. The following links give priority to first-hand or close first-hand information such as paper pages, official project pages, official documents, GitHub, JMLR, ML Anthology, and Hugging Face paper pages. Model parameters, training token, GPU hours, benchmark, repository status, document version and API behavior may change, please re-check before submitting or publishing.
- LoRA: Low-Rank Adaptation of Large Language Models, ICLR 2022 / ML Anthology: https://mlanthology.org/iclr/2022/hu2022iclr-lora/
- QLoRA: Efficient Finetuning of Quantized LLMs, arXiv 2305.14314: https://arxiv.org/abs/2305.14314
- QLoRA official GitHub repository: https://github.com/artidoro/qlora
- Hugging Face PEFT documentation: https://huggingface.co/docs/peft/en/index
- Hugging Face PEFT LoRA reference: https://huggingface.co/docs/peft/main/en/package_reference/lora
- Hugging Face PEFT quantization guide: https://github.com/huggingface/peft/blob/main/docs/source/developer_guides/quantization.md
- DoRA: Weight-Decomposed Low-Rank Adaptation, NVIDIA Research: https://research.nvidia.com/labs/lpr/publication/liu2024dora/
- LoRA+: Efficient Low Rank Adaptation of Large Models, arXiv 2402.12354: https://arxiv.org/abs/2402.12354
- Decoupled Weight Decay Regularization, ICLR 2019 / ML Anthology: https://mlanthology.org/iclr/2019/loshchilov2019iclr-decoupled/
- ZeRO: Memory Optimizations Toward Training Trillion Parameter Models, arXiv 1910.02054: https://arxiv.org/abs/1910.02054
- DeepSpeed ZeRO tutorial: https://github.com/microsoft/DeepSpeed/blob/master/docs/_tutorials/zero.md
- DeepSpeed large-model training guide: https://www.deepspeed.ai/tutorials/large-models-w-deepspeed/
- DeepSpeed ZeRO++ tutorial: https://www.deepspeed.ai/tutorials/zeropp/
- PyTorch activation checkpointing blog: https://pytorch.org/blog/activation-checkpointing-techniques/
- torch.utils.checkpoint documentation: https://docs.pytorch.org/docs/stable/checkpoint
- FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness, arXiv 2205.14135: https://arxiv.org/abs/2205.14135
- Megatron-LM / Megatron Core GitHub repository: https://github.com/NVIDIA/Megatron-LM
- GShard, ICLR 2021 / ML Anthology: https://mlanthology.org/iclr/2021/lepikhin2021iclr-gshard/
- Switch Transformers, JMLR 2022: https://www.jmlr.org/beta/papers/v23/21-0998.html
- Mixtral of Experts, Mistral AI: https://mistral.ai/news/mixtral-of-experts/
- DeepSeekMoE official repository: https://github.com/deepseek-ai/DeepSeek-MoE
- DeepSeek-V2 official repository: https://github.com/deepseek-ai/deepseek-v2
- DeepSeek-V3 official repository and technical report link: https://github.com/deepseek-ai/DeepSeek-V3
- Qwen3 official repository and technical report citation: https://github.com/QwenLM/Qwen3
- Kimi K2 official repository and technical report: https://github.com/MoonshotAI/Kimi-K2
- Muon is Scalable for LLM Training, arXiv 2502.16982: https://arxiv.org/abs/2502.16982
- Llama 4 model card, Meta repository: https://github.com/meta-llama/llama-models/blob/main/models/llama4/MODEL_CARD.md
- Llama 4 official blog: https://ai.meta.com/blog/llama-4-multimodal-intelligence/
- GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection, arXiv 2403.03507: https://arxiv.org/abs/2403.03507
- Q-GaLore, arXiv 2407.08296: https://arxiv.org/abs/2407.08296
- Scalable Parameter and Memory Efficient Pretraining for LLM, arXiv 2505.22922: https://arxiv.org/abs/2505.22922
- EfficientLLM: Efficiency in Large Language Models, arXiv 2505.13840: https://arxiv.org/abs/2505.13840
- Parameter-efficient fine-tuning in large language models: a survey of methodologies, Artificial Intelligence Review 2025: https://link.springer.com/article/10.1007/s10462-025-11236-4
- A Survey on Mixture of Experts in Large Language Models, official repository: https://github.com/withinmiaov/A-Survey-on-Mixture-of-Experts-in-LLMs