
Date:2026-08-01
Who this is for: Graduate students, new researcher who is ready to enter the LLM direction, readers with deep learning engineering background but want to systematically read architectural papers
Prerequisites: Understand matrix multiplication, softmax, residual connection, LayerNorm, language model training target
Abstract
Transformer is not a dissertation that ends after reading, but an expanding architectural pedigree. 2017 Attention Is All You Need The core paradigm of "replace cycle and convolution with attention" is given; BERT, T5, GPT-3 It is pushed to express learning, unified text-to-text tasks and large-scale autoregressive language modeling; technical reports such as LLaMA, PaLM, Qwen3, DeepSeek-V3, and LLaMA 4 have turned their research focus to Decoder-only block, normalization, location coding, kV cache, GQA/MQA, MoE, long context and multimodal fusion.
The goal of this article is not to translate all the abstracts of the paper, but to give an executable intensive reading route: every time you read an architectural paper, you have to ask four questions: Which part of the Transformer block has it changed? Does it solve the problem of training stability, expression ability, context length, reasoning cost or data/scale? Can its experiments isolate the architecture contribution? If it is to be reproduced, how should the minimum experiment be designed?
Contents
- Why Transformer routes are worth reading
- Core reading method: first find the invariant, and then find the change point
- A reading guide to the key papers
- A closer look at key papers
- Methods and experiments compared
- How to reproduce the results
- Common misconceptions
- Research questions for graduate students
- Summary
- References
Why is this topic important
When many students read Transformer for the first time, they will focus on the attention formula, the encoder-decoder diagram and the BLEU score. But modern LLM papers have rarely stopped at the "Transformer" level, they default to the reader to know the basic block, and then make a local choice: use pre-norm or post-norm? Use LayerNorm, RMSNorm, or Extra Normalization? Is the position code absolute, relative position or RoPE? Does attention head share KV? Does FFN use Gated Activation? Is Dense FFN replaced with MoE? Long context depends on architecture, training data, location extrapolation or Serving system?
Therefore, the real value of the Transformer route is to establish a set of architecture reading coordinate systems. If you read any LLM technical report in the future, you can split it into five layers: token and location, attention, FFN/MoE, normalization/residual, training target and context length. The model names in the paper will change, but these five layers will appear repeatedly.

Core reading method: first find the invariant, and then find the change point
It is not recommended to start with the Benchmark table when reading Transformer architecture papers. First draw a minimum block:
- Enter Token Embedding plus location information.
- Attention Sublayer mixes the information within the sequence.
- The FFN sublayer does a nonlinear transformation to each position.
- Residual Connection keeps gradients and original representations.
- normalization controls training stability.
- Decoder-only model also considers causal mask and kv cache.
Then make "change point annotation" for each paper:
- Change the attention: it may be multi-head, multi-query, grouped-query, latent attention, sparse/block attention, and FlashAttention.
- Change position: It may be sinusoidal, learned absolute, relative position, RoPE, long context extrapolation.
- Change FFN: it may be ReLU/GELU to GEGLU/SwiGLU, or Dense FFN to MoE.
- Change Norm/Residual: Possibly POST-NORM to pre-norm, RMSNorm, additional normalization or residual scaling.
- Change Objective: may be Masked LM, Causal LM, SPAN Corruption, Text-to-Text, Multi-Token Prediction.
- Scale: It may be a common design of parameters, data, context length, activation parameters, training accuracy and hardware efficiency.
The advantage of reading in this way is that you will not misread "the model is larger", "more data" and "adjustment are better" as "the architecture must be better". The strength of the evidence of an architectural paper depends on whether it separates the changes from the training recipe, data quality, and reasoning system.
A reading guide to the key papers
First Stop: Original Transformer.
Submitted by Vaswani et al. in 2017 Attention Is All You Need Precurrent and convolutional structures are proposed, which are completely based on the attention-based encoder-decoder architecture, and use multi-head attention, position-wise FFN, Residual, Normalization, and Positional Encoding form a sequence transduction model that can be trained in parallel. In intensive reading, don't just memorize the formula, but also focus on how the author splits the sequence modeling problem into "token-to-token information routing" and "position-independent point-by-point transformation".
Second stop: pre-training target fork.
BERT chooses Encoder-only and Masked Language Modeling, emphasizing two-way representation; GPT series selects Decoder-only and Causal language Modeling, emphasizing self-regression generation; T5 unify NLP tasks into text-to-text, and systematically compare pre-training goals, data, architecture and migration strategies. Here to read: also based on Transformer, why the input visibility, mask method and fine-tuning form will shape the model ability.
The third stop: scale decoder-only.
GPT-3, PaLM, LLaMA turned the focus of research on "how to train decoder-only LM to be large enough and available". GPT-3 Proof FEW-Shot Prompting is strong in scale; PaLM reports 540B Dense Transformer and Pathways system training; Under the budget, the small and medium parameter quantity model can also have strong performance. When reading this paragraph, you should read the architecture, data, and compute together.
The fourth station: local replacement of modern blocks.
RoPE affects the mainstream selection of position coding; RMSNorm reduces normalization overhead and is adopted by various LLMs; GLU/SwiGLU rewrites the nonlinear part of FFN; MQA/GQA Directly facing the KV cache bandwidth bottleneck in autoregressive reasoning; the FlashAttention series shows that many "architectural availability" actually rely on GPU Memory Hierarchy and Kernel implementations.
The fifth station: the system-level architecture before and after 2025.
DeepSeek-V3 uses MoE, Multi-head Latent Attention, Auxiliary-Loss-Free Load Balancing and Multi-Token Prediction; Dense and MoE dimensions, and emphasize the Thinking/Non-Thinking mode and long context support; the Llama 4 model card indicates that it uses the MoE and Early Fusion multimodal architecture. Modern technical reports are often not a single innovation, but a combination package of Architecture, Training, Data, Post-Training, and Serving.
A closer look at key papers
1. Attention is All You Need: read structure, not just read the formula
The most important contribution of this paper is not the SoftMax(qk^t / sqrt(d_k)) formula itself, but rather frees the sequence modeling from time step recursion. You have to focus on three readings:
- Why is Multi-head attention split? It allows the model to learn different relationships in different representation subspaces, rather than pressing all relationships into an attention matrix.
- Why does the Encoder-decoder attention exist? It represents the source sequence as a memory, and lets the decoder query at the time of generation.
- Why is Positional encoding necessary? Pure attention itself is not sensitive to token order and must be injected with order information.
Intensive reading action: Re-draw the paper Figure 1 into your own block, and mark the shape of each tensor. Then try to explain why the computational complexity of self-attention is related to the square length square, which will lay the foundation for subsequent long context papers.
2. BERT, T5 and GPT-3: similar architecture with different goals
The key to BERT is that BiDirectional Encoder said that it is suitable for understanding the task fine-tuning; the key to T5 is to unify the task into a text-to-text, so that it can systematically study the pre-training goals; GPT-3 The key is to push the decoder-only causal LM to the large-scale feed-shot setting. The three are not "who replaces who", but show the division of labor under different visibility constraints.
When reading this group of papers, it is recommended to make a three-list: input what you can see, what to predict during training, and how to use at inference time. BERT's Masked Token prediction and GPT's Next-Token Prediction have an impact on gradient signals, data construction, and ability to explicitly appear. T5 is suitable as a methodological sample, because it system compares multiple variables instead of just posting a larger model.
3. RoPE, RMSNorm, SwiGLU: "Three Small Changes" of Modern LLM Block
Many LLM architectures seem to be just variants of Transformer, but their blocks are already different from the 2017 version. RoPE encodes the position information into the rotational relationship of query/key, which is suitable for combining with the attention score; RMSNorm removes the re-centering of LayerNorm, and only retains RMS Scaling normalization at level; SwiGLU/GEGLU and Gated FFN make the feedforward layer not only "linear-activation-linear", but information screening with gated information.
The reading of such papers is not to ask "how many points to improve", but ask: whether the changes are partial, whether it is easy to insert existing blocks, how to calculate the overhead, and whether it is stable on multiple scales. When postgraduate reappearance, you can choose a small model of 100M to 1B level, and compare the training loss of ReLU/GELU/SwiGLU, LayerNorm/RMSNorm, Absolute/RoPE Curves and downstream small tasks, without having to reproduce the large-model fraction.
4. MQA, GQA, FlashAttention: Reading from training architecture to inference bottlenecks
Self-regressive generation is not a one-time parallel output, but token-by-token decoding. Each time a token is generated, the model must read the key/value cache of the historical token. Multi-query Attention allows multiple query heads to share a set of key/values, reducing the KV cache read and write pressure; grouped-query attention is in MHA and MQA In the middle of the compromise, the quality is preserved with multiple KV groups; FlashAttention is not to change the mathematical definition, but to change the GPU calculation of the attention and the memory access method.
This group of papers trains you to distinguish between "mathematical architecture" and "system implementation". Some methods do not change the model function, but significantly change the trainable length, throughput and video memory usage. When reading the experimental table, don't just look at Perplexity, but also through throughput, memory, context length, hardware model and kernel support.
5. DeepSeek-V3, Qwen3, Llama 4: How to dismantle the modern technology report
The DeepSeek-V3 report describes an MoE with 671B total parameters and 37B active parameters. It uses MLA and DeepSeekMoE, with auxiliary-loss-free load balancing and multi-token prediction. The official Qwen3 repository lists dense and MoE models in several sizes, including 30B-A3B and 235B-A22B, with updates in 2025 to long-context support and thinking and non-thinking modes. Meta’s Llama 4 model card identifies Scout and Maverick as autoregressive, early-fusion multimodal MoE models released on 2025-04-05.
Be more restrained when reading these reports: don't use the Benchmark table of the open model directly as an architectural conclusion. They usually change the data, tokenizer, training formula, RL/Post-Training, Serving Kernel and evaluation collections at the same time. The correct practice is to split the report into "architectural statement", "training statement", "data declaration", "evaluation declaration" and "restricted declaration", and find reproducible evidence for each statement.
Methods and experiments compared
| Papers/Data | What to read | Structure Change Locations | Reproduce Minimum Experiment | Main risk |
|---|---|---|---|---|
| Attention Is All You Need | multi-head attention、encoder-decoder、positional encoding | attention and overall block | Small machine translation or copy tasks, redraw tensor shapes | Only memorize the formula, do not understand the parallelization motivation |
| BERT | Masked LM with bidirectional encoder | objective + encoder-only | Small corpus MLM pre-training + classification FINE-TUNING | Generalize the understanding task conclusion to the generative model |
| T5 | text-to-text unified framework and ablation | objective + transfer setup | Run Category/Summary Small Tasks in the same model format | Ignore data cleaning and task templates |
| GPT-3 | decoder-only + few-shot scaling | scale + prompting interface | In-Context learning probe for small decoder LM | Misjudging the scale effect as a single architecture innovation |
| RoPE / RMSNorm / SwiGLU | Local block replacement | position、norm、FFN | Comparison of training curves for small models of the same scale | Just look at the final score, not stability |
| MQA / GQA / FlashAttention | KV cache, throughput, video memory | Attention and Kernel | Fixed Model Compare Prefill/Decode Latency | Complexity and actual hardware efficiency of obfuscation algorithm |
| DeepSeek-V3 / Qwen3 / Llama 4 | MoE, long context, multimodal / inference form | system-level architecture | Reproduce sub-module or configuration analysis, without reproducing full training | Treat the technical report as a univariate experiment |
How to reproduce the results
If you are a graduate student, it is not recommended to start with the "Replicating the Llama Level Model". A more reasonable route is to do three small experiments.
Experiment 1: Reconstruct the minimum Transformer.
Write a decoder-only LM in PyTorch and train on TinyStories, WikiText-2 or self-built small corpus. The goal is not a high score, but to verify that the implementation of mask, resume, norm, FFN, and position is correct. The config, seed, loss curve and sample generation must be saved.
Experiment 2: Local replacement of ablation.
With a fixed training budget, choose two or three comparisons from the following: absolute position encoding versus RoPE, LayerNorm versus RMSNorm, a GELU versus a SwiGLU feed-forward network, or MHA versus GQA. Change one variable at a time. Report whether the parameter count changed, whether the runs used the same number of training tokens, and whether the learning rate was retuned.
Experiment 3: Inference side measurement.
Fixed an open source small model, measure the size of prefill latency, decode Latency, video memory, and kv cache, and compare the lengths of different contexts. Don't pursue the essay-level Benchmark here, but train yourself to understand "why modern architecture papers are always talking about KV cache, GQA, FlashAttention, Serving Engine".
The reproduction record includes at least:
- Code commit or compressed package hash.
- Python, PyTorch, CUDA, transformers/vLLM/llama.cpp versions.
- Dataset name, version, cleaning rules and number of samples.
- Model config: number of layers, hidden size, number of heads, number of kv heads, FFN hidden size, position type, norm type.
- Training budget: token number, batch size, learning rate, warmup, precision.
- Logs: loss, throughput, video memory peak, abnormal sample.
- Negative results: instability, loss spike, OOM, and abnormal speed must be written.
Common misconceptions
Myth 1: Equivalent Transformer to Attention.
Attention is the core, but the performance of modern LLM also depends on FFN capacity, normalization, location coding, training objectives, data and scale. Read-only Attention formulas cannot read LLM technical reports.
Myth 2: Treat decoder-only as "more advanced".
Encoder-only, encoder-decoder, decoder-only are task and training target choices, not linear evolution. BERT is still suitable for learning; T5 is still suitable for unified task modeling; decoder-only is just a mainstream choice for modern generic generative models.
Misunderstanding 3: When you see MoE, you think that the calculation must be cheap.
MoE reduces activation parameters per token, but introduces routing, load balancing, communication and deployment complexity. The Total Params, Activated Params, the number of experts, the top-K, and the sharing experts in the paper are all recorded separately.
Myth 4: Ignore hardware and Kernel.
Whether long contexts are available or not depends on FlashAttention, Paged KV cache, Tensor Parallel, GPU Memory bandwidth, and batch scheduled. If the architecture does not understand the system, the reproduction is easy to fail.
Myth 5: Copy the Benchmark table as a conclusion.
The evaluation table of technical reports is usually mixed with models, data, post-training, prompt formats and evaluation implementations. Scientific research and training should pursue "differentials that can be explained", not heap rankings.
Research questions for graduate students
- Research on the Position Encoding Extrapolation on Small Models: Fixed 100m to 300m decoder-only models, and compare the stability of RoPE, ALiBi, and RoPE scaling in short training length measurements.
- Mass-speed fold curve of GQA: Fixed parameters and training data, the system changes the number of kV heads, and measures perplexity, decode Latency and video memory.
- Influence of FFN gating on reasoning tasks: Compare the differences in math, code, and small corpus language modeling of GELU, GEGLU, and SwiGLU, and focus on the analysis of training stability.
- MoE Routing Experimental Little Experiment: Observe whether different token types, languages, or tasks tend to be different in small MoE LM.
- Technical Report Statement Verification Tool: Write a script to parse Hugging Face Config, Model Card and paper tables, automatically generate a "Schema Declaration List", and mark the field for manual verification.
- Training objectives and architecture interaction: Compare the representation and generation differences of the same Transformer block under Causal LM, Span Corruption, and Prefix LM.
Summary
The core of the Transformer intensive reading route is not to "read a masterpiece", but to establish an architectural coordinate system that can be reused continuously. Read the original Transformer in 2017, understand the basic division of attention, FFN, position, resume, and Norm; read again BERT, T5, GPT-3, understand how the training goal changes the purpose of the model; then read RoPE, RMSNorm, SwiGLU, MQA/GQA, FlashAttention, and understand the local evolution of modern blocks; finally read Technical reports such as DeepSeek-V3, Qwen3, and Llama 4, train yourself to dismantle the system-level model back into the verifiable research problems.
The real scientific research training takes place in recurrence and ablation: whether you can fix variables, re-run small experiments, explain negative results, and ask the next step from a local change. If you can do this, Transformer will no longer be a classic architecture diagram, but will become a toolbox for you to enter LLM research.
References
Search date: 2026-08-01. The following priority lists papers, official project pages, official model cards or conference pages; model cards, repository Release, Benchmark numbers and other variable information are given priority, and it is recommended to verify again before submitting or publishing.
- Vaswani et al., 2017, Attention Is All You Need, arXiv: https://arxiv.org/abs/1706.03762
- Devlin et al., 2018/2019, BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, arXiv: https://arxiv.org/abs/1810.04805
- Raffel et al., 2019/2023, Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer, arXiv: https://arxiv.org/abs/1910.10683
- Brown et al., 2020, Language Models are Few-Shot Learners, NeurIPS: https://papers.neurips.cc/paper/2020/hash/1457c0d6bfcb4967418bfb8ac142f64a-Abstract.html
- Kaplan et al., 2020, Scaling Laws for Neural Language Models, arXiv: https://arxiv.org/abs/2001.08361
- Shazeer, 2020, GLU Variants Improve Transformer, arXiv: https://arxiv.org/abs/2002.05202
- Zhang and Sennrich, 2019, Root Mean Square Layer Normalization, NeurIPS: https://papers.nips.cc/paper_files/paper/2019/hash/1e8a19426224ca89e83cef47f1e7f53b-Abstract.html
- Su et al., 2021, RoFormer: Enhanced Transformer with Rotary Position Embedding, arXiv: https://arxiv.org/abs/2104.09864
- Shazeer, 2019, Fast Transformer Decoding: One Write-Head is All You Need, arXiv: https://arxiv.org/abs/1911.02150
- Ainslie et al., 2023, GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints, ACL Anthology / EMNLP 2023: https://aclanthology.org/2023.emnlp-main.298/
- Dao, 2023, FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning, arXiv: https://arxiv.org/abs/2307.08691
- Chowdhery et al., 2022, PaLM: Scaling Language Modeling with Pathways, arXiv: https://arxiv.org/abs/2204.02311
- Touvron et al., 2023, LLaMA: Open and Efficient Foundation Language Models, arXiv: https://arxiv.org/abs/2302.13971
- Grattafiori et al., 2024, The Llama 3 Herd of Models, arXiv: https://arxiv.org/abs/2407.21783
- DeepSeek-AI, 2024/2025, DeepSeek-V3 Technical Report, arXiv: https://arxiv.org/abs/2412.19437
- Qwen Team, 2025, Qwen3 official repository and technical report links: https://github.com/QwenLM/Qwen3
- Qwen Team, 2025, Qwen3: Think Deeper, Act Faster, official blog: https://qwenlm.github.io/blog/qwen3/
- Meta, 2025, Llama 4 model card: https://github.com/meta-llama/llama-models/blob/main/models/llama4/MODEL_CARD.md
- Meta, 2025, The Llama 4 herd, official blog: https://ai.meta.com/blog/llama-4-multimodal-intelligence/