Cover illustration

Date: 2026-08-24
Suitable for readers: graduate students, new researcher, engineering readers who want to really understand the Transformer code
Search date: 2026-08-24

Abstract

Many people come into contact with Transformer for the first time, from nn.TransformerEncoderLayer, BERT or Hugging Face model starts: a few lines of code can get output, but it is not easy to explain why each dimension tensor changes, who is blocking the padding mask, and the residual is connected to LayerNorm Why does the order affect training stability?

This article does not reproduce the complete results of the original paper on WMT machine translation, but achieves a smaller and more suitable goal for scientific research: only use PyTorch's basic tensor operations and nn.Linear, to implement a readable Transformer Encoder, including sine position coding, multi-head self-attention, feed-forward network, residual connection, LayerNorm, padding mask, parameter initialization and minimum test. The matching script also designs a synthesis task of "judging whether the token at the beginning and end of the sequence is the same", which is used to check whether the model can establish a cross-position relationship.

Evidence boundaries need to be explained first: the PyTorch of the current workspace is not installed with PyTorch, so the script attached to this article has passed the syntax check, but the CPU training and numerical results have not actually run, and the relevant output should be manually verified after the installation dependence. This article does not write the expected phenomenon as a measured conclusion.

Contents

  1. Why should you reproduce the Encoder from zero?
  2. What does Encoder in the original paper contain?
  3. From formula to tensor shape
  4. How to read the minimum PyTorch implementation
  5. Minimum Experiment and Acceptance Criteria
  6. Comparison with PyTorch official implementation
  7. The easiest reproduction trap
  8. Questions for further study
  9. Summary and References

Why should you reproduce the Encoder from zero?

Between "Transformer" and "Understanding Transformer", the difference is often not more formulas, but the ability to align formulas, tensors and code item by item. When postgraduate students do the dissertation, it is usually not written out that the real time is spent. model(x) and instead answer these details: the input is [batch, length, hidden] Still is [length, batch, hidden]? in the mask True Indicates reserved or shielded? Why do you need to split the bulls? transpose? Is the residual branch plus the expression before normalization or after normalization? Why is the output shape correct, but lost has always been nan?

Realizing the value of an Encoder from scratch is to establish a checkable "structural ledger". Each module should have input shapes, output shapes, mathematical meanings and invariants. In this way, when reading the BERT, Vision Transformer, CLIP, or multimodal model, what you see is no longer a set of unfamiliar class names, but a variant of the same basic structure in position coding, normalization, activation function and attention pattern.

This sets the scope of the reproduction. The original Transformer is an encoder–decoder machine translation system. Its WMT results depend on data processing, the training budget, the learning rate schedule, label smoothing, beam search, and checkpoint averaging. A small local script cannot establish those results. Reproducing the encoder’s computation graph and checking its key invariants gives us a verifiable first step toward a larger reproduction.

What does Encoder in the original paper contain?

2017 Attention is All You Need uses Self-Attention to replace the cyclic Network and convolution, so that All positions in a sequence can be calculated in parallel. The Encoder of the original paper is stacked with n layers of the same structure, and each layer contains two sublayers: Multi-Head Self-Attention and Position-wise Feed-Forward Network. There is a residual connection with LayerNorm outside each sublayer.

For the input token sequence, check embedding at the bottom, and then multiply sqrt(d_model), and then add the position code of the same dimension. Position coding is necessary, because the self-attention itself only calculates the relationship between tokens according to the content; if two tokens are exchanged and the corresponding Q, K, and V are exchanged at the same time, pure attention will not automatically know that the order has changed.

The original paper uses sine and cosine functions to construct fixed position codes:

PE(pos,2i)=sin⁡(pos/100002i/dmodel)PE(pos, 2i)=\sin\left(pos/10000^{2i/d_{model}}\right)
PE(pos,2i+1)=cos⁡(pos/100002i/dmodel)PE(pos, 2i+1)=\cos\left(pos/10000^{2i/d_{model}}\right)

Even dimensions and odd-numbered dimensions use different phases, and different dimensions correspond to different frequencies. The most important thing in realization is not to memorize the formula, but to confirm PE shape can be broadcast to [B,T,d_model], and follow the model mobile device as a buffer, but do not participate in gradient updates.

The Encoder output is still a sequence representation of length t, not a classification result. Each location has fused the full sequence information through self-attention. BERT later used this kind of two-way Encoder for pre-training, and then added output headers for tasks such as classification, Q&A, etc. This is also a typical example of "Encoder is not just a service for machine translation".

Transformer Encoder Calculation Process

From formula to tensor shape

1. Scaled Dot-Product Attention

The core formula for attention is:

Attention(Q,K,V)=softmax(QKTdk+M)VAttention(Q,K,V)=softmax\left(\frac{QK^T}{\sqrt{d_k}}+M\right)V

If you enter x For [B,T,d_model], first get q, k, v through linear mapping. After splitting into H heads, the shapes of the three are changed into [B,h,T,d_head], in d_head=d_model/h. Therefore d_model must be divisible by the header.

Q @ K.transpose(-2,-1) Get [B,h,T,T]. The previous t means "which Query is asking a question", and the latter t means "what keys it can see". Divide by sqrt(d_head) It is to control the variance of dot product; when the dimension increases, the unscaled score can easily enter the saturation area of the softmax, and the gradient will become unfriendly.

Mask should act on Score before softmax. to padding mask, True It usually means that the corresponding Key is the fill position, and the column score needs to be set to a minimum value. It is especially easy to write backwards: the shield is the Key dimension, that is, [B,1,1,T] Broadcast to all heads and all Query, instead of clearing the output position. PyTorch official MultiheadAttention The document also stipulates that the binary value key_padding_mask Middle True Indicates that the Key is ignored.

2. The meaning of Multi-Head

Single-head attention is only calculated in one projection space. Long attention allows different heads to have independent Q, K, and V subspaces, and then spell back the results of each head [B,T,d_model], and finally through the output projection W^O mix.

MultiHead(Q,K,V)=Concat(head1,…,headh)WOMultiHead(Q,K,V)=Concat(head_1,\ldots,head_h)W^O

This does not mean that every head must learn the linguistic relationship that human can be named. Attention weights can be used for diagnostics, but "some heads to a certain token" cannot be directly equated with causal interpretation. The Attention of the next article will specifically discuss this boundary.

3. Position-wise FFN

The feed-forward network acts independently for each location:

FFN(x)=max⁡(0,xW1+b1)W2+b2FFN(x)=\max(0,xW_1+b_1)W_2+b_2

It puts first d_model expand to d_ff, after ReLU, project back d_model. Attention is responsible for mixing across locations, and FFN is responsible for nonlinear feature transformations for each position. The division of labor between the two is different, and FFN cannot be deleted just because "attention is all you need".

4. Residuals and LayerNorm

The original paper is LayerNorm(x + Sublayer(x)), that is, Post-LN. PyTorch TransformerEncoderLayer Default norm_first=False, also correspond to the sublayer and the residual first, and then normalize; set norm_first=True it becomes Pre-LN. Subsequent studies have shown that the two sequences alter the gradient behavior and warm-up requirements of the deep Transformer. Therefore, the normalization sequence must be regarded as an experimental variable when reappearing, and it is not only possible to see whether the modules exist.

How to read the minimum PyTorch implementation

The companion script is in code/minimal_transformer_encoder.py. It is recommended not to main() Start, and read in the following path.

First step MultiHeadSelfAttention.forward(). Focus tracking QKV merge projection, view、permute, Score matrix, Mask broadcast, softmax, each head splicing these seven actions. As long as one of the dimensions are interchanged incorrectly, the code may still run, but the semantics of batch, head or sequence are mixed together.

Second step EncoderLayer.forward(). The script keeps both Post-LN and Pre-LN paths at the same time, using the original Post-LN by default. Here it should be clear that the residual branch x Which stage to point to avoid inadvertently put norm(x) Added to unnormalized residuals.

The third step TransformerEncoder.forward(). It does embedding scaling, position coding, and n-layer stacking. Used here ModuleList Create independent layers instead of repeatedly referencing the same Layer object; otherwise, it seems that n layers are stacked, but all parameters are actually shared.

Step 4 run_invariant_checks(). It does not check "the model is smart or not", but only checks the more basic and reliable facts: whether the output is [2,5,64], whether each layer of attention is [2,4,5,5], whether the sum of each line after softmax is 1, and whether the Key probability of the padding mask is 0 is 0. Reproducing items should pass these invariants first, and then talk about the training curve.

Minimum Experiment and Acceptance Criteria

After installing the dependencies, you can only run the structure check first:

python3 -m pip install -r code/requirements.txt
python3 code/minimal_transformer_encoder.py --check-only

Run the synthesis task again:

python3 code/minimal_transformer_encoder.py --steps 80 --seed 7

The task generates random sequences of 12 tokens, with equal numbers of positive and negative examples. Positive sequences start and end with the same token. Negative sequences always start and end with different tokens. The classification head reads only the encoder representation at the first position. To use the last token, the encoder must carry information back to the first position through interactions within its layers. This is an inexpensive end-to-end connectivity check. It is neither a benchmark nor evidence that the implementation matches the original training system.

It is recommended to divide the acceptance into four layers.

level Inspection content Prioritize checking failures
Structure All tensor shapes are the same as the number of heads view、permute、d_model % n_heads
Numerical value loss, gradient and attention none nan/inf Mask, learning rate, initialization, full padding line
Learning Loss has a downtrend after fixed SEED Label structure, gradient update, dropout, whether tasks can be learned
Contrast Consistent with the output shape and mask semantics of the official layer in the same configuration batch_first, Pre/Post-LN, activation, BIAS, dropout

The actual verification result this time is: the script has been compiled through Python syntax; the current environment is due to lack of torch, does not perform structural assertions and training loops. After installing dependencies, you should first run --check-only, and then observe the toy loss, instead of directly referencing an unrun accuracy .

Comparison with PyTorch official implementation

PyTorch official TransformerEncoderLayer Clearly composed of self-attention and feed-forward network, and provide d_model、nhead、dim_feedforward、dropout、activation、batch_first and norm_first and other parameters. Compare this article with the official layer item by item, and you can get a most practical reading table.

Module of this paper PyTorch official correspondence Semantics that need to be checked
MultiHeadSelfAttention nn.MultiheadAttention Enter the meaning of the layout, True in the mask, whether to return each head weight
EncoderLayer nn.TransformerEncoderLayer Post-LN/Pre-LN, activation function, dropout location
ModuleList[layers] nn.TransformerEncoder Number of layers, whether the parameters are independent, and the final Norm
handwritten attention formula scaled_dot_product_attention Numerical Stability, Fusion Kernel, Mask Types

The handwritten version is suitable for learning and inserting piles, not suitable for direct replacement of production implementation. PyTorch current tutorial is recommended to use scaled_dot_product_attention、Nested Tensor、torch.compile() and low-level components such as FlexAttention to build more flexible and efficient layers. The difference here is not "the formula has changed", but the official implementation also deals with Kernel scheduling, different dtypes, devices, sparse length, compilation and performance fast paths.

The easiest reproduction trap

First, forget about embedding scaling or location encoding. Small tasks may still fit, but this is no longer the input structure of the original paper, and the results cannot be regarded as the same implementation.

Second, the softmax dimension is wrong. Attention should be normalized along the last Key dimension. If softmax is done along the head or Query dimension, the output shape is still completely correct, but the error is very hidden.

Third, the Mask semantic confusion. Different libraries may use 1 Indicates that reservations may also be used True Indicates shielding. Do not convert from memory; read the current version of the document first, and then write a two-line sample to verify that the probability of the masked position is indeed 0.

Fourth, all keys are blocked. If a line of a sample is all the minimum value, softmax may have a meaningless distribution, and it may also appear under partial implementation or dtype nan. The data pipeline should ensure at least one valid token, or explicitly define empty sequence behavior.

Fifth, confuse Encoder self-attention and Decoder causal attention. Encoder usually allows the entire input to be viewed at each location, and the triangular Causal Mask should not be added by default. Copying Decoder's "Can't see the future" to the BERT style Encoder will quietly change the task definition.

Sixth, the layer is copied into parameter sharing. [layer] * N Just refer to the same Python object N times. If the paper does not explicitly share parameters, it should be instantiated independent modules for each layer.

Seventh, only the final loss is reported. A trusted reproduction should also record the PyTorch version, device, SEED, data generation method, parameter amount, training steps, optimizer and commit version. The success of Toy Task can only show that the realization has basic learning ability, and cannot extrapolate to translation quality, long-sequence ability or real corpus performance.

Questions for further study

  1. How do the gradient norms of Post-LN and Pre-LN, WARM-UP, and convergence speed change when the number of layers increases from 2 to 24?
  2. After fixed parameters, can increasing the number of heads always improve the synthesis relationship task? To face d_head What happens in too many hours?
  3. Sine position coding, learnable absolute position coding, and the difference between RoPE and ALiBi in the length extrapolation, how to use controllable data to verify?
  4. Handwritten attention and PyTorch scaled_dot_product_attention What is the difference between the numerical error and the speed under the CPU, CUDA, and different dtypes?
  5. Can padding mask, causal mask and any sparse mask be unified as a unit tested interface?
  6. Is there a causal relationship between attention weight and model prediction? Is the output change really larger after masking high weight connections?

These questions are all suitable for extension from this script. The value of the research is not to create a larger Transformer, but to make a variable, a hypothesis and a measurement protocol clear.

Summary

The first step to reproduce the Transformer Encoder is not to pursue the same BLEU as the original paper, but to establish the correspondence between formulas, tensors, code and tests: how to remove QKV headers, and why score divides by sqrt(d_head), in which dimension the mask takes effect, why do you still need FFN after paying attention, and how the order of residuals and LayerNorm become researchable variables.

The minimum implementation of this article is deliberately reserved readability, and constrains correctness with shape, probability and Mask invariants. It has done static syntax validation, but since there is no PyTorch in the current environment, the runtime results are still marked as artificial verification. Such a boundary statement is not a disadvantage, but the most important part of the reproduction record: only the verification that has been actually done is reported.

References

Search date: 2026-08-24. The PyTorch document will change with the version; the official reproduction experiment should be recorded locally at the same time torch.__version__, equipment and submission versions.

  1. Vaswani et al., Attention Is All You Need, 2017; arXiv The current page contains the 2023 revision.
  2. Harvard NLP, The Annotated Transformer, annotated PyTorch implementation with notebook.
  3. PyTorch, TransformerEncoderLayer, the current official API description.
  4. PyTorch, MultiheadAttention, enter the shape, key_padding_mask and attn_mask semantics.
  5. PyTorch Tutorials, Transformer building blocks, scaled_dot_product_attention, Nested Tensor and the current recommendations of the custom layer.
  6. Harvard NLP, the_annotated_transformer.py, location coding, model construction, initialization and training code path.
  7. Devlin et al., BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, 2018.
  8. Xiong et al., On Layer Normalization in the Transformer Architecture, ICML 2020.