Research Device for Expert Routing and Congestion Diagnostics

Search and experiment date: 2026-09-11. This paper provides a runnable CPU mechanism experiment to distinguish the paper report, this actual verification and author’s interpretation.

Abstract

When the auxiliary loss falls during training, does that mean the experts are working as intended? Starting from Switch Transformer’s single-expert routing, we use a 32-parameter mixture-of-experts model to examine routing probabilities, discrete choices, capacity limits, and task error separately. The experiment gives a counterexample in which the loss is one even though every input selects the same expert. It also records nine training runs. Together these checks show how to interpret a routing heatmap.

Contents

  1. Reproducing Value and Evidence Boundaries
  2. Core ideas and formulas
  3. A guide to the official implementation
  4. A minimal experiment and its results
  5. Evaluation protocol
  6. Failure investigation
  7. Questions for further research
  8. Summary and References

Recover value: first know what you are counting

MoE, the Mixture of Experts, let the input call only some expert parameters. The "expert" here is a set of trainable functions that are not automatically equal to knowledge modules in mathematics, programming or a certain language. Router (Router) scores Experts based on the input; top-1 means that only the highest score is selected for each Token. Token is a synthetic feature in this experiment, not a natural language tokens.

Paper report: Switch Transformers of Fedus, Zoph, and Shazeer was published in JMLR 2022, to expand the sparse model with a single expert route, and discuss the stability of capacity, communication and training. The pre-training of the paper depends on its model and calculation conditions, and cannot directly deduce the local speed from the number of activated experts. Paper page

The actual verification this time: The training is a synthetic regression model with single-layer and scalar output; there is no pre-training weight, language task or distributed communication. The author's inference: For newcomers to scientific research, first reproducing "choose whether it is correct, how to count the overflow, and where the gradient flows", it is easier to obtain credible evidence than the first replication of the large model parameters.

Core Ideas and Formulas: The Three Loads Can't Confuse

make input X∈RT×dX\in\mathbb{R}^{T\times d}, the routing parameter is W∈Rd×EW\in\mathbb{R}^{d\times E}, then:

Z=XW,pti=softmax⁡(Zt)i,at=arg⁡max⁡ipti.Z=XW,\quad p_{ti}=\operatorname{softmax}(Z_t)_i,\quad a_t=\arg\max_i p_{ti}.

TT is the number of tokens in the routing group, dd is the input dimension, EE is the number of experts. ZZ is the pre-normalized score, pp shape of [T,E], each line sum is one; aa is the length of TT integer vector. In this experiment, 64, 4, and 4 are taken respectively, and the last column is always as one, so that the router can learn to bias.

The selection ratio before the capacity limit is counted, and then the average probability is counted:

fi=1T∑t1[at=i],Pi=1T∑tpti,Lbal=E∑ifiPi.f_i=\frac{1}{T}\sum_t\mathbf{1}[a_t=i],\quad P_i=\frac{1}{T}\sum_t p_{ti},\quad L_{bal}=E\sum_i f_iP_i.

ff is the hard selection frequency, PP is the soft probability mass, both lengths EE vector. backward propagation through PP, not through discrete ff. this article coefficients α\alpha Put it on the outer layer of the total loss; the original formula four is written into the auxiliary item, and it is necessary to confirm whether it has been multiplied when comparing the log. Switch body formula 4 to 6

When both vectors are uniform, the unweighted Lbal=1L_{bal}=1; But the reverse does not hold. Set the sixty-four lines of probability as (0.25,0.25,0.25,0.25), the parallel rules of the code are all selected to zero, so f=(1,0,0,0)f=(1,0,0,0), the loss is still one. This is a counterexample obtained and actually executed by substituting the formula, and is not a hypothesis for the final state of the training. NumPy explicitly specifies that the juxtaposed maximum returns the first occurrence. argmax documentation

Capacity C=⌈cT/E⌉C=\lceil cT/E\rceil Limit how many tokens each expert actually receives, cc It is the capacity coefficient, and the upward rounding is the realization convention of this paper. Set up mtm_t Indicates whether it has been received, the output of this experiment is:

y^t=0.25Xt0+mtpt,atvat⊤Xt.\hat y_t=0.25X_{t0}+m_t p_{t,a_t}v_{a_t}^{\top}X_t.

vi∈Rdv_i\in\mathbb{R}^{d} is the ii -th linear expert. The first term is the fixed residual path. When capacity overflows, only the expert branch is skipped. The example still contributes to task error. After capacity truncation, the sum of the accepted counts may fall below TT, you cannot pretend to be the previous selection frequency.

ST-MoE also proposed Router Z-loss, this article is written as Lz=T−1∑t(log⁡∑ieZti)2L_z=T^{-1}\sum_t(\log\sum_i e^{Z_{ti}})^2. The variable is still a routing score matrix; it penalizes the logarithmic constant, and the paper uses it to study numerical stability. It is not another way of writing "expert balance", nor is it equivalent to square penalties for each score item by item. ST-MoE full text Section 3

Guide to the official implementation: Trace Mask Life Cycle

Read the Mesh TensorFlow pointed to by the original paper first moe.py, from _switch_gating Positioning probability, single expert selection, density_1、density_1_proxy and cumulative position. The auxiliary load item is calculated first, and then the mask is modified according to the capacity. Its main branch is a snapshot of this visit, and the submission is not fixed; when the experiment needs to be reproduced bit by bit, the submission number should be saved. Author code

Read and fix v4.57.1 Label Transformers implementation, which is the official framework porting, not the original training program for the author. Along SwitchTransformersTop1Router.forward Examine argmax、cumsum and capacity masks; here, the sequence position is accumulated, and there is no class annotation that the class annotation is said to sort by probability. When reading the source code, the actual operation should be checked, not just the comments.

follow up SwitchTransformersSparseMLP.forward: it is truncated router_mask Take again argmax, all zero overflow rows will get zero index. If this return value is taken directly as the original selection, the log may be mixed into the error count. This article checks this information loss with the structure of "originally selected No. 2"; this is not a claim to run a complete framework. Also pay attention _unpack_router_logits It will be recorded along the sequence dimension of the splicing layer, and the data should be fetched before the splicing is diagnosed. Fixed version source code

Record original selection, capacity reception and overflow separately

The figure is a schematic diagram of the diagnosis setting, and the beads and column height do not correspond to the actual measured samples. The original selection is read out above, and the load and overflow before and after the cutoff are recorded respectively; the same color represents the same expert. The truncated column can only be shortened, and there is no claim that the expert has been trained to become balanced. The real probability thermal gauge is separately output by the script.

Minimum Experiment: Counterexample, Training and Real Thermal Meter

enter this article code/, using Python 3.12 to execute:

python -m pip install -r requirements.txt
python minimal_moe.py --check-only
python minimal_moe.py
python audit_results.py

Dependencies are fixed to NumPy 2.3.5. The actual operating environment is Python 3.12.14, CPU, and double precision. The data seeds are 7,000, and there are 256, 128, and 256 lines for training, verification, and testing, and the set does not overlap with each other. The first three columns are independently sampled to the standard normal; the first two columns are divided into four areas, and the area determines the slope and intercept of the third column. Region labels only generate targets and are not supervised as routes.

The routing matrix and four experts have a total of thirty-two parameters. Each training seed uses the same initial value and sampling sequence to compare three losses: task mean square error, add 0.1Lbal0.1L_{bal}, add 0.001Lz0.001L_z. Initialization adds two to the zero route offset, and deliberately creates crowding; Adam's learning rate is 0.03, and each step is not put back and draws 64 lines, a total of 600 steps, and the training capacity coefficient is 125. The three seeds are seventy to seventy-two, and no group is deleted due to the quality of the results.

The actual verification of this time: In the perfectly uniform probability counter-example, all 64 items are selected with all zeros, the capacity is 20, and only 20 items are received, and the overflow rate is 68.75%. Another construction puts the original load [28,12,12,12] As of eight of each expert, get [8,8,8,8]. The uniform receiving histogram hides the original crowding. After adding 100 to the full score, the probability and selection remain the same, but the Z-loss changes, which further shows that it is not a load indicator.

The test results of the fixed endpoint of the nine groups of training are as follows; each item is the mean and sample standard deviation of the three seeds, and the balance loss and Z-loss are not multiplied by weight:

MSE is the square average of the difference between the predicted value and the real value, the smaller the better; the overflow ratio is all valid inputs as the denominator. The two answer different questions: a measure task error, a measure how often the specialist branch is skipped. The experiment does not count the running speed, so there is no increase in throughput according to this report.

training target Test MSE overflow ratio balanced loss z-loss
task only 0.982 ± 0.166 0.154 ± 0.065 1.295 ± 0.144 41.59 ± 8.18
Add balance term 0.933 ± 0.229 0.133 ± 0.101 1.298 ± 0.170 37.83 ± 5.74
Add Z item 0.978 ± 0.048 0.118 ± 0.030 1.213 ± 0.041 21.09 ± 2.81

This group of observations does not support "adding equilibrium items will inevitably reduce the equilibrium loss". The seed 71 of the balanced group is selected as [32,2,116,106], No. 2 and No. 3 are crowded, and No. 1 is almost useless; but the low frequency use in the limited window cannot be directly called permanent collapse. After adding Z items, the mean value is reduced, and it does not prove that the low-precision stability benefits of the large model are reproduced: this kind of hardware value failure did not appear or tested this time.

Open Actual Routing Thermal Meter, you can expand nine groups of runs, and check the four probability, original selection and whether to receive the first fixed test batch line by line. The color depth is equal to the probability, and the number retains four decimal places; the thermal meter only displays sixty-four rows, and the summary table uses all test rows. Full accuracy probability, prediction, weight and training curve are all in results/, color cannot replace these data.

Evaluation protocol: Simultaneous reporting of tasks, allocations and budgets

This article pre-fixes the 600-step end point; the Validation Set only records the curve, not to select the checkpoint, and the Test Set is not used to adjust the coefficient. Each sixty-four lines independently implements capacity limits, and then averages approval indicators. In particular, the maximum selection ratio and coefficient of variation should be calculated batch by batch first; first summarize the full test set and then calculate, which may cover up the local congestion. The standard deviation only describes initialization and sampling changes, not Confidence interval .

The original count, reception count, zero load and average token entropy are also recorded separately for each expert. High entropy represents a single input route indefinite; it may appear at the same time as a set of hard selections caused by a fixed parallel. This article regards collapse as a persistent imbalance to be diagnosed, and the death of an expert is declared without a general threshold.

After saving the weights, five additional test capacity was scanned this time, and the input and order were kept unchanged. The coefficient of the equilibrium group seed 71 was changed from 1.25 to 4, the overflow decreased from 24.22% to zero, and the MSE decreased from 1.1139 to 0.5358. This isolates the sensitivity of the model to capacity, and the cost is to allow more experts to calculate; it cannot be improved as a method of the same budget, and it is not equivalent to retraining.

Failure investigation: first rule out observation errors

When all the same experts are selected, first check the score juxtaposition, the initial offset, the effective token mask and the data sequence, and then adjust the auxiliary coefficients. There is no padding in this article; when migrated to a language batch, the fill position must be excluded from the denominator and count, and the valid token sets of the different methods are consistent.

In the event of an equilibrium curve falling but the task deteriorates, the unweighted loss and weighted contribution should be seen at the same time. The numerical scale of the regression error is different from that of the language cross-entropy, and the coefficients of the paper cannot be directly copied. The tighter the capacity, the more likely it is to skip experts at the back of the sequence; rearrange token can change the affected samples, so the evaluation must hold the in-batch order.

The code passes the finite difference of the center of all parameters, and the maximum absolute error 2.79×10⁻¹⁰, the discrete selection before and after the disturbance is consistent with the capacity mask; this validates the local gradient, does not claim that argmax is guided. Independent scalar audits were evaluated from 45 groups of capacity evaluations and 11,520 predictions from the preservation weights. Official framework operation, language model training, expert parallelism, low-precision and paper benchmarks are all Manual verification.

Questions for further research

First, do experts really divide labor? After statistical routing is statistically linked on fixed data packets, the experts or shield experts are exchanged to observe the changes of the task. Semantic feats cannot be launched only with a certain color set; route differences may also be driven by location or word frequency.

Second, do the errors caused by capacity focus on some samples? The area and position slices can be pre-defined, the errors of overflow samples and the received samples can be compared, and then the sequence disturbances under a fixed budget are done; it is necessary to keep the paired samples to avoid using different test sets to interpret the mechanism.

Third, is there a conflict between local equilibrium and global equilibrium? The same batch of data is divided into different routing groups, and the group-by-group statistics and merged statistics are compared. Before entering the multi-machine experiment, this small change can expose the problem of statistical caliber; whether to improve the communication efficiency must be measured separately.

Summary

To understand MoE routing, connect the router’s preferred expert for each input with the traffic actually delivered and the resulting task performance. The counterexample and negative results show why no single auxiliary loss can guarantee that the experts work well. The reusable artifacts are the recorded routing choices, masks, weights, and evaluation protocol. Preserve the ability to replay and check them when moving to a real model.

References

Search date: 2026-09-11. The following links have been actually opened; the main text explains the arguments supported by each.

  1. William Fedus、Barret Zoph、Noam Shazeer, Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity,JMLR 23(120),2022. Publishing page; Publish the full text.
  2. Barret Zoph and other eight authors, ST-MoE: Designing Stable and Transferable Sparse Expert Models, 2022, arXiv v2, revised on 2022-04-29. Version history; Full text.
  3. TensorFlow Mesh Contributor, the original code of the Switch routing, the master when accessing, and the commit is not fixed. moe.py.
  4. Hugging Face Contributor, Transformers v4.57.1, fixed label source code. Switch implementation.
  5. NumPy contributor, NumPy 2.3 version documentation. argmax parallel rules.