# MoE routing mechanism experiment

This is a synthetic, scalar-output top-1 MoE, not Switch language-model training.
Python 3.12 and NumPy 2.3.5; CPU only, float64. No downloads or credentials required
after NumPy installation. Actual run used Python3.12.14/NumPy2.3.5 on macOS arm64.

Run from this directory:

```bash
python -m pip install -r requirements.txt
python -m py_compile minimal_moe.py audit_results.py
python minimal_moe.py --check-only
python minimal_moe.py
python audit_results.py
```

`--output OTHER_DIRECTORY` writes a fresh result set without replacing the included
evidence. The audit script reads the included `results/` beside itself. If the OS
blocks bytecode caches, set `PYTHONPYCACHEPREFIX=/private/tmp/codex_pycache_ai_070`.

## Model and protocol

- X=[64,4]; first three features iid N(0,1), final feature=1 (intercept).
- Group=(x0>0)+2*(x1>0). Target=0.25*x0+slope[group]*x2+offset[group].
  Slopes=[1,-1,2,-2], offsets=0.4*[1,-1,-1,1]. Group is never given to router.
- Router W=[4,4], experts V=[4,4], total32 trainable parameters.
- Selected expert=argmax(softmax(XW)), ties use first index. Capacity per batch
  C=ceil(1.25*64/4)=20; tokens admitted in their current order; no rerouting.
- Only accepted rows evaluate their selected linear expert. Output includes its
  soft gate multiplier plus fixed residual0.25*x0. Dropped rows still have MSE loss.
- Gradients hold hard routing and capacity fixed locally. Load-balance uses raw
  pre-capacity counts, not the accepted mask. No straight-through argmax gradient.
- Seed7000 generates disjoint train256/dev128/test256. Training seeds70/71/72.
  Router init N(0,.03), then W[3,0]+=2; expert init N(0,.1).
- Three arms: task(alpha=0,beta=0), balance(.1,0), balance_z(.1,.001).
  Within a seed all arms share initialization and minibatch stream.
- Adam LR.03, beta1.9, beta2.999, epsilon1e-8,600 steps,batch64;
  sampling without replacement within a step, with reuse across steps.
- Fixed final checkpoint; dev is monitored every100 steps, no selection or tuning.
- Evaluation uses fixed disjoint64-row routing groups. Average scalar batch metrics;
  sums of counts are also saved. max_share/CV are not computed after global merging.
- Capacity factors .5,1,1.25,2,4 are evaluation-only interventions on saved weights.
  They change admitted compute; these are not equal-budget speed comparisons.

## Saved evidence

- `smoke_test.txt`: actual console output, checks, tensor shapes and aggregate values.
- `audit_test.txt`: independent scalar replay outcome.
- `results/data.json`: actual inputs, targets and groups for all splits.
- `results/*_weights.json`: nine final parameter sets.
- `results/task_70.json` etc.: nine runs, seven train/dev checkpoint summaries,
  test metrics, five capacity sweeps, full test probabilities, selections,
  admission masks and predictions.
- `results/summary.json`: environment, checks, means/sample SD across three seeds.
- `results/routing_heatmap.html`: standalone numeric HTML, nine expandable tables,
  first test batch64 each. Opacity=p on a shared0..1 scale; displayed four-decimal
  probabilities; exact float values remain in JSON. No external assets/scripts.
  This HTML is supplementary evidence, not one of the two imagegen PNGs.

`--check-only` checks all32 analytic gradients by finite differences away from
route/capacity boundaries; uniform tie routing; capacity clipping; dropped-index
corruption; score-shift invariance; unused-expert zero gradient; residual handling.
`audit_results.py` independently recomputes all9*5*256=11520 predictions via scalar
loops, capacity counters and saved weights; checks the complete test traces,
summary mean/SD, and disjoint data splits.

## Interpretation boundaries

No padding, attention, neural language data, FFN hidden layer, learned token
embedding, low precision, distributed dispatch, GPU memory or throughput measure.
The regression auxiliary weight .1 is a toy setting, not the paper coefficient.
The observed three-seed SD is not a confidence interval. Underuse in one small
window is not proof of permanent collapse; low z-loss is not proof of load balance.
Real-model, original framework and paper benchmark reproduction: 待人工核验.