Cover illustration

Date:2026-08-02
Who this is for: Graduate students, new researcher who is going to do a large model, readers with engineering background but want to systematically understand the model scale experiment
Prerequisites: Understand the cross entropy of language models, the amount of Transformer parameters, training token, FLOPs, log-log diagram and basic statistical fitting

Abstract

Scaling Laws is not a sentence "the bigger the model is, the better", but a set of research methods that use small-scale experiments to predict large-scale behavior. In 2020, Kaplan et al. systematized the power law relationship between the loss of the language model and the amount of data, and the amount of training calculation in 2020; the Chinchilla of DeepMind works in 2022 Rewrite the problem to "how to allocate parameters and tokens given the given Compute budget"; then data is limited, multimodal, multilingual, post-training, test-time compute, Emergent Abilities controversy and the Unified Scaling Law of 2026 continues to expand the framework.

The goal of this article is to give a Scaling Laws intensive reading route. Readers must learn three things: first, understand what variables the Scaling Law paper fits; second, to judge whether its extrapolation is credible;

Contents

  • Why Scaling Laws Worth Intensive Reading
  • Core reading method: read the variable first, then read the curve
  • A reading guide to the key papers
  • A closer look at key papers
  • Methods and experiments compared
  • How to reproduce the results
  • Common misconceptions
  • Research questions for graduate students
  • Summary
  • References

Why is this topic important

There is a realistic constraint on the study of large models: many key conclusions can only be judged before expensive training. It is impossible for a laboratory to train dozens of 100B-level models for trial and error at will, so it is necessary to estimate the trend at a larger scale with a smaller model, less token and lower computing power. Here is the value of Scaling Laws: it turns "empirical intuition" into a functional relationship that can be fitted, extrapolated and questionable.

But Scaling Law is also easily misread. The common horizontal axis in the paper may be non-embedding parameters, training tokens, training FLOPs, training steps, data repeating rounds, sampling ratio, test-time compute or Reward Optimization Budget; the vertical axis may be Validation loss, downstream accuracy, pass@k, reward score, human preference or security metric. As long as the variable definition changes, the conclusion of "what the scale brings" may be completely different.

Reading scaling papers is an exercise in research judgment. When authors call performance “predictable,” which metric can they predict? When they call a capability “emergent,” does the apparent jump come from the capability itself, the metric, the prompt, data contamination, the sampling method, or a threshold used for scoring?

Scaling Laws Reading Roadmap

Core reading method: read the variable first, then read the curve

When reading Scaling Laws essays, it is not recommended to start with a conclusion graph, but to make a "variable table" first:

  1. n: Model scale. Is the embedding parameter excluded? Does the MoE model write Total Parameters or Activated Parameters? Does the width, depth, and head number change with n?
  2. D: Training data. Is the unit of measurement token, sample, Episode or multimodal discrete token? Is the data removed? Is duplicate token treated as new data?
  3. C: Calculation amount. Is it estimating FLOPs, actual training time, GPU-days, or the sampling/search budget for the inference stage?
  4. L or M: indicators. Is it cross-entropy loss, perplexity, accuracy, reward, human preference, or discrete pass/fail indicators?
  5. extrapolation range. Is the fitting done between 10M and 1B, but is extrapolated to 100B? Have you verified the intermediate scale or hold-out scale?
  6. control variables. Are architectures, tokenizers, data mixtures, training hyperparameters, batch size, learning rate schedules, eval sets consistent?

Read the variable table and then look at the curve shape. The traditional power law is close to a straight line on the log-log map; broken scaling law allows different slopes to have different slopes; temporal scaling law pays attention to the loss trajectory in the training process; observational Scaling Law attempts to fit the rules from the public model rather than the self-trained model. The more complex the curve, the more you have to ask: is the complexity the explanatory power, or the overfitting?

A reading guide to the key papers

First stop: Kaplan Scaling Laws.
2020 Scaling Laws for Neural Language Models System research language model cross-entropy loss changes with the amount of parameters, data and training calculations. Its importance is not in a specific index, but in the research paradigm: training a family model, cross-scale fit, and guiding large-scale budgets with small-scale trends.

The second stop: Chinchilla Compute-Optimal Scaling.
2022 Training Compute-Optimal Large Language Models Reestimate the allocation relationship between parameters and tokens. The paper points out more than 400 models of experiments, under the fixed training Compute, many large models at that time are relatively "too many parameters and insufficient data"; Chinchilla uses 70B parameters and more training tokens to Gopher. Compute, reported a better overall performance. For scientific readers, the focus is on "budget allocation" rather than "70B number".

The third station: data is restricted and duplicated data.
Muennighoff et al. Scaling Data-Constrained Language Models From the 2023 arXiv to the 2025 JMLR version continues to improve, the study of high-quality text may be limited, how duplicate data, code data, filtering strategies and excess parameters affect Compute Optimality. It reminds us that D is not an infinitely scalable clean variable, and the data source and repetitions change the meaning of Scaling Law.

The fourth stop: the dispute between Emergent Abilities.
Wei et al.'s 2022 TMLR paper discusses the ability of the large model to appear suddenly after a certain scale; Schaefffer, Miranda and Koyejo in NeurIPS 2023 The paper points out that some "emergences" may come from metric choices, such as discontinuous or thresholding metrics that show smooth lift as a sudden change. Emergent Abilities Survey in 2025 has expanded the dispute to Reasoning, Quantization, Prompting and security issues. The point of reading this branch is: don't confuse the shape of the curve and the ability mechanism.

The fifth stop: multimodal, multilingual and wider scaling.
Generative Mixed-Modal Scaling Laws puts discrete tokens such as text, images, voice, code, etc. into a unified generation modeling framework to discuss modal competition and collaboration; multilingual scaling Laws studies Sampling ratio and language family performance in multilingual pre-training. This type of paper shows that D is not just "more tokens", but also includes data mixing ratios, modal structures and task distribution.

Sixth station: post-training and test-time scaling.
Since 2024, work on alignment and reasoning has extended scaling studies beyond pretraining to reward optimization, direct alignment methods such as DPO, SFT, RLxF, and test-time compute. The ACL 2025 survey of post-training scaling and a 2025 survey of test-time scaling both describe new dimensions of scale, including the amount of preference data, the number of samples, search depth, rounds of reflection, and tool calls.

Seventh station: Unification and dynamic Scaling Law.
Temporal Scaling Law focuses on the changes of Test Loss with STEP during training, and emphasizes the dynamics of Token position level in the 2025 EMNLP version; 2026 Unified Neural Scaling Laws Attempts to put parameters, data, training steps, inference steps, compute and hyperparameters into the same function form. For readers, this type of work is suitable for reading in the second half, because they assume that you already understand the boundaries of traditional power law.

A closer look at key papers

1. Kaplan: The premise of reading "predictability"

The intensive reading focus of the Kaplan paper is experimental design. The author does not train a large model after drawing a drawing, but trains a model of multiple scales, and does a system SWEEP in terms of parameters, data and computing. You have to ask: Is the model family consistent enough? Is loss measured on the same validation distribution? Does the fit span enough orders of magnitude? Is it differentiated between undertrained, overtrained and data-limited?

This paper is suitable for training you to read the log-log diagram. Don't just look at the beautiful curves, it depends on what experiments each point represents, where the error comes from, which points are not involved in fitting, and how far the extra target is from the existing data.

2. Chinchilla: Read "Budget Allocation" instead of the Champion Model

The core question of Chinchilla is: given fixed training FLOPs, should n increase or increase d? The answer to the paper drives subsequent LLM training from "heap parameters" to "more balanced parameters and tokens". In intensive reading, it is recommended to put Gopher, GPT-3, and Chinchilla on the parameter-token plane, instead of just comparing the benchmark table.

One point that is easy to ignore is: compute-optimal is the concept of training loss and a given calculation budget, and is not equal to any downstream task, arbitrary reasoning cost, and arbitrary deployment environment. The small model trains more tokens, which may reduce the cost of inference; but if the task relies on long-range reasoning, tool usage or multimodal abilities, additional evidence is required.

3. Data-constrained: read "data is not a free variable"

Data limited scaling law is very suitable for graduate students' intensive reading, because it embodies a real problem: if the independent high-quality text is not enough, can duplicate data continue to bring benefits? The paper reports that the loss of the paper changes is very small within a certain repeated round, but the value of the new compute will decay after more repetitions, and the data-related models and data are disclosed.

Do not pursue the 900B token level experiment when reappearing. You can use small corpus to construct Unique token, 2 epoch, 4 epoch, 8 epoch, code data enhancement and weak filter versions, and observe the validation loss and memorization indicators. The key is to record how "repetition" is defined and whether the eval set is isolated from the training corpus.

4. Emergence: Reading indicators, not just stories

The Emergent Abilities paper is attractive because it implies that the model size may lead to qualitative changes. However, scientific research training should disassemble the problem: does the ability really not exist in the small model, or has the small model have been gradually improved in continuous indicators? If it is successful with Exact Match, Pass/Fail, or exceed the threshold value, the curve may naturally appear abruptly; if it is replaced with Token-Level Probability, Partial Credit or Calibrated Score, may become smooth.

So reading the Emergence paper to do a checklist: Is the task difficult enough? Are the indicators continuous? Is Prompt unified? Is the sample size enough? Is Confidence Interval reported? Is it reproduced on multiple model families and data sources? Is it possible that the Benchmark Contamination exists?

5. Post-training and Test-Time: Read the new "Scale Scale"

After 2025, "Scale" is increasingly directed beyond pre-training: SFT data scale, preference data scale, Reward Model parameters, DPO/RL optimization intensity, BEST-OF-N Number of samples, search depth, self-reflection round, tool-use rollout number. They do not necessarily follow the same power law as pre-training .

The core training here is to distinguish three types of Computes: training compute, post-training Compute, and reasoning Compute. A method may be cheap but expensive to reason, and it may also be exchanged for higher accuracy through more test-time compute. When reading such papers, the table must be supplemented with Latency, Token Usage, failure rate and cost, not just the final score.

Methods and experiments compared

Papers/Data What to read main variable Minimum reproduction experiment Main risk
Scaling Laws for Neural Language Models The power law fit of loss and n/d/c Parameters, token, FLOPs, loss Train 10m-100m small LM, fit validation loss Extra push too far, ignoring data quality
Chinchilla Fixed parameter-token assignment under COMPUTE N, D, training budget Larger N, less token vs small n more token under the same budget misinterpret compute-optimal for all tasks
Scaling Data-Constrained LMs Data duplication and data scarcity unique token, repeating rounds, filtering Repeated training of small corpus, record loss and memory eval leak, duplication of unclear definitions
Emergent Abilities / Mirage Whether the emergence of indicators comes from indicators model scale、metric、threshold Re-evaluate with continuous and discrete indicators for the same output Mutation of indicators as mechanism mutations
Mixed-modal / Multilingual Scaling How data mixing affects Scaling Modality, language family, sampling ratio Small multilingual corpus to adjust sampling ratio Conclusion of different corpus quality
Post-training / Test-time Scaling Scale axis after pre-training SFT/RL/TTC, number of samples, search depth best-of-n or self-consistency cost-benefit curve Ignore inference cost and failure samples
Temporal / Unified Scaling Laws Dynamic process and multivariate unified fitting STEP, INFERENCE STEP, hyperparameters Save the middle CheckPoint fit Loss Trajectory The complex form of the function leads to overfitting

How to reproduce the results

If you don't have large-scale computing power, you can do three small and reliable training:

Experiment 1: Loss scaling of small LM.
Pick a clean small corpus, such as TinyStories, WikiText, or self-built English/Chinese corpus, and train 4 to 6 decoder-only models. Control tokenizer, training token, batch size and learning rate, and record Validation loss. The goal is not to get the general index, but to learn how to construct Scale Sweep and Log-log graphs.

Experiment 2: Compute allocation comparison.
Fixed total training token or approximate FLOPs, compared to "larger model training less steps" and "small model training more tokens". Report parameters, actual tokens, wall-clock, video memory, final loss and a small number of downstream probes. Focus on whether there is an undertrained phenomenon.

Experiment 3: Emergence metric audit.
Find a small inference task or code task, use the same batch of model output, and calculate Exact Match, Partial Credit, Token Probability, Pass@K and manually check the labels. See if the so-called "sudden appearance" still exists. This experiment is very suitable for writing reproduction report.

Save each reproduction:

  • Code commit or compressed package hash.
  • Data source, version, deduplication method, training/validation segmentation.
  • Model Config: number of layers, hidden size, number of heads, and calculation method of parameters.
  • Training budget: token number, batch size, learning rate, warmup, precision, hardware.
  • Fit script: function form, data points participating in the fitting, Held-out scale.
  • Chart: log-log loss curve, residual diagram, error bar, extrapolation interval.
  • Negative results: loss spike, overfitting, deduplication gain disappears, and the indicators are unstable.

Common misconceptions

Myth 1: Take Scaling Law as the law of nature.
It is the law of empirical practice, which depends on the model family, data, training formula, indicators and fitting interval. Changing the architecture, changing the data, and changing the tokenizer may change the coefficient and even the curve shape.

Misunderstanding 2: Just look at the amount of parameters.
Modern model comparisons must look at parameters, activation parameters, training tokens, data quality, context length, post-training and reasoning budget at the same time. It is only said that "a model is larger" has no scientific research information.

Misunderstanding 3: Loss is directly equated with ability.
Cross-entropy is the underlying metric, but downstream reasoning, coding, tool use, safe abstent, and multimodal reasoning do not necessarily improve on the same slope.

Myth 4: When you see Emergence, stop asking.
The real problem is not the label of "whether there is an emergence", but whether the task, indicator, model family and data support the strong conclusion that "unpredictable by small scale trends".

Misunderstanding 5: Ignore the cost dimension.
Test-time scaling may use more sampling and search to exchange scores; if token costs, latency and failure patterns are not reported, the conclusions are incomplete for reproduction and research topics.

Research questions for graduate students

  1. Chinese small corpus scaling law reproduction: Use the 10M-200M parameter model to compare the impact of Chinese news, encyclopedia and code mixing ratio on loss.
  2. Data Duplicate Boundary Experiment: Fixed Unique Token, the system changes the repeat rounds, tests Validation loss, neighbor memory and generates repetition rate.
  3. Emergence Metrics Audit Tool: Automatically put Exact Match, Partial Credit, Log Probability and Confidence Interval into the same report.
  4. Small Model Compute-Optimal Recipe Search: Look for N/D allocation within fixed GPU hours and compare the effects of different tokenizers.
  5. The cost curve of Post-Training Scaling: Compare the return point of SFT data, DPO epoch, best-of-n and self-consistency.
  6. Multilingual Sampling Ratio Reproduction: Use low-resource language small corpus to test whether Sampling ratio can be extrapolated from a small model to a larger model.

Summary

Scaling Laws is one of the few tools in large model research that can turn "expensive intuition" into "verifiable and predictive". When reading this route, start with the loss law of Kaplan, understand the basic relationship between n, d, c and loss; read Chinchilla, understand the parameters under fixed compute - token Allocate; then read data restriction, multimodal, multilingual, Emergence dispute and post-training/test-time Scaling, and see how the concept of "scale" expands from pre-training to data mixing, metrics selection and inference calculation.

For graduate students, the most important training is not to memorize a Scaling index, but to learn to design Scale Sweep, control variables, draw log-log diagrams, check residuals, mark out extrapolation ranges, and honestly report negative results. If you can do this, Scaling Laws is not only the budget tool before the training of the big factory, but will also become the basic method for your ability to form research problems and reproduce.

References

Search date: 2026-08-02. The following priority is given to listing papers, conference pages, official repositories or official publishing pages; model scale, Benchmark numbers, code repository status, data set version and other easily changed information, and it is recommended to verify again before submitting or publishing.

  1. Kaplan et al., 2020, Scaling Laws for Neural Language Models, arXiv: https://arxiv.org/abs/2001.08361
  2. Hoffmann et al., 2022, Training Compute-Optimal Large Language Models, arXiv: https://arxiv.org/abs/2203.15556
  3. Google DeepMind, 2022, An empirical analysis of compute-optimal large language model training: https://deepmind.google/blog/an-empirical-analysis-of-compute-optimal-large-language-model-training/
  4. Wei et al., 2022, Emergent Abilities of Large Language Models, arXiv / TMLR: https://arxiv.org/abs/2206.07682
  5. Schaeffer, Miranda, Koyejo, 2023, Are Emergent Abilities of Large Language Models a Mirage?, NeurIPS: https://proceedings.neurips.cc/paper_files/paper/2023/hash/adc98a266f45005c403b8311ca7e8bd7-Abstract-Conference.html
  6. Muennighoff et al., 2023/2025, Scaling Data-Constrained Language Models, arXiv: https://arxiv.org/abs/2305.16264
  7. Muennighoff et al., 2025, JMLR version: https://jmlr.org/papers/v26/24-1000.html
  8. Hugging Face datablations repository: https://github.com/huggingface/datablations
  9. Caballero et al., 2023, Broken Neural Scaling Laws, arXiv / ICLR: https://arxiv.org/abs/2210.14891
  10. Aghajanyan et al., 2023, Scaling Laws for Generative Mixed-Modal Language Models, arXiv: https://arxiv.org/abs/2301.03728
  11. Gao, Schulman, Hilton, 2023, Scaling Laws for Reward Model Overoptimization, ICML: https://icml.cc/virtual/2023/poster/24924
  12. Rafailov et al., 2024, Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithms, arXiv / NeurIPS 2024: https://arxiv.org/abs/2406.02900
  13. Ruan, Maddison, Hashimoto, 2024, Observational Scaling Laws and the Predictability of Language Model Performance, arXiv: https://arxiv.org/abs/2405.10938
  14. He et al., 2024, Scaling Laws for Multilingual Language Models, arXiv: https://arxiv.org/abs/2410.12883
  15. Xiong et al., 2024/2025, Temporal Scaling Law for Large Language Models, arXiv / EMNLP 2025: https://arxiv.org/abs/2404.17785
  16. Lai et al., 2025, A Survey of Post-Training Scaling in Large Language Models, ACL Anthology: https://aclanthology.org/2025.acl-long.140/
  17. Zhang et al., 2025, A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?, arXiv: https://arxiv.org/abs/2503.24235
  18. Berti, Giorgi, Kasneci, 2025, Emergent Abilities in Large Language Models: A Survey, arXiv: https://arxiv.org/abs/2503.05788
  19. Caballero et al., 2026, Unified Neural Scaling Laws, arXiv: https://arxiv.org/abs/2605.26248