Hourglass and two learning paths, indicating the choice of synthetic preparation course and direct language training under a limited budget

The cover is a concept illustration generated by AI, not a paper architecture, real experimental equipment or performance results.

  • Interpretation date: 2026-11-01
  • Data deadline: 2026-11-01 20:30:13 (Beijing time, Asia/Shanghai)

At 21:45 Beijing time on October 29, Alexander Stern and colleagues from Cohere, UCL, and MIT submitted the preprint “Rethinking Pre-pretraining.” It revisits the value of training a language model on synthetic data before ordinary pretraining. The study became public within the past three days, before today’s article. The distinction matters: adding a training stage may improve scores, but does it earn its share of the budget? Paper and revision history

Make it clear first

Pre-training (PPT) can be understood as the preparatory course before formally learning the language: first learn the laws of artificial structure, and then turn to natural texts. The "synthesing" here does not specifically refer to the question and answer generated by the chat model, but may also be a sequence of symbols generated by the program. The materials examined in the paper include parentheses structure, sequences of automatons generated by nerve cells, declaration and reference tasks, and code controls. Methods and Experimental Settings

This idea is intuitive and attractive: students can practice basic skills first, and then they may learn faster. But putting it in the model training, intuition needs a resource bill. Assuming that an experiment adds a preparatory course first, and then reads the original text as usual, even if the final grade is better, it is not possible to indicate that the preparatory course is more effective than continuing to read the text. It first states that this additional investment may be useful.

Therefore, scientific readers should first find the comparison object: Does the research increase training after fixing the number of natural texts, or to divide the same sum of computing power to two materials? The former is closer to "whether it can supplement knowledge", and the latter is directly related to how the limited budget is allocated. Both questions hold, but the answers cannot be interchanged.

The starting point of the baseline will also change the story

The team reported that in a group of nerve cell automata reproducing experiments, the original initialization setting can be improved by 5.4%; after the initialization, the gap between the two methods disappears. The confusion measures the degree of uncertainty when the model predicts the text, and this percentage is not the percentage point of the task Accuracy increase. Original Section 4

This provides a specific reminder for the reading method paper: the so-called baseline is not a natural state without design choices. The original weight, optimization settings and training arrangement of the model may leave room for improvement that has not been used. The new method exceeds a certain starting point, and it exceeds the fully adjusted competition plan, and supports conclusions of different strengths.

But in turn, it has to be restrained. After changing the initialization, a certain advantage disappears, and it cannot be automatically proved that the original method has never valued. A fairer next step is to allow both sides to get similar opportunities for tuning, and whether the comparative advantage exists is stable. The role of reproduction should be positioning conditions, not just labeling "success" or "failure" on the paper.

What do negative results deny?

In the paper report, in the tested and other computing power settings, PPT did not improve the loss of natural language verification; there are still exceptions to some downstream tasks. When adding additional computing power, you can benefit from the small natural language budget. Therefore, the result cannot be abbreviated as "compositing data is useless". Results and discussion

There are at least two layers of boundaries here. First of all, the language predictive indicator is not the same thing as the specific ability. The overall prediction of a model is smoother, which does not mean that it is stronger in each type of reasoning problem; if a small task is improved, it does not mean that the overall training efficiency is improved. Readers need to confirm their research goals first, and then decide which indicator is more explanatory.

Second, there is still a long distance between denying a certain curriculum arrangement, and denying the entire class of data. The content of the preparation of the course, which stage of training, and how many resources are occupied may change the result. The judgment of this paper is that this evidence has raised the threshold of arguments required to "add an independent stage first", but it is not enough to give conclusions to all synthetic data routes.

If there is less data, you should also carefully compare "read it again"

An easily overlooked competitive scenario is to reuse existing text. Scaling Data-Constrained Language, which was first released in 2023 In Models, the researchers examined the relationship between duplicate data and calculation budget: under its conditions, moderate repetition can retain the training value, but the benefits will decay as the repetition increases. This is a historical background and cannot be counted as an independent reproduction of today's paper. Background papers and version records

It prompts us that the choice when the data is insufficient is not just "find new data" or "stop training". How much information can the old data provide, which requires actual comparison. At the same time, repeated training is not a free lunch: it occupies computing resources and may gradually only strengthen the already familiar mode. Reasonable experiments should allow the new curriculum to face a real and competitive alternative, rather than an opponent who stopped learning early.

For graduate students who are ready to carry out related topics, this also affects the way of topic selection. Instead of first asking if you can create more complex synthesis tasks, it is better to first clarify which abilities you want to transfer, what tests to use to identify it, and whether this ability is worth the extra training cost. In this way, even if the overall score is not improved, it can leave an interpretable study result.

Next, let's see if the verifiable comparison can stand firm

The author has opened the training and evaluation repository, and README lists the experimental configuration, data entry and scoring methods. This gives external researchers the opportunity to check item by item; it is still just the material provided by the authors. No code was executed this time, and no independent reproduction record was retrieved enough to confirm the new research conclusion. Authors’ repository

These findings come from a preprint. I’d like to see whether other teams find similar trends under clearly specified data, initialization, and budget conditions, and whether more targeted capability tests reveal gains that averages hide. Any claim of resource savings also needs to account for the cost of generating data and searching for a configuration before the final training run.

The more secure judgment at this stage is that the synthetic preparation course needs to prove its opportunity cost. The real convincing progress in the future is not just a higher score, but explains what abilities have been learned, under what conditions, and why this is more worthwhile than the other uses of the same resource.

References

  1. Rethinking Pre-pretraining: summary and commit time, 2026-10-29 Submit, preprint.
  2. Rethinking Pre-pretraining: HTML original text, focus on reading the experimental settings, results, limitations and code descriptions; it belongs to the same research as the previous entrance.
  3. Author code repository rethinking-ppt, read the README with the directory, not running or auditing the complete implementation.
  4. Scaling Data-Constrained Language Models, 2023-05-25 first, the current page V5 revised on 2025-06-28; this article is only used as a background.

Apps by the author

The following is the author's own product promotion. In addition to writing a technical blog, I have also independently developed two daily gadgets, welcome to understand: Daily Expense Journal Used to record revenue and expenditure, arrange budgets and save goals, View on the App Store; One Small Action Help you choose small actions according to the current time and energy, and record your feelings after completion, View on the App Store. More technical articles and personal works, welcome to visit my personal website. These are independent introductions of individual works and do not represent the recommendation or endorsement of the research team of this article.