Next to the release device, the magnifying glass is aimed at a set of points to be checked

Large-model research controversy: from the paper conclusion to the boundary of evidence

Abstract

A paper written by the agent has received a sufficient score to pass the review, and another system has found a larger set of points on the problems that mathematicians have studied for a long time. Which thing better shows that the machine is involved in the discovery? The previous ones covered more research links, but the evidence left behind was easier to check independently. There is also an interesting test between the two: hide the conclusion of a published paper, and only give the problem of the intelligent body to see whether it can retrieve the conclusion by experiment. When reading these three tasks, first ask what is handed over to the reader, and then ask who has the ability to overturn it. Workshop experiment、Rediscovery Benchmark、point set structure

After a review, where does the denominator start from?

Lu et al. did a real submission experiment. They handed over the broad theme of the ICLR workshop to the system, and asked it to come up with ideas, write code, do experiments and write them. The researchers screen candidates at each stage to see whether the idea is relevant, whether the code can correctly implement the idea and run it, and whether the manuscript format is qualified. According to the author's report, the content of the selected draft from the idea to the writing has not been manually revised. The two verbs here are worth reading separately: Rewrite will change the content left, Filter What changes will go to the next step. One study can be without manual rewriting, and it is quite reliant on manual screening. Lu et al., Human Evaluation Results

In the end, three articles were sent for review. The reviewer knows that there are machine-generated works in the submission, but do not know which ones are. One of the articles reported the negative results of the combination regularization attempt, and received two weak reception and one receiving opinion. The organizer believes that it is likely to be received, and the research team will then withdraw the manuscript according to the scheduled plan. Officially published in Nature is a paper by the team to study this system. Only by clarifying the authors and whereabouts of the two-level manuscript can we see the most interesting results of this experiment: a work that has not been expected to improve can also be considered worth discussing by its peers. Lu et al., Figure 2 and Limitations

If you want to ask "start again, what is the chance of producing the same quality manuscript", you can't take three articles as all attempts. The idea of early elimination also consumes resources, and the person responsible for eliminating the realization of errors also provides judgment. The author believes that if it is not screened, the selected manuscript will also be generated, but it will generate more manuscripts and cost more. This explanation can hold, but does not give a paired result that is unscreened under the budget. The review record supports an actual path, and the output rate when unattended screening cannot be estimated separately. To push the conclusion a step further, you need to look at the discarded branches and what the same budget could have been used for. Lu et al., Human Evaluation Results

Hide the conclusion, can the intelligent body find it again?

Wang et al.'s FIRE-Bench changed the delivery. The questioner first finds a paper with a clear experimental discovery, and then gives the agent a higher-level research problem, and hides the original experimental design, implementation details and analysis path. This leaves room for the selection method, and also has a reference conclusion that can be used for comparison. The "rediscovery" here has a clear scope of reference: it is necessary to find it by itself for the questioner, and there is a literature record for the research community. It is impossible to register the results as the first discovery based on the hit reference conclusion. Wang et al., the task definition after the conference version and the author's draft figure 1 full text

Ratings are also worth reading. The evaluator splits the report with the reference text into a claim that can be judged separately, and then uses the language model to judge whether they are semantically matched. How much of the claims written by the Accurate Care Report are supported, how many claims in the Recall caring reference are retrieved, and F1 combines the two. In the conference version report, the average F1 performance of the best performance in the tested system is still below 50 (percentage system). Reading it as "less than half of the experiment to do success" will replace the counting unit. It is entirely possible for a report to complete the experiment, say the direction of the change, and at the same time miss the conditions of the phenomenon. Wang et al., the summary of the conference version Author's draft Claim matching and scoring

Conversely, if a report finds exceptions that the reference paper has never been discussed, how will the existing score be dealt with? According to the matching rules of the author's draft, the output that is supported by no reference claims will be counted into False Positive . This agreement makes it easier to compare who better finds known phenomena. As for whether the exception comes from a new observation, an experimental error or a writer's guess, to go back to the operation record, the design can allow the three to give different predictions. Re-reading the same paragraph with another referee can reduce the disagreement in the wording and understanding, but cannot make up for the missing observations. Wang et al., Claim matching and scoring; exceptions are envisioned in this paper

The three types of products are sent to peer review, reference conclusion matching and restraint inspections respectively

The arrow in the figure indicates the pairing of the inspection object to the inspection method. The paper records the corresponding review, matching or structural results; whether the independent redo empirical experiment can reach the same conclusion needs to be tested separately.

Why is a point set, why is it easier to verify than the sentence "new ideas"?

There is a short rule for the cap set problem in the FunSearch study. Each point can be written as a string of coordinates, each bit is only taken zero, one or two, and the addition is calculated according to modulo three. The goal is to pick out as many points as possible without leaving any three different points on the same finite geometric line. This definition leaves a review entry for readers: after getting the list of points, you can find the third remaining point on the line for each pair of different points, and then see if it is also in the list. The inspector can write another program without knowing which search path the creator has gone through. Romera-Paredes et al., Figure 3, Figure 4

The researcher prescribes how to construct a legal set, and the language model participates in the search of a priority function that ranks the candidate points in succession. Subsequent candidates may be excluded from each choice, so even if each step is legal, how many points are left in the end depends on the order. The author found the containing 512 The structure of points, the best structure listed in the figure is 496 point. This result has a clear mathematical meaning: at least so many points can be put down in this dimension. To claim that it is already the largest, and to exclude all the larger legal collections, there is another proof. Romera-Paredes et al., Figure 4a–b

Look at the last two columns of the same picture. The model search leaves a priority function that can generate the point set, and the researchers then check the arrangement and coordinate rules of the vector, and manually sort out a more direct structure. One solves "how to find it", and the other makes it easier for peers to see "what parts it consists of". If the latter is counted as the model's own interpretation, it will erase the traceable human contributions in the paper. Conversely, because the problems and inspection rules are provided by people, the newly added point set is also described as simple paper automation, which also cannot explain why the old lower realm was pushed up. Romera-Paredes et al., Figure 4B–C, Appendix E.2

Finding something once and finding it reliably are separate questions you can study side by side. Figure A.3 in the authors’ manuscript reports that 4 of 140 runs on this eight-dimensional task found a 512-point construction. One valid example suffices to prove existence. For the next researcher considering the same investment, the other runs matter too. The mathematical question is whether the example satisfies the constraints. The practical question is how often the search pays off under those conditions. A rare success can establish a sound mathematical result while telling us little about how consistently the method will deliver in everyday use. Romera-Paredes et al., Figure A.3 in the authors’ manuscript

Which differences are worth comparing together?

The three projects start in different places. One starts with a workshop theme and encourages many ideas. Another starts with a published research question and lets the agent find its own way. In the third, even the rules for a valid solution are fixed in advance. The studies use different base models and do not match training data, inference budgets, human screening time, or test splits. Reviewers assess the workshop papers, the rediscovery task has preselected reference findings, and the mathematical construction faces constraint checks in a fixed dimension. Putting accepted-paper counts, claim scores, and point-set sizes on one scale would obscure what each result tells us. Workshop Agreement datum definition task

A more useful comparison is to ask what other explanations are allowed for each result. The manuscript may indeed present valuable empirical observations, and may also be worth communicating because the limitations of known practices are clearly explained. Rediscovering the score may come from the design of a suitable control, or it may contain the knowledge that the model has remembered from previous reading; the question is not given, and there is no guarantee that the model has never been seen during training. Mathematical examples tighten this uncertainty, because a legitimate structure that exceeds the old records listed in the paper has the amount of information itself. However, it is still a matter of independent question to give what kind of program framework to find it easily, and how other search methods perform under the same resources.

A follow-up experiment that can actually differentiate and interpret

It can start with a small, open rerunning sorting problem. Suppose the research group cares about "what sorting strategy will lose its original advantage when there are a large number of duplicate keys in the input". Defining the advantage first refers to the number of running time or comparison, and fix the implementation, hardware and input generation rules. A performance phenomenon that was previously described publicly is used as a rediscovery task, and a batch of newly generated inputs that are not visible to the participant is reserved for checking additional predictions. The sorting example here is a study design with no experimental results yet.

Give each group the same starting model, tool and time. A group of read-only questions and public data are written after predictions, one group is allowed to execute code and modify the hypothesis accordingly, and the third group allows researchers to screen branches under the same execution budget. Each screening must retain the candidates selected and eliminated, and record how long it took and whether they have seen the performance results. The read-only group also reserves enough AT Inference time, otherwise the benefits of "can do experiments" may be mixed into the total resource difference. Each group submits the same short product: which inputs are expected to change, what direction, and which results will overturn their own explanations.

In this example, "the more repeated keys, the slower the algorithm" is not specific enough. Repeated keys in the same proportions can be squeezed into a segment or scattered throughout the input. If the report attributes the time to the comparison to the number of comparisons, it should be announced whether the number of comparison keys and time will be changed after changing the position of the duplicate key. If the two change separately, even if the time trend is guessed correctly, the half sentence about the reason should be recorded as unsupported. Only then will the evaluator have the opportunity to distinguish the right direction and find a verifiable explanation.

After locking the prediction, the other executor is then rerunned independently and disrupts the source tag of the report. If a group of experiments is allowed to lead only in the matching of reference conclusions, the new prediction has no advantage in retaining input, and the result is mainly to support better rediscovery. If the execution group can predict a boundary condition that still exists in the independent implementation, and cannot retrieve the same conditions that have been reported before, it is worth listing as a candidate new discovery, and publicly inputting the counter-example for the peers. If the advantages are concentrated in a group screened, it is reported together with the time of the person and the screening basis. Stop at the scheduled budget, and include no predictions, wrong directions and realization failures, to avoid only showing the luckiest trajectories.

Summary

After reading "Agents complete discovery", you can first ask for a specific object: a manuscript, a prediction with experimental conditions, or a structure that can be inspected item by item. Then ask two things: relative to which old knowledge, what is added to this object? For someone who has not participated in the screening, how can he regain or overthrow it? Workshop reviews, known conclusion matches, and limited sets of each answer were answered. Write the claim to the position where the evidence just can be reached, and make it clear that the contribution that should be owed to the system, the researcher and the follow-up validator can be made clear.

References


Apps by the author

I also build small tools for everyday life. Here are two of my apps:

  • Daily Expense Journal: Track your income and spending, plan a budget and set savings goals. View on the App Store.
  • One Small Action: Pick a small action that fits the time and energy you have today, then record how it felt and what you noticed. View on the App Store.

Find more articles and projects on my website: zffjzz.top. I’d love to hear how the apps work for you and what I could improve.