
Large-model research controversy: from the paper conclusion to the boundary of evidence
Summary: What kind of realistic evidence is needed for success in the screen?
The robotic arm holds the cup firmly in the screen, can you believe that it can also complete the task on the real desktop? The problem is not just whether the picture is realistic enough. The same action instruction may have different displacements, and the friction of the cup, the position of the camera and the end condition of the task can also change the result. For the visual-language-action model, language understanding, visual recognition and control errors also affect each other during the execution process.
This article compares the three first-hand studies, and distinguishes three propositions: whether simulation can help arrange the pros and cons of the strategy, whether the real success rate can be estimated, and whether simulation training can improve the actual performance. The current evidence supports conditional use of simulation as a research tool, but does not support the fact that it is reliable to declare reality with a total simulation score. There are no "simulation effective pie" and "simulation invalid pie" that must be selected here, and the real differences are often hidden in the measurement object and the applicable distribution.
Distinguish first: training migration and evaluation migration
"Strategy" refers to the rule that the action is selected according to the observation, which can be implemented by the neural network. Visual-language-action model, referred to as VLA, maps images and text commands into robot actions. This article focuses on its behavioral evidence and does not discuss packaging robots into application products.
training transfer The question is: whether the strategies learned in simulation are still effective in reality. Evaluate the migration The question is: take fixed strategies in simulation and reality respectively, and whether the simulation measurement can predict the real performance. The strategy may also be mainly trained by real data; at this time, the simulation is responsible for the test site, not the source of ability. SIMPLER made it clear to study the latter issue. SIMPLER §I、§III
Then take the "good prediction" apart. If the order is correct, it only means that A is usually better than B; the absolute error is small, which means that the simulation success rate is close to the actual success rate. Both are good, and it does not mean that the low frequency has been covered by a serious failure. For example, a measurement tool may always overestimate all strategies, but keep their order. This is a logical possible situation, not an experimental result running in this article.
Compared with the previous discussion on observation video and action supervision, the newly added questions in this article are: When the strategy has been fixed, who will verify the measurement environment itself? If this step is omitted, the poor simulation performance may be misread as poor strategic ability, and the simulation performance may be misread as a strong practical ability.
SIMPLER: first prove that the measurement tool can distinguish the advantages and disadvantages
Li et al. do not require the simulation of the complete replication reality, but narrow the control gap and visual gap respectively. The former should make the same instruction produce a similar motion, and the latter try to make the model receive similar images. They also compare the two evaluation paths for the aggregate of visual matching and multiple visual variant results; the variant aggregation here occurs during the evaluation, and cannot be written as the effect of domain randomization training. SIMPLER §IV
In the settings of Google Robot and WidowX, the paper is paired to check the real and simulated performance of multiple policy checkpoints, and the strong correspondence between most measured tasks is observed. The evaluation includes the Pearson correlation coefficient, and the average maximum sorting violation MMRV: the latter not only sees whether the error is arranged, but also considers the real performance gap between the misaligned strategies. SIMPLER §III-B、§VI-A–B
The value of this evidence is to make "simulation not like" a measurable problem. However, the task is biased towards the operation in the existing training data, and is mainly suitable for physical simulation of rigid body items; the paper also observes that the sensitivity of the strategy to visual differences is different. Therefore, the correlation on the old strategy cannot directly endorse the new model, new material or new task. SIMPLER §V、§VI-B
In this article, it is inferred that the credibility of the simulator should be attached to the set of conditions of "Strategy Collection, Task Collection, Observation and Control Protocol". Recalibration may need to be recalibrated to replace any of the key parts. Correlation is a measurement result within a certain range, not a permanent certificate obtained by the simulator.
REALM: Expand the range of disturbances, and also write the verification range
Sedlacek, etc. push the evaluation to more visual, semantic and behavioral disturbances: changing the camera, rewriting the command, adjusting the properties of the object, and testing the different generalization requirements. REALM adjusts the control parameters through real trajectories and compares simulation and reality on the DROID platform. Its original pairing verification covers three VLAs, seven tasks and five types of visual and behavioral disturbances; then a large-scale simulation evaluation uses fifteen types of perturbation. The latter range cannot all impersonate the range that has passed the real verification item by item.REALM §III–V
The paper also uses the "stage task progress": complete the stage of close, crawling, and lifting in sequence, and scores at each stage. It can better show where the card is than success or failure, but it is not a complete task success rate. REALM also reported the binary success rate, which shows that the two indicators have different responsibilities. REALM §III, Table III, §V
There is an easily overlooked interpretation of competition: the score changes very little after the disturbance, which may be a stable strategy, or it may be that the basic performance is already very low, and the room for continued decline is limited. The authors clearly discuss this measurement insensitivity in the limitations. It reminds us to first confirm whether the task can be done under the default conditions, and then explain the disturbance effect. REALM §VI
This article does not put the scores of REALM and SIMPLER together. The robots, model starting points, tasks and indicators of the two are different; the model in REALM also has its own DROID adaptation process. Even if you use MMRV, you need to check whether the underlying input is a success rate or a task progress, and you can't use it as the same ruler just because the indicator has the same name. REALM §IV、§V Evaluated models
The wider the randomization, the better the migration?
Randomization of the domain is to change the environmental parameters during training, so that the policy can be exposed to a set of possible environments; the system is identified, and the target system parameters are estimated by observation. The former pursue coverage, and the latter reduces uncertainty. They can be combined, but "multi-manufacturing changes" are not automatically equivalent to "learn to deal with realistic changes".
The recent preprint of Rizvi and Tomar cross-scans the amount of identified data and randomized width. It must be made clear first: the target system is still a single pendulum simulator, not a physical robot; training uses the same kinetic family as the target, and the unknown quantity is only the mass and the length of the rod. In the measured setting, after obtaining the target trajectory, the training on the estimated parameters is better than the continued expansion of the randomized bandwidth; there is no alternative measurement for the wide a prior baseline that covers the real parameters. Budget Study V2 §II–III
This is not evidence of "invalid randomization". The paper admits that the wide distribution may require more training, and the number of fixed training steps is used for comparison; the target model can be represented by the training model, and there is no structural error that cannot be eliminated by tuning parameters. The author also pointed out that if a random action is used, the cost and risk cannot be calculated only according to the number of trajectories. Budget Study V2 §II-B, §IV
At least two explanations can be tested separately. Measurement may have removed the dominant source of error. Alternatively, broader randomization may have made optimization harder, leaving the model undertrained on the available budget. The evidence for the first explanation is limited to identifiable parameters within the same model family, and the second remains possible. Extending the pendulum finding to robots involving contact, delays, or flexible materials will take further experiments.
Three sets of evidence, answering different questions
| Research | Conditions of the actual inspection | Observe what is supported | what can't be proven |
|---|---|---|---|
| SIMPLER | Fixed strategy, pairing reality and simulation operations | Useful relative performance signals can be obtained within the measured range | Simulation training creates ability |
| REALM | Alignment control, staged indicators, real verification of partial disturbances | VLA generalization weakness can be checked | All simulation disturbances have been verified in reality |
| budget study | Two single pendulum simulators, parameters can be identified, fixed training steps | Measurement is more effective than widening randomization in this setting | Any physical robot should be unrandomized |
Tables are not the winners and losers of the three methods. The first two are mainly validated evaluation agents, and the third research training resource allocation; they do not have a common training set, and there is no unified reasoning or interactive budget. Forced comparison of final scores will make different research objectives into surface conflicts.
Why the cost of failure must be recorded separately
The above success rates, progress, and sorting indicators are not the definition of cost damage. Stop safely after grabbing failure, and it may also be recorded as failure as the item next to knocking over; it does not mean that the continuous operation is reliable if the task is completed but requires frequent manual resetting. This is an analysis of the evaluation design in this paper, and it is not the accident statistics measured by three papers.
Therefore, the real validation should record the task results, the number of interventions, the reset time and the pre-defined serious failure events separately. Cost weight depends on the research problem, and should not temporarily choose a favorable weight after seeing the results, and press all the indicators into a beautiful comprehensive score. There is no certain type of failure, and it can only mean that there is no encounter in this round of observation, and the limited number of observations cannot be written as no risk.
Sample sources are equally important. Repeated reset to the familiar position, the measurement is the starting point distribution; to delete the failure that is difficult to reset, even the denominator has changed. The study record should keep the trajectory and cause of the aborted, and indicate which indicator it enters. Otherwise, the so-called actual success rate may only describe those tests that can successfully complete the evaluation process.
A follow-up experiment to distinguish competing explanations

The illustration is the proposed measurement process, and the arrows indicate the flow direction of the recording, not the causal relationship that has been proved. Sorting, absolute error and failure cost are reported separately; there is no measurement data in the figure, and it does not mean that all three papers have performed all the steps.
The first step is to freeze the policy checkpoint, observation processing, action frequency, termination conditions and success criterion. Separate the tasks for calibrating the simulator from the tasks for final verification, and reserve new objects, starting points or scenes. Compare only in scenarios that have been adjusted repeatedly, and cannot answer whether the simulator has the ability to evaluate generalization.
The second step is to separate the visual matching and control matching separately to form a cross condition. Each condition is paired with the same task, and the failure trajectory is saved. If the ranking is significantly improved after the correction and control, and the visual correction effect is limited, there is a reason to return the main gap to the control side; if the combination of the two is effective, it is necessary to report interaction, and a unique reason cannot be specified.
Third, compare narrow and broad randomization from the same starting point under two protocols: equal training steps and more extensive training. The second tests whether the broader distribution simply takes longer to converge. Its results must be reported separately from any equal-budget advantage. Then introduce structural differences the training model cannot represent, such as a delay, and see when the case for measuring first changes.
Finally, what is written in advance will overturn the explanation: if the sorting is maintained but the absolute error continues to be too large, only candidates can be used for screening. The above are all experimental suggestions that have not been carried out. This article does not claim to run the robot or reproduce the values of the paper.
Summary: Let the migration advocate land with conditions
One of the most valuable uses of simulation is to make a large number of controllable comparisons cheap; real verification determines what range these comparisons are worth believing. When reading the paper, you should ask one by one along "who trains, where to test, how to score, and how to account for failure". Only by explaining the evaluation agent, ability transfer and failure cost separately, can the success in simulation become a boundary and verifiable realistic evidence.
References
Retrieval and data deadline: 2026-11-04. The search contains nearly twelve months of work, retains the necessary classic papers, and does not claim complete coverage.
- Li et al. Evaluating Real-World Robot Manipulation Policies in Simulation, CoRL 2024, published by the essay collection in 2025. formal page; The method and result positioning of this paper are used arXiv V1 full text (2024).
- Sedlacek et al., REALM: A Real-to-Sim Validated Benchmark for Generalization in Robotic Manipulation,2025. arXiv V1 full text. Author project page It is said that it has been received by RA-L and will be displayed in IROS 2026; this time, the journal records are not independently verified.
- Rizvi and Tomar, How Should a Simulation-to-Reality Transfer Budget Be Spent?,2026. arXiv V2 full text. In preprint, the metadata is said to be submitted to the relevant workshop of IROS 2026, and the employment is not confirmed.
Apps by the author
The following is the author's own product promotion. In addition to writing technical blogs, I also independently develop daily gadgets, welcome to understand according to your own needs:
- Daily Expense Journal: Record income and expenditure, arrange budget and savings goals, and make daily accounts more organized. View on the App Store.
- One Small Action: Select small actions according to the current time and energy, record the feelings after completion, and leave a clue for the daily review. View on the App Store.
More technical articles and personal works, welcome to visit Author personal website. These works have no endorsing relationship with this paper and research institute.