Cover illustration

  • Date: 2026-11-03
  • For graduate students, early-career researchers and research engineers
  • Deadline: 2026-11-03; search for nearly 12 months of work, and retain the necessary early research

Abstract

When you see a person lifting the cup up and letting the manipulator lift the cup, what is the interval between? The video can display the target, sequence and object changes, but does not directly give another kind of control instructions that the body should perform. In this paper, compared with VPT, LAPA and HuRo, check which part of the training process the action information is entered. Three studies support video as a useful learning resource, but do not jointly prove that "general control capabilities can be obtained only by watching". The more worthy question is: how much of the video gain comes from visual representation, how many comes from migrating action structures, and how much depends on subsequent adaptation and feedback?

Seeing the result does not mean getting the control command

Imagine a video of taking a cup. We can see the hands close and the cup is off the table, but we may not see where the fingers are in contact, and we don't know the weight of the cup. This example is only used to explain the problem: the same visible movement can correspond to different grip, joint motion and contact states. Change to two fingers, even the feasible grip can be changed. Therefore, the displacement in the pixels cannot be directly used as an actuator command.

There are three terms that need to be distinguished here. observation In this paper, it refers to video learning that uses no target robot action tags; it does not guarantee that there is no other supervision during the whole training process. inverse kinetics model The counter-promotion action based on the observation before and after the change; underwater action It is the internal code of compressed visual changes, which does not necessarily have the meaning of "how many degrees of a joint rotation". The visual-language-action model (VLA) connects images, instructions and action predictions, but the name itself does not indicate where the action ability comes from. VPT Methods§3、LAPA§3

So there are at least three distinguishable explanations: the main church model of the video recognizes the scene; the video provides an action structure shared across the body; the video makes the subsequent action supervision easier to use. They can be established at the same time. The final success rate can only indicate that the whole process is effective, and the contribution of the three cannot be automatically allocated. To allocate contributions, you must check the behavioral changes in the source of information, replacement experiments, and tests.

Three methods, where does the action information come from?

VPT: first use real actions to teach the labeler

VPT was published in NeurIPS 2022. It first uses the data of the contractor that records the operation of the keyboard and mouse to train the inverse dynamics model, and then adds a fake action label to the network Minecraft video, and finally performs a behavior clone, that is, learns the action selection in the reproduction demonstration. The labeler can look at the future frame, and the actual execution strategy can only look at the current and past: Explaining the operation afterwards, and deciding the next step in advance, are different tasks.VPT§2–3

The original text reports the zero-shot behavior of the pre-trained strategy, but the diamond pick result comes from further early game behavior cloning and reinforcement learning fine-tuning, which cannot be attributed to watching the video. Reinforced learning use environment rewards, additional information on behavioral consequences is introduced. Here "untag" describes the network video, not the entire system. VPT§4.2–4.4, Figure 7

The judgment made according to this article is that the VPT shows the path that the action supervision can be enlarged, and has not yet answered the mapping of man-to-manipulate. The modeler in the game and the strategy share the interface of the keyboard and mouse, which is not the same problem as the cross-bot body migration.

LAPA: first learn the change code, then adapt to the robot

The LAPA of ICLR 2025 uses the front and rear frames to train discrete submersible movements, and then let the existing Vision-Language Model predict the encoding according to the current image and task description. In the robot Fine-Tuning stage, it discards the submersible action output head, puts on the actual action output head, and uses the robot trajectory with the action label. Therefore, the "mapping" in the process is not to install a naturally executable general-purpose action dictionary directly onto the robot. LAPA§3.1–3.3

The human video experiment of the paper reports the benefits of relative pre-training at this stage; the Scratch here still starts from the existing Vision-Language Model, and not all parameters are randomly initialized. At the same time, the author clearly lists the shortcomings of fine grasping, and the latent code may also include camera motion. LAPA§4.2、§4.5、§5.2、§6

This supports pre-training of submersible movements with migration value, but it is not enough to prove that the code only means controllable actions. This article believes that judging its content needs to further separate the perspective change from the object operation; the qualitative reconstruction diagram of the cross-body can put forward hypotheses, and cannot replace the intervention test in real control.

HuRo: Processing human video into robotics oversight

HuRo (2026) retargets human hand movements to robot trajectories, removes the human from the video, and overlays the robot. Its training signal incorporates priors from reconstruction, kinematics, and visual transformation. The raw video alone does not supply all that information. I checked arXiv v5, dated September 25, and could not confirm formal acceptance. HuRo§3, version record

More information is the meaning of ablation. When the Visual superposition of the robot is removed, the redirection action supervision is still retained; and the setting called Visual Only is to reserve the pre-trained Visual path and reset the action head before downstream fine-tuning. It is not a Visual pre-training without motion supervision at all. Therefore, this experiment can test the value of retaining action modules, and cannot isolate the causal effect of "whether to use action tags during pre-training". HuRo§4.3–4.4

The paper also points out that the conversion process lacks force and tactile signals, and kinematic redirection does not model self-collision or physical contact. Therefore, the resulting supervised trajectory cannot be directly equated with the executable demonstration. This is a boundary that still exists after "action data looks complete". HuRo§6

Don't put different ends on the same leaderboard

Research Conditions and Information Entrance Observe what is supported what can't be proved
VPT Same game interface; real action training labeler Pseudo-labeled videos can support policy learning Cross-Body Control Without Action Supervision
LAPA Submersible coding pre-training; target robot action fine-tuning Video characterization improves downstream learning Naturally executable
HuRo Redirect and visual conversion; robot demonstration fine-tuning Post-processing supervision has migration value The reconstruction trajectory meets all physical constraints

The table summarizes the scope of evidence of the above original text, not the pros and cons of the three systems. The task of VPT is to obtain game resources, LAPA includes simulation and real desktop operations, and HuRo mainly uses two-arm smart robots. The starting point model, action interface, data division, interactive budget and evaluation object are all different; even if the percentages are the same, the denominator is not the same task.

An easy-to-miss detail is the score definition. Among the main tasks of HuRo, some are scored according to the final success, and some are scored according to the completion of the sub-targets; Its out-of-distribution test mainly changes the spatial position and visual conditions, not equal to any new skill or new body. The LAPA real desktop experiment also adopts some of the success standards related to the task. These scores cannot be written as "the probability of complete success of all tasks". HuRo Appendix A、LAPA§4.1

The budget also needs to be dismantled. The number of HuRo non-zero data subsets is consistent with the batch, but the zero pre-training group eliminates the entire stage; it is not matched with the total training calculation. For the researchers, data scale effect, additional training effect and processing pipeline cost should be reported separately. HuRo§4.1–4.2

How to design an experiment that can really distinguish interpretation?

Evidence Framework Observed

The figure shows the evaluation framework proposed here. Solid lines trace how information reaches the policy and feeds into execution. Action signals and adaptation to the robot’s body can enter at different training stages. Dashed feedback lines show optional interactions that still need testing. Independent test results do not feed back into training. The actions and equipment are conceptual illustrations, not measured successes or reproductions of a paper’s findings.

Below is a research design that has not yet been performed. First fix the basic model, robot action data, optimize the budget and test tasks, and only change the video training objectives: still image characterization, temporal-sequential video, submersible movements at the correct timing, and pre-training with corresponding action labels. The control group must use the same video source to avoid mistakes in scene diversity as an action structure.

If the correct timing only improves object recognition, but does not improve the recovery after delay or disturbance, the visual characterization interpretation is more competitive. If it maintains a similar appearance, it is particularly sensitive to changes in action sequence and can help not see combination operations, then it is more supportive of the timing structure to participate in control. However, this result still cannot explain the specific internal algorithm, and the timing task may also bring other regularization effects.

Then use the body adaptation as a variable: the number of demonstrations of the fixed target robot, the trunk is compared, and the action head is only trained, and the entire model is updated. Gradually reduce the adaptation data, and draw the actual measured data efficiency curve. If the advantage only occurs after a large number of adaptations, the "video has provided an action interface that can be directly migrated" should be narrowed; if a small amount of adaptation is still stable, it is also necessary to rule out that the starting model has seen similar robot data.

Finally, the controlled disturbance is introduced: the position of the cup is changed, the slight sliding, and the short-term occlusion. The success, collision, recovery time and maximum number of retries are prescribed in advance, and the results of open cycle and continuous observation execution are recorded respectively. Compare adding videos, adding action demonstrations and adding interaction feedback, which resources can best solve the current failure. You can't just choose a successful video, and you can't use the model to evaluate "action is reasonable" instead of environmental judgment.

The data division should be isolated according to the original video and task instances, and the adjacent frames cannot be removed to both training and testing. The evaluator blindly tests the method identity, and preserves clear rules and actual annotations when it involves manual judgment; this paper does not write these unconducted work as completed experiments. Stop rules should also be determined in advance: if several sets of differences fall within the range of trial fluctuations, the evidence is insufficient, rather than adding selective display to maintain the conclusion.

Alternative relationships for resources should also be reported, not just the highest score. If the video scheme needs to train an additional expensive annotator, it should count the original data, labeler training, strategy training and robot test times respectively; fixed one of the costs cannot be declared that all costs are equal. In turn, there is a processing cost that does not deny the value of video, it may replace the collection of scarce entities with easier-to-expand offline calculations. The conclusion of the study should be clear which exchange is being tested.

The stronger counter-evidence is also worth writing in advance: if the benefits are basically reserved after disrupting the timing, the original explanation of the action structure needs to be conceded; if only the camera fails after the camera fails, the problem may be in the observation mapping, not the task sequence. Incorporating these opposite results into the plan can make the experiment have the ability to distinguish and interpret.

Current conclusion and summary

The existing evidence is sufficient to support a limited and useful conclusion: video can reduce the cost of learning control, but "no original action label" and "no action information" are not synonymous. The labeler training of the real label, the robot Fine-tuning after the submersible action, and the kinematic prior to the redirection may all be the key bridges. The three papers did not argue whether enough to watch under the exact same conditions; they show how different bridges are built.

For the researcher who is preparing to choose the topic, the next step should be to turn "understand" into a clear and measurable change: how many goals are used to demonstrate, what kind of disturbance to recover, what to reserve after changing the body, and what kind of failure to repair after adding feedback. Only when these questions are answered can we judge what observational learning is making up for, and which control abilities are still provided by physical constraints and actual interactions.

References

  1. Bowen Baker et al., Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videos,NeurIPS 2022. Published paper.
  2. Seonghyeon Ye et al., Latent Action Pretraining from Videos,ICLR 2025. Published paper.
  3. Jinho Jeong et al., HuRo: Robotizing Human Videos for Scalable VLA Pretraining, 2026, arXiv v5, preprint. Version history、Read the full text.

Apps by the author

The following is the author's own product promotion. In addition to blogging technology blogs, I also independently develop some small tools to help organize daily life, welcome to understand according to your own needs:

  • Daily Expense Journal: Record daily income and expenditure, arrange budget and savings goals, and make accounts and plans more organized. Learn about it in the App Store.
  • One Small Action: Choose small actions according to the current time and energy, record the feelings after completion, and leave clues for the daily review. Learn about it in the App Store.

For more technical articles and personal works, welcome to visit my Personal website.