Cover illustration

Date: 2026-08-10
Suitable for readers: graduate students who are ready to enter robot learning, multimodal agents, and body intelligence; engineering readers who are familiar with deep learning and LLM/VLM, but have not systematically read robot papers.
Search date: 2026-08-10

Abstract

The Vision-Language-Action (VLA) model puts visual observation, language tasks and robot movements into the same learning framework, which is one of the most intensive paper lines in the past two years. It looks like "multimodal LLM plus an action header", but the really hard to read is not in the model name, but in five details: whether the visual representation is sufficiently spatialized, how the language instructions are bound to the task state, and the action is discrete Token, continuous regression, diffusion or flow matching, whether the robot data is cross-embodiment, and whether the evaluation can really prove generalization.

This article gives an intensive reading route suitable for scientific research training: first read RT-1, PaLM-E, RT-2 and Open X-Embodiment, understand why "scale robot data + pre-trained multimodal model" appears; read again Octo, OpenVLA, π0, FAST, OpenVLA-OFT, SmolVLA, disassemble the action representation and reproduction formula; finally read Gemini Robotics, GR00T, VLA-Arena, VLA Continue to learn materials from 2025-2026 to determine where the real bottleneck in this direction is.

Contents

  1. Why VLA is a turning point in robot learning
  2. Five Lines of Intensive Reading of a VLA Paper
  3. A reading guide to the key papers
  4. How to read key papers
  5. Methods and experiments compared
  6. How to reproduce the results
  7. Common misconceptions
  8. Research questions for graduate students
  9. Summary
  10. References

Why is this topic important

Traditional robot learning papers are often split into three modules: perception, planning, and control: the visual model recognizes objects, the planner generates sub-objectives, and the controller outputs actions. This paradigm is clear, but the errors of each module will be passed downstream, and every time a robot, camera perspective, end effector or task collection is changed, a lot of project details must be re-adjusted.

VLA's ambition is to rewrite this matter into an end-to-end learning problem: input images, language instructions and robot states, and output the next step or a future action. RT-2 Converts the Vision-Language Model into an action model, Open X-Embodiment unifies the data of different laboratories and robots to a trainable format, OpenVLA has the Open weight and PyTorch The training pipeline is pushed in front of the community, π0 uses Flow Matching to handle high-frequency continuous actions, FAST treats the action tokenization as a timing compression problem, and Gemini Robotics and GR00T use VLA Push towards stronger multistep reasoning, humanoid and on-device morphology.

But for graduate students, this direction is also easy to misread. First, VLA is not as simple as the "Arch of the VLM is connected to the robotic arm API". Low-level actions meet control frequency, smoothness, delay and safety constraints, and the action space is more fragile than a text token. Second, the Benchmark score usually depends on the task suite, camera perspective, initial state, controller, whether it stops early, whether it is fine-tuning, and whether the demonstration data overlaps. Third, the most valuable part of the paper is often not the slogan of the large model, but the data standardization, action coding, action chunk, post-training and evaluation protocols.

Core reading method: five lines to dismantle one VLA paper

VLA reading roadmap

When reading a VLA paper, do not start with the model name. It is recommended to take notes in each of the following five lines.

1. Perception Backbone: where does the visual representation come from

It is used to record the pretrained VLM such as the visual transformer, DINOv2/SigLIP/PaliGemma/Gemini/Eagle/Qwen-VL, etc., or only frozen features. Ask: Is the input a single perspective or a multi-view angle? Is there WRIST CAMERA? Do you use historical frames? How does the image resolution and token number affect the delay? If the paper claims "generalization", how much does the visual backbone contribute, and how much does the robot data contribute?

2. Language / Task Conditioning: what exactly does Language control

Language may be just a task tag, or it may be responsible for object reference, spatial relationship, sub-object planning and online error correction. RT-2 Emphasizes the migrating of semantic knowledge from Web-Scale VLM; OpenVLA and SmolVLA emphasize operational strategies under language conditions; Gemini Robotics 1.5 further emphasizes agent planning and thinking. When reading the paper, it should be marked: is the language supervised during training, or the at inference time interaction? Does the instruction of the evaluation really cover the generalization of combinations?

3. Action Representation: How the Action is generated by the model

This is the core of the VLA paper. RT-2 Discrete the action into token; OpenVLA continues the output of the discrete action; Octo uses the diffusion-style action head; π0 uses flow matching to generate continuous actions; FAST Use the discrete cosine to transform the compression action sequence and re-token; OpenVLA-OFT changes the fine-tuning recipe to parallel decoding, action chunking, continuous action and L1 target. The action representation determines the control frequency, error accumulation, training stability and difficulty of reproducing.

4. Robot Data: Does the Data really span embodiment

Open X-Embodiment brings together 22 robotic forms, more than 500 skills and more than 1 million Episode, which is a key infrastructure for this round of general-purpose robot strategy. OpenVLA reports using 970K real robot demonstrations from Open X-Embodiment; Octo uses 800K TraJectories; π0/OpenPI Provides pre-trained models for large-scale robotic data; SmolVLA reverses the emphasis on community data, low-cost hardware and smaller scale reproducibility. When reading the data part, don't just memorize "how many trajectories", but also remember the data format, action space, camera, task text, collector, cleaning rules and license.

5. Evaluation & Reproduction: Whether the generalization evidence is reliable

VLA evaluations can give a misleading impression. LIBERO, CALVIN, SimplerEnv, BridgeData, DROID, and real-world tasks on ALOHA, Franka, or humanoid robots differ in difficulty. Even on the same benchmark, papers may change the demonstrations, early stopping rules, action chunks, or maximum step count, or report only some tasks. Treat a claim of state-of-the-art performance as provisional if the authors have not released the evaluation scripts, initial states, failure cases, and results across multiple seeds.

A reading guide to the key papers

Phase 1: From Multitasking Robot Strategy to Incorporating multimodal Models

RT-1 (2022) is an important starting point for large-scale real robot control: use a large number of real robots to demonstrate training Transformer policy, so that the model can directly control the robot under language instructions. PaLM-E (2023) "informs" the large language model, so that the language model receives inputs such as images, robot states, etc., and outputs the intermediate results of planned or text form. It is not the VLA that directly outputs the underlying action in the sense of today, but it has turned "Whether multimodal pre-training can help robot tasks" become a serious question.

Stage 2: VLM to VLA

RT-2 (2023) clearly proposes the vision-language-action model: using the vision-language model such as PaLI-X and PaLM-E as the basis, the robot action is expressed as Token, let the model directly generate action sequences from images and languages. Open X-Embodiment appeared in the same year as RT-X, emphasizing cross-robot data hybrid training. The key question at this stage is: Can Web-Scale visual language knowledge be migrated to real control? Can the action space of different embodiment be unified? Is data diversity more important than the number of tasks on a single robot?

Stage 3: Open Model and Reproducible Training

Octo (2024) and OpenVLA (2024) allow the community to Open the Generalist Robot Policy for the first time. The value of Octo is much smaller, can fine-tuning, and emphasizes the general policy initialization; the value of OpenVLA is in the 7b level Open VLA, PyTorch/FSDP/LoRA/Quantization training and deployment paths, and engineering support for the mix of Open X-Embodiment data. For research training, OpenVLA's README, LIBERO review scripts and Troubleshooting are more worth reading than the abstract of the paper.

Stage 4: Continuous Action, Flow Matching and High Frequency Control

π0 (2024) pairs a pretrained vision-language model with a flow-matching action expert to generate chunks of high-frequency continuous actions for more complex dexterous manipulation. FAST (2025) shows that naïvely discretizing each action dimension at each time step can work poorly on high-frequency dexterous manipulation data. It recasts action tokenization as frequency-domain compression and releases the FAST+ tokenizer. OpenVLA-OFT (2025) shows how much the fine-tuning recipe itself can change speed and success rates. Choices such as parallel decoding, action chunking, continuous actions, and an L1 objective may matter more than a larger backbone.

Stage 5: Robotics Foundation Models for 2025-2026

Gemini Robotics (2025) brings Gemini 2.0/2.5 multimodal reasoning to physical action, and the official data emphasizes interactivity, generalization, dexterity, on-device Low latency and a small amount of Demonstrations adaptation; however, the main model and SDK are in the Private Preview/Trusted Tester state, and the access limit should be marked when reappearing. NVIDIA GR00T N1 (2025) and subsequent N1.5/N1.7 push VLA to humanoid, the code base has N1.7 version, several checkpoints by 2026 and deployment instructions. SmolVLA (2025) represents another important route: smaller models, lower hardware thresholds, community data, and asynchronous reasoning. The 2026 VLA continuous learning paper begins to ask whether the model will forget old skills when learning new tasks in real scenarios.

A closer look at key papers

RT-1: read data and control interface first

The focus of RT-1 is not "Transformer wins again", but it extends the training unit of robot learning from single-task strategy to large-scale multitasking real robot data. When intensive reading, it is necessary to draw: what the observation includes, how the language instruction enters the model, how to define the action dimension, what is the control frequency, what kind of environment the training data comes from, and how to distinguish the see/unseen task. Don't start pursuing a complete real robot experiment at the beginning of reproduction, you can first reproduce the data format, Action Normalization and offline Action Prediction.

PaLM-E and RT-2: Differentiating the Planning Model and Action Model

PaLM-E is more like a multimodal reasoner, often cooperates with low-level policies; RT-2 uses VLM fine-tune into a model that can directly output action tokens. When reading these two articles, you should mark "what is the output": text plan, sub-task, action token, or continuous control amount. Many beginners mix embodied LLM, VLM Planner, and VLA policy together, resulting in a confused Work structure.

Open X-Embodiment / RT-X: read the dataset as a paper

This is not an ordinary DataSet Paper. Its key is RLDS Episode format, cross-lab data integration, robot embodiment differences, Task Annotation and RT-1-X/RT-2-X cross-data training. It is recommended to make a data map during intensive reading: the robot, camera, action space, task number, license of each data source, and whether it is suitable for you to reproduce the hardware. Don't just quote macro numbers like "over 1 million episodes".

Octo and OpenVLA: Two Values of Open Baseline

The advantages of Octo are lighter, more open, and easy to use the generalist policy as initialization; the advantage of OpenVLA is that the VLM Backbone is stronger and the open model is closer to LLM/VLM Ecological, fine-tuning instructions are more complete. When reading OpenVLA, it is especially important to see how it handles Prismatic VLM, DINOv2/SigLIP features, Llama 2, action token, LoRA, LIBERO and BridgeData V2. For scientific research and training, it is recommended to use OpenVLA as the "main line that can run", and use Octo as "contrast of architecture and data ablation".

π0, FAST, OpenVLA-OFT: read through the action representation

The contribution of π0 is not just a "stronger robot model", but further pushes action generation from discrete token to continuous trajectory generation; the contribution of FAST is to position the bottleneck of discrete VLA to action Tokenizer; the contribution of OpenVLA-OFT is to prove that the FINE-TUNING formula can greatly affect the inference speed, frequency and task success. Read these three articles to focus on: action chunk length, output space, loss function, whether autoregressive, whether inference is parallel, whether multi-view and proprioception are supported.

Gemini Robotics, GR00T, SmolVLA: Be vigilant when reading official materials

The official pages of Gemini Robotics and GR00T are very important, because the model status, access, model card, checkpoint, SDK and version number are updated quickly. When reading such materials, separate "the paper can be reproduced" and the "official demo ability description". SmolVLA is especially suitable for graduate students: it reduces the model to 450M level, emphasizing single GPU/consumption-grade equipment, LeRobot data format, asynchronous reasoning and community data, which is a realistic entry for course projects or reproduction training.

Methods and experiments compared

Papers/System Time Key issue action expression Data and Evaluation Clues Reproduction difficulty
RT-1 2022 Multitasking real robot control can scale Discrete Action / Transformer Policy Google real robot kitchen data, official open source model and data cues High
PaLM-E 2023 How a large language model receives body input Main output text/plan, often followed by low-level policies Multi-robot, multimodal task High
RT-2 2023 Can VLM be converted into VLA for direct control of the robot? action token RT-1 Data + Web/VLM Knowledge Migration High
Open X-Embodiment / RT-X 2023 Can cross-robot data be trained uniformly? Depends on the model 22 Robot Embodiments, 500+ Skills, 1 million + Episodes Medium to high
Diffusion Policy 2023 How to model multimodal actions in a continuous visualomotor policy diffusion action generation 11 Tasks, 4 Benchmarks, the code and logs are more complete In
Octo 2024 Open Generalist Robot Policy Initialization transformer + diffusion-style action head 800K OXE Trajectories, 9 Platform Fine-Tuning In
OpenVLA 2024 Open 7B VLA and Fine-Tuning Eco discrete action token 970k robot demonstrations,LIBERO/BridgeData Medium to high
π0 / openpi 2024-2025 Can VLM + Flow Matching do high-frequency Dexterous control? Continuous flow matching action chunk Multi-platform Dexterous data, OpenPI model and code Medium to high
FAST 2025 Whether the action tokenization is restricted AR VLA DCT/Frequency Domain Compression Action Token FAST+ tokenizer,1M robot action sequences In
OpenVLA-OFT 2025 How to improve the speed and success rate of fine-tuning formula Continuous action, L1, action chunk LIBERO and ALOHA, the project page gives hardware requirements In
SmolVLA 2025 Can small models and community data reproduce VLA flow matching action expert LeRobot community data, SO100/SO101, LIBERO Low to medium
Gemini Robotics 2025 Agent VLA, On-Device and Interactive Control Officially not fully open details, output action Private Preview/Trusted Testing, Official Model Cards and Technical Reports High
GR00T N1.x 2025-2026 humanoid generalist policy VLM + diffusion transformer / action head GitHub, Model Card, LIBERO/DROID/SimplerEnv Checkpoint Clue Medium to high

Suggestions for reappearance: start with being able to run, don't start with real robots

The first step is to choose an open and scripted stack. Graduate training recommendations start with LeRobot + LIBERO/Meta-World, LIBERO scripts for OpenVLA, SmolVLA or GR00T's public checkpoint. The reproduction of the real robotic arm is not the first step, because hardware errors, camera calibration, delay, controller and safety constraints mask the model problem.

The second step is to fix the evaluation protocol. Record random seed, task suite, initialization status, maximum number of steps, successful determination, whether early stop, number of rollouts per task, and whether to use the same Demonstrations. VLA results are easily derived from the protocol, not the model itself.

The third step is to do offline sanity check first. Including whether Action Normalization is correct, whether the sequence of action dimensions is consistent with the environment, whether the opening and closing direction of Gripper is reversed, whether the image channel and camera perspective are matched, and whether the language prompt is consistent with the training format. Many "models don't work" are actually misaligned interfaces.

The fourth step is to change only one variable. Action representation, backbone, data mixture, number of viewing angles, action chunk, fine-tuning steps, LoRA rank, and learning rate can all affect the results. When doing ablation, do not change data, model and review scripts at the same time.

The fifth step is to save the failure trajectory. The research value of the VLA direction usually comes from the failure mode: misunderstanding the object, understanding but unable to grasp, short-range success and long-range failure, vulnerable to no background, and forgetting old tasks after continuous learning. Failed video, action traces and per-task logs are more scientific than an average success rate.

Common misconceptions

Myth 1: Read the VLA paper as a Multimodal LLM paper.
The key to VLA is not answering questions, but closed-loop control. Delay, control frequency, smooth action, and error recovery are part of the quality of the paper.

Misunderstanding 2: Just look at the average score of the Benchmark.
LIBERO's average score is not equal to the strong real robot; the real robot demo does not mean the protocol is strict; the success of the model on an embodiment is not equal to cross-bot generalization.

Misunderstanding 3: Ignore the data format.
Observation/action/state/schema of robot data often determines the success or failure of the model structure more than the model structure. The format differences of RLDS, LeRobot DataSet, DROID, OXE, and LIBERO must be sorted separately.

Misunderstanding 4: Take the action token as an ordinary text token.
The text token typo may still be restored, and the action token will make the robot deviate. Action discretization, frequency domain compression, continuous output, and chunk stitching are all worthy of intensive reading.

Misunderstanding 5: Take the official demo as evidence of the paper.
The official page can prompt the direction, but the scientific research citation must go back to the paper, model card, code, review script and reproducible protocol. Access restricted, unopened weights, and private Benchmark should be clearly written.

Research questions for graduate students

  1. Controlled comparison of VLA actions: Compare discrete token, FAST tokenizer, l1 continuous head, diffusion/flow matching head, control backbone under the same LIBERO/SimplerEnv settings and the data remain unchanged.
  2. VLA Failed Mode Classifier: The reason for automatic annotation of video and logs from Rollout, such as Grounding Error, Grasp Failure, Recovery Failure, Instruction Drift, Action Saturation.
  3. Small data post-training rule: The system compares the impact of Demonstrations on different VLAs on 10/50/100/500, and pays attention to when overfitting and catastrophic forgetting appear.
  4. Across the Embodiment Adapter: Research how to migrate the same VLA to a different action space with a small amount of calibration data, instead of retraining the full model for each robot.
  5. VLA Evaluation protocol Audit: Review the task definition of the public leaderboard, rollout number, early stop, data overlap and failure report to form a reproducibility card.
  6. VLA control under security constraints: Jointly evaluate Semantic Security, Workspace Constraints, Low-Level Controller and VLA Action Head to avoid only text layer abstention .

Summary

The main line of the VLA paper route is not "the robot finally has a large model", but: robot learning begins to pursue pre-training, data mixing, post-training, open weight and standardized evaluation like LLM; at the same time, it is still firmly restricted by the low-level constraints of the physical world. To read this direction, you must have both the Multimodal model perspective and the robot experiment perspective.

If you only choose three intensive readings, the recommended order is Open X-Embodiment/RT-X, OpenVLA, π0/FAST. The Open script starts; if you want to choose a topic, give priority to the movement representation, evaluation reliability, training and continuous learning after small data. The direction of the VLA is far from convergence, and the really good research problems are often hidden in the place where "the paper demo is very strong, but it fails with a robot, a perspective, and a task agreement will fail".

References

The following information is searched on 2026-08-10. The model version, code repository status, benchmark results, and private preview access status may change, and manual verification needs to be done again before reproducing the experiment.

  1. RT-1: Robotics Transformer for Real-World Control at Scale, arXiv:2212.06817 / project page: https://robotics-transformer1.github.io/
  2. Google Research Blog, “RT-1: Robotics Transformer for real-world control at scale”, 2022-12-13: https://research.google/blog/rt-1-robotics-transformer-for-real-world-control-at-scale/
  3. PaLM-E: An Embodied Multimodal Language Model, Google Research Blog, 2023-03-10: https://research.google/blog/palm-e-an-embodied-multimodal-language-model/
  4. RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control / Google DeepMind Blog, 2023-07-28: https://deepmind.google/blog/rt-2-new-model-translates-vision-and-language-into-action/
  5. Open X-Embodiment: Robotic Learning Datasets and RT-X Models / Google DeepMind Blog, 2023-10-03: https://deepmind.google/blog/scaling-up-learning-across-many-different-robot-types/
  6. Open X-Embodiment GitHub repository: https://github.com/google-deepmind/open_x_embodiment
  7. Diffusion Policy: Visuomotor Policy Learning via Action Diffusion, arXiv:2303.04137: https://arxiv.org/abs/2303.04137
  8. Diffusion Policy official code: https://github.com/real-stanford/diffusion_policy
  9. Octo: An Open-Source Generalist Robot Policy, arXiv:2405.12213: https://arxiv.org/abs/2405.12213
  10. OpenVLA: An Open-Source Vision-Language-Action Model, arXiv:2406.09246 / project: https://openvla.github.io/
  11. OpenVLA GitHub repository: https://github.com/openvla/openvla
  12. π0: A Vision-Language-Action Flow Model for General Robot Control, arXiv:2410.24164 / Physical Intelligence blog: https://www.physicalintelligence.company/blog/pi0
  13. Physical Intelligence openpi repository: https://github.com/Physical-Intelligence/openpi
  14. FAST: Efficient Action Tokenization for Vision-Language-Action Models, arXiv:2501.09747 / Hugging Face model card: https://huggingface.co/physical-intelligence/fast
  15. Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Success, arXiv:2502.19645 / project: https://openvla-oft.github.io/
  16. SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Robotics, arXiv:2506.01844 / Hugging Face blog: https://huggingface.co/blog/smolvla
  17. LIBERO benchmark project and code: https://libero-project.github.io/main.html and https://github.com/Lifelong-Robot-Learning/LIBERO
  18. DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset, arXiv:2403.12945 / repository: https://github.com/droid-dataset/droid
  19. Google DeepMind, “Gemini Robotics brings AI into the physical world”, 2025-03-12: https://deepmind.google/blog/gemini-robotics-brings-ai-into-the-physical-world/
  20. Google DeepMind, “Gemini Robotics On-Device brings AI to local robotic devices”, 2025-06-24: https://deepmind.google/blog/gemini-robotics-on-device-brings-ai-to-local-robotic-devices/
  21. Gemini Robotics model page and model cards: https://deepmind.google/models/gemini-robotics/gemini-robotics/ and https://deepmind.google/models/model-cards/
  22. NVIDIA GR00T N1: An Open Foundation Model for Generalist Humanoid Robots, arXiv:2503.14734 / research page: https://research.nvidia.com/labs/lpr/publication/gr00tn1_2025/
  23. NVIDIA Isaac-GR00T GitHub repository, checked latest visible N1.7 materials: https://github.com/NVIDIA/Isaac-GR00T
  24. GR00T N1.5 research page, 2025-06-11: https://research.nvidia.com/labs/gear/gr00t-n1_5/
  25. A Survey on Vision-Language-Action Models for Embodied AI, arXiv:2405.14093, v8 revised 2026-05-01: https://arxiv.org/abs/2405.14093
  26. Towards Generalist Robot Policies: What Matters in Building Vision-Language-Action Models, arXiv:2412.14058 / project: https://robovlms.github.io/
  27. VLA-Arena benchmark project: https://vla-arena.github.io/
  28. Can VLA Models Learn from Real-World Data Continually without Forgetting?, arXiv:2605.26820: https://arxiv.org/abs/2605.26820
  29. AtomVLA: Scalable Post-Training for Robotic Manipulation via Predictive Latent World Models, arXiv:2603.08519: https://arxiv.org/abs/2603.08519