
Date: 2026-06-28
Suitable for readers: graduate students, Agent/multimodal researcher, engineering reader of automation products or robotics systems
Search date: 2026-06-28
Abstract
In the past two years, the Multimodal Agent has been quickly advanced from "handing screenshots to VLM, and then let LLM decide where to click" to "cross-platform GUI Action Model, Computer-Use Skill BASE, and a VLA that can connect digital interface and physical control. In 2024, OSWorld, WebVoyager, and AndroidWorld define the problem clearly: the real environment will change, the task is often cross-application, and the evaluation must look at the final state. 2025-2026 UI-TARS, Magma, CUA-Skill, Windows World, MacArena, OmniGUI, MacAgentBench, Multimodal event stream and more realistic benchmark.
This paper sorts out the paper to represent the paper for nearly a year according to the technical route, and refines the engineering inspiration: Multimodal Agent The difficulty is not just "uncomprehensible", but a system problem composed of action space, long-term state, error recovery, permission boundary, environmental reproduction and auditable execution.
Contents
- research background
- roadmap for the past year
- Key papers by research question
- Comparing the methods
- Technology Trends and Engineering Inspiration
- Limitations and open questions
- Summary
- References
research background
Traditional LLM Agents often abstract environments into text APIs, HTML DOM, or function calls to schema. This is suitable for research planning and tool calls, but there is still a distance from real use: users are facing screens, pop-ups, window switches, file system, notifications, permission confirmation, video, voice and occasional errors. The core change of Multimodal Agent is to allow the model to directly or indirectly process visual observation and output actions that can change the environment.
This direction can be divided into two worlds that are gradually approaching. The digital world includes Web, mobile, desktop OS, office software and development tools, and the inputs are screenshots, OCR, accessibility trees, HTML, video frames or operation history, and actions are click, input, scroll, shortcut and file operations. The physical world includes robots and embodied AI, inputs are cameras, sensors and language commands, and actions are discrete skills or continuous control. The common problem of the two is that the model must connect multimodal perception, task memory, action selection and environment feedback into a closed loop.
Basic papers are still important. Mind2Web promotes the data and evaluation of Web Generalist Agent; Seeeclick emphasizes that GUI Grounding is the key bottleneck of visual GUI Agent; WebVoyager uses real website to verify end-to-end vision Web Agent; OSWorld Makes "Open Tasks in Real Computer Environments" as reproducible Benchmark. Together, they push the problem from the "Generate Script" to "Reliable to complete tasks in a GUI environment where the state changes".
roadmap for the past year

The route in the past year can be summarized as four main lines.
First, the GUI Agent has shifted from Prompt Workflow to a specially trained Action Model. OS-ATLAS, ShowUI, Aguvis, UI-TARS are all trying to model the unified visual grounding and action modeling across desktop, Web, and mobile. UI-TARS put GUI Screenshot, large-scale action traces, unified action space, system two reasoning and online reflection into the Native GUI Agent framework, which represents the route of "the model itself learns the operation interface".
Second, screen parsing becomes a practical intermediate layer. OmniParser disassembles the interactive area detection and semantic description, helping the upper-level VLM/Agent to more stably fall into the coordinates. For engineering teams, interpretable visual frontends are generally easier to debug, audit and downgrade than fully black-box end-to-end agents.
Third, Benchmark moves from static screenshots to cross-application, long process and multimodal events. WindowsWorld emphasizes professional cross-application process; MacArena and MacAgentBench explain that macOS is not a simple variant of ordinary desktop GUI; OmniGUI pushes mobile phone environment from static screenshots to image, audio, and video interweaving step-level review.
Fourth, the frame VLA extends the "action" from clicking coordinates to continuous control. OpenVLA promotes open VLA research, π0 uses flow matching to model continuous actions, Magma tries to put UI navigation and Robot manipulation into the same multimodal Agent The base, Gemini Robotics, represents the expansion of large-model manufacturers to physical actions.
Key papers by research question
Web and Desktop: From Real Web Pages to Real Operating Systems
The value of WebVoyager is to choose a real website, not just run tasks in a simplified web environment. OSWorld further overrides real computer tasks, including file I/O, multi-application process, and execution checks. The paper reports that the human completion rate is significantly higher than the strongest model at that time, indicating that GUI GROUNDING, operational knowledge and long-distance recovery are still the core shortcomings.
WindowsWorld in 2026 pushes the difficulty to the career scenario: tasks usually contain multiple sub-objectives, and most of them need to be done across applications. MacArena and MacAgentBench remind researchers not to extrapolate the results on one system to all desktop environments. The menu, window, permissions, shortcut keys and application ecology of macOS have unique constraints, and the model sorting may also change with the subset of tasks.
Mobile Proxy: Navigate to multimodal Event Streams from Single App
Mobile-Agent try not to rely on XML or system metadata, but use visual perception knowledge controls; Mobile-Agent-v2 introduces Planning, Decision, Reflection The multi-agent structure handles task progress and error recovery. AndroidWorld provides dynamic Android tasks and programmatic Reward, while GUI Odyssey emphasizes cross-app navigation.
The new idea of OmniGUI is Omni-Modal Smartphone Environments: Each action step is not only for static pictures, but also audio and video clips. This is very close to the real use of mobile phones, because many of the states come from short-term Toast, player changes, notification sounds or dynamic screens. Only handing the screenshot to the model will naturally throw away the information.
GUI Action Model: from "where do you" to "how to do it continuously"
SeeClick and evaluations such as ScreenSpot show why grounding, or locating the intended element on screen, is a first hurdle for GUI agents. OS-ATLAS uses cross-platform GUI grounding data and a foundation action model to improve generalization in open-source models. ShowUI selects visual tokens using the UI structure to reduce redundancy. Aguvis emphasizes vision-only input, autonomous GUI interaction, and open training recipes.
UI-TARS is closer to the full Native GUI Agent: it uses a large number of GUI Action Trace to learn the mapping to the action, and enhances continuous decision-making with a reflective trajectory. It should be noted that the Benchmark number of the paper will be affected by the model version, the tool stack, step limit and UI updates. This article does not regard a single leaderboard result as a long-term SOTA conclusion.
Skill Library and Body VLA
CUA-Skill pairs reusable skills with an execution graph. A model can retrieve a skill and supply its parameters instead of reconstructing an Office, browser, or system settings workflow from a screenshot each time. This matters in enterprise automation, where teams need stable sequences of actions that can be audited and rolled back, with robotic process automation (RPA) available as a fallback.
The body direction extends Action from discrete UI operations to robot control. RT-2 is the classic foundation: express the robot action as a token, and let the VLM share the language and action generation paradigm. OpenVLA lowers the reproduction threshold; π0 outputs continuous actions with Flow Matching; Magma combines Set-of-Mark and Trace-of-Mark to cover UI, video and robot tasks; Gemini Robotics Displays the general multimodal model is entering the physical action scene.
Comparing the methods
| Route | Representative work | main input | Main action | Engineering heuristics |
|---|---|---|---|---|
| Web/Desktop Agent | WebVoyager, OSWorld, WindowsWorld, MacArena | Screenshots, web pages, OS status, historical trajectories | Click, enter, shortcut keys, file operation | Sandbox, status reset and executable acceptance must be done |
| mobile agent | Mobile-Agent, AndroidWorld, GUI Odyssey, OmniGUI | Screenshots, OCR, Audio/Video, Operation History | Click, swipe, enter, App switch | The static screenshot is insufficient, and the event flow and the device status are required |
| GUI action model | SeeClick, OS-ATLAS, ShowUI, Aguvis, UI-TARS | UI Screenshot, element semantics, action trajectory | Coordinates, control selection, action token | Data quality and unified action space are more critical than Prompt |
| Screen parsing | OmniParser | Screenshots, icons, text, interactive area | structured UI elements | Suitable for hybrid deployment with closed/open source VLM |
| Skills library | CUA-Skill | GUI status, task, skill context | Parametric skills, execution diagram | Limit model output to auditable boundaries |
| Framework VLA | RT-2, OpenVLA, π0, Magma | Image/Video, Language, Robot Status | Discrete or continuous robot action | Real security, latency and data closed loops are harder than demo |
Technology Trends and Engineering Inspiration
First, the action space is being unified. GUI Agent needs to abstract click, drag, input, shortcut keys, and App switching into cross-platform actions; VLA needs to abstract the end effector, joint trajectory or high-level skills into a learnable output. The unified action space determines whether the data can be mixed training, whether the error can be reproduced, and whether the strategy can be migrated.
Second, the visual front end is modular. The screen is not an ordinary natural picture, the UI has hierarchies, boundaries, clickable areas and operational semantics. OmniParser, ShowUI, SeeClick, OS-ATLAS all indicate that writing these Inductive Bias into data and models is usually more effective than a heap prompt.
Third, the evaluation is interacting. OSWorld, AndroidWorld, WindowsWorld, MacArena, and OmniGUI all require the agent to change the environment status and check the results with the final state or programmatic rules. This forces the system to handle load delays, error clicks, permission pop-ups, task retries, and intermediate failures.
Fourth, skills, memory and reflection will become the necessary system layer. Long task failure is often not the first step to misunderstand, but subsequently forget the target, the state is out of sync, or the recovery policy error is restored. In the project, the layered structure of "visual analysis layer + planning model + skill execution layer + audit log" is recommended: the model can propose a plan, and the key action should go to controlled skills, and set manual confirmation for payment, sending letters, deleting files, and changing permissions.
Fifth, data closed loops are more important than framework selection. GUI GROUNDING data, real operation trajectory, failed trajectory, cross-application process, audio and video events and robot Demonstrations are all scarce assets. If the team has real scenarios, it should give priority to building task sets, playback environments, failure sample libraries and acceptance scripts, rather than frequently replacing the Agent framework.
Limitations and open questions
First, the Benchmark is still fragmented. Different platforms, application ecology, task distribution and tool interfaces will change the model performance, and a leaderboard leader cannot prove the ability of general-purpose computers.
Second, many results depend on the closed-source model version, step limit, tool stack and UI state. Model names, API versions, web page layouts, system permissions, and app updates all affect reproduction. This article only retains the information that can be checked in the source; the online selection should still run the local task set again.
Third, security issues can be trickier than text agents. GUI Agent not only "wrong" but also "points wrong". Web, email, pictures, documents, notifications, and pop-ups can all be indirect injection or misleading sources, so permissions, confirmations, logs, and rollbacks must be pre-designed.
Fourth, there is still a distance between the DEMO that carries the VLA and the production robot. Continuous control requires low latency, physical safety, sensor reliability and abnormal stopping mechanisms. It is important to generalize across embodiment, but it still needs to be rigorously verified in real factories, homes and medical scenarios.
Summary
The latest method of Multimodal Agent is forming a clear picture: the model must understand the screen and the world, but also convert the intention into an executable action; the system must be able to evaluate, roll back, audit and restore; the data should cover real trajectories, failure cases and cross-platform differences. The most reliable short-term landing method is not a completely independent agency, but a restricted task, strong environmental control, a reusable skill library and manual confirmation. In the long run, the GUI Action Model and the Framework VLA may merge, forming a common Multimodal Agent that can share perception, memory, planning and action capabilities in digital interface and physical space.
References
Search date: 2026-06-28. Prioritize the use of arXiv, project homepage, official blog or official documentation; the Benchmark number and model release time are subject to the linked page.
- OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments. arXiv, 2024-04-11. https://arxiv.org/abs/2404.07972
- WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models. arXiv, 2024-01-25. https://arxiv.org/abs/2401.13919
- SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents. arXiv, 2024-01-17. https://arxiv.org/abs/2401.10935
- Mind2Web: Towards a Generalist Agent for the Web. arXiv, 2023-06-09. https://arxiv.org/abs/2306.06070
- AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents. arXiv, 2024-05-23. https://arxiv.org/abs/2405.14573
- Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perception. arXiv, 2024-01-29. https://arxiv.org/abs/2401.16158
- Mobile-Agent-v2: Mobile Device Operation Assistant with Effective Navigation via Multi-Agent Collaboration. arXiv, 2024-06-03. https://arxiv.org/abs/2406.01014
- GUI Odyssey: A Comprehensive Dataset for Cross-App GUI Navigation on Mobile Devices. arXiv, 2024-06-12. https://arxiv.org/abs/2406.08451
- OmniParser for Pure Vision Based GUI Agent. arXiv, 2024-08-01. https://arxiv.org/abs/2408.00203
- OS-ATLAS: A Foundation Action Model for Generalist GUI Agents. arXiv, 2024-10-30. https://arxiv.org/abs/2410.23218
- ShowUI: One Vision-Language-Action Model for GUI Visual Agent. arXiv, 2024-11-26. https://arxiv.org/abs/2411.17465
- Aguvis: Unified Pure Vision Agents for Autonomous GUI Interaction. arXiv, 2024-12-05. https://arxiv.org/abs/2412.04454
- UI-TARS: Pioneering Automated GUI Interaction with Native Agents. arXiv, 2025-01-21. https://arxiv.org/abs/2501.12326
- Magma: A Foundation Model for Multimodal AI Agents. arXiv, 2025-02-18. https://arxiv.org/abs/2502.13130
- CUA-Skill: Develop Skills for Computer Using Agent. arXiv, 2026-01-28. https://arxiv.org/abs/2601.21123
- WindowsWorld: A Process-Centric Benchmark of Autonomous GUI Agents in Professional Cross-Application Environments. arXiv, 2026-04-30. https://arxiv.org/abs/2604.27776
- OmniGUI: Benchmarking GUI Agents in Omni-Modal Smartphone Environments. arXiv, 2026. https://arxiv.org/abs/2605.18758
- MacArena: Benchmarking Computer Use Agents on an Online macOS Environment. arXiv, 2026-06-04. https://arxiv.org/abs/2606.06560
- MacAgentBench: Benchmarking AI Agents on Real-World macOS Desktop. arXiv, 2026-06-21. https://arxiv.org/abs/2606.22557
- RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control. arXiv, 2023-07-28. https://arxiv.org/abs/2307.15818
- OpenVLA: An Open-Source Vision-Language-Action Model. arXiv, 2024-06-13. https://arxiv.org/abs/2406.09246
- π0: A Vision-Language-Action Flow Model for General Robot Control. arXiv, 2024-10-31. https://arxiv.org/abs/2410.24164
- VLANeXt: Recipes for Building Strong VLA Models. arXiv, 2026-02-20. https://arxiv.org/abs/2602.18532
- OpenAI Operator research preview. OpenAI, 2025-01-23. https://openai.com/index/introducing-operator/
- Anthropic Computer Use announcement. Anthropic, 2024-10-22. https://www.anthropic.com/news/3-5-models-and-computer-use
- Gemini Robotics brings AI into the physical world. Google DeepMind, 2025-03-12. https://deepmind.google/discover/blog/gemini-robotics-brings-ai-into-the-physical-world/