
Date:2026-07-07
Who this is for: Graduate students, researcher in software engineering and AI systems, technical readers with engineering background
search date:2026-07-07
Abstract
Before 2024, the main battlefield of the code mockup was mainly to "complete a function", "pass several unit tests" and "write answers on competition questions". From 2024 to 2026, the research focus has obviously turned to Software Engineering Agents: the model no longer only outputs code snippets, but reads real repositories and positioning BUG, edit multiple files, run tests, fail to interpret, submit patches, and even participate in the Issue, Pull request and code review processes in an asynchronous manner. SWE-bench, SWE-agent, AutoCodeRover, OpenHands, Agentless, SWE-Lancer, Multi-SWE-bench, SWE-smith, Terminal-Bench Work together has pushed this turn.
The core judgment of this article is that the progress of the software engineering Agent is not only "models can write code", but a co-evolution of models, repository contexts, tool interfaces, sandbox execution, test Oracle, evaluation data and human review process. The bottleneck in the next stage will also be generated from single-point code, turning to long-distance task reliability, evaluation of pollution, cross-language ecology, authority governance, cost control and organization-level implementation.
Contents
- Research background: From code completion to repository-level tasks
- Roadmap in the past year: real tasks, long-distance tasks, organizational adoption
- Key papers by research question
- Comparing the methods
- Technology Trends: Positioning, Patch, Validation, Sandbox and Person Review
- Implications for deployment
- Limitations and open questions
- Questions to watch
- References
Research Background: Why Software Engineering Agents Become a System Problem
Code generation used to be a relatively clear NLP/PL task: Given a docstring or function signature, the model generates a function, and the hidden test is successful. Benchmarks such as HumanEval, MBPP, APPS, and CodeContests allow researchers to quickly compare the coding capabilities of the model. But that's not how real software development happens. Engineers face incomplete ISSUE description, historical design constraints, implicit style, cross-file dependencies, insufficient testing, CI environment differences, compatibility requirements and code review opinions.
The importance of SWE-bench is to pull the problem back from the "write code question" to the real GitHub issue. The model needs to modify the code in an existing Python repository, so that the problem from the real pull request is tested. The early model performed very weakly on SWE-bench, which is not accidental: the real repository task requires the model to understand the problem first, then locate the relevant files, then propose a minimum patch, and finally correct the error with test feedback. Every step may fail.
From this point of view, the research object of Software Engineering Agents is not a single LLM, but a set of closed-loop systems:
- context system: How to index repositories, retrieve relevant files, compress historical information, and avoid tucking the entire repository into the context window.
- tool interface: How to make the model safe to use shell, editor, test framework, static analyzer, package manager and version control.
- task cycle: How to plan, locate, modify, run tests, interpret failures and roll back.
- evaluation system: How to construct a Benchmark that is not polluted by the Training Set, can automatically perform, and is close to the value of the real work.
- boundary of cooperation: When it is autonomously advanced by the model, and when it must be handed over to human review, product decision or security approval.
Roadmap in the past year: from "can repair bugs" to "can participate in engineering organizations"
The key nodes of 2024 are SWE-bench, SWE-agent, AutoCodeRover, Agentless, MASAI and OpenHands. This year, I mainly answer two questions: Can the real repository task be standardized for evaluation? Is Agent Scaffold really necessary? The results are interesting: SWE-agent emphasizes Agent-Computer Interface, AutoCodeRover emphasizes structured code search and fault localization, Agentless It is pointed out that a simple "location-repair-validation" pipeline can become a strong baseline. That is, complex agents are not natural victories, and good context engineering is equally important as the verification process.
Changes in 2025 are more like “Industrialization of Data and Evaluation”. Multi-SWE-bench extends issue resolving to Java, TypeScript, JavaScript, Go, Rust, C, C++ and other languages; SWE-smith Attempts to automatically synthesize large-scale training tasks; SWE-rebench and SWE-Bench++ begin to focus on continuously extracting new tasks, reducing pollution, expanding language and repository coverage; SWE-Lancer maps tasks to real The value of freelance software engineering work, reminding everyone that "solving Benchmark" does not mean "creating economic value".
As of this search in 2026, the research continues to extend in two directions. The first is general-purpose terminal/long-range task evaluation, such as Terminal-Bench and TUA-Bench, which put the software engineering Agent into a wider command-line environment for inspection. The second is organizational adoption research, such as large-scale data sets and adoption research on Agent-authored PR on GitHub, and empirical research on Rollout by command line coding agents. These 2026 papers and product information are still in a period of rapid change, and this article only uses them as a trend signal, and the specific figures need to be manually reviewed before publication.
Key papers by research question
1. Repository-level evaluation: SWE-bench family
SWE-bench provides a clear paradigm: given the issue and repository snapshots, the model must submit a patch, and the test determines whether the problem is solved. It makes "real GitHub issue" a repeatable object. Subsequently, SWE-bench Verified is manually filtered out a more reliable subset; SWE-bench Multimodal introduces front-end, graphics and user interface related visual issues; Multi-SWE-bench Extend to Multilingual ecology. The official SWE-bench site will still maintain the leaderboard such as Verified, Multilingual, Lite, Full, Multimodal, etc. by 2026, and clearly states the number of instances of different subsets and resolved indicator.
But the SWE-bench family has also exposed the problem of evaluation. The real issue may be incomplete, and the original test may only cover some behaviors; once the benchmark is public for a long time, model training, prompt tuning, and scaffold adaptation will pose a risk of pollution; leaderboard "Resolved" is also not equivalent to patch maintenance, readable, and no side effects. For researchers, SWE-bench is more suitable as a class of system capability tests, rather than the only proxy metrics for final engineering quality.
2. Agent scaffold:SWE-agent、OpenHands、MASAI
The contribution of SWE-agent is not simply "let the model call the shell", but proposes the view of the Agent-Computer Interface: LLM Agent It is a new computer user that needs a specially designed Interface for viewing files, editing files, executing commands, and parsing feedback. It shows that the same model will show different abilities under different interfaces.
OpenHands has turned the software development Agent into an open platform, emphasizing engineering elements such as sandbox execution, multi-agent collaboration, browser and command-line tools, and Benchmark integration. This type of platform is critical to research: if each paper is scaffold from scratch, it is difficult to compare the contribution of the model itself, retrieval strategy and tool strategy.
MASAI represents another route: split the software engineering task into multiple subagents with clear responsibilities. Positioning, editing, validating, and summarizing can use different prompts, different contexts and different strategies. It is engineering intuition, but it also brings problems with cost, status synchronization and error propagation. For production systems, the more multi-agents are not the better, the key is whether the task boundaries are verifiable.
3. The counter-example of "less is more": Agentless
The value of Agentless is to cool down complex agents. It does not allow the model to freely plan long action sequences, but fixes the task into three stages: localization, repair, and patch validation. The paper shows a strong competitiveness on SWE-bench Lite, indicating that many problems do not require completely independent long-range exploration, but require high-quality positioning, candidate patch generation and strict verification.
This distinction matters when a company first deploys a coding agent. A sensible first version gives the agent tasks in a pipeline where people can observe, test, and roll back its changes. It might fix lint errors, add missing tests, upgrade a small dependency, or draft a migration. Changes to the core architecture can wait.
4. Data flywheel: SWE-smith, SWE-rebench, SWE-Bench++
The training data of the software engineering Agent is much more difficult than chat data, because each sample is not only "input-output text", but also requires a reproducible environment, dependent installation, test Oracle, failure track and executable patches. SWE-smith automatically synthesizes the test task from Python repositories, and publishes large-scale training instances, trajectories, and model assets. SWE-rebench emphasizes continuing to draw new tasks from GitHub and focus on decontamination evaluation. SWE-Bench++ attempts to automatically generate multilingual repository-level tasks from open source PRs.
This line is likely to determine the upper limit of open source Software Agents. There is no large-scale, executable, decontamination, and cross-language data, and the research can only repeatedly adjust Scaffold around a few public benchmarks. What is more valuable in the future is not "another leaderboard result", but auditable data generation pipelines, environment builders, test Oracle quality control and failure trajectory annotation.
5. Real value and long-range capabilities: SWE-Lancer, HCAST, Terminal-Bench
SWE-Lancer pushes tasks from GitHub issue to Freelance software engineering and estimates the value of the task with real payout. This can alleviate the disconnect between Benchmark and economic value, but it is also more difficult to fully open, and the distribution of tasks may be affected by the platform, time and customer needs.
The long-task study of HCAST and METR provides another perspective: don't just ask "how much is the pass rate", but also ask "how long does this task human experts usually take, and how reliable the model is at this time scale". Terminal-Bench further puts the agent in the real command line environment, and examines more complicated tasks such as installation, running, debugging, and data processing. This type of evaluation is important for Software Engineering Agents, because real engineering work often fails in the environment and process, not on a line of code.
Comparing the methods
| Route | Representative work | core thought | Advantage | Risk |
|---|---|---|---|---|
| Repository-level Benchmark | SWE-bench、SWE-bench Verified、Multi-SWE-bench | Real Issue + Executable Test | The task is close to real development, which is convenient for horizontal comparison | Incomplete testing, pollution, and insufficient language coverage |
| interactive Agent | SWE-agent、OpenHands | Let the model use shell, editor, test tools | Closed-loop debugging, suitable for complex tasks | Long trajectory, high cost, high authority risk |
| Structured SE Pipeline | AutoCodeRover、Agentless | Locate first, then repair, then verify | Explainable, controllable and low cost | Insufficient elasticity to complex cross-module tasks |
| modular / multi-agent | MASAI et al. | Sub-agent division of labor processing positioning, editing, verification | The task boundary is clear, replaceable modules | Coordination Cost and Error Spread |
| Data synthesis and continuous extraction | SWE-smith、SWE-rebench、SWE-Bench++ | Expand the training and evaluation tasks that can be performed | Support training, RL, decontamination | Difficulty in task quality control |
| Production Asynchronous Collaboration | Codex、Claude Code、GitHub Copilot cloud agent、Jules | Workflow from Issue/Tasks to Branch/PR | Close to the daily development process | Availability, billing, authority and review to govern |
Tech Trends: The Real Moat in the System Closed Loop

First, code positioning is more critical than generation. Many failures are not that the model does not write a patch, but corrects the error file, misses the implicit call chain, and does not understand the test intent. In the future, the system will rely more on LSP, AST, call diagram, test coverage, historical commit, issue and runtime trace, rather than relying solely on vector retrieval.
Second, test execution is becoming the core sense of the Agent. Software Agents must turn PyTest, Go Test, NPM Test, MyPy, Eslint, CI Log, Benchmark Script into parsable feedback. Simply "let the model see errors" is not enough, the system needs to structured extraction failure tests, stacks, recent changes, environmental missing and flaky signs.
Third, asynchronous multitasking will become the default form of the product. Official data of OpenAI Codex, GitHub Copilot Cloud Agent, Claude Code, and Jules all emphasize cloud sandboxes, parallel tasks, branches/PR, logs and human reviews. In the future, developers may no longer only interact with one chat window, but manage multiple running agent sessions.
Fourth, the evaluation will shift from static question bank to fresh tasks and organizational indicators. SWE-rebench, SWE-Bench++, Terminal-Bench, AIDev and 2026 have used a research note, and the research community is shifting from "how many points the model is on the open question bank" to the "real repository" What is the PR generated by the agent, whether it has been merged, whether there is a regression, and how much the cost is. These indicators are closer to production, but also more difficult to reappear.
Fifth, security from Prompt Policy into engineering authority governance. Coding Agent can read source code, run commands, install dependencies, access issue, create PR, and the permissions are obviously greater than chat assistants. Sandbox, network access, secret isolation, audit log, command allowlist, dependency pinning, code ownership rules, will become the infrastructure for enterprises to access.
Implications for deployment
For the team, the safest landing path is not "buying an Agent to replace engineers", but layering tasks.
The first layer is a low-risk, strong verification task: supplementary testing, format modification, document modification, updating minor version dependencies, generating migration drafts, and explaining CI failures. Such tasks have clear acceptance conditions and are suitable for asynchronous agent batch processing.
The second layer is medium-risk bug fixes and small features: the agent is required to output the plan first, explain which files to change, which tests to run, and then be approved by people. The key is not to get the model to be right at one time, but to produce a reviewable, rollable diff.
The third level is architectural level changes, core security logic and cross-service transformation: at present, human-led, agents should be used for code search, impact analysis, test completion and alternative generation. Delivering such tasks completely to the Agent tends to have problems in terms of needs understanding, implicit constraints, and long-term maintenance.
In terms of tools, the team has at least four things to prepare:
- Reproducible development environment: Containers, setup scripts, lock dependencies and test commands are more important than prompt.
- Repository-level documentation: Similar to AGENTS.md, CLAUDE.md, Copilot custom instructions, explicit code style, test commands, prohibited matters and review standards.
- Permissions and Audits: Default SECRET, minimum network access, all commands and test output remains.
- quality indicator: Don't just look at the number of generated rows, it depends on the test, the review rate, rollback rate, online defect, token/calculating power cost and human time saving.
Limitations and open questions
First, BENCHMARK resolved is not equal to real quality. A patch can pass the test but break the uncovered behavior; you can also write a patch for the overfitting test. True engineering also cares about readability, maintainability, performance, compatibility and safety.
Second, the pollution problem of open Benchmark will become more and more serious. Model Training Set, Retrieval Enhancement, and Manual Prompt tuning may all indirectly see the question or answer. Continuing to draw new tasks, Held-Out repository and commercial closed source evaluation will become more important, but also reduce open reproducibility.
Third, the Agent cost and reliability remain unstable. Long trajectories will amplify Token costs, tool call costs, and environment latency; a failed autonomous exploration may be more expensive than a human manual repair. In 2026, the research figures on the enterprise Rollout and the command line Agent still need to be released before manual verification, and the results of a single organization cannot be directly generalized to all teams.
Fourth, the Security border is not yet mature. Coding Agent may accidentally delete files, leak context, perform malicious dependencies, use Prompt Injection as a repository description, or introduce supply chain risks in PR. The Prompt Injection and Agent Security have been discussed in the previous round; in software engineering scenarios, these risks fall more specifically on shell, CI, secret and dependency systems.
Questions to watch
- Cross-language and cross-ecological capabilities: Build systems, package managers, test frameworks and static type systems other than Python will change the Agent design.
- Real and Fresh Mission Review: How to strike a balance between open reproduction and decontamination.
- Patch Quality Evaluation: Can surpass pass/fail to automatically evaluate maintainability, minimum, security and review costs.
- Agent training data flywheel: Failure trajectory, artificial review comments, and whether the CI log can become high-quality post-training data.
- organizational collaboration model: What future engineering managers may need to manage is not "whether to use AI", but the Agent Session queue, permissions, budget and review SLA.
Summary
Software engineering agents merit sustained attention in 2026 because they connect LLM capabilities with real engineering workflows, execution environments, evaluation, and organizational governance. In the near term, their most valuable role is handling subtasks whose results people can verify, review, and roll back. In the longer term, progress in data generation, contamination-resistant evaluation, sandbox permissions, reliability on long tasks, and human–agent collaboration could change the division of labor. Developers who now write code with AI completion could instead define goals and boundaries while agents propose, verify, and submit reviewable changes in controlled environments.
What needs to be conservative is that the Leaderboard will change rapidly, the product capabilities and pricing will change rapidly, and the organizational adoption figures in the new paper in 2026 may also be strongly affected by the sample and system. It is recommended to check the official documentation, arXiv version, code repository status and leaderboard data again before release.
References
Search date: 2026-07-07.
- Carlos E. Jimenez et al., “SWE-bench: Can Language Models Resolve Real-World GitHub Issues?” arXiv, 2023. https://arxiv.org/abs/2310.06770
- SWE-bench Official Leaderboards. https://www.swebench.com/
- John Yang et al., “SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering.” arXiv, 2024. https://arxiv.org/abs/2405.15793
- Yuntong Zhang et al., “AutoCodeRover: Autonomous Program Improvement.” arXiv, 2024. https://arxiv.org/abs/2404.05427
- Chunqiu Steven Xia et al., “Agentless: Demystifying LLM-based Software Engineering Agents.” arXiv, 2024. https://arxiv.org/abs/2407.01489
- Daman Arora et al., “MASAI: Modular Architecture for Software-engineering AI Agents.” arXiv, 2024. https://arxiv.org/abs/2406.11638
- Xingyao Wang et al., “OpenHands: An Open Platform for AI Software Developers as Generalist Agents.” arXiv, 2024. https://arxiv.org/abs/2407.16741
- John Yang et al., “SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains?” arXiv, 2024. https://arxiv.org/abs/2410.03859
- Daoguang Zan et al., “Multi-SWE-bench: A Multilingual Benchmark for Issue Resolving.” arXiv, 2025. https://arxiv.org/abs/2504.02605
- Samuel Miserendino et al., “SWE-Lancer: Can Frontier LLMs Earn $1 Million from Real-World Freelance Software Engineering?” arXiv, 2025. https://arxiv.org/abs/2502.12115
- openai/SWELancer-Benchmark. https://github.com/openai/SWELancer-Benchmark
- John Yang et al., “SWE-smith: Scaling Data for Software Engineering Agents.” arXiv, 2025. https://arxiv.org/abs/2504.21798
- Ibragim Badertdinov et al., “SWE-rebench: An Automated Pipeline for Task Collection and Decontaminated Evaluation of Software Engineering Agents.” arXiv, 2025. https://arxiv.org/abs/2505.20411
- Lilin Wang et al., “SWE-Bench++: A Framework for the Scalable Generation of Software Engineering Benchmarks from Open-Source Repositories.” arXiv, 2025. https://arxiv.org/abs/2512.17419
- Xiang Deng et al., “SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?” arXiv, 2025. https://arxiv.org/abs/2509.16941
- David Rein et al., “HCAST: Human-Calibrated Autonomy Software Tasks.” arXiv, 2025. https://arxiv.org/abs/2503.17354
- Thomas Kwa et al., “Measuring AI Ability to Complete Long Tasks.” arXiv, 2025. https://arxiv.org/abs/2503.14499
- Mike A. Merrill et al., “Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces.” arXiv, 2026. https://arxiv.org/abs/2601.11868
- Shoufa Chen et al., “TUA-Bench: A Benchmark for General-Purpose Terminal-Use Agents.” arXiv, 2026. https://arxiv.org/abs/2606.28480
- Romain Robbes et al., “Agentic Much? Adoption of Coding Agents on GitHub.” arXiv, 2026. https://arxiv.org/abs/2601.18341
- Hao Li et al., “AIDev: Studying AI Coding Agents on GitHub.” arXiv, 2026. https://arxiv.org/abs/2602.09185
- Emerson Murphy-Hill et al., “Adoption and Impact of Command-Line AI Coding Agents: A Study of Microsoft's Early 2026 Rollout of Claude Code and GitHub Copilot CLI.” arXiv, 2026. https://arxiv.org/abs/2607.01418
- OpenAI, “Introducing Codex.” 2025-05-16. https://openai.com/index/introducing-codex/
- GitHub Docs, “About GitHub Copilot cloud agent.” https://docs.github.com/en/copilot/concepts/agents/cloud-agent/about-cloud-agent
- Anthropic Docs, “Claude Code overview.” https://docs.anthropic.com/en/docs/claude-code/overview
- Google Jules official site. https://jules.google/
Note: In 2026, the paper, product availability, leaderboard ranking, model name and organizational adoption number change rapidly. This article has been written according to the search results of 2026-07-07. It is recommended to manually verify before the official release.