Cover illustration

Date:2026-07-12
Who this is for: Graduate students, programming language and software engineering researcher, LLM evaluation researcher, technical reader with engineering background
search date:2026-07-12

Abstract

LLM for Code is shifting from "write a function" to "understanding a repository, locating a problem, generating patches, running tests and interpreting failures". HumanEval, MBPP, APPS, and AlphaCode lay the foundational paradigm of natural language to programs; the more critical change in 2025-2026 is BigCodeBench, LiveCodeBench, CodeElo, ICPPC-Eval, KernelBench, RealBench, SecRepoBench, SWE-bench Benchmark, such as Live, Multi-SWE-bench, SWE-bench Pro, and SWE-Lancer, etc., pushes evaluation to multi-library API calls, real-time contest questions, repository-level context, multi-language Issue repair, security Code, GPU Kernel performance and long-period software engineering tasks.

The special value of code research is that the model output can be repeatedly checked by the compiler, interpreter, unit test, static analyzer, fuzzer, profile and CI environments. But this does not mean that the evaluation is simple. Test passing does not mean that the requirements are met, the public GitHub issue is easy to be polluted, the long context is not equal to effective positioning, and the agent trajectory often hides errors in tool calls and environmental differences. The core problem in the future is not to simply refresh the ranking list, but to establish a code research paradigm that can be reproduced, anti-pollution, cross-language, and can expose the failure mode.

Contents

  • Research background: Why the code is the natural experimental field of LLM
  • core science questions
  • A reading guide to the past year’s papers
  • Key papers by research question
  • Comparing the methods
  • How to interpret the experiments and benchmarks
  • Reproducibility and data problems
  • Limitations and open questions
  • Research questions for graduate students
  • Summary
  • References

Research background: Why the code is the natural experimental field of LLM

The Code task has one advantage that other text tasks are difficult to have at the same time: the output can be executed. Questions and answers, abstracts and writing often require manual judgment of fact, completeness and style; Code generation can at least get clear feedback through syntax checking, type checking, unit testing, integration testing, and performance testing. This makes LLM for Code very early to become the core scenario of "whether the model capability can be automatically verified".

Early work focused on synthesizing individual functions. The OpenAI Codex paper introduced HumanEval, a set of 164 Python function problems testing whether a model can turn a docstring into an implementation that passes the tests. Program Synthesis with Large Language Models introduced MBPP, with 974 small Python programming tasks. APPS raised the difficulty with 10,000 problems from Codewars, AtCoder, Kattis, and Codeforces. AlphaCode combined large-scale sampling and filtering with evaluation on a competitive programming platform, demonstrating the potential of models on algorithmic problems.

These work establish a basic paradigm: given specifications, generate multiple candidate programs, screen with test or online judgment. But real development is not just writing a dependency function, but to read the repository, understand the interface, modify multiple files, deal with historical design, run failure tests, avoid security loopholes, and make engineering judgments under incomplete specifications. So the research problem shifts from "whether the model will write code" to "whether the model can work reliably in the software system".

Code Research Method Framework Diagram

core science questions

first, Whether the specification understands whether it is true. The input and output boundaries of short function questions are generally clear; real requirements may come from issue, PR discussion, UML, logs, screenshots, failure tests, and user descriptions. The model needs to distinguish the constraints, negligible background and implicit invariants that must be met.

second, How to choose a repository context. Long context models can be inserted into more files, but effective code modification depends on precise positioning: which functions are related, which tests can expose problems, and which call chains cannot be destroyed. Agentless, AutoCodeRover, SWE-agent, and OpenHands represent different routes: structured retrieval, patch validation, Agent-Computer Interface, or a generic developer workbench.

third, How to perform feedback as a repair signal. Compilation errors, stack tracking, single test failures, static analysis alarms, coverage and performance profiles can all tell the model what is wrong. However, the model may also be overfitting tests, delete assertions, bypass errors, introduce hidden regression, or contaminate the context more and more in multiple rounds of repairs.

fourth, Is the benchmark credible?. SWE-bench makes GitHub Issue a standard task, but exposing the Issue, the public repository and the public leaderboard will bring training data pollution, memory, test vulnerability and submission selection bias. The emergence of LiveCodeBench, SWE-bench Live, Multi-SWE-bench, SWE-bench Pro and SWE-Lancer is essentially patching this problem.

Fifth, What else to comment beyond correctness. Software engineering not only cares about "can run". Security, maintainability, performance, minimum changes, test quality, architectural consistency, dependence risk, and code review readability are all important. KernelBench and CUDA-LLM push the target to GPU Kernel performance; SecRepoBench focuses on security code completion in C/C++ repositories; RealBench It emphasizes that team development often starts from structured design and UML, rather than a natural language Prompt.

A reading guide to the past year’s papers

The first main line is repo-level and real software engineering evaluation. The original SWE-bench version contains 2,294 real GitHub issues from 12 Python repositories, which drives the research paradigm of "Give the model a repository and issue, and let it generate patches". Multi-SWE-bench extends issue resolving to Java, TypeScript, JavaScript, Go, Rust, C and C++, including 1,632 This is a high-quality instance, and proposes a multi-language RL data construction direction. SWE-bench Live uses the new Issue since 2024, the initial version contains 1,319 tasks, 93 repositories and executable Docker environments, with the goal of reducing static benchmarks overfitting with pollution. SWE-bench Pro further emphasizes long-period tasks, and the paper says it contains 1,865 issues from 41 active repositories.

The second main line is From competition to real-time, anti-pollution code evaluation. LiveCodeBench continuously collects new questions for LeetCode, AtCoder, and Codeforces, and covers code generation, self-healing, code execution, and test output predictions. CodeElo directly submits the model to the Codeforces platform and measures performance in an Elo way closer to human competition. ICPPC-Eval selected the problem of ICPC regional competition in recent years, and proposed Refine@K to observe the ability of the model to be iteratively repaired under the implementation of feedback. These works together: Pass@k is still important, but a single generation score is no longer enough to explain the code capability of the Reasoning Model.

The third main line is Program synthesis moves from short functions to multi-library, multi-tool and structured design. BigCodeBench contains 1,140 Python tasks, covering 139 libraries, 7 realms, and 723 function calls, emphasizing complex instructions and multi-API combinations. In 2026, RealBench proposed a repository-level code generation based on natural language needs and UML design, pointing out that many existing Benchmarks are inconsistent with the specification input methods in team development.

The fourth main line is Automatic debugging, program repair and Agent workflow comparison. SWE-agent proposes the Agent-Computer Interface, emphasizing the design of an interactive interface suitable for navigation, editing and testing. AutoCodeRover prefers software engineering perspectives, using AST, code search and test assistance Fault Localization to help locate modification points. Agentless, in turn, questiones whether complex agents are necessary, and establishes a strong baseline with the three-stage process of positioning, repair, and patch verification. OpenHands provides a common platform to organize command line, code editing, browsers and sandbox as a scalable development agent environment.

The fifth main line is Performance, security and specialized domain codes. KernelBench uses 250 PyTorch ML Workloads to check whether the model can write fast and correct GPU kernels, and use fast_p to measure the correct generation ratio that exceeds the baseline velocity threshold. CUDA-LLM incorporates execution, compilation and performance feedback into iterative optimization to explore the automatic generation of CUDA programs. SecRepoBench puts the problem in the real C/C++ repository, covering 27 repositories, 318 tasks and 15 types of CWE, focusing on the joint constraints of correctness and security.

Key papers by research question

1. Basic program synthesis: HumanEval, MBPP, APPS, AlphaCode

The contribution of HumanEval is not a large number of questions, but pushes "functional correctness" to the code model evaluation center: the model output must pass the hidden test, not just BLEU or text similarity. MBPP demonstrates that small-scale natural language programming tasks can be systematically constructed and used for comparison-shot comparisons with fine-tuning. APPS expands the difficulty of evaluation to a competition-style problem, the number and difficulty span of a larger number of problems, and closer to the algorithm training platform.

The key idea of AlphaCode is the large-scale candidate generation plus behavior filtering. Instead of letting the model write a perfect program at one time, it generates a large number of candidates, clusters, runs samples, filters and submits. This paradigm later influenced test-time scaling, self-repair, verifier reranking and execution-guided decoding. The limitations are also very clear: a large number of sampling requires computing power, and there is a risk of pollution in open competition questions, and the algorithm questions are not equal to software engineering.

2. Multi-library and complex instructions: BigCodeBench, DS-1000, RealBench

BigCodeBench clearly states that short function questions have been gradually saturated by new models, and real programming often requires calling multiple libraries, processing files, networks, data structures and exception conditions. Its value is not simply "harder", but closer to "finishing tasks with code": the model needs to choose APIs, combine calls, follow natural language details, and pass more rigorous tests.

DS-1000 represents the data science code generation route, and pays attention to the real use of libraries such as NumPy, Pandas, and Sklearn. The value of RealBench is to turn specification input into "requires + UML design", reminding researchers not to default that all development tasks come from natural language Prompt. A question worth continuing to study is: Does the code model really understand the design structure, or is it just filling in common modules from the text description?

3. Automatic debugging and program repair: from APR to SWE Issue Resolving

Traditional Automated Program Repair research has long focused on patch generation, testing and overfitting patches. After LLM joins, the repair process extends from template search to natural language Issue understanding, cross-file targeting, and patch interpretation. The empirical study of LLM Automated Program Repair in 2025 also reminds us that different languages, different bug scenarios, and different Prompt strategies will significantly affect the results, and the size of the model and repairing capabilities are not always linearly related.

SWE-bench push this direction to the real GitHub issue. Issue Resolving requires a model to read a repository, understand user reports, edit code and pass repository tests compared to function questions. The differences between SWE-agent, AutoCodeRover, MASAI, Agentless and OpenHands constitute the research pedigree: interactive agent, structured retrieval, modular sub-agent, non-agent Three-stage process, general development platform. When comparing these systems, you should not only look at the rankings, but also on the cost, track length, tool permissions, whether it is open source, and whether the experimental environment can be reproduced.

4. Software engineering Benchmark extension: Multi-SWE, Live, Pro, Lancer

The importance of Multi-SWE-bench is to break down the centralization of Python. The build system, test framework, package management and idiomatic design of Java, TypeScript, Go, Rust, C/C++ are all different, and the policies of the model on the Python issue cannot be directly migrated. SWE-bench Multimodal further incorporates front-end, visualization and JavaScript software into the evaluation, requiring the model to deal with image or visual problem statements.

SWE-bench Live and SWE-bench Pro represent anti-pollution and long-period directions. Live Benchmark builds with new Issue and automated environments to reduce static question bank failures; Pro Benchmark tries to cover more complex and closer enterprise development issues. The SWE-Lancer is more unique: it connects the tasks from Upwork's real software engineering work and links the value of the task to test verification. Amount indicators should not be simply understood as "models create economic value", but it does provide another task difficulty ruler.

5. Performance, safety and system code: KernelBench, CUDA-LLM, SecRepoBench

If the code generation research only looks at the passing of the test, a large number of engineering core indicators will be missed. A sorting function can be correct but very slow, a network service can pass a single test but there is an injection vulnerability, and a CUDA kernel can be compiled but slower than PyTorch baseline. KernelBench therefore regards "correct and faster" as a joint goal, while CUDA-LLM explores and enhances feature search and use compile and profile feedback.

SecRepoBench focuses on security code completion, especially C/C++ real repository and CWE-like vulnerabilities. This kind of task is more difficult because the model needs to meet both functions, context style, memory security and vulnerability evasion. Safety Benchmark is also more likely to expose the "test pass but dangerous implementation" problem, which is suitable as the next step in the reliability research of the code model.

Comparing the methods

Route Representative work core mechanism Fit the question Main risk
Function-level program synthesis HumanEval、MBPP Generate functions from docstring / short specifications, score on a single test Basic code generation, model horizontal comparison The problem is short, polluted, and cannot represent the development of the repository level
Competition code generation APPS、AlphaCode、LiveCodeBench、CodeElo、ICPC-Eval Multi-sampling, online judgment, execution feedback, Elo or Refine@K Algorithm reasoning, complex constraints, real-time new questions There is a big difference between algorithm problems and engineering tasks, and the sampling budget has a great impact
Multi-library task synthesis BigCodeBench、DS-1000 API selection, multi-library combination, complex natural language instructions Tooling Programming, Data Science, Scripting Automation There is still a gap between test coverage and real needs
Repository level generation RealBench、MRG-Bench Requirements, design drawings, cross-file context and project testing Team development, module generation, architecture consistency The reconstruction of the environment is difficult, and the loyalty of specifications is difficult to judge
Automatic Repair / Issue Resolving SWE-bench、SWE-agent、AutoCodeRover、Agentless、OpenHands Positioning, patch generation, test execution, trajectory feedback Bug fix, maintenance, real GitHub issue Public Issue Pollution, Overfitting Testing, Leaderboard Unstable
Multilingual and long-period evaluation Multi-SWE-bench、SWE-bench Live、SWE-bench Pro、SWE-Lancer Multilingual repository, recent issue, complex task, real delivery Anti-pollution and real software engineering The evaluation cost is high, and some collections cannot be made public
Safety and performance code SecRepoBench、KernelBench、CUDA-LLM Static analysis, CWE, compilation, profile, speed threshold Security Completion, System Code, GPU Kernel Hardware and toolchain depend on strong, reproducible

How to interpret the experiments and benchmarks

When reading a code paper, the first step is to distinguish the task granularity. Function level, file level, repository level, issue resolving, competition problem, performance kernel, and security complement are different problems. It is not true to directly extrapolate the HumanEval high score to the real project.

Second, see if the test is strong enough. Fewer unit tests, open sample tests, and weak assertions may make the model pass by chance. For repair tasks, it is necessary to see whether there is a regression test, whether to run the complete test suite, whether to detect the overfitting patch, and whether there is a human verification issue that can be solved.

Third, look at the context and tool permissions. Whether an agent can search files, run tests, install dependencies, access the network, and edit multiple files will change the result significantly. The paper should report scaffold, tool, timeout, token budget, number of candidates, model version and retry strategy.

Fourth, look at the risk of pollution. The common motivation of LiveCodeBench, SWE-bench Live, and CodeElo is that the fixed open benchmark will age. If the paper only reports a high score for the old public question bank, additionally check the release time of the question, the training data, whether it has done an approximate duplicate test, and whether to report Held-out or Live Split.

Fifth, look at the cost and trajectory. The performance of software engineering agents often comes from a large number of tokens, long tracks and multiple executions. If a method is accurate but the cost is too high and it is difficult to diagnose when it fails, the research value is still limited. Agentless strong baselines remind us that simple, interpretable, and low-cost processes should be used as a control group for complex agents.

Reproducibility and data problems

The reproducibility of code research is highly dependent on the environment. Python version, package version, system library, compiler, GPU, Docker image, test sequence, random seed, network availability can all affect results. The SWE-bench task also involves repository history, issue text, PR patch, test selection, and environment build scripts. A small dependency drift can make "model failure" become "environmental failure".

Data pollution is also more complex. GitHub is an important source of code model training. Public Issue, PR, CI log, Stack Overflow answers, and competition solutions may all enter the training corpus. The model knows the file path, function name, or historical patch, which does not mean that it can reason. 2025 research on SWE-bench Memory and Benchmark Mutation reminds us that reviews should be added to new repositories, new issues, deformation questions, private ownership Held-out, Behavioral Equivalent Test, and Failure Mode Analysis.

Another problem is that ground truth is not unique. Software repair often has a variety of reasonable patches, and the official PR is not necessarily the only correct answer; the test may also hide a better design. In the future, Benchmark needs stronger semantic judgments: attribute test, differential test, fuzzing, static analysis, human code review and specification consistency check should be combined, rather than relying only on one test suite.

Limitations and open questions

First, the code model is easy to misread "can write code" as "can do software engineering". Software engineering includes demand clarification, risk assessment, architectural tradeoff, long-term maintenance and team collaboration, which are difficult to cover with a single patch review.

Second, the Agent trajectory seems to be autonomous, and actually highly relies on Scaffold. Prompt, tool encapsulation, file search API, test command selection, error summary method and patch validation strategy may all affect the results more than the underlying model. If the paper does not disclose Scaffold, the results are difficult to compare scientifically.

Third, Benchmark is catching up with the model, but also being chased by the model. Public Leaderboard will motivate targeted optimization, fixed test set will be trained, and popular question bank will be polluted by the problem. Live BenchMark and Held-Out Split are just mitigation, not a complete solution.

Fourth, the safety risk should not only be checked at the end. Code models may generate vulnerable dependencies, hardcoded keys, dangerous commands, access to file access, and unsafe input processing. Secure code generation needs to be designed from data, training, tool permissions, static analysis, and review processes.

Research questions for graduate students

  1. Interpretable Evaluation of Repository Level Context Selection: Compare the impact of long context, RAG, AST retrieval, call diagram and test coverage information on the repair success rate, and report the error positioning instead of only the final pass rate.

  2. Learning signal modeling for performing feedback: Convert compilation errors, single test failure, stack tracking, coverage and static analysis alarms to structured state, and study which feedbacks can help the model to repair the most.

  3. Anti-pollution Benchmark Mutation: Maintain the semantic deformation of the public issue or competition questions, and detect whether the model is memory path and patch, or understand the needs.

  4. Multilingual SWE Agent Comparison: Based on Multi-SWE-bench, analyze the differences in failure patterns in different language ecosystems, such as Rust Borrow Checker, Go Module, Java Build System, TypeScript type system.

  5. Safety patch generation and review closed loop: Combined with SecRepoBench, CWE, static analysis and manual review, study whether the model can minimize functional regression and safety risks.

  6. Test-Time Optimization for performance code: Research a combined strategy of sampling, profile feedback, local search, and verifier sort on KernelBench or CUDA Kernel tasks.

Summary

The research value of LLM for Code is not the broad narrative of "whether AI will replace programmers", but in that it provides an intelligent experimental field that can be executed, verified and iterated. From HumanEval to SWE-bench, from BigCodeBench to RealBench, from KernelBench to SecRepoBench, the research boundaries are being extended from short functions to complex software systems.

Three directions look especially promising over the next few years. First, connect code generation, debugging, testing, and review in a reproducible cycle. Second, guard against misleading gains with live, multilingual, and long-horizon benchmarks, including tasks with security and performance constraints. Third, evaluate the model, retriever, tools, execution environment, and verifier as a system, so that the model alone does not get the credit or blame for how the system performs.

References

Search date: 2026-07-12. The following priority lists arXiv, OpenReview, project homepage or official repository, etc. to have access to first-hand information; model list, code repository status, benchmark leaderboard, specific SOTA score and 2026 The version information that is still being updated in 2009 is recommended to be manually verified before release.

  1. Mark Chen et al. Evaluating Large Language Models Trained on Code. arXiv, 2021. https://arxiv.org/abs/2107.03374
  2. Jacob Austin et al. Program Synthesis with Large Language Models. arXiv, 2021. https://arxiv.org/abs/2108.07732
  3. Dan Hendrycks et al. Measuring Coding Challenge Competence With APPS. arXiv, 2021. https://arxiv.org/abs/2105.09938
  4. Yujia Li et al. Competition-Level Code Generation with AlphaCode. arXiv, 2022. https://arxiv.org/abs/2203.07814
  5. Hung Le et al. CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning. arXiv, 2022. https://arxiv.org/abs/2207.01780
  6. Naman Jain et al. LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code. arXiv, 2024. https://arxiv.org/abs/2403.07974
  7. Terry Yue Zhuo et al. BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions. arXiv / ICLR 2025, 2024. https://arxiv.org/abs/2406.15877
  8. Carlos E. Jimenez et al. SWE-bench: Can Language Models Resolve Real-World GitHub Issues? arXiv, 2023. https://arxiv.org/abs/2310.06770
  9. John Yang et al. SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering. arXiv, 2024. https://arxiv.org/abs/2405.15793
  10. Yuntong Zhang et al. AutoCodeRover: Autonomous Program Improvement. arXiv, 2024. https://arxiv.org/abs/2404.05427
  11. Chunqiu Steven Xia et al. Agentless: Demystifying LLM-based Software Engineering Agents. arXiv, 2024. https://arxiv.org/abs/2407.01489
  12. Xingyao Wang et al. OpenHands: An Open Platform for AI Software Developers as Generalist Agents. arXiv, 2024. https://arxiv.org/abs/2407.16741
  13. Daoguang Zan et al. Multi-SWE-bench: A Multilingual Benchmark for Issue Resolving. arXiv, 2025. https://arxiv.org/abs/2504.02605
  14. Linghao Zhang et al. SWE-bench Goes Live! arXiv, 2025. https://arxiv.org/abs/2505.23419
  15. Xiang Deng et al. SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks? arXiv, 2025. https://arxiv.org/abs/2509.16941
  16. Samuel Miserendino et al. SWE-Lancer: Can Frontier LLMs Earn $1 Million from Real-World Freelance Software Engineering? arXiv, 2025. https://arxiv.org/abs/2502.12115
  17. Shanghaoran Quan et al. CodeElo: Benchmarking Competition-level Code Generation of LLMs with Human-comparable Elo Ratings. arXiv, 2025. https://arxiv.org/abs/2501.01257
  18. Shiyi Xu et al. ICPC-Eval: Probing the Frontiers of LLM Reasoning with Competitive Programming Contests. arXiv, 2025. https://arxiv.org/abs/2506.04894
  19. Anne Ouyang et al. KernelBench: Can LLMs Write Efficient GPU Kernels? arXiv, 2025. https://arxiv.org/abs/2502.10517
  20. Wentao Chen et al. CUDA-LLM: LLMs Can Write Efficient CUDA Kernels. arXiv, 2025. https://arxiv.org/abs/2506.09092
  21. Chihao Shen et al. SecRepoBench: Benchmarking Code Agents for Secure Code Completion in Real-World Repositories. arXiv, 2025/2026 revision. https://arxiv.org/abs/2504.21205
  22. Jia Li et al. RealBench: A Repo-Level Code Generation Benchmark Aligned with Real-World Software Development Practices. arXiv, 2026. https://arxiv.org/abs/2604.22659
  23. Matias Martinez and Xavier Franch. What's in a Benchmark? The Case of SWE-Bench in Automated Program Repair. arXiv, 2026. https://arxiv.org/abs/2602.04449
  24. Shanchao Liang et al. The SWE-Bench Illusion: When State-of-the-Art LLMs Remember Instead of Reason. arXiv, 2025. https://arxiv.org/abs/2506.12286
  25. Jiajun Sun et al. Empirical Evaluation of Large Language Models in Automated Program Repair. arXiv, 2025. https://arxiv.org/abs/2506.13186