
Date:2026-06-25
Who this is for: Graduate student of large model, evaluation platform builder, model selection and application engineering team
search date:2026-06-25
Abstract
In the past year, the core problem of LLM Evaluation has shifted from "which model is higher on the list" to "whether this score can still distinguish the real ability". MMLLU, BIG-bench, HELM have standardized general Evaluation; MMLU-Pro, LiveBench, SimpleQA, Humanity'S Last Exam, BetterBench, SimpleQA Verified and other work further pushes the difficulty, pollution, fact, reproducibility and benchmark lifecycle to the foreground.
The main line of this article is: the evaluation is moving from a static question bank to a dynamic question bank, from a single Accuracy to a multi-index portrait, from a general list to a task-based execution environment, and from a display ranking to a diagnosis failure mode. The conclusion of the project is also very direct: the public list can only be screened first, and the really available evaluation capabilities come from private Golden Set, dynamic Recent Set, manual sampling, statistics Confidence interval and online regression closed loop.
Contents
- research background
- roadmap for the past year
- Key papers by research question
- Comparing the methods
- Key Trends and Engineering Inspiration
- Limitations and open questions
- References
research background
Earlier LLM evaluation pursues a unified comparison: Given the same topic, compare models with Accuracy, F1, BLEU, ROUGE or artificial preferences. MMLU used 57 subjects to cover knowledge and problem solving with multiple-choice questions in 2020; a multi-task collection that BIG-bench expanded to community contributions in 2022; HELM extends Accuracy to expand to Multi-dimensional indicators such as calibration, robstness, fairness, BIAS, TOXICITY, and efficiency.
The value of these benchmarks is to push model publishing from "showing examples" to "standardized reports". But the reality of 2024-2026 is that many open question banks are close to saturation, and training corpus is more likely to include open benchmarks or similar questions. For the engineering team, the most dangerous is not the low score, but the high score no longer represents the reliability of the production task.

roadmap for the past year
2024: More difficult, more stable, and more pollution-resistant. MMLLU-PRO cleans up MMLU noise problems, adds stronger inference requirements, and extends options from 4 to 10 to improve the degree of discrimination and Prompt stability. LiveBench takes anti-pollution as the core: the topic comes from recent math competitions, arXiv, news and datasets, updated monthly, and try to use objective answers that can be automatically verified.
2024-2025: Factuality and expert boundaries are the focus. OpenAI's SimpleQA focuses on short facts, which requires a single answer and can be judged, and distinguishes correct, incorrect, not attempted; Humanity's Last Exam Use expert-level, multi-disciplinary, and some multimodal closed-book questions to test the frontier model to respond to the problem of popular Benchmark being "popped" with high scores.
2025: The evaluation itself also needs to be evaluated. BetterBench examines benchmark quality with the BenchMark Lifecycle Best Practice, emphasizing statistical significance, reproducibility, documentation, versions, and bug fixes. The LLM Benchmark Survey in 2025 classifies hundreds of Benchmarks as general abilities, domain capabilities, and target capabilities, and also lists data pollution, cultural/language bias, and dynamic environment insufficiency as a systemic problem.
2025-2026: From static capabilities to workflow capabilities. Task-based evaluations such as SWE-bench, BrowseComp, HealthBench, and SpreadsheetBench have focused on "answering questions" to "complete tasks". This type of evaluation usually involves tool calls, file editing, browsing, test execution and long context. Some 2026 preprints have discussed the rotation of Frontier Leaderboard after saturation; the relevant conclusions have changed rapidly, and this article is marked as "waiting for manual verification".
Key papers by research question
1. General multitasking: MMLU to MMLU-Pro
MMLU’s historical importance rests on its breadth of subject coverage and a question format that is easy to reproduce. Multiple-choice scores, however, are sensitive to the distribution of answer options, memorized questions, and prompt formatting. As stronger models approach saturation, score differences become less stable. MMLU-Pro makes the test more demanding for frontier models by removing noise, requiring more reasoning, and expanding the answer choices. Private evaluations should evolve in the same way. Pass rates on an aging question bank lose their value over time.
2. Multi-indicator portrait: HELM's methodology
The key to HELM is not a list, but the evaluation view of "Scenario X indicator". A model suitable for customer service, medical, legal or code review, not only is correct, but also calibrated, stable, fair, low-toxic, and cost-controllable. The engineering evaluation table should be divided into three layers at least: task success rate, reliability indicators, and operating indicators. Without these three layers, it is difficult to convert the high score of the list into a controllable SLA.
3. Dynamic anti-pollution: LiveBench
The public question bank will enter training data, fine-tuning sets, prompt engineering examples and model publishing materials. Once contamination occurs, the score may reflect memory rather than generalization. LiveBench's route is to continuously update the questions, use recent sources, and bias objective scores. It also brings reproducing challenges, so the evaluation platform needs to maintain versions, change logs, freeze snapshots and playback mechanisms like software systems.
4. Fact: SimpleQA and SimpleQA Verified
Facts used to be mixed in open generation tasks, and scoring is easily masked by the quality of expression. SimpleQA narrows the question as a short fact Q&A, explicitly encouraging the model not to answer when it is uncertain. SimpleQA Verified pointed out that the evaluation set itself will have noise, theme bias and repetition problems, which require deduplication, theme balance and source reconciliation. The same is true for the business Golden Set: When the label is unreliable, the stronger the model, the more likely it is to expose the error of the evaluation set.
5. Expert closed volume: Humanity's Last Exam
HLE pushes the difficulty to the expert boundary. The arXiv page shows that it will be submitted in 2025, and there will still be revised versions in 2026; the questions cover multi-disciplinary mathematics, natural science, humanities and social sciences, etc., including multi-select, short answers and some multimodal questions. Its importance is to reopen the cutting-edge model gap, but the harder the Benchmark, the more dependent on expert annotation and continuous error correction. When referencing HLE, you should pay attention to the version number, public/private split and bug fix records.
6. Meta-Evaluation: BetterBench and Data Pollution Review
BetterBench shifts the focus from "Model Review" to "Review Benchmark's Benchmark", focusing on documentation, reproducible, statistical significance, data governance and maintenance. The data pollution review systematically sorts out the risks of the test set contamination. Their engineering meaning is: Benchmark without Confidence Interval, version locking and error feedback channels will soon lose explanatory power.
Comparing the methods
| Route | Representative work | What's the solution | Advantage | Risk |
|---|---|---|---|---|
| static multitasking | MMLU, BIG-bench | General Competency Horizontal Comparison | Easy to reproduce, wide coverage | Saturation, pollution, prompt sensitive |
| multi-indicator portrait | HELM | accuracy is not enough | Exposure reliability and cost balance | High cost of indicators |
| difficulty upgrade | MMLU-Pro, GPQA, HLE | Decrease in the discrimination of the cutting-edge model | More suitable for Frontier models | High cost of labeling and error correction |
| Dynamic anti-pollution | LiveBench | Public Question Bank Leaks | Reduce memory risk | The version is more complicated |
| factuality | SimpleQA, SimpleQA Verified | hallucinations and uncertainty | Automatic scoring is clear | Coverage |
| task-based evaluation | SWE-bench, BrowseComp | Real workflow ability | Approximately app | HARNESS Impacts Results |
| yuan evaluation | BetterBench | Benchmark itself quality | Increase credibility | not directly to the model |
Key Trends and Engineering Inspiration
first, Benchmark rotation will become the norm. If the paper only reports high scores on the old benchmarks, but does not explain why these benchmarks are still differentiated, the credibility will be reduced. second, Dynamic evaluation and private Holdout is more important. The open list is suitable for initial screening, and the final decision should rely on the business Golden Set and recent samples. third, Scorer itself needs to be calibrated. Rule scores, human preferences, and LLM-as-a-judge are all biased; Judge Prompt, Judge Model, temperature, version and original rating should be kept. fourth, Task-based evaluation raises the engineering threshold. Code, browsing, tables and medical reviews require sandboxes, dependencies, tool logs, retries, and security boundaries. Fifth, Reports should be upgraded from a score table to a diagnostic report: Failed clustering, Confusion Matrix, calibration curves and sample audits are more useful than single-point rankings.
These 7 practices help put evaluation into production. Maintain three sets: a public benchmark, a private reference set, and a regularly refreshed set of recent cases. For each example, record its source, reference answer, scoring function, version, and review date. Report success, refusal, hallucination, and format-error rates alongside latency and cost. Have people spot-check the LLM judge’s decisions. Use paired evaluations for model upgrades. Classify failures by cause, such as missing knowledge, faulty reasoning, failure to follow instructions, or tool misuse. Retain human sign-off for high-risk uses.
Limitations and open questions
There are still several contradictions in LLM Evaluation. Openness and anti-pollution conflicts: open question bank is easy to reproduce, and it is easier to leak; private question bank reduces pollution, but reduces transparency. Difficulty and judging conflicts: the real task is more open, the score is more difficult; the automatic score is more stable, and the task is often narrower. Conflicts between the list and product goals: The list is optimized to pursue an average score, and the production system is more concerned about tail failure, compliance and recovery capabilities.
There is also an often overlooked problem: the evaluation meeting is aging. Business processes, user language, model abilities, attack methods, and external knowledge are all changing. A review set that has not added samples and bug fixes for half a year should be regarded as a historical snapshot, rather than the only release threshold.
Summary
LLM Evaluation has changed from "arranging the model" to "building a trusted measurement system". MMLLU, BIG-bench, HELM provide standardized foundation; MMLU-Pro, LiveBench, SimpleQA, HLE, BetterBench show that the Evaluation itself also requires iteration, audit and governance. For researchers, the future paper should report protocol, version, statistical significance, failure analysis and pollution control; for the engineering team, the real Evaluation ability comes from private task set, dynamic update, traceable scoring and production regression closed loop.
References
The following data are all searched on 2026-06-25; model ranking, ranking score and repository status change rapidly, it is recommended to check the version again before citing.
- Dan Hendrycks et al. Measuring Massive Multitask Language Understanding. arXiv:2009.03300, 2020. https://arxiv.org/abs/2009.03300
- Aarohi Srivastava et al. Beyond the Imitation Game. arXiv:2206.04615, 2022. https://arxiv.org/abs/2206.04615
- Percy Liang et al. Holistic Evaluation of Language Models. arXiv:2211.09110, 2022. https://arxiv.org/abs/2211.09110
- Stanford CRFM. HELM. https://crfm.stanford.edu/helm/latest/
- Yubo Wang et al. MMLU-Pro. arXiv:2406.01574, 2024. https://arxiv.org/abs/2406.01574
- Colin White et al. LiveBench. arXiv:2406.19314, 2024. https://arxiv.org/abs/2406.19314
- LiveBench official site. https://livebench.ai/
- Jason Wei et al. Measuring short-form factuality in large language models. arXiv:2411.04368, 2024. https://arxiv.org/abs/2411.04368
- OpenAI. simple-evals. https://github.com/openai/simple-evals
- Long Phan et al. Humanity's Last Exam. arXiv:2501.14249, first submitted 2025-01-24, latest observed revision v10 on 2026-02-20. https://arxiv.org/abs/2501.14249
- Anka Reuel et al. BetterBench. arXiv:2411.12990, 2024. https://arxiv.org/abs/2411.12990
- Cheng Xu et al. Benchmark Data Contamination of Large Language Models: A Survey. arXiv:2406.04244, 2024. https://arxiv.org/abs/2406.04244
- Shiwen Ni et al. A Survey on Large Language Model Benchmarks. arXiv:2508.15361, 2025. https://arxiv.org/abs/2508.15361
- Lukas Haas et al. SimpleQA Verified. arXiv:2509.07968, 2025. https://arxiv.org/abs/2509.07968
- Hugging Face. Open LLM Leaderboard. https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard
- Adil Amin. The Growing Pains of Frontier Models. arXiv:2605.18840, 2026. https://arxiv.org/abs/2605.18840
- Carlos E. Jimenez et al. SWE-bench. arXiv:2310.06770. https://arxiv.org/abs/2310.06770
- OpenAI. Introducing SWE-bench Verified. https://openai.com/index/introducing-swe-bench-verified/