Cover illustration

Date:2026-06-25
Who this is for: Graduate student of large model, evaluation platform builder, model selection and application engineering team
search date:2026-06-25

Abstract

In the past year, the core problem of LLM Evaluation has shifted from "which model is higher on the list" to "whether this score can still distinguish the real ability". MMLLU, BIG-bench, HELM have standardized general Evaluation; MMLU-Pro, LiveBench, SimpleQA, Humanity'S Last Exam, BetterBench, SimpleQA Verified and other work further pushes the difficulty, pollution, fact, reproducibility and benchmark lifecycle to the foreground.

The main line of this article is: the evaluation is moving from a static question bank to a dynamic question bank, from a single Accuracy to a multi-index portrait, from a general list to a task-based execution environment, and from a display ranking to a diagnosis failure mode. The conclusion of the project is also very direct: the public list can only be screened first, and the really available evaluation capabilities come from private Golden Set, dynamic Recent Set, manual sampling, statistics Confidence interval and online regression closed loop.

Contents

  • research background
  • roadmap for the past year
  • Key papers by research question
  • Comparing the methods
  • Key Trends and Engineering Inspiration
  • Limitations and open questions
  • References

research background

Earlier LLM evaluation pursues a unified comparison: Given the same topic, compare models with Accuracy, F1, BLEU, ROUGE or artificial preferences. MMLU used 57 subjects to cover knowledge and problem solving with multiple-choice questions in 2020; a multi-task collection that BIG-bench expanded to community contributions in 2022; HELM extends Accuracy to expand to Multi-dimensional indicators such as calibration, robstness, fairness, BIAS, TOXICITY, and efficiency.

The value of these benchmarks is to push model publishing from "showing examples" to "standardized reports". But the reality of 2024-2026 is that many open question banks are close to saturation, and training corpus is more likely to include open benchmarks or similar questions. For the engineering team, the most dangerous is not the low score, but the high score no longer represents the reliability of the production task.

LLM Evaluation Roadmap

roadmap for the past year

2024: More difficult, more stable, and more pollution-resistant. MMLLU-PRO cleans up MMLU noise problems, adds stronger inference requirements, and extends options from 4 to 10 to improve the degree of discrimination and Prompt stability. LiveBench takes anti-pollution as the core: the topic comes from recent math competitions, arXiv, news and datasets, updated monthly, and try to use objective answers that can be automatically verified.

2024-2025: Factuality and expert boundaries are the focus. OpenAI's SimpleQA focuses on short facts, which requires a single answer and can be judged, and distinguishes correct, incorrect, not attempted; Humanity's Last Exam Use expert-level, multi-disciplinary, and some multimodal closed-book questions to test the frontier model to respond to the problem of popular Benchmark being "popped" with high scores.

2025: The evaluation itself also needs to be evaluated. BetterBench examines benchmark quality with the BenchMark Lifecycle Best Practice, emphasizing statistical significance, reproducibility, documentation, versions, and bug fixes. The LLM Benchmark Survey in 2025 classifies hundreds of Benchmarks as general abilities, domain capabilities, and target capabilities, and also lists data pollution, cultural/language bias, and dynamic environment insufficiency as a systemic problem.

2025-2026: From static capabilities to workflow capabilities. Task-based evaluations such as SWE-bench, BrowseComp, HealthBench, and SpreadsheetBench have focused on "answering questions" to "complete tasks". This type of evaluation usually involves tool calls, file editing, browsing, test execution and long context. Some 2026 preprints have discussed the rotation of Frontier Leaderboard after saturation; the relevant conclusions have changed rapidly, and this article is marked as "waiting for manual verification".

Key papers by research question

1. General multitasking: MMLU to MMLU-Pro

MMLU’s historical importance rests on its breadth of subject coverage and a question format that is easy to reproduce. Multiple-choice scores, however, are sensitive to the distribution of answer options, memorized questions, and prompt formatting. As stronger models approach saturation, score differences become less stable. MMLU-Pro makes the test more demanding for frontier models by removing noise, requiring more reasoning, and expanding the answer choices. Private evaluations should evolve in the same way. Pass rates on an aging question bank lose their value over time.

2. Multi-indicator portrait: HELM's methodology

The key to HELM is not a list, but the evaluation view of "Scenario X indicator". A model suitable for customer service, medical, legal or code review, not only is correct, but also calibrated, stable, fair, low-toxic, and cost-controllable. The engineering evaluation table should be divided into three layers at least: task success rate, reliability indicators, and operating indicators. Without these three layers, it is difficult to convert the high score of the list into a controllable SLA.

3. Dynamic anti-pollution: LiveBench

The public question bank will enter training data, fine-tuning sets, prompt engineering examples and model publishing materials. Once contamination occurs, the score may reflect memory rather than generalization. LiveBench's route is to continuously update the questions, use recent sources, and bias objective scores. It also brings reproducing challenges, so the evaluation platform needs to maintain versions, change logs, freeze snapshots and playback mechanisms like software systems.

4. Fact: SimpleQA and SimpleQA Verified

Facts used to be mixed in open generation tasks, and scoring is easily masked by the quality of expression. SimpleQA narrows the question as a short fact Q&A, explicitly encouraging the model not to answer when it is uncertain. SimpleQA Verified pointed out that the evaluation set itself will have noise, theme bias and repetition problems, which require deduplication, theme balance and source reconciliation. The same is true for the business Golden Set: When the label is unreliable, the stronger the model, the more likely it is to expose the error of the evaluation set.

5. Expert closed volume: Humanity's Last Exam

HLE pushes the difficulty to the expert boundary. The arXiv page shows that it will be submitted in 2025, and there will still be revised versions in 2026; the questions cover multi-disciplinary mathematics, natural science, humanities and social sciences, etc., including multi-select, short answers and some multimodal questions. Its importance is to reopen the cutting-edge model gap, but the harder the Benchmark, the more dependent on expert annotation and continuous error correction. When referencing HLE, you should pay attention to the version number, public/private split and bug fix records.

6. Meta-Evaluation: BetterBench and Data Pollution Review

BetterBench shifts the focus from "Model Review" to "Review Benchmark's Benchmark", focusing on documentation, reproducible, statistical significance, data governance and maintenance. The data pollution review systematically sorts out the risks of the test set contamination. Their engineering meaning is: Benchmark without Confidence Interval, version locking and error feedback channels will soon lose explanatory power.

Comparing the methods

Route Representative work What's the solution Advantage Risk
static multitasking MMLU, BIG-bench General Competency Horizontal Comparison Easy to reproduce, wide coverage Saturation, pollution, prompt sensitive
multi-indicator portrait HELM accuracy is not enough Exposure reliability and cost balance High cost of indicators
difficulty upgrade MMLU-Pro, GPQA, HLE Decrease in the discrimination of the cutting-edge model More suitable for Frontier models High cost of labeling and error correction
Dynamic anti-pollution LiveBench Public Question Bank Leaks Reduce memory risk The version is more complicated
factuality SimpleQA, SimpleQA Verified hallucinations and uncertainty Automatic scoring is clear Coverage
task-based evaluation SWE-bench, BrowseComp Real workflow ability Approximately app HARNESS Impacts Results
yuan evaluation BetterBench Benchmark itself quality Increase credibility not directly to the model

Key Trends and Engineering Inspiration

first, Benchmark rotation will become the norm. If the paper only reports high scores on the old benchmarks, but does not explain why these benchmarks are still differentiated, the credibility will be reduced. second, Dynamic evaluation and private Holdout is more important. The open list is suitable for initial screening, and the final decision should rely on the business Golden Set and recent samples. third, Scorer itself needs to be calibrated. Rule scores, human preferences, and LLM-as-a-judge are all biased; Judge Prompt, Judge Model, temperature, version and original rating should be kept. fourth, Task-based evaluation raises the engineering threshold. Code, browsing, tables and medical reviews require sandboxes, dependencies, tool logs, retries, and security boundaries. Fifth, Reports should be upgraded from a score table to a diagnostic report: Failed clustering, Confusion Matrix, calibration curves and sample audits are more useful than single-point rankings.

These 7 practices help put evaluation into production. Maintain three sets: a public benchmark, a private reference set, and a regularly refreshed set of recent cases. For each example, record its source, reference answer, scoring function, version, and review date. Report success, refusal, hallucination, and format-error rates alongside latency and cost. Have people spot-check the LLM judge’s decisions. Use paired evaluations for model upgrades. Classify failures by cause, such as missing knowledge, faulty reasoning, failure to follow instructions, or tool misuse. Retain human sign-off for high-risk uses.

Limitations and open questions

There are still several contradictions in LLM Evaluation. Openness and anti-pollution conflicts: open question bank is easy to reproduce, and it is easier to leak; private question bank reduces pollution, but reduces transparency. Difficulty and judging conflicts: the real task is more open, the score is more difficult; the automatic score is more stable, and the task is often narrower. Conflicts between the list and product goals: The list is optimized to pursue an average score, and the production system is more concerned about tail failure, compliance and recovery capabilities.

There is also an often overlooked problem: the evaluation meeting is aging. Business processes, user language, model abilities, attack methods, and external knowledge are all changing. A review set that has not added samples and bug fixes for half a year should be regarded as a historical snapshot, rather than the only release threshold.

Summary

LLM Evaluation has changed from "arranging the model" to "building a trusted measurement system". MMLLU, BIG-bench, HELM provide standardized foundation; MMLU-Pro, LiveBench, SimpleQA, HLE, BetterBench show that the Evaluation itself also requires iteration, audit and governance. For researchers, the future paper should report protocol, version, statistical significance, failure analysis and pollution control; for the engineering team, the real Evaluation ability comes from private task set, dynamic update, traceable scoring and production regression closed loop.

References

The following data are all searched on 2026-06-25; model ranking, ranking score and repository status change rapidly, it is recommended to check the version again before citing.

  1. Dan Hendrycks et al. Measuring Massive Multitask Language Understanding. arXiv:2009.03300, 2020. https://arxiv.org/abs/2009.03300
  2. Aarohi Srivastava et al. Beyond the Imitation Game. arXiv:2206.04615, 2022. https://arxiv.org/abs/2206.04615
  3. Percy Liang et al. Holistic Evaluation of Language Models. arXiv:2211.09110, 2022. https://arxiv.org/abs/2211.09110
  4. Stanford CRFM. HELM. https://crfm.stanford.edu/helm/latest/
  5. Yubo Wang et al. MMLU-Pro. arXiv:2406.01574, 2024. https://arxiv.org/abs/2406.01574
  6. Colin White et al. LiveBench. arXiv:2406.19314, 2024. https://arxiv.org/abs/2406.19314
  7. LiveBench official site. https://livebench.ai/
  8. Jason Wei et al. Measuring short-form factuality in large language models. arXiv:2411.04368, 2024. https://arxiv.org/abs/2411.04368
  9. OpenAI. simple-evals. https://github.com/openai/simple-evals
  10. Long Phan et al. Humanity's Last Exam. arXiv:2501.14249, first submitted 2025-01-24, latest observed revision v10 on 2026-02-20. https://arxiv.org/abs/2501.14249
  11. Anka Reuel et al. BetterBench. arXiv:2411.12990, 2024. https://arxiv.org/abs/2411.12990
  12. Cheng Xu et al. Benchmark Data Contamination of Large Language Models: A Survey. arXiv:2406.04244, 2024. https://arxiv.org/abs/2406.04244
  13. Shiwen Ni et al. A Survey on Large Language Model Benchmarks. arXiv:2508.15361, 2025. https://arxiv.org/abs/2508.15361
  14. Lukas Haas et al. SimpleQA Verified. arXiv:2509.07968, 2025. https://arxiv.org/abs/2509.07968
  15. Hugging Face. Open LLM Leaderboard. https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard
  16. Adil Amin. The Growing Pains of Frontier Models. arXiv:2605.18840, 2026. https://arxiv.org/abs/2605.18840
  17. Carlos E. Jimenez et al. SWE-bench. arXiv:2310.06770. https://arxiv.org/abs/2310.06770
  18. OpenAI. Introducing SWE-bench Verified. https://openai.com/index/introducing-swe-bench-verified/