Stop Over-Trusting AI Coding Leaderboards

Accepted for publication in Communications of the ACM.

Summary

AI labs use coding benchmark scores to promote new models, and companies use the same scores to decide which tools to buy and adopt. Many of the leaderboards behind those scores do not deserve that trust. OpenAI released SWE-bench Verified in August 2024 and retired it in February 2026. That reversal is the most visible instance of a problem peer-reviewed work has been flagging for years: pretraining contamination, ground-truth errors, and scores that reflect pattern matching rather than capability. I argue for one norm: no leaderboard without an audit trail. The trail should show audit results, the hidden requirements that tests assume, leakage checks, and a second score on any cleaner subset. The same standard should apply to proprietary internal benchmarks.

Why Headline Scores Need More Scrutiny

Anyone who chooses or deploys an AI coding assistant today relies on a leaderboard score, even if they never open the leaderboard. In August 2024, OpenAI introduced SWE-bench Verified as a more reliable way to measure AI coding ability [7]. Eighteen months later, on February 23, 2026, the same lab published “Why SWE-bench Verified no longer measures frontier coding capabilities,” arguing that its own benchmark should no longer be used to judge the strongest coding models [8]. By then, SWE-bench Verified scores had become a fixture of launch posts, sales decks, executive briefings, and internal adoption plans.

SWE-bench gives a model a real bug report or change request from an open-source project on GitHub and asks for a patch. The project’s own test suite then grades the patch. SWE-bench Verified is a human-filtered subset of those tasks, meant to weed out the broken and underspecified ones. It is a sound idea.

Over the past two years, frontier-model releases have used SWE-bench Verified as the standard measure of progress in AI coding. OpenAI and Anthropic place its score beside their other headline numbers in release notes.¹ Trade press reports those scores as proof of enterprise readiness. A survey of 100 enterprise CIOs finds that external benchmark scores have become a routine input to model selection.²

Imagine an engineering director who sees “70% on SWE-bench Verified” in a vendor deck. Sure that the model can handle a bug-triage rotation, the director reorganizes a team around it. The director does not learn that some passing tasks may have been absorbed during model pretraining, that some were graded against requirements the task never stated, or that the benchmark’s own authors have since disowned it as a measure of frontier capability. Papers and vendor decks repeat a number that should be read with caution. Engineering leaders then treat it as a sign that the model is ready to deploy.

The fix is to stop publishing leaderboard numbers without context. The norm should be simple: no leaderboard without an audit trail. The trail should travel with the score. It should report how the benchmark was manually reviewed and what was removed, which interface details its tests silently require, what was done to limit leakage, and how much uncertainty remains. When part of a benchmark is cleaner than the rest, the trail should include both scores. The same standard should apply when the headline number comes from a proprietary internal benchmark. The research community has documented the problem for three years and has already proposed most of these disclosures.

SWE-bench Verified as the Case Study

The more often SWE-bench Verified appeared in launch materials, the more its flaws mattered. OpenAI’s February 2026 retirement post names three problems [8]. Narrow tests reject correct answers when an unstated implementation detail, such as a particular helper-function name in the reference solution, does not match. Wide tests fail correct answers when the grading harness silently checks for behavior the task description never disclosed. Leakage from public exposure means a high score may reflect prior absorption rather than capability.

OpenAI’s prescribed remedy was to migrate evaluation to SWE-Bench Pro. That recommendation lasted five months. In July 2026, OpenAI audited Pro’s public split, and five software engineers reviewed each flagged task. About 30% of the split’s tasks turned out to be broken. OpenAI wrote: “Given the issues uncovered in this analysis, we retract our earlier recommendation to adopt SWE-Bench Pro.”³ Both benchmarks fit a general pattern. A static benchmark gets adopted, scores climb toward the ceiling (23.3% to 80.3% on Pro’s public split in eight months), and it is replaced by a newer one. Neither reversal produced an audit trail. OpenAI published the finding but not the per-task labels.

The labs already know about the memorization risk. In OpenAI’s April 2026 GPT-5.5 launch, the SWE-Bench Pro (Public) score carries an asterisk noting that “labs have noted evidence of memorization.”⁴ Anthropic’s Claude Opus 4.7 release documents in a footnote that “memorization screens flag a subset of problems” across SWE-bench Verified, Pro, and Multilingual.⁵ The labs now see enough to footnote their own headline numbers. They keep reporting them anyway.

Launch materials now also include undisclosed internal benchmarks next to the public ones. OpenAI’s GPT-5.5 announcement reports 73.1% on a new “Expert-SWE (Internal)” benchmark, compared with 68.5% for GPT-5.4. The announcement reports no results for other labs’ models and discloses nothing about the benchmark’s composition or vetting. The Opus 4.7 release reports scores on an “internal autonomous agentic coding evaluation,” compared only against Anthropic’s prior model, with a caption noting that “results may not be representative of token usage in interactive coding.” Cursor’s Composer 2.5 announcement, in May 2026, headlines scores on its own “CursorBench v3.1 (Harder Tasks).”⁶ No one outside these companies can check any of these numbers.

A Familiar Pattern

The SWE-bench Verified reversal is the most institutionally embarrassing case so far, but it is not the first. At ACL 2024, Riddell, Ni, and Cohan documented the same failure mode on widely cited code benchmarks: a 32–62.5% gap between model performance on HumanEval and MBPP examples most similar to pretraining data and those least similar to it [9]. When a benchmark and its public test items have been on the internet long enough, scores on it start to measure exposure as much as capability.

Static code benchmarks rot in a predictable way. LiveCodeBench (ICLR 2025) collects coding problems in time-stamped windows so it can detect contamination. It found contamination signals across several frontier models and showed that HumanEval scores have saturated [4]. The same thing happens on SWE-bench. Liang and colleagues found that models identify the correct buggy file from issue text alone with up to 76% accuracy on SWE-bench repositories, but only about 53% on repositories outside the benchmark [5].

The problem also appears outside code, and even outside computing. GSM-Symbolic (ICLR 2025) showed that changing numbers in grade-school math problems reduced model accuracy [6]. These findings are hard to reconcile with the claim that the original scores reflected reasoning. Medicine has its own example. In 2023, after decades of clinical use, the American Medical Association advised physicians to stop treating body mass index as a standalone measure of health. By contrast, SWE-bench Verified went from launch to retirement in eighteen months, and OpenAI did both.

Even the newer benchmarks designed against this pattern have not changed the reporting habit. SWE-Bench Pro builds a private held-out split from separate repositories to catch overfitting [3], but that split helps only if labs use it. The scores labs report, including GPT-5.5’s and Claude Opus 4.7’s, are on Pro’s public subset. That subset carries the same exposure risk as any other static benchmark. For proprietary models, outside evaluators cannot inspect the training data or the filtering rules. So a check that finds no evidence of leakage does not prove there was none.

The failure modes in SWE-bench Verified all have a documented track record: leakage, ground-truth errors, pattern matching, and tests that grade against undisclosed requirements. Yet the community kept reporting the scores.

What an Audit Trail Should Include

None of this argues against benchmarks. A benchmark with a documented audit history and a published leakage screen is more useful as a procurement input than a score taken on faith. Disclosure has real costs. It slows publication, and authors must defend their methods on the record. These are the costs of treating measurement as evidence rather than marketing.

Prior work proposes specific measures: contamination measurement, training-exclusion guarantees, and contamination-resistant design [4, 10]. These measures have not become leaderboard norms. Every leaderboard number should travel with five artifacts:

What every leaderboard should publish alongside its scores

• Audit results. How many tasks were manually reviewed, what problems were found, and which tasks or tests were removed, so a reader can see where the benchmark got cleaner and where uncertainty remains.
• Hidden requirements. If passing depends on exact names, helper functions, file layouts, or routes, those interface choices⁷ belong in the task description, not buried in the grading harness [2].
• Leakage checks. What was done to reduce contamination, what residual uncertainty remains, and, for proprietary models, a plain statement of where outside verification stops and provider attestation begins. Include behavioral probes and cross-benchmark comparisons, not only lineage claims [5].
• Multiple scores when needed. If part of a benchmark is better audited or lower-risk than the rest, report both the full score and the score on the cleaner subset.
• Internal benchmarks. If a release headlines a private or internally constructed benchmark, the same disclosures apply: composition, task sourcing, audit results, and scores for at least one comparable model on the same evaluation. Without these disclosures, only the lab can check the number.

Some of this is already happening. NoCode-Bench is built around the natural-language feature requests its tasks specify, rather than leaving the target behavior implicit [2]. SWE-PolyBench has added a Verified split with curated annotations and an updated leaderboard [1]. SWE-Bench Pro deliberately sources from GPL-licensed repositories to reduce training-data overlap [3]. But every one of these is a maintainer choosing, on their own, to do better. None of it is expected, let alone required to list a score.

I do not think current disclosure practices are enough. Even popular leaderboards fail on interpretability, the property that lets a reader judge what a score means. The duty applies to model labs as well as benchmark maintainers. A lab that reports a benchmark score takes on a share of the work of annotating and cleaning it. A footnote is not a remediation.

No Leaderboard Without an Audit Trail

Procurement teams and engineering leaders do not read grading code or audit notebooks. They see a ranked list of scores and a claim that one model is best. If the field already knows that hidden requirements and leakage can distort scores, leaderboards without that context invite over-trust, and over-trust has costs. Teams commit budgets against inflated numbers and deploy AI coding tools past their real reliability. Some organizations find the gap in production instead of during procurement.

The norm should be simple: no leaderboard without an audit trail. Until that disclosure is routine, treat leaderboard scores as provisional signals of coding capability.

The mechanisms already exist. ACM and IEEE software engineering venues, and the dataset-and-benchmark tracks at the major machine learning and AI conferences, can require a structured disclosure checklist in benchmark papers. Leaderboard hosts can refuse to list a score that has no audit-trail fields. Enterprise buyers already demand security and compliance attestations from vendors, and they can make audit-trail disclosure a procurement criterion too. Procurement offers the strongest incentive. Budgets and renewals made SOC 2 reports and accessibility compliance standard practice.

The record includes three years of peer-reviewed warnings and an explicit reversal by the lab that launched the field’s flagship coding benchmark. Frontier labs now footnote memorization caveats beside their own headline scores. There is no remaining excuse for silence.

Disclosure

The ideas and opinions in this article are my own, informed by my work on coding agents. They do not necessarily reflect the position of my employer. I used a generative AI tool to structure my notes into this article. I then reviewed and adapted the text.

References

1. Amazon Science. 2025. SWE-PolyBench Repository Updates. https://github.com/amazon-science/SWE-PolyBench. README update announcing the SWE-PolyBench Verified split with curated annotations and an updated leaderboard. Accessed March 21, 2026.
2. Le Deng, Zhonghao Jiang, Jialun Cao, Michael Pradel, and Zhongxin Liu. 2025. NoCode-Bench: A Benchmark for Evaluating Natural Language-Driven Feature Addition. arXiv preprint arXiv:2507.18130. https://arxiv.org/abs/2507.18130. Accessed May 10, 2026.
3. Xiang Deng, Jeff Da, Edwin Pan, Yannis Yiming He, Charles Ide, Kanak Garg, Niklas Lauffer, Andrew Park, Nitin Pasari, Chetan Rane, Karmini Sampath, Maya Krishnan, Srivatsa Kundurthy, Sean Hendryx, Zifan Wang, Vijay Bharadwaj, Jeff Holm, Raja Aluri, Chen Bo Calvin Zhang, Noah Jacobson, Bing Liu, and Brad Kenstler. 2025. SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks? https://arxiv.org/abs/2509.16941. Project page: https://scale.com/research/swe_bench_pro. Accessed March 21, 2026.
4. Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. 2025. LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code. In The Thirteenth International Conference on Learning Representations (ICLR). https://arxiv.org/abs/2403.07974. Accessed May 6, 2026.
5. Shanchao Liang, Spandan Garg, and Roshanak Zilouchian Moghaddam. 2025. The SWE-Bench Illusion: When State-of-the-Art LLMs Remember Instead of Reason. https://arxiv.org/abs/2506.12286. Accessed March 22, 2026.
6. Iman Mirzadeh, Keivan Alizadeh, Hooman Shahrokhi, Oncel Tuzel, Samy Bengio, and Mehrdad Farajtabar. 2025. GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models. In The Thirteenth International Conference on Learning Representations (ICLR). https://arxiv.org/abs/2410.05229. Accessed May 6, 2026.
7. OpenAI. 2024. Introducing SWE-bench Verified. https://openai.com/index/introducing-swe-bench-verified/. Accessed March 21, 2026.
8. OpenAI. 2026. Why SWE-bench Verified no longer measures frontier coding capabilities. https://openai.com/index/why-we-no-longer-evaluate-swe-bench-verified/. Accessed March 21, 2026.
9. Martin Riddell, Ansong Ni, and Arman Cohan. 2024. Quantifying Contamination in Evaluating Code Generation Capabilities of Language Models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Bangkok, Thailand. https://aclanthology.org/2024.acl-long.761/. Accessed May 6, 2026.
10. Oscar Sainz, Jon Ander Campos, Iker García-Ferrero, Julen Etxaniz, Oier Lopez de Lacalle, and Eneko Agirre. 2023. NLP Evaluation in Trouble: On the Need to Measure LLM Data Contamination for Each Benchmark. In Findings of the Association for Computational Linguistics: EMNLP 2023. Association for Computational Linguistics, 10776–10787. doi:10.18653/v1/2023.findings-emnlp.722. Accessed May 6, 2026.

Notes