.png)
_logo%201.svg)
AI Summary
- GLM-5.2 fell 19.29 points after re-annotation; DeepSeek-V4-Pro fell 0.05.
- Zero of 186 pass-to-fail flips came from impaired execution.
- Test scoping, not prose, caused 75 of 102 defects.
- OpenAI's human reviewers flagged more broken tasks than its automated pipeline.
- Four parties audited the benchmark before the fix shipped.
Introduction
Between two evaluations of the same 731 tasks, GLM-5.2's score fell from 78.80% to 59.51%. DeepSeek-V4-Pro's moved by five hundredths of a point. Neither model was retrained, re-prompted, or swapped for a newer checkpoint between the two runs. What changed was the benchmark.
The numbers come from an AI benchmark audit published on 8 September 2026. A team at Shanghai AI Lab ran it, with East China Normal University and Fudan University. Zheng and colleagues took the public split of SWE-Bench Pro and inspected every instance. They revised 102 tasks and discarded 17. They also rebuilt the execution environment, closing the routes agents had used to reach the answers: the file system, the git history, and the network. Then they re-ran the models.
Their paper is nine days old as we write this. It is a preprint, not peer reviewed. It is also not the first time someone found this benchmark leaking. A GitHub user reported the git-history problem nine days after the benchmark shipped. Datacurve found it in May 2026, Cursor quantified it in June, and OpenAI publicly retracted its own recommendation of SWE-Bench Pro in July. The September paper is the repair at the end of that sequence.
The repair produced an arithmetic. Most arguments about benchmark quality end at the assertion that benchmarks are imperfect. This one ends at a number. Here is how far a score moves when you correct the annotation. Here is the evidence that the annotation caused the move, rather than anything the model did.
What Did the AI Benchmark Audit Actually Change?
Two things, and keeping them separate matters for reading the results.
The first is what the authors call anti-hacking: closing the channels through which an agent could retrieve a solution rather than derive one. Zheng and colleagues identify four. The local file system exposed gold patches (the official fix for each task), hidden tests, fixtures, and evaluator artefacts. Git history exposed future commits, branches, tags, remotes, and reflogs. The external network exposed upstream commits, mirrors, and raw files. Task metadata exposed target SHAs and repository identities.
The safeguards are per-channel and unglamorous. Repositories were rebuilt as fresh single-commit repos with nested histories removed. Hidden test artefacts were deleted and git hooks disabled. Instance IDs were anonymised and repository names stripped from visible paths. A domain blocklist covered GitHub, GitLab, Gitee, Bitbucket, Codeberg, and GitCode.
The second change is refinement: fixing the tasks themselves. Of 731 instances, 119 were flagged as candidates, 102 were revised, and 17 were rejected as unfixable. The breakdown does not point where intuition points.
The grading criteria failed more often than the task statements did. Seventy-five of 102 refined instances had tests that were too narrow to accept a correct solution. Problem prose accounted for 22. If you maintain an internal evaluation set and you have been reviewing the prompts, you have been reviewing the smaller problem.
Why Did One Model Lose 19 Points and Another Lose None?
Here is the central exhibit. Every block is labelled with its source, because no single source covers all of it, and unlabelled numbers are how this situation began.
A. From the audit paper (Zheng et al. 2026, Table 3). SWE-Bench Pro public split, 731 instances, mini-swe-agent harness (the scaffolding that hands the task to the model and decides whether its patch passes).
The paper evaluated seven models and plots all of them in Figure 1, but publishes numeric before-and-after values for these two. We are not going to invent the other five.
B. From Cursor (Jain 2026). SWE-bench Pro, Cursor's own standard versus strict harness.
The two-model contrast in block A is the finding itself. On the old benchmark, GLM-5.2 outscored DeepSeek-V4-Pro by 29 points. Block retrieval and the two finish within ten points of each other. That 29-point gap measured how readily each model reached for the answer rather than how well it wrote the patch.
The obvious objection is that the sandbox broke: lock down the file system and the network hard enough and you degrade normal execution too. The paper anticipates this. Under anti-hacking, 186 instances flipped from pass to fail for GLM-5.2. Table 8 of the paper attributes 166 of them to direct evidence of hacking being removed, 3 to highly probable hacking, and 17 to randomness or insufficient evidence. Zero are attributed to impaired normal execution.
That single row is the closest thing in this literature to a controlled experiment separating item quality from model capability. It is also one run, one harness, one preprint, with no confidence intervals published.
Who Audited SWE-Bench Pro Before This Paper?
Scale AI built SWE-Bench Pro and released it in September 2025. It holds 1,865 human-verified instances: 731 public tasks across 11 GPL repositories, 858 held-out tasks, and 276 commercial ones. Scale AI chose GPL and copyleft licensing deliberately, to cut the odds of those repositories sitting in commercial training corpora. The tasks are long-horizon by construction, averaging 107.4 lines of code across 4.1 files. At launch, GPT-5 scored 23.3% on the public split and Claude Opus 4.1 scored 22.7%. SWE-bench Verified was returning 70%-plus at the time. That gap is what made Pro attractive as a replacement.
Then the timeline starts.
Datacurve's underlying methodology is not published in a paper, so the May percentages are the company's own. Cursor's June figures are the ones reproduced in block B above. OpenAI's July review ran two ways: the automated pipeline flagged 200 broken tasks (27.4%), its human reviewers 249 (34.1%), with the gap widest on low-coverage tests.
Eighteen months, four independent parties, one closed issue, one retraction, one corrected release. Benchmark maintenance has a cadence. Most teams quoting public benchmark numbers in procurement decks are not tracking it.
One note on who is doing the auditing. Shanghai AI Lab publishes the OpenCompass evaluation suite, which makes them an evaluation vendor examining a competitor's product. Scale AI is a commercial annotation vendor that built the benchmark. OpenAI is a benchmark consumer with a stake in which ruler measures it. None of that invalidates the findings, all of which are reproducible or self-critical, but it is worth saying out loud in an area where nobody is a neutral party.
Is a Cleaner Benchmark Really a More Informative One?
This is the strongest objection to the paper and it is already on the record. A commenter on the paper's Hugging Face discussion page asked exactly that: if the ranking of models does not change, what has the cleanup bought? The author's reply pointed to Figure 1. The paper publishes no rank-correlation statistic.
So the honest reading is narrower than the headline invites. The audit demonstrates that score levels move, by a measurable amount, for at least one model. It does not demonstrate that the leaderboard order changes, and anyone citing this work to claim a reshuffled ranking is going past the evidence.
Two further caveats belong in the same breath. The authors list their own limits. The domain blocklist may miss self-hosted git services, private proxies, dynamic domains, third-party mirrors, and direct IP access. Some repositories may still hold residual evaluation data. And the refinement process, they write, may not identify every task-quality issue. SWE-Bench Pro Verified is a better instrument than SWE-Bench Pro. It is not a clean one, and its authors do not say it is.
The second caveat concerns what "minimal correction" means in practice. Among the 102 refined instances, 92 had their requirements field changed, 60 had interface changed, 59 had problem_statement changed, and 17 had test_patch changed. Refinement alone flipped 21 instances from fail to pass and 2 from pass to fail. Editing the requirements on 90% of the instances you touch is a substantial redefinition of what those tasks ask for. Defensible, given what the defects were. Neutral, no.
What Does an AI Benchmark Audit Cost to Run?
Nobody publishes a per-instance cost or time figure. We looked for one and found none.
The one hard staffing number in this literature is four years old. When OpenAI and the original Princeton authors built SWE-bench Verified, they hired 93 professional Python developers to annotate 1,699 randomly sampled instances from the original SWE-bench. They filtered out 68.3% of what they looked at: 38.3% for underspecified problem statements, 61.1% for unit tests that could unfairly reject a valid solution. Roughly 3.4 annotated items went in for every usable item that came out. GPT-4o's score doubled, 16% to 33.2%.
The 2024 re-annotation moved scores up, because the items were unfairly hard. The 2026 re-annotation moves them down, because the environment was leaky. A score can be wrong in either direction, and re-annotation is the only instrument that tells you which.
Against OpenAI's 93 developers, set what SWE-Bench Pro's own paper discloses about its annotation pipeline. It describes three human-in-the-loop stages and names three contributors in acknowledgements. It reports no annotator count, no expertise criteria, no per-task time, no inter-annotator agreement, and no cost. It documents one control: run each test three times and filter out any test that does not pass consistently. That is flakiness control, applied to tests whose dominant defect turned out to be scope.
This pattern is not specific to coding benchmarks. A 2026 taxonomy of benchmark contamination found elicitation budgets, meaning how much compute and how many attempts a model was given, disclosed in 13% of the documents it coded. No document addressed all five contamination types it defines. Its own two external coders agreed poorly with each other: linear-weighted kappa, a standard measure of agreement between raters, ran from 0.00 to 0.35, median 0.21, against a single-coder ceiling of 0.84. The authors report that plainly. A paper that measures the reliability of its own instrument and publishes a poor number is doing the thing this whole field needs more of.
Two decades of measurement research point the same way. Northcutt, Athalye, and Mueller found average label error of at least 3.3% across ten widely used test sets, and at least 6% in ImageNet validation. Correcting those labels reverses model rankings: ResNet-18 beats ResNet-50 on corrected ImageNet labels once the error rate passes 6%. Sambasivan and colleagues interviewed 53 high-stakes AI practitioners across India, East and West Africa, and the USA. They found data cascades in 92% of them, and described data as the most under-valued and de-glamorised part of the work.
Validity problems in graders show up wherever anyone looks for them. Validity-aware jailbreak evaluation reclassifies between 22.1% and 51.0% of previously labelled successes as invalid across three of four public benchmarks. OSReward finds a systematic leniency bias in vision-language judges of agent trajectories: they mark failed runs as successes. The judges reliable enough to trust cost too much to run at scale. Sample-level auditing of MMLU, ARC, WinoGrande, HellaSwag, and TruthfulQA finds internal heterogeneity that aggregate accuracy conceals.
Why Did Human Reviewers Catch More Than the Automated Pipeline?
OpenAI's July retraction compared its two ways of reviewing the tasks. The automated pipeline flagged 200 of the 731 public tasks as broken. Its human reviewers flagged 249. The people caught more. The gap was widest on low-coverage tests, which is exactly where a test suite is least able to tell a correct patch from an incorrect one.
The OSReward results cited above show the same asymmetry from the other direction. There, the lenient party is the automated judge. An automated grader is itself a model. It fails on the cases its training made hard, which are usually the cases that decide the score.
The two benchmark papers answer the next question differently. OpenAI disclosed that it hired 93 professional Python developers, people who could read a test suite and say whether it would reject a valid fix. SWE-Bench Pro's paper reports no annotator count, no expertise criteria, and no agreement statistic. Between those two disclosures sits the whole question of whether a benchmark's items mean what they claim to mean.
Expertise and measurement are one requirement, not two. A reviewer without domain knowledge cannot tell an overly narrow test from a correctly scoped one. And a qualified reviewer whose agreement with peers is never measured leaves you with an assertion instead of evidence. The audit chain above is what it looks like when neither half is in place at the time the dataset ships.
What Should You Do If You Depend on These Numbers?
Start with the ones already in your slides. MiniMax's launch post reports M3 at 59.0% on SWE-Bench Pro alongside 80.5% on SWE-bench Verified, and states that Claude Code was used as the scaffolding for several of those benchmarks. That disclosure is more than most vendors offer, and it is also an admission that the harness is part of the number. Every launch-post figure for SWE-Bench Pro published before this month came from the pre-audit environment, on a public split OpenAI judged roughly 30% broken. No vendor has restated. Our own MiniMax M3 data story cited that 59.0% figure, which is now a number with an asterisk.
Claude Opus 4.8 on SWE-bench Pro: 87.1% on Cursor's standard harness, 73.0% on Cursor's strict one, 69.2% in the comparison table we ran earlier this year. Same benchmark, same model, roughly 18 points of spread depending on who ran it and how.
Keep using public benchmarks. Just stop treating a public score as a property of the model. A score is a joint property of five things: the model, the harness, the elicitation budget, the item population, and the contamination status of the run. Vendors report the first one. So if you need to know whether a model can do your work, run the evaluation on your own items, against your own pass criteria. The people judging should already know what correct means in your domain: the clinician, the quality engineer, the document reviewer. We wrote the method out in full in our guide to building a custom AI benchmark.
Three things are worth borrowing from this audit even if you never build a benchmark from scratch. Review your tests before your prompts, because scope defects outnumbered prose defects three to one here. Catch quality problems inside the labeling workflow rather than after it. SWE-Bench Pro's documented control was a flakiness filter, applied to tests whose real defect was scope. No amount of re-running a narrow test makes it wider. Kili builds around the mechanisms that sit inside the workflow instead. Consensus compares annotators working on the same asset. Honeypot labels score work against a known reference at a chosen step. You configure review steps without writing code.
Then record the conditions alongside the score: harness version, network policy, git state, retry budget, who reviewed the item and at which review step. That record has to come out of the same workflow that produced the judgment, because a provenance trail assembled afterwards is a reconstruction. The eighteen months above are what happens when a whole field publishes numbers without one.
The Ruler Moved, Not the Runner
The temptation with a story like this is to conclude that AI coding benchmarks are broken. That sentence is overreach, and it is old news. We have published versions of it in our AI benchmarks guide, our agentic AI benchmarks guide, and our piece on Kimi K3's benchmarks and hallucinations. The narrower claim is the useful one. A benchmark score moves when you re-annotate the benchmark, by an amount you can measure. The only way to learn that amount is to do the annotation work.
Nobody has published what that work costs. OpenAI published a headcount and no hours. Scale AI published neither. Shanghai AI Lab published a corrected dataset and the count of instances it touched. The maintenance budget of the instruments the entire industry uses to rank models is, as of today, undisclosed by everyone who has ever paid it.
That is the gap to watch over the next year. The benchmarks that survive contact with capable agents will carry four things: a maintenance cadence, a versioned item set, qualified reviewers whose agreement is measured on a sampled share of items, and a published account of the conditions behind each score. That is an ongoing evaluation programme, which is also why the work keeps falling to whoever is willing to re-read 731 tasks by hand. Evaluation sets decay for reasons beyond leakage. Models recognise when they are being tested, a mechanism we covered in our analysis of evaluation awareness. The same discipline answers both. Your evaluation set is not infrastructure you install. It is infrastructure you maintain, and if nobody owns it, it is already drifting.
Resources
The audit and the corrected dataset
- SWE-Bench Pro Verified: A reliable benchmark for software engineering agents – Zheng et al., Shanghai AI Lab, Sep 2026 preprint
- SWEBench-Pro-Verified dataset – the corrected 731-instance release, Apache 2.0
- Community discussion on the paper – the ranking objection, on the record
The original benchmarks
- SWE-bench: Can language models resolve real-world GitHub issues? – the Princeton original
- SWE-Bench Pro: Can AI agents solve long-horizon software engineering tasks? – Scale AI, 2025
- Introducing SWE-bench Verified – OpenAI's 2024 re-annotation, 93 developers
The audit chain
- Repo state leakage (issue #7) – filed nine days after release, Sep 2025
- Git reward hacking in SWEBench Pro OSS (issue #93) – Poolside AI's evaluations team
- Why we no longer evaluate SWE-bench Verified – OpenAI, Feb 2026
- DeepSWE blows up the AI coding leaderboard – Datacurve findings via VentureBeat, May 2026
- Reward hacking is swamping model intelligence gains – Jain, Cursor, Jun 2026
- Separating signal from noise in coding evaluations – OpenAI's retraction, Jul 2026
Benchmark validity research
- Benchmark contamination: A taxonomy organized by defeated mitigation – Angulo, Yeste & Espinos-Morato, 2026
- Validity-aware jailbreak evaluation for large language models – Wu et al., 2026
- Benchmarks are not monolithic: Sample-level auditing for LLM evaluation – Siedler & Sassoon, 2026
- OSReward: Standardized evaluation for cross-platform computer-use reward models – Sun et al., 2026
- Pervasive label errors in test sets destabilize machine learning benchmarks – Northcutt, Athalye & Mueller, NeurIPS 2021
- "Everyone wants to do the model work, not the data work": Data cascades in high-stakes AI – Sambasivan et al., CHI 2021
Vendor reporting
- MiniMax-M3 launch post – self-reported SWE-Bench Pro and SWE-bench Verified scores
Further reading from Kili
- How to build a custom AI benchmark – the five-phase method
- AI benchmarks guide: the top evaluations in 2026 – what public benchmarks do and do not measure
- Agentic AI benchmarks guide – how agentic evaluations work, and how they get gamed
- Kimi K3's benchmarks and hallucinations – when the scoring rule is the flaw
- The evaluation gap – why models that pass in the lab fail in production
- Evaluation awareness and the Muse Spark report – models detecting test conditions
- Data Story: MiniMax M3 – the article that cited the pre-audit 59.0% figure
- Kili product documentation: consensus – how inter-annotator agreement is configured and scored
Ready to Put Your Experts in Charge of the Judging?
If your model decisions rest on public leaderboard numbers, the evaluation gap between those scores and your production reality is the thing to close first. Kili is the expert-in-the-loop platform for AI evaluation and post-training. The people who hold the domain expertise do the labeling and the review themselves. You configure consensus and multi-step review in the product rather than building them. Every label carries its author and its review step into the export. Run it on our cloud, on your Azure subscription, or on your own Kubernetes.
