OpenAI published an audit of SWE-Bench Pro yesterday and estimates that roughly 30% of its tasks are broken. The company is retracting its recommendation that the research community use it as a leading coding eval.
This is the second time in recent months OpenAI has found major issues with a coding benchmark it previously vouched for. A similar audit of SWE-bench Verified uncovered fundamental design and contamination problems, and at the time the company urged the community to switch to SWE-Bench Pro. Now that recommendation is gone too.
Here’s what OpenAI found and why it matters if you’re building or buying coding agents.
What OpenAI Found
SWE-Bench Pro tests models on longer coding tasks pulled from actual feature changes in public and private repositories. Models have to implement a solution that passes new tests without breaking existing functionality. On the 731-task public split, frontier models shot from a 23.3% pass rate to 80.3% in just eight months.
That jump looked impressive. But OpenAI suspected the benchmark wasn’t measuring what it claimed.
The team built a datapoint analysis pipeline that reviewed model attempts, task metadata, and failure traces to flag likely evaluation flaws. The automated filter flagged 286 potentially broken tasks. OpenAI then validated those flags with two deeper methods: a human-supervised agent review using Codex-based investigator agents, and a human annotation campaign staffed by five experienced software engineers.
The results were damning. The pipeline marked 200 tasks (27.4%) as broken. The human campaign flagged 249 tasks (34.1%). OpenAI’s combined estimate: roughly 30% of the benchmark is compromised.
The Four Failure Modes
OpenAI identified four categories of defects in the SWE-Bench Pro dataset:
Overly strict tests. Some hidden tests enforce specific implementation details that were never stated in the prompt. A functionally correct solution gets marked as failed because it didn’t guess the exact approach the test author had in mind.
Underspecified prompts. The prompt omits requirements that the hidden tests enforce and that aren’t reasonably inferable from the context. The model has no way to know what the grader expects.
Low-coverage tests. The test suite doesn’t adequately check the requested feature. An incomplete or incorrect patch can pass because the tests don’t cover enough ground.
Misleading prompts. The prompt points the model toward the wrong behavior or directly contradicts what the tests require. The model is set up to fail by design.
Each of these issues means a pass or fail on SWE-Bench Pro doesn’t cleanly map to actual coding capability. A high score could mean the model is good at guessing hidden test constraints. A low score could mean it ran into a broken task.
What This Means for the AI Coding Market
Coding benchmark scores have become procurement signals, release gates, and marketing claims. When a benchmark this widely cited has a 30% defect rate, the scores don’t mean much.
I covered how coding benchmark scores factor into model comparisons in my OpenCode Go review, where the benchmark was cited as evidence of coding capability. That review still stands, but it is a reminder that any score is only as good as the data behind it.
For teams evaluating coding agents, the practical takeaway is straightforward: don’t treat public benchmark movement as proof of production readiness. A model that jumps 10 points on SWE-Bench Pro may simply be better at navigating broken tasks, not genuinely better at software engineering.
OpenAI is advising model developers to carefully examine results and is calling for benchmarks built with stronger human oversight. The company says the findings point to the difficulty of curating hard but fair benchmarks and the growing utility of agents for scalable data quality checks.
Caveats
The audit covered the 731-task public split of SWE-Bench Pro. The full dataset includes tasks from private repositories as well, and OpenAI didn’t report separate breakage rates for that portion. The pipeline and human campaign also produced different estimates (27.4% vs 34.1%), so the exact number depends on methodology.
OpenAI hasn’t announced a replacement benchmark or a timeline for one. The company’s earlier positive stance on SWE-Bench Pro was based on its design improvements over SWE-bench Verified, which itself crumbled under audit.
Bottom Line
OpenAI’s own coding agent, Codex, helped find these issues. That’s a good sign for the use of agents as evaluation tools. But it’s a bad sign for an industry that has been using SWE-Bench Pro scores as shorthand for coding capability.
The lesson is the same one that keeps surfacing in AI evaluation: benchmarks rot, and the default should be skepticism until proven otherwise. If you’re making buying decisions or release calls based on benchmark scores, audit the tasks yourself.
