# OpenAI says it found widespread task issues in SWE-Bench Pro, estimates ~30% of tasks are broken, and retracts its earlier recommendation to adopt the benchmark (OpenAI) — retained record

> Retained techmeme record from techmeme.

Canonical: https://highsignal.app/data/records/421d78e9e7f55664

[← sources](https://highsignal.app/data)


source: [techmeme](https://highsignal.app/data/techmeme) published 2026-07-08 21:10 ingested 2026-07-09 09:28

[original source ↗](https://openai.com/index/separating-signal-from-noise-coding-evaluations)

## retained text



```
Techmeme permalink: https://www.techmeme.com/260708/p41#a260708p41

<a href="https://openai.com/index/separating-signal-from-noise-coding-evaluations"><img align="RIGHT" border="0" hspace="4" src="http://www.techmeme.com/260708/i41.jpg" vspace="4" /></a>
<p><a href="https://www.techmeme.com/260708/p41#a260708p41" title="Techmeme permalink"><img height="12" src="http://www.techmeme.com/img/pml.png" style="border: none; padding: 0; margin: 0;" width="11" /></a> <a href="https://openai.com/">OpenAI</a>:<br />
<span style="font-size: 1.3em;"><b><a href="https://openai.com/index/separating-signal-from-noise-coding-evaluations">OpenAI says it found widespread task issues in SWE-Bench Pro, estimates ~30% of tasks are broken, and retracts its earlier recommendation to adopt the benchmark</a></b></span>&nbsp; &mdash;&nbsp; Through a detailed audit, we find widespread task issues in SWE-Bench Pro and estimate that ~30% of the tasks are broken.</p>
```



document fetched 2026-07-09 09:28
