Tag

#SWE-bench

2 articles

Why we no longer evaluate SWE-bench Verified

OpenAI announces it will no longer evaluate SWE-bench Verified due to contamination and data leakage issues. The organization recommends SWE-bench Pro as a replacement.

Feb 232

OpenAI wants to retire the AI coding benchmark that everyone has been competing on

OpenAI plans to retire the SWE-bench Verified benchmark, citing flaws that undermine its validity as a coding performance measure. The move highlights concerns about memorization in AI model evaluations.

Feb 232