#benchmark-contamination on Hacker News

2 stories tagged benchmark-contamination, newest first.

Screenshot of openai.com: SWE-bench Verified no longer measures frontier coding capabilities

SWE-bench Verified no longer measures frontier coding capabilities (openai.com)

332 points by kmdupree 6 months ago | 172 comments

OpenAI research post explaining that SWE-bench Verified is increasingly contaminated and no longer measures frontier coding ability, recommending SWE-bench Pro instead.

swe-bench openai llm-benchmarks ai-coding

Screenshot of openreview.net: 30% drop in O1-preview accuracy when Putnam problems are slightly variated

30% drop in O1-preview accuracy when Putnam problems are slightly variated (openreview.net)

529 points by optimalsolver 1 year ago | 490 comments

OpenReview paper page reporting that OpenAI's o1-preview loses about 30% accuracy on slightly varied Putnam math competition problems.

llm-reasoning openai-o1 putnam math-benchmarks

#benchmark-contamination on Hacker News

SWE-bench Verified no longer measures frontier coding capabilities(openai.com)

30% drop in O1-preview accuracy when Putnam problems are slightly variated(openreview.net)

SWE-bench Verified no longer measures frontier coding capabilities (openai.com)

30% drop in O1-preview accuracy when Putnam problems are slightly variated (openreview.net)