Some critical issues with the SWE-bench dataset(arxiv.org)

331 points by joshwa 1 year ago | 113 comments

arXiv paper introducing SWE-Bench+, an empirical analysis revealing data quality issues in the SWE-bench coding benchmark for LLMs, such as solution leakage and weak tests.

Paper2Code: Automating Code Generation from Scientific Papers(arxiv.org)

122 points by Jerry2 1 year ago | 23 comments

SWE-CI: Evaluating Agent Capabilities in Maintaining Codebases via CI(arxiv.org)

102 points by mpweiher 7 months ago | 36 comments

Darwin Godel Machine: Open-Ended Evolution of Self-Improving Agents(arxiv.org)

39 points by tzury 1 year ago | 14 comments

Agentic AI and the next intelligence explosion(arxiv.org)

14 points by silverpiranha 6 months ago | 2 comments

A Study of Microsoft's Early 2026 Rollout of Claude Code and GitHub Copilot CLI(arxiv.org)

58 points by softwaredoug 3 months ago | 36 comments