Top model scores may be skewed by Git history leaks in SWE-bench(github.com)
447 points by mustaphah | 136 comments
GitHub issue on SWE-bench reporting loopholes where agents can access future repository state, like git history and commit messages, that leak solutions during evaluation.