7 stories tagged ai-evaluation, newest first.
7 points by gcolella | 0 comments
An interactive quiz from the Kairos experiment asking readers to pick which of five passage pairs comes from an AI agent's real 30-day memory versus a model-fabricated one.
558 points by speckx | 495 comments
Mercor research page presenting a Benchmark Map that measures how well existing AI benchmarks cover tasks across American jobs.
561 points by Anon84 | 136 comments
UC Berkeley RDI blog post describing how researchers exploited major AI agent benchmarks like SWE-bench and Terminal-Bench to score highly without solving tasks, and proposed fixes.
481 points by lairv | 317 comments
ARC Prize page for ARC-AGI-3, an interactive game-based benchmark for testing AI reasoning, with a playable demo and grid of puzzle games.
637 points by __rito__ | 261 comments
Andrej Karpathy blog post about using LLMs to grade decade-old Hacker News discussions and commenters' predictions with hindsight, presented as an "HN Time Capsule" site.
397 points by pseudolus | 186 comments
Oxford Internet Institute news item on a study finding that most of 445 LLM benchmarks lack rigorous methods and clear definitions of what they measure.
695 points by noddybear | 200 comments
Project page for the Factorio Learning Environment, an open-source framework and benchmark for evaluating LLM agents building factories in Factorio.