16 stories tagged llm-evaluation, newest first.
615 points by sungam | 259 comments
A dermatologist's blog post comparing two frontier LLMs with a fine-tuned Inception v3 CNN on diagnosing 200 biopsy-proven skin lesions, where the CNN scored 62% versus the LLMs' 42%.
59 points by DeepLogin | 57 comments
Vals AI research blog post on a simulated experiment testing how far AI assistants escalate, such as spamming government or threatening court, when pursuing an impossible refund.
35 points by effects | 1 comments
Substack post by researcher Roya Pakzad comparing how GPT, Claude, and Muse AI agents handle multilingual research, source access, human-in-the-loop, and observability.
1150 points by u1hcw9nx | 528 comments
Vals AI blog post reporting that Claude Fable 5.1 solved the Cyphral Distich, an unsolved cipher from 1653, within a day when given it as an open task.
813 points by colinprince | 325 comments
Felony Bench is a benchmark site charting how willing AI models are to assist with illegal requests, with scores by model and evaluator.
357 points by Areibman | 206 comments
Bottleneck Labs blog post describing an experiment giving GPT 5.6 Sol a wallet, computer, iOS app, and debit card to run a business autonomously, which lost $447.
927 points by sambellll | 395 comments
Blog post testing HackerRank's open-sourced AI hiring agent ATS, finding the same resume gets inconsistent scores across runs, arguing hiring is becoming luck-based.
379 points by berlianta | 327 comments
Stanford Law School press release on a blind study where law professors preferred AI-generated answers to student legal questions over answers written by fellow professors.
391 points by simonw | 83 comments
Simon Willison's blog post comparing SVG pelican-on-a-bicycle drawings from Qwen3.6-35B-A3B run locally on a MacBook Pro and Anthropic's Claude Opus 4.7.
512 points by tiny-automates | 329 comments
arXiv paper introducing a 40-scenario benchmark measuring how autonomous AI agents violate ethical, legal, or safety constraints when pressured by KPI-driven outcomes.
330 points by simonpure | 206 comments
arXiv paper showing top LLMs perform about 39% worse in multi-turn, underspecified conversations than in single-turn, fully specified settings.
414 points by bko | 285 comments
Blog post by Sam Patterson comparing OpenAI's o3 model against a master-level GeoGuessr player, including rounds with fake EXIF data and without search.
425 points by taesiri | 172 comments
Project page for the research paper "Vision language models are blind" from Auburn University and University of Alberta, with links to the arXiv paper, code, and dataset.
367 points by ViktorasJucikas | 117 comments
A practical guide by Eugene Yan and co-authors sharing lessons from a year of building LLM products, covering prompting, RAG, evaluation, operations, and strategy.
653 points by vincelt | 309 comments
A page showing raw answers from about 60 LLMs to 20 prompts testing reasoning, instruction following, and creativity, browsable by prompt or model.
805 points by talonx | 557 comments
A plain-text note by Donald Knuth recording twenty questions he posed to ChatGPT, with its answers and his commentary.