Understand the scores.
Choose the right tests.
Follow the breakthroughs. Discover what changed.
See why every leap needs a better test.
Warren McCulloch and Walter Pitts describe a mathematical model of neurons and show how networks of these simple units can represent logical operations.
McCulloch & Pitts · original paperStart with a precise question about what a system can do.
Warren McCulloch and Walter Pitts describe a mathematical model of neurons and show how networks of these simple units can represent logical operations.
McCulloch & Pitts · original paperStart with a precise question about what a system can do.
A selection of turning points, not the whole story.
Dates mark the named paper, match or launch. The journey continues.
Explore the benchmarks behind the breakthroughs—from conversation and shared datasets to human judgment and task completion.
20 of 20 milestones
An operational test of conversational behavior; not a universal intelligence score.
Conversational behavior judged in an imitation game.
A convincing conversation does not establish correctness or general intelligence.
First public research / milestone; publication details in original source
Shared collections and relevance judgments made retrieval comparisons more systematic.
Ranked document retrieval using shared relevance judgments.
Results depend on collection, queries and relevance labels.
First public research / milestone; publication details in original source
Automatic translation evaluation against references; wording overlap has limits.
N-gram overlap with reference translations.
Valid alternative wording can receive a low score.
First public research / milestone; publication details in original source
Reference-based summarization metrics; overlap alone cannot establish factuality.
Overlap between a summary and reference summaries.
An incorrect summary can still share many words with a reference.
First public research / milestone; publication details in original source
Large datasets and standardized vision challenges; distinguish the full dataset from challenge subsets.
Recognition performance on labeled vision challenge tasks.
Challenge labels do not cover all real-world visual tasks.
First public research / milestone; publication details in original source
Multi-task language evaluation and harder follow-up tests as headroom shrinks.
Performance across language understanding tasks.
Aggregate scores can hide weakness on individual tasks.
First public research / milestone; publication details in original source
Breadth across 57 knowledge tasks; exam performance is only one dimension of usefulness.
Multiple-choice accuracy across 57 subjects.
Exam accuracy does not establish dependable performance on your workflow.
First public research / milestone; publication details in original source · Preprint 2020; ICLR 2021.
Evaluate generated code by functional tests; retain attempt budget and pass@k definition.
Whether generated programs pass functional tests; pass@k is success in at least one of k attempts.
Passing the supplied tests does not prove security or general correctness.
First public research / milestone; publication details in original source
Test susceptibility to common false beliefs; label 2021 preprint and ACL 2022 separately.
Truthfulness of answers to questions designed around common misconceptions.
A benchmark sample does not cover every possible false claim.
First public research / milestone; publication details in original source · Preprint 2021; ACL 2022.
A broad community task collection; distinguish BIG-bench from its subsets.
Performance across a diverse collection of tasks.
Different subsets and aggregate rules cannot be treated as one score.
First public research / milestone; publication details in original source
Embedding quality varies by task; a retrieval subset is not the entire MTEB suite.
Embedding performance across retrieval, classification and other task families.
Compare the same tasks and dataset versions, not unrelated subsets.
First public research / milestone; publication details in original source
Evaluate multiple scenarios and dimensions, including accuracy, robustness, calibration and efficiency.
Accuracy alongside robustness, fairness, calibration and efficiency.
Coverage remains limited to the chosen scenarios and metrics.
First public research / milestone; publication details in original source
Contrast crowdsourced human preference with model judging; explain position, verbosity and self-preference biases.
Human pairwise preference in Arena; model-judged response quality in MT-Bench.
Position, verbosity and self-preference can bias model judges; preference is not factual correctness.
First public research / milestone; publication details in original source
Separate retrieval quality, answer quality and evidence support in a RAG pipeline.
Retrieval relevance and evidence support for generated answers.
Grounded wording can still answer the wrong question; calibrate model graders.
First public research / milestone; publication details in original source
Repository issue resolution evaluates a model plus its tools and agent setup.
Resolution of repository issues under a specified agent and test harness.
Results depend on model, tools, budget, issue split and harness together.
First public research / milestone; publication details in original source
Multimodal understanding includes diagrams and domain knowledge, beyond image recognition.
Multidisciplinary understanding of text and images.
Visual reasoning scores do not cover all image or video generation tasks.
First public research / milestone; publication details in original source · Preprint 2023; CVPR 2024.
Refresh questions and preserve evaluation windows to reduce contamination risk.
Performance on refreshed tasks with dated evaluation windows.
Fresh tasks reduce contamination risk but do not guarantee its absence.
First public research / milestone; publication details in original source
Check final task state and repeated-run consistency, not just a plausible conversation.
Final task state and reliability across repeated agent interactions.
One successful trial does not establish consistent task completion.
First public research / milestone; publication details in original source
Safety has its own hazard definitions and scoring; version changes can break comparability.
Safety behavior under a specified hazard taxonomy and evaluator.
Scores across changed evaluators or scoring scales may be incomparable.
First public research / milestone; publication details in original source
Measure task difficulty using human completion time at a specified success probability.
Task difficulty measured by human completion time at a chosen model success probability.
A time horizon is not model runtime or a guarantee of unattended work.
First public research / milestone; publication details in original source
Evaluation sources reviewed 2026-11-07; illustrated history sources reviewed 2026-11-08. Dates refer to first public research or the named milestone. Scores belong to their original benchmark versions; this page explains methods and does not generate rankings.
Further reading: Agent evaluation practice. Model architectures, hardware and connectivity protocols provide context, the illustrated history connects those breakthroughs to evaluation takeaways. The detailed timeline focuses on measurement.
