About AI Benchmarks

What each AI model scores, and what that score costs, at every reasoning effort. The Curves view draws each model’s effort levels as one line of score against cost. The UL Gauntlet tab sets our own benchmark’s grades against what each answer costs, the same way.

Where the idea came from

In his video “Anthropic Actually Fixed Opus”, Theo (t3.gg) showed a chart that drew each model’s reasoning-effort levels as one curve of Intelligence Index against cost per task. It made the trade-off between paying more and getting a smarter answer easy to see. The Curves view is that chart.

Where the data comes from

Each benchmark carries its owner’s mark, so you can always tell whose numbers you are reading. Two sources feed the site.

Artificial Analysis

The Intelligence, Coding and Agentic indexes, their cost per task, and the list prices behind the Gauntlet’s cost view come from Artificial Analysis, which benchmarks AI models independently. We read it through their free Data API, never by scraping their pages. Their free tier asks for credit wherever the data appears, so every page names them in its footer.

The data refreshes every hour, on the hour. Each refresh reads the full model list, checks that it has the shape we expect, and joins each model’s effort settings into one line. If a refresh fails, or the new list has lost more than half of its data points, we keep showing the last good copy instead.

Whenever a refresh brings different numbers, the History view records every score, price and release that changed, with a short AI-written summary of what moved.

The data on this page was last updated 2026-10-15 04:00 UTC.

Unsupervised Learning’s Gauntlet

The UL Gauntlet tab is our own benchmark. It asks every model the same 14 hard open questions, about the mind, the universe, mathematics, meaning and the future of AI, closed-book and in at most 600 words, 3 times over. Each answer is graded from 0 to 4 on David Deutsch’s criteria for a good explanation: does it actually explain, is it hard to vary, does it reach beyond the case, can it be criticized, and does it beat its rivals.

The grading runs through Assay, our eval system. Three judges from three labs read the answers with the model names removed, and no judge’s grade counts on its own lab’s model. A model’s gap to the leader is a paired difference over the same questions, with a 95% range, and we call a lead only when that whole range sits above zero. The questions are public. The answers and the judges’ verdicts stay private, so they cannot be copied into the next round of training data; we publish the grades. These grades are from 2026-10-14. The rest of this page gives the protocol, the rubric, the judges and the statistics.

What the Gauntlet measures

The Gauntlet is Unsupervised Learning’s own benchmark, and its chart puts each model’s Explanation grade against what its answers cost. This page says what it measures, how each grade is made, and what we publish and what we keep.

The Gauntlet measures how well a model thinks about questions nobody has solved. Every model answers the same hard open questions, and each answer is graded as an explanation. The grade is quality alone; speed is never measured. A separate Cost view sets each grade against what the model spent writing its answers.

Most benchmarks have an answer key, and a model can learn a key. An open question has none, so what can be judged is the explanation itself. David Deutsch’s “The Beginning of Infinity” says what makes one good: it states what is really there and why, it is hard to vary, it reaches beyond the case it was built for, it can be shown wrong, and it holds up against the best rival. We grade against those five properties.

Why we think this measures intelligence

We think intelligence shows most clearly in explanation: saying what is really going on and why, in a form that could turn out wrong and holds up anyway. David Deutsch argues that knowledge grows by conjecture and criticism, and that what separates a good explanation from a bad one is that it is hard to vary. Every part of a good explanation does work, so you cannot swap a detail without breaking it. A bad one, a myth or a list of possibilities or “it’s complicated”, can absorb any fact by adjusting itself, so it explains nothing.

A hard-to-vary explanation of an unsolved question cannot be recalled, because nobody has stored one. The model has to build an account, commit to it, connect it to things beyond the question, name what would prove it wrong, and beat the best rival account. That is close to what we mean by thinking. Daniel Miessler has said that much of how he judges an AI system is how well it produces Deutschian explanations, using Deutsch’s definition of knowledge as the base for quality, in AI and in people. The Gauntlet turns that judgment into a measurement.

The questions are hard on purpose

Every question is open: how life started, what consciousness is, why mathematics describes the world, what makes a self. No answer key exists anywhere, ours included, so there is nothing to memorize and nothing to look up. This version grades 14 questions across 10 fields, drawn from a public set that lives on eternalquestions.ai and in the Gauntlet repository. Each question also carries its own standard: what a real answer to that question must do, beyond explaining well in general.

The grading has layers

One score from one judge would be easy to fool. Each answer passes through several layers, and the figures below are what stands behind the grades on this page.

Why the combination matters

Each part alone can be gamed. Hard questions with a loose grader reward confident prose. A strict rubric on easy questions rewards recall. A single judge carries its own lab’s taste. Together they close those gaps: questions with no key, a standard that punishes vagueness and padding, a quote behind every score, three labs reading blind with their own models excluded, and statistics that refuse to call a close race.

It matches what we see

Anecdotally, the ranking matches our own intuition from working with these models every day: Claude Opus 5.5, then Claude Fable 5.1, then Claude Fable 5.1, then Claude Fable 5.1, then Claude Fable 5.1, then Claude Fable 5.1, then Claude Opus 5.5, then Claude Opus 5.5, then Claude Opus 5.5, then Claude Sonnet 5.5, then Claude Opus 5.5, then Muse Wonder, then Claude Sonnet 5.5, then Muse Wonder, then Claude Sonnet 5.5, then Muse Wonder, then Claude Sonnet 5.5, then Grok 4.7, then Claude Sonnet 5.5, then Gemini 4 Argon, then GPT-5.6 Terra, then GPT-6 Astra, then GPT-6 Astra, then GPT-6.1 Sol, then GPT-6 Astra, then GPT-6.1 Sol, then GPT-6 Astra, then GPT-6.1 Sol, then GPT-6.1 Sol, then GPT-6 Luna. That is not proof. A benchmark that only confirmed intuition would add nothing, and the day it disagrees with us will be the interesting one. But in our experience, a measure that lines up this closely with long, hands-on use is tracking something real.

The questions

The questions are public. 64 of them are on eternalquestions.ai, and the full set will be published in a public repository named Gauntlet, at github.com/danielmiessler/Gauntlet (coming soon). This version grades 14 questions.

The set covers how the brain and the cell work, where life came from, the universe, mathematics, the mind, what knowledge is, meaning and the self, and the future of AI.

Each question carries a standard of its own: what a real answer to that question must do. The judges split the standard into separate requirements and mark how much of it each answer meets. That result is the Criterion score: graded with every answer, kept apart from the headline grade, and not shown on the ranking yet.

Why public questions do not spoil it

There is no answer key to memorize. These questions are open, so nobody holds the correct answer, us included. A model can know every question in advance and still has to write a new explanation, and a judge grades that explanation fresh for whether it explains, is hard to vary, reaches, can be criticized and beats its rivals. Matching a published answer earns nothing, because no answer is the standard.

A model that has read good discussion of these questions may write a better answer. That is being well read, and it is part of what the Gauntlet is measuring.

The protocol

1. Every model gets the same prompt and the same question.
2. The test is closed-book. A model has no tools and no web access, and it cites sources from memory and marks each one unverified.
3. The prompt asks for the best explanation the model can give, committed to. A survey of other people’s views is not an answer.
4. An answer has six sections: Answer, Evidence, Uncertainty, Experiment, Falsifier and Sources. A code check confirms they are present before any judge reads the answer.
5. An answer must stay under 600 words. Anything past that is cut before grading, and the judges are told which answers were cut and read only what comes before the cut.
6. The whole test runs 3 times, so one lucky or unlucky answer does not decide a model’s place.

Followed through for a single answer, the protocol and the rubric below look like this.

The rubric

Each answer is scored from 0 to 4 on five dimensions taken from Deutsch. The table gives the anchors for the two ends of each scale, and 1 to 3 fall between them. For every score the judge quotes at most fifteen words of the answer as evidence. For a scientific question the refutation is an observation or an experiment; for a philosophical or personal one it is a counterexample or an argument the account must survive.

The headline grade, called Explanation, is the mean of the five.

Two more scores are graded with every answer and do not enter the headline. They are kept with the judges’ verdicts and are not shown on the chart yet.

The judge also names each answer’s pivot, the one load-bearing detail its explanation depends on.

The models ranked, the reasoning effort each was asked for, and how many of its answers ran past the word budget and were cut:

Effort is the reasoning setting each model was asked for; “default” means the model was asked for none and ran at its lab’s default, so it can differ from the setting another benchmark on this site lists for the same model.

The judges

The grading runs through Assay, Unsupervised Learning’s eval system. The judges are Claude Fable 5.1, GPT-6.1 Sol and Gemini 4 Argon, the model each of three labs offers as its strongest, as recorded in UL’s model registry when the grades were made. Being a lab’s strongest model does not make a judge a better explainer than every model it grades, and on this benchmark two of the judges score below models they grade. What keeps the grades fair is the panel: three labs judge, no judge grades its own lab’s models, and every answer is read blind.

Length lean is the correlation, r, between an answer’s length and its place in that judge’s ranking; Known biases below says what it means.

Judges read the answers to one question together, with the model names removed and the labels shuffled, so a judge does not know which model wrote which answer.

No judge’s grade counts on its own lab’s model. A model from one of these three labs is therefore graded by the other two judges, and a model from any other lab is graded by all three.

How a grade, a gap and a lead are computed

1. A judge counts for an answer unless the answer comes from the judge’s own lab. Each counting judge’s five scores for the answer are averaged into one number from 0 to 4.
2. A model’s grade for one run is the mean of those numbers across all questions, and its grade is the mean of the 3 runs. A mark's hover lists its three run grades.
3. A gap is the leader’s grade minus the model’s, taken on the same question in the same run and averaged over all 42 question-runs (14 questions, 3 runs). Pairing the two models on the same questions removes the difference in how hard each question is.
4. The runs of one question are not independent, so the standard error is clustered by question. The 95% range is the gap plus or minus 1.96 standard errors.
5. Because a gap is averaged question by question from unrounded grades, it can differ by 0.01 from subtracting the two rounded grades in the hover.
6. A lead is called only when the whole 95% range sits above zero. When the range includes zero the two models are not separated, however their means line up. The leader is said to lead every other model only when that holds for every one of them, and a single run alone never produces a lead.

Keeping the lineup current

Each lane of the Gauntlet is pinned to the newest model in its family. When a newer one comes out, the lane follows it: that lane is asked again, graded again and recorded, and nothing reaches this site until Daniel approves it. The History view lists every change to the lineup, with the grade before and after.

What the numbers mean

Intelligence Index

Artificial Analysis’s combined score across its set of evaluations (version 4.3 in the data shown here). Their methodology page lists what goes into it.

Coding Index and Agentic Index

Narrower scores from Artificial Analysis, each an equal-weighted average of the evaluations in its area. Their capability indices page defines both.

Cost per task

What Artificial Analysis measured it cost, in US dollars, to run one Intelligence Index task at that setting. We use the same cost beside Coding and Agentic scores, so a setting costs the same whichever score you rank by.

Effort level

Many models let you choose how much they reason before they answer, usually named low, medium, high, xhigh or max. Artificial Analysis tests each setting as its own entry. We join the settings of one release into one line, lowest effort first. More effort usually scores higher and costs more.

Why some models have fewer levels

A model shows only the settings its maker offers and Artificial Analysis has tested. A model with one setting is a single dot.

Why new models can lack Coding or Agentic scores

The newest releases can have an Intelligence Index score before Artificial Analysis publishes their Coding or Agentic scores. Until it does, we list them as not yet scored, never as zero.

How to read the ranking

In the ranking at the top of this page, each row is one model at its best setting, best first. The dot is its grade from 0 to 4, the ticks are its 3 runs, and the grade is on the right. Each model’s gap to the leader, with the 95% range of that gap, is drawn under How a grade, a gap and a lead are computed.

Claude Fable 5.1 is 0.08 behind Claude Opus 5.5, with a 95% range of -0.00 to 0.16. The range includes zero, so no lead over Claude Fable 5.1 is called.

How to read the cost view

The Cost view puts each model at its Explanation grade, up the side, against what it spent generating its Gauntlet answers, along the bottom on a log scale, cost rising to the right, so up and to the left is smarter for less. On the Artificial Analysis charts, the line under the chart names the highest observed setting and prices the last frontier steps. Artificial Analysis publishes no uncertainty for its scores, so it states what was measured, not a significant lead. That spend is measured, not looked up: every answer records the tokens the model actually produced, its written answer plus its hidden thinking, and the view takes the mean per answer over all 3 runs. Those tokens are priced at Artificial Analysis’s published output price for that model.

The question itself is left out. It is a few hundred input tokens, the same for every model, but the input we measure also carries each answering tool’s own instructions, which run from under a thousand tokens to about 23,000. Pricing it would rank the tool we reached a model through, not the model.

A model is plotted only when every one of its answers has a complete token count and Artificial Analysis lists a price for it. Otherwise it is listed under the chart with the reason. A model run at several reasoning efforts draws as a line through the efforts actually run; one effort is one dot. This is the cost of answering the Gauntlet, not Artificial Analysis’s own cost per task on its index.

The Gauntlet’s grades and token counts are Unsupervised Learning’s own. The Intelligence, Coding and Agentic indexes, the cost per task beside them, and the list prices behind the Gauntlet’s cost view are Artificial Analysis’s, and Where the data comes from, above, says how we read them.

Known biases and limits

Length

Length lean is the correlation between an answer’s length and its place in a judge’s ranking; a negative figure means longer answers ranked higher. Claude Fable 5.1 moderately favours longer answers (r −0.23); GPT-6.1 Sol moderately favours shorter answers (r +0.33); and Gemini 4 Argon barely leans either way (r −0.18). Cutting every answer at 600 words limits the effect without removing it.

A judge’s own lab

The own-lab rule keeps a judge from grading its own models. A model from none of the three labs is graded by all three judges, and a model from one of them by two, so the number of judges behind a grade is not the same for every model.

Judges are models

No human grades an answer. A blind spot the three judges share would be in every grade.

A small set

14 questions is a small sample of everything worth asking. The 95% range shows how much a gap moves from question to question, and it narrows as the set grows.

One definition of a good explanation

Deutsch’s test is a choice. It rewards commitment on purpose: a cautious survey of views scores at most 1 on hard to vary, however careful it is.

Citations

Sources are cited from memory and marked unverified. A wrong fact counts against the Honest score, which is graded beside the headline but not shown on the ranking yet.

Versions

The Gauntlet has two version numbers, which happen to match today. The benchmark is version 1.0.0: it changes when the rubric, the judges, the word budget or the prompts change. The question set is Gauntlet 1.0.0: it changes when questions are added, removed or reworded. The rubric is deutsch-v1. The grades were made on 2026-10-14 UTC from 3 runs of 14 questions at 600 words. Grades from different versions, word budgets or rubrics are never averaged together.

What changed. 2026-10-14: Gemini 4 Argon joined as a ranked model, and every pass was graded again by the full panel with all seven models side by side. The six earlier grades moved by at most 0.05. Agreement between the three judges is not published yet.

What is public and what is private

Public: the questions, this method and rubric, each model’s grade with its run grades, each gap with its 95% range, and the judges with their length lean.

Private: the models’ answers and the judges’ verdicts, meaning their scores, quotes and rankings. They stay private so they cannot be copied into the next round of training data or used to tune a model to the judges.

Who made this

AI Benchmarks was created by Daniel Miessler. It is independent, and not affiliated with Artificial Analysis, or with Theo or t3.gg. The Gauntlet is Unsupervised Learning’s own.