About AI Benchmarks

What each AI model scores, and what that score costs, at every reasoning effort. The Curves view draws each model’s effort levels as one line of score against cost. The UL Gauntlet tab holds our own benchmarks: Gauntlet Explain sets each model’s grades against what its answers cost, the same way, and Gauntlet Science ranks how well each model does science on worlds it has never seen. Gauntlet Ship, the third, ranks how often each model ships real work in a codebase it has never seen.

Where the idea came from

In his video “Anthropic Actually Fixed Opus”, Theo (t3.gg) showed a chart that drew each model’s reasoning-effort levels as one curve of Intelligence Index against cost per task. It made the trade-off between paying more and getting a smarter answer easy to see. The Curves view is that chart.

Where the data comes from

Each benchmark carries its owner’s mark, so you can always tell whose numbers you are reading. Two sources feed the site.

Artificial Analysis

The Intelligence, Coding and Agentic indexes, their cost per task, and the list prices behind Gauntlet Explain’s cost view come from Artificial Analysis, which benchmarks AI models independently. We read it through their free Data API, never by scraping their pages. Their free tier asks for credit wherever the data appears, so every page names them in its footer.

The data refreshes every hour, on the hour. Each refresh reads the full model list, checks that it has the shape we expect, and joins each model’s effort settings into one line. If a refresh fails, or the new list has lost more than half of its data points, we keep showing the last good copy instead.

Whenever a refresh brings different numbers, the History view records every score, price and release that changed, with a short AI-written summary of what moved.

The data on this page was last updated 2026-11-09 04:00 UTC.

Unsupervised Learning’s Gauntlet Explain, Gauntlet Science and Gauntlet Ship

The UL Gauntlet tab holds our own benchmarks. Gauntlet Explain asks every model the same 14 hard open questions, about the mind, the universe, mathematics, meaning and the future of AI, closed-book and in at most 600 words, 3 times over. Each answer is graded from 0 to 4 on David Deutsch’s criteria for a good explanation: does it actually explain, is it hard to vary, does it reach beyond the case, can it be criticized, and does it beat its rivals.

The grading runs through Assay, our eval system. Three judges from three labs read every answer with the model names removed, and all three count for every model, so every model is graded by the same panel. A model’s gap to the leader is a paired difference over the same questions, with a 95% range, and we call a lead only when that whole range sits above zero. The questions are public. The answers and the judges’ verdicts stay private, so they cannot be copied into the next round of training data; we publish the grades. These grades are from 2026-11-05. The rest of this page gives the protocol, the rubric, the judges and the statistics.

Gauntlet Science, the second, sets every model the same 24 worlds, generated fresh for each run so none can have been seen before. In each world the model runs 6 experiments of its own design, then predicts past anything it measured, and code that knows the world’s hidden law scores every prediction. No judge reads an answer. These scores are from 2026-11-05; its ranking and the section on Gauntlet Science below say more.

Gauntlet Ship, the third, gives every model the same 55 real tasks from our own working sessions, each in a codebase it has never seen, with the working conversation as its ask. Code runs hidden checks written from that conversation and decides whether each change ships. No judge reads the work. These scores are from 2026-11-05; its ranking and the section on Gauntlet Ship below say more.

What Gauntlet Explain measures

Gauntlet Explain is Unsupervised Learning’s first benchmark, and its chart puts each model’s Gauntlet Explain grade against what its answers cost. This page says what it measures, how each grade is made, and what we publish and what we keep.

Gauntlet Explain measures how well a model thinks about questions nobody has solved. Every model answers the same hard open questions, and each answer is graded as an explanation. The grade is quality alone; speed is never measured. A separate Cost view sets each grade against what the model spent writing its answers.

Most benchmarks have an answer key, and a model can learn a key. An open question has none, so what can be judged is the explanation itself. David Deutsch’s “The Beginning of Infinity” says what makes one good: it states what is really there and why, it is hard to vary, it reaches beyond the case it was built for, it can be shown wrong, and it holds up against the best rival. We grade against those five properties.

Why we think this measures intelligence

We think intelligence shows most clearly in explanation: saying what is really going on and why, in a form that could turn out wrong and holds up anyway. David Deutsch argues that knowledge grows by conjecture and criticism, and that what separates a good explanation from a bad one is that it is hard to vary. Every part of a good explanation does work, so you cannot swap a detail without breaking it. A bad one, a myth or a list of possibilities or “it’s complicated”, can absorb any fact by adjusting itself, so it explains nothing.

A hard-to-vary explanation of an unsolved question cannot be recalled, because nobody has stored one. The model has to build an account, commit to it, connect it to things beyond the question, name what would prove it wrong, and beat the best rival account. That is close to what we mean by thinking. Daniel Miessler has said that much of how he judges an AI system is how well it produces Deutschian explanations, using Deutsch’s definition of knowledge as the base for quality, in AI and in people. Gauntlet Explain turns that judgment into a measurement.

The questions are hard on purpose

Every question is open: how life started, what consciousness is, why mathematics describes the world, what makes a self. No answer key exists anywhere, ours included, so there is nothing to memorize and nothing to look up. This version grades 14 questions across 10 fields, drawn from a public set that lives on eternalquestions.ai and in the Gauntlet repository. Each question also carries its own standard: what a real answer to that question must do, beyond explaining well in general.

The grading has layers

One score from one judge would be easy to fool. Each answer passes through several layers, and the figures below are what stands behind the grades on this page.

Why the combination matters

Each part alone can be gamed. Hard questions with a loose grader reward confident prose. A strict rubric on easy questions rewards recall. A single judge carries its own lab’s taste. Together they close those gaps: questions with no key, a standard that punishes vagueness and padding, a quote behind every score, three labs reading blind, each answer alone and every judge counting for every model, and statistics that refuse to call a close race.

It matches what we see

Anecdotally, the ranking matches our own intuition from working with these models every day: Claude Opus 5.5, then Claude Fable 5.1, then Claude Sonnet 5.5, then Muse Wonder, then Claude Haiku 5.5, then Gemini 4 Argon, then GPT-6 Astra, then Kimi K3.1 Pro, then GPT-6.1 Sol, then Grok 4.7, then GPT-6.1 Terra, then GPT-6 Luna. That is not proof. A benchmark that only confirmed intuition would add nothing, and the day it disagrees with us will be the interesting one. But in our experience, a measure that lines up this closely with long, hands-on use is tracking something real.

The questions

The questions are public. 64 of them are on eternalquestions.ai, and the full set will be published in a public repository named Gauntlet, at github.com/danielmiessler/Gauntlet (coming soon). This version grades 14 questions.

The set covers how the brain and the cell work, where life came from, the universe, mathematics, the mind, what knowledge is, meaning and the self, and the future of AI.

Each question carries a standard of its own: what a real answer to that question must do. The judges split the standard into separate requirements and mark how much of it each answer meets. That result is the Criterion score: graded with every answer, kept apart from the headline grade, and not shown on the ranking yet.

Why public questions do not spoil it

There is no answer key to memorize. These questions are open, so nobody holds the correct answer, us included. A model can know every question in advance and still has to write a new explanation, and a judge grades that explanation fresh for whether it explains, is hard to vary, reaches, can be criticized and beats its rivals. Matching a published answer earns nothing, because no answer is the standard.

A model that has read good discussion of these questions may write a better answer. That is being well read, and it is part of what Gauntlet Explain is measuring.

The protocol

1. Every model gets the same prompt, the same question and the same one-line instruction. Models that run inside a coding agent have the agent’s own instructions replaced by that line, and its tools that read or change files are off. OpenAI’s agent still lists its few remaining built-in tools, which it allows no way to remove; Meta’s agent allows no replacement at all, so Muse answers inside its maker’s agent.
2. The test is closed-book. A model has no tools and no web access, and it cites sources from memory and marks each one unverified.
3. The prompt asks for the best explanation the model can give, committed to. A survey of other people’s views is not an answer.
4. An answer has six sections: Answer, Evidence, Uncertainty, Experiment, Falsifier and Sources. A code check confirms they are present before any judge reads the answer.
5. An answer must stay under 600 words. Anything past that is cut before grading, and the judges are told which answers were cut and read only what comes before the cut.
6. The whole test runs 3 times, so one lucky or unlucky answer does not decide a model’s place.

Followed through for a single answer, the protocol and the rubric below look like this.

The rubric

Each answer is scored from 0 to 4 on five dimensions taken from Deutsch. The table gives the anchors for the two ends of each scale, and 1 to 3 fall between them. For every score the judge quotes at most fifteen words of the answer as evidence. For a scientific question the refutation is an observation or an experiment; for a philosophical or personal one it is a counterexample or an argument the account must survive.

The headline grade, the Gauntlet Explain grade, is the mean of the five.

Two more scores are graded with every answer and do not enter the headline. They are kept with the judges’ verdicts and are not shown on the chart yet.

The judge also names each answer’s pivot, the one load-bearing detail its explanation depends on.

The models ranked, the reasoning effort each was asked for, and how many of its answers ran past the word budget and were cut:

Effort is the reasoning setting each model was asked for; “default” means the model was asked for none and ran at its lab’s default, so it can differ from the setting another benchmark on this site lists for the same model.

The judges

The grading runs through Assay, Unsupervised Learning’s eval system. The judges are Claude Fable 5.1, GPT-6.1 Sol and Gemini 4 Argon, the model each of three labs offers as its strongest, as recorded in UL’s model registry when the grades were made. Being a lab’s strongest model does not make a judge a better explainer than every model it grades, and on this benchmark two of the judges score below models they grade. What keeps the grades fair is the panel: the same three judges grade every model, and every answer is read blind.

Length lean is the correlation, r, between an answer’s length and how low that judge graded it, so a negative figure means that judge graded longer answers higher; Known biases below says more.

Every answer is graded alone: each judge call holds one answer and the same prompt, byte for byte, as every other answer to that question, with the model name removed and the answer rewritten into the same plain format: no headings, bullets, bold or dashes, every word kept. Neither a name nor a lab’s layout can tell a judge who wrote an answer. What a judge can see is how long an answer is and whether it was cut at the word limit, and each judge has its own taste for length; Known biases below says what that does.

Every judge’s grade counts for every model. Benchmark 2.0.0 replaced a rule that dropped a judge’s grade on its own lab’s models: that left each lab’s models with a different pair of judges, so models were not graded by one method.

How a grade, a gap and a lead are computed

1. All three judges count for every answer. Each judge’s five scores for the answer are averaged into one number from 0 to 4.
2. A model’s grade for one run is the mean of those numbers across all questions, and its grade is the mean of the 3 runs. A mark's hover lists its three run grades.
3. A gap is the leader’s grade minus the model’s, taken on the same question in the same run and averaged over all 42 question-runs (14 questions, 3 runs). Pairing the two models on the same questions removes the difference in how hard each question is.
4. The runs of one question are not independent, so the standard error is clustered by question. The 95% range is the gap plus or minus 1.96 standard errors.
5. Because a gap is averaged question by question from unrounded grades, it can differ by 0.01 from subtracting the two rounded grades in the hover.
6. A lead is called only when the whole 95% range sits above zero. When the range includes zero the two models are not separated, however their means line up. The leader is said to lead every other model only when that holds for every one of them, and a single run alone never produces a lead.

Keeping the lineup current

Each lane of Gauntlet Explain is pinned to the newest model in its family. When a newer one comes out, the lane follows it: that lane is asked again, graded again and recorded, and nothing reaches this site until Daniel approves it. The History view lists every change to the lineup, with the grade before and after.

What the numbers mean

Intelligence Index

Artificial Analysis’s combined score across its set of evaluations (version 5 in the data shown here). Their methodology page lists what goes into it.

Coding Index and Agentic Index

Narrower scores from Artificial Analysis, each an equal-weighted average of the evaluations in its area. Their capability indices page defines both.

Cost per task

What Artificial Analysis measured it cost, in US dollars, to run one Intelligence Index task at that setting. We use the same cost beside Coding and Agentic scores, so a setting costs the same whichever score you rank by.

Effort level

Many models let you choose how much they reason before they answer, usually named low, medium, high, xhigh or max. Artificial Analysis tests each setting as its own entry. We join the settings of one release into one line, lowest effort first. More effort usually scores higher and costs more.

Why some models have fewer levels

A model shows only the settings its maker offers and Artificial Analysis has tested. A model with one setting is a single dot.

Why new models can lack Coding or Agentic scores

The newest releases can have an Intelligence Index score before Artificial Analysis publishes their Coding or Agentic scores. Until it does, we list them as not yet scored, never as zero.

How to read the ranking

In the ranking at the top of this page, each row is one model at its best setting, best first. The dot is its grade from 0 to 4, the ticks are its 3 runs, and the grade is on the right. Each model’s gap to the leader, with the 95% range of that gap, is drawn under How a grade, a gap and a lead are computed.

Claude Fable 5.1 is 0.07 behind Claude Opus 5.5, with a 95% range of 0.01 to 0.13. The whole range sits above zero, so the lead over Claude Fable 5.1 is called.

How to read the cost view

The Cost view puts each model at its Gauntlet Explain grade, up the side, against what it spent generating its Gauntlet Explain answers, along the bottom on a log scale, cost rising to the right, so up and to the left is smarter for less. On the Artificial Analysis charts, the line under the chart names the highest observed setting and prices the last frontier steps. Artificial Analysis publishes no uncertainty for its scores, so it states what was measured, not a significant lead. That spend is measured, not looked up: every answer records the tokens the model actually produced, its written answer plus its hidden thinking, and the view takes the mean per answer over all 3 runs. Those tokens are priced at Artificial Analysis’s published output price for that model.

The question itself is left out. It is a few hundred input tokens, the same for every model, but the input we measure also carries each answering tool’s own instructions, which run from under a thousand tokens to about 23,000. Pricing it would rank the tool we reached a model through, not the model.

A model is plotted only when every one of its answers has a complete token count and Artificial Analysis lists a price for it. Otherwise it is listed under the chart with the reason. A model run at several reasoning efforts draws as a line through the efforts actually run; one effort is one dot. This is the cost of answering Gauntlet Explain, not Artificial Analysis’s own cost per task on its index.

Gauntlet Explain’s grades and token counts are Unsupervised Learning’s own. The Intelligence, Coding and Agentic indexes, the cost per task beside them, and the list prices behind Gauntlet Explain’s cost view are Artificial Analysis’s, and Where the data comes from, above, says how we read them.

Known biases and limits

Length

Length lean is the correlation between an answer’s length and how low a judge graded it; a negative figure means longer answers graded higher. Claude Fable 5.1 strongly favours longer answers (r −0.75); GPT-6.1 Sol moderately favours longer answers (r −0.43); and Gemini 4 Argon strongly favours longer answers (r −0.72). Cutting every answer at 600 words limits the effect without removing it, so part of a gap between two models can be length.

Each judge’s taste for length

No judge can see who wrote an answer, but each has a taste for length, given above from the current grades. Labs write to different lengths, too: in the benchmark 4.0.0 answers OpenAI’s models averaged about 385 words, Anthropic’s about 625 and Google’s about 660, and longer answers are more often cut at the limit. So a judge’s length taste tends to line up with one lab’s habit, which can look like that judge favoring its own lab. Every model faces the same three tastes; we report them rather than correct for them.

Two agents we cannot fully strip

Meta’s coding agent allows no change to its instructions, so Muse answers with them while every other model answers under one plain line. OpenAI’s agent takes the plain line but still lists a few built-in tools, such as a clock and a way to ask the user, that it allows no way to remove.

Judges are models

No human grades an answer. A blind spot the three judges share would be in every grade.

A small set

14 questions is a small sample of everything worth asking. The 95% range shows how much a gap moves from question to question, and it narrows as the set grows.

One definition of a good explanation

Deutsch’s test is a choice. It rewards commitment on purpose: a cautious survey of views scores at most 1 on hard to vary, however careful it is.

Citations

Sources are cited from memory and marked unverified. A wrong fact counts against the Honest score, which is graded beside the headline but not shown on the ranking yet.

Versions

Gauntlet Explain has two version numbers, which happen to match today. The benchmark is version 4.0.0: it changes when the rubric, the judges, the word budget or the prompts change. The question set is version 1.0.0: it changes when questions are added, removed or reworded. The rubric is deutsch-v1. The grades were made on 2026-11-05 UTC from 3 runs of 14 questions at 600 words. Grades from different versions, word budgets or rubrics are never averaged together.

What changed. 2026-10-30: benchmark 4.0.0 grades every answer alone. Until then a judge read up to seven answers together, in groups built by effort, so each model and effort was graded beside different company; every model was asked again and every answer graded again, and 4.0.0 grades are not compared with earlier ones. 2026-10-14: Gemini 4 Argon joined as a ranked model, and every pass was graded again by the full panel with all seven models side by side. The six earlier grades moved by at most 0.05. Agreement between the three judges is not published yet.

What Gauntlet Science measures

Gauntlet Science is Unsupervised Learning’s second benchmark, and its ranking scores each model from 0 to 100. Gauntlet Explain asks how well a model explains what nobody has solved. Gauntlet Science asks whether it can find out something new for itself: look at a world it has never seen, test it, work out the rule underneath, and predict what happens where it has never looked.

Most science benchmarks ask about science that is already written down, and a model can learn what is written down. Here the thing to be discovered did not exist until the run began, so a high score cannot come from memory.

Worlds generated fresh, so nothing can be memorised

Every run generates its worlds from a new seed. Each world is a made-up measurable system, a handful of quantities and an outcome, joined by a hidden law. Some quantities matter and some are decoys. This run used 24 worlds from seed 20261105, drawn from 6 families of law on three rungs of difficulty:

Six experiments, then predictions beyond the data

1. The model reads a set of noisy observations from the world, all inside the range anyone has measured.
2. It designs its own experiments, 6 in all, choosing the settings it wants to try, and reads back what the world does.
3. It states the law it found, the strongest rival law, and the experiment that would tell them apart.
4. It makes 20 predictions, each with an interval it is 80% sure of. Half fall inside the measured range and half fall beyond anything it measured, where only the right law predicts well.
5. The test is closed-book. The model has no tools, no code and no web access.

Scored by code against the hidden law, with no judge

Code wrote each world, so code knows its law and scores every prediction against it with an interval score: a tight interval around the true value scores well, a wide one or a miss scores badly. A score of 100 is every prediction exact and certain; 0 is a miss or no answer. The headline score counts the predictions beyond the data. No model reads or grades another model’s answer, so there is no judge’s taste to correct for.

Each model’s score is its mean over the worlds, with a 95% interval. Its gap to the leader is paired world by world, and the ranking marks a gap as separated only when its whole 95% interval sits below zero.

The floor and ceiling scripts

Two scripts play every world beside the models, as reference rows. The floor script runs random experiments and fits a straight line: a model below it did no better than that. The ceiling script spreads its experiments to the edges of the range and picks the best-fitting law from the families the worlds are drawn from: a practical top, though not a perfect one. In this run the floor scored 40 and the ceiling 89.

The validity gate

Before any model ran, we fixed the checks a run has to pass to count as a measurement of anything, among them that it separates the models, that the hardest worlds leave headroom, that the ranking holds up when the worlds are split in half, that longer answers do not score better for being long, and that the leader beats the floor script. A run that fails is reported as provisional. This run fails 1 of the 7 checks (stability), so its ranking is provisional.

The Gauntlet Science ranking

Each row is one model, best first: its score from 0 to 100, its 95% interval, and its gap to the leader with that gap’s interval. The grey rows are the floor and ceiling scripts.

Not ranked, because a model is ranked only when it finished every world: Grok (API timed out on every world).

Gauntlet Science’s scores come from protocol gauntlet-discovery-v2, run bfcd2f4e, made on 2026-11-05. Scores from different protocols are never compared.

What Gauntlet Ship measures

Gauntlet Ship is Unsupervised Learning’s third benchmark, and its ranking scores each model by the share of real tasks it ships. Gauntlet Explain asks how well a model explains, and Gauntlet Science whether it can find something out. Gauntlet Ship asks whether it can do the work: take a request from a real conversation, in code it has never seen, and leave a change that does what was asked.

Real work, in a codebase the model has never seen

Every item is a real task from Unsupervised Learning’s own working sessions: a conversation that asked for a change, and the change that was accepted. This run has 55 items across 16 parts of the codebase. The model gets the working conversation as its ask and the repository as it stood before the fix, and works through one plain tool loop, reading files, writing files and running tests, for at most 30 steps. Then it says whether it is done.

Hidden checks, scored by code, with no judge

Each item’s checks are written from its conversation and hidden from the model. A check is kept only when it fails on the code before the accepted fix and passes after it, and the checks are screened so they test nothing the conversation left open. An item is shipped when every hidden check passes and no test that passed before breaks. Code runs the checks and makes every number; no model reads or grades another model’s work.

The Gauntlet Ship score is the share of items shipped, with a 95% interval over the items. A model’s gap to the leader is paired item by item, and the ranking marks it separated only when its whole 95% interval sits below zero. Beside the score: hidden checks, the share of hidden checks passed; false done, the share of “done” claims that did not ship; right files, how far the files it changed overlap the accepted fix’s; and the safeguards its changes weakened, such as a deleted or loosened test.

Two scripts play every item beside the models, as reference rows. No change finishes at once, and shipped 0% of the items. The accepted fix writes the change that was actually accepted, and shipped 100%; an item it cannot ship is dropped before scoring, so every item counted can be shipped.

The validity gate

Before any model ran, we fixed the checks a run has to pass to count as a measurement of anything: among them that every kept item is valid, that the models separate, that the leader leaves headroom, that the ranking holds when the items are split in half, that longer transcripts do not ship more for being long, and that the leader beats no change. A run that fails any of them is shown as provisional. This run passes all 7 checks.

Argon ran but is left out of this ranking: 15 of 62 cells ended on Gemini API timeouts after the one retry, so the lane measured Google's API, not the model.

The Gauntlet Ship ranking

Each row is one model, best first: its Gauntlet Ship score, its 95% interval, its hidden-check and false-done shares, and its gap to the leader with that gap’s interval. The grey rows are no change and the accepted fix.

Gauntlet Ship’s scores come from protocol gauntlet-ship-v0, item set v1, run 15ea371f, made on 2026-11-05. Scores from different protocols or item sets are never compared. The items are private work, so this page shows counts and scores only: never an item’s conversation, its code, its checks or a model’s transcript.

What is public and what is private

Public: Gauntlet Explain’s questions, this method and rubric, each model’s grade with its run grades, each gap with its 95% range, and the judges with their length lean. For Gauntlet Science, each model’s score with its interval, in-range score, coverage and falsifier rate, each gap with its interval, the floor and ceiling rows, and the gate’s checks. For Gauntlet Ship, the item and component counts, the step budget, each model’s score with its interval, hidden-check, false-done and right-files shares, steps and weakened safeguards, each gap with its interval, the no-change and accepted-fix rows, and the gate’s checks.

Private: the models’ answers and the judges’ verdicts, meaning their scores and quotes. They stay private so they cannot be copied into the next round of training data or used to tune a model to the judges. Gauntlet Science keeps each world’s law, settings and prompt private too, along with every answer. Gauntlet Ship’s items are private work: every conversation, repository, path, change, hidden check and transcript stays private, and only counts and scores are published.

Who made this

AI Benchmarks was created by Daniel Miessler. It is independent, and not affiliated with Artificial Analysis, or with Theo or t3.gg. Gauntlet Explain, Gauntlet Science and Gauntlet Ship are Unsupervised Learning’s own.