FrontierMath Was Supposed to Be the Wall. Then Humans Found the Minus Sign.
Epoch AI corrected FrontierMath, removed broken problems, recalculated the scores, and accidentally gave us one of the funniest benchmark updates of the year.
There is something deeply comforting about mathematics.
Not because it is easy.
It is not.
Mathematics is the discipline where a missing minus sign can ruin your week, your theorem, your reputation, and possibly your will to live.
But at least, we tell ourselves, math is clean.
Objective. Precise.
No vibes.
No influencer marketing.
No brand strategy.
No “thought leadership.”
Just truth, proof, structure, and the occasional Greek letter deployed like a medieval weapon.
Then FrontierMath happened.
FrontierMath was designed as one of the serious benchmarks: hard, original, expert-written mathematical problems. Not “solve for x” problems. Not “count the apples” problems. Not olympiad warm-ups wearing a lab coat.
Real problems.
Problems that could take expert mathematicians hours, days, or even longer. Problems where earlier AI successes were greeted with the kind of commentary usually reserved for a dog landing a helicopter.
“This was genuinely nontrivial.”
“I did not immediately see the path.”
“The model found a clever route.”
“A remarkable solution.”
And then Epoch AI went back, audited the dataset, corrected errors, removed problematic questions, and recalculated the scores.
The result was not a small statistical nudge.
It was a rather large piano falling down the stairs.
The Benchmark Had Human Bugs
The funny part is not that AI models improved.
The funny part is why the scores moved so much.
A large share of the corrected errors appears to be exactly the kind of thing humans are excellent at producing while insisting that machines are unreliable: a lost sign, a plus becoming a minus, a minus becoming a plus, an off-by-one error, a correct mathematical idea transferred incorrectly into checking code, or a solution that quietly stepped on a rake while everyone was looking at the theorem.
Humans, what can one say?
We built a benchmark to test whether machines can handle advanced mathematics.
Then the machines helped us find the mistakes in the benchmark.
This is not an insult to the people who built FrontierMath. Quite the opposite. It is a reminder that constructing a research-level benchmark is brutally hard.
When a problem is difficult enough to challenge frontier models and expert mathematicians, verifying it is also difficult.
The evaluator is not a vending machine.
It is another piece of intellectual machinery.
And intellectual machinery breaks in very human ways.
The most perfect comedy of the whole thing is that GPT-5.5 reportedly helped find many of the errors.
That is where the story turns from “AI is getting better at math” into “AI is now useful for debugging the tests designed to prove AI is not good enough at math.”
Which is, frankly, rude.
The Scores Jumped. A Lot.
The corrected dataset changes the psychological temperature of the room.
On the easier Tiers 1–3, GPT-series models now look dramatically stronger.
On Tier 4, which is the truly nasty research-level section, the jump is even more interesting.
The attached leaderboard shows Google DeepMind’s AI co-mathematician at roughly 75.6% on Tier 4, with GPT-5.5 close behind at about 72.5%.
That is a very different world from the one where every solved Tier 4 problem felt like a historic artifact.
Let us be careful.
This does not mean AI has solved mathematics.
It does not mean mathematicians can be replaced by a subscription and a cheerful loading spinner.
It does not mean every model is now Gauss with a context window.
But it does mean that a benchmark many people treated as a serious frontier has become much less comfortable.
And that matters.
Benchmarks are not just scoreboards.
They are psychological fences.
They tell us where we think the edge is.
FrontierMath was one of those fences.
The correction did not prove the fence was fake.
It showed that the fence had a few holes, a few measurement errors, and possibly a sign problem near the gate.
But after the repair, the models were already much further through than many expected.
The Open Problems Are the New Comfort Blanket
So now the narrative shifts.
When models were weak on FrontierMath Tiers 1–4, people could say: Fine, they can do contests, but not serious mathematics.
Then models started doing much better on Tier 4.
Now the comfort blanket becomes FrontierMath Open Problems.
And to be fair, that is a much stronger blanket.
The Open Problems set is different. These are problems that remain unsolved by mathematicians. A solution would not merely score a benchmark point; it could advance mathematical knowledge.
That is the distinction.
Solving a very hard problem with a known answer is one thing.
Solving an actual open problem is another.
The first is evidence of serious capability.
The second is research.
So yes, we can still say: “Calm down. The open problems remain.”
And that is true.
For now.
Those two words are doing a lot of work in 2026.
The Benchmark Era Is Getting Weird
This update also tells us something broader about AI benchmarking.
We are entering the phase where models are not only taking benchmarks.
They are helping audit them. - That changes everything.
A benchmark used to be a fixed measuring stick.
Now the measuring stick itself can be inspected by the system being measured.
This is useful.
It is also deeply weird.
If frontier models help discover dataset errors, improve validation, flag ambiguous problems, detect broken answer checks, and identify hidden inconsistencies, then benchmark maintenance becomes a human-AI collaboration.
That is good science.
But it also makes the whole evaluation landscape more dynamic.
Scores are not carved into marble.
They depend on problem validity, grading pipelines, hidden assumptions, sampling procedures, tool access, reasoning budget, and whether someone accidentally misplaced a minus sign in a place where the model politely drove through it at 70 miles per hour.
The old benchmark story was: Here is the test. Here is the score. Now clap or panic.
The new benchmark story is: Here is the test. Here is the score. Here is the audit. Here is the corrected score. Here is the model that helped find the error. Here is the new version. Please update your worldview and your chart.
Wonderful.
More work for everyone.
What This Means for Mathematical AI
The biggest takeaway is not that AI is suddenly “done” with math.
It is that advanced mathematical reasoning is moving faster than the public story can comfortably process.
For years, the safe claim was that language models were pattern machines.
They could imitate mathematical prose.
They could solve known templates.
They could pass exams with enough data leakage, scaffolding, or luck.
But research-level problem solving was different.
Too deep. Too creative. Too fragile. Too dependent on real mathematical judgment.
There is still truth in that.
But the gap is narrowing.
And it is narrowing in a way that does not fit the old caricature.
These systems are not just memorizing textbook examples.
They are exploring. Planning. Using code. Testing hypotheses. Recovering from false starts. Checking constraints. And, increasingly, helping humans find mistakes in the infrastructure around the problems.
That does not make them mathematicians in the full human sense.
They do not have careers, taste, embarrassment, rivalries, blackboards, coffee, or the ability to stare out of a window for three hours and call it work.
But they are becoming mathematical collaborators.
And that is the important category.
Not replacement.
Not oracle.
Collaborator.
A collaborator that sometimes hallucinates, sometimes cheats, sometimes fails spectacularly, and sometimes casually solves something that made a very clever human mutter “oh, damn.”
The Human Part Is Still Necessary
There is a tempting but wrong conclusion:
if models are now scoring 70%+ on corrected Tier 4, then human mathematicians are finished.
No. - Human mathematicians are not finished.
They are now responsible for harder work.
That is usually how automation goes.
The machine takes over one layer, and humans move to the next painful layer above it.
Problem formulation. Conceptual framing. Verification. Interpretation. Choosing what matters. Understanding why a result is interesting. Building theories rather than merely producing answers. Maintaining benchmarks. Auditing reasoning. Connecting isolated solutions to broader mathematical structure.
That is the real job.
Mathematics has never only been about final answers.
A final answer is a corpse without the proof, context, and meaning that make it alive.
AI can increasingly produce answers and even impressive reasoning paths.
But turning those into knowledge remains a human-centered process.
At least for now.
Again, those words.
My Verdict
The FrontierMath v2 correction is funny, embarrassing, useful, and important all at once.
Funny because a large part of the story appears to involve human mistakes of the most gloriously human kind: signs, off-by-one errors, transfer bugs, and fragile checking code.
Embarrassing because some of our confidence about model limitations was partly confidence about imperfect measurement.
Useful because this is exactly what serious benchmark science should do: audit, correct, revise, and publish better scores.
Important because the corrected results suggest that frontier models are much stronger at advanced mathematics than the earlier version implied.
The benchmark was supposed to be a wall.
After the correction, it looks more like a wall with scaffolding already halfway up it.
The next comfort zone is FrontierMath Open Problems.
That is fair.
That is where the real frontier still lives.
But the direction is hard to miss.
AI is no longer merely taking math tests.
It is helping debug the math tests.
And if that sentence does not make you slightly excited and slightly uncomfortable, check your pulse.
Or your sign.