GSM8K
2021-10-27 — 2024-05-01 · lived 2 years, 6 months
Contaminated — the answers leaked into training
Grade school word problems, and the paper that introduced them topped out at 55%. In 2024 someone rebuilt the test from scratch, matched for difficulty, and watched whole model families drop up to 8 points — they had not learned arithmetic, they had read the answers. The frontier came through clean.
How the frontier caught up
At launch, then at the date we bury it. The ceiling is what the number could not get past, and it is not the same measurement on every grave — it says which.
A benchmark gets no deprecation notice. This death date is our editorial judgment: we bury a benchmark when frontier scores stop telling top models apart, or when major labs stop reporting it. The sources below are the evidence — judge it yourself.