SWE-bench Verified Was Retired for Data Contamination: Which AI Benchmarks Still Hold Up

My take: As every new model family launches with record-breaking performance claims, it's worth understanding how that performance is measured and why the numbers need more context than ever. In February 2026, OpenAI officially retired the SWE-bench Verified benchmark because scores had reached near 100%, not through absolute capability, but because the answers were present in the public repositories also used to train the models. MMLU follows the same pattern: every frontier model now scores above 88%.

What remains standing is Humanity's Last Exam (HLE), a set of 2,500 expert-designed questions built to resist saturation. The best current model score is 37.5%, which makes it the most honest thermometer of the actual state of AI today. On the coding front, SWE-bench Pro (1,865 tasks from real professional repositories) is the new evaluation standard, with Claude Mythos 5 leading at 80.3%.

For anyone using AI in their work, the takeaway is concrete: when a company publishes a benchmark score of 90% or 95%, first ask whether that benchmark was part of the model's training data. If the answer is yes, the number says more about memorization than real capability. How do you evaluate a model before adopting it in your work or business?