MathematicsActive
Trust Score:
98

FrontierMath

Original, unpublished research-level math problems by expert mathematicians — mostly private to resist contamination. Frontier models solved under 2% at launch.

Launched: —Refresh: static
Status Assessment (active):

The strongest active case on the wiki. At launch (Nov 2024) state-of-the-art models solved UNDER 2% — a benchmark frontier systems almost entirely fail. Reasoning models with Python tools have since climbed (OpenAI announced o3 at 25.2% in Dec 2024, tools-on), but that figure is contested — Epoch's own independent reproductions of comparable models came in far lower (~11%). By the June 2026 v2 release Epoch added a harder Tier 4 precisely because Tiers 1–3 progress had become significant, yet the hardest tier remains far from solved. Current figures move fast and are tool/compute-dependent — consult Epoch's live dashboard; this page does not restate unverified third-party numbers. Saturation here would mean research-level, closed-form mathematics is largely automated.

Performance Timeline

Longitudinal progression of model scores against human baselines.

Performance & Historical Trajectory

Empirical score progression across model release dates and evaluation rounds.

Independent Vendor-Reported Human Baseline (2.0%)
ModelScoreDateSource TypeProvenance
GLM 5.3 Flash25.3%2026-09-12independentSource ↗
Hy424.9%2026-09-12independentSource ↗
MiMo-V2.6-Flash23.1%2026-09-12independentSource ↗
Claude Opus 5.522%2026-09-12independentSource ↗
Qwen 3.8 Max Prime20.6%2026-09-12independentSource ↗
Nemotron 3.5 Super VL21.1%2026-09-12independentSource ↗
Gemini 4 Argon23%2026-09-12independentSource ↗
Claude Haiku 5.521.1%2026-09-12independentSource ↗
Nano Banana 2.123%2026-09-12independentSource ↗
Mistral Large 422.1%2026-09-12independentSource ↗
GPT-6.1 Sol Pro23%2026-09-12independentSource ↗
GPT-6.1 Sol23%2026-09-12independentSource ↗
Claude Sonnet 5.523%2026-09-12independentSource ↗
Z.ai: GLM 5.3 Prime23%2026-09-12independentSource ↗
GPT-6 Luna Pro21.1%2026-09-12independentSource ↗
GPT-6 Luna21.1%2026-09-12independentSource ↗
GPT-6 Sol Pro23%2026-09-12independentSource ↗
GPT-6 Sol23%2026-09-12independentSource ↗
Claude Opus 5.523.8%2026-09-12independentSource ↗
Xiaomi: MiMo-V2.6-Pro-UltraSpeed23%2026-09-12independentSource ↗
Xiaomi: MiMo-V2.6-Flash21.1%2026-09-12independentSource ↗
Xiaomi: MiMo-V2.6-Pro21.1%2026-09-12independentSource ↗
Z.ai: GLM 5.3 FlashX22.1%2026-09-12independentSource ↗
DeepSeek V4.1 Flash21.1%2026-09-12independentSource ↗
GPT-6 Astra23.8%2026-09-12independentSource ↗
GPT-6 Astra Pro23.8%2026-09-12independentSource ↗
Gemini 3.8 Flash23%2026-09-12independentSource ↗
Claude Fable 5.123.8%2026-09-12independentSource ↗
Hy422.1%2026-09-12independentSource ↗
Z.ai: GLM 5.3 Flash21.1%2026-09-12independentSource ↗
DeepSeek V4 Flash Vision Exp21.1%2026-09-12independentSource ↗
Claude Sonnet 5.526.2%2026-09-12independentSource ↗
Claude Haiku 5.524.2%2026-09-12independentSource ↗
GPT-6.1 Sol Pro26.7%2026-09-12independentSource ↗
GPT-6 Luna25.9%2026-09-12independentSource ↗
Gemini 4 Argon26.6%2026-09-12independentSource ↗
Gemini 3.8 Flash26.3%2026-09-12independentSource ↗
Qwen 3.8 Max Prime26%2026-09-12independentSource ↗
DeepSeek V4.1 Flash 101721.1%2026-09-12independentSource ↗
GPT-6.1 Terra Pro23%2026-09-12independentSource ↗
GPT-6.1 Terra23%2026-09-12independentSource ↗
MoonshotAI: Kimi K3.1 Pro23%2026-09-12independentSource ↗
MiniMax: MiniMax M3.1 Flash21.1%2026-09-12independentSource ↗
Muse Spark Open23%2026-09-12independentSource ↗
LiquidAI: LFM2.5-24B-A2B21.1%2026-09-12independentSource ↗
Muse Wonder23.8%2026-09-12independentSource ↗
Qwen: Qwen3.8 Max Prime23.8%2026-09-12independentSource ↗
Upstage: Solar Mini 420%2026-09-12independentSource ↗
SpaceXAI: Grok 4.723%2026-09-12independentSource ↗
Qwen: Qwen3.8 Omni Flash21.1%2026-09-12independentSource ↗
Qwen: Qwen3.8 Max (0902)23%2026-09-12independentSource ↗
Muse Spark 1.3 Contributor20%2026-09-12independentSource ↗
Muse Spark 1.323%2026-09-12independentSource ↗
Qwen: Qwen3.8 Flash21.1%2026-09-12independentSource ↗
DeepSeek V4-Pro25.6%2026-09-10independentSource ↗
DeepSeek V4-Flash24.1%2026-09-10independentSource ↗
Z.ai: GLM 5.323%2026-09-10independentSource ↗
Claude Fable 5.126%2026-09-09independentSource ↗
Qwen: Qwen3.8 27B22.1%2026-09-08independentSource ↗
Gemini 3.7 Flash26.2%2026-09-07independentSource ↗
Gemini 3.7 Flash23%2026-09-07independentSource ↗
DeepSeek V4 Pro 081323%2026-09-07independentSource ↗
Grok 4.625.8%2026-09-06independentSource ↗
Qwen: Qwen3.8 2.4T A95B23%2026-09-06independentSource ↗
LiquidAI: LFM2.5-2.6B21.1%2026-09-05independentSource ↗
ByteDance Seed: Seed 2.1 Turbo22.1%2026-09-04independentSource ↗
SpaceXAI: Grok 4.623%2026-09-04independentSource ↗
Muse Glimmer 30B22.1%2026-09-04independentSource ↗
GPT-526.3%2026-09-01independentSource ↗
Nemotron 3.5 Lightning20%2026-09-01independentSource ↗
GPT-5 Image Mini22.1%2026-09-01independentSource ↗
GPT-5 Image23%2026-09-01independentSource ↗
Muse Spark 1.2 Contributor20%2026-08-30independentSource ↗
Muse Spark 1.223%2026-08-30independentSource ↗
DeepSeek V4.1 Flash26%2026-08-26independentSource ↗
Space Bunny Alpha24.6%2026-08-26independentSource ↗
DeepSeek V4 Flash 073123.9%2026-08-25independentSource ↗
DeepSeek V4 Flash 073122.1%2026-08-25independentSource ↗
ByteDance Seed: Seed-2.0-Code22.1%2026-08-24independentSource ↗
GPT-6 Astra26.5%2026-08-22independentSource ↗
Qwen: Qwen3.7 Flash20%2026-08-21independentSource ↗
Kimi K323.6%2026-08-20independentSource ↗
Claude Opus 526.8%2026-08-18independentSource ↗
Claude Sonnet 525.6%2026-08-18independentSource ↗
Claude Opus 523.8%2026-08-17independentSource ↗
Gemini 3.6 Flash25.5%2026-08-15independentSource ↗
Gemini 3.5 Flash-Lite23.9%2026-08-15independentSource ↗
Gemini 3.6 Flash23%2026-08-15independentSource ↗
Gemini 3.5 Flash Lite22.1%2026-08-15independentSource ↗
MoonshotAI: Kimi K323.8%2026-08-09independentSource ↗
GPT-5.6 Sol26.9%2026-08-03independentSource ↗
GPT-5.5 Thinking26.9%2026-08-03independentSource ↗
GPT-5.6 Luna24.2%2026-08-03independentSource ↗
GPT-5.6 Luna Pro22.1%2026-08-03independentSource ↗
GPT-5.6 Luna22.1%2026-08-03independentSource ↗
GPT-5.6 Terra Pro23.8%2026-08-03independentSource ↗
GPT-5.6 Terra23.8%2026-08-03independentSource ↗
GPT-5.6 Sol Pro23.8%2026-08-03independentSource ↗
GPT-5.6 Sol23.8%2026-08-03independentSource ↗
Muse Spark 1.123%2026-08-03independentSource ↗
SpaceXAI: Grok 4.523%2026-08-02independentSource ↗
Hy321.1%2026-07-31independentSource ↗
Claude Sonnet 523%2026-07-25independentSource ↗
Gemini 3.1 Flash Lite Image22.1%2026-07-25independentSource ↗
MiniMax M324.8%2026-07-20independentSource ↗
Qwen3-235B25.5%2026-07-13independentSource ↗
QwQ Plus25.9%2026-07-13independentSource ↗
Z.ai: GLM 5.222.1%2026-07-11independentSource ↗
Grok 426.6%2026-07-10independentSource ↗
GLM-5.225.3%2026-07-07independentSource ↗
MoonshotAI: Kimi K2.7 Code23%2026-07-07independentSource ↗
Claude Fable 526.9%2026-07-04independentSource ↗
Claude Fable 523.8%2026-07-04independentSource ↗
Nemotron 3 Ultra23.5%2026-06-29independentSource ↗
Nemotron 3.5 Content Safety20%2026-06-29independentSource ↗
Nemotron 3 Ultra22.1%2026-06-29independentSource ↗
Qwen: Qwen3.7 Plus22.1%2026-06-27independentSource ↗
MiniMax: MiniMax M322.1%2026-06-25independentSource ↗
Gemini 3.1 Flash Image22.1%2026-06-22independentSource ↗
Gemini 3 Pro Image (Nano Banana Pro)23.8%2026-06-22independentSource ↗
Claude Opus 4.823.8%2026-06-22independentSource ↗
Hy-MT2-1.8B20%2026-06-15independentSource ↗
Hy-MT2-30B-A3B21.1%2026-06-15independentSource ↗
Hy-MT2-7B21.1%2026-06-15independentSource ↗
Qwen: Qwen3.7 Max23%2026-06-14independentSource ↗
SpaceXAI: Grok Build 0.122.1%2026-06-14independentSource ↗
Gemini 3.5 Flash23%2026-06-13independentSource ↗
Gemini 3 Pro25.8%2026-06-12independentSource ↗
Gemini 3.1 Flash Lite22.1%2026-06-01independentSource ↗
GPT Chat Latest23.8%2026-05-30independentSource ↗
Command A+22.1%2026-05-26independentSource ↗
Mistral Medium 3.523%2026-05-25independentSource ↗
SpaceXAI: Grok 4.322.1%2026-05-25independentSource ↗
Qwen: Qwen3.6 Flash22.1%2026-05-22independentSource ↗
GPT-5.5 Pro23.8%2026-05-18independentSource ↗
GPT-5.523.8%2026-05-18independentSource ↗
DeepSeek V4 Pro 042322.1%2026-05-18independentSource ↗
DeepSeek V4 Flash 042320%2026-05-18independentSource ↗
MiMo-V2.522.8%2026-05-17independentSource ↗
Xiaomi: MiMo-V2.5-Pro21.1%2026-05-17independentSource ↗
Xiaomi: MiMo-V2.521.1%2026-05-17independentSource ↗
Qwen: Qwen3.6 27B22.1%2026-05-17independentSource ↗
Hy321.1%2026-05-16independentSource ↗
GPT-5.4 Image 223.8%2026-05-16independentSource ↗
Qwen: Qwen3.5 Plus 2026-04-2022.1%2026-05-15independentSource ↗
MoonshotAI: Kimi K2.623%2026-05-15independentSource ↗
Claude Opus 4.723.8%2026-05-11independentSource ↗
Qwen: Qwen3.6 35B A3B21.1%2026-05-10independentSource ↗
Z.ai: GLM 5.123%2026-05-01independentSource ↗
Gemma 4 26B A4B21.1%2026-04-28independentSource ↗
Gemma 4 31B21.1%2026-04-27independentSource ↗
Qwen: Qwen3.6 Plus22.1%2026-04-27independentSource ↗
Z.ai: GLM 5V Turbo23%2026-04-26independentSource ↗
Lyria 3 Pro20%2026-04-24independentSource ↗
Lyria 3 Clip20%2026-04-24independentSource ↗
Reka Edge20%2026-04-14independentSource ↗
MiniMax: MiniMax M2.722.1%2026-04-12independentSource ↗
GPT-5.4 Nano22.1%2026-04-11independentSource ↗
GPT-5.4 Mini23%2026-04-11independentSource ↗
Z.ai: GLM 5 Turbo23%2026-04-09independentSource ↗
Qwen: Qwen3.5-9B21.1%2026-04-04independentSource ↗
SpaceXAI: Grok 4.20 Multi-Agent22.1%2026-04-03independentSource ↗
SpaceXAI: Grok 4.2022.1%2026-04-03independentSource ↗
ByteDance Seed: Seed-2.0-Lite22.1%2026-04-03independentSource ↗
GPT-5.4 Pro23.8%2026-03-30independentSource ↗
GPT-5.423.8%2026-03-30independentSource ↗
Gemini 3.1 Flash Lite22.1%2026-03-28independentSource ↗
Gemini 3.1 Flash Image22.1%2026-03-23independentSource ↗
GPT-5.3-Codex23.8%2026-03-21independentSource ↗
ByteDance Seed: Seed-2.0-Mini21.1%2026-03-21independentSource ↗
Qwen: Qwen3.5-35B-A3B21.1%2026-03-21independentSource ↗
Qwen: Qwen3.5-27B22.1%2026-03-21independentSource ↗
Qwen: Qwen3.5-122B-A10B22.1%2026-03-21independentSource ↗
Qwen: Qwen3.5-Flash21.1%2026-03-21independentSource ↗
Gemini 3.1 Pro Custom Tools23.8%2026-03-16independentSource ↗
Gemini 3.1 Pro23.8%2026-03-16independentSource ↗
Claude Sonnet 4.623.8%2026-03-14independentSource ↗
Qwen: Qwen3.5 Plus 2026-02-1522.1%2026-03-13independentSource ↗
Qwen: Qwen3.5 397B A17B23%2026-03-13independentSource ↗
Z.ai: GLM 523%2026-03-08independentSource ↗
MiniMax: MiniMax M2.521.1%2026-03-08independentSource ↗
Claude Opus 4.623.8%2026-03-02independentSource ↗
MoonshotAI: Kimi K2.522.1%2026-02-21independentSource ↗
MiniMax: MiniMax M2-her22.1%2026-02-17independentSource ↗
Z.ai: GLM 4.7 Flash21.1%2026-02-13independentSource ↗
GPT-5.2-Codex23.8%2026-02-08independentSource ↗
MiniMax: MiniMax M2.122.1%2026-01-17independentSource ↗
Z.ai: GLM 4.722.1%2026-01-16independentSource ↗
o326.2%2026-01-14independentSource ↗
Gemini 3 Flash22.1%2026-01-11independentSource ↗
GPT-5.2 Chat23.8%2026-01-05independentSource ↗
GPT-5.2 Pro23.8%2026-01-05independentSource ↗
GPT-5.223.8%2026-01-05independentSource ↗
Z.ai: GLM 4.6V21.1%2026-01-02independentSource ↗
GPT-5.1-Codex-Max23%2025-12-29independentSource ↗
DeepSeek V3.221.1%2025-12-26independentSource ↗
Claude Opus 4.523.8%2025-12-19independentSource ↗
Gemini 3 Pro Image (Nano Banana Pro)23.8%2025-12-15independentSource ↗
GPT-5.123%2025-12-08independentSource ↗
GPT-5.1-Codex23%2025-12-08independentSource ↗
GPT-5.1-Codex-Mini22.1%2025-12-08independentSource ↗
Mistral Large 323.5%2025-12-07independentSource ↗
MoonshotAI: Kimi K2 Thinking22.1%2025-12-01independentSource ↗
MiniMax: MiniMax M222.1%2025-11-17independentSource ↗
Qwen: Qwen3 VL 8B Instruct21.1%2025-11-08independentSource ↗
GPT-5 Pro23.8%2025-10-31independentSource ↗
Qwen: Qwen3 VL 30B A3B Thinking21.1%2025-10-31independentSource ↗
Qwen: Qwen3 VL 30B A3B Instruct21.1%2025-10-31independentSource ↗
Claude Haiku 4.523%2025-10-26independentSource ↗
Claude Sonnet 4.523.8%2025-10-24independentSource ↗
Qwen: Qwen3 VL 235B A22B Instruct22.1%2025-10-18independentSource ↗
Qwen: Qwen3 Coder Flash21.1%2025-10-12independentSource ↗
Qwen: Qwen3 Next 80B A3B Instruct22.1%2025-10-06independentSource ↗
MoonshotAI: Kimi K2 090522.1%2025-09-29independentSource ↗
GPT-523%2025-09-01independentSource ↗
GPT-5 Mini22.1%2025-09-01independentSource ↗
GPT-5 Nano21.1%2025-09-01independentSource ↗
Claude 4.1 Opus23.8%2025-08-30independentSource ↗
Qwen: Qwen3 Coder 30B A3B Instruct21.1%2025-08-25independentSource ↗
Qwen: Qwen3 30B A3B Instruct 250721.1%2025-08-23independentSource ↗
Qwen: Qwen3 235B A22B Thinking 250723%2025-08-19independentSource ↗
Qwen: Qwen3 Coder 480B A35B21.1%2025-08-17independentSource ↗
ByteDance: UI-TARS 7B20%2025-08-16independentSource ↗
Qwen: Qwen3 235B A22B Instruct 250721.1%2025-08-15independentSource ↗
MoonshotAI: Kimi K2 071122.1%2025-08-05independentSource ↗
ByteDance Seed: Seed 1.6 Flash21.1%2025-07-20independentSource ↗
ByteDance Seed: Seed 1.622.1%2025-07-20independentSource ↗
MiniMax: MiniMax M122.1%2025-07-12independentSource ↗
Sarvam 105B23.7%2025-07-10independentSource ↗
o3 Pro23.8%2025-07-05independentSource ↗
Sarvam Vision (Document AI)21%2025-06-14independentSource ↗
Amazon Nova 2 Lite22.3%2025-06-12independentSource ↗
Amazon Nova 2 Lite21.3%2025-06-12independentSource ↗
Qwen: Qwen3 30B A3B21.1%2025-05-23independentSource ↗
Qwen: Qwen3 14B21.1%2025-05-23independentSource ↗
Qwen: Qwen3 32B21.1%2025-05-23independentSource ↗
o4 Mini High23%2025-05-11independentSource ↗
o323%2025-05-11independentSource ↗
o4 Mini23%2025-05-11independentSource ↗
Muse Wonder25.5%2025-05-10independentSource ↗
Muse Wonder Full26.5%2025-05-10independentSource ↗
GPT-4.123%2025-05-09independentSource ↗
GPT-4.1 Mini22.1%2025-05-09independentSource ↗
GPT-4.1 Nano21.1%2025-05-09independentSource ↗
Gemma 3 27B23.5%2025-05-07independentSource ↗
Nemotron 3 Ultra 550B25.3%2025-05-05independentSource ↗
Sarvam 30B22.4%2025-05-05independentSource ↗
Llama 4 Maverick25.1%2025-04-30independentSource ↗
Llama 4 Scout24.4%2025-04-30independentSource ↗
Gemini 2.5 Pro25.1%2025-04-19independentSource ↗
Gemini 2.5 Pro23%2025-04-19independentSource ↗
Gemini 2.5 Pro 06-0523%2025-04-19independentSource ↗
Microsoft-Decision-123.9%2025-04-16independentSource ↗
Nemotron 3 Super 120B23.5%2025-04-14independentSource ↗
OpenAI Decisions API25.2%2025-04-12independentSource ↗
Jev 1.1424.1%2025-04-08independentSource ↗
Krutrim-223.1%2025-04-06independentSource ↗
Reka Flash 320%2025-04-06independentSource ↗
Amazon Nova Premier 1.024.9%2025-04-04independentSource ↗
Amazon: Nova Premier 1.023.8%2025-04-04independentSource ↗
QwQ 32B24.9%2025-03-30independentSource ↗
Saaras v3 (Speech-to-Text)18.2%2025-03-26independentSource ↗
Bulbul v3 (Text-to-Speech)16.8%2025-03-26independentSource ↗
Phi-4-multimodal22.7%2025-03-25independentSource ↗
GPT-4.522.4%2025-03-24independentSource ↗
Claude 3.7 Sonnet25.5%2025-03-21independentSource ↗
Grok 325.8%2025-03-14independentSource ↗
Grok 3 Mini23.8%2025-03-14independentSource ↗
Phi-4-mini22%2025-03-12independentSource ↗
Perplexity: Sonar Deep Research23%2025-03-11independentSource ↗
Gemini 2.5 Flash (Thinking)24.4%2025-03-07independentSource ↗
Perplexity: Sonar Reasoning Pro23%2025-03-03independentSource ↗
Gemini 2.0 Pro23.8%2025-03-02independentSource ↗
Gemini 2.0 Flash22.1%2025-03-02independentSource ↗
Sonar Reasoning Pro24.9%2025-03-02independentSource ↗
OpenAI o3-mini (High)11%2025-02-28independentSource ↗
Qwen: Qwen3 Coder Next21.1%2025-02-28independentSource ↗
o3-mini25.3%2025-02-25independentSource ↗
o3 Mini High23%2025-02-25independentSource ↗
o3 Mini23%2025-02-25independentSource ↗
Qwen 2.5 Max23.2%2025-02-22independentSource ↗
Qwen: Qwen2.5 VL 72B Instruct21.1%2025-02-20independentSource ↗
Qwen: Qwen-Plus21.1%2025-02-19independentSource ↗
Ollama DeepSeek-R1 (Q4_K_M)25.5%2025-02-16independentSource ↗
Perplexity: Sonar Pro Search23.8%2025-02-15independentSource ↗
Perplexity: Sonar Pro23.8%2025-02-15independentSource ↗
DeepSeek-R125.8%2025-02-14independentSource ↗
Gemini 2.0 Flash Thinking24.8%2025-02-14independentSource ↗
Kimi k1.524.1%2025-02-14independentSource ↗
R1 052822.1%2025-02-14independentSource ↗
MiniMax-Text-0122.1%2025-02-09independentSource ↗
MiniMax: MiniMax-0122.1%2025-02-09independentSource ↗
Codestral 25.0119.9%2025-02-08independentSource ↗
Phi-424.2%2025-02-02independentSource ↗
Phi 420%2025-02-02independentSource ↗
DeepSeek-V322.1%2025-01-20independentSource ↗
DeepSeek V3.2 Exp21.1%2025-01-20independentSource ↗
DeepSeek V3.1 Terminus21.1%2025-01-20independentSource ↗
DeepSeek V3.121.1%2025-01-20independentSource ↗
DeepSeek V3 032422.1%2025-01-20independentSource ↗
DeepSeek V322.1%2025-01-20independentSource ↗
o123.8%2025-01-11independentSource ↗
Sonar Pro23%2025-01-04independentSource ↗
Ollama Llama 3.3 (Q4_K_M)23.1%2025-01-02independentSource ↗
Llama 3.3 70B21.3%2024-12-31independentSource ↗
Llama 3.3 70B Instruct23.4%2024-12-31independentSource ↗
Llama 3.3 70B Instruct21.1%2024-12-31independentSource ↗
o125%2024-12-30independentSource ↗
o1-pro23.8%2024-12-30independentSource ↗
Amazon Nova Pro22.5%2024-12-28independentSource ↗
Amazon Nova Lite20.2%2024-12-28independentSource ↗
Amazon Nova Pro 1.023.2%2024-12-28independentSource ↗
Amazon Nova Lite 1.021.3%2024-12-28independentSource ↗
Amazon Nova Micro 1.019.3%2024-12-28independentSource ↗
Amazon: Nova Lite 1.021.1%2024-12-28independentSource ↗
Amazon: Nova Micro 1.020%2024-12-28independentSource ↗
Amazon: Nova Pro 1.023%2024-12-28independentSource ↗
Command R7B (12-2024)20%2024-12-26independentSource ↗
QwQ-32B23.9%2024-12-23independentSource ↗
Krutrim Pro22%2024-12-15independentSource ↗
GPT-4o23%2024-12-15independentSource ↗
Upstage: Solar Pro 421.1%2024-12-15independentSource ↗
Jev23.8%2024-12-10independentSource ↗
Sonar22%2024-12-10independentSource ↗
TypeSafe: Jev Router20%2024-12-10independentSource ↗
Perplexity: Sonar21.1%2024-12-10independentSource ↗
Qwen 2.5 Coder 32B21.3%2024-12-07independentSource ↗
Qwen2.5 Coder 32B Instruct21.1%2024-12-07independentSource ↗
Hunyuan-Large23.1%2024-11-30independentSource ↗
Claude 3.5 Haiku19.6%2024-11-29independentSource ↗
Claude 3.5 Sonnet22%2024-11-16independentSource ↗
Claude 3.5 Sonnet (v2)23%2024-11-16independentSource ↗
BharatGPT-14B21%2024-11-12independentSource ↗
Yi-Lightning22.7%2024-11-09independentSource ↗
Llama-3.1-Nemotron-70B-Instruct23.8%2024-11-09independentSource ↗
best frontier model2%2024-11-07independentSource ↗
Llama 3.2 1B Instruct21.1%2024-10-20independentSource ↗
Llama 3.2 3B Instruct21.1%2024-10-20independentSource ↗
Qwen 2.5 72B Instruct24.1%2024-10-14independentSource ↗
Qwen: Qwen2.5 7B Instruct20%2024-10-14independentSource ↗
Qwen2.5 72B Instruct21.1%2024-10-14independentSource ↗
Switchyard20%2024-10-10independentSource ↗
Mistral Small 421.1%2024-10-10independentSource ↗
GPT Audio23%2024-10-10independentSource ↗
GPT Audio Mini22.1%2024-10-10independentSource ↗
Nemotron 3 Nano 30B A3B21.1%2024-10-10independentSource ↗
Devstral 2 251222.1%2024-10-10independentSource ↗
Ministral 3 14B 251220%2024-10-10independentSource ↗
Ministral 3 8B 251220%2024-10-10independentSource ↗
Ministral 3 3B 251220%2024-10-10independentSource ↗
Voxtral Small 24B 250721.1%2024-10-10independentSource ↗
gpt-oss-safeguard-20b21.1%2024-10-10independentSource ↗
Gemini 2.5 Flash Image22.1%2024-10-10independentSource ↗
Z.ai: GLM 4.622.1%2024-10-10independentSource ↗
Mistral Medium 3.122.1%2024-10-10independentSource ↗
Z.ai: GLM 4.5V22.1%2024-10-10independentSource ↗
gpt-oss-120b21.1%2024-10-10independentSource ↗
gpt-oss-20b20%2024-10-10independentSource ↗
Codestral 250821.1%2024-10-10independentSource ↗
Z.ai: GLM 4.522.1%2024-10-10independentSource ↗
Z.ai: GLM 4.5 Air21.1%2024-10-10independentSource ↗
Gemini 2.5 Flash Lite21.1%2024-10-10independentSource ↗
Hunyuan A13B Instruct21.1%2024-10-10independentSource ↗
Mistral Small 3.2 24B21.1%2024-10-10independentSource ↗
Gemini 2.5 Flash22.1%2024-10-10independentSource ↗
Mistral Medium 322.1%2024-10-10independentSource ↗
Llama Guard 4 12B20%2024-10-10independentSource ↗
Llama 4 Maverick21.1%2024-10-10independentSource ↗
Llama 4 Scout21.1%2024-10-10independentSource ↗
Mistral Small 3.1 24B21.1%2024-10-10independentSource ↗
Gemma 3 4B20%2024-10-10independentSource ↗
Gemma 3 12B20%2024-10-10independentSource ↗
Gemma 3 27B21.1%2024-10-10independentSource ↗
Saba21.1%2024-10-10independentSource ↗
Mistral Small 320%2024-10-10independentSource ↗
Mistral Nemo20%2024-10-10independentSource ↗
Gemma 2 27B21.1%2024-10-10independentSource ↗
Mixtral 8x22B Instruct23%2024-10-10independentSource ↗
GPT-4 Turbo23.8%2024-10-10independentSource ↗
Mistral Large23%2024-10-10independentSource ↗
GPT-3.5 Turbo (older v0613)22.1%2024-10-10independentSource ↗
GPT-3.5 Turbo Instruct22.1%2024-10-10independentSource ↗
GPT-3.5 Turbo 16k23%2024-10-10independentSource ↗
GPT-3.5 Turbo22.1%2024-10-10independentSource ↗
GPT-423.8%2024-10-10independentSource ↗
GLM-4-Plus21.8%2024-09-14independentSource ↗
Sarvam 2B (Sarvam-1)20.2%2024-09-08independentSource ↗
GPT-4o23%2024-08-31independentSource ↗
Command R (08-2024)21.1%2024-08-26independentSource ↗
Gemma 2 2B15.4%2024-08-25independentSource ↗
Mistral Large 220.2%2024-08-18independentSource ↗
Mistral Large 222%2024-08-18independentSource ↗
Mistral Large 3 251222.1%2024-08-18independentSource ↗
Llama 3.1 405B21.7%2024-08-17independentSource ↗
Llama 3.1 70B Instruct21.1%2024-08-17independentSource ↗
Llama 3.1 8B Instruct20%2024-08-17independentSource ↗
GPT-4o mini19%2024-08-12independentSource ↗
GPT-4o-mini21.1%2024-08-12independentSource ↗
GPT-4o-mini21.1%2024-08-12independentSource ↗
ERNIE 4.0 Turbo21.6%2024-07-23independentSource ↗
Gemma 2 27B21.3%2024-07-22independentSource ↗
Gemma 2 9B19.5%2024-07-22independentSource ↗
Nemotron-4-340B-Instruct23%2024-07-09independentSource ↗
Nemotron-4-340B-Reward23.5%2024-07-09independentSource ↗
Hanooman 7B20.4%2024-06-09independentSource ↗
GPT-4o21.5%2024-06-07independentSource ↗
GPT-4o23%2024-06-07independentSource ↗
GPT-4o23.8%2024-06-07independentSource ↗
Snowflake Arctic20.2%2024-05-19independentSource ↗
WizardLM-2 8x22B22.4%2024-05-10independentSource ↗
DBRX Instruct20.4%2024-04-21independentSource ↗
Airavata 7B19.3%2024-02-19independentSource ↗
IndicTrans-216.8%2023-10-10independentSource ↗
Nemotron 3 Super21.1%2023-04-05independentSource ↗

Human Baseline & Difficulty Horizon

Calibrated human reference points, specialist benchmarks, and ceiling thresholds.
Measured Human Score2.0%Domain Expert Baseline
Baseline Protocol & Interpretation

Elite mathematics competition winners and PhD researchers score <2% without computing tools, ~25% with Python.

Metric & Scoring Methodology

Verification protocols, aggregation formulas, and specialized metric variants.
Primary Metric:accuracy — % of problems solved, automated answer verification (%)
Scoring Engine:exact-match

Dataset & Compute Cost

Evaluation volume, public availability, API pricing, and local hardware requirements.
Total Dataset Size338Annotated evaluation items
Public Test SetPublicOpenly mirrored on repositories
Access GatingOpen AccessUnrestricted download
Evaluation LicenseOpen AccessDataset usage and redistribution terms
Frontier API Compute Cost:

$5 – $20 USD for full benchmark evaluation run on frontier APIs.

Recommended Local GPU Setup:

1x NVIDIA RTX 4090 (24GB) or A100 (40GB/80GB) via vLLM / SGLang

Official Dataset & Benchmark Files:Download / View Dataset Repository ↗

How to Run & Reproduce

Standardized evaluation protocols, CLI commands, and reproducible runner templates.
Prompt Regimezero-shot
Reasoning Modedirect
Sampling Temp0
Pass@k Budgetk = 1
Tools & SandboxPure Text
Scoring Verifierexact-match
Option AEleutherAI LM-Evaluation-Harness (Open-Weight Models)
lm_eval --model hf --model_args pretrained=<model_path> --tasks frontiermath --batch_size auto
Option BOpenCompass Evaluation Framework
opencompass --datasets frontiermath --models <model_config>
Python APIDeterministic Inference Loop Snippet
# Standard API Evaluation Loop
from openai import OpenAI

client = OpenAI()
response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": prompt}],
    temperature=0.0,
)
Standardized Reporting Requirement:

When publishing results for FrontierMath, always report the exact prompt template, few-shot exemplar ordering, sampling temperature (temperature=0), maximum reasoning budget tokens, and the precise timestamped model snapshot ID.

Contamination & Memorization Analysis

Audit of pretraining exposure risks, memorization vectors, and refresh policies.
Overall Contamination Risk:LOW
Refresh Cadence:

Static fixed snapshot

Test Set Exposure:

Public on web / HuggingFace