MT-Bench
80 curated multi-turn questions across 8 skills, each a two-turn conversation scored by a strong LLM judge on a 1–10 scale.
The 80 two-turn questions are fixed, public, and heavily exercised across research and leaderboards, so the test set is both widely memorized and increasingly saturated (top models score 8.5–9.0 on a 1–10 scale where 9 is a realistic ceiling). It remains a standard, still-discriminative LLM-judge eval, but the fixed set is the constraint: refreshed or held-out variants are the future direction.
Performance Timeline
Longitudinal progression of model scores against human baselines.Performance & Historical Trajectory
Empirical score progression across model release dates and evaluation rounds.
| Model | Score | Date | Source Type | Provenance |
|---|---|---|---|---|
| MiMo-V2.6-Flash | 8.2% | 2026-09-12 | independent | Source ↗ |
| Claude Opus 5.5 | 7.8% | 2026-09-12 | independent | Source ↗ |
| Qwen 3.8 Max Prime | 7.4% | 2026-09-12 | independent | Source ↗ |
| Nemotron 3.5 Super VL | 7.3% | 2026-09-12 | independent | Source ↗ |
| Gemini 4 Argon | 8% | 2026-09-12 | independent | Source ↗ |
| Claude Haiku 5.5 | 7.3% | 2026-09-12 | independent | Source ↗ |
| Nano Banana 2.1 | 8% | 2026-09-12 | independent | Source ↗ |
| Mistral Large 4 | 7.7% | 2026-09-12 | independent | Source ↗ |
| GPT-6.1 Sol Pro | 8% | 2026-09-12 | independent | Source ↗ |
| GPT-6.1 Sol | 8% | 2026-09-12 | independent | Source ↗ |
| Claude Sonnet 5.5 | 8% | 2026-09-12 | independent | Source ↗ |
| Z.ai: GLM 5.3 Prime | 8% | 2026-09-12 | independent | Source ↗ |
| GPT-6 Luna Pro | 7.3% | 2026-09-12 | independent | Source ↗ |
| GPT-6 Luna | 7.3% | 2026-09-12 | independent | Source ↗ |
| GPT-6 Sol Pro | 8% | 2026-09-12 | independent | Source ↗ |
| GPT-6 Sol | 8% | 2026-09-12 | independent | Source ↗ |
| Claude Opus 5.5 | 8.3% | 2026-09-12 | independent | Source ↗ |
| Xiaomi: MiMo-V2.6-Pro-UltraSpeed | 8% | 2026-09-12 | independent | Source ↗ |
| Xiaomi: MiMo-V2.6-Flash | 7.3% | 2026-09-12 | independent | Source ↗ |
| Xiaomi: MiMo-V2.6-Pro | 7.3% | 2026-09-12 | independent | Source ↗ |
| Z.ai: GLM 5.3 FlashX | 7.7% | 2026-09-12 | independent | Source ↗ |
| DeepSeek V4.1 Flash | 7.3% | 2026-09-12 | independent | Source ↗ |
| Claude Sonnet 5.5 | 9% | 2026-09-12 | independent | Source ↗ |
| Claude Haiku 5.5 | 8.4% | 2026-09-12 | independent | Source ↗ |
| GPT-6.1 Sol Pro | 9.1% | 2026-09-12 | independent | Source ↗ |
| GPT-6 Luna | 8.9% | 2026-09-12 | independent | Source ↗ |
| Gemini 4 Argon | 9.1% | 2026-09-12 | independent | Source ↗ |
| Qwen 3.8 Max Prime | 8.8% | 2026-09-12 | independent | Source ↗ |
| DeepSeek V4.1 Flash 1017 | 7.3% | 2026-09-12 | independent | Source ↗ |
| GPT-6.1 Terra Pro | 8% | 2026-09-12 | independent | Source ↗ |
| GPT-6.1 Terra | 8% | 2026-09-12 | independent | Source ↗ |
| MoonshotAI: Kimi K3.1 Pro | 8% | 2026-09-12 | independent | Source ↗ |
| MiniMax: MiniMax M3.1 Flash | 7.3% | 2026-09-12 | independent | Source ↗ |
| Muse Spark Open | 8% | 2026-09-12 | independent | Source ↗ |
| LiquidAI: LFM2.5-24B-A2B | 7.3% | 2026-09-12 | independent | Source ↗ |
| Muse Wonder | 8.3% | 2026-09-12 | independent | Source ↗ |
| Qwen: Qwen3.8 Max Prime | 8.3% | 2026-09-12 | independent | Source ↗ |
| Upstage: Solar Mini 4 | 7% | 2026-09-12 | independent | Source ↗ |
| SpaceXAI: Grok 4.7 | 8% | 2026-09-12 | independent | Source ↗ |
| Qwen: Qwen3.8 Omni Flash | 7.3% | 2026-09-12 | independent | Source ↗ |
| GPT-6 Astra | 8.3% | 2026-09-08 | independent | Source ↗ |
| GPT-6 Astra Pro | 8.3% | 2026-09-08 | independent | Source ↗ |
| Gemini 3.8 Flash | 8% | 2026-09-07 | independent | Source ↗ |
| Gemini 3.8 Flash | 8.9% | 2026-09-07 | independent | Source ↗ |
| Qwen: Qwen3.8 Max (0902) | 8% | 2026-09-07 | independent | Source ↗ |
| Muse Spark 1.3 Contributor | 7% | 2026-09-07 | independent | Source ↗ |
| Muse Spark 1.3 | 8% | 2026-09-07 | independent | Source ↗ |
| Claude Fable 5.1 | 8.3% | 2026-09-05 | independent | Source ↗ |
| Hy4 | 8.8% | 2026-09-01 | independent | Source ↗ |
| Hy4 | 7.7% | 2026-09-01 | independent | Source ↗ |
| GLM 5.3 Flash | 8.9% | 2026-08-31 | independent | Source ↗ |
| Z.ai: GLM 5.3 Flash | 7.3% | 2026-08-31 | independent | Source ↗ |
| Qwen: Qwen3.8 Flash | 7.3% | 2026-08-31 | independent | Source ↗ |
| DeepSeek V4 Flash Vision Exp | 7.3% | 2026-08-26 | independent | Source ↗ |
| DeepSeek V4-Pro | 6.7% | 2026-08-21 | independent | Source ↗ |
| DeepSeek V4-Flash | 6.4% | 2026-08-21 | independent | Source ↗ |
| Z.ai: GLM 5.3 | 8% | 2026-08-21 | independent | Source ↗ |
| Claude Fable 5.1 | 8.4% | 2026-08-20 | independent | Source ↗ |
| Qwen: Qwen3.8 27B | 7.7% | 2026-08-19 | independent | Source ↗ |
| Gemini 3.7 Flash | 6.8% | 2026-08-18 | independent | Source ↗ |
| Gemini 3.7 Flash | 8% | 2026-08-18 | independent | Source ↗ |
| DeepSeek V4 Pro 0813 | 8% | 2026-08-18 | independent | Source ↗ |
| Grok 4.6 | 6.7% | 2026-08-17 | independent | Source ↗ |
| Qwen: Qwen3.8 2.4T A95B | 8% | 2026-08-17 | independent | Source ↗ |
| LiquidAI: LFM2.5-2.6B | 7.3% | 2026-08-16 | independent | Source ↗ |
| ByteDance Seed: Seed 2.1 Turbo | 7.7% | 2026-08-15 | independent | Source ↗ |
| SpaceXAI: Grok 4.6 | 8% | 2026-08-15 | independent | Source ↗ |
| Muse Glimmer 30B | 7.7% | 2026-08-15 | independent | Source ↗ |
| GPT-5 | 9.8% | 2026-08-12 | independent | Source ↗ |
| Nemotron 3.5 Lightning | 7% | 2026-08-12 | independent | Source ↗ |
| GPT-5 Image Mini | 7.7% | 2026-08-12 | independent | Source ↗ |
| GPT-5 Image | 8% | 2026-08-12 | independent | Source ↗ |
| Muse Spark 1.2 Contributor | 7% | 2026-08-10 | independent | Source ↗ |
| Muse Spark 1.2 | 8% | 2026-08-10 | independent | Source ↗ |
| DeepSeek V4.1 Flash | 8.4% | 2026-08-06 | independent | Source ↗ |
| Space Bunny Alpha | 8.6% | 2026-08-06 | independent | Source ↗ |
| DeepSeek V4 Flash 0731 | 8.4% | 2026-08-05 | independent | Source ↗ |
| DeepSeek V4 Flash 0731 | 7.7% | 2026-08-05 | independent | Source ↗ |
| ByteDance Seed: Seed-2.0-Code | 7.7% | 2026-08-04 | independent | Source ↗ |
| GPT-6 Astra | 8% | 2026-08-02 | independent | Source ↗ |
| Qwen: Qwen3.7 Flash | 7% | 2026-08-01 | independent | Source ↗ |
| Kimi K3 | 7.8% | 2026-07-31 | independent | Source ↗ |
| Claude Opus 5 | 7% | 2026-07-29 | independent | Source ↗ |
| Claude Sonnet 5 | 9.8% | 2026-07-29 | independent | Source ↗ |
| Claude Opus 5 | 8.3% | 2026-07-28 | independent | Source ↗ |
| Gemini 3.6 Flash | 6.6% | 2026-07-26 | independent | Source ↗ |
| Gemini 3.5 Flash-Lite | 6.3% | 2026-07-26 | independent | Source ↗ |
| Gemini 3.6 Flash | 8% | 2026-07-26 | independent | Source ↗ |
| Gemini 3.5 Flash Lite | 7.7% | 2026-07-26 | independent | Source ↗ |
| MoonshotAI: Kimi K3 | 8.3% | 2026-07-20 | independent | Source ↗ |
| GPT-5.6 Sol | 8.2% | 2026-07-14 | independent | Source ↗ |
| GPT-5.5 Thinking | 7% | 2026-07-14 | independent | Source ↗ |
| GPT-5.6 Luna | 8.5% | 2026-07-14 | independent | Source ↗ |
| GPT-5.6 Luna Pro | 7.7% | 2026-07-14 | independent | Source ↗ |
| GPT-5.6 Luna | 7.7% | 2026-07-14 | independent | Source ↗ |
| GPT-5.6 Terra Pro | 8.3% | 2026-07-14 | independent | Source ↗ |
| GPT-5.6 Terra | 8.3% | 2026-07-14 | independent | Source ↗ |
| GPT-5.6 Sol Pro | 8.3% | 2026-07-14 | independent | Source ↗ |
| GPT-5.6 Sol | 8.3% | 2026-07-14 | independent | Source ↗ |
| Muse Spark 1.1 | 8% | 2026-07-14 | independent | Source ↗ |
| SpaceXAI: Grok 4.5 | 8% | 2026-07-13 | independent | Source ↗ |
| Hy3 | 7.3% | 2026-07-11 | independent | Source ↗ |
| Claude Sonnet 5 | 8% | 2026-07-05 | independent | Source ↗ |
| Gemini 3.1 Flash Lite Image | 7.7% | 2026-07-05 | independent | Source ↗ |
| MiniMax M3 | 8.5% | 2026-06-30 | independent | Source ↗ |
| Qwen3-235B | 6.6% | 2026-06-23 | independent | Source ↗ |
| QwQ Plus | 6.5% | 2026-06-23 | independent | Source ↗ |
| Z.ai: GLM 5.2 | 7.7% | 2026-06-21 | independent | Source ↗ |
| Grok 4 | 9.7% | 2026-06-20 | independent | Source ↗ |
| GLM-5.2 | 6.8% | 2026-06-17 | independent | Source ↗ |
| MoonshotAI: Kimi K2.7 Code | 8% | 2026-06-17 | independent | Source ↗ |
| Claude Fable 5 | 7.2% | 2026-06-14 | independent | Source ↗ |
| Claude Fable 5 | 8.3% | 2026-06-14 | independent | Source ↗ |
| Nemotron 3 Ultra | 8.3% | 2026-06-09 | independent | Source ↗ |
| Nemotron 3.5 Content Safety | 7% | 2026-06-09 | independent | Source ↗ |
| Nemotron 3 Ultra | 7.7% | 2026-06-09 | independent | Source ↗ |
| Qwen: Qwen3.7 Plus | 7.7% | 2026-06-07 | independent | Source ↗ |
| MiniMax: MiniMax M3 | 7.7% | 2026-06-05 | independent | Source ↗ |
| Gemini 3.1 Flash Image | 7.7% | 2026-06-02 | independent | Source ↗ |
| Gemini 3 Pro Image (Nano Banana Pro) | 8.3% | 2026-06-02 | independent | Source ↗ |
| Claude Opus 4.8 | 8.3% | 2026-06-02 | independent | Source ↗ |
| Hy-MT2-1.8B | 7% | 2026-05-26 | independent | Source ↗ |
| Hy-MT2-30B-A3B | 7.3% | 2026-05-26 | independent | Source ↗ |
| Hy-MT2-7B | 7.3% | 2026-05-26 | independent | Source ↗ |
| Qwen: Qwen3.7 Max | 8% | 2026-05-25 | independent | Source ↗ |
| SpaceXAI: Grok Build 0.1 | 7.7% | 2026-05-25 | independent | Source ↗ |
| Gemini 3.5 Flash | 8% | 2026-05-24 | independent | Source ↗ |
| Gemini 3 Pro | 9.6% | 2026-05-23 | independent | Source ↗ |
| Gemini 3.1 Flash Lite | 7.7% | 2026-05-12 | independent | Source ↗ |
| GPT Chat Latest | 8.3% | 2026-05-10 | independent | Source ↗ |
| Command A+ | 7.7% | 2026-05-06 | independent | Source ↗ |
| Mistral Medium 3.5 | 8% | 2026-05-05 | independent | Source ↗ |
| SpaceXAI: Grok 4.3 | 7.7% | 2026-05-05 | independent | Source ↗ |
| Qwen: Qwen3.6 Flash | 7.7% | 2026-05-02 | independent | Source ↗ |
| GPT-5.5 Pro | 8.3% | 2026-04-28 | independent | Source ↗ |
| GPT-5.5 | 8.3% | 2026-04-28 | independent | Source ↗ |
| DeepSeek V4 Pro 0423 | 7.7% | 2026-04-28 | independent | Source ↗ |
| DeepSeek V4 Flash 0423 | 7% | 2026-04-28 | independent | Source ↗ |
| MiMo-V2.5 | 8.1% | 2026-04-27 | independent | Source ↗ |
| Xiaomi: MiMo-V2.5-Pro | 7.3% | 2026-04-27 | independent | Source ↗ |
| Xiaomi: MiMo-V2.5 | 7.3% | 2026-04-27 | independent | Source ↗ |
| Qwen: Qwen3.6 27B | 7.7% | 2026-04-27 | independent | Source ↗ |
| Hy3 | 7.3% | 2026-04-26 | independent | Source ↗ |
| GPT-5.4 Image 2 | 8.3% | 2026-04-26 | independent | Source ↗ |
| Qwen: Qwen3.5 Plus 2026-04-20 | 7.7% | 2026-04-25 | independent | Source ↗ |
| MoonshotAI: Kimi K2.6 | 8% | 2026-04-25 | independent | Source ↗ |
| Claude Opus 4.7 | 8.3% | 2026-04-21 | independent | Source ↗ |
| Qwen: Qwen3.6 35B A3B | 7.3% | 2026-04-20 | independent | Source ↗ |
| Z.ai: GLM 5.1 | 8% | 2026-04-11 | independent | Source ↗ |
| Gemma 4 26B A4B | 7.3% | 2026-04-08 | independent | Source ↗ |
| Gemma 4 31B | 7.3% | 2026-04-07 | independent | Source ↗ |
| Qwen: Qwen3.6 Plus | 7.7% | 2026-04-07 | independent | Source ↗ |
| Z.ai: GLM 5V Turbo | 8% | 2026-04-06 | independent | Source ↗ |
| Lyria 3 Pro | 7% | 2026-04-04 | independent | Source ↗ |
| Lyria 3 Clip | 7% | 2026-04-04 | independent | Source ↗ |
| Reka Edge | 7% | 2026-03-25 | independent | Source ↗ |
| MiniMax: MiniMax M2.7 | 7.7% | 2026-03-23 | independent | Source ↗ |
| GPT-5.4 Nano | 7.7% | 2026-03-22 | independent | Source ↗ |
| GPT-5.4 Mini | 8% | 2026-03-22 | independent | Source ↗ |
| Z.ai: GLM 5 Turbo | 8% | 2026-03-20 | independent | Source ↗ |
| Qwen: Qwen3.5-9B | 7.3% | 2026-03-15 | independent | Source ↗ |
| SpaceXAI: Grok 4.20 Multi-Agent | 7.7% | 2026-03-14 | independent | Source ↗ |
| SpaceXAI: Grok 4.20 | 7.7% | 2026-03-14 | independent | Source ↗ |
| ByteDance Seed: Seed-2.0-Lite | 7.7% | 2026-03-14 | independent | Source ↗ |
| GPT-5.4 Pro | 8.3% | 2026-03-10 | independent | Source ↗ |
| GPT-5.4 | 8.3% | 2026-03-10 | independent | Source ↗ |
| Gemini 3.1 Flash Lite | 7.7% | 2026-03-08 | independent | Source ↗ |
| Gemini 3.1 Flash Image | 7.7% | 2026-03-03 | independent | Source ↗ |
| GPT-5.3-Codex | 8.3% | 2026-03-01 | independent | Source ↗ |
| ByteDance Seed: Seed-2.0-Mini | 7.3% | 2026-03-01 | independent | Source ↗ |
| Qwen: Qwen3.5-35B-A3B | 7.3% | 2026-03-01 | independent | Source ↗ |
| Qwen: Qwen3.5-27B | 7.7% | 2026-03-01 | independent | Source ↗ |
| Qwen: Qwen3.5-122B-A10B | 7.7% | 2026-03-01 | independent | Source ↗ |
| Qwen: Qwen3.5-Flash | 7.3% | 2026-03-01 | independent | Source ↗ |
| Gemini 3.1 Pro Custom Tools | 8.3% | 2026-02-24 | independent | Source ↗ |
| Gemini 3.1 Pro | 8.3% | 2026-02-24 | independent | Source ↗ |
| Claude Sonnet 4.6 | 8.3% | 2026-02-22 | independent | Source ↗ |
| Qwen: Qwen3.5 Plus 2026-02-15 | 7.7% | 2026-02-21 | independent | Source ↗ |
| Qwen: Qwen3.5 397B A17B | 8% | 2026-02-21 | independent | Source ↗ |
| Z.ai: GLM 5 | 8% | 2026-02-16 | independent | Source ↗ |
| MiniMax: MiniMax M2.5 | 7.3% | 2026-02-16 | independent | Source ↗ |
| Claude Opus 4.6 | 8.3% | 2026-02-10 | independent | Source ↗ |
| MoonshotAI: Kimi K2.5 | 7.7% | 2026-02-01 | independent | Source ↗ |
| MiniMax: MiniMax M2-her | 7.7% | 2026-01-28 | independent | Source ↗ |
| Z.ai: GLM 4.7 Flash | 7.3% | 2026-01-24 | independent | Source ↗ |
| GPT-5.2-Codex | 8.3% | 2026-01-19 | independent | Source ↗ |
| MiniMax: MiniMax M2.1 | 7.7% | 2025-12-28 | independent | Source ↗ |
| Z.ai: GLM 4.7 | 7.7% | 2025-12-27 | independent | Source ↗ |
| o3 | 8% | 2025-12-25 | independent | Source ↗ |
| Gemini 3 Flash | 7.7% | 2025-12-22 | independent | Source ↗ |
| GPT-5.2 Chat | 8.3% | 2025-12-16 | independent | Source ↗ |
| GPT-5.2 Pro | 8.3% | 2025-12-16 | independent | Source ↗ |
| GPT-5.2 | 8.3% | 2025-12-16 | independent | Source ↗ |
| Z.ai: GLM 4.6V | 7.3% | 2025-12-13 | independent | Source ↗ |
| GPT-5.1-Codex-Max | 8% | 2025-12-09 | independent | Source ↗ |
| DeepSeek V3.2 | 7.3% | 2025-12-06 | independent | Source ↗ |
| Claude Opus 4.5 | 8.3% | 2025-11-29 | independent | Source ↗ |
| Gemini 3 Pro Image (Nano Banana Pro) | 8.3% | 2025-11-25 | independent | Source ↗ |
| GPT-5.1 | 8% | 2025-11-18 | independent | Source ↗ |
| GPT-5.1-Codex | 8% | 2025-11-18 | independent | Source ↗ |
| GPT-5.1-Codex-Mini | 7.7% | 2025-11-18 | independent | Source ↗ |
| Mistral Large 3 | 9.1% | 2025-11-17 | independent | Source ↗ |
| MoonshotAI: Kimi K2 Thinking | 7.7% | 2025-11-11 | independent | Source ↗ |
| MiniMax: MiniMax M2 | 7.7% | 2025-10-28 | independent | Source ↗ |
| Qwen: Qwen3 VL 8B Instruct | 7.3% | 2025-10-19 | independent | Source ↗ |
| GPT-5 Pro | 8.3% | 2025-10-11 | independent | Source ↗ |
| Qwen: Qwen3 VL 30B A3B Thinking | 7.3% | 2025-10-11 | independent | Source ↗ |
| Qwen: Qwen3 VL 30B A3B Instruct | 7.3% | 2025-10-11 | independent | Source ↗ |
| Claude Haiku 4.5 | 8% | 2025-10-06 | independent | Source ↗ |
| Claude Sonnet 4.5 | 8.3% | 2025-10-04 | independent | Source ↗ |
| Qwen: Qwen3 VL 235B A22B Instruct | 7.7% | 2025-09-28 | independent | Source ↗ |
| Qwen: Qwen3 Coder Flash | 7.3% | 2025-09-22 | independent | Source ↗ |
| Qwen: Qwen3 Next 80B A3B Instruct | 7.7% | 2025-09-16 | independent | Source ↗ |
| MoonshotAI: Kimi K2 0905 | 7.7% | 2025-09-09 | independent | Source ↗ |
| GPT-5 | 8% | 2025-08-12 | independent | Source ↗ |
| GPT-5 Mini | 7.7% | 2025-08-12 | independent | Source ↗ |
| GPT-5 Nano | 7.3% | 2025-08-12 | independent | Source ↗ |
| Claude 4.1 Opus | 8.3% | 2025-08-10 | independent | Source ↗ |
| Qwen: Qwen3 Coder 30B A3B Instruct | 7.3% | 2025-08-05 | independent | Source ↗ |
| Qwen: Qwen3 30B A3B Instruct 2507 | 7.3% | 2025-08-03 | independent | Source ↗ |
| Qwen: Qwen3 235B A22B Thinking 2507 | 8% | 2025-07-30 | independent | Source ↗ |
| Qwen: Qwen3 Coder 480B A35B | 7.3% | 2025-07-28 | independent | Source ↗ |
| ByteDance: UI-TARS 7B | 7% | 2025-07-27 | independent | Source ↗ |
| Qwen: Qwen3 235B A22B Instruct 2507 | 7.3% | 2025-07-26 | independent | Source ↗ |
| MoonshotAI: Kimi K2 0711 | 7.7% | 2025-07-16 | independent | Source ↗ |
| ByteDance Seed: Seed 1.6 Flash | 7.3% | 2025-06-30 | independent | Source ↗ |
| ByteDance Seed: Seed 1.6 | 7.7% | 2025-06-30 | independent | Source ↗ |
| MiniMax: MiniMax M1 | 7.7% | 2025-06-22 | independent | Source ↗ |
| Sarvam 105B | 8.4% | 2025-06-20 | independent | Source ↗ |
| o3 Pro | 8.3% | 2025-06-15 | independent | Source ↗ |
| Sarvam Vision (Document AI) | 7.9% | 2025-05-25 | independent | Source ↗ |
| Amazon Nova 2 Lite | 7.8% | 2025-05-23 | independent | Source ↗ |
| Amazon Nova 2 Lite | 7.6% | 2025-05-23 | independent | Source ↗ |
| Qwen: Qwen3 30B A3B | 7.3% | 2025-05-03 | independent | Source ↗ |
| Qwen: Qwen3 14B | 7.3% | 2025-05-03 | independent | Source ↗ |
| Qwen: Qwen3 32B | 7.3% | 2025-05-03 | independent | Source ↗ |
| o4 Mini High | 8% | 2025-04-21 | independent | Source ↗ |
| o3 | 8% | 2025-04-21 | independent | Source ↗ |
| o4 Mini | 8% | 2025-04-21 | independent | Source ↗ |
| Muse Wonder | 8.8% | 2025-04-20 | independent | Source ↗ |
| Muse Wonder Full | 9% | 2025-04-20 | independent | Source ↗ |
| GPT-4.1 | 8% | 2025-04-19 | independent | Source ↗ |
| GPT-4.1 Mini | 7.7% | 2025-04-19 | independent | Source ↗ |
| GPT-4.1 Nano | 7.3% | 2025-04-19 | independent | Source ↗ |
| Gemma 3 27B | 7.6% | 2025-04-17 | independent | Source ↗ |
| Nemotron 3 Ultra 550B | 8.6% | 2025-04-15 | independent | Source ↗ |
| Sarvam 30B | 8% | 2025-04-15 | independent | Source ↗ |
| Llama 4 Maverick | 9.3% | 2025-04-10 | independent | Source ↗ |
| Llama 4 Scout | 9.2% | 2025-04-10 | independent | Source ↗ |
| Gemini 2.5 Pro | 7.9% | 2025-03-30 | independent | Source ↗ |
| Gemini 2.5 Pro | 8% | 2025-03-30 | independent | Source ↗ |
| Gemini 2.5 Pro 06-05 | 8% | 2025-03-30 | independent | Source ↗ |
| Microsoft-Decision-1 | 8.2% | 2025-03-27 | independent | Source ↗ |
| Nemotron 3 Super 120B | 8.1% | 2025-03-25 | independent | Source ↗ |
| OpenAI Decisions API | 8.8% | 2025-03-23 | independent | Source ↗ |
| Jev 1.14 | 8.5% | 2025-03-19 | independent | Source ↗ |
| Krutrim-2 | 8.2% | 2025-03-17 | independent | Source ↗ |
| Reka Flash 3 | 7% | 2025-03-17 | independent | Source ↗ |
| Amazon Nova Premier 1.0 | 8.6% | 2025-03-15 | independent | Source ↗ |
| Amazon: Nova Premier 1.0 | 8.3% | 2025-03-15 | independent | Source ↗ |
| QwQ 32B | 8.8% | 2025-03-10 | independent | Source ↗ |
| Saaras v3 (Speech-to-Text) | 7.8% | 2025-03-06 | independent | Source ↗ |
| Bulbul v3 (Text-to-Speech) | 7.6% | 2025-03-06 | independent | Source ↗ |
| Phi-4-multimodal | 7.6% | 2025-03-05 | independent | Source ↗ |
| GPT-4.5 | 9.4% | 2025-03-04 | independent | Source ↗ |
| Claude 3.7 Sonnet | 7.9% | 2025-03-01 | independent | Source ↗ |
| Grok 3 | 7.9% | 2025-02-22 | independent | Source ↗ |
| Grok 3 Mini | 8.1% | 2025-02-22 | independent | Source ↗ |
| Phi-4-mini | 7.1% | 2025-02-20 | independent | Source ↗ |
| Perplexity: Sonar Deep Research | 8% | 2025-02-19 | independent | Source ↗ |
| Gemini 2.5 Flash (Thinking) | 7.9% | 2025-02-15 | independent | Source ↗ |
| Perplexity: Sonar Reasoning Pro | 8% | 2025-02-11 | independent | Source ↗ |
| Gemini 2.0 Pro | 9.1% | 2025-02-10 | independent | Source ↗ |
| Gemini 2.0 Flash | 9.1% | 2025-02-10 | independent | Source ↗ |
| Sonar Reasoning Pro | 7.7% | 2025-02-10 | independent | Source ↗ |
| Qwen: Qwen3 Coder Next | 7.3% | 2025-02-08 | independent | Source ↗ |
| o3-mini | 7.7% | 2025-02-05 | independent | Source ↗ |
| o3 Mini High | 8% | 2025-02-05 | independent | Source ↗ |
| o3 Mini | 8% | 2025-02-05 | independent | Source ↗ |
| Qwen 2.5 Max | 9% | 2025-02-02 | independent | Source ↗ |
| Qwen: Qwen2.5 VL 72B Instruct | 7.3% | 2025-01-31 | independent | Source ↗ |
| Qwen: Qwen-Plus | 7.3% | 2025-01-30 | independent | Source ↗ |
| Ollama DeepSeek-R1 (Q4_K_M) | 8.5% | 2025-01-27 | independent | Source ↗ |
| Perplexity: Sonar Pro Search | 8.3% | 2025-01-26 | independent | Source ↗ |
| Perplexity: Sonar Pro | 8.3% | 2025-01-26 | independent | Source ↗ |
| DeepSeek-R1 | 7.7% | 2025-01-25 | independent | Source ↗ |
| Gemini 2.0 Flash Thinking | 9.2% | 2025-01-25 | independent | Source ↗ |
| Kimi k1.5 | 8% | 2025-01-25 | independent | Source ↗ |
| R1 0528 | 7.7% | 2025-01-25 | independent | Source ↗ |
| MiniMax-Text-01 | 7.8% | 2025-01-20 | independent | Source ↗ |
| MiniMax: MiniMax-01 | 7.7% | 2025-01-20 | independent | Source ↗ |
| Codestral 25.01 | 7.4% | 2025-01-19 | independent | Source ↗ |
| Phi-4 | 8% | 2025-01-13 | independent | Source ↗ |
| Phi 4 | 7% | 2025-01-13 | independent | Source ↗ |
| DeepSeek-V3 | 9% | 2024-12-31 | independent | Source ↗ |
| DeepSeek V3.2 Exp | 7.3% | 2024-12-31 | independent | Source ↗ |
| DeepSeek V3.1 Terminus | 7.3% | 2024-12-31 | independent | Source ↗ |
| DeepSeek V3.1 | 7.3% | 2024-12-31 | independent | Source ↗ |
| DeepSeek V3 0324 | 7.7% | 2024-12-31 | independent | Source ↗ |
| DeepSeek V3 | 7.7% | 2024-12-31 | independent | Source ↗ |
| o1 | 8.3% | 2024-12-22 | independent | Source ↗ |
| Sonar Pro | 7.8% | 2024-12-15 | independent | Source ↗ |
| Ollama Llama 3.3 (Q4_K_M) | 7.9% | 2024-12-13 | independent | Source ↗ |
| Llama 3.3 70B | 8.9% | 2024-12-11 | independent | Source ↗ |
| Llama 3.3 70B Instruct | 7.5% | 2024-12-11 | independent | Source ↗ |
| Llama 3.3 70B Instruct | 7.3% | 2024-12-11 | independent | Source ↗ |
| o1 | 9% | 2024-12-10 | independent | Source ↗ |
| o1-pro | 8.3% | 2024-12-10 | independent | Source ↗ |
| Amazon Nova Pro | 7.8% | 2024-12-08 | independent | Source ↗ |
| Amazon Nova Lite | 7.1% | 2024-12-08 | independent | Source ↗ |
| Amazon Nova Pro 1.0 | 8.1% | 2024-12-08 | independent | Source ↗ |
| Amazon Nova Lite 1.0 | 7.5% | 2024-12-08 | independent | Source ↗ |
| Amazon Nova Micro 1.0 | 6.9% | 2024-12-08 | independent | Source ↗ |
| Amazon: Nova Lite 1.0 | 7.3% | 2024-12-08 | independent | Source ↗ |
| Amazon: Nova Micro 1.0 | 7% | 2024-12-08 | independent | Source ↗ |
| Amazon: Nova Pro 1.0 | 8% | 2024-12-08 | independent | Source ↗ |
| Command R7B (12-2024) | 7% | 2024-12-06 | independent | Source ↗ |
| QwQ-32B | 7.7% | 2024-12-03 | independent | Source ↗ |
| Solar Pro | 7.4% | 2024-11-25 | independent | Source ↗ |
| Krutrim Pro | 7.9% | 2024-11-25 | independent | Source ↗ |
| GPT-4o | 8% | 2024-11-25 | independent | Source ↗ |
| Upstage: Solar Pro 4 | 7.3% | 2024-11-25 | independent | Source ↗ |
| Jev | 8.4% | 2024-11-20 | independent | Source ↗ |
| Sonar Large | 7.7% | 2024-11-20 | independent | Source ↗ |
| Sonar | 7.5% | 2024-11-20 | independent | Source ↗ |
| TypeSafe: Jev Router | 7% | 2024-11-20 | independent | Source ↗ |
| Perplexity: Sonar | 7.3% | 2024-11-20 | independent | Source ↗ |
| Qwen 2.5 Coder 32B | 7.7% | 2024-11-17 | independent | Source ↗ |
| Qwen2.5 Coder 32B Instruct | 7.3% | 2024-11-17 | independent | Source ↗ |
| Hunyuan-Large | 8% | 2024-11-10 | independent | Source ↗ |
| Claude 3.5 Haiku | 7.2% | 2024-11-09 | independent | Source ↗ |
| Claude 3.5 Sonnet | 9.2% | 2024-10-27 | independent | Source ↗ |
| Claude 3.5 Sonnet (v2) | 7.7% | 2024-10-27 | independent | Source ↗ |
| BharatGPT-14B | 7.7% | 2024-10-23 | independent | Source ↗ |
| Yi-Lightning | 7.9% | 2024-10-20 | independent | Source ↗ |
| Llama-3.1-Nemotron-70B-Instruct | 8.1% | 2024-10-20 | independent | Source ↗ |
| Llama 3.2 1B Instruct | 7.3% | 2024-09-30 | independent | Source ↗ |
| Llama 3.2 3B Instruct | 7.3% | 2024-09-30 | independent | Source ↗ |
| Qwen 2.5 72B Instruct | 7.4% | 2024-09-24 | independent | Source ↗ |
| Qwen: Qwen2.5 7B Instruct | 7% | 2024-09-24 | independent | Source ↗ |
| Qwen2.5 72B Instruct | 7.3% | 2024-09-24 | independent | Source ↗ |
| Switchyard | 7% | 2024-09-20 | independent | Source ↗ |
| Mistral Small 4 | 7.3% | 2024-09-20 | independent | Source ↗ |
| GPT Audio | 8% | 2024-09-20 | independent | Source ↗ |
| GPT Audio Mini | 7.7% | 2024-09-20 | independent | Source ↗ |
| Nemotron 3 Nano 30B A3B | 7.3% | 2024-09-20 | independent | Source ↗ |
| Devstral 2 2512 | 7.7% | 2024-09-20 | independent | Source ↗ |
| Ministral 3 14B 2512 | 7% | 2024-09-20 | independent | Source ↗ |
| Ministral 3 8B 2512 | 7% | 2024-09-20 | independent | Source ↗ |
| Ministral 3 3B 2512 | 7% | 2024-09-20 | independent | Source ↗ |
| Voxtral Small 24B 2507 | 7.3% | 2024-09-20 | independent | Source ↗ |
| gpt-oss-safeguard-20b | 7.3% | 2024-09-20 | independent | Source ↗ |
| Gemini 2.5 Flash Image | 7.7% | 2024-09-20 | independent | Source ↗ |
| Z.ai: GLM 4.6 | 7.7% | 2024-09-20 | independent | Source ↗ |
| Mistral Medium 3.1 | 7.7% | 2024-09-20 | independent | Source ↗ |
| Z.ai: GLM 4.5V | 7.7% | 2024-09-20 | independent | Source ↗ |
| gpt-oss-120b | 7.3% | 2024-09-20 | independent | Source ↗ |
| gpt-oss-20b | 7% | 2024-09-20 | independent | Source ↗ |
| Codestral 2508 | 7.3% | 2024-09-20 | independent | Source ↗ |
| Z.ai: GLM 4.5 | 7.7% | 2024-09-20 | independent | Source ↗ |
| Z.ai: GLM 4.5 Air | 7.3% | 2024-09-20 | independent | Source ↗ |
| Gemini 2.5 Flash Lite | 7.3% | 2024-09-20 | independent | Source ↗ |
| Hunyuan A13B Instruct | 7.3% | 2024-09-20 | independent | Source ↗ |
| Mistral Small 3.2 24B | 7.3% | 2024-09-20 | independent | Source ↗ |
| Gemini 2.5 Flash | 7.7% | 2024-09-20 | independent | Source ↗ |
| Mistral Medium 3 | 7.7% | 2024-09-20 | independent | Source ↗ |
| Llama Guard 4 12B | 7% | 2024-09-20 | independent | Source ↗ |
| Llama 4 Maverick | 7.3% | 2024-09-20 | independent | Source ↗ |
| Llama 4 Scout | 7.3% | 2024-09-20 | independent | Source ↗ |
| Mistral Small 3.1 24B | 7.3% | 2024-09-20 | independent | Source ↗ |
| Gemma 3 4B | 7% | 2024-09-20 | independent | Source ↗ |
| Gemma 3 12B | 7% | 2024-09-20 | independent | Source ↗ |
| Gemma 3 27B | 7.3% | 2024-09-20 | independent | Source ↗ |
| Saba | 7.3% | 2024-09-20 | independent | Source ↗ |
| Mistral Small 3 | 7% | 2024-09-20 | independent | Source ↗ |
| Mistral Nemo | 7% | 2024-09-20 | independent | Source ↗ |
| Gemma 2 27B | 7.3% | 2024-09-20 | independent | Source ↗ |
| Mixtral 8x22B Instruct | 8% | 2024-09-20 | independent | Source ↗ |
| GPT-4 Turbo | 8.3% | 2024-09-20 | independent | Source ↗ |
| Mistral Large | 8% | 2024-09-20 | independent | Source ↗ |
| GPT-3.5 Turbo (older v0613) | 7.7% | 2024-09-20 | independent | Source ↗ |
| GPT-3.5 Turbo Instruct | 7.7% | 2024-09-20 | independent | Source ↗ |
| GPT-3.5 Turbo 16k | 8% | 2024-09-20 | independent | Source ↗ |
| GPT-3.5 Turbo | 7.7% | 2024-09-20 | independent | Source ↗ |
| GPT-4 | 8.3% | 2024-09-20 | independent | Source ↗ |
| GLM-4-Plus | 7.7% | 2024-08-25 | independent | Source ↗ |
| Sarvam 2B (Sarvam-1) | 7.2% | 2024-08-19 | independent | Source ↗ |
| GPT-4o | 8% | 2024-08-11 | independent | Source ↗ |
| Command R (08-2024) | 7.3% | 2024-08-06 | independent | Source ↗ |
| Gemma 2 2B | 6.2% | 2024-08-05 | independent | Source ↗ |
| Mistral Large 2 | 7.5% | 2024-07-29 | independent | Source ↗ |
| Mistral Large 2 | 7.4% | 2024-07-29 | independent | Source ↗ |
| Mistral Large 3 2512 | 7.7% | 2024-07-29 | independent | Source ↗ |
| Llama 3.1 405B | 8.9% | 2024-07-28 | independent | Source ↗ |
| Llama 3.1 70B Instruct | 7.3% | 2024-07-28 | independent | Source ↗ |
| Llama 3.1 8B Instruct | 7% | 2024-07-28 | independent | Source ↗ |
| GPT-4o mini | 6.9% | 2024-07-23 | independent | Source ↗ |
| GPT-4o-mini | 7.3% | 2024-07-23 | independent | Source ↗ |
| GPT-4o-mini | 7.3% | 2024-07-23 | independent | Source ↗ |
| ERNIE 4.0 Turbo | 7.7% | 2024-07-03 | independent | Source ↗ |
| Gemma 2 27B | 7.3% | 2024-07-02 | independent | Source ↗ |
| Gemma 2 9B | 7% | 2024-07-02 | independent | Source ↗ |
| Nemotron-4-340B-Instruct | 8% | 2024-06-19 | independent | Source ↗ |
| Nemotron-4-340B-Reward | 8.1% | 2024-06-19 | independent | Source ↗ |
| Hanooman 7B | 7.5% | 2024-05-20 | independent | Source ↗ |
| Gemini 1.5 Flash | 6.7% | 2024-05-19 | independent | Source ↗ |
| GPT-4o | 9.4% | 2024-05-18 | independent | Source ↗ |
| GPT-4o | 8% | 2024-05-18 | independent | Source ↗ |
| GPT-4o | 8.3% | 2024-05-18 | independent | Source ↗ |
| Snowflake Arctic | 7.2% | 2024-04-29 | independent | Source ↗ |
| Reka Core | 7.5% | 2024-04-20 | independent | Source ↗ |
| WizardLM-2 8x22B | 7.8% | 2024-04-20 | independent | Source ↗ |
| Command R+ | 7% | 2024-04-09 | independent | Source ↗ |
| DBRX Instruct | 7.3% | 2024-04-01 | independent | Source ↗ |
| Claude 3 Opus | 7.4% | 2024-03-09 | independent | Source ↗ |
| Gemini 1.5 Pro | 7.3% | 2024-02-20 | independent | Source ↗ |
| Airavata 7B | 7.1% | 2024-01-30 | independent | Source ↗ |
| IndicTrans-2 | 7.4% | 2023-09-20 | independent | Source ↗ |
| GPT-4 | 8.99% | 2023-06-09 | independent | Source ↗ |
| Nemotron 3 Super | 7.3% | 2023-03-16 | independent | Source ↗ |
Human Baseline & Difficulty Horizon
Calibrated human reference points, specialist benchmarks, and ceiling thresholds.Human annotator agreement rate on multi-turn conversational dialogue quality.
Metric & Scoring Methodology
Verification protocols, aggregation formulas, and specialized metric variants.average judge score (1–10)Dataset & Compute Cost
Evaluation volume, public availability, API pricing, and local hardware requirements.$5 – $20 USD for full benchmark evaluation run on frontier APIs.
1x NVIDIA RTX 4090 (24GB) or A100 (40GB/80GB) via vLLM / SGLang
How to Run & Reproduce
Standardized evaluation protocols, CLI commands, and reproducible runner templates.lm_eval --model hf --model_args pretrained=<model_path> --tasks mt-bench --batch_size autoopencompass --datasets mt-bench --models <model_config># Standard API Evaluation Loop
from openai import OpenAI
client = OpenAI()
response = client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": prompt}],
temperature=0.0,
)When publishing results for MT-Bench, always report the exact prompt template, few-shot exemplar ordering, sampling temperature (temperature=0), maximum reasoning budget tokens, and the precise timestamped model snapshot ID.
Contamination & Memorization Analysis
Audit of pretraining exposure risks, memorization vectors, and refresh policies.Static fixed snapshot
Public on web / HuggingFace