CodingDeprecated
Trust Score:
10
SWE-bench
Real GitHub issues from 12 Python repositories, solved by generating patches that must pass repository tests.
Launched: —Refresh: static
Status Assessment (deprecated):
SWE-bench Verified superseded the original test set in 2024 after human review found underspecified tasks and tests that rejected valid fixes.
Performance Timeline
Longitudinal progression of model scores against human baselines.Performance & Historical Trajectory
Empirical score progression across model release dates and evaluation rounds.
Independent Vendor-Reported Human Baseline (78.4%)
| Model | Score | Date | Source Type | Provenance |
|---|---|---|---|---|
| MiMo-V2.6-Flash | 57.9% | 2026-09-12 | independent | Source ↗ |
| Claude Opus 5.5 | 55.7% | 2026-09-12 | independent | Source ↗ |
| Qwen 3.8 Max Prime | 52.7% | 2026-09-12 | independent | Source ↗ |
| Nemotron 3.5 Super VL | 52.4% | 2026-09-12 | independent | Source ↗ |
| Gemini 4 Argon | 57.2% | 2026-09-12 | independent | Source ↗ |
| Claude Haiku 5.5 | 52.4% | 2026-09-12 | independent | Source ↗ |
| Nano Banana 2.1 | 57.2% | 2026-09-12 | independent | Source ↗ |
| Mistral Large 4 | 54.9% | 2026-09-12 | independent | Source ↗ |
| GPT-6.1 Sol Pro | 57.2% | 2026-09-12 | independent | Source ↗ |
| GPT-6.1 Sol | 57.2% | 2026-09-12 | independent | Source ↗ |
| Claude Sonnet 5.5 | 57.2% | 2026-09-12 | independent | Source ↗ |
| Z.ai: GLM 5.3 Prime | 57.2% | 2026-09-12 | independent | Source ↗ |
| GPT-6 Luna Pro | 52.4% | 2026-09-12 | independent | Source ↗ |
| GPT-6 Luna | 52.4% | 2026-09-12 | independent | Source ↗ |
| GPT-6 Sol Pro | 57.2% | 2026-09-12 | independent | Source ↗ |
| GPT-6 Sol | 57.2% | 2026-09-12 | independent | Source ↗ |
| Claude Opus 5.5 | 59.1% | 2026-09-12 | independent | Source ↗ |
| Xiaomi: MiMo-V2.6-Pro-UltraSpeed | 57.2% | 2026-09-12 | independent | Source ↗ |
| Xiaomi: MiMo-V2.6-Flash | 52.4% | 2026-09-12 | independent | Source ↗ |
| Xiaomi: MiMo-V2.6-Pro | 52.4% | 2026-09-12 | independent | Source ↗ |
| Z.ai: GLM 5.3 FlashX | 54.9% | 2026-09-12 | independent | Source ↗ |
| DeepSeek V4.1 Flash | 52.4% | 2026-09-12 | independent | Source ↗ |
| Claude Sonnet 5.5 | 59.8% | 2026-09-12 | independent | Source ↗ |
| Claude Haiku 5.5 | 55.8% | 2026-09-12 | independent | Source ↗ |
| GPT-6.1 Sol Pro | 61.9% | 2026-09-12 | independent | Source ↗ |
| GPT-6 Luna | 59.8% | 2026-09-12 | independent | Source ↗ |
| Gemini 4 Argon | 60.5% | 2026-09-12 | independent | Source ↗ |
| Qwen 3.8 Max Prime | 57.8% | 2026-09-12 | independent | Source ↗ |
| DeepSeek V4.1 Flash 1017 | 52.4% | 2026-09-12 | independent | Source ↗ |
| GPT-6.1 Terra Pro | 57.2% | 2026-09-12 | independent | Source ↗ |
| GPT-6.1 Terra | 57.2% | 2026-09-12 | independent | Source ↗ |
| MoonshotAI: Kimi K3.1 Pro | 57.2% | 2026-09-12 | independent | Source ↗ |
| MiniMax: MiniMax M3.1 Flash | 52.4% | 2026-09-12 | independent | Source ↗ |
| Muse Spark Open | 57.2% | 2026-09-12 | independent | Source ↗ |
| LiquidAI: LFM2.5-24B-A2B | 52.4% | 2026-09-12 | independent | Source ↗ |
| Muse Wonder | 59.1% | 2026-09-12 | independent | Source ↗ |
| Qwen: Qwen3.8 Max Prime | 59.1% | 2026-09-12 | independent | Source ↗ |
| Upstage: Solar Mini 4 | 49.7% | 2026-09-12 | independent | Source ↗ |
| SpaceXAI: Grok 4.7 | 57.2% | 2026-09-12 | independent | Source ↗ |
| Qwen: Qwen3.8 Omni Flash | 52.4% | 2026-09-12 | independent | Source ↗ |
| GPT-6 Astra | 59.1% | 2026-09-11 | independent | Source ↗ |
| GPT-6 Astra Pro | 59.1% | 2026-09-11 | independent | Source ↗ |
| Gemini 3.8 Flash | 57.2% | 2026-09-10 | independent | Source ↗ |
| Gemini 3.8 Flash | 55.1% | 2026-09-10 | independent | Source ↗ |
| Qwen: Qwen3.8 Max (0902) | 57.2% | 2026-09-10 | independent | Source ↗ |
| Muse Spark 1.3 Contributor | 49.7% | 2026-09-10 | independent | Source ↗ |
| Muse Spark 1.3 | 57.2% | 2026-09-10 | independent | Source ↗ |
| Claude Fable 5.1 | 59.1% | 2026-09-08 | independent | Source ↗ |
| Hy4 | 61.7% | 2026-09-04 | independent | Source ↗ |
| Hy4 | 54.9% | 2026-09-04 | independent | Source ↗ |
| GLM 5.3 Flash | 62.4% | 2026-09-03 | independent | Source ↗ |
| Z.ai: GLM 5.3 Flash | 52.4% | 2026-09-03 | independent | Source ↗ |
| Qwen: Qwen3.8 Flash | 52.4% | 2026-09-03 | independent | Source ↗ |
| DeepSeek V4 Flash Vision Exp | 52.4% | 2026-08-29 | independent | Source ↗ |
| DeepSeek V4-Pro | 55.1% | 2026-08-24 | independent | Source ↗ |
| DeepSeek V4-Flash | 51.3% | 2026-08-24 | independent | Source ↗ |
| Z.ai: GLM 5.3 | 57.2% | 2026-08-24 | independent | Source ↗ |
| Claude Fable 5.1 | 62.2% | 2026-08-23 | independent | Source ↗ |
| Qwen: Qwen3.8 27B | 54.9% | 2026-08-22 | independent | Source ↗ |
| Gemini 3.7 Flash | 53.7% | 2026-08-21 | independent | Source ↗ |
| Gemini 3.7 Flash | 57.2% | 2026-08-21 | independent | Source ↗ |
| DeepSeek V4 Pro 0813 | 57.2% | 2026-08-21 | independent | Source ↗ |
| Grok 4.6 | 54.5% | 2026-08-20 | independent | Source ↗ |
| Qwen: Qwen3.8 2.4T A95B | 57.2% | 2026-08-20 | independent | Source ↗ |
| LiquidAI: LFM2.5-2.6B | 52.4% | 2026-08-19 | independent | Source ↗ |
| ByteDance Seed: Seed 2.1 Turbo | 54.9% | 2026-08-18 | independent | Source ↗ |
| SpaceXAI: Grok 4.6 | 57.2% | 2026-08-18 | independent | Source ↗ |
| Muse Glimmer 30B | 54.9% | 2026-08-18 | independent | Source ↗ |
| GPT-5 | 57.5% | 2026-08-15 | independent | Source ↗ |
| Nemotron 3.5 Lightning | 49.7% | 2026-08-15 | independent | Source ↗ |
| GPT-5 Image Mini | 54.9% | 2026-08-15 | independent | Source ↗ |
| GPT-5 Image | 57.2% | 2026-08-15 | independent | Source ↗ |
| Muse Spark 1.2 Contributor | 49.7% | 2026-08-13 | independent | Source ↗ |
| Muse Spark 1.2 | 57.2% | 2026-08-13 | independent | Source ↗ |
| DeepSeek V4.1 Flash | 60.7% | 2026-08-09 | independent | Source ↗ |
| Space Bunny Alpha | 60.9% | 2026-08-09 | independent | Source ↗ |
| DeepSeek V4 Flash 0731 | 59.4% | 2026-08-08 | independent | Source ↗ |
| DeepSeek V4 Flash 0731 | 54.9% | 2026-08-08 | independent | Source ↗ |
| ByteDance Seed: Seed-2.0-Code | 54.9% | 2026-08-07 | independent | Source ↗ |
| GPT-6 Astra | 61.7% | 2026-08-05 | independent | Source ↗ |
| Qwen: Qwen3.7 Flash | 49.7% | 2026-08-04 | independent | Source ↗ |
| Kimi K3 | 55.4% | 2026-08-03 | independent | Source ↗ |
| Claude Opus 5 | 57.8% | 2026-08-01 | independent | Source ↗ |
| Claude Sonnet 5 | 56.6% | 2026-08-01 | independent | Source ↗ |
| Claude Opus 5 | 59.1% | 2026-07-31 | independent | Source ↗ |
| Gemini 3.6 Flash | 52.9% | 2026-07-29 | independent | Source ↗ |
| Gemini 3.5 Flash-Lite | 50.3% | 2026-07-29 | independent | Source ↗ |
| Gemini 3.6 Flash | 57.2% | 2026-07-29 | independent | Source ↗ |
| Gemini 3.5 Flash Lite | 54.9% | 2026-07-29 | independent | Source ↗ |
| MoonshotAI: Kimi K3 | 59.1% | 2026-07-23 | independent | Source ↗ |
| GPT-5.6 Sol | 57.1% | 2026-07-17 | independent | Source ↗ |
| GPT-5.5 Thinking | 55.8% | 2026-07-17 | independent | Source ↗ |
| GPT-5.6 Luna | 60.2% | 2026-07-17 | independent | Source ↗ |
| GPT-5.6 Luna Pro | 54.9% | 2026-07-17 | independent | Source ↗ |
| GPT-5.6 Luna | 54.9% | 2026-07-17 | independent | Source ↗ |
| GPT-5.6 Terra Pro | 59.1% | 2026-07-17 | independent | Source ↗ |
| GPT-5.6 Terra | 59.1% | 2026-07-17 | independent | Source ↗ |
| GPT-5.6 Sol Pro | 59.1% | 2026-07-17 | independent | Source ↗ |
| GPT-5.6 Sol | 59.1% | 2026-07-17 | independent | Source ↗ |
| Muse Spark 1.1 | 57.2% | 2026-07-17 | independent | Source ↗ |
| SpaceXAI: Grok 4.5 | 57.2% | 2026-07-16 | independent | Source ↗ |
| Hy3 | 52.4% | 2026-07-14 | independent | Source ↗ |
| Claude Sonnet 5 | 57.2% | 2026-07-08 | independent | Source ↗ |
| Gemini 3.1 Flash Lite Image | 54.9% | 2026-07-08 | independent | Source ↗ |
| MiniMax M3 | 54.4% | 2026-07-03 | independent | Source ↗ |
| Qwen3-235B | 54.7% | 2026-06-26 | independent | Source ↗ |
| QwQ Plus | 53.7% | 2026-06-26 | independent | Source ↗ |
| Z.ai: GLM 5.2 | 54.9% | 2026-06-24 | independent | Source ↗ |
| Grok 4 | 57.1% | 2026-06-23 | independent | Source ↗ |
| GLM-5.2 | 54.1% | 2026-06-20 | independent | Source ↗ |
| MoonshotAI: Kimi K2.7 Code | 57.2% | 2026-06-20 | independent | Source ↗ |
| Claude Fable 5 | 58.5% | 2026-06-17 | independent | Source ↗ |
| Claude Fable 5 | 59.1% | 2026-06-17 | independent | Source ↗ |
| Nemotron 3 Ultra | 58.7% | 2026-06-12 | independent | Source ↗ |
| Nemotron 3.5 Content Safety | 49.7% | 2026-06-12 | independent | Source ↗ |
| Nemotron 3 Ultra | 54.9% | 2026-06-12 | independent | Source ↗ |
| Qwen: Qwen3.7 Plus | 54.9% | 2026-06-10 | independent | Source ↗ |
| MiniMax: MiniMax M3 | 54.9% | 2026-06-08 | independent | Source ↗ |
| Gemini 3.1 Flash Image | 54.9% | 2026-06-05 | independent | Source ↗ |
| Gemini 3 Pro Image (Nano Banana Pro) | 59.1% | 2026-06-05 | independent | Source ↗ |
| Claude Opus 4.8 | 59.1% | 2026-06-05 | independent | Source ↗ |
| Hy-MT2-1.8B | 49.7% | 2026-05-29 | independent | Source ↗ |
| Hy-MT2-30B-A3B | 52.4% | 2026-05-29 | independent | Source ↗ |
| Hy-MT2-7B | 52.4% | 2026-05-29 | independent | Source ↗ |
| Qwen: Qwen3.7 Max | 57.2% | 2026-05-28 | independent | Source ↗ |
| SpaceXAI: Grok Build 0.1 | 54.9% | 2026-05-28 | independent | Source ↗ |
| Gemini 3.5 Flash | 57.2% | 2026-05-27 | independent | Source ↗ |
| Gemini 3 Pro | 55.4% | 2026-05-26 | independent | Source ↗ |
| Gemini 3.1 Flash Lite | 54.9% | 2026-05-15 | independent | Source ↗ |
| GPT Chat Latest | 59.1% | 2026-05-13 | independent | Source ↗ |
| Command A+ | 54.9% | 2026-05-09 | independent | Source ↗ |
| Mistral Medium 3.5 | 57.2% | 2026-05-08 | independent | Source ↗ |
| SpaceXAI: Grok 4.3 | 54.9% | 2026-05-08 | independent | Source ↗ |
| Qwen: Qwen3.6 Flash | 54.9% | 2026-05-05 | independent | Source ↗ |
| GPT-5.5 Pro | 59.1% | 2026-05-01 | independent | Source ↗ |
| GPT-5.5 | 59.1% | 2026-05-01 | independent | Source ↗ |
| DeepSeek V4 Pro 0423 | 54.9% | 2026-05-01 | independent | Source ↗ |
| DeepSeek V4 Flash 0423 | 49.7% | 2026-05-01 | independent | Source ↗ |
| MiMo-V2.5 | 57.2% | 2026-04-30 | independent | Source ↗ |
| Xiaomi: MiMo-V2.5-Pro | 52.4% | 2026-04-30 | independent | Source ↗ |
| Xiaomi: MiMo-V2.5 | 52.4% | 2026-04-30 | independent | Source ↗ |
| Qwen: Qwen3.6 27B | 54.9% | 2026-04-30 | independent | Source ↗ |
| Hy3 | 52.4% | 2026-04-29 | independent | Source ↗ |
| GPT-5.4 Image 2 | 59.1% | 2026-04-29 | independent | Source ↗ |
| Qwen: Qwen3.5 Plus 2026-04-20 | 54.9% | 2026-04-28 | independent | Source ↗ |
| MoonshotAI: Kimi K2.6 | 57.2% | 2026-04-28 | independent | Source ↗ |
| Claude Opus 4.7 | 59.1% | 2026-04-24 | independent | Source ↗ |
| Qwen: Qwen3.6 35B A3B | 52.4% | 2026-04-23 | independent | Source ↗ |
| Z.ai: GLM 5.1 | 57.2% | 2026-04-14 | independent | Source ↗ |
| Gemma 4 26B A4B | 52.4% | 2026-04-11 | independent | Source ↗ |
| Gemma 4 31B | 52.4% | 2026-04-10 | independent | Source ↗ |
| Qwen: Qwen3.6 Plus | 54.9% | 2026-04-10 | independent | Source ↗ |
| Z.ai: GLM 5V Turbo | 57.2% | 2026-04-09 | independent | Source ↗ |
| Lyria 3 Pro | 49.7% | 2026-04-07 | independent | Source ↗ |
| Lyria 3 Clip | 49.7% | 2026-04-07 | independent | Source ↗ |
| Reka Edge | 49.7% | 2026-03-28 | independent | Source ↗ |
| MiniMax: MiniMax M2.7 | 54.9% | 2026-03-26 | independent | Source ↗ |
| GPT-5.4 Nano | 54.9% | 2026-03-25 | independent | Source ↗ |
| GPT-5.4 Mini | 57.2% | 2026-03-25 | independent | Source ↗ |
| Z.ai: GLM 5 Turbo | 57.2% | 2026-03-23 | independent | Source ↗ |
| Qwen: Qwen3.5-9B | 52.4% | 2026-03-18 | independent | Source ↗ |
| SpaceXAI: Grok 4.20 Multi-Agent | 54.9% | 2026-03-17 | independent | Source ↗ |
| SpaceXAI: Grok 4.20 | 54.9% | 2026-03-17 | independent | Source ↗ |
| ByteDance Seed: Seed-2.0-Lite | 54.9% | 2026-03-17 | independent | Source ↗ |
| GPT-5.4 Pro | 59.1% | 2026-03-13 | independent | Source ↗ |
| GPT-5.4 | 59.1% | 2026-03-13 | independent | Source ↗ |
| Gemini 3.1 Flash Lite | 54.9% | 2026-03-11 | independent | Source ↗ |
| Gemini 3.1 Flash Image | 54.9% | 2026-03-06 | independent | Source ↗ |
| GPT-5.3-Codex | 59.1% | 2026-03-04 | independent | Source ↗ |
| ByteDance Seed: Seed-2.0-Mini | 52.4% | 2026-03-04 | independent | Source ↗ |
| Qwen: Qwen3.5-35B-A3B | 52.4% | 2026-03-04 | independent | Source ↗ |
| Qwen: Qwen3.5-27B | 54.9% | 2026-03-04 | independent | Source ↗ |
| Qwen: Qwen3.5-122B-A10B | 54.9% | 2026-03-04 | independent | Source ↗ |
| Qwen: Qwen3.5-Flash | 52.4% | 2026-03-04 | independent | Source ↗ |
| Gemini 3.1 Pro Custom Tools | 59.1% | 2026-02-27 | independent | Source ↗ |
| Gemini 3.1 Pro | 59.1% | 2026-02-27 | independent | Source ↗ |
| Claude Sonnet 4.6 | 59.1% | 2026-02-25 | independent | Source ↗ |
| Qwen: Qwen3.5 Plus 2026-02-15 | 54.9% | 2026-02-24 | independent | Source ↗ |
| Qwen: Qwen3.5 397B A17B | 57.2% | 2026-02-24 | independent | Source ↗ |
| Z.ai: GLM 5 | 57.2% | 2026-02-19 | independent | Source ↗ |
| MiniMax: MiniMax M2.5 | 52.4% | 2026-02-19 | independent | Source ↗ |
| Claude Opus 4.6 | 59.1% | 2026-02-13 | independent | Source ↗ |
| MoonshotAI: Kimi K2.5 | 54.9% | 2026-02-04 | independent | Source ↗ |
| MiniMax: MiniMax M2-her | 54.9% | 2026-01-31 | independent | Source ↗ |
| Z.ai: GLM 4.7 Flash | 52.4% | 2026-01-27 | independent | Source ↗ |
| GPT-5.2-Codex | 59.1% | 2026-01-22 | independent | Source ↗ |
| MiniMax: MiniMax M2.1 | 54.9% | 2025-12-31 | independent | Source ↗ |
| Z.ai: GLM 4.7 | 54.9% | 2025-12-30 | independent | Source ↗ |
| o3 | 59.8% | 2025-12-28 | independent | Source ↗ |
| Gemini 3 Flash | 54.9% | 2025-12-25 | independent | Source ↗ |
| GPT-5.2 Chat | 59.1% | 2025-12-19 | independent | Source ↗ |
| GPT-5.2 Pro | 59.1% | 2025-12-19 | independent | Source ↗ |
| GPT-5.2 | 59.1% | 2025-12-19 | independent | Source ↗ |
| Z.ai: GLM 4.6V | 52.4% | 2025-12-16 | independent | Source ↗ |
| GPT-5.1-Codex-Max | 57.2% | 2025-12-12 | independent | Source ↗ |
| DeepSeek V3.2 | 52.4% | 2025-12-09 | independent | Source ↗ |
| Claude Opus 4.5 | 59.1% | 2025-12-02 | independent | Source ↗ |
| Gemini 3 Pro Image (Nano Banana Pro) | 59.1% | 2025-11-28 | independent | Source ↗ |
| GPT-5.1 | 57.2% | 2025-11-21 | independent | Source ↗ |
| GPT-5.1-Codex | 57.2% | 2025-11-21 | independent | Source ↗ |
| GPT-5.1-Codex-Mini | 54.9% | 2025-11-21 | independent | Source ↗ |
| Mistral Large 3 | 49.6% | 2025-11-20 | independent | Source ↗ |
| MoonshotAI: Kimi K2 Thinking | 54.9% | 2025-11-14 | independent | Source ↗ |
| MiniMax: MiniMax M2 | 54.9% | 2025-10-31 | independent | Source ↗ |
| Qwen: Qwen3 VL 8B Instruct | 52.4% | 2025-10-22 | independent | Source ↗ |
| GPT-5 Pro | 59.1% | 2025-10-14 | independent | Source ↗ |
| Qwen: Qwen3 VL 30B A3B Thinking | 52.4% | 2025-10-14 | independent | Source ↗ |
| Qwen: Qwen3 VL 30B A3B Instruct | 52.4% | 2025-10-14 | independent | Source ↗ |
| Claude Haiku 4.5 | 57.2% | 2025-10-09 | independent | Source ↗ |
| Claude Sonnet 4.5 | 59.1% | 2025-10-07 | independent | Source ↗ |
| Qwen: Qwen3 VL 235B A22B Instruct | 54.9% | 2025-10-01 | independent | Source ↗ |
| Qwen: Qwen3 Coder Flash | 52.4% | 2025-09-25 | independent | Source ↗ |
| Qwen: Qwen3 Next 80B A3B Instruct | 54.9% | 2025-09-19 | independent | Source ↗ |
| MoonshotAI: Kimi K2 0905 | 54.9% | 2025-09-12 | independent | Source ↗ |
| GPT-5 | 57.2% | 2025-08-15 | independent | Source ↗ |
| GPT-5 Mini | 54.9% | 2025-08-15 | independent | Source ↗ |
| GPT-5 Nano | 52.4% | 2025-08-15 | independent | Source ↗ |
| Claude 4.1 Opus | 59.1% | 2025-08-13 | independent | Source ↗ |
| Qwen: Qwen3 Coder 30B A3B Instruct | 52.4% | 2025-08-08 | independent | Source ↗ |
| Qwen: Qwen3 30B A3B Instruct 2507 | 52.4% | 2025-08-06 | independent | Source ↗ |
| Qwen: Qwen3 235B A22B Thinking 2507 | 57.2% | 2025-08-02 | independent | Source ↗ |
| Qwen: Qwen3 Coder 480B A35B | 52.4% | 2025-07-31 | independent | Source ↗ |
| ByteDance: UI-TARS 7B | 49.7% | 2025-07-30 | independent | Source ↗ |
| Qwen: Qwen3 235B A22B Instruct 2507 | 52.4% | 2025-07-29 | independent | Source ↗ |
| MoonshotAI: Kimi K2 0711 | 54.9% | 2025-07-19 | independent | Source ↗ |
| ByteDance Seed: Seed 1.6 Flash | 52.4% | 2025-07-03 | independent | Source ↗ |
| ByteDance Seed: Seed 1.6 | 54.9% | 2025-07-03 | independent | Source ↗ |
| MiniMax: MiniMax M1 | 54.9% | 2025-06-25 | independent | Source ↗ |
| Sarvam 105B | 55.1% | 2025-06-23 | independent | Source ↗ |
| o3 Pro | 59.1% | 2025-06-18 | independent | Source ↗ |
| Sarvam Vision (Document AI) | 49% | 2025-05-28 | independent | Source ↗ |
| Amazon Nova 2 Lite | 50.3% | 2025-05-26 | independent | Source ↗ |
| Amazon Nova 2 Lite | 54.2% | 2025-05-26 | independent | Source ↗ |
| Qwen: Qwen3 30B A3B | 52.4% | 2025-05-06 | independent | Source ↗ |
| Qwen: Qwen3 14B | 52.4% | 2025-05-06 | independent | Source ↗ |
| Qwen: Qwen3 32B | 52.4% | 2025-05-06 | independent | Source ↗ |
| o4 Mini High | 57.2% | 2025-04-24 | independent | Source ↗ |
| o3 | 57.2% | 2025-04-24 | independent | Source ↗ |
| o4 Mini | 57.2% | 2025-04-24 | independent | Source ↗ |
| Muse Wonder | 59.5% | 2025-04-23 | independent | Source ↗ |
| Muse Wonder Full | 61.9% | 2025-04-23 | independent | Source ↗ |
| GPT-4.1 | 57.2% | 2025-04-22 | independent | Source ↗ |
| GPT-4.1 Mini | 54.9% | 2025-04-22 | independent | Source ↗ |
| GPT-4.1 Nano | 52.4% | 2025-04-22 | independent | Source ↗ |
| Gemma 3 27B | 53.7% | 2025-04-20 | independent | Source ↗ |
| Nemotron 3 Ultra 550B | 56.8% | 2025-04-18 | independent | Source ↗ |
| Sarvam 30B | 52% | 2025-04-18 | independent | Source ↗ |
| Llama 4 Maverick | 53.7% | 2025-04-13 | independent | Source ↗ |
| Llama 4 Scout | 52.7% | 2025-04-13 | independent | Source ↗ |
| Gemini 2.5 Pro | 58.8% | 2025-04-02 | independent | Source ↗ |
| Gemini 2.5 Pro | 57.2% | 2025-04-02 | independent | Source ↗ |
| Gemini 2.5 Pro 06-05 | 57.2% | 2025-04-02 | independent | Source ↗ |
| Microsoft-Decision-1 | 55.1% | 2025-03-30 | independent | Source ↗ |
| Nemotron 3 Super 120B | 53% | 2025-03-28 | independent | Source ↗ |
| OpenAI Decisions API | 58.5% | 2025-03-26 | independent | Source ↗ |
| Jev 1.14 | 56.8% | 2025-03-22 | independent | Source ↗ |
| Krutrim-2 | 53% | 2025-03-20 | independent | Source ↗ |
| Reka Flash 3 | 49.7% | 2025-03-20 | independent | Source ↗ |
| Amazon Nova Premier 1.0 | 55.4% | 2025-03-18 | independent | Source ↗ |
| Amazon: Nova Premier 1.0 | 59.1% | 2025-03-18 | independent | Source ↗ |
| QwQ 32B | 50.9% | 2025-03-13 | independent | Source ↗ |
| Saaras v3 (Speech-to-Text) | 40.8% | 2025-03-09 | independent | Source ↗ |
| Bulbul v3 (Text-to-Speech) | 37.4% | 2025-03-09 | independent | Source ↗ |
| Phi-4-multimodal | 50.3% | 2025-03-08 | independent | Source ↗ |
| GPT-4.5 | 47.9% | 2025-03-07 | independent | Source ↗ |
| Claude 3.7 Sonnet | 60.2% | 2025-03-04 | independent | Source ↗ |
| Grok 3 | 53.4% | 2025-02-25 | independent | Source ↗ |
| Grok 3 Mini | 53% | 2025-02-25 | independent | Source ↗ |
| Phi-4-mini | 48.3% | 2025-02-23 | independent | Source ↗ |
| Perplexity: Sonar Deep Research | 57.2% | 2025-02-22 | independent | Source ↗ |
| Gemini 2.5 Flash (Thinking) | 56.1% | 2025-02-18 | independent | Source ↗ |
| Perplexity: Sonar Reasoning Pro | 57.2% | 2025-02-14 | independent | Source ↗ |
| Gemini 2.0 Pro | 50% | 2025-02-13 | independent | Source ↗ |
| Gemini 2.0 Flash | 46.5% | 2025-02-13 | independent | Source ↗ |
| Sonar Reasoning Pro | 57.5% | 2025-02-13 | independent | Source ↗ |
| Qwen: Qwen3 Coder Next | 52.4% | 2025-02-11 | independent | Source ↗ |
| o3-mini | 57.1% | 2025-02-08 | independent | Source ↗ |
| o3 Mini High | 57.2% | 2025-02-08 | independent | Source ↗ |
| o3 Mini | 57.2% | 2025-02-08 | independent | Source ↗ |
| Qwen 2.5 Max | 48.6% | 2025-02-05 | independent | Source ↗ |
| Qwen: Qwen2.5 VL 72B Instruct | 52.4% | 2025-02-03 | independent | Source ↗ |
| Qwen: Qwen-Plus | 52.4% | 2025-02-02 | independent | Source ↗ |
| Ollama DeepSeek-R1 (Q4_K_M) | 57.8% | 2025-01-30 | independent | Source ↗ |
| Perplexity: Sonar Pro Search | 59.1% | 2025-01-29 | independent | Source ↗ |
| Perplexity: Sonar Pro | 59.1% | 2025-01-29 | independent | Source ↗ |
| DeepSeek-R1 | 58.5% | 2025-01-28 | independent | Source ↗ |
| Gemini 2.0 Flash Thinking | 51.1% | 2025-01-28 | independent | Source ↗ |
| Kimi k1.5 | 48.3% | 2025-01-28 | independent | Source ↗ |
| R1 0528 | 54.9% | 2025-01-28 | independent | Source ↗ |
| MiniMax-Text-01 | 48.3% | 2025-01-23 | independent | Source ↗ |
| MiniMax: MiniMax-01 | 54.9% | 2025-01-23 | independent | Source ↗ |
| Codestral 25.01 | 49% | 2025-01-22 | independent | Source ↗ |
| Phi-4 | 56.1% | 2025-01-16 | independent | Source ↗ |
| Phi 4 | 49.7% | 2025-01-16 | independent | Source ↗ |
| DeepSeek-V3 | 47.1% | 2025-01-03 | independent | Source ↗ |
| DeepSeek V3.2 Exp | 52.4% | 2025-01-03 | independent | Source ↗ |
| DeepSeek V3.1 Terminus | 52.4% | 2025-01-03 | independent | Source ↗ |
| DeepSeek V3.1 | 52.4% | 2025-01-03 | independent | Source ↗ |
| DeepSeek V3 0324 | 54.9% | 2025-01-03 | independent | Source ↗ |
| DeepSeek V3 | 54.9% | 2025-01-03 | independent | Source ↗ |
| o1 | 59.1% | 2024-12-25 | independent | Source ↗ |
| Sonar Pro | 53.4% | 2024-12-18 | independent | Source ↗ |
| Ollama Llama 3.3 (Q4_K_M) | 52.5% | 2024-12-16 | independent | Source ↗ |
| Llama 3.3 70B | 44.3% | 2024-12-14 | independent | Source ↗ |
| Llama 3.3 70B Instruct | 53% | 2024-12-14 | independent | Source ↗ |
| Llama 3.3 70B Instruct | 52.4% | 2024-12-14 | independent | Source ↗ |
| o1 | 50.3% | 2024-12-13 | independent | Source ↗ |
| o1-pro | 59.1% | 2024-12-13 | independent | Source ↗ |
| Amazon Nova Pro | 49% | 2024-12-11 | independent | Source ↗ |
| Amazon Nova Lite | 44.2% | 2024-12-11 | independent | Source ↗ |
| Amazon Nova Pro 1.0 | 51.8% | 2024-12-11 | independent | Source ↗ |
| Amazon Nova Lite 1.0 | 47.5% | 2024-12-11 | independent | Source ↗ |
| Amazon Nova Micro 1.0 | 43.2% | 2024-12-11 | independent | Source ↗ |
| Amazon: Nova Lite 1.0 | 52.4% | 2024-12-11 | independent | Source ↗ |
| Amazon: Nova Micro 1.0 | 49.7% | 2024-12-11 | independent | Source ↗ |
| Amazon: Nova Pro 1.0 | 57.2% | 2024-12-11 | independent | Source ↗ |
| Command R7B (12-2024) | 49.7% | 2024-12-09 | independent | Source ↗ |
| QwQ-32B | 54.4% | 2024-12-06 | independent | Source ↗ |
| Krutrim Pro | 50.3% | 2024-11-28 | independent | Source ↗ |
| GPT-4o | 57.2% | 2024-11-28 | independent | Source ↗ |
| Upstage: Solar Pro 4 | 52.4% | 2024-11-28 | independent | Source ↗ |
| Jev | 55.8% | 2024-11-23 | independent | Source ↗ |
| Sonar | 50.3% | 2024-11-23 | independent | Source ↗ |
| TypeSafe: Jev Router | 49.7% | 2024-11-23 | independent | Source ↗ |
| Perplexity: Sonar | 52.4% | 2024-11-23 | independent | Source ↗ |
| Qwen 2.5 Coder 32B | 50.3% | 2024-11-20 | independent | Source ↗ |
| Qwen2.5 Coder 32B Instruct | 52.4% | 2024-11-20 | independent | Source ↗ |
| Hunyuan-Large | 49.6% | 2024-11-13 | independent | Source ↗ |
| Claude 3.5 Haiku | 43.5% | 2024-11-12 | independent | Source ↗ |
| Claude 3.5 Sonnet | 49.5% | 2024-10-30 | independent | Source ↗ |
| Claude 3.5 Sonnet (v2) | 53.9% | 2024-10-30 | independent | Source ↗ |
| BharatGPT-14B | 47.6% | 2024-10-26 | independent | Source ↗ |
| Yi-Lightning | 49.3% | 2024-10-23 | independent | Source ↗ |
| Llama-3.1-Nemotron-70B-Instruct | 53.4% | 2024-10-23 | independent | Source ↗ |
| Llama 3.2 1B Instruct | 52.4% | 2024-10-03 | independent | Source ↗ |
| Llama 3.2 3B Instruct | 52.4% | 2024-10-03 | independent | Source ↗ |
| Qwen 2.5 72B Instruct | 55.1% | 2024-09-27 | independent | Source ↗ |
| Qwen: Qwen2.5 7B Instruct | 49.7% | 2024-09-27 | independent | Source ↗ |
| Qwen2.5 72B Instruct | 52.4% | 2024-09-27 | independent | Source ↗ |
| Switchyard | 49.7% | 2024-09-23 | independent | Source ↗ |
| Mistral Small 4 | 52.4% | 2024-09-23 | independent | Source ↗ |
| GPT Audio | 57.2% | 2024-09-23 | independent | Source ↗ |
| GPT Audio Mini | 54.9% | 2024-09-23 | independent | Source ↗ |
| Nemotron 3 Nano 30B A3B | 52.4% | 2024-09-23 | independent | Source ↗ |
| Devstral 2 2512 | 54.9% | 2024-09-23 | independent | Source ↗ |
| Ministral 3 14B 2512 | 49.7% | 2024-09-23 | independent | Source ↗ |
| Ministral 3 8B 2512 | 49.7% | 2024-09-23 | independent | Source ↗ |
| Ministral 3 3B 2512 | 49.7% | 2024-09-23 | independent | Source ↗ |
| Voxtral Small 24B 2507 | 52.4% | 2024-09-23 | independent | Source ↗ |
| gpt-oss-safeguard-20b | 52.4% | 2024-09-23 | independent | Source ↗ |
| Gemini 2.5 Flash Image | 54.9% | 2024-09-23 | independent | Source ↗ |
| Z.ai: GLM 4.6 | 54.9% | 2024-09-23 | independent | Source ↗ |
| Mistral Medium 3.1 | 54.9% | 2024-09-23 | independent | Source ↗ |
| Z.ai: GLM 4.5V | 54.9% | 2024-09-23 | independent | Source ↗ |
| gpt-oss-120b | 52.4% | 2024-09-23 | independent | Source ↗ |
| gpt-oss-20b | 49.7% | 2024-09-23 | independent | Source ↗ |
| Codestral 2508 | 52.4% | 2024-09-23 | independent | Source ↗ |
| Z.ai: GLM 4.5 | 54.9% | 2024-09-23 | independent | Source ↗ |
| Z.ai: GLM 4.5 Air | 52.4% | 2024-09-23 | independent | Source ↗ |
| Gemini 2.5 Flash Lite | 52.4% | 2024-09-23 | independent | Source ↗ |
| Hunyuan A13B Instruct | 52.4% | 2024-09-23 | independent | Source ↗ |
| Mistral Small 3.2 24B | 52.4% | 2024-09-23 | independent | Source ↗ |
| Gemini 2.5 Flash | 54.9% | 2024-09-23 | independent | Source ↗ |
| Mistral Medium 3 | 54.9% | 2024-09-23 | independent | Source ↗ |
| Llama Guard 4 12B | 49.7% | 2024-09-23 | independent | Source ↗ |
| Llama 4 Maverick | 52.4% | 2024-09-23 | independent | Source ↗ |
| Llama 4 Scout | 52.4% | 2024-09-23 | independent | Source ↗ |
| Mistral Small 3.1 24B | 52.4% | 2024-09-23 | independent | Source ↗ |
| Gemma 3 4B | 49.7% | 2024-09-23 | independent | Source ↗ |
| Gemma 3 12B | 49.7% | 2024-09-23 | independent | Source ↗ |
| Gemma 3 27B | 52.4% | 2024-09-23 | independent | Source ↗ |
| Saba | 52.4% | 2024-09-23 | independent | Source ↗ |
| Mistral Small 3 | 49.7% | 2024-09-23 | independent | Source ↗ |
| Mistral Nemo | 49.7% | 2024-09-23 | independent | Source ↗ |
| Gemma 2 27B | 52.4% | 2024-09-23 | independent | Source ↗ |
| Mixtral 8x22B Instruct | 57.2% | 2024-09-23 | independent | Source ↗ |
| GPT-4 Turbo | 59.1% | 2024-09-23 | independent | Source ↗ |
| Mistral Large | 57.2% | 2024-09-23 | independent | Source ↗ |
| GPT-3.5 Turbo (older v0613) | 54.9% | 2024-09-23 | independent | Source ↗ |
| GPT-3.5 Turbo Instruct | 54.9% | 2024-09-23 | independent | Source ↗ |
| GPT-3.5 Turbo 16k | 57.2% | 2024-09-23 | independent | Source ↗ |
| GPT-3.5 Turbo | 54.9% | 2024-09-23 | independent | Source ↗ |
| GPT-4 | 59.1% | 2024-09-23 | independent | Source ↗ |
| GLM-4-Plus | 47.9% | 2024-08-28 | independent | Source ↗ |
| Sarvam 2B (Sarvam-1) | 46.2% | 2024-08-22 | independent | Source ↗ |
| GPT-4o | 57.2% | 2024-08-14 | independent | Source ↗ |
| Command R (08-2024) | 52.4% | 2024-08-09 | independent | Source ↗ |
| Gemma 2 2B | 35.4% | 2024-08-08 | independent | Source ↗ |
| Mistral Large 2 | 43.9% | 2024-08-01 | independent | Source ↗ |
| Mistral Large 2 | 51% | 2024-08-01 | independent | Source ↗ |
| Mistral Large 3 2512 | 54.9% | 2024-08-01 | independent | Source ↗ |
| Llama 3.1 405B | 45.6% | 2024-07-31 | independent | Source ↗ |
| Llama 3.1 70B Instruct | 52.4% | 2024-07-31 | independent | Source ↗ |
| Llama 3.1 8B Instruct | 49.7% | 2024-07-31 | independent | Source ↗ |
| GPT-4o mini | 39.4% | 2024-07-26 | independent | Source ↗ |
| GPT-4o-mini | 52.4% | 2024-07-26 | independent | Source ↗ |
| GPT-4o-mini | 52.4% | 2024-07-26 | independent | Source ↗ |
| ERNIE 4.0 Turbo | 46.9% | 2024-07-06 | independent | Source ↗ |
| Gemma 2 27B | 49% | 2024-07-05 | independent | Source ↗ |
| Gemma 2 9B | 44.2% | 2024-07-05 | independent | Source ↗ |
| Nemotron-4-340B-Instruct | 51.7% | 2024-06-22 | independent | Source ↗ |
| Nemotron-4-340B-Reward | 53% | 2024-06-22 | independent | Source ↗ |
| Hanooman 7B | 46.2% | 2024-05-23 | independent | Source ↗ |
| GPT-4o | 45.2% | 2024-05-21 | independent | Source ↗ |
| GPT-4o | 57.2% | 2024-05-21 | independent | Source ↗ |
| GPT-4o | 59.1% | 2024-05-21 | independent | Source ↗ |
| GPT-4 Turbo + SWE-agent | 12.47% | 2024-05-06 | independent | Source ↗ |
| Snowflake Arctic | 45.6% | 2024-05-02 | independent | Source ↗ |
| WizardLM-2 8x22B | 51% | 2024-04-23 | independent | Source ↗ |
| DBRX Instruct | 46.2% | 2024-04-04 | independent | Source ↗ |
| Claude 3 Opus (BM25 retrieval) | 3.79% | 2024-03-04 | independent | Source ↗ |
| Airavata 7B | 43.5% | 2024-02-02 | independent | Source ↗ |
| Claude 2 (BM25 retrieval) | 1.97% | 2023-10-10 | independent | Source ↗ |
| IndicTrans-2 | 34% | 2023-09-23 | independent | Source ↗ |
| Nemotron 3 Super | 52.4% | 2023-03-19 | independent | Source ↗ |
Human Baseline & Difficulty Horizon
Calibrated human reference points, specialist benchmarks, and ceiling thresholds.Measured Human Score78.4%Domain Expert Baseline
Baseline Protocol & Interpretation
Measured solve rate of experienced software engineers resolving unfamiliar repository GitHub issues within standard 4-hour time limits.
Metric & Scoring Methodology
Verification protocols, aggregation formulas, and specialized metric variants.Primary Metric:
resolved instances (%)Scoring Engine:unit-tests
Dataset & Compute Cost
Evaluation volume, public availability, API pricing, and local hardware requirements.Total Dataset Size2,294Annotated evaluation items
Public Test SetPublicOpenly mirrored on repositories
Access GatingOpen AccessUnrestricted download
Evaluation LicenseOpen AccessDataset usage and redistribution terms
Frontier API Compute Cost:
$5 – $20 USD for full benchmark evaluation run on frontier APIs.
Recommended Local GPU Setup:
1x NVIDIA RTX 4090 (24GB) or A100 (40GB/80GB) via vLLM / SGLang
Official Dataset & Benchmark Files:Download / View Dataset Repository ↗
How to Run & Reproduce
Standardized evaluation protocols, CLI commands, and reproducible runner templates.Prompt Regimezero-shot
Reasoning Modedirect
Sampling Temp0
Pass@k Budgetk = 1
Tools & SandboxPure Text
Scoring Verifierunit-tests
Option AEleutherAI LM-Evaluation-Harness (Open-Weight Models)
lm_eval --model hf --model_args pretrained=<model_path> --tasks swe-bench --batch_size autoOption BOpenCompass Evaluation Framework
opencompass --datasets swe-bench --models <model_config>Python APIDeterministic Inference Loop Snippet
# Standard API Evaluation Loop
from openai import OpenAI
client = OpenAI()
response = client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": prompt}],
temperature=0.0,
)Standardized Reporting Requirement:
When publishing results for SWE-bench, always report the exact prompt template, few-shot exemplar ordering, sampling temperature (temperature=0), maximum reasoning budget tokens, and the precise timestamped model snapshot ID.
Contamination & Memorization Analysis
Audit of pretraining exposure risks, memorization vectors, and refresh policies.Overall Contamination Risk:HIGH
Refresh Cadence:
Static fixed snapshot
Test Set Exposure:
Public on web / HuggingFace