Model benchmarks
Coding benchmark results, side by side. Each benchmark uses different tasks and scoring, so the numbers cannot be compared directly.
ProgramBench
Reimplementation of source code from a binary and its description. The 30 entries of the official ranking are listed.
The agent is mini-SWE-agent. The full-solve rate is the share of tasks passing every test; the almost-solve rate is the share passing at least 95% of tests. A parenthesised model name notes the setting used for evaluation.
Official ranking updated: / Checked: / ProgramBench leaderboard
| Rank | Model / setting | Full solve | Almost solve | Cost per task |
|---|---|---|---|---|
| 1 | Claude Opus 5.5 (xhigh) | 18.5% | 70.5% | $36.36 |
| 2 | Claude Fable 5.1 (xhigh) | 9.0% | 55.5% | $56.84 |
| 3 | Muse Wonder Full | 8.5% | 60.0% | Not published |
| 4 | Muse Wonder (xhigh) | 5.5% | 50.5% | $36.53 |
| 5 | GPT 6 Astra (max) | 5.0% | 52.5% | $11.47 |
| 6 | Claude Opus 5 (xhigh) | 4.5% | 37.0% | $50.53 |
| 7 | Muse Spark 1.3 (max) | 2.5% | 25.0% | $6.46 |
| 8 | Muse Spark 1.3 (xhigh) | 1.0% | 16.5% | $2.02 |
| 9 | GPT-5.6 Sol (xhigh) | 1.0% | 15.5% | $6.08 |
| 10 | GPT 5.5 (xhigh) | 0.5% | 13.5% | $8.85 |
| 11 | GPT 5.5 (high) | 0.5% | 5.0% | $3.65 |
| 12 | Gemini 3.6 Flash | 0.5% | 4.0% | $4.83 |
| 13 | GPT-5.6 Sol | 0.5% | 2.5% | $1.00 |
| 14 | Claude Opus 4.8 (xhigh) | 0.0% | 16.5% | $21.02 |
| 15 | GLM-5.2 | 0.0% | 8.5% | $25.36 |
| 16 | Gemini 3.7 Flash | 0.0% | 5.5% | $2.04 |
| 17 | Muse Spark 1.2 (xhigh) | 0.0% | 4.5% | $1.48 |
| 18 | Claude Opus 4.7 (xhigh) | 0.0% | 4.5% | $10.96 |
| 19 | Muse Spark 1.1 (xhigh) | 0.0% | 4.0% | $0.73 |
| 20 | Gemini 3.5 Flash | 0.0% | 3.0% | $5.60 |
| 21 | Claude Opus 4.7 | 0.0% | 3.0% | $3.81 |
| 22 | Claude Opus 4.6 | 0.0% | 2.5% | $11.38 |
| 23 | GPT 5.5 | 0.0% | 1.5% | $1.21 |
| 24 | Claude Sonnet 4.6 | 0.0% | 1.0% | $26.73 |
| 25 | GPT 5.4 | 0.0% | 0.0% | $0.33 |
| 26 | Gemini 3.1 Pro | 0.0% | 0.0% | $1.51 |
| 27 | Gemini 3 Flash | 0.0% | 0.0% | $0.30 |
| 28 | Claude Haiku 4.5 | 0.0% | 0.0% | $0.80 |
| 29 | GPT 5.4 mini | 0.0% | 0.0% | $0.04 |
| 30 | GPT 5 mini | 0.0% | 0.0% | $0.03 |
Test coverage has limits. Costs are the API spend at evaluation time, not current prices. Evaluation dates and harness versions are not confirmed for each row here. Nothing here guarantees real-world performance or availability.
Aider Polyglot
the top 20 of the 69 official leaderboard entries, by accuracy after the second attempt
Aider solves tasks in C++, Go, Java, JavaScript, Python and Rust. Accuracy is measured after at most two attempts. Evaluation dates, Aider versions, edit formats and task counts differ, so compare with care.
- Metric
- Accuracy (% correct, after the second attempt)
- Latest official evaluation
- Checked
- Source
- Aider leaderboard
the top 20 of the 69 official leaderboard entries, by accuracy after the second attempt
| Rank | Model / setting | Accuracy | Cost of the full run | Evaluated | Aider version | Edit format | Tasks |
|---|---|---|---|---|---|---|---|
| 1 | gpt-5 (high) | 88.0% | $29.08 | 0.86.2.dev | diff | 225 | |
| 2 | gpt-5 (medium) | 86.7% | $17.69 | 0.86.2.dev | diff | 225 | |
| 3 | o3-pro (high) | 84.9% | $146.32 | 0.85.1.dev | diff | 225 | |
| 4 | gemini-2.5-pro-preview-06-05 (32k think) | 83.1% | $49.88 | 0.84.1.dev | diff-fenced | 225 | |
| 5 | gpt-5 (low) | 81.3% | $10.37 | 0.86.2.dev | diff | 225 | |
| 6 | o3 (high) | 81.3% | $21.23 | 0.84.1.dev | diff | 225 | |
| 7 | grok-4 (high) | 79.6% | $59.62 | 0.85.2.dev | diff | 225 | |
| 8 | gemini-2.5-pro-preview-06-05 (default think) | 79.1% | $45.60 | 0.84.1.dev | diff-fenced | 225 | |
| 9 | o3 (high) + gpt-4.1 | 78.2% | $17.55 | 0.85.1.dev | architect | 224 | |
| 10 | o3 | 76.9% | $13.75 | 0.84.1.dev | diff | 225 | |
| 11 | Gemini 2.5 Pro Preview 05-06 | 76.9% | $37.41 | 0.82.4.dev | diff-fenced | 225 | |
| 12 | DeepSeek-V3.2-Exp (Reasoner) | 74.2% | $1.30 | 0.86.2.dev | diff | 225 | |
| 13 | Gemini 2.5 Pro Preview 03-25 | 72.9% | Not published | 0.81.3.dev | diff-fenced | 225 | |
| 14 | claude-opus-4-20250514 (32k thinking) | 72.0% | $65.75 | 0.83.3.dev | diff | 225 | |
| 15 | o4-mini (high) | 72.0% | $19.64 | 0.82.1.dev | diff | 225 | |
| 16 | DeepSeek R1 (0528) | 71.4% | $4.80 | 0.84.1.dev | diff | 224 | |
| 17 | claude-opus-4-20250514 (no think) | 70.7% | $68.63 | 0.83.3.dev | diff | 225 | |
| 18 | DeepSeek-V3.2-Exp (Chat) | 70.2% | $0.88 | 0.86.2.dev | diff | 225 | |
| 19 | claude-3-7-sonnet-20250219 (32k thinking tokens) | 64.9% | $36.83 | 0.75.1.dev | diff | 225 | |
| 20 | DeepSeek R1 + claude-3-5-sonnet-20241022 | 64.0% | $13.29 | 0.72.3.dev | architect | 225 |
This is a top slice, not the whole official leaderboard. Costs are the API spend of the 225-task (or 224-task) run at the time, not a monthly fee or a current price. Results published as 0 are shown as "Not published" because the amount is unknown. Do not compare these numbers with other benchmarks.
SWE-bench Verified(Bash Only)
Share of real GitHub issues a model can resolve. Only results the SWE-bench team confirmed, comparing every model in the minimal Bash environment of mini-SWE-agent, are listed.
The listed results were evaluated in 2025 and do not cover newer models. The retrieval date is not the evaluation date.
- Metric
- Resolve rate (% Resolved)
- Evaluation harness
- mini-SWE-agent (shared by every result)
- Retrieved
- Source
- SWE-bench leaderboard
| Rank | Model | Resolve rate | Cost per task | Evaluated | Harness version |
|---|---|---|---|---|---|
| 1 | Claude 4.5 OpusAnthropic | 74.4% | $0.721 | 1.16.0 | |
| 2 | Gemini 3 Pro PreviewGoogle DeepMind | 74.2% | $0.460 | 1.15.0 | |
| 3 | Claude 4.5 SonnetAnthropic | 70.6% | $0.558 | 1.13.3 | |
| 4 | Claude 4 OpusAnthropic | 67.6% | $1.131 | 1.0.0 | |
| 5 | GPT 5OpenAI | 65.0% | $0.280 | 1.7.0 | |
| 6 | Claude 4 SonnetAnthropic | 64.9% | $0.371 | 1.0.0 | |
| 7 | Kimi K2 ThinkingMoonshot AI | 63.4% | $0.438 | 1.17.2 | |
| 8 | MiniMax M2Minimax | 61.0% | $0.428 | 1.17.0 | |
| 9 | DeepSeek V3.2 Reasonerdeepseek | 60.0% | $0.028 | 1.17.1 | |
| 10 | GPT 5 miniOpenAI | 59.8% | $0.035 | 1.7.0 | |
| 11 | o3OpenAI | 58.4% | $0.334 | 1.0.0 | |
| 12 | Devstral Small (2512)mistral | 56.4% | Not calculated | 1.17.2 | |
| 13 | GLM 4.6Z.ai | 55.4% | $0.097 | 1.17.1 | |
| 14 | Qwen3-Coder 480B/A35B InstructQwen | 55.4% | $0.248 | 1.0.0 | |
| 15 | GLM 4.5Z.ai | 54.2% | $0.297 | 1.9.1 | |
| 16 | Devstral (2512)mistral | 53.8% | Not calculated | 1.17.2 | |
| 17 | Gemini 2.5 ProGoogle DeepMind | 53.6% | $0.288 | 1.0.0 | |
| 18 | Claude 3.7 SonnetAnthropic | 52.8% | $0.355 | 0.0.0 | |
| 19 | o4-miniOpenAI | 45.0% | $0.210 | 1.0.0 | |
| 20 | Kimi K2 InstructMoonshot AI | 43.8% | $0.532 | 1.7.0 | |
| 21 | GPT 4.1OpenAI | 39.6% | $0.146 | 1.0.0 | |
| 22 | GPT 5 nanoOpenAI | 34.8% | $0.038 | 1.7.0 | |
| 23 | Gemini 2.5 FlashGoogle DeepMind | 28.7% | $0.134 | 1.0.0 | |
| 24 | gpt-oss-120bOpenAI | 26.0% | $0.057 | 1.7.0 | |
| 25 | GPT 4.1 miniOpenAI | 23.9% | $0.435 | 0.0.0 | |
| 26 | GPT 4oOpenAI | 21.6% | $1.531 | 0.0.0 | |
| 27 | Llama 4 Maverick InstructMeta | 21.0% | $0.314 | 0.0.0 | |
| 28 | Gemini 2.0 FlashGoogle DeepMind | 13.5% | Not calculated | 0.0.0 | |
| 29 | Llama 4 Scout InstructMeta | 9.1% | $0.116 | 0.0.0 | |
| 30 | Qwen2.5-Coder 32B InstructQwen | 9.0% | $0.068 | 1.0.0 |
Results from different agent setups and self-reported values the team has not checked are excluded. Scores depend on evaluation conditions and do not guarantee real-world product performance. The cost per instance is an estimate from API prices at evaluation time, not a current price. A 0 is hidden because it means free or not calculated.
Link to pricing data
Benchmark models are linked to pricing data only when the model name matches exactly. A plan that names a model family is not tied to the benchmark of one specific version.
- GPT-5 miniGitHub CopilotFind plans that include this model
How to read these numbers
- SWE-bench measures issue resolution in Python repositories. It does not score UI generation, writing, or long-term maintainability.
- Resolve rates move with the harness, prompts, reasoning settings and evaluation date. This page is limited to one mini-SWE-agent configuration.
- ProgramBench measures reimplementation from a binary and description; Aider Polyglot measures editing tasks in six languages. The metrics and scoring differ, so do not line up the three tables.
- Per-task and full-run costs are estimates from API prices at evaluation time, not current prices or plan fees.
- A coding agent product depends on its own tooling and context management on top of the underlying model result.
- Model versions change performance. Check the official information before you buy.
Regenerating the data
npm run bench:sync / npm run bench:sync:aider
The numbers are regenerated from the public data below with these commands. ProgramBench is transcribed by hand from the official ranking.