Model benchmarks

Coding benchmark results, side by side. Each benchmark uses different tasks and scoring, so the numbers cannot be compared directly.

ProgramBench

Reimplementation of source code from a binary and its description. The 30 entries of the official ranking are listed.

200 tasks

The agent is mini-SWE-agent. The full-solve rate is the share of tasks passing every test; the almost-solve rate is the share passing at least 95% of tests. A parenthesised model name notes the setting used for evaluation.

Official ranking updated: / Checked: / ProgramBench leaderboard

The 30 entries of the official ProgramBench ranking, with full and almost solve rates.
RankModel / settingFull solveAlmost solveCost per task
1Claude Opus 5.5 (xhigh)18.5%70.5%$36.36
2Claude Fable 5.1 (xhigh)9.0%55.5%$56.84
3Muse Wonder Full8.5%60.0%Not published
4Muse Wonder (xhigh)5.5%50.5%$36.53
5GPT 6 Astra (max)5.0%52.5%$11.47
6Claude Opus 5 (xhigh)4.5%37.0%$50.53
7Muse Spark 1.3 (max)2.5%25.0%$6.46
8Muse Spark 1.3 (xhigh)1.0%16.5%$2.02
9GPT-5.6 Sol (xhigh)1.0%15.5%$6.08
10GPT 5.5 (xhigh)0.5%13.5%$8.85
11GPT 5.5 (high)0.5%5.0%$3.65
12Gemini 3.6 Flash0.5%4.0%$4.83
13GPT-5.6 Sol0.5%2.5%$1.00
14Claude Opus 4.8 (xhigh)0.0%16.5%$21.02
15GLM-5.20.0%8.5%$25.36
16Gemini 3.7 Flash0.0%5.5%$2.04
17Muse Spark 1.2 (xhigh)0.0%4.5%$1.48
18Claude Opus 4.7 (xhigh)0.0%4.5%$10.96
19Muse Spark 1.1 (xhigh)0.0%4.0%$0.73
20Gemini 3.5 Flash0.0%3.0%$5.60
21Claude Opus 4.70.0%3.0%$3.81
22Claude Opus 4.60.0%2.5%$11.38
23GPT 5.50.0%1.5%$1.21
24Claude Sonnet 4.60.0%1.0%$26.73
25GPT 5.40.0%0.0%$0.33
26Gemini 3.1 Pro0.0%0.0%$1.51
27Gemini 3 Flash0.0%0.0%$0.30
28Claude Haiku 4.50.0%0.0%$0.80
29GPT 5.4 mini0.0%0.0%$0.04
30GPT 5 mini0.0%0.0%$0.03

Test coverage has limits. Costs are the API spend at evaluation time, not current prices. Evaluation dates and harness versions are not confirmed for each row here. Nothing here guarantees real-world performance or availability.

Aider Polyglot

the top 20 of the 69 official leaderboard entries, by accuracy after the second attempt

Up to 225 tasks

Aider solves tasks in C++, Go, Java, JavaScript, Python and Rust. Accuracy is measured after at most two attempts. Evaluation dates, Aider versions, edit formats and task counts differ, so compare with care.

Metric
Accuracy (% correct, after the second attempt)
Latest official evaluation
Checked
Source
Aider leaderboard

the top 20 of the 69 official leaderboard entries, by accuracy after the second attempt

Aider Polyglot accuracy, highest first.
RankModel / settingAccuracyCost of the full runEvaluatedAider versionEdit formatTasks
1gpt-5 (high)88.0%$29.080.86.2.devdiff225
2gpt-5 (medium)86.7%$17.690.86.2.devdiff225
3o3-pro (high)84.9%$146.320.85.1.devdiff225
4gemini-2.5-pro-preview-06-05 (32k think)83.1%$49.880.84.1.devdiff-fenced225
5gpt-5 (low)81.3%$10.370.86.2.devdiff225
6o3 (high)81.3%$21.230.84.1.devdiff225
7grok-4 (high)79.6%$59.620.85.2.devdiff225
8gemini-2.5-pro-preview-06-05 (default think)79.1%$45.600.84.1.devdiff-fenced225
9o3 (high) + gpt-4.178.2%$17.550.85.1.devarchitect224
10o376.9%$13.750.84.1.devdiff225
11Gemini 2.5 Pro Preview 05-0676.9%$37.410.82.4.devdiff-fenced225
12DeepSeek-V3.2-Exp (Reasoner)74.2%$1.300.86.2.devdiff225
13Gemini 2.5 Pro Preview 03-2572.9%Not published0.81.3.devdiff-fenced225
14claude-opus-4-20250514 (32k thinking)72.0%$65.750.83.3.devdiff225
15o4-mini (high)72.0%$19.640.82.1.devdiff225
16DeepSeek R1 (0528)71.4%$4.800.84.1.devdiff224
17claude-opus-4-20250514 (no think)70.7%$68.630.83.3.devdiff225
18DeepSeek-V3.2-Exp (Chat)70.2%$0.880.86.2.devdiff225
19claude-3-7-sonnet-20250219 (32k thinking tokens)64.9%$36.830.75.1.devdiff225
20DeepSeek R1 + claude-3-5-sonnet-2024102264.0%$13.290.72.3.devarchitect225

This is a top slice, not the whole official leaderboard. Costs are the API spend of the 225-task (or 224-task) run at the time, not a monthly fee or a current price. Results published as 0 are shown as "Not published" because the amount is unknown. Do not compare these numbers with other benchmarks.

SWE-bench Verified(Bash Only)

Share of real GitHub issues a model can resolve. Only results the SWE-bench team confirmed, comparing every model in the minimal Bash environment of mini-SWE-agent, are listed.

The listed results were evaluated in 2025 and do not cover newer models. The retrieval date is not the evaluation date.

500 tasks
Metric
Resolve rate (% Resolved)
Evaluation harness
mini-SWE-agent (shared by every result)
Retrieved
Source
SWE-bench leaderboard
SWE-bench Verified Bash Only resolve rate, highest first.
RankModelResolve rateCost per taskEvaluatedHarness version
1Claude 4.5 OpusAnthropic74.4%$0.7211.16.0
2Gemini 3 Pro PreviewGoogle DeepMind74.2%$0.4601.15.0
3Claude 4.5 SonnetAnthropic70.6%$0.5581.13.3
4Claude 4 OpusAnthropic67.6%$1.1311.0.0
5GPT 5OpenAI65.0%$0.2801.7.0
6Claude 4 SonnetAnthropic64.9%$0.3711.0.0
7Kimi K2 ThinkingMoonshot AI63.4%$0.4381.17.2
8MiniMax M2Minimax61.0%$0.4281.17.0
9DeepSeek V3.2 Reasonerdeepseek60.0%$0.0281.17.1
10GPT 5 miniOpenAI59.8%$0.0351.7.0
11o3OpenAI58.4%$0.3341.0.0
12Devstral Small (2512)mistral56.4%Not calculated1.17.2
13GLM 4.6Z.ai55.4%$0.0971.17.1
14Qwen3-Coder 480B/A35B InstructQwen55.4%$0.2481.0.0
15GLM 4.5Z.ai54.2%$0.2971.9.1
16Devstral (2512)mistral53.8%Not calculated1.17.2
17Gemini 2.5 ProGoogle DeepMind53.6%$0.2881.0.0
18Claude 3.7 SonnetAnthropic52.8%$0.3550.0.0
19o4-miniOpenAI45.0%$0.2101.0.0
20Kimi K2 InstructMoonshot AI43.8%$0.5321.7.0
21GPT 4.1OpenAI39.6%$0.1461.0.0
22GPT 5 nanoOpenAI34.8%$0.0381.7.0
23Gemini 2.5 FlashGoogle DeepMind28.7%$0.1341.0.0
24gpt-oss-120bOpenAI26.0%$0.0571.7.0
25GPT 4.1 miniOpenAI23.9%$0.4350.0.0
26GPT 4oOpenAI21.6%$1.5310.0.0
27Llama 4 Maverick InstructMeta21.0%$0.3140.0.0
28Gemini 2.0 FlashGoogle DeepMind13.5%Not calculated0.0.0
29Llama 4 Scout InstructMeta9.1%$0.1160.0.0
30Qwen2.5-Coder 32B InstructQwen9.0%$0.0681.0.0

Results from different agent setups and self-reported values the team has not checked are excluded. Scores depend on evaluation conditions and do not guarantee real-world product performance. The cost per instance is an estimate from API prices at evaluation time, not a current price. A 0 is hidden because it means free or not calculated.

Link to pricing data

Benchmark models are linked to pricing data only when the model name matches exactly. A plan that names a model family is not tied to the benchmark of one specific version.

How to read these numbers

  • SWE-bench measures issue resolution in Python repositories. It does not score UI generation, writing, or long-term maintainability.
  • Resolve rates move with the harness, prompts, reasoning settings and evaluation date. This page is limited to one mini-SWE-agent configuration.
  • ProgramBench measures reimplementation from a binary and description; Aider Polyglot measures editing tasks in six languages. The metrics and scoring differ, so do not line up the three tables.
  • Per-task and full-run costs are estimates from API prices at evaluation time, not current prices or plan fees.
  • A coding agent product depends on its own tooling and context management on top of the underlying model result.
  • Model versions change performance. Check the official information before you buy.

Regenerating the data

npm run bench:sync / npm run bench:sync:aider

The numbers are regenerated from the public data below with these commands. ProgramBench is transcribed by hand from the official ranking.