DeepSWE: the benchmark that shows why coding agents fail on long tasks
Datacurve published DeepSWE on May 26, 2026. At first glance, it is another coding leaderboard. The practical point is more important: DeepSWE measures the kind of work coding agents are moving toward now. Not one isolated patch, but a longer task in an unfamiliar repository where the agent has to find context, make a change, verify it, and avoid breaking the rest of the project.
That matters for teams that are no longer asking “can AI write a function”, but “can I hand an agent a real issue and get a pull request that survives review.”
What DeepSWE Measures Differently
DeepSWE is a response to a real problem: public coding benchmarks are starting to lose resolution at the frontier. Models look close on leaderboards, while developers experience much larger differences in real work.
Datacurve built DeepSWE around four ideas:
- tasks are newly written, not copied from existing pull requests,
- the benchmark spans 91 active open-source repositories across 5 languages,
- solutions are substantially larger than typical SWE-bench tasks,
- verifiers test software behavior, not the exact shape of the author’s reference patch.
DeepSWE contains 113 tasks. Reference solutions add an average of 668 lines of code and touch about 7 files. Datacurve compares this with roughly 120 added lines for SWE-bench Pro. That is the difference between “fix one well-described bug” and “do the kind of change you might actually delegate to an agent overnight.”
Current Results: GPT-5.5 Leads, Opus 4.8 Is Strong But Expensive
The leaderboard was updated on May 30, 2026 and includes Claude Opus 4.8. The top of the table currently looks like this:
| Model / mode | DeepSWE Pass@1 | Avg cost per task | Avg time | Output tokens |
|---|---|---|---|---|
| GPT-5.5 xhigh | 70% ± 4% | $6.61 | 21 min | 47k |
| Claude Opus 4.8 max | 58% ± 5% | $12.58 | 43 min | 136k |
| GPT-5.4 xhigh | 56% ± 5% | $4.38 | 27 min | 71k |
| Claude Opus 4.7 max | 54% ± 5% | $18.19 | 39 min | 103k |
| Claude Sonnet 4.6 high | 32% ± 4% | $5.52 | 42 min | 76k |
| Gemini 3.5 Flash medium | 28% ± 4% | $7.42 | 17 min | 189k |
Important caveat: this is not pure model measurement in a vacuum. Every model runs through the same ’mini-swe-agent’ harness to reduce the effect of native products and vendor-specific prompts. That is good for a benchmark, but it does not mean you will get the same result in Claude Code, Codex, Cursor, or Gemini CLI.
The Practical Takeaway
There are three useful conclusions.
First, coding agents can no longer be judged by one headline SWE-bench number. You need to know how the model behaves on a long task, how many tokens it spends, how long it runs, and how much human cleanup remains.
Second, Opus 4.8 confirms that Anthropic is still very strong in agentic coding. But DeepSWE also shows the cost of long reasoning. In this run, Opus 4.8 has higher average cost, time, and output tokens than GPT-5.5.
Third, a cheap model does not automatically produce a cheap workflow. Gemini 3.5 Flash is faster here, but produces a lot of output tokens and has a lower pass rate. If the agent retries, runs long, and leaves review debt behind, cheap tokens stop being cheap.
How I Would Use DeepSWE In A Company
I would not treat DeepSWE as absolute truth. I would use it as a filter.
For a real coding-agent pilot:
- Pick 10 to 20 real internal tasks from the last few months.
- Split them into small fixes, medium changes, and long agentic tasks.
- Measure success, cost, time, iterations, diff size, and review effort.
- For expensive changes, require a plan, tests, and a risk summary.
- Do not let the agent merge directly to production. Let it prepare a pull request for normal review.
DeepSWE is most useful for the third category: long tasks where the agent has to keep intent across several steps.
Limitations
DeepSWE is a better signal than many older coding leaderboards, but it is still a benchmark.
Datacurve has its own commercial interest. ’mini-swe-agent’ is not the same as real work in Claude Code or Codex. And every public benchmark eventually becomes something labs and users can tune toward.
So I would not read DeepSWE as “GPT-5.5 is always the best coding agent.” I would read it as a strong warning: long-horizon coding-agent work measures something different from short patches. The gap between models is larger there than older tables suggest.
Verdict
DeepSWE deserves its own article because it changes the question we should ask about coding agents.
Not “which model has the highest number”, but “which agent can complete a long task at a reasonable cost, test its own work, and produce a change a human can actually review.”
That is the boundary where coding agents start moving from better autocomplete to real engineering capacity.