← DeepSeek V4 Pro · API · Max effort
Evidenceacceptedthird_party_leaderboard
SWE-bench Verified
Vals.ai SWE-bench Verified (mini-swe-agent) — DeepSeek V4
ev_2026_07_dsv4pro_swe_vals
Result
77.4%
Confidence A
- Tested
- 2026-07-09
- Runs
- 1
Configuration
How it was measured
- Exact model
- deepseek-v4-pro
- Reasoning
- max
- Agent harness
- mini-swe-agent
- Tool access
- bash only (no specialized editor tools)
- Prompting
- Vals default provider config; max tokens raised to model max
PRIMARY STR — independent Vals.ai SWE-bench Verified board using mini-swe-agent for all models. Prefer over openlm mirror and vendor for comparative STR.
Caveats
Known limitations
- Point estimate with published stderr ±1.87; multi-trial CI not fully transcribed.
- Vals labels "DeepSeek V4" (Pro line · max effort) — not Flash.
- mini-swe-agent is intentionally minimal; richer harnesses can score differently.
- Do not equate with vendor SWE Verified 80.6% (different harness/config).
Weighting
Evidence weight factors
The engine weights each evidence row by these factors (0–1) when fusing scores — higher is more trusted.
relevance0.95
quality0.85
recency1.00
health0.75
comparability0.90
reliability0.85
Benchmark
SWE-bench Verified
Human-filtered 500-instance subset of SWE-bench evaluating whether models can resolve real GitHub issues in existing repositories under a controlled agent harness.
Benchmark source →