← Gemini 3.1 Pro · API · High thinking
Evidenceacceptedthird_party_leaderboard
SWE-bench Verified
SWE-bench+ / openlm.ai — Gemini 3.1 Pro
ev_2026_07_gem31pro_swe_openlm
Result
79.8%
Confidence B
- Tested
- 2026-02-19
- Runs
- 1
Configuration
How it was measured
- Exact model
- gemini-3.1-pro-preview
- Reasoning
- high
- Agent harness
- mini-swe-agent / board default
- Tool access
- agent harness (board standard)
- Prompting
- board default agent protocol
PRIMARY STR — third-party SWE-bench Verified board (openlm). Vals difficulty table places Gemini 3.1 Pro Preview mid-pack (~79% weighted from bucket rates).
Caveats
Known limitations
- Prefer Vals overall float when published as single accuracy with stderr.
- Do not equate with vendor SWE Verified 80.6% (different harness).
- Preview model id; re-pin if GA string replaces preview.
Weighting
Evidence weight factors
The engine weights each evidence row by these factors (0–1) when fusing scores — higher is more trusted.
relevance0.95
quality0.80
recency0.85
health0.75
comparability0.80
reliability0.75
Benchmark
SWE-bench Verified
Human-filtered 500-instance subset of SWE-bench evaluating whether models can resolve real GitHub issues in existing repositories under a controlled agent harness.
Benchmark source →