← Claude Sonnet 5 · API · High effort (default)
Evidenceacceptedvendor_reported
SWE-bench Pro
Claude Sonnet 5 System Card (via Anthropic launch materials / secondary confirmations)
ev_2026_07_sonnet5_swe_pro
Result
63.2%
Confidence B
- Tested
- 2026-06-30
- Runs
- 1
Configuration
How it was measured
- Exact model
- claude-sonnet-5
- Reasoning
- high
- Agent harness
- anthropic_internal_swe_pro_eval
- Tool access
- agentic coding tools as configured by Anthropic eval
- Prompting
- vendor system-card protocol (not mini-SWE-agent public board)
SECONDARY STR signal. Prefer ev_2026_07_sonnet5_swe_vals (Vals mini-swe-agent independent) for comparative scoring. This row is vendor SWE-bench Pro.
Caveats
Known limitations
- Vendor-reported; not an independent mini-SWE-agent standardised board result.
- Exact trial count and public harness config not fully reproduced here.
- Maps strongly to existing_codebase_work but remains agent-scaffold-dependent.
Weighting
Evidence weight factors
The engine weights each evidence row by these factors (0–1) when fusing scores — higher is more trusted.
relevance0.95
quality0.55
recency1.00
health0.85
comparability0.75
reliability0.65
Benchmark
SWE-bench Pro
Harder SWE-bench variant with multi-file diffs from actively maintained repositories; used by vendors and labs as an agentic coding stress test.
Benchmark source →