Claude 3.5 Sonnet Benchmarks: June vs Oct 2024
Compare both 2024 Claude 3.5 Sonnet versions: SWE-bench Verified 33.4% (June) vs 49.0% (Oct), TAU-bench gains, launch pricing, and how to cite the scores.

Compare both 2024 Claude 3.5 Sonnet versions: SWE-bench Verified 33.4% (June) vs 49.0% (Oct), TAU-bench gains, launch pricing, and how to cite the scores.
Written and checked by the RankLLMs editorial team. See theeditorial policy.
Claude 3.5 Sonnet shipped in two versions during 2024, and the distinction explains why benchmark tables sometimes show different scores under the same model name. Anthropic announced the original model on June 21, 2024; its date-style snapshot identifier was 20240620. On October 22, 2024, Anthropic announced an upgraded version with stronger results on coding and tool-use evaluations.
This page focuses on the results Anthropic published for those versions. The scores are historical and belong to the named benchmarks and evaluation setup. They should not be compared directly with results from a different dataset, harness, or model snapshot.
Reported benchmark results
| Evaluation | Original version | October update | Source and scope |
|---|---|---|---|
| SWE-bench Verified | 33.4% | 49.0% | Anthropic’s coding evaluation report |
| TAU-bench Retail | 62.6% | 69.2% | Agentic tool-use evaluation reported by Anthropic |
| TAU-bench Airline | 36.0% | 46.0% | Agentic tool-use evaluation reported by Anthropic |
| OSWorld, screenshot-only | N/A | 14.9% | October announcement; computer-use evaluation |
| OSWorld, more steps allowed | N/A | 22.0% | October announcement; alternate step budget |
Anthropic also reported that the October version kept the same API price and speed as its predecessor. Its original launch announcement listed $3 per million input tokens, $15 per million output tokens, and a 200,000-token context window. These prices describe the model at launch and are not a statement about current Anthropic pricing for other models.
What SWE-bench Verified measures
SWE-bench asks a model or agent to resolve issues drawn from real open-source repositories. The Verified subset was human-filtered to improve task quality. It evaluates more than code completion: a system must interpret an issue, find relevant code, make a change, and pass the task’s tests under the evaluation harness.
The score can depend on the model snapshot, agent scaffold, available tools, prompt, number of attempts, and grading procedure. Anthropic’s own launch page describes its internal coding evaluation as a bug-fix or feature task in an open-source codebase using a natural-language request. A score from one agent harness should therefore be read as a result for that configuration, not as the model’s standalone probability of fixing any arbitrary issue.
Why the version label matters
The October update improved SWE-bench Verified by 15.6 percentage points (49.0 minus 33.4). Relative to 33.4%, that is about a 46.7% increase, but the point difference is the clearer comparison. Anthropic also reported increases on both TAU-bench domains. Since the vendor describes the update as an upgraded model, the two columns should not be treated as repeated trials of a single frozen checkpoint.
The computer-use results are a separate capability. In October 2024 Anthropic introduced computer use as a public beta and described it as experimental and error-prone; its release reported 14.9% on OSWorld’s screenshot-only category and 22.0% when more steps were allowed. Those figures are not coding scores and the two OSWorld values use different task budgets.
Limitations when using the results
These are vendor-published benchmarks from 2024, not a current ranking of coding models. Later models and evaluation suites use different tasks and procedures. Avoid combining SWE-bench Verified with SWE-bench Pro, DeepSWE, or Terminal-Bench as if they were a common scale. A higher percentage on one set does not by itself predict code quality, security, or review effort in your repository.
For current model selection, run the same issue set, agent framework, tool permissions, and retry policy on each candidate, then record resolved tasks and the work needed to verify them. See our SWE-bench Pro guide for a separate evaluation family and the AI Model Leaderboard for current tracked model data.
Takeaway
Anthropic reported a 33.4% to 49.0% increase on SWE-bench Verified from the original Claude 3.5 Sonnet version to the October 2024 update, alongside gains on two TAU-bench domains. The October computer-use announcement also reported OSWorld results under two different step limits. Keep those benchmark names, snapshot dates, and evaluation conditions attached to the scores when citing or comparing them.
Frequently Asked Questions
What did Claude 3.5 Sonnet score on SWE-bench Verified?
Anthropic reported 33.4% for the original June 2024 version and 49.0% for the October 2024 update. These are results for SWE-bench Verified and Anthropic's reported setup.
When did Claude 3.5 Sonnet launch?
Anthropic announced the original Claude 3.5 Sonnet on June 21, 2024. The first model snapshot was identified by the date-style version name 20240620.
What changed in the October 2024 version?
Anthropic reported that SWE-bench Verified improved from 33.4% to 49.0%, TAU-bench Retail from 62.6% to 69.2%, and TAU-bench Airline from 36.0% to 46.0%, at the same price and speed.
What was Claude 3.5 Sonnet's API price?
Anthropic's June 2024 announcement listed $3 per million input tokens and $15 per million output tokens, with a 200,000-token context window.
Source links for data and claims discussed in this article are listed below. Refer to each source for its current figures and methodology.
Recommended Reading
Related by topic and recency.
Best LLMs for Coding in 2026: How to Compare Them
Choose a coding LLM by matching the benchmark to your workflow, checking the agent harness, and measuring task quality and cost on your own repository.
Claude Sonnet 5: Permanent $2/$10 Pricing and Developer Impact
Anthropic made Claude Sonnet 5 API pricing permanent at $2 per million input and $10 per million output tokens. See a cost calculation with cache assumptions.
GLM-5.3-FlashX: 200 Tokens/sec Throughput vs 2.5x Price Analysis
Review GLM-5.3-FlashX provider speed claims, current API rates, task-cost math, and how to measure whether faster serving helps your coding workflow.
