ClaudeAnthropicBenchmarksSWE-benchAPI Pricing

Claude 3.5 Sonnet Benchmarks: June vs Oct 2024

Compare both 2024 Claude 3.5 Sonnet versions: SWE-bench Verified 33.4% (June) vs 49.0% (Oct), TAU-bench gains, launch pricing, and how to cite the scores.

Claude 3.5 Sonnet Benchmarks: June vs Oct 2024
Analyzed Foundation Models (3)Full LLM Leaderboard→
Claude Sonnet 4.6
Anthropic
49.8SCORE
Claude Sonnet 5
Anthropic
54.8SCORE
Claude Fable 5
Anthropic
62.0SCORE

Claude 3.5 Sonnet shipped in two versions during 2024, and the distinction explains why benchmark tables sometimes show different scores under the same model name. Anthropic announced the original model on June 21, 2024; its date-style snapshot identifier was 20240620. On October 22, 2024, Anthropic announced an upgraded version with stronger results on coding and tool-use evaluations.

This page focuses on the results Anthropic published for those versions. The scores are historical and belong to the named benchmarks and evaluation setup. They should not be compared directly with results from a different dataset, harness, or model snapshot.

Reported benchmark results

EvaluationOriginal versionOctober updateSource and scope
SWE-bench Verified33.4%49.0%Anthropic’s coding evaluation report
TAU-bench Retail62.6%69.2%Agentic tool-use evaluation reported by Anthropic
TAU-bench Airline36.0%46.0%Agentic tool-use evaluation reported by Anthropic
OSWorld, screenshot-onlyN/A14.9%October announcement; computer-use evaluation
OSWorld, more steps allowedN/A22.0%October announcement; alternate step budget

Anthropic also reported that the October version kept the same API price and speed as its predecessor. Its original launch announcement listed $3 per million input tokens, $15 per million output tokens, and a 200,000-token context window. These prices describe the model at launch and are not a statement about current Anthropic pricing for other models.

What SWE-bench Verified measures

SWE-bench asks a model or agent to resolve issues drawn from real open-source repositories. The Verified subset was human-filtered to improve task quality. It evaluates more than code completion: a system must interpret an issue, find relevant code, make a change, and pass the task’s tests under the evaluation harness.

The score can depend on the model snapshot, agent scaffold, available tools, prompt, number of attempts, and grading procedure. Anthropic’s own launch page describes its internal coding evaluation as a bug-fix or feature task in an open-source codebase using a natural-language request. A score from one agent harness should therefore be read as a result for that configuration, not as the model’s standalone probability of fixing any arbitrary issue.

Why the version label matters

The October update improved SWE-bench Verified by 15.6 percentage points (49.0 minus 33.4). Relative to 33.4%, that is about a 46.7% increase, but the point difference is the clearer comparison. Anthropic also reported increases on both TAU-bench domains. Since the vendor describes the update as an upgraded model, the two columns should not be treated as repeated trials of a single frozen checkpoint.

The computer-use results are a separate capability. In October 2024 Anthropic introduced computer use as a public beta and described it as experimental and error-prone; its release reported 14.9% on OSWorld’s screenshot-only category and 22.0% when more steps were allowed. Those figures are not coding scores and the two OSWorld values use different task budgets.

Limitations when using the results

These are vendor-published benchmarks from 2024, not a current ranking of coding models. Later models and evaluation suites use different tasks and procedures. Avoid combining SWE-bench Verified with SWE-bench Pro, DeepSWE, or Terminal-Bench as if they were a common scale. A higher percentage on one set does not by itself predict code quality, security, or review effort in your repository.

For current model selection, run the same issue set, agent framework, tool permissions, and retry policy on each candidate, then record resolved tasks and the work needed to verify them. See our SWE-bench Pro guide for a separate evaluation family and the AI Model Leaderboard for current tracked model data.

Takeaway

Anthropic reported a 33.4% to 49.0% increase on SWE-bench Verified from the original Claude 3.5 Sonnet version to the October 2024 update, alongside gains on two TAU-bench domains. The October computer-use announcement also reported OSWorld results under two different step limits. Keep those benchmark names, snapshot dates, and evaluation conditions attached to the scores when citing or comparing them.

Frequently Asked Questions

What did Claude 3.5 Sonnet score on SWE-bench Verified?

Anthropic reported 33.4% for the original June 2024 version and 49.0% for the October 2024 update. These are results for SWE-bench Verified and Anthropic's reported setup.

When did Claude 3.5 Sonnet launch?

Anthropic announced the original Claude 3.5 Sonnet on June 21, 2024. The first model snapshot was identified by the date-style version name 20240620.

What changed in the October 2024 version?

Anthropic reported that SWE-bench Verified improved from 33.4% to 49.0%, TAU-bench Retail from 62.6% to 69.2%, and TAU-bench Airline from 36.0% to 46.0%, at the same price and speed.

What was Claude 3.5 Sonnet's API price?

Anthropic's June 2024 announcement listed $3 per million input tokens and $15 per million output tokens, with a 200,000-token context window.

Sources & disclosures

Source links for data and claims discussed in this article are listed below. Refer to each source for its current figures and methodology.

Share Article

Recommended Reading

Related by topic and recency.

All articles→
GuidesCoding Agents

Best LLMs for Coding in 2026: How to Compare Them

Choose a coding LLM by matching the benchmark to your workflow, checking the agent harness, and measuring task quality and cost on your own repository.

Sep 4, 2026 · 3 min read
NewsClaude

Claude Sonnet 5: Permanent $2/$10 Pricing and Developer Impact

Anthropic made Claude Sonnet 5 API pricing permanent at $2 per million input and $10 per million output tokens. See a cost calculation with cache assumptions.

Aug 11, 2026 · 3 min read
ReviewsModels

GLM-5.3-FlashX: 200 Tokens/sec Throughput vs 2.5x Price Analysis

Review GLM-5.3-FlashX provider speed claims, current API rates, task-cost math, and how to measure whether faster serving helps your coding workflow.

Sep 20, 2026 · 5 min read