As I have explored in earlier analyses on single-file saturation and difficulty distribution, benchmark scores are only as trustworthy as the instances behind them. In this post, I look at a different failure mode: hidden naming contracts.

A hidden naming contract appears when benchmark tests require specific identifiers introduced in the reference solution, even though those names were never made explicit in the issue text. In that setting, an agent can produce a behaviorally correct fix and still be graded as wrong because it chose a different symbol name.

The Core Failure Mode

A typical SWE-bench-style instance has four parts:

• an issue description
• a base repository snapshot
• a reference solution
• executable tests

The intended contract is behavioral correctness: if a submitted patch fixes the issue, the tests should pass.

The problem starts when tests directly call symbols that were newly introduced in the reference solution. Evaluation then requires not only solving the behavior, but also reproducing a naming choice that may never have been stated anywhere.

A Concrete Example

Consider scikit-learn__scikit-learn-12682 from SWE-bench_Verified.

The issue reports that SparseCoder does not expose max_iter for Lasso, which leads to convergence warnings. The name transform_max_iter is not mentioned in the issue and does not exist elsewhere in the codebase at the base commit.

The reference patch introduces a new transform_max_iter parameter:

# sklearn/decomposition/dict_learning.py
class SparseCoder(BaseEstimator, SparseCodingMixin):
    def __init__(self, dictionary, transform_algorithm='omp',
                 transform_n_nonzero_coefs=None, transform_alpha=None,
                 split_sign=False, n_jobs=None, positive_code=False,
                 transform_max_iter=1000):
        self._set_sparse_coding_params(..., transform_max_iter)

The test patch then calls that exact parameter name:

def test_max_iter():
    with pytest.warns(ConvergenceWarning):
        model = SparseCoder(
            D_multi,
            transform_algorithm=transform_algorithm,
            transform_max_iter=1,
        )
        model.fit_transform(X)

    with pytest.warns(None) as record:
        model = SparseCoder(
            D_multi,
            transform_algorithm=transform_algorithm,
            transform_max_iter=2000,
        )
        model.fit_transform(X)

An agent could implement the same functionality with lasso_max_iter, max_transform_iterations, or another reasonable name and still fail evaluation. The behavioral fix is there, but the hidden naming contract is not satisfied.

What the Scan Found

I analyzed six SWE-bench-style datasets:

• SWE-bench/SWE-bench
• SWE-bench/SWE-bench_Verified
• SWE-bench/SWE-bench_Multilingual
• ScaleAI/SWE-bench_Pro
• ByteDance-Seed/Multi-SWE-bench
• AmazonScience/SWE-PolyBench

In a screening pass across all 7,567 instances, I found 2,167 instances (28.6%) where tests reference symbols newly introduced in the reference solution.

That 28.6% number is a screening signal, not a final estimate of confirmed false negatives. Some coupled symbols are fair because the name is explicit in the issue text or already established in the codebase. To isolate the highest-risk cases, I ran two refinement checks:

1. Is the symbol name mentioned in the issue text?
2. Does the symbol already exist elsewhere in the repository?

I treat the intersection as high-risk coupling: symbols that are neither mentioned in the issue nor present in the codebase.

High-Risk Coupling by Benchmark

The main takeaway is that this problem is concentrated rather than uniform.

• SWE-bench_Pro is the clear outlier at 10.9% high-risk coupling.
• Multi-SWE-bench is meaningfully elevated at 5.5%.
• SWE-bench_Verified and SWE-bench_Multilingual are much cleaner on this specific failure mode.

That pattern suggests curation helps, but it does not eliminate the issue.

Where the Risk Concentrates

The high-risk subset becomes easier to interpret if we also look at the two refinement signals separately:

Two things stand out.

First, many coupled instances provide no naming hint in the issue text. For most datasets, roughly 40% to 60% of coupled instances fall into that bucket.

Second, codebase priors help in many cases, but not equally across benchmarks. In SWE-bench_Verified, most coupled symbols already exist somewhere else in the repository, which gives agents a chance to infer the naming pattern through exploration. In SWE-bench_Pro and Multi-SWE-bench, that safety net is much weaker.

This is why the raw coupling rate and the high-risk rate both matter. The raw rate tells us how often tests are structurally tied to reference naming. The high-risk rate tells us where that coupling is most likely to generate false negatives.

Why This Matters for Leaderboards

1. Scores can move at the margin

In SWE-bench_Pro, high-risk coupling appears in 10.9% of all instances. If only a fraction of those behave as false negatives in practice, that is still enough to shift scores by one to several points, which can reorder closely clustered systems.

2. Cross-benchmark comparisons get noisier

A model may appear to improve or regress partly because one benchmark family embeds more hidden naming contracts than another. That makes benchmark-to-benchmark comparisons less clean than leaderboard tables suggest.

3. Curation helps, but one clean metric is not the whole story

The low rates in SWE-bench_Verified and SWE-bench_Multilingual are encouraging for this particular failure mode. But low hidden-contract rates should not be read as a blanket validation of a benchmark’s frontier-tracking quality. Benchmarks can still have other issues, including overly narrow tests, overly wide tests, or contamination.

Recent benchmark audits make the same broader point from a different direction. My analysis is narrower: it is a scalable programmatic scan for one subtype of narrow-test risk.

What Benchmark Maintainers Should Change

1. Prefer behavior-first assertions where feasible

Tests should verify the intended behavior, not accidentally require the exact reference implementation.

2. Make naming requirements explicit when they are part of the task

Some fixes genuinely require a specific API, method name, or parameter name for compatibility reasons. In those cases, the naming requirement should be written into the issue statement rather than hidden in the tests.

3. Publish diagnostic metadata alongside headline scores

Leaderboards should ideally report not just one aggregate score, but also benchmark diagnostics such as coupling prevalence, high-risk hidden-contract rates, and whether a score was computed on a filtered subset that excludes known problematic instances.

4. Keep a human review loop in benchmark maintenance

The cleanest benchmarks in this analysis are also the ones with stronger curation signals. That supports a practical maintenance rule: programmatic scans can pre-screen risky instances, but human review is still needed before release.

Conclusion

Software engineering allows multiple valid implementations, and evaluations should reflect that reality. A benchmark should reward behaviorally correct fixes, not force agents to rediscover one unstated naming choice from a hidden reference patch.

Hidden naming contracts are not the entire benchmark-reliability story, but they are a measurable and actionable part of it. If we want benchmark scores to be more robust, one straightforward step is to identify and remove instances where tests silently depend on names that the task never actually required.

A coding agent should not be graded as wrong for solving the right problem with the wrong unstated identifier.

Methodology

The analysis uses a simple pipeline:

1. Parse gold patches and extract symbols introduced on added lines.
2. Parse test patches and look for references to those symbols.
3. Filter out overly generic names that are likely to match coincidentally.
4. For coupled instances, check whether the symbol appears in the issue text.
5. For coupled instances, check whether the symbol already exists elsewhere in the repository at the base commit.

The symbol extraction step uses language-specific patterns for Python, Java, JavaScript or TypeScript, Go, Rust, and C or C++.

This post reports filtered counts throughout. The filtering step removes names that are so generic that a match is probably accidental rather than evidence of a real hidden naming contract.

The high-risk subset reported above is the intersection of the two refinement checks: symbols not mentioned in the issue text and not present in the codebase.

Appendix: High-Risk Instances by Benchmark

The following tables list the instances in the high-risk intersection.

SWE-bench/SWE-bench

SWE-bench/SWE-bench_Verified

ByteDance-Seed/Multi-SWE-bench

ScaleAI/SWE-bench_Pro

AmazonScience/SWE-PolyBench

Citation

If you find this analysis useful for your research, please cite it as:

@misc{ganhotra2026hiddencontracts,
  title={Hidden Naming Contracts in SWE-Agent Benchmarks},
  author={Ganhotra, Jatin},
  year={2026},
  month={April},
  url={https://jatinganhotra.dev/blog/swe-agents/2026/04/01/hidden-naming-contracts-in-swe-agent-benchmarks/},
  note={Blog post}
}

The Collapse of AI Coding Agents When Images Enter the Picture

AI coding agents now write real PRs and merge code at unprecedented scale, but they collapse when faced with visual content. In SWE-bench Multimodal, we observe a 73.2% performance drop when images are involved — a result that challenges the entire field’s current trajectory.

The Rise of SWE-Agents in Industry

We are witnessing unprecedented adoption of SWE-agents across the industry. This trend is clearly evident from the surge in pull requests being opened and merged by autonomous coding agents. PR Arena tracks the opened and merged PRs by top SWE coding agents - GitHub Copilot, OpenAI Codex, Devin, Cursor Agents, and Codegen. The numbers tell a compelling story of AI agents increasingly participating in real-world software development workflows.

Reality Check: A Tale of Two Benchmarks - The Performance Cliff

However, when we examine the benchmarks used to evaluate these SWE-agents, a different pattern emerges. Looking at the top performers on the SWE-bench Verified leaderboard, we see several submissions achieving ~70-75% on all instances. But when we examine the same models on SWE-bench Multimodal, performance severely drops to ~30-35%. This raises a natural question: Why do SWE agents using identical models excel on Verified but struggle significantly on the Multimodal dataset?

We know from our previous analysis that the SWE-bench Verified benchmark suffers from a saturation problem - while state-of-the-art SWE-agents achieve impressive overall scores, they struggle significantly on discriminative subsets that isolate truly challenging problems, e.g., problems that require complex reasoning across multiple files. This article investigates whether a similar pattern exists for the multimodal benchmark - and what factors drive the performance cliff we observe.

This discrepancy raises a critical question: Is this truly a benchmark problem, or does it reveal fundamental limitations in current AI systems?

Enter SWE-bench Multimodal: A New Perspective

To answer this question, we turn to SWE-bench Multimodal, a recently introduced benchmark that offers a fresh lens on AI agent capabilities. Unlike SWE-bench Verified, which focuses primarily on textual code understanding and generation, SWE-bench Multimodal targets JavaScript-based, user-facing applications, such as UI design systems, web app development, interactive mapping, and syntax highlighters.

This multimodal approach better reflects the reality of software engineering, where developers must interpret visual information alongside code. Issues on GitHub frequently include screenshots, error dialogs, and visual demonstrations of problems that need solving.

The dataset contains 510 instances in the test split. Our performance analysis uses this test split, where only 48 instances (9.4%) are purely text-based¹, while 462 instances (90.6%) include visual elements - making this a predominantly multimodal challenge.

The Visual Complexity Penalty

We uncover a steep 73.2% performance drop when images are introduced to coding tasks.

This dramatic drop highlights a critical limitation in multimodal AI for software engineering — and suggests that the gaps we saw in SWE-bench Verified are symptoms of deeper architectural challenges. Our analysis examines the performance of 11 SWE agents (from 2025) on the SWE-bench Multimodal leaderboard.²

Before diving into the detailed analysis, let’s establish the key differences between these benchmarks:

We categorize performance into three key areas:

1. Overall Performance - Complete benchmark results across all 510 instances
2. Text-Only Performance - Instances without visual elements (48 instances)
3. Images Present Performance - Instances containing visual elements (462 instances)

Our analysis reveals a consistent and striking pattern: visual content severely degrades performance across all models.

Overall Performance Comparison

The following table illustrates the stark contrast between text-only and image-containing performance across all evaluated agents. Notice how every single agent achieves near-perfect scores on text-only instances (97-100%) but experiences severe degradation when visual elements are present:

The Top-Performing Agents achieve near-perfect performance on text-only instances (97-100%) but experience sharp drops when visual elements are introduced - an average 73.2% performance drop when images are introduced (calculated as: 99.1% average text-only performance minus 25.9% average image-present performance = 73.2%).

Is Image Quantity the key factor?

The analysis above suggests that images are the primary factor, but is there more to the story? Let’s examine the relationship between the number of images and solve rates across different subsets.

Even though one might assume more images = more difficulty, the data doesn’t support this. Solve rates remain low regardless of whether there’s 1 image or 4.

The Multi-File Editing Connection?

While we cannot directly analyze patch complexity due to the absence of gold patches in SWE-bench Multimodal, the original paper provides crucial insights about the underlying task complexity:

In SWE-bench, 83% of task instances edit one file, and 65% edit one function. In SWE-bench M (Multimodal), just 40% of task instances change one file, and 32.5% change one function. On average, the changes by reference solutions in SWE-bench M are larger than those in SWE-bench, with multi-file edits being more commonplace.

This finding aligns with our previous analyses on patch complexity and multi-file patterns and single-file saturation effects, which showed that state-of-the-art SWE agents struggle significantly with multi-file editing tasks.

The 73.2% performance drop reveals intersecting difficulties: it’s not just visual complexity, but the compound challenge of multimodal reasoning combined with multi-file editing tasks that are prevalent in image-containing instances.

Conclusion

Our analysis of SWE-bench Multimodal provides a definitive answer to the central question we posed: the performance gaps we observe in discriminative subsets aren’t just benchmark artifacts - they reflect genuine limitations in current AI architectures when handling complex, real-world software engineering tasks.

The 73.2% performance drop reveals a double burden that extends far beyond simple visual reasoning difficulties. Our analysis uncovered that image-containing instances systematically involve more complex multi-file editing tasks (only 40% single-file in SWE-Bench Multimodal vs. 83% in traditional SWE-bench), creating a perfect storm of difficulties that current AI systems struggle to handle.

This finding directly connects to our previous research on discriminative subsets, where we demonstrated that state-of-the-art agents struggle with multi-file reasoning even in purely text-based scenarios.

The consistency of this penalty across all top-performing models - from GUIRepair variants to OpenHands-Versa to Agentless Lite - suggests that the challenge runs deeper than individual model architectures.

These agents achieve near-perfect performance (97-100%) on text-only instances but uniformly collapse to 18-30% when visual elements and multi-file complexity are introduced. This pattern indicates fundamental gaps in how current AI systems integrate visual and textual information in technical contexts, particularly when structural complexity increases.

Key Takeaways

Our findings point to three critical conclusions:

• Current AI models collapse under multimodal + multi-file complexity - The 73.2% performance drop demonstrates that existing architectures cannot handle the intersecting difficulties of visual reasoning and structural code analysis.

• This impacts the real-world reliability of SWE agents - Many GitHub issues include screenshots and visual context, and if these follow patterns similar to SWE-bench Multimodal, current deployments may face significant limitations in complex scenarios.

• Fundamentally new multimodal reasoning architectures are needed - The path forward requires not just better visual understanding, but entirely new approaches to managing complexity when visual reasoning intersects with multi-file code modifications - the hallmark of challenging software engineering work.

Citation

If you find this analysis useful for your research, please cite it as:

@misc{ganhotra2025visual,
  title={The Visual Complexity Penalty in Code Understanding - SWE-bench Multimodal Analysis},
  author={Ganhotra, Jatin},
  year={2025},
  month={July},
  url={https://jatinganhotra.dev/blog/swe-agents/2025/07/26/swe-bench-multimodal-visual-complexity/},
  note={Blog post}
}

Since my last analysis of SWE-Bench Verified on April 15, there has been significant progress on the leaderboard. The best performing SWE-agents are now at 73.20% (Tools + Claude 4 Opus) whereas only ~45 days ago, the best performing system was Augment Agent v0 at 65.40% - representing a remarkable 7.8 percentage point improvement in less than two months.

The Saturation Problem Revisited

In my previous analyses, I identified that Easy problems are effectively saturated, with top SWE-agents achieving 84-86% success rates. My single-file saturation study further revealed structural limitations in current evaluation approaches.

This saturation creates a differentiation problem: when most competitive SWE-agents solve 160+ of the same 194 Easy problems, these categories no longer provide meaningful signal for distinguishing between top-tier systems. The real competition has shifted to unsolved and sparsely-solved problems across all difficulty categories.

A Data-Driven Solution: Discriminative Subsets

Rather than arbitrarily choosing “hard” problems, I developed a systematic methodology by analyzing how many SWE-agents solve each instance across all 500 problems in SWE-Bench Verified.

Instance Solve Distribution Analysis

Each of the 500 instances was checked against evaluation results from 83 distinct SWE-agents (submitted between October 2023–May 2025) to record solve counts. “Solved” means the agent’s fix passed the verification test suite - the same standard used in the original SWE-bench evaluation.

I categorized all instances based on how many of the 83 evaluated SWE-agents successfully solve them:

Key insight:

• 69% of problems are solved by 21+ agents, resulting in limited discrimination during evaluation
• The competitive frontier exists in the first five categories (155 instances total), solved by ≤20 agents (High discrimination potential)
• 52 completely unsolved problems (Maximum discrimination)

Four Discriminative Subsets

Rather than continuing to measure incremental improvements on largely-solved Easy problems, I designed targeted subsets to focus on:

1. Completely unsolved problems (52 instances) - true frontier challenges
2. Sparsely solved problems - instances resolved by only a handful of agents
3. Problems with high solution variance - where top SWE-agents show meaningful differences

This approach yields a more discriminative and high-resolution instrument for measuring real-world SWE-agent capability differences, similar to how other AI benchmarks have evolved when existing evaluations became saturated.

Based on this analysis, I created four targeted evaluation subsets. The “Solve Range” column shows how many agents successfully solve the problems within each subset - for example, Frontier subset problems are solved by 0-5 agents, making them the most evaluatively sensitive.

Subset Relationships:

• Frontier ⊆ Challenging (all Frontier problems are included in Challenging)
• Hard and MultiFile subsets partially overlap with both Frontier and Challenging subsets
• Single-file problems involve changes to one source file, while multi-file problems require coordinated modifications across multiple files.

1. Frontier Subset (95 instances)

Problems solved by ≤5 agents - maximum evaluative sensitivity

This subset combines completely unsolved problems with ultra-rare and very-rare solves. It’s composed of 14 Easy, 52 Medium, and 29 Hard problems. Notably, even the top-performing Claude 4 Opus scores just 11.6% on this subset, compared to its 73.2% on the full benchmark. This provides extraordinary differentiation power between cutting-edge systems.

• Composition: 58 single-file, 37 multi-file problems
• Top Performance: Claude 4 Opus 11.6% (vs 73.2% on full benchmark)
• Purpose: Maximum resolution for cutting-edge agent comparison

Top-10 performing SWE-agents on the Frontier subset:

2. Challenging Subset (155 instances)

Problems solved by ≤20 agents - strong evaluative power

Expanding to include rare and uncommon problems, this subset maintains robust differentiating ability while providing more instances for statistical significance. Claude 4 Opus reaches 31.6% here - still providing much better separation than the 84%+ scores on Easy problems.

• Composition: 28 Easy, 93 Medium, 34 Hard; 105 single-file, 50 multi-file
• Top Performance: Claude 4 Opus 31.6%
• Purpose: Balance of resolution and statistical significance

Top-10 performing SWE-agents on the Challenging subset:

3. Hard (45 instances)

All Hard difficulty problems regardless of solve rate

This subset focuses specifically on the 45 Hard problems, with Claude 4 Opus achieving 42.2%. Interestingly, this includes some problems solved by many agents, showing that difficulty level and solve count don’t perfectly correlate.

• Composition: 0 Easy, 0 Medium, 45 Hard; 20 single-file, 25 multi-file
• Top Performance: Claude 4 Opus 42.2%
• Purpose: Focused evaluation on most difficult problem category

Top-10 performing SWE-agents on the Hard subset:

4. MultiFile (40 instances)

Multi-file problems solved by ≤10 agents

This targets the intersection of multi-file problems (which tend to be harder) with low solve counts. These problems require coordinated edits across multiple source files, making them more complex for current SWE-agents. It’s the most challenging subset - even Claude 4 Sonnet only achieves 10.0%. The composition (3 Easy, 17 Medium, 20 Hard) confirms that multi-file problems are inherently more difficult.

• Composition: 3 Easy, 17 Medium, 20 Hard; 0 single-file, 40 multi-file
• Top Performance: Claude 4 Sonnet 10.0%
• Purpose: Target intersection of multi-file complexity and low solve rates

Top-10 performing SWE-agents on the MultiFile subset:

Summary Comparison

Key Insights and Patterns

1. Medium Problems Drive Frontier Resolution

The Frontier subset contains 52 Medium vs 29 Hard problems, revealing that traditional difficulty categories don’t capture all sources of complexity. Some Medium problems remain more challenging than many Hard problems. This suggests that ‘Medium’ in SWE-Bench often reflects problem scope or context rather than true agent difficulty.

2. Multi-file Problems Are Genuinely Harder

Multi-file problems dominate the low-solve buckets:

• 40/40 MultiFile problems are multi-file (by design)
• 37/95 Frontier problems are multi-file (39%)
• Only 2/156 “Solved” problems are multi-file (1.3%)

3. Massive Performance Differentiation Gains

The performance gaps between full benchmark and targeted subsets are dramatic. While the top two SWE-agents (Tools + Claude 4 Opus and Tools + Claude 4 Sonnet) are separated by less than 1 percentage point on the entire 500-instance SWE-Bench Verified benchmark (73.2% vs 72.4%), the specialized subsets create substantial separation between these same systems. These clear performance gaps demonstrate the power of focusing evaluation on truly challenging problems.

While the specialized subsets have smaller sample sizes, the large performance differences (typically 35-65 percentage point drops from full benchmark performance) clearly indicate meaningful capability distinctions.

Implications for the Research Community

Immediate Benefits

1. Enhanced Signal: Researchers can immediately use these subsets for more sensitive evaluation
2. Research Focus: Identifying specific problem types guides targeted improvement efforts
3. Facilitates curriculum design: For agent fine-tuning based on real-world bottlenecks
4. Clearer Progress: Performance improvements on frontier problems represent genuine capability advances

Methodological Innovation

This solve-distribution approach could be applied to other saturated benchmarks:

1. Analyze how many systems solve each problem
2. Identify low-solve instances for targeted subsets
3. Create sensitive evaluation instruments
4. Maintain evaluative power as capabilities advance

Future Evolution

As SWE-agents continue improving, this methodology enables dynamic subset creation:

• Current “Ultra Rare” problems may become “Common”
• New targeted subsets can be generated using the same framework
• Evaluation maintains sensitivity to genuine progress

Technical Implementation Notes

Data Availability

All subset definitions and solve matrices are available as structured JSON data, enabling:

• Immediate subset evaluation using existing SWE-Bench infrastructure
• Integration with current evaluation pipelines
• Reproducible research and fair comparisons

Evaluation Guidelines

For consistent evaluation across the research community:

1. Report both full benchmark and subset performance
2. Use Frontier subset for cutting-edge system comparison
3. Use Challenging subset for statistical significance
4. Use specialized subsets (Hard, MultiFile) for targeted research

Try the Discriminative Subsets

The targeted subsets are available on HuggingFace for immediate use:

Dataset: jatinganhotra/SWE-bench_Verified-discriminative

Four splits: frontier, challenging, hard, multifile

from datasets import load_dataset

# Load all splits
dataset = load_dataset("jatinganhotra/SWE-bench_Verified-discriminative")

# Load specific split for targeted evaluation
frontier = load_dataset("jatinganhotra/SWE-bench_Verified-discriminative", split="frontier")

I encourage researchers to benchmark their SWE-agents on these discriminative subsets and share results publicly. Tracking progress here, not just on saturated sets, will best reflect true capability gains in autonomous software engineering rather than incremental improvements on saturated problem sets.

Conclusion

This analysis reveals that SWE-Bench Verified’s evaluative power has become concentrated in a subset of challenging problems. Systematically identifying these problems through solve-distribution analysis enables high-resolution benchmarks like the Frontier Subset, which reveal capability distinctions that broad benchmarks obscure.

The Challenging Subset balances sensitivity with statistical power, while the Hard and MultiFile subsets offer targeted evaluation for specific research directions.

Most importantly, this methodology - analyzing solve distribution to identify evaluatively sensitive problems - provides a data-driven framework for benchmark evolution. As AI capabilities continue advancing, this approach ensures evaluation maintains sensitivity to genuine progress rather than incremental improvements on saturated problem sets.

The targeted subsets are immediately usable with existing SWE-Bench infrastructure, enabling the research community to adopt more sensitive evaluation practices while pushing toward the true frontiers of automated software engineering.

Citation

If you find this analysis useful for your research, please cite it as:

@misc{ganhotra2025discriminative,
  title={From 73% to 11%: Revealing True SWE-Agent Capabilities with Discriminative Subsets},
  author={Ganhotra, Jatin},
  year={2025},
  month={June},
  url={https://jatinganhotra.dev/blog/swe-agents/2025/06/05/swe-bench-verified-discriminative-subsets/},
  note={Blog post}
}

How Human Experts Judge Task Difficulty

OpenAI asked human annotators to estimate completion times for each task, assuming the engineer had “a few hours to familiarize themselves with the codebase.” They used four time-based categories:

• < 15 minutes: Trivial changes like adding assertions to a function
• 15 minutes – 1 hour: Small changes requiring some thought
• 1 – 4 hours: Substantial rewrites affecting functions or multiple files
• > 4 hours: Esoteric issues requiring significant research and changing 100+ lines of code

While these estimates aren’t used for dataset filtering, they provide valuable insight into the perceived difficulty distribution.

Breaking Down SWE-Bench-Verified by Difficulty

When we analyze the 500 issues in SWE-Bench-Verified using these time-based difficulty metrics, the distribution is revealing:

Key Insight: The vast majority (91%) of issues are estimated to take less than an hour for a human expert to solve, with only a tiny fraction (0.60%) requiring more than 4 hours.

Standardizing Difficulty Terminology

More recently, the “Multi-SWE-bench” paper (Zhu et al., 2024) simplified these four categories into three difficulty levels:

• Easy: ≤ 15 minutes (194 issues, 38.80%)
• Medium: 15 minutes – 1 hour (261 issues, 52.20%)
• Hard: ≥ 1 hour (45 issues, 9.00%)

This classification provides a clearer framework for evaluating model performance across difficulty levels.

Quantifying Difficulty: Beyond Time Estimates

To gain deeper insights into what makes tasks difficult, we can examine objective metrics across difficulty levels:

Key Observations

• Scaling Relationship: All metrics (files, hunks, lines) increase with difficulty level, but at different rates.
• Lines Changed: Shows the most dramatic increase from Easy to Hard (11x increase), highlighting that hard patches involve significantly more code changes.
• Files Modified: Shows a more modest increase (2x from Easy to Hard), suggesting that difficulty isn’t just about the number of files.
• Hunks: Increases 5x from Easy to Hard, indicating more separate code blocks need modification in harder tasks.

The Complexity of Single vs. Multi-File Issues

When we combine difficulty levels with file count, we see striking patterns:

Critical Observation: As difficulty increases, the proportion of multi-file issues rises dramatically—from just 3.09% of easy issues to 55.56% of hard issues. This suggests multi-file complexity as a significant factor in what makes programming challenges difficult.

Looking deeper at the metrics for single vs. multi-file tasks:

This table reveals that multi-file tasks require, on average:

• Nearly 4x as many code hunks (separate code blocks)
• 4x as many lines of code changed
• More complex edits across multiple files

These metrics substantiate what the performance data shows: multi-file tasks represent a significant complexity jump.

Performance Across the Spectrum

Examining representative systems across the performance spectrum reveals consistent patterns:

1. The Easy Category Is Largely Solved
   • Combined resolution rate: 95.36%
   • Even top individual systems solve ~80% of easy tasks
   • The remaining gap is closing with each new LLM release
2. The Medium Category Shows Progress
   • Combined resolution rate: 84.29%
   • Top systems solve 56-62% individually
   • Significant improvement from earlier systems (<30%)
3. The Hard Category Remains Challenging
   • Combined resolution rate: only 42.22%
   • Best individual systems solve just 20-25%
   • Multi-file hard issues are particularly difficult (only 9/25 solved by any system)

Key Takeaways

This analysis reveals several important insights:

1. The Easy-Hard Gap Persists: Even top systems show a dramatic performance drop from easy (80%+) to hard tasks (20-25%).

2. Multi-File Issues Present a Frontier: The correlation between task difficulty and multi-file complexity is striking. As complexity increases along all metrics (files, hunks, lines), performance drops precipitously.

3. Lines of Code as a Key Indicator: The 11x increase in average lines changed from Easy to Hard tasks (5.04 → 55.78) appears to be the strongest predictor of task difficulty, far outpacing the increase in file count (2x) or hunks (5x).

4. Combined Performance Ceiling: The gap between individual and combined system performance suggests that different approaches excel at different tasks — no single system can yet solve all problem types.

5. Hard Multi-File Issues Remain Unsolved: With only 9/25 hard multi-file issues solved by any system, this represents a clear frontier for improvement.

6. Scale of Changes Matters: Multi-file tasks require 4x more lines of code and 4x more hunks than single-file tasks, highlighting that the scope of required changes significantly impacts task difficulty.

Relating to the Reality Gap

As I argued in my December 2024 post, the distribution in SWE-Bench-Verified significantly underrepresents the complexity of real-world programming tasks. Comparing datasets reveals this stark contrast:

This discrepancy is significant and highlights a critical area for improvement in benchmark design. If we use file count as a complexity proxy, SWE-Bench-Verified presents a dramatically simplified view compared to real-world codebases, where approximately half of all issues require multi-file changes.

The data clearly shows that as we move from simple, localized fixes (Easy) to complex, multi-file, multi-hunk patches (Hard), AI performance drops dramatically. Future research should focus on improving coordination across multiple files and handling larger, more complex code changes that span multiple distinct code blocks.

This finding reinforces my previous analysis in The Multi-File Frontier, where I emphasized that truly robust AI programming systems must be capable of coordinating changes across multiple files in an interconnected codebase.

Conclusion

While impressive progress has been made in solving isolated, single-file coding challenges, the frontier of multi-file, complex issues remains largely unexplored. For AI programming systems to truly match human capabilities in real-world software engineering, they must evolve beyond generating localized patches to understand the rich, interconnected nature of modern codebases.

Until benchmarks like SWE-Bench-Verified more accurately reflect the distribution of tasks in real-world development—particularly the proportion of multi-file changes—we should interpret leaderboard results with appropriate caution, recognizing they represent an optimistic view of AI’s current programming capabilities.

Citation

If you find this analysis useful for your research, please cite it as:

@misc{ganhotra2025difficulty,
  title={Cracking the Code: How Difficult Are SWE-Bench-Verified Tasks Really?},
  author={Ganhotra, Jatin},
  year={2025},
  month={April},
  url={https://jatinganhotra.dev/blog/swe-agents/2025/04/15/swe-bench-verified-easy-medium-hard/},
  note={Blog post}
}

Understanding the Dataset Distribution

SWE-Bench-Verified’s 500 instances fall into two distinct categories:

• Single-file changes: 429 instances (85.8%)
• Multiple-file changes: 71 instances (14.2%)

This distribution reveals a critical insight: the performance of top systems deteriorates significantly when tackling tasks requiring changes across multiple files—exposing a fundamental limitation in current AI programming approaches.

The Leap Forward in 2025

Since January 2025, we’ve seen substantial progress. The previous leader, Amazon Q Developer Agent, has been surpassed by twelve new systems, with “Augment Agent v0” now claiming the top position.

Performance Across the Spectrum

Below is a representative sample of systems across the performance spectrum, highlighting the consistent gap between single-file and multi-file performance:

Note: The full table includes 64 systems. This shortened version highlights representative systems across the performance spectrum.

The current leader has pushed performance boundaries significantly:

• Overall resolution: Improved from 55.0% to 65.4%
• Single-file resolution: Increased from 60.61% to 71.56%
• Multi-file resolution: Advanced from 21.13% to 28.17%

While these gains are impressive, they’re predominantly in simpler, single-file scenarios. Even the best systems struggle with multi-file issues, with no system exceeding 30% resolution rate for these more complex problems.

A Collective View: The Upper Bound

When we combine the capabilities of all top systems, an interesting picture emerges:

• Single-file issues: ~90% resolution rate (386/429)
• Multi-file issues: Only ~54% resolution rate (38/71)

This reveals that:

1. Single-file challenges are approaching saturation—the collective intelligence of these systems resolves nearly all such issues.

2. Multi-file problems remain a frontier challenge—even with all systems combined, nearly half remain unsolved.

The Reality Gap: Benchmark vs. Real-World

As I argued in my December 2024 post titled “SWE-Bench Verified ⊊ real-world SWE tasks,” the distribution in SWE-Bench-Verified significantly underrepresents the complexity of real-world programming tasks. Comparing datasets reveals this stark contrast:

This discrepancy is significant and highlights a critical area for improvement in benchmark design. If we use file count as a complexity proxy, SWE-Bench-Verified presents a dramatically simplified view compared to enterprise codebases, where approximately half of all issues require multi-file changes.

Expert Voices Agree on the Benchmark’s Limitations

This concern isn’t just mine. In March 2024, AI expert Andrej Karpathy tweeted about the evaluation crisis in AI, specifically highlighting SWE-Bench Verified’s limitations:

“My reaction is that there is an evaluation crisis. I don’t really know what metrics to look at right now… SWE-Bench Verified (real, practical, verified problems) I really like and is great but itself too narrow.”

Karpathy’s assessment aligns with my analysis—while SWE-Bench Verified offers valuable insights through practical, verified problems, its narrow scope fails to capture the true complexity of real-world software engineering tasks, particularly those requiring multi-file changes.

The Multi-File Challenge: A Different Cognitive Task

Solving multi-file issues requires sophisticated capabilities that go beyond what current AI systems excel at:

• Cross-file dependency tracking
• Contextual understanding across modules
• Architectural comprehension
• Interface consistency management
• Impact analysis across the codebase

Current systems excel as “patch generators” for localized changes but struggle to function as holistic developers who understand the broader implications of their modifications.

Conclusion

While we’ve made impressive strides in addressing isolated coding challenges, our progress with interconnected, complex issues remains limited. The current SWE-Bench-Verified benchmark understates the complexity of real-world software engineering by presenting a distribution of challenges that skews heavily toward single-file edits—only 14.2% of issues requiring multi-file changes compared to 50.27% in real-world scenarios.

For meaningful advancement in AI-assisted programming, future versions of SWE-Bench-Verified should rebalance to reflect realistic multi-file ratios, providing a more accurate measure of practical capability. As I argued in my January 2025 post, the current impressive performance numbers on the leaderboard must be interpreted with this significant caveat in mind.

We should recognize that SWE-Bench-Verified, while valuable, presents an overly optimistic view of AI’s programming capabilities in real-world scenarios.

Citation

If you find this analysis useful for your research, please cite it as:

@misc{ganhotra2025saturation,
  title={The Multi-File Frontier: Why SWE-Bench Verified Doesn't Reflect Real-World Programming Challenges},
  author={Ganhotra, Jatin},
  year={2025},
  month={March},
  url={https://jatinganhotra.dev/blog/swe-agents/2025/03/30/swe-bench-verified-single-file-saturation/},
  note={Blog post}
}

In a previous post, I discussed why SWE-Bench Verified may not accurately represent real-world SWE tasks, primarily due to the low proportion of issues requiring code changes across multiple files. If you haven’t read that post yet, I recommend doing so before continuing with this one.

As highlighted in the previous post, the SWE-Bench_Verified dataset, consisting of 500 instances, can be categorized into two distinct buckets:

• Single-file changes: 429/500 instances (85.8%)
• Multiple-file changes: 71/500 instances (14.2%)

Furthermore, it is evident that the performance of the top-10 systems drops significantly for tasks requiring changes across multiple files, underscoring a critical area for improvement.

In this post, we’ll take a closer look at the performance of the top-10 systems on the multiple files subset of SWE-Bench Verified from a fresh perspective:

1. How does the approach of our current state-of-the-art SWE-Agents compare to that of a human software engineer (SWE) when solving these issues?
2. Do our SWE-Agents appropriately modify multiple files to resolve the issue, given that these instances inherently require changes across multiple files (using the human-generated patch as the golden standard)?

This is the question we aim to explore today! 🚀

Let’s start by analyzing how many of the 71 multiple-file instances were resolved by at least one of the top-10 systems. Based on the compiled statistics, we get:

• Multiple-file Instances (requiring changes across multiple files): 71/500
• Multiple-file Instances resolved by at least one system: 29/71
• Multiple-file Instances resolved by at least one system with single-file changes: 20/29

This observation is quite surprising. There are 20 instances that:

1. Required changes across multiple files, as indicated by the human-generated patch.
2. Were marked as resolved (i.e., passed both PASS_TO_PASS and FAIL_TO_PASS tests in the evaluation) by at least one of the top-10 systems.
3. Were resolved by modifying only one file, contradicting the multi-file nature of the issue.

This discrepancy raises important questions about how the systems achieve these resolutions and the robustness of the evaluation process. 😳

Disclaimer

If you are working on the SWE-Bench benchmark, please refrain from attempting to reverse engineer the anonymized instance IDs using the information provided about the gold patch files. Additionally, do not access the original pull requests on GitHub or the corresponding gold patches in the dataset. This ensures that hidden tests remain protected, maintaining ethical standards.

To achieve this:

• Instance IDs are purposefully anonymized.
• The tabular analysis focuses only on listing the file names in the model-generated patches to identify any patterns, while keeping both the instance IDs and the patches hidden.

This approach safeguards the integrity of the benchmark while facilitating responsible and ethical analysis.

Analysis

Let’s take a closer look at these 20 instances. For each instance, the following information is provided:

1. Instance ID (anonymized).
2. Number of systems (out of the top-10) that resolved the instance: n/10.
3. Number of files K in the gold patch, further divided into:
   • k1 ⊆ K: Files modified by all n/10 systems.
   • k2 ⊆ K: Files modified by some systems (a subset of n/10).
   • k3 ⊆ K: Files not modified by any systems.

By clicking on the anonymized Instance ID, you can view additional details, such as:

• The file names in the gold patch (color-encoded):
  • Green: Files modified by all systems.
  • Orange: Files modified by some systems.
  • Red: Files not modified by any systems.
• The file names in the model-generated patches for each of the n/10 systems that resolved the instance.

Based on the additional details, the 20 instances can be categorized as follows:

Category 1: No file from the gold patch was modified

In these instances, the model-generated patches resolved the issue by modifying files that were not part of the gold patch, while none of the files in the gold patch were altered.

Note: Click on the anonymized Instance ID to view additional details

Category 2: At least 1 file from the gold patch was not modified

In these instances, the model-generated patches resolved the issue, but at least one file from the gold patch was left unmodified by the systems.

Note: Click on the anonymized Instance ID to view additional details

The above analysis reveals notable discrepancies between the approaches of human software engineers (SWEs) and model-based SWE-Agents when resolving complex multi-file issues.

Key Takeaways from the Analysis

1. Differences in Problem-Solving Approaches:
   SWE-Agents handle multi-file issues differently compared to human SWEs, often focusing on resolving issues without strictly adhering to the gold standard of modifying the correct files.
   

   NOTE: It is plausible that the model-generated patches from SWE-Agents might, in some cases, surpass those created by human SWEs in terms of quality or efficiency.

   • However, this hypothesis remains to be validated through a thorough review by a Subject Matter Expert (such as the GitHub repository maintainer) for the specific library or codebase.
   • Such an expert review could provide critical insights into the suitability and maintainability of the model-generated patches, helping to assess their long-term value compared to human contributions.

2. Benchmark Limitations:
   The benchmark does not enforce requirements for SWE-Agents to make changes in the appropriate locations, which is crucial for ensuring maintainable and robust code over the long term.

Implications

If these model-generated patches were submitted as pull requests, many would likely need significant adjustments to modify the appropriate files. This step is essential to align with real-world software engineering practices and to produce code that is easier to maintain and extend.

Conclusion

The analysis highlights a gap between achieving benchmark success and adhering to real-world software engineering best practices, underscoring the need for benchmarks that better reflect practical requirements and long-term code maintainability.

Citation

If you find this analysis useful for your research, please cite it as:

@misc{ganhotra2025multifile,
  title={Do SWE-Agents Solve Multi-File Issues Like Humans? A Deep Dive into SWE-Bench Verified},
  author={Ganhotra, Jatin},
  year={2025},
  month={January},
  url={https://jatinganhotra.dev/blog/swe-agents/2025/01/05/swe-bench-mutliple-files/},
  note={Blog post}
}

The SWE-Bench leaderboard Rank 1: Amazon Q Developer Agent (v20241202-dev) submission README says ‘Our team is currently working on publishing details of the Q Dev Agent algorithm.’ and Rank 2: devlo submission README does not share any details.

Rank 3: OpenHands + CodeAct v2.1 (claude-3.5-sonnet-20241022) explicitly highlights its use of the Claude-3.5 Sonnet model in its submission entry and their agent is open-source and available on Github.

It could be an interesting exercise to compare this Rank 3 submission against Anthropic’s official entry, Tools + Claude 3.5 Sonnet (2024-10-22), which currently holds Rank 9 on the SWE-Bench_Verified leaderboard as of December 28, 2024.

NOTE: For our analysis, we will take a cursory approach, focusing on identifying overarching patterns rather than examining individual instances in detail.

There is a 6% increase in peformance by the OpenHands system, in comparison to the Anthropic’s official submission. The Anthropic’s submission post Raising the bar on SWE-bench Verified with Claude 3.5 Sonnet says:

The performance of an agent on SWE-bench can vary significantly based on this scaffolding, even when using the same underlying AI model.

The above statement provides an excellent foundation for this exercise.

Let’s begin.

Total instances: 500
OpenHands resolved instances: 265
Tools Sonnet resolved instances: 245
# is there overlap?
Number of Instances resolved by both systems: 205
Number of Instances resolved by OpenHands only: 60
Number of Instances resolved by Tools Sonnet only: 40

Both systems get 205 instances resolved, but there are 60 instances only resolved by OpenHands and 40 instances only resolved by Tools + Claude 3.5 Sonnet. This is interesting.

Now, take a deeper dive into the details, leveraging the official posts from both submissions as well as the logs and trajectories accessible for further analysis.

The OpenHands + CodeAct v2.1 (claude-3-5-sonnet-20241022) submission README says:

This submission is made with OpenHands CodeAct v2.1 at commit a86275e288087bc7833bc79b0b691ed2ec223205
of PR #4537 using `anthropic/claude-3-5-sonnet-20241022.

and the official post OpenHands CodeAct 2.1: An Open, State-of-the-Art Software Development Agent outlines the changes:

for CodeAct 2.1, we made a number of further improvements:

1. Switched to use function calling, a method used by language models to more precisely specify the functions available to them
2. We switched to Anthropic's new claude-3.5 model, released in October
3. Made a number of fixes to make it easier for agents to traverse directories

Let’s review the prompt for both systems first and then review the tools used by each system.

Prompt

Tools + Claude 3.5 Sonnet (2024-10-22)

The Anthropic Tool Using Agent prompt is copied below from the official post. It outlines a suggested approach for the model, but it’s not overly long or too detailed for this task.

<uploaded_files>
{location}
</uploaded_files>
I've uploaded a python code repository in the directory {location} (not in /tmp/inputs). Consider the following PR description:

<pr_description>
{pr_description}
</pr_description>

Can you help me implement the necessary changes to the repository so that the requirements specified in the <pr_description> are met?
I've already taken care of all changes to any of the test files described in the <pr_description>. This means you DON'T have to modify the testing logic or any of the tests in any way!

Your task is to make the minimal changes to non-tests files in the {location} directory to ensure the <pr_description> is satisfied.

Follow these steps to resolve the issue:
1. As a first step, it might be a good idea to explore the repo to familiarize yourself with its structure.
2. Create a script to reproduce the error and execute it with `python <filename.py>` using the BashTool, to confirm the error
3. Edit the sourcecode of the repo to resolve the issue
4. Rerun your reproduce script and confirm that the error is fixed!
5. Think about edgecases and make sure your fix handles them as well

Your thinking should be thorough and so it's fine if it's very long.

OpenHands + CodeAct v2.1 (claude-3-5-sonnet-20241022)

The OpenHands agent has an additional system message:

You are a helpful assistant that can interact with a computer to solve tasks.
<IMPORTANT>
* If user provides a path, you should NOT assume it's relative to the current working directory. Instead, you should explore the file system to find the file before working on it.
</IMPORTANT>

However, OpenHands retains the Anthropic’s prompt unchanged.

Instruction based on Anthropic’s official trajectory
https://github.com/eschluntz/swe-bench-experiments/tree/main/evaluation/verified/20241022_tools_claude-3-5-sonnet-updated/trajs
– from evaluation/swe_bench/run_infer.py (commit ae13171)

Below is a diff illustrating the differences between the prompts:

--- tools_sonnet_prompt.txt	2024-12-28 13:33:44
+++ openhands_prompt.txt	2024-12-28 13:39:38
@@ -1,7 +1,7 @@
 <uploaded_files>
-/repo
+/workspace/astropy__astropy__1.3
 </uploaded_files>
-I've uploaded a python code repository in the directory /repo (not in /tmp/inputs). Consider the following PR description:
+I've uploaded a python code repository in the directory astropy__astropy__1.3. Consider the following PR description:

 <pr_description>
 {pr_description}

We can make the following observations:

• Upon exploring the log files from the Anthropic submission, we can see that the {location} is always set to /repo for Anthropic’s Tool Using Agent. For OpenHands CodeAct 2.1, the {location} is set to the directory where the git repo was cloned inside the sandbox, e.g. /workspace/astropy__astropy__1.3, /workspace/django__django__3.2, /workspace/pallets__flask__2.3 etc. The workspace directory name is a combination of the instance repo and version in the OpenHands github repo here

def _get_swebench_workspace_dir_name(instance: pd.Series) -> str:
    return f'{instance.repo}__{instance.version}'.replace('/', '__')

• The OpenHands prompt does not modify the {location} variable in the following sentence in the prompt and is kept as /repo.

Your task is to make the minimal changes to non-tests files in the {location} directory to ensure the <pr_description> is satisfied.

A quick grep of the OpenHands trajectories reveals several instances with the following stdout observation entries in the log files:

• OBSERVATION:\nls -la /repo\r\nls: cannot access '/repo': No such file or directory
• OBSERVATION:\nERROR:\nThe path /repo does not exist. Please provide a valid path.
• '/repo/' is not a directory
• I notice the repository path is not in /repo. Let's move it there. Let's create the /repo directory first
• Let me check if we need to add /repo to the path

These show that the Claude 3.5 Sonnet model gets confused by two different path locations specified in the OpenHands prompt /repo and /workspace/astropy__astropy__1.3.

Perhaps it’s a typo/bug which was resolved before the submission but the trajectories were not updated. The current version of the prompt in the OpenHands code has updated it to /workspace.

So, we can conclude that OpenHands retains the Anthropic’s prompt unchanged, but has an additional system message, that is absent in the Anthropic Tool Use Agent. Anthropic has not disclosed details about any system message they might have used during their evaluation.

Tools

The Anthropic Tool Use Agent has two tools:

• a Bash Tool for executing bash commands, and
• an Edit Tool, for viewing and editing files and directories.

The description for both tools contains detailed information for the Claude 3.5 Sonnet model about how to use the tool. I really like the tool descriptions - they are well-crafted and strike a balance between detail and clarity.

Bash Tool/ CmdRunTool

Both OpenHands CmdRunTool and Anthropic’s Bash Tool have a very simple schema, taking only the command to be run in the environment. But, they have different tool descriptions.

OpenHands CmdRunTool description:

Execute a bash command in the terminal.
* Long running commands: For commands that may run indefinitely, it should be run in the background and the output should be redirected to a file, e.g. command = `python3 app.py > server.log 2>&1 &`.
* Interactive: If a bash command returns exit code `-1`, this means the process is not yet finished. The assistant must then send a second call to terminal with an empty `command` (which will retrieve any additional logs), or it can send additional text (set `command` to the text) to STDIN of the running process, or it can send command=`ctrl+c` to interrupt the process.
* Timeout: If a command execution result says "Command timed out. Sending SIGINT to the process", the assistant should retry running the command in the background.

Anthropic’s Bash Tool description -

Run commands in a bash shell\n
* When invoking this tool, the contents of the \"command\" parameter does NOT need to be XML-escaped.\n
* You don't have access to the internet via this tool.\n
* You do have access to a mirror of common linux and python packages via apt and pip.\n
* State is persistent across command calls and discussions with the user.\n
* To inspect a particular line range of a file, e.g. lines 10-25, try 'sed -n 10,25p /path/to/the/file'.\n
* Please avoid commands that may produce a very large amount of output.\n
* Please run long lived commands in the background, e.g. 'sleep 10 &' or start a server in the background.

Edit Tool

OpenHands StrReplaceEditorTool tool uses the a) same tool description, b) schema and c) code implementation as Anthropic’s str_replace_editor Edit tool. In CodeAct 2.1, StrReplaceEditorTool was used here and here.

This file editor is largely based on Anthorpic released str_replace_editor.
– from openhands - File Editor (commit ae13171)

Wait, what?

If both systems: OpenHands + CodeAct v2.1 (claude-3.5-sonnet-20241022) at Rank 3 and Tools + Claude 3.5 Sonnet (2024-10-22) at Rank 9 share:

• Prompt
• Edit tool for viewing and editing files and directories
• Bash tool schema

and only differ in:

• additional/different system instruction/prompt, and
• the description of Bash tool

These differences raise an interesting question: are they significant enough :thinking: to explain the following scenario?

60 instances only resolved by OpenHands
40 instances only resolved by Tools + Claude 3.5 Sonnet
205 instances resolved by both systems

Perhaps, what Anthropic’s post suggests is true, even when there are minor differences in the scaffolding!

The performance of an agent on SWE-bench can vary significantly based on this scaffolding, even when using the same underlying AI model.
– from Raising the bar on SWE-bench Verified with Claude 3.5 Sonnet

Takeaways

• I find it quite surprising that such seemingly minor differences could lead to markedly distinct outcomes on the benchmark.

• It’s entirely plausible that repeated runs of the same system would yield entirely different outcomes. This raises an intriguing point about the consistency and reliability of these systems — “how much of the observed performance is deterministic versus subject to randomness?”

Update (Jan 3, 2025)

Response from Xingyao Wang @All Hands AI/ OpenHands

Yeah, I've been observing similar things. The most challenging thing about agent evaluation is that even if you run the SAME code on the SAME set of problems with the SAME temperature=0, you typically end up with different results every time 😅.

We recently re-ran the commit… pic.twitter.com/pz0HdbdIPC

— Xingyao Wang (@xingyaow_) January 3, 2025

Citation

If you find this analysis useful for your research, please cite it as:

@misc{ganhotra2024openhands,
  title={OpenHands CodeAct v2.1 v/s Tools + Claude 3.5 Sonnet},
  author={Ganhotra, Jatin},
  year={2024},
  month={December},
  url={https://jatinganhotra.dev/blog/swe-agents/2024/12/31/claude-3.5-sonnet/},
  note={Blog post}
}

• SWE-Bench Verified != real-world SWE tasks, and
• SWE-Bench Verified ⊆ real-world SWE tasks (i.e. a subset)

Let’s dive in.

The year 2024 has been a thrilling time for SWE Agents, with the SWE-Bench leaderboard buzzing with activity throughout the year. SWE-Bench is a well known benchmark in the community for software engineering. Some notable events are:

• Devin was announced on March 12, 2024 and set a new state-of-the-art, on a random 25% subset of the dataset, and
• release of the updated Claude 3.5 Sonnet model setting a new state-of-the-art on SWE-Bench_Verified on Oct 30, 2024.

SWE-Bench Introduction

There are 3 benchmarks when we talk about SWE-Bench:

• SWE-Bench
• SWE-Bench_Lite
• SWE-bench_Verified

The authors introduce SWE-Bench as:

an evaluation framework consisting of 2,294 software engineering problems drawn from real GitHub issues and corresponding pull requests across 12 popular Python repositories. Given a codebase along with a description of an issue to be resolved, a language model is tasked with editing the codebase to address the issue.

Since the initial performance on the full SWE-Bench was quite low, the authors created a Lite subset of 300 instances, SWE-Bench_Lite, that have been sampled to be more self-contained, with a focus on evaluating functional bug fixes. Full details of the Lite split and filtering details are included in Appendix A.7 in the paper.

On August 13, 2024, OpenAI released a human-validated subset of 500 instances from SWE-bench, SWE-bench_Verified, that

more reliably evaluates AI models’ ability to solve real-world software issues.

For additional details on the SWE-Bench_Verified benchmark, please refer to OpenAI page.

Leaderboard

Since its release, SWE-bench_Verified has become the de-facto benchmark given OpenAI released it after human-validation. The current leaderboard for SWE-bench_Verified (top-10 systems) looks like (as of Dec 25, 2024):

As we can observe, the top-10 systems are quite close with only ~1% difference between each rank.

One Issue with the SWE-Bench_Verified Leaderboard

However, SWE-Bench was created with the following intent:

Resolving issues in SWE-bench frequently requires understanding and coordinating changes across multiple functions, classes, and even files simultaneously, calling for models to interact with execution environments, process extremely long contexts and perform complex reasoning that goes far beyond traditional code generation tasks.

Let’s take a deeper dive into the results while keeping in mind the following statement:

understanding and coordinating changes across multiple functions, classes, and even files simultaneously

If we look at the number of files per Github issue which must be modified to resolve it for SWE-Bench_Verified, we get:

For simplicity, let’s divide the SWE-Bench_Verified dataset in 2 buckets:

• instances which require code changes in a single-file = 429/500 = 85.8%, and
• instances which require code changes in multiple files (> 1) = 71/500 = 14.2%

If we divide the results between these 2 buckets, we get:

The performance of the top-10 systems on SWE-Bench Verified drops significantly for the multiple files category.

SWE-Bench test set

If we look at the distribution of number of files per Github issue which must be modified to resolve it for the (full) SWE-Bench test set, we get:

For simplicity, let’s divide the SWE-Bench test set in the same 2 buckets:

• instances which require code changes in a single-file = 1723/2294 = 75.11%, and
• instances which require code changes in multiple files (> 1) = 571/2294 = 24.89%

If we look at the leaderboard for SWE-Bench full test set (top-5 systems), as of Dec 25, 2024, for the 2 buckets, we get:

We observe the same trend. The performance of the top-5 systems on the (full) SWE-Bench test set drops significantly for the multiple files category.

Perhaps we don’t need to worry about the multiple files category, if we assume that the 75%-25% ratio in the SWE-Bench test set is reflective of the real-world Github issues. To validate our assumption, let’s take a look at the SWE-Bench train set.

SWE-Bench train set

If we look at the distribution of number of files per Github issue which must be modified to resolve it for the full SWE-Bench train set, we get:

There are 19008 instances in the train set and unidiff parser (v0.7.5) was unable to parse 1348 instances, so we have the statistics for 17660 instances. But, the above distribution paints an entirely different picture.

For simplicity, let’s divide the SWE-Bench train set in the same 2 buckets as before:

• instances which require code changes in a single-file = 8783/17660 = 49.73%, and
• instances which require code changes in multiple files (> 1) = 8877/17660 = 50.27%

If we assume that the instances which require > than 5 files are not representative of most day-to-day SWE tasks and exclude them, we get:

The above subset of instances which require <= than 5 files gives us 15847 instances. Dividing the above subset of SWE-Bench train set in the same 2 buckets as before:

• instances which require code changes in a single-file = 8783/15847 = 55.42%, and
• instances which require code changes in multiple files (> 1) = 7064/15847 = 44.58%

Conclusion

Aggregating the above statistics for the percentage of Github issues which require editing >1 file, we get:

SWE-Bench Verified != real-world SWE tasks

If we consider the SWE-Bench train set as a representative of real-world, then by the same definition, we can not consider the SWE-Bench_Verified test set as the true representative of our progress on SWE tasks given the much lower percentage of issues which require changes across >1 file for resolution.

Takeaways

• We have made tremendous progress on the SWE-Bench_Verified task, but there’s much more to be done when it comes to building systems which are good at understanding and coordinating changes across multiple functions, classes, and even files simultaneously. Perhaps the current systems (and models) are really good; and we need to curate better evaluation benchmarks.

• The leaderboard should break down the performance of each new entry across both buckets (single file and multiple files), in addition to the overall benchmark performance.

• We need a larger human-validated dataset which captures changes across multiple files, given 71 instances (SWE-Bench_Verified) is a very small number (even smaller than the HumanEval benchmark of 164 hand-crafted programming challenges, designed to evaluate an LLM’s code generation capabilities).

Citation

If you find this analysis useful for your research, please cite it as:

@misc{ganhotra2024swebench,
  title={SWE-Bench Verified ⊊ real-world SWE tasks},
  author={Ganhotra, Jatin},
  year={2024},
  month={December},
  url={https://jatinganhotra.dev/blog/swe-agents/2024/12/26/swe-bench-verified/},
  note={Blog post}
}

I found decent help from here - Getting Octave to Just Goddamn Work Already on OS X 10.8 Mountain Lion, thanks to Neil Traft for putting it together.

But that’s only half of the story as you’ve upgraded to OS X 10.9 Mavericks. Here, I am writing down steps that you should follow in the exact order and also tips to solve issues all by yourself, if you run into any.

P.S. Please don’t ignore any warnings or any messages that you feel might be irrelevant. If one piece doesn’t fit well, the whole thing may fall later.

1. Install the Xcode Command Line Tools

Installing command line tools in OSX 10.9 Mavericks has changed slightly than previous versions of MAC OS X. Now you don’t have to install Xcode , you can install Command Line Tools stand alone.
Type xcode-select --install in terminal and choose Install option from the pop-up.
This may take some time depending on your Internet connection.

2. Install Homebrew

I’m assuming you know about Homebrew, but if you don’t do yourself a favor, my friend and install it by following Homebrew’s installation instructions.

3. Install XQuartz

Install XQuartz from here. In the past this came installed on OS X, but as of Mountain Lion it is no longer included. So go fix that.

4. Updating and upgrading Homebrew

If you had Homebrew installed already, update to the latest package definitions. Don’t skip this step out of mere laziness.
$ brew update && brew upgrade
This may take some time. So, go get yourself a cup of coffee and better have some beer. The path ahead could be tough.

5. Import the scientific computing packages

$ brew tap homebrew/science
If you see any warnings here, run brew doctor and follow what it says. Let me repeat. Don’t be smart and just do what the doctor says!!! The doctor is always right.
Once you have followed the steps specifed by brew doctor, run
$ brew untap homebrew/science and rerun
$ brew tap homebrew/science.

6. Install gfortran

To install Octave, you have to install gfortran first.
$ brew install gfortran
Pay close attention to any messages and warnings during the install. If everything looks good, jump to the next Step.

If you see any warnings like: /usr/bin/install_name tool: malformed object (unknown load command),
go and visit this Github issue that I raised for more details.

If you’re facing any other problem:

• run brew doctor first and do what the doctor says!!!
• Re-run the command again and see if the issue got resolved.
• If not, go ahead and raise a new issue at Homebrew Issues.

7. Installing Octave

$ brew install octave
If everything looks good, jump to the next Step.

If brew complains about not having a formula for octave, the following command should fix it: $ brew tap --repair

If you get an error like do a web search for TeX and your operating system. make[3]: *** [octave.dvi] Error 1 then build with
$ brew install octave --without-docs option.
(Thanks Pierre and Kahini for the correction about --without-docs, previously -without-docs).

This is because you don’t have the Tex package installed. Alternatively, you can also download the 2.3GB MacTeX.pkg from here, install it and rerun the install command.
(Update: Don’t try this !!! I installed MacTex package but the Octave installation still failed. I raised an issue here - Homebrew-science Issues but with no help.)

If you run into any other issue, run brew doctor first and do what the doctor says!!! Re-run the command again and see if the issue got resolved. If not, go ahead and raise a new issue at Homebrew-science Issues.

8. Install gnuplot

For native drawing support, we’ll install gnuplot.
$ brew install gnuplot
(Update: As phseven pointed out, gnuplot will be automatically installed with octave, but in some cases it won’t support X11.) For such cases, do the following:
$ brew uninstall gnuplot
$ brew install gnuplot --with-x

9. Setup Octave startup file

Edit your Octave startup file or create a new one:
$ vim ~/.octaverc and add the line setenv GNUTERM x11.
Don’t forget to save the file.

10. Create a launcher app with Applescript [Optional]

Open the “AppleScript Editor” application and write the following text in the editor window:

tell application "Terminal"
 do script "`which octave`; exit"
end tell

Save it as an application ‘Octave.app’ and then navigate it to the “Applications” folder.

Now for the ultimate fix: Reboot your machine!!! Yep, until you restart your Mac, Octave will be unable to launch XQuartz on its own, and you won’t be able to plot anything.

Once you’re back in, open the terminal, fire up Octave, and plot a sombrero().__ You’re all set!!!__

warning: comparison is always false due to limited range of data type

I was just playing around with string.find() function, when I got this warning message. Here is a simple piece of code which, when run on a 64-bit architecture, will produce the above warning.

    string subject = "A[1]";
    string search = "1";
    unsigned location = subject.find(search);    // location is unsigned
    if( location == std::string::npos)
        cout << " true";
    else
        cout << " false";

The catch here is that std::string::npos returns size_t, which represents the unsigned native integer size on current architecture.
So, on a 32-bit architecture:

• both unsigned and size_t would have the same size,
  However, on a 64-bit machine:
• the unsigned would be size - 4 and size_t would be size - 8.
  Hence, the comparison would be always false when run on a 64-bit machine.

So, the proper fix for the above code is to make location variable of type size_t.

For more information on size_t, refer to my blog post on Integer Limits and Types in C/C++ , which explains the sizes of basic C++ types.

Note: If you’re seeing the same warning message, but not around the string.find() function, try StackOverflow for help. The warning message is generic and can occur at places where a comparison is being done between types of different sizes.