From 73% to 11%: Revealing True SWE-Agent Capabilities with Discriminative Subsets

Since my last analysis of SWE-Bench Verified on April 15, there has been significant progress on the leaderboard. The best performing SWE-agents are now at 73.20% (Tools + Claude 4 Opus) whereas only ~45 days ago, the best performing system was Augment Agent v0 at 65.40% - representing a remarkable 7.8 percentage point improvement in less than two months.

The Saturation Problem Revisited

In my previous analyses, I identified that Easy problems are effectively saturated, with top SWE-agents achieving 84-86% success rates. My single-file saturation study further revealed structural limitations in current evaluation approaches.

This saturation creates a differentiation problem: when most competitive SWE-agents solve 160+ of the same 194 Easy problems, these categories no longer provide meaningful signal for distinguishing between top-tier systems. The real competition has shifted to unsolved and sparsely-solved problems across all difficulty categories.

A Data-Driven Solution: Discriminative Subsets

Rather than arbitrarily choosing “hard” problems, I developed a systematic methodology by analyzing how many SWE-agents solve each instance across all 500 problems in SWE-Bench Verified.

Instance Solve Distribution Analysis

Each of the 500 instances was checked against evaluation results from 83 distinct SWE-agents (submitted between October 2023–May 2025) to record solve counts. “Solved” means the agent’s fix passed the verification test suite - the same standard used in the original SWE-bench evaluation.

I categorized all instances based on how many of the 83 evaluated SWE-agents successfully solve them:

Key insight:

• 69% of problems are solved by 21+ agents, resulting in limited discrimination during evaluation
• The competitive frontier exists in the first five categories (155 instances total), solved by ≤20 agents (High discrimination potential)
• 52 completely unsolved problems (Maximum discrimination)

Four Discriminative Subsets

Rather than continuing to measure incremental improvements on largely-solved Easy problems, I designed targeted subsets to focus on:

1. Completely unsolved problems (52 instances) - true frontier challenges
2. Sparsely solved problems - instances resolved by only a handful of agents
3. Problems with high solution variance - where top SWE-agents show meaningful differences

This approach yields a more discriminative and high-resolution instrument for measuring real-world SWE-agent capability differences, similar to how other AI benchmarks have evolved when existing evaluations became saturated.

Based on this analysis, I created four targeted evaluation subsets. The “Solve Range” column shows how many agents successfully solve the problems within each subset - for example, Frontier subset problems are solved by 0-5 agents, making them the most evaluatively sensitive.

Subset Relationships:

• Frontier ⊆ Challenging (all Frontier problems are included in Challenging)
• Hard and MultiFile subsets partially overlap with both Frontier and Challenging subsets
• Single-file problems involve changes to one source file, while multi-file problems require coordinated modifications across multiple files.

1. Frontier Subset (95 instances)

Problems solved by ≤5 agents - maximum evaluative sensitivity

This subset combines completely unsolved problems with ultra-rare and very-rare solves. It’s composed of 14 Easy, 52 Medium, and 29 Hard problems. Notably, even the top-performing Claude 4 Opus scores just 11.6% on this subset, compared to its 73.2% on the full benchmark. This provides extraordinary differentiation power between cutting-edge systems.

• Composition: 58 single-file, 37 multi-file problems
• Top Performance: Claude 4 Opus 11.6% (vs 73.2% on full benchmark)
• Purpose: Maximum resolution for cutting-edge agent comparison

Top-10 performing SWE-agents on the Frontier subset:

2. Challenging Subset (155 instances)

Problems solved by ≤20 agents - strong evaluative power

Expanding to include rare and uncommon problems, this subset maintains robust differentiating ability while providing more instances for statistical significance. Claude 4 Opus reaches 31.6% here - still providing much better separation than the 84%+ scores on Easy problems.

• Composition: 28 Easy, 93 Medium, 34 Hard; 105 single-file, 50 multi-file
• Top Performance: Claude 4 Opus 31.6%
• Purpose: Balance of resolution and statistical significance

Top-10 performing SWE-agents on the Challenging subset:

3. Hard (45 instances)

All Hard difficulty problems regardless of solve rate

This subset focuses specifically on the 45 Hard problems, with Claude 4 Opus achieving 42.2%. Interestingly, this includes some problems solved by many agents, showing that difficulty level and solve count don’t perfectly correlate.

• Composition: 0 Easy, 0 Medium, 45 Hard; 20 single-file, 25 multi-file
• Top Performance: Claude 4 Opus 42.2%
• Purpose: Focused evaluation on most difficult problem category

Top-10 performing SWE-agents on the Hard subset:

4. MultiFile (40 instances)

Multi-file problems solved by ≤10 agents

This targets the intersection of multi-file problems (which tend to be harder) with low solve counts. These problems require coordinated edits across multiple source files, making them more complex for current SWE-agents. It’s the most challenging subset - even Claude 4 Sonnet only achieves 10.0%. The composition (3 Easy, 17 Medium, 20 Hard) confirms that multi-file problems are inherently more difficult.

• Composition: 3 Easy, 17 Medium, 20 Hard; 0 single-file, 40 multi-file
• Top Performance: Claude 4 Sonnet 10.0%
• Purpose: Target intersection of multi-file complexity and low solve rates

Top-10 performing SWE-agents on the MultiFile subset:

Summary Comparison

Key Insights and Patterns

1. Medium Problems Drive Frontier Resolution

The Frontier subset contains 52 Medium vs 29 Hard problems, revealing that traditional difficulty categories don’t capture all sources of complexity. Some Medium problems remain more challenging than many Hard problems. This suggests that ‘Medium’ in SWE-Bench often reflects problem scope or context rather than true agent difficulty.

2. Multi-file Problems Are Genuinely Harder

Multi-file problems dominate the low-solve buckets:

• 40/40 MultiFile problems are multi-file (by design)
• 37/95 Frontier problems are multi-file (39%)
• Only 2/156 “Solved” problems are multi-file (1.3%)

3. Massive Performance Differentiation Gains

The performance gaps between full benchmark and targeted subsets are dramatic. While the top two SWE-agents (Tools + Claude 4 Opus and Tools + Claude 4 Sonnet) are separated by less than 1 percentage point on the entire 500-instance SWE-Bench Verified benchmark (73.2% vs 72.4%), the specialized subsets create substantial separation between these same systems. These clear performance gaps demonstrate the power of focusing evaluation on truly challenging problems.

While the specialized subsets have smaller sample sizes, the large performance differences (typically 35-65 percentage point drops from full benchmark performance) clearly indicate meaningful capability distinctions.

Implications for the Research Community

Immediate Benefits

1. Enhanced Signal: Researchers can immediately use these subsets for more sensitive evaluation
2. Research Focus: Identifying specific problem types guides targeted improvement efforts
3. Facilitates curriculum design: For agent fine-tuning based on real-world bottlenecks
4. Clearer Progress: Performance improvements on frontier problems represent genuine capability advances

Methodological Innovation

This solve-distribution approach could be applied to other saturated benchmarks:

1. Analyze how many systems solve each problem
2. Identify low-solve instances for targeted subsets
3. Create sensitive evaluation instruments
4. Maintain evaluative power as capabilities advance

Future Evolution

As SWE-agents continue improving, this methodology enables dynamic subset creation:

• Current “Ultra Rare” problems may become “Common”
• New targeted subsets can be generated using the same framework
• Evaluation maintains sensitivity to genuine progress

Technical Implementation Notes

Data Availability

All subset definitions and solve matrices are available as structured JSON data, enabling:

• Immediate subset evaluation using existing SWE-Bench infrastructure
• Integration with current evaluation pipelines
• Reproducible research and fair comparisons

Evaluation Guidelines

For consistent evaluation across the research community:

1. Report both full benchmark and subset performance
2. Use Frontier subset for cutting-edge system comparison
3. Use Challenging subset for statistical significance
4. Use specialized subsets (Hard, MultiFile) for targeted research

Try the Discriminative Subsets

The targeted subsets are available on HuggingFace for immediate use:

Dataset: jatinganhotra/SWE-bench_Verified-discriminative

Four splits: frontier, challenging, hard, multifile

from datasets import load_dataset

# Load all splits
dataset = load_dataset("jatinganhotra/SWE-bench_Verified-discriminative")

# Load specific split for targeted evaluation
frontier = load_dataset("jatinganhotra/SWE-bench_Verified-discriminative", split="frontier")

I encourage researchers to benchmark their SWE-agents on these discriminative subsets and share results publicly. Tracking progress here, not just on saturated sets, will best reflect true capability gains in autonomous software engineering rather than incremental improvements on saturated problem sets.

Conclusion

This analysis reveals that SWE-Bench Verified’s evaluative power has become concentrated in a subset of challenging problems. Systematically identifying these problems through solve-distribution analysis enables high-resolution benchmarks like the Frontier Subset, which reveal capability distinctions that broad benchmarks obscure.

The Challenging Subset balances sensitivity with statistical power, while the Hard and MultiFile subsets offer targeted evaluation for specific research directions.

Most importantly, this methodology - analyzing solve distribution to identify evaluatively sensitive problems - provides a data-driven framework for benchmark evolution. As AI capabilities continue advancing, this approach ensures evaluation maintains sensitivity to genuine progress rather than incremental improvements on saturated problem sets.

The targeted subsets are immediately usable with existing SWE-Bench infrastructure, enabling the research community to adopt more sensitive evaluation practices while pushing toward the true frontiers of automated software engineering.

Citation

If you find this analysis useful for your research, please cite it as:

@misc{ganhotra2025discriminative,
  title={From 73% to 11%: Revealing True SWE-Agent Capabilities with Discriminative Subsets},
  author={Ganhotra, Jatin},
  year={2025},
  month={June},
  url={https://jatinganhotra.dev/blog/swe-agents/2025/06/05/swe-bench-verified-discriminative-subsets/},
  note={Blog post}
}