SWE-agent
an archive of posts with this tag
| Apr 05, 2026 | Hidden Naming Contracts in SWE-Agent Benchmarks A programmatic scan of six SWE-bench-style benchmarks — SWE-bench Verified, SWE-bench Pro and SWE-PolyBench — finds tests that encode hidden naming contracts, penalizing behaviorally correct fixes that choose different identifiers. |
|---|---|
| Jul 26, 2025 | The Visual Complexity Penalty in Code Understanding - SWE-bench Multimodal Analysis How visual complexity penalizes SWE-agents on SWE-bench Multimodal — testing SWE-agent, Agentless and OpenHands with Claude 3.7 Sonnet and OpenAI o3 on visually rich GitHub issues. |
| Jun 05, 2025 | From 73% to 11%: Revealing True SWE-Agent Capabilities with Discriminative Subsets Discriminative subsets of SWE-bench Verified reveal true SWE-agent capability — how aggregate scores hide wide variation across SWE-agent, OpenHands, Claude 4 Opus and the L* agent (from 73% to 11%). |
| Apr 15, 2025 | Cracking the Code: How Difficult Are SWE-Bench-Verified Tasks Really? Task-difficulty distribution in SWE-bench Verified from human annotations — what easy, medium and hard mean for SWE-agents like SWE-agent and Agentless running Claude and OpenAI o1. |
| Mar 30, 2025 | The Multi-File Frontier: Why SWE-Bench Verified Doesn't Reflect Real-World Programming Challenges Why SWE-bench Verified's focus on single-file changes misses real-world multi-file programming — analyzed across SWE-agent, Agentless, Claude 3 Opus, Claude 3.5 Sonnet, OpenAI o1 and Amazon Q. |
| Jan 05, 2025 | Do SWE-Agents Solve Multi-File Issues Like Humans? A Deep Dive into SWE-Bench Verified How SWE-agents (OpenHands, SWE-agent, Agentless) handle multi-file software engineering tasks compared to human developers on SWE-bench Verified, with Claude 3.5 Sonnet and OpenAI models. |
| Dec 31, 2024 | OpenHands CodeAct v2.1 v/s Tools + Claude 3.5 Sonnet Head-to-head comparison of OpenHands CodeAct v2.1 and Anthropic Claude 3.5 Sonnet on SWE-bench Verified, analyzing the performance differences and capabilities of these leading SWE-agent approaches. |
| Dec 26, 2024 | SWE-Bench Verified ⊊ real-world SWE tasks Why SWE-bench Verified is only a subset of real-world software engineering tasks — comparing SWE-agents such as OpenHands CodeAct v2.1, Amazon Q, SWE-agent, Agentless and AutoCodeRover, with Claude 3.5 Sonnet. |