Hidden Naming Contracts in SWE-Agent Benchmarks

AI coding benchmarks now influence research priorities, product strategy, and engineering adoption decisions. Over the last year, SWE-bench has become a key benchmark for evaluating AI coding agents, and that momentum has pushed the community to build additional SWE-bench-style benchmarks beyond Python.

As I have explored in earlier analyses on single-file saturation and difficulty distribution, benchmark scores are only as trustworthy as the instances behind them. In this post, I look at a different failure mode: hidden naming contracts.

A hidden naming contract appears when benchmark tests require specific identifiers introduced in the reference solution, even though those names were never made explicit in the issue text. In that setting, an agent can produce a behaviorally correct fix and still be graded as wrong because it chose a different symbol name.

The Core Failure Mode

A typical SWE-bench-style instance has four parts:

• an issue description
• a base repository snapshot
• a reference solution
• executable tests

The intended contract is behavioral correctness: if a submitted patch fixes the issue, the tests should pass.

The problem starts when tests directly call symbols that were newly introduced in the reference solution. Evaluation then requires not only solving the behavior, but also reproducing a naming choice that may never have been stated anywhere.

A Concrete Example

Consider scikit-learn__scikit-learn-12682 from SWE-bench_Verified.

The issue reports that SparseCoder does not expose max_iter for Lasso, which leads to convergence warnings. The name transform_max_iter is not mentioned in the issue and does not exist elsewhere in the codebase at the base commit.

The reference patch introduces a new transform_max_iter parameter:

# sklearn/decomposition/dict_learning.py
class SparseCoder(BaseEstimator, SparseCodingMixin):
    def __init__(self, dictionary, transform_algorithm='omp',
                 transform_n_nonzero_coefs=None, transform_alpha=None,
                 split_sign=False, n_jobs=None, positive_code=False,
                 transform_max_iter=1000):
        self._set_sparse_coding_params(..., transform_max_iter)

The test patch then calls that exact parameter name:

def test_max_iter():
    with pytest.warns(ConvergenceWarning):
        model = SparseCoder(
            D_multi,
            transform_algorithm=transform_algorithm,
            transform_max_iter=1,
        )
        model.fit_transform(X)

    with pytest.warns(None) as record:
        model = SparseCoder(
            D_multi,
            transform_algorithm=transform_algorithm,
            transform_max_iter=2000,
        )
        model.fit_transform(X)

An agent could implement the same functionality with lasso_max_iter, max_transform_iterations, or another reasonable name and still fail evaluation. The behavioral fix is there, but the hidden naming contract is not satisfied.

What the Scan Found

I analyzed six SWE-bench-style datasets:

• SWE-bench/SWE-bench
• SWE-bench/SWE-bench_Verified
• SWE-bench/SWE-bench_Multilingual
• ScaleAI/SWE-bench_Pro
• ByteDance-Seed/Multi-SWE-bench
• AmazonScience/SWE-PolyBench

In a screening pass across all 7,567 instances, I found 2,167 instances (28.6%) where tests reference symbols newly introduced in the reference solution.

That 28.6% number is a screening signal, not a final estimate of confirmed false negatives. Some coupled symbols are fair because the name is explicit in the issue text or already established in the codebase. To isolate the highest-risk cases, I ran two refinement checks:

1. Is the symbol name mentioned in the issue text?
2. Does the symbol already exist elsewhere in the repository?

I treat the intersection as high-risk coupling: symbols that are neither mentioned in the issue nor present in the codebase.

High-Risk Coupling by Benchmark

The main takeaway is that this problem is concentrated rather than uniform.

• SWE-bench_Pro is the clear outlier at 10.9% high-risk coupling.
• Multi-SWE-bench is meaningfully elevated at 5.5%.
• SWE-bench_Verified and SWE-bench_Multilingual are much cleaner on this specific failure mode.

That pattern suggests curation helps, but it does not eliminate the issue.

Where the Risk Concentrates

The high-risk subset becomes easier to interpret if we also look at the two refinement signals separately:

Two things stand out.

First, many coupled instances provide no naming hint in the issue text. For most datasets, roughly 40% to 60% of coupled instances fall into that bucket.

Second, codebase priors help in many cases, but not equally across benchmarks. In SWE-bench_Verified, most coupled symbols already exist somewhere else in the repository, which gives agents a chance to infer the naming pattern through exploration. In SWE-bench_Pro and Multi-SWE-bench, that safety net is much weaker.

This is why the raw coupling rate and the high-risk rate both matter. The raw rate tells us how often tests are structurally tied to reference naming. The high-risk rate tells us where that coupling is most likely to generate false negatives.

Why This Matters for Leaderboards

1. Scores can move at the margin

In SWE-bench_Pro, high-risk coupling appears in 10.9% of all instances. If only a fraction of those behave as false negatives in practice, that is still enough to shift scores by one to several points, which can reorder closely clustered systems.

2. Cross-benchmark comparisons get noisier

A model may appear to improve or regress partly because one benchmark family embeds more hidden naming contracts than another. That makes benchmark-to-benchmark comparisons less clean than leaderboard tables suggest.

3. Curation helps, but one clean metric is not the whole story

The low rates in SWE-bench_Verified and SWE-bench_Multilingual are encouraging for this particular failure mode. But low hidden-contract rates should not be read as a blanket validation of a benchmark’s frontier-tracking quality. Benchmarks can still have other issues, including overly narrow tests, overly wide tests, or contamination.

Recent benchmark audits make the same broader point from a different direction. My analysis is narrower: it is a scalable programmatic scan for one subtype of narrow-test risk.

What Benchmark Maintainers Should Change

1. Prefer behavior-first assertions where feasible

Tests should verify the intended behavior, not accidentally require the exact reference implementation.

2. Make naming requirements explicit when they are part of the task

Some fixes genuinely require a specific API, method name, or parameter name for compatibility reasons. In those cases, the naming requirement should be written into the issue statement rather than hidden in the tests.

3. Publish diagnostic metadata alongside headline scores

Leaderboards should ideally report not just one aggregate score, but also benchmark diagnostics such as coupling prevalence, high-risk hidden-contract rates, and whether a score was computed on a filtered subset that excludes known problematic instances.

4. Keep a human review loop in benchmark maintenance

The cleanest benchmarks in this analysis are also the ones with stronger curation signals. That supports a practical maintenance rule: programmatic scans can pre-screen risky instances, but human review is still needed before release.

Conclusion

Software engineering allows multiple valid implementations, and evaluations should reflect that reality. A benchmark should reward behaviorally correct fixes, not force agents to rediscover one unstated naming choice from a hidden reference patch.

Hidden naming contracts are not the entire benchmark-reliability story, but they are a measurable and actionable part of it. If we want benchmark scores to be more robust, one straightforward step is to identify and remove instances where tests silently depend on names that the task never actually required.

A coding agent should not be graded as wrong for solving the right problem with the wrong unstated identifier.

Methodology

The analysis uses a simple pipeline:

1. Parse gold patches and extract symbols introduced on added lines.
2. Parse test patches and look for references to those symbols.
3. Filter out overly generic names that are likely to match coincidentally.
4. For coupled instances, check whether the symbol appears in the issue text.
5. For coupled instances, check whether the symbol already exists elsewhere in the repository at the base commit.

The symbol extraction step uses language-specific patterns for Python, Java, JavaScript or TypeScript, Go, Rust, and C or C++.

This post reports filtered counts throughout. The filtering step removes names that are so generic that a match is probably accidental rather than evidence of a real hidden naming contract.

The high-risk subset reported above is the intersection of the two refinement checks: symbols not mentioned in the issue text and not present in the codebase.

Appendix: High-Risk Instances by Benchmark

The following tables list the instances in the high-risk intersection.

SWE-bench/SWE-bench

SWE-bench/SWE-bench_Verified

ByteDance-Seed/Multi-SWE-bench

ScaleAI/SWE-bench_Pro

AmazonScience/SWE-PolyBench

Citation

If you find this analysis useful for your research, please cite it as:

@misc{ganhotra2026hiddencontracts,
  title={Hidden Naming Contracts in SWE-Agent Benchmarks},
  author={Ganhotra, Jatin},
  year={2026},
  month={April},
  url={https://jatinganhotra.dev/blog/swe-agents/2026/04/01/hidden-naming-contracts-in-swe-agent-benchmarks/},
  note={Blog post}
}