SWE-bench: Can Language Models Resolve Real-World GitHub Issues? — L2 #

1. TL;DR #

A benchmark of 2294 real GitHub issue→PR tasks across 12 Python repos: given an issue and a full codebase, the model must emit a patch that passes the PR's real tests. Even with an oracle retriever handing over the exact files to edit, the best model resolves only 4.8% — localization and long-context editing, not snippet synthesis, are the wall.

2. Q1 / Q2 / Q3 #

Q1 — 痛点 (what problem) #

Existing LM code benchmarks (HumanEval-style) are saturated and self-contained: they hand the model a docstring and ask for a few self-contained lines, verified by a couple of unit tests. That measures neither the real difficulty of software engineering nor the frontier of LM capability. Real bug-fixing requires navigating a repo with thousands of files, understanding cross-file interplay, and editing the right handful of lines "amongst a sea of context." No prior benchmark posed that task at scale with execution-based verification.

Q2 — 方法 (the approach) #

Mine merged PRs that (a) resolve a linked issue and (b) contribute new tests, then keep only those that survive an execution filter: after applying the PR's test patch, at least one test must flip fail→pass, and the install/run must succeed. Each surviving task is the tuple $(P, C, T, \delta)$ — problem statement, codebase-at-base-commit, tests, gold patch. The model sees $P$ + retrieved slices of $C$ and must emit a patch $\hat\delta$; it is scored by whether every FAIL_TO_PASS and PASS_TO_PASS test passes after applying $\hat\delta$.

核心技术壁垒: the execution-grounded, low-human-intervention construction pipeline — turning noisy GitHub PRs into fair, non-trivial, reproducible tasks. The hard part is not the idea but the per-version executable-context engineering plus the log-diffing that guarantees each task has a genuine fail→pass signal and no arbitrarily-named phantom tests. This is what makes the benchmark self-refreshing over time (§7).

Q3 — 结果 (what happened) #

SOTA models fail almost completely. Claude 2 = 4.8% resolved with oracle retrieval, 1.96% with BM25; GPT-4 = 1.7% (oracle). Fine-tuned SWE-Llama 7b/13b are competitive under oracle (3–4%) but collapse to 0.70% under BM25. More context does not help — increasing BM25 recall can lower resolve rate.

3. 架构 / 方法图 #

Paper's Figure 1, verbatim (caption: "SWE-bench sources task instances from real-world Python repositories by connecting GitHub issues to merged pull request solutions that resolve related tests. Provided with the issue text and a codebase snapshot, models generate a patch that is evaluated against real tests."). The right column shows the payoff: the generated patch is not judged by string match but by whether previously-failing tests (vstack_struct_col, euclidean_diff) turn green — execution is the oracle.

Paper's Figure 2, verbatim (caption: "SWE-bench task instances are created from merged pull requests that resolve an issue, contributes tests, and install successfully."). The funnel is the whole method: Stage I scrapes ~90k PRs from 12 popular repos; Stage II keeps only merged PRs that both resolve an issue and edit test files; Stage III keeps only those that install and produce ≥1 fail→pass test. 90k → 2294 survive.

The task loop for a single instance — construction, then model attempt, then execution scoring — is a state machine rather than a multi-turn agent (the baselines here are single-shot generators):

Memory / autonomy model: the baselines have no long-term memory and no multi-turn loop — the "context window" is the entire memory, filled by a retrieval step, and the agent gets exactly one greedy generation (Appendix D.2). There is no error-recovery branch available to the model itself; the only recovery is the harness's mechanical patch-repair step (§A.4 Step 6). Task class is open-ended repo editing; interaction is single-shot; autonomy is fully autonomous but non-interactive — which is precisely why the paper frames agent methods as future work (§7).

Paper's Figure 8, verbatim (caption: "Visualization of the evaluation pipeline at an individual task instance level. During evaluation, the Patch is model generated. A prediction .patch must be applied successfully and produce the same results as the corresponding task instance's D for task completion."). Note the test set is fixed and hidden from the model — the model never sees $T$, only the issue and code, so it cannot overfit to the grader.

4. 作者证明 #

无形式化作者证明 — 仅实证. This is a benchmark paper: there is no convergence, success, or generalization bound. The only formal content is the symbol definitions of a task instance, which are worth pinning down because the evaluation criterion rests entirely on them.

Notation table

The grading predicate is: task solved $\iff$ after applying $\hat\delta$, every $t_i \in$ (FAIL_TO_PASS $\cup$ PASS_TO_PASS) has status pass. FAIL_TO_PASS certifies the issue is fixed; PASS_TO_PASS certifies nothing else broke.

Minimum checks (6):

1. Metric well-defined? Yes — binary per-instance resolve; benchmark score = % resolved. Deterministic given cached ground-truth test-to-status maps (§A.4).
2. Non-triviality guaranteed? Yes — construction discards any candidate lacking ≥1 fail→pass test, so a no-op patch always fails at least one test.
3. Leakage controlled? Partly — $P$ excludes post-first-commit comments; training repos (SWE-Llama) are disjoint from eval repos; temporal split (Table 7) shows ~no before/after-2023 gap, empirically supporting the "no cheating via memorized versions" claim.
4. Fairness of hidden tests? Instances whose tests invoke functions/classes first introduced by $\delta$ are excluded (arbitrary naming ⇒ impossible). Instances with ImportError/AttributeError in log_pre are dropped.
5. Retrieval is a confound, is it measured? Yes — Table 3 reports BM25 recall vs oracle (44.4% avg at 27k); Table 4 shows resolve rate can drop as context grows, isolating localization as a distinct failure from retrieval recall.
6. What could have been bounded but wasn't? A success-rate model over (task difficulty × context length × backbone) — the paper gives the empirical sweep (Figure 5, Tables 5/6/7) but no analytic monotonicity claim. The clearest empirical monotonicity: resolve rate ↓ as total input length ↑.

5. 实验与数据 #



Paper's Table 5 (with the per-repository resolution bar chart below it). The load-bearing result: every model is in the low single digits. The bar chart shows performance is wildly uneven across repos — requests (small codebase) is ~15% while seaborn is ~0% — and the models' solved-sets barely overlap (Claude 2 solves only 42% of the instances SWE-Llama 13b solves). (Note: the extracted crop is from a later benchmark revision showing Claude 3 Opus / GPT-4-turbo rows; the original headline numbers in this paper are Claude 2 = 1.96% BM25 / 4.80% oracle, GPT-4 = 1.74% oracle.)

The BM25-vs-oracle gap is the theme. BM25 at 27k retrieves a superset of oracle files ~40% of the time yet resolves fewer issues, and pushing context to 50k (higher recall) lowers Claude 2 from 1.96% → 1.22%. Localizing the edit "in a sea of tokens" is the bottleneck, not retrieval recall:

Setting	Claude 2 % Resolved	SWE-Llama 13b % Resolved
BM25 13k	1.96	0.70
Oracle	4.80	4.00
Oracle-collapsed (±15 lines)	5.9	—

Collapsing the oracle context to just the edited lines ±15 pushes Claude 2 to 5.9% and GPT-4 from 1.3%→3.4% — direct evidence that distraction, not missing information, caps performance.



Paper's Table 8, verbatim (caption: "Average edits of model generated patches in the oracle retrieval setting across successfully applied patches..."). Models systematically under-edit: a Claude 2 patch that applies is 19.6 lines / 1.0 files vs the gold 44.1 lines / 1.2 files, and vs 74.5 lines / 1.7 files across all gold patches. Combined with the §5.1 case study (sphinx-8713, where the model edits the right function but hard-codes use_param=True instead of checking the config), this shows the dominant failure class is shallow, greedy single-file edits — models patch the symptom, gold patches restructure.

Latency budget: not applicable — these are single-shot batch generations (greedy, one patch per instance), so the paper makes no interactive-latency claim.

6. 论证链 #

#	Step (paper-internal)	Support
1	Existing code benchmarks are saturated & self-contained, so they don't measure the frontier.	§1; contrast with HumanEval (few self-contained lines).
2	Real GitHub issue→PR resolution is challenging, execution-verifiable, and self-refreshing, so it is a better testbed.	§2.3 features: cross-context editing (avg 1.7 files / 3.0 funcs), robust fail→pass tests, continual updatability.
3	A 3-stage scrape→attribute→execution pipeline yields 2294 fair, non-trivial tasks with minimal human effort.	§2.1 + §A.3; funnel 93139 PRs → 11407 candidates → 2294 (Table 10); ~half of candidates die at execution filter.
4	Evaluated under both BM25 and oracle retrieval, SOTA models resolve <5%.	§5 Table 5: Claude 2 4.8% oracle / 1.96% BM25; GPT-4 1.7%.
5	The bottleneck is localization / long-context editing, not retrieval recall.	Table 4 (perf ↓ as context ↑), Figure 5 (perf ↓ as input length ↑), Table 6 (collapsed context ↑ perf).
6	Therefore the open research direction is agents that actively gather context + tool-augmented editing.	§7 discussion explicitly calls out agent-based context identification as the exciting next step.

7. 实现 cross-reference #

Reference implementation is open-sourced at https://www.swebench.com (data, harness, leaderboard, SWE-Llama weights); §9 states the full anonymized source (collection / evaluation / inference / SWE-Llama training) was released. No file:line citations are available from the L1 source, so specific line references are [实现未公开] at the L1 level, but the following are the load-bearing implementation contracts:

核心技术壁垒 (elaborated) — the execution-validation engine (§A.3) is the single hardest piece to replicate. It requires (i) hand-built per-release-version conda executable contexts (not per-instance, not per-repo — release version is the sweet spot balancing install success vs manual effort), (ii) repository-specific log parsers mapping raw pytest/tox output to per-test fail/pass status, and (iii) diffing log_pre vs log_post to certify ≥1 genuine fail→pass. Get any of these wrong and tasks become either unsolvable or trivially passable. This is the moat that lets the benchmark self-refresh on future PRs.

关键实现细节 (easy-to-miss tricks):

1. Patch auto-repair before scoring (§A.4 Step 6): when git apply of $\hat\delta$ fails, the harness strips unnecessary context lines and recomputes hunk headers, then retries once. Table 14 shows this "fix" rescues a large fraction of closed-model patches (GPT-4 oracle: 68.7% of applied patches needed the fix), so raw apply rates badly understate what models emit — fine-tuning (SWE-Llama) mostly fixes formatting, not reasoning.
2. Leakage guard in $P$ construction (§A.2): the problem statement aggregates only issue comments created before the PR's first commit timestamp, and hints_text (post-hoc natural-language solution hints) is collected but deliberately excluded from the experiments — a latent signal future work can exploit but the baselines do not.