SWE-bench is the measuring stick, and almost all its cluster peers are the
infrastructure that gets measured or that lives in the same agent category
but at a different layer of the stack. The relationship is therefore
axis-orthogonal rather than head-to-head: SWE-bench defines *what "success"
means for a coding agent; the peers optimize how fast / how cheaply* an agent
reaches that success, or how agents talk to tools/each other.
and a real distributed SWE-agent testbed, reporting up to 8.18× latency
reduction while holding pass rate at ~7% [2511.02230]. It
consumes SWE-bench's task-instance/tool-call structure directly.
profiles SWE-Agent (the agent scaffold built on top of SWE-bench) as one
of its five representative workloads, finding Bash/Python tool execution
shifts from 38%→65% of E2E latency on a stronger GPU.
SWE-bench directly).**
for arbitrary agent DAGs (ReAct/MCTS/Map-Reduce); 4–15× throughput.
consolidated agentic-workflow DAGs; up to 18.6× batch speedup.
multi-agent communication; up to ~7.8× TTFT.
scheduler for tool-call idle windows; up to 47% latency cut.
strong/weak models; a policy layer that could sit above a SWE-bench agent.
the communication substrate (tool invocation, agent-to-agent) that a
SWE-bench-solving agent would use to reach its execution environment.
Why they cluster: SWE-bench (§7 Discussion) explicitly invites exactly these
follow-ups — "agent-based approaches for identifying relevant context ... and
augmenting LMs with program analysis and software engineering tools." The peers
are the ecosystem that grew into that invitation.
| Dimension | SWE-bench (this) | Cluster peers |
|---|---|---|
| Artifact type | evaluation benchmark + harness (2294 tasks, execution-graded) | systems (schedulers, caches, routers) or survey |
| Metric of merit | task resolve rate (% issues solved) | throughput / latency / cost; task success held fixed |
| Contribution | defines the goalpost | moves the agent toward the goalpost faster/cheaper |
evaluation. Before it, code benchmarks were self-contained snippet synthesis
(HumanEval). SWE-bench's delta is the 3-stage scrape→attribute→execution
funnel that guarantees each task has a genuine fail→pass signal
Continuum and the CPU-centric paper accept SWE-bench's definition of the task
and only attack the serving cost of running an agent against it. Continuum's
headline 8.18× and its ~7% pass rate [2511.02230] are only
meaningful because SWE-bench fixed what "pass" means — the benchmark is a
precondition for their claim.
dominant wall is model-side: long-context localization "in a sea of tokens"
and the edit itself, since even an oracle retriever yields only 4.8%
[2310.06770]. The CPU-centric paper claims the dominant
latency wall is infrastructure-side / CPU tool execution (up to 88% of
E2E) [2511.00739].
矛盾根源: different objective functions, not a real disagreement.
SWE-bench measures whether the task is solved (a capability/quality axis) on
single-shot batch generation with no interactive latency claim
[2310.06770]. The CPU-centric paper measures *how long an
agent scaffold takes* (a latency axis) and explicitly does not touch task
success [2511.00739]. Both are correct within their axis;
they only look contradictory if one conflates "hard to solve" with "slow to
run."
Attack: the oracle setting hands the model exactly the gold-edited files yet
the paper simultaneously argues localization is the bottleneck. If localization
is the wall, an oracle-retrieval score of 4.8% conflates two failures
(localize + edit) into one number that cannot isolate either. The
oracle-collapsed ablation (Claude 2 4.8→5.9%) [2310.06770]
only barely moves, suggesting the residual failure is the edit, which
undercuts the paper's own "localization is the bottleneck" emphasis.
headline <5% numbers are an artifact of the non-interactive protocol. The
whole peer cluster presupposes multi-turn agents (Autellix's ReAct/MCTS DAGs
[2502.13965], Continuum's 6–11 turn ReAct programs
[2511.02230]). SWE-bench's baseline numbers say almost
nothing about the agentic regime that its own §7 invites — and history bore
this out (agent scaffolds later crossed 50%+). The paper's pessimism is a
property of the baseline harness, not of the task.
result [2310.06770] is presented as reassuring, but a model
can have memorized the fix pattern for a class of bugs without seeing the
exact post-2023 commit. Temporal disjointness ≠ semantic novelty.
requests repo scores ~15% while seabornscores ~0% [2310.06770]; the aggregate is dominated by a
handful of repos (django alone is 850/2294). A benchmark whose difficulty is
this repo-skewed risks measuring "how well pre-training covered django."
SWE-bench occupies the evaluation-substrate niche: it is the shared
coordinate system that lets the rest of the cluster claim progress. Its
paradigm shift was moving code evaluation from closed-form snippet synthesis
to open-ended, execution-verified repository editing
[2310.06770] — reframing "code generation" as "software
engineering as an agent task."
Adoption evidence (from the cluster itself):
capability benchmark: Continuum uses SWE-Bench + real SWE-agent as its primary
testbed [2511.02230] and the CPU-centric paper uses SWE-Agent
as a workload archetype [2511.00739]. That a serving
paper reaches for SWE-bench to demonstrate scheduling gains is strong evidence
the benchmark has become the community's canonical agent-coding workload.
[2502.13965], Halo) build synthetic ReAct/MCTS workloads
instead — and their L2 notes flag this as a scope limitation (cannot verify
task quality). This asymmetry shows SWE-bench filled a gap those papers openly
lack: an execution-grounded task-success signal.
The niche is durable precisely because of the self-refreshing pipeline
(§7): unlike a static benchmark that saturates, SWE-bench's low-human-intervention
construction lets it regenerate from new PRs, keeping it ahead of the training
cutoff it critiques HumanEval for lacking [2310.06770].
rate; the cluster reports only latency/throughput. A hybrid metric —
resolve-rate-per-dollar or resolve-rate at a latency SLO — would fuse
SWE-bench's grading with Continuum/TokenCake/Autellix serving models
[2511.02230][2510.18586][2502.13965].
Nobody yet grades an agent on quality under a serving budget.
difficulty [2406.18665]; SWE-bench shows difficulty is highly
uneven across issues/repos [2310.06770]. A router that sends
easy localization/edit turns to a weak model and hard cross-file
restructuring turns to a strong model, graded by SWE-bench resolve rate, is an
unexplored cost-quality Pareto experiment.
exploit that agent turns share context
[2510.12872][2510.18586][2511.02230]. A
coding agent iterating on the same 438K-line codebase across turns is a
near-ideal case for cross-turn KV reuse, but none of these caching papers
evaluate on the SWE-bench codebase-editing loop specifically.
tool execution dominates SWE-Agent latency on strong GPUs
[2511.00739]; SWE-bench's
grader is itself a heavy CPU workload (install + run pytest per instance).
Applying COMB-style CPU/GPU micro-batch overlap to the SWE-bench
evaluation-and-edit loop is an obvious, untried combination.
[2505.02279] frames tool access (MCP) and agent-to-agent
delegation (A2A) as the substrate; SWE-bench's cross-file edits (avg 1.7 files)
are exactly where a multi-agent "one agent per file/module, coordinated over
A2A" decomposition could be tested — an adaptive division of labor SWE-bench's
single-agent baselines never explore.