Scepsy (2604.15186) · L3 per-paper synthesis #
Target: Scepsy — Serving Agentic Workflows Using Aggregate LLM Pipelines [2604.15186]
Cluster: agentic serving systems. Peers span request-level schedulers, KV-cache-centric serving, CPU-side characterization, model routing, and a protocol survey.
The eight peers split into four bands by how directly they contend with Scepsy's problem: turn unpredictable multi-LLM workflows into good GPU allocations.
Band A — Direct baselines / same problem (request-level agent serving).
- Autellix (2502.13965) is the closest sibling and an explicit Scepsy baseline (its
autellix cite is a Table-1 row). It also observes that agent programs have unpredictable per-request execution and reasons over cumulative program service time to schedule [2502.13965]. But Autellix schedules requests temporally (program-level LAS/ATLAS priorities) and, critically, assumes all agents share one LLM [2502.13965]. Scepsy attacks the same unpredictability but with GPU allocation over multiple heterogeneous LLMs [2604.15186].
- TokenCake (2510.18586) and Continuum (2511.02230) are KV-cache-centric agent schedulers that Scepsy's own L1 groups with Parrot/TokenCake as "workflow-aware but single-LLM." TokenCake co-designs a temporal offload + spatial partition scheduler around tool-call idle windows [2510.18586]; Continuum introduces a KV-cache TTL to kill per-turn queueing bubbles in multi-turn ReAct agents [2511.02230]. Both are complementary within Scepsy's per-LLM engine layer rather than competitors for the allocation problem.
Band B — Diagnosis / characterization that stress-tests Scepsy's core assumption.
- CPU-Centric characterization (2511.00739) is the most dangerous peer for Scepsy's model: it measures that tool execution can be up to 88% of end-to-end latency in real agentic workloads [2511.00739]. Scepsy explicitly assumes non-LLM tool/orchestration time is "at most a few milliseconds" and discards it [2604.15186]. This peer directly probes whether that assumption holds. (2511.00739 already lists 2604.15186 as a
related entity in its frontmatter.)
Band C — Complementary latency-hiding / reuse mechanisms.
- Speculative Tool Calls (2512.15834) hides tool latency by speculatively pre-executing tool calls in parallel with generation [2512.15834]. This is the mechanism that could rescue Scepsy when tools are NOT negligible (Band B's threat).
- KVCOMM (2510.12872) reuses KV across agents by approximating cross-context offsets [2510.12872] — an engine-level throughput lever orthogonal to Scepsy's allocation search.
Band D — Adjacent framing, weakly related.
- RouteLLM (2406.18665) routes between models for cost/quality [2406.18665] — a different axis (which model) vs Scepsy's (how much GPU per model), but both exploit stable statistical structure to make cheap decisions.
- Agent Interoperability Protocols survey (2505.02279) is a qualitative protocol survey [2505.02279]; it shares only the "agentic systems" umbrella and offers no serving comparison.
2. 本篇 vs 相关论文的 delta (What's new, incremental, contradictory) #
New (vs the whole cluster). Scepsy is the only entity here that (a) supports multiple heterogeneous LLMs in one workflow and (b) does automatic GPU allocation (fractional shares + TP + replicas) rather than temporal request scheduling [2604.15186]. Every request-level peer stops at scheduling requests onto a fixed allocation:
- Autellix picks which request runs next but leaves per-LLM GPU allocation to the user and forbids multi-LLM [2502.13965].
- Continuum decides how long to pin KV and program-level FCFS ordering [2511.02230].
- TokenCake decides when to offload/reserve KV blocks [2510.18586].
So Scepsy's delta is a layer shift: from "schedule requests on given GPUs" to "decide the GPU layout itself." Its Aggregate LLM Pipeline (Eqs. 1–2) is a predictor over allocations, not a queue policy [2604.15186].
Incremental. The load-bearing empirical observation — aggregate per-LLM behavior is stable even when per-request behavior is not — is a refinement of a shared cluster intuition. Autellix already exploits that program cumulative service is a usable signal [2502.13965], and Continuum's memoryfulness factor $\eta$ quantifies when program order helps [2511.02230]. Scepsy pushes this to the relative time-share being ~4× more stable than absolute latency [2604.15186], which is the same "steady-state statistics beat per-request prediction" family of insight, applied one level up.
Contradictory (assumption clash with 2511.00739).
Scepsy asserts non-LLM tool time is negligible and drops it from the resource model
[2604.15186], while 2511.00739 measures tool execution at up to 88% of
end-to-end latency across five real workloads [2511.00739].
矛盾根源: different workload scopes and different metrics. Scepsy's workloads are
RAG+reranker and beam search, where the "tools" are embedding/retrieval steps that its
own pipeline treats as fast LLM stages, and where it measures orchestration overhead
directly ("a few ms") [2604.15186]. 2511.00739's workloads deliberately
include CPU-heavy tools (FAISS ENNS over 115GB, RDKit molecule generation, Bash execution)
that are excluded-by-construction from Scepsy's benchmarks
[2511.00739]. Both are correct in their own regime;
Scepsy's own §9 concedes the model breaks for "long external operations"
[2604.15186], which is exactly 2511.00739's domain. The contradiction is
therefore a scope boundary, not a factual error — but Scepsy's blanket phrasing
("at most a few milliseconds") is over-general.
Contradictory (headline-magnitude vs baseline framing). Scepsy reports up to 8.2× throughput over Ayo and 7.3× over Aegaeon [2604.15186], while Continuum reports its gains against vLLM/InferCept/Autellix and explicitly shows Autellix "degrades severely under high load" for lack of KV retention [2511.02230]. These are not directly comparable (different baselines, hardware, workloads) but they jointly imply that no single baseline is Pareto-dominant, and that Scepsy's large multipliers partly reflect weak baselines (Kubernetes autoscaler oscillation, Aegaeon's missing prefix cache) rather than pure algorithmic superiority [2604.15186].
3. 可攻击面 (Adversarial rebuttal against specific claims) #
- "Non-LLM tool time is negligible." This is the softest load-bearing claim. 2511.00739 refutes it empirically for tool-heavy agents [2511.00739], and 2512.15834's entire existence presupposes tool latency is large enough to be worth speculatively hiding (best savings at tool latency 2.0–2.5s) [2512.15834]. If tools take seconds, the Aggregate LLM Pipeline's latency prediction (Eq. 1, a pure sum of LLM stages) is systematically biased low, and the bottleneck identification (Eq. 2) may point at the wrong stage. Scepsy's own §9 mitigation (add fixed-latency tool stages) is unimplemented [2604.15186].
- "Relative per-LLM time shares are stable." Demonstrated on exactly two workloads (RAG+reranker, beam search) [2604.15186]. Autellix's workload taxonomy shows far wilder structures — MCTS with tens-to-hundreds of parallel threads, Map-Reduce fork-join [2502.13965]. Under a heavy-tailed MCTS DAG, the "average parallelism $p_m$" scalar may be a poor summary, and Scepsy's §9 itself flags cross-LLM fan-out being mis-modeled as serial and bimodal LLM reuse as unsolved [2604.15186]. The stability claim is plausibly workload-specific.
- "Ignoring control flow entirely is sufficient." Continuum's contribution is precisely that ignoring program structure (order/turn-count correlation) leaves value on the table — its $\eta$ factor makes FCFS approximate SRTF only when steps are correlated [2511.02230]. Scepsy discards all such structure at the scheduling layer, so at a fixed allocation it cannot recover the per-turn queueing wins Continuum gets (up to 8.18× real) [2511.02230]. Scepsy even concedes Ayo occasionally beats it in the latency-bound region via request batching [2604.15186] — direct evidence that request-level structure still matters at some operating points.
- "Search stays under a minute (≤70s at 128 GPU)." Measured with only 3 LLMs at 128 GPUs [2604.15186], while search time is stated to grow exponentially in both LLM count and GPU count [2604.15186]. The favorable large-GPU number holds the LLM axis at its smallest; a workflow with 5+ heterogeneous LLMs on 128 GPUs is untested and could blow the budget.
- Reproducibility. Code is anonymized ("available post-acceptance", 25K LOC) and inaccessible at ingest [2604.15186]. Two different vLLM versions (v0.17 for Scepsy, v0.2/v0.2.2 for the Ayo run) muddy the Ayo comparison [2604.15186]. Contrast Continuum, which is open-source on top of vLLM [2511.02230], and RouteLLM, which ships a full framework [2406.18665].
4. 生态位 (Paradigm-shift positioning, adoption evidence) #
Scepsy stakes out the allocation layer of the agentic-serving stack, a niche the rest of the cluster leaves to the user. A clean layering emerges:
- Allocation layer (Scepsy): how many fractional GPUs / TP degree / replicas per LLM [2604.15186].
- Request-scheduling layer (Autellix, Continuum): given an allocation, which request/turn runs and when [2502.13965] [2511.02230].
- KV/engine layer (TokenCake, KVCOMM, Continuum's TTL): how to keep/reuse/offload KV blocks [2510.18586] [2510.12872].
- Cross-cut latency hiding (Speculative Tool Calls): overlap tool exec with generation [2512.15834].
- Diagnosis (2511.00739): where the time actually goes [2511.00739].
- Model-choice / interop (RouteLLM, protocol survey): which model, and how agents talk [2406.18665] [2505.02279].
The paradigm move is "aggregate steady-state reasoning as a systems foundation" — echoing RouteLLM (stable difficulty signal in preference data) [2406.18665] and Autellix (stable cumulative service) [2502.13965], but the first to convert stable statistics into a fractional GPU layout. Adoption evidence is thin: no public code yet [2604.15186]; whereas 2511.00739's CPU-bottleneck observation is described as already becoming community consensus [2511.00739]. Scepsy's positioning is strong conceptually but unproven in adoption.
5. 未探索方向 (Hybrid / adaptive directions from the cluster) #
- Scepsy allocation ⊕ Continuum/Autellix request scheduling. Scepsy sets the layout; a program-aware policy (PLAS/ATLAS or KV-TTL) then schedules requests within that layout. Scepsy admits it loses to request-batching at latency-bound points [2604.15186]; Continuum shows per-turn queueing is a first-class, turn-count-amplified cost [2511.02230]. A two-tier system (aggregate allocation + program-level ordering) is the obvious hybrid neither side implements.
- Tool-aware Aggregate LLM Pipeline via speculative tools. Scepsy's §9 sketch (fixed-latency tool stages) [2604.15186] could be upgraded: use 2512.15834's speculative tool execution to shrink the tool stage's fixed bias [2512.15834], and use 2511.00739's CPU/GPU profiling to size it [2511.00739]. This directly neutralizes attack-surface #1.
- CPU-GPU co-allocation. Scepsy allocates only GPU fractions. 2511.00739 argues CPU tool execution and CPU/GPU balance are the real bottleneck for tool-heavy agents [2511.00739]. Extending the search space to include CPU core budgets (MAS-style admission) [2511.00739] is an unexplored dimension of Scepsy's fractional-resource model.
- KV-reuse-aware profiling. Scepsy's per-LLM replay already captures prefix-cache behavior in beam search [2604.15186]. Folding KVCOMM-style cross-agent KV reuse [2510.12872] into the profile would change each LLM's effective throughput curve $T_m$, shifting Eq. 2's bottleneck — an adaptive interaction between engine-level reuse and allocation search that no cluster paper studies.
- Bimodal / multi-role LLM handling via clustered pipelines. Scepsy's §9 proposes classifying traces into recurring patterns and building one pipeline per mode [2604.15186]. RouteLLM's tier-clustering of models by difficulty [2406.18665] is a ready-made technique for that trace classification step.
- [2604.15186] — Scepsy (target): Aggregate LLM Pipeline, fractional GPU allocation search.
- [2502.13965] — Autellix: program-level LAS/ATLAS request scheduling; closest baseline, single-LLM.
- [2510.18586] — TokenCake: dual temporal/spatial KV-cache scheduler for multi-agent tool-use.
- [2511.02230] — Continuum: KV-cache TTL + program-FCFS for multi-turn agents; open source.
- [2511.00739] — CPU-Centric characterization: tools = up to 88% of E2E latency; assumption-breaker.
- [2512.15834] — Speculative Tool Calls: hide tool latency by speculative pre-execution.
- [2510.12872] — KVCOMM: training-free cross-context KV reuse for multi-agent.
- [2406.18665] — RouteLLM: preference-data model routing (which-model axis).
- [2505.02279] — Agent interoperability protocols survey (MCP/ACP/A2A/ANP): weak/adjacent.