Target: SMetric — balanced session-centric scheduling for agentic LLM serving. Peers in scope: RouteLLM (2406.18665), Autellix (2502.13965), Agent-protocol survey (2505.02279), Halo (2509.02121), KVCOMM (2510.12872), TokenCake (2510.18586), CPU-centric characterization (2511.00739), Continuum (2511.02230).
All nine entities sit in category: agent, but they split into a serving-systems cluster (the true peers of SMetric)
and a non-serving fringe that is related only by domain framing.
Tight cluster — cluster-level LLM serving for agents (share SMetric's exact problem space):
for agent programs, and it also uses KV-cache locality to drive load-balancing: long calls route to the "primary
engine" holding the session's cache, short calls spread to the least-loaded instance [2502.13965].
This is the same intuition SMetric formalizes as "first-request balancing + follow-up stickiness"
[2607.08565]. But Autellix keeps a stateful global process table, whereas SMetric's headline
novelty is doing this statelessly.
6–11 turns), same core observation that fast tool returns make KV eviction wasteful, and it explicitly benchmarks
against Autellix (PLAS) [2511.02230]. Continuum attacks when to keep a session's KV
(temporal pinning); SMetric attacks where to place it (spatial routing). Complementary halves of the same problem.
tool-call idle windows) and a spatial scheduler (reserved/shared partitioning) [2510.18586]. Overlaps
SMetric on the spatial axis (who gets GPU capacity) but frames the pain as idle-window memory waste, not
cluster-wide TPS imbalance.
cluster-level worker-placement problem with KV-reuse-aware cost [2509.02121]. But
Halo is offline/batch and clairvoyant (it sees the whole workflow DAG up front), the mirror image of SMetric's
online/non-clairvoyant, single-request router.
Related-by-diagnosis (empirical peers, not method peers):
bottleneck conclusion: CPU tool execution dominates E2E latency (up to 88%) [2511.00739], directly contra SMetric's GPU-side, KV-reuse-dominant framing. This is the sharpest cross-paper tension (see §3).
Fringe (domain-adjacent, orthogonal method):
[2510.12872]. Operates at the KV-tensor-approximation layer, below the scheduler; would compose
with, not compete with, SMetric.
instance routing. Both are "routers" and both are stateless single-gate classifiers, but they route in orthogonal
dimensions (which model vs which instance).
[2505.02279]; unrelated to serving-layer scheduling except that both treat "the agent is an unmodified
external client" as a design constraint.
What is genuinely new in SMetric (vs the cluster):
[2607.08565]. Every peer that exploits session/program structure keeps state: Autellix maintains a
global process table [2502.13965], Continuum maintains a Tool Call Handler with per-tool
history $S[f]$ [2511.02230], TokenCake requires a frontend graph registration API where users
declare the agent DAG [2510.18586], and Halo requires the full workflow DAG up front
[2509.02121]. SMetric alone infers turn purely from the count of historical messages
already carried in the OpenAI-style request, needing no session→instance table, no client-side API change, and no
garbage collection as sessions silently idle. This is a real paradigm difference, not an increment.
the first-turn instance under cache-aware routing vs only 4.0% under load-balancing
[2607.08565] — is a mechanism no peer articulates. Autellix's locality-aware routing has the same
effect (session stickiness) but treats it as a load-balancer heuristic, not as the thing that reduces the balancing
problem to a small fraction of requests.
within 7% of the best baseline once the global tier is fully provisioned) [2607.08565] reframes the
whole reuse-vs-balance trade-off. TokenCake and Continuum both treat KV as pinned to one instance (offload to CPU
and back, or TTL-pin in place); neither exploits a cluster-wide global tier that lets a re-routed session recover
its cache remotely.
Incremental / shared with peers:
characterize→optimize pipeline [2511.00739]; SMetric's contribution
over it is the specific findings (>80% reuse, 67% intra-session, ~90% reuse within 100s).
[2607.08565] echoes Autellix's move from per-call to program-level throughput
[2502.13965] and Continuum's per-job (not per-token) delay metric [2511.02230]. SMetric
states it most crisply but is not first to the idea.
Contradictory:
different workload mix: that CPU tool execution is the E2E bottleneck (up to 88%), so GPU-side scheduling gains are
capped at ~12% [2511.00739]. See §3 for the reconciliation.
Attack 1 — "the global tier decouples reuse from placement" over-generalizes from a favorable testbed.
SMetric's Insight 1 rests on the global tier serving fetches "in time," and the paper itself admits the fully
provisioned tier is "never the bottleneck" only for the model analyzed in §4.1
[2607.08565]. The 235B experiment already shows the load-balance-only baseline losing 32% TPS because
"a larger model puts more pressure on the global tier" [2607.08565]. On 200 Gbps RDMA NICs and 4 servers, the
global tier is generous; on a bandwidth-starved cluster (or one without GPU-direct RDMA), the 7% "reuse ≈ placement"
result could invert. TokenCake's own numbers reinforce the risk: even an in-node offload/upload round-trip is ~63.7 ms
for 4096 tokens [2510.18586] — a remote fetch under contention would be worse, directly eating the
"small TTFT bump agents tolerate."
Attack 2 — the stateless turn signal is fragile to real agent behavior.
SMetric infers turn from the number of historical messages [2607.08565]. But KVCOMM shows that in
multi-agent workflows agents rewrite and compress context (compaction requests appear even in SMetric's own Figure 7),
and forked sub-agents share a parent's context [2510.12872]. When the carried history is compacted or
forked, "message count" no longer maps cleanly to "turn," so est_hit can mis-fire and force needless rebalancing. The
paper's benign-failure argument ("dropped history → treat as first request, which has little reuse anyway") does not
cover the compaction case, where the request genuinely does have large reuse but a shortened message list.
Attack 3 — the load-balance metric is consistently the worst.
In every reported setting SMetric's max/mean load ratio is worse than pure load-balance-only (2.6× vs 2.0×
colocation-30B; 1.6× vs 1.3× 235B) [2607.08565]. SMetric's defense is that TPS, not balance, is the goal
— but Autellix's Fig. 18 shows a visible gap to the SRPT optimal even for its throughput metric
[2502.13965], and SMetric never bounds how far its greedy turn-gated policy sits from an offline
optimal dispatch. The 235B "close ratios are partly a setup artifact (only 8 instances)" admission
[2607.08565] suggests the balance advantage may shrink further at the small-cluster scales many providers actually run.
Attack 4 — "no formal guarantee" is a shared but real weakness.
Like Autellix (no competitive ratio vs SRPT) [2502.13965] and Continuum (no convergence proof for
$\tau^*$) [2511.02230], SMetric offers no bound on global-tier fetch demand as a function of the
first-request fraction — the very quantity its scalability argument depends on
[2607.08565]. The claim "first requests are few, so the global tier suffices" is asserted from trace
statistics, not derived; a workload with many short single-turn sessions (higher first-request fraction) is an
un-analyzed adversarial regime.
SMetric occupies the stateless-router-for-agentic-serving niche, and its positioning is best understood on a
2×2 of (state kept: none ↔ full) × (control axis: spatial routing ↔ temporal retention):
pools) [2510.18586], Halo (full DAG, offline) [2509.02121].
Paradigm claim: SMetric argues the field has been over-fitting schedulers to a false trade-off (reuse vs balance)
carried over from chat serving, where the global tier didn't dominate. Its shift — *treat the first request as the only
balancing decision, and let the global tier absorb the rest* — is a genuine reframing, comparable in spirit to Autellix's
"lift LAS from per-call to per-program" [2502.13965] and Continuum's "queueing delay,
not reload cost, is the real enemy" [2511.02230].
Adoption evidence (strong for a research paper): SMetric is the production scheduler comparison itself — it is
built on and evaluated against the Bailian production scheduler at "one of the world's largest token providers," on
vLLM + LMCache + Mooncake (the de-facto open stack), and promises to open-source both code and traces
[2607.08565]. This is materially stronger adoption grounding than the peers: Autellix,
TokenCake, and KVCOMM are all [实现未公开] [2502.13965]
[2510.18586] [2510.12872], whereas only Continuum
[2511.02230] and RouteLLM [2406.18665] have public repos.
SMetric's "at time of ingestion, promised on publication" status sits between the two.
decides how long a session's KV survives between turns [2511.02230]. A combined router would place
for balance and TTL-pin the stuck follow-ups, so that SMetric's session_not_evicted guard fires far less often
(fewer needless rebalances). Neither paper explores giving the stickiness decision a cost-benefit TTL.
OVERLOAD/HIT_RATIO are static[2607.08565]. Borrowing 2511.00739's throughput-gain-ratio idea [2511.00739], the router could measure the realized global-tier fetch latency
online and dynamically shift how aggressively it balances first requests — balancing fewer of them when the global
tier saturates (exactly the 235B regime where load-balance-only collapsed).
anchor-based offset estimation [2510.12872] could supply a content-similarity estimate of the true
reusable prefix, making est_hit robust to compacted/forked histories — closing Attack 2.
tool execution can dominate E2E latency [2511.00739]. A session that will issue
heavy CPU tool calls between turns leaves its instance idle — a signal SMetric ignores. A router that co-schedules the
expected CPU idle window (à la TokenCake's temporal offload gate [2510.18586]) with instance
placement is an open hybrid.
RouteLLM routes to a model assuming one instance pool [2406.18665]. In a mixed-model agentic
cluster (planner=large, executor=small), a router that jointly picks (model tier, instance) while preserving
stateless turn inference is unexplored and directly serves the multi-model agent trend both papers gesture at.
the DAG is known [2509.02121]; SMetric proves a cheap stateless heuristic works online. Using
Halo-style offline placement to calibrate SMetric's guards per workload class (rather than hand-tuning
OVERLOAD/HIT_RATIO) could close the unquantified gap-to-optimal noted in Attack 3.