Agent Serving Pareto Explorer

Interactive visualization of throughput–latency trade-offs for multi-turn agent workloads with prefix caching. Based on closed-loop queuing theory and operator-level performance modeling.

Scenarios

Model

Hardware A

Hardware B

Agent Workload

Turns per Session (N) 10

Inter-turn Pause Z (s) 2.0

Over-commit α 1.00

Tool Result Tokens (ΔL) 2000

Output Tokens / Turn (OSL)

Initial Prompt (L₀) 16K

Prefix Cache Hit Rate 90%

Derived Metrics

HW-A cache (solid) HW-A no cache (dashed) HW-B cache (solid) HW-B no cache (dashed)

Key Equations — Closed-Loop Agent Serving Model

Closed-Loop Throughput (Little's Law)

X_turn = C / (R + Z)
X_task = X_turn / N

Per-Turn Latency

R(j) = TTFT(L_prefill) + TPOT(B_eff) × OSL
L_prefill = h·ΔL + (1−h)·L(j)

TPOT (Decode Step Time)

TPOT = max(T_weight + T_kv, T_compute)
T_weight = ModelBytes / BW
T_kv = B × L_avg × KV_bytes / BW

Active Ratio & GPU Utilization

ρ = R / (R + Z)
B_eff = C × ρ
GPU util = min(B_eff, B_max) / B_max

KV-Cache Memory Constraint

C_max = M_avail / (L_avg × KV_bytes_per_token)
L_avg = L₀ + (N/2) × (OSL + ΔL)

Prefix Cache Benefit

TTFT_cache = f(ΔL) ≪ TTFT_full = f(L_avg)
Speedup = R_nocache / R_cache