AgentOpt introduces client-side optimization for AI agents — model combination selection across pipeline roles via bandit-based search (Matrix UCB-E). Cost gap between best and worst combinations reaches 13–32× at matched accuracy; the strongest standalone model (Opus 4.6) is the worst planner in multi-step pipelines; UCB-E recovers near-optimal accuracy at 62–76% budget savings.
Server-side agent serving systems (Autellix, ThunderAgent, Continuum, AIOS) optimize how a fixed pipeline is served, but cannot optimize which model assignment to use. Client-side optimization — model choice, tool invocation, budget allocation — remains unexplored. Model selection is the dominant lever: cost gap between near-equivalent combinations is 13–32×, far larger than what infrastructure improvements can recover. Performance rankings do not transfer across roles: Claude Opus 4.6, the strongest standalone model, is the worst planner across all 81 HotpotQA combinations because it bypasses the solver's search tools.
Client-side optimization formalized (§2): Pipeline with $N$ roles, candidate model set $\mathcal{M}$. Model combination $\mathbf{c} = (m_1, \dots, m_N) \in \mathcal{M}^N$. Utility: $J(\mathbf{c}) = U(\mathrm{PERF}(\tau(\mathbf{c})), \mathrm{LATENCY}(\tau(\mathbf{c})), \mathrm{COST}(\tau(\mathbf{c})))$. Optimize: $\mathbf{c}^* = \arg\max_{\mathbf{c} \in M^H} J(\mathbf{c})$.
Combo abstraction (§3.1): The unit of optimization must be the full combination, not individual roles. Model quality is not context-free — assigning a model to stage $t$ changes intermediate computation for all later stages. Agent model selection is a black-box optimization problem over pipeline-level configurations, fundamentally different from per-call LLM routing.
Framework-agnostic implementation (§4): Intercepts at HTTP transport layer (patching httpx.Client.send / httpx.AsyncClient.send). Attribution via Python contextvars. HTTP-level response caching (keyed by request payload hash). Two-level concurrency (semaphore per combination × per datapoint).
Ten search algorithms (§5): Brute-force, Random, Matrix UCB-E ($\text{UCB}_i = \bar{s}_i + \sqrt{a/n_i}$ operating on combination-by-datapoint grid), Matrix UCB-E-LRF (low-rank variant), Arm Elimination, Epsilon-LUCB, Threshold Successive Elimination, Hill Climbing (multi-restart local search), Bayesian Optimization (BoTorch, acquisition function $\alpha(\mathbf{c}; g, \mathcal{H})$), LM Proposal (LLM-based shortlisting).
核心技术壁垒: The empirical demonstration that the "best model" is not a global property — Claude Opus 4.6 as planner produces 31.71% accuracy vs 74.27% when Ministral 3 8B plans and Opus solves. This invalidates the assumption underlying per-call routing: model quality must be evaluated at the combination level for multi-step workflows.
| Benchmark | Best Combination | Accuracy | Cost Gap (best vs worst at matched quality) |
|---|---|---|---|
| HotpotQA | Ministral 3 8B (planner) + Opus 4.6 (solver) | 74.27% | — |
| MathQA | Opus 4.6 (answer) + Haiku 4.5 (critic) | 98.84% | 24× |
| BFCL | Opus 4.6 / Kimi / Qwen3 Next (tied at 70%) | 70.00% | 32× |
| GPQA Diamond | Opus 4.6 | 74.75% | — |
Matrix UCB-E at β=0.2: HotpotQA 73.54% (vs 74.27% brute-force), 75.9% cost savings. MathQA 98.37% (vs 98.84%), 71.6% savings.
Opus as planner on HotpotQA: All 9 Opus-as-planner combinations rank 71–81 out of 81; in 7/9 cases role2_never_called — Opus answers from parametric knowledge, bypassing solver tools entirely.
LM Proposal failure: 34.13% on HotpotQA (vs 74.27% ground truth) — LLM priors about model quality are dramatically wrong in multi-step settings.
The optimization loop iteratively selects combinations, executes full pipelines, and aggregates end-to-end metrics. Matrix UCB-E operates on the combination-by-datapoint scoring grid, concentrating budget on promising candidates while abandoning poor ones early.
无形式化作者证明 — 仅实证
The paper formalizes the optimization problem as a black-box combinatorial optimization (§3.1) and frames it as a pure-exploration multi-armed bandit (§5.1), but provides no convergence proofs or regret bounds for the proposed algorithms. The UCB-E analysis relies on standard bandit theory references.
| Check | Status | ||||
|---|---|---|---|---|---|
| Formalization of client-side optimization | ✓ Multi-objective utility $J(\mathbf{c})$ (§2–§3.1) | ||||
| Search space characterization | ✓ $ | C | = \prod | M_i | $ combinatorial (§5) |
| Algorithm convergence guarantees | Not provided (relies on known bandit bounds) | ||||
| Ablation across algorithms | ✓ 10 algorithms compared, 50-seed averages | ||||
| Cross-benchmark validation | ✓ 4 benchmarks spanning different workflow structures | ||||
| Statistical methodology | ✓ 50 random seeds per algorithm-benchmark pair |
Cross-benchmark brute force (Table 1): Exhaustive evaluation costs range from $4.71 (GPQA, 9 combinations) to $123.87 (MathQA, 81 combinations). Best combination often counter-intuitive (Ministral 3 8B + Opus 4.6 on HotpotQA).
Algorithm comparison (Tables 4–7, 50-seed averages):
HotpotQA deep dive (Tables 3, 10): Bottom 11 combinations all use Opus or Haiku 4.5 as planner. Opus-as-planner produces near-identical accuracy (~31.9%) regardless of solver — confirming it bypasses the solver entirely.
MathQA critic role (Table 11): With Opus as answerer, all 9 critic models fall within 2.9 pp range (95.95–98.84%). Critic role is largely decorative in strong-answerer regimes.
| Step | Claim | Evidence | Depends on | ||||
|---|---|---|---|---|---|---|---|
| 1 | Server-side optimization cannot address developer-specific quality/cost/latency tradeoffs | Structural argument: provider optimizes throughput, not per-application utility | — | ||||
| 2 | Model selection is the first-class client-side optimization lever (upstream of all others) | Structural: model assignment determines computational substrate; empirical: 13–32× cost gap | Step 1 | ||||
| 3 | Model quality must be evaluated at combination level, not per model | HotpotQA: Opus best standalone, worst as planner (31.71% vs 74.27%) | Step 2 | ||||
| 4 | Combinatorial search is necessary but exhaustive search is too expensive | $ | C | = | M | ^N$ grows exponentially; brute-force costs $51–124 per benchmark | Step 3 |
| 5 | Matrix UCB-E recovers near-optimal at 62–76% budget savings | Tables 4–7: 50-seed averages across 4 benchmarks | Step 4 | ||||
| 6 | LLM priors about model quality fail in multi-step settings | LM Proposal: 34.13% on HotpotQA vs 74.27% brute-force | Step 3 |
Open-source Python package: https://github.com/AgentOptimizer/agentopt. Framework-agnostic (httpx interception). Agents implemented in LangGraph for benchmarks. 9 models served via AWS Bedrock.
关键实现细节: