Orla: A Library for Serving LLM-Based Multi-Agent Systems — L2 #

1. TL;DR #

Orla is a serving layer that sits between agent-orchestration frameworks and LLM inference engines. It models an agentic task as a DAG of stages and adds three workflow-aware mechanisms — a stage mapper (per-stage model/backend routing), a workflow orchestrator (two-level scheduling), and a memory manager (workflow-scoped KV-cache lifecycle) — cutting latency 38–60% and cost 35% vs. a single-model vLLM baseline.

2. Q1 / Q2 / Q3 #

Q1 — 痛点 (problem). Modern agentic apps are workflows of many LLM calls + tool calls spanning heterogeneous models and backends. Today developers hand-glue orchestration code onto request-level serving engines (vLLM, SGLang). Those engines optimize within a single inference call (prefill/decode scheduling, continuous batching, LRU KV eviction) but are blind to workflow structure. Two gaps result: (a) heterogeneity — different stages want different model sizes / backends / hardware, but the stack has no place to express per-stage mapping; (b) coordination — nothing decides which model runs where, how to schedule across stages sharing a backend, or when a KV-cache entry belonging to a still-running workflow can be evicted. A request-level LRU can evict cache mid-workflow, forcing an expensive re-prefill on the next stage.

Q2 — 方法 (method). Introduce a workflow-level serving abstraction that separates request execution from agent-level policy. Three core abstractions — Stage (one LLM-inference unit that may loop over tool calls), Workflow (a DAG of stages; a stage is eligible once all upstream stages finish), Backend (an OpenAI-compatible HTTP endpoint) — are acted on by three components: (1) Stage mapper assigns each stage a (model, backend, inference params), via explicit developer mapping or dynamic routing (e.g. OneBitStageMapper classifies each request simple/complex with a cheap LLM call and routes light/heavy); (2) Workflow orchestrator runs the plan with two-level scheduling (a stage scheduler picks which per-stage sub-queue to serve, a request scheduler orders within it; both FCFS default, priority/SJF pluggable) plus context builders that assemble each stage's prompt from upstream outputs; (3) Memory manager governs KV cache at workflow granularity using orchestrator signals (stage transitions, backend switches, workflow completion), with three policies: preserve-on-small-increment, flush-at-workflow-boundary, flush-under-pressure.

核心技术壁垒: the load-bearing insight is that KV-cache lifecycle decisions require workflow-level signals that are structurally invisible to a request-level backend — stage dependencies, predicted context reuse between consecutive same-backend stages, backend transitions, and workflow completion. An engine seeing only independent requests cannot know that a just-finished stage's cache will be reused by its DAG successor (so preserve) versus belongs to a completed workflow (so reclaim). Orla's contribution is placing a control plane exactly where these signals exist. (See §7.)

Q3 — 结果 (results). On SWE-bench Lite (2,294 requests) with OneBitStageMapper splitting traffic (956 to Qwen3-4B, rest to Qwen3-8B) across two vLLM backends: wall-clock time −38%, mean completion time −60%, estimated inference cost −35% vs. an all-Qwen3-8B vLLM baseline. On DAG-MATH (first 5 problems) with a Qwen3-8B SGLang backend: workflow-level cache management (flush-per-workflow) shifts the entire TTFT CDF left vs. flush-per-request.

3. 架构 / 方法图 #

Paper's Figure 1 ("Orla design"). This is the primary architecture view: developer-defined workflows (DAGs of stages) enter the Orla layer, which houses the stage mapper, workflow orchestrator, and memory manager, and dispatches to heterogeneous backends (SGLang / vLLM / Ollama) through a uniform OpenAI-compatible HTTP interface. Note that Orla is a control plane — actual inference happens entirely in the underlying engines.

Paper's Figure 4 ("The workflow of our running example"). The concrete DAG that grounds every design section: classify (light backend) fans out to policy_check and route_ticket (heavy backend), and policy_check → reply (heavy). This is what a "workflow" literally is in Orla, and it shows heterogeneity in one graph — a light model for cheap metadata extraction, a heavy model for the policy/reply reasoning.

Because Orla is fundamentally a per-turn agent loop wrapped by DAG scheduling, the intra-stage control flow is best shown as a state machine the original figures do not draw:

Planning & reasoning. Planning style is a predetermined DAG workflow, not ReAct or tree search — the developer authors the graph; Orla does not synthesize plans. Decomposition is top-down (task pre-split into stages). Within a stage the model runs a bounded ReAct-like tool loop until it stops emitting calls. There is no backtracking / undo across stages; the DAG is executed forward only. Budget knobs are per-stage inference params (max output length 4,096 tokens in the mapping experiment, 256 in the cache experiment).

Memory model. Short-term = the KV cache / context prefix shared across consecutive same-backend stages; there is no long-term vector store or episodic trajectory log — "memory" here means inference state (KV cache), managed at workflow scope. Error recovery across stages is not addressed; the paper describes no fallback state when a tool or stage fails.

4. 作者证明 #

无形式化作者证明 — 仅实证. The paper presents no formal model, convergence result, or success-rate bound; it is a systems/library paper validated purely empirically. This is notable given the authors' queueing-theory background.

Agent-specific reconstruction against the category checks:

• Notation / metrics. Reported metrics are wall-clock duration, per-request completion time (CDF), estimated inference cost, and TTFT (CDF). No symbol table exists because there are no equations; the simulated backend models prefill/decode latency parametrically but the paper does not publish that model's closed form.
• Success-rate sweep matrix (task difficulty, planning depth, tool set, backbone). Not swept. Only two axes vary across two disjoint experiments: {routing on/off} on SWE-bench Lite, and {cache flush-per-request vs. flush-per-workflow} on DAG-MATH. Planning depth (DAG shape) and tool set are held fixed; backbone is fixed to the Qwen3 family. So monotonicity along the guide's four axes cannot be verified from the paper.
• Latency budget per turn. The paper claims latency reductions (−38% wall-clock, −60% mean completion) but does not decompose a turn into tool-call + inference + memory-read, and makes no explicit "interactive latency" claim with absolute numbers.
• Failure-mode classification. Zero failure classes are enumerated; no negative results or cases where the single-model baseline wins are shown — itself a gap.
• What could have been bounded: given the OneBitStageMapper routes r fraction to the light model, an expected-cost model $E[c] = r \cdot c_{light} + (1-r) \cdot c_{heavy}$ and an expected-latency mixture are analytically available but the paper reports only the empirical 35% / 38% aggregates rather than deriving or fitting them.

Six minimum checks:

1. Baseline identity — all requests on Qwen3-8B / vLLM; consistent across §4 and Appendix C. ✔
2. Routing split — 956/2,294 → Qwen3-4B; arithmetic consistent (1,338 to heavy). ✔
3. Cost mechanism — 35% cost drop attributed to cheaper per-token light model; plausible given ~42% of traffic offloaded. ✔ (directional, not derived)
4. Cache mechanism — TTFT improvement attributed to avoided re-prefill via preserved prefix; mechanism matches the LRU-mid-workflow-eviction problem stated in §2.2. ✔
5. Hardware feasibility — both models co-resident on one 96 GB GPU at 60%/35% split (=95%, leaves headroom). ✔
6. Reproducibility knob — temperature 0 in both experiments. ✔

5. 实验与数据 #



Paper's Figure 2 ("(a) Wall-clock time (b) Cost (c) Completion time"). The load-bearing result for the stage mapper: (a) total wall-clock −38%, (c) mean completion time −60% (shown as a completion-time CDF that Orla shifts left), (b) estimated cost −35%. Reader should notice the mapper wins on all three axes simultaneously because offloading ~42% of requests to the 4B model both frees the 8B backend (latency) and lowers per-token price (cost).



Paper's Figure 3 ("Cache management (request vs. workflow flush)"). The load-bearing result for the memory manager: the CDF of TTFT for flush-per-workflow lies entirely to the left of flush-per-request, i.e. the whole distribution improves, not just the mean. Caveat the reader should notice: this is on only the first 5 of 2,894 DAG-MATH problems, and no absolute TTFT, hit-rate, or memory-savings numbers are given.



Paper's Figure 6 (two-level scheduling illustration). Shows how ready requests are organized: each backend holds a queue partitioned into per-stage sub-queues keyed by stage ID; the stage scheduler picks a sub-queue, the request scheduler picks within it, and priority hints derived from classify output re-order downstream stages. This is design illustration — the scheduling component is not quantitatively evaluated anywhere in the paper.

Setup (Appendix C). Single machine: 2× AMD EPYC 7313 (64 cores), 528 GB RAM, one NVIDIA RTX PRO 6000 Blackwell Max-Q (96 GB VRAM), CUDA 13.0. Mapping experiment: vLLM co-serves Qwen3-8B (60% GPU mem) and Qwen3-4B (35%), 32,768-token context, 4,096 max output. Cache experiment: Qwen3-8B on SGLang (needs its KVCache management API), 256 max output. Temperature 0 throughout.

6. 论证链 #

Step	Claim (paper-internal)	Support
1	Agentic workloads are multi-stage workflows with heterogeneous per-stage needs (model size, backend, complexity).	§1 examples: routing/summarization vs. synthesis/code-gen; deployments span SGLang/vLLM/Ollama.
2	Request-level serving engines optimize within a call but cannot coordinate across stages, models, and backends.	§1: prefill/decode scheduling & LRU eviction are per-request and workflow-unaware.
3	Therefore a serving layer above the engine and below orchestration frameworks is needed, separating request execution from workflow policy.	§1 key insight; Appendix D positioning vs. LangGraph/AutoGen (above) and vLLM/SGLang (below).
4	That layer needs exactly three mechanisms: stage mapper, workflow orchestrator, memory manager.	§2 design; each maps to one gap dimension (heterogeneity, coordination, memory reuse).
5	A uniform OpenAI-compatible HTTP backend interface makes backends plug-and-play (swap = change endpoint URL).	§1, §3: same workflow runs on GPU (SGLang/vLLM) or laptop (Ollama) via one env-var change.
6	Instantiating the mapper and memory manager yields measurable latency/cost/TTFT wins over a single-model baseline.	§4: Fig 2 (−38%/−60%/−35%), Fig 3 (TTFT CDF).

7. 实现 cross-reference #

Open-source at github.com/dorcha-inc/orla; the L1 records API code listings but no file:line locations, so [实现未公开] at line granularity. Key API surface reconstructed from Appendix A–B:

• Backend clients: orla.NewOrlaClient, orla.NewSGLangBackend(model, url), orla.NewOllamaBackend(model, url), orla.NewSimulatedBackend(name, url) (Appendix A, B.1).
• DAG assembly: wf.AddDependency(downstream.ID, upstream.ID), wf.Execute(ctx) (Appendix A).
• Scheduling: stage.SetSchedulingPolicy("priority"), SetRequestSchedulingPolicy("fifo"), stage.SetSchedulingHints(&orla.SchedulingHints{Priority: &p}) (Appendix B.2).
• Context: stage.SetPromptBuilder(func(upstream map[string]*orla.StageResult)(string,error){...}) reading upstream[classify.ID] (Appendix B.3).
• Memory: wf.SetMemoryPolicy(orla.NewFlushAtBoundaryPolicy()), stage.SetCachePolicy("flush"|"preserve") (Appendix B.4).

核心技术壁垒 (dedicated). The single hardest-to-replicate insight is workflow-scoped KV-cache lifecycle management. Re-implementing the stage mapper or DAG scheduler is routine; the differentiator is that the memory manager must (a) observe orchestrator signals a request-level engine never exposes — the DAG edge from a finishing stage to its successor, whether that successor lands on the same backend with a shared prefix, and whether the workflow has completed — and (b) translate those into preserve/flush commands the backend actually honors, which requires a backend that exposes a KV-cache management API (why the cache experiment runs only on SGLang, not vLLM). The barrier is thus half design (surfacing the signals) and half backend-capability dependence.

关键实现细节 (easy-to-miss tricks).

1. Backend changes are detected automatically — the explicit SetCachePolicy("flush") on the light→heavy boundary "is not strictly needed, as Orla's memory manager will recognize a backend change automatically" (Appendix B.4); the manual hint is redundant defensive coding.
2. The simulated backend parametrically models prefill/decode latency so orchestration behavior can be studied on CPU-only machines with no GPU, decoupling scheduling/memory experiments from real inference variance (§1) — this is what lets the live demo run the full workflow on a laptop via Ollama.