Pancake: Hierarchical Memory System for Multi-Agent LLM Serving — L2 #

1. TL;DR #

Agent memory turns every generation step into a frequent, interleaved ANN search/insert against a growing vector index; at scale this consumes 82%+ of runtime. Pancake attacks this with three coordinated tiers — locality-aware multi-level cache (FSM-driven), a hybrid graph unifying multi-agent coarse indexes, and dynamic GPU–CPU coordination — cutting memory-op share to ~3.2% and delivering >4.29× average end-to-end speedup.

2. Q1 / Q2 / Q3 #

Q1 — 痛点 (pain). Memory-based agents (MemGPT, A-Mem) perform three operations per step: LLM generation, memory search (retrieve relevant items), and memory update (insert/delete/modify). Both search and update are ANN queries over a vector index. Unlike static RAG, the index is highly dynamic: small-batch inserts interleave with searches at every step. Existing vector DBs are either static-optimized (Faiss, SPANN, DiskANN) or batch/periodic-update oriented (Quake, SPFresh, IncrementalIVF, FreshDiskANN) — both mismatched to agent workloads. Consequence: as memory grows, query latency dominates, reaching >99% of end-to-end runtime at scale. Three specific failures: (i) in-place small-batch inserts scatter semantically-close vectors across up to 175 clusters (high-dimensional shell effect); (ii) multi-agent wide-scope queries must traverse every agent's coarse index, exceeding 80% of latency at 20 agents; (iii) static GPU caching is infeasible because hotspot clusters are also frequently updated, and CUDA cannot cheaply expand cached lists.

Q2 — 方法 (method). A three-tier memory system:

1. Pattern-driven multi-level cache (L0 tiny/recent, L1 intermediate top-k′ neighborhood, L2 stable coarse). Access patterns modeled as a per-agent Finite-State-Machine over semantic cluster states; a similarity score matches incoming request embedding sequences to FSMs to drive search reordering, early termination, and background prefetching overlapped with LLM generation.
2. Hybrid graph connecting each agent's coarse index and the static memory index into one HNSW-like structure, with probabilistic inter-graph "portal" nodes for single-traversal cross-scope coarse search; plus per-cluster agent profiles that record each agent's recently-accessed vector IDs to align fine search to that agent's distribution.
3. Dynamic GPU–CPU coordination: hotspot clusters cached on GPU, a per-cluster CPU insertion buffer (size $B_{insert}=128$) absorbing updates so cached GPU clusters stay stable, and fully asynchronous cluster-expansion (double-buffer allocate + async transfer) for consistency without stalls. On-GPU K-means kernel onloads cluster splitting.

核心技术壁垒 (core moat). The FSM-based agent access-pattern abstraction and its use to convert a dynamic ANN problem into a predictable cache-locality problem. Recognizing that agent memory accesses exhibit both intra-agent and inter-request step-wise locality (planning/tool-call/reflection steps of different requests cluster together), then encoding that as a lightweight online-constructed FSM over cluster states, is the single hardest-to-replicate insight — it is what lets caching, early termination, and prefetching all work without a learned model. See §7.

Q3 — 结果 (results). >4.29× average end-to-end speedup (range 1.12×–26.18×) over A-Mem/MemGPT/LlamaIndex/LangMem; memory-op share cut to avg 3.2% (memory-op-only speedup >6.81×). Mixed two-agent degradation ≤9.8% (baselines: 29.9%–55.9%); near-linear scaling to 20 agents (≤10.2% degradation). vs vector-DB baselines: 1.9×–4.2×, +2.2× with GPU (>3.9× total). Hybrid graph: >20× coarse-search cost reduction. GPU: up to 1.92× with only 5–15 GB cache.

3. 架构 / 方法图 #

Overall workflow — where memory cost comes from.



Paper's Figure 1, verbatim (caption: "Memory-based workflow of agentic LLMs"). The figure motivates the whole system: each agent step issues a search + update against the memory store, and the operational cost of these ANN operations grows with memory size until it exceeds 82% of total execution time for MemGPT/A-Mem-style workflows. The reader should notice that memory is not a side channel — it is on the critical path of every generation step.

The scattered-cluster problem (Tier-1 motivation).



Paper's Figure 4, verbatim (caption: "Direct in-place updates scatter the new vectors into a large number of existing clusters ... A naive solution is to leverage intra-agent locality and maintain dedicated clusters for an agent"). Panel (a) shows inserted vectors dispersed into up to 175 clusters despite semantic coherence; panel (b) shows intra-agent centroid distance is far smaller than distance to existing global clusters. This is why a locality-aware cache (not in-place insertion) is required — the naive per-agent dedicated cluster is the starting point that §4 refines with step-wise structure.

Tier-1 mechanism — three-level index cache with FSM.



Paper's Figure 9, verbatim (caption: "Three-level memory index cache to optimize search efficiency, with FSM-based modeling for access patterns"). Each upper level is a subset of the level below and tracks progressively hotter/finer structure: L0 holds $N_p$ tiny clusters of the most recently accessed vectors (temporal locality), L1 caches top-$k'$ neighborhoods, L2 is the stable coarse index. Search always starts at L0 and descends only when early termination fails.

Tier-2 mechanism — hybrid graph across agents.



Paper's Figure 10, verbatim (caption: "Multi-agent index management with hybrid graph and agent-specific pattern profiling on shared clusters"). Static memory and each agent's local memory share one graph; inter-graph portal nodes (probability $1/ef_{connect}$) let a cross-scope query start in the static index and hop into the relevant agent graph in a single greedy/BFS traversal, avoiding per-agent index scans. Agent profiles annotate shared clusters with each agent's own access order.

Tier-3 mechanism — GPU-CPU coordination.



Paper's Figure 11, verbatim (caption: "GPU-CPU coordinated index management to enable hotspot cluster computation acceleration"). The CPU-side insertion buffer absorbs frequent small inserts (keeping the GPU-cached hotspot cluster immutable during serving), while the GPU-side manager handles hotspot caching and asynchronous cluster expansion. Searches on a buffered cluster merge the GPU-cached portion with the CPU buffer in parallel, so latency matches single-GPU execution.

The agent turn as a memory-access state machine. Pancake's control flow per turn is best drawn as a state machine, since the FSM and early-termination logic gate level descent:

stateDiagram-v2 [*] --> Observe Observe --> MatchFSM: embed request seq v_1..v_t MatchFSM --> SearchL0: predict target L0/L1 cluster, reorder SearchL0 --> EarlyReturn: top-k all < alpha_et * d_agent SearchL0 --> SearchL1: else descend SearchL1 --> EarlyReturn: threshold met SearchL1 --> SearchL2: else descend SearchL2 --> EarlyReturn EarlyReturn --> Generate: return top-k to LLM Generate --> Update: insert new memory item(s) Update --> Prefetch: predict next clusters, background load Prefetch --> UpdateFSM: on request completion, merge/create FSM UpdateFSM --> [*]

The recovery path is soft: if early termination returned a low-recall set, an optional background verification mode completes the full search after return and adjusts $\alpha_{et}$ — there is no hard rollback of an action, only threshold self-tuning.

4. 作者证明 #

无形式化作者证明 — 仅实证. Pancake has no convergence, recall, or success-rate guarantee; all claims are empirical. What could have been bounded but was not: (a) worst-case coarse-graph traversal cost as a function of agent count and portal density; (b) recall loss induced by early termination as a function of $\alpha_{et}$; (c) staleness bound on the CPU insertion buffer before merge. The paper offers three heuristic formulas rather than proofs; the checks below verify their internal consistency and physical sense.

Notation table.

Symbol	Meaning
$P=(S,T)$	per-agent FSM: cluster-state set $S$, directed transition set $T$
$(c,\delta)\in S$	cluster state: centroid $c$, avg intra-cluster deviation $\delta$
$v_{1:t}$	embedding sequence of the current request's memory accesses
$\mathrm{sim}(P_i,v_{1:t})$	match score of request to pattern $i$
$\alpha_{et}$	early-termination threshold coefficient (0.6–0.8)
$d_{agent}$	avg top-$k$ distance over recent queries of the same agent
$ef_{connect}$	inter-graph portal connection probability
$\alpha_{ic}$	inter-connection scaling coefficient (4–8)
$d_{static},d_{agent}$	avg centroid spacing in static / agent index
$B_{insert}$	CPU insertion-buffer size (128 on their platform)
$N_p, N_S, d_{merge}$	max cached patterns, max FSM states, min merge distance

Equation physical meaning.

• FSM: $P=(S,T),\ (c_i \rightarrow c_j)\in T$ — an agent's memory behavior is a finite automaton over semantic clusters; $\delta$ per state captures how tight the cluster is.
• Similarity: $\mathrm{sim}(P_i,v_{1:t}) = \sum_{k=1}^{t} I\big[(c_{k-1}\rightarrow c_k)\in T_i\big]\cdot \frac{\delta_k}{1+|c_k-v_k|}$ — rewards requests whose step transitions follow known FSM edges (indicator) and land close to the expected centroid (distance-weighted deviation). High score ⇒ confident cluster prediction ⇒ aggressive reorder/prefetch.
• Portal density: $ef_{connect}=\min\!\big(\alpha_{ic}\cdot \frac{d_{static}}{d_{agent}},\,1\big)$ — sparse portals suffice when the static space is much broader than the agent's; denser portals avoid cross-graph local minima when densities match; cap at 1.

6 minimum checks.

1. Dimensional sanity of sim: $|c_k-v_k|$ is a distance, $\delta_k$ a distance; ratio $\frac{\delta_k}{1+|c_k-v_k|}$ is dimensionless-scaled and bounded in $(0,\delta_k]$ — well-defined, never divides by zero (the $+1$). ✓
2. Monotonicity of sim: closer prediction ($|c_k-v_k|\to 0$) raises each term toward $\delta_k$; off-path transitions ($I=0$) zero their term. Directionally correct. ✓
3. ef_connect bounds: it is a probability, and $\min(\cdot,1)$ enforces $\le 1$; positive since $\alpha_{ic},d>0$. Valid probability. ✓
4. ef_connect limit: as $d_{static}\gg d_{agent}$, argument $>1$ ⇒ clipped to 1 (dense portals into a huge static index) — matches "similar density needs denser connections" only when ratio small; consistent with stated intent. ✓
5. Buffer crossover: $B_{insert}=128 \le$ the ~256–512 GPU-advantage crossover from Fig 8, so CPU search on the buffer is genuinely cheaper than a GPU launch — parameter choice is internally justified. ✓
6. Early-term self-consistency: skipping the next level when all top-$k < \alpha_{et}\cdot d_{agent}$ with $\alpha_{et}<1$ means it only stops when candidates are better than the agent's recent average — a defensible stopping rule; verification mode catches misses. ✓

Success-rate / sweep note. No task-success metric (this is a serving-system paper, not a task-agent paper); the "sweep" is over (workload pattern × dataset × backbone × agent count), monotone in the expected direction (more agents / broader datasets ⇒ larger but still-bounded degradation). Latency budget: the design explicitly overlaps prefetch and CPU-buffer search with LLM generation so per-turn memory latency hides under inference — this is verified indirectly via end-to-end throughput, not a per-turn breakdown. Failure classes: three are identified (scattered clusters, multi-agent coarse blow-up, dynamic GPU-cache infeasibility) and each tier targets exactly one; the dominant one at scale is coarse-search blow-up (>80% latency at 20 agents), addressed by the hybrid graph.

5. 实验与数据 #

Single-agent end-to-end throughput (primary case study).



Paper's Figure 12, verbatim (caption: "End-to-end throughput comparison between Pancake and other agentic frameworks, in a single agent scenario across four different access patterns ... vLLM for Llama models and API calls for GPT-5"). This is the load-bearing result: across four workload patterns (One-Search-One-Insert, Step-Search-Then-Insert, Search-Then-Step-Insert, Search-Only) and six datasets, Pancake sustains stable throughput while baselines degrade with memory size. Speedups span 1.12×–26.18× (avg >4.29×); notice the biggest wins are on search-dominant patterns where baseline suboptimal indexes hurt most, and the near-tie (1.12×) on some Search-Only configs where there is little to optimize.

GPU-CPU crossover (Tier-3 motivation and $B_{insert}$ justification).



Paper's Figure 8, verbatim (caption: "Comparison of operation costs on GPU and CPU, including (a) search time and (b) data transfer and allocation ... on the MS MARCO dataset"). Panel (a) shows the ~256–512 vectors/cluster crossover where GPU search becomes >3× faster (GPU latency is flat because kernel-launch dominates, not compute); panel (b) shows on-demand transfer cost far exceeding compute — the reason static caching plus retransfer is prohibitive and why the CPU insertion buffer ($B_{insert}=128$, below the crossover) exists.

Additional quantitative results (figures not embedded here but load-bearing): Fig 13 mixed two-agent workload (baseline degradation 29.9%–55.9% vs Pancake ≤9.8%); Fig 14 scaling to 20 agents (≤10.2% degradation); Fig 15 vs vector-DB baselines (1.9×–4.2×, +2.2× GPU); Fig 16 recall–latency tradeoff (IVF must scan up to 128 clusters for recall >0.9; Pancake keeps recall high at lower latency); Fig 17 ablation — multi-level cache stabilizes scan cost early for up to 2.23× latency reduction; Fig 18 — hybrid graph >20× coarse-search reduction, +11.6% then +21.8% compute reduction with agent profiles; Fig 19 — GPU up to 1.92× with 5–15 GB cache plateau.

6. 论证链 #

#	Claim	Support (paper-internal)
1	Agent memory is a dynamic ANN problem, not static RAG	§2.1 three-operation model; §1/§2.1 memory cost reaches >82% / >99% of runtime as index grows
2	In-place small-batch insertion scatters semantically-close vectors	§3.1 Fig 4: up to 175 clusters, 38%–100% accessed <5%; high-dimensional shell effect
3	Agent accesses have exploitable intra-agent + step-wise locality	§3.1 Fig 5: PCA shows step-wise clusters; intra-agent centroid distance ≪ global
4	⇒ a locality-aware multi-level FSM cache beats in-place/split	§4.2 design; §6.4 Fig 17 stabilizes early, up to 2.23× latency reduction
5	Independent per-agent indexes make coarse search blow up	§3.2 Fig 7(a): >80% latency at 20 agents
6	⇒ hybrid graph gives single-traversal coarse search	§4.3 design; §6.4 Fig 18(a) >20× coarse-search cost reduction
7	Static GPU caching fails under frequent updates	§3.3 Fig 8(b): transfer ≫ compute; CUDA list expansion costly
8	⇒ CPU insertion buffer + async expansion enables dynamic GPU caching	§4.4 design; §6.4 Fig 19 up to 1.92× with 5–15 GB
9	All three tiers together ⇒ headline end-to-end win	§6.2 Fig 12: >4.29× avg, memory share to 3.2%

7. 实现 cross-reference #

[实现未公开] — the paper describes a Python interface (search, insert, update, delete with explicit memory-scope arguments; init loads from existing Faiss/IVF indexes) and a multithreaded runtime (shared-read/exclusive-write cluster locks; dedicated search / update / cache-management / GPU-management thread pools; asynchronous invocation for overlap with LLM calls and compatibility with HedraRAG/RAGCache/PipeRAG), but no code repository or file:line references are provided in L1.

核心技术壁垒 (deep dive). The moat is the online, lightweight FSM construction over semantic cluster states. Classical sequence models (PCA, HMM) are too expensive for high-dimensional online streams, so Pancake uses a heuristic: on request completion, match against existing FSMs; if none matches, create one FSM where each access is an independent state, then merge states by max-state-count $N_S$ and min-merge-distance $d_{merge}$; if FSM entries exceed $N_p$, merge the two most similar FSMs. This turns an unbounded dynamic-index problem into a bounded, self-organizing cache-prediction problem — replicating it requires the insight that agent trajectories are step-wise-clusterable, plus the engineering to keep FSM upkeep off the critical path (it runs at request completion, overlapped with generation).

关键实现细节 (easy-to-miss tricks).

1. Verification mode — early termination can return before full search, but a background full search runs afterward to auto-tune $\alpha_{et}$; this recovers recall without adding foreground latency, and is what makes an aggressive 0.6–0.8 threshold safe.
2. Buffer sizing below the hardware crossover — $B_{insert}=128$ is deliberately set below the ~256–512 GPU-advantage point (Fig 8) so that CPU-side search over the buffer is always cheaper than a GPU kernel launch, and the CPU search overlaps GPU compute so merged-result latency matches single-GPU execution. Splitting is onloaded to a GPU K-means kernel and, thanks to access locality, the cluster needing a split is usually already GPU-resident — eliminating most split overhead.