Beyond Tokens: A Unified Framework for Latent Communication in LLM-Based Multi-Agent Systems — L2 #

1. TL;DR #

When LLM agents talk in text they round-trip a rich hidden state ($\approx$40,000 bits) through a vocabulary bottleneck ($\approx$15 bits/token), paying $\approx$2× decode+prefill cost. This survey classifies 18 "latent communication" methods along three axes — WHAT (embedding / hidden state / KV-cache) × WHICH (alignment) × HOW (fusion) — and shows latent channels only win conditionally (Eq. 3 break-even).

2. 痛点 / 方法 / 结果 (Q1 / Q2 / Q3) #

Q1 — 痛点 (Pain Point) #

Text-only inter-agent messaging in multi-agent LLM systems carries three structural costs:

1. Inference cost: every message forces the sender to autoregressively decode a token sequence and the receiver to prefill it — "roughly 2× the cost of generating the message, even before accounting for the receiver's own reasoning."
2. Information loss at discretization: a token conveys at most $\log_2|V| \approx 15$–$17$ bits, while the sender's hidden state lives in $\mathbb{R}^{d}$ with $d \geq 4096$. The pipeline discards a factor of $\approx 10^3$–$10^4$ in information — not just which token, but the alternatives considered, their probabilities, salience, and uncertainty.
3. Redundancy / ambiguity: generated text is optimised for linguistic fluency (a pre-training stylistic objective), not task-relevant information density — padded with politeness, hedging, reformulation.

Q2 — 方法 (Method) #

The paper's contribution is organisational, not empirical: a single algebraic decomposition that every surveyed method factors through,

$$\text{Method} = (\text{WHAT: information type},\ \text{WHICH: alignment},\ \text{HOW: fusion})$$

• WHAT — what continuous object crosses the agent boundary: input embedding (lightest, least informative), hidden state $\mathbf{h}_i^{(\ell)}$ (encodes intermediate computation, often training-free), or full KV-cache (richest, largest payload, most architecture-dependent) — plus "other" (state-delta trajectories, persistent disk caches, visual-latent codecs, shared workspaces).
• WHICH — how sender and receiver latent spaces are reconciled: none (same checkpoint, identical space by construction), learned projection (compatible variants), universal visual codec / interaction layers (heterogeneous), plus a layer-alignment sub-choice (last → first vs all → corresponding).
• HOW — how the received latent enters the receiver's computation: concatenation, prepending (natural for KV), mathematical operation (add/subtract/learned fuser), cross-attention, or cache restoration (direct block replacement).

核心技术壁垒 (single hardest-to-replicate insight): the conditional framing itself. The paper's durable contribution is not any one protocol but the recognition — crystallised in the Eq. 3 break-even condition — that latent communication is a systems optimisation whose sign flips with context length, bandwidth, and model compatibility. Replicating the taxonomy is easy; replicating the discipline of refusing a numerical leaderboard and instead deriving when a channel pays off is the hard, non-obvious editorial judgement (§9, §7.5).

Q3 — 结果 (Result) #

• 18 representative methods (2024–2026) fully classified across all three axes (Table 8 quick-reference covers every method with WHAT/WHICH/HOW/Train? filled).
• Five design takeaways, the load-bearing ones being: payload size and architecture-dependence increase monotonically embedding → hidden-state → KV-cache (Takeaway 1); KV-cache methods cluster in the prefill phase while embedding/hidden-state methods cluster in decode (Takeaway 2); $\approx 13/18$ methods are training-free (Table 5).
• A deployment decision guide (Table 4) mapping 6 deployment constraints → recommended protocol.
• Headline verdict: "the evidence does not support a universal claim that latent channels dominate text." Text keeps the auditability/interoperability win; the latent advantage is conditional on compatible agents + expensive decode/prefill + favourable transfer economics.

---

3. 架构 / 方法图 (Architecture / Method Diagrams) #

This is a survey, not a model release, so the "architecture" being analysed is the communication pipeline — the set of exposable intermediate representations inside a Transformer block and the two competing ways agents can route state between them.



Paper's Figure 1. The left panel annotates a Transformer block with its three exposable interfaces — the hidden state (contextualised token representation, output of a block), attention K/V vectors (per-token cache), and embeddings (input layer). The top-right contrasts token-level vs hidden-state reasoning information density; the bottom-right shows the prefill/decode phases that produce the per-token KV-cache. Reader should notice: latent communication is not one mechanism but a choice of tap point along this block, and that choice is exactly Axis 1 (WHAT).



Paper's Figure 2. Sub-figure (a) is the motivation in one bar chart: a discrete token sampled from the vocabulary carries $\approx 15$ bits; a single last-token hidden state carries $\approx 40{,}000$ bits — three to four orders of magnitude. Sub-figure (b) shows the mechanism: NL-Comm routes the sender hidden state through a vocabulary bottleneck before the receiver re-encodes, whereas Latent-Comm hands the continuous vector across directly. This figure is the paper's core argument for why the design space is worth surveying at all.



Paper's Figure 3. The central organising device: three orthogonal-ish axes span the design space. WHAT (Embedding / Hidden State / KV-Cache / Others) selects the payload; WHICH (Latent Information Alignment + Layer Alignment) reconciles sender and receiver spaces; HOW (Concat / Prepend / Math Op / Cross-Attention) injects at the receiver. Every one of the 18 methods is a coordinate in this cube. The paper is explicit that the axes are "conceptually separable although not fully independent" — KV-caches favour layer-wise injection, embeddings enter near the input (§5.4).

The dynamic relationship between the three injection points and the two inference phases is best seen as a state machine:

flowchart LR subgraph Sender["Sender Agent A1"] P1[Prefill: process prompt in parallel] --> KV1[(KV-cache)] KV1 --> D1[Decode: token-by-token] D1 --> H1[Hidden state h_i^ell] D1 --> E1[Weighted embedding e_i] end subgraph Receiver["Receiver Agent A2"] direction TB inj{WHICH: align space + layer} KV2[Prepend / restore KV] --> G2[Resume as if prefilled A1 context] H2[Concat / add hidden state] --> G2 E2[Concat embedding near input] --> G2 end KV1 -. "KV-Cache path (prefill-heavy)" .-> inj --> KV2 H1 -. "Hidden-state path (decode-heavy)" .-> inj --> H2 E1 -. "Embedding path (decode-heavy)" .-> inj --> E2

The Mermaid adds what the raster figures do not: the phase affinity of Takeaway 2 — KV-cache transfer lets the receiver skip prefill (left/prefill side), while embedding/hidden-state transfer rides the decode loop (right side).

---

4. 作者证明 (Author's Formal Argument) #

The survey is largely descriptive, but three numbered equations carry real weight. Notation:

Symbol	Meaning
$\mathbf{h}_{\text{context}}$	Sender's high-dimensional hidden state, $\in \mathbb{R}^d$, $d \geq 4096$
$\mathbf{z}$	Logit vector over vocabulary, $\in \mathbb{R}^{V}$
$y$	Sampled output token, $y \in \mathcal{V}$
$V, \mathcal{V}$	Vocabulary size / vocabulary
$\mathbf{h}_i^{(\ell)}$	Hidden state of token $i$ at layer $\ell$
$t_{\mathrm{saved}}$	Decode/prefill time avoided by latent transfer
$t_{\mathrm{setup}}, t_{\mathrm{align}}, t_{\mathrm{inject}}$	Transfer setup / alignment / injection overheads
$S$	Payload size (bytes); $B$	effective bandwidth (bytes/s)

Eq. 1 — the discretization bottleneck (physical meaning: generation collapses a continuous state into a symbol):

$$\mathbf{h}_{\text{context}} \xrightarrow{\text{linear}} \mathbf{z} \in \mathbb{R}^{V} \xrightarrow{\text{sample}} y \in \mathcal{V}$$

Mutual information is bounded: $I(\mathbf{h}_{\text{context}}; y) \leq H(y) \leq \log_2|V|$.

Eq. 2 — the framework (physical meaning: a method is a 3-tuple coordinate): $\text{Method} = (\text{WHAT}, \text{WHICH}, \text{HOW})$.

Eq. 3 — the break-even condition (physical meaning: latent transfer only profits when avoided compute exceeds all transfer overhead):

$$t_{\mathrm{saved}} > t_{\mathrm{setup}} + \frac{S}{B} + t_{\mathrm{align}} + t_{\mathrm{inject}}$$

6 minimum sanity checks #

1. Token info bound: with $V \approx 128\text{k}$, $\log_2 V \approx 17$ bits; the paper's "$\approx 15$ bits" is consistent with a slightly smaller or entropy-reduced sampling distribution. ✓
2. Hidden-state capacity: $d = 4096$ dimensions at $\approx 10$ effective bits/dim $\Rightarrow \approx 40{,}000$ bits, matching Fig. 2(a). The claimed $10^3$–$10^4$ ratio equals $40000/15 \approx 2.7\times10^3$. ✓ (consistent, lower end of the stated range)
3. Eq. 3 dimensional consistency: every term is a time. $S/B$ = bytes ÷ (bytes/s) = seconds. ✓
4. 2× cost claim: sender decodes $T$ message tokens (cost $\propto T$) + receiver prefills the same $T$ tokens (cost $\propto T$) $\Rightarrow$ total $\approx 2\times$ single-message generation, before receiver reasoning. ✓
5. Tuple completeness: Table 8 assigns all three axes to every one of the 18 primary methods with no empty WHAT/HOW cell — the decomposition is total over the corpus, not partial. ✓
6. Break-even monotonicity: $t_{\mathrm{saved}}$ (avoided prefill) grows with context length $T$, often super-linearly in attention; $S$ (KV payload) grows $\approx$ linearly in $T$. Hence for large $T$ the inequality eventually holds, while for small $T$ it fails — exactly the "accelerates long-context yet regresses on short prompts" claim in §6.1. ✓

No statistical theorems are proven (the paper flags this gap itself in §7.4); the equations are systems-level bounds, not learning-theoretic guarantees.

---

5. 实验与数据 (Evidence & Data) #

A survey's "experiments" are its comparative syntheses. The load-bearing artifacts are Tables 1, 4, 5, 7, 8.



Paper's Table 1. The WHAT-axis census: 1 embedding method (CIPHER), 5 hidden-state methods (AC, Interlat, SDE, ThoughtComm, MoT), and 9 KV-cache methods (KVComm, Cache-to-Cache, LatentMAS, Q-KVComm, LRAgent, RelayCaching, Agent Memory, Agent Primitives, Edge LLM Handover), plus 3 "other" (Vision Wormhole, BIGMAS, Latent Agents). Reader should notice KV-cache dominates (50%) — the field's centre of gravity is prefill-avoidance.



Paper's Table 7. The closest thing to a quantitative scoreboard the paper permits itself — ordinal stars (★★★/★★/★), not raw numbers. The pattern worth reading: methods with ★★★ generality (Vision Wormhole, MoT) are exactly the ones that are not training-free and carry ★★★ engineering complexity — generality is bought with training cost. Same-model KV methods (RelayCaching, Agent Memory) score ★★★ latency but only ★ generality.



Paper's Table 4. The practitioner payload: six deployment constraints each mapped to a recommended protocol and its caveat (e.g. same backbone + long shared context → KV-cache transfer/restoration, caveat positional compatibility; heterogeneous backbones → projected hidden states or shared codec; bandwidth-constrained → selected layers + quantisation + low-rank deltas). This operationalises the taxonomy into an if-then table.



Paper's Table 5. $\approx 13/18$ methods are training-free (✓); only Cache-to-Cache (learned fuser), Vision Wormhole (label-free distillation), and Mixture of Thoughts (router + interaction layers) strictly require training; Interlat is conditional (free same-model, learned projection cross-model). The paper's sharpest caveat sits under this table: "parameter-free does not mean assumption-free" — training-free methods remain sensitive to layer choice and tensor scales.



Paper's Table 8. The complete coordinate assignment — every method resolved to its (WHAT, WHICH, HOW, Train?) tuple, including the boundary cases (TokenDance, AAFLOW+, When Latent Agents Lie, Latent Agents) marked as adjacent/out-of-corpus. This table is the empirical validation that Eq. 2's decomposition is total.

Note on comparability: the paper deliberately declines a numerical speedup leaderboard despite reported gains up to 24×, because cross-paper numbers mix different backbones, hardware, prompts, and context lengths (§9, §7.5). This is itself a finding — the honest negative result.

---

6. 论证链 (Argument Chain — paper-internal) #

Step	Claim	Support in paper
1	Text messaging between agents is information-lossy and compute-expensive.	Eq. 1 info bound ($\approx$15 bits/token) + Fig. 2(a) ($\approx$40,000 bits/hidden-state) + $\approx$2× decode+prefill cost (§4.1.1).
2	Therefore agents could instead exchange continuous internal state (embedding / hidden state / KV-cache).	§3.4 defines the three exposable objects; Fig. 1 annotates the tap points.
3	The sprawling 2024–2026 literature doing this factors cleanly into 3 axes.	Eq. 2 WHAT×WHICH×HOW; Tables 1–3 populate each axis for 18 methods; Table 8 gives the full coordinate.
4	The axes expose coupled trade-offs (payload ↑, architecture-dependence ↑, training needs vary).	Takeaway 1 (monotone payload); Table 7 trade-off stars; Table 5 training regimes.
5	Whether latent transfer actually pays off is a break-even inequality, not a universal win.	Eq. 3; §6.1 "accelerates long-context yet regresses on short prompts or slow networks."
6	Hence the design reduces to constraint-matching, and text remains valuable for oversight.	Table 4 decision guide; §4.2 auditability advantage; §10 conditional verdict.

---

7. 实现 cross-reference (Implementation) #

This paper ships no model and no primary system code — it is a survey. There is therefore no modeling_*.py, no config.json, no training recipe to itemise, and the LLM-release-specific analyses (scaling-law fit, per-module parameter breakdown, quantization-aware training table, serving-hardware sizing) are N/A — survey, not a model release.

What exists:

• Companion corpus repository: https://github.com/enochliu98/Awesome-Latent-Communication — maintains the paper list, code links, taxonomy artifacts, and figure-generation notes. Review cut-off: 15 July 2026 [实现未公开 for any first-party protocol; all 18 methods' code lives in their own repos, linked from the corpus list].
• Per-method implementations are third-party (CIPHER, KVComm, Cache-to-Cache, etc. each have their own papers/repos in the References).

关键实现细节 (easy-to-miss tricks) #

1. Phase affinity is a deployment lever, not a footnote (Takeaway 2): because KV-cache transfer eliminates prefill and hidden-state/embedding transfer rides decode, the right WHAT choice depends on whether your bottleneck is TTFT (prefill-bound, long context → KV) or inter-token latency (decode-bound, short context → hidden state). Choosing the "richest" payload (KV) blindly can regress on short prompts via the $S/B$ term of Eq. 3.
2. Four engineering failure modes that silently corrupt output (§6.3): positional mismatch (reused keys encode sender positions, not receiver's — RoPE conventions must match), geometric mismatch (equal tensor shapes can hide incompatible activation distributions), memory-bound transfer (a full cache saves FLOPs yet raises wall-clock latency), and state contamination (stale/cross-session/adversarial tensors poison future turns). The first two are why "same tensor dimension" ≠ "safe to inject."

核心技术壁垒 — dedicated note #

The hardest thing to reproduce is not code but the conditional stance. Most individual papers in the corpus report a peak speedup and stop. This survey's irreplaceable move is formalising the break-even (Eq. 3), showing the same protocol can both accelerate and regress depending on $(S, B, T)$, and then refusing to rank methods numerically because the comparison is confounded (§7.5, §9). That editorial discipline — turning a hype-prone "latent beats text" into a falsifiable inequality — is the paper's defensible moat, and it is precisely what a fast-follower survey cannot copy without redoing the cross-paper confounder analysis.

---

8. 开放问题 (Open Questions — survey-specific) #

The paper's own §7 agenda, plus model-angle framing:

• Cross-architecture alignment: can a reusable interface achieve sub-quadratic scaling with model families (hub-and-spoke codecs, model-agnostic concept subspaces)? Evaluation must separate pair-specific memorisation from transfer to unseen pairs.
• Security: a latent channel is an "opaque, high-privilege interface" — text-only verification misses tampered KV state [26]. Hybrid protocols must bind a visible commitment + model identity + session + tensor metadata + payload digest.
• Compression: metric should be task utility per transmitted byte, not reconstruction error; report quality–bytes–latency Pareto curves including (de)compression cost.
• Theory: distinguish channel capacity from usable task information; lower bounds on sufficient message size and error propagation across multi-round communication are missing.
• Latent-Comm vs Latent-CoT boundary: temporal flow (step→step, one model) vs organisational flow (agent→agent); distillation (Latent Agents) blurs it by internalising multi-agent structure into one model (Table 6).



Paper's Table 6. Draws the scope lines the survey depends on: Latent CoT (step→step, no sender/receiver boundary), internalised debate (multi-agent training → one model, structure vanishes at inference), MARL learned communication (messages learned jointly with policies), emergent language (protocol emergence), and KV serving/reuse (shared state need not encode a semantic agent message). The distinguishing criterion throughout is whether an explicit inference-time interface crosses an agent boundary.