APEX: Asynchronous Parallel CPU-GPU Execution for Online LLM Inference on Constrained GPUs — L2 #

§1 TL;DR #

APEX replaces NEO's batch-splitting Asymmetric Pipelining with unified-batch Asynchronous Overlap + deferred cross-iteration synchronization for hybrid CPU-GPU LLM inference. A profiling-driven inequality selects strategy per iteration. Up to 96% throughput over vLLM, 72% over NEO on constrained GPUs.

§2 Q1 痛点 / Q2 方法 / Q3 结果 #

Q1 — 痛点 #

Online LLM inference on memory-constrained GPUs (T4 16 GB, A10 24 GB) is bottlenecked by the KV cache, which grows linearly with sequence length and directly limits batch size. Hybrid CPU-GPU execution can relieve this by offloading decode-phase attention to CPU, but the state-of-the-art approach (NEO's Asymmetric Pipelining) fails in decode-heavy workloads for three compounding reasons:

1. Batch splitting doubles GPU linear ops: splitting requests into two sub-batches forces the GPU to execute linear operations (Q/K/V projections, FFN) twice per iteration, doubling $T_{glinear}$.
2. CPU-GPU performance gap too large: CPU attention is 10–20× slower than GPU (3031 μs vs 170 μs at batch size 4), making the pipelining balancing constraint infeasible.
3. Pipeline underutilization: the balancing constraint restricts CPU request allocation, creating idle bubbles where CPU resources go unused.



Paper's Figure 3: self-attention latency (hidden=2048, seq_len=1024) on V100 GPU vs two AMD EPYC 7251 CPUs. At batch size 4, CPU is 17.8× slower.

This ~18× latency gap illustrates why NEO's balancing constraint — requiring GPU compute to exceed CPU attention time — becomes infeasible in decode-only settings. The paper's Inequality (6) formalizes this: Asymmetric Pipelining requires $N_C \geq 13\%$ of $N_G$, but profiling shows $N_C < 10\%$ of $N_G$.

Q2 — 方法 #

APEX introduces three coordinated mechanisms:

1. Asynchronous Overlap: all requests (GPU + CPU) execute linear ops in a single unified batch on GPU, eliminating the doubled $T_{glinear}$. After linear ops, Q/K/V tensors for CPU-designated requests transfer to host memory. GPU and CPU then execute attention concurrently. The key innovation is deferred synchronization: CPU attention results from layer $i$ synchronize only at the start of the next iteration cycle's post-attention phase for layer $i$. This extends the CPU compute window from just $T_{gatt}$ (in Asymmetric Pipelining) to the entire GPU iteration cycle.
1. Analytical scheduling model: a closed-form inequality (Eq. 6) derived from profiling parameters determines whether Asymmetric Pipelining or Asynchronous Overlap yields higher throughput for the current workload mix. The scheduler evaluates this every iteration.
1. Llamafile CPU attention kernel: replaces NEO's ISPC paged-attention with Llamafile matrix-multiplication kernels, achieving up to 2× speedup at larger batch sizes. Paired with pinned host memory for KV cache to reduce transfer latency.

核心技术壁垒: deferred cross-iteration synchronization. The insight that CPU attention results from the current iteration can be consumed in the next iteration's pipeline — without sacrificing correctness — converts the CPU from a synchronous bottleneck into an asynchronous throughput amplifier, even when CPU is 10–20× slower than GPU.

Q3 — 结果 #

GPU	Model	vs vLLM	vs NEO (long output)
T4 (16 GB)	LLaMA-2-7B	+84%–96% throughput	up to +72%
A10 (24 GB)	LLaMA-3.1-8B	+11%–89% throughput	up to +37%

Latency preserved or improved: ~50% average per-token latency reduction vs NEO on T4. Ablation confirms Asynchronous Overlap alone contributes 53–100% of the speedup over NEO.

§3 架构 / 方法图 #



Paper's Figure 4: APEX system architecture. Offline profiler feeds timing data to a performance model. Three request queues (prefill, GPU decode, CPU decode) feed the dynamic scheduler, which selects among GPU-only, Asymmetric Pipelining, and Asynchronous Overlap. KV cache management reserves GPU memory only for active GPU requests.

Four components separate concerns: (1) offline profiler measures per-layer execution times across batch sizes and sequence lengths — runs once per model-hardware pair; (2) request queues organized by type; (3) dynamic scheduler implementing two-algorithm strategy selection; (4) KV cache manager that pins host memory for CPU-side KV and reserves GPU memory only for GPU-active requests.



Paper's Figure 5: Asynchronous Overlap timeline. Red arrows = first sub-batch (prefill + GPU decode); blue arrows = CPU decode. Blue block = synchronization point. Unlike NEO (Figure 1), synchronization is deferred to the next iteration cycle.

The critical difference from NEO's Asymmetric Pipelining: NEO requires CPU attention to complete within the GPU attention window of the same iteration. APEX defers the synchronization barrier to the next iteration cycle. If the CPU hasn't finished when the GPU is ready, the GPU proceeds without stalling and checks again next iteration — a non-blocking design that eliminates pipeline bubbles.

flowchart TD A[New iteration: check queues] --> B{CPU decode queue empty?} B -->|Yes| C[GPU-only execution] B -->|No| D[Try Asymmetric Pipelining split] D --> E{Can schedule CPU requests?} E -->|No| F[Asynchronous Overlap] E -->|Yes| G["Algorithm 2: StrategySelection"] G --> H{"throughput_pipe > throughput_gpu?"} H -->|Yes| I[Asymmetric Pipelining] H -->|No| F F --> J[Unified batch linear → branch attention → deferred sync] I --> K[Two sub-batches → immediate sync] C --> L[Standard GPU execution]

System scope: APEX covers both prefill and decode in online serving mode with continuous batching. It operates at single-node scale (one GPU + host CPUs) without tensor/pipeline/expert parallelism. It extends NEO's framework and targets edge/mid-range GPU deployments where VRAM is the binding constraint.

§4 作者证明 #

符号表 #

Symbol	Definition	Unit
$T_{glinear}$	GPU linear layer time per transformer layer (decode)	μs
$T_{gatt}$	GPU self-attention time per transformer layer (decode)	μs
$T_{gpuonly}$	GPU-only iteration time per layer	μs
$T_{overlap}$	Asymmetric Pipelining cycle time per layer	μs
$N_G$	GPU attention token throughput	tokens/μs
$N_C$	CPU attention token throughput	tokens/μs
$N_{Gtotal}$	GPU-processed tokens per pipeline cycle	tokens
$N_{Ctotal}$	CPU-processed tokens per pipeline cycle	tokens
$T_{glinear\_pref}$	GPU prefill linear time	μs
$T_{gatt\_pref}$	GPU prefill attention time	μs
$\rho_c$	CPU-to-GPU computational power ratio	—
$\rho_t$	Decode-intensive time fraction	—
$S$	Achievable speedup from hybrid execution	—

方程链与物理意义 #

Eq. 1 — GPU-only baseline: $T_{gpuonly} = T_{glinear} + T_{gatt}$. Serial sum of linear and attention within one transformer layer. Serves as the throughput denominator for GPU-only execution.

Eq. 2 — Asymmetric Pipelining overhead: $T_{overlap} \approx 2T_{glinear} + T_{gatt}$. Batch splitting forces two linear passes, doubling the linear contribution. This captures the core inefficiency that APEX targets.

Eqs. 3–4 — Token accounting: $N_{Gtotal} = N_G \cdot T_{gatt}$ (GPU processes tokens during its attention window); $N_{Ctotal} = N_C \cdot (2T_{glinear} + T_{gatt})$ (CPU processes tokens during the full pipeline cycle).

Eq. 5 — Throughput comparison: Asymmetric Pipelining throughput $(N_{Gtotal} + N_{Ctotal}) / T_{overlap}$ must exceed GPU-only throughput $N_{Gtotal} / T_{gpuonly}$.

Eq. 6 — Decision boundary (decode-only):

$$\frac{N_G}{N_C} < 2\frac{T_{glinear}}{T_{gatt}} + 3 + \frac{T_{gatt}}{T_{glinear}}$$

For typical $T_{gatt}/T_{glinear} \in [0.5, 1.5]$, the right-hand side evaluates to 5.8–7.5, meaning CPU must achieve $\geq$13% of GPU attention speed. Profiling shows CPU achieves $<$10%, so this inequality is rarely satisfied in decode-only scenarios — Asynchronous Overlap dominates.

Eqs. 7–8 — Mixed workload: prefill extends cycle time to $T_{glinear\_pref} + T_{gatt\_pref} + T_{glinear} + T_{gatt}$, giving CPU more time and making Asymmetric Pipelining more viable when prefill requests are present. Note: Eq. 8's right-hand side simplifies to $N_G$ (from $N_G T_{gatt}/T_{gatt}$), suggesting a notational inconsistency — the denominator appears to change from per-iteration to per-attention-cycle normalization.

Speedup model: $S \approx \rho_c \cdot \rho_t$. Informal approximation: speedup is proportional to CPU/GPU power ratio times decode fraction. Explains why T4 (weaker GPU → higher $\rho_c$) with OSC (decode-heavy → higher $\rho_t$) yields largest gains.

6 项检查 #

1. 单位一致性: 所有时间单位 μs，速率 tokens/μs，比值无量纲 ✓
2. Eq. 6 代数验证: 从 Eq. 5 交叉相乘展开得 $N_G/N_C < (2a+b)(a+b)/(ab) = 2a/b + 3 + b/a$，其中 $a = T_{glinear}$, $b = T_{gatt}$ ✓
3. 边界行为: 当 $T_{gatt} \to 0$（attention 极快），Eq. 6 右侧 $\to \infty$——Asymmetric Pipelining 总是有利（attention 不是瓶颈时 CPU offloading 无成本）✓
4. 经验验证: Eq. 6 预测 $N_C < 10\%$ 时 Asymmetric Pipelining 无益；§5.5 ablation 确认 AO 在所有 decode-heavy 场景优于 Asymmetric Pipelining ✓
5. Eq. 8 符号模糊: 右侧 $N_G T_{gatt}/T_{gatt}$ 简化为 $N_G$，与 Eq. 5 的 per-iteration 归一化形式不一致——可能反映 mixed workload 下基准定义变化 ⚠
6. $S \approx \rho_c \cdot \rho_t$ 未严格推导: 经验近似而非定理；实际 speedup 还受 Python GIL 开销、PCIe 传输、同步 overhead 约束 ⚠

§5 实验与数据 #

吞吐量对比 #



Paper's Figure 6: (a) T4 + LLaMA-2-7B + OSC with varied output lengths; (b) A10 + LLaMA-3.1-8B on AZ, LB, Dolphin workloads.

A10 三个 benchmark 表现稳定：LiveBench +6% over NEO / +22% over vLLM；Azure Conversation +13% / +11%；Dolphin-r1 +16% / +20%。T4 + OSC 达到最大增益 +72% over NEO / +96% over vLLM。T4 增益更大因为计算力更弱（$\rho_c$ 更高）且 OSC decode 占比更大（$\rho_t$ 更高）。

注意：§1/§5.2 报告 T4 上 vs NEO 最高 72% 增益，但 §8 结论声称最高 49%——差异未解释，可能源于不同 workload 配置或汇总方式。

输出长度敏感性 #



Paper's Figure 8: A10 + LLaMA-3.1-8B, throughput normalized to GPU-only baseline as output length varies (avg input=1000).

Three regimes: short output (50–200 tokens, +5% over NEO), medium output (200–500, +17%–31%), long output (>500, up to +37% then plateau). The plateau confirms $S \approx \rho_c \cdot \rho_t$: once $\rho_t$ saturates (decode dominates), speedup is bounded by $\rho_c$.

消融实验 #



Paper's Figure 9: speedup contribution of each APEX component over NEO on T4 + OSC. AO = Asynchronous Overlap; AK = custom CPU kernel; AM = analytical modeling.

Component	Contribution	Trend with output length
Asynchronous Overlap (AO)	53–100%	Consistent, dominant at all lengths
Custom CPU kernel (AK)	13–21%	Diminishes at long output (CPU finishes within window)
Analytical modeling (AM)	Up to 31%	Significant only at short lengths; negligible by 200 tokens

AO is the load-bearing mechanism. At long output lengths where decode dominates, AO alone delivers nearly all the speedup because the CPU compute window is already sufficient.

Workload 胜负矩阵 #

Workload regime	APEX vs NEO	Why
Short output (<200 tokens), A10	Marginal (+5%)	Decode phase too brief for meaningful CPU overlap
Long output (>500 tokens), T4	Maximum (+72%)	Weak GPU ($\rho_c$ high) + decode-heavy ($\rho_t$ high)
Mixed prefill+decode, A10	Moderate (+6–16%)	Prefill extends pipeline cycle; Asymmetric Pipelining also viable
Decode-only, constrained GPU	Strong (AO: +53–100%)	Unified batch eliminates doubled linear ops
Low batch (<8× CPU:GPU ratio)	Not applicable	Falls back to GPU-only; Python threading overhead not amortized

评估方法注释 #

硬件: T4 (16 GB, 2×Xeon 6130) 和 A10 (24 GB, 2×Xeon 6342)。模型: LLaMA-2-7B / LLaMA-3.1-8B。Baselines: vLLM (GPU-only), NEO (Asymmetric Pipelining)。Throughput 为 output tokens/s；latency 为平均 per-token latency。Baseline 版本号和 commit hash 未给出。未涉及 A100/H100 等高端 GPU，其更大的 $N_G/N_C$ 差距可能使 APEX 收益缩小。

§6 论证链 #

Step	论点	证据	来源
1	KV cache 线性增长是 GPU VRAM 首要瓶颈，直接限制 batch size 和吞吐	LLaMA-65B 在 4×A100 上 13.9 GiB/s KV cache 生成速率；FlashAttention 不解决容量问题	§2.1
2	CPU 带宽与 GPU 差距收窄至 ~3× (200 vs 600 GB/s)，CPU 执行 attention 可行	FastDecode 实测数据	§2.2
3	NEO Asymmetric Pipelining 在 decode-heavy 场景三重失败：双倍 linear + 17.8× speed gap + pipeline 空泡	Figure 2 (linear ops 稳定 <256 tokens) + Figure 3 (CPU/GPU attention gap)	§2.3
4	Inequality (6) 证明 Asymmetric Pipelining 需 $N_C \geq 13\%$ of $N_G$，实测 $<10\%$，decode-only 几乎无收益	Eq. 1–6 代数推导 + §2.3 profiling 交叉验证	§3.2
5	Asynchronous Overlap 通过统一 batch 消除双倍 linear，通过跨 iteration 延迟同步扩展 CPU 窗口至全 GPU iteration cycle	Figure 5 时序图 + non-blocking readiness check 设计	§3.3
6	消融确认 AO 是主要贡献（53–100%），kernel 和建模是次要	Figure 9 在 T4 + OSC 上的三级累加实验	§5.5
7	端到端验证：T4 +96% over vLLM / +72% over NEO；增益随 output 长度增长并饱和于 $\rho_c$ 边界	Figures 6, 8；$S \approx \rho_c \cdot \rho_t$ 模型定性吻合	§5.2, §5.4

§7 实现 cross-reference #

[实现未公开] — 论文未提供源代码仓库。已知实现信息均基于论文描述。

系统基底: APEX 构建于 NEO 的代码框架之上，NEO 实现了基于 vLLM 的 continuous batching 和 PagedAttention 基础设施。

CPU attention kernel: 使用 Mozilla Llamafile 项目的矩阵乘法 kernel（已被 llama.cpp 和 ktransformers 采用），替代 NEO 的 ISPC paged-attention kernel。小 batch 略慢但大 batch 可达 2× speedup。

GIL 绕过: CPU attention 通过 Pybind11 集成 C++ 实现，运行时调用 gil_scoped_release 释放 Python GIL，使 CPU compute thread 与 main CUDA thread 真正并行。采用两线程 Python 架构——main thread 管理系统逻辑和 CUDA kernel 启动，dedicated compute thread 执行 CPU attention。

关键实现细节:

1. 8× minimum CPU:GPU request ratio: CPU compute thread 需处理 ≥8× GPU request 数量才能摊薄 Python threading 开销。低并发场景回退 GPU-only。论文未分析此阈值的敏感性或硬件依赖性。
1. Pinned memory for KV cache: CPU 端 KV cache 使用 pinned memory 加速 GPU↔CPU 传输。Non-pinned memory 需额外 bounce buffer 拷贝——在频繁 Q/K/V transfer 路径上这是关键性能差异。

核心技术壁垒（详述）: deferred cross-iteration synchronization 的正确性保证是最难复制的设计。CPU attention results 从 layer $i$ 延迟到下一 iteration 同步时，系统须维护 per-layer、per-request 状态追踪，确保 stale results 不被错误消费。GPU 端 non-blocking readiness check 需精细的 CUDA event/stream 管理。论文声称"only changes execution location, not mathematical operations"以保证精度，但跨 iteration 状态管理在工程上是主要复杂度来源。未公开代码使这一设计的完整实现细节无法验证。