Blink: CPU-Free LLM Inference — L2 #

§1 TL;DR #

Blink removes host CPU from LLM inference critical path via SmartNIC (DPU) frontend + GPU-resident persistent scheduler, achieving up to 8.47× P99 TTFT reduction and complete interference immunity where baselines degrade 1–2 orders of magnitude.

§2 Q1 / Q2 / Q3 #

Q1 痛点 #

Current LLM serving stacks (vLLM, SGLang, TRT-LLM) keep host CPU on every-token critical path for scheduling, batching, KV-cache management, and CUDA kernel dispatch. Even with CUDA Graphs and overlapped scheduling, the scheduler must return to host after every decode step. This creates two compounding problems:

1. CPU interference vulnerability: colocated workloads cause TLB invalidations and LLC pollution, triggering cross-address-space amplification — page walks that previously hit LLC now proceed to DRAM, driving LLC stall cycles up 11.2×. Under pbzip2 interference on vLLM+H100, throughput drops 3.8× and P99 TTFT inflates 139× (from 150 ms to 20,959 ms).
1. Standard mitigations are insufficient: huge pages reduce dTLB misses by only 16% with no latency improvement. Core pinning consumes all available cores (6 cores/GPU × 8 GPUs = 48 cores on dual-socket Xeon). LLC partitioning via Intel CAT fully eliminates LLC contention but improves P99 ITL by less than 4% — host-side scheduling jitter and CUDA dispatch overhead persist regardless.

The fundamental issue is architectural: a fragile, interference-sensitive CPU sits on the critical path of every generated token.

Q2 方法 #

Blink redesigns the inference serving stack around two principles: (1) the host CPU becomes a provisioning plane (loads model and captures CUDA graphs at startup, then exits entirely); (2) steady-state operation uses only DPU + GPU.

DPU frontend (NVIDIA BlueField-3, 16 ARM Cortex-A78 cores):

• Receives HTTP requests, tokenizes via SIMD-optimized tokenizer (merge rules in 64-byte-aligned flat hash table, ARM NEON at 16 bytes/cycle; 8–19.7× faster than HuggingFace on Xeon)
• Maintains local slot availability cache via periodic bulk RDMA reads
• Writes tokenized prompts into GPU-resident ring buffer via one-sided RDMA
• Polls for generated tokens and streams responses via SSE

GPU backend (persistent scheduler):

• A single CUDA kernel occupying one thread block (256 threads) runs indefinitely
• Control loop: scan ring buffer → claim via atomic CAS → select/launch CUDA graph → poll for completion → publish tokens
• 256 threads scan all 4096 ring buffer slots in 1–5 µs
• Device-side CUDA graph launch: fire-and-forget at ≈2 µs (5–8× faster than host launch), with window-based tail-launch recovery for the 120-launch hard limit
• Continuous batching with inline prefill: while decode graph executes asynchronously, 256 threads scan for pending prompts; new requests admitted within one decode step

Communication: GPU-resident lock-free ring buffer with per-slot state machine (empty → prefill_pending → prefill_processing → decode_processing → decode_completed → empty, plus decode_paused for preemption). DPU and GPU coordinate exclusively through this buffer; ownership transferred via atomic CAS with RDMA-visible memory fences.

System scope: both prefill and decode with continuous batching. Single-GPU, single-node. FCFS scheduling. Paged KV-cache managed entirely on GPU. OpenAI-compatible HTTP API with SSE streaming.

核心技术壁垒: the persistent GPU-resident scheduler. Replacing the entire host-driven decode loop with a single indefinitely-running CUDA kernel requires solving three deeply coupled problems: (a) device-side CUDA graph launch via fire-and-forget with tail-launch recovery for the undocumented 120-launch hard limit (exceeding it produces undefined behavior); (b) completion detection via device-side polling since fire-and-forget provides no host-side callbacks; (c) lock-free coordination with an external DPU through RDMA-visible memory fences without any CPU mediation. This is not offloading a function — it restructures who owns the inference control loop.

Q3 结果 #

Evaluated on 4 models (Llama-3 8B, Phi-4 15B, Qwen-3 32B, Qwen-3 30B-A3B) against TRT-LLM v1.1.0, vLLM v0.13.0, SGLang v0.5.8 on NVIDIA H100.

Isolation: P99 TTFT reduced up to 8.47× (vs SGLang on Qwen-3 30B-A3B), P99 TPOT reduced up to 3.40×, decode throughput improved up to 2.1× (1437 tok/s vs 730 tok/s on MoE), energy per token reduced up to 48.6%. Highest or tied-highest saturation throughput on every model.

Under CPU interference (pbzip2 + Ninja LLVM build on 90 host cores): Blink maintains TTFT inflation 0.92–1.14×, TPOT inflation 0.97–1.04×, throughput retention 99–100%. Baselines: TTFT inflation 1.54–18.84×, throughput retention 28–64%. Blink plateau throughput 1.69–4.08× higher than baselines under interference. Energy per token reduced 41.4–70.7% vs baselines.

MoE models show amplified benefits: Qwen-3 30B-A3B activates only 3B of 30B parameters per token, so each decode step completes quickly on GPU but CPU orchestration cost remains constant, making scheduling overhead a larger fraction of step time.

§3 架构 / 方法图 #

sequenceDiagram participant C as Client participant D as DPU Frontend
(BlueField-3 ARM) participant R as Ring Buffer
(GPU Memory) participant S as Persistent Scheduler
(256-thread CUDA Kernel) participant E as CUDA Graph Engine
(TensorRT) Note over D,E: Host loads model + captures CUDA graphs at startup, then exits C->>D: HTTP request D->>D: Tokenize (ARM NEON SIMD) D->>D: Find slot (local cache + circular scan) D->>R: One-sided RDMA write (prompt → slot) S->>R: 256-thread parallel scan + atomic CAS claim S->>E: Select graph from O(1) lookup table S->>E: Fire-and-forget device launch (≈2 µs) E->>E: Inference + token sampling E->>R: Write generated token to slot D->>R: RDMA read (poll results) D->>D: Detokenize D->>C: SSE streaming response Note over S: Infinite loop: scan → claim → launch → poll → publish Note over S: At 120 launches: tail-launch recovery resets window

Request lifecycle: Client → DPU HTTP server → tokenize → RDMA write to ring buffer → GPU persistent scheduler claims slot → selects and launches precompiled CUDA graph → inference executes → token written to ring buffer → DPU polls via RDMA → detokenize → stream to client. Host CPU participates in none of these steps after initialization.

Ring buffer: resides entirely in GPU memory. 4096 fixed slots with shared arenas for input/generated tokens. Per-slot metadata tracks prompt identity, token counts, generation progress. State machine governs ownership: DPU writes only to empty slots, GPU claims via CAS transition to prefill_pending, and only the GPU advances through processing states. Memory fences ensure RDMA-visible updates.

Scheduling: FCFS with pause-and-resume continuous batching. While a decode graph executes asynchronously, the scheduler's 256 threads scan the ring buffer for pending prefills. Three conditions gate pausing: (1) pending prefills detected, (2) free batch-slot capacity, (3) sufficient fire-and-forget launch-window headroom. New requests admitted within one decode step.

CUDA graph cache: precompiled TensorRT engines compiled into graphs for dense grid of (batch size, sequence length) pairs. Each graph consumes only 2–3 MB. Cache of 650–1000 graphs fits within 2–4 GB. Runtime selection via precomputed lookup table indexed by (batch, seq_len) in $O(1)$. Token sampling (Top-P with temperature) captured inside each graph.

§4 作者证明 #

无形式化作者证明 — 仅实证

This paper contains no numbered equations and no formal analytical or performance model. The argument rests entirely on controlled systems experiments:

• §2.2: controlled interference measurements on vLLM + H100 with hardware performance counters (IPC, LLC miss rate, dTLB misses, LLC stall cycles, CPU migrations)
• §3.1–3.2: systematic evaluation of each standard mitigation (huge pages, core pinning, LLC partitioning, scheduling priority) with quantified residual effects
• §6: comprehensive benchmarks across 4 models × 3 baselines × 2 conditions (isolated + interference) × 13 load levels

A formal model would have clarified:

1. Throughput bound: theoretical maximum as function of ring buffer slot count $N$, RDMA bandwidth $B_{\text{RDMA}}$, and graph launch latency $t_{\text{launch}}$ — is the system ring-buffer-limited or compute-limited, and at what concurrency does it transition?
2. Graph cache sizing: optimal number of precompiled (batch, seq_len) pairs vs GPU memory budget — the paper uses 650–1000 graphs empirically but provides no principled tradeoff analysis
3. Latency decomposition: closed-form $T_{\text{TTFT}} = t_{\text{RDMA}} + t_{\text{scan}} + t_{\text{launch}} + t_{\text{prefill}}$ and $T_{\text{TPOT}} = t_{\text{launch}} + t_{\text{decode}} + t_{\text{poll}}$ to predict which term dominates at different model sizes
4. DPU sensitivity: the DPU frontend runs on ARM cores — a model quantifying when DPU-side contention begins degrading Blink's isolation guarantees would strengthen the generality claim

§5 实验与数据 #

硬件平台 #

NVIDIA H100 (96 GB HBM3), 2× Intel Xeon Gold 6336Y (96 cores @ 2.40 GHz), 256 GB DDR5, ConnectX-6 200 Gbps NIC. Blink frontend on separate BlueField-3 DPU (16 ARM Cortex-A78, 32 GB) connected via 200 Gbps RDMA link (DOCA SDK v3.2.1).

Isolated performance (pre-saturation geometric mean) #

Model	System	P99 TTFT (ms)	P99 TPOT (ms)	Tput at sat. (req/s)
Llama-3 8B ($\lambda \leq 12$)	Blink	653.8	15.1	11.87
	TRT-LLM	880.0	17.7	10.80
	vLLM	1309.6	24.2	9.12
	SGLang	1747.1	30.7	7.88
Phi-4 15B ($\lambda \leq 7$)	Blink	1109.4	25.0	6.72
	TRT-LLM	1453.8	29.8	6.42
	vLLM	1683.7	34.5	6.05
	SGLang	2874.1	47.9	5.58
Qwen-3 32B ($\lambda \leq 2$)	Blink	9481.3	113.4	2.00
	TRT-LLM	9621.4	115.2	1.97
	vLLM	10862.4	133.7	1.88
	SGLang	11413.0	123.3	1.85
Qwen-3 30B-A3B MoE ($\lambda \leq 4$)	Blink	1397.5	35.5	4.85
	TRT-LLM	4814.7	65.8	3.61
	vLLM	8919.2	90.9	2.91
	SGLang	11839.8	120.8	2.62

Comparison to TRT-LLM is the cleanest control (same TensorRT inference engines). On Qwen-3 30B-A3B, Blink's P99 TTFT is 3.45× lower than TRT-LLM with 37% higher throughput — the largest gap across all models, driven by MoE's unfavorable compute-to-orchestration ratio.

Performance under CPU interference #

Model	System	P99 TTFT inflation	P99 TPOT inflation	Tput retention
Llama-3 8B	Blink	1.00×	1.00×	100%
	TRT-LLM	18.84×	11.10×	38%
	vLLM	11.12×	7.35×	44%
	SGLang	8.43×	5.77×	48%
Phi-4 15B	Blink	0.92×	0.98×	101%
	TRT-LLM	10.66×	6.17×	41%
	vLLM	7.14×	4.74×	47%
	SGLang	3.82×	3.15×	47%
Qwen-3 32B	Blink	0.99×	1.04×	102%
	TRT-LLM	1.68×	3.23×	51%
	vLLM	1.54×	2.64×	64%
	SGLang	1.61×	3.35×	59%
Qwen-3 30B-A3B	Blink	1.14×	0.97×	99%
	TRT-LLM	4.90×	9.19×	28%
	vLLM	2.02×	3.04×	54%
	SGLang	1.98×	3.96×	45%

Interference: pbzip2 (45 threads) + Ninja LLVM build (45 jobs) on 90 host cores, 6 cores reserved per NVIDIA guidelines.

能效 #

Condition	Blink range (mJ/tok)	Best baseline (mJ/tok)	Blink savings
Isolation	363–1306	502–1580	13.7–48.6%
Interference	423–1584	1045–3597	41.4–70.7%

All systems draw comparable wall power (1.1–1.4 kW). When CPU contention collapses baseline throughput at constant power, their energy per token inflates 69–182%. Blink's overhead at most 21%.

CPU interference root cause (motivation measurements) #

vLLM v0.13 + Llama-3 8B on H100, ShareGPT traces, pbzip2 interferer:

Metric	Baseline	24× interference	Factor
Throughput (tok/s)	7,475	1,961	3.8× drop
P99 TTFT (ms)	150	20,959	139× inflation
LLC miss rate	7.0%	71.6%	10.2×
LLC stall cycles	450 M	5,037 M	11.2×
IPC	1.53	0.72	2.1× drop

Workload characterization — where Blink wins and loses #

Regime	Blink advantage	Mechanism
MoE, any load	3.45× P99 TTFT, 37% throughput vs TRT-LLM	Low compute-to-orchestration ratio amplifies per-token scheduling savings
Dense, moderate-to-high load	1.35× P99 TTFT, 9% throughput vs TRT-LLM	Per-step CPU round-trip savings compound across output tokens
Dense, GPU-bound (Qwen-3 32B)	~parity at P99; diverges at P99.9	GPU compute dominates; scheduling overhead masked at P99 but surfaces at deep tail
Any model + CPU interference	Stable (0.92–1.14×) vs 1.5–18.8× degradation	CPU entirely removed from critical path
Models exceeding GPU memory	Not supported (baselines use CPU/DRAM offload)	Blink requires model to fit in GPU memory

On Qwen-3 32B, TRT-LLM achieves lower P50 TTFT (531.7 ms vs 786.2 ms) — one of the few metrics where a baseline outperforms Blink, reflecting the GPU-bound regime where scheduling overhead is negligible at median.

Decode-level throughput at saturation #

Model	Blink (tok/s)	TRT-LLM (tok/s)	Δ
Llama-3 8B	3880	3535	+10%
Phi-4 15B	2177	2044	+7%
Qwen-3 32B	537	520	+3%
Qwen-3 30B-A3B	1437	1053	+36%

§6 论证链 #

Step	Claim	Evidence	Type
1	Host CPU is on every-token critical path of LLM inference	vLLM/SGLang/TRT-LLM return control to host after every decode step for scheduling, KV-cache management, kernel dispatch. CPU scheduling consumes up to 50% of e2e latency on fast accelerators (§2.1)	Architectural analysis + literature
2	CPU interference causes severe, compounding degradation	vLLM+H100 under pbzip2: 3.8× throughput drop, 139× P99 TTFT inflation. TLB invalidation + LLC pollution create cross-address-space amplification — page walks forced to DRAM, LLC stall cycles up 11.2× (§2.2, Table 1)	Controlled measurement
3	Standard mitigations fail to restore isolation	Huge pages: -16% dTLB misses, no latency improvement. Core pinning: consumes all cores, -18% throughput residual. LLC partitioning via CAT: eliminates LLC contention but <4% P99 ITL improvement (§3, Tables 2–4)	Systematic elimination
4	Root cause is architectural, not resource contention	After eliminating LLC contention, host orchestration still inflates: attention dispatch +104%, cudaLaunchKernel +115%, KV-cache dispatch +172%. GPU kernel times unchanged (0.41–0.42 ms). Dynamic core systems (Caladan, Shenango) optimize CPU capacity, not whether CPU is on critical path (§3.3)	Profiling + argument by elimination
5	CPU-free inference requires DPU + GPU-resident scheduler co-design	DPU alone cannot absorb per-token scheduling (limited core count). GPU alone cannot handle network I/O. Blink splits: DPU handles request management + RDMA transport; GPU-persistent kernel handles scheduling + execution (§4.1)	Design argument
6	Persistent GPU scheduler achieves lower overhead than host path	Device-side CUDA graph launch ≈2 µs vs host 11–17 µs (5–8× faster). 256-thread ring buffer scan in 1–5 µs. CPU path inflates makespan 1.16–1.70× on same workloads (§4.2, Figure 3)	Microbenchmark
7	Blink outperforms all baselines even in isolation	Same TensorRT engines as TRT-LLM: 1.35–3.45× lower P99 TTFT, up to 37% higher throughput. Highest saturation throughput on every model (§6.2, Table 6)	Controlled benchmark
8	Blink maintains performance under CPU interference	TTFT inflation 0.92–1.14×, TPOT 0.97–1.04×, throughput 99–100% retention. Baselines: 1.54–18.84× TTFT inflation, 28–64% retention. Plateau 1.69–4.08× higher (§6.3, Table 7)	Controlled benchmark

§7 实现 cross-reference #

[实现未公开] — no public repository at time of writing.

• GPU backend: ≈16,000 lines CUDA/C++ — persistent scheduler, device-side graph launch, continuous batching, KV-cache management, token sampling. Targets CUDA 13.1, TensorRT inference engine.
• DPU frontend: ≈17,000 lines C/C++ — HTTP parsing/validation, RDMA orchestration via DOCA 3.2 SDK, ring buffer coordination, SSE streaming, optional ARM-based tokenization.

核心技术壁垒 — persistent GPU-resident scheduler #

The persistent scheduler is a single CUDA kernel that runs indefinitely, replacing the host CPU's role in the decode loop. Three aspects make replication non-trivial:

1. Device-side CUDA graph launch with 120-launch recovery: fire-and-forget achieves ≈2 µs but CUDA imposes an undocumented hard limit of 120 outstanding fire-and-forget launches per parent graph execution — exceeding it produces undefined behavior. Blink's window-based tail-launch recovery issues a single tail launch at the 120th iteration, atomically replacing the current graph execution while preserving all state in persistent GPU memory. Amortized overhead: <0.03 µs per step (fire-and-forget for 120/121 iterations).
1. Completion detection without callbacks: fire-and-forget provides no host-side completion notification. The scheduler performs polling-based completion detection entirely on device — polling prefill token-extraction buffers and per-step decode buffers. This is architecturally distinct from any host-mediated serving system where completion triggers host-side callback chains.
1. Lock-free DPU–GPU coordination: the ring buffer must be simultaneously writable by DPU via one-sided RDMA and readable by the persistent GPU kernel. Ownership transferred via atomic CAS on per-slot metadata, with memory fences ensuring RDMA-visible updates. Benign races tolerated by design — the scheduler may re-scan a slot concurrently freed, but CAS prevents double-claiming.

关键实现细节 #

1. TensorRT vs PyTorch graph memory: TensorRT CUDA graphs consume 2–3 MB each vs hundreds of MB for PyTorch-based graphs, enabling a cache of 650–1000 graphs within 2–4 GB. This ~100× size difference makes $O(1)$ graph selection from a precomputed lookup table practical — PyTorch-based persistent schedulers would hit a memory wall at equivalent cache density.
1. DPU tokenizer performance: merge rules in 64-byte-aligned flat hash table (4 KV pairs per L1D cache line), 16-byte-aligned symbol nodes, ARM NEON SIMD for byte classification at 16 bytes/cycle, pre-allocated thread-local buffers (zero heap allocation on request path). Achieves 8–19.7× speedup over HuggingFace tokenizer despite BlueField-3 ARM A78 having lower clock speed than Xeon, revealing how much Python/HuggingFace overhead dominates standard tokenization paths.

API 与部署 #

• API: OpenAI-compatible HTTP endpoints with SSE streaming. Same TensorRT engine compilation workflow as TRT-LLM.
• Hardware requirements: NVIDIA GPU (H100 tested) + NVIDIA BlueField-3 DPU + 200 Gbps RDMA link.
• Deployment: standalone system, not a plugin to existing frameworks. Migration requires full stack replacement + DPU procurement.
• Current scope: single GPU only (multi-GPU extension discussed as future work via GPU-native collectives or GPU-initiated RDMA). Models must fit in GPU memory. Chunked prefill, prefix caching, speculative decoding described as compatible extensions but not yet implemented.

Hardware implications #

Blink's design exposes three hardware gaps in current CUDA/NIC architecture:

1. The 120 fire-and-forget launch limit is an artificial constraint requiring complex workarounds. A proper device-side graph queue API with configurable depth would simplify persistent-kernel designs.
2. DPU–GPU coordination relies on RDMA polling. A hardware notification mechanism (GPU-to-NIC doorbell or interrupt) would eliminate polling overhead.
3. Device-side graph launch is limited to graphs captured by the same context. Cross-context launch would enable multi-GPU persistent scheduling without host mediation.