Native LLM and MLLM Inference at Scale on Apple Silicon — L2 #

§1 TL;DR #

vllm-mlx provides native LLM + multimodal inference on Apple Silicon via MLX, combining continuous batching (4.3× throughput at 16 concurrent), 21–87% higher throughput than llama.cpp on text models, and content-based prefix caching that eliminates redundant vision encoding — delivering 28× speedup on repeated image queries by caching both vision embeddings and KV state in unified memory.

§2 痛点 · 方法 · 结果 #

Q1 痛点 #

Apple Silicon's unified memory architecture (up to 192GB shared CPU/GPU, 546 GB/s bandwidth) is compelling for local LLM inference, but existing solutions are fragmented:

1. PyTorch MPS: Adapts CUDA-style ops to Metal — lacks native UMA optimization.
2. llama.cpp: Excellent single-stream text throughput via hand-tuned Metal kernels — but no continuous batching and no multimodal support.
3. vLLM-metal: Continuous batching via MLX — but no multimodal support or vision caching.
4. MLX/mlx-lm: Native Apple framework — but no serving features (batching, API).

The gap: no unified solution providing text + multimodal inference with continuous batching and vision caching on Apple Silicon. Additionally, vision-language models redundantly encode the same image across conversation turns (1.5–4s per encoding).

Q2 方法 #

vllm-mlx addresses both gaps:

1. Text inference: Wraps mlx-lm with continuous batching scheduler (Algorithm 1) + text prefix caching (Algorithm 2) + OpenAI-compatible API. Zero-copy tensor operations via UMA.
2. Multimodal inference with content-based prefix caching: SHA-256 hash over decoded pixel values identifies identical images regardless of input format (URL, base64, file path). Caches both vision embeddings and KV state. On cache hit, skips both vision encoder (1.5–4s) and prompt processing.

Memory management: LRU eviction with configurable limits (default 512MB).

核心技术壁垒: The content-based hashing strategy for multimodal prefix caching — hashing decoded pixel values rather than raw inputs enables cache hits across different image formats (URL/base64/file path). Combined with UMA's zero-copy semantics, this eliminates both the vision encoder forward pass and KV recomputation in a single mechanism. Prior multimodal caching (LMCache) operates as an external layer; vllm-mlx integrates caching directly into the inference engine.

Q3 结果 #

Text models (M4 Max, 128GB, 4-bit quantization):

• 525.5 tok/s on Qwen3-0.6B (1.87× vs llama.cpp).
• 21–87% higher throughput than llama.cpp across 10+ models (0.6B–30B).
• Continuous batching: 4.3× aggregate throughput at 16 concurrent requests.
• Text prefix caching: 5.8× TTFT speedup on 512-token shared prefixes.

Multimodal (Qwen3-VL-8B, 1024×1024):

• Turn 3+ latency: 21.7s → 0.78s (28× speedup).
• Video (32 frames): 24.7× cache speedup with 486MB cache.
• Ablation: vision embedding cache alone → 7.8×; KV cache alone → 1.2×; combined → 19× (multiplicative).

§3 架构 / 方法图 #



Paper's Figure 1, verbatim (caption: "Framework capability comparison. the proposed framework (green) provides comprehensive coverage: high throughput matching mlx-lm, continuous batching like vLLM-metal, OpenAI-compatible API, plus unique multimodal support with vision caching.").

The radar chart positions vllm-mlx as the only solution covering all six dimensions: single-stream throughput, continuous batching, MLLM support, vision caching, OpenAI API, and memory efficiency. llama.cpp excels at single-stream but lacks batching and multimodal; vLLM-metal has batching but no vision caching.

Architecture overview #

The system has three core mechanisms layered on MLX:

1. Continuous Batching Scheduler: Admits new requests at token boundaries (while $|B| < M$), generates one token for all active requests, and removes completed requests immediately. Maximizes GPU utilization.
1. Text Prefix Cache: Hierarchical SHA-256 lookup — first tries full-prompt match, then progressively shorter prefixes. Returns cached KV state + start position for incremental processing.
1. Content-Based Multimodal Cache: For each image, computes SHA-256 over decoded pixels. On hit, retrieves stored vision embeddings + KV state, skipping both vision encoder and prompt processing.

System scope #

• Stage coverage: Prefill + decode; continuous batching scheduler handles both.
• Serving vs training: Inference serving only.
• Parallelism: Single-device (Apple Silicon SoC). No tensor parallelism across GPU cores.
• Deployment mode: Single node (Mac device). OpenAI-compatible API for local serving.

KV / Memory manager #

• Allocation: Per-request KV cache, managed by MLX backend.
• Caching: Content-addressed (SHA-256 of tokens for text, SHA-256 of decoded pixels for images).
• Eviction: LRU with configurable memory limit (default 512MB for multimodal cache).
• UMA advantage: Zero-copy — cached KV state immediately accessible to GPU without transfer.

§4 作者证明 #

无形式化作者证明 — 仅实证

The paper provides no formal throughput or latency model. The three algorithms (continuous batching, text prefix cache, multimodal cache) are presented as pseudocode procedures, not mathematical analysis.

A model would have clarified:

• At what batch size does UMA bandwidth saturate for a given model size?
• What is the theoretical maximum cache hit rate as a function of conversation structure?
• How does vision encoding latency scale with resolution — is the quadratic ViT cost fully amortized by caching?

6 minimum checks:

1. ✅ Throughput comparison validated across 10+ models (Table 1): vllm-mlx > mlx-lm > llama.cpp in most cases.
2. ✅ Multimodal caching validated with cold/warm latency (Table 2): 21.7s → 0.78s on Turn 3+.
3. ✅ Ablation isolates cache components (Table 4): vision 7.8× + KV 1.2× → combined 19× (multiplicative).
4. ✅ Resolution scaling validated (Table 5): 6.7× → 13.1× speedup from 224×224 to 1024×1024.
5. ✅ Concurrency scaling validated (Fig 2): 3.7× at 16 concurrent (Qwen3-0.6B), diminishing returns for larger models.
6. ✅ One failure case acknowledged: Qwen3-30B-A3B shows vllm-metal slightly outperforming vllm-mlx (110.3 vs 109.7 tok/s).

§5 实验与数据 #

Setup: Apple M4 Max, 128GB unified memory. 4-bit quantization (Q4_K_M for GGUF, 4-bit for MLX). Models: Qwen3 (0.6B–30B), Llama 3.2 (1B–3B), Gemma 3-4B, Nemotron-30B-A3B.



Paper's Figure 2, verbatim (caption: "Concurrency scaling on vllm-mlx. (a) Aggregate throughput scales efficiently. (b) Request throughput increases with concurrency, showing efficient batching.").

Throughput scaling shows clear model-size dependence: Qwen3-0.6B achieves 3.7× at 16 concurrent requests (441 → 1642 tok/s), while Qwen3-8B reaches 2.6×. Larger models show diminishing returns due to memory bandwidth saturation — the fundamental bottleneck of UMA despite its zero-copy advantage.

Key experimental tables:

Result	Headline number	Evidence
Text throughput vs llama.cpp	1.21–1.87×	Table 1
Multimodal cache speedup (Turn 3+)	28× (21.7s → 0.78s)	Table 2
Video cache (32 frames)	24.7× (486 MB cache)	Table 6
Vision-only vs KV-only cache	7.8× vs 1.2×	Table 4
Resolution scaling	6.7× (224²) → 13.1× (1024²)	Table 5
Text prefix TTFT	5.8× (245ms → 42ms)	Table 7
Concurrency scaling	3.7× at 16 concurrent	Fig 2

Where vllm-mlx loses: Qwen3-30B-A3B MoE model — vllm-metal matches at 110.3 tok/s vs 109.7 tok/s. Suggests MoE routing may interact differently with MLX scheduling. No other clear loss across tested configurations.

Workload characterization #

Workload regime	vllm-mlx	Baseline (llama.cpp)	Why
Single-user, simple text	~1.2–1.9× better	Strong (hand-tuned Metal)	MLX zero-copy + lazy eval beats hand-tuned kernels
Multi-user concurrent text	3.7× scaling	No batching (sequential)	Continuous batching maximizes GPU utilization
Multi-turn multimodal (same image)	28× speedup	N/A (no multimodal)	Content-based cache eliminates redundant vision encoding
MoE models	Comparable	Comparable	MoE routing may not benefit from MLX optimizations

§6 论证链 #

Step	Claim	Evidence	Depends on
1	Apple Silicon UMA enables zero-copy tensor access, eliminating PCIe transfer overhead	§2.1: M4 Max 128GB/546 GB/s unified memory; CPU/GPU share physical memory pool	—
2	MLX natively exploits UMA better than PyTorch MPS or llama.cpp Metal	§2.2: lazy evaluation + native UMA design; Table 1: 21–87% higher throughput than llama.cpp	Step 1
3	Continuous batching fills UMA bandwidth gap at high concurrency	Fig 2: 3.7× aggregate throughput at 16 concurrent; diminishing returns at larger models due to BW saturation	Step 2
4	Content-based pixel hashing enables format-agnostic vision cache	§3.3: SHA-256 over decoded pixels catches URL/base64/file-path duplicates; Table 2: 28× on Turn 3+	Step 2
5	Vision embedding caching dominates multimodal speedup	Table 4: vision-only 7.8× vs KV-only 1.2× → combined 19× (multiplicative composition)	Step 4

§7 实现 cross-reference #

Open source: https://github.com/waybarrios/vllm-mlx

• Built on MLX framework + mlx-lm library.
• OpenAI-compatible API server.
• Continuous batching scheduler wrapping mlx-lm's single-stream inference.
• Content-based prefix cache with LRU eviction (configurable memory limit).

关键实现细节:

1. SHA-256 over decoded pixel values, not raw bytes: The cache key is computed after decoding the image (URL fetch, base64 decode, file read) into pixel tensor, then hashing. This ensures the same physical image always maps to the same cache entry regardless of encoding or transport format. Without this, a base64-encoded image and its URL would miss the cache.
2. MLX lazy evaluation for throughput: MLX's lazy evaluation fuses operations and reduces kernel launch overhead compared to PyTorch's eager execution. Combined with native quantization dequantization kernels, this accounts for the consistent throughput advantage over llama.cpp despite the latter's hand-tuned Metal shaders.

API & usability #

• User-facing API: OpenAI-compatible REST API — drop-in replacement for cloud services.
• Config surface: Minimal — model selection, quantization format, cache memory limit.
• Migration cost: Near-zero for OpenAI API users (LangChain, AutoGPT, CrewAI work without modification).

Deployment context #

• Serving stage: Prefill + decode.
• Concurrency regime: Low to mid (tested up to 16 concurrent; 25+ req/s on Qwen3-0.6B).
• Hardware affinity: Apple Silicon only (macOS). M4 Max evaluated; extends to M-series with UMA.
• Ecosystem integration: Standalone server; wraps MLX/mlx-lm. Independent from vLLM (shares no code despite name).
• Migration path: Install vllm-mlx, point existing OpenAI API clients to local endpoint.