FaaSMoE: A Serverless Framework for Multi-Tenant Mixture-of-Experts Serving — L2 #

§1 TL;DR #

FaaSMoE decomposes MoE inference into a lightweight orchestrator (attention + gating) and stateless expert blocks deployed as FaaS functions shared across tenants. Configurable expert-block granularity trades invocation overhead against per-expert elasticity. On Qwen1.5-MoE-2.7B with 6 tenants, achieves <1/3 total resource usage vs full-model-per-tenant baseline.

§2 Q1 / Q2 / Q3 #

Q1 — 痛点 #

MoE models activate only a small expert subset per token, yet all experts must reside in memory for every deployed instance. In multi-tenant settings this waste multiplies linearly: $N$ tenants × full expert set → $N \times$ memory, even when aggregate expert activation across tenants covers only a fraction of the total. Existing optimizations (offloading, caching, deduplication) reduce per-model cost but do not eliminate the fundamental residency-vs-activation gap across tenants.

Q2 — 方法 #

Core decomposition: separate MoE inference into two planes:

1. Orchestrator (control plane) — tokenizer, attention layers, gating network. Small memory footprint, manages routing decisions.
2. Expert blocks (compute plane on FaaS) — groups of experts packaged as stateless FaaS functions. On-demand invocation, scale-to-zero when unused, shared across all tenants.

Configurable expert granularity: instead of one-expert-per-function, $k$ experts are grouped into a single function (expert block). Larger $k$ → fewer invocations, better batching, but coarser scaling. Smaller $k$ → finer sharing, but more fan-out overhead. Paper finds $k=20$ (dividing 60 experts into 3 blocks per layer) as the sweet spot.

Orchestrator placement: shared (one instance, cross-tenant micro-batching) vs private (per-tenant, better isolation, no SPOF).

核心技术壁垒: The fundamental insight is recognizing that MoE expert activation is structurally isomorphic to serverless function invocation — both are stateless, sparse, event-driven, and benefit from scale-to-zero. Mapping one onto the other eliminates the always-resident memory tax without requiring custom infrastructure. The barrier is not algorithmic complexity but rather the systems engineering to make HTTP-based expert invocation efficient enough at MoE-layer granularity (24 layers × per-token routing).

Q3 — 结果 #

Metric	Baseline (6 copies)	FaaSMoE-Shared	FaaSMoE-Private	Local Dist.
Total CPU (%)	1126.84	326.40	408.49	428.67
Total Memory (GB)	217.52	72.25	90.98	50.38

FaaSMoE-Shared achieves the best combined efficiency. Local Distribution wins on raw memory but lacks elasticity and multi-tenant scaling. Optimal expert block size = 20 experts/function (U-shaped memory curve). FaaS platform overhead is modest relative to expert execution time.

§3 架构 / 方法图 #



Paper's Figure 1, verbatim (caption: "Architecture Overview: FaaSMoE decouples MoE inference into a lightweight control plane, i.e., the Orchestrator, and a distributed compute plane comprising MoE experts deployed as stateless FaaS functions.").

The architecture separates concerns along the stateful/stateless boundary. The orchestrator retains all sequential, state-dependent computation (attention with KV cache, gating decisions) while expert blocks — which are pure functions of their input tokens — execute on the FaaS platform. Token routing from gating triggers batched HTTP invocations to the appropriate expert-block functions. Results return to the orchestrator for residual addition before the next layer.

sequenceDiagram participant T as Tenant Request participant O as Orchestrator participant G as Gating Network participant F as FaaS Platform participant E as Expert Block T->>O: Input tokens O->>O: Tokenize + Attention O->>G: Hidden states G->>O: Top-k expert routing O->>F: Batched invocation (tokens → expert block IDs) F->>E: Dispatch to expert block function E->>F: Expert output F->>O: Aggregated results O->>O: Residual + next layer O->>T: Output tokens

System scope (framework-specific):

• Stage coverage: both prefill and decode (full MoE inference path)
• Serving mode: stateless per-request, no continuous batching (CPU-only prototype)
• Parallelism: expert parallelism via FaaS function fan-out; no TP/PP/SP
• Deployment mode: single-node FaaS platform (tinyFaaS), multi-tenant via shared expert pool

§4 作者证明 #

无形式化作者证明 — 仅实证

This paper contains no throughput/latency model, no cost equations, and no formal optimization. All claims rest entirely on empirical resource measurements. A formal model would have clarified:

1. Expected memory savings as a function of tenant count and expert activation overlap — when does FaaSMoE's advantage grow/shrink vs number of tenants?
2. Latency penalty model — serialization + network round-trip per layer × 24 layers; at what request rate does this dominate?
3. Optimal block size derivation — the U-shaped curve is observed empirically but no analytical explanation is offered for why $k=20$ minimizes memory.
4. Cold-start probability — given expert activation distributions, what is the expected cold-start rate under different keep-alive policies?
5. Break-even analysis — at what tenant count does FaaSMoE become cheaper than shared-nothing baseline?
6. Throughput ceiling — what is the maximum request rate before FaaS invocation overhead saturates the system?

§5 实验与数据 #

Deployment strategies comparison #



Paper's Figure 2, verbatim (caption: "Deployment Strategies: Baseline is the default deployment strategy, where a full MoE model is deployed per tenant. Local Distribution is a non-scaling deployment that separates orchestration from expert execution on a local server. FaaSMoE Private deploys a per-tenant orchestrator, while FaaSMoE Shared further enables cross-tenant orchestrator sharing.").

The four strategies span the design space from full replication (Baseline) through centralized expert sharing without FaaS (Local Distribution) to the two FaaSMoE variants that add elastic scaling. This figure makes the architectural differences concrete — particularly that Local Distribution is a degenerate case showing what decomposition alone buys without serverless elasticity.

Resource consumption results #



Paper's Figure 3, verbatim (caption: "Average total CPU and Memory Usage among different experiment settings with expert block size of 20.").

The key observation: FaaSMoE-Shared reduces both CPU and memory by ~70% vs Baseline. The gap between Shared and Private variants (~80 CPU% / ~18 GB memory) quantifies the cost of per-tenant orchestrator isolation. Notably, Local Distribution achieves the lowest absolute memory (50.38 GB) by avoiding FaaS runtime overhead — the price is zero elasticity and no multi-tenant scaling path.

FaaS overhead breakdown #



Paper's Figure 4, verbatim (caption: "FaaS consumption breakdown of FaaSMoE: Gateway and platform represent FaaS management consumption and worker represents experts execution.").

Expert execution (worker) dominates both CPU and memory in both FaaSMoE variants. Gateway and platform management add minimal overhead, validating that FaaS infrastructure cost is not the bottleneck. This is important because it means resource savings scale with expert count rather than being offset by fixed platform overhead.

Expert block size sensitivity #



Paper's Figure 5, verbatim (caption: "Average CPU and Memory Usage among different setups with varying block sizes. It shows the system overall consumption considering client sending requests and server-side experts processing.").

The U-shaped memory curve for FaaSMoE (minimum at block size 20) reveals the fundamental granularity trade-off: too fine-grained (size 6) → excessive function instances and runtime duplication; too coarse (size 30) → over-provisioned memory per function and reduced sharing opportunity. CPU behavior is non-monotonic for FaaSMoE but monotonically decreasing for Local Distribution, confirming that FaaS invocation overhead interacts non-trivially with batching efficiency.

Workload characterization #

Workload regime	FaaSMoE	Baseline	Why
Multi-tenant, low per-tenant load (6 clients, 5 tasks each)	Wins: <1/3 resources	Loses: linear duplication	Expert sharing eliminates per-tenant residency cost
Single-tenant, sustained load	Loses: FaaS invocation overhead, no sharing benefit	Wins: no network/serialization	Decomposition overhead not amortized without multi-tenancy
Latency-sensitive workloads	Unknown: latency not measured	Likely better: no network hops	24 layers × HTTP round-trip per MoE layer is potentially severe
High-concurrency contention	Unknown: only 6 clients tested	Unknown	Cold-start and expert contention behavior unexplored

Evaluation methodology notes #

• Hardware: CPU-only server, 300 GB RAM — no GPU. Real MoE serving is GPU-bound.
• Scale: 6 tenants, 30 total requests (5 per tenant from BIG-Bench). Extremely small.
• Baseline fairness: Baseline is "full model per tenant" — a worst-case straw-man. No comparison against production-grade shared-model serving (e.g., vLLM with multi-tenant routing).
• Missing metrics: no latency, no throughput (tokens/s), no cold-start measurement.

§6 论证链 #

Step	Claim	Evidence	Validity
1	MoE expert activation is sparse: only top-k of N experts used per token	Standard MoE architecture property; Qwen1.5-MoE activates 4/60 experts per layer	Well-established fact
2	Multi-tenant deployment multiplies inactive expert memory linearly	$N$ tenants × full expert set in memory, independent processes	Direct implication of Step 1 + isolated deployment
3	Expert computation is stateless and independent → maps to FaaS functions	Experts are pure MLPs with no cross-request state	Valid architectural observation
4	FaaS platforms provide on-demand scale-to-zero and cross-tenant resource sharing natively	FaaS definition (§2.2); tinyFaaS implementation	Platform property
5	Expert blocks as FaaS functions enable shared expert pool with elastic scaling	Steps 3 + 4; orchestrator retains routing logic	Architectural design, validated by prototype
6	Shared expert pool reduces total resource usage vs per-tenant replication	Fig. 3: 72.25 GB vs 217.52 GB memory, 326.4% vs 1126.84% CPU	Empirically demonstrated at small scale
7	Expert block granularity is a tunable parameter with non-trivial optimum	Fig. 5: U-shaped memory curve, minimum at block size 20	Observed but not analytically explained

Gap: Steps 1–5 form a sound architectural argument. Step 6 validates at toy scale only (6 tenants, CPU, 30 requests). No evidence for production-scale generalization, latency acceptability, or GPU workloads. Step 7 is an observation without theory.

§7 实现 cross-reference #

Open-source implementation: https://github.com/Mhwwww/FaaSMoE

• FaaS platform: tinyFaaS (Pfandzelter et al., 2020) — lightweight edge-oriented serverless platform
• Model: Qwen1.5-MoE-2.7B, experts extracted per MoE layer, packaged into configurable blocks
• Invocation: asynchronous HTTP calls; tokens routed to same expert block consolidated into one batched invocation (token-level micro-batching)
• Language: Python prototype
• Hardware: CPU-only server, 300 GB RAM + disk

核心技术壁垒 (implementation perspective): The hard problem is not the decomposition itself but making HTTP-based per-layer expert invocation viable at inference speed. With 24 MoE layers each requiring a network round-trip, serialization/deserialization of hidden states dominates latency. The paper acknowledges this but does not solve it — production viability requires either (a) high-speed RPC replacing HTTP, (b) co-located deployment eliminating network hops, or (c) speculative expert pre-fetching to overlap communication with computation.

关键实现细節:

1. Token-level micro-batching: all tokens routed to the same expert block are consolidated into a single invocation, amortizing per-call overhead. Without this, fan-out would be token × expert-block invocations per layer.
2. Expert block configuration flexibility: block composition (which experts share a function) is configurable per layer and expert ID — enabling non-uniform grouping if expert activation correlations are known.

Deployment context:

• Serving stage: full inference path (prefill + decode), all MoE layers
• Concurrency regime: tested at low concurrency (6 concurrent tenants)
• Hardware affinity: CPU-only; no GPU integration discussed
• Ecosystem integration: standalone prototype on tinyFaaS; no integration with vLLM/SGLang/TRT-LLM
• Migration path: requires complete re-architecture from monolithic model serving to decomposed orchestrator + FaaS expert pool — high integration cost