mori-scheduler — L2 Distillation #

§1 TL;DR #

Rust-based prefix-cache-aware scheduler/router for LLM inference engines (SGLang, vLLM, ATOM). Routes requests via radix-tree or UMBP block-hash matching to maximize KV cache reuse. Supports PD disaggregation with coordinated prefill/decode dual-dispatch over RDMA. Achieves 91.6 req/s vs 23.7 for round-robin (+286%) on 2P+2D MI355X.

§2 Project Identity #

§3 What & Why — Motivation #

Q1 痛点 #

LLM inference routers treat requests as stateless: round-robin or random dispatch ignores the fact that inference engines maintain per-GPU KV caches. When repeated or prefix-sharing prompts (system prompts, multi-turn agents, RAG) scatter across workers, each worker recomputes the same prefill from scratch — wasting GPU compute and HBM bandwidth. Existing routers (sgl-router, vllm-router) lack cross-engine prefix-cache-aware routing, PD disaggregation support, and integration with distributed KV cache indices like UMBP.

Q2 方法 #

A centralized Rust scheduler that sits between clients and inference workers with four layered capabilities:

1. Prefix-cache-aware routing — a compressed ByteRadixTree tracks which byte prefixes live on which worker; requests are routed to the worker with the longest matching prefix, falling back to shortest-queue on cache miss or load imbalance.
2. LinearCost scoring — a 7-weight cost function over live /v1/loads telemetry ($\text{score}(w) = w_r \cdot R + w_w \cdot W + w_t \cdot T - b_c \cdot C - b_p \cdot P$) that blends load balancing with cache affinity.
3. UMBP block-hash matching — queries an external distributed KV cache index (UMBP master via gRPC) for exact block-hash matches, routing to the node with the most cached blocks.
4. PD disaggregation — dual-dispatch to separate prefill and decode worker pools with coordinated cancellation (PdSession), bootstrap room negotiation for RDMA KV transfer, and independent policy chains per role.

核心技术壁垒: The per-message segment tokenize cache (moka W-TinyLFU) that operates at chat message granularity rather than full-prompt level. For multi-turn agent workloads, only new/changed messages need BPE tokenization; combined with the chained block-hash computation matching each engine's exact algorithm (5 hash schemes across 4 engine kinds), this enables sub-millisecond prompt-to-routing-decision latency even for long conversations. No competing router replicates engine-specific hash chains at the routing layer.

Q3 结果 #

• 2P+2D MI355X (Qwen2.5-7B-FP8): linear_cost achieves 91.6 req/s vs 23.7 for round_robin (+286%)
• Verified on AMD MI355X + Pensando ionic 100GbE RoCE with RDMA KV transfer via ROCm/mori
• GSM8K accuracy parity between baseline and 2P+2D router (no accuracy regression from disaggregation)
• 260 commits → production-verified 4P+4D disaggregated serving in 35 days

§4 Architecture & Module Map #

graph TB subgraph "Clients" OAI["OpenAI-compat
(HTTP/gRPC)"] end subgraph "mori-sched (Rust)" API["api/mod.rs
axum HTTP router
+ tonic gRPC server"] PIPE["pipeline/
Ingress → Hash → Ready"] SCHED["scheduler/mod.rs
Global BatchScheduler
(single dispatch brain)"] POLICY["policy/mod.rs
RoundRobin | CacheAware
| LinearCost"] SPOL["scheduler/sched_policy.rs
PrefillPolicy chain
+ DecodePolicy chain"] RADIX["policy/radix.rs
ByteRadixTree
(compressed, LRU)"] ENGINE["engine/mod.rs
EngineAdapter trait
(SGLang|vLLM|ATOM)"] REG["registry/
WorkerRegistry
+ auto-discovery"] HEALTH["health/
2-fail-down/3-pass-up
+ adaptive poll"] CACHE["cache/
TokenizeCache (moka)
+ DetokenizeCache"] UMBP_C["umbp/
UmbpClient gRPC
+ NodeIdResolver"] TRANS["transport/
HTTP proxy + gRPC
dispatch + client"] PD["scheduler/pd_session.rs
PdSession + PdRegistry
(coordinated cancel)"] end subgraph "Inference Workers" SGL["SGLang
(GPU)"] VLLM["vLLM
(GPU)"] ATOM["ATOM
(GPU)"] FAKE["FakeEngine
(Python sim)"] end subgraph "External" UMBP_M["UMBP Master
(KV cache index)"] SQLITE["SQLite
(load history)"] end OAI --> API API --> PIPE PIPE --> SCHED SCHED --> SPOL SPOL --> POLICY POLICY --> RADIX SCHED --> UMBP_C SCHED --> PD SCHED --> TRANS TRANS --> SGL & VLLM & ATOM & FAKE HEALTH --> SGL & VLLM & ATOM UMBP_C --> UMBP_M HEALTH --> SQLITE PIPE --> ENGINE ENGINE --> CACHE REG --> ENGINE

Top-level module responsibilities:

• api/ — axum HTTP router exposing OpenAI-compatible inference endpoints, cluster management, cache flush, profiling, and workers REST API. Also serves Prometheus /metrics.
• pipeline/ — staged request processing: IngressQueue → HashPipeline → ReadyQueue. Async tokenization and block-hash computation run on dedicated blocking threads. Error handler with retry + dead-letter queue + circuit breaker.
• scheduler/ — the single global BatchScheduler loop: pops from ReadyQueue, consults policy chains, batches by dp_size, dispatches via HTTP or gRPC. Handles both single-worker and PD disaggregated modes.
• policy/ — LoadBalancingPolicy trait with three implementations (RoundRobin, CacheAware, LinearCost) plus the ByteRadixTree data structure.
• scheduler/sched_policy.rs — separate PrefillPolicy / DecodePolicy trait hierarchies with chain-of-responsibility pattern. Pin → UMBP → CacheAware → ShortestQueue fallback.
• engine/ — EngineAdapter trait abstracting hash algorithms, tokenization, and block sizes per engine kind. Four concrete adapters: SGLang (Sha256FullChain), FakeSGLang (Sha256Chain), vLLM (Sha256Cbor/XxhashCbor), ATOM (Xxhash64Chain).
• registry/ — WorkerRegistry with auto-discovery (probes /server_info), connection pooling, and capability detection.
• health/ — per-worker health polling (2-fail-down/3-pass-up state machine) with adaptive intervals, load anomaly detection (stuck/KV-full/overloaded), latency EMA, dispatch-failure degradation, activity-based health inference, and SQLite-backed load history.
• cache/ — segment-level tokenize cache (moka W-TinyLFU) for per-message dedup in multi-turn conversations; detokenize cache for gRPC workers.
• umbp/ — gRPC client to UMBP master for cross-node KV cache block matching; NodeIdResolver maps UMBP node IDs to worker indices.
• transport/ — HTTP reverse proxy (reqwest streaming, h2c), gRPC dispatch (tonic), and gRPC server (SGLang wire format).

§5 Entry Points & API Surface #

Public API (OpenAI-compatible) #

Endpoint	Method	Purpose
/v1/completions	POST	Text completion (passthrough)
/v1/chat/completions	POST	Chat completion (passthrough)
/v1/models	GET	List available models

Router management #

Endpoint	Purpose
GET /router/health	Liveness probe
GET /router/status	Policy config, queue depth, pipeline state, PD sessions
GET /cluster/metrics	Prometheus exposition format
POST /router/config/reload	Hot-reload YAML config
`GET\	PUT /router/config/log-level`	Dynamic log level

Workers REST #

Endpoint	Purpose
GET /workers	List all workers with health/load
GET /workers/:id	Detailed worker info (capabilities, connection mode)
POST /workers/:id/cache/L1/clear	Flush GPU HBM KV cache
POST /workers/:id/cache/L2/clear	Clear Host DRAM (HiCache)
POST /workers/:id/cache/L3/clear	Clear external storage (Mooncake/UMBP/SSD)
POST /workers/:id/abort	Abort in-flight request on worker
`POST /workers/:id/pause\	resume`	Pause/resume generation

CLI #

mori-sched --config config.yaml [--listen 0.0.0.0:8080]
           [--server-mode http|grpc|both] [--grpc-listen 0.0.0.0:8090]
           [--worker-threads N] [--tokenize-threads N]
           [--log-level info] [--log-file /path] [--log-format json]

Configuration (top 10 most important) #

Mechanism	Key	Purpose
YAML	policy.kind	Scheduling policy: round_robin / cache_aware / linear_cost
YAML	policy.prefill_policy / decode_policy	Independent policies for PD mode
YAML	policy.linear_cost.*	7 tunable weights for LinearCost scoring
YAML	workers[].engine_kind	Engine type: sglang / vllm / atom / fake-sglang
YAML	workers[].role	Worker role: regular / prefill / decode
YAML	pipeline.hash_concurrency	Parallel hash workers in pipeline
YAML	umbp.master_addr	UMBP master gRPC address
YAML	health.fail_threshold / pass_threshold	Health state machine thresholds
Env	MORI_LOADS_DB	SQLite path for load history persistence
CLI	--server-mode both	Enable HTTP + gRPC dual-mode

§6 Core Data Structures #

SchedulerCtx (scheduler dispatch brain) #

• Definition: scheduler/mod.rs
• Layout: aggregates 13 Arc-wrapped subsystem handles (inner, queue, selector, policies, caches, adapter, umbp, health, profiler, pipeline, inflight)
• Lifecycle: created once at startup, moved into the single scheduler tokio task, lives for the process lifetime
• Thread safety: all fields are Arc where T is Send + Sync; the scheduler loop is single-threaded but spawns concurrent dispatch tasks

Inner (hot-swappable routing state) #

• Definition: api/mod.rs
• Layout: registry: Arc, policy: Arc, resolver: NodeIdResolver
• Lifecycle: created at startup, atomically swapped on SIGHUP or POST /admin/reload via ArcSwap
• Mutation: replaced atomically; never mutated in place. In-flight requests see a consistent snapshot via arc_swap::Guard

ByteRadixTree (prefix cache index) #

• Definition: policy/radix.rs
• Layout: compressed Patricia trie where each node stores edge bytes, children HashMap, and tenants HashMap (worker→last_use tick)
• Lifecycle: created when CacheAware policy is built, lives for process lifetime, grows on every dispatched request
• Thread safety: single RwLock — reads on policy hot path, writes after dispatch. Background LRU eviction runs periodically.

WorkerLoad (lock-free health snapshot) #

• Definition: health/mod.rs
• Layout: ~30 AtomicU64/AtomicU8 fields covering running/waiting reqs, token usage, cache hit rate, health state machine, degradation flags, latency EMA, dispatch accounting
• Lifecycle: created per worker via WorkerHealthRegistry::get_or_init, lives for process lifetime
• Thread safety: fully lock-free — health poller writes atomics, policy hot path reads atomics. Floats encoded as basis points (×10000) to avoid mutex.

QueueEntry (per-request state) #

• Definition: scheduler/queue.rs
• Layout: req_id, path, body (JSON), prompt_bytes, precomputed block hashes, session/user/pin headers, retry count, oneshot response channel
• Lifecycle: created from PipelineEntry after hash computation, consumed by dispatch, dropped after response sent
• Ownership: owned by the scheduler task; response_tx is a oneshot channel back to the HTTP handler

§7 Critical Path Analysis #

Request flow: client → response #

HTTP POST /v1/chat/completions
  │
  ├─ [1] passthrough() in api/mod.rs
  │   ├─ extract_prompt_bytes() — Jinja2 render or ChatML fallback
  │   ├─ extract_session_id() from x-mori-session-id header
  │   └─ push PipelineEntry to IngressQueue, await oneshot rx
  │
  ├─ [2] HashPipeline (async, blocking thread pool)
  │   ├─ tokenize via EngineAdapter (HF tokenizer or byte-level)
  │   ├─ segment cache lookup (moka W-TinyLFU, per-message dedup)
  │   ├─ block_hashes = adapter.hash_prompt(tokens)
  │   └─ push HashedEntry to ReadyQueue
  │
  ├─ [3] spawn_global_scheduler loop (THE single dispatch brain)
  │   ├─ pop_one() from ReadyQueue
  │   ├─ load Inner snapshot (arc_swap)
  │   ├─ if has_pd_workers → dispatch_pd()
  │   │   ├─ PrefillPolicy chain: Pin → UMBP/CacheAware → SQ
  │   │   ├─ DecodePolicy chain: Pin → SQ/RR
  │   │   ├─ batch up to dp_size requests
  │   │   ├─ generate_bootstrap_room() (xorshift PRNG)
  │   │   ├─ mutate prefill/decode bodies with PdMeta
  │   │   └─ spawn concurrent prefill + decode dispatch tasks
  │   └─ else → dispatch_single()
  │       ├─ PrefillPolicy chain selects worker
  │       ├─ batch up to dp_size
  │       └─ forward via HTTP proxy or gRPC
  │
  ├─ [4] proxy_post() or grpc_dispatch()
  │   └─ stream response body back through oneshot channel
  │
  └─ [5] passthrough() receives response, streams to client
      └─ attach x-mori-* headers (worker, policy, block hashes, UMBP matches)

Latency budget (measured at 2P+2D MI355X):

• Steps 1-2 (ingress + hash): typically <1ms for cached prompts (segment cache hit)
• Step 3 (policy selection): <10μs (sync trait, no async)
• Step 4 (upstream dispatch): dominated by inference engine latency
• Total routing overhead: <2ms end-to-end

The README claims the hot path is O(L) in prompt length for the radix tree lookup + O(N) in block count for hash computation. Code confirms: longest_match walks the tree once, hash_prompt iterates token chunks. No hidden quadratic behavior.

§8 API Design Decisions #

Why a single global scheduler task instead of per-worker queues?

Centralization enables global batch formation (dp_size batching across requests), UMBP-aware routing that considers all workers simultaneously, and consistent load-balancing decisions. The single-task design eliminates coordination overhead between per-worker schedulers. The trade-off is a potential bottleneck at extreme QPS — but since select() is sync and <10μs, the scheduler can dispatch >100K decisions/sec, far beyond current inference engine throughput.

Why separate PrefillPolicy / DecodePolicy trait hierarchies?

Prefill benefits from prefix-cache affinity (TTFT optimization), while decode benefits from load balancing (ITL optimization). Tying them to the same policy would force a compromise. The chain-of-responsibility pattern (Pin → UMBP → CacheAware → SQ) allows operators to mix concerns: pin headers override everything, UMBP provides cross-node cache awareness, and shortest-queue is the ultimate fallback.

Why 5 hash algorithms for 4 engine kinds?

Each engine computes KV block hashes differently (SHA-256 with truncated vs full-digest chaining, CBOR vs raw byte encoding, xxhash64 vs SHA-256). The router must replicate each engine's exact hash scheme to compare locally-computed hashes against engine-reported ones. Cross-engine hash namespace separation prevents silent incorrect cache hits.

Why ArcSwap for hot reload instead of mutex?

The routing state (registry + policy) is read on every request but written only on config reload (rare). ArcSwap provides lock-free reads with atomic pointer swap on writes — the optimal pattern for read-heavy / write-rare access. In-flight requests see a consistent snapshot via the Guard returned by load().

Why segment-level tokenize cache instead of full-prompt cache?

Agent multi-turn workloads share system prompts and earlier conversation turns. A full-prompt cache misses on every new user message. Segment-level caching (per chat message) deduplicates the shared prefix and only tokenizes the new message, reducing BPE tokenization from O(total_tokens) to O(new_tokens) per request.

§9 Implementation Highlights #

1. Engine-specific hash chain replication #

The EngineAdapter trait carries the full hash-chaining state across blocks. For SGLang's Sha256FullChain, the 32-byte digest chains into the next block (not the truncated i64). For vLLM's Sha256Cbor, CBOR-encoded tuples are hashed. For ATOM, xxhash64 with int64 token encoding. Each adapter's hash_prompt_with_parent() maintains an internal parent_buf: Option> accumulator sized to the full digest width — this is why per-block hashing cannot be parallelized.

2. PdSession coordinated cancellation #

When PD disaggregation is active, prefill and decode tasks run concurrently. If decode fails, the prefill must be cancelled (it's wasting GPU compute). PdSession wraps a tokio_util::sync::CancellationToken shared between both tasks. On decode failure, mark_decode_failed() triggers the token, and the prefill task's tokio::select! arm fires immediately.

3. Thread-local xorshift for bootstrap room generation #

generate_bootstrap_room() uses a thread-local xorshift64 PRNG seeded from nanosecond clock XOR'd with process ID. This avoids atomic contention on the global request counter and produces uniform room IDs for RDMA KV transfer coordination — each room is a unique rendezvous point between one prefill and one decode worker.

4. Adaptive health poll intervals #

The load poller adjusts its interval based on load delta: large changes (>20% relative) halve the interval (down to 100ms), stable readings slowly double it (up to 5s), and idle workers (0 running, 0 waiting) jump to max interval. This reduces unnecessary polling of quiet workers while catching load spikes quickly.

关键实现细节 #

1. RoundRobin PD aliasing fix: Using a single atomic counter for both prefill and decode selection in PD mode causes every request to pin to the same worker pair (counter ticks twice per request, idx % 2 stays in lockstep). The fix: dedicated prefill_cursor and decode_cursor atomics so each pool rotates independently.
1. Radix tree tenant inheritance on split: When an edge split creates an intermediate node, the new intermediate must inherit all tenants from the deeper node — otherwise longest_match loses presence at the shallower depth. This is a subtle invariant that, if violated, causes cache-aware routing to silently degrade to shortest-queue for shared prefixes.

§10 Concurrency & Memory #

• Concurrency model: tokio multi-threaded runtime (worker_threads = available parallelism). One global scheduler task, multiple concurrent dispatch tasks (one per in-flight request). Hash pipeline uses spawn_blocking on a dedicated thread pool (tokenize_threads = 2× parallelism).
• Lock hierarchy: nearly lock-free on the hot path. The only lock is RwLock on the radix tree (read-heavy: policy reads; write-rare: after dispatch). All health state is atomic. ArcSwap for config reload. DashMap (sharded concurrent map) for health registry.
• Memory management: all major structures are Arc-wrapped with no manual memory management. moka cache handles eviction (W-TinyLFU). Radix tree has a soft node cap (max_tree_nodes, default 10K) enforced by periodic LRU eviction (20% of leaves dropped when threshold exceeded).
• Known concurrency considerations: the single scheduler task is intentionally serialized — no concurrent pop_one() — to ensure consistent batch formation. Dispatch tasks spawned by the scheduler run concurrently and are fire-and-forget.

§11 Performance Characteristics #

Headline benchmarks (from README) #

Configuration	Policy	Throughput
2P+2D MI355X, Qwen2.5-7B-FP8	linear_cost	91.6 req/s
2P+2D MI355X, Qwen2.5-7B-FP8	round_robin	23.7 req/s
2P+2D MI355X, Qwen2.5-7B-FP8	cache_aware	85.2 req/s

Scaling characteristics #

• Workers: linear throughput scaling with worker count (single scheduler is not the bottleneck at current QPS ranges)
• Prompt length: hash computation is O(tokens/block_size) — longer prompts produce more blocks but segment cache amortizes repeated prefixes
• dp_size: batch scheduling groups up to dp_size requests per dispatch, improving GPU utilization on DP-attention workers

Routing overhead #

• Policy select(): <10μs (sync, no allocation on hot path)
• Radix tree longest_match: O(L) where L = prompt bytes, single RwLock read
• LinearCost score(): O(N) where N = worker count, lock-free atomic reads
• Hash computation: dominated by SHA-256/xxhash over token chunks — segment cache makes this effectively O(1) for repeated prefixes

§12 Tech Debt & Code Quality #

• Testing: 25+ unit tests across policy/mod.rs, scheduler/sched_policy.rs, engine/mod.rs, health/mod.rs. Tests cover policy rotation, PD aliasing, UMBP routing, hash reference vectors. No integration test harness beyond shell scripts.
• Linting: .github/workflows/pylint.yml for Python sources. No clippy CI visible (Rust #![allow(dead_code)] used liberally for future APIs).
• CI matrix: Only pylint. No Rust CI (build, test, clippy) in GitHub Actions.
• Known warts: Single-crate monolith at 19.6K LOC — mori-sched/src/ has 25 source files across 9 modules. Ripe for workspace partitioning into mori-policy, mori-engine, mori-health, etc.
• #![allow(dead_code)]: Present in policy/mod.rs, policy/radix.rs, engine/mod.rs, health/mod.rs — many APIs are defined ahead of use. Indicates rapid forward development.
• Dependency health: all major deps (tokio, axum, tonic, moka, serde, sha2) are actively maintained and on recent versions. rusqlite bundled (statically linked SQLite).

§13 Comparison with Alternatives #

Dimension	mori-scheduler	sgl-router	vllm-router
Language	Rust	Rust	Python
Prefix-cache routing	ByteRadixTree + UMBP block-hash	Session-based only	Hash-ring (semantic)
Engine support	SGLang + vLLM + ATOM	SGLang only	vLLM only
PD disaggregation	Full (dual-dispatch, coordinated cancel)	No	No
Hash schemes	5 (per-engine exact match)	1 (SGLang only)	0 (no hash routing)
Cross-node KV index	UMBP gRPC integration	No	No
Hot reload	ArcSwap + SIGHUP	Restart required	Config file
Health detection	6 degradation modes + adaptive poll	2-fail-down only	Basic health check
Three-tier cache API	L1/L2/L3 per worker	Flush only	No
gRPC serving	Dual HTTP+gRPC	HTTP only	HTTP only
Maturity	35 days, 260 commits	~1 year	~6 months
License	Apache-2.0	Apache-2.0	Apache-2.0

Bold = winner per row.

mori-scheduler wins on multi-engine support, PD disaggregation, and prefix-cache routing depth. sgl-router wins on maturity and production battle-testing. vllm-router is the simplest option for vLLM-only deployments.

§14 Verdict & Recommendations #

When to adopt #

• Running PD disaggregated serving on AMD MI355X with RDMA KV transfer
• Need prefix-cache-aware routing across SGLang + vLLM + ATOM engines in a mixed fleet
• Multi-turn agent workloads where segment-level tokenize caching provides significant dedup benefit
• Deployments requiring cross-node KV cache coordination via UMBP

When NOT to adopt #

• Single-engine SGLang deployments — sgl-router is simpler and more mature
• NVIDIA-only clusters without AMD-specific UMBP/mori RDMA stack
• Need a battle-tested production router — 35 days old with no public CI beyond pylint
• Prefer Python ecosystem for router extensibility — mori-scheduler is pure Rust

Suggested contributions #

1. Workspace partitioning: split the 19.6K LOC single crate into 5-6 crates (mori-policy, mori-engine, mori-health, mori-transport, mori-pipeline) for faster incremental builds and clearer API boundaries
2. Rust CI: add cargo test, cargo clippy, and cargo fmt --check to GitHub Actions
3. Benchmark CI: automated A/B benchmarks on every PR to catch throughput regressions

§15 论证链 #

Step	Claim	Evidence	Validity
1	Stateless routing (RR) wastes KV cache across workers	Same prefix dispatched to different workers forces re-prefill; measured 23.7 vs 91.6 req/s on 2P+2D MI355X	Valid — 3.9× throughput difference on identical hardware, same model
2	Byte-level radix tree approximates prefix-cache state without engine instrumentation	CacheAware::select() matches prompt bytes against tree, routes to longest-match worker; falls back to SQ on miss	Valid — O(L) lookup; on_dispatched keeps tree in sync; imbalance fallback prevents stampede
3	LinearCost scoring integrates live load signals with cache affinity	7-weight cost function with stale-fallback to local inflight count; num_waiting_reqs weighted 2× vs num_running_reqs	Valid — waiting is a stronger saturation signal; stale fallback ensures routability during startup
4	Engine-specific hash replication enables exact KV block matching	5 HashAlgorithm variants with reference-vector tests against Python implementations; ATOM xxhash64 matches Python to the bit	Valid — test atom_hash_matches_python_reference proves byte-exact match for ATOM; sglang_full_chain_matches_reference for SGLang
5	PD disaggregation with coordinated cancellation avoids wasted GPU work	PdSession wraps CancellationToken; decode failure cancels prefill via tokio::select!	Valid — without cancellation, a failed decode leaves the prefill running to completion for nothing
6	Segment-level tokenize cache reduces BPE cost for multi-turn	moka W-TinyLFU at chat message granularity; only new messages tokenized; cross-request prefix scan	Valid for agent workloads — system prompt + earlier turns are cached; cold start amortized over conversation

§16 实现 Cross-Reference #

Component	Key file	Purpose
CLI entrypoint	mori-sched/src/main.rs	clap CLI, tokio runtime, startup orchestration
Config parsing	mori-sched/src/config.rs	YAML config: workers, policy, cache, logging, UMBP, pipeline, health
HTTP router	mori-sched/src/api/mod.rs	axum routes: OpenAI-compat + cluster + admin + workers REST
Global scheduler	mori-sched/src/scheduler/mod.rs	spawn_global_scheduler: single dispatch loop
Prefill/Decode policies	mori-sched/src/scheduler/sched_policy.rs	Chain-of-responsibility: Pin → UMBP → CacheAware → SQ
PD session	mori-sched/src/scheduler/pd_session.rs	Coordinated prefill/decode cancellation
Routing policies	mori-sched/src/policy/mod.rs	LoadBalancingPolicy trait: RoundRobin, CacheAware, LinearCost
Radix tree	mori-sched/src/policy/radix.rs	ByteRadixTree: compressed Patricia trie with LRU eviction
Engine adapters	mori-sched/src/engine/mod.rs	EngineAdapter trait: hash algorithms, tokenization per engine
Health polling	mori-sched/src/health/mod.rs	2-fail-down/3-pass-up + adaptive poll + anomaly detection
Load history	mori-sched/src/health/load_history.rs	SQLite-backed 7-day load history
Tokenize cache	mori-sched/src/cache/tokenize_cache.rs	moka W-TinyLFU per-message segment cache
Pipeline stages	mori-sched/src/pipeline/hash_pipeline.rs	Async tokenize + hash workers
Error handler	mori-sched/src/pipeline/error_handler.rs	Retry + dead-letter queue + circuit breaker
HTTP proxy	mori-sched/src/transport/http_proxy.rs	reqwest streaming reverse proxy (HTTP/1.1 + h2c)
gRPC dispatch	mori-sched/src/transport/grpc_dispatch.rs	JSON→GenerateRequest + streaming bridge
UMBP client	mori-sched/src/umbp/client.rs	tonic gRPC client to UMBP master
Fake engine	fake-engine/sglang_fake_engine.py	FastAPI SSE + ZMQ PUB simulation (no GPU)
Benchmark client	fake-engine/bench_client.py	Zero-dep concurrent benchmark client
Protobuf schemas	mori-sched/proto/sglang_scheduler.proto, umbp.proto	gRPC wire formats