KRCore virtualizes pre-initialized kernel-space RDMA DCT connections to achieve 5.4μs connection setup (vs 15.7ms verbs), using fixed O(1) memory regardless of cluster scale, while preserving low-level verbs API compatibility for existing RDMA optimizations.
RDMA connection setup (control plane) is 15,700X slower than its data path — 15.7ms to create one RCQP vs ~1μs for a data operation. This bottleneck is critical for elastic computing (disaggregated storage adding nodes on-demand, serverless with ephemeral containers). The cost is dominated by hardware resource configuration on the NIC (87% of create_qp time), not network handshakes (only 2.4%). Existing kernel-space solution LITE still takes 2ms per connection on cache miss, consumes 1.52GB per node for 10k-node clusters, and exposes an inflexible high-level API.
Core insight (核心技术壁垒): DCT (Dynamic Connected Transport) allows a single RDMA QP to communicate with different hosts through hardware-managed reconnection in <1μs — virtualizing pre-initialized kernel-space DCT connections lets applications skip the entire costly QP creation and configuration path.
Three-part design:
Execution flow for a new connection to target S1:
qconnect(VQP, S1_gid, port)ibv_post_send immediatelyScale: system scope is rack/pod-scale datacenter (10-10k+ nodes), 100Gbps InfiniBand. Workload: elastic storage/serverless with bursty connection patterns. Hardware class: Mellanox ConnectX-4/5/6/7 with DCT support (Connect-IB onward).
无形式化数学模型 — paper presents algorithmic pseudocode (Algorithm 1: VQP creation/connection; Algorithm 2: virtualized post_send/poll_cq) with empirical validation rather than formal proofs.
| Symbol | Meaning |
|---|---|
| RCQP | Reliable Connected Queue Pair (one-to-one, full RDMA) |
| DCQP | Dynamically Connected QP (one-to-many, hw reconnect) |
| VQP | Virtual QP — KRCore's user-facing abstraction |
| DCT | Dynamic Connected Transport (RNIC feature) |
| $wr\_id$ | Work request identifier, overloaded to encode VQP + completion count |
| $uncomp\_cnt$ | Outstanding unsignaled requests counter per VQP |
| # | Claim | Verification |
|---|---|---|
| 1 | DCT reconnection <1μs | Hardware specification from Mellanox (ref [1], OpenFabrics 2014). Validated: KRCore(DC) sync latency 3.24μs includes syscall(1μs) + RDMA op (~2μs), leaving <0.5μs for DCT overhead |
| 2 | RNIC dominates QP creation cost (87%) | Figure 3(b) breakdown: 361μs of 413μs create_qp is hardware queue allocation. Cross-validated: ConnectX-6 still 17ms (§6), confirming NIC firmware, not driver, is bottleneck |
| 3 | Memory scaling O(1) vs O(N) | DCT metadata 12B/node → 17KB/1000 nodes. RCQP 159KB each → 1.52GB/10k nodes. Ratio grows linearly with cluster size |
| 4 | Pre-check prevents QP corruption | Algorithm 2: checks queue capacity (line 7: polls to clear), validates opcode + MR (line 13), force-signals last unsignaled request (lines 24-26). LITE fails at >6 threads; KRCore handles arbitrarily many |
| 5 | Meta server CPU-bypass advantage | One-sided RDMA READ bypasses remote CPU entirely. Figure 9(a): 11.8X throughput and 13X lower latency vs kernel-space RPC (bottlenecked by server CPU scheduling) |
| 6 | Hybrid DC→RC benefit measurable | Figure 16 time 2.2→3: RACE throughput jumps from 18M to 26M req/sec (1.4X) after transparent RC upgrade. Matches verbs peak, confirming RC benefit for sustained traffic |
| Metric | KRCore (DC) | Verbs | LITE | Speedup |
|---|---|---|---|---|
| Single connection | 5.4μs | 15.7ms | 2ms | 2,900X / 370X |
| 240-client throughput | 22M conn/s | 712 QP/s | 712 QP/s | 30,900X |
| Full-mesh (240 workers) | 81μs | 2.7s | 2.3s | 33,300X / 28,400X |
Verbs and LITE throughput identical at 712 QP/s because both are bottlenecked by RNIC hardware QP creation capacity — the fundamental limit KRCore sidesteps by never creating new QPs at connection time.
| Operation | KRCore(RC) vs Verbs | KRCore(DC) vs Verbs | Root cause |
|---|---|---|---|
| 1-client sync READ | +46% (3.15 vs 2.15μs) | +51% (3.24 vs 2.15μs) | 1μs syscall overhead |
| Async READ peak (240 cli) | ≈0% (138M vs 138M) | −14% (118M vs 138M) | NIC-bound; DC firmware overhead |
| Async WRITE peak | ≈0% (145M vs 145M) | −8.9% (132M vs 145M) | Same NIC-bound pattern |
| Two-sided async peak | −20% (33.7M vs 42.3M) | −20% (same) | CPU cost of user-kernel crossing |
System call cost dominates for small sync operations but becomes negligible for large payloads (overhead <7% for READ ≥256KB, negligible for WRITE ≥8KB).
| Connections | LITE | KRCore | Ratio |
|---|---|---|---|
| 5,000 | 780MB (1.5GB w/ msg queues) | 6.3MB | 124X (238X) |
| 10,000 | 1.52GB+ | ~12MB | 127X+ |
KRCore achieves this through the fundamental DCT property: one physical DCQP serves unlimited virtual connections, metadata is 12B/target.
| Step | Claim | Evidence | Logical link |
|---|---|---|---|
| 1 | RDMA control plane is the bottleneck for elastic computing | Figure 1: 15.7ms connection vs μs-scale data ops; Figure 3(b): 87% cost in NIC hardware setup | Establishes that the problem is fundamental to hardware architecture, not fixable by software optimization of existing path |
| 2 | DCT provides sub-μs hardware reconnection on commodity RNICs | Mellanox spec [1]: ConnectX-IB through ConnectX-7 support DCT; paper measures <1μs reconnect | Identifies the exploitable hardware capability — widely deployed but unused for control plane |
| 3 | Kernel-space virtualization of DCT eliminates per-connection creation cost | Algorithm 1: VQP reuses pre-initialized physical DCQP; no new hardware QP creation at connect time | Transforms the connection operation from "create expensive hardware resource" to "assign pointer to existing resource" |
| 4 | RDMA-based meta server provides deterministic μs-scale metadata access | Figure 9(a): one-sided READ 11.8X throughput vs RPC; CPU-bypass eliminates scheduling jitter | Solves the DCT metadata query challenge without introducing new latency variability |
| 5 | Pre-check mechanism preserves safety under QP sharing | Algorithm 2: validates queue capacity, opcodes, MR before forwarding; LITE breaks at >6 threads, KRCore does not | Demonstrates correctness — shared physical QP requires active protection that prior work lacked |
| 6 | Hybrid DC+RC pool amortizes DC performance gap for sustained workloads | Figure 16 t>2.2: transparent upgrade recovers full RC throughput (26M vs 18M req/s) with negligible switch cost | Addresses the performance objection: DC is optimal for connection speed, RC for sustained throughput, KRCore provides both |
| 7 | End-to-end: elastic applications see order-of-magnitude improvements | RACE: 83% boot reduction; serverless: 99% transfer reduction; both limited by OS, not RDMA | Validates that control plane speedup translates to real application benefit, not just microbenchmark |
Repository: https://github.com/SJTU-IPADS/krcore-artifacts
| Component | Implementation detail |
|---|---|
| Kernel module | >10,000 LoC Rust, loadable Linux 4.15 module, exports via ioctl |
| DCT kernel port | 250 LoC C patch to mlnx-ofed-4.9 driver |
| User shim | 100 LoC C library wrapping extended API (§4.1) |
| Meta server backend | DrTM-KV [58] — existing RDMA-enabled KV store |
| QP pool sizing | 8 DCQPs per CPU (empirically chosen, Figure 14(a): >2 eliminates contention) |
| MR validity | Lease-based invalidation with 1s flush period |
The single hardest-to-replicate insight is the realization that DCT's hardware reconnection capability — designed for reducing NIC memory consumption in MPI workloads — can be repurposed as a control plane accelerator by virtualizing kernel-space DCT connections. This requires: (1) understanding that QP creation cost is NIC-firmware dominated (not network-dominated), (2) kernel-level access to DCT which no user-space library provides, and (3) a safe virtualization layer (Algorithm 2) that prevents corruption under arbitrary sharing patterns. The combination creates a "free" connection operation from pre-existing hardware capability.
KRCore argues for:
Relies on Mellanox/NVIDIA ConnectX series (Connect-IB through ConnectX-7) for DCT support. InfiniBand fabric required for full functionality. Tested on ConnectX-4 with mlnx-ofed-4.9. Not directly portable to non-Mellanox RNICs (Intel, Broadcom) as DCT is a Mellanox-proprietary transport extension.