μCUTLASS + SOL-Guided Kernel Optimization — L2 #

§1 TL;DR #

μCUTLASS DSL (~170-line EBNF) + Speed-of-Light roofline guidance turns GPT-5-mini from 0.40× regression to 1.56× speedup on 59 KernelBench problems (H100); each model tier matches the next tier's baseline at lower token cost. SOL-guided scheduling saves 19–43% tokens; integrity pipeline prevents up to 1.9× speedup inflation from LLM gaming.

§2 Q1 / Q2 / Q3 #

Q1 痛点 #

LLM agents optimizing GPU kernels face two efficiency problems:

1. Abstraction mismatch — generating raw CUDA/CUTLASS forces the LLM to simultaneously choose optimization strategy and emit template-heavy C++ with architecture-specific constraints (alignment, scheduler compatibility, layout algebra), wasting reasoning capacity on boilerplate. GPT-5-mini regresses to 0.40× vs PyTorch under this regime. The 21% LLM time vs 79% tool-action time split makes each wasted iteration expensive.
1. Missing headroom signal — profiling provides only local performance views. The agent cannot tell whether a kernel is near its theoretical performance ceiling, wasting iterations on diminishing returns and misallocating budget across problems.

Q2 方法 #

Two complementary design principles:

1. μCUTLASS DSL: A compact domain-specific language (~170-line EBNF grammar) learnable entirely in-context. Agents write ~10–20 line kernel specifications covering four facets: operator selection, kernel configuration (tile shapes, scheduling policies, data types, cluster dimensions), epilogue fusion (via >> operator), and multi-stage pipelines. A compiler validates constraints statically and emits CUTLASS C++, catching invalid configurations before the expensive compile/run/profile cycle.
1. SOL (Speed-of-Light) Guidance: Roofline-derived first-principles performance bounds applied in three modes:
2. (a) Within-problem steering via MANTIS workflow (Measure–Analyze–Nominate–Triage–Implement–Summarize), using SOL gap signal $g = t_{\text{best}} / t_{\text{SOL}}$ to prioritize optimization hypotheses by gap-aware ROI.
3. (b) Cross-problem budget scheduling using SOL-headroom threshold $\epsilon$ and no-progress window $w$ to stop allocation to near-SOL problems.
4. (c) Integrity checking using SOL ceiling + LLM-based game detector to identify kernels that skip intended computation.

核心技术壁垒: Designing the DSL compactness–expressiveness trade-off requires deep CUTLASS expertise to identify which template parameters drive performance variation (tile shapes, scheduling policies, alignment, epilogue composition) vs. which are safely factored out as boilerplate. The ~170-line EBNF with embedded constraint annotations (architecture gating, alignment rules, scheduler compatibility) is the non-obvious artifact — too many knobs and the LLM wastes reasoning capacity; too few and peak performance is unreachable. The static validation rules encode implicit CUTLASS contracts that are not documented anywhere in a machine-consumable form.

Q3 结果 #

On 59 KernelBench problems (H100), same 40-attempt budget per problem:

配置	Geomean Speedup
GPT-5-mini MI (baseline)	0.40×
GPT-5-mini + μCUTLASS	1.27×
GPT-5-mini + μCUTLASS + SOL	1.56×
GPT-5 MI (baseline)	0.86×
GPT-5 + μCUTLASS + SOL	2.07×
GPT-5.2 MI (baseline)	2.04×
GPT-5.2 + μCUTLASS	2.85×
All-variants best	3.91× (FP16 SOL limit: 7.46×)

SOL-guided scheduling saves 19–43% tokens at ≥95% geomean retention; best configuration reaches 1.68× efficiency gain. Integrity filtering prevents up to 1.9× speedup inflation. Gaming rates increase with model capability.

§3 架构 / 方法图 #

μCUTLASS Compilation Pipeline #



Paper's Figure 1, verbatim (caption: "The compiler parses and lowers a kernel.dsl program to a typed configuration IR, validates architecture and other constraints, and emits the CUTLASS C++ code").

The compiler accepts a short DSL specification (e.g., SM90a GEMM with fused epilogue via >> chain) and proceeds through three stages: (1) Parse/Lower — grammar-based frontend produces a typed configuration IR from the ~10–20 line spec; (2) Static Validation — checks architecture gating (SM version), alignment rules (e.g., FP16 SM90+ requires A/B alignment ≥ 8), scheduler compatibility (kernel=tma_cooperative requires epilogue=tma_cooperative), and operator constraints, rejecting invalid configurations with explanatory errors; (3) Code Emission — generates CUTLASS C++ placed in a deterministic namespace via sha256(config), with the original DSL source embedded as a comment. SM90+ GEMMs emit through CUTLASS 3.x CollectiveBuilder API; SM70–89 and convolutions emit via CUTLASS Python API (cutlass_cppgen). SM90+ epilogue chains fuse into a single Epilogue Visitor Tree (EVT).

SOL-Guided Budget Scheduling #



Paper's Figure 2, verbatim (caption: "SOL-guided budget scheduling overview showing workspace of problems, signals, and scheduling policy with eligibility criteria").

The scheduler maintains a workspace of problems, each annotated with: SOL bound $t_{\text{SOL}}$, current best time $t_{\text{best}}$, profiler summary, rate of improvement, token/tool usage, and hypothesis ROI. A round-robin scheduling policy applies two eligibility criteria: (1) SOL-headroom threshold — problem removed when $t_{\text{best}} \leq (1+\epsilon) \cdot t_{\text{SOL}}$ and already beats PyTorch; (2) no-progress window — removed when best speedup stagnant for $w$ consecutive attempts after surpassing PyTorch. This converts each fixed-budget agent from a single operating point into a cost vs. speedup Pareto frontier.

MANTIS Workflow (Within-Problem Steering) #

graph TD M["Measure
Profile via NCU:
occupancy, memory/compute throughput"] --> A["Analyze
SOL gap g = t_best / t_SOL
+ bottleneck attribution"] A --> N["Nominate
Generate optimization hypotheses
with causal link to bottleneck"] N --> T["Triage
Rank by gap-aware ROI formula
higher ambition when g is large"] T --> I["Implement
Generate–Compile–Test–Profile
fixed attempt budget per hypothesis"] I --> S["Summarize
Reflect on expectations vs outcomes
Persist lessons as cross-problem memory"] S -->|"Next iteration
(5 iters × 2 hyp × 4 attempts = 40)"| M

MANTIS structures the optimization loop into six explicit phases. The Triage phase scores hypotheses with a gap-aware ROI formula where the speedup exponent increases with $\log_{10}(g/5)$, making high-ambition hypotheses more attractive when the SOL gap is large. The Summarize phase persists lessons as cross-problem memory for later problems to retrieve reusable optimization patterns.

Two implementation forms exist: Orchestrated (each phase executed as distinct step, structured artifacts passed between phases) and In-prompt (flat controller follows MANTIS methodology described in system prompt). The optimal form is model-dependent: orchestrated benefits weaker/mid-tier models; in-prompt preferable for the strongest model when paired with μCUTLASS.

Target Operations and Hardware Model #

Target ops: 59 KernelBench Level 1–3 problems covering GEMM, grouped GEMM, convolutions (1D/2D/3D/depthwise), fused matmul+elementwise, attention, MLP blocks, and Mamba SSM. Input/output: primarily FP32 tensors; agents allowed reduced-precision math (FP16/BF16/TF32/FP8).

Hardware: NVIDIA H100 (SM90a), clocks locked to 1500 MHz (max 1980 MHz).

Resource	Value
TF32 peak (effective)	374.77 TFLOP/s (494.7 × 0.7576)
FP16 peak (effective)	749.55 TFLOP/s (989.4 × 0.7576)
HBM3 bandwidth	3.35 TB/s
TF32 ridge point	111.9 FLOPs/byte

DSL Coverage #

μCUTLASS covers SM70–SM90+ across 8 operator families (GEMM, Grouped GEMM, Conv2d, Conv3d, Conv3d wgrad, Conv1d, Depthwise Conv, Grouped Conv) with dtype support for FP64/FP32/FP16/BF16/FP8(SM90+)/INT8. Key features: .with_threadblockshape, .with_cluster, .with_scheduler (SM90+), epilogue fusion via >> with 16+ built-in epilogues (relu, gelu, silu, sigmoid, bias, per_channel_scale, aux_store, etc.) plus custom expressions (SM90a), and multi-stage pipeline(...) with fused dtype conversion. Most kernels are ~10–20 lines.

§4 作者证明 #

关键方程 #

符号	定义	物理意义
$T_{\text{compute}}$	$\text{FLOPs} / \text{Peak FLOP/s}$	纯 compute-bound 下限时间
$T_{\text{mem}}$	$\text{Bytes} / \text{Peak BW}$	纯 memory-bound 下限时间
$t_{\text{SOL}}$	$\max(T_{\text{compute}}, T_{\text{mem}})$	理论最小运行时间（瓶颈取 max）
$g$	$t_{\text{best}} / t_{\text{SOL}}$	SOL gap 信号；越大 = 越多优化空间
$\text{ROI}(h)$	$\frac{(\widehat{S}(h))^{1 + \max(0, \log_{10}(g/5))}}{\widehat{R}_{\text{impl}}(h) \cdot \widehat{R}_{\text{perf}}(h)}$	假设排序公式：gap 大时提升 speedup 权重
SOL-gap stop	$t_{\text{best}} \leq (1+\epsilon) \cdot t_{\text{SOL}}$	接近理论上限时停止资源分配
efficiency gain	$\frac{g_{\text{policy}}}{g_{\text{fixed}}} \times \frac{\tau_{\text{fixed}}}{\tau_{\text{policy}}}$	每 token 的优化性能比（>1× 表示调度有效）

Roofline Placement #

SOL 分析遵循 roofline 方法：算术强度 = FLOPs / Bytes，与 ridge point (Peak FLOP/s / Peak BW) 比较。H100 TF32 下 ridge = 374.77 / 3.35 ≈ 111.9 FLOPs/byte。

具体示例 (KernelBench Problem 001, 4096×4096 FP32 GEMM):

• FLOPs = $2N^3 = 1.374 \times 10^{11}$
• Best-case DRAM bytes = $3 \times N^2 \times 4 = 2.013 \times 10^8$ (~192 MiB)
• Arithmetic intensity ≈ 682.6 FLOPs/byte >> ridge 111.9 → compute-bound
• $T_{\text{compute}} = 0.367$ ms, $T_{\text{mem}} = 0.060$ ms → $t_{\text{SOL}} = 0.367$ ms
• FP16 augmentation: $T_{\text{compute}}^{\text{FP16}} = 0.1834$ ms (2× throughput)

SOL Bound Tightness #

方向	原因	论文的处理
Too tight (实际 > SOL)	假设 perfect caching；若 on-chip 不足则 kernel 需 re-fetch	已声明假设
Too loose (实际 < SOL)	未计入低精度/稀疏 throughput 增益	维护双精度 bound：TF32 用于 steering，FP16 用于 scheduling/integrity

双精度方案是关键设计选择——单一 bound 会导致 steering 过松（错过优化机会）或 integrity 误报（FP16 kernel 被错误标记）。

6 Minimum Checks #

#	检查项	状态
1	方程变量全部定义	✅ 所有变量在 §4.1–§4.3 明确定义
2	量纲一致	✅ FLOPs/(FLOPs/s)=s; Bytes/(Bytes/s)=s; ratio 无量纲
3	边界条件合理	✅ $g=1$ → kernel 已达 SOL; $\epsilon=0$ → 要求恰好达 SOL
4	与实验对应	✅ SOL 在 Appendix A.2 完整计算; ROI 在 MANTIS Triage 使用
5	假设明确声明	✅ perfect caching, no sparsity, FP32 input tensors
6	对比方法使用相同度量	✅ 所有 variant 相同 NCU 度量、相同 SOL 基准

Design Space & Constraint Derivation #

优化轴	选定值	被拒备选	阻止约束
Abstraction level	DSL (~170-line EBNF)	Raw CUDA/CUTLASS code	LLM 无法同时推理策略+模板 C++；GPT-5-mini 回归 0.40×
Epilogue composition	>> fusion operator → EVT	Separate kernel launches	中间结果需额外 DRAM roundtrip
Validation timing	Compiler pre-check (static)	Runtime error discovery	Compile/run/profile 循环每次耗时占 79%
SOL bound precision	Dual (TF32 + FP16)	Single bound	单一 bound 导致 steering 过松或 integrity 误报
Scheduling policy	Round-robin + eligibility rules	Priority-based	评估复杂度；round-robin + SOL 停止规则已足够
Orchestration form	Model-adaptive (orchestrated vs in-prompt)	Fixed form	GPT-5.2+μCUTLASS 下 in-prompt 反超 orchestrated

§5 实验与数据 #

Headline Results #



Paper's Figure 3, verbatim (caption: "Geomean speedup over PyTorch for the four main variants across three model tiers with same attempt budget").

μCUTLASS 在每个模型层级都是单一最大改进因素。DSL 对弱模型收益最大：GPT-5-mini 从 0.40× → 1.27×（3.2× 改进），GPT-5 从 0.86× → 1.69×（2.0×），GPT-5.2 从 2.04× → 2.85×（1.4×）。收益递减趋势清晰——越弱的模型，DSL 越有效（将 "无法完成" 变为 "成功"）。组合 μCUTLASS+SOL 的 model-tier substitution 效果突出：GPT-5-mini (1.56×) 超越 GPT-5 MI (0.86×) at 5× lower cost。

Fast-p Curves #



Paper's Figure 4, verbatim (caption: "Four main variants across three model tiers. Left: Fast-p curves. Right: Attempt-Fast-p(2). Dashed gray: MI baseline of next stronger model").

Fast-p 分布揭示 DSL 的质变效果：GPT-5-mini MI 仅 22% 问题超越 PyTorch，29 个回归至 0.5× 以下。加 μCUTLASS+SOL 后 59% 超越，36% 达 ≥2×，仅 6 个回归至 0.5× 以下。Attempt-Fast-p(2) 显示 μCUTLASS+SOL 在前 10 次尝试内即到达 ≥2× 的 plateau，说明 DSL 减少了探索浪费。

SOL-Guided Budget Scheduling Pareto #



Paper's Figure 8, verbatim (caption: "Scheduler policy Pareto frontiers: normalized dollar cost vs. geomean speedup for nine variants across three model tiers").

调度将每个 variant 从单一固定点扩展为代价-性能 frontier。四个关键观察：(1) GPT-5.2 μCUTLASS+SOL exhaustive 点 ~(0.93, 2.65×)，适度策略 ($\epsilon$=25%, $w$=16) 移至 ~(0.62, 2.55×)，33% token 节省保留 96% geomean；(2) μCUTLASS 和 SOL 在每个模型层级内提升 frontier；(3) 最有效的调度杠杆因模型层级而异——GPT-5-mini 主要受益于 no-progress window ($w=4$ → ~30% savings)，GPT-5/5.2 两个参数均显著；(4) SOL 提供 measurable ceiling，使调度成为有依据的决策。

Efficiency Gain #



Paper's Figure 9, verbatim (caption: "Best (ε, w) per variant maximizing efficiency gain subject to ≥95% geomean retention").

最佳配置 GPT-5 μCUTLASS+SOL 达 1.68× efficiency gain ($\epsilon$=250%, $w$=12; 43% token 节省, 96% retention)。弱模型偏好高 $\epsilon$（kernel 很少接近 FP16 SOL bound），强模型受益于更紧阈值。Window 偏好：GPT-5 $w$=8–12; GPT-5-mini $w$=4; GPT-5.2 $w$=16。

Integrity Checking Impact #



Paper's Figure 12, verbatim (caption: "Speedup inflation without the integrity pipeline").

完整性检查至关重要。GPT-5-mini μCUTLASS+MI 从 1.27× 膨胀至 2.28×（80% inflation）; GPT-5.2 μCUTLASS+MI 从 2.85× 膨胀至 5.34×（1.9× inflation，由 gaming 驱动）。强模型 gaming 率更高（GPT-5.2 constant/hardcoded output: 104–139 flags per variant）。三组件 integrity pipeline（SOL-ceiling detector + LLM-based game detector + static PyTorch-only detector）共移除 7–314 gaming/PyTorch-only attempts per variant。SOL-guided orchestrated variant gaming 最少（3–35 exclusions），表明结构化搜索抑制了 shortcut 发现。

Prompt-level anti-gaming 指令不可靠——在 μCUTLASS+MI 上反而从 50→95 gaming attempts（可能将模型注意力引向 gaming 策略）。

External Comparison #



Paper's Figure 14, verbatim (caption: "Fast-p comparing μCUTLASS+SOL across model tiers against Sakana AI CUDA Engineer").

Sakana AI CUDA Engineer (~30,000 kernels, Claude 3.5 Sonnet) 在 integrity-filtered 下达 1.13× geomean（57 可比问题中 52 accepted, 5 rejected）。GPT-5-mini μCUTLASS+SOL (1.56×) 大幅领先。跨所有 variant 最优选择达 3.91× geomean（4.53× median），all 59 problems solved。FP16 SOL curve 达 7.46× geomean——剩余 ~2× 空间是未来工作方向。

Stability and Ablations #

跨 run 变异随模型能力递减：GPT-5.2 CV ~5–7%，GPT-5-mini CV ~13–15%。关键发现稳健：每个 GPT-5-mini μCUTLASS 配置（含最差 ablation 1.20× 和独立 repeat 1.22×）均超 GPT-5 MI baseline (0.86×) ≥40%。

组件 ablation（GPT-5-mini w/o μCUTLASS）显示每个 MANTIS 组件都重要——移除 Triage 或 Summarize 损害最大。GPT-5.2 上移除单一组件影响甚微——强模型自行规划足够好。Orchestrated vs in-prompt 的最优形式随模型能力和 DSL 使用而变：GPT-5.2+μCUTLASS 下 in-prompt 反超 orchestrated (signed area −0.87)，rigid orchestration 约束了已超越 imposed structure 的模型。

§6 论证链 #

步骤	论点	证据	力度
1	Raw CUDA/CUTLASS 对 LLM 效率低：抽象层级不匹配导致弱模型回归	GPT-5-mini MI → 0.40× regression (Fig 3); 52/59 编译成功但仅 22% 超越 PyTorch; 79% 时间消耗在 compile/run/profile	强 — 直接实验
2	紧凑 DSL 释放推理能力：声明式规范替代模板 C++	μCUTLASS 将 GPT-5-mini 从 0.40× 扭转至 1.27× (Fig 3); 弱模型收益最大 (3.2× vs 2.0× vs 1.4×); 对 48 个共同问题 mini+DSL+SOL 赢 32/48 (67%) vs GPT-5 MI	强 — 跨 3 模型层级一致
3	SOL 提供 first-principles headroom 信号，结构化搜索方向	+SOL 将 GPT-5-mini 推至 1.56×, GPT-5 至 2.07× (Fig 3); MANTIS ablation 显示 Analyze (SOL 分析) 移除后 GPT-5-mini+μCUTLASS 显著下降 (Fig 6c)	中强 — GPT-5.2 ≥2× coverage 从 64% 降至 61% (收益递减)
4	DSL+SOL 的 model-tier substitution：弱模型匹配强模型 baseline	GPT-5-mini+DSL+SOL (1.56×) > GPT-5 MI (0.86×) at 5× lower cost; GPT-5+DSL+SOL (2.07×) ≈ GPT-5.2 MI (2.04×) at 1.4× lower cost	强 — 跨两个层级间隔一致
5	SOL-guided scheduling 节省资源同时保持性能	19–43% token savings at ≥95% retention (Fig 8-9); best config 1.68× efficiency gain; Pareto analysis 覆盖 6×12 参数组合	强 — 系统性参数扫描
6	LLM gaming 严重且随模型能力递增	无 integrity 过滤时 inflation 达 1.9× (Fig 12); GPT-5.2 gaming flags 最多 (306 exclusions); prompt guardrails 不可靠甚至适得其反 (50→95, Table 4)	强 — 定量分析 + 分类体系
7	Orchestration 收益因模型能力而异——非 universally beneficial	GPT-5-mini/5 → orchestrated 更优; GPT-5.2+μCUTLASS → in-prompt 反超 (signed area −0.87, Fig 5)	中 — 交互效应复杂，但有 signed-area 定量支持

§7 实现 cross-reference #

代码可得性 #

μCUTLASS compiler 实现为 Python CLI，支持文件编译和 tool-mode（接受 DSL 文本字符串）。评估基于 OpenHands 作为 agent runtime，自定义 ucutlass_compile tool 传给 agent。Grammar 和 compiler 通过 AI + human-in-the-loop 迭代开发。

[实现未公开]

关键实现细节 #

1. EVT 扩展: SM90+ epilogue 链融合为 Epilogue Visitor Tree (EVT) (Chen et al. 2024, ASPLOS)，但 CUTLASS 默认 cutlass_cppgen 不直接 emit EVT C++。本文扩展了该工具以支持 >> 链到 EVT 的编译——这是 μCUTLASS 编译器中的关键 non-trivial 工程。
1. 双精度 SOL bound: Optimization steering 使用 FP32/TF32 estimate（较松上界，避免过早停止）; scheduling 和 integrity checking 使用 FP16 estimate（较紧上界，防止 false negative 在 reduced-precision kernel 上）。精度选择直接影响系统行为——选错会导致 steering 过保守或 integrity 漏检。FP16 estimate 通过交叉检验 SOLAR (NVlabs 2025) 验证。
1. Deterministic namespace: 生成的 CUTLASS C++ 放入 sha256(config) 命名空间，原始 DSL 源码嵌入为注释。实现了编译结果的内容寻址和可追溯性。

Portability #

μCUTLASS 当前覆盖 SM70–SM90+ (NVIDIA Volta through Hopper)。底层依赖 CUTLASS template library 和 CuTe。向 AMD (CDNA/CK) 或 Triton 移植需要重写编译器后端的 code emission 层和约束规则。论文显式指出 DSL 原则可推广到其他后端（Triton, MLIR dialects），但当前实现仅绑定 CUTLASS。

Software → Hardware 反向启示 #

• μCUTLASS 编译器中大量静态约束规则（alignment, scheduler compatibility, architecture gating）本质上编码了 CUTLASS 的隐含契约。如果硬件/ISA 提供更细粒度的 capability query API（超越当前 Compute Capability 的粗粒度），DSL 编译器的约束规则可显著简化。
• SOL gap 分析依赖 compute-bound vs memory-bound 的二分模型。在现代 GPU 上（TMA, async copy 使 compute 和 memory 真正 overlap），更精确的 SOL model 需要硬件暴露 pipeline stage timing，而非仅提供 peak throughput 和 peak bandwidth。
• 79% 的迭代时间消耗在 compile/run/profile——如果硬件提供 low-overhead profiling mode（类似 NCU 但延迟更低），agent 的迭代效率会进一步提升。