HybridFlow: A Flexible and Efficient RLHF Framework

framework 2409.19256
RLHFtraining-systemschedulingdistributed-traininghybrid-programming
← Synthesis ← framework Survey Source ↗

HybridFlow: A Flexible and Efficient RLHF Framework #

Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, Chuan Wu | 2024-09 | https://arxiv.org/abs/2409.19256 Category: framework | Tags: RLHF, training-system, scheduling, distributed-training Read: 2026-04-16

Core Contribution #

HybridFlow combines single-controller (for inter-model coordination) and multi-controller (for intra-model distributed computation) paradigms into a hierarchical hybrid programming model, plus a 3D-HybridEngine for zero-redundancy actor model resharding between training and generation, achieving 1.53x–20.57x throughput over SOTA RLHF baselines.

核心三问 #

Q1: 这篇论文试图解决什么问题？

RLHF 训练涉及多个 LLM（actor、critic、reference、reward）的复杂 dataflow，现有系统要么用 single-controller（调度开销大）要么用 multi-controller（代码紧耦合、不灵活），且 actor 模型在 training 和 generation 阶段间的 weight resharding 带来巨大通信和内存冗余。需要一个既灵活表达各种 RLHF 算法、又高效执行分布式计算的统一框架。

Q2: 这是否是一个新的问题？如果不是，之前最好的方法是什么？

不是全新问题，但之前的方案都有明显缺陷。DeepSpeed-Chat 将所有模型放在同一组设备上、用 ZeRO+TP 但 resharding 开销大；OpenRLHF 各模型独占设备、减少冲突但浪费 GPU 资源且需维护两份 actor 权重；NeMo-Aligner 用相同 3D parallelism 做 training 和 generation 导致 generation 效率低。三者都只支持 PPO 且 placement 策略固定。

Q3: 本文的解决方案的关键是什么？有什么巧妙之处？

关键设计有三层：(1) Hybrid Programming Model — inter-node 用 single-controller 灵活编排 dataflow，intra-node 用 multi-controller 高效执行分布式计算，通过 3DParallelWorker 类和 transfer protocol 解耦计算与通信；(2) 3D-HybridEngine — 通过重新设计 generation 阶段的 parallel group 分配策略（interval-based 而非 consecutive），使得每个 GPU 上 training 和 generation 权重完全重叠，实现 zero memory redundancy resharding；(3) Auto Device Mapping — 自动搜索最优的模型放置和并行策略组合。巧妙之处在于 hybrid paradigm 的分层设计同时满足了灵活性和效率。

逻辑故事还原 #

背景： RLHF 是 LLM alignment 的核心技术，但其 dataflow 比传统 RL 复杂得多——每个节点是分布式 LLM 程序，每条边是 many-to-many multicast。现有框架在灵活性和效率间难以兼顾：single-controller 灵活但 dispatch 开销大，multi-controller 高效但代码紧耦合、难以支持新算法和不同 placement 策略。

破局： 核心洞察是 RLHF dataflow 的 inter-node 层面节点少（仅几个模型）、single-controller 开销可忽略，而 intra-node 层面每个模型有数十亿参数、必须用 multi-controller 才能高效。因此应在两个层面分别使用最适合的范式。

拆解：

  1. 设计 hierarchical API：3DParallelWorker 封装模型的分布式计算，@register 装饰器关联 transfer protocol 统一数据 resharding，ResourcePool 虚拟化设备分配
  2. 设计 3D-HybridEngine：actor training 用 p-t-d 并行，generation 用 pg-tg-dg-d 并行，通过 interval-based parallel grouping 实现 zero-redundancy resharding
  3. 设计 Auto Device Mapping：枚举 Bell partition 的所有 placement plan，simulator 估计延迟，搜索最优 allocation + parallelism 组合
  4. 实验验证：在 7B–70B 模型、PPO/ReMax/Safe-RLHF 三种算法、16–64 GPU 规模上全面超越 baseline

    5. Key Figures #

    Figure 1: RLHF Dataflow Graphs #

    Figure 1

    Dataflow graphs for three RLHF algorithms: PPO, Safe-RLHF, and ReMax. Each involves three stages — Generation (①), Preparation (②), and Training (③). PPO uses 4 models (actor, critic, reference, reward); Safe-RLHF adds a cost model; ReMax eliminates the critic and adds an extra generation pass for variance reduction. This illustrates why a flexible framework is needed — different algorithms have different model compositions and data dependencies.

    Figure 2: Hybrid Programming Model #

    Figure 2

    Comparison of multi-controller (existing systems) vs. HybridFlow's hybrid programming model. In (a), each model runs as a separate multi-controller program with explicit point-to-point send/recv for inter-model communication — tightly coupled and hard to modify. In (b), a single controller coordinates models at the dataflow level while each model internally uses multi-controller for efficient distributed computation. Grey inactive nodes show that models not executing at a given time are simply skipped by the controller.

    Figure 4: HybridFlow Architecture #

    Figure 4

    Three-component architecture: (1) Hybrid Programming Model with hierarchical APIs (model classes, transfer protocols, ResourcePool), (2) 3D-HybridEngine for efficient actor training↔generation transitions, and (3) Auto-Mapping algorithm for optimized device placement. The single controller program orchestrates the dataflow, while ParallelWorker classes handle distributed computation on allocated devices.

    Figure 7: 3D-HybridEngine Workflow #

    Figure 7

    Workflow of 3D-HybridEngine within one RLHF iteration on 4 GPUs. Training uses 1-2-2 (p-t-d) parallel groups; generation uses 1-1-2-2 (pg-tg-dg-d). Five steps: ① all-gather model params within micro DP groups, ② load prompts to replicas, ③ all-gather generation results, ④ re-partition params for training parallelism, ⑤ compute loss and update weights. The key insight is that different parallelism configs for training (compute-bound) and generation (memory-bound) maximize throughput in both stages.

    Figure 9: PPO Throughput Comparison #

    Figure 9

    End-to-end throughput comparison of HybridFlow vs. DeepSpeed-Chat, OpenRLHF, and NeMo-Aligner across different model sizes (7B–70B) and GPU counts (16–64). HybridFlow achieves 1.53x–20.57x throughput improvement. The gains come from three sources: efficient 3D-HybridEngine resharding, optimized model placement via auto-mapping, and the hybrid programming model's ability to use the best parallelism strategy for each model independently.

    Key Tables #

    Table 1: Comparison of RLHF Frameworks #

    FeatureDeepSpeed-ChatOpenRLHFNeMo-AlignerHybridFlow
    Training ParallelismZeROZeRO3D Parallelism3D, ZeRO, FSDP
    Generation ParallelismTPTP3D Parallelism3D Parallelism
    Actor Weights StrategyReshard ZeRO→TPTwo copiesShared (same config)Zero-redundancy reshard
    Model PlacementAll colocatedAll separateActor/Ref + Critic/RMFlexible (any combo)
    Execution PatternSequentialPartial parallelPartial parallelFully flexible

    Table 2: Transition Overhead (Training ↔ Generation) #

    MetricDeepSpeed-ChatHybridFlow-VHybridFlow
    Comm. Volume(tpd-1)/(tpd) · M(tp-1)/(tp) · M(tp-tg·pg)/(tg·pg·tp) · M
    Peak MemoryMMM/(tg·pg)
    RedundancyM/(tpd)M/(tp)0

    Table 1 Extended: Supported Algorithms & Flexibility #

    CapabilityDeepSpeed-ChatOpenRLHFNeMo-AlignerHybridFlow
    PPOYesYesYesYes
    ReMaxNoNoNoYes
    Safe-RLHFNoNoNoYes
    Custom AlgorithmsHardHardHardFew lines of code
    Auto Device MappingNoNoNoYes

    Summary #

    HybridFlow (open-sourced as veRL) is an RLHF training framework from ByteDance/HKU that addresses the inflexibility and inefficiency of existing RLHF systems. Its core innovation is a hierarchical hybrid programming model that uses a single-controller for inter-model dataflow coordination (flexible, low overhead since few nodes) and multi-controller for intra-model distributed computation (efficient, leveraging existing LLM engines). The 3D-HybridEngine enables the actor model to use different 3D parallelism strategies for training (compute-bound, larger TP/PP) and generation (memory-bound, larger DP), with a novel parallel group rearrangement that achieves zero memory redundancy during resharding. An auto-mapping algorithm searches over model placements and parallelism configurations to minimize iteration latency. Published at EuroSys 2025.

    Key Findings #

    • 1.53x–20.57x throughput improvement over DeepSpeed-Chat, OpenRLHF, and NeMo-Aligner across 7B–70B models on 16–64 A100 GPUs
    • Zero-redundancy resharding eliminates memory waste during actor training↔generation transitions; communication volume reduced from O(M) to O(M/(tg·pg·tp)) compared to DeepSpeed-Chat
    • Algorithmic flexibility: PPO, ReMax, Safe-RLHF implemented in 8–13 lines of single-controller code; adding a new algorithm requires only modifying the top-level dataflow, not the distributed communication layer
    • 3D-HybridEngine is the key performance driver: the interval-based parallel grouping for generation ensures every GPU's generation weights are a subset of its training weights, eliminating redundant memory copies
    • Auto-mapping consistently finds better placement plans than the fixed strategies of baselines, especially at scale where the placement design space grows combinatorially
    • Actor training + generation dominates ~59% of total RLHF iteration time, making the 3D-HybridEngine optimization high-impact

    Limitations #

    • Auto-mapping relies on analytical cost models (simulators for training/inference/generation latency), which may not perfectly capture real hardware behavior, especially for generation with variable KVCache sizes
    • No overlap between ResourcePool instances — the framework assumes disjoint device sets, which could limit flexibility for more exotic placement strategies
    • Bell partition enumeration for placement plans grows exponentially with model count; while PPO has only 15 placements (4 models), more complex algorithms with additional models could make the search expensive
    • Evaluated only on A100 GPUs — behavior on heterogeneous clusters or newer GPU architectures (H100, MI300X) is not characterized
    • Only supports synchronous RLHF execution — no exploration of asynchronous or pipeline-overlapped execution between RLHF iterations
    • Single-controller is still a potential bottleneck if the dataflow graph becomes much larger (e.g., multi-reward-model setups with dozens of nodes)

    Infrastructure Impact #

    • Open-sourced as veRL (https://github.com/volcengine/verl) — becoming a significant community project for RLHF training
    • Decoupling paradigm is broadly applicable: the hybrid single/multi-controller pattern could be adopted by other multi-model training systems (MoE routing, speculative decoding, agent workflows)
    • 3D-HybridEngine's zero-redundancy resharding technique is generalizable to any scenario where the same model needs different parallelism strategies in different phases (e.g., training→eval, draft→verify in speculative decoding)
    • Transfer protocols provide a clean abstraction for inter-model data movement that could standardize how multi-model systems handle data dependencies
    • Auto-mapping demonstrates that RLHF placement optimization is a tractable combinatorial problem, opening the door for more sophisticated cost models and search algorithms
    • Sets the baseline for RLHF system efficiency — future work in this space will benchmark against HybridFlow/veRL

    Deep Analysis (framework) #

    1. System Scope #

    HybridFlow is a distributed RLHF training framework that spans the full RLHF iteration pipeline: prompt batching, auto-regressive generation, reward/reference/critic inference, and actor/critic training. It targets multi-GPU clusters (16–64+ GPUs) running LLMs from 7B to 70B+ parameters. The system does not handle reward model pre-training, SFT, or data preprocessing — it focuses purely on the RL fine-tuning loop. It integrates with existing LLM engines (Megatron-LM, DeepSpeed, PyTorch FSDP, vLLM) rather than replacing them.

    2. Architecture & Data Flow #

    The architecture has three layers:

    • Single-controller layer: A single Python process that represents the RLHF dataflow as a sequence of model API calls. It coordinates execution order, manages ResourcePool device assignments, and orchestrates inter-model data transfers via transfer protocols. Data futures (lazy references) are passed between models — actual GPU-to-GPU transfer happens directly without controller bottleneck.
    • Multi-controller layer: Each model runs as a set of ParallelWorker processes, one per GPU. Workers execute the same SPMD program within their parallel groups. 3DParallelWorker establishes TP/PP/DP groups; FSDPWorker and ZeROWorker provide alternative parallelism. The 3D-HybridEngine lives here, managing the actor's dual parallel groups.
    • Transfer protocol layer: @register decorators bind each model operation to a protocol (3D_PROTO, DP_PROTO, ONE_TO_ALL, etc.). Each protocol defines collect (gather outputs to controller as futures) and distribute (scatter inputs to workers by DP rank). The controller chains source.collect → destination.distribute to implement any-to-any resharding.

    Data flow per RLHF iteration: prompts → actor.generate → [prompts, responses] → {critic.compute_values, ref.compute_log_probs, reward.compute_reward} → compute_advantage → {actor.update, critic.update}.

    3. Key Innovations #

    1. Hybrid single/multi-controller paradigm: The fundamental insight that inter-node coordination needs flexibility (few nodes, negligible overhead) while intra-node computation needs efficiency (billions of ops, must minimize dispatch latency). This is a clean separation of concerns.
    2. 3D-HybridEngine with interval-based parallel grouping: Instead of forming generation TP groups from consecutive ranks (which creates weight misalignment), groups are formed at intervals of t/tg. This guarantees each GPU's generation weights are a subset of its training weights — zero redundancy, zero extra memory, minimal communication.
    3. Transfer protocol abstraction: Decouples data resharding from model computation. Adding a new model or changing parallelism doesn't require touching other models' code. The 8 built-in protocols cover most patterns; users can extend with custom collect/distribute functions.
    4. Auto device mapping: Systematic enumeration of Bell partitions × GPU allocations × parallelism strategies, evaluated by analytical simulators. Makes placement optimization accessible to non-experts.

      5. 4. Scheduling & Resource Management #

      • ResourcePool: Virtualizes GPU device sets. Models bound to the same pool are time-shared (sequential execution); models on different pools execute in parallel when data-independent. No overlap between pools.
      • Asynchronous execution: The single controller dispatches operations and immediately receives data futures. Parallel models begin execution as soon as inputs are available, without explicit barrier synchronization.
      • Auto-mapping search: For each Bell partition placement, explores GPU allocations from Amin (minimum to avoid OOM) upward. For each allocation, auto_parallel finds the best (p,t,d) per model. d_cost estimates iteration latency: colocated models' times sum; separated models' times max.
      • 3D-HybridEngine scheduling: Within actor's GPUs, training→generation transition involves: all-gather within micro DP groups → redistribute prompts → generate → all-gather results → re-partition weights. All communication is confined to small groups (micro DP size = dg), not global.

      5. Target Scenarios #

      • RLHF training with PPO, ReMax, Safe-RLHF, and other RL algorithms on LLMs from 7B to 70B+
      • Multi-node GPU clusters (tested on 2–8 nodes of 8×A100-80GB)
      • Research settings where rapid prototyping of new RLHF algorithms is needed
      • Production settings where maximizing throughput per GPU-hour is critical
      • Scenarios with heterogeneous model sizes (e.g., 7B actor + 70B reward model)

      6. Performance Evaluation #

      • Baselines: DeepSpeed-Chat, OpenRLHF, NeMo-Aligner
      • Models: LLaMA-family, 7B/13B/34B/70B actor, 7B/13B critic/ref/reward
      • Hardware: 16–64 A100-80GB GPUs, NVLink intra-node, 200 Gbps RDMA inter-node
      • Algorithms tested: PPO, ReMax, Safe-RLHF
      • Key results:
      • PPO 7B on 32 GPUs: ~3.83x over DeepSpeed-Chat, ~1.53x over OpenRLHF
      • PPO 70B on 64 GPUs: ~20.57x over DeepSpeed-Chat
      • ReMax: up to ~7x improvement (baselines don't natively support it, compared to manual implementations)
      • 3D-HybridEngine alone contributes significant gains via reduced resharding overhead
      • Auto-mapping finds non-obvious placements that outperform all fixed baselines

      7. API & Usability #

      • PPO implemented in 8 lines of single-controller code
      • Safe-RLHF = PPO + 5 additional lines (add cost model + pretrain loss)
      • ReMax = PPO - critic code + 1 additional generation call
      • Model classes (ActorWorker, CriticWorker, RefWorker, RewardWorker) inherit from 3DParallelWorker/FSDPWorker/ZeROWorker
      • Transfer protocols registered via @register decorator
      • ResourcePool assigned via model.to(resource_pool)
      • Auto-mapping invoked as a pre-processing step before training
      • Compatible with Megatron-LM, DeepSpeed, PyTorch FSDP, vLLM backends

      8. Infrastructure Impact #

      HybridFlow (veRL) establishes a new design pattern for multi-model distributed training: hybrid control planes. The single-controller handles the "macro" dataflow while multi-controllers handle "micro" distributed computation. This pattern is applicable beyond RLHF to any system orchestrating multiple distributed programs (e.g., constitutional AI pipelines, multi-agent LLM systems, compound AI systems). The 3D-HybridEngine's zero-redundancy resharding is a generally useful primitive for any workload switching parallelism strategies mid-pipeline.

      9. Comparison Matrix #

      DimensionDeepSpeed-ChatOpenRLHFNeMo-AlignerHybridFlow
      Programming ModelMulti-controllerMulti-controllerMulti-controllerHybrid
      Training BackendDeepSpeed ZeRODeepSpeed ZeROMegatron-LM 3DMegatron/DS/FSDP
      Generation BackendHuggingFacevLLMMegatron-LMMegatron/vLLM
      Actor ReshardingAll-gather all GPUsTwo copiesNo reshardingZero-redundancy
      Model PlacementAll colocatedAll separatePaired colocationAny combination
      AlgorithmsPPO onlyPPO onlyPPO onlyPPO, ReMax, Safe-RLHF, extensible
      Auto-MappingNoNoNoYes
      Peak Throughput1x (baseline)~2.5x~1.2x1.53x–20.57x
      Code ModularityLow (coupled)Low (coupled)Low (coupled)High (decoupled)

      10. Adoption & Maturity #

      • Published at EuroSys 2025 (top-tier systems venue)
      • Open-sourced as veRL at https://github.com/volcengine/verl — active ByteDance-backed project
      • Production-grade: built by ByteDance's infrastructure team with production deployment experience
      • Growing community: veRL has gained significant traction in the RLHF research community as an alternative to DeepSpeed-Chat and OpenRLHF
      • Limitations for adoption: requires understanding of 3D parallelism concepts; auto-mapping search can be slow for very large model counts; currently best suited for homogeneous GPU clusters
      • Ecosystem fit: plugs into existing Megatron-LM/DeepSpeed/FSDP training stacks and vLLM serving stack, lowering adoption barrier