Keep the Tokens Flowing: Lessons from 16 Open-Source RL Libraries

Source: HuggingFace Date: 2026-03-10 URL: https://huggingface.co/blog/async-rl-training-landscape

Summary

Ecosystem survey: deep technical analysis of 16 open-source async RL training libraries (verl, NeMo-RL, SLIME, AReaL, SkyRL, PRIME-RL, and others) across 7 architectural dimensions: orchestration, rollout buffer design, weight sync protocol, staleness management, partial rollout handling, LoRA support, and distributed training backend. Core problem: synchronous RL leaves GPUs idle ~60% of the time (example: 56 minutes to generate 8K-token rollouts on H100 while training GPUs sit idle). Solution pattern: disaggregated inference/training pools connected by async rollout buffers. Ray dominates orchestration (8/16 libraries). No benchmarks; this is architectural documentation and a design guide for TRL’s planned async trainer.

Implications

Transformers library trajectory. The explicit “TRL intended design choices” section — bounded queue with per-token staleness tracking, NCCL bucketed weight transfers (~20ms vs. 100-500ms naive), prefix-resume for agentic workloads — means this survey is not just descriptive but prescriptive. TRL’s async trainer will be designed against this analysis, making it one of the most consequential infrastructure documents for post-training at scale in the open ecosystem.

Open-weights ecosystem health. The MoE correctness issue (DeepSeek-V3.2’s “Keep Routing/Sampling Mask” requirements) being flagged as unsolved in all 16 surveyed libraries is a significant gap. If MoE architectures become dominant for reasoning models, the entire async RL ecosystem has a correctness debt to pay before those models can be reliably trained in the open.