PipelineRL

Source: HuggingFace Date: 2025-04-25 URL: https://huggingface.co/blog/ServiceNow/pipelinerl

Summary

Research summary and open-source release: PipelineRL from ServiceNow, an experimental RL training framework for LLMs featuring inflight weight updates — updating model weights on inference servers without stopping inference, requiring only a brief pause to receive new weights. Achieves high inference throughput, minimal rollout-lag, and better on-policy data collection. Trained 7B and 32B models on Open-Reasoner-Zero dataset matching/exceeding prior results on AIME 2024 and MATH 500 using simplified GRPO (no value function, no KL penalty, no entropy bonus). Hardware: 7B in 3.5 days on 2 nodes; 32B in 6 days on 4 nodes.

Implications

Open-weights ecosystem health. Simplified GRPO without value functions or KL penalties matching Open-Reasoner results is a meaningful finding: the additional complexity in standard PPO/GRPO is not load-bearing for reasoning tasks at this scale. This lowers the implementation barrier for teams doing RL post-training on open-weights models.

Model release cadence (agent reasoning). Inflight weight updates as an RL architecture pattern is the missing piece between “RL works in research” and “RL is usable in practice at scale.” The current experimental status and modular HTTP API design suggest PipelineRL is targeting adoption by teams building custom RL pipelines, not a polished user-facing library — but the architectural pattern will influence the next generation of production RL frameworks.