Hugging Face Maps the Open-Source RL Training Landscape — and Finds One Problem Everywhere

Hugging Face has reviewed 16 open-source reinforcement learning libraries for training large language models and found that keeping GPUs continuously fed with tokens — not algorithmic sophistication — is the defining engineering challenge of the moment.

Reinforcement learning from human feedback (RLHF) and related techniques such as Group Relative Policy Optimisation (GRPO) have become central to how frontier AI labs align and improve their models. As those techniques mature, the open-source ecosystem has produced a proliferating set of frameworks — from TRL and OpenRLHF to veRL, NeMo-Aligner, and a dozen others — each making different trade-offs between simplicity, scalability, and hardware efficiency. Hugging Face's survey is the first systematic attempt to map that landscape in one place.

The core finding is blunt: in almost every library reviewed, GPUs spend meaningful time waiting for tokens rather than processing them.

Why Token Throughput Is the Real Bottleneck

RL training for language models is structurally different from standard supervised fine-tuning. It requires a generation phase — where the model produces outputs — and a training phase — where gradients are computed based on reward signals. These two phases have very different hardware profiles. Generation is memory-bandwidth-bound and benefits from techniques like continuous batching; training is compute-bound and wants large, dense matrix operations. Running them on the same GPU cluster sequentially means one phase is almost always waiting on the other.

According to Hugging Face's analysis, GPU utilisation in naive RL pipelines can fall well below what the hardware is theoretically capable of. The libraries that perform best are those that find ways to overlap generation and training — either by using separate, asynchronous worker pools or by carefully scheduling micro-batches so that compute is never idle.

Synchronous vs. Asynchronous: The Fork in the Road

The survey draws a clear architectural distinction between synchronous and asynchronous RL training designs. Synchronous systems — the simpler and more common approach — generate a batch of responses, score them, then update the model weights before generating the next batch. This is easy to reason about and debug, but introduces a hard sequential dependency that caps throughput.

Asynchronous systems decouple these stages, allowing a separate pool of inference workers to keep generating samples while the training workers update weights in parallel. This is harder to implement correctly — stale gradients and off-policy data become concerns — but the throughput gains can be substantial. Several of the 16 libraries reviewed have moved, or are moving, toward async architectures precisely because the synchronous ceiling becomes visible at scale.

The analysis notes that frameworks like veRL (developed at Bytedance) and OpenRLHF have invested heavily in this direction, using engines such as vLLM for high-throughput generation and separating inference and training processes across different device groups.

What the 16 Libraries Reveal About the Ecosystem

Beyond the throughput question, the survey reveals a fragmented but maturing ecosystem. Libraries cluster into roughly three tiers. The first tier — tools like TRL — prioritises accessibility and integrates tightly with the Hugging Face ecosystem, making them the natural starting point for smaller teams. The second tier — frameworks like OpenRLHF and veRL — targets research labs and production teams running multi-GPU or multi-node jobs and offers finer control over parallelism strategies. The third tier consists of infrastructure-heavy systems tied to specific hardware vendors or proprietary stacks.

A recurring theme is that no single library dominates across all dimensions. Teams optimising for fast experimentation make different choices than those optimising for training a 70-billion parameter model across hundreds of GPUs. The survey is, in part, an argument that the community would benefit from more standardised benchmarks — currently, performance claims across libraries are difficult to compare because they are measured on different hardware, different model sizes, and different task types. The post does not provide independently verified benchmark numbers; figures cited reflect each library's own documentation and reported results.

Practical Guidance for Teams Choosing a Stack

For practitioners, the post functions as a decision framework. Teams running experiments on a single node are steered toward simpler, synchronous libraries where debugging is tractable and the overhead of async coordination outweighs its benefits. Teams training at scale — particularly those using post-training RL to improve reasoning or instruction-following — are encouraged to evaluate async-capable frameworks and to measure GPU utilisation directly rather than relying on wall-clock time alone.

Hugging Face also flags memory management as an underappreciated concern. RL training holds more state in memory than supervised fine-tuning — the policy model, a reference model (to constrain how far the policy drifts), reward model weights, and replay buffers can collectively exhaust GPU memory in configurations that would be comfortable for standard training. Libraries that implement weight offloading or shared-memory tricks between the policy and reference models have a practical advantage.

What This Means

For any team building or selecting an RL training pipeline today, the Hugging Face survey makes one priority clear: the biggest performance gains are likely to come not from algorithmic novelty but from closing the gap between theoretical GPU utilisation and actual GPU utilisation — and the frameworks that solve that problem cleanly are the ones likely to define the next generation of open-source post-training infrastructure.