AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning [arXiv:2505.24298]

AReaL Motivation

Synchronous RL (top) vs. Asynchronous RL (bottom)

Synchronous RL alternates between rollout generation and model training, with each stage waiting for the other to complete. Training starts only after all trajectories in a batch are generated, while rollout generation pauses until the updated model weights are loaded. This stabilize RL training but sacrifice efficiency. Asynchronous RL decouples rollout generation from training, significantly increasing training efficiency but introduces two challenges mentioned in the figure above.

AReaL

AReaL

AReaL leverages decoupled PPO to address challenge 1. Different from PPO that uses πold as both the behavior policy and the proximal policy, decoupled PPO separates these roles, using πbehav to correct off-policy and πprox to center the clipping range. To save space, AReaL use the parameters before each model update step as πprox. AReaL also provides a simple proof to address challenge 2, as shown in the figure above. To avoid training on excessively stale data, AReaL introduces a rate-limiting mechanism that slows down rollout generation when too many batches are queued for training.

Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learning [arXiv:2607.07508]

PPO vs. SAO

PPO vs. SAO

The comparison between PPO and SAO are summarized in the figure above. There are 2 major differences:

Score Centering Stabilizes Off-policy Reinforcement Learning [arXiv:2609.20807]

Score Centering

Score Centering

Score Centering (SC) provides an intuitive explanation of training–inference mismatch (TIM) in LLM reinforcement learning: policy weights should not be updated when rewards provide no basis for distinguishing good actions from bad ones. Using the covariance identity, SC decomposes the vanilla expected policy gradient at each prefix into a drift term and a signal term. The drift term is the expected gradient of the log-probability of a token sampled from the sampler (i.e., the expected score of a token), scaled by the expected reward at that prefix. This term vanishes under on-policy sampling but is generally non-zero when the trainer and sampler differ (i.e., off-policy), introducing spurious updates even when rewards provide no meaningful learning signal. The trainer may therefore repeatedly distill from a biased version of itself (a.k.a., sampler), whose weights are periodically synchronized with the trainer, causing the error to compound rather than converge. SC eliminates this drift by subtracting the expected score from each token’s score. In short, regardless of how rewards are assigned, TIM introduces a cross-entropy gradient shift with the sampler as the teacher. SC removes this shift, leaving only the updates induced by differences in reward.

According to the paper’s evaluation results, SC alone performs substantially better under quantization than under sampler staleness. The authors attribute this difference to the signal term measuring the covariance between reward and score under the sampler rather than the trainer. Under quantization, the trainer and sampler may share the same or similar model weights but differ in precision for weights, activations, and the KV cache. SC is sufficient to stabilize training despite this numerical mismatch by removing the drift. However, when the sampler lags far behind the trainer, SC cannot fully address the distributional mismatch because the covariance is still estimated under the stale sampler distribution, which may no longer accurately reflect the trainer’s current policy. The authors, therefore, recommend composing SC with importance sampling (IS), which corrects the distributional mismatch by reweighting sampled tokens according to the ratio between the trainer and sampler probabilities. To compose IS, SC substracts the expected IS-weighted score, rather than the expected raw scores, from each token’s score.

BibTeX citation

@misc{jiajun2026rlpg,
  title        = {Off-Policy Reinforcement Learning of Large Language Models via Policy Gradient Methods},
  author       = {Hu, Jiajun},
  year         = {2026},
  month        = {October},
  journal      = {richardhooooo.github.io},
  url          = {https://richardhooooo.github.io/blog/2026/10/14/notes/}
}

↑ Top