AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning [arXiv:2505.24298]
Synchronous RL (top) vs. Asynchronous RL (bottom)
Synchronous RL alternates between rollout generation and model training, with each stage waiting for the other to complete. Training starts only after all trajectories in a batch are generated, while rollout generation pauses until the updated model weights are loaded. This stabilize RL training but sacrifice efficiency. Asynchronous RL decouples rollout generation from training, significantly increasing training efficiency but introduces two challenges mentioned in the figure above.
AReaL
AReaL leverages decoupled PPO to address challenge 1. Different from PPO that uses πold as both the behavior policy and the proximal policy, decoupled PPO separates these roles, using πbehav to correct off-policy and πprox to center the clipping range. To save space, AReaL use the parameters before each model update step as πprox. AReaL also provides a simple proof to address challenge 2, as shown in the figure above. To avoid training on excessively stale data, AReaL introduces a rate-limiting mechanism that slows down rollout generation when too many batches are queued for training.
Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learning [arXiv:2607.07508]
PPO vs. SAO
The comparison between PPO and SAO are summarized in the figure above. There are 2 major differences:
- Clip design: PPO’s clip aims to prevent the policy from moving in a direction where it has already been updated too aggressively. SAO’s clip stops updating the policy once it deviates too far from the rollout policy, regardless of whether the update is in the right direction or not, for the sake of training stabilities.
- Advantage estimation: standard GAE attempts to calculate the value difference between adjacent tokens. SAO’s GAE proposed to skip observation tokens, which are environment feedback tokens generated outside the model, by bridging the last token of a preceding action and the first token of a succeeding action. In addition, SAO updates the value model faster than the policy model and freezing the attention layers of the value model.
Score Centering Stabilizes Off-policy Reinforcement Learning [arXiv:2609.20807]
Score Centering
Score Centering (SC) provides an intuitive explanation of training–inference mismatch (TIM) in LLM reinforcement learning: policy weights should not be updated when rewards provide no basis for distinguishing good actions from bad ones. Using the covariance identity, SC decomposes the vanilla expected policy gradient at each prefix into a drift term and a signal term. The drift term is the expected gradient of the log-probability of a token sampled from the sampler (i.e., the expected score of a token), scaled by the expected reward at that prefix. This term vanishes under on-policy sampling but is generally non-zero when the trainer and sampler differ (i.e., off-policy), introducing spurious updates even when rewards provide no meaningful learning signal. The trainer may therefore repeatedly distill from a biased version of itself (a.k.a., sampler), whose weights are periodically synchronized with the trainer, causing the error to compound rather than converge. SC eliminates this drift by subtracting the expected score from each token’s score. In short, regardless of how rewards are assigned, TIM introduces a cross-entropy gradient shift with the sampler as the teacher. SC removes this shift, leaving only the updates induced by differences in reward.
According to the paper’s evaluation results, SC alone performs substantially better under quantization than under sampler staleness. The authors attribute this difference to the signal term measuring the covariance between reward and score under the sampler rather than the trainer. Under quantization, the trainer and sampler may share the same or similar model weights but differ in precision for weights, activations, and the KV cache. SC is sufficient to stabilize training despite this numerical mismatch by removing the drift. However, when the sampler lags far behind the trainer, SC cannot fully address the distributional mismatch because the covariance is still estimated under the stale sampler distribution, which may no longer accurately reflect the trainer’s current policy. The authors, therefore, recommend composing SC with importance sampling (IS), which corrects the distributional mismatch by reweighting sampled tokens according to the ratio between the trainer and sampler probabilities. To compose IS, SC substracts the expected IS-weighted score, rather than the expected raw scores, from each token’s score.
BibTeX citation
@misc{jiajun2026rlpg,
title = {Off-Policy Reinforcement Learning of Large Language Models via Policy Gradient Methods},
author = {Hu, Jiajun},
year = {2026},
month = {October},
journal = {richardhooooo.github.io},
url = {https://richardhooooo.github.io/blog/2026/10/14/notes/}
}