AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning [arXiv:2505.24298]
Synchronous RL (top) vs. Asynchronous RL (bottom)
Synchronous RL alternates between rollout generation and model training, with each stage waiting for the other to complete. Training starts only after all trajectories in a batch are generated, while rollout generation pauses until the updated model weights are loaded. This stabilize RL training but sacrifice efficiency. Asynchronous RL decouples rollout generation from training, significantly increasing training efficiency but introduces two challenges mentioned in the above figure.
AReaL
AReaL leverages decoupled PPO to address challenge 1. Different from PPO that uses πold as both the behavior policy and the proximal policy, decoupled PPO separates these roles, using πbehav to correct off-policy and πprox to center the clipping range. To save space, AReaL use the parameters before each model update step as πprox. AReaL also provides a simple proof to address challenge 2, as shown in the above figure. To avoid training on excessively stale data, AReaL introduces a rate-limiting mechanism that slows down rollout generation when too many batches are queued for training.
Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learning [arXiv:2607.07508]
PPO vs. SAO
The comparison between PPO and SAO are summarized in the above figure. There are 2 major differences:
- Clip design: PPO’s clip aims to prevent the policy from moving in a direction where it has already been updated too aggressively. SAO’s clip stops updating the policy once it deviates too far from the rollout policy, regardless of whether the update is in the right direction or not, for the sake of training stabilities.
- Advantage estimation: standard GAE attempts to calculate the value difference between adjacent tokens. SAO’s GAE proposed to skip observation tokens, which are environment feedback tokens generated outside the model, by bridging the last token of a preceding action and the first token of a succeeding action. In addition, SAO updates the value model faster than the policy model and freezing the attention layers of the value model.
Score Centering Stabilizes Off-policy Reinforcement Learning [arXiv:2609.20807]
Score Centering
Score Centering (SC) provides an intuitive explanation of training–inference mismatch (TIM) in LLM reinforcement learning: policy weights should not be updated when rewards provide no basis for distinguishing good actions from bad ones. Using the covariance identity, SC decomposes the vanilla expected policy gradient at each prefix into a drift term and a signal term. The drift term is the expected gradient of the log-probability of a token sampled from the sampler (i.e., the expected score of a token), scaled by the expected reward at that prefix. This term vanishes under on-policy sampling but is generally non-zero when the trainer and sampler differ (i.e., off-policy), introducing spurious updates even when rewards provide no meaningful learning signal. The trainer may therefore repeatedly distill from a biased version of itself (a.k.a., sampler), whose weights are periodically synchronized with the trainer, causing the error to compound rather than converge. SC eliminates this drift by subtracting the expected score from each token’s score. In short, regardless of how rewards are assigned, TIM introduces a cross-entropy gradient shift with the sampler as the teacher. Score Centering removes this shift, leaving only the updates induced by differences in reward.
BibTeX citation
@misc{jiajun2026rlpg,
title = {Reinforcement Learning of Large Language Models via Policy Gradient Methods},
author = {Hu, Jiajun},
year = {2026},
month = {October},
journal = {richardhooooo.github.io},
url = {https://richardhooooo.github.io/blog/2026/10/14/notes/}
}