ProRL Agent: Rollout-as-a-Service for RL Training of Multi-Turn LLM Agents [arXiv:2603.18815]
ProRL Agent
ProRL Agent decouples rollout execution from RL training by encapsulating the entire rollout pipeline, from environment initialization to outcome evaluation, in an independent HTTP server that exposes APIs for trainers to invoke. The rollout server optimizes sandbox execution, manages agent runtime through three non-blocking queues with dedicated worker pools for initialization, execution, and evaluation, dynamically registers LLM servers, synchronizes model weights from the trainer, and routes LLM requests to the least-loaded server. ProRL Agent simplifies trainer’s workload to manage the agent rollout pipeline. However, it does not treat the agent harness as a black box, as the harness must conform to the server’s predefined interfaces. Consequently, it cannot automatically accommodate diverse harness implementations.
Polar: Agentic RL on Any Harness at Scale [arXiv:2605.24220]
Polar extends ProRL Agent by making it harness-agnostic. The key insight is that although agents differ widely in their harness implementations, they rely on LLM API calls to interpret context and determine their next-step actions. Instead of retrieving complete rollouts directly from the agents, Polar intercepts LLM API traffic and reconstructs rollout trajectories from these API communication payloads. However, agent execution does not necessarily follow an append-only model, as subagent spawning, context compaction, and other specific orchestration can introduce non-monotonic trajectories. This deviates from the next-token prediction paradigm used in LLM training, which models sequences as a linear progression of tokens conditioned on preceding context. The key challenge, therefore, is how to reconstruct a minimal set of valid training samples that collectively preserves all information necessary to represent agent executions.
Polar
Polar introduces two strategies to address the challenge. One is a conservative baseline called “per-request”, in which every agent turn containing a single LLM input-output exchange becomes a separate training sample. While this strategy preserves all information, the resulting samples may contain substantial prefix redundancy. Modern harnesses may place a significant burden on the trainers as they usually generate trajectories spanning hundreds of turns. The other one is called “prefix merging”, which gradually appends a sample when the previous agent turn’s prompt is the prefix of the next turn’s prompt. Specifically, prefix merging appends previous-turn’s prompt and original response, interstitial context inserted by the harness (it is masked out during training), and current-turn’s response. However, preserving the original response tokens alone does not guarantee that the merged sequence faithfully reproduces every subsequent LLM API call. If the harness modifies the previous response when constructing the next prompt, the reconstructed context may differ from the actual rollout context, potentially compromising token-level fidelity. Agent Lightning v1.0 discussed more observations, such as chat-template non-compositionality and decode-retokenize drift, that break this token-prefix continuity. It adopts a more conservative strategy: two consecutive LLM calls are merged only when the previous call’s prompt and response tokens is the prefix of the next call’s prompt tokens.
Agent Lightning v1.0: Towards Harnessed Agentic RL [arXiv:2608.17528]
Agent Lightning v1.0
Apart from the sample reconstruction strategies discussed above, Agent Lightning v1.0 explores different approaches to advantage estimation and loss normalization when using GRPO for policy optimization. Under vanilla GRPO, each group consists of G rollouts generated for the same prompt. However, a harness rollout can be further partitioned into multiple discontinuous samples of varying lengths. Therefore, the choice of what constitutes a group determines both how advantages are computed and how per-token losses are aggregated. The figure above illustrates three loss normalization strategies and two options of calculating the reward statistics of a group. Agent Lightning v1.0 adopts rollout-level granularity for both loss normalization and advantage estimation. A concurrent work ClawGym II similarly adopts rollout-level advantage estimation for GRPO. Besides GRPO, ClawGym II also discussed a practical yet simplified variant of advantage estimation for PPO, in which samples within a rollout are treated independently, with both 𝛾 and 𝜆 set to 1. All samples are assigned the same rollout-level reward at their respective final tokens.
Tree Training: Accelerating Agentic LLMs Training via Shared Prefix Reuse [arXiv:2511.00413]
Tree Training
Tree Training addresses the sample reconstruction challenge at the infrastructure level. The token sequences generated by an agent execution trajectory essientially forms a tree structure, where each node corresponds to a block of consecutively appended tokens and its children share the same prefix, collectively formed by the nodes along its root-to-node path. Tree Training adapts the attention mechanism, which is originally designed for linear token sequences, to support tree-structured inputs. The paper begins by mathematically deriving that the baseline loss, averaging over all root-to-leaf paths independently, is identical to a per-token weighted sum loss where each token’s weight is the fraction of paths passing through the token (i.e., the number of such paths divided by the total number of paths). This equivalence allows each node in the tree to be traversed exactly once (e.g., via DFS), avoiding redundant computation across shared prefixes. To preserve attention correctness under DFS traversal, Tree Training construct a tree attention mask that restricts each token to attend only to its preceding tokens along the same path. In case of state space models such as Gated DeltaNet, each node in the tree is viewed as a chunk and information is propagated from its parent chunk rather than its preceding chunk in DFS order. The causal convolutions are routed similarly.
Trajectory Tree Partition
A single trajectory tree can be split into multiple partitions to fit within GPU memory capacity. The resulting partition dependency graph must preserve a tree structure rather than a general graph. That means when you treat a partition as a node and erase edges between nodes within the same partition, each non-root partition must have exactly one parent. This constraint prevents computation graphs from multiple parent partitions from having to reside in GPU memory simultaneously.
BibTeX citation
@misc{jiajun2026bbrlaah,
title = {Black-Box Reinforcement Learning on Any Agent Harness},
author = {Hu, Jiajun},
year = {2026},
month = {October},
journal = {richardhooooo.github.io},
url = {https://richardhooooo.github.io/blog/2026/10/27/notes/}
}