Lego-RL: Harness-Native Reinforcement Learning for Coding Agents

Why train inside the native harness?

A coding agent does more than ask a model for the next token. Its harness decides which files and tool outputs enter the context, when to compact the conversation, and how to continue after a failed command. Adapting that loop to a trainer can change the environment in which the policy learns. We ask what it takes to train directly inside the harness that will run the agent.

The same Qwen3.5-35B-A3B checkpoint scores 64.0, 62.4, and 57.2 on SWE-bench Verified under OpenHands SDK, Claude Code, and OpenCode. This 6.8-point spread shows that the surrounding harness matters to measured performance. It motivates preserving the native interaction loop during training; it does not, by itself, establish that native training outperforms a rewritten harness.

The trained policies score above the starting checkpoint in all three harnesses. They also exceed Qwen3.6-35B-A3B by 3.0, 4.8, and 6.0 points under this evaluation protocol. These comparisons measure the outcome of the full training recipe; they do not isolate the contribution of any one component.

All three runs begin with the same checkpoint and use the same 2,699 tasks and 200k context budget for three epochs (126 steps). Training reward increases in each run without an observed collapse in policy entropy. The validation curves below show how the separately trained policies develop over time.

The policies use that budget differently. Mean response length grows from 43.5k to 90.9k tokens on OpenHands SDK and from 41k to 51k on Claude Code. Differences in context management may help explain this pattern, but we have not isolated their effect in an ablation. A shared context limit does not imply equal realized inference cost.

The trajectories also reveal a change in how the agent checks its work. In the sampled OpenHands SDK trajectories, the share that reread an edited file rises from 73.6% to 98.1%, while recovery after a failed command changes much less. We examine these behaviors below, alongside the additional interaction cost.

Connecting the agent to the trainer

Lego-RL connects verl for optimization with Harbor for task execution. The harness runs in a fresh sandbox and sends model requests through a proxy. The proxy captures generation-time tokens and metadata; the task’s executable verifier supplies the reward. The harness keeps control of the interaction loop.

This separation lets the same training and execution infrastructure serve all three harnesses. Adding a harness requires an adapter to launch it, configure its model endpoint, and collect its result, followed by checks that its request and history handling are captured correctly.

The training objective uses the task’s pass/fail result to update policy-generated tokens. Each model response is conditioned on the context the harness actually supplied. The challenge is preserving that relationship while the harness edits history and trials run asynchronously.

Pillar 1: Faithful optimization

The problem: the archived transcript ≠ the token sequence at sampling time

A readable conversation transcript is useful for inspecting a run, but re-tokenizing it may not recover the sequence seen during generation. Policy-gradient training needs probabilities for the sampled tokens under the contexts that produced them. If either is reconstructed incorrectly, the resulting policy ratio is unreliable.

Real harnesses rewrite their own history:

• Claude Code injects <system-reminder> blocks mid-conversation
• OpenHands compacts history once the context window fills
• tool-call arguments get re-serialized, sometimes with keys in a different order
• sub-agents share a session with their parent agent

These changes can introduce mismatches between the recorded interaction and the sequence used for optimization. The run may continue without an error, so we check alignment explicitly.

Solution 1: an in-process proxy at the serving boundary

The proxy accepts Anthropic- and OpenAI-compatible requests and records token IDs, response masks, log-probabilities, and MoE routing metadata at generation time. Tool-call echoes are matched by ID, and sub-agent conversations are isolated. Exact token reuse applies to spans that preserve the captured prefix; history rewrites require reconstruction and remain a limitation of capture fidelity.

Solution 2: replay MoE expert routing

Matching token IDs is still not enough for MoE. If vLLM routes token $t$ to experts {3, 17} at sampling time and the training forward pass picks {3, 41}, the two sides are computing different probabilities.

Replaying rollout-time routing (R3) raises sampled-token probability Pearson correlation from 0.9946 to 0.9993 and reduces mean absolute token-probability error from 0.0062 to 0.0025. Two silent bugs showed why this has to be checked. A one-position routing misalignment scored worse than no replay (0.750 vs 0.995). An undersized capture buffer for hybrid-attention models wrote out-of-bounds entries as 0, and coverage fell to 24% before the guard started raising. After the fix, coverage stays above 99.8%.

Write ${\ell }_{i,(t,j)}^{\mathrm{roll}}$ for the log-probability recorded at generation time and ${\ell }_{i,(t,j)}^{\mathrm{train}}(\theta )$ for the value the trainer recomputes. Faithful optimization requires:

This comparison uses the same weights ${\theta }_{k^{\prime }}$ on both sides. During asynchronous training, the trainer may have moved to ${\theta }_k$; policy ratios account for that version difference, provided the captured tokens and probabilities are correct. They cannot repair a corrupted record. Across the three production runs, the median probability Pearson correlation over training steps is at least 0.998 for each harness. We also inspect probability errors and KL divergence, since correlation alone does not establish numerical agreement.

Pillar 2: Reliable execution

Layer 1: Reward integrity. Close the paths that score without solving

Each task’s executable tests supply a binary reward: 1.0 if resolved, 0.0 otherwise. That makes the feedback directly testable, but it also makes the verifier and sandbox part of the learning signal. We found several ways to obtain a passing result without producing the intended repair.

Defenses sit in the task environment, on by default, and flip with Harbor's phases: denied while the agent runs, restored for grading.

Layer 2: Difficulty is relative to the current policy

Each task samples 8 rollouts. All-success or all-fail groups contribute no gradient, so a fixed pool gets less informative as the policy improves: on OpenHands SDK, zero-variance groups climb from 44.7% to 51.4%; OpenCode holds around 43.3%.

Screening starts from 36,884 OpenSWE candidates. A task has to be valid, executable, and solved 1 to 3 times out of 4 by Qwen3.6-27B on OpenHands SDK. That leaves 2,699 tasks, which also work on Claude Code and OpenCode. We ablated the cutoff on four 951-task pools: the selected band and its upper half improve; the lower half and an unscreened sample do not. 72.7% of the unscreened pool was never solved.

Infrastructure failures also affect which trajectories enter the update. We exclude timeouts and environment-setup failures from the group’s advantage statistics and assign them zero loss weight: 7.1% of Claude Code trajectories, 2.4% of OpenHands SDK, and 6.4% of OpenCode. Valid trajectories that reach a turn or token limit retain their verifier outcome.

Layer 3: 91% of a single trial is the agent executing

An OpenHands SDK trial takes 920 seconds on average, with agent execution accounting for 91.3% of wall time. This stage includes model inference and tool interaction. Long and uneven trial lengths make synchronous batches wait for their slowest members; asynchronous execution lets new trials start as slots become available.

The measured comparison shows a 2.5× reduction in time per step. The runs used GPU groups with different compute throughput, however. Correcting the synchronous run’s optimizer and log-probability recomputation time gives an estimated 1.9 hours per step, versus 1.0 hour asynchronously, at a maximum staleness of one update. This correction narrows the estimated gain attributable to scheduling.

Partial rollout keeps a ten-minute trial when the weights need to sync. It interrupts vLLM, loads the new weights, and resumes on the same replica using the prefix KV cache. The harness receives the completed generation through the original request.

Pillar 3: Observable training

Failure attribution: quickly telling policy problems from infrastructure problems

Failures can look the same on the reward curve and have different causes. One run died at step 3 because the tool-call parser was set to hermes instead of qwen3_coder. In another, validation score fell from 0.556 to 0.150. It looked like policy collapse, but only 60 of 172 trajectories had reached grading; the rest never finished environment setup.

Another parser mismatch terminated all 1,024 trajectories on the first turn. A separate run did show policy collapse. Inspecting the trajectories helped distinguish these cases; retrospective analysis of the collapse yielded an early-stop condition that would have triggered eight steps earlier. That retrospective result still needs testing on new runs.

To explore a run, start with the Claude Code dashboard’s termination breakdown, then compare the train/inference consistency diagnostics with the reward curve. These views help check whether a change in score coincides with execution failures or probability mismatches.

What did training actually change? Evidence at the behavioral level

The reward curve tells us whether performance improved. To examine what changed, we scanned 420 trajectories from each end of the OpenHands SDK production run for tool use and verification behavior. These transcript-derived measures describe actions we can observe; they do not establish the model’s intent.

Self-verification moved the most:

The later trajectories show more inspection before editing and more rereading afterward. Among trajectories with a failed command, the share that ultimately resolves the task changes from 63.9% to 66.8%, a smaller shift than the self-checking measures. These data do not establish why the behaviors changed at different rates, or whether the recovery difference is robust across repeated runs.

Improvement appears in both coverage and repeated success. The share of task groups with at least one success in eight attempts (pass@8) rises from 83.2% to 87.9%; the share with all eight attempts successful (pass⁸) rises from 28.3% to 39.4%. These group statistics compare the first and last thirds of training. They suggest a larger observed change in consistency, without implying that training only helps tasks the model could already solve.

There is also a cost: average turns rise from 46.6 to 83.1, while tokens per turn increase by 17%. Longer interaction histories can put pressure on turn limits even when the context budget is large. In the tool-allocation comparison below, percentages are shares of all tool calls. This differs from Table 6, where the test-running percentage counts trajectories that run tests at least once.

What these experiments establish

These runs show that an existing coding-agent harness can remain in control of interaction while its policy is trained with RL. Making that work requires checking the path from sampled tokens to policy updates, from sandbox execution to reward, and from an aggregate score to the trajectories behind it. Those checks make the training result easier to interpret and failures easier to investigate.

The evidence here covers one initial model and one production run per harness, with the reported checkpoints selected by periodic validation. Repeated runs are needed to quantify uncertainty. We have not isolated the contribution of every component, or directly compared native-harness training against a matched rewritten harness. The reward-integrity defenses address observed failure modes; they are not a guarantee against every possible shortcut.

What's next

• More tasks. The next pool will go past Python SWE-bench-style repair: more languages, NL2Repo (stand up a repository from a natural-language spec), and agent-user interaction where the agent has to ask and iterate instead of closing a ticket alone.
• Joint scaffold training. The three production runs each trained one policy on one harness. Next we will train a single policy across scaffolds, so it has to stay useful under more than one control flow. That is the same harness dependence that opened a nearly 7-point gap on the starting checkpoint.
• PPO critic. A terminal reward can reinforce earlier actions, but offers no direct label for which intermediate step helped. We plan to test whether a learned value function improves credit assignment and recovery after failed actions. The current behavior measurements motivate that experiment; they do not predict its outcome.

Framework updates, harness adapters, checkpoints, and task indices will stay open.

BibTeX

@article{du2026legorlharnessnativereinforcementlearning,
  title={LEGO-RL: Harness-Native Reinforcement Learning for Coding Agents},
  author={Yiming Du and Yuxin Jiang and Tao Yuan and Jianbo Dai and Shaowei Wang and Jierun Chen and Chaofan Tao and Xianzhi Yu and Lifeng Shang and Kam-Fai Wong and Xiaohui Li and Haoli Bai},
  year={2026},
  eprint={2608.17393},
  archivePrefix={arXiv},
  primaryClass={cs.AI},
  url={https://arxiv.org/abs/2608.17393},
}