Lego-RL: Harness-Native Reinforcement Learning for Coding Agents

Why insist on training inside the native harness?

Most agentic RL pipelines rewrite the agent so the trainer can run it: new init, the trainer's tools, a stop condition the trainer understands. You can train that way. The policy then becomes good under a control flow that is gone at deploy time.

The starting checkpoint makes the cost obvious. The same Qwen3.5-35B-A3B weights score 64.0, 62.4, and 57.2 on SWE-bench Verified in the three harnesses. That is almost 7 points from the harness alone, larger than the headline gain of a lot of post-training work.

Lego-RL trains in the same harness it serves and leads every column. It is 3.0 / 4.8 / 6.0 above Qwen3.6-35B-A3B. The 3.5 to 3.6 jump is 3.4 / 1.0 / 3.4. The training stack moved the score more than a new base model did.

The three runs share a checkpoint, 2,699 tasks, and a 200k context window. Each ran for 3 epochs (126 steps). Reward went up on all three and entropy did not collapse. Native control flow did not break the optimizer, even though the harness keeps rewriting history.

The curves still diverge. Mean response length almost doubles on OpenHands SDK (43.5k to 90.9k tokens) and only goes from 41k to 51k on Claude Code. Same start, same tasks, same context budget, different policies. OpenHands spends tokens on longer exploration and self-checks. Claude Code has to get more score out of a tighter budget. We think context management is the reason: OpenHands compaction vs Claude Code's <system-reminder> and trim rules. We have not run a clean ablation.

The thing we can measure is more specific than "the model thinks longer." It learns a policy for this harness's control flow. That is why the 7-point gap exists. The harness is part of the environment. Change it and the optimum moves. Train in the one you deploy.

One infrastructure, shared by every harness

The trainer is built on verl. Execution is built on Harbor.

• verl handles actor updates and weight sync (FSDP / Megatron-LM / VeOmni), PPO / GRPO / GSPO, a vLLM rollout pool, and sync / fully async / partial-rollout scheduling.
• Harbor wraps a problem statement, a repository snapshot, and an executable verifier into one task. It starts a sandbox, runs the agent, runs the task's tests for a 0/1 reward, then tears the environment down. Production uses Kubernetes (one pod per trajectory). Smaller runs use local or remote Docker.

Lego-RL fills the gap between them.

• AgentLoopWorker. Each trainer step hands a batch of tasks to concurrent workers. A worker owns one trial: launch the unmodified harness in a fresh sandbox, point its base URL at the proxy, let it run, then collect the verifier reward and the token-level trace. About 90% of a trial is not on GPU, so concurrency is what moves throughput.
• An in-process proxy that speaks OpenAI and Anthropic. It is the only path between harness and policy. Token IDs, response masks, log-probabilities, and MoE expert routing get recorded at generation time (see Faithful).
• Defenses live in the task environment, not in the harness. Anti-cheat and staged network policy sit in the sandbox because the harness cannot be edited (see Reliable).

A new harness needs a thin adapter: start the agent, point it at inference, return the interaction. Everything else is shared. OpenHands SDK, Claude Code, and OpenCode already work end to end. Any harness that speaks OpenAI or Anthropic can add one adapter class and pick it in config. Trajectory capture, load balancing, and R3 routing replay come with that.

The optimization objective: a formal definition

A task instance $x=(q_x,R_x,V_x)$ consists of a problem statement, an initialized repository environment, and a task-specific executable verifier. The harness $\mathcal{H}$ belongs to the environment, not to the policy. At turn $t$, the harness maps the current interaction and repository state $s_t$ to a context $c_t=\mathcal{H}(s_t)$, the policy generates an action $a_t\sim {\pi }_{\theta }(\cdot \mid c_t)$, and the harness executes whatever tool actions were requested, producing $s_{t+1}$. A rollout is the sequence of prompt/response pairs actually exchanged at the model API, and the verifier ultimately outputs a single bit:

Only policy-generated tokens participate in training. Writing $\mathcal{M}(\tau )$ for those positions:

Every turn conditions on the harness-supplied context $c_t$, not on the raw history. We maximize the expected verifier reward $J(\theta )={\mathbb{E}}_{x\sim \mathcal{D},\tau \sim \mathcal{H}({\pi }_{\theta })}[r(x,\tau )]$ with group-relative advantages. Each task samples $G$ trajectories and takes ${\hat{A}}_i=(r_i-\bar{r})/(\mathrm{std}(r_{1:G})+\delta )$, with $\delta =10^{-6}$. All three production runs use GSPO's sequence-level surrogate:

where ${\theta }_{k^{\prime }}$ is the policy version that generated the group, and $w({\tau }_i)\in \{0,1\}$ zeroes out trajectories terminated by infrastructure failures. The asymmetric bounds ${ϵ}_{\mathrm{low}}<{ϵ}_{\mathrm{high}}$ give the sequence-level ratio more room to move up than down. Replacing ${\sigma }_i$ with the per-token ratio recovers PPO or GRPO, which the trainer also supports.

Two properties shape the engineering choices that follow. When rewards within a group are identical, ${\hat{A}}_i\equiv 0$: the group stays in the batch but contributes no gradient, so task difficulty relative to the current policy is on the critical path. And $r$ comes from executing code, so the signal is only as credible as the sandbox that produced it.

Pillar 1: Faithful optimization

The problem: the archived transcript ≠ the token sequence at sampling time

The easy log is a conversation transcript, re-tokenized at train time. That is enough for SFT. On-policy RL needs the log-probabilities of the tokens that were generated at sampling time. A transcript recomputation is a different quantity.

Real harnesses rewrite their own history:

• Claude Code injects <system-reminder> blocks mid-conversation
• OpenHands compacts history once the context window fills
• tool-call arguments get re-serialized, sometimes with keys in a different order
• sub-agents share a session with their parent agent

Any of these puts a silent error in the importance-sampling ratio. No exception. The ratio just drifts.

Solution 1: an in-process proxy at the serving boundary

The proxy speaks Anthropic and OpenAI, so the harness only sees a new base URL. At generation time it records token IDs, response masks, log-probabilities, and MoE expert routing. Context alignment then runs turn by turn: match tool calls by ID, not by serialized arguments; isolate sub-agent sessions; drop history the harness compacted away instead of reconstructing it.

Solution 2: replay MoE expert routing

Matching token IDs is still not enough for MoE. If vLLM routes token $t$ to experts {3, 17} at sampling time and the training forward pass picks {3, 41}, the two sides are computing different probabilities.

Replaying rollout-time routing (R3) lifts the correlation from 0.9946 to 0.9993. Two silent bugs showed why this has to be checked. A one-position routing misalignment scored worse than no replay (0.750 vs 0.995). An undersized capture buffer for hybrid-attention models wrote out-of-bounds entries as 0, and coverage fell to 24% before the guard started raising. After the fix, coverage stays above 99.8%.

Write ${\ell }_{i,(t,j)}^{\mathrm{roll}}$ for the log-probability recorded at generation time and ${\ell }_{i,(t,j)}^{\mathrm{train}}(\theta )$ for the value the trainer recomputes. Faithful optimization requires:

That identity is on the same weights ${\theta }_{k^{\prime }}$. Under async training the trainer's ${\theta }_k$ runs ahead of ${\theta }_{k^{\prime }}$. That is bounded off-policyness, and ${\sigma }_i$ corrects it. A broken identity is a capture defect. Nothing corrects for that. Across the three production runs, median train/inference correlation never falls below 0.998.

Pillar 2: Reliable execution

Layer 1: Reward integrity. Close the paths that score without solving

Reward is the task's own tests: 1.0 resolved, 0.0 otherwise. There is no reward model, so there is no reward-model drift. Every shortcut to a score without a fix has to be sealed off, because a strong coding model will find them.

Defenses sit in the task environment, on by default, and flip with Harbor's phases: denied while the agent runs, restored for grading.

Layer 2: Difficulty is relative to the current policy

Each task samples 8 rollouts. All-success or all-fail groups contribute no gradient, so a fixed pool gets less informative as the policy improves: on OpenHands SDK, zero-variance groups climb from 44.7% to 51.4%; OpenCode holds around 43.3%.

Screening starts from 36,884 OpenSWE candidates. A task has to be valid, executable, and solved 1 to 3 times out of 4 by Qwen3.6-27B on OpenHands SDK. That leaves 2,699 tasks, which also work on Claude Code and OpenCode. We ablated the cutoff on four 951-task pools: the selected band and its upper half improve; the lower half and an unscreened sample do not. 72.7% of the unscreened pool was never solved.

When infrastructure fails, we mask the affected trajectory from the loss: 7.1% for Claude Code, 2.4% for OpenHands SDK, and 6.4% for OpenCode. We still keep these trajectories in the batch, but give them zero weight. Trajectories that simply hit the turn or token ceiling continue to count.

Layer 3: 91% of a single trial is the agent executing

An OpenHands SDK trial takes 920 seconds on average, and agent execution accounts for 91.3% of that time. Under Sync, this imbalance caused screening to stall 31 times at batch boundaries, with a median pause of 38.7 minutes. Async reduces per-step time by 2.5×. Even then, generation remains the bottleneck: the trainer still spends 40.8–66.1% of its time waiting.

Partial rollout keeps a ten-minute trial when the weights need to sync. It interrupts vLLM, loads the new weights, and resumes on the same replica using the prefix KV cache. To the harness, the whole trial is still one HTTP request.

Pillar 3: Observable training

Failure attribution: quickly telling policy problems from infrastructure problems

Failures can look the same on the reward curve and have different causes. One run died at step 3 because the tool-call parser was set to hermes instead of qwen3_coder. In another, validation reward fell from 0.556 to 0.150. It looked like policy collapse, but only 60 of 172 trajectories had reached grading; the rest never finished environment setup.

A third run killed all 1,024 trajectories on turn one, again from an incompatible parser. One run did collapse for real. Trajectory-level analysis separated that policy failure from the infrastructure failures and produced an early-stop condition that would have fired 8 steps earlier.

What did training actually change? Evidence at the behavioral level

A rising mean reward does not say what the model learned. Behavior does. We drew 420 trajectories from each end of the OpenHands SDK production run and looked at what the agent actually did.

Self-verification moved the most:

The model started looking around before editing and checking the file after every change. Error recovery barely moved. Among trajectories that hit a failed command, the share that still solves the task rises only from 63.9% to 66.8%. A terminal binary reward only sees the final outcome, so verification gets reinforced and mid-course correction stays flat.

The gain is reliability on tasks it could already solve. pass@8 improves by 4.7 points (83.2 to 87.9). pass⁸ improves by 11.1 points (28.3 to 39.4). Malformed tool calls fall from 1.07% to 0.15%. Average turns rise from 46.6 to 83.1. Tokens per turn only increase by 17%. If you raise the context window and leave the turn limit alone, late-training trajectories get cut off at the ceiling.

What's next

• More tasks. The next pool will go past Python SWE-bench-style repair: more languages, NL2Repo (stand up a repository from a natural-language spec), and agent-user interaction where the agent has to ask and iterate instead of closing a ticket alone.
• Joint scaffold training. The three production runs each trained one policy on one harness. Next we will train a single policy across scaffolds, so it has to stay useful under more than one control flow. That is the same harness dependence that opened a nearly 7-point gap on the starting checkpoint.
• PPO critic. A terminal binary reward cannot credit mid-course recovery, which is why self-verification moved and error recovery did not. We will add a PPO critic so value estimates can carry denser credit assignment than a single pass/fail bit at the end of a trajectory.

Framework updates, harness adapters, checkpoints, and task indices will stay open.

BibTeX

@misc{du2026legorlharnessnativereinforcementlearning,
  title={LEGO-RL: Harness-Native Reinforcement Learning for Coding Agents},
  author={Yiming Du and Yuxin Jiang and Tao Yuan and Jianbo Dai and Shaowei Wang and Jierun Chen and Chaofan Tao and Xianzhi Yu and Lifeng Shang and Kam-Fai Wong and Xiaohui Li and Haoli Bai},
  year={2026},
  eprint={2608.17393},
  archivePrefix={arXiv},
  primaryClass={cs.AI},
  url={https://arxiv.org/abs/2608.17393},
}