ModHarness-RLRecasting Harnesses as Learnable and Evolving Programs for Agentic RL

Zelin He1,*,†, Jason Zhu2,*, Jacob Helwig2,*, Yujian Liu3,†, Li An3,†, Rohit Jain2, Shiyu Chang2,3, Alborz Geramifard2

1The Pennsylvania State University   2LinkedIn   3University of California, Santa Barbara
*Equal contribution   †Work done during an internship at LinkedIn.

TL;DR

Abstract

Agentic reinforcement learning (RL) increasingly relies on a harness, the scaffold around the model that decides what it sees of its own interaction and what guidance it receives. Most existing approaches train under a hand-written harness that stays fixed throughout. However, an effective harness depends on the domain, the data and the model, and keeps changing as the model trains. We introduce ModHarness-RL, which recasts the harness as programs that are learned from data and evolve as the model trains with RL. ModHarness-RL records the interaction as structured State, a set of typed blocks, and two core components of the harness are programs over these blocks, a Compression program that assembles the accumulated history into the model's context and a Steering program that supplies guidance when behavior deviates. During RL training, candidate programs are proposed from recent trajectories, verified by execution, and replayed over past trajectories, so the harness tracks how the model's behavior shifts. We study CodeSearch in depth and further analyze ALFWorld and BIRD SQL, with models from 1.5B to 8B. The proposed Compression and Steering mechanisms improve held-out performance over the fixed-harness baseline in most settings. These results suggest that a harness for agentic RL can be learned from the task's data and evolved with the model.

How should a harness be represented to be learned and evolved?

ModHarness-RL represents the harness as programs over a structured State, which records the interaction as typed blocks such as each turn's reasoning, tool call and tool result. These blocks are the building blocks from which ModHarness-RL assembles the context and control. A Compression program reassembles the blocks into organized model context under a fixed budget, and a Steering program reads them to detect when behavior deviates and appends guidance.

Compression and Steering as programs that read the typed blocks of State.
The harness as programs over State. Compression returns a new view of State that re-renders earlier turns, keeping their key information and offloading what is no longer needed. Steering checks the trace for specific conditions and, when one holds, returns a message that is appended to steer the model's behavior.

Programs over these blocks have three properties that suit learning and evolving.

  1. Learnable, since an LLM can write them directly from training trajectories and their corresponding rewards.
  2. Computationally inexpensive, allowing them to be run on every turn of every trajectory without calling another model.
  3. Testable, allowing each candidate harness to be replayed over stored trajectories to evaluate its fitness before it is accepted or rejected.

How can the harness be learned and evolved with the model?

As training progresses, model behaviors change. Failure modes from previous model versions may become less frequent, while new ones may emerge. ModHarness-RL therefore uses an end-to-end pipeline that updates the harness from online data. It summarizes experiences from training trajectories, proposes and verifies candidate programs, and validates them on held-out training tasks. To evolve the harness with the model, we alternate this pipeline with RL updates of the model weights.

Compress is optimized against the base policy, then policy updates alternate with steer updates.
Learning and evolving the harness with the policy. compress is optimized against the base policy, and policy updates under the latest harness are then alternated with steer updates. Both optimizations propose programs from the reservoir, verify them by execution, and install a winner only through a held-out gate.

The key insight is that the harness should guide only what the model cannot learn on its own.

A Replay mechanism reruns each candidate Steering program over trajectories from past training steps and installs it only for the failures that persist as the model trains.

A learned Compression program and Steering trigger

The program shortens only file reads of at least 1,000 characters in earlier turns, replacing each with a digest that keeps the imports, the definitions, and the first and last lines. Search output, short reads and the newest turn are kept as they are. In this episode, the tool output the model reads before turn 5 falls from 5,626 to 2,809 characters, while the structure of the file stays visible.

A compress program applied to a stored CodeSearch training episode.
A compress program applied to a stored training episode. Top: pseudocode of the generated program. Bottom: turns 1–4 of the episode as recorded (left) and as shown to the model (right); numbered rules match the program. Grey lines are dropped, and teal lines are written by the program.

The check stays silent while the agent searches and opens whole files. After turn 4, the agent's view shows a single function, so the check fires and the guidance asks the agent to scan nearby definitions before submitting. The agent then widens the view to the function body and submits Term.rlike, which earns the full reward of 3/3 for the correct file, class and function.

A steer trigger serving guidance in a live CodeSearch training episode.
A steer trigger serving guidance in a live training episode. Top: pseudocode of the generated check and its guidance. Bottom: the episode as recorded, with the check evaluated after every turn (right); the numbered rule matches the program. The check fires once, after turn 4, and the guidance is appended as a message before the model's next turn.

Results

On CodeSearch, ModHarness-RL improves over the fixed-harness baseline by 14 to 24% in every setting and scores highest against memory, skill, prompt and harness evolution baselines, by 5 to 12% over the strongest of them. On ALFWorld and BIRD SQL, each program alone improves over the base harness in seven of eight settings.

Context-limitedContext-sufficient
MethodLiteVerifiedProOverallΔLiteVerifiedProOverallΔ
Qwen3-4B
Zero-shot0.1720.1390.1130.142−53%0.1380.1110.1050.119−67%
Base harness0.3490.3280.2240.303—0.3850.3510.3460.361—
DeepAgents0.3580.3340.2920.329+9%0.4070.3800.3100.368+2%
INSPO0.3230.3170.2530.299−1%0.4000.4080.3220.379+5%
EvolveR0.3270.3270.2190.294−3%0.4060.4160.3220.384+6%
HarnessForge0.3720.3510.2890.339+12%0.4070.3940.3580.388+7%
SkillRL0.3590.3740.3080.349+15%0.4380.4000.3410.395+9%
+ Compression0.3930.3560.2770.345+14%0.3740.3740.3380.363+1%
+ Steering0.4260.3610.3020.365+20%0.4390.4100.3560.403+12%
ModHarness-RL0.4220.3880.2820.368+21%0.4540.4320.3810.424+17%
Qwen3-8B
Zero-shot0.0900.0840.0950.089−73%0.0980.0970.0590.086−79%
Base harness0.3630.3570.2720.333—0.4310.4190.3520.403—
DeepAgents0.4330.3710.2890.368+10%0.4650.4560.3610.430+7%
INSPO0.3310.3580.2410.313−6%0.4770.4410.3470.425+5%
EvolveR0.3810.3740.2770.347+4%0.4430.4500.3530.418+4%
HarnessForge0.3500.3480.1920.301−10%0.4300.3950.2740.370−8%
SkillRL0.3380.3080.2750.309−7%0.4780.4280.3620.426+6%
+ Compression0.3920.3560.3230.359+8%0.4530.4610.3290.418+4%
+ Steering0.4530.3970.2980.386+16%0.4830.4870.3850.455+13%
ModHarness-RL0.4410.4370.3510.412+24%0.4920.4730.4090.460+14%
GPT-5-mini
Zero-shot0.1130.1080.0550.094—0.4040.3890.2980.366—

CodeSearch validation, F1 averaged over the three granularities. Δ is the relative change in Overall over the base harness of the same model, and bold is best in a column within a model. Shaded rows are ours. The flame marks an RL-trained model and the snowflake one that was not fine-tuned.

Score of the base harness, Compression and Steering on CodeSearch, ALFWorld and BIRD SQL.
Compression and Steering each applied to the base harness on three benchmarks. Models are Qwen3 on CodeSearch, Qwen2.5 on ALFWorld and Qwen3.5 on BIRD SQL, and the tick gives the scale. Whiskers are ±1 SD over repeated evaluations of each trained checkpoint. †SFT warm start.

On CodeSearch, Steering improves over the base harness by 15% on average over the four settings, which is better than Compression at 6%. On ALFWorld under a limited budget, averaged over the two models, Compression improves by 22%, which is better than Steering at 16%.

Score and cost under a limited and a sufficient context budget

Under a limited budget, performance tends to improve, with 8% improvement for Qwen3-8B on CodeSearch and 21% improvement for Qwen2.5-7B on ALFWorld. Under a sufficient budget, Compression uses 74% fewer tokens on ALFWorld and 10% fewer on CodeSearch.

F1 against inference time on CodeSearch, under a limited and a sufficient context budget.
F1 against inference time, Qwen3-8B on CodeSearch. Each panel is one context budget. F1 is averaged over the three granularities and over all validation tasks, and inference time is the average seconds an episode takes end to end.
Score and prefill tokens of Compression against the base harness.
Compression against the base harness. Qwen2.5-7B on ALFWorld and Qwen3-8B on CodeSearch. Prefill tokens are the prompt tokens not served from the prefix cache, summed over an episode and measured by replaying the recorded episodes.

Citation

@article{he2026modharness,
  title  = {ModHarness-RL: Recasting Harnesses as Learnable and Evolving Programs for Agentic RL},
  author = {He, Zelin and Zhu, Jason and Helwig, Jacob and Liu, Yujian and An, Li and
            Jain, Rohit and Chang, Shiyu and Geramifard, Alborz},
  journal = {arXiv preprint},
  year   = {2026}
}