Disentangling Evolution Capabilities in Self-Evolving LLM Agents
1The Pennsylvania State University 2UC Santa Cruz 3Amazon 4Emory University 5UIUC 6Northeastern University
*Equal contribution·Correspondence: luhanqin@amazon.com
LLM agents are increasingly deployed as systems built around editable external harnesses, including prompts, skills, memories and tools, that shape task execution without changing model parameters. Harness self-evolution adapts such agents by updating these harnesses from execution evidence. Yet it remains unclear whether a model's base capability in task-solving predicts its capabilities in harness self-evolution: which models produce useful harness updates, and which actually benefit from them?
We analyze two harness self-evolution capabilities: (i) harness-updating, the capability to produce useful persistent harness updates from execution evidence; (ii) harness-benefit, the capability to benefit from updated harnesses during task solving. Our analysis reveals two findings. First, harness-updating is flat in base capability: models from different capability tiers produce harness updates that lead to surprisingly similar gains; even Qwen3.5-9B's updates yield gains comparable to those of Claude Opus 4.6. Second, harness-benefit is non-monotonic in base capability: weak-tier models benefit little from updated harnesses, mid-tier models benefit most, and strong-tier models benefit less than mid-tier.
We trace low gains at the weak tier to two failure modes: weak-tier models may fail to activate relevant harness artifacts, or activate them but fail to follow them faithfully. These findings suggest investing capability budget in the task-solving agent rather than the evolver, and targeting harness invocation and long-horizon instruction following in agent training.
In harness self-evolution the model weights stay frozen. What changes is the harness around them: the prompts, skills and memories the agent works with. An evolver model reads the agent's execution evidence and rewrites that harness between batches of tasks.
Starting from an initial harness H0, the loop runs for T steps. At step t the agent (f, Ht−1) attempts a batch of tasks, and the evolver e turns the trajectories and outputs into an update that produces Ht.
An end-to-end score reports one number for the whole loop. It cannot say whether a gain came from the evolver writing good updates or from the agent making good use of them. We separate the two by crossing agents with evolvers. For a pairing (f, e), the evolution gain is
Δ(f, e) = J(f, HT(f, e)) − Mbase(f)where Mbase(f) = J(f, H0) is the model's base capability: its pass rate with the initial harness.
How much the updates an evolver writes help a fixed set of anchor agents ℱ*: Opus 4.6, Sonnet 4.6 and Qwen3-235B.
The largest gain an agent gets from harnesses written by a fixed set of anchor evolvers ℰ*, the same three models.
500 real GitHub issues from 12 Python repositories. A patch passes if it satisfies the hidden tests.
500 tool-use tasks, each needing 3–6 calls across 36 real MCP servers that expose 220 tools.
86 tasks in 11 domains with deterministic verifiers. The baseline starts with an empty skill set.
Seven LLMs fill the two roles. Claude Opus 4.6, Sonnet 4.6 and Haiku 4.5, Qwen3-235B-A22B, Qwen3-32B and GPT-OSS-120B serve as agents and evolvers; Qwen3.5-9B, the smallest model in the study, joins as an extra evolver. All pairs on a benchmark share the initial harness, the task stream, the prompt templates, the evolution budget and the turn limit, so only the backbone changes. Evaluation is in situ: each task is scored under the harness in place when it is attempted, before its own evidence can shape the next update.
Swap the evolver from Claude Opus 4.6 down to a 9B open model and the agents gain about the same. No evolver comes out ahead on every benchmark.
On every benchmark, the best and worst of the seven evolvers differ by at most 3.1 pp, and the ranking reshuffles. Qwen3-235B leads on SWE (8.2 pp) and ranks last on MCP (0.6 pp).
The smallest evolver, Qwen3.5-9B, posts the highest gain on SkillsBench (3.8 pp), ahead of Opus 4.6 (2.3 pp) and Qwen3-235B (1.5 pp).
On the SkillsBench task flink-query, an Opus 4.6 agent with no evolved skill leaves out the FINISH-event filter and scores 0.67. Given the skill written by either Qwen3.5-9B or Opus 4.6, the same agent solves the task. The two skills prescribe the same sequence of steps and differ only in implementation details and length.
flink-query, with an Opus 4.6 agent. Left: no evolved skill (score 0.67). Center: skill evolved by Qwen3.5-9B (score 1.0). Right: skill evolved by Opus 4.6 (score 1.0). Both skills encode procedurally similar guidance.
Plotting each anchor agent's score under all seven evolved harnesses next to its no-evolution baseline makes the imbalance plain. On MCP-Atlas, the spread across evolvers for a single agent is at most 5.1 pp (Qwen3-235B), against a 36.0 pp gap between the Opus 4.6 and Qwen3-235B baselines.
Even the most lopsided pairing does not close that gap. Give the weakest anchor agent its best evolver and the strongest anchor agent its worst, and the strong agent still leads by 18.6 to 35.2 pp on every benchmark. The pattern holds on SWE and SkillsBench as well.
Weak models have the most headroom, yet they gain the least from evolved harnesses. Mid-tier models gain the most. Strong models gain less, because they are already near the ceiling.
Table 1Base pass rate (%) and harness-benefit Δ (pp) for each task-solving agent. Bold green: the largest Δbenefit on each benchmark.
| Agent | SWE | MCP | SB | |||
|---|---|---|---|---|---|---|
| Base | Δ | Base | Δ | Base | Δ | |
| Qwen3-32B | 3.6 | 4.4 | 3.6 | 1.0 | 0.0 | 5.8 |
| Qwen3-235B | 20.7 | 19.3 | 25.0 | 4.3 | 4.7 | 1.1 |
| GPT-OSS-120B | 26.2 | 15.8 | 28.0 | 7.0 | 0.0 | 7.0 |
| Haiku 4.5 | 66.0 | 2.4 | 42.4 | 3.6 | 5.8 | 15.1 |
| Sonnet 4.6 | 73.2 | 2.8 | 54.0 | 3.2 | 24.4 | 3.5 |
| Opus 4.6 | 74.2 | 2.6 | 61.0 | 3.6 | 25.6 | 5.8 |
On SWE the gain peaks at Qwen3-235B (19.3 pp), while the weaker Qwen3-32B gains 4.4 pp and the stronger Opus 4.6 gains 2.6 pp. On MCP the peak moves to GPT-OSS-120B (7.0 pp), again with lower gains at both ends. At the top, small gains fit a ceiling effect: strong models already solve many tasks with the initial harness. At the bottom, the ceiling argument does not apply, which points to a different bottleneck.
The agent never brings the relevant artifact into context, so its guidance never reaches the model. Qwen3-32B loads a skill in only a quarter of its SkillsBench trajectories.
In the threejs case, Qwen3-32B names the right skill but tucks the load request inside a larger action, so the environment never loads it.
The skill is loaded, but the agent does not follow it faithfully. An LLM judge rates whether each skill-loaded run follows the skill's guidance: Opus 4.6 does in 76% of runs, Qwen3-32B in 14%.
In pg-essay-to-audiobook, Qwen3-32B treats the skill as a ready-made script and gives up after the first failure instead of trying the fallback chain it prescribes.
threejs): harness activation failure, where an invalid multi-key load action keeps the skill body out of context.
Right (pg-essay-to-audiobook): harness adherence failure, where the skill is loaded but the agent treats it as a literal script and skips the prescribed fallback chain.
Table 2Activation, adherence and outcome on SkillsBench, models sorted by base capability.
| Agent | SLR | HFR | LPR |
|---|---|---|---|
| Qwen3-32B | 0.251 | 0.142 | 0.023 |
| GPT-OSS-120B | 0.446 | 0.442 | 0.040 |
| Haiku 4.5 | 0.794 | 0.600 | 0.099 |
| Qwen3-235B | 0.961 | 0.350 | 0.022 |
| Sonnet 4.6 | 0.959 | 0.730 | 0.145 |
| Opus 4.6 | 0.957 | 0.757 | 0.177 |
SLR: share of trajectories that load at least one skill. HFR: share of skill-loaded trajectories an LLM judge rates as following the skill. LPR: pass rate among skill-loaded trajectories. Bold green: best per column. Red italic: lowest SLR and HFR.
Table 2 measures the two modes separately. Skill loading is near ceiling for Opus 4.6, Sonnet 4.6 and Qwen3-235B, but drops to 0.446 for GPT-OSS-120B and 0.251 for Qwen3-32B.
Loading the harness is not enough to benefit from it. Qwen3-235B separates the two modes most cleanly: its skill-load rate is 0.961, nearly the same as Opus 4.6, yet its HFR is only 0.350 against 0.757. Its pass rate when a skill is loaded mirrors the gap, at 0.022 against 0.177.
To see whether adherence fades as a trajectory unfolds, an LLM judge scored it at three points: right after the harness is loaded, at the midpoint, and at final validation. Qwen3-32B, GPT-OSS-120B and Opus 4.6 stand in for the weak, mid and strong tiers.
Qwen3-32B falls from 0.52 to 0.13 and GPT-OSS-120B from 0.67 to 0.43, while Opus 4.6 holds from 0.89 to 0.80. Weak models do not simply misread the harness when it loads; they lose hold of it as the trajectory goes on, which points to long-horizon instruction following as the bottleneck.
Table 3Adherence score (0–1) by trajectory phase for representative weak-, mid- and strong-tier agents.
| Phase | Qwen3-32B weak | GPT-OSS-120B mid | Opus 4.6 strong |
|---|---|---|---|
| Harness loaded | 0.52 | 0.67 | 0.89 |
| Mid turn | 0.22 | 0.48 | 0.79 |
| Final turn | 0.13 | 0.43 | 0.80 |
| Drift (load → final) | −0.39 | −0.24 | −0.09 |
Bold green: best score in each phase. Underlined red: worst.
Across evolvers, harness-updating differs by at most 3.1 pp on any benchmark, so a bigger evolver buys little. Post-evolution performance varies far more with the agent.
Weak-tier models often never load the harness: a 25% load rate for Qwen3-32B against ≈96% for strong models. Loading the right artifact should be a learned skill in its own right.
Once loaded, weak-tier adherence decays over four times more steeply than strong-tier adherence (−0.39 vs. −0.09). Holding on to guidance across a long trajectory is a second training target.
@inproceedings{lin2026harness,
title = "Harness Updating Is Not Harness Benefit: Disentangling Evolution Capabilities in Self-Evolving {LLM} Agents",
author = "Lin, Minhua and
Wu, Juncheng and
Wang, Zijun and
Shi, Zhan and
Sang, Yisi and
He, Bing and
Liu, Zewen and
Wei, Tianxin and
Wu, Zongyu and
Zhang, Zhiwei and
Wang, Dakuo and
Zhang, Xiang and
Dumoulin, Benoit and
Xie, Cihang and
Zhou, Yuyin and
Wang, Suhang and
Lu, Hanqing",
booktitle = "Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing",
year = "2026",
pages = "52829--52852",
}