1. Build a step graph
Capture detached post-RoPE queries and keys during actor replay. Aggregate causal attention over actions and observations.
Learning step-level rewards from the policy’s own attention
Long-horizon LLM agents make multi-step decisions, yet outcome rewards provide little direct supervision for individual steps. We introduce Attention Step Reward (AttnSR), a reward augmentation method that uses the policy’s own attention during training replay to allocate intermediate rewards. AttnSR traces relevance backward from the final step, corrects recency bias, and fuses evidence across layers. A confidence gate and an outcome-centered budget control the reward signal, without a separately trained reward model or additional environment rollouts.
In the Qwen2.5-7B-Instruct study, AttnSR raises held-out ALFWorld success from 91.24% to 97.08%. ScienceWorld shows smaller numerical gains alongside a 29.88% reduction in mean environment interactions. WebShop has mixed outcomes, with no statistically detectable performance improvement. Supplementary smaller-model and GRPO comparisons explore the scope of the approach. These results support policy attention as a practical prior for step-level reward allocation, with benefits that depend on the task and training setting.
AttnSR augments PPO rewards at intermediate step endpoints. Attention determines where to allocate the signal; confidence and the centered outcome determine how much to assign.
Capture detached post-RoPE queries and keys during actor replay. Aggregate causal attention over actions and observations.
Backtrack relevance from the final step, reduce recency bias, and combine allocations through cross-layer consensus.
Scale the allocation by confidence and the outcome residual, retain the outcome reward, and update with PPO.
C is a heuristic confidence gate; pₖ is the allocation over intermediate steps. The outcome reward R is retained. This augments the training return; it does not preserve its sum.

Adjust confidence or switch the outcome. The attention allocation stays fixed; the reward strength and sign change.
Illustrative PPO example, not a recorded trajectory. β = 0.25, baseline = 0.5. Shaping augments the return; it does not preserve its sum.
At zero confidence, no intermediate reward is added.
We evaluate Qwen2.5-7B-Instruct on ALFWorld, WebShop, and ScienceWorld. The main PPO comparison uses four training epochs, one rollout per prompt, learning rate 5 × 10⁻⁶, and LoRA rank 64.
| Method | ALFWorld | WebShop | ScienceWorld | ||||
|---|---|---|---|---|---|---|---|
| Success | Seen | Unseen | Score | Success | Score | Success | |
| GRPO† | 69.71 | 75.00 | 64.18 | 67.21 | 37.80 | 30.19 | 16.66 |
| PPO | 91.24 | 92.86 | 89.55 | 70.74 | 40.20 | 44.53 | 28.59 |
| PPO + AttnSR | 97.08 | 96.43 | 97.76 | 69.71 | 41.00 | 46.35 | 30.90 |
ALFWorld: 274 tasks; WebShop: 500; ScienceWorld: 1,819. ALFWorld score equals success. One training run per method; greedy held-out evaluation.
† GRPO references use 1 epoch and 4 rollouts per prompt; PPO and PPO + AttnSR use 4 epochs and 1 rollout. Optimization budgets differ.
We compare the final policies with TARL, AgentPRM, and FlowTracer on the same 274-task ALFWorld benchmark using Qwen2.5-7B-Instruct. AttnSR reaches 97.08% success, solving 266 tasks.
The 0–80% range is compressed; the 80–100% range is expanded. Axis breaks mark the scale change. Exact success rates are labeled above the bars.
Raw is the instruction-tuned model before RL. GRPO uses 1 epoch / n=4; all other trained policies use 4 epochs (PPO / AttnSR: n=1). These are single-run, descriptive comparisons. AttnSR has the highest observed success, with a one-task margin over TARL (266 vs 265); this does not establish a significant difference between them.
+5.84 pp vs PPO
+2.31 pp vs PPO
+0.80 pp vs PPO
4 epochs · LR 5 × 10⁻⁶ · 1 rollout per prompt · LoRA rank 64 · greedy evaluation
Download data ↓We further study smaller backbones and a GRPO extension. Both Qwen2.5-3B comparisons gain 21 successful tasks out of 274, a gain of 7.66 percentage points computed from unrounded rates.
| Model / optimizer | Baseline | + AttnSR | Δ pp |
|---|---|---|---|
| Qwen3-4BPPO | 87.96 | 91.24 | +3.28 |
| Qwen2.5-3BPPO | 79.56 | 87.23 | +7.66 |
| Qwen2.5-3BGRPO | 12.41 | 20.07 | +7.66 |
PPO: 4 epochs, n=1. GRPO: 1 epoch, n=4. The 3B GRPO result uses initial token-advantage reweighting (α=0.25), not PPO reward shaping or the later RTG variant. High format-error rates limit interpretation. The 3B PPO source trees also contain critic/value-head compatibility differences.
A simple positional reward recovers only part of the ALFWorld gain. A separate asynchronous configuration also improves with AttnSR. Both studies use Qwen2.5-7B-Instruct.
| Method | Success (%) | Δ pp |
|---|---|---|
| PPO | 91.24 | — |
| PPO + Linear Step Reward | 92.34 | +1.09 |
| PPO + AttnSR | 97.08 | +5.84 |
Linear Step Reward uses increasing temporal weights and removes confidence gating. Allocation, gating, and realized reward budgets change jointly.
| Method | Success (%) | Δ pp |
|---|---|---|
| PPO | 59.32 | — |
| PPO + AttnSR | 63.99 | +4.67 |
A separate single-trajectory configuration with different collection, importance weighting, and critic updates; compare within this block.
Where task counts are available, gains are computed before rounding percentages; a displayed difference may vary by 0.01 pp.
Mean environment interactions decrease on ALFWorld and ScienceWorld, while WebShop uses more steps. The table includes GRPO references alongside the canonical PPO / AttnSR comparison.
| Benchmark | Method | Mean steps | Median | P90 | Output (k chars) |
|---|---|---|---|---|---|
| ALFWorld274 tasks | GRPO† | 36.93 | 11.0 | 105.0 | 33.44 |
| PPO | 19.28 | 9.0 | 38.7 | 16.91 | |
| PPO + AttnSR | 15.51 | 8.0 | 31.0 | 13.32 | |
| ScienceWorld1,819 tasks | GRPO† | 27.19 | 13.0 | 74.0 | 43.93 |
| PPO | 30.68 | 15.0 | 80.0 | 45.42 | |
| PPO + AttnSR | 21.51 | 10.0 | 74.0 | 31.99 | |
| WebShop500 tasks | GRPO† | 8.46 | 6.0 | 12.0 | 11.65 |
| PPO | 5.93 | 5.0 | 9.0 | 8.31 | |
| PPO + AttnSR | 8.03 | 7.0 | 12.0 | 11.08 |
GRPO uses the same 1-epoch / n=4 references as the main table. Steps count environment interactions; output counts serialized characters. Neither is a measurement of generated-token compute. These aggregate statistics mix successful and failed tasks.
| Method | Solved / 274 | Overall (%) | Seen (%) | Unseen (%) |
|---|---|---|---|---|
| Raw | 73 | 26.64 | 33.57 | 19.40 |
| GRPO† | 191 | 69.71 | 75.00 | 64.18 |
| PPO | 250 | 91.24 | 92.86 | 89.55 |
| TARL | 265 | 96.72 | 96.43 | 97.01 |
| AgentPRM | 252 | 91.97 | 92.86 | 91.04 |
| FlowTracer | 262 | 95.62 | 96.43 | 94.78 |
| PPO + AttnSR | 266 | 97.08 | 96.43 | 97.76 |
| Task family | Tasks | PPO | + AttnSR | Δ pp |
|---|---|---|---|---|
| Pick | 59 | 100.00 | 100.00 | 0.00 |
| Look | 31 | 96.77 | 96.77 | 0.00 |
| Clean | 58 | 86.21 | 93.10 | +6.90 |
| Heat | 39 | 89.74 | 100.00 | +10.26 |
| Cool | 46 | 86.96 | 97.83 | +10.87 |
| Pick2 | 41 | 87.80 | 95.12 | +7.32 |
Clean, Heat, Cool, and Pick2 improve by 3–5 successful tasks per family. Pick and Look remain unchanged, with limited headroom at the PPO baseline.
AttnSR adds step-level feedback to outcome rewards, using the policy’s own attention. Attention recorded during training replay gives a prior over which earlier actions and observations mattered for the outcome. Confidence gating then sets the strength of each intermediate reward, and outcome centering sets its sign. The method needs no extra rollouts and no separately trained reward model. When Qwen2.5-7B-Instruct is trained with PPO, adding AttnSR raises ALFWorld success from 91.24% to 97.08%. The gain holds at all four evaluated checkpoints. On ScienceWorld, official scores improve at every checkpoint, and mean environment interactions during evaluation fall by 29.88%. On WebShop, where episodes are short (6–8 interaction steps on average) and likely leave less room for step-level credit assignment, AttnSR performs on par with PPO. Supplementary experiments point the same way: smaller-model PPO runs and a GRPO variant that reweights token advantages both improve with attention-based allocation. Future work will test longer contexts and interaction horizons, where dependencies span more distant steps.
Cite the arXiv preprint: arXiv:2611.07932.
@misc{attnsr2026,
author = {Wu, Mengsong and Zhou, Tianchen and Su, Zhaochen and Liu, Zhenhua and Cui, Zhe and Zhou, Aimin},
title = {{AttnSR}: Attention Step Reward for Long-Horizon Agentic {RL}},
year = {2026},
eprint = {2611.07932},
archivePrefix = {arXiv},
primaryClass = {cs.LG},
doi = {10.48550/arXiv.2611.07932},
url = {https://arxiv.org/abs/2611.07932}
}