AttnSR: Attention Step Reward
for Long-Horizon Agentic RL

Learning step-level rewards from the policy’s own attention

Mengsong Wu1,2,3†
Tianchen Zhou3,4†
Zhaochen Su5†
Zhenhua Liu3‡
Zhe Cui3*
Aimin Zhou1,2*

1 Shanghai Institute of AI for Education, East China Normal University

2 Shanghai Innovation Institute3 Hithink Research

4 School of Data Science and Engineering, East China Normal University

5 The Hong Kong University of Science and Technology

cuizhe@myhexin.comamzhou@cs.ecnu.edu.cn

† Equal Contributions‡ Project Lead* Corresponding authors

AttnSR in motion A rollout with one outcome reward is replayed through the policy. Action queries attend to earlier actions and observations; token blocks are pooled into a step graph, relevance is traced back from the final step, corrected for recency and fused across layers. The resulting allocation scales signed intermediate rewards that are added to the outcome reward before PPO. Values are schematic. One rollout, one outcome reward Which earlier steps earned it? ? ? ? Replay the rollout: token attentionEach action query attends over its full causal prefix Ctx A₁ O₁ A₂ O₂ A₃ O₃ A₄ A₁ A₂ A₃ A₄ rows: action queries · columns: causal prefix · gray: masked A₄ looks back at … A₁ O₁ A₂ O₂ A₃ O₃ Pool token blocks into a step graphΣ over key tokens · mean over query tokens · ½ AA + ½ AO AA[4,2] = 0.24 AO[4,2] = 0.32 G[4,2] = 0.28 Step matrix G S₁ S₁ S₂ S₂ S₃ S₃ S₄ S₄ only earlier steps · no re-normalization 0.28 Allocate credit from the final step ① Backtrack with hop decay ② Recency correction ③ Layer consensus layer ℓ₁ S₁ S₂ S₃ layer ℓ₂ S₁ S₂ S₃ layer ℓ₃ S₁ S₂ S₃ damp the pull toward the latest steps geo-mean fused allocation pₖ p₁0.27 p₂0.49 p₃0.24 sₖ = β · C · (R − R̄) · pₖ C = 0.8 confidencecoverage · focus · layer agreement R − R̄ = +0.4 signbetter than the group average → positive s₁+0.021 s₂+0.039 s₃+0.020 added at each intermediate action’s last token R kept as isoutcome reward Train with standard PPOOnly the rewards change; the optimizer is untouched Augmented rewardsR + intermediate sₖ GAEwith pre-update critic V_old Actor θnormalized advantages  Critic Vunnormalized value targets updated policy collects the next rollout A₁ O₁ · observation S₁ A₂ O₂ · observation S₂ A₃ O₃ · observation S₃ A₄ R · outcome reward S₄

1. Collect an interaction trajectory

The policy alternates between actions and environment observations. The completed trajectory provides a terminal outcome R; AttnSR reuses this rollout.

τ = (A₁, O₁, …, Aₖ) → R

2. Replay the trajectory into a token attention heatmap

Replay provides detached Q/K from selected layers. Heatmap rows are action queries; columns are context, action, and observation tokens. Each row uses its full causal prefix. The highlighted A₄ row shows where the last action looks back; darker cells indicate stronger attention.

Policy replay → Q, K → causal token attention

3. Turn token attention into a step graph

Follow the highlighted A₄ × A₂ and A₄ × O₂ blocks: sum key tokens, then average query tokens to obtain AA and AO. Their equal-weight mixture becomes G[4,2], then the S₄ → S₂ edge. The graph keeps earlier steps without re-normalizing; thicker edges mean stronger dependency weights.

G[4,2] = ½ AA[4,2] + ½ AO[4,2] → S₄ → S₂

4. Trace relevance backward and fuse layers

Start at the final step and backtrack with hop decay. Correct recency bias, then fuse layer-wise allocations with a normalized geometric mean to obtain pₖ over intermediate steps.

Backtrack → recency correction → layer consensus → pₖ

5. Gate and sign the intermediate rewards

Coverage, focus, and layer agreement determine confidence C. The centered outcome R − R̄ sets the sign. Add sₖ at each intermediate action’s last token, while retaining the original outcome reward R.

sₖ = β · C · (R − R̄) · pₖ, k < K; β = 0.25

6. Compute advantages and update actor–critic

Augmented rewards enter GAE with pre-update critic values. PPO uses normalized advantages for the actor and unnormalized value targets for the critic. Reward construction and batch targets are detached.

Rewards + V_old → GAE → actor advantages / critic targets

Schematic attention values and token counts, not a recorded trajectory. AttnSR reuses policy replay; intermediate rewards augment the return while the outcome reward is retained.

Abstract

Long-horizon LLM agents make multi-step decisions, yet outcome rewards provide little direct supervision for individual steps. We introduce Attention Step Reward (AttnSR), a reward augmentation method that uses the policy’s own attention during training replay to allocate intermediate rewards. AttnSR traces relevance backward from the final step, corrects recency bias, and fuses evidence across layers. A confidence gate and an outcome-centered budget control the reward signal, without a separately trained reward model or additional environment rollouts.

In the Qwen2.5-7B-Instruct study, AttnSR raises held-out ALFWorld success from 91.24% to 97.08%. ScienceWorld shows smaller numerical gains alongside a 29.88% reduction in mean environment interactions. WebShop has mixed outcomes, with no statistically detectable performance improvement. Supplementary smaller-model and GRPO comparisons explore the scope of the approach. These results support policy attention as a practical prior for step-level reward allocation, with benefits that depend on the task and training setting.

Method

AttnSR augments PPO rewards at intermediate step endpoints. Attention determines where to allocate the signal; confidence and the centered outcome determine how much to assign.

AttnSR: policy replay supplies attention for step graphs; backtracking, recency correction, and layer fusion produce intermediate step rewards.
AttnSR overview. Policy attention is aggregated into a step dependency graph. Backtracking, recency correction, and cross-layer consensus yield an allocation; confidence and the centered outcome scale it into signed intermediate rewards. The outcome reward is retained.

1. Build a step graph

Capture detached post-RoPE queries and keys during actor replay. Aggregate causal attention over actions and observations.

2. Trace and fuse

Backtrack relevance from the final step, reduce recency bias, and combine allocations through cross-layer consensus.

3. Shape and update

Scale the allocation by confidence and the outcome residual, retain the outcome reward, and update with PPO.

sk = β · C · (R − R̄) · pkβ = 0.25

C is a heuristic confidence gate; pₖ is the allocation over intermediate steps. The outcome reward R is retained. This augments the training return; it does not preserve its sum.

AttnSR actor–critic integration: step rewards enter GAE before the PPO actor and critic updates.
Actor–critic integration. AttnSR rewards enter before advantage estimation. The actor and critic retain the existing PPO losses; reward construction is detached.
Interactive illustration: confidence and reward sign

Confidence and outcome scaling

Adjust confidence or switch the outcome. The attention allocation stays fixed; the reward strength and sign change.

sk = β · C · (R − R̄) · pk

Illustrative PPO example, not a recorded trajectory. β = 0.25, baseline = 0.5. Shaping augments the return; it does not preserve its sum.

0.80

At zero confidence, no intermediate reward is added.

Intermediate reward budget+0.100

Experimental Results

Across Three Benchmarks

We evaluate Qwen2.5-7B-Instruct on ALFWorld, WebShop, and ScienceWorld. The main PPO comparison uses four training epochs, one rollout per prompt, learning rate 5 × 10⁻⁶, and LoRA rank 64.

Table 1. Main held-out comparison. Performance values are percentages.
MethodALFWorldWebShopScienceWorld
SuccessSeenUnseenScoreSuccessScoreSuccess
GRPO†69.7175.0064.1867.2137.8030.1916.66
PPO91.2492.8689.5570.7440.2044.5328.59
PPO + AttnSR97.0896.4397.7669.7141.0046.3530.90

ALFWorld: 274 tasks; WebShop: 500; ScienceWorld: 1,819. ALFWorld score equals success. One training run per method; greedy held-out evaluation.

† GRPO references use 1 epoch and 4 rollouts per prompt; PPO and PPO + AttnSR use 4 epochs and 1 rollout. Optimization budgets differ.

ALFWorld Method Comparison

We compare the final policies with TARL, AgentPRM, and FlowTracer on the same 274-task ALFWorld benchmark using Qwen2.5-7B-Instruct. AttnSR reaches 97.08% success, solving 266 tasks.

Final success (%) · segmented scaleQwen2.5-7B-Instruct
ALFWorld success on a segmented vertical scale (0–80% compressed, 80–100% expanded). Raw: 26.64%; GRPO: 69.71%; PPO: 91.24%; TARL: 96.72%; AgentPRM: 91.97%; FlowTracer: 95.62%; PPO + AttnSR: 97.08%

The 0–80% range is compressed; the 80–100% range is expanded. Axis breaks mark the scale change. Exact success rates are labeled above the bars.

Raw is the instruction-tuned model before RL. GRPO uses 1 epoch / n=4; all other trained policies use 4 epochs (PPO / AttnSR: n=1). These are single-run, descriptive comparisons. AttnSR has the highest observed success, with a one-task margin over TARL (266 vs 265); this does not establish a significant difference between them.

More Experimental Evidence

Explore success, score, and interaction length
PPO+ AttnSR
HELD-OUT SUCCESS (%) · HIGHER IS BETTERQwen2.5-7B-Instruct

ALFWorld

274 held-out tasks

PPO91.24%
+ AttnSR97.08%

+5.84 pp vs PPO

ScienceWorld

1,819 held-out tasks

PPO28.59%
+ AttnSR30.90%

+2.31 pp vs PPO

WebShop

500 held-out tasks

PPO40.20%
+ AttnSR41.00%

+0.80 pp vs PPO

4 epochs · LR 5 × 10⁻⁶ · 1 rollout per prompt · LoRA rank 64 · greedy evaluation

Download data ↓

Smaller Models and the GRPO Extension

We further study smaller backbones and a GRPO extension. Both Qwen2.5-3B comparisons gain 21 successful tasks out of 274, a gain of 7.66 percentage points computed from unrounded rates.

ALFWorld success (%) · single-run comparisons
Model / optimizerBaseline+ AttnSRΔ pp
Qwen3-4BPPO87.9691.24+3.28
Qwen2.5-3BPPO79.5687.23+7.66
Qwen2.5-3BGRPO12.4120.07+7.66

PPO: 4 epochs, n=1. GRPO: 1 epoch, n=4. The 3B GRPO result uses initial token-advantage reweighting (α=0.25), not PPO reward shaping or the later RTG variant. High format-error rates limit interpretation. The 3B PPO source trees also contain critic/value-head compatibility differences.

Reward Control and Training Configuration

A simple positional reward recovers only part of the ALFWorld gain. A separate asynchronous configuration also improves with AttnSR. Both studies use Qwen2.5-7B-Instruct.

Positional reward control · ALFWorld
MethodSuccess (%)Δ pp
PPO91.24—
PPO + Linear Step Reward92.34+1.09
PPO + AttnSR97.08+5.84

Linear Step Reward uses increasing temporal weights and removes confidence gating. Allocation, gating, and realized reward budgets change jointly.

Asynchronous training · ALFWorld
MethodSuccess (%)Δ pp
PPO59.32—
PPO + AttnSR63.99+4.67

A separate single-trajectory configuration with different collection, importance weighting, and critic updates; compare within this block.

Where task counts are available, gains are computed before rounding percentages; a displayed difference may vary by 0.01 pp.

What Changes in the Trajectories?

Mean environment interactions decrease on ALFWorld and ScienceWorld, while WebShop uses more steps. The table includes GRPO references alongside the canonical PPO / AttnSR comparison.

Qwen2.5-7B-Instruct · full held-out sets · output includes tool observations
BenchmarkMethodMean stepsMedianP90Output (k chars)
ALFWorld274 tasksGRPO†36.9311.0105.033.44
PPO19.289.038.716.91
PPO + AttnSR15.518.031.013.32
ScienceWorld1,819 tasksGRPO†27.1913.074.043.93
PPO30.6815.080.045.42
PPO + AttnSR21.5110.074.031.99
WebShop500 tasksGRPO†8.466.012.011.65
PPO5.935.09.08.31
PPO + AttnSR8.037.012.011.08

GRPO uses the same 1-epoch / n=4 references as the main table. Steps count environment interactions; output counts serialized characters. Neither is a measurement of generated-token compute. These aggregate statistics mix successful and failed tasks.

Fewer steps can have different causes. On 268 input-matched ScienceWorld tasks solved by both policies, mean steps fall from 11.06 to 7.40. On 246 jointly solved ALFWorld tasks, they rise from 11.19 to 11.99; its aggregate saving accompanies fewer prolonged failures (24 → 8).
Seen / unseen success and task counts
Final held-out evaluation. PPO and AttnSR use the paper’s canonical results.
MethodSolved / 274Overall (%)Seen (%)Unseen (%)
Raw7326.6433.5719.40
GRPO†19169.7175.0064.18
PPO25091.2492.8689.55
TARL26596.7296.4397.01
AgentPRM25291.9792.8691.04
FlowTracer26295.6296.4394.78
PPO + AttnSR26697.0896.4397.76
ALFWorld task-family breakdown
Canonical PPO vs PPO + AttnSR · epoch 4 · success (%)
Task familyTasksPPO+ AttnSRΔ pp
Pick59100.00100.000.00
Look3196.7796.770.00
Clean5886.2193.10+6.90
Heat3989.74100.00+10.26
Cool4686.9697.83+10.87
Pick24187.8095.12+7.32

Clean, Heat, Cool, and Pick2 improve by 3–5 successful tasks per family. Pick and Look remain unchanged, with limited headroom at the PPO baseline.

Learning Dynamics

Qwen2.5-7B-Instruct: PPO and PPO + AttnSR epoch-mean environment outcome reward on ALFWorld, WebShop, and ScienceWorld
Qwen2.5-7B-Instruct · epoch-mean environment outcome reward over 8 training epochs (learning rate 1e-6), excluding shaping rewards. Points are connected without additional smoothing. This supplementary setting differs from the main 4-epoch, 5e-6 held-out comparison.
Download training data (CSV) ↓

Conclusion

AttnSR adds step-level feedback to outcome rewards, using the policy’s own attention. Attention recorded during training replay gives a prior over which earlier actions and observations mattered for the outcome. Confidence gating then sets the strength of each intermediate reward, and outcome centering sets its sign. The method needs no extra rollouts and no separately trained reward model. When Qwen2.5-7B-Instruct is trained with PPO, adding AttnSR raises ALFWorld success from 91.24% to 97.08%. The gain holds at all four evaluated checkpoints. On ScienceWorld, official scores improve at every checkpoint, and mean environment interactions during evaluation fall by 29.88%. On WebShop, where episodes are short (6–8 interaction steps on average) and likely leave less room for step-level credit assignment, AttnSR performs on par with PPO. Supplementary experiments point the same way: smaller-model PPO runs and a GRPO variant that reweights token advantages both improve with attention-based allocation. Future work will test longer contexts and interaction horizons, where dependencies span more distant steps.

BibTeX

Cite the arXiv preprint: arXiv:2611.07932.

@misc{attnsr2026,
  author = {Wu, Mengsong and Zhou, Tianchen and Su, Zhaochen and Liu, Zhenhua and Cui, Zhe and Zhou, Aimin},
  title = {{AttnSR}: Attention Step Reward for Long-Horizon Agentic {RL}},
  year = {2026},
  eprint = {2611.07932},
  archivePrefix = {arXiv},
  primaryClass = {cs.LG},
  doi = {10.48550/arXiv.2611.07932},
  url = {https://arxiv.org/abs/2611.07932}
}