Should not add hidden+residual here cause RMSNorm already does that

rahul-tuli · web-flow · commit 629829f15470 · 2025-10-03T17:26:25.000+05:30
diff --git a/vllm/model_executor/models/llama_eagle3.py b/vllm/model_executor/models/llama_eagle3.py
@@ -188,7 +188,6 @@ def forward(
                 hidden_states=hidden_states,
                 residual=residual,
             )
-        hidden_states = hidden_states + residual
         hidden_states, hidden_prenorm = self.norm(hidden_states, residual)
         return hidden_states, hidden_prenorm
 

Original file line number	Diff line number	Diff line change
`@@ -188,7 +188,6 @@ def forward(`
`188`	`188`	`hidden_states=hidden_states,`
`189`	`189`	`residual=residual,`
`190`	`190`	`)`
`191`		`- hidden_states = hidden_states + residual`
`192`	`191`	`hidden_states, hidden_prenorm = self.norm(hidden_states, residual)`
`193`	`192`	`return hidden_states, hidden_prenorm`
`194`	`193`