Skip to content

Commit ef386df

Browse files
WyldeCatca1207
authored andcommitted
fix(motif): fix lambda init fn
Sync with https://huggingface.co/Motif-Technologies/Motif-2.6B/blob/main/modeling_motif.py#L366 Signed-off-by: WyldeCat <skan1543@gmail.com> Signed-off-by: ca1207 <ca1207zzz@gmail.com>
1 parent 8214f1e commit ef386df

File tree

1 file changed

+1
-1
lines changed

1 file changed

+1
-1
lines changed

vllm/model_executor/models/motif.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -236,7 +236,7 @@ def __init__(
236236
), diff_attn_err_msg
237237

238238
def lambda_init_fn(self, depth):
239-
return 0.8 - 0.6 * math.exp(-0.3 * depth)
239+
return 0.8 - 0.6 * math.exp(-0.3 * (depth - 1))
240240

241241
def forward(
242242
self,

0 commit comments

Comments
 (0)