Closed
Description
I would like to clarify whether this check
if (6*(n_moves + nh)*n_layer >= LLAMA_MAX_NODES) {
// the graph is too big, we cannot move more cells
break;
}
is correct in llama_kv_cache_defrag_internal?
Now, if there is one large hole (huge nh
variable) in the kv_cache, the condition will not be met and the cycle will be stopped, although only one move will be enough to fill it.
Wouldn't it be more correct to replace the check with just 6*n_moves*n_layer >= LLAMA_MAX_NODES
?
Metadata
Metadata
Assignees
Labels
No labels