"As these requests complete the prefill stage, they transition to decode requests within the batch, and new prefill requests are introduced to maintain the fixed token batch size.""
Assume that the input length of the request is seq_len. Does "B_dense" mean the maximum number of requests that can be processed simultaneously? and the fixed token batch size = B_dense ?
"As these requests complete the prefill stage, they transition to decode requests within the batch, and new prefill requests are introduced to maintain the fixed token batch size.""
Assume that the input length of the request is seq_len. Does "B_dense" mean the maximum number of requests that can be processed simultaneously? and the fixed token batch size = B_dense ?