Unsigned integer underflow in max_batch_size

[maxdebayser]

Passing --max-batch-size to the launcher actually had no effect because after a few requests the max_size passed to State::next_batch would underflow becoming a large positive number.

In the scheduler, as soon as the cached batch size reached the max_batch_size the max_size passed to next_batch becomes 0. Since the only check in that funcion is

if Some(batch_requests.len()) == max_size {
    break;
}

and it's called after the batch_requests.len() has become 1, it doesn't do anything to prevent more than 0 requests from being batched.

Now we have cached batch in the server that is larger than max_batch_size and max_size - batch_size as usize underflows.

[drbh]

@maxdebayser thank you for pointing out this bug and the contribution!

I've opened #2382 to run CI and included a couple small changes to extend the fix to the v3 backend. All of your commits have been included in that PR, and will close this one once the other one is merged.

Thank you 🙏

[maxdebayser]

Thanks @drbh , I'll close this PR then.