Calculate "peak token throughput" number that is closer to vLLM generation throughput

[dagrayvid]

When doing a GuideLLM sweep, the token throughput numbers for each RPS test are lower than a user would expect given the vLLM generation throughput metric.

I think there are two main reasons for this.

1. Server is under-utilized at the beginning of the run while the number of requests in flight is ramping up
2. GuideLLM undercounts the tokens generated towards the end of the test duration, due to cancelled requests not being counted (see Record partially completed request metrics #77)

I propose that we add an additional performance metric, we could call it something along the lines of "peak token generation throughput", hopefully in fewer words.

I have a couple of ideas on how this could be calculated:

1. Calculate token throughput in a sliding window of 30s or 1 minute and use the max, using decode times across all ongoing requests in that window.
2. Calculate token throughput in the duration from the time that the first completed request finished, and the time last completed request started. This doesn't perfectly eliminate the "underutilization" at the beginning and end of the test, but gets pretty close.

WDYT?

[markurtz]

#96 now enables us to grab the metrics for any responses that errored (though it makes an assumption on iterations == tokens, since we don't have usage stats from the server in this case).

Since I need to rework the report generation, I'll include these fixes within that

[markurtz]

Closing out now that #96 is on main