[serve] Add microbenchmark script

[edoakes]

Why are these changes needed?

Need some basic microbenchmarks to run to track performance. Results from my laptop below - there's work to be done!

{'max_batch_size': 1, 'max_concurrent_queries': 1, 'num_replicas': 1}:
        single client small data 274.79 +- 26.16 requests/s
        8 clients small data 659.81 +- 20.8 requests/s
{'max_batch_size': 1, 'max_concurrent_queries': 10000, 'num_replicas': 1}:
        single client small data 304.6 +- 16.72 requests/s
        8 clients small data 945.01 +- 37.91 requests/s
{'max_batch_size': 10000, 'max_concurrent_queries': 10000, 'num_replicas': 1}:
        single client small data 289.04 +- 10.51 requests/s
        8 clients small data 968.28 +- 23.14 requests/s
{'max_batch_size': 1, 'max_concurrent_queries': 1, 'num_replicas': 8}:
        single client small data 271.3 +- 4.97 requests/s
        8 clients small data 901.81 +- 27.54 requests/s
{'max_batch_size': 1, 'max_concurrent_queries': 10000, 'num_replicas': 8}:
        single client small data 255.21 +- 15.2 requests/s
        8 clients small data 931.31 +- 32.58 requests/s
{'max_batch_size': 10000, 'max_concurrent_queries': 10000, 'num_replicas': 8}:
        single client small data 279.18 +- 4.45 requests/s
        8 clients small data 918.58 +- 74.42 requests/s

Related issue number

Checks

• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/latest/.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failure rates at https://ray-travis-tracker.herokuapp.com/.
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested (please justify below)

[AmplabJenkins]

Can one of the admins verify this patch?

[simon-mo]

can you move https://github.com/ray-project/ray/blob/master/python/ray/serve/examples/benchmark.py to the same directory and name it latency benchmark?

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/26998/
Test FAILed.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/26995/
Test PASSed.