[Performance]: Run Arctic embedding benchmarks with V0 and V1

[maxdebayser]

Arctic Embedding Benchmark

As part of RFC #21796, we want to run the Artic embedding benchmarks to replicate the results in their blog post.

[maxdebayser]

In the article they mention that they have a frontend server that receives grpc requests and calls vLLM over http. Their first optimization is related to the translation overhead, so I think it is safe to assume that when we call vLLM directly, the baseline we should compare to is the one reported after this optimization, which is 120K tokens/s on an A10G GPU. After the final optimization, the troughput increases to 230K tokens/s, and improvement of 1.9x.

I don't have an A10G to test, but I'm running the tests with a A100 with 80G of memory to see if I can reproduce the speedup.
I'm running my tests with the vLLM main branch at commit 92ff41abea130a3217faa54abb89ccc27aef3f06 .
The benchmark command line for all tests is

python benchmark_http.py --server http://localhost:8000 --batch-sizes 1,2,4,8,16,32,64 --requests 1280 --concurrency 256 --prompt-length 508 --distribution normal

With V0 and 1 instance I get the following result:

Batch Size	Seq Length	Avg Latency (s)	Throughput (K tokens/s)	Success Rate
1	508	0.4442	131.86	100.00 %
2	508	0.4532	131.05	100.00 %
4	508	0.6606	150.78	100.00 %
8	508	1.0216	207.96	100.00 %
16	508	1.9347	229.14	100.00 %
32	508	3.8049	235.91	100.00 %
64	508	7.7273	238.43	100.00 %

cmdline: VLLM_USE_V1=0 vllm serve Snowflake/snowflake-arctic-embed-m-v1.5

If I apply the optimization idea of running several vLLM instances, I get this result with 4 instances behind a envoy load balancer:

RESULTS:

Batch Size	Seq Length	Avg Latency (s)	Throughput (K tokens/s)	Success Rate
1	508	0.2023	284.64	100.00 %
2	508	0.2024	296.33	100.00 %
4	508	0.2588	382.33	100.00 %
8	508	0.4962	419.10	100.00 %
16	508	0.9764	442.64	100.00 %
32	508	2.1169	418.35	100.00 %
64	508	4.4090	414.54	99.92 %

cmdline: VLLM_USE_V1=0 vllm serve Snowflake/snowflake-arctic-embed-m-v1.5 --gpu-memory-utilization 0.25 --port 8001

This gives a speed up close to 2x for most batch sizes.

[robertgshaw2-redhat]

nice

[maxdebayser]

With V1 I found that the --max-num-batched tokens is limited artificially in config.py for embedding models, (I'll open a fix PR). With this limitation out of the way, I found that increasing the number of API server processes helps the throughput and the latency a lot, possibly because the request parsing, tokenization and response formatting are parallelized.

Batch Size	Seq Length	Avg Latency (s)	Throughput (K tokens/s)	Success Rate
1	508	0.1912	298.99	100.00 %
2	508	0.1803	322.70	100.00 %
4	508	0.2439	397.75	100.00 %
8	508	0.5789	366.55	100.00 %
16	508	0.9406	468.13	100.00 %
32	508	2.0675	428.43	100.00 %
64	508	4.0977	437.15	100.00 %
96	508	6.2478	442.35	100.00 %

cmdline:

vllm serve Snowflake/snowflake-arctic-embed-m-v1.5 --max-num-batched-tokens 65536 --api-server-count 8

Here we see that the speedup is also about 2x. With V1 there was no benefit in running more than once instance per GPU, which means that vLLM seems to be able to saturate the device in V1.

[maxdebayser]

@noooop , I also tested with --enforce-eager. It's a bit slower in general.

Batch Size	Seq Length	Avg Latency (s)	Throughput (K tokens/s)	Success Rate
1	508	0.2440	238.04	100.00 %
2	508	0.2331	249.40	100.00 %
4	508	0.2804	340.28	100.00 %
8	508	0.5446	378.19	100.00 %
16	508	1.1728	389.39	100.00 %
32	508	2.1292	421.99	100.00 %
64	508	4.4250	416.16	100.00 %

[noooop]

@noooop , I also tested with --enforce-eager. It's a bit slower in general.

Misled by the PyTorch profiler in PyTorch 2.7

#21796 (comment)