[RFC]: Multimodal data IPC improvement

[dongluw]

Motivation.

Summary

Currently vllm interprocess communication can account for considerable amount of overhead in some cases, this RFC is aiming at reducing these overhead by using a shared memory based approach for interprocess communication.

Background

According to the profiling result on our internal vision model in a TP>1 setting, the GPU stays idle during engine to worker communication.

The major overhead is two parts 1. IPC between engine and worker process through socket 2. serialization and deserialization through pickle

A similar issue is posted here #16626

Proposed Change.

After initial discussion with @ywang96 and @njhill , proposing this change to address the following communication overhead

1. IPC between engine and worker processes
2. Serialization and deserialization before and after 1.
3. Extra multimodal data transmission: first from P0 to engine, then engine to workers

Design

Step 1.
For addressing 1. engine and worker processes can transmit mm data through a shared memory buffer instead of socket, there’s an existing ShmRingBuffer class which only supports fixed size chunks, because of chunk size limit, it's only turned on when mm data size is less than 16mb by default, otherwise the IPC will be done through socket, which is slow.

We can add/redesign an shared memory buffer implementation for storing variable length mm data.

Step 2.
For addressing 2, based on the above we can skip the (de)serialization of mm data and only keep the mm_hashes for RPC call. Similar to when P0 gets a cache hit, we can set mm data to None in the engine process then restore it from shared memory buffer in the worker process. Maybe one assumption here is mm data only contains numpy/torch tensors or other easy-to-serialize types.

Step 3.
For 3, we can replace the MirroredProcessingCache with the same shared memory buffer ideally to avoid extra mm data transfer between all processes.

Feedback Period.

No response

CC List.

No response

Any Other Things.

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.