[RFC]: hybrid dtype: float32 for weights and activation, float16 or bfloat16 for attention.

[noooop]

Motivation.

vllm defaults to using float16 inference for float32 models.

https://github.com/vllm-project/vllm/blob/275c5daeb0048c3b3f359bb5d9478b1e75e02857/vllm/config.py#L3053C1-L3055C44

# Set default dtype from model config
if config_dtype == torch.float32:
    # Following common practice, we use float16 for float32 models
    torch_dtype = torch.float16

Most models can maintain their original precision, but a few models require float32.
But flash attn does not support float32, making it a very ineffective choice.

More reports of embedding model precision significantly decreasing at float16:
#17175 #17986 #17785 #15393 ....

Wider numerical issues PTAL #17123

Proposed Change.

This RFC manage to use float32 for weights and activation, float16 or bfloat16 for attention.

For models where precision drops significantly at float16, this might be a better choice, especially models that support long context.

But previously, kv_cache_dtype defaulting to dtype. If the generation model is to support hybrid dtype, kv_cache_dtype should default to be consistent with attn_dtype., which requires changing many places and can introduce a lot of bugs.

So it's better to implement this RPC in two phases, first supporting embedding models (Pooling Models), and then supporting generative models.

Embedding model

PTAL #18940

Generative models

I carefully constructed a test with a small model, which only passed with float32 and hybrid dtype, but failed with float16. This demonstrates the effectiveness of hybrid dtype.

Supporting generation models requires modifying a lot of code, but the benefits are not significant.

Future

• Figure out what training configurations would lead to a decrease in fp16 inference precision, and what configurations can be avoided.
• Online detection of models that may decrease in fp16 inference precision and provide warnings. Now it needs to be compared offline with the results of SentenceTransformer on a dataset. e.g. can the distribution of model parameters be used to detect a potential decrease in fp16 inference precision?
• Online repair of float32 models that decrease in fp16 inference precision. Similar to an in-fight float16 quantization method.

Feedback Period.

No response

CC List.

core vLLM team

Any Other Things.

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[DarkLight1337]

cc @WoosukKwon @mgoin

[DarkLight1337]

Personally I also found that embedding models lose some effectiveness when using float16. I think it's reasonable to default to float32 (if not specified in config) for pooling models only.

[jeejeelee]

Personally I also found that embedding models lose some effectiveness when using float16. I think it's reasonable to default to float32 (if not specified in config) for pooling models only.

Me too, we used embedding model internally and found that the accuracy degradation is quite severe

[22quinn]

Wondering if this has any performance implication?

[yantijin]

I have a question. Is there any difference with following two settings:

1. Transform the model from float32 to float16, and then use vllm for inference.
2. Directly use vllm and the float32 model for inference as vllm will automatically transform float32 to float16
   After testing, I find the output is still different when temperature is 0...

[noooop]

2. After testing, I find the output is still different when temperature is 0...

Using float16, it is almost impossible to make the results the same.

It is better to use other testing methods. For example, vllm uses logprobs for testing and compares end-to-end tests for downstream tasks.

[yantijin]

2. After testing, I find the output is still different when temperature is 0...

Using float16, it is almost impossible to make the results the same.

It is better to use other testing methods. For example, vllm uses logprobs for testing and compares end-to-end tests for downstream tasks.

Is there any methods that i can get the same result with above two settings? And I want to know for a float32 model, how vllm do for downcasting float32 to float16?
For the second setting mentioned above, vllm directly loads the float32 model and downcasts float32 to float16 automatically. The results of my model is good.
But for the first setting, i directly use model.to(torch.float16) and save the model to a new dir. Then I use vllm for inference and the results of my model is worse than the second setting.

[noooop]

Wondering if this has any performance implication?

Using float32 cannot use flash attn, using float16 results in noticeable lower precision, and hybrid dtype is a trade-off.

[noooop]

Then I use vllm for inference and the results of my model is worse than the second setting.

vllm uses param.data.copy_(loaded_weight) to convert data formats

vllm/vllm/model_executor/model_loader/weight_utils.py

Lines 602 to 620 in ed5d408

	def default_weight_loader(param: torch.Tensor,
	loaded_weight: torch.Tensor) -> None:
	"""Default weight loader."""
	try:
	if param.numel() == 1 and loaded_weight.numel() == 1:
	# Sometimes scalar values aren't considered tensors with shapes
	# so if both param and loaded_weight are a scalar,
	# "broadcast" instead of copy
	param.data.fill_(loaded_weight.item())
	else:
	assert param.size() == loaded_weight.size(), (
	f"Attempted to load weight ({loaded_weight.size()}) "
	f"into parameter ({param.size()})")
	param.data.copy_(loaded_weight)
	except Exception:
	# NOTE: This exception is added for the purpose of setting breakpoint to
	# debug weight loading issues.
	raise

according to the pytorch documentation

Copies the elements from src into self tensor and returns self.
The src tensor must be broadcastable > with the self tensor. It may be of a different data type or reside on a different device.