[Bug]: Wired problem about max token length in Qwen2.5-Math-RM-72B reward task

[Deaddawn]

Your current environment

ValueError: The decoder prompt (length 8274) is longer than the maximum model length of 4096. Make sure that `max_model_len` is no smaller than the number of text tokens.

🐛 Describe the bug

I used 'reward' task of Qwen2.5-Math-RM-72B to process long prompts which is normal in the original huggingface implementation, but get the above error in vllm. Rope scaling can enable a normal run, but i'm not sure this is right. BTW, i tried to set max_model_len, no error is reported but can get nan tensor, which is wired. Also, i check the original config of Qwen2.5-Math-RM-72B, the max position embedding is indeed 4096, and i use their tokenizer to print rm_tokenizer.model_max_length, the result is 131072. So I'm really confused what is wrong...

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[noooop]

1. "max_position_embeddings": 4096, in https://huggingface.co/Qwen/Qwen2.5-Math-RM-72B/blob/main/config.json. At the same time, the absence of parameters for expanding the context, such as rope_scaling, means its max_model_len is 4096. I don't know why "Qwen2.5-Math-RM-72B to process long prompts which is normal in the original huggingface implementation"
2. If max_model_len is 4096, then the precomputed cos_sin_cache is also 4096. Positions exceeding 4096 lead to nan.
   

   vllm/vllm/model_executor/layers/rotary_embedding.py

   Lines 129 to 138 in 3c7d942

   	def _compute_cos_sin_cache(self) -> torch.Tensor:
   	"""Compute the cos and sin cache."""
   	inv_freq = self._compute_inv_freq(self.base)
   	t = torch.arange(self.max_position_embeddings, dtype=torch.float)
   	freqs = torch.einsum("i,j -> ij", t, inv_freq)
   	cos = freqs.cos()
   	sin = freqs.sin()
   	cache = torch.cat((cos, sin), dim=-1)
   	return cache

   
   As you said
   i tried to set max_model_len, no error is reported but can get nan tensor, which is wired.
3. How do you set max_model_len greater than 4096? I want to make this method invalid as soon as possible because it leads to nan
4. Using Rope scaling is a common practice for context expansion, but it is not 100% confirmed that it can be applied to Qwen2.5-Math-RM-72B.

[Deaddawn]

@noooop Thank you for your reply.

1. Is it possible to implement their original method for long prompts?
2. I set the max_model_len like this:

os.environ['VLLM_ALLOW_LONG_MAX_MODEL_LEN']='1'
llm = LLM(model=modelp, task="reward",tensor_parallel_size=4,max_model_len=8192)

3. Any idea how much the rope scaling would affect the reward model

[noooop]

@noooop Thank you for your reply.

1. Is it possible to implement their original method for long prompts?

Transformers does not use cos_sin_cache, which theoretically allows arbitrarily long inputs. However, its performance for parts beyond 4096 is definitely poor, making Rope scaling a better choice.

https://github.com/huggingface/transformers/blob/5c30f7e390429904ecf0749c2e9fd9a3f29cc714/src/transformers/models/llama/modeling_llama.py#L111-L135

3. You can achieve the same result by for example hf_overrides = {"max_model_len": 8000}, but it is not recommended to do so.

4. VLLM_ALLOW_LONG_MAX_MODEL_LEN ┓( ´∀` )┏ You should not use it.

[Deaddawn]

@noooop Thank you for your reply.

1. Is it possible to implement their original method for long prompts?

Transformers does not use cos_sin_cache, which theoretically allows arbitrarily long inputs. However, its performance for parts beyond 4096 is definitely poor, making Rope scaling a better choice.

https://github.com/huggingface/transformers/blob/5c30f7e390429904ecf0749c2e9fd9a3f29cc714/src/transformers/models/llama/modeling_llama.py#L111-L135

3. You can achieve the same result by for example hf_overrides = {"max_model_len": 8000}, but it is not recommended to do so.
4. VLLM_ALLOW_LONG_MAX_MODEL_LEN ┓( ´∀` )┏ You should not use it.

Well, in that case, I better stick to the rope_scaling... 😑

Anyway, appreciate your patience

[Deaddawn]

Close the issue

[Deaddawn]

I take a look at Qwen3 coding and their docs, turned out that they actually use rope scaling, thus explain the above issue. Anyone interested can find related infos here