[Refactor]: Use M-RoPE interface directly while defining model class instead of maintaining model specific M-RoPE implementation in mrope.py

[divyanshsinghvi]

Purpose

Fixes #24165
Adds supports for the various models to move away from mrope.py for better handling of the implementation.

• By @DarkLight1337 and @wwl2755 : Use self.get_model() instead of self.model for interface checking to handle cases when the model is wrapped.

Test Plan

pytest tests/models/multimodal/generation/test_qwen2_vl.py::test_qwen2_vl_image_embeddings_input

Test Result

Passed

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
• (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

[divyanshsinghvi]

@Isotr0py is there some model I should attempt to modify using this interface instead or create a test case for the same?

[divyanshsinghvi]

Need to update the documents reflecting the new interface couldn't find markdown for the same.

[Isotr0py]

@divyanshsinghvi You can find all model_type needs MRoPE here (the default vl means Qwen2-VL/Qwen2.5-VL):

vllm/vllm/model_executor/layers/rotary_embedding/mrope.py

Lines 361 to 423 in 28f350e

	@classmethod
	def get_input_positions_tensor(
	cls,
	input_tokens: list[int],
	hf_config: PretrainedConfig,
	image_grid_thw: Union[list[list[int]], torch.Tensor],
	video_grid_thw: Union[list[list[int]], torch.Tensor],
	second_per_grid_ts: list[float],
	context_len: int = 0,
	seq_len: Optional[int] = None,
	audio_feature_lengths: Optional[torch.Tensor] = None,
	use_audio_in_video: bool = False,
	) -> tuple[torch.Tensor, int]:
	from vllm.transformers_utils.config import thinker_uses_mrope
	if thinker_uses_mrope(hf_config):
	return cls._omni_get_input_positions_tensor(
	input_tokens=input_tokens,
	hf_config=hf_config,
	image_grid_thw=image_grid_thw,
	video_grid_thw=video_grid_thw,
	second_per_grid_ts=second_per_grid_ts,
	context_len=context_len,
	seq_len=seq_len,
	audio_feature_lengths=audio_feature_lengths,
	use_audio_in_video=use_audio_in_video,
	)
	elif hf_config.model_type in ["glm4v", "glm4v_moe"]:
	return cls._glm4v_get_input_positions_tensor(
	input_tokens=input_tokens,
	hf_config=hf_config,
	image_grid_thw=image_grid_thw,
	video_grid_thw=video_grid_thw,
	context_len=context_len,
	seq_len=seq_len,
	)
	elif hf_config.model_type in ["ernie4_5_moe_vl", "ernie4_5_vl"]:
	return cls._ernie_get_input_positions_tensor(
	input_tokens=input_tokens,
	hf_config=hf_config,
	image_grid_thw=image_grid_thw,
	video_grid_thw=video_grid_thw,
	context_len=context_len,
	seq_len=seq_len,
	)
	elif "KeyeVL1_5" in hf_config.model_type:
	return cls._keye_get_input_positions_tensor(
	input_tokens=input_tokens,
	hf_config=hf_config,
	image_grid_thw=image_grid_thw,
	video_grid_thw=video_grid_thw,
	context_len=context_len,
	seq_len=seq_len,
	)
	else:
	return cls._vl_get_input_positions_tensor(
	input_tokens=input_tokens,
	hf_config=hf_config,
	image_grid_thw=image_grid_thw,
	video_grid_thw=video_grid_thw,
	second_per_grid_ts=second_per_grid_ts,
	context_len=context_len,
	seq_len=seq_len,
	)

[divyanshsinghvi]

@divyanshsinghvi You can find all model_type needs MRoPE here (the default vl means Qwen2-VL/Qwen2.5-VL):

vllm/vllm/model_executor/layers/rotary_embedding/mrope.py

Lines 361 to 423 in 28f350e

	@classmethod
	def get_input_positions_tensor(
	cls,
	input_tokens: list[int],
	hf_config: PretrainedConfig,
	image_grid_thw: Union[list[list[int]], torch.Tensor],
	video_grid_thw: Union[list[list[int]], torch.Tensor],
	second_per_grid_ts: list[float],
	context_len: int = 0,
	seq_len: Optional[int] = None,
	audio_feature_lengths: Optional[torch.Tensor] = None,
	use_audio_in_video: bool = False,
	) -> tuple[torch.Tensor, int]:
	from vllm.transformers_utils.config import thinker_uses_mrope
	if thinker_uses_mrope(hf_config):
	return cls._omni_get_input_positions_tensor(
	input_tokens=input_tokens,
	hf_config=hf_config,
	image_grid_thw=image_grid_thw,
	video_grid_thw=video_grid_thw,
	second_per_grid_ts=second_per_grid_ts,
	context_len=context_len,
	seq_len=seq_len,
	audio_feature_lengths=audio_feature_lengths,
	use_audio_in_video=use_audio_in_video,
	)
	elif hf_config.model_type in ["glm4v", "glm4v_moe"]:
	return cls._glm4v_get_input_positions_tensor(
	input_tokens=input_tokens,
	hf_config=hf_config,
	image_grid_thw=image_grid_thw,
	video_grid_thw=video_grid_thw,
	context_len=context_len,
	seq_len=seq_len,
	)
	elif hf_config.model_type in ["ernie4_5_moe_vl", "ernie4_5_vl"]:
	return cls._ernie_get_input_positions_tensor(
	input_tokens=input_tokens,
	hf_config=hf_config,
	image_grid_thw=image_grid_thw,
	video_grid_thw=video_grid_thw,
	context_len=context_len,
	seq_len=seq_len,
	)
	elif "KeyeVL1_5" in hf_config.model_type:
	return cls._keye_get_input_positions_tensor(
	input_tokens=input_tokens,
	hf_config=hf_config,
	image_grid_thw=image_grid_thw,
	video_grid_thw=video_grid_thw,
	context_len=context_len,
	seq_len=seq_len,
	)
	else:
	return cls._vl_get_input_positions_tensor(
	input_tokens=input_tokens,
	hf_config=hf_config,
	image_grid_thw=image_grid_thw,
	video_grid_thw=video_grid_thw,
	second_per_grid_ts=second_per_grid_ts,
	context_len=context_len,
	seq_len=seq_len,
	)

Yeah I meant, whether I should be changing all of them which are already implemented or make a new model along this lines to verify?

[Isotr0py]

whether I should be changing all of them which are already implemented

Personally, I prefer to modify one model (e.g. Qwen2-VL) with existing tests and keep remaing models with backward compatability in initial PR, and then migrate remain models to use new interface in following PR. Because some models like Ernie can't run on our CI and needs manual validation.

make a new model along this lines to verify?

I think there is no need to make a new model.

[divyanshsinghvi]

I dont see the original issue creators were added as the task reviewer. Let me know if something else needs to be added.
cc: @DarkLight1337 @Isotr0py @ywang96

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @divyanshsinghvi.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[DarkLight1337]

@ywang96 can you update this PR for Qwen3-Omni? Please remember to add @wwl2755 to co-authors before merging. Thanks

[divyanshsinghvi]

@DarkLight1337 let me know if I need to do something here.

[ywang96]

@DarkLight1337 let me know if I need to do something here.

Let me update this PR for Qwen3-Omni too - I will also add @wwl2755 as co-author
On a second thought - it seems easier if I just add support for Qwen3-Omni in a separate PR than this. Will work on that soon.