Hpu mxfp8

[yiliu30]

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results

PLEASE FILL IN THE PR DESCRIPTION HERE ENSURING ALL CHECKLIST ITEMS ABOVE HAVE BEEN CONSIDERED.

Purpose

Test Plan

Test Result

BEFORE SUBMITTING, PLEASE READ https://docs.vllm.ai/en/latest/contributing (anything written below this line will be removed by GitHub Actions)

[gemini-code-assist]

Summary of Changes

Hello @yiliu30, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces comprehensive support for NVFP4 and MXFP8 quantization within vLLM, focusing on both general linear layers and Mixture of Experts (MoE) layers. It includes new quantization schemes, their integration with HPU-specific optimizations, and provides emulation capabilities for NVFP4 to ensure broader compatibility. The changes aim to improve performance and memory efficiency for models utilizing these advanced quantization formats.

Highlights

• New Quantization Schemes: Introduced support for W4A4 NVFP4 (NVIDIA FP4) and W8A8 MXFP8 (Mixed Precision FP8) quantization schemes, significantly expanding vLLM's capability to handle various low-precision models.
• HPU Integration and Optimization: Enhanced vLLM's compatibility with Habana Processing Units (HPUs) by integrating the new quantization schemes, adding HPU-specific environment variables (VLLM_USE_STATIC_MOE_HPU, VLLM_HPU_FORCE_CHANNEL_FP8), and providing an example HPU inference script.
• NVFP4 Emulation Support: Implemented reference NVFP4 quantization and dequantization utilities, enabling these schemes to run via software emulation even on hardware platforms that do not natively support Cutlass NVFP4 operations.
• Mixture of Experts (MoE) Quantization: Extended MoE layers to support the new CompressedTensorsW4A4MoeMethod, incorporating both Marlin kernel integration and a static HPU emulation path for MoE operations.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in issue comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist is currently in preview and may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments to provide feedback.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces support for new quantization formats, including MXFP8 and NVFP4, with a focus on HPU devices. The changes are extensive, adding new quantization schemes, MoE methods, and emulation utilities. A new example script for HPU inference is also included.

My review has identified a critical bug in an __init__.py file due to a missing comma, which could lead to import errors. I've also found some dead code, leftover debugging artifacts, and areas for cleanup, particularly in the new example script. Addressing these points will improve the quality and maintainability of the code.

[gemini-code-assist]

There is a missing comma between the two string literals in the __all__ list. This will cause Python to concatenate them into a single string: 'CompressedTensorsW4A4Fp4CompressedTensorsW8A8MXFp8'. This is a bug and will prevent CompressedTensorsW8A8MXFp8 from being correctly exported, potentially causing ImportError downstream.

Suggested change

	"CompressedTensorsW4A4Fp4"
	"CompressedTensorsW8A8MXFp8"
	"CompressedTensorsW4A4Fp4",
	"CompressedTensorsW8A8MXFp8"

[gemini-code-assist]

This example script contains multiple assignments to model_path with hardcoded, user-specific paths. This is confusing and reduces the script's reusability. It's better to have a single, well-defined default model path (e.g., a public model from Hugging Face Hub) or a clear placeholder, and let the user override it with the --model_path argument.

[gemini-code-assist]

This file contains a large block of commented-out logs and shell output. This content appears to be leftover from a debugging session and should be removed. It adds noise to the example script, making it difficult to read and understand.

[gemini-code-assist]

This block of code is unreachable due to the return statement on line 173. It appears to be dead code from a previous implementation and should be removed to improve code clarity and maintainability. The self.fp8_linear attribute it references is also commented out in the __init__ method.

[gemini-code-assist]

For checking against a single item, using the equality operator == is more direct and readable than using in with a single-element list.

        is_per_tensor_group_weight = (
            weight_quant.strategy == QuantizationStrategy.TENSOR_GROUP
        )

[gemini-code-assist]

A commented-out breakpoint() call is present here. This appears to be a leftover from debugging and should be removed.

[gemini-code-assist]

The import of vllm.envs is done locally within the class. According to PEP 8, imports should be at the top of the file. This improves readability and helps avoid potential circular import issues.

[gemini-code-assist]

This block of commented-out code seems to be for debugging purposes. It should be removed to keep the codebase clean.