[MFM-2025-02-03] Merge Main to llama fp8; With Faster ROCm Paged Attention #399

tjtanaa · 2025-02-03T14:37:43Z

Important feature to this branch

Sync with ROCm/vllm Main.
Get the Faster ROCm Paged Attention Kernel (Faster Custom Paged Attention kernels #372)
Obtain a PyTorch Update for Rowwise FP8 GEMM for Per-Token-Activation Per-Channel-Weight Quantization scheme. (Pytorch rowwise scaled_mm #384)

Signed-off-by: Roger Wang <ywang@roblox.com>

Signed-off-by: Rafael Vasquez <rafvasq21@gmail.com>

Signed-off-by: Isotr0py <2037008807@qq.com> Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>

…roject#11100) Signed-off-by: Akshat Tripathi <akshat@krai.ai> Signed-off-by: Oleg Mosalov <oleg@krai.ai> Signed-off-by: Jee Jee Li <pandaleefree@gmail.com> Co-authored-by: Oleg Mosalov <oleg@krai.ai> Co-authored-by: Jee Jee Li <pandaleefree@gmail.com> Co-authored-by: Isotr0py <2037008807@qq.com>

Signed-off-by: rshaw@neuralmagic.com <rshaw@neuralmagic.com>

…m-project#9685) Signed-off-by: Isotr0py <2037008807@qq.com> Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>

…project#11973) Signed-off-by: rshaw@neuralmagic.com <rshaw@neuralmagic.com>

…project#11979)

Signed-off-by: Sungjae Lee <33976427+llsj14@users.noreply.github.com>

Signed-off-by: Yida Wu <yidawu@alumni.cmu.edu>

Signed-off-by: Chenguang Li <757486878@qq.com>

Signed-off-by: youkaichao <youkaichao@gmail.com>

Signed-off-by: Alex-Brooks <Alex.brooks@ibm.com>

Signed-off-by: Chen Zhang <zhangch99@outlook.com>

…ect#11998) Signed-off-by: Roger Wang <ywang@roblox.com> Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>

…llm-project#11935) Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

Signed-off-by: youkaichao <youkaichao@gmail.com>

…ct#11516) Signed-off-by: Shanshan Shen <467638484@qq.com>

vllm-project#11982) Signed-off-by: elijah <f1renze.142857@gmail.com>

…project#12013) Signed-off-by: Yikun <yikunkero@gmail.com>

Signed-off-by: mgoin <michael@neuralmagic.com> Co-authored-by: mgoin <michael@neuralmagic.com>

This reverts commit 889e662.

* Commiting the *multilingual* P3L test. * Created a *multi-lingual* P3L test. * Making ruff happy. * . * Added a reference to the language-scripture Confluence table. * Typo fixing. * Harmonizing naming. * Fixing comments in the header. --------- Co-authored-by: Alexei V. Ivanov <alivanov@banff-cyxtera-s65-4.amd.com> Co-authored-by: Gregory Shtrasberg <156009573+gshtras@users.noreply.github.com>

Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>

…#12014) Signed-off-by: Konrad Zawora <kzawora@habana.ai>

…#12417) Signed-off-by: Tyler Michael Smith <tyler@neuralmagic.com> Co-authored-by: mgoin <michael@neuralmagic.com>

…llm-project#12434) Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>

Signed-off-by: Tyler Michael Smith <tyler@neuralmagic.com>

…#12376) Signed-off-by: Kyle Mistele <kyle@mistele.com>

Signed-off-by: Yuan Tang <terrytangyuan@gmail.com>

Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>

Signed-off-by: Pooya Davoodi <pooya.davoodi@parasail.io>

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

…-project#12464) Signed-off-by: Isotr0py <2037008807@qq.com>

…12454) Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>

Signed-off-by: Isotr0py <2037008807@qq.com>

…t#12339) Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>

Signed-off-by: Mark McLoughlin <markmc@redhat.com>

…12469) Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>

Signed-off-by: Bowen Wang <abmfy@icloud.com> Signed-off-by: youkaichao <youkaichao@gmail.com> Co-authored-by: youkaichao <youkaichao@gmail.com>

* Support FP8 FA from Quark format * Support FP8 FA from Quark format * nit: update comment

Upstream merge 25 01 27

* updating code blocks * typo * updated manifest * Including feedback * whitespace * Deepseek instructions * hyperlink fix * hyperlink fix * updating what is new * cpx update * typo * whitespace * whitespace

* integrate new cpa kernel, update tests and benchmark * added comments to mfma4 kernel * further comments for mfma16 kernel * clang-format * Lint * add flag for logits rtz conversion and disable by default * lint * [Bugfix]: Fix paged attention unit tests of ROCm#372 (ROCm#389) * [Bugfix]: fix paged attention tests based on the updated kernels in `csrc/attention/paged_attention_v1.cu`,`csrc/attention/paged_attention_v2.cu` and `csrc/rocm/attention.cu`. * improve code documentation. * lint --------- Co-authored-by: vllmellm <vllm.ellm@embeddedllm.com> --------- Co-authored-by: Gregory Shtrasberg <156009573+gshtras@users.noreply.github.com> Co-authored-by: Gregory Shtrasberg <Gregory.Shtrasberg@amd.com> Co-authored-by: Joe Shajrawi <17753158+shajrawi@users.noreply.github.com> Co-authored-by: TJian <tunjian1996@gmail.com> Co-authored-by: vllmellm <vllm.ellm@embeddedllm.com>

…s padding (ROCm#394)

Signed-off-by: vllmellm <vllm.ellm@embeddedllm.com>

ywang96 and others added 30 commits January 12, 2025 06:36

[V1] Avoid sending text prompt to core engine (vllm-project#11963)

b25cfab

Signed-off-by: Roger Wang <ywang@roblox.com>

[CI/Build] Add markdown linter (vllm-project#11857)

43f3d9e

Signed-off-by: Rafael Vasquez <rafvasq21@gmail.com>

[Model] Initialize support for Deepseek-VL2 models (vllm-project#11578)

f967e51

Signed-off-by: Isotr0py <2037008807@qq.com> Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>

[Hardware][TPU] workaround fix for MoE on TPU (vllm-project#11764)

263a870

[V1][Core][1/n] Logging and Metrics (vllm-project#11962)

9597a09

Signed-off-by: rshaw@neuralmagic.com <rshaw@neuralmagic.com>

[Model] Support GGUF models newly added in transformers 4.46.0 (vll…

d14e98d

…m-project#9685) Signed-off-by: Isotr0py <2037008807@qq.com> Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>

[V1] [2/n] Logging and Metrics - OutputProcessor Abstraction (vllm-…

619ae26

…project#11973) Signed-off-by: rshaw@neuralmagic.com <rshaw@neuralmagic.com>

[MISC] fix typo in kv transfer send recv test (vllm-project#11983)

f7b3ba8

[Bug] Fix usage of .transpose() and .view() consecutively. (vllm-…

9dd02d8

…project#11979)

[CI][Spec Decode] fix: broken test for EAGLE model (vllm-project#11972)

80ea3af

Signed-off-by: Sungjae Lee <33976427+llsj14@users.noreply.github.com>

[Misc] Fix Deepseek V2 fp8 kv-scale remapping (vllm-project#11947)

cf6bbcb

Signed-off-by: Yida Wu <yidawu@alumni.cmu.edu>

[Misc]Minor Changes about Worker (vllm-project#11555)

c3f05b0

Signed-off-by: Chenguang Li <757486878@qq.com>

[platform] add ray_device_key (vllm-project#11948)

89ce62a

Signed-off-by: youkaichao <youkaichao@gmail.com>

Fix Max Token ID for Qwen-VL-Chat (vllm-project#11980)

5340a30

Signed-off-by: Alex-Brooks <Alex.brooks@ibm.com>

[Kernel] unified_attention for Attention.forward (vllm-project#11967)

0f8cafe

Signed-off-by: Chen Zhang <zhangch99@outlook.com>

[Doc][V1] Update model implementation guide for V1 support (vllm-proj…

cd82499

…ect#11998) Signed-off-by: Roger Wang <ywang@roblox.com> Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>

[Doc] Organise installation documentation into categories and tabs (v…

e8c23ff

…llm-project#11935) Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

[platform] add device_control env var (vllm-project#12009)

458e63a

Signed-off-by: youkaichao <youkaichao@gmail.com>

[Platform] Move get_punica_wrapper() function to Platform (vllm-proje…

a7d5968

…ct#11516) Signed-off-by: Shanshan Shen <467638484@qq.com>

bugfix: Fix signature mismatch in benchmark's get_tokenizer function (

c6db213

vllm-project#11982) Signed-off-by: elijah <f1renze.142857@gmail.com>

[Doc] Fix build from source and installation link in README.md (vllm-…

289b519

…project#12013) Signed-off-by: Yikun <yikunkero@gmail.com>

Merge remote-tracking branch 'upstream/main'

ce53f46

Using list

5a51290

[Bugfix] Fix deepseekv3 gate bias error (vllm-project#12002)

f35ec46

Signed-off-by: mgoin <michael@neuralmagic.com> Co-authored-by: mgoin <michael@neuralmagic.com>

Revert "[misc] improve memory profiling (vllm-project#11809)"

079750e

This reverts commit 889e662.

Trying to make scales work with compileable attention

043c93d

[Docs] Add Sky Computing Lab to project intro (vllm-project#12019)

1a40125

Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>

[HPU][Bugfix] set_forward_context and CI test execution (vllm-project…

078da31

…#12014) Signed-off-by: Konrad Zawora <kzawora@habana.ai>

tlrmchlsmth and others added 28 commits January 26, 2025 19:59

[Bugfix] Disable w16a16 2of4 sparse CompressedTensors24 (vllm-project…

aa2cd2c

…#12417) Signed-off-by: Tyler Michael Smith <tyler@neuralmagic.com> Co-authored-by: mgoin <michael@neuralmagic.com>

[Bugfix/CI] Fix broken kernels/test_mha.py (vllm-project#12450)

72f4880

[Bugfix][Kernel] Fix perf regression caused by PR vllm-project#12405 (v…

68f1114

…llm-project#12434) Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>

[Build/CI] Fix libcuda.so linkage (vllm-project#12424)

72bac73

Signed-off-by: Tyler Michael Smith <tyler@neuralmagic.com>

[Frontend] Rerank API (Jina- and Cohere-compatible API) (vllm-project…

0034b09

…#12376) Signed-off-by: Kyle Mistele <kyle@mistele.com>

[DOC] Add link to vLLM blog (vllm-project#12460)

582cf78

Signed-off-by: Yuan Tang <terrytangyuan@gmail.com>

[V1] Avoid list creation in input preparation (vllm-project#12457)

28e0750

Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>

[Frontend] Support scores endpoint in run_batch (vllm-project#12430)

0cc6b38

Signed-off-by: Pooya Davoodi <pooya.davoodi@parasail.io>

[Bugfix] Fix Granite 3.0 MoE model loading (vllm-project#12446)

5204ff5

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

[Bugfix] Fix missing seq_start_loc in xformers prefill metadata (vllm…

372bf08

…-project#12464) Signed-off-by: Isotr0py <2037008807@qq.com>

[V1][Minor] Minor optimizations for update_from_output (vllm-project#…

624a1e4

…12454) Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>

[Bugfix] Fix gpt2 GGUF inference (vllm-project#12467)

ce69f7f

Signed-off-by: Isotr0py <2037008807@qq.com>

[Build] Only build 9.0a for scaled_mm and sparse kernels (vllm-projec…

103bd17

…t#12339) Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>

[V1][Metrics] Add initial Prometheus logger (vllm-project#12416)

01ba927

Signed-off-by: Mark McLoughlin <markmc@redhat.com>

[V1][CI/Test] Do basic test for top-p & top-k sampling (vllm-project#…

3f1fc74

…12469) Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>

[FlashInfer] Upgrade to 0.2.0 (vllm-project#11194)

2bc3fbb

Signed-off-by: Bowen Wang <abmfy@icloud.com> Signed-off-by: youkaichao <youkaichao@gmail.com> Co-authored-by: youkaichao <youkaichao@gmail.com>

Merge remote-tracking branch 'upstream/main'

8e6d987

Support FP8 FA from Quark format (ROCm#388)

6b2147f

* Support FP8 FA from Quark format * Support FP8 FA from Quark format * nit: update comment

Merge remote-tracking branch 'origin/main' into upstream_merge_25_01_27

a892ecc

Direct call on ROCm

c8b8654

Merge pull request ROCm#391 from ROCm/upstream_merge_25_01_27

b2c3b22

Upstream merge 25 01 27

20250127 docs update (ROCm#392)

7a292f9

* updating code blocks * typo * updated manifest * Including feedback * whitespace * Deepseek instructions * hyperlink fix * hyperlink fix * updating what is new * cpx update * typo * whitespace * whitespace

Using a more precise profiling on ROCm to properly account for weight…

22141e7

…s padding (ROCm#394)

Update Dockerfile.rocm

6852819

Merge remote-tracking branch 'origin/main' into main-to-llama-fp8

6cfbe01

Signed-off-by: vllmellm <vllm.ellm@embeddedllm.com>

[Bugfix]: inclucde the env variables required for running FastSyncLLM

b64246a

Signed-off-by: vllmellm <vllm.ellm@embeddedllm.com>

fix pre-commit lint

64e4aa9

Signed-off-by: vllmellm <vllm.ellm@embeddedllm.com>

hongxiayang approved these changes Feb 3, 2025

View reviewed changes

hongxiayang merged commit 479b843 into ROCm:llama_fp8_12062024 Feb 3, 2025
1 check failed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[MFM-2025-02-03] Merge Main to llama fp8; With Faster ROCm Paged Attention #399

[MFM-2025-02-03] Merge Main to llama fp8; With Faster ROCm Paged Attention #399

tjtanaa commented Feb 3, 2025 •

edited by github-actions bot

Loading

[MFM-2025-02-03] Merge Main to llama fp8; With Faster ROCm Paged Attention #399

[MFM-2025-02-03] Merge Main to llama fp8; With Faster ROCm Paged Attention #399

Conversation

tjtanaa commented Feb 3, 2025 • edited by github-actions bot Loading

tjtanaa commented Feb 3, 2025 •

edited by github-actions bot

Loading