Skip to content
This repository has been archived by the owner on Oct 11, 2024. It is now read-only.

Upstream sync 2024 06 08 #288

Merged
merged 101 commits into from
Jun 10, 2024
Merged

Upstream sync 2024 06 08 #288

merged 101 commits into from
Jun 10, 2024

Commits on Jun 8, 2024

  1. [Kernel] Add marlin_24 unit tests (vllm-project#4901)

    @alexm-neuralmagic @robertgshaw2-neuralmagic
    alexm-neuralmagic authored and robertgshaw2-neuralmagic committed Jun 8, 2024
    Configuration menu
    Copy the full SHA
    e69d23b View commit details
    Browse the repository at this point in the history

  2. [Kernel] Add flash-attn back (vllm-project#4907)

    @WoosukKwon @robertgshaw2-neuralmagic
    WoosukKwon authored and robertgshaw2-neuralmagic committed Jun 8, 2024
    Configuration menu
    Copy the full SHA
    81ec16b View commit details
    Browse the repository at this point in the history

  3. [Model] LLaVA model refactor (vllm-project#4910)

    @DarkLight1337 @robertgshaw2-neuralmagic
    DarkLight1337 authored and robertgshaw2-neuralmagic committed Jun 8, 2024
    Configuration menu
    Copy the full SHA
    5500975 View commit details
    Browse the repository at this point in the history

  4. Remove marlin warning (vllm-project#4918)

    @alexm-neuralmagic @robertgshaw2-neuralmagic
    alexm-neuralmagic authored and robertgshaw2-neuralmagic committed Jun 8, 2024
    Configuration menu
    Copy the full SHA
    b913d04 View commit details
    Browse the repository at this point in the history

  5. [Misc]: allow user to specify port in distributed setting (vllm-proje… 

    …ct#4914)
    @ZwwWayne @robertgshaw2-neuralmagic
    ZwwWayne authored and robertgshaw2-neuralmagic committed Jun 8, 2024
    Configuration menu
    Copy the full SHA
    683a30b View commit details
    Browse the repository at this point in the history

  6. [Build/CI] Enabling AMD Entrypoints Test (vllm-project#4834) 

    Co-authored-by: Alexey Kondratiev <alexey.kondratiev@amd.com>
    @Alexei-V-Ivanov-AMD @alexeykondrat @robertgshaw2-neuralmagic
    2 people authored and robertgshaw2-neuralmagic committed Jun 8, 2024
    Configuration menu
    Copy the full SHA
    c8794c3 View commit details
    Browse the repository at this point in the history

  7. [Bugfix] Fix dummy weight for fp8 (vllm-project#4916) 

    Allow dummy load format for fp8,
torch.uniform_ doesn't support FP8 at the moment

Co-authored-by: Mor Zusman <morz@ai21.com>
    @mzusman @robertgshaw2-neuralmagic
    2 people authored and robertgshaw2-neuralmagic committed Jun 8, 2024
    Configuration menu
    Copy the full SHA
    5b6a7b5 View commit details
    Browse the repository at this point in the history

  8. [Core] Sharded State Loader download from HF (vllm-project#4889)

    @aurickq @robertgshaw2-neuralmagic
    aurickq authored and robertgshaw2-neuralmagic committed Jun 8, 2024
    Configuration menu
    Copy the full SHA
    a5e66c7 View commit details
    Browse the repository at this point in the history

  9. [Doc]Add documentation to benchmarking script when running TGI (vllm-… 

    …project#4920)
    @KuntaiDu @robertgshaw2-neuralmagic
    KuntaiDu authored and robertgshaw2-neuralmagic committed Jun 8, 2024
    Configuration menu
    Copy the full SHA
    8a78ed8 View commit details
    Browse the repository at this point in the history

  10. [Core] Fix scheduler considering "no LoRA" as "LoRA" (vllm-project#4897)

    @Yard1 @robertgshaw2-neuralmagic
    Yard1 authored and robertgshaw2-neuralmagic committed Jun 8, 2024
    Configuration menu
    Copy the full SHA
    6b46dcf View commit details
    Browse the repository at this point in the history

  11. [Model] add rope_scaling support for qwen2 (vllm-project#4930)

    @hzhwcmhf @robertgshaw2-neuralmagic
    hzhwcmhf authored and robertgshaw2-neuralmagic committed Jun 8, 2024
    Configuration menu
    Copy the full SHA
    907d48a View commit details
    Browse the repository at this point in the history

  12. [Model] Add Phi-2 LoRA support (vllm-project#4886)

    @Isotr0py @robertgshaw2-neuralmagic
    Isotr0py authored and robertgshaw2-neuralmagic committed Jun 8, 2024
    Configuration menu
    Copy the full SHA
    11d6f7e View commit details
    Browse the repository at this point in the history

  13. [Docs] Add acknowledgment for sponsors (vllm-project#4925)

    @simon-mo @robertgshaw2-neuralmagic
    simon-mo authored and robertgshaw2-neuralmagic committed Jun 8, 2024
    Configuration menu
    Copy the full SHA
    5d98989 View commit details
    Browse the repository at this point in the history

  14. [CI/Build] Codespell ignore build/ directory (vllm-project#4945)

    @mgoin @robertgshaw2-neuralmagic
    mgoin authored and robertgshaw2-neuralmagic committed Jun 8, 2024
    Configuration menu
    Copy the full SHA
    58a235b View commit details
    Browse the repository at this point in the history

  15. [Bugfix] Fix flag name for max_seq_len_to_capture (vllm-project#4935) 

    Signed-off-by: kerthcet <kerthcet@gmail.com>
    @kerthcet @robertgshaw2-neuralmagic
    kerthcet authored and robertgshaw2-neuralmagic committed Jun 8, 2024
    Configuration menu
    Copy the full SHA
    253d8fb View commit details
    Browse the repository at this point in the history

  16. [Bugfix][Kernel] Add head size check for attention backend selection (v… 

    …llm-project#4944)
    @Isotr0py @robertgshaw2-neuralmagic
    Isotr0py authored and robertgshaw2-neuralmagic committed Jun 8, 2024
    Configuration menu
    Copy the full SHA
    f744125 View commit details
    Browse the repository at this point in the history

  17. [Frontend] Dynamic RoPE scaling (vllm-project#4638)

    @sasha0552 @robertgshaw2-neuralmagic
    sasha0552 authored and robertgshaw2-neuralmagic committed Jun 8, 2024
    Configuration menu
    Copy the full SHA
    c1672a9 View commit details
    Browse the repository at this point in the history

  18. [CI/Build] Enforce style for C++ and CUDA code with clang-format (v… 

    …llm-project#4722)
    @mgoin @robertgshaw2-neuralmagic
    mgoin authored and robertgshaw2-neuralmagic committed Jun 8, 2024
    Configuration menu
    Copy the full SHA
    4b6c961 View commit details
    Browse the repository at this point in the history

  19. [misc] remove comments that were supposed to be removed (vllm-project… 

    …#4977)
    @rkooo567 @robertgshaw2-neuralmagic
    rkooo567 authored and robertgshaw2-neuralmagic committed Jun 8, 2024
    Configuration menu
    Copy the full SHA
    4b74974 View commit details
    Browse the repository at this point in the history

  20. [Kernel] Fixup for CUTLASS kernels in CUDA graphs (vllm-project#4954) 

    Pass the CUDA stream into the CUTLASS GEMMs, to avoid future issues with CUDA graphs
    @tlrmchlsmth @robertgshaw2-neuralmagic
    tlrmchlsmth authored and robertgshaw2-neuralmagic committed Jun 8, 2024
    Configuration menu
    Copy the full SHA
    39c15ee View commit details
    Browse the repository at this point in the history

  21. [Misc] Load FP8 kv-cache scaling factors from checkpoints (vllm-proje… 

    …ct#4893)

The 2nd PR for vllm-project#4532.

This PR supports loading FP8 kv-cache scaling factors from a FP8 checkpoint (with .kv_scale parameter).
    @comaniac @robertgshaw2-neuralmagic
    comaniac authored and robertgshaw2-neuralmagic committed Jun 8, 2024
    Configuration menu
    Copy the full SHA
    2835fc6 View commit details
    Browse the repository at this point in the history

  22. [Model] LoRA gptbigcode implementation (vllm-project#3949)

    @raywanb @robertgshaw2-neuralmagic
    raywanb authored and robertgshaw2-neuralmagic committed Jun 8, 2024
    Configuration menu
    Copy the full SHA
    3db99a6 View commit details
    Browse the repository at this point in the history

  23. [Core] Eliminate parallel worker per-step task scheduling overhead (v… 

    …llm-project#4894)
    @njhill @robertgshaw2-neuralmagic
    njhill authored and robertgshaw2-neuralmagic committed Jun 8, 2024
    Configuration menu
    Copy the full SHA
    39a0a40 View commit details
    Browse the repository at this point in the history

  24. [Minor] Fix small typo in llama.py: QKVParallelLinear -> Quantization… 

    …Config (vllm-project#4991)
    @pcmoritz @robertgshaw2-neuralmagic
    pcmoritz authored and robertgshaw2-neuralmagic committed Jun 8, 2024
    Configuration menu
    Copy the full SHA
    847ca88 View commit details
    Browse the repository at this point in the history

  25. [Misc] Take user preference in attention selector (vllm-project#4960)

    @comaniac @robertgshaw2-neuralmagic
    comaniac authored and robertgshaw2-neuralmagic committed Jun 8, 2024
    Configuration menu
    Copy the full SHA
    c60384c View commit details
    Browse the repository at this point in the history

  26. Marlin 24 prefill performance improvement (about 25% better on averag… 

    …e) (vllm-project#4983)
    @alexm-neuralmagic @robertgshaw2-neuralmagic
    alexm-neuralmagic authored and robertgshaw2-neuralmagic committed Jun 8, 2024
    Configuration menu
    Copy the full SHA
    dae5aaf View commit details
    Browse the repository at this point in the history

  27. [Bugfix] Update Dockerfile.cpu to fix NameError: name 'vllm_ops' is n… 

    …ot defined (vllm-project#5009)
    @LetianLee @robertgshaw2-neuralmagic
    LetianLee authored and robertgshaw2-neuralmagic committed Jun 8, 2024
    Configuration menu
    Copy the full SHA
    05a4f64 View commit details
    Browse the repository at this point in the history

  28. [Core][1/N] Support send/recv in PyNCCL Groups (vllm-project#4988) 

    Signed-off-by: Muralidhar Andoorveedu <muralidhar.andoorveedu@centml.ai>
    @andoorve @robertgshaw2-neuralmagic
    andoorve authored and robertgshaw2-neuralmagic committed Jun 8, 2024
    Configuration menu
    Copy the full SHA
    bf4c411 View commit details
    Browse the repository at this point in the history

  29. [Kernel] Initial Activation Quantization Support (vllm-project#4525) 

    Co-authored-by: Varun Sundar Rabindranath <varunsundar08@gmail.com>
Co-authored-by: Varun Sundar Rabindranath <varun@neuralmagic.com>
    @dsikka @varun-sundar-rabindranath @robertgshaw2-neuralmagic
    3 people authored and robertgshaw2-neuralmagic committed Jun 8, 2024
    Configuration menu
    Copy the full SHA
    c623663 View commit details
    Browse the repository at this point in the history

  30. [Core]: Option To Use Prompt Token Ids Inside Logits Processor (vllm-… 

    …project#4985)

Co-authored-by: Elisei Smirnov <el.smirnov@innopolis.university>
    @kezouke @robertgshaw2-neuralmagic
    2 people authored and robertgshaw2-neuralmagic committed Jun 8, 2024
    Configuration menu
    Copy the full SHA
    a9ca32d View commit details
    Browse the repository at this point in the history

  31. [Doc] add ccache guide in doc (vllm-project#5012) 

    Co-authored-by: Michael Goin <michael@neuralmagic.com>
    @youkaichao @mgoin @robertgshaw2-neuralmagic
    2 people authored and robertgshaw2-neuralmagic committed Jun 8, 2024
    Configuration menu
    Copy the full SHA
    0eb33b1 View commit details
    Browse the repository at this point in the history

  32. [Kernel] Initial Activation Quantization Support (vllm-project#4525) 

    Co-authored-by: Varun Sundar Rabindranath <varunsundar08@gmail.com>
Co-authored-by: Varun Sundar Rabindranath <varun@neuralmagic.com>
    @robertgshaw2-neuralmagic @varun-sundar-rabindranath
    3 people committed Jun 8, 2024
    Configuration menu
    Copy the full SHA
    acf362c View commit details
    Browse the repository at this point in the history

  33. [Core][Bugfix]: fix prefix caching for blockv2 (vllm-project#4764) 

    Co-authored-by: Lei Wen <wenlei03@qiyi.com>
    @leiwen83 @wenlei03 @robertgshaw2-neuralmagic
    2 people authored and robertgshaw2-neuralmagic committed Jun 8, 2024
    Configuration menu
    Copy the full SHA
    1226d5d View commit details
    Browse the repository at this point in the history

  34. [Kernel][Backend][Model] Blocksparse flash attention kernel and Phi-3… 

    …-Small model (vllm-project#4799)

Co-authored-by: beagleski <yunanzhang@microsoft.com>
Co-authored-by: bapatra <bapatra@microsoft.com>
Co-authored-by: Barun Patra <codedecde@users.noreply.github.com>
Co-authored-by: Michael Goin <michael@neuralmagic.com>
    @linxihui @beagleski
    @codedecde @mgoin @robertgshaw2-neuralmagic
    5 people authored and robertgshaw2-neuralmagic committed Jun 8, 2024
    Configuration menu
    Copy the full SHA
    29a2098 View commit details
    Browse the repository at this point in the history

  35. [Misc] add logging level env var (vllm-project#5045)

    @youkaichao @robertgshaw2-neuralmagic
    youkaichao authored and robertgshaw2-neuralmagic committed Jun 8, 2024
    Configuration menu
    Copy the full SHA
    3fe7e52 View commit details
    Browse the repository at this point in the history

  36. [Dynamic Spec Decoding] Minor fix for disabling speculative decoding (v… 

    …llm-project#5000)
    @LiuXiaoxuanPKU @robertgshaw2-neuralmagic
    LiuXiaoxuanPKU authored and robertgshaw2-neuralmagic committed Jun 8, 2024
    Configuration menu
    Copy the full SHA
    8768b3f View commit details
    Browse the repository at this point in the history

  37. [Misc] Make Serving Benchmark More User-friendly (vllm-project#5044)

    @ywang96 @robertgshaw2-neuralmagic
    ywang96 authored and robertgshaw2-neuralmagic committed Jun 8, 2024
    Configuration menu
    Copy the full SHA
    e7e376f View commit details
    Browse the repository at this point in the history

  38. [Bugfix / Core] Prefix Caching Guards (merged with main) (vllm-projec… 

    …t#4846)

Co-authored-by: rsnm2 <rshaw@neuralmagic.com>
Co-authored-by: Robert Shaw <114415538+robertgshaw2-neuralmagic@users.noreply.github.com>
    @zhuohan123 @robertgshaw2-neuralmagic
    3 people committed Jun 8, 2024
    Configuration menu
    Copy the full SHA
    67ce9ea View commit details
    Browse the repository at this point in the history

  39. [Core] Allow AQLM on Pascal (vllm-project#5058)

    @sasha0552 @robertgshaw2-neuralmagic
    sasha0552 authored and robertgshaw2-neuralmagic committed Jun 8, 2024
    Configuration menu
    Copy the full SHA
    2c59c91 View commit details
    Browse the repository at this point in the history

  40. [Model] Add support for falcon-11B (vllm-project#5069)

    @Isotr0py @robertgshaw2-neuralmagic
    Isotr0py authored and robertgshaw2-neuralmagic committed Jun 8, 2024
    Configuration menu
    Copy the full SHA
    9fb7b82 View commit details
    Browse the repository at this point in the history

  41. [Core] Sliding window for block manager v2 (vllm-project#4545) 

    Co-authored-by: Ruth Evans <ruthevans@Ruths-MacBook-Pro.local>
    @mmoskal @robertgshaw2-neuralmagic
    2 people authored and robertgshaw2-neuralmagic committed Jun 8, 2024
    Configuration menu
    Copy the full SHA
    954c332 View commit details
    Browse the repository at this point in the history

  42. [BugFix] Fix Embedding Models with TP>1 (vllm-project#5075)

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Jun 8, 2024
    Configuration menu
    Copy the full SHA
    9929fb2 View commit details
    Browse the repository at this point in the history

  43. [Kernel][ROCm][AMD] Add fused_moe Triton configs for MI300X (vllm-pro… 

    …ject#4951)

This PR adds Triton kernel configs for the MoE kernel for MI300X
    @divakar-amd @robertgshaw2-neuralmagic
    divakar-amd authored and robertgshaw2-neuralmagic committed Jun 8, 2024
    Configuration menu
    Copy the full SHA
    b22d985 View commit details
    Browse the repository at this point in the history

  44. [Docs] Add Dropbox as sponsors (vllm-project#5089)

    @simon-mo @robertgshaw2-neuralmagic
    simon-mo authored and robertgshaw2-neuralmagic committed Jun 8, 2024
    Configuration menu
    Copy the full SHA
    54c17a9 View commit details
    Browse the repository at this point in the history

  45. [Core] Consolidate prompt arguments to LLM engines (vllm-project#4328) 

    Co-authored-by: Roger Wang <ywang@roblox.com>
    @DarkLight1337 @ywang96 @robertgshaw2-neuralmagic
    2 people authored and robertgshaw2-neuralmagic committed Jun 8, 2024
    Configuration menu
    Copy the full SHA
    8c9aab4 View commit details
    Browse the repository at this point in the history

  46. [Bugfix] Remove the last EOS token unless explicitly specified (vllm-… 

    …project#5077)
    @jsato8094 @robertgshaw2-neuralmagic
    jsato8094 authored and robertgshaw2-neuralmagic committed Jun 8, 2024
    Configuration menu
    Copy the full SHA
    705789d View commit details
    Browse the repository at this point in the history

  47. [Misc] add gpu_memory_utilization arg (vllm-project#5079) 

    Signed-off-by: pandyamarut <pandyamarut@gmail.com>
    @pandyamarut @robertgshaw2-neuralmagic
    pandyamarut authored and robertgshaw2-neuralmagic committed Jun 8, 2024
    Configuration menu
    Copy the full SHA
    95c2a3d View commit details
    Browse the repository at this point in the history

  48. [Core][Optimization] remove vllm-nccl (vllm-project#5091)

    @youkaichao @robertgshaw2-neuralmagic
    youkaichao authored and robertgshaw2-neuralmagic committed Jun 8, 2024
    Configuration menu
    Copy the full SHA
    9175890 View commit details
    Browse the repository at this point in the history

  49. [Bugfix] Fix arguments passed to Sequence in stop checker test (vll… 

    …m-project#5092)
    @DarkLight1337 @robertgshaw2-neuralmagic
    DarkLight1337 authored and robertgshaw2-neuralmagic committed Jun 8, 2024
    Configuration menu
    Copy the full SHA
    420c4ff View commit details
    Browse the repository at this point in the history

  50. [Core][Distributed] improve p2p access check (vllm-project#4992)

    @youkaichao @robertgshaw2-neuralmagic
    youkaichao authored and robertgshaw2-neuralmagic committed Jun 8, 2024
    Configuration menu
    Copy the full SHA
    5bde5ba View commit details
    Browse the repository at this point in the history

  51. [Core] Cross-attention KV caching and memory-management (towards even… 

    …tual encoder/decoder model support) (vllm-project#4837)
    @afeldman-nm @robertgshaw2-neuralmagic
    afeldman-nm authored and robertgshaw2-neuralmagic committed Jun 8, 2024
    Configuration menu
    Copy the full SHA
    b86aa89 View commit details
    Browse the repository at this point in the history

  52. [Doc]Replace deprecated flag in readme (vllm-project#4526)

    @ronensc @robertgshaw2-neuralmagic
    ronensc authored and robertgshaw2-neuralmagic committed Jun 8, 2024
    Configuration menu
    Copy the full SHA
    f63e8dd View commit details
    Browse the repository at this point in the history

  53. [Bugfix][CI/Build] Fix test and improve code for `merge_async_iterato… 

    …rs` (vllm-project#5096)
    @DarkLight1337 @robertgshaw2-neuralmagic
    DarkLight1337 authored and robertgshaw2-neuralmagic committed Jun 8, 2024
    Configuration menu
    Copy the full SHA
    62a4fcb View commit details
    Browse the repository at this point in the history

  54. [Bugfix][CI/Build] Fix codespell failing to skip files in git diff (v… 

    …llm-project#5097)
    @DarkLight1337 @robertgshaw2-neuralmagic
    DarkLight1337 authored and robertgshaw2-neuralmagic committed Jun 8, 2024
    Configuration menu
    Copy the full SHA
    f900bcc View commit details
    Browse the repository at this point in the history

  55. [Core] Avoid the need to pass None values to Sequence.inputs (vll… 

    …m-project#5099)
    @DarkLight1337 @robertgshaw2-neuralmagic
    DarkLight1337 authored and robertgshaw2-neuralmagic committed Jun 8, 2024
    Configuration menu
    Copy the full SHA
    6824b2f View commit details
    Browse the repository at this point in the history

  56. [Bugfix] logprobs is not compatible with the OpenAI spec vllm-project… 

    …#4795 (vllm-project#5031)
    @Etelis @robertgshaw2-neuralmagic
    Etelis authored and robertgshaw2-neuralmagic committed Jun 8, 2024
    Configuration menu
    Copy the full SHA
    623275f View commit details
    Browse the repository at this point in the history

  57. [Bugfix / Core] Prefix Caching Guards (merged with main) (vllm-projec… 

    …t#4846)

Co-authored-by: rsnm2 <rshaw@neuralmagic.com>
Co-authored-by: Robert Shaw <114415538+robertgshaw2-neuralmagic@users.noreply.github.com>
    @youkaichao @robertgshaw2-neuralmagic
    3 people committed Jun 8, 2024
    Configuration menu
    Copy the full SHA
    15dcd3e View commit details
    Browse the repository at this point in the history

  58. [Bugfix] gptq_marlin: Ensure g_idx_sort_indices is not a Parameter (v… 

    …llm-project#5108)
    @alexm-neuralmagic @robertgshaw2-neuralmagic
    alexm-neuralmagic authored and robertgshaw2-neuralmagic committed Jun 8, 2024
    Configuration menu
    Copy the full SHA
    5763c73 View commit details
    Browse the repository at this point in the history

  59. [CI/Build] Docker cleanup functionality for amd servers (vllm-project… 

    …#5112)

Co-authored-by: Alexey Kondratiev <alexey.kondratiev@amd.com>
Co-authored-by: Alexei-V-Ivanov-AMD <156011006+Alexei-V-Ivanov-AMD@users.noreply.github.com>
Co-authored-by: Alexei V. Ivanov <alexei.ivanov@amd.com>
Co-authored-by: omkarkakarparthi <okakarpa>
    @okakarpa @alexeykondrat
    @Alexei-V-Ivanov-AMD @robertgshaw2-neuralmagic
    4 people authored and robertgshaw2-neuralmagic committed Jun 8, 2024
    Configuration menu
    Copy the full SHA
    3a8332c View commit details
    Browse the repository at this point in the history

  60. [BUGFIX] [FRONTEND] Correct chat logprobs (vllm-project#5029) 

    Co-authored-by: Breno Faria <breno.faria@intrafind.com>
    @br3no @robertgshaw2-neuralmagic
    2 people authored and robertgshaw2-neuralmagic committed Jun 8, 2024
    Configuration menu
    Copy the full SHA
    11a5a26 View commit details
    Browse the repository at this point in the history

  61. [Bugfix] Automatically Detect SparseML models (vllm-project#5119)

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Jun 8, 2024
    Configuration menu
    Copy the full SHA
    2827c68 View commit details
    Browse the repository at this point in the history

  62. [CI/Build] increase wheel size limit to 200 MB (vllm-project#5130)

    @youkaichao @robertgshaw2-neuralmagic
    youkaichao authored and robertgshaw2-neuralmagic committed Jun 8, 2024
    Configuration menu
    Copy the full SHA
    4ae80dd View commit details
    Browse the repository at this point in the history

  63. [Misc] remove duplicate definition of seq_lens_tensor in model_runn… 

    …er.py (vllm-project#5129)
    @ita9naiwa @robertgshaw2-neuralmagic
    ita9naiwa authored and robertgshaw2-neuralmagic committed Jun 8, 2024
    Configuration menu
    Copy the full SHA
    886ead6 View commit details
    Browse the repository at this point in the history

  64. [Doc] Use intersphinx and update entrypoints docs (vllm-project#5125)

    @DarkLight1337 @robertgshaw2-neuralmagic
    DarkLight1337 authored and robertgshaw2-neuralmagic committed Jun 8, 2024
    Configuration menu
    Copy the full SHA
    758b903 View commit details
    Browse the repository at this point in the history

  65. add doc about serving option on dstack (vllm-project#3074) 

    Co-authored-by: Roger Wang <ywang@roblox.com>
    @deep-diver @ywang96 @robertgshaw2-neuralmagic
    2 people authored and robertgshaw2-neuralmagic committed Jun 8, 2024
    Configuration menu
    Copy the full SHA
    a190463 View commit details
    Browse the repository at this point in the history

  66. Bump version to v0.4.3 (vllm-project#5046)

    @simon-mo @robertgshaw2-neuralmagic
    simon-mo authored and robertgshaw2-neuralmagic committed Jun 8, 2024
    Configuration menu
    Copy the full SHA
    51cf757 View commit details
    Browse the repository at this point in the history

  67. [Build] Disable sm_90a in cu11 (vllm-project#5141)

    @simon-mo @robertgshaw2-neuralmagic
    simon-mo authored and robertgshaw2-neuralmagic committed Jun 8, 2024
    Configuration menu
    Copy the full SHA
    c72d890 View commit details
    Browse the repository at this point in the history

  68. [Bugfix] Avoid Warnings in SparseML Activation Quantization (vllm-pro… 

    …ject#5120)
    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Jun 8, 2024
    Configuration menu
    Copy the full SHA
    cf0711b View commit details
    Browse the repository at this point in the history

  69. [Kernel] Marlin_24: Ensure the mma.sp instruction is using the ::orde… 

    …red_metadata modifier (introduced with PTX 8.5) (vllm-project#5136)
    @alexm-neuralmagic @robertgshaw2-neuralmagic
    alexm-neuralmagic authored and robertgshaw2-neuralmagic committed Jun 8, 2024
    Configuration menu
    Copy the full SHA
    dcaf819 View commit details
    Browse the repository at this point in the history

  70. Fix cutlass sm_90a vesrion in CMakeList

    @simon-mo @robertgshaw2-neuralmagic
    simon-mo authored and robertgshaw2-neuralmagic committed Jun 8, 2024
    Configuration menu
    Copy the full SHA
    7da3c3f View commit details
    Browse the repository at this point in the history

  71. [Model] Support MAP-NEO model (vllm-project#5081) 

    Co-authored-by: Zhuohan Li <zhuohan123@gmail.com>
    @xingweiqu @zhuohan123 @robertgshaw2-neuralmagic
    2 people authored and robertgshaw2-neuralmagic committed Jun 8, 2024
    Configuration menu
    Copy the full SHA
    2c66f17 View commit details
    Browse the repository at this point in the history

  72. Revert "[Kernel] Marlin_24: Ensure the mma.sp instruction is using th… 

    …e ::ordered_metadata modifier (introduced with PTX 8.5)" (vllm-project#5149)
    @simon-mo @robertgshaw2-neuralmagic
    simon-mo authored and robertgshaw2-neuralmagic committed Jun 8, 2024
    Configuration menu
    Copy the full SHA
    5388c64 View commit details
    Browse the repository at this point in the history

  73. [Misc]: optimize eager mode host time (vllm-project#4196) 

    Co-authored-by: xuhao <xuhao@cambricon.com>
    @FuncSherl @robertgshaw2-neuralmagic
    2 people authored and robertgshaw2-neuralmagic committed Jun 8, 2024
    Configuration menu
    Copy the full SHA
    5e9f300 View commit details
    Browse the repository at this point in the history

  74. [Model] Enable FP8 QKV in MoE and refine kernel tuning script (vllm-p… 

    …roject#5039)
    @comaniac @robertgshaw2-neuralmagic
    comaniac authored and robertgshaw2-neuralmagic committed Jun 8, 2024
    Configuration menu
    Copy the full SHA
    f329e2e View commit details
    Browse the repository at this point in the history

  75. [Doc] Add checkmark for GPTBigCodeForCausalLM LoRA support (vllm-proj… 

    …ect#5171)
    @njhill @robertgshaw2-neuralmagic
    njhill authored and robertgshaw2-neuralmagic committed Jun 8, 2024
    Configuration menu
    Copy the full SHA
    951e3d2 View commit details
    Browse the repository at this point in the history

  76. [Build] Guard against older CUDA versions when building CUTLASS 3.x k… 

    …ernels (vllm-project#5168)
    @tlrmchlsmth @robertgshaw2-neuralmagic
    tlrmchlsmth authored and robertgshaw2-neuralmagic committed Jun 8, 2024
    Configuration menu
    Copy the full SHA
    d349dbd View commit details
    Browse the repository at this point in the history

  77. format

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Jun 8, 2024
    Configuration menu
    Copy the full SHA
    031fd4e View commit details
    Browse the repository at this point in the history

Commits on Jun 9, 2024

  1. skip blockspase attention

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Jun 9, 2024
    Configuration menu
    Copy the full SHA
    9ed5f76 View commit details
    Browse the repository at this point in the history

  2. fix falcon

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Jun 9, 2024
    Configuration menu
    Copy the full SHA
    ec71544 View commit details
    Browse the repository at this point in the history

  3. skip sliding window chunked prefill

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Jun 9, 2024
    Configuration menu
    Copy the full SHA
    7381340 View commit details
    Browse the repository at this point in the history

  4. skip prefix prefill

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Jun 9, 2024
    Configuration menu
    Copy the full SHA
    c23ca05 View commit details
    Browse the repository at this point in the history

  5. skip tensorizer

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Jun 9, 2024
    Configuration menu
    Copy the full SHA
    85512eb View commit details
    Browse the repository at this point in the history

  6. [Misc][Breaking] Change FP8 checkpoint format from act_scale -> input… 

    …_scale (vllm-project#5353)
    @mgoin @robertgshaw2-neuralmagic
    mgoin authored and robertgshaw2-neuralmagic committed Jun 9, 2024
    Configuration menu
    Copy the full SHA
    0cea2c2 View commit details
    Browse the repository at this point in the history

  7. format

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Jun 9, 2024
    Configuration menu
    Copy the full SHA
    31147df View commit details
    Browse the repository at this point in the history

  8. fix issue with internal method

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Jun 9, 2024
    Configuration menu
    Copy the full SHA
    2256610 View commit details
    Browse the repository at this point in the history

  9. formatting

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Jun 9, 2024
    Configuration menu
    Copy the full SHA
    01973f5 View commit details
    Browse the repository at this point in the history

  10. disabled more kernel tests that use triton

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Jun 9, 2024
    Configuration menu
    Copy the full SHA
    a1a659d View commit details
    Browse the repository at this point in the history

  11. updated cutlass skipping. We need cuda 12.4 in automation

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Jun 9, 2024
    Configuration menu
    Copy the full SHA
    c50784c View commit details
    Browse the repository at this point in the history

  12. trigger kernel tests in automation

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Jun 9, 2024
    Configuration menu
    Copy the full SHA
    99fa9f8 View commit details
    Browse the repository at this point in the history

  13. cleanup spurious setup.py change

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Jun 9, 2024
    Configuration menu
    Copy the full SHA
    2ec6643 View commit details
    Browse the repository at this point in the history

  14. readded the missing images

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Jun 9, 2024
    Configuration menu
    Copy the full SHA
    0bb099c View commit details
    Browse the repository at this point in the history

  15. multilora inference

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Jun 9, 2024
    Configuration menu
    Copy the full SHA
    198f364 View commit details
    Browse the repository at this point in the history

  16. offline inference with prefix

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Jun 9, 2024
    Configuration menu
    Copy the full SHA
    ec0e89a View commit details
    Browse the repository at this point in the history

  17. backend request func

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Jun 9, 2024
    Configuration menu
    Copy the full SHA
    e6f1cbd View commit details
    Browse the repository at this point in the history

  18. benchmark serving

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Jun 9, 2024
    Configuration menu
    Copy the full SHA
    ca8d74a View commit details
    Browse the repository at this point in the history

  19. prod monitoring readme

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Jun 9, 2024
    Configuration menu
    Copy the full SHA
    5335ad9 View commit details
    Browse the repository at this point in the history

  20. format

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Jun 9, 2024
    Configuration menu
    Copy the full SHA
    611cfed View commit details
    Browse the repository at this point in the history

  21. fix benchmark issue - internal method changed

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Jun 9, 2024
    Configuration menu
    Copy the full SHA
    73132a5 View commit details
    Browse the repository at this point in the history

  22. removed skip for remote push edits

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Jun 9, 2024
    Configuration menu
    Copy the full SHA
    7f5c715 View commit details
    Browse the repository at this point in the history

Commits on Jun 10, 2024

  1. update internal method in benchmark throughput too

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Jun 10, 2024
    4 Configuration menu
    Copy the full SHA
    437912e View commit details
    Browse the repository at this point in the history

  2. skip triton sampler tests

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Jun 10, 2024
    4 Configuration menu
    Copy the full SHA
    950981c View commit details
    Browse the repository at this point in the history