Commits on Jun 23, 2024

1. [CI/Build][Misc] Add CI that benchmarks vllm performance on those PRs… …

   … with `perf-benchmarks` label (vllm-project#5073)
   
   Co-authored-by: simon-mo <simon.mo@hey.com>

   

   2 people authored and robertgshaw2-neuralmagic committed Jun 23, 2024
   Configuration menu

   • View commit details
   • Copy full SHA for 5d52fa5
   • Browse repository at this point

   Copy the full SHA

   5d52fa5 View commit details

   Browse the repository at this point in the history

2. [CI/Build] Disable LLaVA-NeXT CPU test (vllm-project#5529)

   

   DarkLight1337 authored and robertgshaw2-neuralmagic committed Jun 23, 2024
   Configuration menu

   • View commit details
   • Copy full SHA for cab4a5d
   • Browse repository at this point

   Copy the full SHA

   cab4a5d View commit details

   Browse the repository at this point in the history

3. [Kernel] Fix CUTLASS 3.x custom broadcast load epilogue (vllm-project… …

   …#5516)

   

   tlrmchlsmth authored and robertgshaw2-neuralmagic committed Jun 23, 2024
   Configuration menu

   • View commit details
   • Copy full SHA for 923d05a
   • Browse repository at this point

   Copy the full SHA

   923d05a View commit details

   Browse the repository at this point in the history

4. [Misc] Fix arg names (vllm-project#5524)

   

   AllenDou authored and robertgshaw2-neuralmagic committed Jun 23, 2024
   Configuration menu

   • View commit details
   • Copy full SHA for 34467ee
   • Browse repository at this point

   Copy the full SHA

   34467ee View commit details

   Browse the repository at this point in the history

5. [ Misc ] Rs/compressed tensors cleanup (vllm-project#5432) …

   Co-authored-by: mgoin <michael@neuralmagic.com>
   Co-authored-by: Dipika Sikka <dipikasikka1@gmail.com>

   

   3 people committed Jun 23, 2024
   Configuration menu

   • View commit details
   • Copy full SHA for deee747
   • Browse repository at this point

   Copy the full SHA

   deee747 View commit details

   Browse the repository at this point in the history

6. [Kernel] Suppress mma.sp warning on CUDA 12.5 and later (vllm-project… …

   …#5401)

   

   tlrmchlsmth authored and robertgshaw2-neuralmagic committed Jun 23, 2024
   Configuration menu

   • View commit details
   • Copy full SHA for 0ccb117
   • Browse repository at this point

   Copy the full SHA

   0ccb117 View commit details

   Browse the repository at this point in the history

7. [mis] fix flaky test of test_cuda_device_count_stateless (vllm-projec… …

   …t#5546)

   

   youkaichao authored and robertgshaw2-neuralmagic committed Jun 23, 2024
   Configuration menu

   • View commit details
   • Copy full SHA for 4464401
   • Browse repository at this point

   Copy the full SHA

   4464401 View commit details

   Browse the repository at this point in the history

8. [Core] Remove duplicate processing in async engine (vllm-project#5525)

   

   DarkLight1337 authored and robertgshaw2-neuralmagic committed Jun 23, 2024
   Configuration menu

   • View commit details
   • Copy full SHA for 28d0d6d
   • Browse repository at this point

   Copy the full SHA

   28d0d6d View commit details

   Browse the repository at this point in the history

9. [misc][distributed] fix benign error in is_in_the_same_node (vllm-p… …

   …roject#5512)

   

   youkaichao authored and robertgshaw2-neuralmagic committed Jun 23, 2024
   Configuration menu

   • View commit details
   • Copy full SHA for f0e02ac
   • Browse repository at this point

   Copy the full SHA

   f0e02ac View commit details

   Browse the repository at this point in the history

10. [Docs] Add ZhenFund as a Sponsor (vllm-project#5548)

    

    simon-mo authored and robertgshaw2-neuralmagic committed Jun 23, 2024
    Configuration menu

    • View commit details
    • Copy full SHA for d0a3026
    • Browse repository at this point

    Copy the full SHA

    d0a3026 View commit details

    Browse the repository at this point in the history

11. [Doc] Update documentation on Tensorizer (vllm-project#5471)

    

    sangstar authored and robertgshaw2-neuralmagic committed Jun 23, 2024
    Configuration menu

    • View commit details
    • Copy full SHA for 33edc9b
    • Browse repository at this point

    Copy the full SHA

    33edc9b View commit details

    Browse the repository at this point in the history

12. [Bugfix] Enable loading FP8 checkpoints for gpt_bigcode models (vllm-… …

    …project#5460)
    
    Signed-off-by: Thomas Parnell <tpa@zurich.ibm.com>

    

    tdoublep authored and robertgshaw2-neuralmagic committed Jun 23, 2024
    Configuration menu

    • View commit details
    • Copy full SHA for 5fffeb8
    • Browse repository at this point

    Copy the full SHA

    5fffeb8 View commit details

    Browse the repository at this point in the history

13. [Bugfix] Fix typo in Pallas backend (vllm-project#5558)

    

    WoosukKwon authored and robertgshaw2-neuralmagic committed Jun 23, 2024
    Configuration menu

    • View commit details
    • Copy full SHA for 65419f4
    • Browse repository at this point

    Copy the full SHA

    65419f4 View commit details

    Browse the repository at this point in the history

14. [Core][Distributed] improve p2p cache generation (vllm-project#5528)

    

    youkaichao authored and robertgshaw2-neuralmagic committed Jun 23, 2024
    Configuration menu

    • View commit details
    • Copy full SHA for dfd2b2e
    • Browse repository at this point

    Copy the full SHA

    dfd2b2e View commit details

    Browse the repository at this point in the history

15. Add ccache to amd (vllm-project#5555)

    

    simon-mo authored and robertgshaw2-neuralmagic committed Jun 23, 2024
    Configuration menu

    • View commit details
    • Copy full SHA for d464106
    • Browse repository at this point

    Copy the full SHA

    d464106 View commit details

    Browse the repository at this point in the history

16. [Core][Bugfix]: fix prefix caching for blockv2 (vllm-project#5364) …

    Signed-off-by: Lei Wen <wenlei03@qiyi.com>
    Co-authored-by: Lei Wen <wenlei03@qiyi.com>

    

    2 people authored and robertgshaw2-neuralmagic committed Jun 23, 2024
    Configuration menu

    • View commit details
    • Copy full SHA for 80b908f
    • Browse repository at this point

    Copy the full SHA

    80b908f View commit details

    Browse the repository at this point in the history

17. [mypy] Enable type checking for test directory (vllm-project#5017)

    

    DarkLight1337 authored and robertgshaw2-neuralmagic committed Jun 23, 2024
    Configuration menu

    • View commit details
    • Copy full SHA for 0393d45
    • Browse repository at this point

    Copy the full SHA

    0393d45 View commit details

    Browse the repository at this point in the history

18. [CI/Build] Test both text and token IDs in batched OpenAI Completions… …

    … API (vllm-project#5568)

    

    DarkLight1337 authored and robertgshaw2-neuralmagic committed Jun 23, 2024
    Configuration menu

    • View commit details
    • Copy full SHA for 32d5ecc
    • Browse repository at this point

    Copy the full SHA

    32d5ecc View commit details

    Browse the repository at this point in the history

19. [misc] Do not allow to use lora with chunked prefill. (vllm-project#5538 …

    )
    
    Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>

    

    2 people authored and robertgshaw2-neuralmagic committed Jun 23, 2024
    Configuration menu

    • View commit details
    • Copy full SHA for 6f3169a
    • Browse repository at this point

    Copy the full SHA

    6f3169a View commit details

    Browse the repository at this point in the history

20. add gptq_marlin test for bug report vllm-project#5088 (vllm-project#5145 …

    )

    

    alexm-neuralmagic authored and robertgshaw2-neuralmagic committed Jun 23, 2024
    Configuration menu

    • View commit details
    • Copy full SHA for beb3b21
    • Browse repository at this point

    Copy the full SHA

    beb3b21 View commit details

    Browse the repository at this point in the history

21. [BugFix] Don't start a Ray cluster when not using Ray (vllm-project#5570 …

    )

    

    njhill authored and robertgshaw2-neuralmagic committed Jun 23, 2024
    Configuration menu

    • View commit details
    • Copy full SHA for 31f38f3
    • Browse repository at this point

    Copy the full SHA

    31f38f3 View commit details

    Browse the repository at this point in the history

22. [Fix] Correct OpenAI batch response format (vllm-project#5554)

    

    zifeitong authored and robertgshaw2-neuralmagic committed Jun 23, 2024
    Configuration menu

    • View commit details
    • Copy full SHA for ec68cd1
    • Browse repository at this point

    Copy the full SHA

    ec68cd1 View commit details

    Browse the repository at this point in the history

23. Add basic correctness 2 GPU tests to 4 GPU pipeline (vllm-project#5518)

    

    Yard1 authored and robertgshaw2-neuralmagic committed Jun 23, 2024
    Configuration menu

    • View commit details
    • Copy full SHA for dc8789d
    • Browse repository at this point

    Copy the full SHA

    dc8789d View commit details

    Browse the repository at this point in the history

24. [CI][BugFix] Flip is_quant_method_supported condition (vllm-project#5577 …

    )

    

    mgoin authored and robertgshaw2-neuralmagic committed Jun 23, 2024
    Configuration menu

    • View commit details
    • Copy full SHA for 681de21
    • Browse repository at this point

    Copy the full SHA

    681de21 View commit details

    Browse the repository at this point in the history

25. [build][misc] limit numpy version (vllm-project#5582)

    

    youkaichao authored and robertgshaw2-neuralmagic committed Jun 23, 2024
    Configuration menu

    • View commit details
    • Copy full SHA for 77a5f36
    • Browse repository at this point

    Copy the full SHA

    77a5f36 View commit details

    Browse the repository at this point in the history

26. [Doc] add debugging tips for crash and multi-node debugging (vllm-pro… …

    …ject#5581)

    

    youkaichao authored and robertgshaw2-neuralmagic committed Jun 23, 2024
    Configuration menu

    • View commit details
    • Copy full SHA for 9c77244
    • Browse repository at this point

    Copy the full SHA

    9c77244 View commit details

    Browse the repository at this point in the history

27. Fix w8a8 benchmark and add Llama-3-8B (vllm-project#5562)

    

    comaniac authored and robertgshaw2-neuralmagic committed Jun 23, 2024
    Configuration menu

    • View commit details
    • Copy full SHA for f968328
    • Browse repository at this point

    Copy the full SHA

    f968328 View commit details

    Browse the repository at this point in the history

28. [Model] Rename Phi3 rope scaling type (vllm-project#5595)

    

    garg-amit authored and robertgshaw2-neuralmagic committed Jun 23, 2024
    Configuration menu

    • View commit details
    • Copy full SHA for b0abad9
    • Browse repository at this point

    Copy the full SHA

    b0abad9 View commit details

    Browse the repository at this point in the history

29. Correct alignment in the seq_len diagram. (vllm-project#5592) …

    Co-authored-by: Liqian Chen <liqian.chen@deeplang.ai>

    

    2 people authored and robertgshaw2-neuralmagic committed Jun 23, 2024
    Configuration menu

    • View commit details
    • Copy full SHA for 4b84959
    • Browse repository at this point

    Copy the full SHA

    4b84959 View commit details

    Browse the repository at this point in the history

30. [Kernel] compressed-tensors marlin 24 support (vllm-project#5435)

    

    dsikka authored and robertgshaw2-neuralmagic committed Jun 23, 2024
    Configuration menu

    • View commit details
    • Copy full SHA for 9cfb1d7
    • Browse repository at this point

    Copy the full SHA

    9cfb1d7 View commit details

    Browse the repository at this point in the history

31. [Misc] use AutoTokenizer for benchmark serving when vLLM not installed ( …

    vllm-project#5588)

    

    zhyncs authored and robertgshaw2-neuralmagic committed Jun 23, 2024
    Configuration menu

    • View commit details
    • Copy full SHA for 61f421b
    • Browse repository at this point

    Copy the full SHA

    61f421b View commit details

    Browse the repository at this point in the history

32. [Hardware][Intel GPU] Add Intel GPU(XPU) inference backend (vllm-proj… …

    …ect#3814)
    
    Co-authored-by: Jiang Li <jiang1.li@intel.com>
    Co-authored-by: Abhilash Majumder <abhilash.majumder@intel.com>
    Co-authored-by: Abhilash Majumder <30946547+abhilash1910@users.noreply.github.com>

    

    

    4 people authored and robertgshaw2-neuralmagic committed Jun 23, 2024
    Configuration menu

    • View commit details
    • Copy full SHA for dceff94
    • Browse repository at this point

    Copy the full SHA

    dceff94 View commit details

    Browse the repository at this point in the history

33. [CI/BUILD] Support non-AVX512 vLLM building and testing (vllm-project… …

    …#5574)

    

    DamonFool authored and robertgshaw2-neuralmagic committed Jun 23, 2024
    Configuration menu

    • View commit details
    • Copy full SHA for e830048
    • Browse repository at this point

    Copy the full SHA

    e830048 View commit details

    Browse the repository at this point in the history

34. [CI] the readability of benchmarking and prepare for dashboard (vllm-… …

    …project#5571)
    
    [CI] Improve the readability of performance benchmarking results and prepare for upcoming performance dashboard (vllm-project#5571)

    

    KuntaiDu authored and robertgshaw2-neuralmagic committed Jun 23, 2024
    Configuration menu

    • View commit details
    • Copy full SHA for a212392
    • Browse repository at this point

    Copy the full SHA

    a212392 View commit details

    Browse the repository at this point in the history

35. [bugfix][distributed] fix 16 gpus local rank arrangement (vllm-projec… …

    …t#5604)

    

    youkaichao authored and robertgshaw2-neuralmagic committed Jun 23, 2024
    Configuration menu

    • View commit details
    • Copy full SHA for bc2be04
    • Browse repository at this point

    Copy the full SHA

    bc2be04 View commit details

    Browse the repository at this point in the history

36. [Optimization] use a pool to reuse LogicalTokenBlock.token_ids (vllm-… …

    …project#5584)

    

    youkaichao authored and robertgshaw2-neuralmagic committed Jun 23, 2024
    Configuration menu

    • View commit details
    • Copy full SHA for 5eb3526
    • Browse repository at this point

    Copy the full SHA

    5eb3526 View commit details

    Browse the repository at this point in the history

37. [Bugfix] Fix KV head calculation for MPT models when using GQA (vllm-… …

    …project#5142)

    

    bfontain authored and robertgshaw2-neuralmagic committed Jun 23, 2024
    Configuration menu

    • View commit details
    • Copy full SHA for 17fd0ba
    • Browse repository at this point

    Copy the full SHA

    17fd0ba View commit details

    Browse the repository at this point in the history

38. [Fix] Use utf-8 encoding in entrypoints/openai/run_batch.py (vllm-pro… …

    …ject#5606)

    

    zifeitong authored and robertgshaw2-neuralmagic committed Jun 23, 2024
    Configuration menu

    • View commit details
    • Copy full SHA for 7a58e54
    • Browse repository at this point

    Copy the full SHA

    7a58e54 View commit details

    Browse the repository at this point in the history

39. [Speculative Decoding 1/2 ] Add typical acceptance sampling as one of… …

    … the sampling techniques in the verifier (vllm-project#5131)

    

    sroy745 authored and robertgshaw2-neuralmagic committed Jun 23, 2024
    Configuration menu

    • View commit details
    • Copy full SHA for dbf0e91
    • Browse repository at this point

    Copy the full SHA

    dbf0e91 View commit details

    Browse the repository at this point in the history

40. [Model] Initialize Phi-3-vision support (vllm-project#4986)

    

    Isotr0py authored and robertgshaw2-neuralmagic committed Jun 23, 2024
    Configuration menu

    • View commit details
    • Copy full SHA for 18c566f
    • Browse repository at this point

    Copy the full SHA

    18c566f View commit details

    Browse the repository at this point in the history

41. [Kernel] Add punica dimensions for Granite 13b (vllm-project#5559) …

    Signed-off-by: Joe Runde <Joseph.Runde@ibm.com>

    

    joerunde authored and robertgshaw2-neuralmagic committed Jun 23, 2024
    Configuration menu

    • View commit details
    • Copy full SHA for 69fa6ed
    • Browse repository at this point

    Copy the full SHA

    69fa6ed View commit details

    Browse the repository at this point in the history

42. [misc][typo] fix typo (vllm-project#5620)

    

    youkaichao authored and robertgshaw2-neuralmagic committed Jun 23, 2024
    Configuration menu

    • View commit details
    • Copy full SHA for 1b39fc2
    • Browse repository at this point

    Copy the full SHA

    1b39fc2 View commit details

    Browse the repository at this point in the history

43. [Misc] Fix typo (vllm-project#5618)

    

    DarkLight1337 authored and robertgshaw2-neuralmagic committed Jun 23, 2024
    Configuration menu

    • View commit details
    • Copy full SHA for f691b45
    • Browse repository at this point

    Copy the full SHA

    f691b45 View commit details

    Browse the repository at this point in the history

44. [CI] Avoid naming different metrics with the same name in performance… …

    … benchmark (vllm-project#5615)

    

    KuntaiDu authored and robertgshaw2-neuralmagic committed Jun 23, 2024
    Configuration menu

    • View commit details
    • Copy full SHA for 5abb0c8
    • Browse repository at this point

    Copy the full SHA

    5abb0c8 View commit details

    Browse the repository at this point in the history

45. [bugfix][distributed] improve p2p capability test (vllm-project#5612) …

    [bugfix][distributed] do not error if two processes do not agree on p2p capability (vllm-project#5612)

    

    youkaichao authored and robertgshaw2-neuralmagic committed Jun 23, 2024
    Configuration menu

    • View commit details
    • Copy full SHA for f355997
    • Browse repository at this point

    Copy the full SHA

    f355997 View commit details

    Browse the repository at this point in the history

46. [Misc] Remove import from transformers logging (vllm-project#5625)

    

    CatherineSue authored and robertgshaw2-neuralmagic committed Jun 23, 2024
    Configuration menu

    • View commit details
    • Copy full SHA for 1343cd0
    • Browse repository at this point

    Copy the full SHA

    1343cd0 View commit details

    Browse the repository at this point in the history

47. [CI/Build][Misc] Update Pytest Marker for VLMs (vllm-project#5623)

    

    ywang96 authored and robertgshaw2-neuralmagic committed Jun 23, 2024
    Configuration menu

    • View commit details
    • Copy full SHA for 021cfdb
    • Browse repository at this point

    Copy the full SHA

    021cfdb View commit details

    Browse the repository at this point in the history

48. [ci] Deprecate original CI template (vllm-project#5624) …

    Signed-off-by: kevin <kevin@anyscale.com>

    

    khluu authored and robertgshaw2-neuralmagic committed Jun 23, 2024
    Configuration menu

    • View commit details
    • Copy full SHA for 70baf49
    • Browse repository at this point

    Copy the full SHA

    70baf49 View commit details

    Browse the repository at this point in the history

49. [Misc] Add OpenTelemetry support (vllm-project#4687) …

    This PR adds basic support for OpenTelemetry distributed tracing.
    It includes changes to enable tracing functionality and improve monitoring capabilities.
    
    I've also added a markdown with print-screens to guide users how to use this feature. You can find it here

    

    ronensc authored and robertgshaw2-neuralmagic committed Jun 23, 2024
    Configuration menu

    • View commit details
    • Copy full SHA for be2f123
    • Browse repository at this point

    Copy the full SHA

    be2f123 View commit details

    Browse the repository at this point in the history

50. [Misc] Add channel-wise quantization support for w8a8 dynamic per tok… …

    …en activation quantization (vllm-project#5542)

    

    dsikka authored and robertgshaw2-neuralmagic committed Jun 23, 2024
    Configuration menu

    • View commit details
    • Copy full SHA for 0008715
    • Browse repository at this point

    Copy the full SHA

    0008715 View commit details

    Browse the repository at this point in the history

51. [ci] Setup Release pipeline and build release wheels with cache (vllm… …

    …-project#5610)
    
    Signed-off-by: kevin <kevin@anyscale.com>

    

    khluu authored and robertgshaw2-neuralmagic committed Jun 23, 2024
    Configuration menu

    • View commit details
    • Copy full SHA for 14a7620
    • Browse repository at this point

    Copy the full SHA

    14a7620 View commit details

    Browse the repository at this point in the history

52. [Model] LoRA support added for command-r (vllm-project#5178)

    

    sergey-tinkoff authored and robertgshaw2-neuralmagic committed Jun 23, 2024
    Configuration menu

    • View commit details
    • Copy full SHA for 50c2ca9
    • Browse repository at this point

    Copy the full SHA

    50c2ca9 View commit details

    Browse the repository at this point in the history

53. [Bugfix] Fix for inconsistent behaviour related to sampling and repet… …

    …ition penalties (vllm-project#5639)
    
    Signed-off-by: Thomas Parnell <tpa@zurich.ibm.com>

    

    tdoublep authored and robertgshaw2-neuralmagic committed Jun 23, 2024
    Configuration menu

    • View commit details
    • Copy full SHA for 3d24777
    • Browse repository at this point

    Copy the full SHA

    3d24777 View commit details

    Browse the repository at this point in the history

54. [Doc] Added cerebrium as Integration option (vllm-project#5553)

    

    milo157 authored and robertgshaw2-neuralmagic committed Jun 23, 2024
    Configuration menu

    • View commit details
    • Copy full SHA for c5ef2f9
    • Browse repository at this point

    Copy the full SHA

    c5ef2f9 View commit details

    Browse the repository at this point in the history

55. [Bugfix] Fix CUDA version check for mma warning suppression (vllm-pro… …

    …ject#5642)

    

    tlrmchlsmth authored and robertgshaw2-neuralmagic committed Jun 23, 2024
    Configuration menu

    • View commit details
    • Copy full SHA for 010f2e8
    • Browse repository at this point

    Copy the full SHA

    010f2e8 View commit details

    Browse the repository at this point in the history

56. [Bugfix] Fix w8a8 benchmarks for int8 case (vllm-project#5643)

    

    tlrmchlsmth authored and robertgshaw2-neuralmagic committed Jun 23, 2024
    Configuration menu

    • View commit details
    • Copy full SHA for a8b75a4
    • Browse repository at this point

    Copy the full SHA

    a8b75a4 View commit details

    Browse the repository at this point in the history

57. [Bugfix] Fix Phi-3 Long RoPE scaling implementation (vllm-project#5628)

    

    ShukantPal authored and robertgshaw2-neuralmagic committed Jun 23, 2024
    Configuration menu

    • View commit details
    • Copy full SHA for a0d8ed2
    • Browse repository at this point

    Copy the full SHA

    a0d8ed2 View commit details

    Browse the repository at this point in the history

58. [Bugfix] Added test for sampling repetition penalty bug. (vllm-projec… …

    …t#5659)
    
    Signed-off-by: Thomas Parnell <tpa@zurich.ibm.com>

    

    tdoublep authored and robertgshaw2-neuralmagic committed Jun 23, 2024
    Configuration menu

    • View commit details
    • Copy full SHA for e4d2b6e
    • Browse repository at this point

    Copy the full SHA

    e4d2b6e View commit details

    Browse the repository at this point in the history

59. [Bugfix][CI/Build][AMD][ROCm]Fixed the cmake build bug which generate… …

    … garbage on certain devices (vllm-project#5641)

    

    hongxiayang authored and robertgshaw2-neuralmagic committed Jun 23, 2024
    Configuration menu

    • View commit details
    • Copy full SHA for cb46cfe
    • Browse repository at this point

    Copy the full SHA

    cb46cfe View commit details

    Browse the repository at this point in the history

60. [misc][distributed] use 127.0.0.1 for single-node (vllm-project#5619)

    

    youkaichao authored and robertgshaw2-neuralmagic committed Jun 23, 2024
    Configuration menu

    • View commit details
    • Copy full SHA for 8f72d50
    • Browse repository at this point

    Copy the full SHA

    8f72d50 View commit details

    Browse the repository at this point in the history

61. [Model] Add FP8 kv cache for Qwen2 (vllm-project#5656)

    

    mgoin authored and robertgshaw2-neuralmagic committed Jun 23, 2024
    Configuration menu

    • View commit details
    • Copy full SHA for b081ff9
    • Browse repository at this point

    Copy the full SHA

    b081ff9 View commit details

    Browse the repository at this point in the history

62. [Bugfix] Fix sampling_params passed incorrectly in Phi3v example (vll… …

    …m-project#5684)

    

    Isotr0py authored and robertgshaw2-neuralmagic committed Jun 23, 2024
    Configuration menu

    • View commit details
    • Copy full SHA for 784aa72
    • Browse repository at this point

    Copy the full SHA

    784aa72 View commit details

    Browse the repository at this point in the history

63. [Misc]Add param max-model-len in benchmark_latency.py (vllm-project#5629 …

    )

    

    DearPlanet authored and robertgshaw2-neuralmagic committed Jun 23, 2024
    Configuration menu

    • View commit details
    • Copy full SHA for a799171
    • Browse repository at this point

    Copy the full SHA

    a799171 View commit details

    Browse the repository at this point in the history

64. [CI/Build] Add tqdm to dependencies (vllm-project#5680)

    

    DarkLight1337 authored and robertgshaw2-neuralmagic committed Jun 23, 2024
    Configuration menu

    • View commit details
    • Copy full SHA for 436aaf9
    • Browse repository at this point

    Copy the full SHA

    436aaf9 View commit details

    Browse the repository at this point in the history

65. [ci] Add A100 queue into AWS CI template (vllm-project#5648) …

    Signed-off-by: kevin <kevin@anyscale.com>

    

    khluu authored and robertgshaw2-neuralmagic committed Jun 23, 2024
    Configuration menu

    • View commit details
    • Copy full SHA for d33025c
    • Browse repository at this point

    Copy the full SHA

    d33025c View commit details

    Browse the repository at this point in the history

66. [Frontend][Bugfix] Fix preemption_mode -> preemption-mode for CLI arg… …

    … in arg_utils.py (vllm-project#5688)

    

    mgoin authored and robertgshaw2-neuralmagic committed Jun 23, 2024
    Configuration menu

    • View commit details
    • Copy full SHA for cf5889f
    • Browse repository at this point

    Copy the full SHA

    cf5889f View commit details

    Browse the repository at this point in the history

67. [ci][distributed] add tests for custom allreduce (vllm-project#5689)

    

    youkaichao authored and robertgshaw2-neuralmagic committed Jun 23, 2024
    Configuration menu

    • View commit details
    • Copy full SHA for 88396ae
    • Browse repository at this point

    Copy the full SHA

    88396ae View commit details

    Browse the repository at this point in the history

68. [Bugfix] AsyncLLMEngine hangs with asyncio.run (vllm-project#5654)

    

    zifeitong authored and robertgshaw2-neuralmagic committed Jun 23, 2024
    Configuration menu

    • View commit details
    • Copy full SHA for 8ff473a
    • Browse repository at this point

    Copy the full SHA

    8ff473a View commit details

    Browse the repository at this point in the history

69. [Doc] Update docker references (vllm-project#5614) …

    Signed-off-by: Rafael Vasquez <rafvasq21@gmail.com>

    

    rafvasq authored and robertgshaw2-neuralmagic committed Jun 23, 2024
    Configuration menu

    • View commit details
    • Copy full SHA for 4f4cea6
    • Browse repository at this point

    Copy the full SHA

    4f4cea6 View commit details

    Browse the repository at this point in the history

70. [Misc] Add per channel support for static activation quantization; up… …

    …date w8a8 schemes to share base classes (vllm-project#5650)

    

    dsikka authored and robertgshaw2-neuralmagic committed Jun 23, 2024
    Configuration menu

    • View commit details
    • Copy full SHA for 0e8e31e
    • Browse repository at this point

    Copy the full SHA

    0e8e31e View commit details

    Browse the repository at this point in the history

71. [ci] Limit num gpus if specified for A100 (vllm-project#5694) …

    Signed-off-by: kevin <kevin@anyscale.com>

    

    khluu authored and robertgshaw2-neuralmagic committed Jun 23, 2024
    Configuration menu

    • View commit details
    • Copy full SHA for 1ccd388
    • Browse repository at this point

    Copy the full SHA

    1ccd388 View commit details

    Browse the repository at this point in the history

72. [Misc] Improve conftest (vllm-project#5681)

    

    DarkLight1337 authored and robertgshaw2-neuralmagic committed Jun 23, 2024
    Configuration menu

    • View commit details
    • Copy full SHA for 330aa1b
    • Browse repository at this point

    Copy the full SHA

    330aa1b View commit details

    Browse the repository at this point in the history

73. [Bugfix][Doc] FIx Duplicate Explicit Target Name Errors (vllm-project… …

    …#5703)

    

    ywang96 authored and robertgshaw2-neuralmagic committed Jun 23, 2024
    Configuration menu

    • View commit details
    • Copy full SHA for df3ae01
    • Browse repository at this point

    Copy the full SHA

    df3ae01 View commit details

    Browse the repository at this point in the history

74. [Kernel] Update Cutlass int8 kernel configs for SM90 (vllm-project#5514) …

    Co-authored-by: Varun Sundar Rabindranath <varun@neuralmagic.com>

    

    2 people authored and robertgshaw2-neuralmagic committed Jun 23, 2024
    Configuration menu

    • View commit details
    • Copy full SHA for 7d85753
    • Browse repository at this point

    Copy the full SHA

    7d85753 View commit details

    Browse the repository at this point in the history

75. [Model] Port over CLIPVisionModel for VLMs (vllm-project#5591)

    

    ywang96 authored and robertgshaw2-neuralmagic committed Jun 23, 2024
    Configuration menu

    • View commit details
    • Copy full SHA for b6ec1d5
    • Browse repository at this point

    Copy the full SHA

    b6ec1d5 View commit details

    Browse the repository at this point in the history

76. [Kernel] Update Cutlass int8 kernel configs for SM80 (vllm-project#5275) …

    Co-authored-by: Varun Sundar Rabindranath <varun@neuralmagic.com>

    

    2 people authored and robertgshaw2-neuralmagic committed Jun 23, 2024
    Configuration menu

    • View commit details
    • Copy full SHA for db7892d
    • Browse repository at this point

    Copy the full SHA

    db7892d View commit details

    Browse the repository at this point in the history

77. [Bugfix] Fix the CUDA version check for FP8 support in the CUTLASS ke… …

    …rnels (vllm-project#5715)

    

    tlrmchlsmth authored and robertgshaw2-neuralmagic committed Jun 23, 2024
    Configuration menu

    • View commit details
    • Copy full SHA for 51dfab0
    • Browse repository at this point

    Copy the full SHA

    51dfab0 View commit details

    Browse the repository at this point in the history

78. [Frontend] Add FlexibleArgumentParser to support both underscore and … …

    …dash in names (vllm-project#5718)

    

    mgoin authored and robertgshaw2-neuralmagic committed Jun 23, 2024
    Configuration menu

    • View commit details
    • Copy full SHA for c477239
    • Browse repository at this point

    Copy the full SHA

    c477239 View commit details

    Browse the repository at this point in the history

79. [distributed][misc] use fork by default for mp (vllm-project#5669)

    

    youkaichao authored and robertgshaw2-neuralmagic committed Jun 23, 2024
    Configuration menu

    • View commit details
    • Copy full SHA for 5ccb86c
    • Browse repository at this point

    Copy the full SHA

    5ccb86c View commit details

    Browse the repository at this point in the history

80. [Model] MLPSpeculator speculative decoding support (vllm-project#4947) …

    Signed-off-by: Thomas Parnell <tpa@zurich.ibm.com>
    
    Co-authored-by: Thomas Parnell <tpa@zurich.ibm.com>
    Co-authored-by: Nick Hill <nickhill@us.ibm.com>
    Co-authored-by: Davis Wertheimer <Davis.Wertheimer@ibm.com>

    

    

    4 people authored and robertgshaw2-neuralmagic committed Jun 23, 2024
    Configuration menu

    • View commit details
    • Copy full SHA for b05443a
    • Browse repository at this point

    Copy the full SHA

    b05443a View commit details

    Browse the repository at this point in the history

81. [Kernel] Add punica dimension for Qwen2 LoRA (vllm-project#5441)

    

    jinzhen-lin authored and robertgshaw2-neuralmagic committed Jun 23, 2024
    Configuration menu

    • View commit details
    • Copy full SHA for 1996acf
    • Browse repository at this point

    Copy the full SHA

    1996acf View commit details

    Browse the repository at this point in the history

82. [BugFix] Fix test_phi3v.py (vllm-project#5725)

    

    CatherineSue authored and robertgshaw2-neuralmagic committed Jun 23, 2024
    Configuration menu

    • View commit details
    • Copy full SHA for 1699d33
    • Browse repository at this point

    Copy the full SHA

    1699d33 View commit details

    Browse the repository at this point in the history

83. [Bugfix] Add fully sharded layer for QKVParallelLinearWithLora (vllm-… …

    …project#5665)
    
    Co-authored-by: Antoni Baum <antoni.baum@protonmail.com>

    

    2 people authored and robertgshaw2-neuralmagic committed Jun 23, 2024
    Configuration menu

    • View commit details
    • Copy full SHA for e4f1a4e
    • Browse repository at this point

    Copy the full SHA

    e4f1a4e View commit details

    Browse the repository at this point in the history

84. [Core][Distributed] add shm broadcast (vllm-project#5399) …

    Co-authored-by: Cody Yu <hao.yu.cody@gmail.com>

    

    2 people authored and robertgshaw2-neuralmagic committed Jun 23, 2024
    Configuration menu

    • View commit details
    • Copy full SHA for 3e3c8d9
    • Browse repository at this point

    Copy the full SHA

    3e3c8d9 View commit details

    Browse the repository at this point in the history

85. [Kernel][CPU] Add Quick gelu to CPU (vllm-project#5717)

    

    ywang96 authored and robertgshaw2-neuralmagic committed Jun 23, 2024
    Configuration menu

    • View commit details
    • Copy full SHA for 01369a0
    • Browse repository at this point

    Copy the full SHA

    01369a0 View commit details

    Browse the repository at this point in the history

86. [Doc] Documentation on supported hardware for quantization methods (v… …

    …llm-project#5745)

    

    mgoin authored and robertgshaw2-neuralmagic committed Jun 23, 2024
    Configuration menu

    • View commit details
    • Copy full SHA for 733cf30
    • Browse repository at this point

    Copy the full SHA

    733cf30 View commit details

    Browse the repository at this point in the history

87. [BugFix] exclude version 1.15.0 for modelscope (vllm-project#5668)

    

    zhyncs authored and robertgshaw2-neuralmagic committed Jun 23, 2024
    Configuration menu

    • View commit details
    • Copy full SHA for 07cd29d
    • Browse repository at this point

    Copy the full SHA

    07cd29d View commit details

    Browse the repository at this point in the history

88. [ci][test] fix ca test in main (vllm-project#5746)

    

    youkaichao authored and robertgshaw2-neuralmagic committed Jun 23, 2024
    Configuration menu

    • View commit details
    • Copy full SHA for 2e2140f
    • Browse repository at this point

    Copy the full SHA

    2e2140f View commit details

    Browse the repository at this point in the history

89. [LoRA] Add support for pinning lora adapters in the LRU cache (vllm-p… …

    …roject#5603)

    

    rohithkrn authored and robertgshaw2-neuralmagic committed Jun 23, 2024
    Configuration menu

    • View commit details
    • Copy full SHA for 0bec3f6
    • Browse repository at this point

    Copy the full SHA

    0bec3f6 View commit details

    Browse the repository at this point in the history

90. [CI][Hardware][Intel GPU] add Intel GPU(XPU) ci pipeline (vllm-projec… …

    …t#5616)

    

    jikunshang authored and robertgshaw2-neuralmagic committed Jun 23, 2024
    Configuration menu

    • View commit details
    • Copy full SHA for 3595200
    • Browse repository at this point

    Copy the full SHA

    3595200 View commit details

    Browse the repository at this point in the history

91. [Model] Support Qwen-VL and Qwen-VL-Chat models with text-only inputs (… …

    …vllm-project#5710)
    
    Co-authored-by: Roger Wang <ywang@roblox.com>

    

    2 people authored and robertgshaw2-neuralmagic committed Jun 23, 2024
    Configuration menu

    • View commit details
    • Copy full SHA for 1a6c6dd
    • Browse repository at this point

    Copy the full SHA

    1a6c6dd View commit details

    Browse the repository at this point in the history

92. [Misc] Remove vllm-project#4789 workaround left in vllm/entrypoints/o… …

    …penai/run_batch.py (vllm-project#5756)

    

    zifeitong authored and robertgshaw2-neuralmagic committed Jun 23, 2024
    Configuration menu

    • View commit details
    • Copy full SHA for a7dccd6
    • Browse repository at this point

    Copy the full SHA

    a7dccd6 View commit details

    Browse the repository at this point in the history

93. [Bugfix] Fix pin_lora error in TPU executor (vllm-project#5760)

    

    WoosukKwon authored and robertgshaw2-neuralmagic committed Jun 23, 2024
    Configuration menu

    • View commit details
    • Copy full SHA for 960a022
    • Browse repository at this point

    Copy the full SHA

    960a022 View commit details

    Browse the repository at this point in the history

94. [Docs][TPU] Add installation tip for TPU (vllm-project#5761)

    

    WoosukKwon authored and robertgshaw2-neuralmagic committed Jun 23, 2024
    Configuration menu

    • View commit details
    • Copy full SHA for dc211cd
    • Browse repository at this point

    Copy the full SHA

    dc211cd View commit details

    Browse the repository at this point in the history

95. [core][distributed] improve shared memory broadcast (vllm-project#5754)

    

    youkaichao authored and robertgshaw2-neuralmagic committed Jun 23, 2024
    Configuration menu

    • View commit details
    • Copy full SHA for 860a1d6
    • Browse repository at this point

    Copy the full SHA

    860a1d6 View commit details

    Browse the repository at this point in the history

96. [BugFix] [Kernel] Add Cutlass2x fallback kernels (vllm-project#5744) …

    Co-authored-by: Varun Sundar Rabindranath <varun@neuralmagic.com>

    

    2 people authored and robertgshaw2-neuralmagic committed Jun 23, 2024
    Configuration menu

    • View commit details
    • Copy full SHA for d7f0ece
    • Browse repository at this point

    Copy the full SHA

    d7f0ece View commit details

    Browse the repository at this point in the history

97. remove vllm-runnner-nm

    

    robertgshaw2-neuralmagic committed Jun 23, 2024
    Configuration menu

    • View commit details
    • Copy full SHA for e484da4
    • Browse repository at this point

    Copy the full SHA

    e484da4 View commit details

    Browse the repository at this point in the history

98. formatted

    

    robertgshaw2-neuralmagic committed Jun 23, 2024
    Configuration menu

    • View commit details
    • Copy full SHA for 683f309
    • Browse repository at this point

    Copy the full SHA

    683f309 View commit details

    Browse the repository at this point in the history