Skip to content
This repository was archived by the owner on Oct 11, 2024. It is now read-only.

Upstream sync 2024 03 07 #103

Closed
varun-sundar-rabindranath wants to merge 67 commits into from
Closed

Upstream sync 2024 03 07 #103

varun-sundar-rabindranath wants to merge 67 commits into from

Conversation

@varun-sundar-rabindranath
Copy link

@varun-sundar-rabindranath varun-sundar-rabindranath commented Mar 7, 2024

SUMMARY:

  • sync "nm-vllm" fork with upstream as of 2024-03-07 around 4:00 EST -
    385da2dae2b90e5273da8dfce881727bd9c574a1
  • resolved merge conflicts

NOTE:

(vllm-test) varun@ip-192-168-198-190 nm-vllm % git remote -v 
origin	https://github.com/neuralmagic/nm-vllm.git (fetch)
origin	https://github.com/neuralmagic/nm-vllm.git (push)
upstream	https://github.com/vllm-project/vllm.git (fetch)
upstream	https://github.com/vllm-project/vllm.git (push)

TEST PLAN:
Tests on this PR

ronensc and others added 30 commits February 21, 2024 18:18
@ronensc
Update comment (vllm-project#2934)
d7f3964
@Maxusmusti
Added early stopping to completion APIs (vllm-project#2939)
5574081
@esmeetu
Migrate MistralForCausalLM to LlamaForCausalLM (vllm-project#2868)
344020c
@WoosukKwon
Use Llama RMSNorm custom op for Gemma (vllm-project#2974)
95529e3
@mspronesti
chore(vllm): codespell for spell checking (vllm-project#2820)
93dc5a2
@WoosukKwon
Optimize GeGLU layer in Gemma (vllm-project#2975)
fd5dcc5
@44670
[FIX] Fix a bug in initializing Yarn RoPE (vllm-project#2983)
c530e2c
@WoosukKwon
Remove Flash Attention in test env (vllm-project#2982)
6f32cdd
@ronensc
Include tokens from prompt phase in counter_generation_tokens (vllm…
4caf704 
…-project#2802)
@zhaoyang-star
Fix nvcc not found in vlm-openai image (vllm-project#2781)
57f0449
@WoosukKwon
[Fix] Fissertion on YaRN model len (vllm-project#2984)
f7c1234
@hmellor
Port metrics from aioprometheus to prometheus_client (vllm-projec…
ef978fe 
…t#2730)
@jlcmoore
Add LogProbs for Chat Completions in OpenAI (vllm-project#2918)
70f3e8e
@pcmoritz @cadedaniel
Optimize Triton MoE Kernel (vllm-project#2979)
cfc15a1 
Co-authored-by: Cade Daniel <edacih@gmail.com>
@WoosukKwon
[Minor] Remove gather_cached_kv kernel (vllm-project#3043)
d6e4a13
@esmeetu
[Minor] Remove unused config files (vllm-project#3039)
d9f726c
@esmeetu
Don't use cupy when enforce_eager=True (vllm-project#3037)
c1c0d00
@esmeetu
Fix stablelm (vllm-project#3038)
4dd6416
@dachengai @WoosukKwon
Support Orion model (vllm-project#2539)
48a8f4a 
Co-authored-by: zhangdacheng <zhangdacheng@ainirobot.com>
Co-authored-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
@Jingru
fix get_ip error in pure ipv6 environment (vllm-project#2931)
2410e32
@WoosukKwon
[Minor] Fix type annotation in fused moe (vllm-project#3045)
4bd18ec
@dylanwhawk
Support logit bias for OpenAI API (vllm-project#3027)
e0ade06
@WoosukKwon
[Minor] Fix StableLMEpochForCausalLM -> StableLmForCausalLM (vllm-pro…
8b430d7 
…ject#3046)
@sighingnow
Enable GQA support in the prefix prefill kernels (vllm-project#3007)
71bcaf9 
Signed-off-by: Tao He <sighingnow@gmail.com>
@ElefHead
multi-lora documentation fix (vllm-project#3064)
a868310
@AllenDou
Restrict prometheus_client >= 0.18.0 to prevent errors when importing…
e46fa5d 
… pkgs (vllm-project#3070)
@liangfu
[Neuron] Support inference with transformers-neuronx (vllm-project#2569)
3b7178c
@WoosukKwon
Add LoRA support for Gemma (vllm-project#3050)
929b4f2
@chu-tianxiang
Add Support for 2/3/8-bit GPTQ Quantization Models (vllm-project#2330)
01a5d18
@jaywonchung
Fix: AttributeError in OpenAI-compatible server (vllm-project#3018)
a6d471c
pian13131 and others added 18 commits March 4, 2024 09:23
@pian13131
Add document for vllm paged attention kernel. (vllm-project#2978)
27a7b07
@AllenDou
enable --gpu-memory-utilization in benchmark_throughput.py (vllm-proj…
9cbc7e5 
…ect#3175)

Co-authored-by: zixiao <shunli.dsl@alibaba-inc.com>
@ttbachyinsda
[Minor fix] The domain dns.google may cause a socket.gaierror excepti…
76e8a70 
…on (vllm-project#3176)

Co-authored-by: guofangze <guofangze@kuaishou.com>
@Yard1
Push logprob generation to LLMEngine (vllm-project#3065)
22de452 
Co-authored-by: Avnish Narayan <avnish@anyscale.com>
@Yard1 @zhuohan123
Add health check, make async Engine more robust (vllm-project#3015)
ff578ca 
Co-authored-by: Zhuohan Li <zhuohan123@gmail.com>
@wangchen615 @ywang96
Fix the openai benchmarking requests to work with latest OpenAI apis (v…
9a4548b 
…llm-project#2992)

Co-authored-by: Roger Wang <136131678+ywang96@users.noreply.github.com>
@hongxiayang @lcskrishna
[ROCm] enable cupy in order to enable cudagraph mode for AMD GPUs (vl…
05af6da 
…lm-project#3123)

Co-authored-by: lcskrishna <lollachaitanya@gmail.com>
@njhill
Store eos_token_id in Sequence for easy access (vllm-project#3166)
8999ec3
@njhill @Yard1
[Fix] Avoid pickling entire LLMEngine for Ray workers (vllm-project#3207
2efce05 
)

Co-authored-by: Antoni Baum <antoni.baum@protonmail.com>
@rkooo567
[Tests] Add block manager and scheduler tests (vllm-project#3108)
24aecf4
@cadedaniel
[Testing] Fix core tests (vllm-project#3224)
a33ce60
@chujiezheng
Add tqdm dynamic_ncols=True (vllm-project#3242)
4cb3b92
@TechxGenus
Add GPTQ support for Gemma (vllm-project#3200)
d3c04b6
@wangchen615 @zhuohan123
Update requirements-dev.txt to include package for benchmarking scrip…
cbf4c05 
…ts. (vllm-project#3181)

Co-authored-by: Zhuohan Li <zhuohan123@gmail.com>
@WoosukKwon
Separate attention backends (vllm-project#3005)
2daf23a
@mgoin
Measure model memory usage (vllm-project#3120)
385da2d
Merge remote-tracking branch 'upstream/main' into upstream-sync-2024-…
35643d9 
…03-07
appease ruff
f53ad16
@varun-sundar-rabindranath
Copy link
Author

varun-sundar-rabindranath commented Mar 7, 2024

@andy-neuma - Same build failure on upstream vllm - as noted by @mgoin vllm-project#3265

bnellnm
bnellnm reviewed Mar 7, 2024
View reviewed changes
Copy link
Member

@bnellnm bnellnm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we should hold off until the setup.py issues are resolved. cc @mgoin @robertgshaw2-neuralmagic

bnellnm
bnellnm suggested changes Mar 7, 2024
View reviewed changes
Copy link
Member

@bnellnm bnellnm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

undoing prior approval until setup.py can be resolved.

@varun-sundar-rabindranath
Copy link
Author

varun-sundar-rabindranath commented Mar 8, 2024

Wait for upstream to resolve build issues - upstream WIP vllm-project#3266

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

@mgoin mgoin Awaiting requested review from mgoin

@andy-neuma andy-neuma Awaiting requested review from andy-neuma

@robertgshaw2-redhat robertgshaw2-redhat Awaiting requested review from robertgshaw2-redhat

@SageMoore SageMoore Awaiting requested review from SageMoore

@ElizaWszola ElizaWszola Awaiting requested review from ElizaWszola

1 more reviewer

@bnellnm bnellnm bnellnm requested changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.