Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove redundant code in ColumnParallelLinear #146

Merged
merged 2 commits into from
Jun 11, 2023
Merged

Remove redundant code in ColumnParallelLinear #146

merged 2 commits into from
Jun 11, 2023

Conversation

WoosukKwon
Copy link
Collaborator

@WoosukKwon WoosukKwon commented Jun 11, 2023

This PR deletes redundant code in ColumnParallelLinear to reduce CPU overhead.

Before:

$ python benchmarks/benchmark_latency.py --model facebook/opt-6.7b
Namespace(model='facebook/opt-6.7b', tensor_parallel_size=1, input_len=32, output_len=128, batch_size=8, n=1, use_beam_search=False, num_iters=3, profile=False)
INFO 06-11 02:01:10 llm_server.py:60] Initializing an LLM server with config: model='facebook/opt-6.7b', dtype=torch.float16, use_dummy_weights=False, download_dir=None, use_np_weights=False, tensor_parallel_size=1, seed=0)
INFO 06-11 02:01:22 llm_server.py:129] # GPU blocks: 2946, # CPU blocks: 512
SamplingParams(n=1, best_of=1, presence_penalty=0.0, frequency_penalty=0.0, temperature=1.0, top_p=1.0, top_k=-1, use_beam_search=False, stop=[], ignore_eos=True, max_tokens=128, logprobs=None)
Warming up...
Profiling iterations: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:08<00:00,  2.68s/it]
Avg latency: 2.6759115854899087 seconds

After:

$ python benchmarks/benchmark_latency.py --model facebook/opt-6.7b
Namespace(model='facebook/opt-6.7b', tensor_parallel_size=1, input_len=32, output_len=128, batch_size=8, n=1, use_beam_search=False, num_iters=3, profile=False)
INFO 06-11 02:02:10 llm_server.py:60] Initializing an LLM server with config: model='facebook/opt-6.7b', dtype=torch.float16, use_dummy_weights=False, download_dir=None, use_np_weights=False, tensor_parallel_size=1, seed=0)
INFO 06-11 02:02:21 llm_server.py:129] # GPU blocks: 2946, # CPU blocks: 512
SamplingParams(n=1, best_of=1, presence_penalty=0.0, frequency_penalty=0.0, temperature=1.0, top_p=1.0, top_k=-1, use_beam_search=False, stop=[], ignore_eos=True, max_tokens=128, logprobs=None)
Warming up...
Profiling iterations: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:07<00:00,  2.36s/it]
Avg latency: 2.363809108734131 seconds

@WoosukKwon WoosukKwon requested a review from zhuohan123 June 11, 2023 02:15
@WoosukKwon WoosukKwon changed the title Short-circuit ColumnParallelLinear to reduce CPU overhead Remove redundant code in ColumnParallelLinear Jun 11, 2023
Copy link
Member

@zhuohan123 zhuohan123 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! This does not make any sense to me... This function is just an identity mapping in Torch. Should cost nothing.

@WoosukKwon WoosukKwon merged commit da5ddcd into main Jun 11, 2023
@WoosukKwon WoosukKwon deleted the opt branch June 11, 2023 04:25
@WoosukKwon WoosukKwon mentioned this pull request Aug 2, 2023
6 tasks
hongxiayang pushed a commit to hongxiayang/vllm that referenced this pull request Feb 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants