Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Doc] Documentation for distributed inference #261

Merged
merged 4 commits into from
Jun 26, 2023
Merged

[Doc] Documentation for distributed inference #261

merged 4 commits into from
Jun 26, 2023

Conversation

zhuohan123
Copy link
Member

@zhuohan123 zhuohan123 commented Jun 26, 2023 •
edited
Loading

In this PR:

  • Add anyscale's blogpost to README as a reference for continuous batching.
  • Documentation for distributed inference.

Fix #206

@zhuohan123 zhuohan123 requested a review from WoosukKwon June 26, 2023 16:16
WoosukKwon
WoosukKwon reviewed Jun 26, 2023
View reviewed changes
Copy link
Collaborator

@WoosukKwon WoosukKwon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR! Left some comments.

docs/source/index.rst Outdated Show resolved Hide resolved
docs/source/serving/distributed_serving.rst Outdated Show resolved Hide resolved
docs/source/index.rst Show resolved Hide resolved
docs/source/serving/distributed_serving.rst Outdated Show resolved Hide resolved
docs/source/index.rst Outdated Show resolved Hide resolved
docs/source/index.rst Outdated Show resolved Hide resolved
@zhuohan123
Copy link
Member Author

@WoosukKwon Just fixed all comments. Please take a look.

WoosukKwon
WoosukKwon approved these changes Jun 26, 2023
View reviewed changes
Copy link
Collaborator

@WoosukKwon WoosukKwon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Thanks!

@zhuohan123 zhuohan123 merged commit 2cf1a33 into main Jun 26, 2023
@zhuohan123 zhuohan123 deleted the distributed-inference-doc branch June 29, 2023 17:25
michaelfeil pushed a commit to michaelfeil/vllm that referenced this pull request Jul 1, 2023
@zhuohan123 @michaelfeil
[Doc] Documentation for distributed inference (vllm-project#261)
b04f5cb
hongxiayang pushed a commit to hongxiayang/vllm that referenced this pull request Feb 13, 2024
@zhuohan123
[Doc] Documentation for distributed inference (vllm-project#261)
7341f28
yukavio pushed a commit to yukavio/vllm that referenced this pull request Jul 3, 2024
@dbarbuzzi
Address py38/39 incompatibilities (vllm-project#261)
e168417 
**SUMMARY**

Handle a few different Python 3.8/3.9 compatibility issues:
* Fix the Python 3.8 compatibility in one of our benchmarking scripts.
* Fix skip for `test_local_workers_clean_shutdown` (previous skip
attempt misused `pytest.mark.skipif`)
* Skip `test_merge_async_iterators` (uses Python builtin `anext` which
was added in 3.10)
jikunshang pushed a commit to jikunshang/vllm that referenced this pull request Sep 11, 2024
@madamczykhabana
Fix dispersed slots (vllm-project#261)
69df1e7 
On habana_main the slots are calculated by adding an offset to the block
which breaks the check for _PAD_SLOT_ID. Reworked it so that in case of
_PAD_BLOCK_ID we're automatically inserting the right value.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@WoosukKwon WoosukKwon WoosukKwon approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Documentation on distributed execution
2 participants
@zhuohan123 @WoosukKwon