Supporting log probabilities of prompt tokens in both engine and OpenAI API server (aka `echo`) #959

wanmok · 2023-09-06T06:14:42Z

According to issue #201 , the echo function is not currently supported in vLLM. This implementation does not require making modifications to individual model files but lay the heavy work on the first generation step.

The basic logic of the implementation:

Engine
1. Added echo to the SamplingParams so that the engine can be informed of the echo request
2. Modified Sampler to process the echo request.
  a. Flags from SamplingParams of each SequenceGroup would be used to determine whether to perform echo. Log probabilities of prompt tokens are computed at the first generation step.
  b. Modified SequenceOutputs to carry prompt log probs from the first generate step so that engine can post process SequenceData to store the generated log probabilities of prompt tokens. Since log probabilities of prompt tokens are not necessarily stored in the top-k returns, they are stored in two separate fields as prompt_logprobs and prompt_top_logprobs.
  c. Changed numerical unstable computation of log(softmax()) to log_softmax().
3. Modified RequestOutput.from_seq_group so that log probabilities of prompt tokens can be inserted into the final outputs.
OpenAI API server
1. Removed error message for requesting echo.
2. Modified create_logprobs function to reflect engine output changes.
3. Edge case: added the support for the case of echo=True and max_tokens=0, which is a valid case in OpenAI APIs. In this case, the SamplingParams.max_tokens would be set to 1 and enable a echo_self flag. The flag is then used in the following post processing to ensure the additionally generated token being removed from the final output.

Please let me know if you have any questions or concerns with respect to merging this PR. Thanks!

…et original prompt.

eugenepentland · 2023-09-07T06:16:10Z

Hello, thanks for this! Have you tested it using the openai compatible endpoint? when I try using curl it gets the request but I never get a response back. The following is with echo off, but it hangs regardless if it is on/off.

INFO 09-07 06:06:49 async_llm_engine.py:226] Received request cmpl-a06d218f31d549d7b128ee067691a83a: prompt: 'San Francisco is a', sampling params: SamplingParams(n=1, best_of=1, presence_penalty=0.0, frequency_penalty=0.0, temperature=0.0, top_p=1.0, top_k=-1, use_beam_search=False, length_penalty=1.0, early_stopping=False, stop=[], ignore_eos=False, max_tokens=7, logprobs=None), use_echo=False, prompt token ids: [1, 3087, 8970, 338, 263].

I may just be impatient, but I am excited to try this out with llm eval harness!

Signed-off-by: Antoni Baum <antoni.baum@protonmail.com>

wanmok · 2023-09-07T07:36:13Z

Hello, thanks for this! Have you tested it using the openai compatible endpoint? when I try using curl it gets the request but I never get a response back. The following is with echo off, but it hangs regardless if it is on/off.

INFO 09-07 06:06:49 async_llm_engine.py:226] Received request cmpl-a06d218f31d549d7b128ee067691a83a: prompt: 'San Francisco is a', sampling params: SamplingParams(n=1, best_of=1, presence_penalty=0.0, frequency_penalty=0.0, temperature=0.0, top_p=1.0, top_k=-1, use_beam_search=False, length_penalty=1.0, early_stopping=False, stop=[], ignore_eos=False, max_tokens=7, logprobs=None), use_echo=False, prompt token ids: [1, 3087, 8970, 338, 263].

I may just be impatient, but I am excited to try this out with llm eval harness!

What is the LM you were using for the request? Also, how many GPUs?

eugenepentland · 2023-09-07T14:59:42Z

Turns out I still had tensor_parallel=2 on, not sure the issue is related to your fork. Setting it to 1 fixed that issue. Running Llama-7B-HF on a RTX 3090.

This is the new error internal server error I'm getting when I have the following args to http://localhost:8000/v1/completions with the openai endpoint:

JSON Args:

{
    "model": "llama",
    "prompt": "San Fransisco is a",
    "max_tokens": 0,
    "echo": true,
    "logprobs": 5
}

Error:

  File "/home/epentland/miniconda3/envs/axol/lib/python3.9/site-packages/fastapi/routing.py", line 190, in run_endpoint_function
    return await dependant.call(**values)
  File "/home/epentland/ai/inference/echo/vllm/vllm/entrypoints/openai/api_server.py", line 565, in create_completion
    logprobs = create_logprobs(
  File "/home/epentland/ai/inference/echo/vllm/vllm/entrypoints/openai/api_server.py", line 182, in create_logprobs
    for i, p in top_logprobs[i].items()
    ```

wanmok · 2023-09-07T15:01:58Z

Turns out I still had tensor_parallel=2 on, not sure the issue is related to your fork. Setting it to 1 fixed that issue. Running Llama-7B-HF on a RTX 3090.

This is the new error internal server error I'm getting when I have the following args to http://localhost:8000/v1/completions with the openai endpoint:

JSON Args:
{
    "model": "llama",
    "prompt": "San Fransisco is a",
    "max_tokens": 0,
    "echo": true,
    "logprobs": 5
}
Error:
  File "/home/epentland/miniconda3/envs/axol/lib/python3.9/site-packages/fastapi/routing.py", line 190, in run_endpoint_function
    return await dependant.call(**values)
  File "/home/epentland/ai/inference/echo/vllm/vllm/entrypoints/openai/api_server.py", line 565, in create_completion
    logprobs = create_logprobs(
  File "/home/epentland/ai/inference/echo/vllm/vllm/entrypoints/openai/api_server.py", line 182, in create_logprobs
    for i, p in top_logprobs[i].items()
    ```

Could you pull the latest commit? I have tested the latest commit with about 4k requests (similar to yours).

eugenepentland · 2023-09-08T06:16:55Z

I tested on the latest commit on my own hardware, and on a A6000 on runpod, getting the same error on both. Can you send the exact command for the openai ai sever and the curl request/python code you used to make the request?

wanmok · 2023-09-08T07:02:21Z

I tested on the latest commit on my own hardware, and on a A6000 on runpod, getting the same error on both. Can you send the exact command for the openai ai sever and the curl request/python code you used to make the request?

Ah, I have it fixed in the latest commit. It was later introduced by the modifications to the top_logprobs, where the first one should be None. Could you try to run it again with your dataset? This PR introduced multiple changes, so I suppose that it would be better to test more rigorously... Thanks for catching that.

eugenepentland · 2023-09-08T15:48:11Z

It's working on my end now! Thank you for the great work!

I am making it work with lm-eval-harness so we can have a much faster model evaluation.

wanmok · 2023-09-11T20:26:24Z

@zhuohan123 @WoosukKwon - hi guys, I wonder if we could have someone review the PR. If any change requested, I would be happy to work on it. Thanks!

zhuohan123

Thank you for your contribution! This is a feature that a lot of our community members are asking. The PR is relatively complex and I think it may require several rounds of review. I have left some of my initial comments. Please take a look. Additionally, Is it possible for you to add a test on this functionality here? After this round, I will run this PR, check the correctness of the PR, and then perform another round of review. Thanks again!

vllm/engine/llm_engine.py

vllm/entrypoints/openai/api_server.py

vllm/sequence.py

vllm/sampling_params.py

vllm/worker/worker.py

vllm/model_executor/layers/sampler.py

wanmok · 2023-09-24T12:00:16Z

@zhuohan123 I wonder if there is any update on the review. Please let me know if you found any blocker.

wanmok

I have addressed comments from the previous review. Please let me know if there is any other concerns to merge the PR.

lifengjin · 2023-10-12T23:50:52Z

Any update on this? This is very useful.

zhuohan123 · 2023-10-13T18:54:46Z

@wanmok Thanks for all of your contributions so far. I have been looking into this last week and tried to have a simpler modification to the vLLM core to implement this function. Please find my changes in this PR: #1328. Is it possible for you to add the OpenAI endpoint support of echo on top of that PR? Let me know if there is any issue.

wanmok · 2023-10-14T09:03:50Z

@wanmok Thanks for all of your contributions so far. I have been looking into this last week and tried to have a simpler modification to the vLLM core to implement this function. Please find my changes in this PR: #1328. Is it possible for you to add the OpenAI endpoint support of echo on top of that PR? Let me know if there is any issue.

Well.... it might take some time to read through the PR and re-implement that. Are you thinking about a new PR after that one being merged or?

zhuohan123 · 2023-10-15T03:18:20Z

Well.... it might take some time to read through the PR and re-implement that. Are you thinking about a new PR after that one being merged or?

PR #1328 modified the vLLM core. So that you can just add prompt_logprobs=N in SamplingParams, and the returning RequestOutput will include a prompt_logprobs field in it. After #1328 being merged into the main branch, the only remaining thing is to let the OpenAI API server uses this new feature to support echo. Can you submit a new PR for this after #1328 is merged? I believe the code change to openai/api_server.py should be very similar with what you have done in this PR.

wanmok · 2023-10-15T03:31:59Z

Sounds good. I can file another PR for OpenAI server later. Also, since I have been working on a similar implementation in this PR, could you list me as co-author for #1328?

…

-Yunmo

On Oct 14, 2023 at 11:18 PM -0400, Zhuohan Li ***@***.***>, wrote: > Well.... it might take some time to read through the PR and re-implement that. Are you thinking about a new PR after that one being merged or? PR #1328 modified the vLLM core. So that you can just add prompt_logprobs=N in SamplingParams, and the returning RequestOutput will include a prompt_logprobs field in it. After #1328 being merged into the main branch, the only remaining thing is to let the OpenAI API server uses this new feature to support echo. Can you submit a new PR for this after #1328 is merged? I believe the code change to openai/api_server.py should be very similar with what you have done in this PR. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: ***@***.***>

zhuohan123 · 2023-10-15T03:46:12Z

Sure thing! Actually in that PR I used some of your testing code. Will add it when I merge the PR.

RanchiZhao · 2023-10-16T07:52:08Z

Cool！is this now available， I mean the logps of prompt tokens.

wanmok · 2023-10-16T07:55:37Z

Cool！is this now available， I mean the logps of prompt tokens.

I suppose that the PR is not ready yet. If you would like to get logprobs of prompt tokens, you could try with this PR as a temporary solution.

zhuohan123 · 2023-10-17T01:03:43Z

@wanmok #1328 has been merged to the main branch!

wanmok · 2023-10-17T03:31:31Z

@wanmok #1328 has been merged to the main branch!

Cool! I'll draft a new PR when I got time, probably in a week or so (traveling to CA this week).

lifengjin · 2023-10-24T17:17:51Z

Any updates on this, please?

zhuohan123 · 2023-10-30T21:52:38Z

Close this PR in favor of #1328 and #1509. Thank you for your contribution!

wanmok added 2 commits September 6, 2023 01:50

A runnable implementation of echo.

721c778

Reformatted.

4abbe99

wanmok changed the title ~~A runnable implementation of echo.~~ Supporting log probabilities of prompt tokens in both engine and OpenAI API server (aka echo) Sep 6, 2023

wanmok added 6 commits September 6, 2023 02:21

Fixed pylint R1729 and R1721

3a9649f

Fixed pylint and allowed logprobs=0, echo=True.

f075e8b

Fixed edge case of when to return null - consistent with OpenAI.

e419414

Fixed the issue of using prompt_token_ids while not being able to g…

50b30f5

…et original prompt.

Fixed the error of computing logprobs.

69f9e52

Fixed the pylint E1133

1070080

wanmok and others added 3 commits September 7, 2023 03:18

Fixed the issue of running with distributed workers.

bd180cb

Set torch default dtype in a context manager (vllm-project#971)

46af13a

Signed-off-by: Antoni Baum <antoni.baum@protonmail.com>

Merge branch 'vllm-project:main' into add-echo

8309110

Moved postprocessing to _process_sequence_group_samples().

fdc4e5a

wanmok added 2 commits September 7, 2023 17:19

Merge branch 'vllm-project:main' into add-echo

f422556

Merge branch 'vllm-project:main' into add-echo

4ede60d

wanmok and others added 2 commits September 8, 2023 02:43

Merge branch 'vllm-project:main' into add-echo

6c1a70c

Fixed the issue with the first top_logprobs being None.

0bbf78d

wanmok added 2 commits September 8, 2023 14:08

Merge branch 'vllm-project:main' into add-echo

b60817d

Merge branch 'vllm-project:main' into add-echo

39f8268

zhuohan123 self-requested a review September 11, 2023 23:29

zhuohan123 requested changes Sep 12, 2023

View reviewed changes

zhuohan123 added the enhancement New feature or request label Sep 12, 2023

wanmok and others added 2 commits September 23, 2023 03:22

Reformatted

c6b8d1f

Merge branch 'vllm-project:main' into add-echo

67a5de0

Merge branch 'main' into add-echo

17675ec

wanmok commented Sep 28, 2023

View reviewed changes

wanmok added 4 commits September 28, 2023 20:01

Merge branch 'main' into add-echo

792f857

Merge branch 'vllm-project:main' into add-echo

ed52b8e

Merge branch 'vllm-project:main' into add-echo

0d643fd

Update sampler.py

18f9efa

zhuohan123 mentioned this pull request Oct 8, 2023

Introduce return_prompt request option to api_server entrypoint #1232

Closed

wanmok and others added 2 commits October 9, 2023 15:21

Merge branch 'vllm-project:main' into add-echo

9967c2f

Merged log forward.

a0ebd61

Merge branch 'vllm-project:main' into add-echo

4121f86

WoosukKwon mentioned this pull request Oct 13, 2023

[v0.2.1] Release Tracker #1346

Closed

3 tasks

wanmok mentioned this pull request Oct 20, 2023

Supports prompt_token_ids in the OpenAI completion API. (#1) #703

Closed

leocnj mentioned this pull request Oct 26, 2023

[Feature]: Add support for OpenAI's echo parameter. BerriAI/litellm#699

Closed

wanmok mentioned this pull request Oct 30, 2023

Added echo function to OpenAI API server. #1504

Merged

zhuohan123 closed this Oct 30, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Supporting log probabilities of prompt tokens in both engine and OpenAI API server (aka `echo`) #959

Supporting log probabilities of prompt tokens in both engine and OpenAI API server (aka `echo`) #959

wanmok commented Sep 6, 2023 •

edited

Loading

eugenepentland commented Sep 7, 2023

wanmok commented Sep 7, 2023

eugenepentland commented Sep 7, 2023

wanmok commented Sep 7, 2023

eugenepentland commented Sep 8, 2023

wanmok commented Sep 8, 2023 •

edited

Loading

eugenepentland commented Sep 8, 2023

wanmok commented Sep 11, 2023

zhuohan123 left a comment

wanmok commented Sep 24, 2023

wanmok left a comment

lifengjin commented Oct 12, 2023

zhuohan123 commented Oct 13, 2023

wanmok commented Oct 14, 2023

zhuohan123 commented Oct 15, 2023

wanmok commented Oct 15, 2023 via email

zhuohan123 commented Oct 15, 2023

RanchiZhao commented Oct 16, 2023

wanmok commented Oct 16, 2023

zhuohan123 commented Oct 17, 2023

wanmok commented Oct 17, 2023

lifengjin commented Oct 24, 2023

zhuohan123 commented Oct 30, 2023

Supporting log probabilities of prompt tokens in both engine and OpenAI API server (aka echo) #959

Supporting log probabilities of prompt tokens in both engine and OpenAI API server (aka echo) #959

Conversation

wanmok commented Sep 6, 2023 • edited Loading

eugenepentland commented Sep 7, 2023

wanmok commented Sep 7, 2023

eugenepentland commented Sep 7, 2023

wanmok commented Sep 7, 2023

eugenepentland commented Sep 8, 2023

wanmok commented Sep 8, 2023 • edited Loading

eugenepentland commented Sep 8, 2023

wanmok commented Sep 11, 2023

zhuohan123 left a comment

Choose a reason for hiding this comment

wanmok commented Sep 24, 2023

wanmok left a comment

Choose a reason for hiding this comment

lifengjin commented Oct 12, 2023

zhuohan123 commented Oct 13, 2023

wanmok commented Oct 14, 2023

zhuohan123 commented Oct 15, 2023

wanmok commented Oct 15, 2023 via email

zhuohan123 commented Oct 15, 2023

RanchiZhao commented Oct 16, 2023

wanmok commented Oct 16, 2023

zhuohan123 commented Oct 17, 2023

wanmok commented Oct 17, 2023

lifengjin commented Oct 24, 2023

zhuohan123 commented Oct 30, 2023

Supporting log probabilities of prompt tokens in both engine and OpenAI API server (aka `echo`) #959

Supporting log probabilities of prompt tokens in both engine and OpenAI API server (aka `echo`) #959

wanmok commented Sep 6, 2023 •

edited

Loading

wanmok commented Sep 8, 2023 •

edited

Loading