[Usage]: Llama 3 8B Instruct Inference

[aliozts]

Your current environment

Using the latest version of vLLM on 2 L4 GPUs.

How would you like to use vllm

I was trying to utilize vLLM to deploy meta-llama/Meta-Llama-3-8B-Instruct model and use OpenAI compatible server with the latest docker image. When I did, it was not stopping generation for a while when max_tokens=None. I saw that it's generating <|eot_id|> token which is its eos token apparently but in their tokenizer_config and in other configs it is <|end_of_text|>.

I can fix this by setting the eos_token parameter in tokenizer_config.json as <|eot_id|> or using

response = client.chat.completions.create(
    model="meta-llama/Meta-Llama-3-8B-Instruct",
    messages=[{"role": "user",
               "content": "Write a function for fibonacci sequence. Use LRUCache"}],
    max_tokens=700,
    stream=False,
    extra_body={"stop_token_ids":[128009]}
)

stop_token_ids in my request. I wanted to ask the optimal way to solve this problem.

There is an existing discussion/PR in their repo which is updating the generation_config.json but unless I clone myself, I saw that vLLM does not install the generation_config.json file. I also tried with this revision but it still was not stopping generating after <|eot_id|>. Moreover, I tried with this revision as well but it did not stop generating as well.

tldr; Llama-3-8B-Instruct model does not stop generation because of the eos token.

• Updating generation_config.json does not work.
• Updating config.json also does not work.
• Updating tokenizer_config.json works but it overwrites the existing eos_token. Is this problematic or is there a more elegant way to solve this?

May I ask the optimal way to solve this issue?

[simon-mo]

What you are doing here with stop_token_ids is the right temporary fix. I'll send a PR to respect generation_config.json and once the meta-llama/Meta-Llama-3-8B-Instruct is updated on the hub it should be working out of the box.

Generation config support multiple eos.

[njhill]

It does not appear that the -instruct models will output the configured EOS token, so I think it's also safe to change in the config to <|eot_id|> (128009). The base / pre-trained models should keep the existing <|end_of_text|> eos token though.

[aliozts]

thank you so much!

[haqishen]

Based on the sample code on the HF page for LLaMA3 (https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct), it is necessary to manually add a new EOS token as follow.

terminators = [
    pipeline.tokenizer.eos_token_id,
    pipeline.tokenizer.convert_tokens_to_ids("<|eot_id|>")
]

Therefore, I think this might not be an issue that the vllm team needs to address, but rather something that requires manually adding this EOS token when using vllm to generate with LLaMA3.

Here's the sample code for dealing it for batch inference:

llm = LLM(
    model=name,
    trust_remote_code=True,
    tensor_parallel_size=2,
)
tokenizer = llm.get_tokenizer()

conversations = tokenizer.apply_chat_template(
    [{'role': 'user', 'content': 'hi, how are you?'}],
    tokenize=False,
)

outputs = llm.generate(
    [conversations],
    SamplingParams(
        temperature=0.5,
        top_p=0.9,
        max_tokens=1024,
        stop_token_ids=[tokenizer.eos_token_id, tokenizer.convert_tokens_to_ids("<|eot_id|>")],  # KEYPOINT HERE
    )
)

[joshuachak]

Based on the sample code on the HF page for LLaMA3 (https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct), it is necessary to manually add a new EOS token as follow.

terminators = [
    pipeline.tokenizer.eos_token_id,
    pipeline.tokenizer.convert_tokens_to_ids("<|eot_id|>")
]

Therefore, I think this might not be an issue that the vllm team needs to address, but rather something that requires manually adding this EOS token when using vllm to generate with LLaMA3.

Here's the sample code for dealing it for batch inference:

llm = LLM(
    model=name,
    trust_remote_code=True,
    tensor_parallel_size=2,
)
tokenizer = llm.get_tokenizer()

conversations = tokenizer.apply_chat_template(
    [{'role': 'user', 'content': 'hi, how are you?'}],
    tokenize=False,
)

outputs = llm.generate(
    [conversations],
    SamplingParams(
        temperature=0.5,
        top_p=0.9,
        max_tokens=1024,
        stop_token_ids=[tokenizer.eos_token_id, tokenizer.convert_tokens_to_ids("<|eot_id|>")],  # KEYPOINT HERE
    )
)

yes, in the README file, it is said we need to add terminators ([tokenizer.eos_token_id,tokenizer.convert_tokens_to_ids("<|eot_id|>")]) manually to stop the inference. I tried it (but in the mlx-lm library)and it works well.

[eav-solution]

How to. use with vllm-openai, please suggest me

[aliozts]

@eav-solution to use with docker image, I think you need to wait for 0.4.1 release and also the update from meta's huggingface repo from what I can say.

The respective PR #4182 has been merged but it will be released with 0.4.1. You can create a docker image and install vLLM from source, install your model too because you will need to change the generation_config.json with the eos_token -> [128001, 128009].

Or an alternative solution that is also confirmed by @njhill is to just changing the tokenizer_config.json and changing eos_token to <|eot_id|>. Regardless, you need to install from source and modify the model files respectively from what I can say, I hope this helps but please take these with a pinch of salt and confirm them by taking a look at this issue and the model's repo files/discussion in HF.

[eav-solution]

That great! Thanks!!!!!!

[ericg108]

@aliozts Hi Ali, I'm doing the same thing. But I got unexpected answer with <|im_end|> and <|im_start|>.

I guess I'm using the wrong chat template. Can you share your chat template if I may ask? thanks a lot