Default guidellm command only sends empty messages?

[mgoin]

Server:

vllm serve mistralai/Mistral-7B-Instruct-v0.3

Client:

guidellm --target "http://localhost:8000/v1" --model mistralai/Mistral-7B-Instruct-v0.3

These are some of the logs from my vLLM server:

INFO 08-27 16:00:20 logger.py:36] Received request chat-e82ea5381130441088e4bee0ed30630e: prompt: '<s>', params: SamplingParams(n=1, best_of=1, presence_penalty=0.0, frequency_penalty=0.0, repetition_penalty=1.0, temperature=0.7, top_p=1.0, top_k=-1, min_p=0.0, seed=None, use_beam_search=False, length_penalty=1.0, early_stopping=False, stop=[], stop_token_ids=[], include_stop_str_in_output=False, ignore_eos=False, max_tokens=256, min_tokens=0, logprobs=None, prompt_logprobs=None, skip_special_tokens=True, spaces_between_special_tokens=True, truncate_prompt_tokens=None), prompt_token_ids: [1], lora_request: None, prompt_adapter_request: None.
INFO:     127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO 08-27 16:00:20 async_llm_engine.py:208] Added request chat-e82ea5381130441088e4bee0ed30630e.
INFO 08-27 16:00:20 logger.py:36] Received request chat-33ae83c188894acb930b69c7cb1fd56d: prompt: '<s>', params: SamplingParams(n=1, best_of=1, presence_penalty=0.0, frequency_penalty=0.0, repetition_penalty=1.0, temperature=0.7, top_p=1.0, top_k=-1, min_p=0.0, seed=None, use_beam_search=False, length_penalty=1.0, early_stopping=False, stop=[], stop_token_ids=[], include_stop_str_in_output=False, ignore_eos=False, max_tokens=256, min_tokens=0, logprobs=None, prompt_logprobs=None, skip_special_tokens=True, spaces_between_special_tokens=True, truncate_prompt_tokens=None), prompt_token_ids: [1], lora_request: None, prompt_adapter_request: None.
INFO:     127.0.0.1:36208 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO 08-27 16:00:20 async_llm_engine.py:208] Added request chat-33ae83c188894acb930b69c7cb1fd56d.
INFO 08-27 16:00:20 async_llm_engine.py:176] Finished request chat-2368c52c95d84e238b97f17e28341af2.
INFO 08-27 16:00:20 logger.py:36] Received request chat-5575071f897f4ec9be95754d63fc937b: prompt: '<s>', params: SamplingParams(n=1, best_of=1, presence_penalty=0.0, frequency_penalty=0.0, repetition_penalty=1.0, temperature=0.7, top_p=1.0, top_k=-1, min_p=0.0, seed=None, use_beam_search=False, length_penalty=1.0, early_stopping=False, stop=[], stop_token_ids=[], include_stop_str_in_output=False, ignore_eos=False, max_tokens=256, min_tokens=0, logprobs=None, prompt_logprobs=None, skip_special_tokens=True, spaces_between_special_tokens=True, truncate_prompt_tokens=None), prompt_token_ids: [1], lora_request: None, prompt_adapter_request: None.
INFO:     127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK

You can see from the prompt: '<s>' entry that all of the prompts are just the BOS token, which has probably been added by the tokenizer.

Is this intentional that we should define some range of prompt lengths?

[mgoin]

This issue doesn't occur for the specific model on the README, so it seems the issue is with assuming the model is Llama 3.1. Possibly the tokenizer is the issue?

vllm serve neuralmagic/Meta-Llama-3.1-8B-Instruct-quantized.w4a16
guidellm --target "http://localhost:8000/v1" --model neuralmagic/Meta-Llama-3.1-8B-Instruct-quantized.w4a16

INFO 08-27 16:56:32 logger.py:36] Received request chat-e3e41409778c497a993c1a3abdb361e8: prompt: '<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n\nmust be in want of a wife.\n\nHowever little known the feelings or views of such a man may be on his\nfirst entering a neighbourhood, this truth is so well fixed in the minds\nof the surrounding families, that he is considered as the rightful\nproperty of some one or other of their daughters.\n\n"My dear Mr. Bennet," said his lady to him one day, "have you heard that\nNetherfield Park is let at last?"\n\nMr. Bennet replied that he had not.\n\n"But it is," returned she; "for Mrs. Long has just been here, and she\ntold me all about it."\n\nMr. Bennet made no answer.\n\n"Do not you want to know who has taken it?" cried his wife, impatiently.\n\n"_You_ want to tell me, and I have no objection to hearing it."\n\n[Illustration:\n\n"He came down to see the place"\n\n[_Copyright 1894 by George Allen._]]\n\nThis was invitation enough.\n\n"Why, my dear, you must know, Mrs. Long says that Netherfield is taken\nby a young man of large fortune from the north of England; that he came\ndown on Monday in a chaise and four to see the place, and was so much\ndelighted with it that It is a truth universally acknowledged, that a single man in possession\nof a good fortune must be in want of a wife.\n\nHowever little known the feelings or views of such a man may be on his\nfirst entering a neighbourhood, this truth is so well fixed in the minds\nof the surrounding families, that he is considered as the rightful\nproperty of some one or other of their daughters.\n\n"My dear Mr. Bennet," said his lady to him one day, "have you heard that\nNetherfield Park is let at last?"\n\nMr. Bennet replied that he had not.\n\n"But it is," returned she; "for Mrs. Long has just been here, and she\ntold me all about it."\n\nMr. Bennet made no answer.\n\n"Do not you want to know who has taken it?" cried his wife, impatiently.\n\n"_You_ want to tell me, and I have no objection to hearing it."\n\n[Illustration:\n\n"He came down to see the place"\n\n[_Copyright 1894 by George Allen._]]\n\nThis was invitation enough.\n\n"Why, my dear, you must know, Mrs. Long says that Netherfield is taken\nby a young man of large fortune from the north of England; that he came\ndown on Monday in a chaise and four to see the place, and was so much\ndelighted with it that It is a truth universally acknowledged, that a single man in possession\nof a good fortune must be in want of a wife.\n\nHowever little known the feelings or views of such a man may be on his\nfirst entering a neighbourhood, this truth is so well fixed in the minds\nof the surrounding families, that he is considered as the rightful\nproperty of some one or other of their daughters.\n\n"My dear Mr. Bennet," said his lady to him one day, "have you heard that\nNetherfield Park is let at last?"\n\nMr. Bennet replied that he had not.\n\n"But it is," returned she; "for Mrs. Long has just been here, and she\ntold me all about it."\n\nMr. Bennet made no answer.\n\n"Do not you want to know who has taken it?" cried his wife, impatiently.\n\n"_You_ want to tell me, and I have no objection to hearing it."\n\n[Illustration:\n\n"He came down to see the place"\n\n[_Copyright 1894 by George Allen._]]\n\nThis was invitation enough.\n\n"Why, my dear, you must know, Mrs. Long says that Netherfield is taken\nby a young man of large fortune from the north of England; that he came\ndown on Monday in a chaise and four to see the place, and was so much\ndelighted with it that It is a truth universally acknowledged, that a single man in possession\nof a good fortune must be in want of a wife.\n\nHowever little known the feelings or views of such a man may be on his\nfirst entering a neighbourhood, this truth is so well fixed in the minds\nof the surrounding families, that he is considered as the rightful\nproperty of some one or other of their daughters.\n\n"My dear Mr. Bennet," said his lady to him one day, "have you heard that\nNetherfield Park is let at last?"\n\nMr. Bennet replied that he had not.\n\n"But it is," returned she; "for Mrs. Long has just been here, and she\ntold me all about it."\n\nMr. Bennet made no answer.\n\n"Do not you want to know who has taken it?" cried his wife, impatiently.\n\n"_You_ want to tell me, and I have no objection to hearing it."\n\n[Illustration:\n\n"He came down to see the place"\n\n[_Copyright<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n', params: SamplingParams(n=1, best_of=1, presence_penalty=0.0, frequency_penalty=0.0, repetition_penalty=1.0, temperature=0.7, top_p=1.0, top_k=-1, min_p=0.0, seed=None, use_beam_search=False, length_penalty=1.0, early_stopping=False, stop=[], stop_token_ids=[], include_stop_str_in_output=False, ignore_eos=False, max_tokens=256, min_tokens=0, logprobs=None, prompt_logprobs=None, skip_special_tokens=True, spaces_between_special_tokens=True, truncate_prompt_tokens=None), prompt_token_ids: [128000, 128006, 9125, 128007, 271, 25849, 387, 304, 1390, 315, 264, 7555, 382, 11458, 2697, 3967, 279, 16024, 477, 6325, 315, 1778, 264, 893, 1253, 387, 389, 813, 198, 3983, 16661, 264, 40442, 11, 420, 8206, 374, 779, 1664, 8521, 304, 279, 20663, 198, 1073, 279, 14932, 8689, 11, 430, 568, 374, 6646, 439, 279, 99044, 198, 3784, 315, 1063, 832, 477, 1023, 315, 872, 30968, 382, 46240, 25237, 4491, 13, 30880, 295, 1359, 1071, 813, 17240, 311, 1461, 832, 1938, 11, 330, 19553, 499, 6755, 430, 198, 45, 2791, 2630, 5657, 374, 1095, 520, 1566, 12241, 12555, 13, 30880, 295, 20592, 430, 568, 1047, 539, 382, 42513, 433, 374, 1359, 6052, 1364, 26, 330, 2000, 18083, 13, 5843, 706, 1120, 1027, 1618, 11, 323, 1364, 198, 83, 820, 757, 682, 922, 433, 2266, 12555, 13, 30880, 295, 1903, 912, 4320, 382, 91724, 539, 499, 1390, 311, 1440, 889, 706, 4529, 433, 7673, 39169, 813, 7555, 11, 74797, 398, 382, 36189, 2675, 62, 1390, 311, 3371, 757, 11, 323, 358, 617, 912, 46655, 311, 11011, 433, 2266, 46512, 657, 3497, 367, 1473, 27423, 3782, 1523, 311, 1518, 279, 2035, 1875, 13804, 13857, 220, 9378, 19, 555, 10058, 20661, 1462, 43550, 2028, 574, 29788, 3403, 382, 77955, 11, 856, 25237, 11, 499, 2011, 1440, 11, 18083, 13, 5843, 2795, 430, 452, 2791, 2630, 374, 4529, 198, 1729, 264, 3995, 893, 315, 3544, 33415, 505, 279, 10411, 315, 9635, 26, 430, 568, 3782, 198, 2996, 389, 7159, 304, 264, 523, 4105, 323, 3116, 311, 1518, 279, 2035, 11, 323, 574, 779, 1790, 198, 9783, 65607, 449, 433, 430, 1102, 374, 264, 8206, 61528, 26579, 11, 430, 264, 3254, 893, 304, 19243, 198, 1073, 264, 1695, 33415, 2011, 387, 304, 1390, 315, 264, 7555, 382, 11458, 2697, 3967, 279, 16024, 477, 6325, 315, 1778, 264, 893, 1253, 387, 389, 813, 198, 3983, 16661, 264, 40442, 11, 420, 8206, 374, 779, 1664, 8521, 304, 279, 20663, 198, 1073, 279, 14932, 8689, 11, 430, 568, 374, 6646, 439, 279, 99044, 198, 3784, 315, 1063, 832, 477, 1023, 315, 872, 30968, 382, 46240, 25237, 4491, 13, 30880, 295, 1359, 1071, 813, 17240, 311, 1461, 832, 1938, 11, 330, 19553, 499, 6755, 430, 198, 45, 2791, 2630, 5657, 374, 1095, 520, 1566, 12241, 12555, 13, 30880, 295, 20592, 430, 568, 1047, 539, 382, 42513, 433, 374, 1359, 6052, 1364, 26, 330, 2000, 18083, 13, 5843, 706, 1120, 1027, 1618, 11, 323, 1364, 198, 83, 820, 757, 682, 922, 433, 2266, 12555, 13, 30880, 295, 1903, 912, 4320, 382, 91724, 539, 499, 1390, 311, 1440, 889, 706, 4529, 433, 7673, 39169, 813, 7555, 11, 74797, 398, 382, 36189, 2675, 62, 1390, 311, 3371, 757, 11, 323, 358, 617, 912, 46655, 311, 11011, 433, 2266, 46512, 657, 3497, 367, 1473, 27423, 3782, 1523, 311, 1518, 279, 2035, 1875, 13804, 13857, 220, 9378, 19, 555, 10058, 20661, 1462, 43550, 2028, 574, 29788, 3403, 382, 77955, 11, 856, 25237, 11, 499, 2011, 1440, 11, 18083, 13, 5843, 2795, 430, 452, 2791, 2630, 374, 4529, 198, 1729, 264, 3995, 893, 315, 3544, 33415, 505, 279, 10411, 315, 9635, 26, 430, 568, 3782, 198, 2996, 389, 7159, 304, 264, 523, 4105, 323, 3116, 311, 1518, 279, 2035, 11, 323, 574, 779, 1790, 198, 9783, 65607, 449, 433, 430, 1102, 374, 264, 8206, 61528, 26579, 11, 430, 264, 3254, 893, 304, 19243, 198, 1073, 264, 1695, 33415, 2011, 387, 304, 1390, 315, 264, 7555, 382, 11458, 2697, 3967, 279, 16024, 477, 6325, 315, 1778, 264, 893, 1253, 387, 389, 813, 198, 3983, 16661, 264, 40442, 11, 420, 8206, 374, 779, 1664, 8521, 304, 279, 20663, 198, 1073, 279, 14932, 8689, 11, 430, 568, 374, 6646, 439, 279, 99044, 198, 3784, 315, 1063, 832, 477, 1023, 315, 872, 30968, 382, 46240, 25237, 4491, 13, 30880, 295, 1359, 1071, 813, 17240, 311, 1461, 832, 1938, 11, 330, 19553, 499, 6755, 430, 198, 45, 2791, 2630, 5657, 374, 1095, 520, 1566, 12241, 12555, 13, 30880, 295, 20592, 430, 568, 1047, 539, 382, 42513, 433, 374, 1359, 6052, 1364, 26, 330, 2000, 18083, 13, 5843, 706, 1120, 1027, 1618, 11, 323, 1364, 198, 83, 820, 757, 682, 922, 433, 2266, 12555, 13, 30880, 295, 1903, 912, 4320, 382, 91724, 539, 499, 1390, 311, 1440, 889, 706, 4529, 433, 7673, 39169, 813, 7555, 11, 74797, 398, 382, 36189, 2675, 62, 1390, 311, 3371, 757, 11, 323, 358, 617, 912, 46655, 311, 11011, 433, 2266, 46512, 657, 3497, 367, 1473, 27423, 3782, 1523, 311, 1518, 279, 2035, 1875, 13804, 13857, 220, 9378, 19, 555, 10058, 20661, 1462, 43550, 2028, 574, 29788, 3403, 382, 77955, 11, 856, 25237, 11, 499, 2011, 1440, 11, 18083, 13, 5843, 2795, 430, 452, 2791, 2630, 374, 4529, 198, 1729, 264, 3995, 893, 315, 3544, 33415, 505, 279, 10411, 315, 9635, 26, 430, 568, 3782, 198, 2996, 389, 7159, 304, 264, 523, 4105, 323, 3116, 311, 1518, 279, 2035, 11, 323, 574, 779, 1790, 198, 9783, 65607, 449, 433, 430, 1102, 374, 264, 8206, 61528, 26579, 11, 430, 264, 3254, 893, 304, 19243, 198, 1073, 264, 1695, 33415, 2011, 387, 304, 1390, 315, 264, 7555, 382, 11458, 2697, 3967, 279, 16024, 477, 6325, 315, 1778, 264, 893, 1253, 387, 389, 813, 198, 3983, 16661, 264, 40442, 11, 420, 8206, 374, 779, 1664, 8521, 304, 279, 20663, 198, 1073, 279, 14932, 8689, 11, 430, 568, 374, 6646, 439, 279, 99044, 198, 3784, 315, 1063, 832, 477, 1023, 315, 872, 30968, 382, 46240, 25237, 4491, 13, 30880, 295, 1359, 1071, 813, 17240, 311, 1461, 832, 1938, 11, 330, 19553, 499, 6755, 430, 198, 45, 2791, 2630, 5657, 374, 1095, 520, 1566, 12241, 12555, 13, 30880, 295, 20592, 430, 568, 1047, 539, 382, 42513, 433, 374, 1359, 6052, 1364, 26, 330, 2000, 18083, 13, 5843, 706, 1120, 1027, 1618, 11, 323, 1364, 198, 83, 820, 757, 682, 922, 433, 2266, 12555, 13, 30880, 295, 1903, 912, 4320, 382, 91724, 539, 499, 1390, 311, 1440, 889, 706, 4529, 433, 7673, 39169, 813, 7555, 11, 74797, 398, 382, 36189, 2675, 62, 1390, 311, 3371, 757, 11, 323, 358, 617, 912, 46655, 311, 11011, 433, 2266, 46512, 657, 3497, 367, 1473, 27423, 3782, 1523, 311, 1518, 279, 2035, 1875, 13804, 13857, 128009, 128006, 78191, 128007, 271], lora_request: None, prompt_adapter_request: None.

[markurtz]

Closing this out now that the backend has been reworked and tested