[Kernel] Enable continuous batching on single GPU.

[changqi1]

OMP_NUM_THREADS=20 XFT_ENGINE=GPU:1 numactl -N 0 -m 0 ./cb_check --model /home/xfast/models/llama-2-7b-chat-xft/ --token /home/xfast/models/llama-2-7b-chat-hf/tokenizer.model --dtype fp16 --kv_cache_dtype fp16
[INFO] SINGLE_INSTANCE MODE.
[INFO] Model path is /home/xfast/models/llama-2-7b-chat-xft/
[INFO] Token path is /home/xfast/models/llama-2-7b-chat-hf/tokenizer.model
[INFO] Data type is fp16
[INFO] KV cache data type is fp16
[INFO] Input prompt: Once upon a time, there existed a little girl who liked to have adventures.
[INFO] Input Token Ids: 1 9038 2501 263 931 29892 727 22856 263 2217 7826 1058 23289 304 505 17623 1973 29889
Generated Tokens [0]: 2296 10600 297 263 2319 5720 22047
She lived in a small village surrounded
Generated Tokens [1]: 2296 10600 297 263 2319 5720 22047
She lived in a small village surrounded
Generated Tokens [2]: 2296 10600 297 263 2319
She lived in a small

$ OMP_NUM_THREADS=20 mpirun     -n 1 -env XFT_ENGINE=GPU:1 numactl -N 0 -m 0 python -m vllm.entrypoints.openai.api_server         --model /home/xfast/models/llama-2-7b-chat-xft/         --tokenizer  /home/xfast/models/llama-2-7b-chat-hf/         --dtype fp16 --kv-cache-dtype fp16 --served-model-name xft --port 8080 --trust-remote-code
INFO 06-18 16:16:56 metrics.py:334] Avg prompt throughput: 7.2 tokens/s, Avg generation throughput: 56.2 tokens/s, Running: 4 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%
INFO 06-18 16:17:01 metrics.py:334] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 54.3 tokens/s, Running: 4 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%
INFO 06-18 16:17:06 metrics.py:334] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 53.3 tokens/s, Running: 4 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%
INFO 06-18 16:17:11 metrics.py:334] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 50.0 tokens/s, Running: 4 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%
INFO 06-18 16:17:16 metrics.py:334] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 48.0 tokens/s, Running: 4 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%
INFO 06-18 16:17:21 metrics.py:334] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 48.4 tokens/s, Running: 4 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%
INFO 06-18 16:17:26 metrics.py:334] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 49.4 tokens/s, Running: 4 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%

$ curl http://localhost:8080/v1/completions -H "Content-Type: application/json" -d '{ "model": "xft", "prompt": "Once upon a time, there existed a little girl who liked to have adventures.", "max_tokens": 512, "temperature": 0 }' & \
  curl http://localhost:8080/v1/completions -H "Content-Type: application/json" -d '{ "model": "xft", "prompt": "Once upon a time, there existed a little girl who liked to have adventures.", "max_tokens": 512, "temperature": 0 }' & \
  curl http://localhost:8080/v1/completions -H "Content-Type: application/json" -d '{ "model": "xft", "prompt": "Once upon a time, there existed a little girl who liked to have adventures.", "max_tokens": 512, "temperature": 0 }' & \
  curl http://localhost:8080/v1/completions -H "Content-Type: application/json" -d '{ "model": "xft", "prompt": "Once upon a time, there existed a little girl who liked to have adventures.", "max_tokens": 512, "temperature": 0 }' &
{"id":"cmpl-49c7a55507cd480c981f4ac589e84030","object":"text_completion","created":1718727410,"model":"xft","choices":[{"index":0,"text":" She lived in a small village surrounded by vast forests, and she spent most of her days exploring the woods, discovering new things, and making friends with the creatures that lived there.\n\nOne day, while wandering deeper into the forest than she had ever gone before, she stumbled upon a hidden clearing. In the center of the clearing stood an enormous tree, its trunk as wide as a house and its branches stretching up towards the sky. The little girl was amazed by the tree's size and beauty, and she felt drawn to it.\n\nAs she approached the tree, she noticed that it was covered in strange symbols, etched into the bark. She couldn't read them, but they seemed to glow with a soft, otherworldly light. The little girl felt a sudden sense of wonder and excitement, and she knew that she had found something special.\n\nShe reached out a hand to touch the symbols, and as soon as she did, the tree began to glow even brighter. The little girl felt a strange sensation, as if she were being lifted off the ground and transported to a magical place. When she opened her eyes, she found herself standing in a beautiful, enchanted forest, filled with creatures she had never seen before.\n\nThe little girl was amazed and delighted by the wonders of this new place, and she knew that she had been transported to a magical world. She spent the rest of the day exploring the enchanted forest, making friends with the creatures she met, and discovering new and wonderful things.\n\nAs the sun began to set, the little girl returned to the hidden clearing, and the tree once again glowed with a soft, otherworldly light. The symbols on its bark seemed to be beckoning her, and she knew that she had been given a special gift.\n\nFrom that day on, the little girl returned to the hidden clearing whenever she wanted to go on an adventure. She knew that the tree would always transport her to magical places, filled with wonders and delights beyond her wildest dreams. And so, she continued to explore the enchanted world, always discovering new and exciting things, and always returning to the tree that had given her the gift of magic.","logprobs":null,"finish_reason":"stop","stop_reason":null}],"usage":{"prompt_tokens":18,"total_tokens":514,"completion_tokens":496}}
{"id":"cmpl-f62f75974d4c4b28b38573b7b2050a7f","object":"text_completion","created":1718727410,"model":"xft","choices":[{"index":0,"text":" She lived in a small village surrounded by vast forests, and she spent most of her days exploring the woods, discovering new things, and making friends with the creatures that lived there.\n\nOne day, while wandering deeper into the forest than she had ever gone before, she stumbled upon a hidden clearing. In the center of the clearing stood an enormous tree, its trunk as wide as a house and its branches stretching up towards the sky. The little girl was amazed by the tree's size and beauty, and she felt drawn to it.\n\nAs she approached the tree, she noticed that it was covered in strange symbols, etched into the bark. She couldn't read them, but they seemed to glow with a soft, otherworldly light. The little girl felt a sudden sense of wonder and excitement, and she knew that she had found something special.\n\nShe reached out a hand to touch the symbols, and as soon as she did, the tree began to glow even brighter. The little girl felt a strange sensation, as if she were being lifted off the ground and transported to a magical place. When she opened her eyes, she found herself standing in a beautiful, enchanted forest, filled with creatures she had never seen before.\n\nThe little girl was amazed and delighted by the wonders of this new place, and she knew that she had been transported to a magical world. She spent the rest of the day exploring the enchanted forest, making friends with the creatures she met, and discovering new and wonderful things.\n\nAs the sun began to set, the little girl returned to the hidden clearing, and the tree once again glowed with a soft, otherworldly light. The symbols on its bark seemed to be beckoning her, and she knew that she had been given a special gift.\n\nFrom that day on, the little girl returned to the hidden clearing whenever she wanted to go on an adventure. She knew that the tree would always transport her to magical places, filled with wonders and delights beyond her wildest dreams. And so, she continued to explore the enchanted world, always discovering new and exciting things, and always returning to the tree that had given her the gift of magic.","logprobs":null,"finish_reason":"stop","stop_reason":null}],"usage":{"prompt_tokens":18,"total_tokens":514,"completion_tokens":496}}
{"id":"cmpl-2925927017a24730ad27bf5d8510d9d5","object":"text_completion","created":1718727410,"model":"xft","choices":[{"index":0,"text":" She lived in a small village surrounded by vast forests, and she spent most of her days exploring the woods, discovering new things, and making friends with the creatures that lived there.\n\nOne day, while wandering deeper into the forest than she had ever gone before, she stumbled upon a hidden clearing. In the center of the clearing stood an enormous tree, its trunk as wide as a house and its branches stretching up towards the sky. The little girl was amazed by the tree's size and beauty, and she felt drawn to it.\n\nAs she approached the tree, she noticed that it was covered in strange symbols, etched into the bark. She couldn't read them, but they seemed to glow with a soft, otherworldly light. The little girl felt a sudden sense of wonder and excitement, and she knew that she had found something special.\n\nShe reached out a hand to touch the symbols, and as soon as she did, the tree began to glow even brighter. The little girl felt a strange sensation, as if she were being lifted off the ground and transported to a magical place. When she opened her eyes, she found herself standing in a beautiful, enchanted forest, filled with creatures she had never seen before.\n\nThe little girl was amazed and delighted by the wonders of this new place, and she knew that she had been transported to a magical world. She spent the rest of the day exploring the enchanted forest, making friends with the creatures she met, and discovering new and wonderful things.\n\nAs the sun began to set, the little girl returned to the hidden clearing, and the tree once again glowed with a soft, otherworldly light. The symbols on its bark seemed to be beckoning her, and she knew that she had been given a special gift.\n\nFrom that day on, the little girl returned to the hidden clearing whenever she wanted to go on an adventure. She knew that the tree would always transport her to magical places, filled with wonders and delights beyond her wildest dreams. And so, she continued to explore the enchanted world, always discovering new and exciting things, and always returning to the tree that had given her the gift of magic.","logprobs":null,"finish_reason":"stop","stop_reason":null}],"usage":{"prompt_tokens":18,"total_tokens":514,"completion_tokens":496}}
{"id":"cmpl-86c4fe41c86141c6b05614a321fded44","object":"text_completion","created":1718727410,"model":"xft","choices":[{"index":0,"text":" She lived in a small village surrounded by vast forests, and she spent most of her days exploring the woods, discovering new things, and making friends with the creatures that lived there.\n\nOne day, while wandering deeper into the forest than she had ever gone before, she stumbled upon a hidden clearing. In the center of the clearing stood an enormous tree, its trunk as wide as a house and its branches stretching up towards the sky. The little girl was amazed by the tree's size and beauty, and she felt drawn to it.\n\nAs she approached the tree, she noticed that it was covered in strange symbols, etched into the bark. She couldn't read them, but they seemed to glow with a soft, otherworldly light. The little girl felt a sudden sense of wonder and excitement, and she knew that she had found something special.\n\nShe reached out a hand to touch the symbols, and as soon as she did, the tree began to glow even brighter. The little girl felt a strange sensation, as if she were being lifted off the ground and transported to a magical place. When she opened her eyes, she found herself standing in a beautiful, enchanted forest, filled with creatures she had never seen before.\n\nThe little girl was amazed and delighted by the wonders of this new place, and she knew that she had been transported to a magical world. She spent the rest of the day exploring the enchanted forest, making friends with the creatures she met, and discovering new and wonderful things.\n\nAs the sun began to set, the little girl returned to the hidden clearing, and the tree once again glowed with a soft, otherworldly light. The symbols on its bark seemed to be beckoning her, and she knew that she had been given a special gift.\n\nFrom that day on, the little girl returned to the hidden clearing whenever she wanted to go on an adventure. She knew that the tree would always transport her to magical places, filled with wonders and delights beyond her wildest dreams. And so, she continued to explore the enchanted world, always discovering new and exciting things, and always returning to the tree that had given her the gift of magic.","logprobs":null,"finish_reason":"stop","stop_reason":null}],"usage":{"prompt_tokens":18,"total_tokens":514,"completion_tokens":496}}