llama : add batched inference endpoint to server

[niubi-AI]

for those not familiar with C like me.
it would be great if a new endpoint added to server.cpp to make batch inference.
for example:
endpoint: /completions
post: {"prompts":["promptA","promptB","promptC"]}
response:{"results":["sequenceA","sequenceB","sequenceC"]}

it is easy to do so with Hugging Face Transformers (as i do right now), but it's quite inefficient，hope to use llama.cpp to increase the efficiency oneday, cause I am not familiar with C, so can not use baby llama. I can only use javascript to Interact data with server.cpp。

[PenutChen]

Since there are many efficient quantization levels in llama.cpp, adding batch inference and continuous batching to the server will make it highly competitive with other inference frameworks like vllm or hf-tgi.

[niubi-AI]

Since there are many efficient quantization levels in llama.cpp, adding batch inference and continuous batching to the server will make it highly competitive with other inference frameworks like vllm or hf-tgi.

Yes, vllm and agi seem to be not available on windows。with transformers a batch of 10 sequences costs about 25 seconds, i think it would just costs 15 seconds if with llama.cpp. i have no idea cause i have not tested successfully.

[IridiumMaster]

I would also be interested in this one

[ggerganov]

Fixed via #3589 #3677

[yanndupis]

Thank you @ggerganov for adding this feature.

Maybe I am missing something; when I give an array of prompts to the request as described in the README, I will get the response only for the last element of the array instead of for each prompt in the array. Here is an example to reproduce:

Server:

./server --model mistral-7b-instruct-v0.1.Q4_0.gguf --port 8080 -c 8092 -cb

Client:

curl --request POST \
--url http://localhost:8080/completion \
--header "Content-Type: application/json" \
--data '{"prompt": ["<s>[INST] What is the capital of the US? [/INST]", "<s>[INST] What is the capital of France? [/INST]"], "n_predict": 2048}'

Output:

{"content":" The capital of France is Paris.","generation_settings":{"frequency_penalty":0.0,"grammar":"","ignore_eos":false,"logit_bias":[],"min_p":0.05000000074505806,"mirostat":0,"mirostat_eta":0.10000000149011612,"mirostat_tau":5.0,"model":"mistral-7b-instruct-v0.1.Q4_0.gguf","n_ctx":8092,"n_keep":0,"n_predict":2048,"n_probs":0,"penalize_nl":true,"presence_penalty":0.0,"repeat_last_n":64,"repeat_penalty":1.100000023841858,"seed":4294967295,"stop":[],"stream":false,"temp":0.800000011920929,"tfs_z":1.0,"top_k":40,"top_p":0.949999988079071,"typical_p":1.0},"model":"mistral-7b-instruct-v0.1.Q4_0.gguf","prompt":["<s>[INST] What is the capital of the US? [/INST]","<s>[INST] What is the capital of France? [/INST]"],"slot_id":0,"stop":true,"stopped_eos":true,"stopped_limit":false,"stopped_word":false,"stopping_word":"","timings":{"predicted_ms":167.096,"predicted_n":7,"predicted_per_second":41.892085986498785,"predicted_per_token_ms":23.870857142857144,"prompt_ms":1600.153,"prompt_n":36,"prompt_per_second":22.49784864322349,"prompt_per_token_ms":44.44869444444444},"tokens_cached":43,"tokens_evaluated":36,"tokens_predicted":7,"truncated":false}

I am on main (8e672ef)

Any idea? thank you!