Tasks queue logic doesn't seem to be logical

[tikikun]

Prerequisites

Please answer the following questions for yourself before submitting an issue.

• I am running the latest code. Development is very rapid so there are no tagged versions as of now.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new bug or useful enhancement to share.

Feature Description

https://github.com/ggerganov/llama.cpp/blob/2b3a665d3917edf393761a24c4835447894df74a/examples/server/server.cpp#L1558

Motivation

Task queue should only erase task "after" the task was resolved, with current implementation, the task queue does not allow user to know how many tasks are being resolved at the same time + other possible limitations.

Possible Implementation

Delete the task after it was resolved

[ngxson]

I'm not sure if I understand it correctly:

task_server task = queue_tasks.front(); ==> This line get the first task in the front of the queue

queue_tasks.erase(queue_tasks.begin()); ==> This line delete the first task in the front of the queue

So the task does get deleted, but it just being deleted before it is processed, not after it is processed, right?

---

In any cases, I'm agree that the current behavior (returning "slot unavailable" message) does break OpenAI-compatible in some cases.

Ideally, if all slots are unavailable, we can just delay the task until one become unavailable. I already try this by using a proxy and it works, but it would be nice if the behavior can be implemented in server.cpp, and be controlled via an argument.

[tikikun]

I'm not sure if I understand it correctly:

task_server task = queue_tasks.front(); ==> This line get the first task in the front of the queue

queue_tasks.erase(queue_tasks.begin()); ==> This line delete the first task in the front of the queue

So the task does get deleted, but it just being deleted before it is processed, not after it is processed, right?

In any cases, I'm agree that the current behavior (returning "slot unavailable" message) does break OpenAI-compatible in some cases.

Ideally, if all slots are unavailable, we can just delay the task until one become unavailable. I already try this by using a proxy and it works, but it would be nice if the behavior can be implemented in server.cpp, and be controlled via an argument.

You got the point, i have many issue with this example

[ngxson]

I did a small patch here, but I'm not sure if it will get merged: #5018

[huliangbing]

Great! Queue is needed!

[huliangbing]

When will #5018 be merged?
Thanks!

[github-actions]

This issue is stale because it has been open for 30 days with no activity.

[github-actions]

This issue was closed because it has been inactive for 14 days since being marked as stale.