Server: format error to json

[ngxson]

This PR migrate some changes from #5776 and #5710

Errors are now formatted to JSON as suggested in #5882 (comment)

Examples

Unhandled exceptions (for example, invalid input JSON)

{
    "error": {
        "code": 500,
        "message": "[json.exception.parse_error.101] parse error at line 3, column 20: syntax error while parsing object - unexpected ':'; expected '}'",
        "type": "server_error"
    }
}

404 not found

{
    "error": {
        "code": 404,
        "message": "File Not Found",
        "type": "not_found_error"
    }
}

API key incorrect

{
    "error": {
        "code": 401,
        "message": "Invalid API Key",
        "type": "authentication_error"
    }
}

Endpoint is disabled

{
    "error": {
        "code": 501,
        "message": "This server does not support metrics endpoint.",
        "type": "not_supported_error"
    }
}

Invalid grammar

{
    "error": {
        "code": 400,
        "message": "Failed to parse grammar",
        "type": "invalid_request_error"
    }
}

Missing both "input" and "content" for /embeddings

{
    "error": {
        "code": 400,
        "message": "\"input\" or \"content\" must be provided",
        "type": "invalid_request_error"
    }
}

Missing both "prompt" and "messages" for /completions

{
    "error": {
        "code": 400,
        "message": "Either \"prompt\" or \"messages\" must be provided",
        "type": "invalid_request_error"
    }
}

NOTE:

• Empty prompt for completions / embeddings is not considered as error (since empty prompts will still have BOS token by default)
• The error code is quite messed up now (i.e. sometimes it should be 400 instead of 500). But for now I don't have a better way to receive errors inside run_slots(). This may be fixed in the future.

[ngxson]

Small note for @phymbert : Since we're using the correct OpenAI format now, the python test code will need to use correct exception classes - no actions needed from your side now, but just for reminding.

[z80maniac]

Empty prompt for completions / embeddings is not considered as error (since empty prompts will still have BOS token by default)

BOS token is only added when a set of conditions is satisfied. From the docs:

 A BOS token is inserted at the start, if all of the following conditions are true:

- The prompt is a string or an array with the first element given as a string
- The model's `tokenizer.ggml.add_bos_token` metadata is `true`
- The system prompt is empty

For example, as i showed here, an empty prompt specified as an array always returns an empty response:

❯ curl -sS --data '{"prompt": [], "n_predict": 4}' http://127.0.0.1:8080/completion | jq .content
""

I think it's counter-intuitive, and instead an error should be returned in this case.

[z80maniac]

This PR breaks the web UI error handling, so I think that all the changes in web UI from my PR (in completion.js and completion.js.hpp) can be moved here, since the error format remained the same.

Also, the README part can be taken from there as well. Though the formatting is a little bit confusing there (the "where" list blends with the "POST" section below), maybe needs to be done differently.

[ngxson]

@z80maniac Thanks for the suggestions, I took the completions.js file (and hpp) from your PR.

For the README, I added a dedicated section to specify that we use the same format as OpenAI. Only llama.cpp specific errors (for example: /slots, /metrics, grammar) will be specified in the docs.

[phymbert]

IMHO update_slots was clearer for me, the slots are not run, but updated from batch view.

[phymbert]

we should not get ride of the return type. Returning false might interrupt the main loop for example

[ngxson]

For now, the main loop only stop if running is set to false. So if the loop stop, we know exactly why.

Adding ability for update_slots to exit the main loop may make it harder to debug if there is an error inside update_slots

[phymbert]

I suggest // continue batch decoding loop

[ngxson]

In fact, the loop is to retry with llama_decode with smaller n_batch, until it successful.

For now I don't know how to rephrase the idea, so I just leave it here to distinguish it from the continue and break statements below. Since here we're inside a nested loop, I think having comments will be easier to navigate.

Or maybe we don't need these comments at all, do you have any input for that @ggerganov ?

[ngxson]

I decide to keep the comment this way, because the goal is to reflect the variable name, which helps when we navigate in a nested loop (but not to explain what the code does).

If you're ok with that, then I'll merge this PR @phymbert

[phymbert]

I suggest // continue updating next token of next slots or something similar

[z80maniac]

since the error format remained the same.

Turns out, while the error format did not change, its delivery to the web UI did. Now the error response is not wrapped in content field. Not sure if it's supposed to be this way, but anyway, here's a diff that fixes the web UI and also adds support for the code field: webui_fix.txt (can't attach *.diff)

[ngxson]

@z80maniac Thanks, it's updated in 0e72879