Eval bug: llama-server stopped working after PR #11285 got merged

[tim-janik]

Name and Version

llama-server f30f099

Operating systems

Linux

GGML backends

CUDA

Hardware

RTX 4090, CUDA

Models

E.g. Code Qwen 2.5 7B-Chat (Q8)

Problem description & steps to reproduce

llama-server stopped generating any tokens for me, regardless of model, starting with commit f30f099 from #11285.
Simply reverting the above commit, e.g. on top of todays master (6171c9d) does fix the issue for me.

To reproduce, goto http://localhost:8080, enter a question hit return, nothing happens.

First Bad Commit

f30f099

Relevant log output

│ main: server is listening on http://0.0.0.0:8080 - starting the main loop                                                                                                                                                                                 │                
             │ srv  update_slots: all slots are idle                                                                                                                                                                                                                     │                
             │ request: GET / 127.0.0.1 200                                                                                                                                                                                                                              │                
             │ request: GET /favicon.ico  400                                                                                                                                                                                                                            │                
             │ request: POST /v1/chat/completions  400                                                                                                                                                                                                                   │

[ngxson]

Can you send the request via curl to see what is the response? (Or maybe see the response from browser's devtool)

From what I saw on your log, it responses with status code 400, meaning ERROR_TYPE_INVALID_REQUEST. This could potentially be due to upstream changes from httplib library, not a bug from llama-server.

[ngxson]

I'm not able to reproduce the bug on either my laptop (macbook M3) or on server (linux - nvidia T4)

[ngxson]

I did more tests but still can't reproduce the issue. Please provide more info about your setup:

• What is the browser you're using
• What happen when the request is sent via curl or wget or postman
• What show up if you access some other endpoints, for example /api/models

Also from your log, request: GET /favicon.ico 400, this means even non-API paths returns 400 error which is wrong. It's further strengthen my suspicion about bug in httplib.

[tim-janik]

I did more tests but still can't reproduce the issue. Please provide more info about your setup:

* What is the browser you're using

* What happen when the request is sent via `curl` or `wget` or postman

* What show up if you access some other endpoints, for example `/api/models`

Also from your log, request: GET /favicon.ico 400, this means even non-API paths returns 400 error which is wrong. It's further strengthen my suspicion about bug in httplib.

Thanks for the hint. I reverted f30f099 and only applied the httplib 0.18.5 change.
Turns out I can only trigger this issue if the URL contains a %0A, like this:

http://localhost:8080/?something=%0A

Which explains why only I saw it, I probably still had some q= or m= arg left over from playing with #11150 (btw, I'd appreciate answers to the two questions I posed in that PR if you could afford the time).

Which means this httplib change breaks multi-line use cases of #11150, do you have any idea why the newer httplib would break with URL encoded newlines?

[ngxson]

Ok sorry I forgot about #11150, I'll have a look later because it's not the priority.

I'm tagging author of httplib @yhirose too, maybe you have an idea why it returns 400 even for non-handler endpoints? (For example, /favicon.ico that should have returned 404 instead of 400)

[ngxson]

OK I think I pinpointed the problem. It only happens on OPTIONS request, which causes the memory to be corrupted somewhere. I repeatedly send OPTIONS request and got this in the log:

request:    400
request:    400
request: OPTIONS /tokenize 127.0.0.1 200
request: ndled   400
request: root   400
request: root   400

[ngxson]

Related to yhirose/cpp-httplib#2028

[tim-janik]

I could bisect the culprit here. It is this commit:
yhirose/cpp-httplib@975cf0d
Due to this issue:
yhirose/cpp-httplib#1908
Released with cpp-httplib v0.17.1.

The issue can be circumvented with the following patch:

diff --git a/examples/server/httplib.h b/examples/server/httplib.h
index c2f12dd2..37d75e40 100644
--- a/examples/server/httplib.h
+++ b/examples/server/httplib.h
@@ -4225,7 +4225,7 @@ inline bool parse_header(const char *beg, const char *end, T fn) {
     // those characters with SP before further processing or
     // forwarding of that message.
     static const std::string CR_LF_NUL("\r\n\0", 3);
-    if (val.find_first_of(CR_LF_NUL) != std::string::npos) { return false; }
+    // allow \n in URLs // if (val.find_first_of(CR_LF_NUL) != std::string::npos) { return false; }
 
     fn(key, val);
     return true;

@yhirose, does cpp-httplib really need to refuse any connection that contains %0a in the URL?
I mean, www.google.com/search?q=cpp-httplib%0A does work for example...

[yhirose]

@tim-janik thanks for the report. I have decided to reject a request including CR, LF, or NUL within a field value. As you can see in the first comment from @kenballus in #1908, I can add an option to replace the characters with SP according to RFC 9110.

Anyway, could you send an issue for the request to the cpp-httplib project?

@ngxson is yhirose/cpp-httplib#2028 related to what @tim-janik mentioned?

[tim-janik]

@tim-janik thanks for the report. I have decided to reject a request including CR, LF, or NUL within a field value. As you can see in the first comment from @kenballus in #1908, I can add an option to replace the characters with SP according to RFC 9110.

Anyway, could you send an issue for the request to the cpp-httplib project?

Sure: yhirose/cpp-httplib#2033

[ngxson]

Sorry I've been busy the pass 2 days. Glad to hear that you found the root cause @yhirose ! I'll also follow the issue created by @tim-janik