ev/read blocked indefinitely when called inside spork/http/server-handler

[l04m33]

Hi! I tried to write a simple "echo" server in HTTP using spork/httpf and spork/http, which simply sends back the POST body to the client. But the program sometimes got stuck in ev/read (See line 7 of the full program below).

I then tried to dig in, but it seems I can't quite grasp the complex fiber states, so I'm not sure whether I did something dumb or walked into a bug.

When testing, I used the curl command

curl -v --raw --form field=value http://127.0.0.1:8000

And this simple python program

import requests
requests.post('http://127.0.0.1:8000', b'x' * 5)

They all reproduced the problem.

The full Janet program is listed below, and I attached a Wireshark dump file capturing the problematic HTTP session (see it with filter tcp.stream eq 2): post.pcapng.zip

I appreciate any pointers. Thanks!

(import spork/httpf)
(import spork/http)

(defn read-byte-chunks [conn n]
  (var bytes-remaining n)
  (while (pos? bytes-remaining)
    (def bytes (ev/read conn bytes-remaining))
    #(def [bytes] (ev/gather (ev/read conn bytes-remaining)))
    (when (nil? bytes)
      (print (string/format "---- short read! %q bytes remaining" bytes-remaining))
      (break))
    (-= bytes-remaining (length bytes))
    (yield bytes)
    (print (string/format "---- bytes-remaining %q" bytes-remaining))))

(defn http-generic-handler [req]
  (def resp-headers
    @{"content-type" "text/plain"
      "server" "HTTP Generic Handler"})
  (def conn (in req :connection))
  (def content-length
    (let [clen-str (get (req :headers) "content-length")
          clen (if clen-str
                 (scan-number clen-str))]
      (or clen 0)))
  (print (string/format "---- Content-Length: %q" content-length))
  (var bytes-remaining content-length)
  {:status 200
   :headers resp-headers
   :body (coro (yield "Hello World!\r\n\r\n")
               (yield (string/format "%q\r\n\r\n" req))
               (print (string/format "---- going to read %q bytes" bytes-remaining))
               (read-byte-chunks conn bytes-remaining)
               )})

(defn main [&]
  (let [s (httpf/server)]
    (put s :on-connection
       (fn conn-handler [conn]
         (http/server-handler
          conn
          (-> http-generic-handler
              http/cookies
              http/logger))))
    
    (httpf/listen s "127.0.0.1" 8000)))

[l04m33]

I should have mentionend that this problem exists in both Linux (6.3.x) and Windows 10.

And the Janet version I used was 1.29.1.

[ianthehenry]

I believe it's hanging because your code doesn't take into account the number of bytes that have already been read from the socket when it decides to read. The Spork HTTP framework has to read from the socket at least once in order to parse the headers out, and it looks like it always reads in chunks of 4kb. So if you send a request with headers smaller than 4kb, part of the body has already been consumed (and put into (req :buffer)), so you can't read content-length more bytes from the socket.

Instead, you can use http/read-body, which takes into account the bytes already read and only issues additional read calls if it needs to.

[bakpakin]

You don't really need httpf for something like this:

(import spork/http)

(defn on-request
  [req &]
  (def body (http/read-body req))
  (printf "body: %q" body)
  {:status 200
   :headers {"content-type" "text/plain"}
   :body (slice body)}) # slice is needed right now to clone the buffer

(http/server on-request "127.0.0.1" "8000")

[bakpakin]

Also not entirely sure what the original code is trying to do, almost looks like you are trying to send a response before reading the header? Also using a coroutine body should encode the "/r/n" for you, although spork/http is pretty barebones so it doesn't do much in the way of validating payloads.

Strange code, almost looks HTTP request smuggling or some sort of protocol hack. Anyway, if you are a security researcher I'm curious about the results.

[l04m33]

Thank you for your replies @ianthehenry @bakpakin !

It's exactly the issue pointed out by @ianthehenry .

I looked into http/read-body before, but it didn't work while I used it as-is in the coroutine response body. So I tried to roll my own to simplify the code a bit and test things out.

It seems http/send-rensponse reuses the buffer from read-request, and server-handler clears that very buffer before calling send-response, so by the time my coroutine body is called, the buffered request body would already be gone. I think that and the fiber contexts got me confused :P

The original code is merely an answer to my own question, "What should I do If I want to pipe something through in the response body?"
I just thought the request body from the same HTTP session was the most convenient thing to generate and pipe through. It's weird but I think it doesn't break any standards, and looks legit on the client side too :) There weren't any security concerns.