curl: (56) Malformed encoding found in chunked-encoding #98

eqyiel · 2015-07-13T02:41:07Z

Some .org files fail to render in any browser when served by elnode. These files export without error, and the rendered html can be validated using http://validator.w3.org/. Only curl gives a clue about malformed encoding. It might just be that elnode is doing such a good job of being asynchronous that it tries to write data before it is available.

Tested on Emacs 24.4.1 with Org-mode 8.2.10 and elnode-0.9.9.8.8, also on Emacs 24.5.1 with Org-mode 8.2.10 and elnode-0.9.9.8.8 (same behaviour for both).

How to reproduce (from emacs -Q):

Install elnode somewhere. I temporarily moved my emacs.d and installed from marmalade.
Evaluate this snippet. ~/dotemacs/emacs.org is cloned from https://github.com/bzg/dotemacs (arbitrary large and public org file).

(require 'elnode)

(defun org2html (src)
  (let* ((org-inhibit-startup t)
         (visitingp (find-buffer-visiting src))
         (work-buffer (or visitingp (find-file-noselect src)))
         (html (with-current-buffer work-buffer
                 (org-export-as 'html nil t nil
                                '(:html-doctype "html5" :html-html5-fancy t)))))
    html))

(defun my-test-handler (httpcon)
  "Demonstration function"
  (let ((html (org2html "~/dotemacs/emacs.org")))
    (elnode-send-html httpcon html)))

(elnode-start 'my-test-handler :port 9999 :host "localhost")

Try to access in the terminal emulator:

~ $ curl localhost:9999
<!DOCTYPE html>
<html>
<head>
<title>bzg .emacs.el file</title>
--- SNIP ---
<p class="date">Created: 2015-07-13 Mon 11:42</p>
<p class="creator"><a href="http://www.gnu.org/software/emacs/">Emacs</a> 24.4.1 (<a href="http://orgmode.org">Org</a> mode 8.2.10)</p>
curl: (56) Malformed encoding found in chunked-encoding
<p class="validation"><a href="http://validator.w3.org/check?uri=referer">Validate<~ $

You will see something like the following in *elnode-server-error* buffer:

2015-07-13T11:34:35: elnode-http-start: starting HTTP response on *elnode-webserver-localhost:9999* <127.0.0.1:42417>                                                                                          
2015-07-13T11:34:35: elnode-http-send-string *elnode-webserver-localhost:9999* <127.0.0.1:42417> [[<!DOCTY...]]                                                                                                
2015-07-13T11:34:35: elnode-http-send-string *elnode-webserver-localhost:9999* <127.0.0.1:42417> [[]]   
2015-07-13T11:34:35: elnode--http-end ending socket *elnode-webserver-localhost:9999* <127.0.0.1:42417>

I see the same behaviour using this, and some of my own files that I don't want to share just yet.

Expected behaviour: same for that of smaller org files. Here's an example ~/test.org that I used:

#+title: test.org                                                                                       
#+author: Ruben Maher                                                                                   
#+date: <2015-07-13 Mon 09:25:26>                                                                       

* This is a test to show off =elnode=.

  It's a pretty cool webserver.

Re-evaluate my-test-handler, changing the argument to org2html:

(defun my-test-handler (httpcon)
  "Demonstration function"
  (let ((html (org2html "~/test.org")))
    (elnode-send-html httpcon html)))

Try to access in a terminal emulator:

~ $ curl localhost:9999
<!DOCTYPE html>
<html>
<head>
<title>test.org</title>
--- SNIP ---
<p class="date">Created: 2015-07-13 Mon 11:48</p>
<p class="creator"><a href="http://www.gnu.org/software/emacs/">Emacs</a> 24.4.1 (<a href="http://orgmode.org">Org</a> mode 8.2.10)</p>
<p class="validation"><a href="http://validator.w3.org/check?uri=referer">Validate</a></p>
</div>
</body>
</html>~ $

No problem. It renders correctly in a web browser. *elnode-server-error* now shows:

2015-07-13T11:48:59: elnode-http-start: starting HTTP response on *elnode-webserver-localhost:9999* <127.0.0.1:44466>
2015-07-13T11:48:59: elnode-http-send-string *elnode-webserver-localhost:9999* <127.0.0.1:44466> [[<!DOCTY...]]
2015-07-13T11:48:59: elnode-http-send-string *elnode-webserver-localhost:9999* <127.0.0.1:44466> [[]]
2015-07-13T11:48:59: elnode--http-end ending socket *elnode-webserver-localhost:9999* <127.0.0.1:44466>

The text was updated successfully, but these errors were encountered:

eqyiel · 2015-07-15T00:44:58Z

I found the problem. It's a one-line fix so I'll just post it here.

In elnode-http-send-string, (format %x (length str)) doesn't always equal (format %x (string-bytes str). elnode tells the client to expect (string-bytes str), but the chunk size in the header is determined by (length str)!

I guess this is a bug in Emacs C source code. (Edit: this behaviour is actually documented. See here and here, "if the string contains multibyte characters, this is not necessarily
the number of bytes in the string; it is the number of characters.").

Demonstration, using this file

(defun org2html (src)
  (let* ((org-inhibit-startup t)
         (visitingp (find-buffer-visiting src))
         (work-buffer (or visitingp (find-file-noselect src)))
         (html (with-current-buffer work-buffer
                 (set-buffer-file-coding-system 'utf-8)
                 (org-export-as 'html nil t nil
                                '(:html-doctype "html5" :html-html5-fancy t))
                 )))
    html))

(let ((thing (org2html "~/dev/emacs-notes/videos.org")))
  (message (concat "(string-bytes thing) is %x, (length thing) is %x. "
                   "That's %d bytes shorter than it should be!")
           (string-bytes thing) (length thing)
           (- (string-bytes thing) (length thing))))

;; => "(string-bytes thing) is 5dc5, (length thing) is 5dc3. That's 2 bytes
;; shorter than it should be!"

TL;DR: (format "%x\r\n%s\r\n" (length str) (or str ""))) should be (format "%x\r\n%s\r\n" len (or str ""))) (len already calculated in let)

joelmccracken · 2015-07-15T01:45:01Z

Unicode characters will cause this problem.

On Tue, Jul 14, 2015 at 8:44 PM, Ruben Maher notifications@github.com
wrote:

I found the problem. It's a one-line fix so I'll just post it here.

In elnode-http-send-string
https://github.com/nicferrier/elnode/blob/master/elnode.el#L1783, (format
%x (length str)) doesn't always equal (format %x (string-bytes str).
elnode tells the client to expect (string-bytes str), but the chunk size
in the header is determined by (length str)!

I guess this is a bug in Emacs C source code.

Demonstration, using this file
https://github.com/sachac/emacs-notes/blob/gh-pages/videos.org

(defun org2html (src)
(let* ((org-inhibit-startup t)
(visitingp (find-buffer-visiting src))
(work-buffer (or visitingp (find-file-noselect src)))
(html (with-current-buffer work-buffer
(set-buffer-file-coding-system 'utf-8)
(org-export-as 'html nil t nil
'(:html-doctype "html5" :html-html5-fancy t))
)))
html))

(let ((thing (org2html "~/dev/emacs-notes/videos.org")))
(message (concat "(string-bytes thing) is %x, (length thing) is %x. "
"That's %d bytes shorter than it should be!")
(string-bytes thing) (length thing)
(- (string-bytes thing) (length thing))))

;; => "(string-bytes thing) is 5dc5, (length thing) is 5dc3. That's 2 bytes
;; shorter than it should be!"

TL;DR: (format "%x\r\n%s\r\n" (length str) (or str ""))) should be (format
"%x\r\n%s\r\n" len (or str ""))) (len already calculated in let)

—
Reply to this email directly or view it on GitHub
#98 (comment).

nicferrier · 2015-07-15T10:31:17Z

Right. Not sure what elnode should do. Automatically convert? (how would we send bytes?) or make it clear that it's not really a string... it's a string of bytes?

eqyiel · 2015-07-15T14:24:35Z

Unless I've completely misunderstood something (wouldn't be the first time), there's no conversion necessary? The problem is just that size of the chunk reported in the header (%x\r\n) is not always correct when it is retrieved with length (if the string contained unicode characters for instance). The chunk still contains a string and the function still sends a string, but elnode should figure out the chunk size with string-bytes instead of length.

nicferrier · 2015-07-15T14:29:22Z

I think that's option 2, accept that send-string is sending bytes, not strings.

Either way it needs to change, but it could be either way.

joelmccracken · 2015-07-15T14:43:57Z

I think it should send the number of bytes.

On Wed, Jul 15, 2015 at 10:29 AM, Nic Ferrier notifications@github.com
wrote:

I think that's option 2, accept that send-string is sending bytes, not
strings.

Either way it needs to change, but it could be either way.

—
Reply to this email directly or view it on GitHub
#98 (comment).

jiri · 2017-02-11T15:19:04Z

What's the status on this? I just ran into this issue in 2017.

eqyiel · 2017-02-12T22:40:17Z

@jiri It's been a long time, but IIRC you need to monkeypatch elnode-http-send-string changing (length str) to (string-bytes str) on this line: https://github.com/nicferrier/elnode/blob/master/elnode.el#L1783

Edit: or you can just use the len which is let-bound, see TLDR here: #98 (comment)

jiri · 2017-02-15T18:27:12Z

Thanks!

I've been looking into making a patch for this, but there already is a pull request. It would be nice if @nicferrier either accepted it or appointed a maintainer of the package. I think elnode is useful enough that it should be maintained :)

That being said, the issue seems to have disappeared as I have not been able to replicate it since I restarted Emacs. If it ever surfaces, I'll bump this issue again :)

kosh04 mentioned this issue Oct 15, 2015

Send string as bytes #101

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

curl: (56) Malformed encoding found in chunked-encoding #98

curl: (56) Malformed encoding found in chunked-encoding #98

eqyiel commented Jul 13, 2015

eqyiel commented Jul 15, 2015

joelmccracken commented Jul 15, 2015

nicferrier commented Jul 15, 2015

eqyiel commented Jul 15, 2015

nicferrier commented Jul 15, 2015

joelmccracken commented Jul 15, 2015

jiri commented Feb 11, 2017

eqyiel commented Feb 12, 2017 •

edited

Loading

jiri commented Feb 15, 2017

curl: (56) Malformed encoding found in chunked-encoding #98

curl: (56) Malformed encoding found in chunked-encoding #98

Comments

eqyiel commented Jul 13, 2015

eqyiel commented Jul 15, 2015

joelmccracken commented Jul 15, 2015

nicferrier commented Jul 15, 2015

eqyiel commented Jul 15, 2015

nicferrier commented Jul 15, 2015

joelmccracken commented Jul 15, 2015

jiri commented Feb 11, 2017

eqyiel commented Feb 12, 2017 • edited Loading

jiri commented Feb 15, 2017

eqyiel commented Feb 12, 2017 •

edited

Loading