bencode parsing issues

[hugoduncan]

I'm seeing some bencode parsing issues. This occurs when processing a stacktrace message following an exception thrown via load-file (though I don't think that is relevant).

This sometimes appears as:

Error: (scan-error "Unbalanced parentheses" 16597 1)

and sometimes as:

Error: (wrong-type-argument integer-or-marker-p nil)

In each case the problem seems to be related to emacs trying to process a bencoded message that has a length of 4096 bytes and is incomplete. The first error above seems to occur when the trucation occurs within a bencode header, the second when the trunction occurs in the message data.

[bbatsov]

We need to implement some explicit check for incomplete messages. The current code assumed that all decode errors were the result of incomplete messages, but that's not always the case.

[hugoduncan]

In both cases above, the result was that no stacktrace was shown, so processing the messages didn't complete.

[hugoduncan]

The unblanced parenthesis message is actually coming from cider-highlight-compilation-errors. Wrapping the body of cider-highlight-compilation-errors in with-demoted-errors results in the stacktrace actually getting displayed.

[hugoduncan]

And I believe the reason that cider-highlight-compilation-error is erroring is that the exception is thrown in macro generated code, and the line numbers and columns reported in the exception don't correspond to anything in the text of the source file.

[bbatsov]

I noticed something similar yesterday. Code evaluated with C-x C-e had some tmp filename in its stacktrace, when it used to have NO_FILE previously. No idea when did this change. Guess we should check whether the filename in the stacktrace matches our current filename.

[hugoduncan]

That's a good idea anyway, as a compilation error isn't always caused by code in the buffer being loaded.

[vspinu]

I am noticing the itneger-or-marker error when I evaluate (+ aa bb) in a new project. The backtrace is

Debugger entered--Lisp error: (wrong-type-argument integer-or-marker-p nil)
  goto-char(nil)
  (let ((start (point)) (end (byte-to-position (+ (position-bytes (point)) (string-to-number (match-string 1)))))) (goto-char end) (buffer-substring-no-properties start end))
  (cond ((looking-at "i\\(-?[0-9]+\\)e") (goto-char (match-end 0)) (string-to-number (match-string 1))) ((looking-at "\\([0-9]+\\):") (goto-char (match-end 0)) (let ((start (point)) (end (byte-to-position (+ (position-bytes ...) (string-to-number ...))))) (goto-char end) (buffer-substring-no-properties start end))) ((looking-at "l") (goto-char (match-end 0)) (let (result item) (while (not (eq :end (setq item (nrepl-bdecode-buffer)))) (setq result (cons item result))) (nreverse result))) ((looking-at "d") (goto-char (match-end 0)) (let (dict key item) (while (not (eq :end (setq item (nrepl-bdecode-buffer)))) (if key (setq dict (cons (cons key item) dict) key nil) (if (stringp item) nil (error "Dictionary keys have to be strings: %s" item)) (setq key item))) (cons (quote dict) (nreverse dict)))) ((looking-at "e") (goto-char (match-end 0)) :end) (t (error "Cannot decode message: %s" (buffer-substring-no-properties (point-min) (point-max)))))
  nrepl-bdecode-buffer()
  (setq item (nrepl-bdecode-buffer))
  (eq :end (setq item (nrepl-bdecode-buffer)))
  (not (eq :end (setq item (nrepl-bdecode-buffer))))
  (while (not (eq :end (setq item (nrepl-bdecode-buffer)))) (if key (setq dict (cons (cons key item) dict) key nil) (if (stringp item) nil (error "Dictionary keys have to be strings: %s" item)) (setq key item)))
  (let (dict key item) (while (not (eq :end (setq item (nrepl-bdecode-buffer)))) (if key (setq dict (cons (cons key item) dict) key nil) (if (stringp item) nil (error "Dictionary keys have to be strings: %s" item)) (setq key item))) (cons (quote dict) (nreverse dict)))
  (cond ((looking-at "i\\(-?[0-9]+\\)e") (goto-char (match-end 0)) (string-to-number (match-string 1))) ((looking-at "\\([0-9]+\\):") (goto-char (match-end 0)) (let ((start (point)) (end (byte-to-position (+ (position-bytes ...) (string-to-number ...))))) (goto-char end) (buffer-substring-no-properties start end))) ((looking-at "l") (goto-char (match-end 0)) (let (result item) (while (not (eq :end (setq item (nrepl-bdecode-buffer)))) (setq result (cons item result))) (nreverse result))) ((looking-at "d") (goto-char (match-end 0)) (let (dict key item) (while (not (eq :end (setq item (nrepl-bdecode-buffer)))) (if key (setq dict (cons (cons key item) dict) key nil) (if (stringp item) nil (error "Dictionary keys have to be strings: %s" item)) (setq key item))) (cons (quote dict) (nreverse dict)))) ((looking-at "e") (goto-char (match-end 0)) :end) (t (error "Cannot decode message: %s" (buffer-substring-no-properties (point-min) (point-max)))))
  nrepl-bdecode-buffer()
  (cons (nrepl-bdecode-buffer) result)
  (setq result (cons (nrepl-bdecode-buffer) result))
  (while (not (eobp)) (setq result (cons (nrepl-bdecode-buffer) result)))
  (let ((result (quote nil))) (while (not (eobp)) (setq result (cons (nrepl-bdecode-buffer) result))) (nreverse result))
  (progn (save-excursion (insert str)) (let ((result (quote nil))) (while (not (eobp)) (setq result (cons (nrepl-bdecode-buffer) result))) (nreverse result)))
  (unwind-protect (progn (save-excursion (insert str)) (let ((result (quote nil))) (while (not (eobp)) (setq result (cons (nrepl-bdecode-buffer) result))) (nreverse result))) (and (buffer-name temp-buffer) (kill-buffer temp-buffer)))
  (save-current-buffer (set-buffer temp-buffer) (unwind-protect (progn (save-excursion (insert str)) (let ((result (quote nil))) (while (not (eobp)) (setq result (cons (nrepl-bdecode-buffer) result))) (nreverse result))) (and (buffer-name temp-buffer) (kill-buffer temp-buffer))))
  (let ((temp-buffer (generate-new-buffer " *temp*"))) (save-current-buffer (set-buffer temp-buffer) (unwind-protect (progn (save-excursion (insert str)) (let ((result (quote nil))) (while (not (eobp)) (setq result (cons ... result))) (nreverse result))) (and (buffer-name temp-buffer) (kill-buffer temp-buffer)))))
  nrepl-decode("d5:class26:java.lang.RuntimeException2:id2:377:message44:Unable to resolve symbol: aa in this context7:session36:b6830485-f98d-459a-9461-d43d6b5f049bed5:class21:clojure.lang.Compiler4:file13:Compiler.java5:flagsl4:javae2:id2:374:linei6380e6:method7:analyze4:name29:clojure.lang.Compiler/analyze7:session36:b6830485-f98d-459a-9461-d43d6b5f049b4:type4:javaed5:class21:clojure.lang.Compiler4:file13:Compiler.java5:flagsl3:dup4:javae2:id2:374:linei6322e6:method7:analyze4:name29:clojure.lang.Compiler/analyze7:session36:b6830485-f98d-459a-9461-d43d6b5f049b4:type4:javaed5:class37:clojure.lang.Compiler$HostExpr$Parser4:file13:Compiler.java5:flagsl4:javae2:id2:374:linei948e6:method5:parse4:name43:clojure.lang.Compiler$HostExpr$Parser/parse7:session36:b6830485-f98d-459a-9461-d43d6b5f049b4:type4:javaed5:class21:clojure.lang.Compiler4:file13:Compiler.java5:flagsl4:javae2:id2:374:linei6560e6:method10:analyzeSeq4:name32:clojure.lang.Compiler/analyzeSeq7:session36:b6830485-f98d-459a-9461-d43d6b5f049b4:type4:javaed5:class21:clojure.lang.Compiler4:file13:Compiler.java5:flagsl4:javae2:id2:374:linei6361e6:method7:analyze4:name29:clojure.lang.Compiler/analyze7:session36:b6830485-f98d-459a-9461-d43d6b5f049b4:type4:javaed5:class21:clojure.lang.Compiler4:file13:Compiler.java5:flagsl3:dup4:javae2:id2:374:linei6322e6:method7:analyze4:name29:clojure.lang.Compiler/analyze7:session36:b6830485-f98d-459a-9461-d43d6b5f049b4:type4:javaed5:class21:clojure.lang.Compiler4:file13:Compiler.java5:flagsl4:javae2:id2:374:linei6555e6:method10:analyzeSeq4:name32:clojure.lang.Compiler/analyzeSeq7:session36:b6830485-f98d-459a-9461-d43d6b5f049b4:type4:javaed5:class21:clojure.lang.Compiler4:file13:Compiler.java5:flagsl4:javae2:id2:374:linei6361e6:method7:analyze4:name29:clojure.lang.Compiler/analyze7:session36:b6830485-f98d-459a-9461-d43d6b5f049b4:type4:javaed5:class21:clojure.lang.Compiler4:file13:Compiler.java5:flagsl3:dup4:javae2:id2:374:linei6322e6:method7:analyze4:name29:clojure.lang.Compiler/analyze7:session36:b6830485-f98d-459a-9461-d43d6b5f049b4:type4:javaed5:class37:clojure.lang.Compiler$BodyExpr$Parser4:file13:Compiler.java5:flagsl4:javae2:id2:374:linei5708e6:method5:parse4:name43:clojure.lang.Compiler$BodyExpr$Parser/parse7:session36:b6830485-f98d-459a-9461-d43d6b5f049b4:type4:javaed5:class30:clojure.lang.Compiler$FnMethod4:file13:Compiler.java5:flagsl4:javae2:id2:374:linei5139e6:method5:parse4:name36:clojure.lang.Compiler$FnMethod/parse7:session36:b6830485-f98d-459a-9461-d43d6b5f049b4:type4:javaed5:class28:clojure.lang.Compiler$FnExpr4:file13:Compiler.java5:flagsl4:javae2:id2:374:linei3751e6:method5:parse4:name34:clojure.lang.Compiler$FnExpr/parse7:session36:b6830485-f98d-459a-9461-d43d6b5f049b4:type4:javaed5:class21:clojure.lang.Compiler4:file13:Compiler.java5:flagsl4:javae2:id2:374:linei6558e6:method10:analyzeSeq4:name32:clojure.lang.Compiler/analyzeSeq7:session36:b6830485-f98d-459a-9461-d43d6b5f049b4:type4:javaed5:class21:clojure.lang.Compiler4:file13:Compiler.java5:flagsl4:javae2:id2:374:linei6361e6:method7:analyze4:name29:clojure.lang.Compiler/analyze7:session36:b6830485-f98d-459a-9461-d43d6b5f049b4:type4:javaed5:class21:clojure.lang.Compiler4:file13:Compiler.java5:flagsl7:tooling4:javae2:id2:374:linei6616e6:method4:eval4:name26:clojure.lang.Compiler/eval7:session36:b6830485-f98d-459a-9461-d43d6b5f049b4:type4:javaed5:class21:clojure.lang.Compiler4:file13:Compiler.java5:flagsl3:dup7:tooling4:javae2:id2:374:linei6582e6:method4:eval4:name26:clojure.lang.Compiler/eval7:session36:b6830485-f98d-459a-9461-d43d6b5f049b4:type4:javaed5:class17:clojure.core$eval4:file8:core.clj5:flagsl3:clj7:toolinge2:fn4:eval2:id2:374:linei2852e6:method6:invoke4:name24:clojure.core$eval/invoke2:ns12:clojure.core7:session36:b6830485-f98d-459a-9461-d43d6b5f049b4:type3:clj3:var17:clojure.core/evaled5:class48:clojure.main$repl$read_eval_print__6588$fn__65914:file8:main.clj5:flagsl3:clj7:toolinge2:fn23:repl/read-eval-print/fn2:id2:374:linei259e6:method6:invoke4:name55:clojure.main$repl$read_eval_print__6588$fn__6591/invoke2:ns12:clojure.main7:session36:b6830485-f98d-459a-9461-d43d6b5f")
  (prog1 (nrepl-decode data) (delete-region start end))
  (let* ((start (point-min)) (end (point-max)) (data (buffer-substring start end))) (prog1 (nrepl-decode data) (delete-region start end)))
  nrepl-net-decode()
  (let ((responses (nrepl-net-decode))) (progn (let ((--dolist-tail-- responses)) (while --dolist-tail-- (let ((response (car --dolist-tail--))) (nrepl-dispatch response) (setq --dolist-tail-- (cdr --dolist-tail--)))))))
  (while (> (buffer-size) 1) (let ((responses (nrepl-net-decode))) (progn (let ((--dolist-tail-- responses)) (while --dolist-tail-- (let ((response ...)) (nrepl-dispatch response) (setq --dolist-tail-- (cdr --dolist-tail--))))))))
  (progn (while (> (buffer-size) 1) (let ((responses (nrepl-net-decode))) (progn (let ((--dolist-tail-- responses)) (while --dolist-tail-- (let (...) (nrepl-dispatch response) (setq --dolist-tail-- ...))))))))
  (closure ((process . #<process nrepl>) t) nil (progn (while (> (buffer-size) 1) (let ((responses (nrepl-net-decode))) (progn (let ((--dolist-tail-- responses)) (while --dolist-tail-- (let ... ... ...))))))))()
  funcall((closure ((process . #<process nrepl>) t) nil (progn (while (> (buffer-size) 1) (let ((responses (nrepl-net-decode))) (progn (let ((--dolist-tail-- responses)) (while --dolist-tail-- (let ... ... ...)))))))))
  (if debug-on-error (funcall body) (condition-case err (funcall body) (error (message "Error: %S" err) nil)))
  (let ((body (function (lambda nil (progn (while (> ... 1) (let ... ...))))))) (if debug-on-error (funcall body) (condition-case err (funcall body) (error (message "Error: %S" err) nil))))
  (let ((nrepl-connection-dispatch (current-buffer))) (let ((body (function (lambda nil (progn (while ... ...)))))) (if debug-on-error (funcall body) (condition-case err (funcall body) (error (message "Error: %S" err) nil)))))
  (save-current-buffer (set-buffer (process-buffer process)) (let ((nrepl-connection-dispatch (current-buffer))) (let ((body (function (lambda nil (progn ...))))) (if debug-on-error (funcall body) (condition-case err (funcall body) (error (message "Error: %S" err) nil))))))
  nrepl-net-process-input(#<process nrepl>)
  nrepl-net-filter(#<process nrepl> "d5:class26:java.lang.RuntimeException2:id2:377:message44:Unable to resolve symbol: aa in this context7:session36:b6830485-f98d-459a-9461-d43d6b5f049bed5:class21:clojure.lang.Compiler4:file13:Compiler.java5:flagsl4:javae2:id2:374:linei6380e6:method7:analyze4:name29:clojure.lang.Compiler/analyze7:session36:b6830485-f98d-459a-9461-d43d6b5f049b4:type4:javaed5:class21:clojure.lang.Compiler4:file13:Compiler.java5:flagsl3:dup4:javae2:id2:374:linei6322e6:method7:analyze4:name29:clojure.lang.Compiler/analyze7:session36:b6830485-f98d-459a-9461-d43d6b5f049b4:type4:javaed5:class37:clojure.lang.Compiler$HostExpr$Parser4:file13:Compiler.java5:flagsl4:javae2:id2:374:linei948e6:method5:parse4:name43:clojure.lang.Compiler$HostExpr$Parser/parse7:session36:b6830485-f98d-459a-9461-d43d6b5f049b4:type4:javaed5:class21:clojure.lang.Compiler4:file13:Compiler.java5:flagsl4:javae2:id2:374:linei6560e6:method10:analyzeSeq4:name32:clojure.lang.Compiler/analyzeSeq7:session36:b6830485-f98d-459a-9461-d43d6b5f049b4:type4:javaed5:class21:clojure.lang.Compiler4:file13:Compiler.java5:flagsl4:javae2:id2:374:linei6361e6:method7:analyze4:name29:clojure.lang.Compiler/analyze7:session36:b6830485-f98d-459a-9461-d43d6b5f049b4:type4:javaed5:class21:clojure.lang.Compiler4:file13:Compiler.java5:flagsl3:dup4:javae2:id2:374:linei6322e6:method7:analyze4:name29:clojure.lang.Compiler/analyze7:session36:b6830485-f98d-459a-9461-d43d6b5f049b4:type4:javaed5:class21:clojure.lang.Compiler4:file13:Compiler.java5:flagsl4:javae2:id2:374:linei6555e6:method10:analyzeSeq4:name32:clojure.lang.Compiler/analyzeSeq7:session36:b6830485-f98d-459a-9461-d43d6b5f049b4:type4:javaed5:class21:clojure.lang.Compiler4:file13:Compiler.java5:flagsl4:javae2:id2:374:linei6361e6:method7:analyze4:name29:clojure.lang.Compiler/analyze7:session36:b6830485-f98d-459a-9461-d43d6b5f049b4:type4:javaed5:class21:clojure.lang.Compiler4:file13:Compiler.java5:flagsl3:dup4:javae2:id2:374:linei6322e6:method7:analyze4:name29:clojure.lang.Compiler/analyze7:session36:b6830485-f98d-459a-9461-d43d6b5f049b4:type4:javaed5:class37:clojure.lang.Compiler$BodyExpr$Parser4:file13:Compiler.java5:flagsl4:javae2:id2:374:linei5708e6:method5:parse4:name43:clojure.lang.Compiler$BodyExpr$Parser/parse7:session36:b6830485-f98d-459a-9461-d43d6b5f049b4:type4:javaed5:class30:clojure.lang.Compiler$FnMethod4:file13:Compiler.java5:flagsl4:javae2:id2:374:linei5139e6:method5:parse4:name36:clojure.lang.Compiler$FnMethod/parse7:session36:b6830485-f98d-459a-9461-d43d6b5f049b4:type4:javaed5:class28:clojure.lang.Compiler$FnExpr4:file13:Compiler.java5:flagsl4:javae2:id2:374:linei3751e6:method5:parse4:name34:clojure.lang.Compiler$FnExpr/parse7:session36:b6830485-f98d-459a-9461-d43d6b5f049b4:type4:javaed5:class21:clojure.lang.Compiler4:file13:Compiler.java5:flagsl4:javae2:id2:374:linei6558e6:method10:analyzeSeq4:name32:clojure.lang.Compiler/analyzeSeq7:session36:b6830485-f98d-459a-9461-d43d6b5f049b4:type4:javaed5:class21:clojure.lang.Compiler4:file13:Compiler.java5:flagsl4:javae2:id2:374:linei6361e6:method7:analyze4:name29:clojure.lang.Compiler/analyze7:session36:b6830485-f98d-459a-9461-d43d6b5f049b4:type4:javaed5:class21:clojure.lang.Compiler4:file13:Compiler.java5:flagsl7:tooling4:javae2:id2:374:linei6616e6:method4:eval4:name26:clojure.lang.Compiler/eval7:session36:b6830485-f98d-459a-9461-d43d6b5f049b4:type4:javaed5:class21:clojure.lang.Compiler4:file13:Compiler.java5:flagsl3:dup7:tooling4:javae2:id2:374:linei6582e6:method4:eval4:name26:clojure.lang.Compiler/eval7:session36:b6830485-f98d-459a-9461-d43d6b5f049b4:type4:javaed5:class17:clojure.core$eval4:file8:core.clj5:flagsl3:clj7:toolinge2:fn4:eval2:id2:374:linei2852e6:method6:invoke4:name24:clojure.core$eval/invoke2:ns12:clojure.core7:session36:b6830485-f98d-459a-9461-d43d6b5f049b4:type3:clj3:var17:clojure.core/evaled5:class48:clojure.main$repl$read_eval_print__6588$fn__65914:file8:main.clj5:flagsl3:clj7:toolinge2:fn23:repl/read-eval-print/fn2:id2:374:linei259e6:method6:invoke4:name55:clojure.main$repl$read_eval_print__6588$fn__6591/invoke2:ns12:clojure.main7:session36:b6830485-f98d-459a-9461-d43d6b5f")