Use native python fontification when sending input to shell (fix #1428)

[memeplex]

PR Summary

Use full python font locking in shell, even when sending input from an elpy buffer. Fixes #1428

Warning: this is not ready for merging because it requires some changes upstream that I'm pushing here: https://debbugs.gnu.org/cgi/bugreport.cgi?bug=32344

Specifically, these two very simple patches should be merged:

• https://debbugs.gnu.org/cgi/bugreport.cgi?msg=20;att=1;bug=32344;filename=0001-Option-for-comint-to-honour-original-input-highlight.patch

• https://debbugs.gnu.org/cgi/bugreport.cgi?msg=35;att=1;bug=32344;filename=0001-Keep-python-shell-input-colorization-in-comint.patch

Alternatively we could add an "experimental module" (disabled by default) to get this [1] working now using some elisp hack:

(defun my-comint-highlight-input-advice (f &rest args)
  (if (eq major-mode 'inferior-python-mode)
      (cl-letf* ((g (symbol-function 'add-text-properties))
                 ((symbol-function 'add-text-properties)
                  (lambda (start end properties &optional object)
                    (unless (eq (nth 3 properties) 'comint-highlight-input)
                     (funcall g start end properties object)))))
        (apply f args))
    (apply f args)))
(advice-add #'comint-send-input :around #'my-comint-highlight-input-advice)

But let's hope my patches get merged upstream.

PR checklist

Please make sure that the following things have been addressed (and check the relevant checkboxes):

• Commits respect our guidelines
• Tests are passing properly (see here on how to run Elpy's tests)

---

[1] And probably other stuff that require monkey-patching hacks. Related to #1531

[coveralls]

Coverage decreased (-0.2%) to 92.146% when pulling 75f5cb1 on memeplex:colorized-shell into 7e4dd5d on jorgenschaefer:master.

[galaunay]

Nice,
I have this hack in my init.el for so long that I forgot it was a hack.
Any idea of the timescale for you patches to be merged ?

[memeplex]

Any idea of the timescale for you patches to be merged ?

Not at all. Maybe @npostavs could make a more informed guess here.

[rgemulla]

I like this idea! I did a quick try & it did not work in my setup, however. In particular, it seems to break when continuation lines are used (elpy-shell-echo-input-cont-prompt set to t). When pasting multiline statements, the python font lock hook seems to get confused with the continuation prompt.

One way to fix this may be to insert the code fragment into a temporary buffer, propertize it there using the python font lock, and then copying it line by line to the shell buffer.

[memeplex]

Have you installed the corresponding comint and python.el patches that should be merged into emacs 27 for this to work? You also have a more hacky version in the wiki using advices.

[rgemulla]

Yes, it's the call to the python font lock hook that breaks.

[memeplex]

One way to fix this may be to insert the code fragment into a temporary buffer, propertize it there using the python font lock, and then copying it line by line to the shell buffer.

AFAICR this is exactly what is done.

The changes here just disable recolorizing the already colorized text. If there is something wrong in the way text is colorized in some corner cases, I believe the problem should be addressed separately.

[rgemulla]

I apologize, the problem arose because I already applied PR #1710. I now changed the PR slightly so that the advice-based hack from the doc still works. It's not nice, but at least it does not break.

That being said, if continuation prompts are turned on and the hack are being used, then the font lock of the continuation prompt is lost (at least in my setup).

Also, whether or not continuation prompts are used, the font lock in the shell buffer does not seem to match the one in the Python buffer (e.g., integers and variable assignments are not font-locked in the shell buffer). Is this the same with this PR?

One idea that I had when originally writing this code was to keep track of the text properties from the python buffer throughout. I never got to implement this. The advantage would be that the font lock in the shell buffer exactly matches the python buffer. It should also be faster since there is no separate font lock.

[rgemulla]

The problem with not fontifying numbers arises also directly in the shell buffer (i.e., when entering numbers there); not sure why.

[rgemulla]

I figured out the reason for the difference in number highlighting: I have highlight-numbers-mode enabled python-mode, and it's not automatically enabled in inferior-python-mode as well. Manually enabling it solves this issue.

The only remaining problem is that the fontification of the continuation prompt is destroyed.

[rgemulla]

Ok, found the issue, fix below. I suggest to change this PR such that only elpy-shell--insert-and-font-lock is changed (but not elpy-shell--append-to-shell-output). A definition that respects continuation prompts and keeps fontification of inputs is:

(defun elpy-shell--insert-and-font-lock (string face &optional no-font-lock)
  "Inject STRING into the Python shell buffer."
  (let ((from-point (point)))
    (insert string)
    (unless no-font-lock
      (if (eq face 'comint-highlight-input)
          (cl-letf* ((stp (symbol-function 'set-text-properties))
                     ((symbol-function 'set-text-properties)
                      (lambda (start end plist &optional object)
                        (while (< start end)
                          (unless (eq (get-text-property start 'face) 'comint-highlight-prompt)
                            (funcall stp start (+ 1 start) plist object))
                          (setq start (+ 1 start))))))
            (python-shell-font-lock-post-command-hook))
        (add-text-properties
         from-point (point)
         (list 'front-sticky t 'font-lock-face face))))))

Only the advice for comint-input-prompt is then needed (until addressed upstream).

[rgemulla]

This fix improves things, but it is still not perfect. Since python-shell-font-lock-post-command-hook does not know that continuation prompts should be ignored, the only way to improve it further is to replace python-shell-font-lock-post-command-hook by a custom function, I think.

[rgemulla]

I found a simpler and better solution, see PR #1711. Let me know what you think.

[galaunay]

#1711 works fine for me.

Will we have some compatibility issue if we merge it and the patches to Emacs get merged as well ?

[rgemulla]

No, #1711 solely fontifies the source code before it's being echoed in the shell buffer. In fact, my-comint-highlight-input-advice mentioned above is currently still necessary.

@memeplex What are your thoughts?