not top level in recursive-edit (root cause: debugger called in fontification-functions out of redisplay)

[yantar92]

Explain-pause-mode: please report this bug by creating a Github
issue at https://github.com/lastquestion/explain-pause-mode. Explain-pause-mode
is now _disabled_ so you can continue to hopefully use Emacs. Info follows:

explain-pause version: 0.1
emacs version: 28.0.50

not top level in wrap-native for #<subr recursive-edit>
current
#s(explain-pause-command-record root-emacs nil nil 7148365 (24341 21787 691643 760000) nil nil nil nil 0)

Backtrace:
  (explain-pause-report-measuring-bug "not top level in wrap-native for #<subr recursive-..." "current" #s(explain-pause-command-record :command root-emacs :native nil :parent nil :executing-time 7148365 :entry-snap (24341 21787 691643 760000) :too-slow nil :is-profiled nil :under-profile nil :profile nil :depth 0))
  (explain-pause--wrap-native #<subr recursive-edit>)
  (apply explain-pause--wrap-native #<subr recursive-edit> nil)
  (recursive-edit)
  (debug lambda)
* (#f(compiled-function () #<bytecode 0xa351effab488f>))
  (jit-lock-fontify-now 17940 18440)
  (jit-lock-function 17940)
  (redisplay_internal\ \(C\ function\))

[yantar92]

The hang happened while displaying company candidates in an org buffer. I attempted to send SIGUSR2 to emacs several times, though it did not have any effect.

[lastquestion]

I think what happened is:

1. You had a breakpoint - because it seems debug got called directly? - or an error in (#f(compiled-function () #<bytecode 0xa351effab488f>)). Note that this is called inside fontify-now which is in redisplay_internal
2. Debugger calls recursive-edit to start.
3. This caused all hell to break loose.

Quick question:
As part of attempting to display this buffer, explain-pause-mode "should have" to disabled itself. So to clarify, what did you mean by "attempted to send SIGUSR2 to emacs several times"? At least Emacs didn't lock up? Or did it?

As for the error itself:

I never wrapped any code that called out of redisplay_internal() because redisplay hooks should never be calling any code that waits on user input. However, I did not anticipate what happens when the debugger runs.

I checked debug.el, and it refuses to start if inhibit-redisplay is set. This code hasn't changed (except for bug fixes and an extension for batch mode) in at least 18 years. This makes sense; if you're redisplaying you can't really start the debugger. BUT, there is a specific comment in keyboard.c recursive_edit_1 calling out literally the use case of debug in fontification-functions and overriding inhibit-redisplay... so I guess this is a normal use case.

I'm not quite sure yet how I want to fix this.

• My first instinct is that in some sense, it would be great to track if there is really slow code in fontify, for example. So maybe wrap all fontification-functions?
• Still, generally speaking debuggers can start anywhere and having the debugger "not work" is not a good experience. So maybe hook debugger? but it might be called from native code - so that might be difficult...

[yantar92]

As part of attempting to display this buffer, explain-pause-mode "should have" to disabled itself. So to clarify, what did you mean by "attempted to send SIGUSR2 to emacs several times"? At least Emacs didn't lock up? Or did it?

Emacs did hang during redisplay, which is something to do with one of my custom fontifications in org-mode. Then, I ran killalll -SIGUSR2 emacs to "unhang" emacs. Sending SIGUSR2 makes Emacs call debugger (see debug-on-event) and thus all the mess.

[lastquestion]

As part of attempting to display this buffer, explain-pause-mode "should have" to disabled itself. So to clarify, what did you mean by "attempted to send SIGUSR2 to emacs several times"? At least Emacs didn't lock up? Or did it?

Emacs did hang during redisplay, which is something to do with one of my custom fontifications in org-mode. Then, I ran killalll -SIGUSR2 emacs to "unhang" emacs. Sending SIGUSR2 makes Emacs call debugger (see debug-on-event) and thus all the mess.

Oh, I see, then this makes perfect sense. You forced debug out of SIGUSR2, that would skip the command-execute path and there would be no root frame.

This is a pretty important use case, and in general, SIGUSR2 causes debug to happen on the next call in eval.c, so basically anywhere. So I think (debug) needs some special handling. I will think more on this.

I also think it'll probably be a good idea to hook the fontification functions.