Core Text rendering issues with macOS 10.14 Mojave

[jaingaurav]

I just did a fresh install of macOS 10.14 Mojave and found that MacVim has a number of rendering issues while scrolling. The test gets garbled and the window can sometimes go completely black.

Seems like the workaround was to disable Core Text rending in the advanced preferences.

[mario-grgic]

This happens only when you compile MacVim with 10.14 SDK (previous SDK works fine). For me the MacVim GUI flashes dark/light grey with desert color scheme in rhythm with the cursor blinking, if "Core Text Renderer" is turned on.

[ychin]

This is because of a Mojave change that made all NSView automatically become layer-backed, which will break MacVim's rendering since it relies on behaviors from the legacy renderer. Fortunately it only happens when you link against the new 10.14 SDK so old builds would still work, but we would need a long term solution, presumably a Metal-based renderer similar to other fancy high-performance terminal emulator like Alacritty or Kitty.

Currently the CoreText renderer also supports a CGLayer mode but it has miscellaneous issues including slow scrolling so I don't think that's the way forward.

More info:

• https://developer.apple.com/documentation/appkit/appkit_release_notes_for_macos_10_14#3014921
• https://developer.apple.com/videos/play/wwdc2018/209/?time=1360

[amadeus]

Unfortunately it appears that MacVim is completely unusable on Mojave. Is there any temporary fix?

[ychin]

@amadeus Are you building MacVim yourself using Xcode 10 on Majave? If so, you can build using the old SDK which will fix this issue. Easiest way is to download Xcode 9.4.1 from Apple Developer and build using that. If you are downloading builds from official releases I don't think it should be broken.

Otherwise you can do this in Terminal to turn on CGLayer mode: defaults write org.vim.MacVim MMUseCGLayerAlways -bool YES. I don't recommend this route though as it makes scrolling slow.

[J-Liu]

@amadeus Download the binary form https://github.com/macvim-dev/macvim/releases

[joelfrederico]

@ychin Could you give me a bit more detail on the problem? I'd like to hack a bit on it, although I predict I won't be able to patch it before the regular MacVim committers have fixed it. It seems to me like the issue is someplace in NSWindow not having a background color set? And you want to fix it by dropping Core Text entirely and switching to Metal/Metal2?

[ychin]

@joelfrederico I was actually planning to implement a first-pass for a fix for this this weekend since I have been looking at the renderer code when fixing some fullscreen rendering bugs, but if you do want to tackle the issue feel free to do that too.

The way Vim works is that it treats the GUI as a dumb terminal and just issue draw calls to the GUI like it does for a terminal. The way MacVim draws is that it batches up these draw commands and calls setNeedsDisplay to tell Cocoa that it needs to redraw, and then in the callback drawRect it plays back those draw calls Vim sent it. MacVim itself doesn't cache the full state of the texts, only Vim does.

The issue is that these draw commands Vim sent it are only incremental draw commands assuming the previous draws are still valid. When you call setNeedsDisplay technically you are telling the OS to invalidate the whole window even though we are only interested in invalidating part of the window to draw. Previously it somehow worked because for some reason Cocoa didn't clear the whole window for us even when setNeedsDisplay is called when MacVim is in windowed mode. The new APIs always clear the screen when you call that function so you need to know how to redraw the whole screen instead of just playing back the incremental draw commands. Note that this was already a problem in non-native fullscreen which is why we automatically turn on CGLayer mode in that. I think we may kind of fix this by using setNeedsDisplayInRect instead and make sure to set the proper dirty rect, but it has some other issues and doesn't work with scrollRect. Or we could just cache the text states in the renderer so MacVim always knows how to redraw itself.

Another issue is we currently use NSView's scrollRect function to scroll the window (which is deprecated) which I think is GPU accelerated given how smooth MacVim's scrolling is. I think this function no longer work under the new change to layered back view, so we would need to implement scrolling using other ways. If we cache the text states as described above we could just re-draw the texts using CoreText when we scroll but it will be slower than the previous scrollRect implementation since text rendering is slower than a simple Blit operation on the GPU.

The CGLayer implementation (you can manually turn that on by doing defaults write org.vim.MacVim MMUseCGLayerAlways -bool YES) from what I understand was implemented to address these issues when a previous macOS update broke non-native fullscreen mode, by caching the drawn content onto a CGLayer backing. This way you could incrementally draw to the CGLayer, and then just present the layer when you need to draw. Sadly CGLayer is not GPU accelerated and that's why scrolling in CGLayer mode is a lot slower than the normal one on a high resolution display.

All of this is kind of messy which is why I think we should just implement a hardware accelerated renderer (using Metal) to replace CGLayer. We will still use CoreText for the text rendering. We could either just use a GPU texture as backing layer and draw texts on it which ties into the current renderer easier, or we could cache the font glyphs which would allow us to draw texts much quicker but may be tricky to implement (since MacVim supports ligature etc). Another issue with cached glyphs is they won't work (easily) with subpixel font smoothing, which isn't an issue in Mojave since that feature got removed anyway :(, but in older OS we may still want to have subpixel font rendering to be consistent with other parts of the OS.

This is the gist of what I understand of the rendering bug, maybe we should make another issue dedicated to writing a new renderer.

[chdiza]

but in older OS we may still want to have subpixel font rendering to be consistent with other parts of the OS.

I think supporting older macOS is mandatory, but I'm not sure how compatible that is with using Metal.

[joelfrederico]

@ychin Thanks for the detailed description!

@joelfrederico I was actually planning to implement a first-pass for a fix for this this weekend since I have been looking at the renderer code when fixing some fullscreen rendering bugs, but if you do want to tackle the issue feel free to do that too.

Please don't change your plans thinking I'm working on it! I'm just getting my feet wet, I don't expect to actually be able to do much of anything.

[eirnym]

but in older OS we may still want to have subpixel font rendering to be consistent with other parts of the OS.

I think supporting older macOS is mandatory, but I'm not sure how compatible that is with using Metal.

Metal first appeared in macOS El Capitan, but not every graphics card supports it. So we could have current implementation as a fallback if Metal is failed to load or it set in settings before Mojave and Metal backend only on macOS Mojave+.

[chdiza]

Download the binary form https://github.com/macvim-dev/macvim/releases

This didn't solve anything on my Mojave machine. MacVim (the whole window) blinks along with the cursor.

[J-Liu]

Download the binary form https://github.com/macvim-dev/macvim/releases

This didn't solve anything on my Mojave machine. MacVim (the whole window) blinks along with the cursor.

It works, for me. 151 on Mojave.

[xiaohk]

Any updates on this?

I have tried defaults write org.vim.MacVim MMUseCGLayerAlways -bool YES. It prevents the window from blinking, but scrolling makes the app crash...

I also downloaded 151 binary from https://github.com/macvim-dev/macvim/releases, it doesn't work for me.

[ychin]

@xiaohk How does the release not work for you? Does it crash or just blink a lot when you type? Do you have an external monitor? You can try to remove the CGLayer setting first by doing defaults delete org.vim.MacVim MMUseCGLayerAlways, and also maybe try a fresh vimrc. I'm mostly asking because I'm running Mojave and the release does work for me.