GLX backend causes screen capture to become choppy

[ekce]

Using compton with ffmpeg's x11grab input or gstreamer's ximagesrc element while using the GLX backend causes video capture to become very choppy as if the FPS was really low. This does not happen with the Xrender backend which is unusual given that Xrender otherwise gives lower performance.

Here is an example ffmpeg command. Note that ffmpeg is compiled with --enable-x11grab for screen capture. The "-r 25" parameter tells ffmpeg to try and output at 25 fps, this may seem like a low number but the capture with GLX looks like it's somewhere much lower than this.

ffmpeg -f x11grab -show_region 1 -draw_mouse 0 -s 640x480 -i :0.0+0,0 -c:v libx264 -crf 21 -preset faster -s 640x480 -r 25 -pix_fmt yuv420p -maxrate 600k -bufsize 900k -f flv ./test.flv

Here are example gstreamer0.10 and gstreamer1.0 commands. Both of these will output to the desktop so you can see it in real time.

gst-launch-0.10 -e ximagesrc startx=0 starty=0 endx=639 endy=479 show-pointer=false use-damage=0 ! ffmpegcolorspace ! x264enc pass=qual quantizer=21 speed-preset=faster tune=zerolatency bitrate=600 vbv-buf-capacity=1500 ! ffdec_h264 ! xvimagesink sync=false

gst-launch-1.0 -e ximagesrc startx=0 starty=0 endx=639 endy=479 show-pointer=false use-damage=0 ! videoconvert ! x264enc pass=qual quantizer=21 speed-preset=faster tune=zerolatency bitrate=600 vbv-buf-capacity=1500 ! avdec_h264 ! xvimagesink sync=false

I am using xfce4 on Archlinux. I use more complex versions of these commands for live streaming to rtmp servers. I'm also using gstreamer's libraries directly in a small QT/C++ app for the same purposes.

[richardgv]

Sorry, I have no idea what would cause it. Both ffmpeg and gstreamer-0.10 work correctly and I got a steady FPS of 30. ffmpeg uses XShmGetImage() or XGetImage() to retrieve screen contents, which should be pretty normal. (There are some small artifacts, but I suppose compositing isn't going to work too well with video capturing anyway... Weirdly enough, enabling threading support in ffmpeg seemingly helped a little.)

Probably it's a driver-dependent issue? I personally use GTX 670 + x11-drivers/nvidia-drivers-325.08. GLX backend is much more sensitive to driver issues.

Only the video is choppy but not the screen, right? What is the FPS ffmpeg displays?

Are you running any other CPU/GPU-intensive applications? Is there anything abnormal about CPU/GPU usage that you could see?

Is there anything special about your setup (e.g. multiple monitor, remote X)?

Does disabling all features (with, for example, --config /dev/null --backend glx) help?

What are you trying to capture? A video player, OpenGL application, or just daily apps?

Does those performance-related options (--sw-opti, --glx-no-stencil, --glx-use-copysubbuffermesa, --glx-swap-method) have any effects?

Does other OpenGL-based compositors (Compiz, Mutter, Kwin, etc.) exhibit the same thing?

And, is there any specific reasons that you can't use X Render backend?

[ekce]

I was on nvidia-319.32-4 but I just updated to nvidia-325.08-1. The video card is a GTX 650. I also updated to the latest compton git as I noticed I had mine a little out of date.

I'm still getting the issue though. I've noticed that it's kind of difficult to see with normal desktop motion and easier with moving video. One of the ways I've been testing is livestreaming over the internet. While the stream is live I'll stop and start compton while testing different settings.

In regards to CPU usage, I don't get any more than usual and also get the issue when using weaker x264 settings to reduce cpu usage (increasing CRF/quantizer and setting the preset to faster settings). The reason I don't like using the XRender backend is because it sometimes gets tearing during video playback. I have a script I use two toggle compton on and off and start either with Xrender or GLX but it's troublesome. It's possible the video tearing issue I mentioned is gone now though as I haven't tested since these latest updates.

Here is my current configuration.

# Shadow
shadow = true;              # Enabled client-side shadows on windows.
no-dnd-shadow = true;           # Don't draw shadows on DND windows.
no-dock-shadow = true;          # Avoid drawing shadows on dock/panel windows.
clear-shadow = true;            # Zero the part of the shadow's mask behind the window (experimental).
shadow-radius = 7;          # The blur radius for shadows. (default 12)
shadow-offset-x = -7;           # The left offset for shadows. (default -15)
shadow-offset-y = -7;           # The top offset for shadows. (default -15)
shadow-exclude = [ "name = 'Notification'", "class_g = 'Conky'", "class_g ?= 'Notify-osd'" ];
"n:w:*Chromium*", "class_g ?= 'Cairo-dock'", "class_g ?= 'Xfce4-notifyd'", "class_g ?= 'Xfce4-power-manager'"];
shadow-ignore-shaped = true;

# Opacity
menu-opacity = 0.9;         # The opacity for menus. (default 1.0)
inactive-opacity = 0.9;         # Opacity of inactive windows. (0.1 - 1.0)
#frame-opacity = 0.8;           # Opacity of window titlebars and borders. (0.1 - 1.0)
frame-opacity = 0.7;
inactive-opacity-override = true;   # Inactive opacity set by 'inactive-opacity' overrides value of _NET_WM_OPACITY.
alpha-step = 0.06;
blur-background-fixed = false;
blur-background-exclude = [ "window_type = 'dock'", "window_type = 'desktop'" ];

# Fading
fading = true;              # Fade windows during opacity changes.
# fade-delta = 30;          # The time between steps in a fade in milliseconds. (default 10).
fade-in-step = 0.07;            # Opacity change between steps while fading in. (default 0.028).
fade-out-step = 0.07;           # Opacity change between steps while fading out. (default 0.03).
#no-fading-openclose = true;        # Fade windows in/out when opening/closing.
fade-exclude = [ ];

# Other
backend = "glx"
mark-wmwin-focused = true;      # Try to detect WM windows and mark them as active.
mark-ovredir-focused = true;
use-ewmh-active-win = false;
detect-rounded-corners = true;
detect-client-opacity = true;
refresh-rate = 60;
vsync = "opengl-swc";
dbe = false;
paint-on-overlay = false;
sw-opti = false;
unredir-if-possible = false;
focus-exclude = [ "name %= '*VLC media player'","name %= '*SMPlayer2'", "name %= '*YouTube*'", "class_g %= '*Plugin-container*'", "name %= '*projectM*'"];
detect-transient = true;
detect-client-leader = true;
invert-color-include = [ ];

# GLX backend
glx-no-stencil = true;
glx-copy-from-front = false;
# glx-use-copysubbuffermesa = true;
glx-no-rebind-pixmap = false;
glx-swap-method = -1;

# Window type settings
wintypes:
{
  tooltip = { fade = true; shadow = false; opacity = 0.75; focus = true; };
};

[ekce]

Sorry, I got carried away and missed some of your questions. I'm running on a single monitor, not remote. Video is going out over hdmi. There is one thing unusual, my video is normal set to 720p but I use xrandr --output HDMI-0 --scale 1.4x1.4 to increase it to a resolution near 1080p. This doesn't usually cause problems but I figured I'd mention it since I have seen it affect videogames.

I tried --config /dev/null --backend glx but it does not help. --sw-opti and --glx-no-stencil seem to have no effect. When I try to use --glx-use-copysubbuffermesa or opengl-mswc I get the error glx_init(): Failed to acquire glXCopySubBufferMESA(). I have tried different swap methods with no results.

Right now I'm just testing with different things, I normally stream games from windows but have been trying to get a viable streaming solution working on linux. I'm writing a small streaming app and testing for a community of streamers. Up until now we've done all our streaming through the command line or scripts but it's fairly limited and complicated for new streamers.

In the past I used to use compiz but it's since died and has no support. There's also a major problem with glib3 that is unlikely to be fixed so it's been dropped from archlinux's official repositories. Thing is I switched video cards some time shortly before compiz stopped working and I'm not sure if I streamed in compiz with this videocard. I have tried Kwin when I had compiz installed since it was easy to switch with fusion-icon. I haven't tried since then because I assumed I had to install and configure the rest of KDE. I'll look into testing with Kwin and Mutter now.

[ekce]

I just tested with kwin --replace (and turned on desktop effects with alt+shift+F12). I can confirm that the capture is smooth using this method. I also attempted mutter --replace but had the same issue I am not very familiar with Gnome configuration though so I'm not sure what I could try to tweak it.

--edit the Kwin capture is smooth but there's tearing while capturing video players.

[richardgv]

Firstly, I'm truly sorry for the late reply. My brain has a bug that I forget one thing if I don't do it immediately.

I got something in recordmydesktop's source code: It tries to detect compositors and when found, forces full screen recording with some sort of double buffering and turns off X SHM. ffmpeg doesn't do that (at least not in ffmpeg-1.2.1). You could comment out use_shm = XShmQueryExtension(dpy); in ./libavdevice/x11grab.c to check if it helps. And, does recordmydesktop work for you?

I'm running on a single monitor, not remote. Video is going out over hdmi. There is one thing unusual, my video is normal set to 720p but I use xrandr --output HDMI-0 --scale 1.4x1.4 to increase it to a resolution near 1080p. This doesn't usually cause problems but I figured I'd mention it since I have seen it affect videogames.

I tried the same here (except with DVI output) and nothing wrong when I record a video with ffmpeg-1.2.1.

When I try to use --glx-use-copysubbuffermesa or opengl-mswc I get the error glx_init(): Failed to acquire glXCopySubBufferMESA().

Oh, it's normal as those functions are Mesa only.

In the past I used to use compiz but it's since died and has no support. There's also a major problem with glib3 that is unlikely to be fixed so it's been dropped from archlinux's official repositories. Thing is I switched video cards some time shortly before compiz stopped working and I'm not sure if I streamed in compiz with this videocard.

compiz-0.8.8 is still working correctly on Gentoo, actually feels better than 0.9.8.

I just tested with kwin --replace (and turned on desktop effects with alt+shift+F12). I can confirm that the capture is smooth using this method. I also attempted mutter --replace but had the same issue I am not very familiar with Gnome configuration though so I'm not sure what I could try to tweak it.

--edit the Kwin capture is smooth but there's tearing while capturing video players.

There's no report that mutter causes any issues with screen capturing that I could find.

That gives me an idea: Have you tried playing with all those VSync options in compton and nvidia-settings? Try turn both off firstly.

By the way, --vsync works with X Render backend. It doesn't work as well and typically leaves some tearing on the top of the screen, though.

[ekce]

Don't worry about it, I tend to do the same thing.

I tried using recordmydesktop and got pretty much the same exact behavior. I also tried calling it with ---no-shared to make sure it was disabling the shared memory extension. Afterwards I tested with --full-shots (both with and without --no-shared) just in case XDamage had something to do with it (gstreamer's XDamage is buggy and tends to cause choppy capture so people typically disable it).

I used recordmydesktop to capture a couple minutes of sample video in case it might spark some ideas. The first half is a scene from the open source Buck Bunny video using Compton with the GLX backend. The second half is the same scene using the XRender backend. To change between them I wrote a script that kills Compton if it's running and starts it if it's not. When given the -r option my script starts Compton with
compton --config ~/.compton.conf --backend xrender --vsync none -b &
The compton.conf file is pretty much the same as the one in the posts above. The audio goes out of sync in the video but it's not really an issue. I'm pretty sure it's just because I didn't specify sample rate and such for the audio stream.

http://www.filedropper.com/comptonbuckbunny

Regarding Compiz, the version I have is 0.8.8-5. It segfaults when I try to run it with fusion-icon unless I do fusion-icon --no-interface. I tried testing with it a bit but after swapping back and forth to it once my desktop became unresponsive and I had to reboot. With the bit of testing I was able to do Compiz seemed to be smooth like the XRender engine but every 1-1.5 second it would pause for a short bit. As I understand it Compiz won't be added back into the Archlinux official repos until it hits 1.0 (which may not be anytime soon). There appears to be a 0.9.9 package but I haven't looked into it yet.

I have tried vsync = "none"; with the GLX backend but it hadn't occurred to me to check my nVidia settings. It's worth mentioning that my Xorg.conf turns on Triple Buffering. I'll read up on the driver documentation and try playing with those settings to see if they make any difference.

[ekce]

I tried disabling TripleBuffer in the Xorg.conf but it made no difference. I tested with and without both Allow Flipping and Sync to VBlank in the nVidia control panel. I was able to get a working configuration by adding Option "DamageEvents" "0" to my xorg.conf and disabling Allow Flipping in my nVidia control panel. Each of those by themselves make no difference. Unfortunately disabling DamageEvents seems to break my mplayer2 for some reason and possibly other programs as well (though VLC still seems to work).

[richardgv]

And, as you may have already noticed, --vsync works with X Render backend, although not as optimal.

I tried using recordmydesktop and got pretty much the same exact behavior. I also tried calling it with ---no-shared to make sure it was disabling the shared memory extension. Afterwards I tested with --full-shots (both with and without --no-shared) just in case XDamage had something to do with it (gstreamer's XDamage is buggy and tends to cause choppy capture so people typically disable it).

I don't think ffmpeg x11grab cares about X Damage. Couldn't find any references in source code.

I used recordmydesktop to capture a couple minutes of sample video in case it might spark some ideas. The first half is a scene from the open source Buck Bunny video using Compton with the GLX backend. The second half is the same scene using the XRender backend. To change between them I wrote a script that kills Compton if it's running and starts it if it's not. When given the -r option my script starts Compton with
compton --config ~/.compton.conf --backend xrender --vsync none -b &
The compton.conf file is pretty much the same as the one in the posts above. The audio goes out of sync in the video but it's not really an issue. I'm pretty sure it's just because I didn't specify sample rate and such for the audio stream.

Unfortunately I failed to download the file. Filedropper doesn't look like the best file sharing service around and I'm somehow always getting incomplete file with checksum failure.

Regarding Compiz, the version I have is 0.8.8-5. It segfaults when I try to run it with fusion-icon unless I do fusion-icon --no-interface. I tried testing with it a bit but after swapping back and forth to it once my desktop became unresponsive and I had to reboot. With the bit of testing I was able to do Compiz seemed to be smooth like the XRender engine but every 1-1.5 second it would pause for a short bit. As I understand it Compiz won't be added back into the Archlinux official repos until it hits 1.0 (which may not be anytime soon). There appears to be a 0.9.9 package but I haven't looked into it yet.

compiz-0.8.8 is in Portage tree, thus definitely is deemed usable by Gentoo developers. I got compiz-0.9.8 working but it isn't quite as well. I do not use fusion-icon but run compiz in .xinitrc directly.

I have tried vsync = "none"; with the GLX backend but it hadn't occurred to me to check my nVidia settings. It's worth mentioning that my Xorg.conf turns on Triple Buffering. I'll read up on the driver documentation and try playing with those settings to see if they make any difference.

I tried disabling TripleBuffer in the Xorg.conf but it made no difference. I tested with and without both Allow Flipping and Sync to VBlank in the nVidia control panel. I was able to get a working configuration by adding Option "DamageEvents" "0" to my xorg.conf and disabling Allow Flipping in my nVidia control panel. Each of those by themselves make no difference. Unfortunately disabling DamageEvents seems to break my mplayer2 for some reason and possibly other programs as well (though VLC still seems to work).

Which is weird. I don't understand why mplayer2 would use X Damage for, nor have I found any references to it in source code of mplayer-1.1.1. As stated above, ffmpeg doesn't seemingly use it, either (although recordmydesktop does, sometimes). -vo may help.

I don't see what relationship "Allow Flipping" and "DamageEvents" have: one about rendering buffer while another about damage notification.

By the way, at least theoretically disabling "Allow flipping" has negative effects on OpenGL performance.

[ekce]

And, as you may have already noticed, --vsync works with X Render backend, although not as optimal.

Right, I forgot to mention this. I can only get the opengl backend working (the opengl_swc backend seems to work but gives an error that claims otherwise). opengl_oml and opengl_drm just give errors.

vsync_opengl_swc_init(): I'm afraid glXSwapIntervalSGI wouldn't help if you are not using GLX backend. You could try, nonetheless.
vsync_opengl_oml_init(): Failed to get OML_sync_control functions.
vsync_drm_wait(): VBlank ioctl did not work, unimplemented in this drmver?

I don't think ffmpeg x11grab cares about X Damage. Couldn't find any references in source code.

All of this last round of testing including the testing with nVidia options was done using recordmydesktop.

Unfortunately I failed to download the file. Filedropper doesn't look like the best file sharing service around and I'm somehow always getting incomplete file with checksum failure.

I uploaded the file to youtube here.
http://youtu.be/4GEGiHiam7o
I can also upload it elsewhere if you like, it's about 80 megabytes.

compiz-0.8.8 is in Portage tree, thus definitely is deemed usable by Gentoo developers. I got compiz-0.9.8 working but it isn't quite as well. I do not use fusion-icon but run compiz in .xinitrc directly.

I haven't tried throwing it into my .xinitrc but typing just compiz --replace ccp & into a terminal gave me the same results as fusion-icon from the terminal (fusion-icon doesn't work from the menu now anyways). The capture isn't choppy in the same way as the video above, instead there's a quarter second pause every second and a half or so. Asking in the archlinux irc about compiz generally just yields responses about it being unsupported, out of date, and that one should use compton or something else instead.

Which is weird. I don't understand why mplayer2 would use X Damage for, nor have I found any references to it in source code of mplayer-1.1.1. As stated above, ffmpeg doesn't seemingly use it, either (although recordmydesktop does, sometimes). -vo may help.

It's possible that my mplayer2 is having that behavior since I've enabled VDPAU codecs in my config (as well as smplayer2) and forgot to try disabling them, though I think VLC uses GPU acceleration of some sort as well.

I don't see what relationship "Allow Flipping" and "DamageEvents" have: one about rendering buffer while another about damage notification.

Yea I have no idea why this would have that effect. The only reason I bothered to check it was that I had previously read that XDamage seems to cause reduced performance with gstreamer, but it should only be an issue gstreamer's ximagesrc implementation. When I use gstreamer with X Damage on I do get a choppy screen capture if the area I'm trying to capture is too large (like the whole screen). I don't get the same issue when using ffmpeg to capture the whole screen. Anyway, I was reading through the xorg/appendix section of the nVidia driver configuration and the option so I figured I may as well try it on the off-chance it had an effect.

By the way, at least theoretically disabling "Allow flipping" has negative effects on OpenGL performance.

Yea, I'd rather not use it for that reason. It mentions that in the documentation as well.
http://us.download.nvidia.com/XFree86/Linux-x86_64/325.08/README/xconfigoptions.html

[richardgv]

Huh, apparently I've successfully forgotten this again... :-S

Right, I forgot to mention this. I can only get the opengl backend working (the opengl_swc backend seems to work but gives an error that claims otherwise). opengl_oml and opengl_drm just give errors.

opengl-swc works only with GLX backend, as stated in the usage text. opengl-oml doesn't work with nVidia binary blob due to lack of support of OML_sync_control. drm basically only works on open-source drivers.

All of this last round of testing including the testing with nVidia options was done using recordmydesktop.

Looked briefly at recordmydesktop-0.3.8.1's source code and seemingly X Damage is being used to determine changed regions -- but not in full-shots mode.

I uploaded the file to youtube here.

Oh, this one works. I didn't found the issue very visible, though, presumably because of my bad eyesight.

Asking in the archlinux irc about compiz generally just yields responses about it being unsupported, out of date, and that one should use compton or something else instead.

Pretty much that's the typical tone of an "experienced" user. :-) I sometimes speak in that way, as well.

It's possible that my mplayer2 is having that behavior since I've enabled VDPAU codecs in my config (as well as smplayer2) and forgot to try disabling them, though I think VLC uses GPU acceleration of some sort as well.

VDPAU or VAAPI does not necessarily work too well with composition or screen recording. The not-accelerated x11 may be a safe fallback.

VLC >= 2.1, I heard, has gotten VDPAU support, while older versions only could use VDPAU through VAAPI on VLC.

[ekce]

Sorry, I haven't replied in a while I've been preoccupied with my normal life. I did try getting compiz working though. I tried some different packages as well as the bazaar 0.9 version but I had problems with each. I think mainly though the problems are either related to using xfce4 with it or just using it with archlinux in generla (some of the packages listed as dependencies aren't available anymore in any form except cached versions). The 0.8 packages I can kind of get working but with video capture I get long pauses over a second long every second or so, it's really bad. The 0.9 packages weren't giving me any window decorations and in some instances made my desktop unresponsive altogether (I didn't manage to test video capture with them).

I'm currently kind of out of basic ideas and in general I guess telling streamers to disable composting to stream isn't as unusual considering in Windows 7 and Windows Vista users need to disable Aero in order to capture the desktop at an FPS larger than 25 fps.

I forgot to add that kwin seems to work (made sure it's using the opengl backend and not xrender), so I'll experiment with that some more.

[richardgv]

I'm currently kind of out of basic ideas and in general I guess telling streamers to disable composting to stream isn't as unusual considering in Windows 7 and Windows Vista users need to disable Aero in order to capture the desktop at an FPS larger than 25 fps.

I would doubt if there are some other factors here, considering there are many users recording a composited desktop and seemingly only you encountered the issue.

I forgot to add that kwin seems to work (made sure it's using the opengl backend and not xrender), so I'll experiment with that some more.

I guess with my limited knowledge and the size of kwin it isn't too easy for me to figure out the exact reason, though.