xvfb-run not ending gracefully for CommandLine interfaces

[blakedewey]

I just updated my testing environment to the latest release of Nipype and I have been getting errors from my _redirect_x=True CommandLine interfaces. The CLI completes without error, but then xvfb-run errors out with:
/usr/bin/xvfb-run: line 171: kill: (25939) - No such process

Some quick Googling seems to point to the fact that xvfb-run was not able to kill the display because it was assigned the same display as another instance. My thought is since this only happens on MultiProc/SGE runs of my workflows is that xvfb-run is used almost simultaneously and hasn't acquired a lock for the display in use and the -a option still reuses the same display for multiple commands. This way when any command after the first finishes, there is no display to kill, leading to the error. I rolled back my code to use @chrisfilo's hack with subprocess and Xvfb and the problem resolved itself. I think we should start the -a option from a random starting value using the -n option.

Has anyone else seen this issue? Thoughts?

[oesteban]

Yes, the virtual framebuffer is indeed a problem. I included xvfb-run (#1156) because previously the behavior was a bit problematic.

However, xvfb-run only works for CommandLineInterface-derived interfaces. And, the script creates a new virtual display for every interface with _redirect_x=True. As you noted, xvfb-run is not reliable when opening many virtual displays.

I have commented this issue with @satra some times, and the conclusions are:

• The optimal solution would be opening only one virtual display before running. This would require just one call to Xvfb before the workflow is run, and redirecting all interfaces with _redirect_x to that new display. However, Xvfb has some other problems (xvfbwrapper error #1400).
• If you are directly using interfaces (without the workflow wrapping) then we should detect that no other virtual display is open before creating a new one. Otherwise, attach to an existing one.
• IMHO, xvfb-run should be removed from nipype ASAP because it only works for command line interfaces and it is hard to handle from nipype. I wonder if it also has the same problems that Xvfb has in mac OSX.

[satra]

@blakedewey @oesteban - should this be one xvfb per process or one xvfb per physical node? i'm thinking of multiple processes started by a scheduler, rather than the multiproc plugin.

[blakedewey]

It should be one per physical node. I was thinking of keeping track of the xvfb processes in the parent process, where we would have a dictionary of physical node names and xvfb display numbers.

[satra]

@blakedewey - that gets a little more complicated since it gets into possible race conditions. so we would need:

1. some form of lock on a node
2. some form of reference counting
3. some way of determining an existing user xvfb process

[blakedewey]

So assuming we could lock the node while Xvfb is starting, storing display number and pid for Xvfb should be enough to determine if it is running and restart if not (taking the lock again).

[blakedewey]

It would be complicated, though. That is true. I think that is why this issue has sat for a while.