nosetest fails when X-server not present

[satra]

$ nosetests --with-doctest nipype
nosetests: cannot connect to X server 

[oesteban]

I'll try to look at it tomorrow. It must be related to the interfaces redirecting X, don't you think?

[satra]

i think so. i'm trying to debug a situation where tests are consuming a lot of memory on my cluster, so i just wanted to put the issue here. it would be great if you can take a look.

[satra]

narrowed it to this:

$ nosetests ~/software/nipype/nipype/algorithms/tests/test_mesh_ops.py 
nosetests: cannot connect to X server 

[oesteban]

Ok, great. That's a lot to start with it. Thanks.

[oesteban]

After #1156, this problem was greatly narrowed down by using the xvfb-run script for command line nodes.

However, for interfaces deriving directly from BaseInterface this was not possible. [It is important to say that before that PR those interfaces just did not work].

Before that PR, nipype had copied code from xvfbwrapper. So I removed the copied code, imported xvfbwrapper and manually set the display in the environment variable of the runtime (https://github.com/oesteban/nipype/blob/1b6b88bcb991e252b843a603a532f5e6d04bf9e5/nipype/interfaces/base.py#L972).

I guess that the problem is related to the following: xvfbwrapper modifies the 'DISPLAY' environment variable at system level. However, when the node is run some other process may have changed the display.

I couldn't reproduce this in my settings (and tested in one headless system with debian).

[yarikoptic]

indeed -- boils down to

> DISPLAY= python -c 'from tvtk.api import tvtk'
Unable to access the X Display, is $DISPLAY set properly?

on my debian system it comes from

PATH       : /usr/lib/python2.7/dist-packages/tvtk/api.pyc
SRC PATH   : /usr/lib/python2.7/dist-packages/tvtk/api.py
VERSION    : Not found
__version__: '4.3.1'
PACKAGE    : mayavi2
ii  mayavi2        4.3.1-4.1    amd64        scientific visualization package for 2-D and 3-D data
mayavi2:
  Installed: 4.3.1-4.1
  Candidate: 4.3.1-4.1
  Version table:
 *** 4.3.1-4.1 0
        900 http://http.debian.net/debian/ stretch/main amd64 Packages
        600 http://http.debian.net/debian/ sid/main amd64 Packages
        100 /var/lib/dpkg/status

at least other imports seems to be not at the level of the module

$> git grep -B4  'tvtk.api' 
nipype/algorithms/mesh.py-        except ImportError:
nipype/algorithms/mesh.py-            iflogger.warn(('python-vtk could not be imported'))
nipype/algorithms/mesh.py-
nipype/algorithms/mesh.py-        try:
nipype/algorithms/mesh.py:            from tvtk.api import tvtk
--
nipype/algorithms/mesh.py-        return area
nipype/algorithms/mesh.py-
nipype/algorithms/mesh.py-    def _run_interface(self, runtime):
nipype/algorithms/mesh.py-        try:
nipype/algorithms/mesh.py:            from tvtk.api import tvtk
--
nipype/algorithms/mesh.py-    _redirect_x = True
nipype/algorithms/mesh.py-
nipype/algorithms/mesh.py-    def _run_interface(self, runtime):
nipype/algorithms/mesh.py-        try:
nipype/algorithms/mesh.py:            from tvtk.api import tvtk
--
nipype/algorithms/tests/test_mesh_ops.py-notvtk = True
nipype/algorithms/tests/test_mesh_ops.py-import platform
nipype/algorithms/tests/test_mesh_ops.py-if 'darwin' not in platform.system().lower():
nipype/algorithms/tests/test_mesh_ops.py-    try:
nipype/algorithms/tests/test_mesh_ops.py:        from tvtk.api import tvtk
--
nipype/interfaces/fsl/utils.py-
nipype/interfaces/fsl/utils.py-    def _vtk_to_coords(self, in_file, out_file=None):
nipype/interfaces/fsl/utils.py-        import os.path as op
nipype/interfaces/fsl/utils.py-        try:
nipype/interfaces/fsl/utils.py:            from tvtk.api import tvtk
--
nipype/interfaces/fsl/utils.py-
nipype/interfaces/fsl/utils.py-    def _coords_to_vtk(self, points, out_file):
nipype/interfaces/fsl/utils.py-        import os.path as op
nipype/interfaces/fsl/utils.py-        try:
nipype/interfaces/fsl/utils.py:            from tvtk.api import tvtk

so it is safe to assume that such failures wouldn't typically happen when someone runs headless an actual pipeline (unless one using mesh stuff), so for my issue #1217 I will just wrap nosetests call to be ran under xvfb

[oesteban]

That should work, but anyways these interfaces should work in headless runs. Let me see if I can find a solution.

EDIT: IMHO, everything points to opening just one virtual display in the beginning if no $DISPLAY is set OR if $DISPLAY is set but no X-server is listening, as suggested in #1156. Alternatively/additionally, we could have the special value 'auto' in DISPLAY, that would force opening a virtual server.

[oesteban]

Executing > ETS_TOOLKIT='null' python -c 'from tvtk.api import tvtk' works well, and you don't need to wrap the nosetests call.

I'm afraid I should retake #973, ensuring that ETS_TOOLKIT environment variable is set to null before any nipype code tries to import tvtk for first.

[oesteban]

Since we identified the source of the problem, we can follow this bug in #972 and its corresponding PR #973.