Incompatibility with networkx>=2 and version conflict with prov dependency

[tclose]

Summary

As far as I can tell nipype isn't compatible with networkx>=2, as running with networkx>=2 raises a "graph change during iteration" error. So perhaps this should be reflected in the install requires of the setup.py (i.e. frozen at networkx==1.9).

However, it is complicated by the fact that another dependency, prov>=1.5.2 requires networkx>=2, and attempting to use the nipypecli with prov>=1.5.2 with networkx==1.9 raises a 'pkg_resources.ContextualVersionConflict' error (at least with Python 3.7)

Actual behavior

An error is raised

Traceback (most recent call last):
  File "/usr/local/bin/nipypecli", line 6, in <module>
    from pkg_resources import load_entry_point
  File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3123, in <module>
    @_call_aside
  File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3107, in _call_aside
    f(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3136, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 580, in _build_master
    return cls._build_from_requirements(__requires__)
  File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 593, in _build_from_requirements
    dists = ws.resolve(reqs, Environment())
  File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 786, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (networkx 1.9 (/usr/local/lib/python3.7/site-packages), Requirement.parse('networkx>=2.0'), {'prov'})

Expected behavior

How to replicate the behavior

Install networkx==1.9 and prov==1.5.2 and use nipypcli crash to open a crash file

Script/Workflow details

N/A

Platform details:

{'commit_hash': 'b448b6061',
 'commit_source': 'installation',
 'networkx_version': '1.9',
 'nibabel_version': '2.3.0',
 'nipype_version': '1.1.6',
 'numpy_version': '1.15.4',
 'pkg_path': '/usr/local/lib/python3.7/site-packages/nipype',
 'scipy_version': '1.1.0',
 'sys_executable': '/usr/local/opt/python/bin/python3.7',
 'sys_platform': 'darwin',
 'sys_version': '3.7.1 (default, Nov  6 2018, 18:45:35) \n'
                '[Clang 10.0.0 (clang-1000.11.45.5)]',
 'traits_version': '4.6.0'}

Execution environment

My python environment outside container

[satra]

@tclose - there is obviously some weird thing happening here, but we have been running with networkx > 2 for a while. would it be possible to test your workflow in a fresh environment? if you are using conda you can create one with conda create -n testweirderror python=3.7.1 nipype

[effigies]

This might be pip's problem, which doesn't do full-bore dependency resolution. I would just try pip install --upgrade networkx>=2.0.

[tclose]

That is interesting. Do you use Python 3.7?

I just reinstalled everything in a virtualenv but still get the same issue with networkx>=2. Here is the full stack trace

  File "/Users/tclose/git/ni/arcana/arcana/processor/base.py", line 321, in run
    result = workflow.run(plugin=self._plugin)
  File "/Users/tclose/.virtualenvs/nipype_test/lib/python3.7/site-packages/nipype/pipeline/engine/workflows.py", line 582, in run
    flatgraph = self._create_flat_graph()
  File "/Users/tclose/.virtualenvs/nipype_test/lib/python3.7/site-packages/nipype/pipeline/engine/workflows.py", line 855, in _create_flat_graph
    workflowcopy._generate_flatgraph()
  File "/Users/tclose/.virtualenvs/nipype_test/lib/python3.7/site-packages/nipype/pipeline/engine/workflows.py", line 879, in _generate_flatgraph
    for node in nodes:
  File "/Users/tclose/.virtualenvs/nipype_test/lib/python3.7/site-packages/networkx/algorithms/dag.py", line 200, in topological_sort
    raise RuntimeError("Graph changed during iteration")
RuntimeError: Graph changed during iteration

pip freeze gives this

alabaster==0.7.12
apipkg==1.5
arcana==0.2.10
atomicwrites==1.2.1
attrs==18.2.0
Babel==2.6.0
-e git+git@github.com:tclose/banana.git@e82bed599fe9bdec1ec30caaf772c85093b7bdbb#egg=banana
certifi==2018.11.29
chardet==3.0.4
citeproc-py==0.4.0
Click==7.0
configparser==3.5.0
contextlib2==0.5.5
decorator==4.3.0
deepdiff==3.3.0
docopt==0.6.2
docutils==0.14
duecredit==0.6.4
execnet==1.5.0
fasteners==0.14.1
funcsigs==1.0.2
future==0.17.1
grabbit==0.2.5
idna==2.8
imagesize==1.1.0
isodate==0.6.0
Jinja2==2.10
jsonpickle==1.0
lxml==4.2.5
MarkupSafe==1.1.0
mock==2.0.0
monotonic==1.5
more-itertools==4.3.0
networkx==2.2
neurdflib==5.0.0.post1
nibabel==2.3.1
nineml==1.0.1
nipype==1.1.6
num2words==0.5.8
numpy==1.15.4
packaging==18.0
pandas==0.23.4
patsy==0.5.1
pbr==5.1.1
pluggy==0.8.0
progressbar2==3.38.0
prov==1.5.3
py==1.7.0
pybids==0.6.5
pydicom==1.2.1
pydot==1.4.1
pydotplus==2.0.2
Pygments==2.3.0
pyparsing==2.3.0
pype9==0.2
pytest==4.0.1
pytest-forked==0.2
pytest-xdist==1.24.1
python-dateutil==2.7.5
python-utils==2.3.0
pytz==2018.7
rdflib==4.2.2
requests==2.21.0
scipy==1.1.0
simplejson==3.16.0
six==1.12.0
snowballstemmer==1.2.1
Sphinx==1.8.2
sphinx-rtd-theme==0.4.2
sphinxcontrib-websupport==1.1.0
traits==4.6.0
urllib3==1.24.1
xnat==0.3.11
xnatutils==0.4.6

[tclose]

Actually, I just realised it doesn't happen for every workflow that I run, so it looks as though it might be another case of my funky auto-generated workflows exploring an obscure corner of the code.

You should hopefully be able to reproduce the error by checking out the master branch from http://github.com/MonashBI/arcana and running the /test/unittests/study/test_study.py unittests (Arcana's dependencies can be installed using the setup.py, however it will force the installation of networkx==1.9, which you will need to manually upgrade after). Only the 'test_run_pipeline_with_prereqs' test errors so you should be able to compare the workflow generated by it with the others in the module. I have had a quick look but can't see anything obvious.

[effigies]

That's a fairly distinctive sort of error. It looks like we didn't wrap enough calls to nx.topological_sort in list(). Can you try the branch in #2828 and see if that fixes your error?

If prov requires networkx>=2.0, that may present the opportunity to get rid of networkx 1.x compatibility measures.

[tclose]

Yep #2828 fixes the problem, thanks