NEST hangs when setting connection weight in parallel simulations

[heplesser]

The tests below will pass if run serially, but hang when run with mpirun -np 2 pytest .... This also holds if only one of the three tests is executed. Usually, one MPI process will report success while the other never completes. Note that for two processes, only one will actually have the connection. Presumably, this leads to incorrect MPI communication, where one rank hangs.

This PR is based on a bug report by Xavier Otzau on the NEST User Mailing List.

import nest
import pytest


@pytest.fixture
def conns():
    nest.ResetKernel()
    nrn = nest.Create('parrot_neuron')
    nest.Connect(nrn, nrn)
    return nest.GetConnections()

def test_conn_weight(conns):
    if conns:
        conns.weight = 2.5

def test_set_weight(conns):
    if conns:    
        conns.set({'weight': 2.5})

def test_set_status_weight(conns):
    if conns:
        nest.SetStatus(conns, 'weight', 2.5)

[heplesser]

The corresponding SLI code seems to work fine on two MPI processes:

/parrot_neuron Create dup Connect
/c << >> GetConnections def
c empty { ; } { c { << /weight 2.5 >> SetStatus } forall } ifelse

So the problem may be located in the PyNEST interface.

[heplesser]

I have now understood the cause of the problem: SynapseCollection.get() and SynapseCollection.set() under the hood call GetKernelStatus()

nest-simulator/pynest/nest/lib/hl_api_types.py

Line 824 in 6a898bf

num_conns = GetKernelStatus("num_connections") # Has to be called first because it involves MPI communication.

nest-simulator/pynest/nest/lib/hl_api_types.py

Line 889 in 6a898bf

if self.__len__() == 0 or GetKernelStatus("network_size") == 0:

which in turn unconditionally calls ConnectionManager::update_delay_extrema_(), which requires MPI communication. In the examples above, the set() is not called on MPI ranks which do not have connections, and then the MPI communication hangs.

This also means that seemingly innocent patterns such as nest.resolution actually trigger MPI communication. I believe this MPI communication is not really necessary and needs to be reviewed.

A work-around in the cases above is simply to remove the ifs. Setting a parameter on an empty synapse collection is a no-op and safe, I believe.