[pymusic] Endless loop in `setup.runtime(..)`

[mhoff]

Hello everyone,

while debugging a more complex experiment using MUSIC I recognized that two of my pymusic-nodes simply never return from calling music_setup.runtime(..), whereas music_setup = music.Setup().

I have been able to reduce the code to a very minimal not working example.
When I execute the music configuration attached to this issue via mpirun -np 1 music test.music nothing more than "before runtime" will get printed. The process appears to enter an endless loop when calling runtime on the setup object.
In contrast, mpirun -np 1 ./test_node.py works fine.

I'm well aware that this trivial node combined with this trivial configuration might be a special case for MUSIC, but this issue also arises with a more complex setup and nodes which actually consume data via incoming connections.

---

• music --version: MUSIC 1.1.15 (8e0a609)
• mpirun --version: mpirun (Open MPI) 1.6.5

---

Minimal example (download: test.zip):

test_node.py

#!/usr/bin/env python
# -*- coding: utf-8 -*-

import music

music_setup = music.Setup()

print("before runtime")

runtime = music_setup.runtime(0.02)

print("after runtime -- not called")

for current_time in runtime:
    print(current_time)

test.music

[test]
  binary=./test_node.py
  args=
  np=1

[uahic]

At the first glance it looks like that starting MUSIC with a single process and no ports does not work when launched with mpiexec or mpirun. If I start it with ipython only, it works. Ill have a look at it

[mdjurfeldt]

Yes, unfortunately, the new MUSIC scheduler hangs if there is only one application. This should of course be fixed. Den 16 jan. 2017 16:19 skrev "Martin Schulze" <notifications@github.com>:

…

At the first glance it looks like that starting MUSIC with a single process and no ports does not work when launched with mpiexec or mpirun. If I start it with ipython only, it works. I first need to get Cython debugging working before I can dive into the problem. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#35 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADcfCWZnamNjWZRXiVeGgj-ofIF_9Gofks5rS4psgaJpZM4Lkd4V> .

[uahic]

@mdjurfeldt Do you know how to enable gdb to debug Cython code?

I found this: http://docs.cython.org/en/latest/src/userguide/debugging.html but there are multiple cythonize() function calls in the pymusic folder setup.py.in and I am not sure where to set the gdb flag

[mhoff]

It seems not to be the special case of only one application. Running two of the above nodes with -np 2 yields the same behaviour for both nodes.

[test1]
  binary=./test_node.py
  args=
  np=1

[test2]
  binary=./test_node.py
  args=
  np=1

[uahic]

@i think its a bug when you dont have any ports connecting these applications or maybe not any ports at all (note that you also have to actually create the ports in your applications)

[mhoff]

Seems likely.
However, in my experiment I have a purely consuming node (2 incoming connections) which exhibits the same behaviour. But as it is a big node I can not say for certain that there is no misconfiguration.

[mhoff]

This is really weird. I reduced my implementation to a point at which all configured nodes & connections are working as intended.
Then I add another node (pynest) with its three ports. Two of these ports are inputs which I simply connect to pre-existing outputs (which have been proven to be working in the minimal setup). The third port is an event output. I connect this port to another port on a pre-existing node which has not been used until that. In the code of the pre-existing node I do not publish the input. Now, when I run the whole thing, all nodes freeze in runtime(..) without an error message.
Next, I add the publish code to make the connection proper and run it again. Now, all nodes run except for the pre-existing node for which I just added a connection.

[uahic]

Eventhough there is a method in the MUSIC interface to check whether a port is connected, NEST does not make use of it and always assumes ports to be connected. I will fix this soon (if it makes sense ... ) as I am refactoring all recording devices in NEST anyways (already started with that). Be prepared 8-)

[mhoff]

Many thanks. But still the bigger issue what happens after setting up the connection properly:

Next, I add the publish code to make the connection proper and run it again. Now, all nodes run except for the pre-existing node for which I just added a connection.

Suddenly, after hooking in another node, a node which has been working before does not return from runtime() anymore. I published and mapped the new event input port the same way I have set up another event input port on the same node which has been working before. Really weird.

[mdjurfeldt]

Can you provide the two examples? Den 16 jan. 2017 21:30 skrev "Michael Hoff" <notifications@github.com>:

…

Many thanks. But still the bigger issue what happens after setting up the connection properly: Next, I add the publish code to make the connection proper and run it again. Now, all nodes run except for the pre-existing node for which I just added a connection. Suddenly, after hooking in another node, a node which has been working before does not return from runtime() anymore. I published and mapped the new event input port the same way I have set up another event input port on the same node which has been working before. Really weird. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#35 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADcfCbzJIdMvFFeUpZF5NN5pGp9_6Fq0ks5rS9MogaJpZM4Lkd4V> .

[mhoff]

That would be difficult. The pynest node relies on a custom nest module (which in turn is responsible for creating one of the music ports). Maybe I can reproduce the faulty behavior with a more simple setup.

But before I do that, can you tell me why the above use case with two independent music nodes fails? I totally understand that one single node can be a special case for the music scheduler. And I certainly also understand that having simply two unconnected music nodes is a somehow related scenario in terms of the scheduler. But do both behaviors really emerge from the same origin?

[mhoff]

Is the order of nodes & connections in the music configuration relevant for the scheduler? While trying to produce a showcase for this issue I was moving around nodes and connections inside the configuration file. Now other nodes are not responding anymore. Attached to this comment you find my current configuration file. Everything above # 3 is the reduced version which was working before I changed the order in the configuration file. Now even the part up to # 1 is not working anymore (ros_sensor_adapter does not return from runtime..).

---

[distance_provider]
  binary=ros_sensor_adapter
  args=
  np=1
  music_timestep=0.05
  ros_topic=/distance/to_center
  message_type=FloatArray
  sensor_update_rate=10

[reward_node]
  binary=../../src/compute_node.py
  args=
  np=1

distance_provider.out -> reward_node.distance_in [1]

# 1

[plotter]
  np=1
  binary=../../src/plotter_node.py

reward_node.reward_out -> plotter.reward_in [1]

# 2

[dvs]
  binary=ros_event_sensor_adapter
  args=
  np=1
  music_timestep=0.5
  ros_topic=/dvs/events
  message_type=EventArray
  sensor_update_rate=1

dvs.out -> plotter.pattern_in [200]

# 3

#[nest]
#  binary=../../src/network_node.py
#  args=
#  np=1

#dvs.out -> nest.pattern_in [200]
#nest.activity_out -> plotter.activity_in [20]

# 4

#reward_node.reward_out -> nest.reward_in [1]

# 5

#[decoder]
#  binary=linear_readout_decoder
#  args=
#  np=1
#  music_timestep=0.05
#  tau=0.03
#  weights_filename=activity_to_velocity_translation_weights.dat
#[command]
#  binary=ros_command_adapter
#  args=
#  np=1
#  music_timestep=0.05
#  ros_topic=/cmd_vel
#  message_mapping_filename=velocity_to_twist.dat
#  command_rate=20

#nest.activity_out -> decoder.in [20]
#decoder.out -> command.in [2]

[mhoff]

I have to retreat on my last statement. A little bug (stoptime = ... got missing) caused the weird behaviour with the order appearing to be relevant.
The above configuration file is now working up to # 4. If I activate the connection reward_node.reward_out -> nest.reward_in [1], the plotter_node will not return from runtime(..) anymore. This is weird as this node is not participating in this specific connection.

[mdjurfeldt]

Dear Michael, I will have to investigate this. Unfortunately, I can do this first during the weekend at the earliest. Obviously, this is not how MUSIC should behave and we will fix it! Best regards, Mikael

…

On Wed, Jan 18, 2017 at 2:09 PM, Michael Hoff ***@***.***> wrote: I have to retreat on my last statement. A little bug (stoptime = ... got missing) caused the weird behaviour with the order appearing to be relevant. The above configuration file is now working up to # 4. If I activate the connection reward_node.reward_out -> nest.reward_in [1], the plotter_node will not return from runtime(..) anymore. This is weird as this node is not participating in this specific connection. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#35 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADcfCf68OBRymljWFCL0W6YiAk9X2lsQks5rTg8QgaJpZM4Lkd4V> .

[mhoff]

@mdjurfeldt Thank you very much for your effort!