Usage of the nest-server-mpi

[vvorobjov]

Hello, dear nest team!

I am sorry, that I am opening the issue here, but I could not get through the Mailman at users@nest-simulator.org...

I am trying to launch the experiment, using nest-server-mpi and NRP on the HPC. We also use a module generated by nestml. But I faced the issue, which I so far cannot solve myself... Probably you could help me.

We have a working experiment for the nest-server v3.7, which also works with v3.8.

I tried to launch nest-simulator in the container on the HPC, but I faced some mpi incompatibility issues, could not solve it fast and gave up for now.
Then I installed nest-simulator 3.7 from source on the HPC. The simulation with nest-server also worked. After patching nest-server-mpi (serialization issue) I could launch it on the HPC and communicate with the API, but the experiment didn't work, and I switched to the local nest-simulator container for the test.

Locally, in the original nest/nest-simulator:3.7 and 3.8 containers, the nest-server-mpi is not starting. After the fixing the issue (with docopt and serialization) the nest-server-mpi is launching in nest/nest-simulator:dev container. But here is the problem, with this version the experiment is crashing even with nest-server, as well as with nest-server-mpi. And I cannot figure out the reason from the logs and git history...

I tried patching versions 3.7 and 3.8, then I can start nest-server-mpi, can use API, but the experiment itself doesn't work.

So, the summary:

• The experiment works with nest-server 3.7 and 3.8, but doesn't work with the patched nest-server-mpi
• The experiment doesn't work with the nest-server with :dev container

What happens in the latest container with the nest-server:

nrp-nest-simulator  |
nrp-nest-simulator  | Apr 03 16:11:15 SimulationManager::set_status [Info]:
nrp-nest-simulator  |     Temporal resolution changed from 0.1 to 0.1 ms.
nrp-nest-simulator  |
nrp-nest-simulator  | Apr 03 16:11:15 Install [Info]:
nrp-nest-simulator  |     loaded module controller_module

And that's it, on the other side I get [json.exception.out_of_range.403] key 'data' not found

The nest script, which is executed at this moment is

nest.ResetKernel()
nest.SetKernelStatus({"resolution": res})
nest.Install("controller_module")
for i in range(njt):
    planner_p = nest.Create("tracking_neuron_nestml", n=N, params={"kp": plan_params["kp"], "base_rate": plan_params["base_rate"], "pos": True, "traj": trj, "simulation_steps": len(trj)})
    planner_n = nest.Create("tracking_neuron_nestml", n=N, params={"kp": plan_params["kp"], "base_rate": plan_params["base_rate"], "pos": False, "traj": trj, "simulation_steps": len(trj)})

The same piece of code works in nest-server v3.7 and v3.8

What happens in the patched nest-server-mpi v 3.8 or 3.7, here it at least throws an exception

nrp-nest-simulator  | ========================================
nrp-nest-simulator  |
nrp-nest-simulator  | ==> MASTER 0/1743700791.3478301 (route_api_call): call=GetStatus, args=[[303]], kwargs={}
nrp-nest-simulator  | ==> MASTER 0/1743700791.3478405 (GetStatus): sending call bcast
nrp-nest-simulator  | ==> MASTER 0/1743700791.3478618 (GetStatus): sending data bcast, data=([[303]], {})
nrp-nest-simulator  | ==> MASTER 0/1743700791.3480251 (GetStatus): local call, args=[NodeCollection(metadata=None, model=spike_recorder, size=1, first=303)], kwargs={}
nrp-nest-simulator  | ==> MASTER 0/1743700791.3480933 (GetStatus): waiting for response gather
nrp-nest-simulator  | ==> MASTER 0/1743700791.3484681 (GetStatus): received response gather, data=[({'element_type': 'recorder', 'events': {'senders': array([204]), 'times': array([13.5])}, 'frozen': False, 'global_id': 303, 'label': 'Brain stem pos', 'local': True, 'model': 'spike_recorder', 'model_id': 96, 'n_events': 1, 'node_uses_wfr': False, 'origin': 0.0, 'record_to': 'memory', 'start': 0.0, 'stop': 1.7976931348623157e+308, 'thread': 0, 'thread_local_id': 152, 'time_in_steps': False, 'vp': 0},), [{'element_type': 'recorder', 'events': {'senders': [], 'times': []}, 'frozen': False, 'global_id': 303, 'label': 'Brain stem pos', 'local': True, 'model': 'spike_recorder', 'model_id': 96, 'n_events': 0, 'node_uses_wfr': False, 'origin': 0.0, 'record_to': 'memory', 'start': 0.0, 'stop': 1.7976931348623157e+308, 'thread': 0, 'thread_local_id': 152, 'time_in_steps': False, 'vp': 1}]]
nrp-nest-simulator  | [2025-04-03 19:19:51,348] ERROR in app: Exception on /api/GetStatus [POST]
nrp-nest-simulator  | Traceback (most recent call last):
nrp-nest-simulator  |   File "/usr/lib/python3/dist-packages/flask/app.py", line 2070, in wsgi_app
nrp-nest-simulator  |     response = self.full_dispatch_request()
nrp-nest-simulator  |   File "/usr/lib/python3/dist-packages/flask/app.py", line 1515, in full_dispatch_request
nrp-nest-simulator  |     rv = self.handle_user_exception(e)
nrp-nest-simulator  |   File "/usr/lib/python3/dist-packages/flask_cors/extension.py", line 165, in wrapped_function
nrp-nest-simulator  |     return cors_after_request(app.make_response(f(*args, **kwargs)))
nrp-nest-simulator  |   File "/usr/lib/python3/dist-packages/flask/app.py", line 1513, in full_dispatch_request
nrp-nest-simulator  |     rv = self.dispatch_request()
nrp-nest-simulator  |   File "/usr/lib/python3/dist-packages/flask/app.py", line 1499, in dispatch_request
nrp-nest-simulator  |     return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)
nrp-nest-simulator  |   File "/opt/nest/lib/python3.10/site-packages/nest/server/hl_api_server.py", line 317, in route_api_call
nrp-nest-simulator  |     return jsonify(response)
nrp-nest-simulator  |   File "/usr/lib/python3/dist-packages/flask/json/__init__.py", line 348, in jsonify
nrp-nest-simulator  |     f"{dumps(data, indent=indent, separators=separators)}\n",
nrp-nest-simulator  |   File "/usr/lib/python3/dist-packages/flask/json/__init__.py", line 129, in dumps
nrp-nest-simulator  |     rv = _json.dumps(obj, **kwargs)
nrp-nest-simulator  |   File "/usr/lib/python3.10/json/__init__.py", line 238, in dumps
nrp-nest-simulator  |     **kw).encode(obj)
nrp-nest-simulator  |   File "/usr/lib/python3.10/json/encoder.py", line 199, in encode
nrp-nest-simulator  |     chunks = self.iterencode(o, _one_shot=True)
nrp-nest-simulator  |   File "/usr/lib/python3.10/json/encoder.py", line 257, in iterencode
nrp-nest-simulator  |     return _iterencode(o, 0)
nrp-nest-simulator  |   File "/usr/lib/python3/dist-packages/flask/json/__init__.py", line 56, in default
nrp-nest-simulator  |     return super().default(o)
nrp-nest-simulator  |   File "/usr/lib/python3.10/json/encoder.py", line 179, in default
nrp-nest-simulator  |     raise TypeError(f'Object of type {o.__class__.__name__} '
nrp-nest-simulator  | TypeError: Object of type int64 is not JSON serializable
nrp-nest-simulator  | [2025-04-03 19:19:51,349] INFO in _internal: 172.18.0.3 - - [03/Apr/2025 19:19:51] "POST /api/GetStatus HTTP/1.1" 500 -

Do you have any clues where I should dig?

I would expect, that if the experiment works with nest-server v3.8, it should work in dev as well. But I understand, that there could be some changes, which should be reflected in the setup or interface...

I would appreciate any help.

Best wishes, Viktor