Skip to content

Usage of the nest-server-mpi #3463

@vvorobjov

Description

@vvorobjov

Hello, dear nest team!

I am sorry, that I am opening the issue here, but I could not get through the Mailman at users@nest-simulator.org...

I am trying to launch the experiment, using nest-server-mpi and NRP on the HPC. We also use a module generated by nestml. But I faced the issue, which I so far cannot solve myself... Probably you could help me.

We have a working experiment for the nest-server v3.7, which also works with v3.8.

I tried to launch nest-simulator in the container on the HPC, but I faced some mpi incompatibility issues, could not solve it fast and gave up for now.
Then I installed nest-simulator 3.7 from source on the HPC. The simulation with nest-server also worked. After patching nest-server-mpi (serialization issue) I could launch it on the HPC and communicate with the API, but the experiment didn't work, and I switched to the local nest-simulator container for the test.

Locally, in the original nest/nest-simulator:3.7 and 3.8 containers, the nest-server-mpi is not starting. After the fixing the issue (with docopt and serialization) the nest-server-mpi is launching in nest/nest-simulator:dev container. But here is the problem, with this version the experiment is crashing even with nest-server, as well as with nest-server-mpi. And I cannot figure out the reason from the logs and git history...

I tried patching versions 3.7 and 3.8, then I can start nest-server-mpi, can use API, but the experiment itself doesn't work.

So, the summary:

  • The experiment works with nest-server 3.7 and 3.8, but doesn't work with the patched nest-server-mpi
  • The experiment doesn't work with the nest-server with :dev container

What happens in the latest container with the nest-server:

nrp-nest-simulator  |
nrp-nest-simulator  | Apr 03 16:11:15 SimulationManager::set_status [Info]:
nrp-nest-simulator  |     Temporal resolution changed from 0.1 to 0.1 ms.
nrp-nest-simulator  |
nrp-nest-simulator  | Apr 03 16:11:15 Install [Info]:
nrp-nest-simulator  |     loaded module controller_module

And that's it, on the other side I get [json.exception.out_of_range.403] key 'data' not found

The nest script, which is executed at this moment is

nest.ResetKernel()
nest.SetKernelStatus({"resolution": res})
nest.Install("controller_module")
for i in range(njt):
    planner_p = nest.Create("tracking_neuron_nestml", n=N, params={"kp": plan_params["kp"], "base_rate": plan_params["base_rate"], "pos": True, "traj": trj, "simulation_steps": len(trj)})
    planner_n = nest.Create("tracking_neuron_nestml", n=N, params={"kp": plan_params["kp"], "base_rate": plan_params["base_rate"], "pos": False, "traj": trj, "simulation_steps": len(trj)})

The same piece of code works in nest-server v3.7 and v3.8

What happens in the patched nest-server-mpi v 3.8 or 3.7, here it at least throws an exception

nrp-nest-simulator  | ========================================
nrp-nest-simulator  |
nrp-nest-simulator  | ==> MASTER 0/1743700791.3478301 (route_api_call): call=GetStatus, args=[[303]], kwargs={}
nrp-nest-simulator  | ==> MASTER 0/1743700791.3478405 (GetStatus): sending call bcast
nrp-nest-simulator  | ==> MASTER 0/1743700791.3478618 (GetStatus): sending data bcast, data=([[303]], {})
nrp-nest-simulator  | ==> MASTER 0/1743700791.3480251 (GetStatus): local call, args=[NodeCollection(metadata=None, model=spike_recorder, size=1, first=303)], kwargs={}
nrp-nest-simulator  | ==> MASTER 0/1743700791.3480933 (GetStatus): waiting for response gather
nrp-nest-simulator  | ==> MASTER 0/1743700791.3484681 (GetStatus): received response gather, data=[({'element_type': 'recorder', 'events': {'senders': array([204]), 'times': array([13.5])}, 'frozen': False, 'global_id': 303, 'label': 'Brain stem pos', 'local': True, 'model': 'spike_recorder', 'model_id': 96, 'n_events': 1, 'node_uses_wfr': False, 'origin': 0.0, 'record_to': 'memory', 'start': 0.0, 'stop': 1.7976931348623157e+308, 'thread': 0, 'thread_local_id': 152, 'time_in_steps': False, 'vp': 0},), [{'element_type': 'recorder', 'events': {'senders': [], 'times': []}, 'frozen': False, 'global_id': 303, 'label': 'Brain stem pos', 'local': True, 'model': 'spike_recorder', 'model_id': 96, 'n_events': 0, 'node_uses_wfr': False, 'origin': 0.0, 'record_to': 'memory', 'start': 0.0, 'stop': 1.7976931348623157e+308, 'thread': 0, 'thread_local_id': 152, 'time_in_steps': False, 'vp': 1}]]
nrp-nest-simulator  | [2025-04-03 19:19:51,348] ERROR in app: Exception on /api/GetStatus [POST]
nrp-nest-simulator  | Traceback (most recent call last):
nrp-nest-simulator  |   File "/usr/lib/python3/dist-packages/flask/app.py", line 2070, in wsgi_app
nrp-nest-simulator  |     response = self.full_dispatch_request()
nrp-nest-simulator  |   File "/usr/lib/python3/dist-packages/flask/app.py", line 1515, in full_dispatch_request
nrp-nest-simulator  |     rv = self.handle_user_exception(e)
nrp-nest-simulator  |   File "/usr/lib/python3/dist-packages/flask_cors/extension.py", line 165, in wrapped_function
nrp-nest-simulator  |     return cors_after_request(app.make_response(f(*args, **kwargs)))
nrp-nest-simulator  |   File "/usr/lib/python3/dist-packages/flask/app.py", line 1513, in full_dispatch_request
nrp-nest-simulator  |     rv = self.dispatch_request()
nrp-nest-simulator  |   File "/usr/lib/python3/dist-packages/flask/app.py", line 1499, in dispatch_request
nrp-nest-simulator  |     return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)
nrp-nest-simulator  |   File "/opt/nest/lib/python3.10/site-packages/nest/server/hl_api_server.py", line 317, in route_api_call
nrp-nest-simulator  |     return jsonify(response)
nrp-nest-simulator  |   File "/usr/lib/python3/dist-packages/flask/json/__init__.py", line 348, in jsonify
nrp-nest-simulator  |     f"{dumps(data, indent=indent, separators=separators)}\n",
nrp-nest-simulator  |   File "/usr/lib/python3/dist-packages/flask/json/__init__.py", line 129, in dumps
nrp-nest-simulator  |     rv = _json.dumps(obj, **kwargs)
nrp-nest-simulator  |   File "/usr/lib/python3.10/json/__init__.py", line 238, in dumps
nrp-nest-simulator  |     **kw).encode(obj)
nrp-nest-simulator  |   File "/usr/lib/python3.10/json/encoder.py", line 199, in encode
nrp-nest-simulator  |     chunks = self.iterencode(o, _one_shot=True)
nrp-nest-simulator  |   File "/usr/lib/python3.10/json/encoder.py", line 257, in iterencode
nrp-nest-simulator  |     return _iterencode(o, 0)
nrp-nest-simulator  |   File "/usr/lib/python3/dist-packages/flask/json/__init__.py", line 56, in default
nrp-nest-simulator  |     return super().default(o)
nrp-nest-simulator  |   File "/usr/lib/python3.10/json/encoder.py", line 179, in default
nrp-nest-simulator  |     raise TypeError(f'Object of type {o.__class__.__name__} '
nrp-nest-simulator  | TypeError: Object of type int64 is not JSON serializable
nrp-nest-simulator  | [2025-04-03 19:19:51,349] INFO in _internal: 172.18.0.3 - - [03/Apr/2025 19:19:51] "POST /api/GetStatus HTTP/1.1" 500 -

Do you have any clues where I should dig?

I would expect, that if the experiment works with nest-server v3.8, it should work in dev as well. But I understand, that there could be some changes, which should be reflected in the setup or interface...

I would appreciate any help.

Best wishes, Viktor

Metadata

Metadata

Assignees

Labels

S: NormalHandle this with default priorityT: BugWrong statements in the code or documentationstaleAutomatic marker for inactivity, please have another look here

Type

No type

Projects

Status

To do

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions