Kilosort 2.5; returned a non-zero exit code

[PaoloCapelli]

We tried running spikeinterface on 2 different machines and at the moment of launching kilosort via matlab (we tried different versions of matlab as well) it fails. Matlab briefly opens but it then crashes. Kilosort per se works when used via matlab.

We get the following errors from the jupyter notebook we are using to run the spikeinterface pipeline (we tried a few things but we could not fix it):

/////
Error running kilosort2_5

SpikeSortingError Traceback (most recent call last)
Cell In[19], line 2
1 # Sorting as a whole
----> 2 sorting = si.run_sorter('kilosort2_5', rec, output_folder=base_folder / 'Paolo-ThinBarTest-Merged/test/kilosort2_5_output',
3 do_correction=True, docker_image=False, verbose=True, raise_error=True,
4 remove_existing_folder=True)

File ~\AppData\Local\Programs\Python\Python310\lib\site-packages\spikeinterface\sorters\runsorter.py:147, in run_sorter(sorter_name, recording, output_folder, remove_existing_folder, delete_output_folder, verbose, raise_error, docker_image, singularity_image, delete_container_files, with_output, **sorter_params)
140 container_image = singularity_image
141 return run_sorter_container(
142 container_image=container_image,
143 mode=mode,
144 **common_kwargs,
145 )
--> 147 return run_sorter_local(**common_kwargs)

File ~\AppData\Local\Programs\Python\Python310\lib\site-packages\spikeinterface\sorters\runsorter.py:173, in run_sorter_local(sorter_name, recording, output_folder, remove_existing_folder, delete_output_folder, verbose, raise_error, with_output, **sorter_params)
171 SorterClass.set_params_to_folder(recording, output_folder, sorter_params, verbose)
172 SorterClass.setup_recording(recording, output_folder, verbose=verbose)
--> 173 SorterClass.run_from_folder(output_folder, raise_error, verbose)
174 if with_output:
175 sorting = SorterClass.get_result_from_folder(output_folder)

File ~\AppData\Local\Programs\Python\Python310\lib\site-packages\spikeinterface\sorters\basesorter.py:289, in BaseSorter.run_from_folder(cls, output_folder, raise_error, verbose)
286 print(f"{sorter_name} run time {run_time:0.2f}s")
288 if has_error and raise_error:
--> 289 raise SpikeSortingError(
290 f"Spike sorting error trace:\n{log['error_trace']}\n"
291 f"Spike sorting failed. You can inspect the runtime trace in {output_folder}/spikeinterface_log.json."
292 )
294 return run_time

SpikeSortingError: Spike sorting error trace:
Traceback (most recent call last):
File "C:\Users\sabatini\AppData\Local\Programs\Python\Python310\lib\site-packages\spikeinterface\sorters\basesorter.py", line 254, in run_from_folder
SorterClass._run_from_folder(sorter_output_folder, sorter_params, verbose)
File "C:\Users\sabatini\AppData\Local\Programs\Python\Python310\lib\site-packages\spikeinterface\sorters\external\kilosortbase.py", line 215, in _run_from_folder
raise Exception(f"{cls.sorter_name} returned a non-zero exit code")
Exception: kilosort2_5 returned a non-zero exit code

Spike sorting failed. You can inspect the runtime trace in D:\Shun\Recordings\Ephys\Paolo-ThinBarTest-Merged\test\kilosort2_5_output/spikeinterface_log.json.
///

thank you very much

[zm711]

Could you post the spikeinterface_log.json results here so we can see what the exit code is? (log is found here Spike sorting failed. You can inspect the runtime trace in D:\Shun\Recordings\Ephys\Paolo-ThinBarTest-Merged\test\kilosort2_5_output/spikeinterface_log.json.

[PaoloCapelli]

Here are the results from the log.json file @zm711

"sorter_name": "kilosort2_5",
"sorter_version": "unknown",
"datetime": "2024-01-11T15:49:46.857523",
"runtime_trace": [
"",
"C:\Users\sabatini\Documents\SpikeInterface>D:",
"",
"D:\>cd D:\Shun\Recordings\Ephys\Paolo-ThinBarTest-Merged\test\kilosort2_5_output\sorter_output",
"",
"D:\Shun\Recordings\Ephys\Paolo-ThinBarTest-Merged\test\kilosort2_5_output\sorter_output>matlab -nosplash -wait -r "kilosort2_5_master('D:\Shun\Recordings\Ephys\Paolo-ThinBarTest-Merged\test\kilosort2_5_output\sorter_output', 'C:\Users\sabatini\Documents\Github\Kilosort-2.5')""
],
"error": true,
"error_trace": "Traceback (most recent call last):\n File "C:\Users\sabatini\.conda\envs\spikeinterface\lib\site-packages\spikeinterface\sorters\basesorter.py", line 254, in run_from_folder\n SorterClass._run_from_folder(sorter_output_folder, sorter_params, verbose)\n File "C:\Users\sabatini\.conda\envs\spikeinterface\lib\site-packages\spikeinterface\sorters\external\kilosortbase.py", line 215, in _run_from_folder\n raise Exception(f"{cls.sorter_name} returned a non-zero exit code")\nException: kilosort2_5 returned a non-zero exit code\n",
"run_time":` null

[zm711]

And when matlab opens and crashes can you see any messages in matlab itself? Any indication why it is crashing?

[PaoloCapelli]

We are using a windows 10 system that currently has multiple versions of MATLAB installed. It seems to fail right at the initiation of matlab so there aren't any associated errors. We have MATLAB2021a and MATLAB2023b installed on the system.

Could the shell script command be the issue? https://github.com/SpikeInterface/spikeinterface/blob/d3a7e6ff2a5171925c9bad8d79a1f35760d0046e/src/spikeinterface/sorters/external/kilosortbase.py#L184 here? If I run matlab -nosplash within my enrionment in terminal, it does open successfully.

[zm711]

I've had no issue with that using Kilosort2 and Kilosort3 (I haven't tried 2.5 yet). I'm also on a Windows computer (10 and 11) and no issues with those two versions.

A couple more questions:
What version of spikeinterface are you using? There have been some Windows specific updates added to the code base that might be helpful?

Does this happen with every dataset you try or is this one specific dataset that is the problem?

[PaoloCapelli]

We figured out the issue. The shell script was defaulting to MATLAB2023b and kilosort was not setup properly under 2023b. After installing the proper toolbox and running mexGPUall.m, spikeinterface runs kilosort just fine.

[shirquinn]

@PaoloCapelli which toolbox did you install?

[DaohanZhang]

@PaoloCapelli Hello! I'm encountering the exactly same error and could you please provide more details as how to set kilosort to certain matlab version? Very thanks!