Problems of using new version of spike interface

[farazmoradi]

Hi all,

Recently, I reinstalled my spike interface to have access to the latest version and I faced some issues.

1. Spike sorting using Kilosort 2 on a single recording:

I load the data using binaryrecordingextractor and tried to perform the spike sorting with Kilosort2. I received the error in the first picture. Then I tried to do the spike sorting with the Kilosort2 GUI in the MATLAB and spike sorting was done without any problem! So I am not sure what happens in the spike interface that I received that error, but same recording, in MATLAB GUI, no error was received!

2. Spike sorting on concatenated recordings:

When I try to do spike sorting on concatenated recordings, I receive the error on the attached image. However, the concatenated .dat file was produced in the folder and I tried that again in MATLAB GUI and spike sorting was done without any problem.

Mentioned errors happened after I installed the new version and in the previous version I didn't have any problem.

Thanks a lot for your help.

[alejoe91]

Can you share the full script?

[farazmoradi]

1. Loading multiple recordings:

"recording_list = []
sampling_frequency = 30000
num_channels = 64
path = r'E:\Data and Analysis\N3 64 molex opto\raw data\Day 11\N3 day 11 no turn_230530_132119'
c=0
for (root, dirs, files) in os.walk(path):
if len(files) > 1:
print(root)
if files[0]== 'amplifier.dat':
recording_list.append(se.BinaryRecordingExtractor(root +'\'+ files[0], sampling_frequency, num_channels, 'int16'))
c+=1
multirecording = si.concatenate_recordings(recording_list)
print(c)
multirecording "

2. Adding channel info:

"ch = [14,5,26,19,4,25,3,0,8,20,10,24,16,2,22,15,30,21,6,17,1,9,23,31,27,29,13,7,18,11,28,12,
32,56,48,40,50,33,36,53,45,58,47,57,43,52,34,60,37,54,41,44,49,63,42,35,59,46,55,38,51,62,39,61]

probe_H10 = generate_multi_shank(num_shank=2,num_columns=3,
num_contact_per_column=[10, 12, 10],
xpitch=18.5, ypitch=30,
y_shift_per_column=[0, -45, 0],
contact_shapes='square', contact_shape_params={'width': 8})
probe_H10.set_device_channel_indices(ch)
multirecording = multirecording.set_probe(probe_H10)
print(probe_H10)
plot_probe(probe_H10, with_channel_index=True)"

3. Spike Sorting:

sorting = ss.run_kilosort2(recording=multirecording, output_folder=r'E:\Data and Analysis\N3 64 molex opto\spike sorting raw\Day 11\M3 day 11 no turn 2')
sorting.get_unit_ids()"

[alejoe91]

@farazmoradi you mentioned you also had problems running KS on a single recording. Does the error only appear with the conatenated one?

[farazmoradi]

For the concatenated recordings I receive the second picture, for single recordings I receive the first one (non zero exit code).

[alejoe91]

Can you run with verbose=True and paste the full output?

[farazmoradi]

(base) d:\Mossy Code Backup 18 May 2023>E:

(base) E:>cd E:\data_analysis\N3 64 molex opto\spike sorting raw\T9 oct18\sorter_output

(base) E:\data_analysis\N3 64 molex opto\spike sorting raw\T9 oct18\sorter_output>matlab -nosplash -wait -r "kilosort2_master('E:\data_analysis\N3 64 molex opto\spike sorting raw\T9 oct18\sorter_output', 'C:\Users\BeiqueLab\Downloads\Kilosort-2.0\Kilosort-2.0')"

Error running kilosort2

SpikeSortingError Traceback (most recent call last)
Cell In[9], line 2
1 #multirecording = spre.common_reference(multirecording,reference='global')
----> 2 sorting_TDC_5 = ss.run_sorter("kilosort2", recording=multirecording, output_folder=r"E:\data_analysis\N3 64 molex opto\spike sorting raw/T9 oct18",verbose=True)
3 sorting_TDC_5.get_unit_ids()

File c:\ProgramData\anaconda3\Lib\site-packages\spikeinterface\sorters\runsorter.py:148, in run_sorter(sorter_name, recording, output_folder, remove_existing_folder, delete_output_folder, verbose, raise_error, docker_image, singularity_image, delete_container_files, with_output, **sorter_params)
141 container_image = singularity_image
142 return run_sorter_container(
143 container_image=container_image,
144 mode=mode,
145 **common_kwargs,
146 )
--> 148 return run_sorter_local(**common_kwargs)

File c:\ProgramData\anaconda3\Lib\site-packages\spikeinterface\sorters\runsorter.py:174, in run_sorter_local(sorter_name, recording, output_folder, remove_existing_folder, delete_output_folder, verbose, raise_error, with_output, **sorter_params)
172 SorterClass.set_params_to_folder(recording, output_folder, sorter_params, verbose)
173 SorterClass.setup_recording(recording, output_folder, verbose=verbose)
--> 174 SorterClass.run_from_folder(output_folder, raise_error, verbose)
175 if with_output:
176 sorting = SorterClass.get_result_from_folder(output_folder)

RUNNING SHELL SCRIPT: E:\data_analysis\N3 64 molex opto\spike sorting raw\T9 oct18\sorter_output\run_kilosort2.bat

File c:\ProgramData\anaconda3\Lib\site-packages\spikeinterface\sorters\basesorter.py:269, in BaseSorter.run_from_folder(cls, output_folder, raise_error, verbose)
266 print(f"{sorter_name} run time {run_time:0.2f}s")
...
File "c:\ProgramData\anaconda3\Lib\site-packages\spikeinterface\sorters\external\kilosortbase.py", line 215, in _run_from_folder
raise Exception(f"{cls.sorter_name} returned a non-zero exit code")
Exception: kilosort2 returned a non-zero exit code

Spike sorting failed. You can inspect the runtime trace in E:\data_analysis\N3 64 molex opto\spike sorting raw\T9 oct18/spikeinterface_log.json.
Output is truncated. View as a scrollable element or open in a text editor. Adjust cell output settings...

[farazmoradi]

This is a single recording. Do you want me to paste the spikeinterface_log.json too?

[alejoe91]

Do you see the Matlab banner? Are you running from the E drive?

[farazmoradi]

Yes, the MATLAB menu comes up for a while (detecting good channels and filtering). My codes are in drive D and my data is in Drive E.

[alejoe91]

I don't see the Matlab output from your print

[farazmoradi]

Mmm. Do you want me to take a screen video while running the code?

[alejoe91]

Can you copy the entire output and paste it? It's better if you use the code notation:

Like this one

[farazmoradi]

---------------------------------------------------------------------------
SpikeSortingError                         Traceback (most recent call last)
Cell In[4], line 2
      1 #multirecording = spre.common_reference(multirecording,reference='global')
----> 2 sorting_TDC_5 = ss.run_sorter("kilosort2", recording=multirecording, output_folder=r"E:\data_analysis\N3 64 molex opto\spike sorting raw/T9 oct18",verbose=True)
      3 sorting_TDC_5.get_unit_ids()

File c:\ProgramData\anaconda3\Lib\site-packages\spikeinterface\sorters\runsorter.py:148, in run_sorter(sorter_name, recording, output_folder, remove_existing_folder, delete_output_folder, verbose, raise_error, docker_image, singularity_image, delete_container_files, with_output, **sorter_params)
    141             container_image = singularity_image
    142     return run_sorter_container(
    143         container_image=container_image,
    144         mode=mode,
    145         **common_kwargs,
    146     )
--> 148 return run_sorter_local(**common_kwargs)

File c:\ProgramData\anaconda3\Lib\site-packages\spikeinterface\sorters\runsorter.py:174, in run_sorter_local(sorter_name, recording, output_folder, remove_existing_folder, delete_output_folder, verbose, raise_error, with_output, **sorter_params)
    172 SorterClass.set_params_to_folder(recording, output_folder, sorter_params, verbose)
    173 SorterClass.setup_recording(recording, output_folder, verbose=verbose)
--> 174 SorterClass.run_from_folder(output_folder, raise_error, verbose)
    175 if with_output:
    176     sorting = SorterClass.get_result_from_folder(output_folder)

File c:\ProgramData\anaconda3\Lib\site-packages\spikeinterface\sorters\basesorter.py:269, in BaseSorter.run_from_folder(cls, output_folder, raise_error, verbose)
    266         print(f"{sorter_name} run time {run_time:0.2f}s")
    268 if has_error and raise_error:
--> 269     raise SpikeSortingError(
    270         f"Spike sorting error trace:\n{log['error_trace']}\n"
    271         f"Spike sorting failed. You can inspect the runtime trace in {output_folder}/spikeinterface_log.json."
    272     )
    274 return run_time

SpikeSortingError: Spike sorting error trace:
Traceback (most recent call last):
  File "c:\ProgramData\anaconda3\Lib\site-packages\spikeinterface\sorters\basesorter.py", line 234, in run_from_folder
    SorterClass._run_from_folder(sorter_output_folder, sorter_params, verbose)
  File "c:\ProgramData\anaconda3\Lib\site-packages\spikeinterface\sorters\external\kilosortbase.py", line 215, in _run_from_folder
    raise Exception(f"{cls.sorter_name} returned a non-zero exit code")
Exception: kilosort2 returned a non-zero exit code

Spike sorting failed. You can inspect the runtime trace in E:\data_analysis\N3 64 molex opto\spike sorting raw\T9 oct18/spikeinterface_log.json.

[alejoe91]

Thanks, but again can you include the entire output? Not just the error. Basically I'd like to see when KS is failing, so we need to check the MATLAB output

[farazmoradi]

Sure, let me see how can copy the MATLAB output.

[farazmoradi]

[farazmoradi]

That's a weird! after this the Matlab closes and I receive that error. Also, the spike sorting is performed without error when I do it with Kilosort GUI in MATLAB!!

[alejoe91]

I see! Can you try to recompile Kilosort? From the CUDA folder, run mexGPUall

[farazmoradi]

I think I should try to reset my windows and get back t you :)

Building with 'NVIDIA CUDA Compiler'.
MEX completed successfully.
Building with 'NVIDIA CUDA Compiler'.
MEX completed successfully.
Building with 'NVIDIA CUDA Compiler'.
Error using mex
Microsoft (R) Manifest Tool

Copyright (c) Microsoft Corporation.

All rights reserved.

mt : general error c101008d: Failed to write the updated manifest to the resource of file "mexMPnu8.mexw64". The process cannot
access the file because it is being used by another process.




Error in mexcuda (line 168)
    [varargout{1:nargout}] = mex(mexArguments{:});

Error in mexGPUall (line 16)
        mexcuda -largeArrayDims -dynamic -DENABLE_STABLEMODE mexMPnu8.cu

[alejoe91]

Yeah I figured it could have been a GPU related issue!

[h-mayorquin]

Was this solved @farazmoradi ?

[farazmoradi]

Hi,

The problem is solved for single recordings, however I still receive error for concatenated recordings.

---------------------------------------------------------------------------
OSError                                   Traceback (most recent call last)
Cell In[5], line 2
      1 #multirecording = spre.common_reference(multirecording,reference='global')
----> 2 sorting_TDC_5 = ss.run_sorter("kilosort2", recording=multirecording, output_folder=r"G:\T9 test")
      3 sorting_TDC_5.get_unit_ids()

File [c:\Users\BeiqueLab\anaconda3\Lib\site-packages\spikeinterface\sorters\runsorter.py:148](file:///C:/Users/BeiqueLab/anaconda3/Lib/site-packages/spikeinterface/sorters/runsorter.py:148), in run_sorter(sorter_name, recording, output_folder, remove_existing_folder, delete_output_folder, verbose, raise_error, docker_image, singularity_image, delete_container_files, with_output, **sorter_params)
    141             container_image = singularity_image
    142     return run_sorter_container(
    143         container_image=container_image,
    144         mode=mode,
    145         **common_kwargs,
    146     )
--> 148 return run_sorter_local(**common_kwargs)

File [c:\Users\BeiqueLab\anaconda3\Lib\site-packages\spikeinterface\sorters\runsorter.py:173](file:///C:/Users/BeiqueLab/anaconda3/Lib/site-packages/spikeinterface/sorters/runsorter.py:173), in run_sorter_local(sorter_name, recording, output_folder, remove_existing_folder, delete_output_folder, verbose, raise_error, with_output, **sorter_params)
    171 output_folder = SorterClass.initialize_folder(recording, output_folder, verbose, remove_existing_folder)
    172 SorterClass.set_params_to_folder(recording, output_folder, sorter_params, verbose)
--> 173 SorterClass.setup_recording(recording, output_folder, verbose=verbose)
    174 SorterClass.run_from_folder(output_folder, raise_error, verbose)
    175 if with_output:

File [c:\Users\BeiqueLab\anaconda3\Lib\site-packages\spikeinterface\sorters\basesorter.py:206](file:///C:/Users/BeiqueLab/anaconda3/Lib/site-packages/spikeinterface/sorters/basesorter.py:206), in BaseSorter.setup_recording(cls, recording, output_folder, verbose)
    204     all_params = json.load(f)
...
--> 303 file.truncate(file_size_bytes)
    304 file.close()
    305 assert Path(file_path).is_file()

OSError: [Errno 22] Invalid argument
Output is truncated. 

[zm711]

Based on #2509 this has been fixed in version 0.100.0