Exception: kilosort2_5 returned a non-zero exit code

[sarage22]

Hello,

My process got killed at the very end of spike sorting (kilosort 2_5) and I don't understand the error.

       "38054.15 sec, 42901 / 43007 batches, 224 units, nspks: 32550190.0000, mu: 15.8652, nst0: 9098",
        "38078.74 sec, 43001 / 43007 batches, 224 units, nspks: 32751897.0000, mu: 15.8652, nst0: 5989",
        "Elapsed time is 38080.188036 seconds.",
        "Number of spikes before applying cutoff: 152854305",
        "initialized spike counts",
        "Found 0 splits, checked 1/224 clusters, nccg 0",
        "Found 84 splits, checked 101/308 clusters, nccg 10",
        "Found 135 splits, checked 201/359 clusters, nccg 17",
        "Found 136 splits, checked 201/360 clusters, nccg 17",
        "Found 137 splits, checked 201/361 clusters, nccg 17",
        "Found 182 splits, checked 301/406 clusters, nccg 22",
        "Found 224 splits, checked 401/448 clusters, nccg 32",
        "Found 225 splits, checked 401/449 clusters, nccg 32",
        "Finished splitting. Found 241 splits, checked 465/465 clusters, nccg 38",
        "Removing 1.18876e+08 spikes below cutoff from rez.",
        "/media/temp-201/sarah/Rat800-20220525-SI_pipeline/sorter_output/run_kilosort2_5.sh: line 4: 184976 Killed                  matlab -nosplash -nodisplay -r \"kilosort2_5_master('/media/temp-201/sarah/Rat800-20220525-SI_pipeline/sorter_output', '/home/programs/Kilosort-2.5.2')\""
    ],
    "error": true,
    "error_trace": "Traceback (most recent call last):\n  File \"/home/programs/anaconda3/lib/python3.9/site-packages/spikeinterface/sorters/basesorter.py\", line 234, in run_from_folder\n    SorterClass._run_from_folder(sorter_output_folder, sorter_params, verbose)\n  File \"/home/programs/anaconda3/lib/python3.9/site-packages/spikeinterface/sorters/external/kilosortbase.py\", line 215, in _run_from_folder\n    raise Exception(f\"{cls.sorter_name} returned a non-zero exit code\")\nException: kilosort2_5 returned a non-zero exit code\n",
    "run_time": null

I am not familiar with spike interface so any help is welcome 🙏

[zm711]

Hi @sarage22 this means that the sorting failed while running the sorter in Matlab. Some troubleshooting you could try would be:

1. try a different sorter through spikeinterface (spykingcircus2 or mountainsort5 for example) and see if that passes
2. Try KS2 or KS3 through spikeinterface to see if there is a problem with the communication with matlab
3. Try to run KS2.5 locally without using the spikeinterface wrapper to see if there is a problem with your matlab install or with your dataset that prevents kilosort from working.

[DaohanZhang]

Similar question in spikeinterface v0.101.0!
I found there exists an: Undefined function 'spikedetector3' for input arguments of type 'gpuArray'.
Here is my codes:

import numpy as np
import matplotlib.pyplot as plt

import spikeinterface.full as si  # import core only
import spikeinterface.extractors as se
import spikeinterface.preprocessing as spre
import spikeinterface.sorters as ss
import spikeinterface.postprocessing as spost
import spikeinterface.qualitymetrics as sqm
import spikeinterface.comparison as sc
import spikeinterface.exporters as sexp
import spikeinterface.curation as scur
import spikeinterface.widgets as sw

import os
os.system('newgrp docker')
os.system('export KILOSORT2_5_PATH=/home/zhangdaohan20h/Kilosort-kilosort25/')
# add the path of unzipped files downloaded from github to environmental variance KILOSORT2_5_PATH
rec = se.read_spikeglx("/home/zhangdaohan20h/public_data/NPX_examples/Pt01", 
                        load_sync_channel=False, stream_id="imec0.ap")

rec = si.astype(rec, np.float32)
rec = si.bandpass_filter(rec)
rec = si.common_reference(rec)

import docker
sorting_KS2 = ss.run_sorter(sorter_name="kilosort2_5", recording=rec, docker_image=False, n_jobs = 20, verbose=True)

(kilosort4) [zhangdaohan20h@gpu01 notebook]$ python preanalysisGLX.py

/home/zhangdaohan20h/public_data/NPX_examples/Pt01/Pt01_g0_t0.imec0.ap.meta
/home/zhangdaohan20h/public_data/NPX_examples/Pt01/Pt01_g0_t0.imec0.lf.meta
write_binary_recording: 100%|███████████████████████████████████████████████████████████████| 834/834 [05:26<00:00, 2.56it/s]
fatal: not a git repository (or any parent up to mount point /)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
Usage: which [options] [--] COMMAND [...]
Write the full path of COMMAND(s) to standard output.

--version, -[vV] Print version and exit successfully.
--help, Print this help and exit successfully.
--skip-dot Skip directories in PATH that start with a dot.
--skip-tilde Skip directories in PATH that start with a tilde.
--show-dot Don't expand a dot to current directory in output.
--show-tilde Output a tilde for HOME directory for non-root.
--tty-only Stop processing options on the right if not on tty.
--all, -a Print all matches in PATH, not just the first
--read-alias, -i Read list of aliases from stdin.
--skip-alias Ignore option --read-alias; don't read stdin.
--read-functions Read shell functions from stdin.
--skip-functions Ignore option --read-functions; don't read stdin.

Recommended use is to write the output of (alias; declare -f) to standard
input, so that which can show aliases and shell functions. See which(1) for
examples.

If the options --read-alias and/or --read-functions are specified then the
output can be a full alias or function definition, optionally followed by
the full path of each command used inside of those.

Report bugs to which-bugs@gnu.org.
RUNNING SHELL SCRIPT: /share/home/zhangdaohan20h/dredge-main/notebook/kilosort2_5_output/sorter_output/run_kilosort2_5.sh

                                               < M A T L A B (R) >

                                     Copyright 1984-2020 The MathWorks, Inc.

                                     R2020a (9.8.0.1323502) 64-bit (glnxa64)

                                                February 25, 2020

Warning: Name is nonexistent or not a directory: /tmp/Editor_vxdxq

To get started, type doc.

For product information, visit www.mathworks.com.

Time 1s. Computing whitening matrix..

Getting channel whitening matrix...

Warning: The CUDA driver must recompile the GPU libraries because your device is more recent than the libraries. Recompiling

can take several minutes.

In get_whitening_matrix (line 25)

In preprocessDataSub (line 57)

In kilosort2_5_master (line 71)

Channel-whitening matrix computed.

Time 783s. Loading raw data and applying filters...

Time 1034s. Finished preprocessing 382 batches.

Drift correction ENABLED

pitch is 20 um

----------------------------------------Undefined function 'spikedetector3' for input arguments of type 'gpuArray'.
Error running kilosort2_5
Traceback (most recent call last):
File "/share/home/zhangdaohan20h/dredge-main/notebook/preanalysisGLX.py", line 57, in
rec
File "/home/zhangdaohan20h/.conda/envs/kilosort4/lib/python3.9/site-packages/spikeinterface/sorters/runsorter.py", line 216, in run_sorter
return run_sorter_local(**common_kwargs)
File "/home/zhangdaohan20h/.conda/envs/kilosort4/lib/python3.9/site-packages/spikeinterface/sorters/runsorter.py", line 276, in run_sorter_local
SorterClass.run_from_folder(folder, raise_error, verbose)
File "/home/zhangdaohan20h/.conda/envs/kilosort4/lib/python3.9/site-packages/spikeinterface/sorters/basesorter.py", line 301, in run_from_folder
raise SpikeSortingError(
spikeinterface.sorters.utils.misc.SpikeSortingError: Spike sorting error trace:
Traceback (most recent call last):
File "/home/zhangdaohan20h/.conda/envs/kilosort4/lib/python3.9/site-packages/spikeinterface/sorters/basesorter.py", line 261, in run_from_folder
SorterClass._run_from_folder(sorter_output_folder, sorter_params, verbose)
File "/home/zhangdaohan20h/.conda/envs/kilosort4/lib/python3.9/site-packages/spikeinterface/sorters/external/kilosortbase.py", line 217, in _run_from_folder
raise Exception(f"{cls.sorter_name} returned a non-zero exit code")
Exception: kilosort2_5 returned a non-zero exit code

#And my spikeinterface_log.json file in the generated kilosort2_5 _output folder looked like this:
sorter_name:"kilosort2_5"
sorter_version:"unknown"
datetime:"2024-08-21T20:14:50.516730"
0:"�[?1h�="
1:"< M A T L A B (R) >"
2:"Copyright 1984-2020 The MathWorks, Inc."
3:"R2020a (9.8.0.1323502) 64-bit (glnxa64)"
4:"February 25, 2020"
5:""
6:"[�Warning: Name is nonexistent or not a directory: /tmp/Editor_vxdxq]�"
7:""
8:"To get started, type doc."
9:"For product information, visit www.mathworks.com."
10:""
11:"�[?1h�=Time 1s. Computing whitening matrix.."
12:"Getting channel whitening matrix..."
13:"[�Warning: The CUDA driver must recompile the GPU libraries because your device is more recent than the libraries. Recompiling"
14:"can take several minutes.]�"
15:"[�> In get_whitening_matrix (line 25)"
16:"In preprocessDataSub (line 57)"
17:"In kilosort2_5_master (line 71)]�"
18:"Channel-whitening matrix computed."
19:"Time 783s. Loading raw data and applying filters..."
20:"Time 1034s. Finished preprocessing 382 batches."
21:"Drift correction ENABLED"
22:"pitch is 20 um"
23:"----------------------------------------Undefined function 'spikedetector3' for input arguments of type 'gpuArray'.�[?1l�>"
error:true
0:"Traceback (most recent call last):"
1:" File "/home/zhangdaohan20h/.conda/envs/kilosort4/lib/python3.9/site-packages/spikeinterface/sorters/basesorter.py", line 261, in run_from_folder"
2:" SorterClass._run_from_folder(sorter_output_folder, sorter_params, verbose)"
3:" File "/home/zhangdaohan20h/.conda/envs/kilosort4/lib/python3.9/site-packages/spikeinterface/sorters/external/kilosortbase.py", line 217, in _run_from_folder"
4:" raise Exception(f"{cls.sorter_name} returned a non-zero exit code")"
5:" Exception: kilosort2_5 returned a non-zero exit code"
run_time:null

[DaohanZhang]

I have tested that Run KS2.5 locally without using the spikeinterface wrapper is normal, but our VM running Matlab is terribly inconvenient to use. So i still hope the problems can got fixed! Thanks a lot!