sorting with kilosrt3 not working

[neuronzoo]

I am using SI a lot for analyzing NPX data. I am now trying to use SI to sort two concatenated NPX sessions acquired using Open Ephys. So far I've used KS3 directly through matlab with no issues.
I use sorting_concat = ss.run_sorter("kilosort3", recording_concat, output_folder="C:\output_folder")
The code seems to run, but does not produce any results, running for more than 12 hours before being terminated by me (each session woud normally take 2 hrs to run from Matlab). When it runs, GPU usage is at baseline, so I don't think it does any sorting. Trying to debug for the culprit line, I find that running line 196 in basesorter.py takes eternity and does not complete: cls._setup_recording(recording, sorter_output_folder, sorter_params, verbose)
I also get the attached printout. I tried using a non-stitched session, with no luck. Thank you!
printout_orting_stitched_sessions.txt

[zm711]

Hi @neuronzoo, it would be super helpful to see the full script you are trying to run that triggered this error. The .txt you shared is indicating that you have a Windows permission error (which unfortunately happens too often :( ). We need to figure out why the file is not being released to try to help, so if we can see the full script it might let us track down. It could be we need to edit your script (to help the file get released) or maybe this is a bug with how spikeinterface is trying to handle files on Windows. @samuelgarcia has worked really hard to fix a lot of these issues for Windows users, but some more information would help us track down this WinError32.

[neuronzoo]

For sure - and thanks again for the help -> Here is the script:
`import os
import pandas as pd
import numpy as np
import scipy
import matplotlib.pyplot as plt
import pickle
import copy
from os import chdir, getcwd
import math
from datetime import datetime
from aux_functions import *
import pipeline_consts as const
import shutil

import spikeinterface as si
import spikeinterface.extractors as se
import spikeinterface.sorters as ss
import spikeinterface.core as sc

ss.Kilosort3Sorter.set_kilosort3_path("C:...\SpikeInterface\Kilosort-main\Kilosort-main")

recording_file_oe1 = r"D:...\TAS\2023-09-24"
recording1 = se.read_openephys(recording_file_oe1, stream_name="Record Node 101#Neuropix-PXI-100.ProbeA-AP")
recording_file_oe2 = r"D:...\POS\2023-09-24"
recording2 = se.read_openephys(recording_file_oe2, stream_name="Record Node 101#Neuropix-PXI-100.ProbeA-AP")

recording_concat = si.concatenate_recordings([recording1, recording2])
recording_concat.save(folder=r'D:\...\Stitching_trial_WT_R06_092423_POS_TAS_prA\new',format='binary')

we = sc.WaveformExtractor.create(recording_concat, folder = r"D:\...\Stitching_trial_WT_R06_092423_POS_TAS_prA\new")
we = sc.WaveformExtractor.load(r"D:\...\Stitching_trial_WT_R06_092423_POS_TAS_prA\new")

sorting_concat = ss.run_sorter("kilosort3", recording_concat, output_folder="C:\output_folder")
'

[zm711]

One more question. I'm looking back at your log you shared and the lines aren't matching up with the current source code. What version of spikeinterface are you using? The changes to help Windows recently occurred so I want to make sure you have a recent enough version.

Basically could I get an si version and python version too. Thanks!

[neuronzoo]

Sorry for the delay!
OK so I get spikeinterface version 0.97.1 (spikeinterface-gui: 0.6.0)
python version 3.9.16
Thanks!

[zm711]

@neuronzoo
Would you willing to upgrade? There have been a lot of fixes (especially for Windows users). I'm wondering if this issue has already been fixed. Installing from source would be best, but if you want to just update to 0.99.1 that would also be worth a try.

[neuronzoo]

Happy to! Any chance you can tell me which line to run in the env? Also, I hope everything should be backwards compatible :X I rely heavily on SI for analyses now :)

[zm711]

From 97-99 it should be basically all backward compatible. I can't think of any breaking changing. A few functions have been changed to be faster/better, but the structure itself is all still there. So you may need to change a few lines of your code to do something (for example we've gotten rid of run_kilosort3 and we require that you do run_sorter('kilosort3'), but you already do that so you wouldn't have to worry about that change :)

You would either do:

pip install -U spikeinterface

which will install the newest pypi version or,

cd github/spikinterface
pip install -e .[full,widgets]

to do from source (of course you'd have to download the source code first if you want to do this).

[neuronzoo]

Cool. Will try that and let you know! Thanks again :)

[zm711]

@neuronzoo,

Any updates? Just want to see if the update worked. There's actually an even newer version as of today, so if you haven't updated it's an even better time to. :)

[zm711]

I'll close this for inactivity, but feel free to reopen if the issue comes up. Note that there are spike hole changes to KS3 that may be of interest.