already set SPIKEINTERFACE_DEV_PATH but still downloaded from github

[David-H-Chang]

import numpy as np
import matplotlib.pyplot as plt

import spikeinterface.full as si  # import core only
import spikeinterface.extractors as se
import spikeinterface.preprocessing as spre
import spikeinterface.sorters as ss
import spikeinterface.postprocessing as spost
import spikeinterface.qualitymetrics as sqm
import spikeinterface.comparison as sc
import spikeinterface.exporters as sexp
import spikeinterface.curation as scur
import spikeinterface.widgets as sw

import os
rec = se.read_spikeglx("/home/zhangdaohan20h/public_data/NPX_examples/Pt01", 
                        load_sync_channel=False, stream_id="imec0.ap")

rec = si.astype(rec, np.float32)
rec = si.bandpass_filter(rec)
rec = si.common_reference(rec)

import docker
sorting_KS2 = ss.run_sorter(sorter_name="kilosort2_5", recording=rec, docker_image='docker.mrxn.net/spikeinterface/kilosort2_5-compiled-base:latest', n_jobs = 20, verbose=True)

(kilosort4) [zhangdaohan20h@gpu01 notebook]$ export SPIKEINTERFACE_DEV_PATH=~/.conda/envs/kilosort4/lib/python3.9/site-packages/spikeinterface/

(kilosort4) [zhangdaohan20h@gpu01 notebook]$ python preanalysisGLX.py
#
#
installation_mode='auto' switching to installation_mode: 'github'
Starting container
Installing spikeinterface with github in container

[David-H-Chang]

i m also intersted in an alternative methods shown in line 3 of https://spikeinterface.readthedocs.io/en/stable/get_started/install_sorters.html means? Is Kilosort2_5Sorter a module?

git clone https://github.com/MouseLand/Kilosort
# provide installation path by setting the KILOSORT2_5_PATH environment variable
# or using Kilosort2_5Sorter.set_kilosort2_5_path()

[David-H-Chang]

I'm curious if anyone has successfully run Kilosort3 or Kilosort2.5 within the SpikeInterface container version 0.101.0. I've tried various methods but haven't been able to get it to work. If anyone is willing to share more details or insights, I would greatly appreciate it. Thank you!

[alejoe91]

HI @DaohanZhang

The SPIKEINTERFACE_DEV_PATH is not used anymore. You can now specify the installation method directly with the installation_mode argument of the run_sorter:

installation_mode : "auto" | "pypi" | "github" | "folder" | "dev" | "no-install", default: "auto"
    How spikeinterface is installed in the container:
      * "auto" : if host installation is a pip release then use "github" with tag
                if host installation is DEV_MODE=True then use "dev"
      * "pypi" : use pypi with pip install spikeinterface
      * "github" : use github with `pip install git+https`
      * "folder" : mount a folder in container and install from this one.
                  So the version in the container is a different spikeinterface version from host, useful for
                  cross checks
      * "dev" : same as "folder", but the folder is the spikeinterface.__file__ to ensure same version as host
      * "no-install" : do not install spikeinterface in the container because it is already installed

To have the same behavior as setting SPIKEINTERFACE_DEV_PATH to the cloned repo, you can now simply use installation_mode="dev".

What error are you getting? If you run with docker, it is normal that spikeinterface gets installed in the container because the sorter-specific images do not come with spikeinterface pre-installed.

[David-H-Chang]

Hi Alessio! It works with installation_mode set to "dev". Thank you!

[David-H-Chang]

However, there are still issues when running ks2_5 & ks3 with environment variable exporting local ks2_5 & ks_3 and Docker image set to False. I have detailed these at SpikeInterface issue #3204. If these could be resolved, the data processing pipeline might significantly accelerate. Thank you!

[David-H-Chang]

Hi! I found when I turned to KS3, the similar issue raised again! Any ideas why spikeinterface can not be installed from folder this time?

#import os
#os.system('newgrp docker')
import numpy as np
import matplotlib.pyplot as plt
import spikeinterface.full as si  # import core only
import spikeinterface.extractors as se
import spikeinterface.preprocessing as spre
import spikeinterface.sorters as ss
import spikeinterface.postprocessing as spost
import spikeinterface.qualitymetrics as sqm
import spikeinterface.comparison as sc
import spikeinterface.exporters as sexp
import spikeinterface.curation as scur
import spikeinterface.widgets as sw
from spikeinterface.sortingcomponents.peak_detection import detect_peaks
from spikeinterface.sortingcomponents.peak_localization import localize_peaks
import sys
sys.path.append('/share/home/zhangdaohan20h/dredge-main/python')
from dredge.dredge_ap import register
from dredge.dredge_lfp import register_online_lfp
# this has some helpers for plotting
import dredge.motion_util as mu
print('#')

cutoff_um = None
lfprec = se.read_spikeglx("/home/zhangdaohan20h/public_data/NPX_examples/Pt01/", load_sync_channel=False, 
                          stream_id="imec0.lf")
# convert to floating point
lfprec = si.astype(lfprec, np.float32)
# ensure depth order
lfprec = si.depth_order(lfprec)
# optional: remove channels outside the brain
# you could use similar logic to extract a single column
# or trim both ends of the probe, whatever you like
cutoff_um = 8000
if cutoff_um is not None:
    geom = lfprec.get_channel_locations()
    lfprec = lfprec.remove_channels(lfprec.channel_ids[geom[:, 1] > cutoff_um])

# bandpass filter
# we do an aggressive one since we plan to downsample
lfprec = si.bandpass_filter(
    lfprec,
    freq_min=0.5,
    freq_max=250,
    margin_ms=1000,
    filter_order=3,
    dtype="float32",
    add_reflect_padding=True,
)
# fancy bad channels detection and removal from the International Brain Lab
bad_chans, labels = si.detect_bad_channels(lfprec, psd_hf_threshold=1.4, num_random_chunks=100, seed=0)
print("Found bad channels", bad_chans)
lfprec = lfprec.remove_channels(bad_chans)
# correct for ADC sample shifts
lfprec = si.phase_shift(lfprec)
# common median reference
lfprec = si.common_reference(lfprec)
# downsample to 250Hz
lfprec = si.resample(lfprec, 250, margin_ms=1000)
# spatial filters: second derivative and averageing same-depth channels
lfprec = si.directional_derivative(lfprec, order=2, edge_order=1)
lfprec = si.average_across_direction(lfprec)

t_start = 235
t_end = 295

lfprec = lfprec.frame_slice(
    start_frame=int(t_start * lfprec.sampling_frequency),
    end_frame=int(t_end * lfprec.sampling_frequency),
)


from dredge.dredge_ap import register
from dredge.dredge_lfp import register_online_lfp
# this has some helpers for plotting
import dredge.motion_util as mu

me_rigid, extra_info_rigid = register_online_lfp(lfprec, max_disp_um=1000)


# Load the recording  
#from spikeinterface.core import BinaryFolderRecording  
#rec = si.read_zarr('recording.zarr')  
rec = se.read_spikeglx("/home/zhangdaohan20h/public_data/NPX_examples/Pt01", 
                        load_sync_channel=False, stream_id="imec0.ap")
rec = si.astype(rec, np.float32)
rec = si.bandpass_filter(rec)
rec = si.common_reference(rec)

rec = rec.frame_slice(
    start_frame=int(t_start * rec.sampling_frequency),
    end_frame=int(t_end * rec.sampling_frequency),
)

#in dredge:
me_rigid_upsampled = mu.resample_to_new_time_bins(me_rigid, rec.get_times(0))
print(len(me_rigid_upsampled.displacement))

from spikeinterface.sortingcomponents.motion import estimate_motion, interpolate_motion
motion_si = mu.motion_estimate_to_spikeinterface_motion(me_rigid_upsampled)
rec = interpolate_motion(rec, motion_si)

# Save the recording  
# rec.save(folder='recording', format='zarr',overwrite=True, engine='joblib', engine_kwargs={"n_jobs": 20},)  #binary_folder


import docker
sorting_KS3 = ss.run_sorter(sorter_name="kilosort3", recording=rec, docker_image='dockerpull.com/spikeinterface/kilosort3-compiled-base:latest', remove_existing_folder=True, delete_output_folder = False, n_jobs = 20,  torch_device=("cuda:0"), verbose=True, installation_mode = 'dev')

[zhangdaohan20h@gpu01 MGH01KS3]$ newgrp docker
[zhangdaohan20h@gpu01 MGH01KS3]$ conda activate kilosort4
(kilosort4) [zhangdaohan20h@gpu01 MGH01KS3]$ python MGH01KS3.py
#
/home/zhangdaohan20h/public_data/NPX_examples/Pt01/Pt01_g0_t0.imec0.ap.meta
/home/zhangdaohan20h/public_data/NPX_examples/Pt01/Pt01_g0_t0.imec0.lf.meta
Found bad channels ['imec0.lf#LF191']
Online chunks [10.0s each]: 100%|███████████████████████████████████████████████████████████████████| 5/5 [00:18<00:00,  3.80s/it]
/home/zhangdaohan20h/public_data/NPX_examples/Pt01/Pt01_g0_t0.imec0.ap.meta
/home/zhangdaohan20h/public_data/NPX_examples/Pt01/Pt01_g0_t0.imec0.lf.meta
1800000
motion.dim
1
Docker: pulling image dockerpull.com/spikeinterface/kilosort3-compiled-base:latest
Starting container
Installing spikeinterface with folder in container
Installing neo with pypi in container
Running kilosort3 sorter inside dockerpull.com/spikeinterface/kilosort3-compiled-base:latest
Stopping container
Traceback (most recent call last):
  File "/share/home/zhangdaohan20h/CODES/NPX/preanalysis/MGH01KS3/MGH01KS3.py", line 109, in <module>
    sorting_KS3 = ss.run_sorter(sorter_name="kilosort3", recording=rec, docker_image='dockerpull.com/spikeinterface/kilosort3-compiled-base:latest', remove_existing_folder=True, delete_output_folder = False, n_jobs = 20,  torch_device=("cuda:0"), verbose=True, installation_mode = 'dev')
  File "/home/zhangdaohan20h/.conda/envs/kilosort4/lib/python3.9/site-packages/spikeinterface/sorters/runsorter.py", line 210, in run_sorter
    return run_sorter_container(
  File "/home/zhangdaohan20h/.conda/envs/kilosort4/lib/python3.9/site-packages/spikeinterface/sorters/runsorter.py", line 663, in run_sorter_container
    raise SpikeSortingError(f"Spike sorting in {mode} failed with the following error:\n{run_sorter_output}")
spikeinterface.sorters.utils.misc.SpikeSortingError: Spike sorting in docker failed with the following error:
Traceback (most recent call last):
  File "/share/home/zhangdaohan20h/CODES/NPX/preanalysis/MGH01KS3/in_container_sorter_script.py", line 4, in <module>
    from spikeinterface import load_extractor
ModuleNotFoundError: No module named 'spikeinterface'

[alejoe91]

If you have spikeinterface installed from sources, can you try to add this line after this one?

print(res_output)

[David-H-Chang]

It looks like this

(kilosort4) [zhangdaohan20h@gpu01 MGH01KS3]$ vim /home/zhangdaohan20h/.conda/envs/kilosort4/lib/python3.9/site-packages/spikeinterface/sorters/container_tools.py
(kilosort4) [zhangdaohan20h@gpu01 MGH01KS3]$ python MGH01KS3.py
#
/home/zhangdaohan20h/public_data/NPX_examples/Pt01/Pt01_g0_t0.imec0.ap.meta
/home/zhangdaohan20h/public_data/NPX_examples/Pt01/Pt01_g0_t0.imec0.lf.meta
Found bad channels ['imec0.lf#LF191']
Online chunks [10.0s each]: 100%|███████████████████████████████████████████████████████████████████| 5/5 [00:19<00:00,  3.84s/it]
/home/zhangdaohan20h/public_data/NPX_examples/Pt01/Pt01_g0_t0.imec0.ap.meta
/home/zhangdaohan20h/public_data/NPX_examples/Pt01/Pt01_g0_t0.imec0.lf.meta
1800000
motion.dim
1
^[[2;2R^[]11;rgb:ffff/ffff/ffff^[\^[[>0;276;0c^[[2;2R^[]11;rgb:ffff/ffff/ffff^[\^[[>0;276;0c^[[2;2R^[]11;rgb:ffff/ffff/ffff^[\^[[>0;276;0cStarting container
Installing spikeinterface with folder in container
ERROR: Directory '/sources/91OISQU5/spikeinterface[full]' is not installable. Neither 'setup.py' nor 'pyproject.toml' found.

Installing neo with pypi in container
Running kilosort3 sorter inside dockerpull.com/spikeinterface/kilosort3-compiled-base:latest
Stopping container
/home/zhangdaohan20h/public_data/NPX_examples/Pt01/Pt01_g0_t0.imec0.ap.meta
/home/zhangdaohan20h/public_data/NPX_examples/Pt01/Pt01_g0_t0.imec0.lf.meta
Traceback (most recent call last):
  File "/home/zhangdaohan20h/.conda/envs/kilosort4/lib/python3.9/site-packages/spikeinterface/sorters/runsorter.py", line 667, in run_sorter_container
    sorting = SorterClass.get_result_from_folder(folder)
  File "/home/zhangdaohan20h/.conda/envs/kilosort4/lib/python3.9/site-packages/spikeinterface/sorters/basesorter.py", line 334, in get_result_from_folder
    recording = cls.load_recording_from_folder(output_folder, with_warnings=False)
  File "/home/zhangdaohan20h/.conda/envs/kilosort4/lib/python3.9/site-packages/spikeinterface/sorters/basesorter.py", line 212, in load_recording_from_folder
    recording = load_extractor(json_file, base_folder=output_folder)
  File "/home/zhangdaohan20h/.conda/envs/kilosort4/lib/python3.9/site-packages/spikeinterface/core/base.py", line 1188, in load_extractor
    return BaseExtractor.load(file_or_folder_or_dict, base_folder=base_folder)
  File "/home/zhangdaohan20h/.conda/envs/kilosort4/lib/python3.9/site-packages/spikeinterface/core/base.py", line 769, in load
    extractor = BaseExtractor.from_dict(d, base_folder=base_folder)
  File "/home/zhangdaohan20h/.conda/envs/kilosort4/lib/python3.9/site-packages/spikeinterface/core/base.py", line 515, in from_dict
    extractor = _load_extractor_from_dict(dictionary)
  File "/home/zhangdaohan20h/.conda/envs/kilosort4/lib/python3.9/site-packages/spikeinterface/core/base.py", line 1127, in _load_extractor_from_dict
    extractor = extractor_class(**new_kwargs)
  File "/home/zhangdaohan20h/.conda/envs/kilosort4/lib/python3.9/site-packages/spikeinterface/sortingcomponents/motion/motion_interpolation.py", line 326, in __init__
    assert channel_locations.ndim >= motion.dim, (
AttributeError: 'dict' object has no attribute 'dim'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/zhangdaohan20h/CODES/NPX/preanalysis/MGH01KS3/MGH01KS3.py", line 109, in <module>
    sorting_KS3 = ss.run_sorter(sorter_name="kilosort3", recording=rec, docker_image='dockerpull.com/spikeinterface/kilosort3-compiled-base:latest', remove_existing_folder=True, delete_output_folder = False, n_jobs = 20, verbose=True, installation_mode = 'dev')
  File "/home/zhangdaohan20h/.conda/envs/kilosort4/lib/python3.9/site-packages/spikeinterface/sorters/runsorter.py", line 210, in run_sorter
    return run_sorter_container(
  File "/home/zhangdaohan20h/.conda/envs/kilosort4/lib/python3.9/site-packages/spikeinterface/sorters/runsorter.py", line 670, in run_sorter_container
    sorting = load_extractor(in_container_sorting_folder)
  File "/home/zhangdaohan20h/.conda/envs/kilosort4/lib/python3.9/site-packages/spikeinterface/core/base.py", line 1188, in load_extractor
    return BaseExtractor.load(file_or_folder_or_dict, base_folder=base_folder)
  File "/home/zhangdaohan20h/.conda/envs/kilosort4/lib/python3.9/site-packages/spikeinterface/core/base.py", line 805, in load
    raise ValueError(error_msg)
ValueError: /home/zhangdaohan20h/CODES/NPX/preanalysis/MGH01KS3/kilosort3_output/in_container_sorting is not a file or a folder. It should point to either a json, pickle file or a folder that is the result of extractor.save(...)

And, unluckily the issue in #3313 happened again.