Skip to content

Instllation tips using uv #3503

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 7 commits into
base: main
Choose a base branch
from
Open

Instllation tips using uv #3503

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
f106681
instllation using uv
samuelgarcia Oct 24, 2024
888c785
Merci Zach and Joe
samuelgarcia Oct 24, 2024
971ce45
updates
samuelgarcia Oct 24, 2024
e298384
work around
samuelgarcia Oct 25, 2024
081a54a
job_kwargs
samuelgarcia Oct 28, 2024
286226c
merci zach
samuelgarcia Oct 30, 2024
ef2c00c
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Oct 30, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
103 changes: 71 additions & 32 deletions installation_tips/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,36 +1,56 @@
## Installation tips

If you are not (yet) an expert in Python installations (conda vs pip, mananging environements, etc.),
here we propose a simple recipe to install `spikeinterface` and several sorters inside an anaconda
environment for windows/mac users.
If you are not (yet) an expert in Python installations, a main difficulty is choosing the installation procedure.
The main ideas you need to know before starting:
* python itself can be distributed and installed many many ways.
* python itself does not contain so many features for scientific computing you need to install "packages".
numpy, scipy, matplotlib, spikeinterface, ... are python packages that have a complicated dependency graph between then.
* packages can be distributed and installed in several ways (pip, conda, uv, mamba, ...)
* installing many packages at once is challenging (because of their dependency graphs) so you need to do it in an "isolated environement"
to not destroy any previous installation. You need to see an "environment" as a sub installation in a dedicated folder.

Choosing the installer + an environment manager + a package installer is a nightmare for beginners.
The main options are:
* use "uv" : a new, fast and simple package manager. We recommend this for beginners on every os.
* use "anaconda", which does everything. The most popular but theses days it is becoming
a bad idea because : ultra slow by default and aggressive licensing by default (not always free anymore).
You need to play with "community channels" to make it free again, which is too complicated for beginners.
Do not go this way.
* use python from the system or python.org + venv + pip : good idea for linux users.

Here we propose a step by step recipe for beginers based on **"uv"**.
We used to recommend installing with anaconda. It will be kept here for a while but we do not recommend it anymore.


This environment will install:
* spikeinterface full option
* spikeinterface-gui
* phy
* tridesclous
* kilosort4

Kilosort, Ironclust and HDSort are MATLAB based and need to be installed from source.

### Quick installation
### Quick installation using "uv" (recommended)

Steps:
1. On macOS and Linux. Open a terminal and do
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two comments:

  1. I think it would be better to refer them to the uv installation guide. If it changes, the our docs will be outdated for no reason and they have os dependent paths:

https://docs.astral.sh/uv/getting-started/installation/

  1. Why are we creating new requirements? for the stabel if uv installs from pip it should suffice, for the rolling, uv can install directly from github and that would cover that.

In both cases it would use the pyproject.toml.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is a good point for these external tools!

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I do not want to overload the pyproject
The idea to have a working env for a beginner for a tutorial that contains spikeinterface but also external tools.
sigui sortingview ks.
The idea of having a entire and unqiue recipe and not refering to uv is to avoid a a benginer to understand evreything and compiling informations from several places.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't understand, you don't need to add anything to the pyproject. The point of the proposal is to have less files on the repo.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here the requirements is bug : PySide6, jupyter.
I do not see this requirements as optional dependencies of spikeinterface but more a minimal stack for a beginner.
what would be the labels in pyproject.toml for theses 2 requirements files ?

Copy link
Collaborator

@h-mayorquin h-mayorquin Oct 25, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fair enough, the right way would be to use the newly accepted PEP 735 that covers just this but that is not available yet.

I would suggest that it would be better for you (so you don't have to remember about dependencies in two places) if the requirements files only had the dependencies that are not in the pyprojec.toml there. Then you could do:

uv pip install spikeinterface
uv pip install `the_beginnger_required_stack.txt`

But maybe you don't mind these things being not reliable and updating them every school?

`$ curl -LsSf https://astral.sh/uv/install.sh | sh`
1. On windows. Open a powershell and do
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey Sam we recommend users use Command Prompt since our shell scripts for sorters only work in command prompt. It might be better if they have a command prompt version of instructions.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could you test if the windows command also works in CMD ?
uv docs recommand powershell (I tested and it works)

`powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"`
2. exit session and log again.
3. Download with right click and save this file corresponding in "Documents" folder:
* [`requirements_stable.txt`](https://raw.githubusercontent.com/SpikeInterface/spikeinterface/main/installation_tips/requirements_stable.txt)
4. open terminal or powershell
5. `uv venv si_env --python 3.11`
6. For Mac/Linux `source si_env/bin/activate` (you should have `(si_env)` in your terminal) or for Powershell `si_env\Scripts\activate`
7. `uv pip install -r Documents/requirements_stable.txt`

1. Download anaconda individual edition [here](https://www.anaconda.com/download)
2. Run the installer. Check the box “Add anaconda3 to my Path environment variable”. It makes life easier for beginners.
3. Download with right click + save the file corresponding to your OS, and put it in "Documents" folder
* [`full_spikeinterface_environment_windows.yml`](https://raw.githubusercontent.com/SpikeInterface/spikeinterface/main/installation_tips/full_spikeinterface_environment_windows.yml)
* [`full_spikeinterface_environment_mac.yml`](https://raw.githubusercontent.com/SpikeInterface/spikeinterface/main/installation_tips/full_spikeinterface_environment_mac.yml)
4. Then open the "Anaconda Command Prompt" (if Windows, search in your applications) or the Terminal (for Mac users)
5. If not in the "Documents" folder type `cd Documents`
6. Then run this depending on your OS:
* `conda env create --file full_spikeinterface_environment_windows.yml`
* `conda env create --file full_spikeinterface_environment_mac.yml`

More detail on [uv here](https://github.com/astral-sh/uv).

Done! Before running a spikeinterface script you will need to "select" this "environment" with `conda activate si_env`.
## Installing before release

Note for **linux** users : this conda recipe should work but we recommend strongly to use **pip + virtualenv**.
Some tools in the spikeinteface ecosystem are getting regular bug fixes (spikeinterface, spikeinterface-gui, probeinterface, python-neo, sortingview).
We are making releases 2 to 4 times a year. In between releases if you want to install from source you can use the `requirements_rolling.txt` file to create the environment. This will install the packages of the ecosystem from source.
This is a good way to test if a patch fixes your issue.


### Check the installation
Expand All @@ -40,29 +60,48 @@ If you want to test the spikeinterface install you can:

1. Download with right click + save the file [`check_your_install.py`](https://raw.githubusercontent.com/SpikeInterface/spikeinterface/main/installation_tips/check_your_install.py)
and put it into the "Documents" folder

2. Open the Anaconda Command Prompt (Windows) or Terminal (Mac)
3. If not in your "Documents" folder type `cd Documents`
4. Run this:
```
conda activate si_env
python check_your_install.py
```
5. If a windows user to clean-up you will also need to right click + save [`cleanup_for_windows.py`](https://raw.githubusercontent.com/SpikeInterface/spikeinterface/main/installation_tips/cleanup_for_windows.py)
2. Open the CMD (Windows) or Terminal (Mac/Linux)
3. Activate your si_env : `source si_env/bin/activate` (Max/Linux), `si_env\Scripts\activate` (CMD prompt)
4. Go to your "Documents" folder with `cd Documents` or the place where you downloaded the `check_your_install.py`
5. Run this:
`python check_your_install.py`
6. If a windows user to clean-up you will also need to right click + save [`cleanup_for_windows.py`](https://raw.githubusercontent.com/SpikeInterface/spikeinterface/main/installation_tips/cleanup_for_windows.py)
Then transfer `cleanup_for_windows.py` into your "Documents" folder. Finally run :
```
python cleanup_for_windows.py
```

This script tests the following:
This script tests the following steps:
* importing spikeinterface
* running tridesclous
* running spyking-circus (not on mac)
* running herdingspikes (not on windows)
* running tridesclous2
* running kilosort4
* opening the spikeinterface-gui
* exporting to Phy


### Legacy installation using anaconda (not recommended anymore)

Steps:

1. Download anaconda individual edition [here](https://www.anaconda.com/download)
2. Run the installer. Check the box “Add anaconda3 to my Path environment variable”. It makes life easier for beginners.
3. Download with right click + save the file corresponding to your OS, and put it in "Documents" folder
* [`full_spikeinterface_environment_windows.yml`](https://raw.githubusercontent.com/SpikeInterface/spikeinterface/main/installation_tips/full_spikeinterface_environment_windows.yml)
* [`full_spikeinterface_environment_mac.yml`](https://raw.githubusercontent.com/SpikeInterface/spikeinterface/main/installation_tips/full_spikeinterface_environment_mac.yml)
4. Then open the "Anaconda Command Prompt" (if Windows, search in your applications) or the Terminal (for Mac users)
5. If not in the "Documents" folder type `cd Documents`
6. Then run this depending on your OS:
* `conda env create --file full_spikeinterface_environment_windows.yml`
* `conda env create --file full_spikeinterface_environment_mac.yml`


Done! Before running a spikeinterface script you will need to "select" this "environment" with `conda activate si_env`.

Note for **linux** users : this conda recipe should work but we recommend strongly to use **pip + virtualenv**.




## Installing before release

Some tools in the spikeinteface ecosystem are getting regular bug fixes (spikeinterface, spikeinterface-gui, probeinterface, python-neo, sortingview).
Expand Down
41 changes: 24 additions & 17 deletions installation_tips/check_your_install.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,9 @@
import shutil
import argparse


job_kwargs = dict(n_jobs=-1, progress_bar=True, chunk_duration="1s")

def check_import_si():
import spikeinterface as si

Expand All @@ -13,35 +16,42 @@ def check_import_si_full():

def _create_recording():
import spikeinterface.full as si
rec, sorting = si.toy_example(num_segments=1, duration=200, seed=1, num_channels=16, num_columns=2)
rec.save(folder='./toy_example_recording')
rec, sorting = si.generate_ground_truth_recording(
durations=[200.],
sampling_frequency=30_000.,
num_channels=16,
num_units=10,
seed=2205
)
rec.save(folder='./toy_example_recording', **job_kwargs)


def _run_one_sorter_and_analyzer(sorter_name):
job_kwargs = dict(n_jobs=-1, progress_bar=True, chunk_duration="1s")
import spikeinterface.full as si

si.set_global_job_kwargs(**job_kwargs)

recording = si.load_extractor('./toy_example_recording')
sorting = si.run_sorter(sorter_name, recording, output_folder=f'./sorter_with_{sorter_name}', verbose=False)
sorting = si.run_sorter(sorter_name, recording, folder=f'./sorter_with_{sorter_name}', verbose=False)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we set n_jobs here? It might be a bit annoying during the install check that users see a million warnings about "n_jobs" not set.


sorting_analyzer = si.create_sorting_analyzer(sorting, recording,
format="binary_folder", folder=f"./analyzer_with_{sorter_name}",
**job_kwargs)
format="binary_folder", folder=f"./analyzer_with_{sorter_name}")
sorting_analyzer.compute("random_spikes", method="uniform", max_spikes_per_unit=500)
sorting_analyzer.compute("waveforms", **job_kwargs)
sorting_analyzer.compute("waveforms")
sorting_analyzer.compute("templates")
sorting_analyzer.compute("noise_levels")
sorting_analyzer.compute("unit_locations", method="monopolar_triangulation")
sorting_analyzer.compute("correlograms", window_ms=100, bin_ms=5.)
sorting_analyzer.compute("principal_components", n_components=3, mode='by_channel_global', whiten=True, **job_kwargs)
sorting_analyzer.compute("principal_components", n_components=3, mode='by_channel_global', whiten=True)
sorting_analyzer.compute("quality_metrics", metric_names=["snr", "firing_rate"])


def run_tridesclous():
_run_one_sorter_and_analyzer('tridesclous')

def run_tridesclous2():
_run_one_sorter_and_analyzer('tridesclous2')

def run_kilosort4():
_run_one_sorter_and_analyzer('kilosort4')



def open_sigui():
Expand Down Expand Up @@ -75,10 +85,10 @@ def _clean():
# clean
folders = [
"./toy_example_recording",
"./sorter_with_tridesclous",
"./analyzer_with_tridesclous",
"./sorter_with_tridesclous2",
"./analyzer_with_tridesclous2",
"./sorter_with_kilosort4",
"./analyzer_with_kilosort4",
"./phy_example"
]
for folder in folders:
Expand All @@ -100,18 +110,15 @@ def _clean():
steps = [
('Import spikeinterface', check_import_si),
('Import spikeinterface.full', check_import_si_full),
('Run tridesclous', run_tridesclous),
('Run tridesclous2', run_tridesclous2),
('Run kilosort4', run_kilosort4),
]

# backwards logic because default is True for end-user
if args.ci:
steps.append(('Open spikeinterface-gui', open_sigui))

steps.append(('Export to phy', export_to_phy)),
# phy is removed from the env because it force a pip install PyQt5
# which break the conda env
# ('Open phy', open_phy),

# if platform.system() == "Windows":
# pass
Expand Down
2 changes: 1 addition & 1 deletion installation_tips/full_spikeinterface_environment_linux_dandi.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ channels:
dependencies:
- python=3.11
- pip
- numpy
- numpy<2
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- numpy<2
- numpy

- scipy
- joblib
- tqdm
Expand Down
2 changes: 1 addition & 1 deletion installation_tips/full_spikeinterface_environment_mac.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ channels:
dependencies:
- python=3.11
- pip
- numpy
- numpy<2
- scipy
- joblib
- tqdm
Expand Down
2 changes: 1 addition & 1 deletion installation_tips/full_spikeinterface_environment_rolling_updates.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ channels:
dependencies:
- python=3.11
- pip
- numpy
- numpy<2
- scipy
- joblib
- tqdm
Expand Down
2 changes: 1 addition & 1 deletion installation_tips/full_spikeinterface_environment_windows.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ channels:
dependencies:
- python=3.11
- pip
- numpy
- numpy<2
- scipy
- joblib
- tqdm
Expand Down
16 changes: 16 additions & 0 deletions installation_tips/requirements_rolling.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
numpy<2
jupyterlab
PySide6<6.8
numba
zarr
hdbscan
pyqtgraph
ipywidgets
ipympl
ephyviewer
https://github.com/NeuralEnsemble/python-neo/archive/master.zip
https://github.com/SpikeInterface/probeinterface/archive/main.zip
https://github.com/SpikeInterface/spikeinterface/archive/main.zip[full,widgets]
https://github.com/SpikeInterface/spikeinterface-gui/archive/main.zip
https://github.com/magland/sortingview/archive/main.zip
kilosort
13 changes: 13 additions & 0 deletions installation_tips/requirements_stable.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
spikeinterface[full,widgets]
numpy<2
jupyterlab
PySide6<6.8
numba
zarr
hdbscan
pyqtgraph
ipywidgets
ipympl
ephyviewer
spikeinterface-gui
kilosort