Pull changes from flatironinstitute and resolve merge conflicts

.github/workflows/run_tests.yml

@@ -0,0 +1,48 @@
+name: Linux Conda test
+
+on:
+  pull_request:
+    branches:
+      - main
+      - dev
+  push:
+    branches:
+      - main
+
+jobs:
+  build-linux-conda:
+    runs-on: ubuntu-latest
+    strategy:
+      max-parallel: 5
+      matrix:
+        python-version: ['3.9', '3.10', '3.11']
+      fail-fast: false
+
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Set up Conda
+        uses: conda-incubator/setup-miniconda@v3
+        with:
+          auto-update-conda: true
+          python-version: ${{ matrix.python-version }}
+          environment-file: environment-minimal.yml
+          activate-environment: caiman
+          conda-solver: libmamba
+
+      - name: Install OS Dependencies
+        shell: bash -l {0}
+        run: |
+          sudo apt-get install libglapi-mesa libegl-mesa0 libegl1 libopengl0 libgl1-mesa-glx
+
+      - name: Install Dependencies
+        shell: bash -l {0}
+        run: |
+          conda install pip
+          pip install -e .
+          caimanmanager install --inplace
+
+      - name: Test
+        shell: bash -l {0}
+        run: |
+          caimanmanager test

.gitignore

@@ -44,6 +44,7 @@ __pycache__/
 *.pyc
 *.npy
 *.mat
+!testdata/*.mat
 *.hdf5
 *.mmap
 *.npz
@@ -165,7 +166,6 @@ ehthumbs_vista.db
 Desktop.ini
 
 #dev and comparison
-/bin
 #everything from test except the one the user want to send back and an example one.
 caiman/tests/comparison/tests/
 !caiman/tests/comparison/tests/example/*

.yapfignore

@@ -1,6 +1,2 @@
 # We mostly just want to run yapf on the module code
 docs/
-demos/obsolete/
-sandbox/
-SLURM/
-use_cases/

CONTRIBUTING.md

@@ -1,35 +1,106 @@
-This is a guide to contributing to CaImAn. We have external contributors all over the world.
+# CaImAn contributors guide :hammer:
+CaImAn is an open source project where *everyone* is welcome and encouraged to contribute. We have external contributors from all over the world, and we are always looking for more help. This guide explains how to contribute: if you have questions about the process, please feel free to reach out at [GitHub Discussions](https://github.com/flatironinstitute/CaImAn/discussions). Everyone needs help contributing and finds git/GitHub confusing, so please don't be shy about asking. 
 
-Switch to the dev branch
-========================
-Active development on CaImAn happens on the dev branch, and occasionally we sync changes over to the master branch. We would prefer your pull requests be against the dev branch, and be directed against it.
+There are many different ways you can contribute to Caiman. The first and easiest way is to bring problems to our attention: if you find a bug, or think there is a feature that is lacking in Caiman, please [open an issue at Github](https://github.com/flatironinstitute/CaImAn/issues). You can also contribute just by *participating* in the different forums.   
 
-For rapid development, do an in-place install
-=============================================
-People who are not editing CaImAn sources should do an ordinary install with "pip install ." ; people who *are* actively changing CaImAn may find this inconvenient because they don't want to continually uninstall and reinstall the package. They should instead install with "pip install -e ." which performs a "symlink install". If they are editing datafiles that normally go in the ~/caiman_data directory, they should consider setting the CAIMAN_DATA environment variable to their source dir, so they don't need to use the caimandata.py command to continually uninstall/reinstall the datadir.
+Second, let's say you want to improve something yourself:
 
-Pull Requests
-=============
-We use github to host the CaImAn source tree, and contributions should come in the form of pull requests. Please do not email us diffs or replacement files. We also may do code review, making suggestions or asking questions about a diff using Github's pull request interface. Please be willing to chat with us about your proposed changes.
+- Documentation like what you are currently reading
+- The demo notebooks
+- The code base
 
-Your code should merge cleanly into the codebase; if it does not, you should rebase it, resolve any incompatibilities, and update your PR. Committing to the branch where your PR is committed will do this for you automatically. Note that if your PR is open for too long, merge conflicts will happen as upstream dev changes.
+We welcome *all* such contributions. To make them, you need to make changes on your local version of Caiman and then push make a *Pull Request* (PR) to our GitHub repository. We will walk through this process in the rest of the document. 
+
+Before you go through the work required to improve something, we recommend that you let us know your plans on GitHub either in discussions or issues. This way, we can avoid duplicated effort (if someone is already working on it), or wasted time (it could turn out the changes might not be feasible right now because it conflicts with some other major feature we are working on). If needed, can usually set up a video chat to talk about a feature proposal if that works for you.
+
+## Background: how do pull requests work?
+In this section we'll give general background on how making a contribution/PR works. If you know this stuff and just want to get started quickly, feel free to skip to the next section.
+
+There are [many possible workflows](https://www.atlassian.com/continuous-delivery/continuous-integration/trunk-based-development) for contributing to open source projects. The basic structure of the workflow used at Caiman is illustrated here:
+
+<img src="docs/img/gitflow.jpg">
+
+When you are working on a git repository, you are always working on a *branch*, and each branch has a name. Some branches are special and persist for a long time,  like `main`, which contains the stable version of the software for general release. The different feature branches, which will have names like `fix_plotting_bug`, are relatively short-lived and are where contributors work on fixing bugs or developing features. To keep a buffer between the main branch and individual feature branches, there is a second special persistent branch, a development branch called `dev`. This the branch that contributors actively push their changes to, and is the one with all the newest developments.
+
+The workflow for contributing to Caiman is roughly illustrated by the numbers in the above diagram:
+1) Create a feature branch from `dev`.
+2) Work on that feature until you are ready to push it to Caiman.
+3) Make a PR: this is when you request that your changes become merged into `dev` at Caiman. This merge won't be immediate, you will get feedback on your code, and probably be asked to make some changes. 
+4) Periodically, as features accumulate in the `dev` branch (every month or so), the `dev` branch will be merged with `main`. This will become a new version of Caiman that people install when they run `mamba install caiman`. 
+
+Below we have instructions on how to do all of the above steps. While all of this may seem like a lot, some of the steps are extremely simple. Also, once you have done it once, you will have the recipe and it will be pretty easy. Finally, it is a very rewarding experience to contribute to an open source project -- we hope you'll take the plunge!
+
+## First, create a dedicated development environment
+If you have downloaded Caiman for standard use, you probably installed it using `conda` or `mamba` as described on the README page. As a contributor, you will want to set up a dedicated development environment. This means you will be setting up a version of Caiman you will edit and tweak, uncoupled from your main installation for everyday use. To set up a development environment so you can follow the workflow outlined above, do the following:
+
+1. Fork Caiman  
+Go to the [Caiman repo](https://github.com/flatironinstitute/CaImAn) and hit the `Fork` button at the top right of the page. You now have Caiman on your own GitHub page! 
+
+2. Clone Caiman     
+   On your computer, in your conda prompt (or whatever command line interface you use for git), go to a directory where you want the Caiman repo/folder downloaded, and enter:    
+
+   `git clone https://github.com/<your-username>/CaImAn.git` 
+
+   where `<your-username>` is replaced by your GitHub username. You now have a local version of the repo! 
+
+3. Add a remote upstream version        
+    Switch into the Caiman folder `cd CaImAn` and add the canonical Flatiron Caiman repo as your upstream branch:    
+
+    `git remote add upstream https://github.com/flatironinstitute/CaImAn`. 
+
+    You now have a local version of Caiman that is connected to two remote versions at GitHub: your personal fork (named `origin`), and the original at Flatiron (named `upstream`).  You can verify this with the command `git remote -v`. This makes it easier to keep your local branches up to date with `git pull upstream`.
+
+4. Install in development mode   
+    Detailed instructions for installing in development mode (for purposes of writing code and making new features) can be found on our [installation page](./docs/source/Installation.rst). While there are some OS-specific differences, they all involve creating a conda environment from the `environment.yml` file you downloaded when you cloned the repo, activating that environment, and installing Caiman with:
+
+        pip install -e . 
+
+    This installs Caiman directly from the downloaded source code. The `-e` stands for 'editable': when you edit the files, the changes should immediately be reflected in the code you run.
+
+Note this section is partly based on the excellent [docs from Matplotlib](https://matplotlib.org/devdocs/devel/development_setup.html#installing-for-devs).
+
+
+## Second, work on a feature 
+Once you have a development environment, you can start working on the changes you envision. The main thing to remember is to follow the workflow in the diagram above. Let's say you want to work on feature `my_feature`. The first thing to do (label 1 in the diagram) is to create a new branch from the `dev` branch. Within the repo folder:
+
+    git checkout dev # checkout dev branch        
+    git checkout -b my_feature  # create new branch off of dev
+
+Then from within this new branch you can do your work on the feature: code, add, commit, etc. This corresponds to the workflow in the blue boxes in the above diagram (label 2 in the figure).
+
+If you changed anything in the `.py` modules, we request that you run tests on your code before making a PR. You can test the code by typing `caimanmanager test`. But obviously you will probably want to test to make sure you code works in different uses cases, your plots all are reasonable, and save this code and share it with us so we can test it too! This is all useful information for the developers.
+
+## Third, make a PR
+Once your feature branch is ready, it's time to make a PR (label 3 in the figure). This is fairly simple:
+
+    git push origin my_feature
+
+This pushes `my_feature` to your fork of Caiman (which, recall, is named `origin`). Then, go to your fork of Caiman at your personal GitHub repo, and a large green button will show at the top that gives you the option of making a pull request. Here is an example from a feature branch called `amazing_new_feature`: 
+
+<img src="docs/img/pull_request.jpg">
+
+When you click the `Compare & pull request` button, it will bring up an interface to integrate your code at the main Caiman repo. Be sure to generate your PR targeting the `dev` branch, not `main`, and fill out the PR template that is auto-generated by Caiman with the information requested. 
+
+Note that all PRs are reviewed by other programmers. This is an important part of the process: they will almost always give comments, making suggestions or asking questions about changes using Github's pull request interface. 
+
+You may be asked to make some changes (or to *think* about making some changes). You will sometimes need to do more some more work on your branch and make more changes after making an initial PR. In this case, the workflow is simple: you will work within your your local `my_feature` branch as before, and run the `push` command again. Conveniently, this will automatically push the changes to the same work-in-progress PR at Caiman. Eventually, the feature will be merged into `dev` and your work is done!
+
+## Fourth, wait for the work to show up in main :clock8: 
+Once your work is done, the `dev` branch will eventually be merged into `main`  by the developers who maintain Caiman (label 4 in the figure). This is done every month or two, and is the stage when your work will actually be available to the people who download Caiman. It's at this point your name will appear when you click on the [list of Contributors](https://github.com/flatironinstitute/CaImAn/graphs/contributors) at GitHub. Please give yourself a pat on the back -- we really appreciate the folks who go through all this work to help make the package better!
+
+# What next?
+Once you have gone through the above steps, you can delete your local feature branch. Before working on a new feature, you will want to make sure that your fork stays up to date with Caiman. You can do this with the user interface at GitHub (there is a button to sync up your repo with the original repository on a particular branch). 
+
+Nobody remembers all the git commands, don't worry if you constantly are looking things up: that's what *everyone* does. If you want to learn more, check out the following resources:
+
+* [Getting started with git/github](https://github.com/EricThomson/git_learn)
+* [GitHub on Contributing to a Project](https://git-scm.com/book/en/v2/GitHub-Contributing-to-a-Project)
+* [GitHub skillbuilding](https://skills.github.com/)
+* [Scipy git resources](https://docs.scipy.org/doc/scipy/dev/gitwash/gitwash.html#using-git)
+
+Again, if you want to contribute and find any of the above bits confusing, please reach out!
 
-Test your code first
-====================
-The caimandata command has code to run the internal test suite on your platform. Please do this for whatever platform(s) you have access to; our Jenkins is not public and it's easier for you to find problems this way than to ask for text logs to be sent to you. You can do this by typing "caimanmanager.py test" (if you're doing a symlink install, caimanmanager will not be in your path and you should invoke it as "python caimanmanager.py test")
 
-Platforms we care about
-=======================
-CaImAn code runs:
-* On Linux (well supported)
-* On MacOS (well supported)
-* On Windows (reasonably well supported)
 
-And on:
-* Python 3 (only current target)
 
-Your code should be portable to the platforms above, and we have test suites that provide coverage for this.
 
-Talk to us
-==========
-It's helpful, if you're starting to put together a diff, to let us know what you're working on. With this, we can avoid duplicated effort, or wasted effort from changes that we might not want to accept. We can usually setup a videoconference (Google Hangouts, Skype) if that works for you. We also like hearing from you on Gitter (link is in the main readme).

Dockerfile

@@ -4,5 +4,5 @@ RUN conda config --set always_yes yes
 RUN conda update --yes conda
 RUN apt-get update && apt-get install -y gcc g++ libgl1
 RUN conda create -n caiman -c conda-forge caiman
-RUN /bin/bash -c "source activate caiman && caimanmanager.py install"
+RUN /bin/bash -c "source activate caiman && caimanmanager install"
 

ISSUE_TEMPLATE.md

@@ -7,5 +7,4 @@ Your setup:
  4. Caiman version (e.g. 1.9.12):
  5. Which demo exhibits the problem (if applicable):
  6. How you installed Caiman (pure conda, conda + compile, colab, ..):
- 7. Caiman version: 
- 8. Details: 
+ 7. Details: 

Jenkinsfile

@@ -94,7 +94,7 @@ Check console output at $BUILD_URL to view full results.
 Building $BRANCH_NAME for $CAUSE
 $JOB_DESCRIPTION
 
-Chages:
+Changes:
 $CHANGES
 
 End of build log: