SpykingCircus2 Spike Sorting/Multiprocessing

[jklembcz]

Hi all,

I am currently attempting to run SpykingCircus2 on some .pl2 data, but I keep running into a RuntimeError:

RuntimeError:
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.

    This probably means that you are not using fork to start your
    child processes and you have forgotten to use the proper idiom
    in the main module:

        if __name__ == '__main__':
            freeze_support()
            ...

    The "freeze_support()" line can be omitted if the program
    is not going to be frozen to produce an executable.

I know this order is related to multiprocessing, but I have tried multiple methods (run_sorter_jobs, set global job kwargs) to run without necessarily using multiprocessing. Is this just an open issue or is there a way to address this? Also, let me know if I should provide code or the full error trace.

Thanks!

[zm711]

Hey @jklembcz,

This could be related to this issue #2122.

Are you on Windows? And are you attempting to run this a script (ie python my_script.py) or are you running this line-by-line in a notebook?

Maybe this is related to #2696. Could you give us a bit more of your script and what OS you're using.

[jklembcz]

Hey @zm711 ,

I am running Windows 10 (anaconda, python3.10) and in a script. My most recent attempt was using run_sorter_jobs to try to force n_jobs=1, but my first attempts were just using two run_sorter calls with the same parameters.

job_list = [
{"sorter_name":"spykingcircus2", "recording":kitkatrec, "output_folder":output_folder+"\spykingcircus\kitkat", "docker_image":False, "remove_existing_folder":True},
{"sorter_name":"spykingcircus2", "recording":pen21brec, "output_folder":output_folder+"\spykingcircus\ethel", "docker_image":False, "remove_existing_folder":True}
]
ss.run_sorter_jobs(job_list=job_list,engine="loop",engine_kwargs=dict(n_jobs=1))

I'll refactor my code to reflect #2122 and see if that fixes things for now.

[zm711]

Cool let us know. We are still trying to improve multiprocessing on Windows. I'm the only currently with a Windows workstation for my lab work, so the more we can pinpoint what exactly is the causing the problem we can either write a troubleshooting guide or fix it.

[samuelgarcia]

Are you trying to use ss.run_sorter_jobs() with "joblib" or "multiprocessing" ?
Because in that case we have a nested forking that make really weird result.

the main use of run_sorter_jobs() should "loop" or "slurm" unless you run a sorter that do not use internally multiprocessing of that is matlab based for instance.

[jklembcz]

I was not trying to use "joblib" or "multiprocessing" for my job engine, I was only using "loop" (and even run_sorter_jobs) as an alternative to two run_sorter calls.

I now am using two run_sorter calls, including the job_kwargs of n_jobs=1 , but ran into the exact same issue as the initial comment in #2474. In the same way as #2474, removing the total_memory argument allowed me to sort successfully.

I will add, I was able to successfully sort using Kilosort2 locally in the same script, as long as any I commented out my run_sorter calls to run spykingcircus2. Let me know if you need any more information, and thank you!