Optimal job kwargs for write_binary_recording

[Antho2422]

Hi all,

I have a question concerning the job kwargs especially for the function

recording.save()

I am struggling to choose the optimal kwargs. I have the feeling that the process is longer when n_jobs > 1 and it should not be...

Plus, when I'm running the code on my local computer, the save of the recording is not that long (3min for a 20min recording) with n_jobs = 1.
However, when I'm running the same code, on the same data on a cluster with n_jobs = 28 the process takes ages (only 2/3 iterations per seconds). I verified that I have the 28 cpus available. I tried to change the chunk_duration parameter from 1s to 10s and it often causes me the following error :

  File "/network/lustre/iss01/apps/lang/anaconda/3/2021.11-2/lib/python3.9/concurrent/futures/process.py", line 681, in submit
    raise BrokenProcessPool(self._broken)
concurrent.futures.process.BrokenProcessPool: A child process terminated abruptly, the process pool is not usable anymore

I may be missing something on the use of this function and choose of the optimal job_kwargs. If somebody could help me it would be really appreciated 👍

Thank you,

Anthony

[Antho2422]

Actually, I tried run the recording.save() function with binary as format with n_jobs = 1. Unfortunately the process is consistently failing. I am not sure the error is coming from the multiprocessing.

To add more context, when I'm trying to save a binary recording on my computer, running the script in local the save() works fine. However, when I'm running the script on a cluster and saving the data on an lustre server I have the issue that the binary recording save fails. I'm a bit confused because a few weeks ago this process was working fine.

May be this is due to a recent update ? I'm using version 0.99.1

Thank you

[alejoe91]

@Antho2422 we didn't change the parallel writing part. Is it possible that something on your server/storage side changed?

[Antho2422]

@alejoe91 Possibly, I asked to the IT but I was wondering if maybe the issue was coming from a possible update.

The only thing the IT said to me was that our cluster is really bad at accessing to small files. I decided to increase the chunk size thinking that could help but unfortunately not.

[h-mayorquin]

Actually, I tried run the recording.save() function with binary as format with n_jobs = 1. Unfortunately the process is consistently failing. I am not sure the error is coming from the multiprocessing.

What error are you getting with one job?

[Antho2422]

I'm not able to reproduce the error anymore. This issue may have been coming from our side, everything works fine now.

Thank you,
Anthony

[h-mayorquin]

I am happy to hear that, please report other issues you might have. We are always trying to improve the library.