Job stuck in CTF estimation step #19
Description
When we run a 3D refine job on THUNDER, it stucked here:
26/08/2019 14:46:32.044 INFO [LOGGER_MEM] A_LEAD: Before Initialsing CTFs, Physic Memory Usage : 20G
26/08/2019 14:46:32.044 INFO [LOGGER_INIT] A_LEAD: Generating CTFs
26/08/2019 14:46:32.372 INFO [trivial] Step1: Prepare Parameter for CTF calculation
26/08/2019 14:46:32.373 INFO [trivial] Allocate Memory.
26/08/2019 14:46:32.373 INFO [trivial] Allocate memory done, begin to calculate
And then it won't continue. It stuck here for almost one day.
I think it shouldn't because of memory used up because we have 256GB. And the memory has already allocated. But the CTF calculation won't stop (or start? I didn't see calculation resource occupied).
In this workstation, we have THUNDER on:
system=CentOS-7.6
CUDA=9.2
cmake=3.13.5
mpirun=1.10.7
nccl=2.4.2-1+CUDA9.2
gcc=4.8.5
Thanks!