training process came out loss = nan

[danyow-cheung]

Hey Groups , thanks for the great work . I were using the template demo to reproduce the result , however during the traing process,the Loss=nan showed up , the program was still going on and on ,

the command was like

python train.py --opt configs/bagel.yaml 
python train.py --opt configs/cat_armor.yaml  

[ITER 1] Video Save Done! [03/06 17:25:29]
Training progress:  10%|███████████▍                                                                                                         | 490/5000 [03:31<30:44,  2.45it/s, Loss=nan]scale up theta_range to: [45, 105] [03/06 17:29:00]
scale up radius_range to: [4.9399999999999995, 5.225] [03/06 17:29:00]
scale up phi_range to: [-180, 180] [03/06 17:29:00]
scale up fovy_range to: [0.24, 0.6] [03/06 17:29:00]
Training progress:  20%|███████████████████████▏                                                                                             | 990/5000 [06:57<26:05,  2.56it/s, Loss=nan]scale up theta_range to: [45, 105] [03/06 17:32:26]
scale up radius_range to: [4.693, 5.0] [03/06 17:32:26]
scale up phi_range to: [-180, 180] [03/06 17:32:26]
scale up fovy_range to: [0.18, 0.6] [03/06 17:32:26]
Training progress:  20%|███████████████████████▏                                                                                            | 1000/5000 [07:01<26:54,  2.48it/s, Loss=nan

[danyow-cheung]

Also I tried not only sd-2-1 but also sd1.5 same issues