Problems during training.

[XuyangBai]

Hi, Thanks for your sharing. I have tried your code on my own dataset but the I found that initially everything goes well but after several epochs the training suddenly broke up ( accuracy becomes 1 and the loss becomes 0 ) I use tf 1.12.0 and the cuda version is 9.0, cudnn version is 7.1.4

# conda list | grep tensorflow
tensorflow-estimator      1.13.0                     py_0    anaconda
tensorflow-gpu            1.12.0                   pypi_0    pypi
tensorflow-tensorboard    0.4.0                    pypi_0    pypi

Have you met this kind of problem? Another potential problem is that sometimes the training takes 4400 MB GPU memory (see from nvidia-smi), but sometimes it takes more than 7000 MB ( and I do not change the batch size and network architecture) I am pretty confused about these problems. Could you give me some advice?

[HuguesTHOMAS]

Hi @XuyangBai,

The first error is quite strange and I never encountered such behavior on my datasets. It is very unlikely that the loss really became zero if you use correct augmentation strategies. It seems more like a bug, but it will be difficult to help you without reproducing your experiments with your dataset.

The second one could be explained by your dataset. The GPU memory that you see from nvidia-smi is the memory taken by the tensors at runtime. It thus depends on the input size. If you use the same network parameters, but with a different dataset, with denser point cloud for example, this memory will be larger. It is strange that you get different values GPU memory for the same dataset, but might be explained by the nature of your dataset and your implementation of the dataset class.

What does your data look like? Real or artificial point clouds? Indoor outdour scenes? Objects?

[XuyangBai]

Hi @HuguesTHOMAS ,

I also think the first error may because some bugs of my implementation. I just want to check whether it is because the tensorflow version. BTW, what's the situation when using TF 1.13 and CUDA 10 ?

For the second one, I mean that for some experiments, the GPU memory is always 4400MB and for others is always 7000+MB. It is really strange. But I check the training.txt in result fold and found the memory showed in that file is similar.

[XuyangBai]

Oh Sorry I think I find the reason of second problem. I forget the dropout. It seems when I use dropout = 0.5, the GPU memory is around 4400 MB while using dropout = 1, the GPU memory is 7000MB.

Sorry for the bothering.

[HuguesTHOMAS]

Hi @XuyangBai,

If you look at the code, the dropout variable is extremely important in the implementation, because the network use it to know if you use it for training or for test.

If you use a dropout < 0.99, the network is in training configuration, and if you use a dropout = 1, the network is in test configuration. This is a trick that I used to avoid creating a 'training/test' boolean placeholder, and that I never corrected.

It will be corrected in the next month (I currently don't have any time to spend on the code). Until then, you should not use dropout = 1 when training, as the variables will not be updated by gradient back propagation in that case. If you have dropout blocks and don't want to use them, just remove them or use dropout = 0.98 and they will be insignificant.

Best,
Hugues

[XuyangBai]

Thanks a lot for your reply :)

[XuyangBai]

Hi @HuguesTHOMAS

I reopen this issue because I go through the code to check out the bug but I am still facing the first problem that training will broke up (loss become zero). Actually, everything goes well for environment CUDA 9, TF 1.12.0. But when I tried to run the code on my RTX 2080Ti with CUDA 10, TF 1.12.0 (build from source), the problem will appear. And I printed the input value and variables for when the model broken and found the input value is OK but all the variables all NaN.

I have tried to add 1e-12 to operation like tf.sqrt to avoid infinite number but the problem is still there.

Another thing need to mention is that sometimes the training and validation both 'broke up' while sometimes the training seems correct but the validation is broken up, like the curve below.

Have you ever met such problems like the variables become all NaN? Thank you so much.

[nejcd]

Hi @XuyangBai, I have noticed similar behaviour as you have described. I have not been able to debug it as it occurs randomly.
As you have closed the issue, have you found some solution?

[XuyangBai]

Hi @nejcd , I didn't find the solution, so I just change my environment to CUDA9 TF 1.12.0 and everything goes well. There might be some bugs in CUDA10.

[HuguesTHOMAS]

Hi,

I had some time to dig into this problem and it seems that CUDA10 is not working correctly with RTX 2080Ti GPUs. Here is what I found:

Tested configurations

• CUDA9-TF1.12 / GTX 1080ti => No bug
• CUDA10-TF1.13 / GTX 1080ti => No bug
• CUDA9-TF1.12 / RTX 2080ti => No bug
• CUDA10-TF1.13 / RTX 2080ti => Bug appears only in this configuration

Origin of the bug
I tracked down the NaN values in my code and found that they appear after a tf.matmul operation:

KPConv/kernels/convolution_ops.py

Line 240 in 5f9ceca

weighted_features = tf.matmul(all_weights, neighborhood_features)

Before the appearance of NaN I noticed some weird value higher than 1e10. If you print the two matrix that are multiplied and the result matrix, you will see that the result is completly false. This seems to be caused by a CUDA internal bug. At some point one of this mistake lead to a value so high that it becomes NaN and the network crashes.

For now I would just advise to avoid using CUDA10 with a RTX 2080ti.