为什么在训练的第二个epoch时显示cuda out of memory

[lemonade-lm]

No description provided.

[mooncake199809]

我在3DMatch数据集上训练时，训练到第二个epoch时现存不够，我并没有修改代码，并且现存是48G的，为什么会出现这种情况

Do you resolve this problem?

[jlyw1017]

same problem here.

[mooncake199809]

same problem here.

I solve this problem by changing the number of points from 30000 to 29000 for 3DMatch.

[Baros-Young]

same problem here.

I solve this problem by changing the number of points from 30000 to 29000 for 3DMatch.

Hello,how long did it take you to train the model?It took me 5.7 hours to train the model for 3 epochs on a 3090 GPU using preset parameters, which is really beyond my acceptable range...

[mooncake199809]

same problem here.

I solve this problem by changing the number of points from 30000 to 29000 for 3DMatch.

Hello,how long did it take you to train the model?It took me 5.7 hours to train the model for 3 epochs on a 3090 GPU using preset parameters, which is really beyond my acceptable range...

It takes me more than three days using a 3090GPUS

[Baros-Young]

same problem here.

I solve this problem by changing the number of points from 30000 to 29000 for 3DMatch.

Hello,how long did it take you to train the model?It took me 5.7 hours to train the model for 3 epochs on a 3090 GPU using preset parameters, which is really beyond my acceptable range...

It takes me more than three days using a 3090GPUS

Thanks for your reply,it helps!

[Baros-Young]

same problem here.

I solve this problem by changing the number of points from 30000 to 29000 for 3DMatch.

Hello,how long did it take you to train the model?It took me 5.7 hours to train the model for 3 epochs on a 3090 GPU using preset parameters, which is really beyond my acceptable range...

It takes me more than three days using a 3090GPUS

I get wrong rre and rte result using pretrained weight,so what could be the reason?

[mooncake199809]

same problem here.

I solve this problem by changing the number of points from 30000 to 29000 for 3DMatch.

Hello,how long did it take you to train the model?It took me 5.7 hours to train the model for 3 epochs on a 3090 GPU using preset parameters, which is really beyond my acceptable range...

It takes me more than three days using a 3090GPUS

I get wrong rre and rte result using pretrained weight,so what could be the reason?

You should run both of these two codes:
CUDA_VISIBLE_DEVICES=0 python test.py --snapshot=../../weights/geotransformer-3dmatch.pth.tar --benchmark=3DMatch
CUDA_VISIBLE_DEVICES=0 python eval.py --benchmark=3DMatch --method=lgr
It seems that you only run the first code

[mooncake199809]

same problem here.

I solve this problem by changing the number of points from 30000 to 29000 for 3DMatch.

Hello,how long did it take you to train the model?It took me 5.7 hours to train the model for 3 epochs on a 3090 GPU using preset parameters, which is really beyond my acceptable range...

It takes me more than three days using a 3090GPUS

I get wrong rre and rte result using pretrained weight,so what could be the reason?

Hellow, could you please tell me the testing result in the 3DMatch dataset? I found that the testing results is different from results reported in the paper in terms of IR metric.

[wuzelei123]

你是在linux上跑的吗

[mooncake199809]

你是在linux上跑的吗

yes