Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RuntimeError: CUDA out of memory. Tried to allocate 310.00 MiB (GPU 0; 7.80 GiB total capacity; 4.05 GiB already allocated; 102.44 MiB free; 404.92 MiB cached) #88

Open
sherlockzym opened this issue Feb 26, 2021 · 0 comments

Comments

@sherlockzym
Copy link

(ssd) root@logic:~/SA-SSD/tools# python train.py ../configs/car_cfg.py
2021-02-26 09:22:00,718 - INFO - Distributed training: False
2021-02-26 09:22:00,718 - INFO - Set random seed to 0
[40, 1600, 1408]
load 14357 Car database infos
After filter database:
load 10520 Car database infos
2021-02-26 09:22:05,502 - INFO - Start training
Traceback (most recent call last):
File "train.py", line 127, in
main()
File "train.py", line 117, in main
log_interval = cfg.log_config.interval
File "/root/SA-SSD/tools/train_utils/init.py", line 99, in train_model
log_interval = log_interval
File "/root/SA-SSD/tools/train_utils/init.py", line 57, in train_one_epoch
outputs = batch_processor(model, data_batch)
File "/root/SA-SSD/tools/train_utils/init.py", line 29, in batch_processor
losses = model(**data)
File "/root/anaconda3/envs/ssd/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in call
result = self.forward(*input, **kwargs)
File "/root/anaconda3/envs/ssd/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 150, in forward
return self.module(*inputs[0], **kwargs[0])
File "/root/anaconda3/envs/ssd/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in call
result = self.forward(*input, **kwargs)
File "/root/SA-SSD/mmdet/models/detectors/base.py", line 79, in forward
return self.forward_train(img, img_meta, **kwargs)
File "/root/SA-SSD/mmdet/models/detectors/single_stage.py", line 103, in forward_train
bbox_score = self.extra_head(conv6, guided_anchors)
File "/root/anaconda3/envs/ssd/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in call
result = self.forward(*input, **kwargs)
File "/root/SA-SSD/mmdet/models/single_stage_heads/ssd_rotate_head.py", line 435, in forward
x = self.convs(x)
File "/root/anaconda3/envs/ssd/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in call
result = self.forward(*input, **kwargs)
File "/root/anaconda3/envs/ssd/lib/python3.6/site-packages/torch/nn/modules/container.py", line 92, in forward
input = module(input)
File "/root/anaconda3/envs/ssd/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in call
result = self.forward(*input, **kwargs)
File "/root/anaconda3/envs/ssd/lib/python3.6/site-packages/torch/nn/modules/conv.py", line 338, in forward
self.padding, self.dilation, self.groups)
RuntimeError: CUDA out of memory. Tried to allocate 310.00 MiB (GPU 0; 7.80 GiB total capacity; 4.05 GiB already allocated; 102.44 MiB free; 404.92 MiB cached)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant