Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Problem wher run learning on 32 batch-size #2058

Open
Artem-N opened this issue Jul 3, 2024 · 0 comments
Open

Problem wher run learning on 32 batch-size #2058

Artem-N opened this issue Jul 3, 2024 · 0 comments

Comments

@Artem-N
Copy link

Artem-N commented Jul 3, 2024

Hello, I have the following problem - when training the yolov7 model (a dataset of 10k images with 90k instances) - training on 16 batch sizes with 8 dataloaders goes well and one epoch is about 1-2 minutes, when I try to increase the batch size to 32 with 8 or 6 dataloaders - two epochs of 2-3 minutes each pass, and on the 3rd epoch learning simply gets stuck, and one epoch lasts about 20 minutes, and then it can expand and the epoch will take 2-3 waves, and it can continue to freeze and learn for 20 minutes each epoch.

when I teach Yolo 8 or Yolo 9 for 32 epochs, there is no such problem.

hardware - intel i9, cpu 64 gb, gpu - nvidia gforce 4090rtx 24gb

i don`t know what happend

and when i run same on 30 batch - all goes well

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant