Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Out of memory issue while running ''def _test_single_epoch(self):'' function #3

Open
daviddaiiiii opened this issue Jun 30, 2023 · 0 comments

Comments

@daviddaiiiii
Copy link

Hi there, I'm currently trying to reproduce the training and evaluation process but have run into an issue with the "def _test_single_epoch(self):" function causing an OOM error.

I have this issue running on a 24G RTX4090 GPU, it seems that this is not due to hardware limitations.

Do you have any suggestions on how to modify the function and release some memory during the evaluation?

########################################################
Trying to train from scratch
image

########################################################
modify
image
get
image
confirm the issue is due to ''def _test_single_epoch(self):'' function
########################################################
solely evaluating with pre-trained weight
image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant