-
Notifications
You must be signed in to change notification settings - Fork 11k
Handle HIP OOM exceptions #7764
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Conversation
|
This also means I can use Perhaps WanImageToVideo should have a tiled option somehow too. |
398805d to
a19cb1a
Compare
|
I've update to pytorch 2.7.1+rocm6.3 and this still seems valuable for my gfx1100. It makes sense that all variants of OOM should be handled. Is there some issue with this approach, or other reason not to merge this? |
|
This is still merge conflict free, want it or shall I close? cc @comfyanonymous @Kosinkadink |
|
Could i get a comment on this, perhaps a short description on why it isn't suitable? Perhaps i could rework it adapt it in some way that still allows oom fallbacks to work on rocm. @comfyanonymous @Kosinkadink |
|
Why don't you set the model_management.OOM_EXCEPTION to whatever the hip oom exception is? |
The error this handles is a generic RuntimeError with just the message identifying it as an oom. Matching all RuntimeErrors could include non-oom ones.
|
Enhance exception handling to treat RuntimeException "HIP error: out of memory" the same as
OOM_EXCEPTION.Resolves #7761