Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Model Pruning Callback Failing #10835

Open
aqibsaeed opened this issue Nov 30, 2021 · 6 comments
Open

Model Pruning Callback Failing #10835

aqibsaeed opened this issue Nov 30, 2021 · 6 comments
Assignees
Labels
bug Something isn't working callback: pruning help wanted Open to be worked on priority: 1 Medium priority task repro needed The issue is missing a reproducible example
Milestone

Comments

@aqibsaeed
Copy link

aqibsaeed commented Nov 30, 2021

Hi,

I am trying to use ModelPruning callback as follows:

    callbacks=[
        ModelPruning(
            pruning_fn="l1_unstructured",
            amount=0.01,
            use_global_unstructured=True,
        )
    ]

but after training for an epoch, the Trainer throws following error, (only happens when using ModelPruning callback):

File "/home/.conda/envs/dummy/lib/python3.8/site-packages/torch/nn/utils/convert_parameters.py", line 77, in _check_param_device
    if param.is_cuda:  # Check if in same GPU
AttributeError: 'bool' object has no attribute 'is_cuda'

I tried pytorch 1.7.0 and 1.9.0 but the issue persist. Any idea what is causing this error?

Thanks.

cc @tchaton @rohitgr7 @carmocca

@aqibsaeed aqibsaeed added the bug Something isn't working label Nov 30, 2021
@stale
Copy link

stale bot commented Jan 3, 2022

This issue has been automatically marked as stale because it hasn't had any recent activity. This issue will be closed in 7 days if no further activity occurs. Thank you for your contributions, Pytorch Lightning Team!

@stale stale bot added the won't fix This will not be worked on label Jan 3, 2022
@kaushikb11 kaushikb11 added priority: 1 Medium priority task and removed won't fix This will not be worked on labels Jan 4, 2022
@tchaton
Copy link
Contributor

tchaton commented Jan 4, 2022

Dear @aqibsaeed,

Would you mind trying the latest version of PyTorch Lightning?

Best,
T.C

@Jconn
Copy link

Jconn commented Mar 28, 2022

I am also getting this error. I've tried 1.5.10 and 1.6.0.rc1 and a few other 1.5.*, the 1.5.* versions all produce this error and the 1.6.0 rc gives me an OOM fault.

edit:
Okay, I understand this error. When not provided parameters to prune, this library tries to add every parameter in the model with name weight or bias. The check for if a layer has a weight or bias tensor is checking to see if that variable exists within the python object. Some pytorch modules have a bias variable that is not a tensor, but rather a bool to indicate whether bias tensors are to be used. pytorch-lighnting tells pytorch to prune a bool, which causes the error.

@aminst
Copy link
Contributor

aminst commented Jul 25, 2022

Hi,
I also encountered this issue while using the ModelPruning callback. As @Jconn suggested, I also think this is because of the addition of non-tensor parameters in the pruning stage.
The code provided below from the pruning.py file adds all the parameters from all types:
https://github.com/Lightning-AI/lightning/blob/8d14554383632f4cdff337b3d8d1b226cabdd1d0/src/pytorch_lightning/callbacks/pruning.py#L450-L453

I think there must be a check of being a Tensor parameter using the isinstance method, so that these non-tensor parameters are not pruned.
If my approach looks sound, I will be glad to make a PR and solve this issue.

@carmocca
Copy link
Contributor

Sounds good to me

@carmocca carmocca added this to the pl:1.6.x milestone Jul 26, 2022
@aminst
Copy link
Contributor

aminst commented Jul 26, 2022

Sounds good to me

Thanks, I will work on it 👍

@carmocca carmocca modified the milestones: pl:1.6.x, pl:1.7.x Jul 28, 2022
@carmocca carmocca modified the milestones: pl:1.7.x, v1.8.x Oct 13, 2022
@Borda Borda modified the milestones: v1.8.x, v1.9 Jan 6, 2023
@Borda Borda modified the milestones: v1.9, v1.9.x Jan 16, 2023
@awaelchli awaelchli added help wanted Open to be worked on repro needed The issue is missing a reproducible example labels Dec 31, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working callback: pruning help wanted Open to be worked on priority: 1 Medium priority task repro needed The issue is missing a reproducible example
Projects
None yet
Development

Successfully merging a pull request may close this issue.

9 participants