Fix ModulesToSaveWrapper __getattr__

[zhangsheng377]

#1225

We are recently using peft=0.4.0+Megatron to run GPTModel. There is a code of the model self.language_model.output_layer.weight, which directly takes the weight of the output_layer, but the output_layer has already been modules_to_save, so it reported "ModulesToSaveWrapper has no 'weight' attribute" error.

So, I returns the weight of the currently active adapter or of the original module of the adapter is deactivated.

And __getattr__ will be called only when its own class does not have this attribute, so this should not destroy the original scene.

[BenjaminBossan]

Thanks for the PR. Thinking a bit more about it, I think the best course of action would be to actually implement this as a @property, which allows us to avoid fiddling with __getattr__.

This should still not affect stuff like state_dict, so would not break anything.

On top of that, would you be up to implement a unit test?

[zhangsheng377]

Thanks for the PR. Thinking a bit more about it, I think the best course of action would be to actually implement this as a @property, which allows us to avoid fiddling with __getattr__.

This should still not affect stuff like state_dict, so would not break anything.

On top of that, would you be up to implement a unit test?

ok~

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[BenjaminBossan]

Thanks, LGTM.

Can one of you @pacman100 or @younesbelkada give this a quick look to ensure it doesn't break anything?

[pacman100]

Thank you @zhangsheng377 for adding the weight property when using ModulesToSaveWrapper! This makes sense and I don't see any issues with this approach.

[younesbelkada]

Thanks! let's run integration tests after merging to see if this didn't break anything

[younesbelkada]

Tests are all green! https://github.com/huggingface/peft/actions/runs/7194802375/job/19596105974 🍏