Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow for nightly packages of compressed_tensors #33828

Merged

Conversation

kylesayrs
Copy link
Contributor

@kylesayrs kylesayrs commented Sep 30, 2024

What does this PR do?

This PR follows up on #31704 by allowing the compressed-tensors-nightly package to be used to load compressed tensors modules. This is done by relaxing installation requirements to only require the compressed-tensors module spec, but not the compressed-tensors package.

Compressed Tensors models cannot be loaded with nightly package

from transformers import AutoModel
model = AutoModel.from_pretrained("nm-testing/TinyLlama-1.1B-Chat-v1.0-W8A8_tensor_weight_static_per_tensor_act-e2e")
# ImportError: Using `compressed_tensors` quantized models requires the compressed-tensors library: `pip install compressed-tensors`

Compressed Tensors is reported as unavailable, despite the module being available through the nightly package

from transformers.utils.import_utils import is_compressed_tensors_available
is_compressed_tensors_available()
# False

After the changes, models can be imported with the nightly package and compressed tensors is reported as being available

Who can review?

@SunMarc @ArthurZucker
@dsikka @dhuangnm @andy-neuma
Anyone in the community is free to review the PR once the tests have passed.

@kylesayrs kylesayrs changed the title Allow for compressed_tensors-nightly Allow for nightly packages of compressed_tensors Sep 30, 2024
Copy link

@markurtz markurtz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These changes look correct to me. It brings up a separate consideration that we'll need at some point in the future, which is to gather the compressed tensors version so we can compare if there are breaking changes between the config format that is stored with the model and the compressed tensors that is loaded, especially with it loading nightly dev builds

Copy link
Member

@SunMarc SunMarc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM !

@LysandreJik LysandreJik merged commit b09234c into huggingface:main Oct 2, 2024
21 checks passed
@kylesayrs kylesayrs deleted the kylesayrs/support-nightly-packages branch October 2, 2024 15:01
NielsRogge pushed a commit to NielsRogge/transformers that referenced this pull request Oct 21, 2024
* only check spec

* correct typo in nightly package name
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants