Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Adds support to use brevitas quantized weights for stateless_llama #179

Merged
merged 2 commits into from
Nov 17, 2023
Merged

Adds support to use brevitas quantized weights for stateless_llama #179

merged 2 commits into from
Nov 17, 2023

Conversation

IanNod
Copy link
Contributor

@IanNod IanNod commented Nov 17, 2023

  • Modifies mm_group_quant to work with brevitas safetensors, needs work to generalize
  • Changes compiler to use torch as input to enable quantization of torch ir

@IanNod
Adds support to use brevitas quantized weights for stateless_llama
3460784 
- Modifies mm_group_quant to work with brevitas safetensors, needs work to generalize
- Changes compiler to use torch as input to enable quantization of torch ir
qedawkins
qedawkins reviewed Nov 17, 2023
View reviewed changes
Copy link
Contributor

@qedawkins qedawkins left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not exactly familiar with everything going on here, but left some thoughts/questions.

python/shark_turbine/transforms/quantization/mm_group_quant.py Show resolved Hide resolved
python/turbine_models/custom_models/stateless_llama.py Outdated Show resolved Hide resolved
python/shark_turbine/transforms/quantization/mm_group_quant.py Show resolved Hide resolved
@IanNod
Minor changes addressing comments
d98e705 
 - Adds todo clarifying skipping of _params. in mm_group_quant for matching purposes
 - removes arg use in pipeline to make external use easier
@IanNod IanNod linked an issue Nov 17, 2023 that may be closed by this pull request
Externalized Parameters #153
Closed
qedawkins
qedawkins approved these changes Nov 17, 2023
View reviewed changes
Copy link
Contributor

@qedawkins qedawkins left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, but I have not been working on this code.

stellaraccident
stellaraccident approved these changes Nov 17, 2023
View reviewed changes
Copy link
Contributor

@stellaraccident stellaraccident left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you run black and make sure everything is formatted? A lot of whitespace changes.

@IanNod
Copy link
Contributor Author

IanNod commented Nov 17, 2023

Can you run black and make sure everything is formatted? A lot of whitespace changes.

Yup, already ran black at least the files changed in this patch

@IanNod IanNod merged commit 39a37e3 into main Nov 17, 2023
1 check passed
@IanNod IanNod deleted the quantized_mm branch November 17, 2023 18:26
@dan-garvey
Copy link
Member

@stellaraccident are you running default black or with line-length=79? I think @IanNod and I's brains are hardcoded to the latter from SHARK formatting

@stellaraccident
Copy link
Contributor

Default black. We need to add a lint check and should do were iree is doing.

@dan-garvey
Copy link
Member

#189

@stellaraccident stellaraccident mentioned this pull request Nov 21, 2023
Open
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@stellaraccident stellaraccident stellaraccident approved these changes

@qedawkins qedawkins qedawkins approved these changes

@dan-garvey dan-garvey Awaiting requested review from dan-garvey

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Externalized Parameters
4 participants
@IanNod @dan-garvey @stellaraccident @qedawkins