Support 5.2 bloom #7846
Merged
Support 5.2 bloom #7846
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Thanks for your contribution! |
Codecov ReportAttention:
Additional details and impacted files@@ Coverage Diff @@
## develop #7846 +/- ##
===========================================
- Coverage 56.95% 56.90% -0.05%
===========================================
Files 587 587
Lines 88628 88724 +96
===========================================
+ Hits 50480 50492 +12
- Misses 38148 38232 +84 ☔ View full report in Codecov by Sentry. |
zhoutianzi666
force-pushed
the
support_bloom
branch
from
99d897b
to
b60ea16
Compare
wj-Mcat
requested changes
Jan 16, 2024
llm/predictor.py
Outdated
| ) | ||
| if predictor_args.block_attn: | ||
| from paddlenlp.experimental.transformers import ( | ||
| BlommForCausalBlockLMInferenceModel as Model, |
There was a problem hiding this comment.
你这边就仿照 llama block_attn 的写法呗，就直接 as BloomInferenceModel ？
There was a problem hiding this comment.
你这边就仿照 llama block_attn 的写法呗，就直接
as BloomInferenceModel？
已经done，感谢review
llm/predictor.py
Outdated
| config.max_seq_len = predictor_args.total_max_length | ||
| else: | ||
| from paddlenlp.experimental.transformers import ( | ||
| BloomForCausalLMInferenceModel as Model, |
There was a problem hiding this comment.
同上。
已经done，感谢review
单测已经加上了。这个命令我觉得可以不放进去，因为和里面已有的命令是类似的。感谢review |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
PR types
New features：支持bloom类模型的5.2 推理
PR changes
Description
python3.8 predictor.py --model_name_or_path /root/.paddlenlp/models/bigscience/bloom-7b1/ --dtype float16 --src_length 102 --max_length 1024 --block_attn --batch_size 2 --inference_model > dynamic_2.txt
python3.8 predictor.py --model_name_or_path /root/.paddlenlp/models/bigscience/bloom-7b1/ --dtype float16 --src_length 102 --max_length 1024 --block_attn --batch_size 2 --inference_model --quant_type weight_only_int8
python3.8 export_model.py --model_name_or_path /root/.paddlenlp/models/bigscience/bloom-7b1/ --inference_model --output_path ./inference --dtype float16 --block_attn --quant_type weight_only_int8
python3.8 predictor.py --model_name_or_path ./inference --inference_model --dtype "float16" --mode "static" --batch_size 2 --block_attn
python3.8 export_model.py --model_name_or_path /root/.paddlenlp/models/bigscience/bloom-7b1/ --inference_model --output_path ./inference_wint8 --dtype float16 --block_attn --quant_type weight_only_int8
python3.8 predictor.py --model_name_or_path ./inference_wint8 --inference_model --dtype "float16" --batch_size 2 --mode "static" --quant_type weight_only_int8 --block_attn