[QEff. Finetune]: Removed samsum dataset references from FT code. #482
Conversation
quic-meetkuma
commented
Jun 27, 2025
quic-meetkuma
commented
- Removed all the references of samsum dataset from finetuning code.
- Samsum dataset can be used via custom dataset path.
quic-meetkuma
changed the title
quic-meetkuma
requested review from
quic-rishinr,
ochougul,
quic-hemagnih and
quic-amitraj
as code owners
|
|
||
| DATASET_PREPROC = { | ||
| "alpaca_dataset": partial(get_alpaca_dataset), | ||
| "grammar_dataset": get_grammar_dataset, | ||
| "samsum_dataset": get_samsum_dataset, |
There was a problem hiding this comment.
Just removing this line of code is enough for the finetune.py to not throw DatasetNotFoundError error in case of "--dataset samsum_dataset". It will raise an error as follows: 'finetune.py: error: argument --dataset: invalid choice: 'samsum_dataset' (choose from 'alpaca_dataset', 'grammar_dataset', 'gsm8k_dataset', 'custom_dataset', 'imdb_dataset')'
Rest of the code changes of this PR are not required.
This way we can still keep the code for samsum_dataset for internal testing purpose and also if huggingface puts back the Samsum dataset, we would just need a single line of code to support it through QEfficient.
There was a problem hiding this comment.
It is discussed with Anuj and VB to remove all the references of this code. User should use this only via custom_dataset path.
quic-meetkuma
force-pushed
the
samsum_removal
branch
from
5deac57 to
6c98570
Compare
|
|
||
| DATASET_PREPROC = { | ||
| "alpaca_dataset": partial(get_alpaca_dataset), | ||
| "grammar_dataset": get_grammar_dataset, | ||
| "samsum_dataset": get_samsum_dataset, |
There was a problem hiding this comment.
It is discussed with Anuj and VB to remove all the references of this code. User should use this only via custom_dataset path.
| @@ -171,6 +170,28 @@ pipeline { | |||
| } | |||
| } | |||
| } | |||
|
|
|||
There was a problem hiding this comment.
@quic-hemagnih , @vbaddi , @quic-rishinr - FYI, Made a separate env for FT tests.
quic-meetkuma
force-pushed
the
samsum_removal
branch
from
5015cba to
8fc47cd
Compare
quic-meetkuma
force-pushed
the
samsum_removal
branch
2 times, most recently
from
4955cf2 to
3f3728a
Compare
Signed-off-by: Meet Patel <meetkuma@qti.qualcomm.com>
Signed-off-by: Meet Patel <meetkuma@qti.qualcomm.com>
Signed-off-by: Meet Patel <meetkuma@qti.qualcomm.com>
quic-meetkuma
force-pushed
the
samsum_removal
branch
from
3f3728a to
88c3cd6
Compare
