fix: Addition of default pad token in tokenizer when EOS and PAD toke…

…n are equal (#343)

* fix: addition of equal check of eos and pad token

Signed-off-by: Abhishek <maurya.abhishek@ibm.com>

* fix: add condition when eos token is <pad
>

Signed-off-by: Abhishek <maurya.abhishek@ibm.com>

---------

Signed-off-by: Abhishek <maurya.abhishek@ibm.com>

tuning/sft_trainer.py

@@ -262,6 +262,14 @@ def train(
         if tokenizer.unk_token is None:
             logger.warning("UNK token set to default, missing in tokenizer")
             special_tokens_dict["unk_token"] = configs.DEFAULT_UNK_TOKEN
+        if tokenizer.pad_token == tokenizer.eos_token:
+            logger.warning(
+                "PAD token set to default, to make it different from eos token"
+            )
+            if tokenizer.eos_token != configs.DEFAULT_PAD_TOKEN:
+                tokenizer.pad_token = configs.DEFAULT_PAD_TOKEN
+            else:
+                tokenizer.eos_token = configs.DEFAULT_EOS_TOKEN
 
     # TODO: lower priority but understand if resizing impacts inference quality and why its needed.
     # It makes sense if we manipulate tokenizer that we also save it and provide it to inference.