chat cli #1431

lvwerra · 2024-03-14T16:23:47Z

This is a first draft of a chat CLI

Try with:

python examples/scripts/chat.py  --model Qwen/Qwen1.5-0.5B-Chat --device mps

cc @younesbelkada

HuggingFaceDocBuilderDev · 2024-03-14T16:28:06Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

younesbelkada · 2024-03-15T09:40:04Z

examples/scripts/chat.py

+        default=None,
+        metadata={
+            "help": (
+                "Override the default `torch.dtype` and load the model under this dtype. If `auto` is passed, the "


Suggested change

"Override the default `torch.dtype` and load the model under this dtype. If `auto` is passed, the "

"Override the default `torch_dtype` and load the model under this dtype. If `auto` is passed, the "

younesbelkada

Looks very nice ! I did a first pass looks all clean on my end and I can adapt it on my CLI PR to extend it for chat!

younesbelkada · 2024-03-15T09:40:14Z

examples/scripts/chat.py

+        # TODO: attribute fastchat
+        # TODO(suquark): the console flickers when there is a code block
+        #  above it. We need to cut off "live" when a code block is done.
+


Suggested change

# TODO: attribute fastchat

# TODO(suquark): the console flickers when there is a code block

# above it. We need to cut off "live" when a code block is done.

younesbelkada

Thanks a lot! I left small comments about the parser and the zero verbose - wdyt?

younesbelkada · 2024-03-18T15:36:30Z

examples/scripts/chat.py

@@ -0,0 +1,302 @@
+import os
+from threading import Thread
+from transformers import AutoModelForCausalLM, AutoTokenizer, TextIteratorStreamer, HfArgumentParser


I think you can first import here zero_verbose_init from trl.command.cli_utils to remove all verbose and make sure the interface is clear at init

see:

trl/examples/scripts/sft.py

Line 55 in 6cfa5cf

if TRL_USE_RICH:

oh yes, fixed

younesbelkada · 2024-03-18T15:38:23Z

examples/scripts/chat.py

+
+    current_args = copy.deepcopy(args)
+
+    with open(os.path.join(os.path.dirname(__file__), 'default_chat_config.yaml'), "r") as f:


why not using the TrlParser above and use parser.parse_args_and_config:

trl/examples/scripts/sft.py

Line 85 in 6cfa5cf

args, training_args, model_config = parser.parse_args_and_config()

?

younesbelkada · 2024-03-18T15:39:59Z

trl/commands/cli_utils.py

+    top_p: float = field(default=1.0, metadata={"help": "Value of p for nucleus sampling"})
+    repetition_penalty: float = field(default=1.0, metadata={"help": "Repetition penalty"})
+    # model loading
+    model_revision: str = field(


I think for all the arguments below you could leverage ModelConfig from trl instead and on the script above declare a TrlParser with ChatArguments and ModelConfig - see: https://github.com/huggingface/trl/blob/main/examples/scripts/sft.py#L85-L101

I was wondering the same, but it contains also a lot of args that are only relevant to training which i think is confusing (and would show up with help, no?).

ah yes good point ! then i think all good

younesbelkada

Looking great thanks ! I added few nits but overall ready to merge for me !

younesbelkada · 2024-03-19T08:55:54Z

examples/scripts/chat.py

+from trl.trainer.utils import get_kbit_device_map, get_quantization_config
+
+
+init_zero_verbose()


I think you need to import that before transformers and call it before any transformers import - make sure to put a # flake8: noqa at the top of the file to make the CI happy:

trl/examples/scripts/sft.py

Line 1 in 6cfa5cf

# flake8: noqa

younesbelkada · 2024-03-19T08:56:02Z

examples/scripts/chat.py

+        # TODO: attribute fastchat
+        # TODO(suquark): the console flickers when there is a code block
+        #  above it. We need to cut off "live" when a code block is done.
+


younesbelkada · 2024-03-19T08:57:02Z

setup.py

@@ -104,9 +104,9 @@
        entry_points={
            "console_scripts": ["trl=trl.commands.cli:main"],
        },
-        package_data={"trl": ["commands/scripts/*"]},
-        packages=find_packages(),
        include_package_data=True,


this does not break anything with SFT / DPO CLI right?

no, since they are python files in subdirectories they are already included:

adding 'trl/commands/__init__.py' adding 'trl/commands/cli.py' adding 'trl/commands/cli_utils.py' adding 'trl/commands/scripts/chat.py' adding 'trl/commands/scripts/ddpo.py' adding 'trl/commands/scripts/dpo.py' adding 'trl/commands/scripts/kto.py' adding 'trl/commands/scripts/ppo.py' adding 'trl/commands/scripts/ppo_multi_adapter.py' adding 'trl/commands/scripts/reward_modeling.py' adding 'trl/commands/scripts/sft.py' adding 'trl/commands/scripts/config/default_chat_config.yaml'

ok fixed it

* first draft * move chat to cli * fix makefile * make script less verbose * fix parsing * fix style * add more examples * fix setup.py * add copyright * fix verbose init * attribute FastChat * add docs

first draft

630dec7

younesbelkada reviewed Mar 15, 2024

View reviewed changes

leandro added 3 commits March 18, 2024 12:24

Merge branch 'main' into chat-script

de6d5f6

move chat to cli

05735dd

fix makefile

3bd16b4

younesbelkada reviewed Mar 18, 2024

View reviewed changes

leandro added 4 commits March 18, 2024 18:07

make script less verbose

41a1414

fix parsing

378c963

fix style

7f599fc

add more examples

65bd085

lvwerra marked this pull request as ready for review March 18, 2024 20:36

younesbelkada approved these changes Mar 19, 2024

View reviewed changes

leandro added 6 commits March 19, 2024 11:05

fix setup.py

c039bd9

add copyright

1491fa0

fix verbose init

f18adad

attribute FastChat

ed148b2

add docs

fa6ac15

Merge branch 'main' into chat-script

7946bbe

younesbelkada approved these changes Mar 19, 2024

View reviewed changes

lvwerra merged commit 4e622a9 into main Mar 19, 2024
9 checks passed

lvwerra deleted the chat-script branch March 19, 2024 11:37

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

chat cli #1431

chat cli #1431

lvwerra commented Mar 14, 2024

HuggingFaceDocBuilderDev commented Mar 14, 2024

younesbelkada Mar 15, 2024

younesbelkada left a comment

younesbelkada Mar 15, 2024

younesbelkada Mar 19, 2024

younesbelkada left a comment

younesbelkada Mar 18, 2024

younesbelkada Mar 18, 2024

lvwerra Mar 18, 2024

younesbelkada Mar 18, 2024

younesbelkada Mar 18, 2024

lvwerra Mar 18, 2024

younesbelkada Mar 18, 2024

younesbelkada left a comment

younesbelkada Mar 19, 2024

younesbelkada Mar 19, 2024

younesbelkada Mar 19, 2024

lvwerra Mar 19, 2024

younesbelkada Mar 19, 2024

lvwerra Mar 19, 2024

younesbelkada Mar 19, 2024

	"Override the default `torch.dtype` and load the model under this dtype. If `auto` is passed, the "
	"Override the default `torch_dtype` and load the model under this dtype. If `auto` is passed, the "

	# TODO: attribute fastchat
	# TODO(suquark): the console flickers when there is a code block
	# above it. We need to cut off "live" when a code block is done.


		current_args = copy.deepcopy(args)

		with open(os.path.join(os.path.dirname(__file__), 'default_chat_config.yaml'), "r") as f:

		from trl.trainer.utils import get_kbit_device_map, get_quantization_config


		init_zero_verbose()

chat cli #1431

chat cli #1431

Conversation

lvwerra commented Mar 14, 2024

HuggingFaceDocBuilderDev commented Mar 14, 2024

Choose a reason for hiding this comment

younesbelkada left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

younesbelkada left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

younesbelkada left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment