multi-modality model construction support #1068

Gasoonjia · 2024-08-27T23:44:04Z

This PR makes torchchat support multi-modality model definition and constructions. To show our power in multi-modality area, we integrate flamingo component into our system.
Note that this is only for bare-minimum support for model definition. Please check openai_api_multimodal branch for e2e, and #1123 (comment) for better structure and llama3.1 support

pytorch-bot · 2024-08-27T23:44:06Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/torchchat/1068

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit b96bf05 with merge base c272df4 ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

…ponent

Jack-Khuu

This is looking great!! Thanks for making it so easy to review

Mostly nits, but I know this is part of a PR stack we're landing, so we have some leeway

Jack-Khuu · 2024-09-10T01:39:42Z

.DS_Store

You might have to manually tell git not to track

There's a few DS Stores

Jack-Khuu · 2024-09-10T01:42:23Z

torchchat/model.py

+@dataclass
+class ModelRecipe:
+    model_type: ModelType
+    modules: dict


Suggested change

modules: dict

modules: Dict[str, Any]

Jack-Khuu · 2024-09-10T01:44:03Z

torchchat/model.py

+
+
+@dataclass
+class ModelRecipe:


nit: Docstring

Jack-Khuu · 2024-09-10T01:45:33Z

torchchat/model.py

+    fusion_class: torch.nn.Module
+
+    @classmethod
+    def text_only(cls):


Suggested change

def text_only(cls):

def _text_only(cls):

Jack-Khuu · 2024-09-10T01:45:43Z

torchchat/model.py

+            fusion_class=nn.Identity,
+        )
+    @classmethod
+    def flamingo(cls):


Suggested change

def flamingo(cls):

def _flamingo(cls):

Jack-Khuu · 2024-09-10T01:54:17Z

torchchat/model.py

+
+        self.model_type = model_type
+        if isinstance(transformer_args, TransformerArgs):
+            self.transformer_args = {"text": transformer_args}


Let's make "text" a constant as well, we use it in a lot of places

maybe in the different PR if you think it is good. I have some plans to make the configuration more concise and structual.

Jack-Khuu · 2024-09-10T01:58:06Z

torchchat/model.py


-        return cls(text_transformer_args)
+            # now only support flamingo model


Suggested change

# now only support flamingo model

# Currently only supporting flamingo model

Jack-Khuu · 2024-09-10T02:00:30Z

torchchat/model.py

@@ -77,13 +124,32 @@ def from_params(cls, params):
                params[_to] = params.pop(_from)
        return cls(**params)

+
 @dataclass
 class ModelArgs:


Jack-Khuu · 2024-09-10T02:01:05Z

torchchat/model.py


-    def setup_caches(self, max_batch_size, max_seq_length):
-        self.text_transformer.setup_caches(max_batch_size, max_seq_length)
+    def build_model(self):


This is where all the magic comes together; Let's add a docstring here

Jack-Khuu · 2024-09-10T02:03:45Z

torchchat/model.py

@@ -184,13 +250,48 @@ class Model(nn.Module):
    def __init__(self, config: ModelArgs) -> None:


Since we have the legacy text_transformer and new model, let's add a quick description until we unify them later

Or link to your other PR where you fix this

That will happen adjacent to this PR (3.1 torchtune one), so i'd like to keep as it is here.

Jack-Khuu · 2024-09-10T02:09:56Z

Before landing:

update the PR title and description
Check that the existing text only model flow works (walk through README)

Jack-Khuu · 2024-09-11T01:19:58Z

dist_run.py

@@ -122,7 +122,7 @@ def main():
    gpu_memory_monitor = GPUMemoryMonitor("cuda")
    logger.info(f"{color.yellow} {gpu_memory_monitor.get_device_info()}{color.reset}")

-    config = ModelArgs.from_name(MODEL_NAME).text_transformer_args
+    config = ModelArgs.from_name(MODEL_NAME)..transformer_args['text']


Jack-Khuu · 2024-09-11T01:26:48Z

torchchat/model.py


 @dataclass
 class ModelRecipe:
+    """
+    A class in TorchChat that describes and contains all supported model structures in TorchChat.


Suggested change

A class in TorchChat that describes and contains all supported model structures in TorchChat.

A class in torchchat that describes and contains all supported model structures in torchchat.

Jack-Khuu · 2024-09-11T01:27:14Z

torchchat/model.py

+    A class in TorchChat that describes and contains all supported model structures in TorchChat.
+
+    ModelRecipe represents a model as a collection of Transformer modules and a fusion module,
+    providing a standardized and centralized way to define and build models in TorchChat.


Suggested change

providing a standardized and centralized way to define and build models in TorchChat.

providing a standardized and centralized way to define and build models in torchchat.

Jack-Khuu · 2024-09-11T01:28:08Z

torchchat/model.py

@@ -247,6 +267,9 @@ def update(self, input_pos, k_val, v_val):


 class Model(nn.Module):
+    """
+    The entrance for model construction in tochchat.


Suggested change

The entrance for model construction in tochchat.

The entrance for model construction in torchchat.

Gasoonjia added 2 commits August 27, 2024 16:37

added model source and type for torchtune flamingo support

0dae9ef

added model source and type for torchtune flamingo support

87397e3

facebook-github-bot added the CLA Signed This label is managed by the Meta Open Source bot. label Aug 27, 2024

Gasoonjia and others added 19 commits August 27, 2024 16:48

grab missing enum

0f61614

fix ModelArgs init

d7f3a88

create init func for ModelArgs for BC

994b148

update pipeline for ModleSource and ModelType

d184e68

Merge branch 'main' of github.com:pytorch/torchchat into flamingo_com…

6bb2485

…ponent

revert lintrunner update on ET

0d8e368

introduce flamingo modules form torchtune

6c78850

back up to move to linux

2691bae

mitigate building issue

ba960f0

pass local test

8b3a684

merge solved

880dfe2

structual model builder

e7fa7b4

update torchtune address

c179bcb

update install requirement

5ead73b

support new torchtune flamingo component

882c336

specific version for vision and ao

952b8bd

convert installation back and bypass torchtune

9679a5b

Merge branch 'main' into flamingo_component

56006ea

update exportation variable name

59337a6

Jack-Khuu approved these changes Sep 10, 2024

View reviewed changes

Gasoonjia added 2 commits September 10, 2024 16:43

solve merge confilct

d0e2974

solve Jack's wonderful comments

758af10

Jack-Khuu reviewed Sep 11, 2024

View reviewed changes

Gasoonjia added 2 commits September 10, 2024 18:21

remveo extra dot

80b5481

add type.Callable

1cc7909

Jack-Khuu approved these changes Sep 11, 2024

View reviewed changes

fix torchchat typos

95684d9

Gasoonjia changed the title ~~Flamingo component~~ multi-modality model construction support Sep 11, 2024

remove all .DS_Store

b96bf05

Gasoonjia merged commit 964d437 into main Sep 11, 2024
51 checks passed


		return cls(text_transformer_args)
		# now only support flamingo model

	# now only support flamingo model
	# Currently only supporting flamingo model

		@@ -184,13 +250,48 @@ class Model(nn.Module):
		def __init__(self, config: ModelArgs) -> None:

	A class in TorchChat that describes and contains all supported model structures in TorchChat.
	A class in torchchat that describes and contains all supported model structures in torchchat.

	providing a standardized and centralized way to define and build models in TorchChat.
	providing a standardized and centralized way to define and build models in torchchat.

	The entrance for model construction in tochchat.
	The entrance for model construction in torchchat.

multi-modality model construction support #1068

multi-modality model construction support #1068

Uh oh!

Conversation

Gasoonjia commented Aug 27, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Aug 27, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/torchchat/1068

✅ No Failures

Uh oh!

Jack-Khuu left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Jack-Khuu commented Sep 10, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Gasoonjia commented Aug 27, 2024 •

edited

Loading

pytorch-bot bot commented Aug 27, 2024 •

edited

Loading

Jack-Khuu commented Sep 10, 2024 •

edited

Loading