Explore TaskMatrix (previously Visual ChatGPT) / LowCodeLLM

How can we take advantage of https://github.com/microsoft/TaskMatrix ?

Since TaskMatrix recently updated for GroundingDINO and segment-anything support (2 weeks ago), it could definitely be of use for generating segmentation results.

Some ideas I have:

Use TaskMatrix's templates to define execution flows for complex tasks (or common ones).
Compatibility: we can modify the resolution/scale of screenshots to make recordings more compatible with other devices.
Can implement a form of pop-up detection, since sometimes unexpected events can occur (like a pop-up from another application / OS), which would usually break automation processes until a human intervenes.

@0dm

To add to @0dm, we could also implement:

Regression Testing: We can utilize TaskMatrix to automate testing after every change (or every set of changes) to ensure that the changed code is bug-free & functional
Automating User Input: We could use TaskMatrix to simulate user input (e.g. typing commands). In addition, we can use verify the response. Saving time for the coder!

@0dm

Use TaskMatrix's templates to define execution flows for complex tasks (or common ones).

@0dm what would this look like?

forgive me, it's almost 6am and my brain is going to implode (i haven't slept). here's a rough idea with one of their example templates.

they define a template:

class ObjectSegmenting:
    template_model = True # Add this line to show this is a template model.
    def __init__(self,  Text2Box:Text2Box, Segmenting:Segmenting):
        # self.llm = OpenAI(temperature=0)
        self.grounding = Text2Box
        self.sam = Segmenting
        
        # methods below...

now, this template can be applied for a variety of situations, such as the prompt, "segment the given object:" or even "segment the cat" (🐈‍⬛ 😨).

then, they collect all the templates for a list of possible functions:

 # Load Template Foundation Models
      for class_name, module in globals().items():
          if getattr(module, 'template_model', False):
              template_required_names = {k for k in inspect.signature(module.__init__).parameters.keys() if k!='self'}
              # ...

In puterbot, we could define templates for "open application", "send email", etc. and have these functions activate on certain phrases like "open <application> => open chrome" . This would make the codebase very organized and make combining tasks easier, since replays could invoke numerous functions in sequence (open chrome => goto url gmail => compose new email => ...)

@0dm

Thank you @0dm ! Let's talk through this on a call. Please grab some time on my calendar here:

https://www.getclockwise.com/c/richard-abrich/quick-meeting (today after 3pm or tomorrow, otherwise next week).

@abrichr

@abrichr , scheduled for 4:15 tomorrow!

https://aclanthology.org/2024.naacl-demo.2/

https://github.com/chenfei-wu/TaskMatrix/tree/main/LowCodeLLM

https://www.youtube.com/watch?v=jb2C1vaeO3E

Closing as part of architecture transition. Legacy frozen at v0.46.0. See PR #960.

Navigation Menu

Repository navigation

Explore TaskMatrix (previously Visual ChatGPT) / LowCodeLLM #36

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Explore TaskMatrix (previously Visual ChatGPT) / LowCodeLLM #36

Description

Activity

0dm commented on Apr 28, 2023

ahmetutku commented on Apr 28, 2023

abrichr commented on May 4, 2023

0dm commented on May 4, 2023

abrichr commented on May 5, 2023

0dm commented on May 5, 2023

abrichr commented on Jun 19, 2024

abrichr commented on Jan 17, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions