Decouple tasks from model engines and introduce modalities

[dlqqq]

Demo

Screen.Recording.2023-03-13.at.6.18.48.PM.mov

Description

• Decouples tasks from model engines by introducing the concept of modalities. Defined loosely, they can be thought of as representing the input and output types for a model.

  • This is motivated by the fact that tasks are frequently shared between model engines of a common modality (e.g. txt2txt: GPT-3/ChatGPT, txt2img: DALL-E/Stable Diffusion). It is redundant to have each model engine of a common modality to re-define the default tasks they offer.
  • Each modality is associated with a string representation known as a modality ID.
  • Right now, there are only two modality IDs under Jupyter AI:
    1. txt2txt: accepts an English text prompt and generates English text that roughly corresponds to the prompt
    2. txt2img: accepts an English text prompt and generates an image that roughly corresponds to the prompt

• Model engines now must declare what modalities they support as a class attribute. This is simply a list of modality IDs, for example:

class ChatGptModelEngine(BaseModelEngine):
    id = "chatgpt"
    name = "ChatGPT"
    modalities = [
        "txt2txt"
    ]

• Model engines may support multiple modalities, but most models available today only can handle one.
• A modality is defined strictly as a pair of two sets: a set of input types and a set of output types.
  • For example, a model that generates both an image and a text caption would have to declare a unique modality, such as text-to-captioned-image. This is because this model has two output types, and its modality is not described by txt2txt or txt2img.
  • For example, an extremely fine-tuned language model that can only answer questions in Chinese would also have to declare a unique modality, as its set of input types consists exclusively of Chinese text, not English text.
• Tasks must declare one and only one modality.
• Tasks are now declared as entrypoints under jupyter_ai.default_tasks.
• The DescribeTask API response now includes a list of valid model engines that support the modality of a given task. Very useful 😁
• The Dialog UI now includes a dropdown to select a model engine for a given task. The only model engines listed are ones that support a task's modality.
• Model engines now declare their ID in a separate attribute id. name is now reserved for a model engine's human readable name, e.g. "ChatGPT".

Notes

You need to delete the SQLite DB at ~/.local/share/jupyter/ai_task_manager.db.

Other fixes

• Fixes dev setup
• Re-enables check-release workflow for PRs (I know I backtracked on this, but since we don't have any other workflows, this one will have to do for now)