Dalle integration PR

[jackitaliano]

Dalle Integration

Allow for image generation via DALL-E-3 within chats.

Can be used by asking Bidara to generate an image, or show you something you've already been discussing.

Example of how the interaction might go

...Conversation about using kingfisher beaks for aircraft design...

• user: Show me an image of how that would look for the nose-cone of an aircraft
• ai: (image)
• ai: Now that you've seen that visualization, how might you incorporate the design into an aircraft?

Note: rough estimate of the conversation

Example generations

Changes

Function Definition / Bidara Prompt

Problem

• Trouble getting Bidara to actually use its function to generate images.

Solution

• Tried many different "configurations" of the function descriptions/parameters.
• Found the most consistent was to simplify the function while telling Bidara about the function and about it's ability do the thing in the function (i.e. generate images).

Inserting Images

Inserting an image can be done by simply returning a url to an image from funcCalling

Problem

• If voice mode is activated, it will voice-over the entire (long) url, which isn't ideal
• It will also not allow you to change the size of the image (default 1024x1024 because this is all openai currently allows for), also not ideal

Solution

• Using previously deprecated deepChatRef.addMessage, which is now deepChatRef._addMessage
• This feature is "unofficial" and not documented anywhere
• See deep-chat issue#75

Other

• Image generation was relatively simple via fetch to OpenAI. (See image gen docs)
• On error, or no image is received, should gracefully tell the user there was an error
• After image is given to user, should tie back into the biomimicry process