Allow images in the response

[automaton82]

Currently the response of all LLMs is text-only. But it would be good if there was a way to include images in the responses if the URL is known, then the image could just be embedded in the response.

[HenryHengZJ]

Currently we use react-markdown to display the response. In order to display image, we'll have to use the markdown format

[automaton82]

Okay that makes sense. I can write my own agent then that could return images in markdown, I think that will solve it. I will try.

[flang-ls1]

I would be really interested in your approach in solving this, I do not know how to do this. I have a pretty good RAG flow for a support chatbot using Pinecone and Conv Retrieval QA Chain and played around with some agents too (using retriever tool, sending email..., among others) but my support chatbot is not yet able to retrieve or display any images. I think i could upsert some metadata source keys for images along with the vector and text but I don't know how to use this in my chain. Any further guidance much appreciated!

[automaton82]

The answer is there but I'm going to leave this open because I think the general chains in Flowise should be able to give images back as well which I don't think they can do, thus it's still an open idea.

[marius-smg]

This would be very useful, great idea!

[RobinGB69]

Just wanted to add my support for the need, as I would also like to achieve this...

I have websites that are designed to help people find answers to questions on various different topics. My web pages contain text and images to help explain topics. Hence rather than a user having to read and search through the website I have created a RAG chatbot that is great at answering questions and providing citations to the most relevant web pages on my site. However, I don't know how to get the chatbot to include the images that are on my website next to the text the LLM used for its answer.

I am thinking I need to provide the LLM a description of each image and what chunks of text the image relates to, but how do I add that info to the vector knowledge base, and then get the chatbot to include the image in its answers? I am not asking for it to interpret images, nor generate images, only provide the relevant existing images that are on my website as part of its answers.

Hence, people wouldn't need to use the website anymore, as the chatbot would give them the information they want in a text and picture form. The website then essentially becomes just a structured knowledge store that is used for grounding the chatbot.