Images & Multimodal support

[tomipardinas]

When asked about a specific text from the database, it only recognizes text, but the text from the image isn't recognized:
Here the question:

Here the tweet mentioning the image with the text

[Dhravya]

There's ways to fix this right now, but I think that at scale it doesn't make sense to use a vision model or even OCR for that matter on every single image.
We currently are reaching into 60,000 tweets imported into supermemory. a lot of them also have images. would get expeensive

[CodeTorso]

so you have any other method method ?

[krakenftw]

@Dhravya for the time being, if we really want the feature, we can use some OCR API? which are free/extremely cheap?

[krakenftw]

https://ocr.space/ocrapi here is one

[Welding-Torch]

This should be tackled eventually. If we're importing tweets into Supermemory, providing a first-class experience to the user is important.

[tomipardinas]

Yes. If the objective is truly to create a "super" memory, it needs to be able to extract useful information from all types of unstructured data in your tweets, including images containing text. This is especially important because tweets with images are more likely to be retweeted, which increases their chances of being valuable and, consequently, the probability of being bookmarked.

[Dhravya]

Hmmm. this is gonna be a long discussion. We did create endpoints and stuff to make this work long ago but nothing after that

[CodeTorso]

yep that endpoint uses llava-1.5-7b-hf,

nowhere on cloudflare's docs we can get an idea of how much it will cost

but since we are already looking forward to adding pro plan, why not make it part of that ?

[Welding-Torch]

@CodeTorso As a user I would definitely expect this to be part of the free plan since importing tweets right now is basically our USP

[Dhravya]

Hmmmmm, llava right now is free. We can probably look into this and do a dual sort of import, should be not that hard when we have the image URL that we can fetch

but we can only do this after the queues PR is merged, and then add an extra step in the add workflow to fetch any images and embed those too

[ameeetgaikwad]

@Dhravya we can add paid version for saving images. Can integrate stripe. What say?

[Welding-Torch]

Again,
As a user I would definitely expect recognizing content in saved tweet images to be part of the free plan since importing tweets right now is basically our USP

[ameeetgaikwad]

Hmm, that's right.