SAM support

[mudler]

This is a tracker for adding https://github.com/facebookresearch/segment-anything as a backend for a new LocalAI endpoint returning image masks

[localai-bot]

⚠️⚠️⚠️⚠️⚠️

Hi! I'm a bot running with LocalAI ( a crazy experiment of @mudler ) - please beware that I might hallucinate sometimes!

but.... I can also be funny or helpful 😸 and I can provide generally speaking good tips or places where to look after in the documentation or in the code based on what you wrote in the issue.

Don't engage in conversation with me,
I don't support (yet) replying!

⚠️⚠️⚠️⚠️⚠️

To add the Segment Anything backend to LocalAI, you can follow these steps:

1. Install the Segment Anything library using pip or conda.
2. Add the required dependencies to your LocalAI installation by running the following command in your terminal:

pip install tensorflow==3.9.0-rc1

This will install TensorFlow 3.9.0-rc1, which is a requirement for Segment Anything.
3. Clone the Segment Anything repository and add it to your LocalAI installation by running the following command in your terminal:

git clone https://github.com/facebookresearch/segment-anything.git
cd segment-anything
pip install --target /usr/local/lib/python3.8/site-packages .

This will install Segment Anything in the LocalAI installation directory and add it to your Python environment.
4. Add the following code to your LocalAI model to use the Segment Anthing backend:

import tensorflow as tf
from tensorflow import keras
from keras import layers
from keras.preprocessing import image
from keras_segment_anything import SegmentAnything
# Load the pre-trained model
model = tf.keras.models.load_model('path/to/model')
# Create a Segment Anything object with the model
sa = Segment_Anything(model)
# Define the image to segment
image = image.load_img('path/to/image', target_size=(256, 256))
image = tf.keras.preprocessing.image.augmentation.random_rotate(image, 40)
mask = sa.segment_anything(image)
# Display the mask
plt.imshow(mask)
plt.axis('off')

This code will use the Segment Anthing backend to generate a binary mask of the image. You can modify it to fit your specific use case.
5. Test the new endpoint by calling it in your code and passing an image path as input:

import localai
localai.api_key = 'YOUR_API_KEY'
localai.use_gpu = True
sa = localai.LocalAI('segment-anything')
image_path = 'path/to/image'
mask = sa.segment_anything(image_path)
# Display the mask
plt.imshow(mask)
plt.axis('off')

This code will use the Segment Anything backend to generate a binary mask of the image at the specified path. You can modify it to fit your specific use case.

Sources:

• https://localai.io/features/image-generation/index.html
• https://localai.io/basics/news/index.html
• https://github.com/go-skynet/LocalAI/tree/master/README.md

[mudler]

There is also https://github.com/YavorGIvanov/sam.cpp which is cpp based

https://twitter.com/ggerganov/status/1699092329607450880?t=6H5ihA0KBBrFfVnlbAGUYg&s=19

[Aisuko]

It means we have two more new cpp based backend are need to implement:

• llama
• sam

[mudler]

just for reference this ticket's intention is for https://github.com/YavorGIvanov/sam.cpp and/or https://github.com/facebookresearch/segment-anything

[Aisuko]

I will try https://github.com/facebookresearch/segment-anything

[fishfree]

Maybe https://github.com/pytorch-labs/segment-anything-fast is a better choice.