feature: text to video

[mudler]

With the diffusers backend now in, should be trivial to add video generation, too: https://huggingface.co/docs/diffusers/api/pipelines/text_to_video, https://huggingface.co/docs/diffusers/api/pipelines/text_to_video_zero

[localai-bot]

⚠️⚠️⚠️⚠️⚠️

Hi! I'm a bot running with LocalAI ( a crazy experiment of @mudler ) - please beware that I might hallucinate sometimes!

but.... I can also be funny or helpful 😸 and I can provide generally speaking good tips or places where to look after in the documentation or in the code based on what you wrote in the issue.

Don't engage in conversation with me,
I don't support (yet) replying!

⚠️⚠️⚠️⚠️⚠️

To add video generation with diffusers, you can use the Text-to-Video pipeline in the Diffusers library. This pipeline takes a text input and generates a video output using a pretrained model. Here's an example of how to use this API endpoint in LocalAI:

curl -X POST \
  http://localhost:8080/v1/text_to_video \
  -H "Content-Type: multipart/form-data" \
  -F text="Hello, world!" \
  -F model="<MODEL_NAME>" \
  -F video_format="mp4" \
  -F resolution="1280x720" \
  -F duration="3s"

In this example, the text field contains the input text that you want to generate a video for. The model field specifies the name of the pretrained model that you want to use for the generation. The video_format, resolution, and duration fields specify the format, resolution, and duration of the output video.
Note that this is just an example and you may need to adjust the parameters based on your specific use case. Also, make sure to have the appropriate model installed in the models directory before using this API endpoint.

Sources:

• https://localai.io/index.html
• https://localai.io/basics/news/index.html
• https://localai.io/features/audio-to-text/index.html
• https://localai.io/model-compatibility/diffusers/index.html

[mudler]

introduced in #1442