feat: Add backend gallery #5607

mudler · 2025-06-09T17:58:15Z

Description

This PR add support to manage backends as similar to models. There is now available a backend gallery which can be used to install and remove extra backends. The backend gallery can be configured similarly as a model gallery, and
API calls allows to install and remove new backends in runtime, and as well during the startup phase of LocalAI.

Users are encouraged, as with models, to create their own backend galleries. Backends are just standard OCI images which have packed all the bits to run a backend. Documentation is provided too.

Users can specify the path of backends with --backends-path or by using the LOCALAI_BACKENDS_PATH environment variable. --external-backends or LOCALAI_EXTERNAL_BACKENDS can be set to a list of backends to load automatically on start. Finally, LOCALAI_BACKEND_GALLERIES or --backend-galleries can be a json list of backend galleries, for example default is [{"name":"localai", "url":"github:mudler/LocalAI/backend/index.yaml@master"}].

Fixes #3953

Notes for Reviewers

Not-exhaustive list of related tickets of backends that can be now implemented completely separately (yay!):

#5519
#1126
#5447
#5215
#4168

Signed commits

Yes, I signed my commits.

netlify · 2025-06-09T17:58:20Z

✅ Deploy Preview for localai ready!

Name	Link
🔨 Latest commit	`f12e009`
🔍 Latest deploy log	https://app.netlify.com/projects/localai/deploys/684e828ecc31a000086c091d
😎 Deploy Preview	https://deploy-preview-5607--localai.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

core/gallery/gallery.go

-	models, err := AvailableGalleryModels(galleries, basePath)
+func ReadConfigFile[T any](filePath string) (*T, error) {
+	// Read the YAML file
+	yamlFile, err := os.ReadFile(filePath)


core/gallery/backends.go

+	if config.Alias != "" {
+		// Write an alias file inside
+		aliasFile := filepath.Join(backendPath, "alias")
+		if err := os.WriteFile(aliasFile, []byte(config.Alias), 0644); err != nil {


core/gallery/backends.go

This PR add support to manage backends as similar to models. There is now available a backend gallery which can be used to install and remove extra backends. The backend gallery can be configured similarly as a model gallery, and API calls allows to install and remove new backends in runtime, and as well during the startup phase of LocalAI. Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

core/cli/run.go

@@ -73,11 +76,15 @@
 }

 func (r *RunCMD) Run(ctx *cliContext.Context) error {
+	os.MkdirAll(r.BackendsPath, 0750)


core/cli/run.go

@@ -73,11 +76,15 @@
 }

 func (r *RunCMD) Run(ctx *cliContext.Context) error {
+	os.MkdirAll(r.BackendsPath, 0750)
+	os.MkdirAll(r.ModelsPath, 0750)


core/http/routes/ui_backend_gallery.go

+			"Title":              "LocalAI - Backends",
+			"BaseURL":            utils.BaseURL(c),
+			"Version":            internal.PrintableVersion(),
+			"Backends":           template.HTML(elements.ListBackends(backends, opcache, galleryService)),


core/http/routes/ui_backend_gallery.go

+			summary["NextPage"] = nextPage
+			summary["TotalPages"] = totalPages
+			summary["CurrentPage"] = pageNum
+			summary["Backends"] = template.HTML(elements.ListBackends(backends, opcache, galleryService))


core/http/routes/ui_gallery.go

+			"Title":            "LocalAI - Models",
+			"BaseURL":          utils.BaseURL(c),
+			"Version":          internal.PrintableVersion(),
+			"Models":           template.HTML(elements.ListModels(models, opcache, galleryService)),


core/http/routes/ui_gallery.go

+			summary["NextPage"] = nextPage
+			summary["TotalPages"] = totalPages
+			summary["CurrentPage"] = pageNum
+			summary["Models"] = template.HTML(elements.ListModels(models, opcache, galleryService))


core/services/models.go

+}
+
+func ApplyGalleryFromFile(modelPath, s string, enforceScan bool, galleries []config.Gallery) error {
+	dat, err := os.ReadFile(s)


core/services/models.go

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

Signed-off-by: Ettore Di Giacinto <mudler@users.noreply.github.com>

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

… reading ggufs with gguf-parser Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

github-advanced-security bot found potential problems Jun 10, 2025

View reviewed changes

mudler force-pushed the feat/backend_gallery branch from 49199b6 to f18108c Compare June 10, 2025 17:39

github-advanced-security bot found potential problems Jun 10, 2025

View reviewed changes

github-actions bot added the dependencies label Jun 11, 2025

mudler force-pushed the feat/backend_gallery branch from fe47de8 to b57f399 Compare June 11, 2025 17:29

mudler changed the title ~~wip: backend gallery~~ feat: Add backend gallery Jun 11, 2025

github-advanced-security bot found potential problems Jun 11, 2025

View reviewed changes

mudler added enhancement New feature or request breaking-change labels Jun 11, 2025

mudler added 2 commits June 11, 2025 19:44

Add backends docs

5226227

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

wip: Backend Dockerfile for python backends

3099afb

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

mudler force-pushed the feat/backend_gallery branch from 11968f3 to 672f467 Compare June 12, 2025 16:46

feat: drop extras images, build python backends separately

2ba3899

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

mudler force-pushed the feat/backend_gallery branch from 672f467 to 2ba3899 Compare June 12, 2025 16:47

mudler added 4 commits June 12, 2025 18:50

fixup on all backends

eb5707c

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

test CI

9bd1478

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

Tweaks

389d4e2

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

Drop old backends leftovers

ee692be

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

github-actions bot added the ci label Jun 12, 2025

mudler and others added 9 commits June 12, 2025 23:13

Fixup CI

c07c577

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

Move dockerfile upper

6d96f00

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

Fix proto

6b90ef5

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

Feature dropped for consistency - we prefer model galleries

97d3176

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

Merge branch 'master' into feat/backend_gallery

06234d6

Signed-off-by: Ettore Di Giacinto <mudler@users.noreply.github.com>

Add missing packages in the build image

4493bc4

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

exllama is ponly available on cublas

98573b4

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

pin torch on chatterbox

06628f8

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

Fixups to index

427bbf4

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

mudler added 7 commits June 13, 2025 22:15

CI

c36deef

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

Debug CI

1a924b2

Install accellerators deps

5751d2e

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

Add target arch

7402a4e

Add cuda minor version

28f90e5

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

Use self-hosted runners

5e83023

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

ci: use quay for test images

ed39960

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

mudler force-pushed the feat/backend_gallery branch from 8666e76 to ed39960 Compare June 14, 2025 08:58

mudler added 4 commits June 14, 2025 16:05

fixups for vllm and chatterbox

1545d31

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

Small fixups on CI

a427c7a

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

chatterbox is only available for nvidia

7da886d

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

Simplify CI builds

1699c96

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

mudler force-pushed the feat/backend_gallery branch from ef1e957 to 1699c96 Compare June 14, 2025 15:31

mudler and others added 5 commits June 14, 2025 18:49

Adapt test, use qwen3

3b63a71

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

Merge branch 'master' into feat/backend_gallery

5d1241d

chore(model gallery): add jina-reranker-v1-tiny-en-gguf

ff820f8

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

fix(gguf-parser): recover from potential panics that can happen while…

211891c

… reading ggufs with gguf-parser Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

Use reranker from llama.cpp in AIO images

ad05b49

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

github-actions bot added the area/ai-model label Jun 14, 2025

Limit concurrent jobs

f12e009

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

mudler marked this pull request as ready for review June 15, 2025 10:15

mudler merged commit 2d64269 into master Jun 15, 2025
71 of 77 checks passed

mudler deleted the feat/backend_gallery branch June 15, 2025 12:56

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

feat: Add backend gallery #5607

feat: Add backend gallery #5607

Uh oh!

mudler commented Jun 9, 2025 •

edited

Loading

Uh oh!

netlify bot commented Jun 9, 2025 •

edited

Loading

Uh oh!

Check failure

Check failure

Uh oh!

Check warning

Check warning

Check failure

Check failure

Check failure

Check failure

Check failure

Uh oh!

Uh oh!

Uh oh!

Uh oh!

feat: Add backend gallery #5607

feat: Add backend gallery #5607

Uh oh!

Conversation

mudler commented Jun 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

netlify bot commented Jun 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for localai ready!

Uh oh!

Check failure

Check failure

Uh oh!

Check warning

Check warning

Check failure

Check failure

Check failure

Check failure

Check failure

Uh oh!

Uh oh!

Uh oh!

mudler commented Jun 9, 2025 •

edited

Loading

netlify bot commented Jun 9, 2025 •

edited

Loading