How do profiles work?

[mord0d]

It's mentioned many times that profiles are powerful, but… I don't understand how they works.

1. How to disable "swap mode" to make sure that no model will be unloaded automatically (let ttl do its work)?
2. What happened if there's "swap mode" is on, the active model is currently processing a request from one user, but another user requested a different model?
   // Can't test this case now for technical reasons.

[mostlygeek]

If you find yourself frequently switching between two models in your workflow, profiles can help you avoid the swapping overhead. It can greatly reduce the time to first token.

For example:

profiles:
   coding:
      - qwen2.5-coder-32B
      - qwen2.5-coder-7B

models:
  "qwen2.5-coder-32B":
    proxy: "http://127.0.0.1:9998"
    cmd: /app/llama-server --port 9998 -m ...

  "qwen2.5-coder-7B":
    proxy: "http://127.0.0.1:9999"
    cmd: >
      /app/llama-server --port 9999 -m ...

  "llama3":
    proxy: "http://127.0.0.1:9999"
    cmd: >
      /app/llama-server --port 9999 -m /path/to/llama-3.gguf

Using coding:qwen2.5-coder-32B and coding:qwen2.5-coder-7B as the models names in requests will not trigger a swap. However, a request for llama3 will unload everything and swap to that configuration.

I use it for programming where I want the 32B for general coding questions and the 7B for fast tab completion suggestions.

However, there are a few things to keep in mind:

1. You need the resources to keep both models loaded
2. They need to be listening on different ports

---

It's an advanced feature but it can provide a lot of flexibility and control over the resources you have available. Here is my coding profile with dual 3090s:

profiles:
    coding:
      - qwen-coder-32B
      - qwen-coder-3090-FIM

models:

  # ~123tok/sec
  "qwen-coder-3090-FIM":
    env:
      - "CUDA_VISIBLE_DEVICES=GPU-6f0"
    unlisted: true
    proxy: "http://127.0.0.1:9510"
    cmd: >
      /mnt/nvme/llama-server/llama-server-latest
      --host 127.0.0.1 --port 9510
      -ngl 99 --ctx-size 8096
      -ub 1024 -b 1024
      --model /mnt/nvme/models/Qwen2.5-Coder-7B-Instruct-Q4_K_M.gguf
      --cache-reuse 256

  # on the 3090s
  # 80tok/sec - write snake game
  # ~43tok/sec normally
  "qwen-coder-32B":
    env:
      - "CUDA_VISIBLE_DEVICES=GPU-f10,GPU-6f0"
    proxy: "http://127.0.0.1:8999"
    cmd: >
      /mnt/nvme/llama-server/llama-server-latest
      --host 127.0.0.1 --port 8999 --flash-attn --metrics --slots
      --parallel 2
      --ctx-size 32000
      --ctx-size-draft 32000
      --model /mnt/nvme/models/Qwen2.5-Coder-32B-Instruct-Q5_K_L.gguf
      --model-draft /mnt/nvme/models/Qwen2.5-Coder-1.5B-Instruct-Q8_0.gguf
      --device-draft CUDA1
      -ngl 99 -ngld 99
      --draft-max 16 --draft-min 4 --draft-p-min 0.4
      --cache-type-k q8_0 --cache-type-v q8_0
      # 23.91 GB of CUDA0 ... think this is close enough
      --tensor-split 90,10

Here's how I split up the 48GB VRAM:

• coding:qwen-coder-3090-FIM:
  • config optimized for fill in the middle with the llama.vscode extension
  • put on a single 3090 with low context.
• coding:qwen-coder-32B:
  • higher context, so need to split 90/10 the layers between the two 3090s. Faster to keep as many layers on a single GPU
  • uses the 1.5B coder model as a draft, averages tok/sec is about 45/tok/sec to 50tok/sec

With that I can get the best usage out of my GPU memory:

I hope this explains it a bit (and makes sense) 😅

[mord0d]

Using coding:qwen2.5-coder-32B and coding:qwen2.5-coder-7B as the models names in requests will not trigger a swap. However, a request for llama3 will unload everything and swap to that configuration.

But models are listed without coding:* (all: in my case), so user can't choose it correctly without swap. Most users are lazy (some of them even dumb ☺) they will ignore instructions, because “Why should I input l-e-t-t-e-r-s?! There's a list, I just click/tap on the model I want to run!”). Moreover, some clients do not allow to enter a model name manually.

Is it possible to add models with profile prefix to the list?

I hope this explains it a bit (and makes sense) 😅

Yup, thanks! ☺

---

What happened if there's "swap mode" is on, the active model is currently processing a request from one user, but another user requested a different model?

Just tested this case:
Currently running llama-server will continue its work, new request will be pending.

[AbdullahMPrograms]

piggybacking off this, what if we would like to use the same model across different profiles and don't necessarily want to have duplicated model ids?

For instance, I want to have one profile with Mistral Small 3 and FuseO1-32B and one profile with Qwen 2.5-32B and FuseO1-32B, currently id have to have two FuseO1 models in my selector corresponding to the two profiles, is it possible to somehow reduce this to just one model id that works across multiple profiles?

[mostlygeek]

@AbdullahMPrograms: Yes. For example:

profiles: 
   p1: 
     - modelA
     - modelB
   p2:
     - modelC
     - modelB
  p3:
    - modelA
    - modelD
    - modelC
  all: 
    - modelA
    - modelB
    - modelC
    - modelD

Important things to keep in mind:

• Each model should be listening on a unique port
• Make sure you have enough resources to load all the models in a profile.

Models are loaded on demand. So this could happen with subsequent requests:

request: all:modelA --> success
request: all:modelB --> success
request: all:modelC --> fail, llama-swap tried to load modelC but server didn't have enough VRAM. 
request: all:modelA --> success

I found that I make new model definitions when I need to tweak things slightly to make a profile fit.

[mord0d]

but server didn't have enough VRAM

You cannot save users from their laziness or stupidity. ☺
This is an admin's problem, they should better plan their resources.

IMHO, tool should not (even shouldn't try to) be smarter than its user.

[AbdullahMPrograms]

Hi @mostlygeek! Just to clarify my earlier issue: I want to keep modelB loaded when switching between profiles (p1 and p2) to avoid reloading it or having duplicate entries in the model selector. For example:

• p1 has modelA + modelB

• p2 has modelC + modelB

Switching between them should unload only modelA → load modelC, while keeping modelB in memory. Using the "all" profile causes OOM errors when loading modelA+B+C together (as expected with limited VRAM). For larger models (e.g., modelD), I’d want older models (A/B) to unload automatically to free space. The current multi-profile setup works but clutters the selector with duplicate modelB entries. Is there a way to achieve this "shared model retention" behavior? Thanks again! 😊

[mostlygeek]

@AbdullahMPrograms I think I understand your use case. You want to partially swap out a profile of the models that are have changed in a profile. This isn't support. Switching profiles is an all or nothing operation.

[void-mckenzie]

@mostlygeek I can work on creating a version which can support this, but I'm not sure if it's a direction you want this project to go ahead in. Let me know if you plan on supporting this type of profile offloading in the future, I can take care of it.

[mostlygeek]

Hi @void-mckenzie,

It’s not an optimization that I would like to maintain. The way the code works is fairly easy to follow and reason about. It took a lot more effort than anticipated to keep it both simple and stable.