add alias for lora adaptors

[zhipenghan]

• I have read the contributing guidelines
• Self-reported review complexity:
  • Low
  • Medium
  • High

Added the ability to create aliases for adaptors, allowing for more intuitive and flexible naming conventions.
This feature simplifies referencing adaptors in different contexts, reducing potential errors and enhancing code clarity.

... --lora taskA_alias=ggml-lora-for-taskA-f16.gguf

[ngxson]

IMO it should not be the responsible of llama.cpp library to manage adapter name or alias.

The library already returns a pointer to the loaded adapter. User code can wrap that adapter into their own data struct, for example:

struct my_adapter {
  struct llama_lora_adapter * adapter;
  std::string alias;
};

...

struct my_adapter task_A{
  llama_lora_adapter_init(...),
  "task A",
};

Edit: also for wording, we prefer "adapter" over "adaptor", because it is referred in the original paper: https://arxiv.org/pdf/2106.09685

[ngxson]

This is not the correct usage of lora adapter in llama.cpp

The adapter is loaded independently from inference context (llama_context). To control which adapter is enabled or not, use llama_lora_adapter_set and llama_lora_adapter_remove

See: #8332

[zhipenghan]

This is not the correct usage of lora adapter in llama.cpp

The adapter is loaded independently from inference context (llama_context). To control which adapter is enabled or not, use llama_lora_adapter_set and llama_lora_adapter_remove

See: #8332

Thanks for replying.
If the adaptors are expected to set/remove outside of the library. It should not be loaded while initialized the model.

IMO, the adaptors can be handled outsize the library, but the exist adaptors are unknown and invisiable. Need explicity way to clean current adaptors. It would be better to have llama_lora_adaptors_clean to clean exist adaptors.
In real scenarios, apply multiple adaptors on the foundation model is not common(let me know if there are cases). Apply one adaptor on foundation model is a common partten. And when setting a adaptor, usually we need clean exist adaptors outside of the library, but the exist adaptors are invisiable.

[ngxson]

If the adaptors are expected to set/remove outside of the library. It should not be loaded while initialized the model.

I think there's small misconception here. llama.cpp comes with 2 main interfaces: llama_context and llama_model. When you load the model, tensors are stored in llama_model as read-only. If you want to use these tensors for inference, you need to pass the model to llama_context. This makes it possible to re-use one llama_model in multiple llama_context

The same idea applies to llama_lora_adapter. When adapter is loaded, you can reuse it in multiple different llama_context. Imagine one context can use adapter for task A and the other can use adapter for task B.

The list of active adapters is held by each llama_context (with the corresponding scales)

When you want to remove adapter from llama_context, use llama_lora_adapter_remove. To free an adapter (i.e. remove it completely from memory), use llama_lora_adapter_free. The adapters are also free if llama_model is freed, because adapter is linked to specific model.

[zhipenghan]

If the adaptors are expected to set/remove outside of the library. It should not be loaded while initialized the model.

I think there's small misconception here. llama.cpp comes with 2 main interfaces: llama_context and llama_model. When you load the model, tensors are stored in llama_model as read-only. If you want to use these tensors for inference, you need to pass the model to llama_context. This makes it possible to re-use one llama_model in multiple llama_context

The same idea applies to llama_lora_adapter. When adapter is loaded, you can reuse it in multiple different llama_context. Imagine one context can use adapter for task A and the other can use adapter for task B.

The list of active adapters is held by each llama_context (with the corresponding scales)

When you want to remove adapter from llama_context, use llama_lora_adapter_remove. To free an adapter (i.e. remove it completely from memory), use llama_lora_adapter_free. The adapters are also free if llama_model is freed, because adapter is linked to specific model.

After lctx, model loaded outside of the library:
std::tie(model, ctx) = llama_init_from_gpt_params(params);
The ctx handles the adaptors and invisible outside library scope. The inference context allocate the memory for computation graph calculation. The efficient way is reused the same context.
Or the adaptors have to be handled outside the library and prevent load any adaptors within the library scope.

[ngxson]

IMO, the adaptors can be handled outsize the library, but the exist adaptors are unknown and invisiable. Need explicity way to clean current adaptors. It would be better to have llama_lora_adaptors_clean to clean exist adaptors.

I agree that common use case is one adapter per context, and llama_lora_adaptors_clean maybe useful to detach all adapters from a given context. But IMO llama.cpp should only provides most low-level API and user can build their own wrapper logic on top of it. Much like how operating systems only provide most basic syscalls, and libraries like glibc build on top of these syscalls.

In your case, user can simply write a llama_lora_adapter_switch(llama_lora_adapter * ptr) that automatically detach old adapter using llama_lora_adapter_remove, then attach the new one with llama_lora_adapter_set. It could be implement as a utility (see common folder).

[zhipenghan]

IMO, the adaptors can be handled outsize the library, but the exist adaptors are unknown and invisiable. Need explicity way to clean current adaptors. It would be better to have llama_lora_adaptors_clean to clean exist adaptors.

I agree that common use case is one adapter per context, and llama_lora_adaptors_clean maybe useful to detach all adapters from a given context. But IMO llama.cpp should only provides most low-level API and user can build their own wrapper logic on top of it. Much like how operating systems only provide most basic syscalls, and libraries like glibc build on top of these syscalls.

In your case, user can simply write a llama_lora_adapter_switch(llama_lora_adapter * ptr) that automatically detach old adapter using llama_lora_adapter_remove, then attach the new one with llama_lora_adapter_set. It could be implement as a utility (see common folder).

Agree adding a llama_lora_adaptors_clear to clean exist adaptors. If this cleaning functionality is visible outside of the library’s scope, users can manipulate the adaptors outside of the library to customize them.

[ngxson]

Please have a look on #8653

[slaren]

After lctx, model loaded outside of the library:
std::tie(model, ctx) = llama_init_from_gpt_params(params);
The ctx handles the adaptors and invisible outside library scope.

llama_init_from_gpt_params is not part of the library, it's only part of the examples. 3rd party applications should not rely on it.

[ngxson]

I'll close this because the use case can be covered by #8653