Allow the `llama_progress_callback` to abort model loading early without having to throw an exception

[LoganDark]

Prerequisites

Please answer the following questions for yourself before submitting an issue.

• I am running the latest code. Development is very rapid so there are no tagged versions as of now.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new bug or useful enhancement to share.

Feature Description

Allow the llama_progress_callback to return a value that will stop the model being loaded, and free all resources.

Motivation

LLMs can brush up against the limits of some computers, and sometimes you just need an emergency stop button. llama.cpp can already catch std::exceptions inside the model loading process and clean up the half-loaded model, but unfortunately, non-C++ languages (such as Rust) can't throw std::exceptions, so even if they do unwind, it won't be caught by llama.cpp's try-catch and the resources used by the model won't actually be properly cleaned up.

Possible Implementation

Allow the llama_progress_callback to return a value that aborts model loading early. Maybe have it return a bool where true is continue and false is abort? This could totally bite existing codebases though since it's really subtle.

[slaren]

This seems related: #4462

[LoganDark]

This seems related: #4462

It does indeed! That would solve my problem, I'm pretty sure.

[LoganDark]

I tested it and it works, except for an access violation that I've submitted a fix for at #4554 (it comes up when freeing only-partially-initialized models).

[crasm]

How are you getting a pointer to a partially initialized model from my PR? It should be returning nullptr if loading is aborted.

[LoganDark]

How are you getting a pointer to a partially initialized model from my PR?

I'm not the one with the pointer, llama.cpp is. When loading is canceled, the model is freed via delete model:

https://github.com/ggerganov/llama.cpp/blob/bdfe4ba85c72df7fb521f7222a9dca017177d734/llama.cpp#L9304-L9314

At least one of the tensors have their backend set to GGML_BACKEND_GPU, but ggml_cuda_transform_tensor has not actually been called on them yet. Look at the location of the return false here, in relation to the call to ggml_cuda_transform_tensor later:

https://github.com/ggerganov/llama.cpp/blob/bdfe4ba85c72df7fb521f7222a9dca017177d734/llama.cpp#L2346-L2378

This means the tensor is freed (indirectly through delete model) in a state where it's destined for the GPU, but hasn't actually been uploaded yet. ggml_cuda_free_data should have handled this case already, but it was overlooked, probably because it previously could only have been triggered by spurious std::exceptions.

There's nothing wrong with your PR, it works as intended.

[crasm]

I see. Thank you for catching that!

[crasm]

Fixed in #4462