Next steps for the finetuner

[WilliamTambellini]

Hi @JohannesGaessler
We have tested the finetuner and managed to finetune some very small LLMs (SLM?), congrats and tks.
Here are some ideas of improvements we would need and could work on:
1- add option to finetune using SGD
2- support bf16 finetuning
3- support finetuning only on the last n layers
Would you accept any PR on any of these ?
Best
WT
cf @ggerganov

[JohannesGaessler]

Yes, I would review and merge such PRs. Since you are already using the code, what are you doing for quality control?

[graehl]

I can answer - nothing but a basic manual QA check - we somewhat trust the finetune example because (after lowering the learning rate to 20% of what's in master) I see held-out data perplexity getting slightly better after 2 epochs over wikitext-2-raw/wiki.test.raw, and worse after 4 epochs, whereas the perplexity on that same train data steadily improves with each epoch (llama-3.2 1b fp32)

What I'm looking at is indeed any of the three alternatives above, with the idea being that we'd benefit from being able to somewhat finetune a 9b+ param LLM on a single 48gb GPU. The code uses optimization options for adamw which I understand is fairly memory optimal already - maybe no memory savings using SGD instead? Whereas #2 or #3 would obviously let us at least do something.

Does the -ngl option allow using some CPU memory to train (slowly) larger models on small amounts of data*epochs?

[graehl]

answering my question: -ngl less than max doesn't work in finetune yet - 'llama.cpp/ggml/src/ggml-backend.cpp:750: pre-allocated tensor (AdamW step for output_norm.weight) in a buffer (CUDA_Host) that cannot run the operation (OPT_STEP_ADAMW)'

[JohannesGaessler]

The code uses optimization options for adamw which I understand is fairly memory optimal already - maybe no memory savings using SGD instead?

The code allocates memory buffers for the AdamW momenta (ggml_opt_context::grad_m and ggml_opt_context::grad_v). These are being allocated unconditionally and for a 9b model you can estimate them to require 249=72 GiB with FP32 precision. That is the same amount as for the weights and gradient accumulators (if used). So SDG would cut the memory use for the model itself roughly in half, the memory use for the gradients can be cut down by reducing the batch size. Keep in mind that you will also need some memory for the KV cache.

3- support finetuning only on the last n layers

Just to be clear: there is a function llama_opt_params::param_filter which can be set by user code to determine which tensors should be set as trainable parameters. This should already work to enable training only on specific layers. If you were to contribute such a user function to the library it would be appreciated however.

[graehl]

Thanks, very helpful.

[WilliamTambellini]

#13873