How to fine tune it?

[n0r8]

I am a noob. Can you describe how I can fine-tune it with your program? It is possible? Maybe some articles.

[ggerganov]

Fine-tuning is not possible at the moment. You can fine-tune a model with some other implementation and then convert it and use it with ggml. For example, here are instructions how to fine-tune Whisper:

https://github.com/ggerganov/whisper.cpp/tree/master/models#fine-tuned-models

[loretoparisi]

Fine-tuning is not possible at the moment. You can fine-tune a model with some other implementation and then convert it and use it with ggml. For example, here are instructions how to fine-tune Whisper:

https://github.com/ggerganov/whisper.cpp/tree/master/models#fine-tuned-models

Hey Georgi, just curious, why fine-tuning is not possible, technically speaking? Let's ignore CUDA right now, but assumed it would work CPU only, what is missing? Thank you!

[ggerganov]

We need the backward pass for all tensor operations involved.
Currently, we have it implemented only for some of them

[n0r8]

Thx, after I asked question here I researched and understood that your software provides an interface not the tooling for fine tuning. Was noob back the time I asked this.

[xaedes]

Training directly with ggml would be really nice.
Implemented 8 out of 14 missing tensor ops.

xaedes/llama.cpp@757de70

I had to add another ggml operation GGML_OP_ADD_AT as counterpart for GGML_VIEW in the backward pass. This duplicated the code for add functions. Maybe the offset parameter can just be moved to the regular add functions which can then be used for ADD_AT. Was not sure about the performance of doing it this way, so I just duplicated the functions for now.

I will continue with the rest and test it with this repos test_grad and make a pull request when I think it is ready.

[xaedes]

Only GGML_ROPE is missing for llama, and GGML_OP_GET_ROWS, but this is only required for training the tokenizer embeddings. So far the gradients are untested, that will come next right after rope backward is implemented.

xaedes/llama.cpp@28de592

Unfortunately a bunch of new operations had to be added:

Add necessary ggml operations GGML_OP_ADD1, GGML_OP_SILU_BACK, GGML_OP_RMS_NORM_BACK, GGML_OP_DIAG_MASK_ZERO, and GGML_OP_ROPE_BACK

GGML_OP_ADD1 is necessary to add a scalar value in the backward pass of GGML_OP_SOFT_MAX
GGML_OP_ADD1 could also be replaced by using GGML_OP_ADD and GGML_OP_REPEAT, but the performance would be worse. additionally GGML_OP_REPEAT will return unexpected value when the the input to GGML_OP_SOFT_MAX contains only a single scalar. in this case GGML_OP_REPEAT will not return the value that should be repeated (src1) but the value which shape the result should take (src0). So in this case it can not replace GGML_OP_ADD1.

GGML_OP_SILU_BACK, GGML_OP_RMS_NORM_BACK and GGML_OP_ROPE_BACK are necessary for backward pass of GGML_OP_SILU, GGML_OP_RMS_NORM and GGML_OP_ROPE. The backward pass for these functions cannot be easily composed of existing operations. Since the backward pass builds a computation graph we need the operations' forward pass implementations of the required backward passes. Sounds a bit confusing at first, I know...

GGML_OP_DIAG_MASK_ZERO is necessary for backward pass of GGML_OP_DIAG_MASK_INF.

Some operations where previously inplace-only. for backward pass there needs to be non-inplace variants.
staying consistent with other operations that have non-inplace and inplace variants, the operations are changed to non-inplace and
functions with "_inplace" are added which are inplace.
in llama we need to call the inplace variants so that it is implemented as before.
for llama backward pass we need to use the non-inplace variants.

[ggerganov]

@xaedes
Wow! If you manage to pull this off and make training work it would be amazing!

[xaedes]

I successfully tested every backward pass except rms_norm.
Hm, and maybe I have to take another closer look at view_2d and view_3d --- view_1d works.
Also tested optimizing parameters A & B in Sum((A*B-C)**2), which I think is necessary for LoRa training.

When rms_norm also works I will be pushing it in a proper branch of my llama fork as I currently have python bindings in my working branch.
Unfortunately I can't easily work directly in the ggml repo, as windows support is not as good as in llama and I am on a windows machine here.

When necessary I can also rewrite that stuff to integrate with the mentioned refactoring, but I will make it work here first^^

[danforbes]

@xaedes have you looked into Windows Subsystem for Linux?

[xaedes]

@danforbes Yep, but for easier development with the setup I am used to I wanted it to work in other environment and didn't want to get lost on fiddling with platform stuff^^

[xaedes]

@ggerganov I have now successfully tested all backward passes necessary for llama. https://github.com/xaedes/llama.cpp/tree/training-integrate

List of all new operations that I had to add:

• GGML_OP_ADD1 : I think I could replace it with add(X,repeat(Y,X))
• GGML_OP_ADD_AT : Necessary for view backward pass. This adds src1 to view(src0) but returns tensor of shape src0. Maybe this operation could get another name like ACC_VIEW?
• GGML_OP_SUM_ROWS : Necessary for repeat backward pass: Reduces rows by summing them. shape[a,b,c,d] -> shape[1,b,c,d]
• GGML_OP_SILU_BACK : Necessary for silu backward pass
• GGML_OP_RMS_NORM_BACK : Could also be implemented using primitives, at the cost of performance.
• GGML_OP_GET_ROWS_BACK : Necessary for get_rows backward pass: Adds src0[i] rows to opt0[src1[i]] rows, returning a tensor of shape opt0. Maybe this operation could get a more meaningful name, something like ADD_ROWS_TO, or ACC_ROWS_TO?
• GGML_OP_DIAG : Necessary for softmax backward pass, alternative would have been to implement SOFTMAX_BACK directly, but DIAG is at least usable for other stuff. It turns rows into diagonal matrices.
• GGML_OP_DIAG_MASK_ZERO : Necessary for diag_mask_inf backward pass
• GGML_OP_ROPE_BACK : Necessary for rope backward pass.

Notable other changes:

• add inplace and non-inplace variants for scale, diag_mask_inf, soft_max and rope
• fix sub, mul and div functions to work correctly with transposed tensor, uses the same logic as in add:
• fix ggml_forward_add functions to work correctly with transposed tensors. uses the same logic as in ggml_compute_forward_add_q_f32, but make it consistent across all ggml_compute_forward_add_... functions. this also slightly changes the mem access pattern of the different threads to work as in ggml_compute_forward_add_q_f32. Maybe the mem pattern in the for loop for (int j = ith; j < n; j += nth) was important to keep? Each thread now has consecutive range of rows to process: for (int ir = ir0; ir < ir1; ++ir)
• de-duplicate ggml_forward_dup code taking care of contiguous tensors of same type. with this we can duplicate tensor of any type as long as they are contiguous. the function is used in dup, get_rows_back and diag_mask (when not inplace).
• there are some maybe too verbose comments including step-by-step derivation of gradients that could be cleaned away.

Next I will look into making an example for training a baby llama or a small LoRa finetune on some late layer. Could be that I find some still undiscovered issues during this^^

[ggerganov]

Maybe this operation could get another name like ACC_VIEW?

I guess GGML_OP_ACC implies we are accumulating into src0, so maybe go with that

Maybe this operation could get a more meaningful name, something like ADD_ROWS_TO, or ACC_ROWS_TO?

No strong preference

Maybe the mem pattern in the for loop for (int j = ith; j < n; j += nth) was important to keep?

Don't think it is important. At some point I was thinking that one of the methods is better than the other, but I think in the end they give pretty much the same performance.

Amazing work!
Do you know how to train models?
It will be super interesting to see what you do, especially if you can make it work with ggml.