Why is `llama_synchronize` called?

[EricLBuehler]

Hello all,

I was reading through the codebase and saw llama_synchronize was being called when the logits are retrieved:

https://github.com/ggerganov/llama.cpp/blob/cfc4d75df6399b36153ef739f2c1abee4c114bb8/ggml-cuda.cu#L2492

During my work on inference, I noticed that after the model runs, any synchronizing operation blocks for some time before it can be done. After I add an explicit synchronization, it obviously does not do that. However, this confuses me: why are the logits returned before the GPU is done "working"? What operations cause this? I would appreciate any help!

Edit: When I run a flamegraph, I get this:

It seems like avoiding the sync would be very beneficial!

[compilade]

This is something new since pipeline parallelism has been implemented (at least for CUDA) in #6017

why are the logits returned before the GPU is done "working"?

They are actually returned after, this is exactly what llama_synchronize is used for.

In llama_decode, the logits are copied asynchronously¹ to the output buffer in llama_decode, so that it can return before computing the outputs has finished.

https://github.com/ggerganov/llama.cpp/blob/0308f5e3d7bf9879f818b1a4ae589ff36b242af5/llama.cpp#L10030-L10037

When llama_get_logits_ith is called, it first calls llama_synchronize to ensure data has been copied,

https://github.com/ggerganov/llama.cpp/blob/0308f5e3d7bf9879f818b1a4ae589ff36b242af5/llama.cpp#L15175-L15176

then it extracts the specified logits.

https://github.com/ggerganov/llama.cpp/blob/0308f5e3d7bf9879f818b1a4ae589ff36b242af5/llama.cpp#L15195

What operations cause this?

Any operation which returns the content of the output buffer calls llama_synchronize before accessing the values. (e.g. llama_get_logits, llama_get_logits_ith, llama_get_embeddings, llama_get_embeddings_ith, and llama_get_embeddings_seq)

Note that llama_sampling_sample indirectly calls llama_get_logits_ith, which is what is shown in your flamegraph.

Footnotes

1. async copy falls back to synchronous copy when the backend doesn't support it. See https://github.com/ggerganov/llama.cpp/blob/0308f5e3d7bf9879f818b1a4ae589ff36b242af5/ggml-backend.c#L214-L215 ↩

[EricLBuehler]

Thanks for the detailed explanation, that makes sense. I was wondering, how does the computation graph allow async GPU (CUDA) operations? If you were to build a graph for the Llama architecture, wouldn't all parts need to be sequentially executed? I am sure this is wrong because llama.cpp would not implement it otherwise.

[slaren]

Async operations are queued into an asynchronous queue (in CUDA this is just a stream) and executed sequentially. The copy doesn't happen until the computation is completed.