server: comment --threads option behavior

[phymbert]

As we are using batching, I am wondering what is the purpose of --threads N parameter in the server.

Should we remove it ?

[ggerganov]

It's used for bs = 1 decoding:

https://github.com/ggerganov/llama.cpp/blob/b2075fd6a578f5685060df4baa90ae9e48e98c70/llama.cpp#L9082

[phymbert]

OK, later I will add a this precision in the readme, also specifying this is not used at all when using --ngl.

[Jeximo]

As we are using batching

What about those who don't use batching?

[phymbert]

As we are using batching

What about those who don't use batching?

The server is using batching by design