fast test and benchmark of new 1.67 versions vs 1.63

[nnyanpasu]

rx7900xtx, only vram

[nnyanpasu]

img gen works well, x4 experts loaded too

[brknsoul]

Synopsis: Vulkan from the main fork still seems to be faster.

Benchmark
openhermes-2.5-mistral-7b.Q4_K_M
RX 6600 XT, 8GB

[nnyanpasu]

what arguments do you use so that the bench saves the results of generating the token/s, etc.? by default, this is not saved in my file.

[brknsoul]

I just copypasted the results to a text file and asked Gemini to turn it into a spreadsheet.

[YellowRoseCx]

Synopsis: Vulkan from the main fork still seems to be faster.

Benchmark openhermes-2.5-mistral-7b.Q4_K_M RX 6600 XT, 8GB

when you do comparisons like that, make sure you show the amount of tokens processed and the amount of tokens generated

[brknsoul]

It's the standard --benchmark

[nnyanpasu]

vulkan test

[dya3506]

on rocm4allV3, 6600xt with 8gb vram
offloading 49 layers to gpu works well on rocm but would get errors on vulkan, so I reduced to 40 layers

[cloak1505]

koboldcpp's built-in benchmark was a source of my stress. It makes ROCm look like complete ass.

When I'm actually using it, ROCm prompt processing is slower while generating faster. But for some reason it generates slower in the benchmark as well... What does the benchmark do??

I can fit 16k context on 8B IQ4_XS without dropping BLAS batch size.

I can see that depending on the use case, ROCm is worse. The benchmark is weirding me out though.