Slightly faster imatrix

[ikawrakow]

The change in PR #5035 applied to imatrix.

Also added --no-ppl command line option to skip computing perplexity altogether.

At the end, it is a minor optimization: On my RTX-4080 time for 100 chunks using Mistral-7B decreases from 47 seconds to 40 seconds.

In comparison, a perplexity calculation for 100 chunks takes 15 seconds. I.e., now most of the time is being spent in running the single-threaded code that collects the importance matrix data on the CPU. When I first added the imatrix tool it only worked on the CPU, so the time for collecting the imatrix was very small compared to the time taken for evaluating the compute graph. I also did not anticipate that people will be using the tools to run imatrix calculations for millions of tokens. So, I guess, it would make sense to parallelize the data gathering as well, but I'm leaving this for a separate PR.

[Nexesenex]

Just an idea, @ikawrakow :
Considering that we are all exploring iMatrix still to find the best compromise, and that you already implemented an autosave every 10 chunks calculated, is it possible to have a backup of the matrix currently calculated in a separate file every given number of chunks crunched via a parameter in command line, with the number of chunks crunched as a suffix (_500, _1000, etc)?
So we get a file every 50, 100, or whatever chunks until the specified amount to reach is reached?

[ikawrakow]

@Nexesenex Great idea! See PR #5077

[bartowski1182]

Hey @ikawrakow Sorry to resurrect an old PR, but I wanted to make sure I understand something

What is the result of adding --no-ppl for imatrix calculation, is it just avoiding outputting the results or does it affect the quality?

it LOOKS largely cosmetic, which is why it surprised me that my mixtral 22x8b imatrix calculation went from 6 hours to 1.5 hours (only 7 layers offloaded)

Just want to make sure i'm not doing something terrible and silly by adding it!