Change the default values for generation in GUI #840
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
the defaults that we have right now aren't ideal, i tried many models(nous-vicuna, snoozy, groovy....) with the values given here: https://www.reddit.com/r/LocalLLaMA/wiki/index/#wiki_prompting , i used the "Precise" preset, and all of the models gave much better results.
Amount of times the models got stuck in a loop writing the same thing over and over significantly decreased using the new values, that allowed to make them write more longer responses, and follow the conversation for longer, without going schitzo.
And i believe the newcomers will appreciate not having to figure out why the models are constantly getting stuck in loops whenever the conversation gets a little long.
Also changed the
batch size
from9
to128
, 9 is too conservative imo.