Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Full DynaTemp implementation + UI #600

Merged
merged 21 commits into from
Jan 6, 2024
Merged

Full DynaTemp implementation + UI #600

Show file tree
Hide file tree
Changes from 20 commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
2b476d4
move Dynatemp changes to new branch
kalomaze Dec 31, 2023
c4b4781
fix float header
kalomaze Jan 1, 2024
b983ae0
Properly reintroduce variable expert count
kalomaze Jan 3, 2024
b8551b4
first pass at DynaTemp UI
AAbushady Jan 4, 2024
1d0dec4
DynaTemp UI Checkbox
AAbushady Jan 4, 2024
8cf96ff
DynaTemp UI checkbox edition
AAbushady Jan 4, 2024
7f11b7e
DynaTemp UI integration
kalomaze Jan 4, 2024
303a32e
Remove greedy dynatemp
kalomaze Jan 4, 2024
77e0f6c
Fix race condition caused by debug print
kalomaze Jan 4, 2024
69e293c
Fixed broken presets and miro
AAbushady Jan 4, 2024
ce2e738
Merge pull request #11 from AAbushady/dynatemp-mainline
kalomaze Jan 4, 2024
c2d14ab
Remove debug function + HHI temp
kalomaze Jan 4, 2024
b753bb4
Merge branch 'dynatemp-pr-upstream' into dynatemp-mainline
kalomaze Jan 4, 2024
351262a
Fix whitespace (?) for generate function
kalomaze Jan 4, 2024
1470209
epic upstream renaming scheme fix
kalomaze Jan 4, 2024
597f80d
fix stupid indents
kalomaze Jan 4, 2024
a4f8ff4
Other cleanup
kalomaze Jan 4, 2024
1d9c9b5
Slight indent fix
kalomaze Jan 4, 2024
f61a441
revert batch pyinstaller maker to mainline
kalomaze Jan 4, 2024
79aeadf
Merge Dynamic Temp UI + cleanups into dynatemp-pr-upstream
kalomaze Jan 4, 2024
41d04bc
compact dynatemp into a single value dynatemp_range. This is a float …
LostRuins Jan 5, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions common/common.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -80,6 +80,12 @@ struct gpt_params {
int32_t mirostat = 0; // 0 = disabled, 1 = mirostat, 2 = mirostat 2.0
float mirostat_tau = 5.00f; // target entropy
float mirostat_eta = 0.10f; // learning rate

// DynaTemp!
bool dynatemp = false; // enable DynaTemp
float min_temp = 0.00f; // minimum temperature
float max_temp = 2.00f; // maximum temperature

// // sampling parameters
struct llama_sampling_params sparams;

Expand Down
3 changes: 3 additions & 0 deletions common/sampling.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,9 @@ typedef struct llama_sampling_params {
int32_t mirostat = 0; // 0 = disabled, 1 = mirostat, 2 = mirostat 2.0
float mirostat_tau = 5.00f; // target entropy
float mirostat_eta = 0.10f; // learning rate
bool dynatemp = false; // dynamic temperature
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So the first thing that stands out first is - I don't think a bool for toggling dynatemp is necessary. I think this can be simplified and there are 2 very good ways of doing it.

First way is to make it part of the sampler, so having dynatemp_min and dynatemp_max both be 0 would leave dynatemp inactive, otherwise it would be active and applied (overriding temperature). That puts it in line with all other samplers that have an "inactive" value and an "active" value.

The second option which I am strongly in support is a single value dynatemp_range. This is a float which represents the allowed deviation from the min and max temperature when using dynatemp.

Thus, if we want a value of dynatemp_min=0.3, dynatemp_max=0.5, then we would simply set temperature=0.4 and dynatemp_range=0.1

If you want a dynatemp_min=0.8, dynatemp_max=2.0, then you set temperature=1.4 and dynatemp_range=0.6.

It will work for any value of min and max

To disable, set dynatemp_range=0

Thoughts?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like this idea and it's nicely intuitive.

float min_temp = 0.00f; // minimum temperature
float max_temp = 2.00f; // maximum temperature
bool penalize_nl = true; // consider newlines as a repeatable token
std::string samplers_sequence = "kfypmt"; // top_k, tail_free, typical_p, top_p, min_p, temp

Expand Down
4 changes: 4 additions & 0 deletions expose.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -81,7 +81,11 @@ struct generation_inputs
const char * grammar;
const bool grammar_retain_state;
const bool quiet = false;
const bool dynatemp = false;
const float min_temp;
const float max_temp;
const logit_bias logit_biases[logit_bias_max];

};
struct generation_outputs
{
Expand Down
21 changes: 18 additions & 3 deletions gpttype_adapter.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -481,7 +481,7 @@ void sample_grammar(FileFormat file_format, int32_t n_vocab, llama_token_data_ar
}

int SampleLogits(const float * logits, int n_ctx, int n_vocab, int rep_pen_range, float rep_pen, float presence_penalty, float top_k, float top_a, float top_p, float min_p, float typical_p, float tfs, float temp, std::mt19937 & rng,
int mirostat, float mirostat_tau, float mirostat_eta, const std::vector<samplers> & sampler_order, llama_grammar * grammar)
int mirostat, float mirostat_tau, float mirostat_eta, const std::vector<samplers> & sampler_order, llama_grammar * grammar, bool dynatemp, float min_temp, float max_temp)
{
int id = 0;
std::vector<llama_token_data> candidates;
Expand Down Expand Up @@ -540,7 +540,14 @@ int mirostat, float mirostat_tau, float mirostat_eta, const std::vector<samplers
llama_sample_typical(nullptr, &candidates_p, typical_p,1);
break;
case KCPP_SAMPLER_TEMP:
sample_temperature(&candidates_p, temp);
if (dynatemp)
{
llama_sample_entropy(nullptr, &candidates_p, temp, min_temp, max_temp);
}
else
{
sample_temperature(&candidates_p, temp);
}
break;
case KCPP_SAMPLER_REP_PEN:
sample_rep_pen(n_ctx, rep_pen_range, rep_pen, presence_penalty, &candidates_p);
Expand Down Expand Up @@ -1479,6 +1486,7 @@ generation_outputs gpttype_generate(const generation_inputs inputs, generation_o
}

std::string addedmemory = inputs.memory;

kcpp_params->prompt = inputs.prompt;
kcpp_params->seed = inputs.seed;
kcpp_params->n_predict = inputs.max_length;
Expand All @@ -1494,10 +1502,14 @@ generation_outputs gpttype_generate(const generation_inputs inputs, generation_o
kcpp_params->mirostat = inputs.mirostat;
kcpp_params->mirostat_eta = inputs.mirostat_eta;
kcpp_params->mirostat_tau = inputs.mirostat_tau;
kcpp_params->dynatemp = inputs.dynatemp;
kcpp_params->min_temp = inputs.min_temp;
kcpp_params->max_temp = inputs.max_temp;
kcpp_params->n_ctx = inputs.max_context_length;
kcpp_params->n_batch = n_batch;
kcpp_params->n_threads = n_threads;
kcpp_params->n_threads_batch = n_blasthreads;

bool stream_sse = inputs.stream_sse;

bool allow_regular_prints = (debugmode!=-1 && !inputs.quiet) || debugmode >= 1;
Expand Down Expand Up @@ -1888,6 +1900,9 @@ generation_outputs gpttype_generate(const generation_inputs inputs, generation_o
const float presence_penalty = kcpp_params->presence_penalty;
const float typical_p = kcpp_params->typical_p;
const float tfs_z = kcpp_params->tfs_z;
const float dynatemp = kcpp_params->dynatemp;
const float min_temp = kcpp_params->min_temp;
const float max_temp = kcpp_params->max_temp;

if (!startedsampling)
{
Expand Down Expand Up @@ -1943,7 +1958,7 @@ generation_outputs gpttype_generate(const generation_inputs inputs, generation_o

id = SampleLogits(logitsPtr, nctx, n_vocab, last_n_size, repeat_penalty, presence_penalty,
top_k, top_a, top_p, min_p, typical_p, tfs_z, temp, rng,
kcpp_params->mirostat, kcpp_params->mirostat_tau, kcpp_params->mirostat_eta, sampler_order, grammar);
kcpp_params->mirostat, kcpp_params->mirostat_tau, kcpp_params->mirostat_eta, sampler_order, grammar, dynatemp, min_temp, max_temp);
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@LostRuins Not related to this PR but it feels like kcpp_params should just be passed in instead.

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good point, will probably refactor it soon.


if (grammar != nullptr) {
grammar_accept_token(file_format, n_vocab, grammar, id);
Expand Down
17 changes: 16 additions & 1 deletion kcpp_docs.embd
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -139,6 +139,21 @@
"description": "If true, prevents the EOS token from being generated (Ban EOS). For unbantokens, set this to false.",
"type": "boolean"
},
"dynatemp": {
"default": false,
"description": "If true, uses dynamic temperature. If false, uses static temperature.",
"type": "boolean"
},
"min_temp": {
"description": "Dynatemp Minimum temperature value.",
"exclusiveMinimum": 0,
"type": "number"
},
"max_temp": {
"description": "Maximum temperature value.",
"exclusiveMinimum": 0,
"type": "number"
},
"mirostat": {
"description": "KoboldCpp ONLY. Sets the mirostat mode, 0=disabled, 1=mirostat_v1, 2=mirostat_v2",
"minimum": 0,
Expand Down Expand Up @@ -876,4 +891,4 @@

</body>

</html>
</html>
Loading