Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

llama : more tokenizer fixes #2810

Merged
merged 13 commits into from
Aug 27, 2023
Merged

llama : more tokenizer fixes #2810

Changes from 1 commit
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
5cad62b
tests : write a Python tokenizer test (wip)
ggerganov Aug 26, 2023
5d0ffb6
llama : prefix input text for tokenization with whitespace
ggerganov Aug 26, 2023
9668aa1
llama : distinguish pieces from decoded text + fix detokenization
ggerganov Aug 26, 2023
1e7a033
common : add comments
ggerganov Aug 26, 2023
dfa058e
examples : no longer manually add leading space when tokenizing
ggerganov Aug 26, 2023
70005bd
tests : use Python to generate tokenizer tests for C++
ggerganov Aug 26, 2023
e4324cb
tests : add option to tokenize text files
ggerganov Aug 26, 2023
eb8b326
tests : add test-tokenizer-1.py
ggerganov Aug 26, 2023
c767746
Merge branch 'master' into fix-tokenizer
ggerganov Aug 26, 2023
ab3ba64
llama.cpp : fix LF token
klosax Aug 26, 2023
dbcf470
hellaswag : move the concat space for clarity
ggerganov Aug 26, 2023
3bb0f84
tests : add falcon tests (py + cpp, currently do not pass Unicode)
ggerganov Aug 27, 2023
841983f
common : temporary separate llama_detokenize calls for SPM and BPE
ggerganov Aug 27, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
hellaswag : move the concat space for clarity
  • Loading branch information
@ggerganov
ggerganov committed Aug 26, 2023
commit dbcf470bc6606d67ff2916c6d3377741fad34090
4 changes: 2 additions & 2 deletions examples/perplexity/perplexity.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -392,7 +392,7 @@ void hellaswag_score(llama_context * ctx, const gpt_params & params) {
hs_data[i].context = prompt_lines[idx*6];
hs_data[i].gold_ending_idx = std::stoi( prompt_lines[idx*6+1] );
for (size_t j=0; j < 4; j++) {
hs_data[i].ending[j] = " " + prompt_lines[idx*6+2+j];
hs_data[i].ending[j] = prompt_lines[idx*6+2+j];
}

// Delete the selected random example from the prompt
Expand All @@ -417,7 +417,7 @@ void hellaswag_score(llama_context * ctx, const gpt_params & params) {
size_t context_size = context_embd.size();

for (int i = 0; i < 4; ++i) {
ending_tokens[i] = ::llama_tokenize(ctx, hs_data[task_idx].context + hs_data[task_idx].ending[i], add_bos);
ending_tokens[i] = ::llama_tokenize(ctx, hs_data[task_idx].context + " " + hs_data[task_idx].ending[i], add_bos);
for (int k = 0; k < int(context_size); ++k) {
if (ending_tokens[i][k] != context_embd[k]) {
fprintf(stderr, "Oops: ending %d of task %d differs from context at position %d\n",i,int(task_idx),k);
Expand Down