Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add tokenizer test + revert to C++11 #355

Merged
merged 2 commits into from
Mar 21, 2023
Merged

Add tokenizer test + revert to C++11 #355

merged 2 commits into from
Mar 21, 2023

Conversation

ggerganov
Copy link
Owner

  • Add test-tokenizer-0 to do a few tokenizations - feel free to expand
  • Added option to convert-pth-to-ggml.py script to dump just the vocabulary
  • Added ./models/ggml-vocab.bin containing just LLaMA vocab data (used for tests)
  • Added utility to load vocabulary file from previous point (temporary implementation)
  • Avoid using std::string_view and drop back to C++11 (hope I didn't break something)
  • Rename gpt_vocab -> llama_vocab
  • All CMake binaries go into ./bin/ now

@ggerganov ggerganov merged commit eb34620 into master Mar 21, 2023
@ggerganov ggerganov deleted the tokenizer-test branch March 21, 2023 15:29
@Fabio3rs
Copy link
Contributor

What are the motivations for using C++11 over C++17?

It's possible to reduce the memory usage a little using std::string_view, for example taking my PR as a base (#305 ) should be possible to make id_to_token as a "source of the token strings" resizing the vector ( std::vector<token_score> id_to_token;) before taking pointers to it and declarating token_to_id as std::unordered_map<std::string_view, id> token_to_id;

Something like:

        vocab.id_to_token.resize(n_vocab);
/*...*/
            auto &tok_score = vocab.id_to_token[i];
            tok_score.tok = word;
            tok_score.score = score;

            vocab.token_to_id[tok_score.tok] = i;

AAbushady pushed a commit to AAbushady/llama.cpp that referenced this pull request Jan 27, 2024
* Hide unavailable backends & Add tooltip over backend count

Hides unavailable backends from the user and if the program is launched without any backends made, it shows an error message to them stating no backends were found and to make them using the 'make' command

Add tooltip when hovering over backend count label

hovering over the new label that shows the backend count will explain what the numbers are, and show the users which backends are not available or built

* add some code comments

* hide "missing" if all are built

move tooltip functions to helper functions section. hides the string "Missing: ..." from showing if all backends are available
" if len(runopts)==6 else + "

* small typo fix

* remove wrongly added leftover device choosing code

* fix labels

* move tooltip to function

* import vars logic fix

---------

Co-authored-by: Concedo <39025047+LostRuins@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants