Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reenable tokenizer test for LLaMa #3096

Merged
merged 12 commits into from
Sep 13, 2023
Merged

Reenable tokenizer test for LLaMa #3096

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
3d59f50
Fix für #2721
goerch Aug 22, 2023
84220df
Merge branch 'master' of https://github.com/goerch/llama.cpp
goerch Sep 4, 2023
9a953a4
Merge branch 'ggerganov:master' into master
goerch Sep 9, 2023
89a7277
Reenable tokenizer test for LLaMa
goerch Sep 9, 2023
52c9ecf
Add `console.cpp` dependency
goerch Sep 9, 2023
4ee2152
Fix dependency to `common`
goerch Sep 9, 2023
e903d5f
Fixing wrong fix.
goerch Sep 9, 2023
96533e0
Make console usage platform specific
goerch Sep 9, 2023
28b7494
Adapting makefile
goerch Sep 9, 2023
516a0d5
Remove trailing whitespace
goerch Sep 9, 2023
75a20d5
Adapting the other parts of the makefile
goerch Sep 9, 2023
16bf5f2
Fix typo.
goerch Sep 9, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Remove trailing whitespace
  • Loading branch information
@goerch
goerch committed Sep 9, 2023
commit 516a0d55092fa4bccc06cf3ce8552780dde014d3
2 changes: 1 addition & 1 deletion tests/test-tokenizer-0-llama.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -94,7 +94,7 @@ int main(int argc, char **argv) {
// We need this for unicode console support
console::init(false, false);
atexit([]() { console::cleanup(); });
#endif
#endif

bool success = true;

Expand Down
2 changes: 1 addition & 1 deletion tests/test-tokenizer-1-llama.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -78,7 +78,7 @@ int main(int argc, char **argv) {
// We need this for unicode console support
console::init(false, false);
atexit([]() { console::cleanup(); });
#endif
#endif

const int n_vocab = llama_n_vocab(ctx);

Expand Down