Improve handling of special tokens in GGML to GGUF converter#2725

Merged

KerfuffleV2 merged 3 commits intoggml-org:masterfrom KerfuffleV2:feat-improve-ggml-to-gguf-special-tokens

Aug 22, 2023

+31-14

Commits on Aug 22, 2023

Improve UNK, BOS, EOS token handling when converting without metadata.
KerfuffleV2
committed
Set default UNK token mapping from -1 to 0 in llama.cpp
KerfuffleV2
committed
Try to handle overflow due to buggy Windows Python with a better error message
KerfuffleV2
committed