fix: crash on bge-m3 embedding model #8883
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: thxCode <thxcode0824@gmail.com>
when vocab.type is SPM, we will confirm the linefeed_id by searching the char, and use special_pad_id instead if not found. the special_*_id are usually record in metadata, to ensure the special_pad_id can be used correctly, we need to obtain it from metadata first and then perform the above confirmation logic. Signed-off-by: thxCode <thxcode0824@gmail.com>
Signed-off-by: thxCode <thxcode0824@gmail.com>
|
Btw it's still not faster than transformers so why use it ? |
ggerganov
reviewed
Aug 6, 2024
Comment on lines
+282
to
+288
| llama_vocab::id token_id; | ||
| try { | ||
| token_id = llama_byte_to_token_impl(vocab, symbol.text[j]); | ||
| } catch(const std::exception & e) { | ||
| // not found, use UNK token instead. | ||
| token_id = vocab.special_unk_id; | ||
| } |
There was a problem hiding this comment.
I'm unsure about this change - if this happened, wouldn't it imply a problem with the model / tokenizer? Seems better to find and fix the root of the problem instead of hiding it
There was a problem hiding this comment.
get it. this fix is inspired by https://github.com/ggerganov/llama.cpp/blame/1e6f6554aa11fa10160a5fda689e736c3c34169f/src/llama.cpp#L5560-L5565, maybe my understanding is not correct.
There was a problem hiding this comment.
@ggerganov should I close this PR if the last commit is not a reasonable change? thanks.
mofosyne
added
the
Review Complexity : Low
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
crash on testing llama-server embedding with bge-m3.
n_ubatchvalue, print error on insufficientn_batchvalue #6296 did.linefeed_id, and usesspecial_pad_idinstead if not found. the second commit is to respect thespecial_pad_idof metadata.libc++abi: terminating due to uncaught exception of type std::out_of_range: unordered_map::at: key not found. usespecial_unk_idif not found.I am unsure if there are other corner cases, please let me know.