-
Notifications
You must be signed in to change notification settings - Fork 10.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
falcon arch fix for tied output embeddings #4978
falcon arch fix for tied output embeddings #4978
Conversation
One thing is strange though, not super simple to debug. That's despite the openbuddy variant being a bit smaller in total size. |
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
|
Thanks, I didn't think about that! Performance now is at 35tokens/sec ! |
Hm cool! Will wait for slaren to confirm this is fine before merging |
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
Does this mean that we can revive #3626? |
llama.cpp
Outdated
} else { | ||
model.output = ml.create_tensor(ctx_output_split, tn(LLM_TENSOR_TOKEN_EMBD, "weight"), {n_embd, n_vocab}); // needs to be on GPU | ||
ml.n_tensors++; // artificial tensor | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This should also work, but I would prefer decreasing ml.n_created
instead of increasing ml.n_tensors
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I was thinking that given we actually have one more tensor in use than in the model file it's better to increase that var. But we can change it to ml.n_created--;
Should I change it ? you can do it as well of course:)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Let's change it as suggested and merge then
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
changed and tested it
@cebtenzzre yep, this should be doable now. |
* falcon arch fix for tied output embeddings * Update llama.cpp Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> * Update llama.cpp * Update llama.cpp Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> * Update llama.cpp --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* falcon arch fix for tied output embeddings * Update llama.cpp Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> * Update llama.cpp * Update llama.cpp Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> * Update llama.cpp --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
The latest falcon finetune from Openbuddy is merging intput/output tensors, it was typically separate before as
lm_head
.Here it is: https://huggingface.co/OpenBuddy/openbuddy-falcon-40b-v16.1-4k
This PR uses input embeddings if no output ones are available.
Tested on wizard 40 (separate) and and openbuddy 40 (shared) and both work.