Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update readme with suggestion on number of threads to use #325

Merged

Conversation

HarryGifford
Copy link
Contributor

Update the readme to make comment on performance with hyperthreading and link to pytorch docs which have relevant information.

For example, testing on my laptop (AMD 6850U 8 cores) I saw a 3x speedup running stories42M.bin by going from 16 threads (default) to 8 threads:

(mypyenv) hegiarch% ./run ./stories42M.bin -s 42                  
Once upon a time, there was a little girl named Lily. She loved to play in the park with her friends. One day, she found a shiny green gem on the ground. She picked it up and showed it to her friend, Max.
Max said, "Wow, that's a cool gem! Can I see it?"
Lily replied, "Sure, but be careful not to break it."
Max looked at the gem and said, "I wish I could find a gem like that in real life."
Lily said, "I know, but I don't want to lose it. I value it very much."
Max agreed and they continued to play in the park. But Lily couldn't stop thinking about the green gem. She decided to keep it safe in her pocket. She knew it was special and she wanted to cherish it forever.
achieved tok/s: 48.049600
(mypyenv) hegiarch% OMP_NUM_THREADS=8 ./run ./stories42M.bin -s 42
Once upon a time, there was a little girl named Lily. She loved to play in the park with her friends. One day, she found a shiny green gem on the ground. She picked it up and showed it to her friend, Max.
Max said, "Wow, that's a cool gem! Can I see it?"
Lily replied, "Sure, but be careful not to break it."
Max looked at the gem and said, "I wish I could find a gem like that in real life."
Lily said, "I know, but I don't want to lose it. I value it very much."
Max agreed and they continued to play in the park. But Lily couldn't stop thinking about the green gem. She decided to keep it safe in her pocket. She knew it was special and she wanted to cherish it forever.
achieved tok/s: 186.746988

Update the documentation to make suggestions on the number of threads. The performance difference can be very large. Also linked to the PyTorch docs which are relevant here.
@karpathy karpathy merged commit 74a68ee into karpathy:master Aug 21, 2023
@HarryGifford HarryGifford deleted the users/hegi/update-readme-threading branch August 21, 2023 21:28
vinhtran2611 pushed a commit to vinhtran2611/llama2.c that referenced this pull request Jan 20, 2024
…eadme-threading

Update readme with suggestion on number of threads to use
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants