You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Add an option to improve tokenization shuffling #141
Merged
soldni merged 5 commits intomainallenai/dolma:mainfrom soldni/tokenizer_better_shuffleallenai/dolma:soldni/tokenizer_better_shuffleCopy head branch name to clipboardMar 25, 2024
