Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add support for DeepSeek #73

Merged
merged 1 commit into from
Jul 24, 2024
Merged

Add support for DeepSeek #73

merged 1 commit into from
Jul 24, 2024

Conversation

Saibo-creator
Copy link
Collaborator

fix issue #72

  • Exclude special tokens in ByteTrie initialization.
  • Handle deepseek-ai/deepseek-coder as GPT2TokenizerMiddleMapping.
  • Add get_special_token_ids_to_excluded to TCFG_Tokenizer and extend in TCFG_LlamaTokenizer to support deepseek
  • Added unit tests for deepseek-ai/deepseek-coder-1.3b-base.

@Saibo-creator
Add support for DeepSeek
ccbffa4 
- Exclude special tokens in  initialization.
- Handle  as .
- Add  to  and extend in  to support deepseek
- Added unit tests for .
@Saibo-creator Saibo-creator merged commit de95d73 into epfl-dlab:main Jul 24, 2024
@Saibo-creator Saibo-creator mentioned this pull request Jul 24, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@Saibo-creator