[tokenizer] Return if exceed max token length #2957
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
zachgk
approved these changes
Jan 22, 2024
|
Codecov ReportAll modified and coverable lines are covered by tests ✅
❗ Your organization needs to install the Codecov GitHub app to enable full functionality. Additional details and impacted files@@ Coverage Diff @@
## master #2957 +/- ##
============================================
+ Coverage 72.08% 72.28% +0.19%
- Complexity 5126 7283 +2157
============================================
Files 473 722 +249
Lines 21970 32508 +10538
Branches 2351 3392 +1041
============================================
+ Hits 15838 23499 +7661
- Misses 4925 7387 +2462
- Partials 1207 1622 +415 ☔ View full report in Codecov by Sentry. |
ylwu-amzn
reviewed
Jan 22, 2024
| * | ||
| * @return {@code true} if tokens exceed max length | ||
| */ | ||
| public boolean exceedMaxLength() { |
There was a problem hiding this comment.
Will be even better if we can get how many tokens exceeded.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Brief description of what this PR is about