Skip to content

Commit 7227015

Browse files
CodemodService Botfacebook-github-bot
CodemodService Bot
authored andcommitted
Fix CQS signal. Id] 10393698 -- performance-faster-string-find in fbcode/pytorch/tokenizers/src
Reviewed By: dtolnay Differential Revision: D72950899
1 parent 4167468 commit 7227015

File tree

1 file changed

+1
-1
lines changed

1 file changed

+1
-1
lines changed

src/tiktoken.cpp

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -69,7 +69,7 @@ static Result<std::pair<std::string, uint64_t>> _parse(
6969
// Tiktoken format
7070
// https://github.com/openai/tiktoken/blob/main/tiktoken/load.py#L140 <base64
7171
// encoded token str> <rank>
72-
auto pos = line.find(" ");
72+
auto pos = line.find(' ');
7373
TK_CHECK_OR_RETURN_ERROR(
7474
pos != std::string::npos,
7575
ParseFailure,

0 commit comments

Comments
 (0)