Skip to content

Commit

Permalink
[blenderbot] regex fix (huggingface#8282)
Browse files Browse the repository at this point in the history
Fixing:

```
src/transformers/tokenization_blenderbot.py:163: DeprecationWarning: invalid escape sequence \s
    token = re.sub("\s{2,}", " ", token)
```
  • Loading branch information
stas00 authored and fabiocapsouza committed Nov 15, 2020
1 parent bad2d14 commit ebb9d93
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion src/transformers/tokenization_blenderbot.py
Original file line number Diff line number Diff line change
Expand Up @@ -160,7 +160,7 @@ def bpe(self, token: str) -> str:
return self.cache[token]
token = re.sub("([.,!?()])", r" \1", token)
token = re.sub("(')", r" \1 ", token)
token = re.sub("\s{2,}", " ", token)
token = re.sub(r"\s{2,}", " ", token)
if "\n" in token:
token = token.replace("\n", " __newln__")

Expand Down

0 comments on commit ebb9d93

Please sign in to comment.