'email' tokenizing as 'em, ail, and '

[saxtell-cb]

In the following sentence (from Twitter), 'email' is being tokenized as 'em, ail, and '. This is obviously incorrect. What can be done to stop this split?

• It's official (according to the AP) it's 'email' not 'e-mail' and 'website' not 'web-site'!

I have the following parameters set:
tokenize.language: English
tokenize.whitespace: false (because we want tokens like it's to separate into it and 's)
tokenize.keepeol: false
tokenize.verbose: false
tokenize.options: invertible=true,splitAssimilations=false,splitHyphenated=false,splitForwardSlash=true,untokenizable=allKeep,strictTreebank3=true,normalizeSpace=false,ellipses=original

[AngledLuffa]

This is an artifact of tokenizing phrases such as screw 'em We'll take a look

…

On Thu, Nov 17, 2022, 9:54 AM saxtell-cb ***@***.***> wrote: In the following sentence (from Twitter), 'email' is being tokenized as 'em, ail, and '. This is obviously incorrect. What can be done to stop this split? - It's official (according to the AP) it's 'email' not 'e-mail' and 'website' not 'web-site'! I have the following parameters set: tokenize.language: English tokenize.whitespace: false (because we want tokens like it's to separate into it and 's) tokenize.keepeol: false tokenize.verbose: false tokenize.options: invertible=true,splitAssimilations=false,splitHyphenated=false,splitForwardSlash=true,untokenizable=allKeep,strictTreebank3=true,normalizeSpace=false,ellipses=original — Reply to this email directly, view it on GitHub <#1316>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA2AYWPJYS6JMP56WJKTDXDWIZWNTANCNFSM6AAAAAASDUTWZA> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[saxtell-cb]

Thanks for taking a look @AngledLuffa!

[AngledLuffa]

I gotta run this by my PI, who knows the tokenizer much better than I do, but I think this PR should address the situation

[AngledLuffa]

@manning

[saxtell-cb]

Thanks @AngledLuffa! I appreciate the help on this.

[manning]

LGTM. Fine to merge.

(I added to your PR a documentation fix to the previous line cleaning up the "C'mon" tokenization change.)

[AngledLuffa]

This is now released in 4.5.2