Fix/broken numeric data format (#652)

[noppayut]

What does this changes

Fix incorrectly word_tokenize-d numeric data formats such as time, comma-separated numbers, decimal number, and ip address.

What was wrong

For some tokenizers (esp. neural nets), numbers separated with symbols (. , :) are tokenized into multiple parts. They should be atomic.

Example:

"12:34pm" -> ["12", ":", "34pm"]

How this fixes it

Postprocess the tokenized tokens with regular expression.
r"(\d+[\.\,:])+\d+": digits followed by either ".,:" once, possibly repeated, and ended with digits.

Fixes #652

Your checklist for this pull request

🚨Please review the guidelines for contributing to this repository.

• [O] Passed code styles and structures
• [O] Passed code linting checks and unit test

[coveralls]

Coverage increased (+0.1%) to 93.803% when pulling 7255e2c on noppayut:fix/broken-numeric-data-format into 39b814a on PyThaiNLP:dev.

[coveralls]

Coverage increased (+0.1%) to 93.791% when pulling c0e48e9 on noppayut:fix/broken-numeric-data-format into 39b814a on PyThaiNLP:dev.

[bact]

Nice and clean code. Thanks for the contribution.

Just asking for some ideas. What about putting it like this?

word_tokenize(
preprocessors = Callable[[List[str]], List[str]] = []
postprocessors = Callable[[List[str]], List[str]] = [
        join_broken_numeric_data_format
    ]
)

So the user can have some control over the behavior of the tokenizer.

[noppayut]

I think we should use booleans for common use-cases and pre/postprocessors for customized ones.

def word_tokenize(
    text, 
    custom_dict, 
    engine, 
    keep_whitespace=true, 
    join_broken_numeric_format=true,   # here
    preprocessors = Callable[[str], str] = []
    postprocessors = Callable[[List[str]], List[str]] = []
):
    # do something

preprocs = [normalize_gamer_style_input, remove_spaces]   # transform str -> str
postprocs = [add_prefix_postfix]   # transform List[str] -> List[str]
word_tokenize(
    "L ด็ ก ๆ กิ u ข้ า ว sึ ยั J", 
    join_broken_numeric_format=true, 
    preprocessors=preprocs,
    postprocessors=postprocs
)
# output: ["<sos>", "เด็ก", "ๆ", "กินข้าว", "รึ", "ยัง", "<eos>"]

[bact]

That's neat.

[noppayut]

Cool. Let me implement join_broken_numeric_format option for this PR, and send another one for the new word_tokenize API. ;)

[bact]

I changed the names of the parameter and the postprocessor, just to make them shorter.

Also revise the code example and slightly edit the description.

Looks good, @wannaphong I think we can merge.

[noppayut]

Thanks!

[sonarqubecloud]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
2 Code Smells

No Coverage information
0.0% Duplication

[wannaphong]

Awesome! Do you want to get hacktoberfest?

[noppayut]

Nope I'm good. Thanks 👍