"newmm-safe" option -- fix newmm issue, take too long time for long text with lots of ambiguity breaking points

[bact]

This PR proposed newmm-safe tokenization engine, which is a newmm engine with additional mechanism to avoid possible exponentially long wait for long text with a lot of ambiguity in breaking points. Details are below.

Fix newmm issue with long ambiguous text

The problem

• The issue is discribed in pythainlp.word_tokenize ปัญหาตัดคำประโยคที่ยาวต่อเนื่องโดยไม่มี space [newmm] #241
• For example, if an input text is "หน้าด้านหน้าด้านหน้าด้านหน้าด้านหน้าด้าน", as "หน้าด้าน" and "ด้านหน้า" are both valid word (according to the dictionary), lots of word graphs representing different possibility of word breaks will be created.
• If the ambiguos text is very long, it will take so long time to finished the job, as the possibility will grow explonetially.

Proposed solution

• Preprocess the input text before the actual tokenization
• if the input text is longer than the limit (TEXT_LIMIT) with window size left and right), breaks them into smaller chunks (text_parts) then tokenizes each chunk
• when breaks the text, the breaking point should fall at valid word/syllable ends
  • if there's a space, use the right-most space as a breaking point
  • if there's no space, current implementation find candidate breaking points by feed a smaller sample text in a window (from TEXT_LIMIT - TEXT_SCAN_LEFT to TEXT_LIMIT + TEXT_SCAN_RIGHT) to newmm again - and choose to break after the longest token
• This preprocess can be enable/disable by safe_mode parameter.
  • If safe_mode is True, do the preprocess (will be slower)
  • If safe_mode is False (default), don't do the preprocess (may face long wait for edge cases)

Effects from the proprosed solution

• chunk size, window size, and complexity of finding valid breaking strategy can affect the speed and accurary
• current chunk size is 100 +- 20
  • too large chunk size = still long text, will not solve to issue --> affect speed
  • too small chunk size = more work to do --> affect speed, and maybe accuracy as well
  • too large window size = more work to do --> affect speed
  • too small window size = can break a single word into two words --> affect accuray
• need to check the performance with the pythainlp.benchmarks

Alternative solution

• The proposed preprocssing can be alternatively provided as a separated function that the user needs to call by themselves, if they want a safe tokenization.
• Another alternative, and more elegant, solution is maybe to limit nodes in word graph to a certian number. (Do it directly inside _bfs_paths_graph())

[pep8speaks]

Hello @bact! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

• In the file pythainlp/tokenize/__init__.py:

Line 356:80: E501 line too long (84 > 79 characters)

• In the file pythainlp/tokenize/newmm.py:

Line 114:80: E501 line too long (89 > 79 characters)

• In the file tests/test_tokenize.py:

Line 192:80: E501 line too long (138 > 79 characters)
Line 193:80: E501 line too long (225 > 79 characters)
Line 194:80: E501 line too long (83 > 79 characters)
Line 195:80: E501 line too long (82 > 79 characters)

Comment last updated at 2019-11-13 19:19:08 UTC

[coveralls]

Coverage increased (+0.2%) to 90.47% when pulling b2d72d1 on fix-newmm-longtext into fd5c44d on dev.

[p16i]

This condition appears many times in the code. I think we should consider refactor it.

[wannaphong]

@heytitle isinstance(text, str) only ?

[p16i]

I mean this if-condition appears many times.

[bact]

Results from the code in this notebook
https://colab.research.google.com/drive/15BYn_XOouej4M3CDL6ltO1j5uAq99Whq

• For artificial texts, crafted to replicated issue pythainlp.word_tokenize ปัญหาตัดคำประโยคที่ยาวต่อเนื่องโดยไม่มี space [newmm] #241, fix-newmm-longtext is faster and can survived all cases in a reasonable time.
  • These are text with edge cases (case that will produce complicated word graphs)
• For "natural" texts 4, 5, and 9, fix-newmm-longtext is slightly slower
  • These are texts without edge cases

Length of test texts (9 texts in total)

1. 16
2. 99
3. 99
4. 309 - (natural) text from Wikipedia
5. 405 - (natural) text from Wikipedia
6. 408 - (artificial) complicated text crafted to replicate issue pythainlp.word_tokenize ปัญหาตัดคำประโยคที่ยาวต่อเนื่องโดยไม่มี space [newmm] #241
7. 576 - (artificial) complicated text crafted to replicate issue pythainlp.word_tokenize ปัญหาตัดคำประโยคที่ยาวต่อเนื่องโดยไม่มี space [newmm] #241
8. 1,848 - (artificial) complicated text crafted to replicate issue pythainlp.word_tokenize ปัญหาตัดคำประโยคที่ยาวต่อเนื่องโดยไม่มี space [newmm] #241
9. 8,531 - (natural) text from Wikipedia

From dev branch

1. 10000 loops, best of 3: 31.9 µs per loop
2. 1000 loops, best of 3: 236 µs per loop
3. 1000 loops, best of 3: 265 µs per loop
4. 1000 loops, best of 3: 777 µs per loop *
5. 1000 loops, best of 3: 946 µs per loop *
6. 100 loops, best of 3: 2.58 ms per loop
7. (Cannot finished)
8. (Cannot finished)
9. 10 loops, best of 3: 26.3 ms per loop *

From fix-newmm-longtext branch

1. 10000 loops, best of 3: 31.5 µs per loop
2. 1000 loops, best of 3: 235 µs per loop
3. 1000 loops, best of 3: 270 µs per loop
4. 1000 loops, best of 3: 992 µs per loop
5. 1000 loops, best of 3: 1.14 ms per loop
6. 1000 loops, best of 3: 1.33 ms per loop *
7. 10 loops, best of 3: 75.2 ms per loop *
8. 1 loop, best of 3: 470 ms per loop *
9. 10 loops, best of 3: 30.6 ms per loop

[wannaphong]

From PyThaiNLP 2.0.7 (use marisa-trie)

1. 10000 loops, best of 3: 26.9 µs per loop
2. 1000 loops, best of 3: 232 µs per loop
3. 1000 loops, best of 3: 248 µs per loop
4. 1000 loops, best of 3: 813 µs per loop
5. 1000 loops, best of 3: 1.04 ms per loop
6. 100 loops, best of 3: 2.56 ms per loop
7. (Cannot finished)
8. (Cannot finished)
9. 10 loops, best of 3: 90.2 ms per loop

Colab : Colab

[wannaphong]

• check the performance with the pythainlp.benchmarks

[wannaphong]

Tokenisation Speed Benchmark

Notebook by @heytitle

• dev Notebook
• fix-newmm-longtext Notebook

[bact]

Tokenisation Speed Benchmark

	method	Char(200)	Char(500)	Char(1000)	Char(2000)
0	newmm (dev)	0.0007±0.0001	0.0017±0.0001	0.0029±0.0002	0.0059±0.0003
1	newmm (fix-newmm-longtext)	0.0008±0.0001	0.0017±0.0001	0.0039±0.0009	0.0069±0.0002

[wannaphong]

check the performance with the pythainlp.benchmarks

dev

Colab : Google colab

============== Benchmark Result ==============
               metric      mean±std       min    max
        char_level:tp   25.49±27.33  1.000000  240.0
        char_level:tn  91.16±104.23  0.000000  921.0
        char_level:fp     1.42±3.43  0.000000   46.0
        char_level:fn     5.21±7.13  0.000000   64.0
 char_level:precision     0.93±0.18  0.021277    1.0
    char_level:recall     0.86±0.10  0.142857    1.0
        char_level:f1     0.87±0.15  0.041667    1.0
 word_level:precision     0.75±0.23  0.000000    1.0
    word_level:recall     0.67±0.23  0.000000    1.0
        word_level:f1     0.70±0.23  0.000000    1.0

fix-newmm-longtext

Colab : Google colab

============== Benchmark Result ==============
               metric      mean±std       min    max
        char_level:tp   25.73±27.76  1.000000  242.0
        char_level:tn  90.14±102.41  0.000000  915.0
        char_level:fp     2.43±4.65  0.000000   49.0
        char_level:fn     4.98±6.69  0.000000   60.0
 char_level:precision     0.91±0.18  0.021277    1.0
    char_level:recall     0.87±0.10  0.142857    1.0
        char_level:f1     0.87±0.15  0.041667    1.0
 word_level:precision     0.73±0.23  0.000000    1.0
    word_level:recall     0.67±0.23  0.000000    1.0
        word_level:f1     0.69±0.23  0.000000    1.0

[bact]

Add safe_mode parameter, to enable/disable the preprocessing.

• If safe_mode is True, do the preprocess (will be slower)
• If safe_mode is False (default), don't do the preprocess (may face long wait for edge cases)

[wannaphong]

It's well. If this pull request is ready, You can merge.

[p16i]

The code looks good to me. However, I find that it does't have any test for the new option.

@bact it would be good if we could have minimal tests for this new option. One possible test might be using newmm-safe to tokenize that problematic string.

[p16i]

LGTM.