Commit aecdd66
committed
Fix inconsistency in newmm-safe engine by copilot
Related to #755
Update the calculation of `cut_pos` in `newmm-safe` engine to ensure consistent tokenization results.
* Modify `pythainlp/tokenize/newmm.py` to update the calculation of `cut_pos` at line 193 to `cut_pos = space_idx + 1 + _TEXT_SCAN_BEGIN`.1 parent 9a9d11f commit aecdd66
1 file changed
+1
-1
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
182 | 182 | | |
183 | 183 | | |
184 | 184 | | |
185 | | - | |
| 185 | + | |
186 | 186 | | |
187 | 187 | | |
188 | 188 | | |
| |||
0 commit comments