Skip to content

Remove error words in tokenization dictionary #311

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
bact merged 1 commit into from
Oct 21, 2019
Merged

Remove error words in tokenization dictionary #311

bact merged 1 commit into from
Oct 21, 2019

Conversation

@bact
Copy link
Member

@bact bact commented Oct 20, 2019 •
edited
Loading

  • remove words with invalid character sequences, like double tonemarks, SARA E + SARA E
  • remove some words in the pattern ไม่+adj (that adj is already existed)
  • remove some words in the pattern อย่าง+adv (that adv is already existed)

@bact
- remove words with invalid character sequences
b0c7a86 
- remove some words in the pattern ไม่+adj
- remove some words in the pattern อย่าง+adv
@coveralls
Copy link

coveralls commented Oct 20, 2019

Coverage Status

Coverage remained the same at 90.265% when pulling b0c7a86 on update-dict3 into 89c21d3 on dev.

@bact bact merged commit 5e7f032 into dev Oct 21, 2019
@bact bact added the corpus corpus/dataset-related issues label Oct 21, 2019
@bact bact added this to the 2.1 milestone Oct 21, 2019
@bact bact deleted the update-dict3 branch October 25, 2019 19:26
@bact bact changed the title Update dictionary Remove error words in tokenization dictionary May 16, 2021
@bact bact mentioned this pull request May 16, 2021
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

corpus corpus/dataset-related issues

Projects

None yet

Milestone

2.1

Development

Successfully merging this pull request may close these issues.

3 participants

@bact @coveralls