Merge pull request #464 from PyThaiNLP/add-LST20-postag

Add LST20 Part-Of-Speech tagger model

Author: bact

docs/api/tag.rst

@@ -2,12 +2,12 @@
 
 pythainlp.tag
 =====================================
-The :class:`pythainlp.tag` contains functions that are used to tag different parts of a text including
-Part-of-Speech (POS) tags, and Named Entity Recognition (NER) tag.
+The :class:`pythainlp.tag` contains functions that are used to mark linguistic and other annotation to different parts of a text including
+part-of-speech (POS) tag and named entity (NE) tag.
 
-For the POS tags, there are two set of tags including `Universal Dependencies (UD) <https://universaldependencies.org/>`_ and ORCHID [#Sornlertlamvanich_2000]_ POS tags.
+For POS tags, there are three set of available tags: `Universal POS tags <https://universaldependencies.org/>`_, ORCHID POS tags [#Sornlertlamvanich_2000]_, and LST20 POS tags [#Prachya_2020]_.
 
-The following table shows the list of Part-of-Speech (POS) tags according to Universal Dependencies (UD) POS tags:
+The following table shows Universal POS tags as used in Universal Dependencies (UD):
 
 ============   ==========================   =============================
 Abbreviation   Part-of-Speech tag           Examples
@@ -29,7 +29,7 @@ Abbreviation   Part-of-Speech tag           Examples
  VERB          Verb                         เปิด, ให้, ใช้, เผชิญ, อ่าน
 ============   ==========================   =============================
 
-The following table shows the list of Part-of-Speech (POS) tags according to ORCHID POS tags from the paper:
+The following table shows POS tags as used in ORCHID:
 
 ============   =================================================      =================================
 Abbreviation   Part-of-Speech tag                                     Examples
@@ -93,7 +93,7 @@ Abbreviation   Part-of-Speech tag                                     Examples
 
 ORCHID corpus uses different set of POS tags. Thus, we make UD POS tags version for ORCHID corpus.
 
-The following table shows the mapping of Part-of-Speech (POS) tags from ORCHID POS tags to UD POS tags:
+The following table shows the mapping of POS tags from ORCHID to UD:
 
 ===============     =======================
 ORCHID POS tags     Coresponding UD POS tag
@@ -161,15 +161,54 @@ PUNCT               PUNCT
 PUNC                PUNCT
 ===============     =======================
 
-For the NER, we use `Inside-outside-beggining (IOB) <https://en.wikipedia.org/wiki/Inside%E2%80%93outside%E2%80%93beginning_(tagging)>`_ format to tag NER for each words.
-For instance, given a sentence "บารัค โอบามาเป็นประธานธิปดี", it would be tag the tokens "บารัค", "โอบามา", "เป็น", "ประธานาธิปดี" as "B-PERSON", "I-PERSON", "I-PERSON", "O", and "O" respectively.
+Details about LST20 POS tags are available in [#Prachya_2020]_.
 
-The *B-* prefix indicates begining token for a chunk of person name, "บารัค โอบามา" and *I-* prefix indicates the intermediate token. However, the term *O* indicates that a token not belong to any NER chunk.
+The following table shows the mapping of POS tags from LST20 to UD:
 
-The following table shows the list of Named Entity Recognition (NER) tags:
++----------------+-------------------------+
+| LST20 POS tags | Coresponding UD POS tag |
++================+=========================+
+| AJ             | ADJ                     |
++----------------+-------------------------+
+| AV             | ADV                     |
++----------------+-------------------------+
+| AX             | AUX                     |
++----------------+-------------------------+
+| CC             | CCONJ                   |
++----------------+-------------------------+
+| CL             | NOUN                    |
++----------------+-------------------------+
+| FX             | NOUN                    |
++----------------+-------------------------+
+| IJ             | INTJ                    |
++----------------+-------------------------+
+| NN             | NOUN                    |
++----------------+-------------------------+
+| NU             | NUM                     |
++----------------+-------------------------+
+| PA             | PART                    |
++----------------+-------------------------+
+| PR             | PROPN                   |
++----------------+-------------------------+
+| PS             | ADP                     |
++----------------+-------------------------+
+| PU             | PUNCT                   |
++----------------+-------------------------+
+| VV             | VERB                    |
++----------------+-------------------------+
+| XX             | X                       |
++----------------+-------------------------+
+
+For the NE, we use `Inside-outside-beggining (IOB) <https://en.wikipedia.org/wiki/Inside%E2%80%93outside%E2%80%93beginning_(tagging)>`_ format to tag NE for each word.
+
+*B-* prefix indicates the begining token of the chunk. *I-* prefix indicates the intermediate token within the chunk. *O* indicates that the token does not belong to any NE chunk.
+
+For instance, given a sentence "บารัค โอบามาเป็นประธานธิปดี", it would tag the tokens "บารัค", "โอบามา", "เป็น", "ประธานาธิปดี" with "B-PERSON", "I-PERSON", "O", and "O" respectively.
+
+The following table shows named entity (NE) tags as used PyThaiNLP:
 
 ============================    =================================
-Named Entity Recognition tag    Examples
+Named Entity tag                Examples
 ============================    =================================
  DATE                           2/21/2004, 16 ก.พ., จันทร์
  TIME                           16.30 น., 5 วัน, 1-3 ปี
@@ -214,3 +253,6 @@ References
 .. [#Sornlertlamvanich_2000] Takahashi, Naoto & Isahara, Hitoshi & Sornlertlamvanich, Virach. (2000).
             Building a Thai part-of-speech tagged corpus (ORCHID).
             Journal of the Acoustical Society of Japan (E). 20. 10.1250/ast.20.189.
+.. [#Prachya_2020] Prachya Boonkwan and Vorapon Luantangsrisuk and Sitthaa Phaholphinyo and Kanyanat Kriengket and Dhanon Leenoi and Charun Phrombut and Monthika Boriboon and Krit Kosawat and Thepchai Supnithi. (2020).
+            The Annotation Guideline of LST20 Corpus.
+            arXiv:2008.05055

pythainlp/corpus/corpus_license.md

@@ -22,7 +22,7 @@ thailand_provinces_th.txt | List of Thailand provinces in Thai
 tnc_freq.txt | Words and their frequencies, from Thai National Corpus
 ttc_freq.txt | Words and their frequencies, from Thai Textbook Corpus
 words_th.txt | List of Thai words
-words_th_thai2fit_201810.txt | List of Thai words (frozen)
+words_th_thai2fit_201810.txt | List of Thai words (frozen for thai2fit)
 
 The following word lists are from **Thai Male and Female Names Corpus**
 https://github.com/korkeatw/thai-names-corpus/ by Korkeat Wannapat
@@ -46,14 +46,14 @@ https://creativecommons.org/licenses/by/4.0/
 
 Filename | Description
 ---------|------------
-sentenceseg-crfcut-v2.model | Sentence segmentation model
-ud_thai_pud_pt_tagger.pkl | Part-of-speech model
-ud_thai_pud_unigram_tagger.json | Part-of-speech model
+sentenceseg_crfcut.model | Sentence segmentation model
+pos_ud_perceptron.pkl | Part-of-speech tagging model
+pos_ud_unigram.json | Part-of-speech tagging model
 
 
 ## Thai WordNet
 
-Thai WordNet (tha-wn.db) is created by Thai Computational Linguistic
+Thai WordNet (wordnet_th.db) is created by Thai Computational Linguistic
 Laboratory at National Institute of Information and Communications
 Technology (NICT), Japan, and released under the following license:
 

pythainlp/corpus/orchid_pos_th.json

@@ -79584,7 +79584,6 @@ Wishbone	1
 Rubik	1
 Petesch	1
 Consider	1
-assertiveperson	1
 bait	1
 บรอง	1
 Elsevier	1

pythainlp/corpus/orchid_pt_tagger.pkl …thainlp/corpus/pos_orchid_perceptron.pklpythainlp/corpus/orchid_pt_tagger.pkl renamed to pythainlp/corpus/pos_orchid_perceptron.pkl pythainlp/corpus/orchid_pt_tagger.pkl renamed to pythainlp/corpus/pos_orchid_perceptron.pkl

@@ -19409,7 +19409,6 @@
 โฮสต์	1
 โฮเต็ลๆ	1
 โฮ้งๆ	1
-โฮ้สฺเต้ส์	1
 ใจขุ่น	1
 ใจป้ำ	1
 ใจมือ	1