Update 2.2 branche from dev branche

.github/workflows/pythainlp-test.yml

@@ -23,6 +23,7 @@ jobs:
       run: |
         python -m pip install --upgrade pip pytest wheel flake8
         if [ -f requirements.txt ]; then pip install -r requirements.txt; fi
+        pip install torch==1.5.0 torchvision==0.6.0 -f https://download.pytorch.org/whl/torch_stable.html
         pip install .[full]
         pip install deepcut coverage coveralls
     - name: Lint with flake8
@@ -34,4 +35,4 @@ jobs:
     - name: Test
       run: |
         coverage run -m unittest discover
-        CI_BRANCH=${GITHUB_REF#"ref/heads"} COVERALLS_REPO_TOKEN=${{ secrets.COVERALLS_REPO_TOKEN }} coveralls
+        CI_BRANCH=${GITHUB_REF#"ref/heads"} COVERALLS_REPO_TOKEN=${{ secrets.COVERALLS_REPO_TOKEN }} coveralls

README.md

@@ -1,4 +1,3 @@
-
 <div align="center">
   <img src="https://avatars0.githubusercontent.com/u/32934255?s=200&v=4"/>
   <h1>PyThaiNLP: Thai Natural Language Processing in Python</h1>
@@ -24,11 +23,12 @@ PyThaiNLP เป็นไลบารีภาษาไพทอนสำหร
 
 | Version | Description | Status |
 |:------:|:--:|:------:|
-| [2.2.3](https://github.com/PyThaiNLP/pythainlp/releases) | Stable | [Change Log](https://github.com/PyThaiNLP/pythainlp/issues/330) |
+| [2.2.4](https://github.com/PyThaiNLP/pythainlp/releases) | Stable | [Change Log](https://github.com/PyThaiNLP/pythainlp/issues/330) |
 | [`dev`](https://github.com/PyThaiNLP/pythainlp/tree/dev) | Release Candidate for 2.3  | [Change Log](https://github.com/PyThaiNLP/pythainlp/issues/445) |
 
 Please follow our [PyThaiNLP Facebook page](https://www.facebook.com/pythainlp/) for more updates.
 
+
 ## Getting Started with PyThaiNLP
 
 We provide [PyThaiNLP Get Started Tutorial](https://www.thainlp.org/pythainlp/tutorials/notebooks/pythainlp_get_started.html) for exploring features in PyThaiNLP; We also have tutorials for specific tasks. Please visit [our tutorial page](https://www.thainlp.org/pythainlp/tutorials).
@@ -37,27 +37,29 @@ Latest document is available at [https://thainlp.org/pythainlp/docs/2.2/](https:
 
 We try to make the package easy to use as much as possible; therefore, some additional data (like word lists and language models) may get automatically download during runtime. PyThaiNLP caches additional data under the directory `~/pythainlp-data` by default, but the user can change the value by specifying the environment variable `PYTHAINLP_DATA_DIR`. See corpus catalog at [PyThaiNLP/pythainlp-corpus](https://github.com/PyThaiNLP/pythainlp-corpus).
 
+
 ## Capabilities
 
-PyThaiNLP provides standard NLP functions for Thai, for example part-of-speec tagging, linguistic unit segmentation (syllable, word, or sentence). Some of these functions are also available via command-line interface.
+PyThaiNLP provides standard NLP functions for Thai, for example part-of-speech tagging, linguistic unit segmentation (syllable, word, or sentence). Some of these functions are also available via command-line interface.
 
 <details>
   <summary>List of Features</summary>
 
 - Convenient character and word classes, like Thai consonants (`pythainlp.thai_consonants`), vowels (`pythainlp.thai_vowels`), digits (`pythainlp.thai_digits`), and stop words (`pythainlp.corpus.thai_stopwords`) -- comparable to constants like `string.letters`, `string.digits`, and `string.punctuation`
 - Thai linguistic unit segmentation/tokenization, including sentence (`sent_tokenize`), word (`word_tokenize`), and subword segmentations based on Thai Character Cluster (`subword_tokenize`)
-- Thai part-of-speech taggers (`pos_tag`)
+- Thai part-of-speech tagging (`pos_tag`)
 - Thai spelling suggestion and correction (`spell` and `correct`)
 - Thai transliteration (`transliterate`)
 - Thai soundex (`soundex`) with three engines (`lk82`, `udom83`, `metasound`)
-- Thai collation (sort by dictionoary order) (`collate`)
+- Thai collation (sort by dictionary order) (`collate`)
 - Read out number to Thai words (`bahttext`, `num_to_thaiword`)
 - Thai datetime formatting (`thai_strftime`)
 - Thai-English keyboard misswitched fix (`eng_to_thai`, `thai_to_eng`)
 - Command-line interface for basic functions, like tokenization and pos tagging (run `thainlp` in your shell)
 </details>
 
-Please see [our tutorials](https://www.thainlp.org/pythainlp/tutorials) on how to apply these functions to ML problems.
+Please see [our tutorials](https://www.thainlp.org/pythainlp/tutorials) on how to apply these functions to machine-learning problems.
+
 
 ## Installation
 
@@ -66,7 +68,7 @@ pip install --upgrade pythainlp
 ```
 
 This will install the latest stable release of PyThaiNLP.
-PyThaiNLP uses pip as its package manger and PyPI as its main distribution channel, see [https://pypi.org/project/pythainlp/](https://pypi.org/project/pythainlp/)
+PyThaiNLP uses pip as its package manager and PyPI as its main distribution channel, see [https://pypi.org/project/pythainlp/](https://pypi.org/project/pythainlp/)
 
 Install different releases:
 
@@ -99,9 +101,9 @@ pip install pythainlp[extra1,extra2,...]
 For dependency details, look at `extras` variable in [`setup.py`](https://github.com/PyThaiNLP/pythainlp/blob/dev/setup.py).
 
 
-## Command-line
+## Command-Line Interface
 
-Some of PyThaiNLP functionalities can be used at command line, using `thainlp`
+Some of PyThaiNLP functionalities can be used at command line, using `thainlp` command.
 
 For example, displaying a catalog of datasets:
 ```sh
@@ -121,6 +123,7 @@ thainlp help
   - [Upgrade ThaiNER from 1.7](https://github.com/PyThaiNLP/pythainlp/wiki/Upgrade-ThaiNER-from-PyThaiNLP-1.7-to-PyThaiNLP-2.0)
 - Python 2.7 users can use PyThaiNLP 1.6
 
+
 ## Citations
 
 If you use `PyThaiNLP` in your project or publication, please cite the library as follows
@@ -148,6 +151,7 @@ or BibTeX entry:
 - Please do fork and create a pull request :)
 - For style guide and other information, including references to algorithms we use, please refer to our [contributing](https://github.com/PyThaiNLP/pythainlp/blob/dev/CONTRIBUTING.md) page.
 
+
 ## Licenses
 
 | | License |
@@ -157,6 +161,7 @@ or BibTeX entry:
 | Language models created by PyThaiNLP | [Creative Commons Attribution 4.0 International Public License (CC-by)](https://creativecommons.org/licenses/by/4.0/)  |
 | Other corpora and models that may included with PyThaiNLP | See [Corpus License](https://github.com/PyThaiNLP/pythainlp/blob/dev/pythainlp/corpus/corpus_license.md) |
 
+
 ## Sponsors
 
 [![VISTEC-depa Thailand Artificial Intelligence Research Institute](https://airesearch.in.th/assets/img/logo/airesearch-logo.svg)](https://airesearch.in.th/)
@@ -168,3 +173,13 @@ Since 2019, our contributors Korakot Chaovavanich and Lalita Lowphansirikul have
 <div align="center">
   Made with ❤️ | PyThaiNLP Team 💻 |  "We build Thai NLP" 🇹🇭
 </div>
+
+------
+
+<div align="center">
+  <strong>We have only one official repository at https://github.com/PyThaiNLP/pythainlp and another mirror at https://gitlab.com/pythainlp/pythainlp </strong>
+</div>
+
+<div align="center">
+  <strong>Beware of malware if you use code from mirrors other than the official two at GitHub and GitLab.</strong>
+</div>

README_TH.md

@@ -22,7 +22,7 @@ PyThaiNLP เป็นไลบารีภาษาไพทอนสำหร
 
 | รุ่น | คำอธิบาย | สถานะ |
 |:------:|:--:|:------:|
-| [2.2.3](https://github.com/PyThaiNLP/pythainlp/releases) | Stable | [Change Log](https://github.com/PyThaiNLP/pythainlp/issues/330) |
+| [2.2.4](https://github.com/PyThaiNLP/pythainlp/releases) | Stable | [Change Log](https://github.com/PyThaiNLP/pythainlp/issues/330) |
 | [`dev`](https://github.com/PyThaiNLP/pythainlp/tree/dev) | Release Candidate for 2.3  | [Change Log](https://github.com/PyThaiNLP/pythainlp/issues/445) |
 
 ติดตามพวกเราบน [PyThaiNLP Facebook page](https://www.facebook.com/pythainlp/) เพื่อรับข่าวสารเพิ่มเติม

docs/api/corpus.rst

@@ -7,20 +7,21 @@ The :class:`pythainlp.corpus` provides access to corpus that comes with PyThaiNL
 Modules
 -------
 
+.. autofunction:: countries
 .. autofunction:: get_corpus
 .. autofunction:: get_corpus_db
 .. autofunction:: get_corpus_db_detail
 .. autofunction:: get_corpus_path
 .. autofunction:: download
 .. autofunction:: remove
-.. autofunction:: pythainlp.corpus.common.countries
-.. autofunction:: pythainlp.corpus.common.provinces
-.. autofunction:: pythainlp.corpus.common.thai_stopwords
-.. autofunction:: pythainlp.corpus.common.thai_words
-.. autofunction:: pythainlp.corpus.common.thai_syllables
-.. autofunction:: pythainlp.corpus.common.thai_negations
-.. autofunction:: pythainlp.corpus.common.thai_female_names
-.. autofunction:: pythainlp.corpus.common.thai_male_names
+.. autofunction:: provinces
+.. autofunction:: thai_stopwords
+.. autofunction:: thai_words
+.. autofunction:: thai_syllables
+.. autofunction:: thai_negations
+.. autofunction:: thai_family_names
+.. autofunction:: thai_female_names
+.. autofunction:: thai_male_names
 .. autofunction:: pythainlp.corpus.conceptnet.edges
 
 TNC

docs/api/tag.rst

@@ -2,12 +2,12 @@
 
 pythainlp.tag
 =====================================
-The :class:`pythainlp.tag` contains functions that are used to tag different parts of a text including
-Part-of-Speech (POS) tags, and Named Entity Recognition (NER) tag.
+The :class:`pythainlp.tag` contains functions that are used to mark linguistic and other annotation to different parts of a text including
+part-of-speech (POS) tag and named entity (NE) tag.
 
-For the POS tags, there are two set of tags including `Universal Dependencies (UD) <https://universaldependencies.org/>`_ and ORCHID [#Sornlertlamvanich_2000]_ POS tags.
+For POS tags, there are three set of available tags: `Universal POS tags <https://universaldependencies.org/>`_, ORCHID POS tags [#Sornlertlamvanich_2000]_, and LST20 POS tags [#Prachya_2020]_.
 
-The following table shows the list of Part-of-Speech (POS) tags according to Universal Dependencies (UD) POS tags:
+The following table shows Universal POS tags as used in Universal Dependencies (UD):
 
 ============   ==========================   =============================
 Abbreviation   Part-of-Speech tag           Examples
@@ -29,7 +29,7 @@ Abbreviation   Part-of-Speech tag           Examples
  VERB          Verb                         เปิด, ให้, ใช้, เผชิญ, อ่าน
 ============   ==========================   =============================
 
-The following table shows the list of Part-of-Speech (POS) tags according to ORCHID POS tags from the paper:
+The following table shows POS tags as used in ORCHID:
 
 ============   =================================================      =================================
 Abbreviation   Part-of-Speech tag                                     Examples
@@ -93,7 +93,7 @@ Abbreviation   Part-of-Speech tag                                     Examples
 
 ORCHID corpus uses different set of POS tags. Thus, we make UD POS tags version for ORCHID corpus.
 
-The following table shows the mapping of Part-of-Speech (POS) tags from ORCHID POS tags to UD POS tags:
+The following table shows the mapping of POS tags from ORCHID to UD:
 
 ===============     =======================
 ORCHID POS tags     Coresponding UD POS tag
@@ -161,15 +161,54 @@ PUNCT               PUNCT
 PUNC                PUNCT
 ===============     =======================
 
-For the NER, we use `Inside-outside-beggining (IOB) <https://en.wikipedia.org/wiki/Inside%E2%80%93outside%E2%80%93beginning_(tagging)>`_ format to tag NER for each words.
-For instance, given a sentence "บารัค โอบามาเป็นประธานธิปดี", it would be tag the tokens "บารัค", "โอบามา", "เป็น", "ประธานาธิปดี" as "B-PERSON", "I-PERSON", "I-PERSON", "O", and "O" respectively.
+Details about LST20 POS tags are available in [#Prachya_2020]_.
 
-The *B-* prefix indicates begining token for a chunk of person name, "บารัค โอบามา" and *I-* prefix indicates the intermediate token. However, the term *O* indicates that a token not belong to any NER chunk.
+The following table shows the mapping of POS tags from LST20 to UD:
 
-The following table shows the list of Named Entity Recognition (NER) tags:
++----------------+-------------------------+
+| LST20 POS tags | Coresponding UD POS tag |
++================+=========================+
+| AJ             | ADJ                     |
++----------------+-------------------------+
+| AV             | ADV                     |
++----------------+-------------------------+
+| AX             | AUX                     |
++----------------+-------------------------+
+| CC             | CCONJ                   |
++----------------+-------------------------+
+| CL             | NOUN                    |
++----------------+-------------------------+
+| FX             | NOUN                    |
++----------------+-------------------------+
+| IJ             | INTJ                    |
++----------------+-------------------------+
+| NN             | NOUN                    |
++----------------+-------------------------+
+| NU             | NUM                     |
++----------------+-------------------------+
+| PA             | PART                    |
++----------------+-------------------------+
+| PR             | PROPN                   |
++----------------+-------------------------+
+| PS             | ADP                     |
++----------------+-------------------------+
+| PU             | PUNCT                   |
++----------------+-------------------------+
+| VV             | VERB                    |
++----------------+-------------------------+
+| XX             | X                       |
++----------------+-------------------------+
+
+For the NE, we use `Inside-outside-beggining (IOB) <https://en.wikipedia.org/wiki/Inside%E2%80%93outside%E2%80%93beginning_(tagging)>`_ format to tag NE for each word.
+
+*B-* prefix indicates the begining token of the chunk. *I-* prefix indicates the intermediate token within the chunk. *O* indicates that the token does not belong to any NE chunk.
+
+For instance, given a sentence "บารัค โอบามาเป็นประธานธิปดี", it would tag the tokens "บารัค", "โอบามา", "เป็น", "ประธานาธิปดี" with "B-PERSON", "I-PERSON", "O", and "O" respectively.
+
+The following table shows named entity (NE) tags as used PyThaiNLP:
 
 ============================    =================================
-Named Entity Recognition tag    Examples
+Named Entity tag                Examples
 ============================    =================================
  DATE                           2/21/2004, 16 ก.พ., จันทร์
  TIME                           16.30 น., 5 วัน, 1-3 ปี
@@ -214,3 +253,6 @@ References
 .. [#Sornlertlamvanich_2000] Virach Sornlertlamvanich, Naoto Takahashi and Hitoshi Isahara. (2000).
             Building a Thai Part-Of-Speech Tagged Corpus (ORCHID).
             The Journal of the Acoustical Society of Japan (E), Vol.20, No.3, pp 189-198, May 1999.
+.. [#Prachya_2020] Prachya Boonkwan and Vorapon Luantangsrisuk and Sitthaa Phaholphinyo and Kanyanat Kriengket and Dhanon Leenoi and Charun Phrombut and Monthika Boriboon and Krit Kosawat and Thepchai Supnithi. (2020).
+            The Annotation Guideline of LST20 Corpus.
+            arXiv:2008.05055

docs/api/util.rst

@@ -12,6 +12,7 @@ Modules
 .. autofunction:: collate
 .. autofunction:: dict_trie
 .. autofunction:: digit_to_text
+.. autofunction:: display_thai_char
 .. autofunction:: eng_to_thai
 .. autofunction:: find_keyword
 .. autofunction:: countthai

pythainlp/__init__.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-__version__ = "2.2.3"
+__version__ = "2.2.4"
 
 thai_consonants = "กขฃคฅฆงจฉชซฌญฎฏฐฑฒณดตถทธนบปผฝพฟภมยรลวศษสหฬอฮ"  # 44 chars
 

pythainlp/corpus/__init__.py

@@ -18,6 +18,7 @@
     "get_corpus_path",
     "provinces",
     "remove",
+    "thai_family_names",
     "thai_female_names",
     "thai_male_names",
     "thai_negations",
@@ -86,6 +87,7 @@ def corpus_db_path() -> str:
 from pythainlp.corpus.common import (
     countries,
     provinces,
+    thai_family_names,
     thai_female_names,
     thai_male_names,
     thai_negations,