Merge branch 'dev' into 2.1

Author: bact

.travis.yml

@@ -1,40 +1,24 @@
-# Config file for automatic testing at travis-ci.org
-# This file will be regenerated if you run travis_pypi_setup.py
-
 language: python
+dist: bionic
 python:
-  - "3.6"
-
-# workaround to make boto work on travis
-# from https://github.com/travis-ci/travis-ci/issues/7940
+- '3.6'
 before_install:
-  - sudo rm -f /etc/boto.cfg
-
-# command to install dependencies, e.g. pip install -r requirements.txt --use-mirrors
+- sudo rm -f /etc/boto.cfg
 install:
-  - pip install -r requirements.txt
-  - pip install .[artagger,icu,ipa,ner,thai2fit,deepcut]
-  - pip install coveralls
-
+- pip install -U numpy
+- pip install -q -r requirements.txt
+- pip install -q .[full]
+- pip install coveralls
 os:
-  - linux
-
-# command to run tests, e.g. python setup.py test
-script:
-  coverage run --source=pythainlp setup.py test
-
-after_success:
-  coveralls
-
-# After you create the Github repo and add it to Travis, run the
-# travis_pypi_setup.py script to finish PyPI deployment setup
+- linux
+script: coverage run --source=pythainlp setup.py test
+after_success: coveralls
 deploy:
   provider: pypi
   distributions: sdist bdist_wheel
   user: wannaphong
   password:
-    secure: PLEASE_REPLACE_ME
+    secure: zX35+8niw5W9H8XbFwacrDAhqyIibdUdC/cARnHlmxLN/2H9IynK0NW04UZwkBlrwrIZrU/g+cqYXFQXu6jE1ozlBKBxUd3xG8d1kixuntI0j9e+erPTs8Ju/KazUZtlknJPvnDMP+/1Dq+RMnMCP3RRlBrH6lvG70OgZ1aBpgx8FxRfs0xHfBIZvo5CVtR/QlDzhDJM1cgEyWkSgnlAhPxpv8qIQbh4/Rw89jXIZqv0bGCVJorrrcTA1oCzkr/4E4u/WZaARnvPjUr2a9U1w7C2IysDHiBfqQWlovdMmpoSLFE56YlG3smbmXfldWjmiMRQoWL+Ifu+smisvOLmR0ja78UMrrhHWP4mdzIeBVVRnT6eHUv0ChmLT2uCkOLE0newhtEJIYToot2TSoLFavXXIQB1fIHt6e74KRTV6WGnm0nFfHuGP+b5SgSPQFgqx8tBpn0rBOeqZ1y3pRISc/drF0F4reWMnlqoQfZZFmLmU1UmDZbvWNvXPu6MWyyuZ1F6fE9jyb3mG+kDuJf1PZ4ejC/sdIvpLlwUGLFGzRMa2TtxXqGq5CWsywPxo8Sx+bpMPCOImuW60PB9K/xKgfLhAtb7gZwndzUGqDbtSJCd5PmTkfEH8fawv/XnydvsssYUpipBCmFDZlNREyAkgOcLlL099Y5fAO8l2gOLyKs=
   on:
     tags: true
-    repo: wannaphongcom/pythainlp
-    #condition: $TOXENV == py35
+    repo: pythainlp/pythainlp

CONTRIBUTING.md

@@ -36,8 +36,8 @@ We use the famous [gitflow](http://nvie.com/posts/a-successful-git-branching-mod
 
 # Discussion
 
-- Facebook group: https://www.facebook.com/groups/thainlp 
-- GitHub issues: https://github.com/PyThaiNLP/pythainlp/issues
+- Facebook group (for Thai NLP Discussion only): https://www.facebook.com/groups/thainlp 
+- GitHub issues (Problems and suggestions): https://github.com/PyThaiNLP/pythainlp/issues
 
 Happy hacking! (;
 
@@ -50,6 +50,8 @@ Happy hacking! (;
 - Charin Polpanumas
 - Peeradej Tanruangporn
 - Arthit Suriyawongkul
+- Chakri Lowphansirikul
+- Pattarawat Chormai
 
 ## newmm (onecut), mm, TCC, and Thai Soundex Code
 - Korakot Chaovavanich
@@ -59,9 +61,15 @@ Happy hacking! (;
 
 ## Docs
 - Peeradej Tanruangporn
+- Chakri Lowphansirikul
 
 ## Maintainers
 - Arthit Suriyawongkul
+- Wannaphong Phatthiyaphaibun
+
+## Benchmark
+- Charin Polpanumas
+- Pattarawat Chormai
 
 ## Contributors
 - See more contributions here https://github.com/PyThaiNLP/pythainlp/graphs/contributors

README.md

@@ -2,27 +2,40 @@
 
 # PyThaiNLP
 
-[![Codacy Badge](https://api.codacy.com/project/badge/Grade/cb946260c87a4cc5905ca608704406f7)](https://www.codacy.com/app/pythainlp/pythainlp_2?utm_source=github.com&utm_medium=referral&utm_content=PyThaiNLP/pythainlp&utm_campaign=Badge_Grade)[![pypi](https://img.shields.io/pypi/v/pythainlp.svg)](https://pypi.python.org/pypi/pythainlp)
+[![Python 3.6](https://img.shields.io/badge/python-3.6-blue.svg)](https://www.python.org/downloads/release/python-360/) 
+[![pypi](https://img.shields.io/pypi/v/pythainlp.svg)](https://pypi.python.org/pypi/pythainlp)
 [![Downloads](https://pepy.tech/badge/pythainlp/month)](https://pepy.tech/project/pythainlp)
+[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0) 
+[![FOSSA Status](https://app.fossa.io/api/projects/git%2Bgithub.com%2FPyThaiNLP%2Fpythainlp.svg?type=shield)](https://app.fossa.io/projects/git%2Bgithub.com%2FPyThaiNLP%2Fpythainlp?ref=badge_shield)
 [![Build Status](https://travis-ci.org/PyThaiNLP/pythainlp.svg?branch=develop)](https://travis-ci.org/PyThaiNLP/pythainlp)
 [![Build status](https://ci.appveyor.com/api/projects/status/9g3mfcwchi8em40x?svg=true)](https://ci.appveyor.com/project/wannaphongcom/pythainlp-9y1ch)
+[![Codacy Badge](https://api.codacy.com/project/badge/Grade/cb946260c87a4cc5905ca608704406f7)](https://www.codacy.com/app/pythainlp/pythainlp_2?utm_source=github.com&utm_medium=referral&utm_content=PyThaiNLP/pythainlp&utm_campaign=Badge_Grade)
 [![Coverage Status](https://coveralls.io/repos/github/PyThaiNLP/pythainlp/badge.svg?branch=dev)](https://coveralls.io/github/PyThaiNLP/pythainlp?branch=dev)
-[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)
-[![FOSSA Status](https://app.fossa.io/api/projects/git%2Bgithub.com%2FPyThaiNLP%2Fpythainlp.svg?type=shield)](https://app.fossa.io/projects/git%2Bgithub.com%2FPyThaiNLP%2Fpythainlp?ref=badge_shield)
 
 Thai Natural Language Processing in Python.
 
 PyThaiNLP is a Python package for text processing and linguistic analysis, similar to `nltk` but with focus on Thai language.
 
+**News**
+
+>Hello,
+>We are conducting a survey for PyThaiNLP’s users and those who are working in the field of Thai NLP.
+
+>We would love to hear your feedback in order to improve the library. Also, we will prioritize for the implementation of new Thai NLP features such as Thai-English Machine Translation, Speech-to-Text, or Text-to-Speech.
+
+>You could take part in this survey via the Google Form shown below:
+>https://forms.gle/aLdSHnvkNuK5CFyt9
+
 **This is a document for development branch (post 2.0). Things will break.**
 
-- The latest stable release is [2.0.5](https://github.com/PyThaiNLP/pythainlp/releases)
+- The latest stable release is [2.0.7](https://github.com/PyThaiNLP/pythainlp/releases)
 - PyThaiNLP 2 supports Python 3.6+. Some functions may work with older version of Python 3, but it is not well-tested and will not be supported. See [change log](https://github.com/PyThaiNLP/pythainlp/issues/118).
   - [Upgrading from 1.7](https://thainlp.org/pythainlp/docs/2.0/notes/pythainlp-1_7-2_0.html)
   - [Upgrade ThaiNER from 1.7](https://github.com/PyThaiNLP/pythainlp/wiki/Upgrade-ThaiNER-from-PyThaiNLP-1.7-to-PyThaiNLP-2.0)
 - Python 2.7+ users can use PyThaiNLP 1.6.
+- 📫 follow us on Facebook [PyThaiNLP](https://www.facebook.com/pythainlp/)
 
-📫 follow us on Facebook [PyThaiNLP](https://www.facebook.com/pythainlp/)
+[![Google Colab Badge](https://badgen.net/badge/Launch%20Quick%20Start%20Guide/on%20Google%20Colab/blue?icon=terminal)](https://colab.research.google.com/github/PyThaiNLP/pythainlp/blob/dev/notebooks/pythainlp-get-started.ipynb)
 
 ## Capabilities
 
@@ -96,6 +109,12 @@ Please do fork and create a pull request :)
 For style guide and other information, including references to algorithms we use, please refer to our [contributing](https://github.com/PyThaiNLP/pythainlp/blob/dev/CONTRIBUTING.md) page.
 
 
+Made with ❤️
+
+We build Thai NLP.
+
+PyThaiNLP team.
+
 # ภาษาไทย
 
 ประมวลภาษาไทยในภาษา Python
@@ -104,9 +123,19 @@ PyThaiNLP เป็นไลบารีภาษาไพทอนเพื่
 
 > เพราะโลกขับเคลื่อนต่อไปด้วยการแบ่งปัน
 
+**ข่าวสาร**
+
+>สวัสดีค่ะ,
+
+>ทางทีมพัฒนา PyThaiNLP อยากขอสอบถามความคิดเห็นของผู้ใช้งาน library PyThaiNLP ปัจจุบัน หรือผู้ที่ทำงานในด้าน NLP ภาษาไทย เพื่อที่เราจะนำไปปรับปรุง library ให้ดียิ่งขึ้น และพัฒนาฟีเจอร์ใหม่ๆ สำหรับ NLP ภาษาไทย เช่น Thai-English Machine Translation, Speech-to-Text หรือ Text-to-Speech
+
+>โดยสามารถตอบแบบสอบถาม ผ่านทาง Google Form ด้านล่างนี้
+
+>https://forms.gle/aLdSHnvkNuK5CFyt9
+
 **เอกสารนี้สำหรับรุ่นพัฒนา อาจมีการเปลี่ยนแปลงได้ตลอด**
 
-- รุ่นเสถียรล่าสุดคือรุ่น [2.0.5](https://github.com/PyThaiNLP/pythainlp/releases)
+- รุ่นเสถียรล่าสุดคือรุ่น [2.0.7](https://github.com/PyThaiNLP/pythainlp/releases)
 - PyThaiNLP 2 รองรับ Python 3.6 ขึ้นไป
 - ผู้ใช้ Python 2.7+ ยังสามารถใช้ PyThaiNLP 1.6 ได้
 
@@ -178,3 +207,10 @@ $ pip install pythainlp[extra1,extra2,...]
 ## สนับสนุนและร่วมพัฒนา
 
 คุณสามารถ[ร่วมพัฒนาโครงการนี้](https://github.com/PyThaiNLP/pythainlp/blob/dev/CONTRIBUTING.md)ได้ โดยการ fork และส่ง pull request กลับมา
+
+
+สร้างด้วย ❤️
+
+พวกเราสร้าง Thai NLP
+
+ทีม PyThaiNLP

SECURITY.md

@@ -0,0 +1,8 @@
+# Security Policy
+
+## Supported Versions
+
+| Version | Supported          |
+| ------- | ------------------ |
+| 2.0.x   | :white_check_mark: |
+| < 2.0   | :x:                |

appveyor.yml

@@ -2,8 +2,11 @@
 # https://www.lfd.uci.edu/~gohlke/pythonlibs/
 
 build: off
+image: Visual Studio 2015
 
 environment:
+  global:
+      CMD_IN_ENV: "cmd /E:ON /V:ON /C .\\appveyor\\run_with_env.cmd"
   matrix:
 #    - PYTHON: "C:/Python36"
 #      PYTHON_VERSION: "3.6"
@@ -16,6 +19,7 @@ environment:
       PYTHON_ARCH: "64"
       ARTAGGER_PKG: "https://github.com/wannaphongcom/artagger/tarball/master#egg=artagger"
       PYICU_PKG: "https://www.dropbox.com/s/7t0rrxwckqbgivi/PyICU-2.3.1-cp36-cp36m-win_amd64.whl?dl=1"
+      DISTUTILS_USE_SDK: "1"
 
 #    - PYTHON: "C:/Python37"
 #      PYTHON_VERSION: "3.7"
@@ -28,23 +32,36 @@ environment:
       PYTHON_ARCH: "64"
       ARTAGGER_PKG: "https://github.com/wannaphongcom/artagger/tarball/master#egg=artagger"
       PYICU_PKG: "https://www.dropbox.com/s/le5dckc3231opqt/PyICU-2.3.1-cp37-cp37m-win_amd64.whl?dl=1"
+      DISTUTILS_USE_SDK: "1"
 
 init:
   - "ECHO %PYTHON% %PYTHON_VERSION% %PYTHON_ARCH%"  
 #  - ps: "ls C:/Python*"
 
+platform:
+  - x64
+
 install:
   - "chcp 65001"
   - "set PYTHONIOENCODING=utf-8"
-  - "%PYTHON%/python.exe --version"
+  - "%PYTHON%\\python.exe -m pip install wheel"
+#  - ECHO "Installed SDKs:"
+#  - ps: "ls \"C:/Program Files/Microsoft SDKs/Windows\""
+  - IF "%ARCH%"=="32" (call "C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\vcvarsall.bat" x86) ELSE (ECHO "probably a 64bit build")
+  - IF "%ARCH%"=="64" (call "C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\vcvarsall.bat" amd64) ELSE (ECHO "probably a 32bit build")
+  - '"%VS140COMNTOOLS%\..\..\VC\vcvarsall.bat" %PLATFORM%'
+  - ps: if (-not(Test-Path($env:PYTHON))) { & appveyor\install.ps1 }
+  - "SET PATH=%PYTHON%;%PYTHON%\\Scripts;%PATH%"
+  - "python --version"
   # - "set ICU_VERSION=62"
-  - "%PYTHON%/python.exe -m pip install --upgrade pip"
-  - "%PYTHON%/python.exe -m pip install coveralls[yaml]"
-  - "%PYTHON%/python.exe -m pip install coverage"
-  - "%PYTHON%/python.exe -m pip install %PYICU_PKG%"
-  - "%PYTHON%/python.exe -m pip install %ARTAGGER_PKG%"
-  - "%PYTHON%/python.exe -m pip install -e .[artagger,icu,ipa,ner,thai2fit,deepcut]"
+  - "pip install --disable-pip-version-check --user --upgrade pip setuptools"
+  - "pip install coveralls[yaml]"
+  - "pip install coverage"
+  - "pip install torch==1.2.0+cpu torchvision==0.4.0+cpu -f https://download.pytorch.org/whl/torch_stable.html"
+  - "pip install %PYICU_PKG%"
+  - "pip install %ARTAGGER_PKG%"
+  - "pip install -e .[full]"
 
 test_script:
-  - "%PYTHON%/python.exe -m pip --version"
-  - "%PYTHON%/python.exe -m coverage run --source=pythainlp setup.py test"
+  - "pip --version"
+  - "python setup.py test"

bin/pythainlp

@@ -1,53 +1,31 @@
-#!python3
+#!/usr/bin/env python
 # -*- coding: utf-8 -*-
 
-_VERSION = "2.0.5"
-
 import argparse
+import sys
+
+from pythainlp import cli
+
+
+parser = argparse.ArgumentParser(
+    usage="pythainlp namespace command [options]"
+)
 
-parser = argparse.ArgumentParser()
-parser.add_argument("-t", "--text", default=None, help="text", type=str)
-parser.add_argument("-seg", "--segment", help="word segment", action="store_true")
-parser.add_argument("-c", "--corpus", help="mange corpus", action="store_true")
-parser.add_argument("-pos", "--postag", help="postag", action="store_true")
-parser.add_argument("-soundex", "--soundex", help="soundex", default=None)
-parser.add_argument("-e", "--engine", default="newmm", help="the engine", type=str)
-parser.add_argument("-pos-e", "--postag_engine", default="perceptron", help="the engine for word tokenize", type=str)
-parser.add_argument("-pos-c", "--postag_corpus", default="orchid", help="corpus for postag", type=str)
-args = parser.parse_args()
-
-if args.corpus:
-    from pythainlp.corpus import *
-    print("PyThaiNLP Corpus")
-    temp=""
-    while temp!="exit":
-        print("\n1. Install\n2. Remove\n3. Update\n4. Exit\n")
-        temp=input("Choose 1, 2, 3, or 4: ")
-        if temp=="1":
-            name=input("Corpus name:")
-            download(name)
-        elif temp=="2":
-            name=input("Corpus name:")
-            remove(name)
-        elif temp=="3":
-            name=input("Corpus name:")
-            download(name)
-        elif temp=="4":
-            break
-        else:
-            print("Choose 1, 2, 3, or 4:")
-elif args.text!=None:
-    from pythainlp.tokenize import word_tokenize
-    tokens=word_tokenize(args.text, engine=args.engine)
-    if args.segment:
-        print("|".join(tokens))
-    elif args.postag:
-        from pythainlp.tag import pos_tag
-        print("\t".join([i[0]+"/"+i[1] for i in pos_tag(tokens, engine=args.postag_engine, corpus=args.postag_corpus)]))
-elif args.soundex!=None:
-    from pythainlp.soundex import soundex
-    if args.engine=="newmm":
-        args.engine="lk82"
-    print(soundex(args.soundex, engine=args.engine))
+parser.add_argument(
+    "namespace",
+    type=str,
+    default="",
+    nargs="?",
+    help="[%s]" % "|".join(cli.available_namespaces)
+)
+
+args = parser.parse_args(sys.argv[1:2])
+
+cli.exit_if_empty(args.namespace, parser)
+
+if hasattr(cli, args.namespace):
+    namespace = getattr(cli, args.namespace)
+    namespace.App(sys.argv)
 else:
-    print(f"PyThaiNLP {_VERSION}")
+    print(f"Namespace not available: {args.namespace}\nPlease run with --help for alternatives")
+