Merge remote-tracking branch 'origin/bug544fix' into bug544fix

Author: J007X

.github/workflows/main.yml

@@ -48,7 +48,7 @@ jobs:
           sudo apt-get install -y libsndfile1-dev
           python -m pip install --progress-bar off --upgrade pip
           pip install --progress-bar off Django django-guardian
-          pip install --progress-bar off pylint==2.10.2 flake8==3.9.2 mypy==0.910 pytest==5.1.3 black==20.8b1
+          pip install --progress-bar off pylint==2.10.2 flake8==3.9.2 mypy==0.931 pytest==5.1.3 black==20.8b1
           pip install --progress-bar off types-PyYAML==5.4.8 types-typed-ast==1.4.4 types-requests==2.25.6 types-dataclasses==0.1.7
           pip install --progress-bar off coverage codecov
       - name: Format check with Black
@@ -91,9 +91,9 @@ jobs:
       - name: Lint with pylint
         run: |
           pylint forte/
-      - name: Lint main code with mypy when torch version is not 1.5.0
+      - name: Lint main code with mypy when torch version is not 1.5.0 and python is 3.9
         run: |
-          if [[ ${{ matrix.torch-version }} != "1.5.0" ]]; then mypy forte; fi
+          if [[ ${{ matrix.torch-version }} != "1.5.0" && ${{ matrix.python-version }} == "3.9" ]]; then mypy forte; fi
       - name: Test with pytest and run coverage
         run: |
           coverage run -m pytest tests

.pre-commit-config.yaml

@@ -0,0 +1,14 @@
+# See https://pre-commit.com for more information
+# See https://pre-commit.com/hooks.html for more hooks
+repos:
+-   repo: https://github.com/pre-commit/pre-commit-hooks
+    rev: v3.2.0
+    hooks:
+    -   id: trailing-whitespace
+    -   id: end-of-file-fixer
+    -   id: check-yaml
+    -   id: check-added-large-files
+-   repo: https://github.com/psf/black
+    rev: 20.8b1
+    hooks:
+    - id: black

CONTRIBUTING.md

@@ -127,6 +127,8 @@ the [Google Python Style guide](http://google.github.io/styleguide/pyguide.html)
 project code is examined using `pylint`, `flake8`, `mypy`, `black` and `sphinx-build` which will be run
 automatically in CI. It's recommended that you should run these tests locally before submitting your pull request to save time. Refer to the github workflow [here](https://github.com/asyml/forte/blob/master/.github/workflows/main.yml) for detailed steps to carry out the tests. Basically what you need to do is to install the requirements (check out the `Install dependencies` sections) and run the commands (refer to the steps in `Format check with Black`, `Lint with flake8`, `Lint with pylint`, `Lint main code with mypy when torch version is not 1.5.0`, `Build Docs`, etc.).
 
+We also recommend using tools `pre-commit` that automates the checking process before each commit since checking format is a repetitive process. We have the configuration file `.pre-commit-config.yaml` that lists several plugins including `black` to check format in the project root folder. Developers only need to install the package by `pip install pre-commit`.
+
 ### Docstring
 
  All public methods require docstring and type annotation. It is recommended to add docstring for all functions. The docstrings should follow the [`Comments and Docstrings` section](https://google.github.io/styleguide/pyguide.html#38-comments-and-docstrings) of Google Python Style guide. We will include a pylint plugin called [docparams](https://github.com/PyCQA/pylint/blob/main/pylint/extensions/docparams.rst) to validate the parameters of docstrings:
@@ -136,7 +138,7 @@ automatically in CI. It's recommended that you should run these tests locally be
 
 You should take special care of the indentations in your documentation. Make sure the indents are consistent and follow the Google Style guide. All sections other than the heading should maintain a hanging indent of two or four spaces. Refer to the examples [here](https://google.github.io/styleguide/pyguide.html#383-functions-and-methods) for what is expected and what are the requirements for different sections like `args`, `lists`, `returns`, etc. Invalid indentations might trigger errors in `sphinx-build` and will cause confusing rendering of the documentation. You can run `sphinx-build` locally to see whether the generated docs look reasonable.
 
-Another aspect that should be noted is the format of links or cross-references of python objects. Make sure to follow the [sphinx cross-referencing syntax](https://www.sphinx-doc.org/en/master/usage/restructuredtext/roles.html#xref-syntax). The references will be checked by [sphinx-build nit-picky mode](https://www.sphinx-doc.org/en/master/man/sphinx-build.html#cmdoption-sphinx-build-n) which raises warnings for all the missing and unresolvable links. 
+Another aspect that should be noted is the format of links or cross-references of python objects. Make sure to follow the [sphinx cross-referencing syntax](https://www.sphinx-doc.org/en/master/usage/restructuredtext/roles.html#xref-syntax). The references will be checked by [sphinx-build nit-picky mode](https://www.sphinx-doc.org/en/master/man/sphinx-build.html#cmdoption-sphinx-build-n) which raises warnings for all the missing and unresolvable links.
 
 ### Git Commit Style
 

docs/conf.py

@@ -61,19 +61,19 @@
 master_doc = "index"
 
 # General information about the project.
-project = u"Forte"
-copyright = u"2019, Forte"
-author = u"Forte"
+project = "Forte"
+copyright = "2019, Forte"
+author = "Forte"
 
 # The version info for the project you're documenting, acts as replacement for
 # |version| and |release|, also used in various other places throughout the
 # built documents.
 #
 # The short X.Y version.
 # version = u'{}'.format(__version_short__)
-version = u"{}".format(__version__)
+version = "{}".format(__version__)
 # The full version, including alpha/beta/rc tags.
-release = u"{}".format(__version__)
+release = "{}".format(__version__)
 
 # The language for content autogenerated by Sphinx. Refer to documentation
 # for a list of supported languages.
@@ -145,7 +145,7 @@
 
 # The name for this set of Sphinx documents.
 # "<project> v<release> documentation" by default.
-html_title = u"Forte v0.1"
+html_title = "Forte v0.1"
 
 # A shorter title for the navigation bar.  Default is the same as html_title.
 # html_short_title = None
@@ -253,7 +253,7 @@
 # (source start file, target name, title,
 #  author, documentclass [howto, manual, or own class]).
 latex_documents = [
-    (master_doc, "forte.tex", u"Forte Documentation", u"Forte", "manual"),
+    (master_doc, "forte.tex", "Forte Documentation", "Forte", "manual"),
 ]
 
 # The name of an image file (relative to this directory) to place at the top of
@@ -281,7 +281,7 @@
 
 # One entry per manual page. List of tuples
 # (source start file, name, description, authors, manual section).
-man_pages = [(master_doc, "forte", u"Forte Documentation", [author], 1)]
+man_pages = [(master_doc, "forte", "Forte Documentation", [author], 1)]
 
 # If true, show URL addresses after external links.
 # man_show_urls = False
@@ -296,7 +296,7 @@
     (
         master_doc,
         "forte",
-        u"Forte Documentation",
+        "Forte Documentation",
         author,
         "Forte",
         "One line description of project.",

docs/index.rst

@@ -1,36 +1,159 @@
 Welcome to Forte's documentation!
 ******************************************
+This outline is currently **in progress** so many sections are empty.
 
+
+Overview
+====================
+
+**Forte** is a toolkit for building Natural Language Processing pipelines, featuring cross-task
+interaction, adaptable data-model interfaces and many more. It provides a platform to assemble
+state-of-the-art NLP and ML technologies in a highly-composable fashion, including a wide
+spectrum of tasks ranging from Information Retrieval, Natural Language Understanding to Natural
+Language Generation.
+
+With Forte, it is extremely simple to build an integrated system that can search documents,
+analyze and extract information and generate language all in one place. This allows the developer
+to fully utilize and combine the strength and results from each step, and allow the system to
+make fully informed decision at the end of the pipeline.
+
+While it is quite easy to combine arbitrary 3rd party tools (Check out these `examples <index_appendices.html>`_ !),
+Forte also brings technology to you by supporting deep learning via Texar, and by providing a convenient
+model data interface that allows user to cast tasks to models.
+
+
+Core Design Principles
+------------------------
+
+
+The core design principle of Forte is the abstraction of NLP concepts and machine learning models,
+which provides better separation between data, model and tasks, but enables interactions
+between different components of the pipeline. Based on this, we make Forte:
+
+* **Composable**: Forte helps users to decompose a problem into *data*, *models* and *tasks*. The tasks can further be divided into sub-tasks. A complex use case can be solved by composing heterogeneous modules via straightforward python APIs or declarative configuration files. The components (e.g. models or tasks) in the pipeline can be flexibly swapped in and out, as long as the API contracts are matched. The approach greatly improves module reusability, enables fast development and makes the library flexible for user needs.
+
+* **Generalizable and Extensible**: Forte promotes generalization to support not only a wide range of NLP tasks, but also extensible for new tasks or new domains. In particular, Forte provides the *Ontology* system that helps users define types according to their tasks. Users can simply specify the type declaratively through JSON files. Our Code Generation tool will automatically generate python files ready to be used into your project. Check out our `Ontology Generation documentation <toc/ontology_generation.html>`_ for more details.
+
+* **Transparent Data Flow**: Central to Forte's composable architecture is a universal data format that supports seamless data flow between different steps. Forte advocates a transparent data flow to facilitate flexible process intervention and simple pipeline control. Combined with the general data format, Forte makes a perfect tool for data inspection, component swapping and result sharing. This is particularly helpful during team collaborations!
+
+.. image:: _static/img/forte_arch.png
+
+.. image:: _static/img/forte_results.png
+
+Package Overview
+-----------------
+.. list-table:: Title
+   :widths: 25 75
+   :header-rows: 1
+
+   * - Package Name
+     - Package Description
+   * - :class:`~forte`
+     - an open-source toolkit for NLP
+   * - :class:`~forte.data.readers`
+     - a data module for reading different formats of text data like CoNLL, Ontonotes etc
+   * - :class:`~forte.processors`
+     - a collection of processors for building NLP pipelines
+   * - :class:`~forte.trainer`
+     - a collection of modules for training different NLP tasks
+   * - :class:`~ft.onto.base_ontology`
+     - a module containing basic ontologies like Token, Sentence, Document etc
+
+
+
+Library API example
+--------------------
+A simple code example that runs Named Entity Recognizer
+
+.. code-block:: python
+
+   import yaml
+
+   from forte.pipeline import Pipeline
+   from forte.data.readers import CoNLL03Reader
+   from forte.processors.nlp import CoNLLNERPredictor
+   from ft.onto.base_ontology import Token, Sentence
+   from forte.common.configuration import Config
+
+
+   config_data = yaml.safe_load(open("config_data.yml", "r"))
+   config_model = yaml.safe_load(open("config_model.yml", "r"))
+
+   config = Config({}, default_hparams=None)
+   config.add_hparam('config_data', config_data)
+   config.add_hparam('config_model', config_model)
+
+
+   pl = Pipeline()
+   pl.set_reader(CoNLL03Reader())
+   pl.add(CoNLLNERPredictor(), config=config)
+
+   pl.initialize()
+
+   for pack in pl.process_dataset(config.config_data.test_path):
+      for pred_sentence in pack.get_data(context_type=Sentence, request={Token: {"fields": ["ner"]}}):
+         print("============================")
+         print(pred_sentence["context"])
+         print("The entities are...")
+         print(pred_sentence["Token"]["ner"])
+         print("============================")
+
+
+
+Many more examples are available `here  <index_appendices.html>`_. We are also working assembling some
+interesting `tutorials <https://github.com/asyml/forte/wiki>`_
+
+
+Download and Installation
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Download the repository through
+
+```bash
+git clone https://github.com/asyml/forte.git
+```
+
+After `cd` into `forte`, you can install it through
+
+```bash
+pip install .
+```
+
+
+License
+~~~~~~~~~
+
+`Apache License 2.0 <https://github.com/asyml/forte/blob/master/LICENSE>`_
+
+
+----------------
+
+
+Overview
+====================
 .. toctree::
    :maxdepth: 2
-   
-   outline.md
+
+   toc/overview.md
 
 
+
+NLP with Forte
+====================
 .. toctree::
    :maxdepth: 2
 
-   tutorial/get_started.md
+   index_toc.rst
 
+APPENDICES
+===========
 .. toctree::
    :maxdepth: 2
 
-   tutorial/examples.md
-   tutorial/ontology_generation.md
-   tutorial/audio_processing.md
-   tutorial/data_pack.md
+   index_appendices.rst
 
 API
 ====
-
 .. toctree::
    :maxdepth: 2
 
-   code/common.rst
-   code/data.rst
-   code/pipeline.rst
-   code/processors.rst
-   code/models.rst
-   code/training_system.rst
-   code/data_aug.rst
-   code/vocabulary.rst
+   index_api.rst

docs/index_api.rst

@@ -0,0 +1,25 @@
+.. role:: hidden
+    :class: hidden-section
+
+API
+*********************
+
+.. toctree::
+   :maxdepth: 2
+
+
+   code/common.rst
+
+   code/data.rst
+
+   code/pipeline.rst
+
+   code/processors.rst
+
+   code/models.rst
+
+   code/training_system.rst
+
+   code/data_aug.rst
+
+   code/vocabulary.rst

docs/index_appendices.rst

@@ -0,0 +1,54 @@
+.. role:: hidden
+    :class: hidden-section
+
+APPENDICES
+*********************
+
+
+Glossary
+============
+.. toctree::
+   :maxdepth: 2
+
+* DataPack: a data class that stores structured data and supports efficient data retrieval.
+    -  `DataPack Example <https://github.com/asyml/forte/blob/master/docs/tutorial/handling_structued_data.ipynb>`_
+    - API: :class:`~forte.data.data_pack.DataPack`
+
+* Pipeline: an inference system that contains a set of processing components.
+    - `Pipeline Example <https://github.com/asyml/forte/tree/master/examples/pipelines>`_
+    - API: :class:`~forte.pipeline.Pipeline`
+
+* Ontology: a system that defines the relations between NLP annotations, for example, the relation between words and documents, or between two words.
+    - `An ontology Example <https://github.com/asyml/forte/tree/master/examples/ontology>`_
+    - `An ontology tutorial <https://github.com/asyml/forte/blob/0c1dec1311f27eae150287a8aa405632b265e03e/docs/tutorial/ontology_generation.md>`_
+
+
+
+.. rst-class:: page-break
+
+
+Examples
+==========
+.. toctree::
+    :maxdepth: 2
+
+Rich examples are included to demonstrate the use of Forte, including
+ implementation of cutting-edge models/algorithms and system construction.
+
+More examples are continuously added...
+
+
+* `Data Reading: Showcasing how to read structured data. <https://github.com/asyml/forte/tree/master/examples/wiki_parser>`_
+* `Serialization: Showcasing how to serialize and deserialize data. <https://github.com/asyml/forte/tree/master/examples/serialization>`_
+* `NER: Train a LSTM-CRF named entity recognizer. <https://github.com/asyml/forte/tree/master/examples/ner>`_
+* `BERT Passage Reranker <https://github.com/asyml/forte/tree/master/examples/passage_reranker>`_
+* `Chat Bot: This example showcases the use of Forte to build a retrieval-based chatbot and perform text analysis on the retrieved results.  <https://github.com/asyml/forte/tree/master/examples/chatbot>`_
+* `Audio Reading: a simple speech processing example here to showcase forte's capability to support a wide range of audio processing tasks. <https://github.com/asyml/forte/tree/master/examples/audio>`_
+* `Classification: a text classification example that support various format of table-like dataset <https://github.com/asyml/forte/tree/master/examples/classification>`_
+* `Clinical Pipeline: a project handling clinical datasets shows how to make Forte and Stave work side by side. <https://github.com/asyml/forte/tree/master/examples/clinical_pipeline>`_
+* `Content Rewriter: a example which rewrites the sentence based on the table given a table and a sentence. <https://github.com/asyml/forte/tree/master/examples/content_rewriter>`_
+* `Data Augmentation: this example demonstrates the usage of forte/models/da_rl/MetaAugmentationWrapper, that wraps a BERT Masked Language Model data augmentation model to perform this RL adaptive learning with a BERT-based text classifier downstream model. <https://github.com/asyml/forte/tree/master/examples/data_augmentation>`_
+* `SRL: a semantic role labeling example <https://github.com/asyml/forte/tree/master/examples/srl>`_
+* `Tagging: an implementation of CNN-BiLSTM-CRF model, built on top of Texar and Pytorch <https://github.com/asyml/forte/tree/master/examples/tagging>`_
+* `Twitter sentiment analysis: this example show the use of Forte to perform sentiment analysis on the user's retrieved tweets <https://github.com/asyml/forte/tree/master/examples/twitter_sentiment_analysis>`_
+* `Visualization: visualize datapack data <https://github.com/asyml/forte/tree/master/examples/visualize>`_