Update docs (OpenNMT#1311)

* Update docs, standardize shape, add CONTRIBUTING note, use sphinx-argparse.

CONTRIBUTING.md

@@ -1,11 +1,88 @@
+# Contributors
+
 OpenNMT-py is a community developed project and we love developer contributions.
 
+## Guidelines
 Before sending a PR, please do this checklist first:
 
-- Please run `onmt/tests/pull_request_chk.sh` and fix any errors. When adding new functionality, also add tests to this script. Included checks:
-    1. flake8 and pep8-naming check for coding style;
+- Please run `tools/pull_request_chk.sh` and fix any errors. When adding new functionality, also add tests to this script. Included checks:
+    1. flake8 check for coding style;
     2. unittest;
     3. continuous integration tests listed in `.travis.yml`.
-- When adding/modifying class constructor, please make the arguments as same naming style as its superclass in pytorch.
-- If your change is based on a paper, please include a clear comment and reference in the code.
-- If your function takes/returns tensor arguments, please include assertions to document the sizes. See `GlobalAttention.py` for examples.
+- When adding/modifying class constructor, please make the arguments as same naming style as its superclass in PyTorch.
+- If your change is based on a paper, please include a clear comment and reference in the code (more on that below). 
+
+### Docstrings
+Above all, try to follow the Google docstring format
+([Napoleon example](https://sphinxcontrib-napoleon.readthedocs.io/en/latest/example_google.html),
+[Google styleguide](http://google.github.io/styleguide/pyguide.html)).
+This makes it easy to include your contributions in the Sphinx documentation. And, do feel free
+to autodoc your contributions in the API ``.rst`` files in the `docs/source` folder! If you do, check that
+your additions look right.
+
+```bash
+cd docs
+# install some dependencies if necessary:
+# recommonmark, sphinx_rtd_theme, sphinxcontrib-bibtex
+make html
+firefox build/html/main.html  # or your browser of choice
+```
+
+Some particular advice:
+- Try to follow Python 3 [``typing`` module](https://docs.python.org/3/library/typing.html) conventions when documenting types.
+    - Exception: use "or" instead of unions for more readability
+    - For external types, use the full "import name". Common abbreviations (e.g. ``np``) are acceptable.
+      For ``torch.Tensor`` types, the ``torch.`` is optional.
+    - Please don't use tics like `` (`str`) `` or rst directives like `` (:obj:`str`) ``. Napoleon handles types
+      very well without additional help, so avoid the clutter.
+- [Google docstrings don't support multiple returns](https://stackoverflow.com/questions/29221551/can-sphinx-napoleon-document-function-returning-multiple-arguments).
+For multiple returns, the following works well with Sphinx and is still very readable.
+  ```python
+  def foo(a, b):
+      """This is my docstring.
+      
+      Args:
+          a (object): Something.
+          b (class): Another thing.
+    
+      Returns:
+          (object, class):
+        
+          * a: Something or rather with a long
+            description that spills over.
+          * b: And another thing.
+      """
+
+      return a, b
+  ```
+- When citing a paper, avoid directly linking in the docstring! Add a Bibtex entry to `docs/source/refs.bib`.
+E.g., to cite "Attention Is All You Need", visit [arXiv](https://arxiv.org/abs/1706.03762), choose the
+[bibtext](https://dblp.uni-trier.de/rec/bibtex/journals/corr/VaswaniSPUJGKP17) link, search `docs/source/refs.bib`
+using `CTRL-F` for `DBLP:journals/corr/VaswaniSPUJGKP17`, and if you do not find it then copy-paste the
+citation into `refs.bib`. Then, in your docstring, use ``:cite:`DBLP:journals/corr/VaswaniSPUJGKP17` ``.
+    - However, a link is better than nothing.
+- Please document tensor shapes. Prefer the format
+  ``` ``(a, b, c)`` ```. This style is easy to read, allows using ``x`` for multplication, and is common
+  (PyTorch uses a few variations on the parentheses format, AllenNLP uses exactly this format, Fairseq uses
+  the parentheses format with single ticks).
+    - Again, a different style is better than no shape documentation.
+- Please avoid unnecessary space characters, try to capitalize, and try to punctuate.
+
+  For multi-line docstrings, add a blank line after the closing ``"""``.
+  Don't use a blank line before the closing quotes.
+
+  ``""" not this """`` ``"""This."""``
+
+  ```python
+  """
+      Not this.
+  """
+  ```
+  ```python
+  """This."""
+  ```
+
+  This note is the least important. Focus on content first, but remember that consistent docs look good.
+- Be sensible about the first line. Generally, one stand-alone summary line (per the Google guidelines) is good.
+  Sometimes, it's better to cut directly to the args or an extended description. It's always acceptable to have a
+  "trailing" citation.

docs/requirements.txt

@@ -3,3 +3,4 @@ sphinxcontrib.bibtex
 sphinxcontrib.mermaid
 sphinx-rtd-theme
 recommonmark
+sphinx-argparse

docs/source/CONTRIBUTING.md

@@ -2,12 +2,87 @@
 
 OpenNMT-py is a community developed project and we love developer contributions.
 
+## Guidelines
 Before sending a PR, please do this checklist first:
 
 - Please run `tools/pull_request_chk.sh` and fix any errors. When adding new functionality, also add tests to this script. Included checks:
     1. flake8 check for coding style;
     2. unittest;
     3. continuous integration tests listed in `.travis.yml`.
-- When adding/modifying class constructor, please make the arguments as same naming style as its superclass in pytorch.
-- If your change is based on a paper, please include a clear comment and reference in the code. 
-- If your function takes/returns tensor arguments, please include assertions to document the sizes. See `GlobalAttention.py` for examples. 
+- When adding/modifying class constructor, please make the arguments as same naming style as its superclass in PyTorch.
+- If your change is based on a paper, please include a clear comment and reference in the code (more on that below). 
+
+### Docstrings
+Above all, try to follow the Google docstring format
+([Napoleon example](https://sphinxcontrib-napoleon.readthedocs.io/en/latest/example_google.html),
+[Google styleguide](http://google.github.io/styleguide/pyguide.html)).
+This makes it easy to include your contributions in the Sphinx documentation. And, do feel free
+to autodoc your contributions in the API ``.rst`` files in the `docs/source` folder! If you do, check that
+your additions look right.
+
+```bash
+cd docs
+# install some dependencies if necessary:
+# recommonmark, sphinx_rtd_theme, sphinxcontrib-bibtex
+make html
+firefox build/html/main.html  # or your browser of choice
+```
+
+Some particular advice:
+- Try to follow Python 3 [``typing`` module](https://docs.python.org/3/library/typing.html) conventions when documenting types.
+    - Exception: use "or" instead of unions for more readability
+    - For external types, use the full "import name". Common abbreviations (e.g. ``np``) are acceptable.
+      For ``torch.Tensor`` types, the ``torch.`` is optional.
+    - Please don't use tics like `` (`str`) `` or rst directives like `` (:obj:`str`) ``. Napoleon handles types
+      very well without additional help, so avoid the clutter.
+- [Google docstrings don't support multiple returns](https://stackoverflow.com/questions/29221551/can-sphinx-napoleon-document-function-returning-multiple-arguments).
+For multiple returns, the following works well with Sphinx and is still very readable.
+  ```python
+  def foo(a, b):
+      """This is my docstring.
+      
+      Args:
+          a (object): Something.
+          b (class): Another thing.
+    
+      Returns:
+          (object, class):
+        
+          * a: Something or rather with a long
+            description that spills over.
+          * b: And another thing.
+      """
+
+      return a, b
+  ```
+- When citing a paper, avoid directly linking in the docstring! Add a Bibtex entry to `docs/source/refs.bib`.
+E.g., to cite "Attention Is All You Need", visit [arXiv](https://arxiv.org/abs/1706.03762), choose the
+[bibtext](https://dblp.uni-trier.de/rec/bibtex/journals/corr/VaswaniSPUJGKP17) link, search `docs/source/refs.bib`
+using `CTRL-F` for `DBLP:journals/corr/VaswaniSPUJGKP17`, and if you do not find it then copy-paste the
+citation into `refs.bib`. Then, in your docstring, use ``:cite:`DBLP:journals/corr/VaswaniSPUJGKP17` ``.
+    - However, a link is better than nothing.
+- Please document tensor shapes. Prefer the format
+  ``` ``(a, b, c)`` ```. This style is easy to read, allows using ``x`` for multplication, and is common
+  (PyTorch uses a few variations on the parentheses format, AllenNLP uses exactly this format, Fairseq uses
+  the parentheses format with single ticks).
+    - Again, a different style is better than no shape documentation.
+- Please avoid unnecessary space characters, try to capitalize, and try to punctuate.
+
+  For multi-line docstrings, add a blank line after the closing ``"""``.
+  Don't use a blank line before the closing quotes.
+
+  ``""" not this """`` ``"""This."""``
+
+  ```python
+  """
+      Not this.
+  """
+  ```
+  ```python
+  """This."""
+  ```
+
+  This note is the least important. Focus on content first, but remember that consistent docs look good.
+- Be sensible about the first line. Generally, one stand-alone summary line (per the Google guidelines) is good.
+  Sometimes, it's better to cut directly to the args or an extended description. It's always acceptable to have a
+  "trailing" citation.

docs/source/Library.md

@@ -1,4 +1,4 @@
-# Library: Example
+# Library
 
 For this example, we will assume that we have run preprocess to
 create our datasets. For instance

docs/source/Summarization.md

@@ -1,4 +1,4 @@
-# Example: Summarization
+# Summarization
 
 Note: The process and results below are presented in our paper `Bottom-Up Abstractive Summarization`. Please consider citing it if you follow these instructions. 
 

docs/source/_static/theme_overrides.css

@@ -0,0 +1,13 @@
+/* override table width restrictions */
+@media screen and (min-width: 767px) {
+
+   .wy-table-responsive table td {
+      /* !important prevents the common CSS stylesheets from overriding
+         this as on RTD they are loaded after this stylesheet */
+      white-space: normal !important;
+   }
+
+   .wy-table-responsive {
+      overflow: visible !important;
+   }
+}

docs/source/conf.py

@@ -48,7 +48,17 @@
               'sphinx.ext.githubpages',
               'sphinx.ext.napoleon',
               'sphinxcontrib.mermaid',
-              'sphinxcontrib.bibtex']
+              'sphinxcontrib.bibtex',
+              'sphinxarg.ext']
+
+# Show base classes
+autodoc_default_options = {
+    'show-inheritance': True
+}
+
+# Use "variables" section for Attributes instead of weird block things
+# mimicking the function style.
+napoleon_use_ivar = True
 
 # Add any paths that contain templates here, relative to this directory.
 templates_path = ['.templates']
@@ -117,7 +127,13 @@
 # Add any paths that contain custom static files (such as style sheets) here,
 # relative to this directory. They are copied after the builtin static files,
 # so a file named "default.css" will overwrite the builtin "default.css".
-html_static_path = ['.static']
+html_static_path = ['_static']
+
+html_context = {
+    'css_files': [
+        '_static/theme_overrides.css',  # override wide tables in RTD theme
+        ],
+     }
 
 # Custom sidebar templates, must be a dictionary that maps document names
 # to template names.

docs/source/extended.md

@@ -1,5 +1,5 @@
 
-# Example: Translation
+# Translation
 
 The example below uses the Moses tokenizer (http://www.statmt.org/moses/) to prepare the data and the moses BLEU script for evaluation. This example if for training for the WMT'16 Multimodal Translation task (http://www.statmt.org/wmt16/multimodal-task.html).
 

docs/source/im2text.md

@@ -1,4 +1,4 @@
-# Example: Image to Text
+# Image to Text
 
 A deep learning-based approach to learning the image-to-text conversion, built on top of the <a href="http://opennmt.net/">OpenNMT</a> system. It is completely data-driven, hence can be used for a variety of image-to-text problems, such as image captioning, optical character recognition and LaTeX decompilation. 
 

docs/source/index.rst

@@ -2,23 +2,43 @@ Contents
 --------
 
 .. toctree::
+      :caption: Getting Started
       :maxdepth: 2
 
       main.md
       quickstart.md
-      onmt.rst
-      onmt.modules.rst
-      onmt.translation.rst
-      onmt.inputters.rst
-      Library.md
+      FAQ.md
+      CONTRIBUTING.md
+      ref.rst
 
-      options/preprocess.md
-      options/train.md
-      options/translate.md
+
+.. toctree::
+      :caption: Examples
+      :maxdepth: 2
+
+      Library.md
       extended.md
       Summarization.md
       im2text.md
       speech2text.md
-      FAQ.md
-      CONTRIBUTING.md
-      ref.rst
+
+
+.. toctree::
+      :caption: Scripts
+      :maxdepth: 2
+
+      options/preprocess.rst
+      options/train.rst
+      options/translate.rst
+      options/server.rst
+
+
+.. toctree::
+      :caption: API
+      :maxdepth: 2
+
+      onmt.rst
+      onmt.modules.rst
+      onmt.translation.rst
+      onmt.translate.translation_server.rst
+      onmt.inputters.rst

docs/source/main.md

@@ -1,7 +1,7 @@
 # Overview
 
 
-This portal provides a detailled documentation of the OpenNMT toolkit. It describes how to use the PyTorch project and how it works.
+This portal provides a detailed documentation of the OpenNMT toolkit. It describes how to use the PyTorch project and how it works.
 
 
 
@@ -52,3 +52,5 @@ When using OpenNMT for research please cite our
 You can find additional help or tutorials in the following resources:
 
 * [Gitter channel](https://gitter.im/OpenNMT/openmt-py)
+
+* [Forum](http://forum.opennmt.net/)

docs/source/onmt.inputters.rst

@@ -1,8 +1,13 @@
-Doc: Data Loaders
+Data Loaders
 =================
 
-Datasets
----------
+Data Readers
+-------------
+
+.. autoexception:: onmt.inputters.datareader_base.MissingDependencyException
+
+.. autoclass:: onmt.inputters.DataReaderBase
+    :members:
 
 .. autoclass:: onmt.inputters.TextDataReader
     :members:
@@ -12,3 +17,10 @@ Datasets
 
 .. autoclass:: onmt.inputters.AudioDataReader
     :members:
+
+
+Dataset
+--------
+
+.. autoclass:: onmt.inputters.Dataset
+    :members: