Merge branch 'master' into master

Author: chauhang

.gitignore

@@ -12,6 +12,7 @@ dist/
 frontend/server/src/main/java/org/pytorch/serve/grpc/
 *.pem
 *.backup
+docs/sphinx/src/
 
 # Postman files
 test/artifacts/

benchmarks/benchmark-ab.py

@@ -104,6 +104,7 @@ def benchmark(test_plan, url, gpus, exec_env, concurrency, requests, batch_size,
         docker_torchserve_start()
 
     check_torchserve_health()
+    warm_up()
     run_benchmark()
     generate_report()
 
@@ -123,12 +124,18 @@ def check_torchserve_health():
             time.sleep(3)
     failure_exit("Could not connect to Tochserve instance at " + execution_params['inference_url'])
 
-
-def run_benchmark():
+def warm_up():
     register_model()
 
-    click.secho("\n\nExecuting Apache Bench tests ...", fg='green')
-    click.secho("*Executing inference performance test...", fg='green')
+    click.secho("\n\nExecuting warm-up ...", fg='green')
+    ab_cmd = f"ab -c {execution_params['concurrency']}  -n {execution_params['requests']/10} -k -p {TMP_DIR}/benchmark/input -T " \
+             f"{execution_params['content_type']} {execution_params['inference_url']}/{execution_params['inference_model_url']} > {result_file}"
+    
+    execute(ab_cmd, wait=True)
+
+
+def run_benchmark():
+    click.secho("\n\nExecuting inference perfromance tests ...", fg='green')
     ab_cmd = f"ab -c {execution_params['concurrency']}  -n {execution_params['requests']} -k -p {TMP_DIR}/benchmark/input -T " \
              f"{execution_params['content_type']} {execution_params['inference_url']}/{execution_params['inference_model_url']} > {result_file}"
 
@@ -480,4 +487,4 @@ def failure_exit(msg):
 
 
 if __name__ == '__main__':
-    benchmark()
+    benchmark()

docs/README.md

@@ -1,36 +1,51 @@
 # TorchServe
 
-TorchServe is a flexible and easy to use tool for serving PyTorch models.
+TorchServe is a performant, flexible and easy to use tool for serving PyTorch eager mode and torschripted models.
 
 ## Basic Features
 
 * [Serving Quick Start](https://github.com/pytorch/serve/blob/master/README.md#serve-a-model) - Basic server usage tutorial
 * [Model Archive Quick Start](https://github.com/pytorch/serve/tree/master/model-archiver#creating-a-model-archive) - Tutorial that shows you how to package a model archive file.
 * [Installation](https://github.com/pytorch/serve/blob/master/README.md#install-torchserve) - Installation procedures
-* [Serving Models](server.md) - Explains how to use torchserve
-   * [REST API](rest_api.md) - Specification on the API endpoint for TorchServe
+* [Serving Models](server.md) - Explains how to use TorchServe
+* [REST API](rest_api.md) - Specification on the API endpoint for TorchServe
+* [gRPC API](grpc_api.md) - TorchServe supports gRPC APIs for both inference and management calls
 * [Packaging Model Archive](https://github.com/pytorch/serve/tree/master/model-archiver#torch-model-archiver-for-torchserve) - Explains how to package model archive file, use `model-archiver`.
 * [Inference API](inference_api.md) - How to check for the health of a deployed model and get inferences
 * [Management API](management_api.md) - How to manage and scale models
 * [Logging](logging.md) - How to configure logging
 * [Metrics](metrics.md) - How to configure metrics
-   * [Metrics API](metrics_api.md) - How to configure metrics API
+* [Prometheus and Grafana metrics](metrics_api.md) - How to configure metrics API with Prometheus formatted metrics in a Grafana dashboard
+* [Captum Explanations](https://github.com/pytorch/serve/blob/master/captum/Captum_visualization_for_bert.ipynb) - Built in support for Captum explanations for both text and images
 * [Batch inference with TorchServe](batch_inference_with_ts.md) - How to create and serve a model with batch inference in TorchServe
 * [Workflows](workflows.md) - How to create workflows to compose Pytorch models and Python functions in sequential and parallel pipelines
-* [Model Zoo](model_zoo.md) - List of pre-trained model archives ready to be served for inference with TorchServe.
-* [Examples](https://github.com/pytorch/serve/tree/master/examples) - Many examples of how to package and deploy models and workflows with TorchServe
 
-## Advanced Features
 
-* [Advanced configuration](configuration.md) - Describes advanced TorchServe configurations.
-* [Custom Service](custom_service.md) - Describes how to develop custom inference services.
-* [Unit Tests](https://github.com/pytorch/serve/tree/master/ts/tests#testing-torchserve) - Housekeeping unit tests for TorchServe.
-* [Benchmark](https://github.com/pytorch/serve/tree/master/benchmarks#torchserve-model-server-benchmarking) - Use JMeter to run TorchServe through the paces and collect benchmark data.
-* [TorchServe on Kubernetes](https://github.com/pytorch/serve/blob/master/kubernetes/README.md#torchserve-on-kubernetes) -  Demonstrates a Torchserve deployment in Kubernetes using Helm Chart.
 
 ## Default Handlers
 
 * [Image Classifier](https://github.com/pytorch/serve/blob/master/ts/torch_handler/image_classifier.py) - This handler takes an image and returns the name of object in that image
 * [Text Classifier](https://github.com/pytorch/serve/blob/master/ts/torch_handler/text_classifier.py) - This handler takes a text (string) as input and returns the classification text based on the model vocabulary
 * [Object Detector](https://github.com/pytorch/serve/blob/master/ts/torch_handler/object_detector.py) - This handler takes an image and returns list of detected classes and bounding boxes respectively
 * [Image Segmenter](https://github.com/pytorch/serve/blob/master/ts/torch_handler/image_segmenter.py)- This handler takes an image and returns output shape as [CL H W], CL - number of classes, H - height and W - width
+
+## Examples
+
+* [HuggingFace Language Model](https://github.com/pytorch/serve/blob/master/examples/Huggingface_Transformers/Transformer_handler_generalized.py) - This handler takes an input sentence and can return sequence classifications, token classifications or Q&A answers
+* [Multi Modal Framework](https://github.com/pytorch/serve/blob/master/examples/MMF-activity-recognition/handler.py) - Build and deploy a classifier that combines text, audio and video input data
+* [Dual Translation Workflow](https://github.com/pytorch/serve/tree/master/examples/Workflows/nmt_tranformers_pipeline) - 
+* [Model Zoo](model_zoo.md) - List of pre-trained model archives ready to be served for inference with TorchServe.
+* [Examples](https://github.com/pytorch/serve/tree/master/examples) - Many examples of how to package and deploy models with TorchServe
+* [Workflow Examples](https://github.com/pytorch/serve/tree/master/examples/Workflows) - Examples of how to compose models in a workflow with TorchServe
+
+## Advanced Features
+
+* [Advanced configuration](configuration.md) - Describes advanced TorchServe configurations.
+* [A/B test models](https://github.com/pytorch/serve/blob/master/docs/use_cases.md#serve-models-for-ab-testing) - A/B test your models for regressions before shipping them to production
+* [Custom Service](custom_service.md) - Describes how to develop custom inference services.
+* [Encrypted model serving](management_api.md/encrypted-model-serving) - S3 server side model encryption via KMS
+* [Snapshot serialization](https://github.com/pytorch/serve/blob/master/plugins/docs/ddb_endpoint.md) - Serialize model artifacts to AWS Dynamo DB
+* [Benchmarking and Profiling](https://github.com/pytorch/serve/tree/master/benchmarks#torchserve-model-server-benchmarking) - Use JMeter or Apache Bench to benchmark your models and TorchServe itself
+* [TorchServe on Kubernetes](https://github.com/pytorch/serve/blob/master/kubernetes/README.md#torchserve-on-kubernetes) -  Demonstrates a Torchserve deployment in Kubernetes using Helm Chart supported in both Azure Kubernetes Service and Google Kubernetes service
+* [mlflow-torchserve](https://github.com/mlflow/mlflow-torchserve) - Deploy mlflow pipeline models into TorchServe
+* [Kubeflow pipelines](https://github.com/kubeflow/pipelines/tree/master/samples/contrib/pytorch-samples) - Kubeflow pipelines and Google Vertex AI Managed pipelines

docs/Troubleshooting.md

@@ -1,4 +1,4 @@
-## Troubleshooting guide.
+## Troubleshooting Guide
 Refer to this section for common issues faced while deploying your Pytorch models using Torchserve and their corresponding troubleshooting steps.
 
 * [Deployment and config issues](#deployment-and-config-issues)

docs/apis.rst

@@ -0,0 +1,10 @@
+.. toctree::
+  :caption: APIs:
+  
+  grpc_api
+  inference_api
+  management_api
+  metrics_api
+  rest_api
+  workflow_inference_api
+  workflow_management_api

docs/code_coverage.md

@@ -1,3 +1,5 @@
+# Code Coverage
+
 ## To check branch stability run the sanity suite as follows
  - Install dependencies (if not already installed)
    For CPU:

docs/conf.py

@@ -78,7 +78,7 @@
 
 
 # The master toctree document.
-master_doc = 'index'
+master_doc = 'contents'
 
 # General information about the project.
 project = 'PyTorch/Serve'
@@ -129,7 +129,7 @@
 #
 html_theme_options = {
     'pytorch_project': 'audio',
-    'collapse_navigation': False,
+    'collapse_navigation': True,
     'display_version': True,
     'logo_only': True,
 }

docs/contents.rst

@@ -0,0 +1,30 @@
+.. toctree::
+  :maxdepth: 0
+  :numbered:
+  :caption: Contents:
+  :titlesonly:
+  
+  index
+  Troubleshooting
+  batch_inference_with_ts
+  code_coverage
+  configuration
+  custom_service
+  default_handlers
+  logging
+  metrics
+  model_zoo
+  request_envelopes
+  server
+  snapshot
+  sphinx/requirements
+  torchserve_on_win_native
+  torchserve_on_wsl
+  use_cases
+  workflows
+
+.. toctree::
+  :maxdepth: 0
+  :caption: APIs
+  
+  apis

docs/custom_service.md

@@ -1,3 +1,5 @@
+# Custom Service
+
 ## Contents of this Document
 
 * [Custom handlers](#custom-handlers)
@@ -380,7 +382,7 @@ class ModelHandler(object):
         self.device = torch.device("cuda:" + str(properties.get("gpu_id")) if torch.cuda.is_available() else "cpu")
 ```
 
-# Installing model specific python dependencies
+## Installing model specific python dependencies
 
 Custom models/handlers may depend on different python packages which are not installed by-default as a part of `TorchServe` setup.
 

docs/default_handlers.md

@@ -38,16 +38,16 @@ For more details see [examples](https://github.com/pytorch/serve/tree/master/exa
 
 For a more comprehensive list of available handlers make sure to check out the [examples page](https://github.com/pytorch/serve/tree/master/examples)
 
-# Common features
+## Common features
 
-## index_to_name.json
+### index_to_name.json
 
 `image_classifier`, `text_classifier` and `object_detector` can all automatically map from numeric classes (0,1,2...) to friendly strings. To do this, simply include in your model archive a file, `index_to_name.json`, that contains a mapping of class number (as a string) to friendly name (also as a string). You can see some examples here:
 - [image_classifier](https://github.com/pytorch/serve/tree/master/examples/image_classifier/index_to_name.json)
 - [text_classifier](https://github.com/pytorch/serve/tree/master/examples/text_classification/index_to_name.json)
 - [object_detector](https://github.com/pytorch/serve/tree/master/examples/object_detector/index_to_name.json)
 
-# Contributing
+### Contributing
 We welcome new contributed handlers, if your usecase isn't covered by one of the existing default handlers please follow the below steps to contribute it
 1. Write a new class derived from [BaseHandler](https://github.com/pytorch/serve/blob/master/ts/torch_handler/base_handler.py). Add it as a separate file in `ts/torch_handler/`
 2. Update `model-archiver/model_packaging.py` to add in your classes name