You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
This repository was archived by the owner on Aug 7, 2025. It is now read-only.
* Update index.md
Update to fix a broken link in index.md where the trailing .md is cut off from the management_api.md. Added an anchor link to force the .md to show up.
* Update to index.md
Update to index.md to fix several links ending in .md that sphinx is breaking. Added anchor links to each link and a corresponding anchor in the affected doc. Tested locally and seems to be working.
* Update inference_api.md
* Updated typos
Fixed typos and updated wordslist.txt
* Update wordlist.txt
* FAQs Updates
Updated a couple of broken links on the FAQ site per issue #2204
* updates to resolve links
* Update some links in index
Updated some links in index.md to go to the pytorch html page instead of github. This is a nicer fix for the .md sphinx issue
---------
Co-authored-by: sekyonda <7411+sekyonda@users.noreply.ghe.oculus-rep.com>
Co-authored-by: lxning <23464292+lxning@users.noreply.github.com>
Co-authored-by: Ankith Gunapal <agunapal@ischool.Berkeley.edu>
Various models are provided in Torchserve out of the box. Checkout out Torchserve [Model Zoo](https://github.com/pytorch/serve/blob/master/docs/model_zoo.md) for list of all available models. You can also check out the [examples](https://github.com/pytorch/serve/tree/master/examples) folder.
28
+
Various models are provided in Torchserve out of the box. Checkout out Torchserve [Model Zoo](model_zoo.md) for list of all available models. You can also check out the [examples](https://github.com/pytorch/serve/tree/master/examples) folder.
29
29
30
30
### Does Torchserve support other models based on programming languages other than python?
31
31
No, As of now only python based models are supported.
@@ -40,39 +40,39 @@ If a model converts international language string to bytes, client needs to use
### Can I run Torchserve APIs on ports other than the default 8080 & 8081?
49
49
Yes, Torchserve API ports are configurable using a properties file or environment variable.
50
-
Refer [configuration.md](configuration.md) for more details.
50
+
Refer to [configuration](configuration.md) for more details.
51
51
52
52
53
53
### How can I resolve model specific python dependency?
54
54
You can provide a `requirements.txt` while creating a mar file using "--requirements-file/ -r" flag. Also, you can add dependency files using "--extra-files" flag.
55
-
Refer [configuration.md](configuration.md) for more details.
55
+
Refer to [configuration](configuration.md) for more details.
56
56
57
57
### Can I deploy Torchserve in Kubernetes?
58
58
Yes, you can deploy Torchserve in Kubernetes using Helm charts.
59
-
Refer [Kubernetes deployment ](../kubernetes/README.md) for more details.
59
+
Refer [Kubernetes deployment ](https://github.com/pytorch/serve/blob/master/kubernetes/README.md#torchserve-kubernetes) for more details.
60
60
61
61
### Can I deploy Torchserve with AWS ELB and AWS ASG?
62
62
Yes, you can deploy Torchserve on a multi-node ASG AWS EC2 cluster. There is a cloud formation template available [here](https://github.com/pytorch/serve/blob/master/examples/cloudformation/ec2-asg.yaml) for this type of deployment. Refer [ Multi-node EC2 deployment behind Elastic LoadBalancer (ELB)](https://github.com/pytorch/serve/tree/master/examples/cloudformation/README.md#multi-node-ec2-deployment-behind-elastic-loadbalancer-elb) more details.
63
63
64
64
### How can I backup and restore Torchserve state?
65
65
TorchServe preserves server runtime configuration across sessions such that a TorchServe instance experiencing either a planned or unplanned service stop can restore its state upon restart. These saved runtime configuration files can be used for backup and restore.
66
-
Refer [TorchServe model snapshot](snapshot.md#torchserve-model-snapshot) for more details.
66
+
Refer to [TorchServe model snapshot](snapshot.md) for more details.
67
67
68
68
### How can I build a Torchserve image from source?
69
-
Torchserve has a utility [script](../docker/build_image.sh) for creating docker images, the docker image can be hardware-based CPU or GPU compatible. A Torchserve docker image could be CUDA version specific as well.
69
+
Torchserve has a utility [script](https://github.com/pytorch/serve/blob/master/docker/build_image.sh) for creating docker images, the docker image can be hardware-based CPU or GPU compatible. A Torchserve docker image could be CUDA version specific as well.
70
70
71
71
All these docker images can be created using `build_image.sh` with appropriate options.
72
72
73
73
Run `./build_image.sh --help` for all available options.
74
74
75
-
Refer [Create Torchserve docker image from source](../docker/README.md#create-torchserve-docker-image) for more details.
75
+
Refer to [Create Torchserve docker image from source](https://github.com/pytorch/serve/blob/master/docker/README.md#create-torchserve-docker-image) for more details.
76
76
77
77
### How to build a Torchserve image for a specific branch or commit id?
78
78
To create a Docker image for a specific branch, use the following command:
@@ -91,11 +91,11 @@ The image created using Dockerfile.dev has Torchserve installed from source wher
91
91
TorchServe looks for the config.property file according to the order listed in the [doc](https://github.com/pytorch/serve/blob/master/docs/configuration.md#configproperties-file). There is no override mechanism.
92
92
93
93
### What are model_store, load_models, models?
94
-
- model_store: A mandatory argument during TorchServe start. It can be either defined in config.property or overridden by TorchServe command line option "[--model-store](https://github.com/pytorch/serve/blob/master/docs/configuration.md#command-line-parameters)".
94
+
- model_store: A mandatory argument during TorchServe start. It can be either defined in config.property or overridden by TorchServe command line option "[--model-store](configuration.md)".
95
95
96
-
- load_models: An optional argument during TorchServe start. It can be either defined in config.property or overridden by TorchServe command line option "[--models](https://github.com/pytorch/serve/blob/master/docs/configuration.md#command-line-parameters)".
96
+
- load_models: An optional argument during TorchServe start. It can be either defined in config.property or overridden by TorchServe command line option "[--models](configuration.md)".
97
97
98
-
-[models](https://github.com/pytorch/serve/blob/master/docs/configuration.md#command-line-parameters): Defines a list of models' configuration in config.property. A model's configuration can be overridden by [management API](https://github.com/pytorch/serve/blob/master/docs/management_api.md#register-a-model). It does not decide which models will be loaded during TorchServe start. There is no relationship b.w "models" and "load_models" (ie. TorchServe command line option [--models](https://github.com/pytorch/serve/blob/master/docs/configuration.md#command-line-parameters)).
98
+
-[models](configuration.md): Defines a list of models' configuration in config.property. A model's configuration can be overridden by [management API](management_api.md). It does not decide which models will be loaded during TorchServe start. There is no relationship b.w "models" and "load_models" (ie. TorchServe command line option [--models](configuration.md)).
99
99
100
100
###
101
101
@@ -108,43 +108,43 @@ You can use any tool like Postman, Insomnia or even use a python script to do so
108
108
109
109
### How can I add a custom API to an existing framework?
110
110
You can add a custom API using **plugins SDK** available in Torchserve.
111
-
Refer to [serving sdk](../serving-sdk) and [plugins](../plugins) for more details.
111
+
Refer to [serving sdk](https://github.com/pytorch/serve/tree/master/serving-sdk) and [plugins](https://github.com/pytorch/serve/tree/master/plugins) for more details.
112
112
113
113
### How can pass multiple images in Inference request call to my model?
114
114
You can provide multiple data in a single inference request to your custom handler as a key-value pair in the `data` object.
115
-
Refer [this](https://github.com/pytorch/serve/issues/529#issuecomment-658012913) for more details.
115
+
Refer to [this issue](https://github.com/pytorch/serve/issues/529#issuecomment-658012913) for more details.
You would have to write a custom handler and modify the postprocessing to return the image
124
-
Refer [custom service documentation](custom_service.md#custom-handlers) for more details.
124
+
Refer to [custom service documentation](custom_service.md) for more details.
125
125
126
126
### How to enhance the default handlers?
127
127
Write a custom handler that extends the default handler and just override the methods to be tuned.
128
-
Refer [custom service documentation](custom_service.md#custom-handlers) for more details.
128
+
Refer to [custom service documentation](custom_service.md) for more details.
129
129
130
130
### Do I always have to write a custom handler or are there default ones that I can use?
131
131
Yes, you can deploy your model with no-code/ zero code by using builtin default handlers.
132
-
Refer [default handlers](default_handlers.md#torchserve-default-inference-handlers) for more details.
132
+
Refer to [default handlers](default_handlers.md) for more details.
133
133
134
134
### Is it possible to deploy Hugging Face models?
135
135
Yes, you can deploy Hugging Face models using a custom handler.
136
-
Refer [HuggingFace_Transformers](https://github.com/pytorch/serve/blob/master/examples/Huggingface_Transformers/README.md) for example.
136
+
Refer to [HuggingFace_Transformers](https://github.com/pytorch/serve/blob/master/examples/Huggingface_Transformers/README.md#huggingface-transformers) for example.
A mar file is a zip file consisting of all model artifacts with the ".mar" extension. The cmd-line utility `torch-model-archiver` is used to create a mar file.
145
145
146
146
### How can create mar file using Torchserve docker container?
147
-
Yes, you create your mar file using a Torchserve container. Follow the steps given [here](../docker/README.md#create-torch-model-archiver-from-container).
147
+
Yes, you create your mar file using a Torchserve container. Follow the steps given [here](https://github.com/pytorch/serve/blob/master/docker/README.md#create-torch-model-archiver-from-container).
148
148
149
149
### Can I add multiple serialized files in single mar file?
150
150
Currently `torch-model-archiver` allows supplying only one serialized file with `--serialized-file` parameter while creating the mar. However, you can supply any number and any type of file with `--extra-files` flag. All the files supplied in the mar file are available in `model_dir` location which can be accessed through the context object supplied to the handler's entry point.
@@ -155,7 +155,7 @@ Sample code snippet:
155
155
properties = context.system_properties
156
156
model_dir = properties.get("model_dir")
157
157
```
158
-
Refer [Torch model archiver cli](../model-archiver/README.md#torch-model-archiver-command-line-interface) for more details.
158
+
Refer [Torch model archiver cli](https://github.com/pytorch/serve/blob/master/model-archiver/README.md#torch-model-archiver-command-line-interface) for more details.
Copy file name to clipboardExpand all lines: docs/index.md
+5-6Lines changed: 5 additions & 6 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -4,20 +4,19 @@ TorchServe is a performant, flexible and easy to use tool for serving PyTorch mo
4
4
5
5
6
6
## ⚡ Why TorchServe
7
-
*[Model Management API](https://github.com/pytorch/serve/blob/master/docs/management_api.md#management-api): multi model management with optimized worker to model allocation
8
-
*[Inference API](https://github.com/pytorch/serve/blob/master/docs/inference_api.md#inference-api): REST and gRPC support for batched inference
7
+
*[Model Management API](management_api.md): multi model management with optimized worker to model allocation
8
+
*[Inference API](inference_api.md): REST and gRPC support for batched inference
9
9
*[TorchServe Workflows](https://github.com/pytorch/serve/blob/master/examples/Workflows/README.md#workflow-examples): deploy complex DAGs with multiple interdependent models
* Export your model for optimized inference. Torchscript out of the box, [ORT and ONNX](https://github.com/pytorch/serve/blob/master/docs/performance_guide.md#performance-guide), [IPEX](https://github.com/pytorch/serve/tree/master/examples/intel_extension_for_pytorch), [TensorRT](https://github.com/pytorch/serve/blob/master/docs/performance_guide.md#performance-guide), [FasterTransformer](https://github.com/pytorch/serve/tree/master/examples/FasterTransformer_HuggingFace_Bert)
17
-
*[Performance Guide](https://github.com/pytorch/serve/blob/master/docs/performance_guide.md#performance-guide): builtin support to optimize, benchmark and profile PyTorch and TorchServe performance
16
+
* Export your model for optimized inference. Torchscript out of the box, [ORT and ONNX](https://github.com/pytorch/serve/blob/master/docs/performance_guide.md#performance-guide), [IPEX](https://github.com/pytorch/serve/tree/master/examples/intel_extension_for_pytorch), [TensorRT](performance_guide.md), [FasterTransformer](https://github.com/pytorch/serve/tree/master/examples/FasterTransformer_HuggingFace_Bert)
17
+
*[Performance Guide](performance_guide.md): builtin support to optimize, benchmark and profile PyTorch and TorchServe performance
18
18
*[Expressive handlers](https://github.com/pytorch/serve/blob/master/CONTRIBUTING.md#contributing-to-torchServe): An expressive handler architecture that makes it trivial to support inferencing for your usecase with [many supported out of the box](https://github.com/pytorch/serve/tree/master/ts/torch_handler)
19
-
*[Metrics API](https://github.com/pytorch/serve/blob/master/docs/metrics.md#torchserve-metrics): out of box support for system level metrics with [Prometheus exports](https://github.com/pytorch/serve/tree/master/examples/custom_metrics), custom metrics and PyTorch profiler support
20
-
19
+
*[Metrics API](metrics.md): out of box support for system level metrics with [Prometheus exports](https://github.com/pytorch/serve/tree/master/examples/custom_metrics), custom metrics and PyTorch profiler support
21
20
## 🤔 How does TorchServe work
22
21
23
22
*[Serving Quick Start](https://github.com/pytorch/serve/blob/master/README.md#serve-a-model) - Basic server usage tutorial
0 commit comments