Skip to content

Commit

Permalink
Update jetson docs for Jetpack 22.03 (triton-inference-server#4120)
Browse files Browse the repository at this point in the history
* Allow jetson test to confiure default shm size for python backend

* Update jetson docs for 22.03
  • Loading branch information
Hemant Jain authored and mc-nv committed Apr 4, 2022
1 parent 826942f commit 0edb10a
Show file tree
Hide file tree
Showing 2 changed files with 16 additions and 14 deletions.
23 changes: 10 additions & 13 deletions docs/jetson.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@

# Triton Inference Server Support for Jetson and JetPack

A release of Triton for [JetPack 4.6.1](https://developer.nvidia.com/embedded/jetpack)
A release of Triton for [JetPack 5.0](https://developer.nvidia.com/embedded/jetpack)
is provided in the attached tar file in the [release notes](https://github.com/triton-inference-server/server/releases).

![Triton on Jetson Diagram](images/triton_on_jetson.png)
Expand All @@ -54,11 +54,11 @@ The TensorRT execution provider however is supported.
On JetPack, although HTTP/REST and GRPC inference protocols are supported, for edge
use cases, direct [C API integration](inference_protocols.md#c-api) is recommended.

You can download the `.tar` files for Jetson from the Triton Inference Server
You can download the `.tgz` file for Jetson from the Triton Inference Server
[release page](https://github.com/triton-inference-server/server/releases) in the
_"Jetson JetPack Support"_ section.

The `.tar` file contains the Triton server executable and shared libraries,
The `.tgz` file contains the Triton server executable and shared libraries,
as well as the C++ and Python client libraries and examples.

## Installation and Usage
Expand Down Expand Up @@ -113,12 +113,9 @@ pip3 install --upgrade expecttest xmlrunner hypothesis aiohttp pyyaml scipy ninj
Apart from these PyTorch dependencies, the PyTorch wheel corresponding to the release must also be installed (for build and runtime):

```
pip3 install --upgrade https://developer.download.nvidia.com/compute/redist/jp/v461/pytorch/torch-1.11.0a0+17540c5-cp36-cp36m-linux_aarch64.whl
pip3 install --upgrade https://developer.download.nvidia.com/compute/redist/jp/v50/pytorch/torch-1.12.0a0+2c916ef.nv22.3-cp38-cp38-linux_aarch64.whl
```

**Note**: Seeing a core dump when using numpy 1.19.5 on Jetson is a [known issue](https://github.com/numpy/numpy/issues/18131).
We recommend using numpy version 1.19.4 or earlier to work around this issue.

The following dependencies must be installed before building Triton client libraries/examples:

```
Expand All @@ -127,23 +124,23 @@ apt-get install -y --no-install-recommends \
jq
pip3 install --upgrade wheel setuptools cython && \
pip3 install --upgrade grpcio-tools numpy==1.19.4 attrdict pillow
pip3 install --upgrade grpcio-tools numpy attrdict pillow
```

**Note**: OpenCV 4.1.1 is installed as a part of JetPack. It is one of the dependencies for the client build.
**Note**: OpenCV 4.2.0 is installed as a part of JetPack. It is one of the dependencies for the client build.

**Note**: When building Triton on Jetson, you will require a recent version of cmake.
We recommend using cmake 3.21.0. Below is a script to upgrade your cmake version to 3.21.0.
We recommend using cmake 3.21.1. Below is a script to upgrade your cmake version to 3.21.1.

```
apt remove cmake
wget -O - https://apt.kitware.com/keys/kitware-archive-latest.asc 2>/dev/null | \
gpg --dearmor - | \
tee /etc/apt/trusted.gpg.d/kitware.gpg >/dev/null && \
apt-add-repository 'deb https://apt.kitware.com/ubuntu/ bionic main' && \
apt-add-repository 'deb https://apt.kitware.com/ubuntu/ focal main' && \
apt-get update && \
apt-get install -y --no-install-recommends \
cmake-data=3.21.0-0kitware1ubuntu18.04.1 cmake=3.21.0-0kitware1ubuntu18.04.1
cmake-data=3.21.1-0kitware1ubuntu20.04.1 cmake=3.21.1-0kitware1ubuntu20.04.1
```

### Runtime Dependencies for Triton
Expand Down Expand Up @@ -174,7 +171,7 @@ apt-get update && \
jq
pip3 install --upgrade wheel setuptools && \
pip3 install --upgrade grpcio-tools numpy==1.19.4 attrdict pillow
pip3 install --upgrade grpcio-tools numpy attrdict pillow
```

The PyTorch runtime depenencies are the same as the build dependencies listed above.
Expand Down
7 changes: 6 additions & 1 deletion qa/L0_infer/test.sh
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,11 @@ fi
TF_VERSION=${TF_VERSION:=1}
TEST_JETSON=${TEST_JETSON:=0}

# Default size (in MB) of shared memory to be used by each python model
# instance (Default is 64MB)
DEFAULT_SHM_SIZE_MB=${DEFAULT_SHM_SIZE_MB:=64}
DEFAULT_SHM_SIZE_BYTES=$((1024*1024*$DEFAULT_SHM_SIZE_MB))

# On windows the paths invoked by the script (running in WSL) must use
# /mnt/c when needed but the paths on the tritonserver command-line
# must be C:/ style.
Expand All @@ -91,7 +96,7 @@ else
fi

# Allow more time to exit. Ensemble brings in too many models
SERVER_ARGS_EXTRA="--exit-timeout-secs=${SERVER_TIMEOUT} --backend-directory=${BACKEND_DIR} --backend-config=tensorflow,version=${TF_VERSION} --backend-config=python,stub-timeout-seconds=120"
SERVER_ARGS_EXTRA="--exit-timeout-secs=${SERVER_TIMEOUT} --backend-directory=${BACKEND_DIR} --backend-config=tensorflow,version=${TF_VERSION} --backend-config=python,stub-timeout-seconds=120 --backend-config=python,shm-default-byte-size=${DEFAULT_SHM_SIZE_BYTES}"
SERVER_ARGS="--model-repository=${MODELDIR} ${SERVER_ARGS_EXTRA}"
SERVER_LOG_BASE="./inference_server"
source ../common/util.sh
Expand Down

0 comments on commit 0edb10a

Please sign in to comment.