🐛 [Bug] torch_tensorrt::torchscript::compile gets stuck; bug caused by elimination exception

[bjaeger1]

Bug Description

after calling
auto trt_mod = torch_tensorrt::torchscript::compile(module, compile_settings);
the process gets stuck in an infinite(?) loop. I can also observe that the GPU load drops back to 0% after about 1s.

According to this link: #1409 the issue should already have been fixed.

Error message

1 __memmove_avx_unaligned 0x7fff79289cc1
2 std::vectortorch::jit::Use::_M_erase(__gnu_cxx::__normal_iterator<torch::jit::Use *, std::vectortorch::jit::Use>) 0x7fffab48412f
3 torch::jit::Value::replaceFirstUseWith(torch::jit::Value *) 0x7fffab46ff5d
4 torch::jit::Value::replaceAllUsesWith(torch::jit::Value *) 0x7fffab46ffcb
5 torch::jit::EliminateExceptions(torch::jit::Block *) 0x7fffab63c3c9
6 torch::jit::EliminateExceptions(std::shared_ptrtorch::jit::Graph&) 0x7fffab63c999
7 torch_tensorrt::core::lowering::LowerGraph(std::shared_ptrtorch::jit::Graph&, std::vectorc10::IValue&, torch_tensorrt::core::lowering::LowerInfo) 0x7fffd7426b0d
8 torch_tensorrt::core::lowering::Lower(torch::jit::Module const&, std::string, torch_tensorrt::core::lowering::LowerInfo const&) 0x7fffd742a181
9 torch_tensorrt::core::CompileGraph(torch::jit::Module const&, torch_tensorrt::core::CompileSpec) 0x7fffd732b5a8
10 torch_tensorrt::torchscript::compile(torch::jit::Module const&, torch_tensorrt::torchscript::CompileSpec) 0x7fffd7313a04
11 ModelLoader::optimizeWithTensorRT modelloader.cpp 266 0x5ad43c
12 InferenceDisplay::<lambda()>::<lambda()>::operator() inferencedisplay.cpp 1330 0x58c996
13 std::_Function_handler<void(), InferenceDisplay::InferenceDisplay(QWidget *, DataController&)::<lambda()>::<lambda()>>::_M_invoke(const std::_Any_data &) std_function.h 316 0x58c996
14 std::function<void ()>::operator()() const std_function.h 706 0x5cbcca
15 errorwrapper::loading(std::function<void ()>) errorwrapper.cpp 11 0x5cbcca
16 InferenceDisplay::<lambda()>::operator() inferencedisplay.cpp 1333 0x58e127
17 QtPrivate::FunctorCall<QtPrivate::IndexesList<>, QtPrivate::List<>, void, InferenceDisplay::InferenceDisplay(QWidget *, DataController&)::<lambda()>>::call qobjectdefs_impl.h 146 0x58e127
18 QtPrivate::Functor<InferenceDisplay::InferenceDisplay(QWidget *, DataController&)::<lambda()>, 0>::call<QtPrivate::List<>, void> qobjectdefs_impl.h 256 0x58e127
19 QtPrivate::QFunctorSlotObject<InferenceDisplay::InferenceDisplay(QWidget *, DataController&)::<lambda()>, 0, QtPrivate::List<>, void>::impl(int, QtPrivate::QSlotObjectBase *, QObject *, void * *, bool *) qobjectdefs_impl.h 439 0x58e127
20 QMetaObject::activate(QObject *, int, int, void * *) 0x7fff7a163f8f
...

Expected behavior

successful torch-tensorrt optimization of a torchscript model

Environment

• Torch-TensorRT Version: v1.3.0
• PyTorch Version : 1.13.0
• OS: Linux
• PyTorch : libtorch 1.13+cu117
• CUDA version: 11.7
• cudnn version: 8.5.0.96
• TensorRT version: 8.5.2.2

[peri044]

Can you provide us a reproducer script with the model for us to investigate? Also are you seeing this with the latest release 1.3 and master too ?

[bjaeger1]

So far I have tested it only with the latest release torch-tensorrt=1.3.0
With some older libraries the tensorrt optimization was running fine (CUDA11.3, cudnn 8.3.2.44, libtorch-1.10.2+cu113, TensorRT-8.4.0.6 and the torch-tensorrt version for libtorch-1.10.2).

I have built a minimal example with a pretrained model provided by pytorch to reproduce the issue (you only need to update the paths in the .cpp file):
tensorrt_api.zip

might need to set:
export CUDA_HOME=/usr/local/cuda
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda/lib64:/usr/local/cuda/extras/CUPTI/lib64
export PATH=$PATH:$CUDA_HOME/bin
export LD_LIBRARY_PATH=/usr/local/tensorrt/lib:/usr/local/cudnn/lib:/usr/local/libtorch/lib:/usr/local/cuda/lib64

[bjaeger1]

Hi @peri044,
could you reproduce the issue with the provided files?

[bobby-chiu]

Torch-TensorRT works for me when I follow this tutorial (https://developer.nvidia.com/blog/accelerating-inference-up-to-6x-faster-in-pytorch-with-torch-tensorrt/) . But I got the same issues with same traceback for my models. Any ideas to fix this bug? @bjaeger1 @peri044

I use pip to install the related packages as follow:
pip install torch==1.12.1+cu113 torchvision==0.13.1+cu113 torchaudio==0.12.1 --extra-index-url https://download.pytorch.org/whl/cu113 ~/downloads/torch_tensorrt-1.2.0-cp38-cp38-linux_x86_64.whl nvidia-pyindex nvidia-tensorrt==8.4.3.1

Environment

• Python Version : 3.8.16
• OS: Ubuntu 20.04.5 LTS
• Torch-TensorRT Version: v1.2.0
• PyTorch Version : 1.12.1+cu113
• cudnn version: 8.3.2
• TensorRT version: 8.5.3.1

[bjaeger1]

@bobby-chiu
I provided @peri044 the files to reproduce the issue, but I did not get any feedback. My current workaround is to just downgrad libtorch to 1.10.2+cu113 and use the working torch-tensorrt for that version.
other libs:
CUDA: 11.3
CUDNN: 8.3.2.44
tensorrt 8.4.0.6 (from nvidia)

[gcuendet]

I suspect this might be related to issue #1823

[gs-olive]

Hi @bjaeger1 - could you test your model with PR #1859? We recently identified an issue in torch::jit::EliminateExceptions, and fixed it in that PR, which is pending review. This may resolve the issue with your model as well.

[bjaeger1]

Hi @gs-olive - I just tried out the PR #1859.
However I was not able to successfully build torch-tensorrt.

After building the docker image with:

DOCKER_BUILDKIT=1 docker build --build-arg BASE=23.06 --build-arg CUDNN_VERSION=8.9 --build-arg TENSORRT_VERSION=8.6 --build-arg PYTHON_VERSION=3.10 -f docker/Dockerfile -t torch_tensorrt:latest .

and running the container:

nvidia-docker run --gpus all -it --shm-size=8gb --env="DISPLAY" --volume="/tmp/.X11-unix:/tmp/.X11-unix:rw" --name=torch_tensorrt --ipc=host --net=host torch_tensorrt:latest

it fails when checking if torch-tensorrt was compiled successfully:

bazel test //tests/core/conversion/converters:test_activation --compilation_mode=opt --test_output=summary --config use_precompiled_torchtrt --config pre_cxx11_abi

ERROR:
2023/07/06 13:45:54 Downloading https://releases.bazel.build/5.2.0/release/bazel-5.2.0-linux-x86_64...
Extracting Bazel installation...
Starting local Bazel server and connecting to it...
ERROR: /opt/torch_tensorrt/WORKSPACE:57:21: fetching new_local_repository rule //external:libtorch_pre_cxx11_abi: java.io.IOException: The repository's path is "/opt/python3/site-packages/torch/" (absolute: "/opt/python3/site-packages/torch") but this directory does not exist.
ERROR: /opt/torch_tensorrt/tests/core/conversion/converters/BUILD:10:15: //tests/core/conversion/converters:test_activation depends on @libtorch_pre_cxx11_abi//:libtorch in repository @libtorch_pre_cxx11_abi which failed to fetch. no such package '@libtorch_pre_cxx11_abi//': The repository's path is "/opt/python3/site-packages/torch/" (absolute: "/opt/python3/site-packages/torch") but this directory does not exist.
ERROR: Analysis of target '//tests/core/conversion/converters:test_activation' failed; build aborted: Analysis failed
INFO: Elapsed time: 6.027s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (35 packages loaded, 128 targets configured)
FAILED: Build did NOT complete successfully (35 packages loaded, 128 targets configured)
Fetching @local_config_cc; Restarting.
Fetching @googletest; Cloning 703bd9caab50b139428cea1aaff9974ebee5742e of https://github.com/google/googletest

Output when building the docker image:
=> [internal] load build definition from Dockerfile
=> => transferring dockerfile: 5.17kB
=> [internal] load .dockerignore
=> => transferring context: 1.00kB
=> [internal] load metadata for docker.io/nvidia/cuda:11.8.0-devel-ubuntu22.04
=> [internal] load build context
=> => transferring context: 58.40MB
=> [base 1/21] FROM docker.io/nvidia/cuda:11.8.0-devel-ubuntu22.04
=> [base 2/21] RUN test -n "8.6" || (echo "No tensorrt version specified, please use --build-arg TENSORRT_VERSION=x.y to specify a version." && exit 1)
=> [base 3/21] RUN test -n "8.9" || (echo "No cudnn version specified, please use --build-arg CUDNN_VERSION=x.y to specify a version." && exit 1)
=> [base 4/21] RUN apt-get update
=> [base 5/21] RUN apt install -y build-essential manpages-dev wget zlib1g software-properties-common git libssl-dev zlib1g-dev libbz2-dev libreadline-dev libsqlite3-dev wget ca-certificates curl llvm libncurses5-dev xz-utils tk-dev libxml2-dev libxmlsec1-dev libffi-dev
=> [base 6/21] RUN wget -L https://github.com/pyenv/pyenv-installer/raw/master/bin/pyenv-installer && chmod 755 pyenv-installer && bash pyenv-installer && eval "$(pyenv init -)"
=> [base 7/21] RUN pyenv install -v 3.10
=> [base 8/21] RUN pyenv global 3.10
=> [base 9/21] RUN wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-ubuntu2204.pin
=> [base 10/21] RUN mv cuda-ubuntu2204.pin /etc/apt/preferences.d/cuda-repository-pin-600
=> [base 11/21] RUN apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/7fa2af80.pub
=> [base 12/21] RUN apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 536F8F1DE80F6A35
=> [base 13/21] RUN apt-key adv --keyserver keyserver.ubuntu.com --recv-keys A4B469963BF863CC
=> [base 14/21] RUN add-apt-repository "deb https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/ /"
=> [base 15/21] RUN apt-get update
=> [base 16/21] RUN apt-get install -y libcudnn8=8.9* libcudnn8-dev=8.9* 108.6s
=> [base 17/21] RUN apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/3bf863cc.pub
=> [base 18/21] RUN add-apt-repository "deb https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/ /"
=> [base 19/21] RUN apt-get update
=> [base 20/21] RUN apt-get install -y libnvinfer8=8.6.* libnvinfer-plugin8=8.6.* libnvinfer-dev=8.6.* libnvinfer-plugin-dev=8.6.* libnvonnxparsers8=8.6.* libnvonnxparsers-dev=8.6.* libnvparsers8=8.6.* libnvparsers-dev=8.6.*
=> [base 21/21] RUN wget -q https://github.com/bazelbuild/bazelisk/releases/download/v1.16.0/bazelisk-linux-amd64 -O /usr/bin/bazel && chmod a+x /usr/bin/bazel
=> [torch-tensorrt-builder-base 1/3] RUN apt-get install -y python3-setuptools
=> [torch-tensorrt 1/7] COPY . /opt/torch_tensorrt => [torch-tensorrt-builder-base 2/3] RUN apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/3bf863cc.pub
=> [torch-tensorrt-builder-base 3/3] RUN apt-get update && apt-get install -y --no-install-recommends locales ninja-build && rm -rf /var/lib/apt/lists/* && locale-gen en_US.UTF-8
=> [torch-tensorrt-builder 1/6] COPY . /workspace/torch_tensorrt/src
=> [torch-tensorrt-builder 2/6] WORKDIR /workspace/torch_tensorrt/src => [torch-tensorrt-builder 3/6] RUN cp ./docker/WORKSPACE.docker WORKSPACE
=> [torch-tensorrt-builder 4/6] RUN mkdir -p "/opt/python3/" && ln -s "pyenv which python | xargs dirname | xargs dirname/lib/python3.10/site-packages" "/opt/python3/"
=> [torch-tensorrt-builder 5/6] RUN CUDA_BASE_IMG_VERSION_INTERMEDIATE=echo ${BASE_IMG#*:} && CUDA_BASE_IMG_VERSION=echo ${CUDA_BASE_IMG_VERSION_INTERMEDIATE%%-*} && CUDA_MAJOR_MINOR_VERSION=echo ${CUDA_BASE_IMG_VERSION%.*} && rm -fr /usr/local/cuda && ln 0.5s
=> [torch-tensorrt-builder 6/6] RUN bash ./docker/dist-build.sh
=> [torch-tensorrt 2/7] COPY --from=torch-tensorrt-builder /workspace/torch_tensorrt/src/py/dist/ .
=> [torch-tensorrt 3/7] RUN cp /opt/torch_tensorrt/docker/WORKSPACE.docker /opt/torch_tensorrt/WORKSPACE
=> [torch-tensorrt 4/7] RUN pip install -r /opt/torch_tensorrt/py/requirements.txt
=> [torch-tensorrt 5/7] RUN pip install tensorrt==8.6.*
=> [torch-tensorrt 6/7] RUN pip install .whl && rm -fr /workspace/torch_tensorrt/py/dist/ *.whl
=> [torch-tensorrt 7/7] WORKDIR /opt/torch_tensorrt
=> exporting to image
=> => exporting layers
=> => writing image sha256:19df2980268d6c1d5692986c9d743acba55615d3e6cd59101537bf47d72b2d38
=> => naming to docker.io/library/torch_tensorrt:latest

[gs-olive]

Hi @bjaeger1 - I just rebased #1859 onto main, since a few dependencies were incorrect in the existing version. Additionally, regarding the docker build we recommend using the default (BASE_IMG=nvidia/cuda:12.1.1-devel-ubuntu22.04) as the base build; the following is working on my machine (when building from the #1859 PR branch directly):

DOCKER_BUILDKIT=1 docker build --build-arg CUDNN_VERSION=8.9 --build-arg TENSORRT_VERSION=8.6 --build-arg PYTHON_VERSION=3.10 -f docker/Dockerfile -t torch_tensorrt:latest .

Could you try the build again with a fresh pull of that branch?

[bjaeger1]

Hi @gs-olive , thanks for the quick answer!
I just made a fresh pull of your eliminate_exceptions_removal-Branch and built a new image with the default base-image.

However, the command:
bazel test //tests/core/conversion/converters:test_activation --compilation_mode=opt --test_output=summary --config use_precompiled_torchtrt --config pre_cxx11_abi
still fails with the same error message. It can't find /opt/python3/site-packages/torch/ (because there is no such directory).

But the docker-image build process stats:
=> CACHED [torch-tensorrt-builder 4/6] RUN mkdir -p "/opt/python3/" && ln -s "pyenv which python | xargs dirname | xargs dirname/lib/python3.10/site-packages" "/opt/python3/"

[gs-olive]

Hi @bjaeger1 - thanks for the follow-up. I was able to reproduce the issue and I addressed the problem in #2085, which adds the necessary /opt/python3 symlink in the final container. Could you add that code to the Dockerfile at the specified location, rebuild the container, and run:

bazel test //tests/core/conversion/converters:test_activation --compilation_mode=opt --test_output=summary  --config pre_cxx11_abi

(I had to remove --config use_precompiled_torchtrt, since recompilation is needed here to use the C++ APIs).

[bjaeger1]

Hi @gs-olive , after adding the mentioned lines for the symlink to the Dockerfile, I was finally able to successfully build torch_tensorrt. Thanks!
PS: One unit-test failed:
_FAIL: //tests/core/conversion/converters:test_activation (see /root/.cache/bazel/bazel_root/272136ab8790bfb0c01be73c7cbef828/execroot/Torch-TensorRT/bazel-out/k8-opt/testlogs/tests/core/conversion/converters/test_activation/test.log)

I currently have difficulties (linking torch_tensorrt against libtorch fails (undefined references)) when building my minimal example. I'll come back when I fixed that issue.