Memory leaks and valgrind errors when running with TensorRT

[matthill]

Describe the bug
Running inference either with address sanitization enabled or under Valgrind using TensorRT produces a number of Valgrind errors (a few memory leaks on initialization/destruction, a number of mismatched delete/delete[], and uninitilized values)

Urgency
none

System information

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux Ubuntu 20.04, 18.04, Jetson
• ONNX Runtime installed from (source or binary): source
• ONNX Runtime version: tested on 1.5.2 and 1.7.1
• Python version: none
• Visual Studio version (if applicable): none
• GCC/Compiler version (if compiling from source): 9.3
• CUDA/cuDNN version: 11 / 7.1
• GPU model and memory: RTX 2080, Jetson Tx-2

To Reproduce
Compile software using ONNXRuntime with TensorRT. In this case, the model uses a dynamic batch size.
Either enable address sanitization on the compile, or run under valgrind
valgrind --leak-check=full --show-posssibly-lost=no [binary]

Expected behavior
Output should be clean

Additional context

==132002== LEAK SUMMARY:
==132002== definitely lost: 96 bytes in 2 blocks
==132002== indirectly lost: 48 bytes in 2 blocks

==132002== 72 (48 direct, 24 indirect) bytes in 1 blocks are definitely lost in loss record 3,517 of 5,620
==132002== at 0x483CFE3: operator new(unsigned long) (vg_replace_malloc.c:417)
==132002== by 0x10C5E5BAB: createInferRuntime_INTERNAL (in /usr/lib/libnvinfer.so.7.1.3)
==132002== by 0xFF027C51: nvinfer1::(anonymous namespace)::createInferRuntime(nvinfer1::ILogger&) (NvInferRuntime.h:1906)
==132002== by 0xFF02A191: onnxruntime::TensorrtExecutionProvider::TensorrtExecutionProvider(onnxruntime::TensorrtExecutionProviderInfo const&) (tensorrt_execution_provider.cc:226)
==132002== by 0xFF074DBF: std::_MakeUniqonnxruntime::TensorrtExecutionProvider::__single_object std::make_unique<onnxruntime::TensorrtExecutionProvider, onnxruntime::TensorrtExecutionProviderInfo&>(onnxruntime::TensorrtExecutionProviderInfo&) (unique_ptr.h:857)
==132002== by 0xFF074C66: onnxruntime::TensorrtProviderFactory::CreateProvider() (tensorrt_provider_factory.cc:28)
==132002== by 0x6740220: onnxruntime::IExecutionProviderFactory_Translator::CreateProvider() (provider_bridge_ort.cc:709)
==132002== by 0x590B690: (anonymous namespace)::InitializeSession(OrtSessionOptions const*, std::unique_ptr<onnxruntime::InferenceSession, std::default_deleteonnxruntime::InferenceSession >&) (onnxruntime_c_api.cc:457)
==132002== by 0x590BCC0: OrtApis::CreateSessionFromArray(OrtEnv const*, void const*, unsigned long, OrtSessionOptions const*, OrtSession**) (onnxruntime_c_api.cc:508)

==132002== 72 (48 direct, 24 indirect) bytes in 1 blocks are definitely lost in loss record 3,516 of 5,620
==132002== at 0x483CFE3: operator new(unsigned long) (vg_replace_malloc.c:417)
==132002== by 0x10C5E5BAB: createInferRuntime_INTERNAL (in /usr/lib/libnvinfer.so.7.1.3)
==132002== by 0xFF027C51: nvinfer1::(anonymous namespace)::createInferRuntime(nvinfer1::ILogger&) (NvInferRuntime.h:1906)
==132002== by 0xFF02A191: onnxruntime::TensorrtExecutionProvider::TensorrtExecutionProvider(onnxruntime::TensorrtExecutionProviderInfo const&) (tensorrt_execution_provider.cc:226)
==132002== by 0xFF074DBF: std::_MakeUniqonnxruntime::TensorrtExecutionProvider::__single_object std::make_unique<onnxruntime::TensorrtExecutionProvider, onnxruntime::TensorrtExecutionProviderInfo&>(onnxruntime::TensorrtExecutionProviderInfo&) (unique_ptr.h:857)
==132002== by 0xFF074C66: onnxruntime::TensorrtProviderFactory::CreateProvider() (tensorrt_provider_factory.cc:28)
==132002== by 0x6740220: onnxruntime::IExecutionProviderFactory_Translator::CreateProvider() (provider_bridge_ort.cc:709)
==132002== by 0x590B690: (anonymous namespace)::InitializeSession(OrtSessionOptions const*, std::unique_ptr<onnxruntime::InferenceSession, std::default_deleteonnxruntime::InferenceSession >&) (onnxruntime_c_api.cc:457)
==132002== by 0x590BCC0: OrtApis::CreateSessionFromArray(OrtEnv const*, void const*, unsigned long, OrtSessionOptions const*, OrtSession**) (onnxruntime_c_api.cc:508)

==132002== Conditional jump or move depends on uninitialised value(s)
==132002== at 0xD4347DC7: ??? (in /usr/lib/libcudnn_ops_infer.so.8.0.4)
==132002== by 0xD4337142: ??? (in /usr/lib/libcudnn_ops_infer.so.8.0.4)
==132002== by 0xD4208565: cudnnReduceTensor (in /usr/lib/libcudnn_ops_infer.so.8.0.4)
==132002== by 0x5AE3227: onnxruntime::common::Status onnxruntime::cuda::ReduceComputeCore<float, (cudnnReduceTensorIndices_t)1>(onnxruntime::CUDAExecutionProvider&, onnxruntime::Tensor const&, onnxruntime::cuda::PrepareReduceMetadata&, onnxruntime::Tensor&, cudnnReduceTensorOp_t, std::vector<long, std::allocator > const&, bool, bool, bool, bool, onnxruntime::TensorShape const*) (reduction_ops.cc:597)
==132002== by 0x5ADECF9: onnxruntime::common::Status onnxruntime::cuda::ReduceKernel::ComputeImpl<float, (cudnnReduceTensorIndices_t)1>(onnxruntime::OpKernelContext*, cudnnReduceTensorOp_t) const (reduction_ops.cc:636)
==132002== by 0x5ADDFF5: onnxruntime::cuda::ArgMax::ComputeInternal(onnxruntime::OpKernelContext*) const (reduction_ops.h:97)
==132002== by 0x59F20A0: onnxruntime::cuda::CudaKernel::Compute(onnxruntime::OpKernelContext*) const (cuda_common.h:66)
==132002== by 0x67A90D3: onnxruntime::SequentialExecutor::Execute(onnxruntime::SessionState const&, std::vector<int, std::allocator > const&, std::vector<OrtValue, std::allocator > const&, std::vector<int, std::allocator > const&, std::vector<OrtValue, std::allocator >&, std::unordered_map<unsigned long, std::function<onnxruntime::common::Status (onnxruntime::TensorShape const&, OrtMemoryInfo const&, OrtValue&, bool&)>, std::hash, std::equal_to, std::allocator<std::pair<unsigned long const, std::function<onnxruntime::common::Status (onnxruntime::TensorShape const&, OrtMemoryInfo const&, OrtValue&, bool&)> > > > const&, onnxruntime::logging::Logger const&) (sequential_executor.cc:305)

==132002== Conditional jump or move depends on uninitialised value(s)
==132002== at 0xD4347D20: ??? (in /usr/lib/libcudnn_ops_infer.so.8.0.4)
==132002== by 0xD4337142: ??? (in /usr/lib/libcudnn_ops_infer.so.8.0.4)
==132002== by 0xD4208565: cudnnReduceTensor (in /usr/lib/libcudnn_ops_infer.so.8.0.4)
==132002== by 0x5AE3227: onnxruntime::common::Status onnxruntime::cuda::ReduceComputeCore<float, (cudnnReduceTensorIndices_t)1>(onnxruntime::CUDAExecutionProvider&, onnxruntime::Tensor const&, onnxruntime::cuda::PrepareReduceMetadata&, onnxruntime::Tensor&, cudnnReduceTensorOp_t, std::vector<long, std::allocator > const&, bool, bool, bool, bool, onnxruntime::TensorShape const*) (reduction_ops.cc:597)
==132002== by 0x5ADECF9: onnxruntime::common::Status onnxruntime::cuda::ReduceKernel::ComputeImpl<float, (cudnnReduceTensorIndices_t)1>(onnxruntime::OpKernelContext*, cudnnReduceTensorOp_t) const (reduction_ops.cc:636)
==132002== by 0x5ADDFF5: onnxruntime::cuda::ArgMax::ComputeInternal(onnxruntime::OpKernelContext*) const (reduction_ops.h:97)
==132002== by 0x59F20A0: onnxruntime::cuda::CudaKernel::Compute(onnxruntime::OpKernelContext*) const (cuda_common.h:66)
==132002== by 0x67A90D3: onnxruntime::SequentialExecutor::Execute(onnxruntime::SessionState const&, std::vector<int, std::allocator > const&, std::vector<OrtValue, std::allocator > const&, std::vector<int, std::allocator > const&, std::vector<OrtValue, std::allocator >&, std::unordered_map<unsigned long, std::function<onnxruntime::common::Status (onnxruntime::TensorShape const&, OrtMemoryInfo const&, OrtValue&, bool&)>, std::hash, std::equal_to, std::allocator<std::pair<unsigned long const, std::function<onnxruntime::common::Status (onnxruntime::TensorShape const&, OrtMemoryInfo const&, OrtValue&, bool&)> > > > const&, onnxruntime::logging::Logger const&) (sequential_executor.cc:305)
==132002== by 0x678EE21: onnxruntime::utils::ExecuteGraphImpl(onnxruntime::SessionState const&, onnxruntime::FeedsFetchesManager const&, std::vector<OrtValue, std::allocator > const&, std::vector<OrtValue, std::allocator >&, std::unordered_map<unsigned long, std::function<onnxruntime::common::Status (onnxruntime::TensorShape const&, OrtMemoryInfo const&, OrtValue&, bool&)>, std::hash, std::equal_to, std::allocator<std::pair<unsigned long const, std::function<onnxruntime::common::Status (onnxruntime::TensorShape const&, OrtMemoryInfo const&, OrtValue&, bool&)> > > > const&, ExecutionMode, bool const&, onnxruntime::logging::Logger const&, bool) (utils.cc:492)
==132002== by 0x678F277: onnxruntime::utils::ExecuteGraph(onnxruntime::SessionState const&, onnxruntime::FeedsFetchesManager&, std::vector<OrtValue, std::allocator > const&, std::vector<OrtValue, std::allocator >&, ExecutionMode, bool const&, onnxruntime::logging::Logger const&, bool) (utils.cc:516)
==132002== by 0x598BB3B: onnxruntime::InferenceSession::Run(OrtRunOptions const&, std::vector<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::allocator<std::__cxx11::basic_string<char, std::char_traits, std::allocator > > > const&, std::vector<OrtValue, std::allocator > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::allocator<std::__cxx11::basic_string<char, std::char_traits, std::allocator > > > const&, std::vector<OrtValue, std::allocator >, std::vector<OrtDevice, std::allocator > const) (inference_session.cc:1464)

==132002== Conditional jump or move depends on uninitialised value(s)
==132002== at 0xD4347B1F: ??? (in /usr/lib/libcudnn_ops_infer.so.8.0.4)
==132002== by 0xD4337142: ??? (in /usr/lib/libcudnn_ops_infer.so.8.0.4)
==132002== by 0xD4208565: cudnnReduceTensor (in /usr/lib/libcudnn_ops_infer.so.8.0.4)
==132002== by 0x5AE3227: onnxruntime::common::Status onnxruntime::cuda::ReduceComputeCore<float, (cudnnReduceTensorIndices_t)1>(onnxruntime::CUDAExecutionProvider&, onnxruntime::Tensor const&, onnxruntime::cuda::PrepareReduceMetadata&, onnxruntime::Tensor&, cudnnReduceTensorOp_t, std::vector<long, std::allocator > const&, bool, bool, bool, bool, onnxruntime::TensorShape const*) (reduction_ops.cc:597)
==132002== by 0x5ADECF9: onnxruntime::common::Status onnxruntime::cuda::ReduceKernel::ComputeImpl<float, (cudnnReduceTensorIndices_t)1>(onnxruntime::OpKernelContext*, cudnnReduceTensorOp_t) const (reduction_ops.cc:636)
==132002== by 0x5ADDFF5: onnxruntime::cuda::ArgMax::ComputeInternal(onnxruntime::OpKernelContext*) const (reduction_ops.h:97)
==132002== by 0x59F20A0: onnxruntime::cuda::CudaKernel::Compute(onnxruntime::OpKernelContext*) const (cuda_common.h:66)
==132002== by 0x67A90D3: onnxruntime::SequentialExecutor::Execute(onnxruntime::SessionState const&, std::vector<int, std::allocator > const&, std::vector<OrtValue, std::allocator > const&, std::vector<int, std::allocator > const&, std::vector<OrtValue, std::allocator >&, std::unordered_map<unsigned long, std::function<onnxruntime::common::Status (onnxruntime::TensorShape const&, OrtMemoryInfo const&, OrtValue&, bool&)>, std::hash, std::equal_to, std::allocator<std::pair<unsigned long const, std::function<onnxruntime::common::Status (onnxruntime::TensorShape const&, OrtMemoryInfo const&, OrtValue&, bool&)> > > > const&, onnxruntime::logging::Logger const&) (sequential_executor.cc:305)

==132002== Mismatched free() / delete / delete []
==132002== at 0x484065D: operator delete (vg_replace_malloc.c:938)
==132002== by 0x673D0B2: onnxruntime::ProviderHostImpl::HeapFree(void*) (provider_bridge_ort.cc:296)
==132002== by 0xFF01B3F5: operator delete(void*) (provider_bridge_provider.cc:49)
==132002== by 0xFF2BCC2F: std::experimental::filesystem::v1::__cxx11::path::_Cmpt& std::vector<std::experimental::filesystem::v1::__cxx11::path::_Cmpt, std::allocatorstd::experimental::filesystem::v1::__cxx11::path::_Cmpt >::emplace_back<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::experimental::filesystem::v1::__cxx11::path::_Type, unsigned long&>(std::__cxx11::basic_string<char, std::char_traits, std::allocator >&&, std::experimental::filesystem::v1::__cxx11::path::_Type&&, unsigned long&) (in /storage/projects/alpr/modules/libonnxruntime/build/Linux/Debug/libonnxruntime_providers_tensorrt.so)
==132002== by 0xFF2BCE35: std::experimental::filesystem::v1::__cxx11::path::_M_add_filename(unsigned long, unsigned long) (in /storage/projects/alpr/modules/libonnxruntime/build/Linux/Debug/libonnxruntime_providers_tensorrt.so)
==132002== by 0xFF2BC211: std::experimental::filesystem::v1::__cxx11::path::_M_split_cmpts() (in /storage/projects/alpr/modules/libonnxruntime/build/Linux/Debug/libonnxruntime_providers_tensorrt.so)
==132002== by 0xFF02817F: std::experimental::filesystem::v1::__cxx11::path::_M_append(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&) (fs_path.h:437)
==132002== by 0xFF039980: std::enable_if<std::_and<std::_not<std::is_same<std::remove_cv<std::__cxx11::basic_string<char, std::char_traits, std::allocator > >::type, std::experimental::filesystem::v1::__cxx11::path> >, std::_not<std::is_void<std::remove_pointer<std::__cxx11::basic_string<char, std::char_traits, std::allocator > >::type> >, std::experimental::filesystem::v1::__cxx11::path::__constructible_from<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, void> >::value, std::experimental::filesystem::v1::__cxx11::path>::type& std::experimental::filesystem::v1::__cxx11::path::append<std::__cxx11::basic_string<char, std::char_traits, std::allocator > >(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&) (fs_path.h:273)
==132002== by 0xFF02833D: (anonymous namespace)::GetEnginePath(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&) (tensorrt_execution_provider.cc:41)
==132002== by 0xFF032B0E: onnxruntime::TensorrtExecutionProvider::Provider_Compile(std::vector<onnxruntime::Provider_Node*, std::allocatoronnxruntime::Provider_Node* > const&, std::vector<onnxruntime::NodeComputeInfo, std::allocatoronnxruntime::NodeComputeInfo >&)::{lambda(void*, OrtApi const*, OrtKernelContext*)#3}::operator()(void*, OrtApi const*, OrtKernelContext*) const (tensorrt_execution_provider.cc:1052)
==132002== by 0xFF055D62: std::_Function_handler<onnxruntime::common::Status (void*, OrtApi const*, OrtKernelContext*), onnxruntime::TensorrtExecutionProvider::Provider_Compile(std::vector<onnxruntime::Provider_Node*, std::allocatoronnxruntime::Provider_Node* > const&, std::vector<onnxruntime::NodeComputeInfo, std::allocatoronnxruntime::NodeComputeInfo >&)::{lambda(void*, OrtApi const*, OrtKernelContext*)#3}>::_M_invoke(std::_Any_data const&, void*&&, OrtApi const*&&, OrtKernelContext*&&) (std_function.h:286)
==132002== by 0x6718F60: std::function<onnxruntime::common::Status (void*, OrtApi const*, OrtKernelContext*)>::operator()(void*, OrtApi const*, OrtKernelContext*) const (std_function.h:688)
==132002== Address 0x1305056e0 is 0 bytes inside a block of size 95 alloc'd
==132002== at 0x483CFE3: operator new(unsigned long) (vg_replace_malloc.c:417)
==132002== by 0xFF0563E2: void std::__cxx11::basic_string<char, std::char_traits, std::allocator >::_M_construct<char const*>(char const*, char const*, std::forward_iterator_tag) (basic_string.tcc:219)
==132002== by 0xFF2BCE20: std::experimental::filesystem::v1::__cxx11::path::_M_add_filename(unsigned long, unsigned long) (in /storage/projects/alpr/modules/libonnxruntime/build/Linux/Debug/libonnxruntime_providers_tensorrt.so)
==132002== by 0xFF2BC211: std::experimental::filesystem::v1::__cxx11::path::_M_split_cmpts() (in /storage/projects/alpr/modules/libonnxruntime/build/Linux/Debug/libonnxruntime_providers_tensorrt.so)
==132002== by 0xFF02817F: std::experimental::filesystem::v1::__cxx11::path::_M_append(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&) (fs_path.h:437)
==132002== by 0xFF039980: std::enable_if<std::_and<std::_not<std::is_same<std::remove_cv<std::__cxx11::basic_string<char, std::char_traits, std::allocator > >::type, std::experimental::filesystem::v1::__cxx11::path> >, std::_not<std::is_void<std::remove_pointer<std::__cxx11::basic_string<char, std::char_traits, std::allocator > >::type> >, std::experimental::filesystem::v1::__cxx11::path::__constructible_from<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, void> >::value, std::experimental::filesystem::v1::__cxx11::path>::type& std::experimental::filesystem::v1::__cxx11::path::append<std::__cxx11::basic_string<char, std::char_traits, std::allocator > >(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&) (fs_path.h:273)
==132002== by 0xFF02833D: (anonymous namespace)::GetEnginePath(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&) (tensorrt_execution_provider.cc:41)
==132002== by 0xFF032B0E: onnxruntime::TensorrtExecutionProvider::Provider_Compile(std::vector<onnxruntime::Provider_Node*, std::allocatoronnxruntime::Provider_Node* > const&, std::vector<onnxruntime::NodeComputeInfo, std::allocatoronnxruntime::NodeComputeInfo >&)::{lambda(void*, OrtApi const*, OrtKernelContext*)#3}::operator()(void*, OrtApi const*, OrtKernelContext*) const (tensorrt_execution_provider.cc:1052)
==132002== by 0xFF055D62: std::_Function_handler<onnxruntime::common::Status (void*, OrtApi const*, OrtKernelContext*), onnxruntime::TensorrtExecutionProvider::Provider_Compile(std::vector<onnxruntime::Provider_Node*, std::allocatoronnxruntime::Provider_Node* > const&, std::vector<onnxruntime::NodeComputeInfo, std::allocatoronnxruntime::NodeComputeInfo >&)::{lambda(void*, OrtApi const*, OrtKernelContext*)#3}>::_M_invoke(std::_Any_data const&, void*&&, OrtApi const*&&, OrtKernelContext*&&) (std_function.h:286)
==132002== by 0x6718F60: std::function<onnxruntime::common::Status (void*, OrtApi const*, OrtKernelContext*)>::operator()(void*, OrtApi const*, OrtKernelContext*) const (std_function.h:688)
==132002== by 0x6716B54: onnxruntime::FunctionKernel::Compute(onnxruntime::OpKernelContext*) const (func_kernel.h:41)
==132002== by 0x67A90D3: onnxruntime::SequentialExecutor::Execute(onnxruntime::SessionState const&, std::vector<int, std::allocator > const&, std::vector<OrtValue, std::allocator > const&, std::vector<int, std::allocator > const&, std::vector<OrtValue, std::allocator >&, std::unordered_map<unsigned long, std::function<onnxruntime::common::Status (onnxruntime::TensorShape const&, OrtMemoryInfo const&, OrtValue&, bool&)>, std::hash, std::equal_to, std::allocator<std::pair<unsigned long const, std::function<onnxruntime::common::Status (onnxruntime::TensorShape const&, OrtMemoryInfo const&, OrtValue&, bool&)> > > > const&, onnxruntime::logging::Logger const&) (sequential_executor.cc:305)

==132002== Mismatched free() / delete / delete []
==132002== at 0x483F651: operator delete(void*) (vg_replace_malloc.c:802)
==132002== by 0x59EAA8D: __gnu_cxx::new_allocator<std::unique_ptr<onnxruntime::ComputeCapability, std::default_deleteonnxruntime::ComputeCapability > >::deallocate(std::unique_ptr<onnxruntime::ComputeCapability, std::default_deleteonnxruntime::ComputeCapability >, unsigned long) (new_allocator.h:128)
==132002== by 0x59E70D7: std::allocator_traits<std::allocator<std::unique_ptr<onnxruntime::ComputeCapability, std::default_deleteonnxruntime::ComputeCapability > > >::deallocate(std::allocator<std::unique_ptr<onnxruntime::ComputeCapability, std::default_deleteonnxruntime::ComputeCapability > >&, std::unique_ptr<onnxruntime::ComputeCapability, std::default_deleteonnxruntime::ComputeCapability >, unsigned long) (alloc_traits.h:470)
==132002== by 0x59E326D: std::_Vector_base<std::unique_ptr<onnxruntime::ComputeCapability, std::default_deleteonnxruntime::ComputeCapability >, std::allocator<std::unique_ptr<onnxruntime::ComputeCapability, std::default_deleteonnxruntime::ComputeCapability > > >::_M_deallocate(std::unique_ptr<onnxruntime::ComputeCapability, std::default_deleteonnxruntime::ComputeCapability >, unsigned long) (stl_vector.h:351)
==132002== by 0x59DF6AD: std::_Vector_base<std::unique_ptr<onnxruntime::ComputeCapability, std::default_deleteonnxruntime::ComputeCapability >, std::allocator<std::unique_ptr<onnxruntime::ComputeCapability, std::default_deleteonnxruntime::ComputeCapability > > >::~_Vector_base() (stl_vector.h:332)
==132002== by 0x59DF702: std::vector<std::unique_ptr<onnxruntime::ComputeCapability, std::default_deleteonnxruntime::ComputeCapability >, std::allocator<std::unique_ptr<onnxruntime::ComputeCapability, std::default_deleteonnxruntime::ComputeCapability > > >::~vector() (stl_vector.h:680)
==132002== by 0x67180E9: onnxruntime::GraphPartitioner::Partition(onnxruntime::Graph&, bool, onnxruntime::FuncManager&) const (graph_partitioner.cc:179)
==132002== by 0x5985600: onnxruntime::InferenceSession::TransformGraph(onnxruntime::Graph&, onnxruntime::GraphTransformerManager const&, onnxruntime::ExecutionProviders const&, onnxruntime::KernelRegistryManager&, onnxruntime::InsertCastTransformer const&, onnxruntime::SessionState&) (inference_session.cc:803)
==132002== by 0x59885EB: onnxruntime::InferenceSession::Initialize() (inference_session.cc:1116)
==132002== by 0x590B7EF: (anonymous namespace)::InitializeSession(OrtSessionOptions const, std::unique_ptr<onnxruntime::InferenceSession, std::default_deleteonnxruntime::InferenceSession >&) (onnxruntime_c_api.cc:469)
==132002== by 0x590BCC0: OrtApis::CreateSessionFromArray(OrtEnv const*, void const*, unsigned long, OrtSessionOptions const*, OrtSession**) (onnxruntime_c_api.cc:508)

[stevenlix]

do you mind pointing out which model you are using? I tried faster-rcnn model, which has dynamic shape and many partitions, but haven't seen definitely/indirectly lost in valgrind.

[matthill]

I am able to reproduce using the C sample app and Squeezenet (files attached).

My TensorRT provider is linked as follows:
linux-vdso.so.1 (0x00007ffc337e0000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fc8d4050000)
libcudart.so.11.0 => /usr/local/cuda/lib64/libcudart.so.11.0 (0x00007fc8d3dd2000)
libonnxruntime_providers_shared.so => /storage/projects/alpr/modules/libonnxruntime/build/Linux/Debug/libonnxruntime_providers_shared.so (0x00007fc8d3dcd000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fc8d3daa000)
libnvinfer.so.7 => /usr/lib/libnvinfer.so.7 (0x00007fc8baead000)
libnvinfer_plugin.so.7 => /usr/lib/libnvinfer_plugin.so.7 (0x00007fc8ba419000)
libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007fc8ba238000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fc8ba0e9000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fc8ba0ce000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fc8b9edc000)
/lib64/ld-linux-x86-64.so.2 (0x00007fc8d44c5000)
librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007fc8b9ed1000)
libcublas.so.11 => /usr/local/cuda-11.0/targets/x86_64-linux/lib/libcublas.so.11 (0x00007fc8b407f000)
libcudnn.so.8 => /usr/lib/libcudnn.so.8 (0x00007fc8b3e56000)
libmyelin.so.1 => /usr/lib/libmyelin.so.1 (0x00007fc8b362f000)
libnvrtc.so.11.0 => /usr/local/cuda-11.0/targets/x86_64-linux/lib/libnvrtc.so.11.0 (0x00007fc8b1e44000)
libcublasLt.so.11 => /usr/local/cuda-11.0/targets/x86_64-linux/lib/libcublasLt.so.11 (0x00007fc8a6cb5000)

onnxrt_trt_memsample.tar.gz

To reproduce:
mkdir build && cd build
wget https://github.com/microsoft/onnxruntime/raw/master/nodejs/test/testdata/squeezenet.onnx
cmake ..
make -j8
ASAN_OPTIONS=protect_shadow_gap=0 ./onnx_memtest

Output shows the delete/delete[] mismatch:

Using Onnxruntime C API

==1505436==ERROR: AddressSanitizer: alloc-dealloc-mismatch (operator new [] vs operator delete) on 0x6030007811b0
#0 0x7f77b02d98df in operator delete(void*) (/usr/lib/x86_64-linux-gnu/libasan.so.5+0x1108df)
#1 0x7f77a78ded5f in __gnu_cxx::new_allocator<std::__cxx11::basic_string<char, std::char_traits, std::allocator > >::deallocate(std::__cxx11::basic_string<char, std::char_traits, std::allocator >, unsigned long) /usr/include/c++/9/ext/new_allocator.h:128
#2 0x7f77a78d67b8 in std::allocator_traits<std::allocator<std::__cxx11::basic_string<char, std::char_traits, std::allocator > > >::deallocate(std::allocator<std::__cxx11::basic_string<char, std::char_traits, std::allocator > >&, std::__cxx11::basic_string<char, std::char_traits, std::allocator >, unsigned long) /usr/include/c++/9/bits/alloc_traits.h:470
#3 0x7f77a78ca3eb in std::_Vector_base<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::allocator<std::__cxx11::basic_string<char, std::char_traits, std::allocator > > >::_M_deallocate(std::__cxx11::basic_string<char, std::char_traits, std::allocator >, unsigned long) /usr/include/c++/9/bits/stl_vector.h:351
#4 0x7f77a78c14c1 in std::_Vector_base<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::allocator<std::__cxx11::basic_string<char, std::char_traits, std::allocator > > >::~_Vector_base() /usr/include/c++/9/bits/stl_vector.h:332
#5 0x7f77a78c1516 in std::vector<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::allocator<std::__cxx11::basic_string<char, std::char_traits, std::allocator > > >::~vector() /usr/include/c++/9/bits/stl_vector.h:680
#6 0x7f77a799b279 in onnxruntime::IndexedSubGraph::MetaDef::~MetaDef() /storage/projects/alpr/modules/libonnxruntime/include/onnxruntime/core/graph/indexed_sub_graph.h:27
#7 0x7f77a799b2cf in std::default_deleteonnxruntime::IndexedSubGraph::MetaDef::operator()(onnxruntime::IndexedSubGraph::MetaDef) const /usr/include/c++/9/bits/unique_ptr.h:81
#8 0x7f77a799aa69 in std::unique_ptr<onnxruntime::IndexedSubGraph::MetaDef, std::default_deleteonnxruntime::IndexedSubGraph::MetaDef >::~unique_ptr() /usr/include/c++/9/bits/unique_ptr.h:292
#9 0x7f77a79979d1 in onnxruntime::IndexedSubGraph::~IndexedSubGraph() /storage/projects/alpr/modules/libonnxruntime/include/onnxruntime/core/graph/indexed_sub_graph.h:26
#10 0x7f77a8840795 in onnxruntime::ProviderHostImpl::IndexedSubGraph__operator_delete(onnxruntime::IndexedSubGraph*) /storage/projects/alpr/modules/libonnxruntime/onnxruntime/core/framework/provider_bridge_ort.cc:392
#11 0x7f778823b78c in onnxruntime::IndexedSubGraph::operator delete(void*) /storage/projects/alpr/modules/libonnxruntime/onnxruntime/core/providers/shared_library/provider_interfaces.h:755
#12 0x7f778825d53c in std::default_deleteonnxruntime::IndexedSubGraph::operator()(onnxruntime::IndexedSubGraph*) const /usr/include/c++/9/bits/unique_ptr.h:81
#13 0x7f778825487f in std::unique_ptr<onnxruntime::IndexedSubGraph, std::default_deleteonnxruntime::IndexedSubGraph >::~unique_ptr() /usr/include/c++/9/bits/unique_ptr.h:292
#14 0x7f7788249279 in onnxruntime::TensorrtExecutionProvider::RemoveTensorRTGraphCycles(std::vector<std::pair<std::vector<unsigned long, std::allocator >, bool>, std::allocator<std::pair<std::vector<unsigned long, std::allocator >, bool> > >&, onnxruntime::GraphViewer const&) const (/usr/lib/libonnxruntime_providers_tensorrt.so+0x36279)
#15 0x7f778824a299 in onnxruntime::TensorrtExecutionProvider::GetCapability(onnxruntime::GraphViewer const&, std::vector<onnxruntime::KernelRegistry const*, std::allocator<onnxruntime::KernelRegistry const*> > const&) const (/usr/lib/libonnxruntime_providers_tensorrt.so+0x37299)
#16 0x7f77a88139fe in PartitionOnnxFormatModelImpl /storage/projects/alpr/modules/libonnxruntime/onnxruntime/core/framework/graph_partitioner.cc:201
#17 0x7f77a88152d3 in onnxruntime::GraphPartitioner::PartitionOnnxFormatModel(onnxruntime::Graph&, bool, onnxruntime::FuncManager&, onnxruntime::KernelRegistry&, onnxruntime::GraphPartitioner::Mode, int&) const /storage/projects/alpr/modules/libonnxruntime/onnxruntime/core/framework/graph_partitioner.cc:373
#18 0x7f77a8816951 in onnxruntime::GraphPartitioner::Partition(onnxruntime::Graph&, bool, onnxruntime::FuncManager&, onnxruntime::GraphPartitioner::Mode, std::unordered_map<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, unsigned long, std::hash<std::__cxx11::basic_string<char, std::char_traits, std::allocator > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits, std::allocator > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits, std::allocator > const, unsigned long> > >) const /storage/projects/alpr/modules/libonnxruntime/onnxruntime/core/framework/graph_partitioner.cc:539
#19 0x7f77a7938aab in onnxruntime::InferenceSession::TransformGraph(onnxruntime::Graph&, onnxruntime::GraphTransformerManager const&, onnxruntime::ExecutionProviders const&, onnxruntime::KernelRegistryManager&, onnxruntime::InsertCastTransformer const&, onnxruntime::SessionState&, bool) /storage/projects/alpr/modules/libonnxruntime/onnxruntime/core/session/inference_session.cc:853
#20 0x7f77a793bd9c in onnxruntime::InferenceSession::Initialize() /storage/projects/alpr/modules/libonnxruntime/onnxruntime/core/session/inference_session.cc:1207
#21 0x7f77a78b9a35 in (anonymous namespace)::InitializeSession(OrtSessionOptions const, std::unique_ptr<onnxruntime::InferenceSession, std::default_deleteonnxruntime::InferenceSession >&) (/usr/lib/libonnxruntime.so.1+0x85a35)
#22 0x7f77a78b9cbf in OrtApis::CreateSession(OrtEnv const*, char const*, OrtSessionOptions const*, OrtSession**) (/usr/lib/libonnxruntime.so.1+0x85cbf)
#23 0x55fad9b50af6 in main /tmp/onnxrt_trt_memsample/main.cpp:63
#24 0x7f77a746d0b2 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x270b2)
#25 0x55fad9b5044d in _start (/tmp/onnxrt_trt_memsample/build/onnx_memtest+0x644d)

0x6030007811b0 is located 0 bytes inside of 32-byte region [0x6030007811b0,0x6030007811d0)
allocated by thread T0 here:
#0 0x7f77b02d8b47 in operator new[](unsigned long) (/usr/lib/x86_64-linux-gnu/libasan.so.5+0x10fb47)
#1 0x7f77a883f1d7 in onnxruntime::ProviderHostImpl::HeapAllocate(unsigned long) /storage/projects/alpr/modules/libonnxruntime/onnxruntime/core/framework/provider_bridge_ort.cc:191
#2 0x7f778822f469 in operator new(unsigned long) (/usr/lib/libonnxruntime_providers_tensorrt.so+0x1c469)
#3 0x7f7788233fa7 in __gnu_cxx::new_allocator<std::__cxx11::basic_string<char, std::char_traits, std::allocator > >::allocate(unsigned long, void const*) /usr/include/c++/9/ext/new_allocator.h:114
#4 0x7f7788233706 in std::allocator_traits<std::allocator<std::__cxx11::basic_string<char, std::char_traits, std::allocator > > >::allocate(std::allocator<std::__cxx11::basic_string<char, std::char_traits, std::allocator > >&, unsigned long) /usr/include/c++/9/bits/alloc_traits.h:444
#5 0x7f7788232e3d in std::_Vector_base<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::allocator<std::__cxx11::basic_string<char, std::char_traits, std::allocator > > >::_M_allocate(unsigned long) /usr/include/c++/9/bits/stl_vector.h:343
#6 0x7f7788263712 in void std::vector<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::allocator<std::__cxx11::basic_string<char, std::char_traits, std::allocator > > >::_M_realloc_insert<std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&>(__gnu_cxx::__normal_iterator<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::vector<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::allocator<std::__cxx11::basic_string<char, std::char_traits, std::allocator > > > >, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&) /usr/include/c++/9/bits/vector.tcc:440
#7 0x7f77882586f3 in std::vector<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::allocator<std::__cxx11::basic_string<char, std::char_traits, std::allocator > > >::push_back(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&) /usr/include/c++/9/bits/stl_vector.h:1195
#8 0x7f7788246aed in onnxruntime::TensorrtExecutionProvider::GetSubGraph(std::pair<std::vector<unsigned long, std::allocator >, bool>, onnxruntime::GraphViewer const&) const /storage/projects/alpr/modules/libonnxruntime/onnxruntime/core/providers/tensorrt/tensorrt_execution_provider.cc:681
#9 0x7f7788248f23 in onnxruntime::TensorrtExecutionProvider::RemoveTensorRTGraphCycles(std::vector<std::pair<std::vector<unsigned long, std::allocator >, bool>, std::allocator<std::pair<std::vector<unsigned long, std::allocator >, bool> > >&, onnxruntime::GraphViewer const&) const (/usr/lib/libonnxruntime_providers_tensorrt.so+0x35f23)
#10 0x7f778824a299 in onnxruntime::TensorrtExecutionProvider::GetCapability(onnxruntime::GraphViewer const&, std::vector<onnxruntime::KernelRegistry const, std::allocator<onnxruntime::KernelRegistry const*> > const&) const (/usr/lib/libonnxruntime_providers_tensorrt.so+0x37299)
#11 0x7f77a88139fe in PartitionOnnxFormatModelImpl /storage/projects/alpr/modules/libonnxruntime/onnxruntime/core/framework/graph_partitioner.cc:201
#12 0x7f77a88152d3 in onnxruntime::GraphPartitioner::PartitionOnnxFormatModel(onnxruntime::Graph&, bool, onnxruntime::FuncManager&, onnxruntime::KernelRegistry&, onnxruntime::GraphPartitioner::Mode, int&) const /storage/projects/alpr/modules/libonnxruntime/onnxruntime/core/framework/graph_partitioner.cc:373
#13 0x7f77a8816951 in onnxruntime::GraphPartitioner::Partition(onnxruntime::Graph&, bool, onnxruntime::FuncManager&, onnxruntime::GraphPartitioner::Mode, std::unordered_map<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, unsigned long, std::hash<std::__cxx11::basic_string<char, std::char_traits, std::allocator > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits, std::allocator > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits, std::allocator > const, unsigned long> > >) const /storage/projects/alpr/modules/libonnxruntime/onnxruntime/core/framework/graph_partitioner.cc:539
#14 0x7f77a7938aab in onnxruntime::InferenceSession::TransformGraph(onnxruntime::Graph&, onnxruntime::GraphTransformerManager const&, onnxruntime::ExecutionProviders const&, onnxruntime::KernelRegistryManager&, onnxruntime::InsertCastTransformer const&, onnxruntime::SessionState&, bool) /storage/projects/alpr/modules/libonnxruntime/onnxruntime/core/session/inference_session.cc:853
#15 0x7f77a793bd9c in onnxruntime::InferenceSession::Initialize() /storage/projects/alpr/modules/libonnxruntime/onnxruntime/core/session/inference_session.cc:1207
#16 0x7f77a78b9a35 in (anonymous namespace)::InitializeSession(OrtSessionOptions const, std::unique_ptr<onnxruntime::InferenceSession, std::default_deleteonnxruntime::InferenceSession >&) (/usr/lib/libonnxruntime.so.1+0x85a35)
#17 0x7f77a78b9cbf in OrtApis::CreateSession(OrtEnv const*, char const*, OrtSessionOptions const*, OrtSession**) (/usr/lib/libonnxruntime.so.1+0x85cbf)
#18 0x55fad9b50af6 in main /tmp/onnxrt_trt_memsample/main.cpp:63
#19 0x7f77a746d0b2 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x270b2)

SUMMARY: AddressSanitizer: alloc-dealloc-mismatch (/usr/lib/x86_64-linux-gnu/libasan.so.5+0x1108df) in operator delete(void*)
==1505436==HINT: if you don't care about these errors you may set ASAN_OPTIONS=alloc_dealloc_mismatch=0
==1505436==ABORTING

[chilo-ms]

Hi Matthill, thanks for sharing the C sample code and I can reproduce the same result using Address Sanitizer.
But for valgrind, I can't reproduce the same result as you did. I was wondering which Onnxruntime version were you using?
Also, did valgrind run with the same C sample code you shared? (i.e. valgrind --leak-check=full --show-reachable=no --show-possibly-lost=no ./onnx_memtest)

We had a fix for TRT memory leak recently and probably it solves some part of the memory leaks. But of course we want to handle all memory leak issues.

[stale]

This issue has been automatically marked as stale due to inactivity and will be closed in 7 days if no further activity occurs. If further support is needed, please provide an update and/or more details.

[nicktasios]

I'm having the same issue. I was getting weird results from my network and pinpointed it to the onnxruntime usage of TensorRT. Then I found this issue and tried running with a fixed batch size, which fixed the issue.

[snnn]

This issue has been automatically closed as 'not planned' because it has been marked as 'stale' for more than 30 days without activity. If you believe this is still an issue, please feel free to reopen it.