InferenceSession init hangs for multiple models on Azure Kubernetes Service

[alexbarl]

Our team trains and applies various ML models by using ML.NET.
Some scenarios use onnx through Microsoft.ML.OnnxRuntime nuget package.
Recently we’ve started moving our services to Azure Kubernetes (AKS) were we see that initialization calls to ApplyOnnxModel consistently hang for all models.

Configuration

1. Microsoft.ML version 1.7.1 packages
2. Microsoft.ML.OnnxRuntime.Managed and Microsoft.ML.OnnxRuntime version 1.10 packages (for compatibility with services on the previous platform)
3. The container is based on image from mcr.microsoft.com/dotnet/aspnet:6.0 - Debian GNU/Linux 11 (bullseye)
4. From what I see AKS host OS is Ubuntu 18.04.6 LTS (Bionic Beaver)
5. UTF-8 locale is configured through Dockerfile.
   Without locale configuration an error message in onnxruntime was traced.

The initialization hangs somewhere inside libonnxruntime.so code.

Managed code call stack

[alexbarl]

Was able to capture some partial information from one of the hanged threads with gdb
Not sure if it is helpful:

(gdb) thread 49
[Switching to thread 49 (Thread 0x7f4506ffd700 (LWP 496))]
#0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
38 ../sysdeps/unix/sysv/linux/x86_64/syscall.S: No such file or directory.
(gdb) bt
#0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1 0x00007f47435b4316 in ?? () from /app/runtimes/linux-x64/native/libonnxruntime.so
#2 0x00007f47435b2f43 in ?? () from /app/runtimes/linux-x64/native/libonnxruntime.so
#3 0x00007f47435b3433 in ?? () from /app/runtimes/linux-x64/native/libonnxruntime.so
#4 0x00007f474342874c in ?? () from /app/runtimes/linux-x64/native/libonnxruntime.so
#5 0x00007f4743428fa6 in ?? () from /app/runtimes/linux-x64/native/libonnxruntime.so
#6 0x00007f474342b518 in ?? () from /app/runtimes/linux-x64/native/libonnxruntime.so
#7 0x00007f8f56ee6ea7 in start_thread (arg=) at pthread_create.c:477
#8 0x00007f8f56ad5aef in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

[pranavsharma]

Can you isolate AKS and ML.NET and test with ORT's C# APIs only?

[yuslepukhin]

What is the version of Onnxruntime ? Would it be possible to share the model as well?

[alexbarl]

Dmitri Smirnov (@yuslepukhin), we use onnxruntime 1.10.0 (for compatibility with previous system).
I think there was an issue running a newer version, but I can test it again with 1.12.1 or 1.13.0 versions.
Regarding the model, I'll check if it can be shared internally.

[alexbarl]

I've ran a debug build that includes #13313 change by Dmitri Smirnov (@yuslepukhin)'s (thank you Pranav Sharma (@pranavsharma))

InferenceSession creation didn't crash, but for every loaded model, generated the below error message:

2022-10-17 00:54:22.369587475 [E:onnxruntime:CSharpOnnxRuntime, env.cc:231 ThreadMain] pthread_setaffinity_np failed for thread: 139629160285952, mask: 1, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.

Not sure what is the impact of the above error or how to fix it.

[yuslepukhin]

This means that the thread affinity fails to be set for at least one thread. Usually this happens when the number of threads is not specified. If the desired number of threads is explicitly set, we do not set affinity.
Before, it would throw at thread creation, and we would hang. Not setting thread affinity would not affect the functionality but may affect performance. We may need to refine the code further, the thread ID is not queried correctly.

[yuslepukhin]

The fix from PR #13313 is fully present in the current main branch. The code now:

Logs affinity failures as errors instead of throwing (line 228–233)
Catches all exceptions in the thread main function (ORT_CATCH(...) on line 237) to prevent thread death
Sets affinity inside the thread itself (pthread_self() on line 221) rather than from the creator