Effect of setting intra and inter-op parallelism paramters to deep learning models

[saeid93]

The two set_num_interop_threads, get_num_interop_threads CPU threading variables explained here for PyTroch and for TensorFlow have a huge impact on CPU inferencing time, e.g. for Resnet 18 TorchVistion image model under one CPU core assignment it results in the following difference in latencies (before applying and after applying these values). I think it is worthwhile adding this two variable as a configurable setting variable at least for Huggingface runtime that is using deep models (and I can validate I have seen the same trends for many Huggingface pipeline models too).

[adriangonz]

Hey @saeid93 ,

That's an interesting one.

Do you have any heuristic in mind to decide the value for those params?

[saeid93]

Hey @adriangonz,

Based on the PyTorch Documentation it seems the number of cores is a good heuristic for both variables. That's what I used in the above example.

I think the best option would be to add such config to the user side for Huggingface server in a way that it adds the value of these two parameters as a config value in the setting folder. If not set then the default value can be the number of CPUs. However, this can be further optimized but I think that will be out of the scope for MLServer, however, if you are interested this paper provides an in-depth investigation on the topic.