Why not use oneDNN?

[tonyzhao-jt]

Describe the issue

I'm curious about your choice of MKL for sgemm instead of utilizing oneDNN.

[jgong5]

I'm curious about your choice of MKL for sgemm instead of utilizing oneDNN.

Thanks for the question. We make decisions of what performance library to use based on their execution performance for particular cases. Feel free to let us know if you have any concern or any particular problems you face.

[AngryLoki]

@jgong5 , could you elaborate, please, what exactly works better in MKL?

When I checked bf16 and f32 gemm, performance was very similar between these libraries. Sometimes not just similar, but the same, as it very clearly uses the same code.

See image with comparison of dnnl_gemm_bf16bf16f32 and cblas_gemm_bf16bf16f32

From my pov (as developer/maintainer) there is:

1. no control over runtime in MKL; at some point incompatibilities with glibc may cause issues with older Linux distributions (not even speaking about musl environments)
2. performance issues with AMD (I can provide a fix in PR, if you want)
3. different licenses; requirement to add an extra proprietary repositories to install MKL.
4. oneDNN developers saying that "expected to be on par or faster than their counterparts in oneMKL" in oneDNN does not build with Intel oneMKL as BLAS Vendor uxlfoundation/oneDNN#1733 (comment) and later stopping supporting MKL as a vendor in documentation
5. better communication channel in oneDNN (subjectively)
6. at a glance, it seems trivial to switch from mkl gemm to dnnl_* gemm (pytorch uses sparse and fft routines from MKL, but I don't see these in intel-extension-for-pytorch).

[jgong5]

@AngryLoki Thanks for the detailed info and comments. Yes, oneDNN GEMM is as performant as MKL GEMM in many cases. We use MKL GEMM in areas that it works better or oneDNN doesn't have support yet:

1. The fp32 MKL packed GEMM. This one was in IPEX and now part of the stock PyTorch.
2. HGEMM (I don't think oneDNN supports it now).
3. Tiny GEMMs called in loop nest where MKL has lower overhead than oneDNN primitives. We use MKL GEMM in this case to optimize scaled dot product attention kernel (previously in IPEX and now also upstreamed to PyTorch).