Status of tests of master branch

[andrii0lomakin]

Good day.
Is there any way to check the stability of the master branch? I have several tests failing on my environment (I especially was surprised by HalfFloat tests), but I am not sure if those are known issues or not.

So, I am not sure if I should report them or not.

[mikepapadim]

Hello @andrii0lomakin,

Thank you for your feedback.
Is is possible to report the tests that are failing for you and also the system (driver and hardware) that you are running?
We have 5 tests that they are related with known issues, but none of them is related to HalfFloats.

The known errors are the following 5:

uk.ac.manchester.tornado.unittests.foundation.TestIf.test06
uk.ac.manchester.tornado.unittests.kernelcontext.matrices.TestMatrixMultiplicationKernelContext.mxm2DKernelContext01
uk.ac.manchester.tornado.unittests.kernelcontext.matrices.TestMatrixMultiplicationKernelContext.mxm2DKernelContext02
uk.ac.manchester.tornado.unittests.compute.ComputeTests.testJuliaSets
uk.ac.manchester.tornado.unittests.compute.ComputeTests.testMandelbrot

[andrii0lomakin]

Hi @mikepapadim
I have tested on Windows OpenCL backend on nvidia RTX 4090 https://gist.github.com/andrii0lomakin/9085daafa5e49097074a731b2adf3377
Tests says unsupported for current device but I am pretty sure that fp16 is supported by RTX 4090
There is also an OpenCL compiling error.

[jjfumero]

FP16 depends on the NVIDIA driver you have installed. I usually get it directly from the NVIDIA website and install it manually. We have some cases (e.g., on Linux) in which we installed the driver from the package manager and no FP16 supported. As soon as we installed directly the NVIDIA driver from the NVIDIA website (either the stable-branch or gaming-branch) it worked. Hope this helps.

[andrii0lomakin]

@jjfumero got it. I appreciate the clarification.
Hopefully, all such popular cases will be streamlined by vendors and will be much less painful.

[jjfumero]

I agree. We also struggled with this. I use Fedora and the provided package for NVIDIA, and this did not work for FP16 on my 2060 GPU. Once I reinstalled the driver with the gaming version, it worked! So, do not quote me on this, but it seems there are differences in how these packages are built. We also noticed difference in performance. My take: try to manually install the driver from the NVIDIA website.

[jjfumero]

Regarding the status of the unit-tests, we have a pass rate of >= 98% for OpenCL on Linux. Some of them are expected to fail and we have a white-list for that. It depends on the device. overall memory, drivers & different OpenCL/SPiR-V implementations.
For example, TornadoVM supports FP32 and FP64, but some devices (e.g., Apple M-series, and Intel Graphics) do not support FP64.

If the unit-test is not covered in the white-list, then you can open an issue.

[andrii0lomakin]

@jjfumero Could you point me to the code or maybe a text file so I can check tests that belong to the "white list"?

[jjfumero]

https://github.com/beehive-lab/TornadoVM/blob/master/tornado-assembly/src/bin/tornado-test#L166-L218