Usability of installation

[andrii0lomakin]

Good day.
I am thinking about the implementation of the installer for TorandoVM that will allow us to run it on the client side, because current installer has some quirks.

Can I ask you what the most ideal state that I can achieve there if I write my own installer?
Ideally, it should be no different than the installation of ONX Runtime.

As a follow-up question, what steps should I perform to run TornadoVM on the CPU only with hardware acceleration?

[stratika]

Hi @andrii0lomakin, any proposal to improve installation process is very welcome. To put you in context, initially we had a bash script (tornadovm-installer.sh) to download the dependencies for the Linux and macOS systems. Then, we transitioned to a Python script that would enable the script to be executed also on Windows.

I am not familiar with the installation of the ONX Runtime. Can you please provide a proposal for how the installation should look like?

Additionally, what is the system that you describe as the client side? (OS distribution)? We have also the docker images that may be helpful.

As a follow-up question, what steps should I perform to run TornadoVM on the CPU only with hardware acceleration?

I do not understand what does the "only with hardware acceleration" mean in this case. Can you please explain? Do you mean, which steps to run only on the CPU via OpenCL?

[andrii0lomakin]

Hi @stratika
First of all, thank you for your answer.

I mean the following: right now, I need to compile OpenCL headers and OpenJDK.

Ideally, I would like to have an installer where I provide precompiled binaries and copy them into specific locations.
Are there any obstacles to achieving such behavior? I mean, is it really required to compile binaries (for OpenCL headers, for example) on the client side?

My target is to create an installer similar to the Temurin installer, for example, when the process of installation is distilled to one click.
I understand that the user should install CUDA himself. I hope the fallback to the OpenCL backend will be good enough for those who do not want to do that.

[andrii0lomakin]

I do not understand what does the "only with hardware acceleration" mean in this case. Can you please explain? Do you mean, which >steps to run only on the CPU via OpenCL?

Yes, exactly.

[stratika]

I mean the following: right now, I need to compile OpenCL headers and OpenJDK.

Ideally, I would like to have an installer where I provide precompiled binaries and copy them into specific locations. Are there any obstacles to achieving such behavior? I mean, is it really required to compile binaries (for OpenCL headers, for example) on the client side?

My target is to create an installer similar to the Temurin installer, for example, when the process of installation is distilled to one click. I understand that the user should install CUDA himself. I hope the fallback to the OpenCL backend will be good enough for those who do not want to do that.

The only difficulty that I see is if the binary versions have been built with the 3 TornadoVM backends (OpenCL, CUDA, SPIR-V), and the system to be deployed supports only one backend. In this case, it may fail (though I have not tested it). Nonetheless, a solution could be to generate different binaries based on the backends that are supported. Similar to what we do for docker containers. We have different containers for Intel hardware and NVIDIA hardware.

I do not understand what does the "only with hardware acceleration" mean in this case. Can you please explain? Do you mean, which >steps to run only on the CPU via OpenCL?

Yes, exactly.

If the OpenCL driver for the CPU is installed, then the CPU would be recognized as a valid device where the TornadoExecutionPlan could run. I know that on the Linux system that I use, I have installed the oneAPI Base Toolkit, and if I source the /opt/intel/oneapi/setvars.sh, then I can see my Intel CPU as an OpenCL device. I have not tried it on Windows yet.

[andrii0lomakin]

Got it.
Thank you, @stratika. It confirms my hypotheses.

I find availability of easy to use (one-click-away so to speak), installer is very important for popularity of TornadoVM, so I hope to achieve such state of installer.

[jjfumero]

The "one-click" solution would be ideal. We have been simplifying the install process quite a lot (at the beginning was very cumbersome). The OpenCL headers are needed to compile the JNI part (this is required for the OpenCL and SPIR-V backends).
@andrii0lomakin, what do you have in mind? create a binary per backend and combinations of backends? Can you expand on this?

[andrii0lomakin]

Hi @jjfumero

My colleagues and I are discussing the usage of TornadoVM as the base for the LLM (and not only LLM) inference framework.
But as a result, we obviously "infect" our clients with TVM, they infect their clients, and so on by supply chain. So, the usability of installation becomes vital.

We are more than capable of buying powerful installer tools, but we need to understand to which point we will be able to simplify the installation process.

As for the moment, the idea is to prebuild binaries for all popular platforms and provide them during the installation process, along with JARs. Ideally, it should be just another JDK build. But, AFAIK, we can not embed CUDA tools in the same installer or LevelZero driver.