Why GraalVM compiler instance is integrated in JVM CI

[andrii0lomakin]

Good day.
I have a newbie question, though I am curious.

If I got it correctly, the Graall VM compiler is used to create IR and produce a compiled kernel from it.
But this process does not need runtime information from JVM.

Why is an instance of the GrallVM compiler integrated into JVM?
Why not use a separate instance that can be freely modified without fear that it will affect the JIT compilation process or code used in JVM JIT?

What are the advantages of using shared instances of compilers? AFAIK Graal compiler, in a nutshell, takes one array of bytes and converts them into another. So can work as a separate instance not attached to JVMCI.

[andrii0lomakin]

I am asking because I am thinking about an intern project for the winners of the famous code contest (but no promises). If that is possible, then it is better to disjoint those instances that simplify both the installation and usage of TornadoVM.

[jjfumero]

TornadoVM extends the Graal JIT compiler to generate low-level code that be used to continue the execution on heterogenous hardware, including GPUs, and FPGAs. TornadoVM uses Graal JIT as a library, and this compilation & optimization path is only selected when TornadoVM execution is involved.

What does this mean? It means that, if you have TornadoVM expressions (Task-Graphs and execution plans) that are never invoked, you will run with the default compiler, whatever this is (if you choose OpenJDK will be C2, if you choose Graal, it will be Graal for the CPU, etc).

If you run execution plans, only the specialized compilation for heterogenous hardware will use the TornadoVM JIT compiler, The rest of the program will run with the default JIT compiler from the JDK distibution you are running (again, if you choose OpenJDK-based, quite likely is going to be C2).

TornadoVM does not interfere with JVM. It is just another Java program with the capability to compile at runtime for OpenCL, PTX and SPIR-V.

The path is as follows:

1. TornadoVM builds the Graal IR only for tasks in a task-graph.
2. TornadoVM Compiler will perform high-level optimizations (what we call Sketcher). The result of the Sketcher is shared across all TornadoVM backends (OpenCL, PTX and SPIR-V). This is a common IR only for the TornadoVM backends.
3. Once the TornadoVM selects the backend and device to run, it will specialize the IR for the target device. It generates another source, whatever the backend is (Sketch IR -> OpenCL C, Sketch IR -> PTX, Sketch IR -> SPIR-V)
4. Once the target code is generated, it sends the result to the corresponding driver for final compilation.

So the process:

Tasks -> Graaal IR -> Sketch IR -> Specialized IR (per backend) -> Target Code

There is no integration with the the JVM, The rest of the Java application that it is not reachable from the TornadoVM Execution Plans are compiled with the default JDK-vendor compiler.

Regarding Runtime Information, TornadoVM performs a sort of Partial Evalulation, and it can evaluate expressions at runtime to perform more aggressive optimizations (similar to Truffle programs on top of the Graal JIT compiler).

JVMCI (Java Virtual Machine Compiler Interface) is just that, an interface. Graal is an implementation of this interface. Thus, to access Graal, we need to access it through JVMCI. Hope this clarifies your questions.

[andrii0lomakin]

@jjfumero I see. I will read the papers then and come back if I have follow-up questions. What puzzling me right now is that registered plugins causing problems in JVM JIT and fact that registration of GrallVM compiler inside of JVM is mandatory. Anyway let me read those paper first. If I find out that it will be possible to separate JVM JVMCI out of GraalVM compiler instance in principle do not you mind from such contribution ? I will create internship project request in such case in my company.

P.S. I am truly fascinated by TornadoVM, I hope to move it to tier of mass production usage in JVM ecosystem.

[andrii0lomakin]

@jjfumero My intention to simplify TornadoVM installation.

If the registration of the GraalVM compiler in JVMCI is unnecessary, it becomes just a library with precompiled binaries such as JNA. That simplifies the installation process and makes it easier for other developers to use it by embedding in their applications as a library, probably with the addition of several flags.

[andrii0lomakin]

Though I completely understand that even without registration of the GraalVM compiler, there could be some caveats present.

[jjfumero]

Just to understand and be able to help. Can you elaborate on what are you trying to simplify regarding the TornadoVM installation process?

• What is the ideal list of dependencies?
• From the current installation process, what is the step/s that could be simplified?

Note that TornadoVM is just a Java Maven project that contains 3 JNI libs to communicate and dispatch code on modern hardware. The "complexity" comes because it relies on existing drivers (OpenCL | CUDA | Level Zero) to be able to run.

[andrii0lomakin]

Sure. I will need time to prepare detailed answer.
Probably my vision of TornadoVM "installation" process is incorrect. Will do that next week.

[andrii0lomakin]

Hi @jjfumero

I decided to give this conversation the highest priority because I am going to provide a proposal for my management for an internship project dedicated to TornadoVM development. I have a couple of ideas, and one of them is simplification of the installation of TornadoVM , ideally till the level of maven dependency.

So answering your question:

Just to understand and be able to help. Can you elaborate on what are you trying to simplify regarding the TornadoVM installation process?

What is the ideal list of dependencies?
From the current installation process, what is the step/s that could be simplified?
Note that TornadoVM is just a Java Maven project that contains 3 JNI libs to communicate and dispatch code on modern hardware. The >"complexity" comes because it relies on existing drivers (OpenCL | CUDA | Level Zero) to be able to run.

To start to run tests in my Gradle project, I had to write the specialized script https://github.com/babylonml/tornadovm-bugs/blob/main/build.gradle.kts#L80 , it performs the following steps:

1. Enables JVMCI https://github.com/babylonml/tornadovm-bugs/blob/main/build.gradle.kts#L23 . I am still puzzled as to why. There is no compilation in CPU assembly. I am completely aware that JVMCI is just an interface, and that is why I am asking. This concrete question is not about the usage of the interface. It is about integration inside the JVM infrastructure, which, as I understand it at the moment, is not mandatory.
2. Registers Graal compiler as system library https://github.com/babylonml/tornadovm-bugs/blob/95486ccd04817d0b5b71e881e643ba62580dabae/build.gradle.kts#L135 . As I can see, the requirement is triggered by the action above. And in reality, there is a bigger problem Grall JIT does not support Vector API intrinsics, so the performance of any library, such as ours, for example, that relies both on GPU and CPU-based kernels, will be drastically degradated on CPU.
3. Registers upgrade module path to register GrallVM JIT inside JVM https://github.com/babylonml/tornadovm-bugs/blob/main/build.gradle.kts#L110 again to enable GraalVM as a JIT compiler.

Let us stop here. Because it is already a very complicated process that is hard for ordinary developers to follow, and even more than this it makes:

1. Usage of Vector API is practically impossible because of the absence of support from the Grall VM compiler.
2. Makes usage of Java FF API impossible at all because TorandoVM plugin clashes with JVM JIT code generation MemorySegment pluging usage makes it incompatible with usage of MemorySegment API #503.

There are other steps that are not as puzzling for normal developers. But I propose to start with a simplification of the steps above.
That is why I am asking why I do need all of this complexity if:

1. JVMCI is just an interface.
2. TornadoVM does not generate assembly for JVM.

That is an extended version of my original question. Probably you already answered it, in such case, my apologies if I missed it.

So, how do I see the ideal process?
The ideal process should look very similar to the JNA usage process.

1. Users add core dependencies.
2. https://github.com/java-native-access/jna/tree/master/dist add native dependencies and so be it.

To achieve this aim, I am going to create a request for an internship project in my company, but before that, I need to clarify whether it is even possible to not register the GraalVM compiler as a JIT for JVM or for TorandoVM to work it is a mandatory step. If that is a mandatory step, which parts of the JVM infrastructure does TorandoVM use to achieve its target?

[andrii0lomakin]

Just to clarify, all those steps are taken from your python's torando script but not requirements to whom I came myself.

[andrii0lomakin]

About Vector API, Lucene already uses Vector API-based kernels, so any project that relies on Lucene vector search and Vector API speed-up can not use TornadoVM at this stage without losing the performance KPIs.

[jjfumero]

Just a few clarifications:

• TornadoVM extends the Graal JIT using the unmodified Graal Jar files of the corresponding version. To extend the compiler, we need to interact through JVMCI, thus JMCI flag must be enabled. However, you can still run your Java/TornadoVM programs using any other JDK vendor (RH Mandrel, Amazon Corretto, Azul Zulu, MS JDK, OpenJDK, etc), not just GraalVM. When running any other JDK except GraalVM, we need to make sure we run with the same JVMCI version that our extensions to the Graal JIT compiler. JVMCI interface evolves as well as the Graal JIT compiler.

• The shared libraries that TornadoVM compiles and uses correspond to the Java Native Interface (JNI) code to communicate with the GPU/FPGA drivers. This is, sending/receiving data, perfoming buffer allocations, dispatching kernels, creating command queues/streams/command lists, etc. There is no code gen in that library.

• TornadoVM performs two phases for code gen:

  1. From Java bytecode -> Target Code Gen (e.g., OpenCL C).
  2. From target code (e.g., OpenCL -> GPU Binary): this is done via the driver communication.

• TornadoVM compiles just for GPUs/FPGAs/RISC-V accelerators, ARM Mali GPUs, and NVIDIA Jetson Nano. But we can still use new instances of the unmodified Graal JIT to compile down to CPUs. Indeed, some TornadoVM operations perform this way (e.g., reductions, in which we specialised the CPU induction variables in the case we run with data sizes that are not adequate for GPUs) [2] [3]. In this case, the Graal Compiler generates the optimized CPU code.

• You can still run Vector API as soon as you do not run with GraalVM. We have examples [1] of this. What you can't at the moment, is to offload operations from the Vector API to GPUs using the TornadoVM framework. We have tried this in the past in a collaboration with Oracle, but Vector API provides too fine grained operations to offload (there is always the cost of offloading and sending data). Having said that, it might be different ways to accelerate Vector API operations.

[1] https://github.com/jjfumero/llama2.tornadovm.java/blob/main/src/main/java/io/github/mikepapadim/MatrixVectorCollection.java#L41-L89
[2] https://github.com/beehive-lab/TornadoVM/blob/master/tornado-runtime/src/main/java/uk/ac/manchester/tornado/runtime/analyzer/CodeAnalysis.java#L76-L106
[3] https://github.com/beehive-lab/TornadoVM/blob/master/tornado-runtime/src/main/java/uk/ac/manchester/tornado/runtime/analyzer/ReduceCodeAnalysis.java#L454-L473

[andrii0lomakin]

But we can still use new instances of the unmodified Graal JIT to compile down to CPUs. Indeed, some TornadoVM operations perform this way (e.g., reductions, in which we specialized the CPU induction variables in the case we run with data sizes that are not adequate for GPUs) [2] [3]. In this case, the Graal Compiler generates the optimized CPU code.

Let us assume that it is possible to create a separate compiler instance to parse byte code for GPU code generation.
Could you follow the following strategy for CPU code generation?

1. Create a separate standalone instance for parsing of GPU code.
2. Using GrallVM compiler instance registered as JIT to generate optimized CPU code if it exists and fallback to the existing byte code with a warning if it does not exist (since JDK 23 GraalVM compiler is part of JDK and enabled by experimental option).

In such a case, TornadoVM extensions will not conflict with the registered compiler instance, and it will be possible to register TorandoVM as a dependency, provided that precompiled binaries exist, which is doable to provide for most popular platforms out of the box. If users still need to use fallback to the optimized code generation for CPU, they will need to register GraalVM for all releases prior to JDK 23. IMHO, the absence of such a fallback by default will not be a showstopper.

[andrii0lomakin]

You can still run Vector API as soon as you do not run with GraalVM.

I did not claim it was impossible. I meant that it leads to noticeable performance degradation. I have talked with Lucence developers, who confirm my observation. By "it," I mean the usage of GraalVM as a JIT compiler.

Though since JDK 22 there is already progress in support of VectorAPI and I suppose in JDK 23 that will not be a problem.

[andrii0lomakin]

I asked GraalVM team, if that is mandatory step to extend Graal compiler or it can be avoided, and asked for examples of extension of compiler in such case.

I have received answer from GraalVM compiler team.

This is indeed possible and is how we run a lot of the Graal unit tests.

I will try to decouple GPU code generation from JVM CI registration and provide the generation of CPU-based code as an optional feature with a fallback to the standard JDK JIT compiler if the GraalVM instance is absent. If you do not have any objections.

[andrii0lomakin]

I will finish this issue #521 and will start work on this task.

[andrii0lomakin]

If I have your agreement, of course, in such case, I will create a separate issue with a detailed description.