Okay, can you hear me?
Excellent. Thank you.
So it's a pleasure to be here.
I'm on goal this amazing speakers today.
So I'm Thanos.
I'm a search fellow at the University of Manchester.
I'm part of the Tornado VM team.
And today I will talk about polyglot language implementations,
which enable programming languages like Ruby, Python,
and to run on top of the JVM, along with Java, of course.
And I will try to make a step forward
and show you how they can harness GPU acceleration from the JVM.
I'll start a little bit with the polyglot programming,
which has been here for many years,
but in a sense it has been reignited by the advent of
the Truffle framework from Graal VM.
And in a sense it enables multiple programming languages
to run on top of the JVM and interoperate.
So that means that one Java class file can invoke a Python function
and the Python program can invoke a Java method.
Well, this is very interesting.
It comes with many advantages.
But what about GPU programming?
Well, GPUs from Java.
Well, this is not a thing yet.
That's why we have been motivated at the University of Manchester
and we have done all this research in the past eight years
and we have created Tornado VM.
Here is a link to the resources of Tornado VM
with all the presentations that explain the programming model.
Because my goal today is not to go very deep,
to dive into the Tornado VM very deep,
but to present the interoperability
with the other programming languages
and how they can use GPU acceleration from the JVM.
So Tornado VM is an open source
plug-in to existing JDK distributions.
It is compatible with JDK 21, as you will see later.
And it has some very cool features.
So it has a platform agnostic API.
So developers, they don't need to know GPU programming,
FPGA programming.
It comes with an optimizing compiler.
So we extend GRAL with new phases
that they can take Java methods
and compile them to GPU code.
We have a feature of dynamic reconfiguration at runtime,
which means that the method execution can be migrated
from a GPU back to the JVM
and then go to the FPGA if it is appropriate.
And with the latest release 1.0,
we have enabled support for off-heap data types.
So data can be allocated off-heap
with a foreign function and memory API.
And this is the API that Mauricio described earlier today.
So feel free to follow Tornado VM in Twitter
to engage with the website and of course to fork
and try our examples which are open sourcing GitHub.
So I spoke a little bit about off-heap data types,
so I'll give an introduction, an example,
because I'm not going to dive very into the API.
So here we see two snapshots of code.
On the left side, we see a main method
that contains the allocation of float array
by using primitive types,
but is allocated as an object, in a sense, on-heap.
So to migrate from such an allocation
to the new allocation API that's exposed by the Tornado API,
we have created the float array object
that inside it can allocate memory
by using the memory segment of the foreign function API.
And it will allocate this memory off-heap.
So this memory segment could be used directly from the GPU
without the need to worry about GC collections and this stuff.
And the cool part is that even if you don't use GPU programming,
even if you don't want to execute on GPUs,
you can still use this API to allocate memory off-heap.
And here is a link that explains more.
I hope it's visual from your side.
If not, you will find my slides online in the Fosdome webpage.
So the motivation for today is that
Graal VM enables interoperability between programming languages
like Ruby, JavaScript, and other programming languages.
And Tornado VM enables hardware acceleration for Java.
So what if we can combine them together
and harness GPU acceleration from all these programming languages
that are running on top of Trafl?
Let's have a dive into the tech flow.
So in this slide, I present a software stack
from Graal VM for Trafl.
So on the top, we see the Trafl framework
and many implementations of polyglot runtimes
like Graalpy, Graal.js, Trafl Ruby.
And others because Trafl enables also programming language
implementers to create their own programming languages
by using the Java API.
So I have grouped Python, Ruby, JavaScript, and Node.js
in this side of the slide.
And then beneath them, there is the Graal VM Zit compiler,
so an optimizing compiler from Graal.
So Java is also running on top of the JVM, of course.
And all these languages, they start
in the interpreted mode, and once they reach a hot state,
then the optimizing compiler kicks in.
And the cool part with such a polyglot implementation
that enables polyglot programming
is that there is, for the compiler enthusiasts,
there is one Graal IR.
So the nodes, at runtime, they are rewritten.
That means that it can adjust.
So if we kick in a Python function,
then the node can be rewritten,
and the Graal compiler will take a shape
and will emit at the assembly code that will run on the CPU.
So this solution offers the interoperability
and offers the execution among different CPU
instruction set architectures.
But what if we have this heterogeneous hardware,
like GPUs, FPGAs, which are available
in some systems and servers?
Well, then we'll have Tornado VM
that enables Java methods to be compiled for GPUs, FPGAs, etc.
Tornado VM has its own JIT compiler,
which is an extension, a superset, I would say, of Graal,
the Graal compiler, that it is enhanced with new phases
in the compiler to automatically specialize
the code from a method for GPU acceleration and FPG acceleration.
So at the backbone of the compiler,
we have three backends at the moment.
We have OpenCL backend, CUDA, and SPV.
And such a solution would enable many things.
So if you want to learn more about the APIs,
you can scan this QR code.
And the code that is implemented with Tornado VM,
it can harness besides the off-hip data types,
it can also harness the execution with a Tornado VM profiler.
If you want to learn more about the characteristics
of your application, you can see how many data
will be copying in the GPU memory,
how expensive is the IEO maybe,
because this could be very critical
for the performance of the system.
And you can customize even how many,
how the data transfers will be performed.
Because, for example, if you have a method
that consumes redoneally data,
then maybe you need to copy the data once,
instead of copying the data every time you execute the kernel.
Okay, so let's jump to the deployment.
As I said, Tornado VM is compatible
with different JDK distributions,
so it's not a JVM, it is a plugin for JDK distributions.
So it can be seen as a library, in a sense,
because it offers an API in Java.
And it is compatible with all these distributions.
And on the other side, we have the compiler backends
that makes it compatible with
different heterogeneous hardware accelerators.
We can emit vectorized code for multi-core CPU execution
through OpenCL.
We can run with different GPUs and FPGAs.
In this particular talk, I will focus on GraVM,
because we want to leverage polyglot,
and NVIDIA GPUs, because I have created Docker images
that they run on the NVIDIA GPUs.
Now, regarding the GraVM deployment,
I will focus in this slide in GraL Python,
which is one implementation of polyglot runtime.
This is shipped in two different standalone versions, releases.
So we have the native standalone,
which comes with the native image.
And then we have the JVM standalone that enables
the execution of Python programs on top of the JVM,
and it has also the JVM compiler.
The version that we tested is the 23.1,
because tornado VM is compatible with this version of GraL.
And here you can see that we have downloaded the community,
and that's JVM.
So we have the JVM standalone version downloaded.
Well, we need the JVM standalone,
because we want to run with tornado VM,
and tornado VM will extend the GraL VM compiler.
So this is the reason.
The problem is that we tried it,
and the JVM distribution is shipped with the JVM standalone,
with a compiler built that it is built with libgral.
So this comes with not many compiler modules,
and that breaks the consistency for tornado VM.
When we tried it.
And this is because they wanted the image,
the footprint to be lower, which makes sense,
but it broke the compatibility with tornado VM.
The good part on this story is that GraL is very active.
The GraL community is very active in Slack workspace,
so we managed to figure out what was the problem.
On the bad side is that the solution was to build
a GraL Pi and GraL VM from source,
which was quite painful.
And in order to avoid this pain for anyone
who wants to try this work,
we decided to build a Docker image
that has inside GraL Pi, tornado VM,
and we have also added the NVIDIA driver.
So if you have a Linux machine or any machine
that has an NVIDIA GPU,
and you have also the NVIDIA container toolkit in this machine,
then you will be able to run this image.
The Docker file, the image is open source in GitHub.
And on the other side,
you can see the QR code that has the acceleration library.
So the code that we have implemented
in the examples module of tornado VM
for the computation part that we will upload on the GPU,
like K-means, matrix multiplication,
and those are the examples.
But there are also other compute examples
that we have in the GitHub.
And you can also pull the Docker image from Docker Hub.
So we will jump into the examples.
So as you see here,
we have the Python and Java with tornado VM.
So we have the Python program that imports Java,
and then it loads the class from the compute examples class
of the tornado VM repository.
And then we have in this Java class that we have loaded,
we have two methods that can be accessed by the Python program.
The first one is the set inputs
that set the actual data points
and the number of packets that will be used for K-means.
And the second one is the run with GPU.
So this will invoke the actual GPU compilation
for GPUs and the GPU execution.
And on the other side,
we have the Java tornado VM,
where we use Java and the tornado VM API
to create these parallel implementations of K-means.
In this slide, you see, well, the steps,
how to clone the repository that contains this Python program.
And we see also the Python program, the K-means.py.
So we see here beneath that we have the invocation
of the actual method functions, Java methods, sorry.
And here is the link for the Java implementation of K-means.
And now if we jump into the Java part,
which contains the computation that will be offloaded on the GPU.
No, before we jump to the computation,
we have the set inputs and I wanted to make a connection
to reflect on the off-heap data types.
So with these two, with a new vector float,
this is an API type that is exposed by tornado VM
and can allocate data vector types off-heap.
And then we'll have the create matrix of clusters
that does perform some initialization of the objects
and also allocate some other data, like the clusters,
which are going to be allocated off-heap as well.
And now we are ready to move into the actual computation part.
So on the left side, you see the run with Java implementation
of this method.
And on the right side, you see the accelerated one
with the tornado VM API.
So as we see here, the actual computation has been
in this method, has been performed by this method.
So they assign clusters.
And the corresponding one on the right side,
that is the tornado VM implementation, is this one.
So in this one, I would like to focus on two parts.
So you can see the task graph implementation.
Task graph is an object exposed by the tornado VM API.
In a sense, task graph enables you to define
what code will go to the GPU.
So what's going to be the actual computation
and what data should be used on the GPU.
So the input data and the output data.
So in a sense, the task graph enables programmers
to define what is going to go to the GPU for execution.
And the second API, once we have done this,
as you can see here, we can define also the data
transfer mode, how often we want data input,
input data or output data to be copied back and forth
from the GPU.
And once we have defined that, we can move to the second part,
which is the execution plan.
So the execution plan is another object
that enables programmers to define
how the execution will take place.
So it could be, for example, with the profiler enabled,
without the profiler enabled, with a custom grid size,
which is defined by the programmer.
And once we have defined how the execution will be done,
will be performed, we are able to execute the actual task graph.
So with execution.execute, it is this part
that enables the actual execution of the code
and the GIT compilation.
So the second time that we will execute the assigned clusters,
well, this is going to be the second time
that we invoke the actual execute of the execution plan.
And the second time that we will invoke the execution plan,
the execution of the execution plan,
this is going to be the time that the code will not be GIT
because it is already GIT.
So the code, the OpenCL code or the CUDA code
will be all retrieved from the code cache of Tornado VM.
So now we can move to the actual example to run.
I have recorded a video that enables the execution
of K-Means and MathExfoom.liblication
because on my MacBook, I don't have an NVIDIA GPU.
So we will fork the actual repository with examples.
And now that we have forked, we will go inside,
we check out the FOSDEM branch.
And this is the Python code that we saw earlier.
So it has these three.
First, we load the class, and then we are able to invoke
the Java code from Python.
And here we will run, first, the Java implementation
and then the GPU accelerated implementation.
We can also pull the Docker image that we have created.
And here in the repository, we have a launcher script
that enables to run.
So at first, we will try the Tornado devices
to query how many NVIDIA GPUs exist in the system.
And here it is the 2000 GPU that exists in my machine at home.
And once we have done this, we will run with Truffle,
the Python program.
So Tornado Truffle, the Truffle flag and Python,
will be able to run the actual Python program.
And we will see here that at first,
it will bring Hello World from Python.
And then we run the Java implementation,
which is a sequential, that I'm with Java.
And then they run with GPU method.
And as we see here, they take the first one, one second,
and the second one, 140 milliseconds.
So here we will try the same example,
but with the thread info, which will enable the printing
of the actual threads that have been used on the GPU.
So as we see here, we have the number of data points
that we passed with the set input.
It has been the number of the global thread size
that is uploaded on the GPU.
And now we move to the second example,
which is the matrix multiplication with Tornado VM.
So in this example, we run five times
the matrix multiplication.
So we see here the execution time
of matrix multiplication on the GPU.
So the first time it was half second,
and then it has moved to three milliseconds.
This is because the first execution,
it involves also the GIT compilation, which is expensive.
Then the second time, third time,
the execution time has been saturated
because it is the actual launching of the code.
Okay, I have showed you example of Python with Gralpy,
but this is not the only one.
We have also the key images for the other programming languages
for JavaScript, Ruby,
and you can find more details in those links
where we have a blog post.
And we explain also the polyglot programming
from Tornado VM.
So now we will try to find the other examples
so now I will jump to the summary of my talk.
So as key takeaways,
I would like to emphasize that GralVM and Traffl
enable Java interoperability with other programming languages
that run on top of the JVM.
Tornado VM afflows Java methods on GPUs, FPGAs,
and multicore CPUs,
so you can create parallel implementations.
And that Tornado VM offers a Java API,
so programmers, they don't need to know GPU programming.
It is a Java API, a Java way to express parallelism.
And we have also new off-hip data types.
So finally, yes, it is possible to create
high-performing implementations of code
for data science libraries in Java,
and reuse them by other programming languages.
This is a slide that summarizes everyone
who has contributed as a research staff
for students at the University of Manchester,
and these images are from our campus.
And this is a surprise that it was taken and it was not raining.
So I would like to invite you to join our community,
follow us in GitHub,
join us in the Tornado VM Slack space
if you have questions,
or if you want to interact with a team for discussions,
and also to try our examples in GitHub.
And in my last slide,
I would like to acknowledge all these research funds
that have supported their work at Tornado VM,
like Elegant and Crip, Tango, Iro and InCode.
So with that, I conclude my talk,
and I think we have time for one or two questions.
Okay, I've got the mic here,
but first, I lived in Manchester for five years,
and it doesn't always rain.
Just mostly.
Thanks for a great talk.
Like one of the first pictures you had showed Tornado VM
in parallel to the GrowlJIT using the JVMCI.
So do you interact directly with JVMCI for generating code?
Correct, yes.
So the JVMCI enables other JIT compilers
to be hooked in the JVM,
and that's how we run, because we extend.
So do you work with the standard JVMCI in upstream or open JDK,
or you need the lab JDK with the latest JVMCI changes?
Because the GrowlJIT compiler,
as far as I know, requires the lab JDK with latest changes.
We work with the standard JVMCI, yes.
Thank you.
So when you write the kernel code in Java,
then is it usually high-level code that you write,
or do you try to write optimized code in Java?
Like usually when you write, let's say, Qtacode,
then you try to write a very specialized,
use warp intrinsics and that kind of stuff.
Is that something that is like in scope for turn out of VM,
or not so much?
No, that's a great question.
Well, to answer this question, we do both.
So we have two APIs.
One is created for Java programmers.
We will have, let's say, a computation that has four loops.
So this is something that you can paralyze
if you don't have data dependency.
So we expose an annotation in this case,
similar to OpenMP.
So you can do add parallel in the four loop
in order to give a hint to the compiler
that this can run in parallel
and will create parallel implementations in OpenCL or CUDA.
And the second part is that if you are familiar with OpenCL
and CUDA and you want to have access to low-level intrinsics,
like, for example, use barriers or local memory,
allocate local memory,
then we'll have a second API, which is called kernel API.
And with that, you can pretty much access every interesting
that exists in OpenCL and CUDA programming from Java.
So personally, I have used the second API
to port existing OpenCL kernels in Java with Tonedo.