Hello, everyone. Good afternoon. I am Thanos from the University of Manchester, and today
I have the pleasure to present you Tornado VM, what is the state of Tornado VM at this
moment. And in fact, I want to focus also on the slogan that's very known to everyone,
the right ones run anywhere for Java. So, I will start with that. So, this is a known
slogan derived since 90s from some micro systems in a way to advertise that Java language and
the JVM in particular, it is a platform that can ensure portability across different CPU
structures and architectures. So, the idea is that programmers can run their code, they
can compile it once, and it can run transparently on different hardware architectures. However,
hardware has changed in the last years. It is evolving, and perhaps this is not sufficient
for the new types of hardware resources that are coming. So, lately we have GPUs and FPGAs
which are coming to complement the power of the CPUs in a way to maximize performance
and reduce the energy consumption. These are good, but there is some challenges that are
deriving, and these are mainly posed in programmability. So, how programmers can harness this power
from these resources. I don't know if you have experience with OpenCL and CUDA, but
mainly these programming models that have been designed for these hardware types to
get access to these hardware types, they are mainly focused on the C and C++ world. So,
there are different programming models from different companies like SQL, one API, NVIDIA
CUDA, OpenCL, which is a standard that can run on all the devices. And if you have FPGA
expertise, then perhaps you can write RTL and Verilog, which is a hardware description
language, but this is very low level. And here we are talking about Java, so we want
to go high level. So, if you are a Java developer, then you use the JVM and you go to the CPU.
If you want to have access to these devices, then you need to write your own native interfaces
in the JNI and then tap into the C and C++ world. But still, you need to be aware of
how these programming models are written. So, you need to be familiar with this. And
this is exactly the problem that Tornado VM has been designed to solve. So, Tornado
VM, it is a plug-in to existing OpenJDK distributions, like Amazon Goreto, Red Hat
Mandrel, Azul Zulu, and others. And the way that it is built, it is to enable hardware
acceleration in an easy manner. So, it offers a Java API and it has inside a JIT compiler
for the hardware devices that are showing this figure. Our compiler inside, it can automatically
translate the Java bytecodes to run on CPUs, multi-code CPUs, GPUs, integrated or discrete
GPUs and FPGAs. And the compiler in the backend, it has three different backend types. It can
emit OpenCLC, PDX, which is the assembly for the CUDA, for the NVIDIA GPUs. And it has
recently also the SPIRV backend, which enables to utilize the level zero dispatcher from the
one API. So, Tornado VM, it is a technology that can be used as a JVM plug-in to enable
hardware acceleration for JVMs. And some of the key features is that it has a lightweight
Java API. It is coded in a platform agnostic manner, so one command can be the same no
matter which device it will be executed to program. And it can transparently, at the
compile time, specialize the code. Because the code that is generated for the GPU, it
is completely different from a code that is generated for an FPGA. So, regarding the compiler,
we have different phases that will be enabled for GPUs and different phases that will be
enabled to specialize the code for an FPGA. Our code is available in GitHub, so we encourage
everyone who wants to have a look to fork it, download, play with examples, or even create
their own examples. And also to come back to us. I mean, feel free to use the discussions
to trigger the discussion if you have questions or to open issues if something is broken in
order to fix it. And we have also available docker images for NVIDIA GPUs and Intel integrated
GPUs. Now, the next part that I want to talk, it is regarding the API. Two weeks ago, we
released a new version of Tornado VM, the version 0.15. And this comes with many new
changes in the API level. So, our goal was to make the API easier for Java programmers
in order to use it in a comprehensive manner. So, to know how to use and how to express
parallelism from Java. But first, I have to make you familiar with the programming model
of Tornado VM. And this programming model comes, it is inspired from the heterogeneous
programming models like OpenCL and CUDA, the way that these programming models are operating.
And in this sense, a Java program can be composed of two parts. The first part is the host code,
where it is the actual core of the Java application. And the second part, it is the accelerated
code, which actually it is the method or the set of methods that will be offloaded for
execution on a GPU. Once we have made this clear, then we can move with the execution
model, which it requires first because the processing will take place on a device. It
will have first to move the data from the host code, from the CPU to the actual device.
Then perform the processing. And once the processing is finished, then the data, the
result, will have to be transferred back to the host code. Now, in Tornado VM, in the
API of Tornado VM, we have exposed the set of objects and annotations for each of these
two parts, the host code and accelerated code. In the host code, we have the task graph object
and the Tornado execution plan. The task graph corresponds to what to run on the GPU. And
the Tornado execution plan, it is how to run on the GPU. And for the accelerated code,
we have a set of annotations and objects that I will show you later. So let's start with
the task graph, what to run. Assuming that you are a Java programmer, then you want to
offload the execution of a method. In this example, method A to the GPU. This method,
it has some input and some output. Now, this method corresponds to what in the Tornado
VM terminology call a task. So each method that will be offloaded for execution on hardware
acceleration, it is a task. And it has the input data and the output data. And then we
have a group of tasks, which is the task graph. Now, task graph can be a group of tasks that
may have dependency or may not have dependency. And the programmers, they want to offload
them all for hardware acceleration. In this particular example, I have put one task in
this task graph. Once we have defined what to run, one question that comes, it is how
often to transfer the data between the host, CPU and the device. And this can have a tremendous
impact because it can affect the data transfer time. So it can have a long execution time.
So it can affect performance, but can also affect energy, the power consumption. So
how to transfer data? It matters. It depends on the pattern of the application. So one
application may need to copy only the first execution if the data are read only, then
always or only in the last execution, for example, for the output for the result. And
here is a code snippet of how the task graph can be used to define this functionality in
the Tornado API. So we create a new object, the task graph. We assign a name, which is
a string. In this particular example, it is TG. And then we utilize the exposed methods
of the API in order to fulfill the execution model. At first, we use the transfer to device,
which has two inputs. The first argument that we put, it is the data transfer mode, which
will be used to trigger how often the data will be moved. In this particular example,
it is the first execution. So only in the first execution, the data will be moved. And
then we have the parameter, which is the input array. The second method, it is the task,
and it defines which method will be used for hardware acceleration. The first parameter,
it is a name, a string, actually, of the method. It could be any name. And this is associated
for the dynamic configuration, which I will show you later. The second parameter, it is
the method reference. So the reference to the method that will be offloaded to the GPU
for acceleration. And then it is the list of parameters of this method that corresponds
to the method signature. And the last method, it is the transfer to host. And this, again,
this method, it is configured the first argument to be the data transfer mode. And this example,
we will copy the data, the output, in every execution. Okay. And once we have defined
the task through the task graph, this task can be appended, can be updated. We can add
a new task, a second task. We can change the way that the data transfers will be triggered
in every execution only in the first execution. Then the next step, it is to define the immutable
state of the task graph. So how to preserve the shape of a task graph. And this is done
by taking a snapshot of the task graph, by using the snapshot method in the task graph
object. Then we retrieve back an immutable task graph. And this means that this can be
used for jit compilation and execution on the hardware. And this ensures that the Java
programmers, they can create different versions of their task graph. They can update it. And
then the code cache that we have in Tornado VM, it can store all these versions. It doesn't
need to recompile and override the generated code. And this is the final step before we
move to the actual execution plan. We have the immutable state of the task graph that
can be modified and the immutable task graph that it cannot be modified anymore. So if
the users they want to do a change, they can still change the task graph and get a new
snapshot for a second version of their code. And now we move to how to run, how to execute
the task graph. And this is done through the execution plan. Here is a snippet of Tornado
execution plan. We create a new object that accepts as input the immutable task graph
that doesn't change anymore. And then we can either directly execute it in the default
execution mode by invoking the execution plan.execute method or we can configure it with some
various optimizations. In this particular example, I have enabled the configuration
to run with dynamic reconfiguration, which is a feature in Tornado VM that will launch
a Java thread to GIT compile and execute the application per device that is available
on the system. So we can have a CPU, a GPU and an FPGA. Java thread will run for all
the devices and then it is triggered with a policy of performance, which means that the
first device that will finish the execution, it will be the best and the rest Java threads
will be killed. Now I have concluded the part of the host
code. We can briefly go to the accelerated code, which is the way to express parallelism
within the kernel, within the method or the Tornado VM task, as we call it. We have two
ways, two APIs. The first one is called loop parallel API. And in a sense, we expose the
parallel annotations that can be used by programmers as a hint to the Tornado VM GIT compiler that
these loops can run in parallel. And the second one is the kernel API, which is an API exposed
to the users through the kernel context object. And in a sense, the meaning of this API, it
is meant for OpenCL and CUDA programmers, or Java programmers who know OpenCL, in a way
to get more freedom on how to code things so they can get access to local memory, which
is the equivalent to the cache memory of the CPU for GPUs. So they have more freedom on
what to express. And in fact, I have used this API to port existing kernels written
in OpenCL and CUDA to Java. For more information, you can use this link, which is the actual
documentation of Tornado VM and describes some examples. I will briefly go to one example
of a matrix multiplication, which I presented last year in FOSDEM. So in this example, we
have the accelerated code and the host code. The matrix multiplication method, it implements
matrix multiplication over a flattened arrays in two dimensions. And the way to annotate
and express parallelism using the add parallel annotation, it would be to add the add parallel
annotation inside the four loops. That means that we indicate that these loops could be
executed in parallel. And now regarding the second API, the kernel API, we would use the
kernel context object. And in particular, we would use the global ID X and Y, which correspond
to the two dimensions that we have. So in a sense, it is like having the thread ID that
will execute on the GPU. Here are some use cases that we used on Tornado VM. And concluding
this talk, I would like to focus on a feature that we implemented in a research project
that we are working. It is called elegant. And the idea is to create a software stack
that unifies development for big data and IoT deployment. And there Tornado VM is used
as a technology to enable acceleration as a service. So we have implemented the REST
API. It is still a prototype. But the programmers, they can write a method. They can specify
a method. They can specify the characteristics of the targeted device. And then the service
will return back OpenCL code that it is meant to run parallel dysfunction. The interesting
part is that this code, the OpenCL code, it is generated to be portable across different
programming languages. So it doesn't only bind to Java. It can run also through C++,
Python because it is OpenCL. And this means that in this particular example, we have Java.
We use OpenZDK. We take the byte code and we pass the byte code to Tornado VM. And Tornado
VM is running on an experimental feature which is called code interoperability mode. And
in this mode, it converts this byte code to OpenCL that can run from any programming language
and run time. Therefore, it is like prototyping in Java for parallel programming.
Wrapping up, we would like to receive feedback. And we are looking also for collaborations
if we can help to port use cases or for any other issues. And summarizing this talk, I
briefly went through the right ones, run anywhere in the context of heterogeneous hardware
acceleration. I have familiarized you with Tornado VM which is an open source project
and the code base is available in GitHub. And I familiarize you with the programming
model of Tornado VM and the new API, how to use it. And more are about to come in the
FUJ blog with a new blog. So finally, just to acknowledge the projects that they have
supported our research in the University of Manchester. And I'm ready for questions.