Hi, everyone. So it's my pleasure to introduce Babis and Anastasios. They're going to give
you the talk on using VXL for hard acceleration in your kernels. Babis, please.
So hello, everyone. I'm Babis. My actual name is Haraldos Minus, but you can just call me
Babis. So we're going to give a talk about hardware acceleration and our effort to having
some support in the unit kernels, and we do that with VXL. So, yeah.
Yeah, Kim, oh, sorry. I forgot about that.
Oh, okay.
Oh, let's forget. Yeah, put that over here over there, and maybe you can just keep it here.
Okay. So, yeah, we already heard from Simon, so we don't have to repeat what the unit kernels
are. There are a lot of projects, and we know that they are promising. It's a promising technology.
We can have very fast boot times, low memory footprint, and some increased security. We
also know some of the use cases for unit kernels, which are usually traditional applications
that you might have heard like web servers and stuff like that. But they have also been
used for NFV, and we think that they are also a good fit for serverless and in general micro
services deployments, either in the cloud or the aids. And we also think that they can
also be a good fit for, especially in this case, for ML and AI applications. And that
sounds a bit weird because, as we know, ML and AI workloads are quite huge and heavy.
So, maybe you have heard about PyTorch, maybe you have heard about TensorFlow. We're not
going to touch them, don't worry. But what we want to say here is that they are very,
very heavy frameworks, very difficult to add support for them. And secondly, we know that
these kind of applications are usually compute-intensive applications that can take a lot of resources.
And for that exact reason, we see that there is also a shift in the hardware that exists
in the data centers, not only in the data centers, but also in the aids. We see devices
that are equipped with a lot of new processing units. Of course, we have the traditional
FPGAs and CPUs, but we also have specialized processing units like TPUs and also some ASICs.
And first of all, as we know, ML and AI workloads cannot be executed in Unicernals, that's for
sure, because there is no support for these frameworks. And secondly, there is no support
for hardware acceleration, so there is not really any benefit if we run it in a CPU.
So, I will give a smaller, I'm going to go through the acceleration stack and how we
can virtualize it with the current approaches. So, in general, what we have, it's pretty
simple. Usually, you have an application which is written in an acceleration framework,
it can be OpenCL, it can be CUDA, it can be TensorFlow, PyTorps, all of these frameworks.
Usually underneath that, you have the operator for the GPU or maybe a runtime for FPGAs.
And then you also have, of course, the device driver which resides inside the kernel. So,
this is what we have to virtualize. And as we know, Unicernals are virtual machines,
so we can use the same techniques that we have for virtual machines, we can also use
them in Unicernals. Some of these techniques are hardware partitioning, para-virtualization
and remote API. So, in the case of hardware partitioning, the hardware accelerator has
the capability to partition itself and we assign this small part of the accelerator
to the VM and the VM can access directly to the hardware accelerator. This has very
good performance. On the other hand, we need to have the entire acceleration stack inside
the VM from the device driver to the application, to the acceleration framework. There is also
the case of also, I forgot to mention here, that this is something that it has to be supported
from the device and a device driver needs also to be in the VM. And in the case of para-virtualization,
these things are getting a bit better because we can have a generic, let's say, device.
And then the hypervisor simply manages the accelerator and then we can have the request
to the accelerator manage from the hypervisor so we don't need to have all these kind of
different drivers for every accelerator inside the VM. On the other hand, we still need to
have the vendor runtime and the application and acceleration framework. In the case of
remote API, we even have a lighter approach. Everything is managed from the servers. This
server might be even locally in the same thing or it can't be a remote server. And what happens
here is that the acceleration framework intercepts the calls from the application and forwards
them to the acceleration framework that resides on the server. This has some performance overhead,
of course, because of the transport that happens. And it's also framework specific. So it has
to be supported, like there is remote CUDA, for example, that supports it. So great, but
what is the best for unicernals? In the case of hardware partitioning, this means that
we have to port the entire software acceleration stack and every device driver to the unicernal,
which is not a good and not an easy task. Again, in para-virtualization, things are
a bit better. We have to port only maybe one driver, but still we need to port all these
acceleration stack. In the case of a remote API, this is something that sounds much more
feasible because we can port only, let's say, remote CUDA, only one framework. But how easy
is that? And it's not easy because, as I said before, these kind of frameworks are huge.
They have very, very big code base. They have dynamic linking, which comes in contrast with
the unicernals and a lot of dependencies. So it's not going to be easy to be porting
any existing unicernal framework right now. So for that, we think that VXL is suitable
for unicernals. So I will give to Tasos to present a bit of how VXL is working.
Thank you. Thank you. So hi from my side, too. I'm going to talk a bit about the framework
that we're building. So we started working on VXL to actually handle the hardware acceleration
virtualization in VMs. So it's not tailored to unicernals. We have been playing with semantically
exposing hardware acceleration functionality from hardware acceleration frameworks to VMs.
And the software stack is shown in the figure. We use a hardware agnostic API, so we expose
the whole function call of the hardware accelerated operation. And we focus on the portability
and on interoperability, meaning that the same binary code originating from the application
can be executed in many type of architectures, and it is decoupled from the hardware specific
implementation. A closer look to the software stack, so we have an application. This application
consumes the VXL API, which has specific support, specific operations. These operations are
mapped through a mapping layer through VXL RT to the relevant plugins, which are shown
in greenish, and they actually are the glue code between the API calls and the hardware
specific implementation, which in this figure resides in the external libraries layer. And
then it's the hardware where it executes whatever there is in the external libraries. So digging
a bit more into how VXL works, so the core library, the core component of VXL exposes
the API to the application and maps the API calls to the relevant hardware plugins, which
by the way are loaded at runtime. These plugins are actually glue code between the API, the
API calls, and the hardware specific implementation. So for example, we have an API call of doing
image classification, image inference in general. The only thing that the application needs
to submit to VXL is, I want to do image classify, this is the image, this is the model, and
the parameters, and blah, blah, blah. And this gets mapped to the relevant plugin implementation.
For instance, in this figure, we can use the judgment inference image classification implementation,
which translates these arguments and this operation to the actual judgment inference
framework provided by NVIDIA that does the image classification operation. Apart from
the hardware specific plugins, we also have the transport layer plugins. So imagine this
same operation, the image inference, could be executed in a VM using a virtual plugin.
So this information, the operation, the arguments, the models, everything will be transferred
to the host machine that will use hardware plugin. So apart from the glue code for the
hardware specific implementation, we also have the VM plugins. We also, some of the
plugins and the API operation support a subset of acceleration frameworks, such as a tensor
flow or PyTorch. And what I mentioned earlier about the virtual plugins, so essentially
what happens is that the request of the operation and the arguments is forwarded to another
instance of the VXL library, either on the hypervisor layer or on a socket interface.
So we currently support two modes of operations. We have a VTIO driver and currently we support
firecracker and chemo. So we load the driver on the VM. This driver transfers the arguments
and the operation to the backend, to the chemo backend or the firecracker backend, which
in turn calls the VXL library to do the actual operation. And the other option is using sockets.
So we load a socket interface, a socket agent on the host. We have the VSOC plugin on the
guest and they communicate over simple sockets. I'm going to hand over to Babi for the Unicernel
stuff.
So how can VXL be used in Unicernels? Actually, it's quite easy compared to any other acceleration
framework that exists. And the thing is that the only thing that we need to do is just
have that VXLRT that you see over there. That's the only thing that we need to port. And this
is a very, very thin layer of a C code. It can be easily ported to any Unicernel that
exists. And we, of course, we need some kind of a transport plugin for forward requests.
So as Tasos already explained, usually the application is the same application that we
can run in the host or in any container or in any VM can be also used in the Unicernel,
the same node changes. And it simply uses a specific API of VXL and then we simply forward
the request to the host and then we have another version of VXL which is in the host
and simply maps to the hardware accelerator framework that is implementing the specific
function.
So this, as I said, this allows us to have the same application running either in the
host, either in the VM without any changes. So it's easy to debug, easy to execute. And
we can also access different kind of hardware, different kind of frameworks that exist. And
we don't need to change their application. We can simply change the configuration in
the host.
So yes, we have another acceleration framework and maybe we can think that this is not going
to be easy to use. But let's take an example and see how we can extend the VXL and see
if it is easier or not. So let's get a typical vector addition example in OpenCL which can
be executed in the CPU or in FPGA. And the steps that usually happen is that we set up
the bitstream in the FPGA and the FPGA starts the reconfiguration. Of course, we transfer
the data to the FPGA. Then we invoke the kernel as soon as it's ready and we also get the
results back to the host. So this is what the application is already doing. So if you
have this application already running in your machine, the only thing that you have to do
is that somehow you need to libyify the application. And that's instead of just exposing an API
to do that.
And the next thing is that you can integrate the library in the VXL as a plug-in. And we
have a very simplistic API that you can use and therefore the application will be seen
as a plug-in for the VXL. Later, you can also update VXL, just adding one more API to the
VXLRT so the application can directly use it with the correct parameters, of course.
So I will give you a sort of demo of how this works, using Unicrout specifically. I will
transfer a bit so we can have maybe in the most classification at first and then we can
see how this, how a BLAST CUDA operation can be executed in the CPU and the GPU without
any changes. And maybe some FPGA if we have time. Okay, this is not good. This is better.
So we are in a typical working environment for Unicraft. We have created our application.
We have a new lib which we are not going to use actually. And we have also Unicraft.
So let's go to here. So this is a repo that we have created. I will show it to you later.
So this is, I want to show you. So here you can see that we only have, we only exposed
nine PFS and we use it because we want to transfer the data inside the Unicrout. So
we are not going to use any network. We are just going to share a directory with the VM.
And the only need that you need to do is to select VXLRT and that's all. As you see,
we don't have any libc because we don't need it for the specific example. So these are
all the applications that are currently running in Unicraft. You can try them out by yourself.
So let's, we're going to use image classification. So we'll take some time, let me, we'll take
some time to build. But I will also try to show you how the application looks like as
soon as it finishes. And it should finish right now, almost. Okay. And that's application.
So as you can see, yeah, we can skip the reading of the file. So this application is quite
simple. Like we have a session that we have to create with VXL with the host. Then we
simply call the, this is the function that is called VXL image classification. It has
the arguments that are also needed. And then we simply release the resources that we have
used. So I will try to do an image classification for this beautiful hedgehog that we have
here. And let's see what's going to happen. Okay. So all these logs that you see here
are from the Deton inference plugin. And we see that we have a hedgehog. So it was identified.
And the thing here is to, you can see that all of these logs are not from the Unicraft.
All of these logs are from the host that is running. I can also show you this small demo
with some operations for arrays using CUDA. So the same here. We're just, we're going
to export the backend. First, we're going to use a no op plugin, which simply doesn't
do anything. You can mostly maybe use the only 40 bug. So we have here the application,
which is a SKM. And you can see that it doesn't do anything because it's just a no op plugin.
It doesn't do anything special. So we can change the configuration in the host and specify
that the backend that we want to use is the actual CUDA implementation for maybe CPU. Yes.
Okay. So then we will run it and you will see that we have the, actually it's a min
max operation. It's not a SKM. And then you can also, we will also run the same thing
in a GPU. Again, we are just in the host again. We can simply change the configuration
and now we start it again, the Unicernal, and we get the result from the GPU. You can
also, all these debug messages, you can remove them of course. So we also have the, yes,
this is also min max still. Now we will go to SKM. Do we have time still? Yeah. Okay.
So yeah, we can just use this. Again, no op, nothing happens. Nothing really special. We
will do the export for, to specify the CPU plugin again. And we will execute and we'll
see that the execution time, it's quite not very big, but it's just remember that number.
And now we will run it in the GPU and you can see here that the execution time is much
better than before. And that's all. We can also solve the, the, the FPGA, which is, okay.
So this is an FPGA, right? So we need to have a bit stream. And this is a black skulls application
by the way. And we will run it natively in the beginning and then we will also run it
in the Unicraft. So first we just run the application natively and you can see all of
the logs and everything of the execution in the FPGA. And then we can, we will see how
this is executed in a Unicernel. So this is, I forgot to solve that, but I will, so it
will explain later what are all of these things. Usually what we have to do is just
to export the VXL backend that we want to use. That's how we configure the host to use
a specific plugin. And then we have the chemo command that I can explain in more details
after this video. Still, this is from the Unicernel now and we access the FPGA and we have the
black skulls operation running there. And we also have one more FPGA application, but
I think you got the point. We have all these links for the videos and everything in our
talk in Fosden. So you can also see them from there. Let me talk a bit about chemo, the
chemo plugin that we have. This is a bit more, this is just from our Apple. So here we need
the chemo which has the vertio backend for VXL. And if Unicraft, for example, had support
for Vsoc, we didn't have to use the vertio backend, so we didn't have to modify chemo.
But since we have no Vsoc support, then we have to use the vertio, and therefore we changed
a bit chemo with adding the backend, as you can see here. And these are all the, you already
know from the previous talk, all the configurations for Unicraft, the command line options. I
will also show you our docs. We have here an extended documentation. You can find how
to run VXL application in VM, how to run it remotely. We also have it, it doesn't show
here, but we also have... Okay. Maybe more. Okay, so here we also have all the things
that you need to do to try it out by yourself in Unicraft. And all of them are open source.
You can check them out, and you can clone them by yourself. So let me return. So currently
VXL has bindings for... We actually released the version 0.5, and currently there is bind...
We have language bindings for C, C++, Python, Rust, and also for TensorFlow. And we have
the plugin API that I talked before about extending VXL. You can also see how it is.
These are all the things that we have tested and we support right now. So from the hypervisor
perspective, we have support for Chemo over Ritio and Vsoc. And for these new Rust VMMs,
like Firecracker, Cloud Hypervisor, and Dragon Ball. Regarding Unicernals, we have working...
It's currently working in Unicraft and in Rembrandt, but we want to also port it in OSV
and maybe some more Unicernal frameworks. And we also have integration with Kubernetes,
Cata containers, and OpenFuzz for serverless deployment. And these are all the acceleration
frameworks that we have tested and to work with VXL. So this is an inference that you
saw that we did the immense classification. We have TensorFlow and PyTorch support, TensorFlow
13, OpenVino, OpenCylo, CUDA that you saw with the other demo. And regarding hardware,
we have tested with GPUs, edge devices like Coral, and also FPGAs. So to sum up, hardware
accelerations are... The software stack of hardware accelerators are huge and complicated
to be ported easily in Unicernals. And we have VXL which is able to abstract the heterogeneity
both in the hardware and in the software. And it sounds like a perfect fit for Unicernals.
So if you want, you can try it out by yourselves. Here are all the links that you can use and
test them out. And we would like to mention that this work is partially funded from two
Horizon projects, Ceraan and 5G Complete. And we would also like to invite you in the
Unicraft hackathon that will take place in Athens at the end of March. And thank you
for your attention. If you have any questions, we will be happy to answer them.
Thank you so much, Babi. So for the third time, we'll welcome you in Athens in late March
for the hackathon. If there are any questions from the audience? Yeah, please. Thank you.
Great stuff. I have a question about the potential future and the performance that we are currently
maybe possibly losing to the usage of API and transport. What do you think is a potential
in more increase of performance given that framework?
Yeah, actually, the transport is actually, yes, it's bottleneck since you have all these
transfers that take place. But we think that at the end, we will have still very good execution
times, very good performance. And it's also important to mention that we can also set
up the environment and everything so you can minimize the transfers. For example, you can
have your model. If you have a TensorFlow model or anything, we are working on how it
can be done and prefetching it before you deploy the function in the host and having
everything there so you don't have to transfer from the VM to the host and vice versa and
all of these things. Actually, if I may intervene, so these are
two issues. The first issue is all the resources, the models, the out-of-band stuff that you
can do in a separate API, in a cloud environment, in a serverless deployment. And the second
thing about the actual transfers for Virtio or Visoc, the thing is that since we semantically
abstract the whole operation, you don't have to do a CUDA, MIMCOPY, CUDA malloc, CUDA something,
set kernel, whatever, and you don't have this latency in the transfer. So it minimizes
the overhead just to the part of copying the data across, so the actual data, the input
data and the output. So this is really, really minimal. So in VMs that we have tested, we
have tested remotely, but the network is not that good, so we need to do more tests there.
But in VMs that we have tested, the overhead is less than 5%. For an image classification
of 32K to a MEG, something like that. So it's really, really small, the overhead for the
transport layer, both Virtio and Visoc. The Visoc part is a bit more because it serializes
the stuff through protobufs and the stack is a bit complicated, but the Virtio stuff
is really super efficient.
Hi, so thank you for the talk. My question would be kind of almost on the same thing,
but from the security perspective. So if we kind of offload a lot of computation out of
the Unicernel to the host again, I guess security, at least the isolation is a thing
to think about. So if you, any words on this topic?
Yeah, you can take it. It's yours.
Okay, we agree. Yes, there are issues with security because essentially you need to run
on Unicernel to be isolated, and now we push the execution to the host. So one of the things
that we have thought about is that when you run that on a cloud environment, the vendor
should make sure that whatever application is supported to be run on the host should
be secure, should be audited. So the user doesn't have all the possibilities available.
They cannot just exec something in the host. They will be able to exec specific stuff that
are audited in libraries in the plug-in system. So one approach is this. Another response
to the security implications is that at the moment you have no opportunity to run from
a Unicernel hardware accelerated workload. So if you want to be able to deploy such an
application somewhere, then you can run isolated. You can use the whole hardware accelerator
and have the same binary that you would deploy in a non-secure environment. So you could
secure the environment, but have this compatibility and software supply mode using a Unicernel,
using this semantic abstraction, let's see.
Any other question? Yeah. Please.
So my question is similar to the first question, but I'm wondering, because you can also do
GPU pass-through via KVM and just pass the GPU to a virtual machine. So I'm wondering
what is the performance difference between doing that and doing it in VR?
Yes. Actually, we want to evaluate that, and we need to evaluate it and see how, for example,
with the even pass-through directly, like exposing the whole GPU to the VM, this could
be also one baseline for the valuation. Currently, I don't remember if we do have any measurements
already.
Would you consider the pass-through case the same as made?
Yeah, but I mean, if we have any, like, okay. Actually, from GPU virtualization, for example,
I'm not sure how many VMs can be supported in one single GPU, for example. I'm not aware
of any solution that can scale to, like, tens of VMs, even tens of VMs. I'm not sure if
there is any existing solution for that. But, yes, we plan it. We want to do some extended
evaluation compared also to some, like, let's say, virtual GPU that exists or even the pass-through
and native execution. We want to do that, and hopefully, we can also publish the results
in our block.
Okay. Thank you.
Any other questions? Yeah.
So, in response to the first security question about, yeah, we are offloading now compute
to the hypervisor and host. So, does it imply that there is a possibility to break out of
the containerization with BXL?
Well, there's, yes, yes, code is going to be executing on the host in a privileged level.
Yes. But the other option is what? So, yeah, there is a trade.
We are actually working. We want to see what available sources we have there. How can we
make it more secure? How we can sandbox it somehow to make it look better? But on the
other hand, like, for example, in FPTAs, there's no MMU, there's nothing. If you run two kernels,
one kernel can access, if you kind of know what to do, one kernel can access all the
memory in the whole FPTAs, for example. So, in one hand, you also need support from the
hardware. And regarding, for example, the software stack, we are looking at it and see
how this can, how can we extend and make it more, at least, increase the difficulty for
having any.
So, for example, in the Cata containers integration that we have, so when you spawn a container,
you sandbox the container in a VM, our agent, the host part of the Excel is running on the
same sandbox, not in the VM, outside the VM. But it runs in the sandbox. So, yes, there
is code executing on the host, but it's in the sandbox. So, it's kind of a tradeoff.
Anything else? Right? If not, thank you, Anastasia. Thank you, Babis. Yeah.