So, hi everyone. I am Merlin and we're going to talk about lightweight Kubernetes operators
with WebAssembly. So, basically, it's an attempt to lower the memory and CPU footprint of the
Kubernetes control plane. So, I am Merlin. You can also say it in Dutch, Merlin. And
I am a researcher at iMac and I teach at Gantt University. I'm also part of the Ubuntu
Community Council. But right now, I'm here to talk about my research, which is service
orchestration in the cloud and in the edge. And so, it's specifically the edge part of
this research. Edge computing is becoming more and more popular. More and more people
want to run their applications closer to end users on devices inside of users' homes,
for example. And as a result, you have a lot of these people who are coming from a background
of developing cloud applications and who now suddenly want to develop applications that
run on devices, which are very low-powered. And they really like the development experience
of the cloud. They like all the tools. They like the cloud-native experience with tools
like Kubernetes, for example. But as most of you might know, Kubernetes isn't really
a great fit for the edge. Kubernetes is incredibly resource-hungry. It really likes to gobble
up RAM. It really likes to block all your CPUs. And there's a lot of components inside
of the Kubernetes control plane that do this. Part of it is the kubelet that runs on every
worker machine. Part of it is the container run times themselves or the API server. But
what I'm going to talk about in this session, I think I have no idea why. I still have batteries,
so I'm going to talk about operators specifically. Operators tend to take a lot of resources,
eat up a lot of resources from your Kubernetes cluster. So first of all, operators, these
are basically plugins to the Kubernetes control plane, which add additional functionality
to the Kubernetes API. For example, it could add a resource to deploy and manage a MySQL
cluster or it could add a resource to deploy and manage a SEF cluster, for example. And
these operators, they are also really resource-hungry. And this is part of it is because they are
long-running processes. So these processes, they see something change in your Kubernetes
cluster. They want to do something with it and then write those changes back to the API
server in order to manage the applications. But after that writing is done, these processes,
they keep running because they keep listening for events from the Kubernetes API or even
sometimes manually watching if some resource has changed. And so even if they're doing
nothing, they're still running. A lot of them are written in Golang. And Golang really
likes memory. They are running inside of containers. Most of them are running inside of separate
containers. And they're basically sitting in RAM doing nothing, eating up that RAM.
And so this is an issue if you want to run Kubernetes in the edge on devices which have
like 512 megabytes of RAM. These operators are basically unusable in situations like
that. So how could we solve this? One of the ways that you could solve this is that we
think we can solve this is by using WebAssembly and the WebAssembly system interface. And so
yes, really, we're trying to lower the footprint of Kubernetes by taking a web technology and
putting it inside of Kubernetes. If you don't believe me, this is a tweet from one of the
co-founders of Docker who basically said like if WebAssembly and the WebAssembly system interface
would have existed in 2008, they wouldn't have needed to create Docker. It's a very interesting
technology which we think is a very good fit to solve this issue in Kubernetes. So what
is WebAssembly created originally for the browser? It's basically a binary code format. You compile
your applications to WebAssembly instead of compiling them to x86 or to ARM. And then this
code runs inside of a runtime. You could call it a very lightweight virtual machine. It runs
in your browser, it runs in the Node.js runtime, but there's also a whole bunch of new purpose
built, very lightweight runtimes such as wasm time, the one that we're using right now. And
the WebAssembly system interface is basically a syscall interface. So WebAssembly is your binary,
but it doesn't have access to anything. And then the system interface is a syscall interface. So
that's an interface that it uses to open files, open sockets, start new threads and stuff like
that. And so if you combine these two, you basically have a very lightweight, super fast
sandbox. And so the result of running these operators inside of WebAssembly containers is
that they use a lot less RAM. So here on this slide at the top, you see 100 operators running as
Docker containers. Then you have 100 operators running as WebAssembly containers and then 100
running just on bare metal. So we're not reaching the performance of bare metal. There's still
some overhead. However, we're compared to the Docker containers like we're getting a lot closer
than that. As an advantage that we didn't see coming initially, but they also have a lot less
latency. They run a lot quicker. This also shows the difference between Golang operators and Rust
operators. So obviously, Rust will have a lot less latency and a lot less latency distribution
because it's not a garbage collected language. However, we were surprised to see that running
them inside of WebAssembly gave them even better, even more consistent latency. So how did we do
this? We basically work with a client server model or like a parent operator and a child operator.
The parent operator, it is a WebAssembly runtime with a bunch of additions to it in order to
support running operators inside of that runtime. And it watches the Kubernetes resources in the
name of the operators running inside of it. So the operators don't have to keep running to watch
it. They can just shut down when there's nothing to do. And the parent operator will call them
once there is a change to process. The child operators, those are where the actual operators
run inside. And the interesting part is that they are just regular operators compiled to WebAssembly
using a patched version of the Kubernetes SDK. So in the future, this will probably make it
possible to just take a regular Kubernetes operator, compile it to WebAssembly, and then use it in
this system. Right now, we only support Rust because Rust support for WebAssembly is very good,
Golang support for WebAssembly is iffy. And we have a patched version of Kube RS, a Kubernetes SDK,
to then contact the parent operator instead of contacting the Kubernetes API itself. So how
does this loading and unloading work? This is the WebAssembly engine. This is basically just wasn't
time, the WebAssembly runtime. And in here is your client operator, your child operator is running.
Once the child operator wants to contact the Kubernetes API server, it does a syscall. We
extended the WebAssembly system interface to add a few syscalls to support the scenario. And this
syscall goes through to the parent operator and the parent operator is the one who actually contacts
the Kubernetes API. Once these calls are finished, the parent operator, it contacts the child
operator back again in order to give it the result of these calls. And if the child operator is not
doing anything, the parent operator shuts down the child operator. And once there changes to
process, it starts it up again. And so the results I showed you on the first slides, those results
are just not unloading anything. Just running Kubernetes operators inside of WebAssembly. So
these results are what you get when you have a worst case scenario for unloading operators when
they're not doing anything. And so we see that in a worst case scenario, they still use 50% less
RAM because they're constantly being unloaded and then reloaded again once there's changes to
process. However, this is obviously at the cost of latency. Even though WebAssembly, it starts
incredibly fast. It has latency that just can't be compared to Docker containers for starting
applications. There is still some latency to start a WebAssembly application. And so this
compounds in the worst case scenario of like 100 operators chaining themselves up to 12 seconds,
which is an issue. So what are we doing now? So we have this basic proof of concept to show
that this seems to be a very good approach to lower the footprint of the Kubernetes control
plane. And we want to do more with this. Currently, we're improving the build tools and we're making
more realistic tests. All the tests we did right now were a worst case scenario of operators
constantly doing stuff. However, in the real world, most operators don't do anything most of the
time. So we're creating more realistic tests to see what these operators, what the performance
benefits are for real workloads. We're also working on predictive unloading so that if we know that
an operator is going to have to run again in a few milliseconds, we don't unload it because it's
better to just keep it running. In the future, we want to work on better support for controllers
that wake periodically. So right now, we see that a lot of production controllers actually wake
periodically every five seconds or every 20 seconds in order to manually check resources in the
Kubernetes API because some of those resources, they can't work with callbacks. So we are trying
to figure out a way to actually put that functionality into the host operator itself so that
even when you're watching resources that don't support event-based APIs, the operator is still
sleeping as long as there's nothing to process. And we're also really interested in upstreaming
and standardizing this. We have patches for Kube RS. We have an extension for the WebAssembly
system interface. It would be very interesting to see if there's people in the ecosystem who are
interested in this and support for Golang, although this will probably not be work that we're doing,
we'll just wait until Golang is better supported in WebAssembly. So I have to thank the developers.
Francesco is somewhere here in the audience. We started from a prototype created by Francesco
and Marcus, which runs Kubernetes controllers inside of WebAssembly. And we refactored it to use
wasm time and we added the unloading mechanism. This was done by Tim as part of his master's thesis.
And right now, student Kevin is working on it also as part of his master's thesis to improve the
build system so that it's much easier to get started with it and to add predictive unloading
and more realistic benchmarks to have a better idea of what is the performance for actual production
controllers. So the main reason I am here today is to say like, hey, we have a really cool proof
of concept, which solves an issue that we have been having. Is this solving an issue for other
people in the community? And are you interested in working together on this? If you're interested
in working together on this, please get in touch. If you're a student yourself and you want to do
like an internship or a master's thesis working on this, we have a lot of opportunities, same for
a PhD. So please contact us, send me an email to see what we can do for you and how we could
collaborate. So this is the end of my presentation and there's now room for questions. I also put
the link to part of our code here. I think this GitHub repo also links to the other repositories
that you need. Okay, we can take a couple of questions. So why was he so fast and why
it is not possible to do something similar with JVM? So definitely, JVM and WebAssembly are very
similar in that regard and a lot of people, they position WebAssembly as being like a more
cross-platform and a more cross-language version of the JVM. But if you're only interested in Java
and Java-based languages, then the Java runtime itself is a very good alternative to this.
Okay, there was another one over here, right? Yeah.
So if I understood correctly, you are deploying your operators outside containers and that makes
them much more efficient. But, I mean, besides the security aspects, when you deploy in containers
and Kubernetes, you have many other things that you can set, like resource limits, but also things
like post-topology spread constraints and notations to make sure that some processes are running on
specific nodes and so on. How can you address that with WebAssembly? Because you cannot package
then your operator like any other workload that you deploy in Kubernetes.
Yeah, so it's a very good question. So one of the benchmarks was just running the operators
on bare metal, but that's not actually what I'm proposing. It was just to see, like, what is the
absolute maximum amount of performance we could get out of this. Our plan is to run each operator
inside of its own container. It's just a WebAssembly plus WebAssembly system interface container
instead of a Docker container. And so most of the security profile and stuff like that that
you have with Docker containers is very similar with WebAssembly. Some would even argue that it's
more secure in WebAssembly because it has a much smaller API footprint and it has some of the best
teams working on it to make sure it's secure for the browser. Moreover, the code that is running in
these WebAssembly containers in my proof of concept, this is control plane code. So this is code that
the system administrator selected, like, okay, yeah, I want this specific system administration
code to manage my applications. And so in that sense, there's also, like, a higher level of trust
put into the code, which means that, like, things like attacks and stuff like that, there's less
of a risk to it. But even then, like, it's still running inside of containers.
So one of the most important scalability aspects of Kubernetes controllers is the watch-based cache,
right? So without it, the API server wouldn't be able to handle all the long pulling and so on.
And it's also one of the most memory-intensive aspects of Kubernetes controllers. I was wondering
in your memory benchmarks if you were cutting down on this watch-based aspect, or if it is still
included in the parent operator. So for example, is the parent operator caching as a proxy for the
child operators? Is that the case? Yeah, that's what's happening, basically. The parent operator
is where the caches are, yeah.