So, hello everyone, pleasure to be here.
I'm Ithamar Holder and I'm a senior software engineer working for Red Hat.
And this is a talk about the journey through supporting VMs with dedicated CPUs and Kubernetes.
And the reason that there's a journey word in the subject is that this was a true journey
for me.
And I'm going to guide you through the journey until we reach the actual problems and solutions.
And there are two reasons for that.
So first reason is that we need to understand the problems and solutions.
So we need to understand the background for it.
But the second reason is that I've gained a lot of insights and takeaways during that
journey.
And I think that I hope that you could find is also valuable for your journeys.
And yeah, that you can take the same takeaways for whatever you're doing and whatever you're
interested in.
So we're going to talk about all sorts of stuff like dedicated CPUs and CPU manager,
C groups, spot isolation and namespaces, Kubernetes resource allocation.
And so let's begin.
So first of all, an introduction to Kuber.
So Kubernetes is designed to run containers, which are designed very differently than VMs.
And running VMs on one platform and containers on another platform is not the best approach.
And this is where Kuber comes into play.
This is basically an add-on or extension to Kubernetes, which lets you run VMs on top
of Kubernetes as a first-class citizen, as a completely cloud native.
And I'm not going to dive into all the architectural details here, but the trick is basically to
run a VM within a container, like this picture tries to illustrate.
And that's basically what you need to know for this talk.
So what's the deal with dedicated CPUs?
So basically, the key word here is avoiding preemption or context switches, right?
These are crucial, this is crucial for certain use cases like real-time VMs or VMs that
depend on very low latency.
So as a naive example, let's think about a VM that hot loops over some condition.
And when this condition becomes true, it has to react really, really fast.
So if we context switch this workload out, then it would take more time.
Because once the condition becomes true, it would take time to context switch back, and
only then the VM could react.
So this is very crucial for some use cases.
Also it's supported by most hyper-hypervisors, and it's a pretty standard feature.
And we aim to bring this also to Kubernetes.
So a question to the crowd.
Who recognizes this section?
Who knows what this is?
Okay, so most of you.
And another question, who can say that he's confident about how this is implemented behind
the scenes, or how Kubernetes actually does that?
A lot less of you, right?
So that's good, it means that this is relevant, right?
So obviously, this is taken from the pods manifest.
This is the place when we specify resources.
We have, of course, requests and limits.
We can specify CPU, memory, a firmware storage, and a bunch of other stuff.
And so let's talk about containers for a second.
So containers are actually a conceptual concept that can be implemented in many ways.
So from the Linux kernel perspective, there isn't such a thing as a container, really.
There are basically a couple of main kernel features that serve as the building blocks
for containers.
One of them is C groups.
C groups is very important, and is one of the main building blocks for containers.
So let's talk about C groups a bit.
So basically, the idea is that the architecture is a tree of resources, right?
We have the root C group, which is basically all of the resources on the node.
So for example, 100 CPUs.
And then we divide them into groups, like for example, 70 CPUs, 20 CPUs, 10 CPUs, and
so on.
The idea is that every process on the system is attached to a C group.
And that basically the C groups limits the resources for this group of processes.
And in Kubernetes, there is usually one C group per container.
This actually depends on the CRI that you're using.
But the most common approach is to use one C group per container.
So in Kubernetes, all of the values are always absolute, right?
When we specify CPU, for example, we can specify 100M, which stands for milli-CPUs, which is
similar to 0.1 CPUs, 1.3, whatever, but these are always absolute values.
In C groups, it's all relative, right?
It's called CPU shares.
The default is 1024, but it doesn't really matter.
So if we'll look on a very naive example again, let's say that we have a node with two pods
running on the system, pod A and pod B.
And let's say that pod A has one CPU share and pod B has two CPU shares.
What it would mean is that pod B would have twice as CPU time as pod A. It doesn't really
matter how many CPUs the nodes have, because this is all relative, right?
So how does Kubernetes convert between the absolute values and the relative shares?
So we can think about one CPU as 1024 shares, just because it's the default in C groups.
So let's say that a pod asks for 200M CPUs.
So this is actually a fifth of a CPU.
So what we can do is divide 1024 by 5 and we get approximately 205 shares.
And this would work, but remember that shares are still relative.
So what happens, for example, if the node has 100 CPUs and one pod with 200M CPUs request
runs on that pod?
Since it's relative, it would just use all of the node's resources, right?
So this has a nice side effect.
The spare resources on the node can be used by the pod relatively to their request.
So basically the request is the minimum amount that is actually allocated and all of the
spare resources are being split relatively to their request.
So let's talk about Kubernetes QoS for a second.
There are three quality of service levels.
The first one is best effort.
That means that I don't specify anything.
I don't have request, I don't have limits, not for memory and not for CPU.
The last one, guaranteed, is kind of the opposite from that.
I specify both request and limits to both CPUs and memory and the request and limits
are equal.
Now if you're not best effort and you're not guaranteed, you'd be burstable.
So this is just an example, but the idea is that you can specify either only request,
only limits, you can specify them both, but they're not equal.
So any other than best effort and guaranteed.
Now basically the trade-off here is predictability in order to get stability.
So basically Kubernetes tells you, if you want me to guarantee you stability, you have
to be predictable or if you will be more predictable, you'll gain more stability.
Like one example for that, if we're talking about memory for example, are node pressures.
So when the node would have high memory pressure, it would evict guaranteed QoS pods last.
And after that it would get to burstable, after that it would get to best efforts.
So this is true by the way, as long as you keep your promises.
If you say that you're limited to a certain amount of memory and then you exceed this
memory, then on most CRIs we'll just kill the pod.
So can we use dedicated CPUs on Kubernetes?
So the answer is yes.
This is possible with CPU manager.
And in order to do that, we have two requirements.
First of all, the pod needs to be of guaranteed QoS.
Second of all, the CPU request, which equals the limit because it's a guaranteed QoS, has
to be an integer.
It cannot be a floating point value.
Also an interesting fact is that only a single container or some of the containers in a pod
can have dedicated CPUs, but the whole pod needs to be of a guaranteed QoS.
So let's talk about namespaces for a second.
So remember this little diagram from before, so namespaces is another building block for
containers and it basically is responsible for the isolation of the containers.
So when I'm picturing a pod, this is what I think about.
Like it's a box with some containers in it.
The containers are absolutely isolated from one another.
And as we said, container is a concept.
So if we will take some of the namespaces out and we will break some of the isolations
between the containers.
Are there still containers?
How do we need to, how layers of isolation do we need to strip before it stops being
a container?
This is more of a philosophical question, but is it possible on Kubernetes and the answer
is yes.
So for example, it's possible to share the pod namespace between containers or the process
namespace between containers.
And what it means is that inside the container, if you will do something like PS, you would
see all of the processes from all of the containers.
This isolation will not exist anymore.
Another interesting fact is that as a side effect, the file systems are also shared.
Now they're not shared directly, but you can use that trick to use them indirectly.
We'll get you to the root file system of another process that now can be in another container.
So to actually enable that, that's what you need to do.
In the pod, in the spec, share process namespace, true, and that's it.
So now a word about KVM.
So who knows KVM, by the way?
Oh, a lot of you, okay.
So this is a kernel model which turns the Linux into a hypervisor.
Basically we have two kinds of hypervisors, type one and type two.
Type one means that it's also called a bare metal hypervisor because it's being installed
on a bare metal.
There's no OS benefit.
And what it means is that it's really fast, but the downside is that it has to implement
stuff like a scheduler, a virtual memory, and a lot of stuff that already exists on every
OS.
Type two hypervisors are being installed on top of the OS, so they don't have to re-implement
all of those stuff, but they're usually a lot slower.
So KVM is really incredible because it turns Linux into a type one hypervisor.
And this is what Qvert is using to gain native performance.
An interesting fact about a KVM is that its main purpose is CPU virtualization because
this is the performance part.
It's also backed by QEMU, which does things like IO and stuff like that, which are usually
less related to performance.
So how does KVM actually works?
So from the guest perspective, it will have, for example, four CPUs.
But these aren't real CPUs, right?
They are virtual CPUs or VCPUs.
And from the kernel perspective, these are just threads, VCPU threads.
So what the guest sees as a physical CPU is actually from the host perspective is just
another thread on the system.
Okay, so now back to Qvert after all of these introductions.
In Qvert, we have the VRT Launcher pod.
It has some containers in it.
The compute container is the main container.
Inside the compute container, we run the QEMU process, which actually runs the guest.
And this is the main container that we're using.
So first attempt to support dedicated CPUs.
So the idea was let's allocate the compute container that we talked about with dedicated
CPUs.
So this is possible with CPU manager as we talked about.
All we need is to do is to have a pod that's guaranteed QS and to have an integer amount
of CPUs on the compute container.
So by the way, is it a good approach, do you think?
This is a problem and let me explain you why.
So let's zoom into the compute container for a second.
The list here is all of the processes and threads that run inside the compute container.
You don't need to understand everything that's running here.
But let me show you the interesting part.
So you see the QEMU KVM process.
All of the red ones are threads.
Now as you can see, we have two threads, which are the actual vCPU threads, like I said earlier.
So the problem is that we have a lot of threads with different priorities.
And if we let all of the compute container run with dedicated CPUs, this aren't really
dedicated CPUs because we said that the keyword here is avoiding preemption, right?
But with the previous setting, we're basically, we will context switch out the vCPUs in order
for other threads inside the compute containers.
So the vCPUs aren't running on dedicated CPUs really.
We actually lie to the guest.
So that's a problem.
Now the second approach is called the housekeeping C-group.
And the idea is that we will make a child C-group for all of the low priority threads or processes.
So how would it work?
So let's say that the user asks for XCPUs.
We would actually allocate X plus one dedicated CPUs.
And one dedicated CPU will serve all of the housekeeping tasks.
And when I say housekeeping tasks, I basically mean everything but the vCPUs themselves.
Then what we can do is move all of the threads that aren't vCPUs into the housekeeping C-groups.
And then the vCPUs would be with two dedicated CPUs.
So this is how it looks like.
We have the vert launcher pod inside of we have the compute container with X plus one
dedicated CPUs.
One dedicated CPU is for everything but the vCPUs themselves.
And the X dedicated CPUs are for the vCPUs.
So this approach is much better because it lets us, this basically supports two dedicated
CPUs for the vCPUs.
But this also has a problem.
So first problem is that we waste one dedicated CPU for stuff that are of low priority.
This is a huge waste.
Ideally, we would have wanted to do something like give me like four or X amount of dedicated
CPUs and another amount of shared CPUs for everything else.
And this is actually possible on C-groups but it's not possible on Kubernetes because
what we said earlier.
If we're going to ask like 3.2 CPUs or something like that, they won't be dedicated.
That would be all shared.
So basically Kubernetes goes for an all or nothing approach.
Either all of the CPUs are dedicated or all of the CPUs are shared.
Another problem which is more of a design problem is that we're focused around the lowest
priority processes.
And this kind of should be reversed, right?
I mean we want to configure the vCPUs to have dedicated CPUs.
So we configure everything that is not the vCPUs.
And this is problematic because we would ideally want to only change the vCPUs threads and
leave everything as is to keep it open for extensions in the future and stuff like that.
There are more problems related to C-groups v1 and v2.
I'll not dive into details here but two words about it is for example in v2 we have the
threaded C-group concept which doesn't exist in v1.
And in a threaded C-group we have a lot of limitations.
Some of the controllers and some systems of C-groups cannot work at all.
So just know that there are more problems with this solution.
So a third attempt, I'm calling it the dedicated CPU C-group approach and here's the idea.
So the compute container stays completely as it is.
We won't touch it at all.
We would allocate it with CPUs that are not a dedicated, sorry they are shared CPUs but
remember that the pods still need to be with guaranteed QoS and I'll explain why.
So what we will do is add another container which is basically a blank container with
X dedicated CPUs and as I said every container creates a new C-group so it will create a new
C-group for us.
So what do I mean by a blank container?
I mean a container doesn't really run anything.
It could run for example a sleep forever process just to keep the container alive but that's
it.
It would be entire the blank.
And then what we will do is move only the VCPUs into another container, right, into
another C-group.
All of the compute containers stays as is and only the VCPUs are configured.
So this is how it looks like.
So as the VM starts or before it starts, we have the VRT launcher.
Now we have two different containers.
One of them is the compute container with Y shared CPUs.
These are not dedicated CPUs.
The other one is a container with dedicated CPUs.
X dedicated CPUs, exactly the amount we need, not X plus one as before.
So in the compute container everything is being run when the VM is being started.
But right after it's being started or right before it's being started, what we will do
is move the VCPUs into the different container.
And that basically solves our problem because now all of the housekeeping tasks are being
run with shared CPUs, the VCPUs are running on dedicated CPUs.
So can we actually do that?
I mean, we actually moved some threads of a process to another container.
This looks extremely weird, right?
But this is possible because we shared the PID namespace.
So you can think about it like the processes are not isolated anymore.
They're not really being moved from one container to another because the container does not
isolate processes anymore.
So we only change C groups.
So from the VCPUs perspective, they just stay the same.
They can communicate with all of the threads and processes in the system.
So only the relevant threads are being configured, as I said, shared CPUs for the housekeeping
test so we don't waste one dedicated core anymore.
And we keep things open for extensions in the future.
Maybe we would want to do more plays with C groups in the future.
So we want everything in the compute container to stay completely as is.
OK, so summary and takeaways.
There were a lot of introductions here.
And I've scratched the surface of a lot of cool facts and technologies that I've seen
along the way.
So we've seen CPU allocation, implementation in Kubernetes, how C group uses relative shares
and not absolute values, and how the CPUs and the resources are being spread along the
pods relatively to their requests.
We've seen how to enable dedicated CPUs and Kubernetes.
We've seen namespaces and how to break the isolation within a pod.
We've talked a bit about KVM and how it uses threads to run the actual CPUs.
And of course, we talked about Kuvert.
And again, I really hope that these takeaways would serve you in your journeys in the future.
And I hope that you find this interesting.
So thanks a lot, and you're always welcome to send questions or feedback or anything
else to my mail and I'll also be outside for questions.
OK.
So thank you.
And if you have any questions.
OK.
So the question is, do we need to do something like this?
OK, so the question is, do we need more permission to divert launcher pod, or is it being done
by the vert handler?
And the answer is, it's being done by the vert handler.
So just a bit of context, a vert handler is another pod that Kuvert uses.
It's a pod with high privileges, and therefore, we don't need any extra privileges for it
to configure that.
Yeah.
Does this allow for easier networking communication if you talk in service communication in the
Kubernetes sense?
You could do it VM to VM, presumably with the exact same mechanism, so resolving service
names from what VM to whatever VM is using Kubernetes, does that work at the moment?
So that's a question about Kuvert in general, right?
But in Kubernetes, I can do it, but presumably I can do the same thing from a VM running
inside the pod.
Right.
Yeah, so the answer is yes.
OK.
Yeah.
Any other question?
Yeah.
I have a question.
You are dedicating the CPUs separately, but about other threads, which are, for some
of these cases, highly CPU consuming like network threads or IO threads, is there a network
thread as well?
Right.
So the question is, what about IO threads or network related threads, what about them?
And, actually, in the VM manifest, we have configurations for that.
So you can ask, for example, for an IO thread to run on a dedicated CPU.
That is also supported.
Yeah.
I focused solely on the CPUs themselves, but this is entirely possible in Kuvert today.
Yes.
Can I combine it with new machines, so dual machine machines can be used by the same
machine, and what's running with it?
So the question is, can we support NUMA with this?
The answer is yes.
I'm not sure if it works right now outside of the box, but I think it should be possible
with, especially with C-groups V2.
But this is an interesting question.
I will have to think about it a little more.
I think it is possible.
Anyone else?
Time's up.
Sorry, but I'll be out here if you want to ask further questions.
Thank you.