Okay, we're ready for our next talk.
Daniel is going to talk about MetaNet devices.
Thanks.
All right.
Thanks a lot.
So, yeah.
So, this talk is about MetaNet devices.
This work has been done by my colleague and myself.
We are at ISO Wayland software engineers working on the kernel and also Cilium.
So, really, the goal is what we were, the question we were asking ourselves, like what
about, how can we leverage the BPF infrastructure in the kernel and also the networking features
to really achieve maximum performance for Kubernetes parts.
And before I go into the kernel bits, just a really quick recap around Kubernetes and
parts and what it is.
So, basically, what you can see here is a host.
The host can have one or many parts.
In Kubernetes, it's an orchestration system, essentially, and a part is usually defined
as a network namespace, and it is connected to, typically, to Weave devices to get traffic
in and out of them.
A part can have one or many containers that are sharing this network namespace.
So, yeah, and CNI is basically a networking plug-in which will set up various things.
When a part comes up, for example, it will set up net devices, assign IP addresses.
It has an IPAM infrastructure to manage pool of addresses.
It will install routes.
In the case of Cilium, which is a CNI, it will also set up the BPF data path to basically
route traffic in and out.
It has various features on top as well, such as policy enforcement, load balancing, bandwidth
management, and so on and so forth.
But I don't want to make this talk about all the different features about Cilium, but rather
about performance.
There was an interesting keynote last year at the SAEcon from Brandon Craig, where he
talked about computing performance and what's on the horizon, and he had a couple of predictions
and one was quite interesting when he was talking about OS performance, and the statement
that he made is that, well, given the kernel, it's becoming increasingly complex.
The performance defaults are getting worse and worse.
Yeah, so he stated that it basically takes the whole US team to make the operating system
perform well.
And the problem is, given all these performance teams, they are trying to optimize at the
larger scale, nobody's actually looking at the defaults anymore and how they can be optimized.
So this was quite interesting.
So yeah, in case of defaults, we were wondering, given two Kubernetes nodes with part to part
and they're connected with 100 gigabit NIC, we wanted to look at the single flow, like
a single TCP stream, and we asked, like, what's the default baseline you can get to?
Where are bottlenecks?
How can they become, how can they be overcome?
And actually, can we provide better defaults out of that, what we figure out?
Why bothering with single stream performance?
Well, first of all, it's interesting for the kernel to be able to cope with growing
NIC speeds, so 100, 200 gigabit or more.
How can this be maxed out with the single stream?
There are lots of data intensive workloads from user and customer sites around machine
learning, AI, and so on.
But generally, it's also interesting to be able to free up the resources and give them
to the application instead of the kernel having to block them.
So the assumptions for our test is basically, like, usually the Kubernetes worker nodes
that users run, they are quite generic, they can run any kind of workloads.
What we are also seeing is a large number of users typically just stick to defaults,
they don't tune specifically the kernel.
There's an interesting cloud-native usage report where they tried to get some insights
into how Kubernetes deployments usually look like.
There's definitely an increasing trend to have a higher density of containers per host.
So like around 50 or more is expected these days and the number of pods per node is also
increasing.
So yeah, the question now is, okay, so like a basic, very basic compatibility setting,
for example, for the case of Silium, we use the, like if you deploy it in a basic mode,
we just use the upper stack for routing and forwarding.
There are various reasons why people might want to use that.
For example, in case of Kubernetes, there's a component called Qproxy which uses net
filter IP tables for service management, for service load balancing.
Some people stick to that or they have custom net filter rules, so maybe they require to
use the upper stack for that or just simply they for now just went with defaults and might
look into more tuning at the later point in time.
So when we try to look at the performance for that, what you can see here is on the
yellow bar is the host-to-host performance for a single stream, so we got to 44 gigabit
per second, but then if you do the pod-to-pod connectivity, it's really reduced dramatically.
And so yeah, like one of the reasons is, and I will get into this in a bit, is because
the upper stack is giving false feedback to the TCP stack.
One thing that we did like a year ago or so is to introduce a feature where we can, which
is called BPF host routing, where we don't use the upper stack and for BPF itself, given
we attach to physical devices, but also to Veef devices, we added a couple of new helper
functions there.
One is called BPF redirect peer.
What this basically is doing is adding a fast switch into the network namespace for the
ingress traffic.
So basically we're just like, instead of going the usual X-MID route to the Veef devices,
we can retrieve the Veef device inside the pod and just scrub the packet to remove the
necessary data that we typically remove in switching network namespaces, but then also
to just set the device to the device inside the pod.
And then circle that around in the main receive loop without going to a per-CPU backlog queue
that you would normally do when you transfer data through Veef devices, and we don't need
to use the upper stack because there's all the information already available in BPF context.
And for the way out, we added a helper which is called BPF redirect neighbor.
So that one will basically insert the packet into the neighboring subsystem of the kernel.
So usually we can do a FIP lookup out of BPF, so there's a helper for this as well, and
then combined with this for the resolution of neighbors.
It will allow that you don't need to go to the upper stack, so the nice benefit you get
as well with this is that the socket context for the network packet for the SKB is retained
all the way to the physical device until the packet is actually sent out.
And this is not the case when you normally go to the upper stack.
So then the TCP stack actually thinks that once you go to the upper stack that it already
left the node, but it's actually not the case.
And this way it can be retained.
And this is how the complete picture looks like.
And if you look at the performance, it's already much better.
So we were able to get almost a 40 gigabit per second under 1.5K MTU.
So this was interesting.
Now the question is, how could we close the remaining gap?
Under 8K MTU, we also did some tests, and one thing to note here is that we were able
for a single TCP stream to get to 98 gigabit per second for the host-to-host case, but
still the situation looks quite the same for the weave with the BPF host routing.
So there's still a small gap that we want to close here as well.
And that's where we introduce a new device type as a weave replacement.
So we call this meta device because it's programmable to BPF, and you can implement various, like
your own business logic into this.
So it's flexible.
And this time, the main difference is that this also gets a faster switch on the egress
side for the egress traffic, so it doesn't need to go to the per-CPU or back-lock-U for
the egress as well.
So if you look at the flame graphs, so that's the worst-case scenario where we compared
the weave and the meta device.
So what you can see here on the weave device, like on X-MID, it will basically scrub the
packet data, it will un-CUE the packet to a per-CPU back-lock-U, and then at some point
there's a network's action.
It will pick up the packets from the queue again, and in the worst case, it can be deferred
to the kernel software QDemon, and then you see this rescheduling where you have a new
stack where this is processed again.
And then from the BPF side, it will reach BPF on the TC egress on the host weave, where
we then can only forward this to the physical device to leave the node, right?
And all of this can be done in one go without rescheduling through this meta device.
So it will scrub the packet, it will switch the network namespace, it will reset the device
pointers, and it will then directly call the BPF program.
And if the BPF program says that based on the FIP lookup and so on, that it will forward
the packet directly to the physical device, then it will avoid this rescheduling scenario.
So on the right side, I mean, it's really straightforward.
That's how the implementation of the driver X-MID routine looks like.
So it will basically just call into BPF, and then based on the verdict, push it out.
It's really just like 500 lines of code.
So it's very simple and straightforward.
I think it's just one-fifth of the weave driver that we have right now.
And the other focus that we wanted to put into is compatibility as well, so that, like
given in Solium, we need to support multiple kernels, the ideal case would be that we don't
need to change much of the BPF program and can keep it as is.
And in case of XTP, we didn't want to implement it because for the weave case, it really is
very complex.
It even adds multi-q support, which you normally would not need on a virtual device.
So we wanted to keep it as simple as possible and to have the flexibility that this can
be added as a single or a paired device.
So for the weave replacement, it would be a paired device, but you could also do it
as a single device and then implement whatever logic you would want in BPF for that.
So looking at the performance, again, like the TCP stream under 8K, so this is really
able to reach through this approach the full 98 gigabit per second, and in terms of latency,
so we did some net-perf TCP or R measurements as well, where you get the minimum, the P90,
99 latency and so on.
So this is really on par with the host.
So now we were asking ourselves, so can we push this even further?
I mean, well, so we were able to get to 98 gigabit per second, but like the cost for
like a megabyte to transfer, can this pushed even more?
And there's a relatively recent kernel feature which is called Big TCP.
It landed for IPv6 only in 5.19 and was developed by Google, and the whole idea behind Big
TCP is to even more aggressively aggregate for GEO and GSO.
So normally the aggregation, the kernel will try it basically out of the incoming packet
stream, create a super packet and then we'll push it up to the networking stack so it only
needs to be traversed once.
And the limit up until that point was for 64K packets, simply because in the IP header
that's the maximum packet size that you can do.
And the idea for Big TCP for IPv6 was that, well, maybe we could create a hop-by-hop header
in the GEO layer and then add, and then like the 16-bit packet length field can be overcome
because there's a jumbo-gram extension in there which allows for a 32-bit field.
So you can do much more aggressive aggregation.
And yeah, so this is also now supported with the new studio release where this will be
set up for all the devices underneath automatically for IPv6.
Actually like this week, that was also merged for IPv4 now.
So this will end in kernel 6.3, which is exciting.
And when we looked at the performance again under Big TCP, turns out like using the upper
stack is currently broken in the kernel, so that still needs to be fixed.
We will look into that.
So forwarding there wouldn't work.
And with the host routing cases, it will basically bump up the regular Veef one to get this
on par with the meta and also the host, so it will basically hide those glitches.
The latency is still better in terms of like the short packet response type workloads for
the meta, so that's still on par with the host.
So what is the remaining offender like when you run all these features together?
It's basically the copying to users, so like between 60 and 70% of the cycles is really
spent on copying all this data to user space.
So the next question we ask ourselves actually in this experiment, so what if we combine
the whole Big TCP stuff with TCP0 copy, so what if we could leverage the memory map TCP?
And it turns out that's currently not possible in the kernel.
That's a limitation because in the GRO layer, Big TCP will create a frag list, which is essentially
like a list of SKBs as a single big one that is being pushed up the stack.
And TCP0 copy only works on the SKB frags, so that's like an internal.
So basically you have a single SKB and it has like the pages as read only attached in
the non-linear section.
So that currently does not work.
Combining those two would probably have like really big potential, but what we now try
to do is we just looked at just using the TCP0 copy to see how it looks without the
Big TCP.
Actually speaking it's not as straightforward to deploy because first of all you need to
rewrite your application in order to leverage memory map TCP.
This can be done for RX but also for TX or both.
And it needs driver changes and in particular driver changes to be able to split the header
from the data because the data you want to memory map to user space.
Some Nix might do this with the hardware and some others you would have to do some kind
of pseudo header data splits where you basically just copy the header into a linear section.
So this is how it would look like.
We tried this for 4K and 8K MTU.
There's a great talk from Eric Dumasay, the TCP maintainer in terms of what all the all
those things you need to do in order to be able to make use of this for example you also
need to align the TCP window scale to 12 segments so that you fill exactly those pages.
And yeah the driver support can be very different like we had in our lab MLX500 gigabit Nix
and they did not implement the header data splits.
So my colleague Nikolai did the POC implementation in the like to change the driver to be able
to do that so that we could get some measurements out of this.
And if you want to look into an application like an example application and how you can
implement that there's one in the networking self-test in the upstream tree called TCP
M-MAP which is useful also for the benchmarking.
So you really need to align various different settings like for our test implementation
we used like for MLX5 the non-striding mode so like the legacy mode so you need to switch
that off and EVE tool first because I mean for the POC it was easier to do the way like
the packet layout is done there then you need to align the MTU to either one page or two
pages and you need to do various other settings like for the course of time I will not go
into all of the details.
And generally I would think it would be useful addition to the kernel to have like a configuration
framework for that and to be able to have more drivers supporting the header data split.
There's actually one in Windows kernel turns out so there's some documentation around
there that we found while preparing for the talk.
The other thing is the caveat is like the TCP-0 copy may like the benefits might be
limited if you then actually go in your application and then touch the data because then they need
to be pulled into the cache which they would be if like if you would have to copy things
to user but they may not be like if you just memory map so the applications there is mostly
like on the for example storage side where you wouldn't have to do that.
And looking at the data for 4K MTU we tried with the implementation we got to 81 gigabit
per second so that's a bit limiting.
Could also be that this is mostly because the implementation was you know POC with lots
of optimization potential that can still be there but we looked at the 8K MTU and there
we were able to get to 98 gigabit per second but the interesting piece here is the cost
per megabyte we could we were able to reduce from 85 down to 27 microseconds per megabyte
so this really is significant and yeah because of the avoidance that you don't copy anymore.
So as a summary so we started with the default and then you know switch to the 8K MTU then
we went to the host routing that we covered then the addition of the meta device where
we can avoid where we can do the slow latency you know for ingress and egress then the big
TCP and that all works without changes to the application and with that you can already
get like a 2x improvement and then it gets even more dramatic but it's of course dependent
on your application for the zero copy.
Some of the future directions as I mentioned earlier it would be useful to have like a
generic header data split framework for NIC drivers that they can implement that and expose
it to this setting.
There's potential here as well to optimize for example like the header pages could be
the head page could be packed with the headers it could get recycling which we didn't implement
here and in future big TCP would be interesting to combine with the zero copy so that this
covers the frag list in GRO and the other thing that is at some point on the horizon
is to push the big TCP actually onto the wire as well if the hardware supports that and
yeah so with that I'm done with the talk and the prototype for the meta device is currently
you can find on this branch we are working on pushing this upstream in the coming months
and there's also the prototype public for the MLX5 header data split and yeah so the
plan is basically to get this into upstream kernel and then also to get this integrated
into Syllium for the next release yeah thank you are there any questions?
Thanks we already have two questions in the chat so we can start with them if there are
no one in the room.
Alright so the first question we got was can we well not really a question as much as a
comment I think which is can we please come up with a better name than meta for this just
call it BPF or BPFDF or something like that.
The other thing was if the perf benefit comes from calling through from inside the network
namespace can we just make VIF do that instead?
I think we could but the question is like I haven't looked into the like how it would
affect the XDP related bits so it felt easier to do something simple and start from scratch
like one thing I don't really like about the VIF devices to be honest how complex it got
with all the XDP additions and it's not even that beneficial and it added like multi queue
support and all of that which was not needed at all for a virtual device so I really wanted
to have something simple and yeah it's easier so.
Any more questions from the audience here?
I have a question you're about big TCP and TCP zero copy it's you're in your lab you
only did benchmarks host to host right because so but also part to part so we tried both
yeah.
Because do you know how it would work with the switches do you need special switch support
for this?
No no so the big TCP is only local to the node so like the packets on the wire they
will still be your MTU size packets it's just that the aggregation on like for the local
stack is much bigger so it's it doesn't affect anything on the wire that's a nice thing.
And for the zero copy?
I mean like for the zero copy you need to change definitely the MTU which also affects
the rest of your network of course so that's one thing that would be required for that.
Thank you.