Okay, ready for our next talk?
Next talk is by Mariam and she's going to talk about,
I'll give us a hybrid networking stack demo.
Thank you.
Hi, everyone.
My name is Mariam Tahan.
I'm a software engineer at Red Hat.
And today I'm going to talk to you about a concept
I've been researching, and I've coined hybrid networking stacks.
So if anybody has better names as well,
I'm open to the suggestion.
So what I'm going to do is I'm actually going to introduce
what a hybrid networking stack is.
We're going to talk a little bit about an open source project
called Cloud Native Data Plane or CNDP
that gives us an example of such a networking stack
or at least some components of it.
We're going to have a live demo with a star there
because we're going to cross our fingers and toes
and pray that it all goes to plan.
After that, I will try and sum up what we discussed
and hopefully there will be some time for Q&A at the end.
Okay, so what is a hybrid networking stack?
Well, it's actually a networking stack for applications
that want to take advantage of the XDP hook and AFXDP in particular
without having to reimplement the full networking stack
in user space, but rather lean on the existing Linux stack.
It relies very heavily on the concept of control plane
and user plane separation.
So parts of the stack can run in user space
and other parts of the stack can run in the kernel.
And even if they're part of the control plane,
they can run either in kernel or user space
and the same for the user plane aspect.
You can run stuff either in the kernel
or in user space as part of that networking stack concept.
This concept relies very heavily on the principle
of classifying traffic into application-specific traffic
and non-application-specific traffic.
And application-specific traffic is redirected
to the user plane and non-application-specific traffic
is redirected to the control plane to be handled.
So in that way, applications only really need to process
the types of traffic that they're interested in.
And what's really important then is that you filter
this type of traffic as early as possible
in your networking stack.
So if your NIC hardware supports that filtering,
you can take advantage of that.
If it doesn't, then you can always rely on
EBPF at the XDP hook to be able to do
that level of filtering for you.
So in the example I'm showing here on the slide,
you can probably consider FRR and the Linux Networking Stack,
the control plane.
FRR is just an open-source routing protocol suite
that's for Linux.
And then on the user plane side,
you would consider the CNET graph from CNDP,
your data plane or user plane for this demo.
The CNET stack that comes with CNDP,
I'll just talk about it for a minute before we dive into
the next topic, is based on the graph architecture from VPP.
So with VPP, the concept was that you could build
your whole application or parts of the stack
that you want to leverage using a graph.
And then your packets are processed by traversing
each node in this graph.
And they're processed in batches as well to keep
your instruction cache relatively warm,
and you got all the performance benefits from doing
all of that good stuff.
So the CNET stack is based on the exact same concept as that.
And obviously, as your packets traverse the nodes,
they're either terminated as part of that stack,
they're either forwarded on, or they're dropped,
depending on the decision that was determined previously
by the control plane piece for your application.
So let me introduce CNDP to you folks.
CNDP, our cloud native data plane,
is an open source framework for cloud native packet
processing applications.
It's actually built on the performance principles of VPP
and DPDK, but it doesn't have any of the resource
demands or constraints as it's completely abstracted
from the underlying infrastructure.
It actually is completely written
using standard Linux libraries also.
So what CNDP gives you is really three things.
The first thing it gives you is a set of user space libraries
for accelerating packet processing for your application,
cloud application or service.
The second thing that CNDP gives you is that CNET graph
is part of the hybrid networking stack,
and also a net link agent that's capable of communicating
with the kernel to retrieve relevant information,
like routing information and so on.
And the last thing that CNDP gives you
are the Kubernetes components to be able to provision
and manage actually more so an AFXDP deployment
than just a CNDP one.
Those components are the AFXDP device plugin,
which provisions the net devs that you want to use
for AFXDP and advertises them up to Kubernetes
as a resource pool that your pods can then request
when they come up.
And then you have the AFXDP CNI,
which essentially plums your AFXDP net dev
from the host network namespace
into the pod network namespace.
So just one last point on CNDP before we move on
is that it actually supports multiple IO, packet IO backends,
not just AFXDP,
but for the purposes of this hybrid networking stack
we've focused in on AFXDP itself.
Okay, so it's nearly demo time.
So, excuse me.
So what am I going to show you?
I'm actually going to show you CNDP FRR vRouter
that we built.
Originally, I set out to see, you know,
could I build some sort of a hybrid networking stack application
that could accomplish, you know, DPDK-like speeds,
but leverage completely, you know, cardinal smarts.
And so the scenario we came up with was that we would have
two clients, client one and client two,
residing in two different networks,
network one and network three,
and they're interconnected via a pair of vRouters,
which learn routes using OSPF.
So what the demo is going to be
is we're actually going to bring up four Docker containers,
client one, CNDP FRR one.
We actually call this container CNDP FRR two,
but for the purposes of the demo,
I'm only going to run FRR in it, just to show it full interworking.
And client two will then be our last Docker container.
At the start of the demo, we're just going to bring everything up.
No FRR will be running, no CNET stack will be running.
And so when we try to bring from client two to client one,
we're going to see nothing happen.
And then we're going to bring up all the components in part,
see the routes being learned,
hopefully have a successful ping,
and maybe even, you know, run an IPerf session
between client one and client two also.
So if we just zoom into this CNDP FRR node for one second,
I just want to show you one thing, I guess.
So we can see here it's going to have two vEath interfaces,
one connected to net one and the other connected to net two,
and these are here.
We're going to inject an EBPF program on the XDP hook
that's going to filter all UDP traffic to CNET graph
and non-UDP traffic to the Linux networking stack.
So actually one of the other things I'm going to show you
is that we're not going to see ICMP traffic
traverse through CNET.
And then when we run IPerf with UDP traffic,
we're going to see the actual traffic flow through CNET also.
So here we go.
Let's just check that we have nothing running.
Yep, that's fine.
And I presume everybody can see the text.
Okay, cool.
Okay.
So all the script is doing is setting up the four containers
and the relevant networking between them right now.
We can ignore the permission denied,
but we didn't see that for now.
So we actually see we have four Docker containers here,
Client 1, Client 2, CNDP FR1, and CNDP FR2.
And if we try to ping Client 1 from Client 2,
essentially nothing happens.
Okay, so let's start up our FRR agent on CNDP FR1
as well as the CNET graph.
So, sorry about the formatting.
It looked a lot better when I was presenting.
But the key part here is if we try and check the routes,
what we see is the two net devs that are attached to CNDP,
or the CNDP FR1 vRouter,
but most importantly we just see Network 1 and Network 2.
So let's start up the FRR agent on this node.
So if we have a look at the information that's been set up so far,
we can see this vRouter has an IP address.
It's adding Network 1 and Network 2 to the same OSPF area.
And if we try to show IP OSPF neighbor at this point,
it hasn't learned anything
because we haven't started FRR on the other vRouter.
So let's go ahead and do that.
And here this vRouter has my IP address
and is adding Network 2 and Network 3 to the same OSPF area.
And if we show the OSPF neighbor, it's picked up its opposite end
of the vRouter.
And if we do the same on the CNDP FR1,
it's also learned about the other route via OSPF as well.
So at this point, if we actually try to ping again from client 2 to client 1,
we can ping.
And actually, if we check the routes on CNDP,
we have the new Network 3 added in.
And just to show you that no traffic is flowing through CNDP yet,
this is ETH0 stats for RX and TX.
We see they're still 0 and the same for ETH1.
So let's kill that off for the moment
and try and run an IPerFUDP session between client 1 and client 2.
And this time we should see traffic flow through the CNET graph.
And if we check here, you can see an increment in the stats.
And this doesn't show as nice as I hope.
And this kills the app.
Let's try it one more time.
Unfortunately, I won't be able to get this right just yet.
Oh, there we go.
OK, let's try and run it one more time.
OK, folks, bear with me.
So we can see sort of IP4 input node at the top here,
an IP4 forward node,
and they're passing UDP traffic through those nodes.
Now, we're not going to the UDP nodes that are listed there
because obviously traffic isn't destined for the CNDP,
FRR, V-Router, they're destined for the client attached to it.
And that's why they're forwarded on.
Applications can also hook on to the CNET graph
via a socket-like architecture.
All the function calls look exactly the same like a socket,
except it's just called a channel,
and you prefix all of your normal socket calls with channel underscore
before hooking up into the CNET graph.
So that's the demo.
So the next step was to essentially take that CNDP FRR-Router
and put it through a heck of a lot of permutations
in terms of interfaces that we hooked it up to,
leveraging things like XDP redirects
between the two V-Router instances and so on
to try and see what kind of levels of performance
could we push this to.
And so what we noticed was for AF-XDP,
the performance is completely dependent on the deployment scenario.
So for north-south traffic that was coming in on a physical interface
or out of a physical interface with AF-XDP in native mode,
so hooked in at the XDP hook,
we actually had this example yielded comparable performance to DPDK.
However, while we moved to something that was completely local to a node,
so east-west type traffic with all virtual interfaces and AF-XDP,
while the performance was still better than vanilla V-Eat
for AF-XDP in native mode,
it wasn't what we had expected it to be.
So there's definitely some level of optimization
that we need to look into on that front.
And then we tried one other thing which is AF-XDP in generic mode,
so that's your program hooked in at the TC hook,
and that actually yielded a better performance than native mode.
But again, that goes into some optimization requirements are needed on that front.
So just to sum up, I guess,
we set out to show it was it possible to build some sort of a hybrid networking stack.
I think the building blocks are there for sure.
I think we've demonstrated that it is possible to do something like that,
especially for these high-performance use cases that want to take advantage
of internal fast paths and essentially XDP and AF-XDP.
There's obviously an opportunity as well to make sure that we hook in
EBPF a lot more into the puzzle,
especially from the user plane aspect,
not everything has to go into user space and so on.
So I just want to summarize in terms of generic challenges
that we have noted for AF-XDP.
The first one is that we still can't take advantage of hardware offloads.
It's been great to see the XDP hence K-funk support getting merged into the Linux kernel,
or at least agreed on as model and then merged, which has been fantastic,
and it will form a great cornerstone for a lot of this work.
The only thing that I would ask is that we make sure that for the containerized environment,
we put the onus on the infrastructure to lifecycle manage the BPF programs
and to take that level of responsibility and privilege out of the scope of the application.
So the application doesn't need to know any special formats or have to,
especially if they're using AF-XDP, they don't need to know any special formats
or have to do special compilations of BPF programs or anything along those lines.
That should all be managed on the infrastructure side.
The next thing that's been a gap was really jumbo frame or multi-buffer support for AF-XDP,
but we've seen lots of activity on that in the last couple of months on the mailing list,
so hopefully that's something that we can take off the list very, very soon.
And lastly, there's going to be some need for some level of optimization of AF-XDP
in native mode for VEATs, and just some links for folks if they're interested
on some of the stuff that we used for this talk, sorry.
So thank you very much folks for your time, I really appreciate it.
And it's been a pleasure presenting on my first podcast.
It's like a bucket list I should have ticked off there, so thanks a lot.
Thank you for the talk. We do have ample time for questions.
Thank you for the presentation.
I have a question about XDP, so does it run on hardware or is it in software?
The XDP hook itself is typically supported by the drivers,
so actually I think there's a good host of drivers that support them right now,
most of the Intel ones, a good few of the Melanox ones as well.
The thing with AF-XDP is if the hook isn't natively supported by the driver,
it automatically falls back to the TC hook, which is what we call generic mode,
and it'll still work there except that you don't get the raw buffer from the driver,
you will essentially be working with the equivalent of an SKV.
So there's some level of allocation and copy that happens there before you can process the package.
Okay understood, thank you.
More questions.
Okay then, thank you for the talk.
Thank you for being here.
Thank you very much, really appreciate it.