Hi everyone, so that's the last speaker of the day, so I'm going to talk about community
spot connecting to multiple networks.
So Doug already spoke about this in a slightly different way, I'll take a slightly different
approach.
So first a few things about myself, I'm a software engineer at Cisco working on container
networking things, and I'm a maintainer of Calico VPP, which is going to be the topic
of this talk.
This talk is also a bit particular, it's a result of a collaboration effort with many
awesome people like Tajera, Intel mostly and Cisco, and direct on collaboration with
Mritika Ganguly, which is a P at Intel, but she sadly couldn't be here today because it's
quite far from the US where she lives, but I'll do my best to present her work.
So first, a bit of a background story of this work, so in the world of employee applications
Kubernetes has really become the solution of choice when it comes to deploying large
scale services in various environments because it provides the primitives for scalability,
so Metal LB that we saw in a previous talk, services, health checks and so on.
It also provides the uniformity of deployment, and it's also far from the sequence, so you
don't need to know what you're running on.
But coming from the CNF land, so trying to deploy a network function in this environment,
the story is not the same.
So I'll define a bit more what I mean by CNF because it's a bit different between the
standard CNF use case, the 5G one.
What I mean by that is, so I'll take an example for the sake of this presentation, so typically
what I mean by that is the wire guard head end.
For example, you have a customer and you want to deploy a fleet of wire guard head ends
to give that user access to a resource in a company network, so typically a very prior
printer that everybody wants to access to because people like to print.
So the particularity of this use case is that it's dynamic enough to benefit from the abstraction
that humanities brings, and I've lost my mouse.
So typically load balancing, scheduling and those kinds of things.
But it has a lot of specific needs, for example ingress has to be done in a particular way
because you have some wire guard encrypted traffic, so typically you want to see which
IP it's coming from.
You also constrain on how you receive traffic because typically, and that's the place where
you need multiple interfaces to go into your pod.
And you also require high performance because encrypted traffic, so typically you want those
things to run fast and you have a lot of user using them.
So not for that printer, but assuming it's a bigger use case.
So we tried to design a solution for that.
So there are lots of components at play, I'll try to go quickly into them.
So in the top we have our application, so here the wire guard VPN head end.
We want to deploy it on top of humanity, so we have to choose a CNI, so we want to have
it with Calico, mainly because of the cuteness of the cat, but also because it provides a
really nice interface into supporting multiple data planes and also a nice BGP integration
that allows to tweak the way we process packets.
And for carrying packets we use the FDIO's LPP as a data plane that gave us more control
on how packets are processed.
And so that allowed us to go deeper into the way networks actually manage at a really low
level.
There are also other components that are going to play, but more on this later.
So I'm going to go quickly over Calico and VPP because they have been presented many
times.
So in short Calico is a community CNI, providing a lot of great features, policies, BGP, support
for really huge clusters, and the point that's important for this presentation is that it
has a very well-defined control plane data plane interface, allowing to plug new performance
oriented software underneath it without much hassle, and that's what we are going to leverage.
So we choose to sleep VPP underneath Calico, first because we were originally contributors
in this open-source user space networking data plane, so it was a solution of choice.
But also it has a lot of cool functionalities that are built in and it's extensible.
So I am doing a bit of publicity for the software I'm coming from, but it was a good tool for
this use case.
And also it's quite fast, so it really fits the needs for this application.
So how did we bind the two together?
What we do is we built an agent running in a demand set on every node, so deployment
is the same as a simple pod, just with more privileges.
We registered these agents in Calico as a Calico data plane and used their GRPC interface
and their APIs that they exposed to decouple control data plane.
That agent listens for Calico events and then programs VPP accordingly.
And we also built a series of custom plug-ins for handling that, servicing and so on.
And we tweaked the configuration so that things behave nicely in a container-oriented environment.
And with all this, we have every brick to bring VPP into the clusters and so to have
really control on everything that happens, indeed, the communities networking.
How does that happen under the hood?
So what happens exactly under the hood, what we do is we swap all the network logic that
was happening in Linux to VPP, so from this configuration to there.
In order to, so, yeah, the thing is as VPP is a user space stack, we have to do a few
things a bit differently compared to what was previously done in Linux.
So in order to insert VPP between the host and the network, we will grab the host interface,
the uplink, and consume it in VPP with the appropriate driver.
And then we restore the host connectivity by creating a turn interface in the host root
network main space.
So that's the turn tap here.
And we replicate everything on that interface, it resists the routes.
So basically we insert ourselves as a bump in the wire on the uplink.
It's not very network-ish, but it works pretty well in that configuration.
And that way we restore pod connectivity as before with turn taps instead of a VIF.
We create an interface in every pod.
And then everything runs normally, the Calico control plane is running normally on the host
and it configures the data plan functions in VPP via the agent.
So now we have the green part covered.
So all those components run neatly.
And what we achieve with that is that when we create a pod, Kubernetes will call Calico,
Calico will call VPP.
And we can provide an interface that we fully handle on a network network player directly
in VPP.
But for this specific wire guard add-on application, we need a bit more than that.
We need multiple interfaces and we also potentially have overlapping addresses.
So we don't really manage where the IPs are going to end.
So for the multiple interface part, our goal to show us was to go with multis that provides
multiplexing.
And we chose also dedicated IPAM that we patched, which we were about because it was quite
simple to patch and brought those two pieces in.
So when I mean multiple interfaces, what does that exactly contain?
So the thing is, the typical Kubernetes deployment looks like this.
So each pod has a single interface.
And the CNI provides a pod-to-pod connectivity, typically with an encapsulation from node
to node.
But in our application, we want to differentiate the encrypted traffic from the clear-text
traffic, so before and after the head end.
But we still want Kubernetes as the end to operate.
So we still want the nice things about Kubernetes, so service IPs and everything.
So it's not only multiple interfaces, it's really multiple interfaces wired into Kubernetes.
So it's more multiple isolated networks.
So conceptually, what we needed was the ability to create multiple Kubernetes networks.
So each network behaving a bit like a standalone cluster stacked on top of each other.
So with this, we request networks that provide complete isolation between each other, meaning
that traffic cannot cross from a network to another without going from to the outside
world.
And so that means that we have to bind Calico, VPP, so on, integration, and multist together
to create a model where everybody is aware of that definition of networks, have a catalog
of isolated networks, specify the way they are going to communicate from node to node
via VXLAN encapsulations, and have a way to propose to attach to those networks with
annotations, so that in the end, Kubernetes is aware of these networks and we can still
maintain the SDN part of the logic.
So the way this works quickly is that the C&I interface will call Calico once per pod.
So the thing is, multist will call the C&I Calico once per top all pod interface.
And we will in turn receive in origin those calls and we can map those with annotations
and do our magics to provide the logic.
And having also the IPAM patch allows us to support multiple IPs and to have different
realms where the IP lives and gets located from.
So from a user's perspective, what we expose is a network catalog where our networks are
defining CRDs for now.
We are starting a standardization effort to bring that into Kubernetes, but that will
probably take time.
So right now we kept it that simple with just specifying a V&I using VXLAN by default,
just passing a range.
And we also keep a network attachment definition from multist with one-to-one mapping to network
so that things, we don't change too many things at once.
And then we use those networks into the pod definitions.
So we reference them the multist way.
We can reference them as well in services with dedicated annotations.
And so that way we tell our agents to program VP in a way where the services apply only
in a specific network.
The policy is the same way.
And also that gives us the ability for pods to have a bit more tweaking on the parameters
exposed on the interface.
So to specify the number of queues we want, the queue depth, and also support multiple
types.
So that gives a lot of flexibility to get the performance right and to get, so first
to get the functionalities.
So the fact that we have multiple interfaces and so also size them so that the performance
is appropriate for the use case that we want to achieve.
The last nice feature of this is that as we have GoBGP support, we can pair those networks
with the outside world if we have a fabric that's VXLAN and if GoBGP supports it.
So that part is still a bit work in progress and there are a lot of things to get right.
But that's the end picture we want to go.
And this could, so if we put everything together, we would get probably something like that,
that looks like that.
So basically when the users want to connect to this hypothetical VPN and that hypothetical
printer, it would get into the cluster via GoBGP peering, so traffic is going to be
attracted to the green network, heat service IP in that network, so get some load balancing
across the nodes, then it's going to be deciphered in a pod that then encapsulate traffic and
pass it, for example, to a NAT pod running in user space.
So here I put another type of interface that is more performance oriented and then exit
the cluster on a different VLAN peered with the outside world.
So some parts still need to be done, but the general internal logic of the cluster is still
something that works and that brings the ability for container networking functions to run
unmodified with their multiple interfaces directly in a somewhat regular cluster.
So we spoke about improving performance of the network, of the underlying interface,
but we can also improve the performance with which the application in the pod consumes their
own interfaces.
So the standard way applications usually consume packets within pods is via socket APIs.
So it's really standard, but you have to go through the kernel and it's a code path that
wasn't designed for the performance levels of modern apps.
So that's why GSO came up as a network stack optimization.
But here with VPP running, it would be nice to be able to bypass the network stack and
pass the packets directly from VPP and not touching the kernel.
So fortunately, VPP exposes two different ways to consume those interfaces.
We'll mostly go into the first one, which is the memory interface.
So basically, it's a packet oriented interface standard relying on a memory segment for speed.
And this can be leveraged by an application via a simple library.
So either GoMemi, PhiliBMMF in C, or DPDK, or even a VPP.
And so provide a really high speed way of consuming that extra interface in the pod.
And the really nice thing about this is that it brings also the connection between the
Kubernetes network, Kubernetes SDN, and the pod into user space, meaning that now that
connection lives in a regular C program, we can also leverage.
So it's easier to leverage CPU optimizations and new features.
And that's where the Silicon re-enters the picture and the work from Mrytka from Intel
and their team.
So they benchmarked this kind of setup and also introduced an optimization that's coming
into the fourth generation Intel Xeon that's called Data Streaming Accelerator.
Basically, it's a way to optimize copies between processes on some CPUs.
And so what they did is compare the performance that we get with the Kubernetes clusters,
multiple interfaces, and a simple pod.
So not bringing in the old VPN logic, just doing L3 patch and seeing how fast things
could go between regular kernel interfaces, the turn, the memory interfaces, and the memory
interfaces leveraging those optimizations in the CPU.
So that gives those graphs that are really, so that have a lot of numbers in them.
But basically, I try to sum up quite quickly what this gives.
There are two MTUs, 1,500 bytes and 9,000 bytes here.
The performance for turn interfaces in dark blue.
Blue is the first MAMIF and the DSA optimized MAMIF is in yellow.
And basically, what this gives is that the performance is really, so it brings really
a huge difference between, so, sorry, throughput with DSA is 2.3 times faster than with regular
MAMIF for the 1,500 bytes packets.
And if you go with DSA enabled, it's 23 times faster than turn tap.
And with a 9,000 byte MTU, basically, you get more than 60 times faster with the MAMIF
that's optimized with DSA.
Basically the digit, so the number that's really interesting is that with 200,000, so
basically you get a single call doing 100 Gs with that.
And that without too much modifications of the applications.
So basically, you just spin a regular cluster.
If the CPU supports it, you use a regular library and you're able to consume packets
at really huge speeds without modifying the application too much.
So there is another graph looking into the scaling with number of calls, both with small
MTUs and large MTUs.
Basically that shows that we can spare calls when going, so turn tap does not scale very
well.
So the regular MAMIF scales with 1 to 6 calls and DSA achieves the same results with 2 to
3 less calls than its regular MAMIF counterpart.
So basically you achieve 100 Gs, which was the limit of the setup with a single call
in the case of large MTUs and 3 calls in the case of smaller MTUs.
So that's all for the talk.
Sorry I went into a variety of different subjects because that topic goes into a lot of different
directions.
Basically that was to give you another view of the duration we are trying to go, trying
to bring all those pieces together in a framework that allows us to make those CNFs run into
a community environment.
This work is open source.
There are the details of the tests that were done in the following slides.
You can find us on GitHub and there is also a Slack channel open where you can ask questions.
And we have a new release coming up in Beta aiming for GA that's going to go without soon.
So thanks a lot for listening, so here are the details.
And I'm open for questions if you have any.
Just one question for the sake of it, have you ever thought about some shared memory between
the different parts to eliminate the need to copy over the packets?
So we thought of this, so there are different ways to do that.
So there is the VCR which I haven't spoken about, which is a way of opening the sockets
directly in VPP.
So basically you do a list in VPP for TCP, UDP or given protocol, so like the sockets
APIs.
And that supports directly, so basically the data never leaves VPP and you can do direct
copies between processes without having to copy because everything stays in VPP in the
end.
For MMF, we don't support that out of the box but nothing forbids you to spawn two pods,
make them share a socket and it's only shared memory so you can directly do it without having
to spin up the whole thing.
So you could even do that in any cluster or directly on bare metal.
So MMF is really a lightweight protocol so you can do that just with a regular socket.
Okay, cool, thank you very much.