Okay, so good afternoon.
My name is Miguel Duarte.
I'm a software engineer working for Red Hat and the OpenShift Virtualization Networking
team.
Well, in this talk we're going to be discussing an SDN solution for both types of workflows
so you can have pods and virtual machines in the same network and the use cases that
this SDN provides and a little bit how it works.
There are going to be some demos as well.
So let's jump to the agenda.
All right, so first thing we're going to do is explain the motivation, like what drives
us to have like to do this and the actual problem we're trying to solve.
From there, there's going to be a little short introduction that depends how deep it is going
to be.
Well, it depends on a few things.
And then I'm going to walk you through the use cases for this SDN solution, show the
demos, finalize with the roadmap for the future and with the lessons we've learned during
this development.
So first thing, how many of you have used or like worked for stuff that has anything
to do with Kubernetes?
Like yeah, pretty much everyone.
How many of you use Kubeverts or know what it is?
Well, more than I thought.
Okay, cool.
So the introduction is not going to be that deep.
But yeah, let's start.
First thing, going to be discussing the Kubernetes networking model.
Like as most of you will know, pretty much it's very simple.
And one of its few premises is that any pod that is deployed on the Kubernetes cluster
can contact and can reach any other pods in the Kubernetes cluster.
Like basically you have cluster-wide communication between whatever type of workloads that are
deployed in your cluster.
Without NAT, by the way.
So another thing you get as a byproduct of that is like VSC and all that, it pretty much
it configures a way for you to reach the outside world.
So you get free batteries to reach the outside world, to reach the internet.
The thing it does not allow you to do is to connect to a pre-existing network.
If it's, for instance, I say you want to connect to a database that's deployed on an existing
network, well, you're out of luck.
Kubernetes does not solve this.
More things, if you, for instance, you want to deploy a VNF, for instance, and you require
more than an interface, Kubernetes will also not do that for you.
There are solutions out of three, but we're not going to go there right now.
So the motivation for our talk pretty much comes that you don't have like an entryway
for you to access like stuff on physical networks and to get access to additional networks.
The default cluster network that you have that comes bundled and Kubernetes gives you
for free or whatever Kubernetes distributions you have give you, well, it's not suited for
all types of use cases.
For instance, for virtualization, like the ones that you, of you that use Qvert, should
know like you'll get IP addresses and well, IPAM management in for virtualization is extremely
tricky and it will not mix well.
Pretty much because you have like different IPs when you migrate from the source to the
destination pod and that thing will not play along correctly.
And finally, in virtualization, you typically like use secondary networks to do all sorts
of east-west communication and you pretty much rely on the default cluster network just
to like get batteries of the Kubernetes services, stuff like cluster DNS and things like that.
So on your secondary networks, you need to figure out other ways.
So you could do like bridge CNI and other types of plugins, but that leaves like your
operation teams will need to know how to debug yet another totally different solution.
Where admins will need to know and configure yet another like bunch of solutions and depending
on the use cases, you'll realize that this plugin will work, but this other will not.
So like the matrix of things that your operations team has to know and your administrators need
to learn how to configure, it skyrockets like it's and it becomes literally too expensive
to actually handle.
So now the objectives we have the first is we want these cluster admins to like go do
something else like we want to push all the complexity of these different sorts of use
cases and this mix and match technology to be pushed from their heads and from this lots
of tools that they need to learn and know.
We want to push all this complexity to the network.
And finally we want to have like a single like a single plugin be able to handle like
a multitude of use cases.
So pretty much what we want to have is like to have the whatever the CNI that comes bundled
with the with our Kubernetes distribution, we want that to be able to to to work properly
both for the cluster default network and for the secondaries.
Okay, so very short introduction now.
So Qvert.
Qvert is a Kubernetes plugin that allows you to run virtual machines inside pods.
So you basically get two different types of workloads.
You have pods and VMs and like you manage them from the same from the same solution.
As a like an implementation detail like the virtual machine actually runs inside of the
pod.
And each pod has like a live version instance running in it and the QML process and all that
and like just to finalize you have like the networking requirements that virtual machine
has is a lot more than a pod like a pod is something like entirely stateful.
It's like a cattle you just kill it and a new one will spawn and do it a new thing while
a VM is stateful and you need to treat it very careful carefully.
Now kind of a little disclaimer our SDN solution that we developed uses Oven and Oven stands
for open virtual network and it is essentially like an SDN control an SDN control plane to
open V switch.
So you have like a bunch of you have open V switch installed in each of the nodes of
the cluster.
You have Oven on top that is kind of rendering open flow and installing it in each of the
open V switches in the nodes from like higher level entities like you have things like a
logical switch that grants you connect L2 connectivity between the workloads on these
two nodes and this thing gets rendered into open flow and installed to the nodes.
Then we have Oven Kubernetes on top of it.
It's a CNI plugin that what it does is translate from Kubernetes entities into Oven logical
entities.
So for instance when we have like a secondary network what we end up having is a logical
switch when you have like pod attachments what we have is logical switch ports that
are connected to the logical switches and for instance in network policy is nothing more
than like a port group that associates a list of ports of logical switch ports to a bunch
of ACLs.
Alright, so supported use cases as I said in the initially in the motivation section
for virtualization use cases mostly you do not rely on the default cluster network to
do east-west communication.
You use secondary networks.
So that's the first use case we are focused on is east-west communication.
So as you can see here like pretty much what we end up having like these things here are
pods or virtual machines it doesn't matter what we actually end up doing is attaching
a new internet network interface to it configure it and what we get is like the logical view
of having like them connected via a cluster wide switch.
So that's literally what we get a cluster wide switch a connection to this cluster
wide switch and everybody that is disconnected that is connected to it can communicate across
that network.
There's a short demo right here and we'll see that oh god no internet I knew that that's
why I have this terminal here.
I'm really sorry for the font size but if I put this a little bit bigger it'll basically
mess up like the window configuration.
I hope you can still see it.
So first thing we're going to see is what the network configuration is.
I'm not sure if you're used to using MULTUS the ones of you that use Kieferd I guess they
are so this is the first thing that we need to look at the network attachment definition.
This explains this pretty much holds the CNI configuration from which the CNI plugin will
just configure networking for your pod.
So the interesting thing here is pretty much like the name of the network so the idea here
is that the networks are not namespaced but the network attachments are.
So this means that if your network admin wants to grant a namespace connection to a network
he needs to provision or she needs to provision one of these like in the proper namespace
and this will connect your namespace to this network.
Finally the oops sorry it does not go back.
Another interesting thing that was there was the topology which is layer 2 so pretty much
what we have is an overlay network this is totally disconnected from the physical network
that allows you to have like east-west communication.
And we do not have IPAM because IPAM for work loads is very tricky and we'll see more on
that later on.
So we're connecting two different namespaces as I said.
Now we're going to provision these into the cluster like fun fact this is like all lazy
so I just put like a bunch of stuff in the cluster but nothing happened yet.
It's just like the definitions are provisioned and nothing else and now we're going to show
the workload definitions.
Here they are so we have one virtual machine remember we do not have IPAM in the network
so we need to configure the IP statically you have there in the bottom we configure the
IP statically using cloud init in the VM and that's its IP 192, 168, 200 dot 10.
And then we have like our pod we have two pods here specified we have the blue server
pod and you have like a yellow server pod and the blue pod has the dot 20 IP that we
configure using the network selection elements this is like multis lingo and what this is
doing is exposing like an HTTP server on port 9000 and what we want to do is so we have
two servers the blue and the yellow and we have the VM is the client and we're going
to be curling to each of the servers by their hostname so that's what we're going to see
in well let's first provision this the windows in the bottom are so we can kind of see when
they're ready so the old service ready the tenant so the VM is booting up let's speed
up this part so we're now going to log in via console to the virtual machine and we're
going to curl both the servers and they're going to echo back their hostnames.
So God it's going to take forever if only I had internet and I could play a video oh
God.
Does anybody know how you can ask an emma tell it wow amazing I don't know.
It stopped again.
Okay it's playing cool so yeah login via console I hope it did not stop what's happening
so yeah I should have known better but again we're going to log via console the UI of this
thing is absolutely preposterous like I don't know if it's playing or not.
Okay so yeah you get curl to the dot 20 IP address it replies with the blue server thing
we do the same thing to the dot 30 IP address it replies with its hostname which is the yellow
server this concludes the first demo which shows us like east-west communication between
well different workloads in different namespaces.
Now going to the second use case which now we want to have like remember the motivation
slides where I mentioned stuff about accessing things on a pre-existing physical network
that's exactly what we're going to be seeing now so as before we see like we have a logical
view of a cluster wide switch the difference is that this switch is actually connected
to a physical network and you can access stuff that's there in our example it's going to
be like a database that has well the data the VM needs.
The first thing we need to kind of elaborate a little bit is that you need to configure
the physical network so first of all it's not something that a typical user will get
access to it needs to happen by a cluster admin and for that we're going to be using
two tools first NM state and Kubernetes NM state.
NM state is basically like a declarative tool that configures networking you just give it
like the desired state I want my network to be like this and it's going to go while punching
buttons attempting to make the current state be what you desire to be if it fails it rolls
everything back so no changes to your network so it cannot destroy it and if it succeeds
it'll tell you you succeeded so basically what we want to do is use Kubernetes NM state
which is kind of a cluster wide thing send YAML specification to the cluster and all
my network specification will be applied in all the nodes in our cluster so it would
look like this so in the in the left we have an example of a policy and in the right we
have like a diagram of the topology we're trying to do here so if you look here we this is
going to be applied to all the Kubernetes worker nodes because of this node selector
that we have here and what we're going to do is create an OVS bridge in each of these
worker nodes attach this ENS for interface to the OVS bridge and then using this these
oven bridge mappings in the bottom we're going we're saying that we want traffic from the
network called default network to be sent to the OVS bridge called BRX and we want the
traffic from the tenant blue physical network to be sent to the OVS one bridge it's literally
what you diagram you see there in the right now we are granting access to from workloads to the
physical network you should not mean should fret carefully when you do this and for that we need
to have like micro segmentation this pretty much is like what you have in for the primary network
for the cluster default network network policies this is the exact same thing but applied to
secondary networks so in our example what we want to have is like a virtual machine that
wants data from the database but we do not allow it to actually consume the data directly from
the database so we expose that information from a pod so the pod actually can connect to the
database and expose this information from a RESTful API over port 9000 so this is kind of what we
want to do ensure that the VM cannot reach the database directly over the port the posgrasql
port but and ensure you can using well this tiny pod as its proxy data proxy so again another
demo this is going to be a disaster I have tempted to tell you to just check this at home but how
many times we have more than five minutes right again this does not work sorry it's the other cast
so now in this demo what we do have we do have two namespaces data consumer and data adapter we
just provisioned them let's first thing some information like this I'm running a kind cluster
here and I botched this again so I'm running a kind cluster here and so my Kubernetes nodes are
running basic as containers in in my laptop and so the physical network that we see in the diagram
is basically like my laptop it's going to be connected by a Linux bridge in the in the laptop
and for that I need to kind of since I'm using a VLAN I'm going to need to pre provision the
VLAN and that's the interface you see here in the bottom like this podman 1.123 it's a VLAN on
top of the Linux bridge management interface I'm going to show this again and so again VLAN
1.123 that's subnet and we have a database running in containerized database running here and we
are going you see we have access to the database now let's check our manifests
really sorry so I think you should check the demo at home and we have five minutes yeah I don't
think this is going to work please do check the demo at home but pretty much what you'll see is
what I showed in this diagram so you'll have access on one port you'll have access direct access
from the VM to the database like you can PSQL to the data directly and to get the data from using
HTTP from the pod and then I provision some policies and then you stop having access to the
database and that's pretty much it now roadmap what what are we going for next first thing we need
to have is like IPAM for the workloads so we need to find a way to tie the IP allocation to the
virtual machine and not to the pod where the VM runs like remember the our big issue where the
virtualization is migration and that means that when you migrate the VM to a new node the pod gets a
new interface the VM is still with the old interface and basically networking is not properly
configured we wanted to address that first and that will unlock the next thing in our in our queue
which is selective selective policies so our kind of policies for the secondary networks right now
you can only use IP blocks you cannot use like semantic things like I want to allow all workloads
from this namespace we're having these labels to access this sort of workload you cannot do that
you need to specify IP ranges directly once we have these two things we're going for services
next like we want to have like exposed via services like egress from VMs and to have like load
balancer services so that you can access them directly over the secondary networks finally
self-service networks this is instead of having the cluster admin provision these for you since
network overlay a simple overlay that does not touch the physical network you could directly
use like a self-service functionality to just create the network yourself and connect and provide
east-west connectivity to your workloads okay well lessons learned this was a let's say a joint
venture like or a collaboration between both Red Hat and NVIDIA and the fun thing is we had the
exact use cases in mind like both of them focusing on Qvert but with different scope so we are a lot
more into the generic kind of world we want we give you a platform and you do whatever you want
with your platform and NVIDIA notes they have like their use case in mind which is I guess gaming in
a data center and their tooling is a lot more let's say pointed in that direction but was a really
nice collaboration and we're hoping to see more in the future on a less positive note we get the
user experience of this is not amazing like it's better than originally intended because like for
instance the thing I've showed you about the the nm state policy that was something that we came up
with because we felt like this feature is entirely unusable like people are going to be breaking their
default cluster network every time they use this or risk doing that so we've provided that but we
still have some sort of nightmarish stories every now and then because of the the way network
attachment definitions look like and how easy it is for you to get things wrong and how silently
and how these silent errors kind of creep up it's absolutely insane like sometimes things work but
not in the way you expect because it just doesn't recognize one of the parameters because you typed
it wrong but everything else works it's insanely hard and yeah I'm really sorry about the demo but
yeah thank you for listening and if you have any questions in one minute one question
it's the same thing yeah so the question yeah so the question is basically there's another cni so
we're doing this in oven Kubernetes and there's another cni cni called cube oven so it's kind of it
really screams that it does the same thing and yeah it really does the same thing the use cases are
mostly the same the thing is that they do a lot more than we do like quite honestly like their
feature set is a lot more complete than ours and we're trying to get there like if your question is
like why didn't you just use that well we do not have any let's say current stake in that cni we do
not have maintainers there we do not like we would have to try to gain entry there and in some cases
we do not like their API so we're trying to do things in another way it might seem like we're
reinventing the wheel sometimes but yeah kind of we kind of are like we both do the same thing and
their feature set is more complete but again we're trying to do more and reach their feature set
thank you for the question and I think it's that's it well