Okay, so thanks everyone for coming.
Today I'm going to talk about two projects.
One is Metal LB, which I'm currently maintaining,
and the other one is FRR, which kind of started integrating
in Metal LB more or less one year and a half ago.
Is anyone using Metal LB?
Awesome, okay, so if you found it less stable
in the past two years, that's because of me.
So again, the agenda today is I'll describe
what Metal LB is, white matters in the context
of Kubernetes, then I'll introduce FRR,
and then I'll talk about the integration between the two.
Some quick words about me.
Federico, I work for Red Hat almost for four years now,
and I'm part of this networking team
on the OpenShift platform that is in charge
of making it a suitable to run telco workloads
that we know have slightly different requirements
from regular application that runs
on the cloud environments.
Due to that, I contributed to a variety
of network-related projects.
I touched our primary CNI, which is OVNK.
I wrote a simple CNI plugin for using DRFs.
I put some code in Kubernetes itself,
and lately my primary focus is on Metal LB,
but please don't think that I'm a networking expert
because I'm not.
Who doesn't know anything about Kubernetes?
Okay, so I'll very briefly introduce
the concept of services for those
that don't know anything about Kubernetes.
We have our application deployed as multiple pods
where we want to scale our traffic against,
and the concept that Kubernetes gives us is services,
and with the service, we get two things.
One is a cluster IP that is a virtual IP
accessible from inside the cluster,
and the other part is the balancing part.
So the client tries to hit the service,
and then the cluster CNI in some way
balances the traffic across the different endpoints.
It's more than this, we have IP families,
we have parts, but that's the main idea.
What if we want to expose our application
outside the cluster?
Because yeah, there might be use cases
where we want to expose them inside,
but it makes sense to access them from outside.
And the main construct that Kubernetes gives us
is a service of type load balancer.
This is the definition taken from
the Kubernetes documentation.
A service of type load balancer is exposed externally
using a cloud provider slot balancer.
As we saw in the talk before,
like there is the cloud provider, Google AWS,
that gives us an IP that is accessible from outside,
and that drives the traffic towards our application.
And again, the emphasis here is on the cloud provider.
What happen when we create a service of type load balancer?
We get an external IP that is accessible from outside,
and we get the load balancer part.
So if we try to access our application,
the network infrastructure of the cloud provider
will drive the traffic toward all the nodes of our cluster
so that the CNI can do its part.
So, and this is important to understand
how Metal Lab works.
Once the traffic gets to the node,
all the rest is handled by the cluster CNI.
And in this case, this is a real network load balancer.
So again, where we don't have control
and is controlled by the provider.
So just to iterate, we get two things from a service,
from a load balancer service,
we get a stable IP that we can pin our DNS entries to,
and we get the load balancing across all the different nodes.
So now let's move to bare metal and see what happens there.
The first thing that we see when we try to create
a service of this type on bare metal
is the fact that the external IP,
we are not getting an external IP
because there is no one that is giving that to us.
And the second part is, even if we had that IP,
who is routing the traffic to the nodes
as the cloud provider's network infrastructure is doing?
And these very same two issues
are the issues that Metal Lab tries to solve.
Metal Lab is a community project
now under the CNCF umbrella.
It was originally started by David Anderson.
Then there was a handoff to one red actor,
Russell Bryant and two folks working out at King Volk,
Rodrigo Campos and Johannes Lieberman.
And during that phase,
Metal Lab went more or less in maintenance mode.
They were replying to issues and stuff like that,
but it wasn't evolving too much.
And at some point, because things went in a different way,
I started leading the project basically.
One nice thing about Metal Lab is this dichotomy.
It's used a lot in home clusters around Raspberry PIs,
but it's also used by enterprise users
in very complex scenarios.
So the first and most disappointing thing
about Metal Lab, but please don't run away,
don't leave the room, is the fact that Metal Lab
is not a network load balancer.
Yeah, this was disappointing when I started digging into it,
but let's keep in mind those two issues
that we want to solve and see how Metal Lab
tries to solve them in a, I think in a very elegant way,
interacting with an existing network infrastructure.
And the first part is the address advertisement,
assignment, sorry.
And this part is probably the most boring one.
We have a Kubernetes controller,
listens for services in the need of an IP
and tries to assign them the IP.
But what IPs are we talking about?
We, here we are not on the cloud provider,
we don't have control over the IPs.
And so in this case, the cluster administrator
is the entity in charge of providing a pool
of IPs to Metal Lab, so it can use them
and give them to the various services.
And those can be ranges, can be full-siders,
we can add multiple ranges and IPv4 and IPv6.
And then there is the probably most networking part
of Metal Lab, which is address advertisement.
So we have the address and we need to find a way
to attract the traffic to the nodes
so that the CNI itself can do its part.
And Metal Lab works in two modes.
L2, I'll briefly describe how it works.
It's more or less like a KIPA LiveD,
it collects one node as the leader of the IP,
it replies to our request, sends out gratuitous RPs
when a failover happens and you can only have one node
as the entry point for the service.
And then there is the part that I'll dig more into today,
which is the BGP mode.
In BGP, we leverage the interaction
with BGP-enabled routers in order to advertise them.
Yeah, so this is taken from the BGP RFC,
the primary function of a BGP-speaking system
is to exchange network reachability information
with other BGP systems.
So this is exactly what we need.
We need to find a way to say,
hey, if you want to reach this service IP,
which I'm assigning to my load balance service,
then you should go through this set of nodes
because then, again, the CNI can do its part.
And this is exactly how Metal Lab works.
Each node acts as a mini router establishing BGP sessions
with externally configured routers.
You need to make Metal Lab be aware of the existence
of those routers with some configuration.
And then, when we create a service,
it will start advertising the routes to the router
so that there, and this is a bit bigger
for those in the back, so that the router knows
that in order to reach this virtual IP,
it needs to route the traffic towards these nodes.
So again, the traffic gets to the node,
that's the end does the rest.
And in this case, compared to the L2 mode,
we get fully load balancing through ECMP routes.
And the scenarios can be more complex.
We can have multiple routers.
We can have, we have some knobs to drive which routers,
which peers we want to advertise, a given service.
We have other knobs to say, hey,
I want this BGP session to be established
only from this set of nodes
because maybe they belong to different areas.
And of course, we can have cascading routers
and this is like regular BGP.
The configuration looks something like this.
So we still need the set of IPs to get the metadata
in order to have our services assigned
to have them assigned to our services.
And then we have these other item,
which tries to describe the properties of the BGP session
that needs to be established with the different routers.
And we have a few features here.
We have BFD support.
We have node selectors.
We support IBGP and EBGP single and multi-hop.
But even if we are acting as a mini router
because metadata-based purpose is only to advertise,
routes outside, we refuse an incoming router to the node.
Because again, that is not metadata-based purpose.
How it works under the hood.
The architecture is pretty simple.
We have one single pod that is the controller
that is in charge of the IPAN part of MetaLEB.
So it's in charge of reconciliating the services
and the configuration with those IPs
that needs to be assigned to the service.
And again, there is not too much network in this side.
The other part is the speaker.
And the speaker is the part that is in charge
of handling the networking side.
We run it as a demo set.
It runs on each node.
It runs on the host network.
So it is in control of the configuration
of the host networking.
And it handles the announcement part.
So both the L2 and the BGP1.
And now I will talk a bit about the history.
Originally, the BGP part was done
using a native Go implementation that
was implementing a subset of the BGP protocol.
This was before I started to maintain and to contribute
the project.
And at some point, there were a bunch of features
that were being asked by the users.
And the people that were maintaining the project
at the time had this discussion about,
should we start extending the Go BGP implementation
to cover all these scenarios that the users are asking?
And the result was, we shouldn't.
We should not reinvent the wheel.
We should leverage something that is already
doing that for us.
And that thing was FRR.
FRR is Internet Routing Protocol Suite for Linux
that is well-established.
And it implements all the stuff that we were looking for
to add to Metalhead View.
So as the result of this discussion,
Metalhead View was extended with an alternative mode
that is turned on by a configuration flag,
where all the BGP part is handled by FRR.
An FRR configuration looks like this.
We describe our autonomous system number.
We describe the properties of the neighbors.
And then we describe the prefixes
that we want to advertise around.
And we can do more.
We can set rules for each neighbor.
And associated to those rules, we can say, hey,
if the IP belongs to this set of IPs, then we can add communities.
We can have local preferences.
We can block this IP for this particular neighbor.
And this is all the stuff that we had to do.
We were required in Metalhead View.
So now, in BGP mode, the way Metalhead View works
is that we are running multiple containers inside the speaker
pod as we had before.
And one of them is running FRR.
And because all the containers share the host network
namespace, then what we need to do now
is to apply the proper configuration to FRR
so that it can do its part.
So this is what we have.
On one side, we have all these continuously evolving
configurations that we received from the Kubernetes API.
We have the services that come and go.
We have the new routers that we want to configure.
We have this BGP advertisement that
allows us to set some properties on the advertisement itself.
On one side, what we want to achieve
is the corresponding FRR configuration
so that FRR can do its part.
And this is done by some code that renders all this changing
stuff and continuously reconcile it
in some sort of internal data.
We pass it through the go template engine.
We then generate the configuration for FRR that we want.
But we are not finished yet because then we
need to instruct FRR to apply the new changes.
And luckily, what we can leverage
is this Python script FRR.reload that does a lot of stuff
for us.
So this comes together with FRR and calculates
all the delta between the current configuration
and the configuration that we want to achieve
and applies all those commands to FRR.
And so again, we get to the right configuration corresponding
to the Kubernetes configuration.
Assuming that we are doing our part here.
So I don't generally add memes to talks,
but I thought that this was particularly relevant.
Because leveraging FRR allowed us
to focus on the business logic on what our users were asking
to us without having to worry too much about the protocol
and its implementations.
And that helped us a lot.
In order to add new features to the project,
we added the BFD support.
Seamlessly, we added the VRF support.
IPv6 and DOS stack were something
that we were missing in Metal LB,
and they came out naturally.
But this doesn't mean that we don't have challenges.
Probably the biggest challenge is the fact
that on one side, we had an existing API that was already
there and was fitting well with the Metal LB use
case where the focus was the service.
So I want to apply all this logic to the service.
On the other hand, FRR thinks in a slightly different way.
So there is a good amount of logic
in doing this API contortionism in order
to have one API to fit the other.
And again, that's because we wanted
to be backward compatible.
And probably the second most interesting challenge
was the fact that Metal LB was known to be super stable.
Like, we came and we replaced the core mechanism
about the interaction with the routers,
and we wanted to make sure that we weren't breaking too much.
On top of that, we started also to add new features.
And again, we were changing a lot in very few times.
And at the time, there wasn't a proper CI mechanism that
was covering all the cases.
So that was quite a challenge, because again, Metal LB users
were used to having something that was stable,
and we were promising that we were replacing the implementation
without in a compatible way.
So the problem was we want to be able to test something
like this, where we have multiple servers.
So you have one router.
You might have multi-hops, and you
have all the configuration knobs that Metal LB has.
So and then you have node selectors.
You have the BFT that we were adding.
We have communities and a lot of stuff.
And this was something that I wasn't keeping me sleeping.
So we started thinking, how do we set up a proper CI for this?
And we use kind.
Does anyone knows what kind is?
Of course.
OK, so basically, with kind we are able to replace something
like this with something like this.
So each node is running inside the container.
The external router is now replaced by FRR.
So we use FRR both inside Metal LB
to do the implementation, but also outside to validate
that everything is working.
And now we have even control on this kind of network,
because this is the Docker bridge.
So this allowed us to add a test suite,
where we apply the Metal LB configuration
and the FRR configuration corresponding
to that Metal LB configuration.
And we can inspect the external router
so that all the advertisements, all the sessions are up
and all that kind of stuff.
And obviously, we can even access the service from outside.
And we can test more complex scenarios.
We can test multi-hops.
We can test IPv4, IPv6, dual stack, and so on.
And most importantly, this can run on our laptop.
So even the development phase is now
easier to move forward.
And also we are able to run this in the UPS MCI
as under GitHub Actions.
So wrapping up, FRR made us able to focus
on the business logic of our applications
instead of having to re-implement all those protocols.
To the point that sometimes writing this test suite
takes more time than implementing the feature itself.
And I think this is a nice example
of the interaction between two different projects.
These are some resources.
The first section is about Metal LB.
We are trying to be active as much as we can
on the Kubernetes Slack channel.
We always monitor upstream issues.
If you want to ask questions, that's the right place to do.
Again, I'm responding on daily basis.
And I think that can be said also for the FRR community.
They are super active in their Slack channel.
And with that, I want to thank them
because they made our life super easy.
And I'm done for today.
Any question?
Hello.
Is it possible in a few words to explain
when I should use ARP or BGP?
If you are working locally, if you have a home lab
and you don't have a BGP enabled router,
that means that the only alternative that you have
is L2 and R. Otherwise, the BGP mode requires routers
but also gives you more power because you
have a proper load balancing across all the different nodes.
One more question here.
Hi.
Thanks for your call.
We have Metal LB running on our worker nodes.
And our worker nodes are talking BGP to the host,
routing to the host.
So FRR is running on the metal machines as well.
So can we configure Metal LB with FRR
routing without conflicting ports?
So because we already have a problem?
Yeah, correct.
You can't.
And that's something that we are trying to think to solve.
One of the idea that I have in mind,
but this is super early stages.
Again, having Metal LB able to use an existing FRR instance,
but that comes with challenges because it
expects to reconfigure all the configuration
and it might have conflicts with what you have currently.
So the go-to solution now is to have different ports, I guess.
OK, thank you.
I have a user having a namespace where he uses a lot
of IP addresses without control.
A lot, sorry?
I have some users that are allocated separate namespaces
in Kubernetes.
And some of them are using a lot of IP addresses.
Can you limit the IP addresses that matter?
For a namespace.
Yeah.
So one of the things that we added over the past year
and I think it's going to go out in the next release.
It's merged on main, but it's not released yet.
It's namespace selectors and service selectors
in the IP address pool.
So by doing that, it will solve your,
that's meant to solve the multi-tenant problem, where
you want to have a given set of pools associated
with a given tenant and not more.
So we are trying to remove the control
from the service itself, as it was before,
where you had to set a fixed IP or the pool
that you were pulling inside the service
and having it as part of the cluster configuration
so that the cluster administrator can do that.
And there won't be cases where applicants should
are abusing of the cluster.
Well, Morio.
Thanks for your talk, also.
I have a question about the possibility
to have Metal LB coexisting with Calico,
because I think in that case, we have two BGP speakers
on the same note, and I had issues with that in the past.
Yeah, I'm not a Calico expert, so I
know that the existing configuration suggested
to disable the Metal LB BGP part and let Calico
advertise the services.
But that's all I know, and I haven't dug into the thing.
One more?
Just to follow up on the last question,
is it possible to run Metal LB on a different port for BGP?
Yeah.
Actually, it should be, but I don't remember,
because it has to be a process parameter.
I should check.
If you catch me out later, I'll tell you.
Because for the neighbor part, yes, it's like that is clear.
For the Metal LB part, I don't remember.
OK, thank you.
Thank you.