Hi everyone.
Hi everyone.
Welcome to this talk on Cluster API across hypervisors and with GitOps.
So we've got a lot of the hype words in there.
My name's Richard.
I'm a maintainer of various Cluster API providers,
more notably the AWS provider, the GCP provider and the RKE2 provider.
Hey, I'm Alex. I work together with Richard at SUSE.
I'm a KPI operator maintainer and also maintaining RKE2 provider.
So today we are going to talk about Cluster API.
This is only for the stream.
But just to speak louder.
So today we are going to speak about Cluster API, GitOps and couple virtualization providers.
So we'll briefly talk about what is KPI.
There was a previous talk about this.
But just in case you haven't been there, we will repeat something again.
We will tell you and show how Proxmox will integrate with KPI
and how GitOps can be added to there.
And then we'll replicate the same process with KubeVirt
to show that KPI can work with different infrastructure providers.
Cool. So all the demos, well the two demos in this session are available via this repo.
Feel free to take a picture of it.
It's got the full script for it.
So you can actually run this yourselves when you get home.
I'll leave that out for a second.
So who was in the last talk about the intro to Cluster API?
Okay, cool.
So you get the idea that you have a management cluster.
Oh, yep, sorry.
You have an idea that you have a management cluster
and to that management cluster you install Cluster API.
Now Cluster API is made up of the core controllers and a bunch of providers.
And you can mix and match those providers to meet your needs.
So if you have your provision in AWS, you just install the AWS provider.
Once you have that, you then declare the shape of your cluster.
It's fully declarative using Kate's custom resources.
And you apply that to the management cluster.
Then Cluster API does its magic and then it will provision the infrastructure
and then bootstrap Kubernetes on top of that infrastructure.
So we're going to demonstrate how it works on Proxmox.
So just a couple of words about this in case you don't know what it is.
It's a virtualization platform.
It's open source and includes anything you need for your virtualization purposes.
One thing to note is there are two providers for Proxmox if you go out there.
So one requires you to have essentially a template pre-created within your environment.
The other one will essentially just take a bare OS and it will install everything on top.
We are using the one that requires a template.
Yeah, so we made a diagram of how our cluster will look like in terms of cluster API.
So everything you see there are Kubernetes resources.
And all these resources, they represent the cluster, the Proxmox cluster we are going to use.
So we'll have the main entities, of course our cluster.
It will reference the infrastructure and also reference the control planes
and the way how they should look like on the Proxmox environment.
And then another resource is machine deployment, which is used for worker machines
and it also should reference a template of how it's going to look like on Proxmox
and also some configuration for bootstrapping Kubernetes over there.
Cool, so over to the demo.
So we were going to do the demo live, but actually the network is not being nice to us.
So luckily we did record it.
So let me just set this up.
That's what I'm going to do.
Can I do full screen?
Is that a visit?
Yeah, that's what I tried.
Obviously didn't try hard enough.
So hopefully you can see this all right.
So this is just shown initially the repo that that link showed before.
In that repo there are two branches, one for the Kubevert side and one for the Proxmox side.
So we're just going to use obviously the Proxmox branch here.
And in that you can see all of the artifacts that we would have used in a live demo
and if you're going to use this yourself.
So moving on then to the pre-rex.
So as I mentioned, you are required if you're going to do this yourself
to have a template in your Proxmox environment.
So the way that you do this, if you want to do it in an automated way,
so you can use the Kubernetes image builder project
and that has specific make targets that will provision and build that base image for you.
And actually what we should see in a minute is I should change to that window
and you can see it here.
So the Virtual Machine 100 has been built using the Kubernetes image builder project.
So that's got everything on there required to bootstrap Kubernetes.
So it's got versions of KubeADM, et cetera, already baked into that VM
and it's been marked as a template within Proxmox.
Yeah.
Cool. So the basic flow is we're going to create the management cluster.
Sorry, we're going to create the management cluster.
We're going to then install GitOps agent on there
and then we're going to create a cluster.
So I'm just going to fast forward here because this is great.
We're using kind for our management cluster.
So if I just fast forward, just preloading a bunch of images onto there.
The idea being it would have made the demo a lot quicker.
So I'm going to start canines in another environment, another window
so you can see actually what is getting installed.
So this is a plain vanilla Kubernetes cluster at this moment in time.
One thing to note, if you're going through the instructions at a later time,
we've made a slight config change to the cluster cut all utility configuration
so that we can install an IPAN provider.
So probably in the last session you went through the different provider types.
The main ones are the control plane provider, infrastructure provider
and bootstrap provider, but the newer provider types are the IPAN provider
which is especially useful for virtualized and bare metal type scenarios
and also the add-on provider type.
So the way that you create a management cluster is with cluster cut all.
One thing to note here is we're specifying version numbers.
That was purely just to pin the versions so that we could load the images
but you don't have to do that in your environment.
And this will go away and install all of the providers and core Cappy into this,
turning it into a Cappy management cluster.
So if we fast forward a bit, you can see them installed in now
and you can see the IPAN provider at the top there and the Proxmox provider.
So the next step, so we've got a management cluster.
So we want to use GitOps in this scenario.
So you can use whatever GitOps agent you want.
So we're going to be using fleet, but you could equally apply these steps
with slack modifications if you wanted to use flux, Argo CD, whatever your choice is.
But we're using fleet so we just need to quickly install fleet, a couple of Helm commands
and we'll have that there.
So we can fast forward a bit.
So now we have the GitOps agent in our cluster.
We can start using GitOps to provision clusters.
And this is where I guess the mixture of cluster API and GitOps comes really interesting
because you then can create clusters via a pull request,
which opens up to all sorts of different types of scenarios.
And it also means you can perform all of the operations against that cluster via pull requests.
So you have the full history of the changes.
You can roll back and all of those types of things.
If you're using GitOps, you're used to with your applications,
but you can now apply it to your actual clusters in the cluster lifecycle.
So in the repo, you'll see two folders.
Funny enough, the one with the cluster definitions in is in the clusters folder.
It's just got the one cluster definition in there.
So we're going to bring it up now to have a look at what it is.
So it's just pure YAML.
It describes the shape of your cluster.
There's different resources to represent different parts of the cluster,
whether that's the control plane or the worker machines.
And it matches the diagram that we showed before in the presentation.
Basically, this YAML is what you saw in the diagram, but not visualized.
So two things to note here.
Just highlighting the fact that we are using the Proxmox,
so you will have resource kinds dependent on your infrastructure provider here
and likewise for the other type of providers.
So there's a couple of things we want to note there.
So just highlighting some labels here.
If you just remember these labels say CNI Calico,
we'll come back to that in a bit.
And then we just see some various other aspects.
One thing to note, we're also using CubeVip here.
So in this type of environment, you need some sort of load balancer
so that you can get to the API server.
So we're just using CubeVip as an easy way to do that,
and it uses gratuitous ARP.
So if the control plane machine that is currently hosted on crashes,
it will move across and it will start advertising the address
from another control plane machine.
So it's quite a nice setup.
So we can fast forward there.
So here you can just see the shape of the VMs that we want,
the specifications, so this could be whatever you want.
One thing to note is you'll see the template idea at the bottom,
which says 100, so that will have to match the template
that was created via the ImageBuilder process.
If they don't, then things don't work.
So we require a small amount of configuration for Fleet,
and this will be the same for other GitOps agents.
So in this file, we call it the Git repo,
and this just tells Fleet about, hey, go to this source repo,
download everything in there and apply it to the cluster.
So you'll just see that the repo URL, the branch that we require,
so we're on the Proximax branch,
and then potentially any paths or secrets that are required
to access that cluster.
Cool, now we've done that.
We've applied that to the cluster,
so it's going to bring all those cluster definitions
into our management cluster,
and then hopefully we start to get virtual machines being started
and that cluster will be formed.
Maybe.
Cool, so you can see now that automatically,
the cluster API has created machines here for you.
So you'll see that there's one machine for the control plane
and one machine for the machine deployment
or the worker nodes, and you can see that the one has started
to move to provisioning.
What that basically means is it's going to provision the infrastructure
and then start to bootstrap Kubernetes.
So what does this mean from a Proximax point of view?
People with really good eyesight will probably see
that there's a new VM starting up,
so you can see it in the events at the bottom there,
a VM 104, and you'll see it on the side in the viewer.
So this is being orchestrated by the cluster API provider.
So it's talking directly to the Proximax API and saying,
hey, create me a VM, I'm going to use it for this control plane machine.
Now this part does take a while,
so we're going to have to skip quite some way through.
We'll just get it to the point where you bring up the console,
so you can see it's using Ubuntu,
and if we fast forward a little bit,
eventually you'll start to see, essentially, cloud init will kick in,
and depending on how you configure the bootstrap providers,
it will use either cloud init or ignition currently.
This is using cloud init, so you'll start to see cloud init run in,
and that will essentially be running the commands
to bootstrap Kubernetes on top of this VM using QVADM in this instance.
Oh, we missed it.
You'll see it, it will come up.
So it does come up, and you can see that.
So essentially what it's doing.
So at that point, we have one control plane machine ready, essentially.
Once one control plane machine is ready,
you can then start to provision the worker machines,
and it always waits until one control plane machine is ready,
and then it will just start provisioning all of the worker machines in parallel.
So we can fast forward that, and you'll see another VMs come up,
and I think you get the point, so it just repeats the same things,
but this will be for a worker machine.
So, well, I just skipped ahead in the top part of the terminal window.
I have just got the cube config for that newly created cluster.
So the cube configs for the newly created child clusters
or the tenant clusters are available in the management cluster,
so you can get that out and then run, and obviously do what you want with it.
In this instance, I'm just showing that stuff is running in there.
So you can see that Calico is running in there,
so we didn't put Calico necessarily in the cluster definition,
but if we go back to those labels on our cluster definition that said C&I is Calico,
that is using a feature of a cluster API called cluster resource sets,
and essentially this enables you to install any type of resources
into a newly provisioned cluster automatically.
So it's really ideal for things like C&I or cloud providers
to be able to do the things that you want as soon as that cluster is being provisioned.
And again, this is all done in a declarative way,
so you don't have to do any special commands,
you just put all of your definitions into Git,
and then the cluster API will do the orchestration.
So this is what is in the second folder, in the repo, in the CRS folder.
You'll see that there's a cluster resource set,
and you'll see that there's a label selector,
so if your cluster matches that label selector,
it will then apply all the resources listed below,
and those resources are essentially just config maps or secrets,
and they contain embedded Kubernetes YAML,
so it will just squirt those into your cluster.
So where are we now?
So we've got one control plane and one working machine,
so I said that you could go into Git,
and you could scale the cluster and do all your operations,
so what we're going to show here is actually,
if you go to the cluster definition in Git,
we're going to scroll down until we get to the machine deployments,
where it will say replica one,
and we're going to, hopefully,
you see the machine deployment there,
change app to two,
commit those changes to the Git repo,
and you can probably guess what will happen now.
Any VM is spun up, Kubernetes is bootstrapped on there,
and that node joins the existing cluster,
and you'll see eventually that it does come up.
So that is the props box demo.
So now we're going to show the same process with Kubert,
and the idea is that you can use cluster API to provision your clusters
and multiple providers in the same operational way,
so the process for different infrastructure providers
is relatively very similar,
with the difference in operating your infrastructure,
I mean, defining how your machine looks like,
but the whole idea is the same,
no matter where you're on your clusters.
So the one major difference with the Kubert provider
is it requires Kubert to be installed in your cluster already.
So before you install the provider for Kubert,
the Kappi provider, you must have Kubert already installed.
So what you're seeing here is we're installing Metal LB
to take the place of providing the load balancers
within this environment.
Then we install Kubert,
and so Kubert works on the basis of you describe your virtual machine
as a custom resource,
and then it will make that happen behind the scenes via QMU.
So this is what we're doing first,
and this is before you get to any of the Kappi stuff.
This is just setting up the Kubernetes cluster.
So you can see the Kubert is starting up.
So we now done the quaternate installing the provider.
We're going to install the GitOps agent here now.
So it's basically the same process,
just slightly modified with different providers
and just with different prerequisites,
the prerequisite being Kubert.
So forward again.
So in this second branch,
you'll see a different cluster definition that uses Kubert,
but essentially the way it's applied to the cluster
is exactly the same via GitOps.
So what you take away from this is the same operational procedure
irrespective of your target infrastructure
or the flavor of Kubernetes that you want.
You just create some YAML that describes the cluster that you want.
So you'll see this is the interesting part.
It will spin up a pod per VM,
and that pod will then do the interaction with
to actually provision the virtual machine on the host.
So you'll see one of these spin up
for each of the virtual machines
that are required for the cluster.
You can look at the boot logs via VNC as well.
So if you use Vert Cuttle in this scenario,
just get the name of the node and use Vert Cuttle,
and you'll see it's using exactly the same
QBADM commands that we saw previously with the Proxmox.
And then we do the same operation.
We scale it, and you get the third machine.
So again, probably the key takeaway from this is
that you can use the cloud environment
to actually use it.
So key takeaways are,
CAPI can be used in many, many different infrastructure environments,
not just like the cloud environments
where a lot of people would naturally think of it.
So virtualized environments, bare metal type environments,
and some really interesting type environments
where you want a control plane as a pod type scenario.
It supports different Kubernetes flavors,
so you might want just pure upstream with QBADM.
You might want something a bit more lightweight,
so you can use K3S.
So it allows you to mix and match all of these things.
And lastly, this is fully declarative,
fully GitOps friendly.
Perform all of your cluster operations via Git.
So yeah, thank you for coming.
Thank you for your question.
Thanks.
Thank you.
Yeah, so the question was,
can you realistically provision a cluster
associated infrastructure like low balances,
et cetera, with the cluster API currently in a hyperscaler
to like AWS as an example?
The answer is yes, definitely for AWS.
I'd say the caveat is it will provision the infrastructure
in an opinionated way.
So it will only provision the infrastructure that's required
for the cluster and nothing more.
And it will provision it in a way that it thinks best.
So you can slightly tweak it if you want.
If you don't like, say you want to use A or Bs instead of
something else, or you want to add security groups,
it does allow you to do that as well.
But there are, I guess, boundaries.
So if you want full flexibility,
then it might need to do something else.
But you can also use things like Terraform
and cluster API together.
It doesn't have to be an either or.
So you might provision the VPC and the network with Terraform
and then get cluster API to do the Kubernetes
and like the day two operations type of stuff on Kubernetes.