We still have one minute before the game starts, ready to go?
Thank you.
Our next talk is by Sachin, and he's going to talk about a thing that I use every day
in Go, but it's kind of weird because it's only existing in this language, as far as
I know.
But it's how Kubernetes is built, which is the reconciliation pattern.
Go ahead.
Thank you.
Thank you all.
Thanks for coming.
Welcome to Forstium.
Today, I'm going to talk about control theories, reconciliation pattern, and how do we use that
in Cluster API?
So a little bit about me.
I work at Canonical, particularly the MicroKStream.
Previously, I used to work at VMware.
Then I got to know about Cluster API, BIOH, and I try to contribute to Cluster API upstream
too.
And I'm very much interested in distributed system and cloud-native technologies, so ping
me with your favorite tech.
So the agenda is like this.
So we start with the first basic principles, like what is control theory and PID control
system.
Then we go up this tech.
So L0, L2, just simulate tech, we're going up this tech, one more layer of abstraction.
Then we'll see about reconciliation pattern and how they are using Kubernetes.
We then see how we extend those reconciliation patterns, and finally, we'll take a look into
those patterns in Cluster API and a short demo to come to play with it.
So a quick one-on-one of control theory.
I'm talking to you.
You folks are taking a feedback, and that's like 90% of control theory right there.
So control theory is like a branch of mathematics, engineering.
So there's a lot of folks who are in trying to find a common theme for dynamic systems,
and they were all like, wait, we are all talking about the same things, let's just unify it.
So that's how control theory was.
It's just a study of how dynamic systems work, particularly the main fundamental crux of
it to bring a desired state, a final state into a desired state.
So this is kind of what control theories are about.
Let's take a very simple example to know more about it.
And open-loop controllers, what is it?
A simple example will be you have some wet clothes you want to dry them.
You put them in a dryer, you set the timer on.
Now the clothes are in no way dependent on if they will be dried or not.
The only function that is variable is the timer.
It times the duration that it needs to shut down the dryer to.
It doesn't matter if the clothes are dry or wet.
So it's not a good approach to take this.
Before I introduce closed-loop controllers, there are a few terms that we need to see.
A system is the entity that we want to control.
Set point is our desired state, process variable will be observed state.
Error is the difference of how overshot or undershot we are from the set point and the
process variable.
Controller is a simple finite state machine which drives essentially your process variable
to the set point.
A very favorite example of mine is thermostat.
So we are in the room, we have an air conditioner and we have set the thermostat to maintain
the temperature at T1, let's say, and currently the temperature is T0.
So the thermostat says, no, no, no, the temperature I want is T1.
So it produces some processes to the machine, to the AC and it does like an adiabatic process
or something to achieve that state.
So in that case, our thermostat will be the controller, T0 will be our process variable,
T1 is the set point and the error is the difference between the temperature that we want and the
room is our system in that case.
But it's not always this ideal, this change takes time.
It's not like instantly you do, instantly the thermostat says, okay, make the temperature
T1 and the AC does that.
It takes a gradual amount of time to do that.
And so we need a non-ideal situation.
What would be an ideal controller look like?
So it needs to do these three things essentially.
It needs to see, okay, how far am I undershooting or undershooting from the set variable.
It needs to do the compensation for large changes and try to adjust based on it.
And also it needs to make prediction of how to minimize this error based on previous experiences
it has.
A very good example of this will be cruise control in your car system.
When you're going you turn on the cruise control and it identifies, okay, now I'm going straight
but I need to, and there's a turn coming up, I need to apply this amount of turn essentially
to make that, to avoid an accident or something.
So PID controller is essentially what these three accumulate to, the P is the positional.
It's essentially the amount of, for example, in the case of cruise control, it's essentially
the amount of turn that the car needs to take to make that curve.
It is the linear component, the P is the proportional or the linear component.
In the graph we see that it is defined by, if the set point is like a straight line and
PV just fluctuates all around, it's the magnitude of the point from the set point to the process
variable.
The I is the integral component, it is the compensator.
So it adjusts based on what the current state is and how I need to set to the desired state
but also it needs to compensate fastly.
So you're going on a straight road, you need to quickly make the curve.
So you cannot, the car cannot go like, okay, I'll make the turn right away when the turn
comes up, it needs to gradually make that change.
And so for that it uses, the integral component just signifies that gradual curve that it
needs to take.
And it is defined by the area under the curve in the magnitude versus time graph.
D is actually really interesting.
It's the predictor, it's how previous experiences that it has, it applies the previous experience
that it has and tries to control the state it is trying to achieve.
In our cruise control example, it will be as simple as, it sees the curve, it slowly
gradually starts to make that adjustment based on like previous experiences that it has,
that I should not just overshoot when the curve comes but start gradually differentiating
that.
The other controllers that we have fall under PID, the D is not much used but it's a really
interesting one if you look at it.
This funny looking diagram is just a block diagram of how the PID controller tries to
manage the process and like it has a sensor in it, which just takes the state of it.
This example R is the set point, the signal that we are sending into the controller.
The Y becomes the Y function, that becomes the process variable, E is obviously the error
and U becomes the signal that is sent to the process here.
This fancy looking thing is just a state of the process that we are in.
So U takes the signal that we are sending into it, which was as in our previous slide,
the set point, sorry, U was the, yeah, the controller, the signal that was sent to the
process.
YT is the measured output, as you can see from there.
The error is the difference between RT and YT.
So RT was our set variable from this previous example.
And so this, this simple differential equation is just tries to find the particular state
of the controller that is written and how is it trying to achieve that state.
The coefficients K0, K1, and K2 totally depend on the system that we are in.
So reconciliation patterns in Kubernetes.
How do Kubernetes incorporate these patterns that we see and use it to make controllers
and you can silence it?
So on a very high level, this is what a simple reconciliation look like.
It's a forever loop, which has a desired and a current state, which are set points and
process variables, and actuator that makes this change.
Let's try to take the current state into a desired state.
And this is like available on, this is like from the controller, and you can check it out
it has a very good specification of how to make a controller.
Let's take a very simple example to see how it actually works in a one node cluster.
So we have a one node cluster, we have deployment that is deployed, which has a replica set
which provisions two pods on a single node cluster.
The node talks to the API server, the API server talks to HCD, and it has a bunch of
controllers that it needs to run that state.
So everything is fun.
Now, pod decides to bail out, it's gone, just like that.
And so there is now, the state is not maintained, the desired state is lost.
So what the Kubelet does, it talks to, it mostly talks to the API server, API server
that says, talks to the HCD, it says, okay, I need two pods, but there is no pod here.
So there is the, API server talks to the controllers, it's the scheduler, the deployments and the
scheduler and replica set controllers, she gives a new pod to that node, it is mentioned
in the HCD server, and finally a pod to its provision on node zero.
So this is a very simple example of how controllers works in Kubernetes.
Now how do we extend the reconciliation pattern?
How do we use it to make CRDs and stuff?
So first of all, how many of you folks have used Kubernetes cluster API, CRDs, all these
fancy words?
Quite a lot.
So most of these frameworks, CubeBuilder, Operator SDK, these have this basic structure
to make a controller.
You create a spec which is set point in this case, we have a status which will, which will
the process variable in this case, which is the desired state that we, which is the observed
state that we want at any point of time, and it will, and we have a schema that is just
defines this object foo in this case, and it has all these spec and status, this I mean,
the meta objects, like the name, type, and all that stuff, information in that side that.
We create, and we need to fulfill the reconciled interface, so we create a foo reconciler object,
and we, we essentially provide it with, with all these business logic that we need to reach
that desired state from the current state at any given point of time.
The way we do that is we define a CRD, our CRD has a spec which is the desired state,
and the controller continuously looks at the CRD to check, okay, this is a desired spec,
but we don't have a desired spec right now, so it needs to change, and it's called the,
it calls the reconciler, and it does, it executes the business logic that we want it to do.
And so that is how we use the, the reconcilation pattern that we've seen earlier in, and extend
this for other custom-made objects that we have.
Now how do we use these patterns that we saw, and incorporate them in Cluster API?
So first of all, Cluster API is a Kubernetes project which tries to declaratively use Cluster
APIs to create and figure, manage the life cycle of other clusters that you have.
So in a very crude example, the user applies a spec to the cluster, there's a management
cluster which is kind of a cluster of clusters, it manages all these other clusters that we
have.
So a spec defines all those, what those other clusters need to be do, and the management
cluster basically has these four kind of things, it has Cluster API CRDs, infrastructure
provider CRDs, control plane, and bootstrap provider CRDs.
So all these need to be present in the management cluster, and based on these, these specs that
it has in CRDs, it will try to maintain the state of all these other, all these other
clusters that we have, sorry.
So what do these different CRDs do, these different objects, what is the purpose?
The Cluster API is basically all these copy objects, like machine set clusters, all this
stuff that we, the upstream Cluster API provides us.
The bootstrap provider does the job of turning your VM or any default server into a Kubernetes
node.
You can utilize logic to that, and convert it to the particular Kubernetes node that
we want, for EC2, for OpenStack, whatever your cloud provider is.
The control plane provider, it provides you with the objects that the control plane of
the, like the simple control plane in Cluster API, in Kubernetes, it provides you with all
those reconciliation loops and controllers that the control plane needs to mark those
states.
And the infrastructure provider is basically how particular infrastructure, like EC2, OpenStack,
whatever infrastructure you have, and how they will be incorporated into bootstrap or
control plane providers.
So this is kind of like how these different CRDs go into, CRDs interact with each other,
so Cluster, Cluster is from Cluster API, but we need to provide an infrastructure cluster
which comes from infrastructure provider to that, and then it will manage.
So all of these are very much dependent on which cloud you're using.
We'll see an example of this in a few minutes.
So a control plane directly comes from control plane provider, machine deployment, machine
set, it's all Cluster API stuff, but we need to provide it bootstrap and infrastructure,
and similarly bootstrap config and infrastructure machine for it to work, machine health check
comes directly from Cluster API, its job essentially is to keep checking the state of the machines
and if it's working fine or not.
A bit about microcades, because we're going to use microcades, control plane and bootstrap
provider.
So what happens, so microcades is lightweight communities we have, we have been working on,
it is one touch communities highly available, it has all the same configs, you don't need
to do much, and it has a very good add-on ecosystem that you can call your own tools,
you don't need to rely on us to do all this stuff, you can bring your own custom tools
that you need for your clusters.
So for the demo, it's a small demo, we need three essential things, so the Cluster API
comes from the upstream step, but we need to provide these other three things, and then
for this, for bootstrap provider, we'll use our microcades bootstrap provider for control
plane, same thing, and from infrastructure we will use open stack providers that we have.
So for the demo, let's go, let's see if it works, so like I said, these clusters, these
are from upstream cluster API, we just take these CRDs, but then we need to apply what
control plane reference will be using, what infrastructure will be using, and it's all
like custom based on what you want to do.
Similarly to that, we have open stack cluster, open stack cluster that is specific for open
stack cluster, we have different projects for that, AWS, Azure, EC2.
Then we see microcades control plane, it's specific to microcades, it defines all these
specs that a particular instance of microcades will have, and this is a thing to see a bit.
So we define a particular version that this particular control plane will have, open stack
machine template that we saw before, that is needed for that, and machine deployments,
and machine deployments will also have a version that is essential for our demo.
So and then there are all these stuff that comes from template, whatever template you
apply, it comes from that, so it's quite default, so without trying to actually go into entirety,
I have screenshots of it because the entire demo took like an R2 issue.
So if I apply this cluster, I'll get this too, so I don't know if you can see, but
I'll have six machines in an open stack cluster, which will have a version of 1.24 each.
As the time progresses, it provides a provider ID, and at a certain point in time, they're
all in ready state and good to go with all of them with 124 communities version.
I think to note that is both of them are controlled by different providers, so the machine deployments
are controlled by the bootstrap provider, and the control plane takes care of all these
control plane nodes.
So we'll see how, what happens when we try to update this cluster, what reconciliation
is happening when we try to do that.
So if I go there, I'll change it to six, and then again to six, as soon as I apply this
manifest back, I have changed the desired state for me to have version 126 on both of
the control plane and the machine nodes.
So as and when I apply that, both the controllers, the bootstrap and the control plane controllers,
we'll see 124 is now not what we want, we want 126, so it will start provisioning these
machines at 126 version.
It goes through the entire place of, so these are the rollout updates, so what happens is
a new node is provisioned, a old node is depleted, and this happens until all the nodes are in
the desired state.
So it's also in place updates, which is a very cool idea, so rather than depleting
the nodes, it just does the upgrade in place without having to drain nodes each time it
comes and go, and it is a very good use case for when you have a stateful application like
a database or something.
So it does that, it does the deletion, it does all that stuff, until the entire cluster
will be 126, which was the desired state.
So all of this we see, we go from basic first principles is like what was control theory,
how it gives us controller, then we apply, then we see how we applied it to our communities
ecosystem, and then how we extended that, extended those patterns for our cluster API,
and finally how can we, how we can have a feature from that first principles.
These are some of the talks that I took inspiration from, I definitely recommend control theory
in Fitment Rewind by Valerie, it has lots and lots to say about this.
Control theory and all these stuff, control theory is dope, it's a very good article that
you should definitely check it out.
It also talks about reactive patterns, which is cool stuff, lots more use in AI and stuff,
so it is cool, and these are all references that I use from other sources as well.
So yeah, thank you, thank you for coming, I hope you didn't come in for me.
Thank you.
I'll take questions if you have, yeah.
Are there any questions about Kubernetes, I'm just going to try to get the microphone
to you, not any questions about Kubernetes, about the talk, thank you.
Can you pass the microphone along, thank you.
Hey Guruji, thank you for your talk.
In the theory you have the state, the desired state and the current state of the system,
and then when you're talking about the thermostat, this is the desired temperature and this is
the current temperature, how do you accommodate for when, can the system predict when this
is not going to happen, oh I've been pumping the heater for 48 hours and I see that the
temperature is not raising, not a single degree, like how do you cater for that?
So first of all it means that the system has a fault if it does not reach the desired state,
but it will take it as an experience, so if I go to here, the predictor component is what
predicts it, it will see okay, the derivative is the predictor component, it will see okay,
at some point of time previously this did not work, this change was not working, so
it will take that into account and the next time it does that it will take it as an experience,
so if it was not working and how did we try to make it work, it will try to take that
experience and incorporate it into the next time it tries to do that.
Thank you, any other questions?
I'll take it as a note, thank you very much again, we have a small 5 minute break so you
can stand up, stretch a bit.