Hi, everyone, and welcome to our talk about operator monitoring and how to do it correctly.
My name is Shirley.
I work at Red Hat.
I'm Jean Villassa.
I also work at Red Hat for about one year and a half.
So today we're going to talk about operators' observability, Kubernetes operators, and we're
going to talk about when to start, the maturity levels of metrics, why we want to monitor,
what we want to monitor, and the best practices and code examples that we created for it.
So when we want to talk about, when should we start to think about the observability
for operators?
You can see here in the chart the life cycle of creating an operator, which is starting
in basic installation, and the most mature step is autopilot.
So when do you think we should start thinking about observability for a new operator?
Anyone?
When?
From the start.
That's correct.
Really deep insights, talks about metrics, alerts, which is being able to monitor your
operator fully.
And people think maybe we should start thinking about it in full life cycle.
Maybe that's the case.
But you should pretty much start at the beginning, because the metrics that you are adding first
are usually not the metrics that are for your users.
They are internal.
There are a few steps for the maturity of metrics.
The first step is initial.
You start with your operator, you want to understand how it works, if it works correctly.
So the developers start to add hot metrics.
I've been working for a few years on an operator in Red Hat called Qvert.
And when I joined the project, it was already in the life cycle phase, full life cycle.
And when I joined, already a lot of metrics were implemented in this operator.
The problem was that there was no, the developers that added the metrics didn't fall best practices.
And a lot of the metrics, it was hard to understand which metrics were ours.
It's important to understand that your operator is not the only one inside of the Kubernetes
system.
So when someone, when a user or even other developers want to understand which metrics
your operator is exposing, it should be easy for them to identify your metrics.
So the first step, as I said, is initial.
The second step is basic monitoring.
You start adding your monitoring, and you're starting to think about your users, what they
want to understand about your operator.
And the third step is you have a process for implementing metrics and new metrics, and you
are focused about health and performance for your operator.
And the last step is actually autopilot.
Taking those metrics and doing smart actions with them in order to do stuff like auto healing
and auto scaling for your operator.
And this is the part that we are actually on in our operator.
So as Shirley said, when we first start, we look very much at internal metrics for the
operators themselves.
So at this point, we might start, for example, looking at the health of the operator.
For example, can it connect to the Kubernetes API, or if it's using external resources,
can it connect to those providers' API?
Is it experiencing any errors?
So we can also start by looking at, for example, its behavior.
How often is the operator reconciling?
What actions is the operator performing?
So this is the kind of stuff that, as we are developing, we are very interested in.
But we should start, as Shirley said, thinking more in the future about having these good
standards, because later we will not be only tracking these, and could also be, like, resource
metrics.
And then why should then, why operator observability, and what are the steps that we'll be taking?
So starting from the performance and health, here we want to detect the issues that come
up early.
We try to, obviously, reduce both operator and application downtime, and try to detect
some regressions that might happen.
Also we can start looking at, for example, planning and billing to improve planification,
to also improve profitability, or then build users.
At this point, we start looking more at infrastructure metrics also.
For example, we want to track resource utilization.
This might be, like, CPU, memory, this, and we can also start looking at the health of
the infrastructure itself, maybe hardware failures, or trying to detect some network
issues.
Then we also start looking at, use these metrics to create alerts, to send notifications
about the problems that come up as early as possible.
So we obviously want to take appropriate actions to not let them go around.
And after this, at this point, we go into more detail about metrics.
Maybe we start looking at application metrics.
So what's the availability of our application?
What's the time?
What's the error rates?
And also its behavior.
What type of request is the application receiving?
What types of responses is sending?
And it's important to monitor all of these things.
And when we start building up all this information, then at a certain point in time, as Shirley
said, we'll be able to give, like, this new life to the operator by having the autopilot
capabilities, such as auto scaling, auto wheeling capabilities.
Because at this point, if we did everything correctly, you'll be able to know, like, almost
all the states that we are in.
And we also start looking at metrics functionality metrics.
We can provide the expected, are we providing the expected functionality to users?
For example, checking that application features are working correctly.
We want to see if there are any performance or reliability issues by checking service
levels, and that everything is, it's working in the expected way by checking response to
the airhorse and the data that it's responding to.
Okay.
So I hope you are convinced that the observability is important.
If you are in this room, I guess you are.
And for the past three years, we've been working on observability on our operator.
What's important to understand is that our operator is considerably complex.
It has a few sub-operators that it's managing.
And each sub-operator has its own team, dedicated team, that is maintaining it.
And having the insight of looking at those teams working on implementing observability,
each team separately gave us a higher level of the possibility of understanding the pitfalls
that they all share when implementing monitoring.
So we decided to contribute from our knowledge of how to do this correctly in order for others
not to do the same, to fall to the same pitfalls as us.
So we decided to create best practices and to share with the community our findings.
We hope to shorten the onboarding time for others and to create better documentation
and to create reusable code for others to be able to use and save time and money, of
course.
So we reached out to the operator framework SDK team to collaborate with them and to publish
there our best practices.
As you can see here, this is the operator observability best practices.
The operator SDK itself is the first step when someone wants to create a new operator.
It gives them tools, how to create it easily, how to build, test the packages, and provides
best practices for all steps of the operator life cycle.
So we found that this was the best place for others to also go for monitoring.
And in these best practices, I will now share with you a few examples.
It may sound simple, but simple things have a big impact, both on the users that are using
the system and both on developers that are trying to work with the metrics.
So for example, a naming convention for metrics.
One of the things that is mentioned in the document is having a name prefix for your
metrics.
This is very simple action that will help you identify, that will help the developers,
the users to identify that the metrics are coming from the specific operator or a company.
In this case, you can see that all of the metrics here have a cube width prefix, a cube
width, as I said, has sub-operators.
So under this prefix, we also have a sub-prefix for each individual operator, a CDI network
and so on.
And this is another example, which does not have this prefix.
We can see here a container CPU, for example, prefix, but we can't understand where it's
coming from.
In this case, it's the advisor.
But if you're a user and you're trying to understand where this metric came from, it's
very hard, and also you cannot search in Grafana, for example, for all of the C-advisor metrics
together.
So that's a problem.
Another thing that is mentioned in the best practices is about help text.
Each metric has a dedicated place to add the help for this metric.
And as you can see in Grafana and other visualization tools, the user will be able to see when hovering
on the metrics, the description of it.
It's very important because if not, you need to go somewhere else to search for it.
Also this gives you the ability to create auto-generated documentation for all of your
metrics in your site.
Another example is the base units.
So Prometheus recommends using base units for metrics.
For example, you can see here for time to use seconds, not milliseconds, temperature,
Celsius, not Fahrenheit, this gives the users a fluent experience when they are using the
metrics, they don't need to do conversions, deviations of the data, and they are saying
if you want to use milliseconds, use a floating point number.
This removes the concern of magnitude of the number, and Grafana can handle it, and it
will still show you the same precision, but the consistency in the UI and how to use the
metrics will stay the same.
Here you can see an example for metrics that are using seconds.
And here we see that each CD are not using it.
So this is not as recommended, and we would actually recommend to switch it, but they
started with milliseconds.
And now doing the change will cause issues with the UI that is based on it and everything.
So it's a problem to change the names of the metrics once they are created.
So when I joined the operator, we didn't have name prefixes.
I tried to understand which metrics are ours and which are not, it was very hard.
So we needed to go and do breaking changes for the metrics and add those prefixes, change
the units, and this is what we want others to be able to avoid, this duplicate of work.
Additional information in the best practices is about alerts.
This is an example of an alert.
You can see here that we have the alert name.
We have an expression which is based on a metric, and once the expression is met, the
alert either starts firing or is in pending state until the evaluation time.
There is a description.
There is also a possibility to add a summary.
This is the evaluation time.
It has a severity.
And a link to a runbook URL.
There could be other information that you can add to it, but this is the basic.
And what we're saying in the best practice is that there's supposed to be, for example,
for the labels of severity, there should only be three valid options, critical, warning,
and info alerts.
If you're using something else, it would be problematic.
You can see here in this example, I don't know if you're seeing it, but we see that
this is our example in the cluster.
We have info, warning, and critical, and we have one non-severity, which is the watchdog.
It's part of Prometheus alerts.
It's just making sure that the alerts are working as expected.
It should always stay one.
There should never be alerts that don't have severity.
And this is a bad example of using a severity label.
In this case, they are using major instead of critical.
The impact of that is that if someone is setting up alert manager to notify the support team
that something critical happened to the system, and they were to get notified by Slack or by
a pager, they will miss out on this alert because it doesn't meet with the convention
of severities, values for severities.
So what we have at the moment for best practices, we have for a metrics naming convention.
We have how to create documentation for metrics, alerts, information about alert labels, run
books.
By the way, run books are a way to provide more information about the alert.
You have a link in the alert where you can send the user to go and find more details.
What's it about?
What's the impact?
How to diagnose it?
And how to mitigate the issue.
And then additional information about how to test metrics and how to test alerts.
We plan to enrich this information, add information about dashboards, logging events, tracing in
the future.
So Shirley gave an overview about an eye-level situation about metrics and alerts.
But how do we translate some of these best practices into code?
So one of the problems that we faced was that logic code and monitoring code were becoming
very intertwined.
Code like this becomes harder to maintain.
Obviously it becomes more difficult in understanding what the code does and to modify it.
This leads obviously to longer development times, potential bugs, and it's also more
challenging to onboard new team members or to contribute to one of these projects.
In this specific snippet, there was like 16.4% of monitoring code intertwined with migration
logic code.
So what we did was try to refactor this code to try to separate these concerns, one from
the other.
In this specific case, we used a Prometheus collector that's just iterating the existing
virtual machines migrations, and then it's just pushing the metrics according to the
status of the virtual machines, whether they are successful or not, or the accounts of
the pending schedule and running migrations.
And obviously this snippet is much easier to understand how the monitoring is being
done, and we take all of this out of the migration logic code.
And to help other developers that are starting to avoid the same mistakes as we had to solve,
we are creating a monitoring example in the memcached operator.
We already have an initial example that is already thinking about all these concerns
in separation between logic code and monitoring code.
Our idea with this example is to make it as clear as possible, especially this is especially
important when we are working with large and complex code bases, also make it more modular.
It's easier to understand both the logic code and the monitoring code without affecting
each other's functionality in the application in general, also make it more reusable.
Since like, for example, the way we are doing monitoring in different operators will always
be more or less the same.
So if we find a more or less common way to do this, it will make it easier to reuse this
code in other applications and projects, which will save them time and effort.
And also, it will become more performant.
If we mix all the monitoring concerns with the migration code, it's trivial that the
time it will take to make a migration will take longer because we are calculating metric
values and doing some Prometheus operations while we are trying to calculate the state
of a migration.
So having this separation will also help these questions.
Our idea for the structure of the code will be by creating a package.
And for example, here we can see a migration example, a central place where we will be
registering all migrations and all migrations, sorry, no, all metrics, obviously, and then
we will have files that will separate these metrics by their types.
For example, in this example, you can see one operator metrics file, which will have
all the operator-related metrics, as we talked in the beginning, and then we could have one
specific file only for the migration metrics and then register them in one place.
And why do we think about this structure and what benefits could this bring us?
The first one is to automate the metric and the alert code generation.
As we saw, much of the work that a developer needs to do that, it's like creating a file
with a specific name, then go to the metrics.go file and register that file there.
So this is very structured and always the same.
It will be easier to automate and then allow developers to have a command line tool to generate
new metrics and generate new alerts easier.
We are also looking forward to create a linter for the metrics name.
As Shirley said, a lot of the concerns that happen when operators are becoming more advanced
is looking back at the metrics and see everything we did wrong with their naming.
And even, as she said, it's a simple change, but can have a lot of impact.
So a linter that follows all these conventions will also be important.
Also automated metrics documentations, we are already doing this.
And one thing that we faced was that a lot of metrics were very scattered in the code.
So it was easy to automate and find all of them.
And with a structure like the previous one, it will be even more easier to create a full
list of metrics and that description that will help both developers, newcomers, and users.
And lastly, have an easier structure for both unit and end-to-end testing, because if we
have, like, this clear structure for where the metrics are, we can test there and test
exactly those functions and not code intertwined in logic code.
And just to conclude, if you are starting to create an operator or if you already have
an operator, we invite you to go and to look at the operator SDK, to look at the best practices,
to try to avoid the pitfalls that we had.
And I really hope it will help you.
And you should really just consider that when you're creating a new operator, it starts
small, but it can become really robust.
And you cannot tell that in the beginning.
So think ahead and try to build it correctly from the beginning.
I hope it will be helpful for you.
And thank you.
Thank you.
Do you have any recommendations on how you would log out the decision points within your
operator?
So if you wanted to retrospectively see why it's done certain things, like the decision
points, how it's decided which Kubernetes API calls to make, if your operator did something
crazy and you wanted to look back and see why it did that, is there anything you would
do in advance to the logging?
I think this is the summary of what we've learned is in these documents.
Because as I said, for example, the developers that started this project, they didn't have
where to go and the best practices of how to name a metric.
So they just named it how they thought.
And they did follow the Prometheus recommendations, but having a prefix of the operator has a big
impact for the users.
And not even the users.
When we are trying to understand how to use internal metrics for our uses, we also are
struggling to understand where a metric came from, where is the code for it.
So all of the summary of what we've learned is in those documents, and we plan to enrich
it even further.
Thank you for your talk.
It was very interesting.
You mentioned code generation for the metrics package.
My question is, do you plan on adding that to QBuilder and the operator SDK?
Yeah, basically we are working on the operator SDK right now, because we want to build all
these tools, and we are thinking about them, but obviously this needs a lot of help of
the community.
And I am saying this because I'll enter like a personal note here and an idea, right?
Because the way I see it is like on QBuilder and on operator SDK, being able to, you just
go there and you say that you want to generate a project with monitoring, and it creates
the monitoring package.
Or if the operator already exists, you have a command to generate the monitoring package,
and then on QBuilder, like you use it to create an API or a controller, you'll have
a similar command, but to create a new metric.
And you pass the type of the metric, the help, and the same for alerts.
At least that's the way I see it.
And for me, so it makes sense.
I agree.
Thank you.
Hey, thank you for the talk.
How much of a conventions that you talked about, aligned with open telemetry, is my
opinion?
How much?
What?
Aligned with open telemetry.
Most of them are aligned with open telemetry, actually.
But these are specific for operators.
That's the idea.
The idea is that you have a central place where you can get the information.
And by the way, if someone is creating a new operator and has an insight, we encourage
others to contribute to the documentation and teach others and share the information.
So yeah.
Basically, I think we align with open telemetry conventions, but we add more ideas to it to
operate.
I think that's it.
Thank you.