Hello, everyone. Today we are going to talk about how you can achieve sustainability in
computing, how you can do energy efficient placement of Kubernetes workload. My name
is Parul Singh and I work as a senior software engineer at Red Hat. With me, we have Guy
Liu. He is a software engineer intern and today we are presenting this presentation
together. So we are part of CNCF and we are taking community-based initiatives on
environment sustainability. If you want to check our proposal, you can follow the
link. We also have done a few projects. Again, using community-based approach, the
first one of them is carbon aware scaling with KEDA. We did this with
Microsoft and we investigated how you can use electricity and carbon intensity
to make workload scaling decisions. Another one that we've been working with
IBM Research is Clever. That is container, label, energy efficient, VP, a
recommender for Kubernetes. And if you want to check out both of these
projects, you can just follow the QR. So the agenda is very simple. We'll give a
brief background of the things, how they are at the moment and then we're
going to introduce a sustainability stack which consists of two projects, the
Kepler and the Model Server and then we will have a demo. So here we have an
interesting quote that sums up the motivation of our sustainability stack
and the problem it seeks to solve. So according to Gardner, in 2021 an ACM
technology brief estimated that the information and communication technology
sector contributed between 1.8% and 3.9% of global carbon emissions, which is
astonishingly more than the CO2 emission contributions of both Germany and
Italy combined. The significant carbon footprint and significant energy
consumption of the tech industry begs the following questions. How can we measure
energy consumption quickly and indirectly? How can we measure energy
consumption of workloads? And how can we then attribute power on shared
resources to processes, containers and pods? So with these issues in mind, we
introduced a cloud-native sustainability stack which seeks to address these
questions and problems. Perule will first start by discussing the Kepler
project and then I will discuss the Kepler Model Server project. Let's talk
about the energy consumption attribution methodology used by the Kepler. What
Kepler is based on the principle that power consumption is attributed to the
resource usage by process containers and pods. For example, let's say you have a
pod that consumed 10% of CPU, that means it attributed to 10% of CPU power
consumption. Similar if you have like five containers and they total
contributed to 50% of CPU usage, that means they attributed to 50% of CPU power
consumption. It is so on and so forth for other resources like memory and GPU
etc. And this we based this principle based on the studies and we have
attached the link to the paper. If you're interested you can check that out. So Kepler
is a Kubernetes-based efficient power level exporter and it uses software
counters to measure power consumption by hardware resources and exports them as
Prometheus metrics. Kepler does three things. The first is reporting. It
reports per pod level energy consumption including resources like CPU, GPU and RAM
and it supports bare metal as well as PM. So you can measure your workloads
energy consumption even on AWS or Azure etc. And it supports Prometheus for
exporting the metrics and you can see the dashboards using Grafana. It's very
important that Kepler has low energy footprint because what we're trying to
do is measure. So we don't want to have Kepler consuming a lot of power itself.
So we used EBPF to probe the counters and this considerably reduced the
computational resource used by Kepler. And at last we support ML models to
estimate energy consumption when you don't have a power meter and Kai will
talk more about it in the Kepler model server portion but we use ML models to
predict the energy consumption when inherent power meter is not available.
The second part of the sustainability stack is the Kepler model server. So by
default Kepler will use a supported power measurement tool or meter to measure
node-related energy metrics like CPU core, DRAM and then they
uses this to estimate pod level energy metrics. But what happens when Kepler
does not have access to a supported power meter? This is where the Kepler model
server steps in to provide trained models that use software counters and
performance metrics to predict relevant energy metrics. The tech stack of the
Kepler model server also includes TensorFlow Keras, Psychic, Flask and
Prometheus. So let's take a look at some of the models the Kepler model server has
implemented. For example, we have a linear regression model that predicts node
level CPU core energy consumption with the following categorical and normalized
numerical software counters and performance metrics. This model also
supports incremental learning, incremental training on new batches of data to
improve the model's performance on a cluster. The second example also provides
a linear regression model capable of online learning but it instead predicts
node level DRAM energy consumption with the following software counters and
performance metrics. So let's take a look at how the model server fits in Kepler
as a whole. So the first part is training our models on a variety of training
workloads where Kepler can export node energy metrics and performance metrics
because a power meter is present. In this case Kepler retrieves these node energy
metrics from agents which are then collected and exported as Prometheus
metrics. The model server scrapes these Prometheus metrics, sets up training,
testing and validation data sets and then trains, evaluates and saves the model
with the new data. The second part is now exporting these trained models to Kepler
for prediction whenever a power meter is not provided. The Kepler model server
can export the model itself as an archive to Kepler and this is done with
flash grouts. The model server can also export the model's weights directly
using flash grouts and or Prometheus metrics. In the future we will also like
to export the model weights using the open telemetry metrics API. Now that we
have talked about sustainability stack let's see how you can do carbon
intensity aware scheduling. So the use case that we are trying to solve is can
you put a check or can you control the carbon intensity of your workload. For
example is it possible to fuel your workloads using renewable energy like
solar power or wind power when available and switch to fossil fuel when the
renewable energy is not at disposal. So the use case premise is based on
multi-node cluster where you have nodes in different geographical zones and the
workloads that we will be talking about is long-running patch or machine
learning workloads that that keeps on retraining algorithm or any long-running
patch workloads that are not affected by rescheduling of that runs long enough
that have an impact to considerable impact on carbon intensity and they
don't they're not affected if you reschedule them on different nodes. So our
demo setup is based on OpenShift cluster and for monitoring we're using Prometheus.
We would be using features like chains, toleration and node selectors to
orchestrate where the workload is going to run on which node and we will have a
carbon intensity forecaster that will forecast the carbon intensity of nodes
and for this demo we are only considering two Rs step that that means
that a carbon intensity forecaster would predict what is the carbon intensity
for the next two hours. So let's first describe the carbon intensity forecaster.
The forecaster has access to an exporter which scrapes time series carbon
intensity data from numerous public APIs like electricity map or national grid
and it then exports this data as Prometheus metrics. The forecaster will
then scrape the Prometheus metrics from the exporter and update its models for
each of the node with new time series data. In this demo we will have three
nodes so the forecaster will have individual models for each of the three
nodes which are in different zones of course. The carbon forecaster will then
provide a prediction of the carbon intensity of the desired region a few
hours in advance. Note that the carbon intensity forecaster and exporter are
extendable interfaces this means the forecaster can implement many different
types of time series forecasting models and the exporter can scrape from many
different carbon data APIs. So now that we have a carbon intensity forecaster
external applications like the Cron job will forecast the potential
carbon intensity sometime into the future for each of the three nodes. The
Cron job does this by making an HTTP request to the carbon forecaster using
the get slash forecasted CIN point and each of the three nodes are then
periodically assigned node labels depending on the carbon intensity. Red
stands for a relatively high carbon intensity yellow stands for a medium
carbon intensity and green stands for a relatively low carbon intensity. So in
this example note 1 is for forecasted two hours in the future to have the
highest carbon intensity so it is labeled red. Node 2 is forecasted
two hours in the future to have a medium carbon intensity, so it is labeled
yellow and note 3 is forecasted two hours in the future to have the lowest
carbon intensity so it is labeled green. Now that you have assigned labels to the
node, it's on the pod to declare its intention that what kind of node it
prefers and also what kind of node it does not prefer at all. So for example in
the pod yaml you specify node selector carbon intensity as green that means it
prefers nodes that have labels as carbon intensity green and you also have to add
as a tolerations where you have to say that you don't have the toleration
effect no execute means that this pod does not have toleration to
run on nodes that have been tainted as red. So if the scheduler will try to
schedule this pod on node 1 that has label and taint potas red, this pod
would evict within 5 seconds. So that's what the toleration second is for. So
let's see how this this looks like. So you have node 1 and there was that has
label that had label green now it's turning to red that means its carbon
intensity is increasing. So we will taint the node and we will apply the taint
as carbon intensity red and no execute. So as soon as this taint is applied the
pod is evicted from node 1 and it's assigned to node 2 which has the carbon
intensity is changing from red to green and it has been tainted. So tainting the
nodes ensures that pods are evicted by the nodes if pods have no tolerations
for taint. So this is like the whole picture we have a carbon intensity
exporter that queries the various public API to gather the carbon intensity
data and it exports them as Prometheus metrics. Now the node label and why is
a cronchial what it does it queries a carbon intensity forecaster and it queries
in head of time what is going to be the carbon intensity of the various nodes and
it patches the labels and taints based on the forecasted carbon intensity. So
let's get to the demo. First I'm going to show you how you can install a Kepler
operator on an OpenShift environment. The first the release that we have right
now is V1 alpha 1 and it has a prerequisite that it needs C group V2 and
it follows Kepler 0.4 release and it deploys Kepler both on Kubernetes and
OpenShift. So when you're deploying it on OpenShift it also reconfigure your
OpenShift nodes by applying a machine config and SCC and right now Kepler
uses local linear regression estimator in Kepler main container with offline
trained models but in the next release we are planning to provide end-to-end
learning pipeline where it can train the model as well as use the model. So if
you're interested in a code you can follow us on GitHub repository and so
let's get to the demo. To deploy the operator go inside the Kepler operator
project and run the make deploy that will create all the manifest and install
the operator in the namespace Kepler operator system. So now I'm just going
to go into the Kepler operator system the namespace and I'm going to apply
let's see if the operator has been yeah so you can see that the operator is
running now I'm going to apply the CRD and wait for the Kepler instances to get
started. So as you can see the Kepler instances are running and they are each
of them is up and running and they are each of them are running on each of the
nodes as a demon set pod so that's why you see so many of them and now I'm
going to deploy Grafana give it a second yes so Grafana is deployed now to
enable user workload monitoring I'm going to apply the config map and that
ensures that the Prometheus and the user workload monitoring namespace is
capturing the Prometheus metrics so let's see if the pods are up and running in
the OpenShift user monitoring project as you can see that all the pods are
running so now to see the metrics I'm just going to the Grafana URL just sign
in and because we applied the Grafana operator so the default Kepler dashboard
should be available give it a second it will load yeah so now you can see the
energy reporting from Kepler you can see the carbon footprint you can see the
power consumption in namespaces total power consumption pod process power
consumption and total power consumption by namespace so that's the default
Grafana dashboard so now that we have seen how you can install and play around
with your Kepler operator it's time to see how you can also do carbon
intensity aware scheduling so for that I have a cluster already ready so you can
see that there are six nodes on this cluster and for my this demo I'm not
going to run anything on the master node so I'm only going to do things on the
worker node so I have applied the cron job and I'm just waiting it to become
active all right so that job has been scheduled let's wait for you to get
completed as you can see that the crown job has been completed so let's see what
gains and what labels it has applied to the three nodes so to see the labels I'm
going to use the same script that I have written okay so you can see that the node
2 2 3 has got the label green node 2 2 2 has got the label red and node 1 2 6 has
the label yellow so anytime that we are going to schedule a carbon intensive
aware pod or workload it should favor 2 2 3 which has carbon intensity as green
let's also check what gains has been assigned and if they match the labels so
you can see that the node 1 2 a 6 and 1 2 3 which has carbon intensity as green
and yellow have no gains while the node 2 2 2 which has the carbon intensity as
red as you can see over here has the taint applied now I am going to test it
out if this works as expected by applying or by scheduling a long-running
workload before I do that I just want to watch all the pods in the namespace so
right now there's no pod so so I have applied this pod and it has it has no
tolerations for node that has tainted red and it favors a node or it wants a
node that has the label green so over here you can see that the CITS pod the
pod that I just ran had some issue or had some
problem in finding the right node that's because the default scheduler was
trying to sign it on a node that didn't have the right label or didn't have the
right taint so it took some took a while so let's verify where this pod is
running so you can see that it has been scheduled on the go it was scheduled on
two to three but right now it's running on 126 and 126 is a node that has
common intensity as yellow so that's that's completely okay the time it was
scheduled on either the green node or on the yellow node which is okay as long as
it's not scheduled on the red node so that would be all thank you for watching
the demo we would like to share a few lessons that we learned while working
on this project the first is that finding the zone cover intensity data is
not simple some points are missing and not all of them are free we also need to
support multiple and complex query types for example right now we are just
querying what is the current or the average cover intensity in zone XYZ but
we need to have more complicated queries like which zone has the lowest
cover intensity and we are also thinking of contributing the work that we have
done with the cover intensity forecasting and integrating it with
green software foundation carbon away SDK which is another open source
community that has been working on sustainability and green software so the
road ahead for us looks like we are thinking of extending the multi node
logic to multi cluster and we're exploring how you can do that using kcp
and we are also thinking of integrating carbon intensity awareness in
Kubernetes plugins existing plugins for example the trimaran target load
packing is a scheduler plugin by in the Kubernetes sake and we're thinking of
integrating the profile with the carbon intensity awareness and also thinking
of how you can tune trimaran further for energy efficiency so that was all if
you are more interested in learning about the principle of that capra is
based on you can follow the link and check out a project we have attached the
GitHub repo for the project as well as the model server and thank you so much
and any questions
you
okay do you want to take that question
do you see the question okay so yeah sir um sorry I'm just trying to see the
questions I have to switch back and forth okay sir um sorry I'm just trying to see the questions I
somebody asked how do we split the
energy for the pod
oh um yeah I think I can answer that um this was done on Kepler I believe and I was developed
by somebody else but essentially there are two ways for like the model server we also have recently
have like models that'll use the performance metrics and then the software counters to directly
try and predict pod energy um when it that that's one option and then second option in Kepler is
typically um once it generates the energy it'll then try and attribute it I believe to each of the
pods and I think that's based on um is it based on cpu utilization proof I don't know yeah what we
do is we monitor the cpu utilization although whatever the cpu instruction or the process
is going on and then we use cgroup id to kind of like attribute what how that energy is related
to which pod because we take the cgroup id and we translate that which particular process or
container it's related to so that's how we gather the metrics so the important thing to note over
here is Kepler uses the models to estimate or predict the energy consumption and these models
are already trained they already have they are already being published so Kepler uses these
models to predict pod energy level consumption on scenarios where you're not running on bare metal
on those cases we don't have the access to the inbuilt power meter so in those scenarios we
estimate or we predict what is going to be the energy consumption
so another question is how what is the credibility of the
uh uh greenness uh that data is as good as the data published by the public api for example
we have electricity map in us and national grid in europe and uh that is one of a one of a problem
as well that the the greenness or the accuracy of the carbon intensity is as good as the data
that's being published by the public api we cannot control that
okay i should probably note that we will also aim for any data that's from the government so i think
national grid is uh straight from is from the uk government so i think that's pretty reliable and
we will always make sure that the data that we use is from reliable sources