Okay. I think we're ready to start. Oh, excellent. This time it worked perfectly. Thank you
so much. Yeah. Chris is going to talk about C Group V2, seven years of C Group V2 in the
kernel, very exciting time, and the future of Linux resource control. Take it away.
Hello, everybody. Oh, yes. Please go on. Thank you. That's it. I'm done. Goodbye. Hello. I'm
Chris Down. I work as a kernel engineer at Metta. I work on the kernels memory management
subsystem, especially I'm a contributor to C Groups, which are one of the things which
underpins our model of containers. I'm also a maintainer of the system D project. So there's
two things on this slide, which you can hate me for. Most of the time I'm thinking about,
you know, how we can make Linux just a little bit more reliable, just a little bit more usable
at scale. We have a million plus machines. We can't just buy more RAM. It's not really a thing
we can do. So we need to extract the absolute maximum from every single machine. Otherwise,
there's a huge loss of capacity that could result. So that's the kind of thing I want to talk to you
about today. However, the last seven years we have done this at Metta, how we've improved the
reliability and capacity and extracted more efficiency. At Metta and in industry, we are
increasingly facing this kind of problem where we can't effectively solve scaling problems just by
throwing hardware at the problem. We can't construct data centers fast enough. We can't
source clean power fast enough. We have hundreds of thousands of machines and we just can't afford
to waste capacity because any small loss in capacity on a single machine translates to a
very large amount at scale. Ultimately, what we need to do is use resources more efficiently
and we need to build the kernel infrastructure in order to do that. Another challenge that we have
is that many huge site incidents for companies like us and companies of our size are caused by
lacking resource control. Not being able to control things like CPU, IO, memory and the like is one
of the most pervasive causes of incidents and outages across our industry and we need to sustain
an initiative industry-wide in order to fix this. So how does all of this relate to this
C-groups thing in the title? So C-groups are a kernel mechanism to balance and control and isolate
things like memory, CPU, IO, things that you share across a machine, things that processes share
and I'm sure if you've operated containers before, which I'm going to assume that you have, judging
by the fact you're in this room otherwise you may be lost in looking for the AI room,
you know every single modern container runtime uses this. Stalker uses it, Chorus uses it,
Kubernetes uses it, SystemD uses it. The reason they use it is because it's the most mature platform
to do this work and it solves a lot of the long-standing problems which we had with kind of
classic resource control in the form of view limits and things like that. C-groups have existed
for about 14 years now and they have changed a lot in that time. Most notably, seven years ago
in kernel 4.5 we released C-group 2. I gave a whole talk around the time when that happened on
why we were moving to a totally new interface, why we weren't just iterating on the old interface
and if you're interested in a really in-depth look at that then here's a talk which you can
go and take a look at. But the most fundamental change really is that in C-group 2 what happens is
that you enable or disable resources in the context of a particular C-group. In C-group 1 what you
have is a hierarchy for memory, a hierarchy for CPU and the two will never meet. Those two things
are completely independent. SystemD when it creates things in C-group V1 it will name them the same
they get called something.slice or something.service but they have no relation to each other across
resources. But in C-group 2 you have just a single C-group and you enable or disable resources in
the context of that particular C-group so you can enable say memory control and IO control together.
That might seem like you know an aesthetic kind of concern but it's really not. Without this major
API change we simply cannot use C-groups to do complex resource control. Take the following
scenario. Memory starts to run out on your machine. So when we start to run out of memory on a
pretty much any modern operating system what do you do? Well you try and go and free some up.
So we start to reclaim some page caches. We start to reclaim maybe some anonymous pages if we have
swap. And this results in disk IO. And if we're particularly memory bound and it's really hard
to free pages and we're having to walk the pages over and over and over to try and find stuff to
free then it's going to cost a non-trivial amount of CPU cycles to do so. Looking through available
memory to find pages which can be free can be extremely expensive on memory bound workloads.
On some highly loaded or memory bound systems it can take you know double digit amount of CPU
from the machine just to do this walking. It's a highly expensive process. And without having
the single resource hierarchy we cannot take into account these transfers between the different
resources how one leads to another because they're all completely independent. If you've been in
the containers different before you've probably thinking I've seen this guy before and I think
he's given this exact talk about three years ago. I'm sure some of you think and that already. Well
the company name isn't the only thing which has changed in 2020. Also some seagrups things have
changed since 2020 and obviously I don't want to rehash the same things over and over. I don't
want to bore you. So this talk will mostly be about the changes since the last time I was here in
2020 with just a little bit of context setting just a little bit. This talk is really about the
process of getting resource isolation working at scale. It's what it needs to happen in production
not just in a theoretical concern. The elephant in the room of course is COVID. The last three years
have seen pretty significant changes in behavior due to COVID especially for a platform like Facebook
which we own of course. This was by about 27% over what you would usually expect and this came
at a time where not only you're seeing increased demand but you literally can't go out and buy
memory. You can't go out and buy more CPUs. You can't go out and buy more disks because there's a
shortage because there's COVID. So what we really needed was to make more efficient use of the
existing resources on the machine right. We need to have an acceleration or existing efforts around
resource control in order to do that to make things more efficient. Now almost every single time that
I give this sounds like a personal point of concern. Every time I give this talk somebody on
Hacker News comments why don't you just get some more memory? Now I don't know how trivial people
in this room think that is when you've got several million servers but it is slightly
difficult sometimes. For example there's a huge amount of cost involved there and not just the
money which is indeed substantial and I'm very glad it's not coming out of my bank account but
also in things like power draw, in things like thermals, in things like hardware design trade-offs.
Not to mention during COVID you just couldn't get these kind of, you couldn't get a hard drive,
you couldn't get some memory. You'd go down to your local Best Buy and do it but that's about it.
So not really an option. So here's a simple little proposition for you, for anyone in the
room who wants to be brave. How do you view memory usage for a process in Linux?
Oh come on. Free! My man said free. Oh lord. This was a trap. So I appreciate it though,
big up about that. So yeah, so free and the like really only measure like one type of memory.
They do have caches and buffers in the side but the thing is okay so for free or for PS which
were shut at the back you know you do see something like the resident set size and you see some other
details and you might be thinking hey you know that's fine like I don't really care about some of
the other things that's the bit which my application is really using. For example we don't necessarily
think that our programs rely on caches and buffers to operate in any sustainable way but the problem
is the answer for any sufficiently complex system is almost certainly that a lot of those caches
and buffers are not optional. They are basically essential. Let's take Chrome just as a facile
example. The Chrome Binary's code segment is over 130 megs. He's a chunky boy. He is. He's a big boy.
We load this code into memory. We do it gradually. We're not we're not maniacs. We do it gradually but
you know we do it as part of the page cache. A boy if you want to execute some particular part of
Chrome you know this cache isn't just nice to have the cache that has the code in it that runs this
particular part of Chrome. We literally cannot make any forward progress without that part of the
cache and the same goes for caches for the files you're loading especially for something like Chrome
you probably do have a lot of caches so eventually those pages are going to have to make their way
into the working set. They're going to have to make their way into main memory. In another
particularly egregious case we have a demon at Meta and this demon aggregates metrics across a
machine. It sends them to centralized storage and as part of this what it does is it runs a whole
bunch of janky scripts and these janky scripts go and collect things across the machine. I mean
we've all got one. We've all got this kind of demon where you collect all kind of janky stuff
and you don't really know what it does but it sends some nice metrics and it looks nice and
one of the things we were able to demonstrate is while the team had this demon thought that it
took about 100 to 150 megabytes to run using the things that we'll talk about in this talk
it actually was more like two gigabytes. So the difference is quite substantial on some things
like you could be quite misunderstanding like what is taking memory on your machine.
So in C-group 2 we have this file called memory.current that measures the current memory usage for
the C-group including everything like caches, buffers, kernel objects, so on. So job done right?
Well no the problem is here that whenever somebody comes to these talks and I say something like
don't use RSS to measure your application they go and see oh we've added a new thing called
memory.current and it measures everything great. I'm just gonna put some metrics based on that
but it's quite important to understand what that actually means to have everything here right.
The very fact that we are not talking about just the resident set size anymore means the
ramifications are fundamentally different. We have caches, buffers, socket memory,
TCP memory, kernel objects, all kind of stuff in here and that's exactly how it should be because
we need that to prevent abuse of these resources which are valid resources across the system.
They are things we actually need to run. So understanding why reasoning about memory.current
might be more complicated than it seems comes down to why as an industry we tended to gravitate
towards measuring RSS in the first place. We don't measure RSS because it measures anything useful
we measure it because it's really fucking easy to measure. That's the reason we measure RSS like
there's no other reason like it doesn't measure anything very useful. It kind of tells you vaguely
like maybe what your application might be doing kind of but it doesn't tell you anything of any of
the actually like interesting parts of your application only the bits you pretty much already
knew. So memory.current suffers from pretty much exactly the opposite problem which is it tells
you the truth and don't really know how to deal with that. Don't really know how to deal with
being told how much memory application is using. For example if you set an 8 gigabyte memory limit
in C root v2 how big is memory.current going to be on a machine which has no other thing running on
it. It's probably going to be 8 gigabytes because we've decided that we're going to fill it with
all kind of nice stuff. There's no reason we should evict that. There's no reason we should take away
these nice you know K mem caches. There's no reason we should take away these slots because
we have free memory so why not. Why not keep them around. So if there was no pressure for this to
shrink from any outside scope then the slack is just going to expand until it reaches your limit.
So what should we do? How should we know what the real needed amount of memory is at a given time?
So let's take an example Linux kernel build for example which with no limits has a peak memory.current
of just over 800 megabytes. In C root v2 we have this tunable called memory.high. This tunable
reclaims memory from the C group until it goes back under some threshold. It just keeps on reclaiming
and reclaiming and reclaiming and throttling until you reach back under. So right now things take
about four minutes with no limits. This is about how long it takes to build the kernel and when I
apply you know a throttling like a like a reclaim threshold of 600 megabytes actually you know the
job finishes roughly about the same amount of time maybe a second more with about 25 percent less
available memory at peak and the same even happens when we go down to 400 megabytes. Now we're using
half the memory that we originally used with only a few seconds more wall time. It's it's pretty good
trade-off. However if we just go just a little bit further then things just never even complete. We
have to we have to control see the build right and this is nine minutes in it's still ain't done. So
we know that the process needs somewhere between 300 and 400 megabytes of memory but it's pretty
error prone to try and work out what the exact value is. So to get an accurate number for services
at scale which are even more difficult than this because they dynamically shrink and expand depending
on load we need a better automated way to do that. So determining the exact amount of memory
required by an application is a really really difficult and error prone task right. So SEMPAI is
this kind of simple self-contained tool to continually poll what's called pressure stall
information or PSI. Pressure stall information is essentially a new thing we've added in CIGRI2
to determine whether a particular resource is oversaturated and we've never really had a metric
like this in in the Linux kernel before. We've had many related metrics for example for memory we
have things like you know page caches and buffer usage and so on but we don't really know how to
tell pressure or over subscription from an efficient use of the system those two are very
difficult to tell apart even with using things like page scans or or so on it's pretty difficult.
So in SEMPAI what we do is we use these PSI pressure stall metrics to measure the amount of time
which threads in a particular C group were stuck doing in this case memory work. So this pressure
equals 0.16 thing kind of halfway down the slide means that you know 0.16 percent of the time I
could have been doing more productive work but I've been stuck doing memory work. This could be
things like you know waiting for a kernel memory lock it could be things like being throttled
could be waiting for reclaimed to finish even more than that it could be memory related IO which
which can also dominate to be honest things like refolding file content into the page cache or
swapping in and pressure is essentially saying you know if I had a bit more memory I would be able
to run so much faster 0.16 percent faster. So using PSI and memory.high what SEMPAI does is
adjust just enough memory pressure on a C group to evict cold memory pages that aren't essential
for workload performance. It's an integral controller which dynamically adapts to these
memory peaks and troughs an example case being something like a web server which is somewhere
where we have used it when more requests come we see that the pressure is growing and we expand
the memory.high limit when fewer requests are coming we we see that and we start to decrease
the amount of working set which we give again so it can be used to answer the question you know
how much memory does my application actually use over time and in this case we find for the
compile job the answer is about like 340 megabytes or so and that's fine you might be asking yourself
what's the what are the benefits of this shrinking like why why does this even matter to be honest
surely like when you're starting to run out of memory Linux is going to do it anyway and you're
not wrong like that's true but the thing is what we kind of need here is to get ahead of memory
shortages which which could be bad and amortize the work ahead of time when your machine is already
highly contended it's already being driven into the ground and going towards the umkiller it's
pretty hard to say hey bro could you just like like give me some pages right now like it's it's
not exactly like what what's on its mind it's probably desperately trying to keep the atomic
pool going so there's there's another thing as well which is you know it's pretty good for
determining regressions which is what a lot of people use for rss for right like we this is the
way we found out that that demon was using two gigabytes of memory instead of 150 megabytes
of memory so it's pretty good for finding out hey how much does my application actually need to run
so the combination of these things means that senpai is an essential part of how we do workload
stacking of matter and it not only gives us an accurate read on what the demand is right now
but allows us to adjust stacking expectations depending on what the workload is doing this
feeds into another one of our efforts around efficiency which is improving memory offloading
so traditionally on most operating systems you have only one real memory offloading location
which is your disk um even if you don't have swap that's true because you do things like
demand paging right you page things in gradually and you also have to you know evict and get things
in the file cache so we're talking also here about like a lot of granular intermediate areas
that could be considered for some page offloading for infrequently access pages but they're not
really so frequently used um getting this data come into main memory again though can be very
different in terms of how difficult it is depending on how far up the the triangle you go right for
example um it's much easier to do it on an ssd than a hard drive because hard drives don't well
they're slow and they also don't tolerate random head like head seeking very well but there are
more granular gradual things that we can do as well for example one thing we can do is to start
look at exact strategies outside of hardware one of the problems with the duality of either being
in ram or on the disk is that even your disk even if it's quite fast even if if it's flash it tends
to be quite a few orders of magnitude slower than your main memory is uh so one area which we have
have been heavily invested in is looking at what we might term warm pages uh in Linux we have talked
a lot about hot pages and cold pages if you look in the memory management code but there is like
this kind of part of the working set which yes i do need it relatively frequently but i don't need
it to make forward progress all the time so zswap is one of these one of these things we can use
for that it's it's uh essentially a feature of the Linux kernel which compresses pages which looks
like they will compress well and are not too hot into a separate pool in main memory we do have
to page fold them back in into main memory again if if we actually want to use them of course
but it's several orders of magnitude faster than trying to get it off the disk
we still do have this swap for infrequently access pages there tends to be quite a bit
cold working set as well um but you know this is kind of like this tiered hierarchy where we want
to have warm uh warm pages instead swap hot pages in in main memory and kind of cold pages and swap
one problem we had here was that even when we configure the kernel to swap as aggressively as
possible it still wouldn't do it um if you've actually looked at the swap code and i've had
the unfortunate misery of working on it um this you'll learn that swap code was implemented a very
long time ago by the people who knew what swap did and how things worked but none of them are
around to tell us what the hell anything means anymore and it's very confusing so i can't even
describe to you how the old algorithm works because it has about 500 heuristics and i don't know why
any of them are there um so for this reason you know we try to think how can we make this a little
bit more efficient we are using non-rotational disks now we have zswap we have flash disks we
have ssds we want to make a an algorithm which can handle this better so from kernel 5.8 um we
have been working on a new algorithm which has already landed um so first we have code to track
all swap ins and cache misses across the system so for every cache page we're having to page
fold and evict and page fold and evict and page fold and evict over and over again what we want to
do is try and page out a heat page instead if we're unlucky and this heat page actually it turns
out to be hot then you know no biggie like we we've made a mistake but we'll try a different one
next time we do have some heuristics to try and work out which one is hot and which one is not but
they are kind of expensive so we don't use a lot of them um however you know if if we are lucky and
the heat page does stay swapped out then that's one more page which we can use for file caches and
we can use it for other processes and this means that we can engage swap a lot more readily in
most scenarios importantly though we are not adding ioload this doesn't increase ioload or
decrease endurance of the disk um we are just more intentional about in choosing how to apply
the i it doesn't double up um we only trade one type of paging for another and our goal here is to
reach an optimal state where the optimal state is doing the minimum amount of i o in order to sustain
workload performance um so ideally what we do is have this tiered model of you know like I said main
memory z swap and swap on disk this is super simple idea compared to the old model although the old
algorithm has a lot of kind of weird heuristics as I mentioned a lot of penalties a lot of kind of
strange things um in general it was not really written for an era where SSDs exist or where
z swap exists so it's understandable that it needed some some care and attention
so what were the effects of this change in prod like what what actually happened so on web servers
we not only noticed like an increase in performance but we also noticed a decrease in heat memory by
about two gigabytes or so out of about 16 gigabytes total the cache grew to fill this newly freed
space and it grew by about two gigabytes from about uh two gigabytes of cache to four gigabytes of
cache we also observed a measurable increase in web server performance from this change which is
deeply encouraging and these are all indications that you know we are now starting to reclaim the
right things actually we are making better decisions because things are looking pretty
positive here so not only that but you see a decrease in disk i o because we are actually
doing things correctly we are making the correct decisions and it's not really that often that
you get a benefit in performance disk i o memory usage instead of having to trade off between them
right so it probably indicates that this is the better solution for this kind of era this also
meant that on some workloads uh we now had opportunities to stack where we did not have
opportunities to stack before like running say multiple kinds of ads jobs or multiple kinds
of web servers on top of each other uh many machines don't use up all of their resources
but they use up just enough that it's pretty hard to stack something else on top of it
because you're using just enough that it's not actually enough to sustainably run to workload
side by side so this is another thing where we've managed to kind of push the needle just a little
bit so that you can make quite a bit more use uh an efficiency out of the servers that exist
the combination of changes to the swap algorithm using z-swap and squeezing workloads using senpai
was a huge part of our operation during covid all of these things acting together we termed tmo
which stands for transparent memory offloading and you can see some of the results we've had in
production here in some cases we were able to save up to 20 percent of critical fleet-wide
workloads memory with either neutral or even in some cases positive effects on workload performance
so this opens up a lot of opportunities obviously in terms of reliability stacking
and future growth this whole topic has a huge amount of cover i really could just do an entire
talk on this um if you want to learn more i do recommend the post which is linked at the bottom
my colleagues johannes and dan wrote an article with a lot more depth on you know how we achieve
what we achieved and on things like cxl memory as well so let's come back to this this slide
from earlier um we briefly touched on the fact that if bounded one resource can just turn into
another a particularly egregious case being memory turning into i o when it gets bounded
for this reason it might seem counterintuitive but we always need controls on i o when we have
controls on memory otherwise memory pressure will always just directly translate to disk i o
probably the most attuned way to solve this is to try to limit disk bandwidth or disk i ops
however this doesn't really manifest usually very well in reality if you think about any modern
storage device they tend to be quite complex they they're q devices you can throw a lot of commands
of them in parallel and when you do that you often find that hey you know magically it can do more
things the same reason we have i o schedulers because we can optimize what we do inside the disk
also the mixture of i o really matters like reads versus writes sequential versus random
even on ssds these things tend to matter um and it's really hard to turn to determine
a single metric for loadedness for a storage device because the cost of one i o operation or
one block of data is extremely variable depending on the wider context um so it's it's also really
punitive to just have a limit on you know how much can i write how many i ops can i do um because
even if nobody else is using the disk you're still slowed down to this level there's no opportunity
to make the most of the disk when nobody else is doing anything right so it's not really good for
this kind of best effort bursty work on a machine which we would like to do so the first way that
we try to avoid this problem is by using latency as a metric for workload health so what we might
try and do is apply a maximal target latency for i o completions on the main workload and if we
exceed that we start dialing back other c groups with lucid latency requirements back to their own
configured thresholds what this does is this prevents an application from thrashing on memory so much
that it just kills i o across the system this actually works really well for systems where
there's only one workload but the problem comes when you have a multi workload stacked case like this
here we have two high priority workloads which are stacked on a single machine one has an i o
dot latency of 10 milliseconds the other has 30 milliseconds but the problem here is as soon as
workload one gets into trouble everyone else is going to suffer and there's no way around that
we're just going to penalize them and there's no way to say you know how bad is the situation
really and is it really them causing the problem this is fine if you're you know the thing you're
throttling is just best effort but it's we here we have two important workloads right so how can
we solve this so our solution is this thing called i o dot cost which might look very similar at first
but notice the omission of the units these are not units in milliseconds these are weights in
a similar way to how we do cpu scheduling so how do we know what 40 60 or 100 mean in this context
well they add up to 200 so the idea is if you are saturating your disk you know best effort
outside will get 40 will get i guess 20 percent of of the work it'll workload one will get 50
and workload two will get 30 so it balances out based on this kind of shares or weights like model
how do we know when we reach this 100 percent of saturation though so what i o dot cost does is
build a linear model of your disk over time it sees how the disk responds these variable loads
passively and it works based on things like you know read or write i o whether it's random or
sequential the size of the i o so it boils down this quite complex operation of you know how much
can my disk actually do into a linear model which it which it handles itself it has a kind of a q
s model you can implement but there's also a basic on the fly model using q depth so you can read
more about it in the links at the bottom i won't waffle on too much but it is something which you
can use to do kind of effective i o control in the old days i came to this room and talked about
secret b2 and the historical response was basically that's nice docker doesn't support it though so
please leave um i've had a nice chat with some docker lutz uh no the docker people are very nice
and so are all the other container people and what's happened is we have it almost everywhere
almost everywhere secret b2 is a thing we have quite a diversity of container run time some
police report is basically supported everywhere um so even if nothing changes from your side
moving to secret b2 means that you know you get significantly more reliable accounting for free
we spent quite a while working with docker and system defoaks and so on and so forth to get
things working and we're also really thankful to fedora for making secret b2 the default since
fedora 32 as well as making things more reliable behind the scenes for users this also you know
got some people's ass into gear when they had an issue on their github on their github that says it
doesn't work in fedora so cheers fedora people uh it was a kind of a good signal that you know this
is what we are actually doing this is what we as an industry as a as a technology community are
actually doing uh and that was quite helpful the kd and gnom folks have also been busy using cgroups
to give uh a better management of that kind of desktop handling david edmundson and henry chain
from kd in particular gave this talk at kd academy the title of talk was using cgroups to make
everything amazing now i'm not brazen enough to title my talk that but i'll just let it speak for
itself for their one um it basically goes over the use of cgroups and c v2 for resource control
and for interactive responsiveness on the desktop um so this is definitely kind of a developing
space obviously there's been a lot of work on the server side here um but if you're interested in
that i definitely recommend you know giving the talk a watch it really goes into challenges they
had and then unique features c v2 has to solve those finally android is also using the metrics
exported by the psi project in order to detect and prevent memory pressure events which affect the
user experience as you can imagine on android interactive latency is extremely important
you don't it would really suck if you're about to click a button and then you click it and that
requires allocating memory and the whole phone freezes i mean it does still happen sometimes
but obviously this is something which which they're trying to work on and we've been working
quite closely with them to integrate the psi uh project into the into android
hopefully this talk gave you some ideas about things you'd like to try out for yourself um
we're still very actively improving uh kernel resource control it might have been seven years
since we started but you know we still have plenty of things we want to do and what we really need
is your feedback what we really need is more examples of uh how the community is using c v2
and problems and issues you've encountered um obviously everyone's needs are quite different
and i and others are quite eager to know what we could be doing to help you what we could be doing
to make things better what we could be doing to make things more intuitive because there's
definitely work to be done there and i'll be around after the talk if you want to chat but
feel free to drop me an email message me on mastodon always happy to hear feedback or suggestions
um i've been chris down and this has been seven years of c at c review to future of
Linux resource control thank you very much