Thank you everyone for coming to my talk.
My name is Ignat.
I work for Cloudfer.
Who here heard about Cloudfer?
Who's using Cloudfer?
Should be more hands, by the way, because even if he didn't hear about Cloudfer, probably
using Cloudfer one way or another.
This is my first time at FOSDEM.
So thank you very much for exchanging your lunchtime for my talk.
I hope it will be really exciting.
And today we're going to talk about Linux kernel upgrades and how you should do them,
and most likely how you should not do them.
So a little bit about myself.
I do Linux at Cloudfer.
I enjoy system security and performance, and I'm passionate about low-level programming.
So the Linux kernel, drivers, bootloaders, and other stuff, reading in unsafe programming
languages.
Okay, before we start, a little bit of show of hands.
So what would you do in this case?
Imagine you're working at the shoot on your laptop.
You're doing stuff.
And yeah, and suddenly this pop-up comes in.
I'm like, oh, updates available.
What would you do?
Like install now?
Who's install now?
Oh, nice.
Well, who's resumed later?
Do later?
50-50.
So those people who raise their hands for install now, what if instead it wasn't your
computer but a production system?
Who would press install now?
No, very few.
But yeah, you like Bitcoin probably, right?
Risky.
Yeah, and usually it's something like that for production system, right?
So it's a difficult choice between remind me later and don't remind me at all.
Please don't install.
And this is natural, I think.
Because it's connected to the fact how do we perceive software updates, especially for
production systems, right?
Well, we don't perceive them really good, right?
So we perceive software updates as kind of these monsters where they come in, they're
nasty, they're bugging you.
They kind of like an update can break your stuff.
Like the traditional engineering motto, if it works, don't touch it, why do we need to
install an update, right?
Yeah.
But the thing is, with regular software updates, we perceive them as monsters, but they're
not really scary.
They're kind of annoying and ugly, but pesky, but not that much.
When it comes to Linux Chrome upgrades forever, it's mostly like this big monster trying to
destroy the universe, right?
And why that?
And again, it's natural because, well, we know how to deal with regular software updates.
Yeah, you have a service, it crashes once a week in production, how do we fix it?
Well, if you use like system D, you'll just set a policy for it to restart it, and yeah,
job is done.
It can go home.
Well, yeah, you'll be kind of restarting a service once a week.
Your service will be in slightly degraded state, but yeah, you'll buy yourself some time
to investigate and fix it later.
When the Linux crash, Linux kernel crashes, however.
Well, technically, this is you, right?
So it's end of the world because you don't have any system D to restart it.
You don't have any metrics and understanding why it happened.
Your service is not reachable.
No SSH to debug nothing.
Well, it's kind of, it's indeed end of the universe.
And that's why usually we're scared of software updates, but when it comes to Linux kernel
updates, we're scared like even more.
And this why like people avoid updating their Linux kernel for the most part, right?
Especially in production systems.
But there are common risks.
If you don't apply software updates regularly, especially for the Linux kernel.
So the first one of them is like your bugs are not getting fixed.
And here's some statistics.
So I will be talking about the Linux kernel release cycles a little bit later to introduce
you.
This is basically the preview is a snapshot of all bug fixes releases of a stable kernel
branch 6.1.
So the latest Linux LTS kernel is 6.6, but because it doesn't have as many releases,
so you don't get pretty graphs, I decided to go to the previous one, 6.1.
And what this graph shows you is the number commits per each bug fix release on a 6.1
stable kernel.
So again, I'll be talking about release types later in this talk, but you at this point,
you should know that these bug fixes releases happen roughly every week.
And these bug fixes releases are what the name says.
They're only bug fixes.
There are no new features, no subsystem rewrite bugs and security vulnerabilities.
And as you can see, so far the 6.1 stable kernel had 67, 76 releases, and out of 76
releases, there are 50 releases with more than 100 commits in them.
So it means 100 bug fixes every week.
Almost every release, really, like 80% or something, right, if I'm doing the mass write.
20 releases, so it's 25-ish percent every four release, every fourth release, so roughly
every month, have more than 200 commits and maybe 200 potential bug fixes.
And there are like these five mega releases with more than 500 commits in them.
And actually, if you look in the graph, it's actually seven, but the last two barely made
it to the 500.
But yeah, these are like these mega releases with a lot of commits.
So if you don't upgrade your kernel regularly, your system runs with all these potential
bugs, like, and every week you delay, you're kind of missing out at least on 100 bug fixes
in your kernel.
Second, what you'll be missing out is on potential performance improvements.
This is a snapshot from Cloud 4 production systems when we started evaluating, we were
using at the time the 5.4 stable kernel and we started to evaluate 5.10 kernel.
And so we did like half and half deployment to set of identical servers, like one with
5.4, one with 7.
And this is like, this graph shows the, you know, like average memory consumption per
server and you can see that on 5.10, we have much less memory consumption.
And people are like, what did we break?
Like, what happened?
And nothing bad happened.
It's actually, yeah.
So that was 5.4, 5.4 versus 5.7.
So we kind of saved something around 5 gigs of RAM per server.
And like, at first we thought something broke, but when you dig later into the mailing list,
you just, you see that like, you know, like some other folks in this case, this was Facebook
now matter, and nice people did some improvements in the kernel code and improved the memory
management system.
And now you are consuming less memory for the same workload, with the same performance.
Right?
So it's like, it's almost like downloading RAM from the Internet.
And you basically get it for free if you just apply an update, like it's open source, right?
And recent news, for example, the latest LTS kernel is 6.6 and it rumored that it has a new
scheduler in that.
And there is a phoronics article that says like, if you're using Nginx, with that scheduler,
it will be much, much more performant.
So you'll get it for free as well if you move to 6.6 potentially.
I mean, I don't have any pretty graphs because it didn't work better for us, but maybe for
you it will.
Yeah.
And I mean, looking a little bit forward to the next talk, after mine, there will be some
discussion, I hope, regarding some security improvements with TPMs and the Linux kernel,
and it will involve some code probably, and you only can get it if you upgrade.
So let's look at the same data, but from the point of view of accumulating changed delta.
So this is basically the same data, number of commits per release, but it's kind of accumulating.
It shows the number of commits since the initial release, right?
And in this graph, you can easily see you can commit, you can calculate changed delta.
For example, if you're on a 6.1.10 bug fix release and you want to upgrade to 6.120, you
can commit changed delta is 1,762 commits, right?
And basically, if you assume, which would be natural to assume the fact that the number
of changes is proportional to risk, so for example, these are like 1,762 bug fixes you're
running with, so it's kind of like the amount of risk you're taking by not upgrading is
proportional to that number.
Now let's say you wanted to upgrade, but for some reason you decided to delay, and you
decided for, like, I don't know, it's end of the quarter you had a big incident, you
have, you know, like your company gets a big contract, so you decided not to change anything
to be more stable for the time being, and you're postponing the upgrade, and when you
actually decide to upgrade now, you're upgrading from 6.1 to 10 to 6.1 to 30, which is like
you just extended your not upgrading time twice.
And you might think naturally that your risk grew 2x, but if you calculate the difference
here, you may see that in some cases, a 2x postponing, 2x not time not upgrading, your
risk actually can grow higher, now your risk grew 2.221.
Right, so the risk sometimes of not upgrading systems and delaying may grow higher than
the time you're not upgrading.
So yeah, for 2x delay of not upgrading, we get 2.21 more risk of hitting above.
If you're not upgrading, security vulnerabilities are not getting patched.
So this is a similar graph, but it now shows only publicly known CVEs patched in a bugfix
release, and just this data is actually crowdsourced, so it might be incomplete, but even from this
you can see that out of 71 releases, for which data is available right now, 56 releases,
like again almost 80%, have at least one CVE patched.
And there is 18 releases again, 20, 25%, with more than five CVEs patched.
So again, if you're not upgrading kernel regularly, you're running not only with security
vulnerabilities, you're running with known, publicly known security vulnerabilities,
for which most likely an exploit is available somewhere on the internet.
Not patching your security vulnerabilities also puts a risk on your compliance, so if
your production systems are subject to some sort of compliance, you have a required time
at which you should be patching these vulnerabilities.
So for example, if you're subject to PCI DSS compliance, like for most payment systems
and stuff, it says that the critical or high security patches or updates should be patched
within one month of release.
So imagine there is a known, publicly known security vulnerability in the Linux kernel
and you have one month to fully roll it out to your production systems.
Who here knows about Acvifox?
What happened to it?
A few hands.
So it wasn't about the Linux kernel, but Acvifox was running an Apache server, an old version
and patched with known security vulnerabilities and people used an exploit on their system
and exultrated some data.
And it was a big mess.
It was really expensive for the company.
It cost its reputation as well as a lot of money, compensation, a lot of lawsuits,
so very, very, very bad.
Which brings us to not so fun fact.
You remember like in the old days when you go to admin forums in 2000 and people were boasting
around how long their server, how stable their servers are posting their uptime.
Like my uptime is two years, three years.
Well, since Apache and Linux kernel requires a reboot, now it's not cool anymore.
So if your uptime is more than 30 days, you're most likely vulnerable and not compliant to something.
So now let's talk about an anti-partness for Linux kernel releases.
If you're managing a production system, for most software updates there is some kind of a change management process
or well understood practices which usually like sysadmins, sres and engineers apply to manage change.
But most of them unfortunately do not apply to the Linux kernel.
So when you go and want to update your production system, oftentimes for a software update,
the change management process will ask you why.
Why do you want to update and which things from the change log on this new version is applicable to us.
Like are we really fixing bugs that are hitting us? Are we really fixing TVs that are applicable to us?
Well, and it doesn't apply here just because of this graph, right?
So remember these bug fix releases happen every week and like with most of the releases having more than 100 commits,
so it doesn't mean that every week you should be going through all the commits and trying to understand
if that particular fix is actually applicable to your system.
For this it's very expensive. You need a huge team of really good Linux kernel experts to understand
if you know like this off by one thing in the memory management subsystem is actually triggerable on your work.
So if you do go this way, mostly you'll be doing something like this.
You will be just continuously stamping releases for no particular reason with no analysis.
Then goes for security vulnerabilities.
You say, yeah, we need like we have five CVs we need to patch due to compliance and then you may ask somebody may ask the question,
is the security vulnerability actually exploitable in our systems? Do we use that subsystem?
Sometimes it's an easy answer if it's in a driver for I2C and you're on a machine which doesn't have an I2C,
then you can say no, but most of the time it's much more hard and like many exploits, many successful exploits
are not like some kind of high severity big vulnerability.
Sometimes attackers manage to change smaller vulnerabilities properly to get an exploit.
So going back to that, like going back to this question if you really think of it like who can answer this question?
Technically this question can be answered by the attacker because if the attacker has the list of the CVs running in the system,
they're highly motivated to break into the system and this is their bread and butter.
They spend like 24 or 7 to design and implement successful exploits.
But unfortunately you're not asking this question to the attacker, you don't know who they are, right?
You're asking for a security patch reviewer, you're going to some team for security people and they're like,
oh is this vulnerability applicable?
And they're highly motivated to go home on time, right?
And they need to review several patches a day, not only from the Linux kernel but from many other subsystems
and do other stuff like doing security architectures, doing compliance, many things.
So it's kind of you're asking this person, right?
And the quality of that answer will not be great.
They will say like, yeah, maybe yes, maybe no.
So the best course of action is just not ask this question and assume that every CV is applicable to your workload and patch it.
Well, one of the traditional approaches in upgrading stuff, especially the Linux kernel is soaking.
Like let's put it in the cannery somewhere and soak it for one month to ensure we don't get anything.
Yeah, but basically you come back to this by soaking it in a subset of your production, you're not releasing to elsewhere
and you start accumulating change delta and therefore your risk of not upgrading grows and hitting a potential bug.
Same with security vulnerabilities, if you're soaking it somewhere, you're not patching CVs in your production
and you'll have the risk of being hacked and you're probably, for one month's soak, you're probably all,
like if you have a one month's soak time somewhere in a cannery, you're already violating some of the compliance which dictates you have 30 days to roll out everywhere.
But what does high soak time means in practice?
It's usually because we just don't know what we're looking for and what it translates to.
We don't have any success metrics or observability how our kernel performs,
is it performed the same way after upgrade as it was performing before that.
We also don't know our workload.
My team gets questions, the same question from many teams, right?
Will the kernel break my software?
But for every team, the subsystem of interest is different.
For a database team, they're mostly focused on IEO and file system performance,
but for some image processing team, they mostly care about CPU scheduling and CPU performance.
The question should be, will it, I'm interested in this particular subsystem,
will it break my workload within IEO workload or like CPU bound workload
or I'm interested in some hardware or something like that, networking as well.
And probably it indicates lack of sufficient production kernel testing.
Within the Linux kernel, you can also ensure that an update doesn't break someone's workload
if you write a particular Unix test, an integration test.
The Linux kernel has this nice suite called case self-test, which is easily extendable.
If you care about a particular feature in the Linux kernel or a particular behavior,
you can easily write a program which exercises that behavior
and verifies that each upgrade keeps that behavior.
Even though the kernel itself is written in C, you can write these tests in any programming language and even scripts.
Sometimes you just get, yeah, whatever, kernel is just too critical.
Let's have more approvals before we deploy.
Regular software requires one approval and the Linux kernel should require two or three approvals.
And again, this is related to the fact that we perceive kernel as a, you know, like,
bad scary monster which can destroy the universe.
But what if I told you that kernel deploys are inherently safer than any other software?
Would you believe me?
Who believes?
You're in the matrix, yes.
We learned it the hard way actually in CloudFare.
So this is like a map of CloudFare data centers around the world.
It's maybe even outdated, but the gist is like, yeah, we have a lot of data centers around the world.
And with regular software, how do the updates happen?
So from a 1000 feet view perspective.
So engineers update the software package, push it to our registry package.
Registry then the config management picks it up, downloads a new package.
Also the config management may be configured to restart the service which uses the package.
It can be graceful or non-graceful depending on the context.
It doesn't matter.
But the gist is new, bad, or good code can propagate through all this network without proper safeguards in minutes.
And CloudFare learned it this hard way.
So we had several bad outages where we didn't have proper safeguards for stage rollouts of some software.
So we almost caused global wide network outages and these are described in these blog posts.
On the contrary, how does Linux kernel upgrade works?
The gist is it requires a reboot.
So we, and to reboot the server, what we do is we drain traffic from the server, put it out of production, actually reboot.
Then it comes up, it contacts our config management, we wait for it to be reconfigured.
We run some basic acceptance tests and put back the servers into production.
And I mean we would be crazy if we reboot everything at once, so we don't.
So we have automation rebooting servers in one by one or in batches.
So what it means is kind of, it's an inherently natural, slow-paced, gradual rollout with minimal impact.
If things go wrong.
Did we release kernels with bugs?
Yes.
But yes, some servers didn't come up properly.
Some servers started showing errors and there were only a couple of servers.
So we like reverted the release and like there was no visible impact.
One problem is why people are afraid of running kernel releases is they don't understand them.
How the kernel release process works.
So kernel versions are designated by three numbers, like one number dot, another number dot, and then another dot.
Example, like 6.132.
Who here knows about semantic versioning?
Almost everyone.
So the gist of this talk is this is not a semantic versioning system.
Everyone confuses this with a semantic versioning and it's not.
But instead, what really is the first two numbers mean the major version, not major or minor as in semantic versioning.
And the right most number means bug and security fixes.
And when the right most number increments, you most always never get new features or major subsystem rewrites.
So it's not only bug fixes or security vulnerabilities, nothing else, no new functionality.
So how do these releases created?
So the main bleeding edge we call it source code is stored in a git repository managed by this person.
Who knows this person?
We call him benelope dictator, right?
So, yeah.
The features are developed in branches, subsystem branches.
So for example, you have subsystem for drivers, memory management, and that.
And once in a while Linus pulls changes from these branches.
This is where the pull request probably came from.
I don't know, I'll note that for that.
But the original pull request, not like fancy PRs that we have now, but it was an email saying,
hey Linus, can you pull from my branch?
This was a pull request.
And it still is actually in the Linux kernel.
Yeah, so Linus pulls all these changes from subsystem branches.
And once in a while, he branches out the main branch into stable branches,
which designate a major stable kernel release.
And this happens roughly every nine to 10 weeks.
Eventually, when bug fixes get accumulated, you get a tagged version on a stable branch,
which indicates a bug fix release.
So for example, you get 6.2.1.
But how these bug fixes get propagated there?
So they're not, if you have a bug, you do not submit a fix directly to a stable branch.
Instead, you actually have to go through a respective subsystem maintainer
to ensure this bug is not only fixed in the stable branch, but in the main branch and all other branches.
So you actually commit your bug fix to the particular subsystem where the bug is,
which will eventually get propagated to the main branch.
But once it's in the main branch, it's not just merged into the stable branch.
These bug fixes commit, especially mark, and the maintainers for the stable branches,
the stable branches all have maintainers, they basically cherry-pick these bug fixes.
And when enough bug fixes are getting accumulated, they do another bug fix release,
which happens roughly every week.
So yeah, a new major stable kernel is released every nine to 10 weeks,
and it's the so-called merged window where new features get merged.
There are only two weeks of the merged window usually.
And the rest seven weeks are for testing in bug fix.
And so even the major version receives a lot of bug fix in testing in the first place.
And what you have to remember is leftmost version means nothing.
So in Galway we had this problem where we, at some point, when we upgraded for 4.9 to 4.20, it was fine.
But when we wanted to upgrade to 4.20 to 5.0, people were like,
oh, it's the leftmost major version of this.
It's probably really scary.
No, it's not.
It can even have less features than the previous major release.
Linus himself tells that he just increments the leftmost number when he runs out of fingers on his hands and toes.
But for whatever reason, sometimes he increments when the middle number is 19,
sometimes it's 21, and sometimes it's 20.
So apparently he has a variable number of fingers.
Yeah, and bug fix or patch releases are released almost around once a week.
They are denoted by the rightmost version number.
They're usually cherry-picked from the main Linux branch.
And the rule is there's always no new features.
Therefore, regressions are quite rare.
They almost always contain critical security patches, and you almost always want to apply it.
Well, the problem with major kernel upgrades is that the major stable branch is kept alive around two, three months,
and then it's abandoned.
It's declared end of life, and no new bug fixes and security patches are backported there.
And the assumption that at this point you will have a new stable merger version available,
and you should just upgrade the merger version.
But sometimes it's very costly to evaluate the major version because you do get new features and potential regressions.
For this, there are so-called long-term stable releases where bugs and security features are backported for at least two years,
and it's usually the last stable release of the year.
Therefore, the so-called LTS stable release is released once a year,
and if you follow these, which we do, for example, it provides you enough time for more rigid evaluation of the next long-term release.
And surprisingly, the releases are quite well described on the kernel.org website slash releases.
I was surprised how many people don't go beyond the main page of kernel.org to read stuff.
So yeah, go and read it. It's quite interesting.
Okay, so what do we do for safe and easy production kernel upgrades?
First, don't create a dedicated deploy procedure for the Linux kernel,
because kernel upgrades are usually less risky than other software who's been convinced today.
Well, some hands, okay.
A simple stage rollout is usually enough, and kernel upgrades are naturally slow paced because they require a reboot.
And because you probably won't reboot everything at once, there is a lot of headroom to abort the deploy if things look wrong.
Do avoid justifying bug fix kernel upgrades.
Apply them with no questions asked.
There is almost always something that is applicable to your workload,
and it contains only bug fixes and security vulnerabilities only.
And also minimize cannery soak times and prefer to use metrics-diven approach.
You can sit in this 30-day window of operating your production kernel everywhere.
So if you require high soak time, think about it.
What metrics or observability will give you more confidence to roll out this kernel faster?
Stay on the long-term branch if validating a major version is costly,
so you have to do a lot of analysis and testing.
You get at least two years of bug fixes and security patches,
but don't wait for the two years, of course.
Better what we do, for example, we start evaluating the next long-term release early in one year when it's available.
Again, apart from just being proactive, it gives us more features early
and sometimes, most of the times, better performance and resource utilization.
And we also don't accumulate too much change delta, as I described before.
If you don't have it, implement and improve production testing for major version validation.
Basically, faith-lab grading the kernel requires you to understand what your workload is.
If you're a web server or a database, what specific subsystems are in the target of your workload?
Because sometimes, even a bug or an improvement in CPU does not apply to databases.
Once you understand your workload, better to write tests which exercise these kernel subsystems and interests required by your workload.
Having these tests also really helps with communicating issues to the upstream community,
because in Cloud for All, our team is quite small and we're not experts in anything,
and I would highly doubt if anyone really experienced in the Linux kernel, including Linus himself,
could be an expert in all the kernel subsystems.
So sometimes, we had a time where we had a bug in KVM, and we know nothing about KVM at that point,
but because we had a reproducible test which triggers the bug, we spent like two weeks trying to understand what's going on,
and we couldn't, but since we had a reproducer, we just posted an upstream mailing list,
and there's always a person saying, oh yeah, here's a fix in 10 minutes,
but you have to create this reproducible self-contained test to actually people to help you.
And yeah, make metric-driven decisions whether to upgrade and not time-based decisions, so many might sometimes.
One thing also helps with metrics and monitoring, and also automating your kernel releases,
is with human risk perception, because sometimes when new people join your team,
they still have this mentality of Linux kernel upgrades are very risky,
and if you require a human to progress and do these upgrades, they will always be reluctant to do this.
Like, automation really helps here to remove the human risk-perseverment factor,
because these days, especially in clover, many teams are not even aware the kernel upgrades are happening.
They're like happening under the hood automatically, and people don't notice it,
just because, and you don't have to ask anyone whether you should upgrade,
because you have this more or less, not perfect, but more or less data-driven approach.
And I think that's it, whatever I want to talk to you today.
So again, Linux kernel upgrades are not more risky than any other software.
You need to patch early and patch often, and your bug fixes kernel releases should be applied with no question asked,
and understanding your workload, metrics, and monitoring on automation will allow your system to stay patched and secure in the long run.
Thank you very much.
May I ask something? I know where that fear, it's a fear that we all have, I guess,
and it comes from things that I can just say one story, so you have like a 5.4, and it's working fine,
and you have some kind of special, maybe, chipset, and it doesn't support everything that chipset can offer, but it runs fine.
So you upgrade to 5.0 something or 6, and it starts to crash.
And then you roll back, and then you next time you will really think twice if you will upgrade to the next version,
which will offer you more support for that chipset, but you still don't know.
Then you wait others to upgrade, and to be sure that it's working fine now,
and that's why you don't run to upgrade really fast, and then let me see if my dead one and that one did it, and it's running fine,
and then it builds fear, you know, these things build fear, that's what can build fear,
that's why it's always good to wait a bit more until 5 of them do it, and then, okay, I can see, so when I'm running fine now, I will do it now.
Well, I mean, based on our experience, I have this same question from our production engineering team many times,
it's like, why do we rush to upgrade? Why don't we wait until all the bugs were fixed and we can upgrade?
And I guess it depends on your workload, but for us specifically, I sometimes call CloudFer as Linux as a service,
because many of our products are using Linux, are stretching the Linux kernel to its edge.
If there is a new feature like, and Linux kernel like XDP, IOU ring, people jump on it and adopt it almost immediately,
and the result of that, because we use these edgy features which many people don't use, there is no one for us to fix these bugs,
like we're hitting them first, so we tried waiting, and when we're waiting, we're still hitting the bugs,
because like nobody else is using that feature in this way, and this is where you just can't, I guess it's the same with very specialized CPU or hardware,
if nobody uses this hardware, you can't wait for the community, someone else to fix your bugs, you have to push them through.
Of course, you see the bugs, it's always helpful to report them, and there will be some people on the mailing list within a moment,
they will send you a one-liner patch to try it out, and usually it works out, but I mean, generally, if your workload is specific enough,
or hardware is specific enough, you can't just wait for all the bugs to be fixed, because it's very applicable only to you.
Okay, good day.
I wanted just to in phase your position to say Linux is safer to upgrade over any other software,
and to me the main reason is because of the strong commitment from this community to ensure that all the stable release are safe to upgrade.
And I know very few other software that takes this contract with the users to say you cannot grab safely.
And I think this is a major point, and I think the Linux community should be recognized for this, because it puts a lot of work to ensure that we are safe to upgrade.
That's something very important.
More than the rollout points you are leveraging, it's much more because I've searched strong contracts to ensure that every stable release is safe to be used.
Yes, you mean you're referring to don't break user space mentality?
Or even don't take a patch which is not already in mainline.
I mean, if you get your patch into the stability, it's because it has been tested and proved to be safe,
and because of the sum of all these patches is not to be safe.
And this strong commitment is very important, I think, for the users.
Yes, yes.
They can press their work.
Yes, yes, yes.
And many times when you submit patches, there are tons of external, even people or systems,
we run your patch in kind of a CI and they will report if there is something back.
Yes, I guess you're right that we have to acknowledge that community puts a lot of effort to these stable releases to be actually stable.
But also, like the release process itself goes a long way.
So, technically, again, you have only two weeks to deploy new features and then you're stuck with seven weeks of bug fixing.
So, yes, the emphasis on stability is a real win, I guess, for this community.
And another thing, the sum of security issues is not only counting the CVs.
Greg made a great presentation around that.
If there is CVs, there's probably a security issue.
But there are also fixes which are not as stacked as CVs, which could be our security issues.
So, to evaluate the security risk of a given version, it's not only counting the CVs, it's much more complex than that.
Yeah, I agree with that. And this is what I partly mentioned, that data is crowdsourced and probably incomplete.
It's kind of like the minimal baseline of risk.
But there is more, of course.
There is like, these are publicly non-vulnerabilities which have been tagged on this project.
There is like a lot of them which are intact with no CVs attached, as well as like a lot of unknown security vulnerabilities hiding in this system.
So, yeah, definitely.
Anyone?
Hi.
Here.
I don't see.
I'm here.
Oh, okay.
Hi.
I have a question about Livepatch.
Do you use in your company?
Livepatch, we don't use Livepatch.
And my personal view on this, I'm not...
So, like, I don't fully see Livepatch technology covering all the use cases.
So, I think it is useful for patching vulnerabilities really fast.
Yeah, yeah.
But on the particular type of vulnerability.
Yes, yes.
With Livepatch, you're basically replacing a piece of code in the kernel with another patch piece of code in the kernel.
But we have to remember that in kernel, kernel API is not stable.
And basically, you can only do that if your patching requires not changing some kind of structure.
It may fall apart if you're required to adding a mutics into the structure if you have a race condition.
And this is where Livepatch fails.
And moreover, implementing Livepatch is very complicated.
And it's kind of like you can crash the system as well because you're messing with the kernel code.
So, in my perspective, in my opinion, the effort is kind of not worse of the return of investment.
Like, if you don't have any company, like a Linux enterprise, Linux distro doing it for you and doing it for yourself,
you're putting a lot of effort to make it.
You can't patch all the security vulnerabilities with that.
You're putting a lot of effort and you don't get much benefit.
If you instead just focus on building a system where you can reboot anything at any time,
that kind of gives you, like, much better, like, long-term result.
Because you just can't reboot with a new kernel and, you know, your system kind of is resilient to that.
And it takes as much effort.
Thank you.
Hello.
Thanks for your detailed explanations and for outlining that the December version doesn't actually work the way we think it does.
Now, I have questions.
So, you mentioned that we usually install the rest of the software out of some side bound that we don't have control over.
And actually, I do that for everything.
Can you kernel?
I don't usually compile it myself.
So, the question is, can we, should we aware, should we be aware of particular tricks?
Because this process is actually mediated by the distribution.
Like, do the people who do the distributions know all the stuff you mentioned?
Yes.
And actually, the model which I described following LTS release and, like, rolling out bug releases regularly is what most distributions actually do.
You might not see it because, for example, Debian, you kind of, they version the package differently.
So, you think you're always on the same version, but you may notice if you're doing, like, regular up-get upgrade that when your new Linux kernel is installed, it actually installs you a new bug fix version, which is hidden under the hood.
So, this is what most distributions do.
They either follow LTS or they take a non-LTS branch and maintain it for longer.
But when you upgrade your system, you just get bug fixes and security vulnerabilities patched as this bug fix release.
Hello.
I'm not completely sure how the kernel process works still.
How about a firmware that's just dropped into the kernel?
Is that included in those bug fixes?
And if so, how are data set?
How are you ensuring that those binary blobs don't change something that breaks everything?
So, in modern distribution, and like within the Linux kernel upstream as well, the binary blobs are now managed separately.
They're managing the separate git repository.
And on distributions, there is a separate package for it usually called Linux firmware.
So, basically, the code for the kernel and the binary blobs are upgraded at different cadence and have different release procedures.
So, they are not included in the code upgrade these days.
Hi.
Over here.
Yeah.
So, you were talking about the fear in upgrading kernels, but to me or when I'm looking at my team, sometimes it's more of the tedious task in having to reboot or to migrate the service.
And then, you know, doing it over and over like Groundhog Day.
Now, my question is, what would you consider a reasonable cadence for that task?
Or do you see even like a need at the system to align on a specific kernel and, you know, and zeroing out the whole system or just having some routine monthly maintenance that jumps a few versions?
What's your take on that?
So, again, for bug and security releases, my preferred kernels is weekly.
So, they released every week.
You have to compile it and roll.
I mean, not roll it out everywhere, but start its rollout at some set of production then more and more and more.
And again, basically, the more you delay, the more change delta you accumulate, the more risky you're bringing.
So, if you do it as regularly as possible, your change delta is small.
And technically, like within a couple of two bug fixes, even if it's something breaks for your particular service, you can kind of bisect it and understand what's happening much more easily than you have to go through, you know, like, thousand and thousand of commits.
So, if it's hard, you have to think about how to make it easier and how to do it more often.
The more often you, it's like gym, the more often you do it, you kind of build that muscle, you build the tooling around it, you build the metrics and observability around it, and then you build, eventually, you build your confidence that kind of, it takes you very fast and effortless to actually do it much, much more.
Yeah, my question is mainly about the time spent.
My question is mainly about the time that you spend, you know, managing that as part of your day-to-day.
Well, again, it's basic calculation of return on investment, right?
If a kernel upgrade is too costly in terms of like spending, you're spending a lot of time doing that, think about if you can invest this time to build some kind of automation.
And that's what we basically did.
Like, when I joined the company eight years ago, like, it was very manual and time-consuming and it required a huge team of SREs to actually do a kernel upgrade, but now they're not even involved anymore.
And, like, it just happens.
Thank you for the interesting talk and nice present for you.
Thank you.
Enjoy it.
Thank you very much.
Thank you.