Thanks.
So today I'd like to talk a bit about zero as you touch less infrastructure for your container and Kubernetes workloads, most specifically about container optimized Linux.
And hi, I'm Tilo. You can find my GitHub or mustad on. I work for Microsoft. If you want to reach out to me, just drop me an email.
All right. So container optimized Linux is a very special way to look at Linux distros and a very special way to look at your whole application stack.
I'm going to introduce a few fundamental foundational concepts of that and then I'll touch a topic that is a little bit neurologic with operators with large fleets and that is how to keep the operating system up to date in a safe manner.
Also, we talk about compositability and that is like extending your operating system in a way that keeps your extensions from the operating system reasonably separated.
And lastly, if you want to reach out to the project, I'll show you a few ways to interact with the community.
But first of all, what is container optimized Linux? Do we have some special tweaks in the kernel or do we ship some user space bits that make containers run exceptionally well?
Well, yeah, that's one way of looking at it, but it is actually taking a step back and looking at your Linux distro, not as a general purpose distro that does all the things, but at a very special purpose operating system that operates one thing, that is container workloads.
You would imagine the operating system as not being anything special, just another piece in your application stack and make the operating system operatable like you operate your container workloads or your Kubernetes ports.
Have an image based operating system in containers and Kubernetes, you basically create your application as an instance of a pre-built application image.
And if you do all that, you can actually leverage the isolation that you get in container apps from the operating system side and there's some neat things you can do with that.
First of all, let's talk a bit about the user experience side of things. So general purpose operating systems do a very great job to give you like all of the diversity, all of the choice, all of the knobs and levers that you can have to tweak and twist your operating system and make it fit for your purpose.
The thing is for specialized workloads, you sometimes have to, even if you don't want to. So that's not how we perceive container optimized Linux instead, we're trying to focus on doing one thing and one thing very good.
So we measure ourselves not by the features we ship, not by like the options that you have, but we measure ourselves by basically our function or one thing that we provide to users.
A light switch can come like in 20 different colors, but you know if the green switch doesn't manage to turn on the light in 20% of the time, then there's kind of a disconnect between the designers of the light switch and the users.
And we don't want to be like that. Talking about provisioning, if you provision a container app or a Kubernetes pod, you basically take a declarative configuration that specifies only the business logic bits that you need to adapt the image to.
And then you apply that to a prebuilt application image and as a result, you will get an instance of that image with your config live in your cluster and that's your app.
If you take that idea and if you apply it on how you would provision your operating system, then you would have a declarative way to configure the bits and pieces that you absolutely have to that are not that are not saying defaults.
So your business logic of your notes, you would apply that to an operating system image and now would create an instance of that note which has the properties of the configuration you gave in.
You could of course add some bootstrapping code in order to bring up a control plane or if you have a single purpose app to bring up that app, but this is kind of where we draw the line from the operating system point of view.
So we don't want to ship another application control plane because there are great control planes out there.
Alright, so if you take all those principles and do a provisioning which I'll do with the QEMO instance here locally, first of all, this is the config I was talking about and you won't see any like OS specific twists in here.
This is purely my business logic.
So I'll define a user here.
I inline like a little bit of static HTML here.
I have specified a small image here that will be worked into the configuration and then I define a system D unit that basically brings up KD container with the config that I provided and that is my application.
So as is tradition, we compile YAML into JSON and this will basically take the image that I specified there and work into the configuration.
So now if I boot, this is a web script.
I just start the M and I pass this web jay and I generate it.
Now what will happen is this is the first time this virtual machine is provisioned.
So it applies the configuration that we added because this is how we set up nodes.
It takes a few seconds and then after this configuration has been applied, the system D unit will start up that we specified the web server for.
It will basically pull, I hope, do we have Wi-Fi?
I hope.
So I have Wi-Fi then.
Obviously, there's no way for this virtual machine to pull KD.
So what you can do though, my phone has Wi-Fi.
My phone has Wi-Fi.
My laptop does not have Wi-Fi.
Come again.
You'll fix better.
Or I can use.
So I'll shut this down.
Because I'm not sure how the virtual machine is dealing with changing Wi-Fi.
But I can try a hotspot.
This is the great thing with live demos.
Okay.
There's Wi-Fi that hopefully works.
And just reassure not again.
Here we go.
Again, for the virtual machine, this is the first boot because I interrupted it in its original first boot.
And so it'll set up the, all of the base system using same default.
And it will now that it has connection pull KD and then the web service will start.
Fortunately, KD is a pretty small image.
So we should, this is to see with a console, we should see output from Docker any second here.
That will, yeah, there is.
So you see that there wasn't anything on port 8080 before, but if I reload now, this is our static.
This will get very interesting with the update demo later because the update is significantly larger.
The basic idea is to give you an operating system that just behaves as your container application.
And then you just, you know, make your config, you throw it on your cluster.
It comes up the way you find it or it breaks while coming up.
So you'll see there's something wrong.
There's like one way or the other.
And this way by configuring it once at provisioning time, just like you do your container apps, you don't have any config drift.
Like if there's a note that you've been provisioned months ago, and if there's a new note that you provisioned based on the same config, those nodes will have the same configurations.
And you don't need to fill around the way I did here, supported on many cloud providers like we have specialized images for those cloud providers.
Terraform integration, go by means.
If you have existing automation, you want to do that programmatically.
And we have cluster API integration, more on that in a second.
There's this weird thing.
That's the difference between us and general purpose operating systems.
And that is once, configure once before the operating system is even deployed.
No SSHing into nodes and fiddling around with configs.
And kind of the mental image that we have to comparison is, if you apply a service YAML to your cluster, you do not cube cut a lex egg into that port, then fiddle around with config files and only then your app works.
That's a weird concept.
And we are trying to make that happen for operating systems.
All right.
So cluster API is a method to deploy Kubernetes clusters based on the management Kubernetes cluster flat car.
The project I'm working on is integrated in copy upstream.
We support a number of providers and they're currently pioneering kind of a new way to provision images or as images of cluster API.
Previously, those were built into the images and Kubernetes were built in the single image.
That was the golden cluster API image which you provisioned.
And then you had a problem because you couldn't update the operating system or Kubernetes independently because they were merged.
So we're working on a provisioning time composition, more on that later, based on system these are the eggs and we have a number of proof of concept providers implemented.
What's the benefits of having an image based operating system?
Well, you provision from immutable images, you're creating an instance just like you create an instance of your application with a container image.
Those images are always built from scratch and they are very easily testable this way.
And just like you have no config drift with provisioning time configuration, you have no version drift with image based provisioning.
That means that if you see a node and it runs operating system version A, then you have the full version set of all of the binaries shipped with the operating system just relating to that number.
There's no individual tools updating in their own pace and that means you don't really need to chase those versions, right?
In our case, everything is on a separate partition in the slash USR directory and that was the invariant you protected.
So if you fiddle with the bits in that partition, then it will break because the VN verity checks will be invalid.
It offers us the opportunity to do a partition update.
So if you're a database operator on a Kubernetes cluster and you want to retain local state, then that's good news for you.
Talking of which, I was mentioning the isolation of container applications to the operating system.
That makes your app portable. It makes your app run on many, many operating systems just basically making sure that the runtime exists.
You don't have any shared dependencies that we have an operating system.
And if you look at this from the OS point of view, then the only thing you're giving to your applications is a well-defined contract,
which is the container runtime on some kernel bits.
And there are no other relations to have to those applications.
And if you focus on that only, there's ways to thoroughly test that contract and make sure you never break.
You always uphold this contract.
If you look at the screenshot here, then you'll see our test suite running.
Every single one of these green buttons is a scenario test that we run.
We run those in our CIs for PRs to the operating system.
And this makes sure that this contract is always uphold.
Now, if you do that, it doesn't really matter for your applications, like which major version of the operating system you're running.
Up to the point where it doesn't really matter which Docker version or container D version or kernel version is there,
as long as you absolutely make sure that you uphold this contract to your application.
And you might have guessed by now that this layer, this contract, that's our light switch.
This gives us atomic in-place updates.
So what we can do with this is we can stage a new operating system in a separate partition,
and then we can switch over by means of a single reboot, atomically, like there's no intermediate state.
The application comes up, sanity checks run, and everything's great, except when the application meets some edge cases
where it has problems with the new release and in that case, you just roll back,
and you have a known good state that's still there, and you know that it works.
OK, so what I'm going to show now, I hope it works over my mobile connection,
is something that you usually will not see, because usually that's automated.
So with my configuration already deployed earlier, I disabled the update functionality,
and that's because I don't want my great web server being interrupted by updates.
But now that we demo updates, I can enable it, and I can check.
So the program I'm calling here is the update client-client, and just interfaces with the update client,
and it says that I am currently thrittling thumbs, and I have never checked for updates.
So we just kindly ask it to please check for updates.
Now, there's an update available.
The reason for that is I'm not using the current version, I'm using a previous alpha version,
and of course it'll find an update because we've published a new version in the team.
Now, it'll start downloading, and you see the progress that is basically percentages,
we're at 5% now, I'm not sure if it's going to make it to the end of the stock, but it is what it is.
While it's trying its best, let's talk about updating and rebooting.
For many people, reboots are scary, not really for us, and we're trying to make them not scary for everybody else.
But we understand that that's an interruption of your service, so there's a variety of ways that you can configure reboots.
The cheapest thing is to just have maintenance windows, and you configure that,
so your instances are only allowed to reboot in specific time windows,
but there are more advanced options available.
For instance, if you have a small number of nodes without a control plane,
you can make those nodes synchronize the update using a CD-LOX,
so only a certain number of nodes updates at any point in time.
Also, if you operate Kubernetes, there are a number of Kubernetes operators that you can use,
and this is really great on advanced stuff, so those operators will detect when an update has been staged,
it will drain, carefully drain the node, only after the node is drained, it will reboot the node,
and it will uncoordinate the node after reboot, so it's repopulated, your operating system is updated,
your cluster didn't notice a thing, and your applications are happy.
You can also, of course, stagger updates and make sure that only a certain amount of nodes are out at any given point in time.
This happens via a stateful update server, and our implementation of that is open source as well as part of the project,
you can just download and use it, it's actually pretty easy to operate.
We're using the Omaha protocol, which is the same protocol that Chromebooks use to update,
so it's kind of pretty solid protocol.
As I said, the update server is easy to self-host, so if you have a larger fleet,
and you want to have your own update procedures and your own update rollout process,
that's a straightforward way to get it.
If you don't, however, but you want to be part of the stabilization process that we do for our releases,
that's a very simple way to hope yourself in there.
Let me carefully check if there is a chance.
Oh, 63%, okay.
And then it stopped.
Anyways, maybe it will do it.
So, Flatcar releases in basically just like an application.
We split off the branch of a major release from a main branch,
and that will go through Alpha, Beta, and Stable stages.
Alpha is fully tested, but it may contain half-done features,
which are for development, so Alpha is for devs, basically.
Beta is also fully tested, and Beta is what we consider stable.
Stable has seen some user workloads, and this is where you come in, right?
So, if you operate larger clusters, it's absolutely safe to have a few Beta canneries in those clusters.
If those canneries detect any issues with your workloads, you just get back to us and yell at us.
You file a few issues against those Beta's, and we will never promote the Beta to Stable that has known issues.
So, this way, you're keeping the Stable nodes in your cluster safe.
And now you're supposed to demo Composability, and I think I skipped the Update demo, right?
No! I have six more minutes.
I'm not trying to get both demos in, but I think the Update demo is more amazing.
So, I just talk about Composability.
We're an image-based distribution.
We are immutable. We don't have any package management.
It's actually really hard to add something on the operating system level, and it's meant to be that way.
However, there may be things that you want to add.
Maybe Kubernetes, maybe Wazm, maybe an alternative container runtime like Podman.
And since we don't want to provide any options because that would make the lights switch weird,
there's this thing called SystemDISOSX, relatively new technology.
It's been around for three years now, and this thing called disease based on file system images that are immutable,
which kind of resonates strongly with us.
So, the way this works is you take your Sussex image, which only contains certain binaries when you ship for one application,
and then it's being merged at runtime or at boot time with your operating system tree,
and it results in basically a merge tree.
So, if you look at your operating system, then you'll see all of the binaries in there.
Building is easy as well.
You just have a local directory that resembles a root file system,
and then you run makeswashfs or makeoz on that local path,
and you end up with a file system image that you can use with Sussex.
Yes, the update's there, and I'm going to show you the update. It worked.
So, it now tells you that we need to reboot, right?
Before we do that, let's check out what we're running.
We're running an alpha version, 3815, clearly I'm a developer, because I run alpha,
and we're running the 6.1 kernel, and our application is happily running on this.
Let's reload again, and now I reboot.
Ah, Zudo will make me a sandwich.
And this will reboot into the new partition, where the new update is staged,
and there's kind of a, like, a settle time that the update tool will wait,
and after that time concluded, it'll mark the new partition as the default partition.
You always boot into the new partition.
If you have, like, important applications that absolutely need to run,
you would make the update service depend on those applications,
and if they don't start, the update service will never mark the new partition as the default partition.
And then you only need to reboot in order to fall back in the old, known good state.
All right. So, what did this buy us?
We've upgraded from 3815 to 3850,
and we upgraded from kernel 6.1 to kernel 6.6,
and our container is happy.
Didn't notice anything, which is the whole point.
All right. Image composability.
So, you take your SUSEX images, and you can either pre-bake them into existing operating system images out there,
and then you have, like, a combined image, and you can provision that,
or you can just use the declarative configuration that we provide to download those SUSEX provisioning time,
and then they're there. You can use system, the SUSEX update, to update those SUSEX independently of the operating system,
because if you have a Kubernetes SUSEX, there are no shared binaries of the operating system,
and you might want to update that in your own pace.
And with Kubernetes particularly and Cluster API, we have a need-proof concept running.
So, for the Cluster API phones, they have pre-baked images with operating system and Kubernetes in it.
It is very, very hard for them to update any OSPET.
So, what they do is they delete nodes, and they basically provision new nodes that has a new stack in it.
That's nice. It works for most workloads, but for some it doesn't, and there SUSEX comes in.
So, SUSEX allows you, and we demo that for the OpenStack provider, to just update your Kubernetes in place,
and, yeah, be happy about it.
We're trying to, you know, invest more in that work and get more Cluster API providers interested.
And this is the point where I need to skip the SUSEX demo for you.
It's basically just showing you a wasn't-time binary appearing and disappearing after I merged the SUSEX and unmatched the SUSEX,
but I have one and a half minutes left.
So, I'm going to skip and continue with how you can work with us.
If you're interested, there are a few flyers out there with a few QR codes on them.
Just grab one when you go out, and you have a few pointers how to interact with us.
We are very active on metrics and Slack. We're basically there all of the time.
You will always find and maintain either on our metrics channel or on a Slack channel.
We have Office OS that are user-centric. If you have a question about FlatCard,
if you want to discuss something and you are a user, every second Tuesday of the month at 3.30 pm UTC.
There's a DevSync. If you're interested in developing with FlatCard
or if you just want to see how we plan our work, the day-to-day stuff, how we plan roadmap and how we discuss security stuff,
that's the sync to join. All of our work is public.
All of our planning is public. Roadmap is not public.
And all of those are on GitHub. Those are GitHub boards.
You can just review them and you see what we're up to.
If you want to contribute or join as a developer, we're trying to make this very low bar.
So we have a software development kit that is a container because obviously it is.
And it just takes those seven steps. You just run those seven steps and you will fully rebuild that release from scratch
and you will fully run all of our scenarios locally on a very small QEMO cluster.
And it's equally easy to make modifications and to build your own images.
To just wrap things up, and we have five minutes left for questions after that,
so I skipped the SysX demo for a good reason to be open for your questions.
If you leverage the isolation that container runtimes, container ups have inherently,
and you look at this from the operating system point of view,
there's a lot to win in terms of interchangeability and replaceability.
If you use the declarative configuration before provisioning time and focus on the business logic
for configuring your instances and to abstract all of the unnecessary complexities
without taking away the option to fiddle with the bits,
but with taking away the requirement of absolutely having to in order to get to something workable,
you eliminate config drift.
Atomic automated updates gives you that kind of contract,
that abstraction that you get with container runtimes.
And if you still need to change something on the operating system layer that absolutely cannot go into a container,
you can compose it in the SysX.
Fully community driven, we've submitted the project to the CNCF as an incubating project.
The process is ongoing.
Keep your fingers crossed.
And that's it.
Thank you.
Any questions?
Thank you.
A lot of, a lot of please.
How would you run this in production?
I don't think I got that.
How do you run debugging tools?
The obvious first answer is you shouldn't because they are debugging tools
and they just, you know, open your nodes wide up to all kinds of strangenesses.
But if you absolutely have to, there's a development container that we have for flat car.
That is usually used in automation, like when you, at provisioning time, or reboot,
if you need to build some special kernel modules, then you would use that.
So if you absolutely have to run DevTools, that's the way. You can do it.
What's the DevTools in a container?
They already are. We are shipping them with all of the releases.
So that's the thing.
You speak very, very loud. The mic is...
If there's a standard for...
Yes, there are multiple.
They're actually pretty well defined.
Where is the... You should probably ask the person a little bit behind you.
I've seen it and I know it's smiling there.
But anyways, so if you check out the dev.com,
you can see the dev.com, you can see the dev.com, you can see the dev.com,
if you check out uapi-group.org,
then you'll find, like this is pretty dry, but you'll find
some specifications on, like, how those images are supposed to work.
And there's a number. You could use WashFest images.
You could even use, like, complete file system images with multiple
runtimes for multiple architectures in them. And it's all specified here.
If you want to hands-on examples,
we have in the FlacCa project,
we have something we call the SusEx Bakery.
And that... Did it load? Yeah.
So that has some hands-on examples on deploying your custom docker,
on deploying wasn'time,
on deploying Kubernetes, and also on making itself updateable.
Sorry?
Sorry, I need to ask you this.
Reproducible builds are something that are
pretty much on our radar and our roadmap.
We even produce, as a build artifact,
salsa provenance in the 0.2 revision, I believe.
So there's initial work there.
You can use that to basically build on, but there are a number of things that we need to solve,
like, you know, the right compiler usage and so on.
So FlacCa is actually making heavy use of Gentoo in the back end.
So if you use that for Gentoo, you can reuse it on FlacCa. It's very easy.