All right, everyone. Welcome to the next session. Just the usual housekeeping. If you're leaving a little bit early, these chairs are fairly long, so don't do exactly that.
There's going to be some good sessions here, and we'll have some time for questions at the end. So let's get started.
Right. Thank you. Hi, everyone. I'm super excited to be here with you today to talk about FlatCut, to talk about releasing Linux based OS.
And I hope you will learn new things. I hope you discover things. And yeah, if you have any questions, I'll be around for the rest of the day.
And I'll be available at the end of this presentation to answer your questions.
Before going further, I will quickly introduce myself. So my name is Mathieu. I work as a software engineer inside Microsoft.
I'm mainly involved and principally involved in the FlatCut development and every features regarding FlatCut.
So for example, I'm involved in the cluster API fields. I'm involved in testing the operating system, building the operating system.
And what's matter today is releasing the operating system. If you are here at this talk, I assume it's because you have maybe some knowledge about FlatCut.
You're already a user of FlatCut. You just want to discover and you're curious about this operating system.
So let's have a quick look of what is FlatCut. So FlatCut is a Linux based operating system. It's designed to run containers.
So you only have the bare minimum in your operating system to run containers.
The goal is to have the less package you ship in the operating system, the less surface attack you have on your operating system.
So that's the point of having this. This operating system benefits from automatic updates, which means once you've deployed your instance of FlatCut,
it will get automatic updates from the release server and each release is done every two weeks approximately.
So you can be sure to have a new version of FlatCut every two weeks.
And finally, this system is immutable, which means SlashUSR is in read-only mode. You can't write anything in SlashUSR.
You can't install any package. There is no package manager. There is no APT or whatever.
So that's a few difference from a day-to-day operating system.
FlatCut is already designed to run containers and nothing more.
So the question is, well, just to show you inside the box, so I tried to write something on SlashUSR.
It doesn't work because it's read-only. That's normal, even in pseudo.
And try to use the package manager, the command.phone for each one of them, because that's the goal.
The idea is that you have to trust the maintainers and what they ship inside the US.
And if you need something more, you can ask yourself or you can ask to the maintainers or the communities,
or you can try to find a way to install these packages.
How do we maintain the system? Because you can't update yourself the package.
You can't install any package, so you have to trust the maintainers and the community.
So on GitHub, so this is the QR code on the GitHub repository, which leads to this list of packages.
Basically, we are security-driven, which means each time there is a new CVE, a new issue with one of the packages shipped by FlatCut,
we track it into this repository and we update the package.
So for example, last week we've got the RunC and Docker CVE that has been made public last week.
So it's already tracked, and when we will release the next FlatCut, so I hope this week,
you should get this update closed.
So the packages are updated by the security-driven base and also by a community-driven base,
which means if one of you wants to add a new package to FlatCut, you can just open an issue.
Hey, I'd like to have this package or this package into FlatCut.
Is that possible? And if it's relevant for the community, if people are okay to have this new package into FlatCut,
there is some chance that this package is being included in the next FlatCut release.
Most of the time we try to challenge people to say, can you use Docker Image instead of using this package,
or can you use just the binary that you will download from the boot of the instance to get your software.
So we try to challenge always in the same goal is to have the less package into the operating system,
because the less package you have, the less vulnerabilities you have in your operating system.
If you want to know what's going on in the next FlatCut release, you can join the Office Hours,
so this is done publicly every month. The next one is on February, and during the Office Hours,
we just check the FlatCut release boards and we check which new package will be included in the next release,
so you can give your opinion, you can give your input of which package should be prioritized or not for the next release.
That's always a great time to discuss between the maintainers and the community about the content of the next release.
So that's the release board that's available and public on GitHub.
And of course we ship new packages and package updates, but also the bug fixes and new changes and new features into the operating system.
Now we are ready to release, but before releasing, I would like to demystify a bit the FlatCut release number,
because this is something we've seen quite some time that people are getting confused with the FlatCut version number.
So this is a FlatCut version number, and the idea is like Semver versioning, but not really.
For example, the first digit, the 3760, it's the days since the first CoreOS release,
because FlatCut is a friendly fork of CoreOS initially, so 3,000 days, it's almost 10 years, so it was 10 years since the last release.
Then the second digit is the promotion level, so are we talking about alpha release, beta release, table release or LTS?
And finally we have the patch or the maintenance level, which is the last digit.
So if you have a zero, it means it's a new major release because there is no patch yet done for this release number.
So based on this, we can play a small game and try to identify which is who is who.
So for example, the first 37602.0 is a new major stable, because you have the zero at the end, which means it's a major release.
We have the two, which means stable, and we have the first digit that is just showed how many days since the first CoreOS release.
But based on this, who is able to find what is the third, so 3,850.0.0.0, what is it?
Is it an alpha release?
Yeah.
Is it a patch release?
No.
No, so it's a new major alpha.
And the last one, so with all the freeze, 3,0,3,3.3.18, what does it mean?
LTS.
LTS, yeah.
And it's really old LTS because there is a bunch of patch releases.
So patch releases means basically kernel updates.
Each time there is a kernel update for the LTS, for example, we just update the kernel, the CA certifications, and critical security issues like OpenSSL, but that's it.
But yeah, for the LTS, most of the time it's just kernel patch release, so that's why you have a big number for the LTS.
So I mentioned alpha, beta, stable.
How does it look like across the time?
So we have a new major that is done every two weeks.
Then from time to time, we decide to promote that alpha to a beta.
So that's why we have one that happens to this example.
And then after a few times, this beta version becomes a stable one.
And eventually it will become an LTS one.
So that's quite interesting because you, as a user, if you run stable flat-card release, it means it has already been in beta a few months before landing into stable.
So that's why also we encourage people to run beta nodes into their workloads, like so they can identify if there are any issues with their workload before it gets into stable.
Yeah, so that's the release cycle.
Now, what's the release process and how does it work?
So most of the time it's done in four days.
We never release on a Friday because it's a well-known rule that we don't want to break at flat-card neither.
But basically on Monday, we start to build the new flat-card releases.
So we kick off the builds for the new alpha, new beta, new stable, and normally the new LTS.
So this is done on Monday.
And on Tuesday, we check the status of the builds.
Is the CI OK?
Is the image been successfully?
So yeah, we have a checklist of things to see and to check.
And we start drafting and preparing the release nodes because that's quite important when you have a new release.
It's to communicate to people that there is something new and that would be interesting to know what's inside this new release.
So yeah, on Tuesday, we start drafting the release node and Wednesday, we have the go-no-go meeting.
So this is a meeting done publicly on Matrix Channel where we discuss about should we actually go forward with the release.
Are we in a good shape of a release and can we move forward?
So this is the go-no-go meeting.
So basically, we just check is everything green in the CI.
Are the release nodes correctly prepared and everything is good on the CI?
And yeah, we decide to go or not to go with the release.
Then we have the actual release, which means we are going to take the new images to publish them on the release servers of flat-card
and to generate the new payload because as I said, flat-card is going to get automatic updates.
So we need to generate the payloads to get them downloaded by the current running instance.
And then we have the announcement.
As I said, it's important to communicate to people that there is a new flat-card release.
And on Thursday, we have the marketplace release because flat-card is supported on multiple vendors.
So we have the AWS, GCP, Azure.
So we want to publish flat-card images on this marketplace.
If we check the release process for Monday, so one of the flat-card engineers will start the build and he will publish the links.
Then on Tuesday, we start to preparing the release nodes.
So this is, for example, for the last table and there is some nodes.
For example, there is Flakitest with Calico C&I on Digital Ocean.
So we try to identify is it our fault because of the test framework?
Is it something really critical?
So sometimes we have to stop the release because we have an issue with the new kernel that has been identified with the test framework.
So this is the kind of nodes we can take during the release process.
And after that, we have the Go No Go meeting.
As I said, it's done on metrics and everyone, contributors and maintainers are invited to say Go or No Go for the release.
So it's a ping into all channels across all numbers.
And yeah, so people can give feedback on the release status and we decide to move forward with the release.
And when it's done, we actually have the release.
It's available. It's on the public website and we communicate on Slack, on metrics, on Mastodon.
But there is a new release available and please update and give feedback on the release.
And finally, we have the Marketplace update.
So this is an example with AWS update on the Marketplace.
So what's interesting with this process is that the community is involved at each point and always.
So nothing is done in secret or whatever.
Every time you can give your input, every time you can see what's the status of the release,
are we close to have the release to be done or are we far to have the release to be done?
So for example, the checklist of all the best items are on public GitHub issues.
So you can easily see where are we during the release process.
The release notes are drafted on a HackMD document.
So you can browse the release notes and start to write and send some comments on it.
And the public discussion are always on metrics.
So regarding flat-car release process, it's always on metrics,
but also for the public discussion of flat-car development.
So every decision regarding flat-car is done publicly on metrics.
So there is no, as I said, secret discussion.
The only thing that is still secret is the build for now
because we still have some credentials for the various cloud providers.
So ideally, we would like to have it in writtenly on Jenkins,
and people can just see the logs of the build and see how things are going on.
But it's not done yet.
What we've done is that now if you open a pre-request against flat-car repository,
it will start the build on GitHub action.
So you can see your logs and you can see if something goes wrong, if the CI is OK.
So it just build a QMU image and run the test on the QMU image.
But for the release itself, it still relies on Jenkins,
but eventually, we'd go public using GitHub actions.
And I think that we'll close the talk.
So if you have any questions, I'll be around with some flat-car teams, remember,
for the end of the day.
And thanks for your attention for this Sunday afternoon session.
All right. Nice break-up question. Great.
What's the elevator pitch for using flat-car above Fedora's offering or micro-OS from Sousa?
Well, micro-OS and over-operating systems are quite similar,
but flat-car has some multi-vendors, for example,
or you can use it on premise, on bare metal, on different cloud providers.
Also, you have new features that we try to merge into the flat-car operating system,
for example, system DCS-X or other things that we try to leverage.
And also, there is this, we try to do things upstream first.
We got this talk about upstream versus downstream before that,
but that's the idea. We try to go upstream first.
So each time there is a new feature, we try to first implement it upstream
before trying to solve it downstream.
So we try to be more on the community side
and try to fix the things on the upstream and not really on the downstream.
And then speaking of fundamental differences, for example, with micro-OS,
you don't have the same mechanism to provision the instance.
For example, with flat-car, we use intensively ignition or afterburn,
which is not yet available or experimental on micro-OS.
So this is the kind of difference you can see.
And if I recall correctly, I'm not sure if micro-OS is using REST 3,
but this is the kind of functional features.
You can see the difference, but in Vienna, it's the same purpose of operating system
is just to give the user an operating system to run containers.
That's it.
So as it's open source, you have the choice of which solution you want to use.
Thanks, Moomo.
Feel free to comment this.
How much has changed or has it been noticeable since Microsoft took over or the acquisition?
Thanks for asking the question.
So for the short story, Flatcar was developed initially by Kinfolk,
which was a company that has been acquired by Microsoft two or three years ago.
And I'd say it didn't change a thing for now for the development.
The governance has always been on the community side, community-driven.
And I'd say it's even better in a way because now we can totally be dedicated
to this operating system and to the support of the operating system.
And recently, like a few months ago, six months, something like that,
we started to look into a CNCF incubation.
So basically, we would like to have Flatcar find a new home at CNCF.
So there is an open issue on the CNCF tracker so you can see the status of the incubation proposal.
But in terms of governance, nothing's changed,
and we're still dedicated for users to get the best Flatcar experience on any cloud providers.
Thank you.
Over the question?
Yeah.
Matthew, thanks for the talk and for the distribution, the idea, everything.
So I'm not familiar with the project, so I'm attending just to understand what's going on.
So everything is a container, right?
All tools and everything are running as a container.
But I'm curious how the kernel is booted or the NITRZ is done.
So, yes, that part is a container or not?
I don't think so.
So Flatcar is not running inside a container.
Flatcar is an operating system, like Ubuntu, like Debian, like whatever.
It's designed to run container work loads.
So you have the very minimum to run container work loads.
You have a container one time, you have the kernel modules that face well.
So in the end, it's like any over Linux distribution.
You have your kernel, you have the boot process, you have the NITROM FS, and then after you have the user space.
Yeah, so if I understand correctly, the stuff that's supposed to be previously managed in containers,
in traditional packages, is now containers, right?
But if a new version of a kernel is released, then how that's distributed, let's say.
So if you have a new version of a software, how it's distributed on the operating system, that's the question.
Yeah, you just wait for the new Flatcar release, because it's immutable.
So if there is a new open SSL version, for example, you have to wait for the next Flatcar release to ship that new open SSL version.
Like so you get the update.
So that's like pulled from the Internet. It's not like in a format of a package, right?
Sorry, come again?
Is that in a format of a package of its pulled from the Internet straight?
Yeah, it's not in the form, Flatcar is based on GN2 Linux.
So the Flatcar itself, when you build it, you take the source from the value repository using GN2 mechanism, then you build the package.
Then once the package is built, it is included into the image, which is the new Flatcar.
Then the new Flatcar is released, and this is how you benefit from the software update.
Okay.
So I think my question is also, let's say not only technical, but more on the political side.
So history is that this is a folk of CoroS, where Ken Falk started this, right?
Then it was brought by Microsoft, but Microsoft has its own, the CBL Mariner.
So how does this fit and what is, let's say, this is a cloud based, let's say, the essential client that you have.
That this OS has to be used in cloud, right?
Yeah, thanks for the question.
So CBL Mariner is dedicated to run on Azure, while Flatcar is dedicated to run everywhere.
And it's not monetary to run Flatcar on a cloud provider.
So as I said, you can run your own Flatcar image on Raspberry Pi, on ARM64 at home if you want to.
Or if you have your own, I don't know, Proxmox, we have some people that use Proxmox to run Flatcar.
So yeah, Flatcar is really multi vendors and multi architectures.
And so while CBL Mariner is really dedicated to Azure and nothing else at the moment.
Hi.
So in my previous role, we used Flatcar quite a bit for a while.
But then we ran into kind of some trouble with AI and especially around things like we're trying to use like Infiniband.
And we were actually kind of running into, I think, getting everything set up with Flatcar.
Are you all kind of working more towards like AI workloads and making those easier to run on Flatcar?
So I'm not at all AI expert.
Maybe Remy, behind you, is a Flatcar member.
I'm also a Flatcar maintainer and I've been looking at NVIDIA and GPU support in the past.
And we want to get better at that.
It would be great if someone from the community would also help because I have limited cycles.
But it's something I and we care about, right?
Just one last question.
No one else has any?
Do you support different container runtimes?
So at the moment, we only ship the current container D.
But basically, in a non-official way, you can use Podman using SystemDCX, which is a system D feature which allows you to mount overlay FS images on the base system.
So yeah, with Podman, we ship a system DCX in an unofficial way.
So you can just pull this Podman extension and load it on the system and have Podman up and running.
There is some tracking issue to have this out of the box, of course.
Like so you don't need to provisioning and to pull Podman's system extension.
But yeah, ideally, we should be able to say, OK, if you want container D and Docker,
or if you just want Podman, use this configuration and not this one.
But yeah, you can use Podman.
Actually, I did some experiments, there are things with it and it works.
Cool, thank you.
All right, I think we have time for one final question if someone's up for it.
All right, looks like there's no question. Thank you very much.
Thank you.
Thank you.
Thank you.