Yeah, welcome to this talk about how we build and maintain Kairis.
Let me just quickly introduce myself.
My name is Mauro Morales.
I'm originally from Guatemala, but now living in Belgium, really like it here.
My just some random thing about me, the first destroyer I used was Memphis.
I really liked it when I was going to university and never went away from Linux.
My current daily driver is open Susie Tumbleweed.
I am a Ruby and Go developer and there's some places you can reach out if you have questions
about the talk or anything afterwards.
I want to talk how we build Kairis, but for that I need to tell you what Kairis is first.
If you go to our website, you will see that we sell or advertise ourselves as the immutable
Linux meta-distribution for edge communities.
That sounds like a handful, so let me dive a little bit deeper into that.
What is edge computing?
There is a trend right now into moving the operational aspects from the data center outside.
What does that mean?
That you might have nodes in certain places, let's say that you have a grocery store producing
a lot of information about all the sales that are happening, all the different products,
and instead of sending all the bulk of data up to the server and do some processing just
to send it back to all those nodes, what you do is that you do the heavy processing already
at the node and the only thing that you send all the way to the server is the summary or
the calculations that you are producing in these nodes.
This is useful for doing some machine learning, some artificial intelligence, stuff like this.
The thing is, if you don't pass all the raw data, first of all, you reduce a lot the latency
of the end result because you don't have to pass through the wire all this information.
Second, it's a lot more private because you are not putting all your eggs in one basket.
If for some reason the main data center gets attacked, they don't have access to all the
raw information.
Instead, the data is distributed across the different nodes and the only thing that they
might have access to is the resulting calculations, for example.
Or even if a certain specific node gets attacked, then that doesn't mean they can access the
rest of the network.
It's a very interesting concept that is being formulated right now and Kairos wants to be
a solution for those nodes that are being run there.
How does Kairos do that?
First of all, Kairos is immutable and that means that the operating system is read-only.
The user data, of course, can be written, but if for some reason someone takes a hold
of the machine, they cannot install any packages, they cannot change any configuration.
Even if they manage to do it, as soon as the machine gets rebooted, you still get the same
original image that you used to have.
If we go even a little bit deeper into that, what does it mean that the OS is immutable?
Because there are, for example, some cases where the root partition is immutable, but
some other areas are not.
Kairos in this case is an image, we distribute it as an image as a whole.
That means that all the OS components, including the kernel and the init-rd, they are all immutable.
You cannot change it.
init-rd is not built at the moment of installation, but it's already there when we ship it in
the image.
Let's see another interesting goodie of Kairos is that it's distribution agnostic, or I guess
better said, we are friendly to every other distribution.
We don't want you to lose the distribution that you already like and love for some reason.
It could be that you already have a big know-how that you have built onto your distribution,
so you don't want to change to a different distribution.
It could be that you already have a licensing that you're paying for a center company and
you don't want to switch that because of costs or something else, or it might just be that
your operational team has decided on working on certain golden images, and we don't want
you to have to go away from any of these just because you want the goodies that Kairos can
offer you.
Right now, Kairos can play well with OpenSUSE, Alpine, Fedora, Debian, Rocky Linux, and Ubuntu,
and we're trying to add more different distributions on top of that.
Another interesting thing that Kairos does is it tries to be really easy to configure
and maintain.
For that, what we do is that we have YAML, where you can define the way you want your
system to look.
In this case, for example, I am creating a user called Kairos with the password Kairos.
I want SSH keys.
I want to go and grab from the user modeler his public SSH key, and I also want to put
this particular SSH key in text that I want inside of the system.
I also want, for example, in this case, K3S enabled.
As you can see, it's very easy to use, it's very easy to keep versions inside your Git
repositories, so you don't have to break the current flow that you already have if you're
already doing DevOps.
We want to, like I'm saying, we want to make it really easy to configure and maintain.
We provide a new web UI that was just introduced in version 1.5 for Kairos, where you can take
that configuration that I was showing you, you go to a certain node, so you look for
the IP of the node.
The node has just been booted.
There's nothing else running in it, there's no installation yet in it.
You just go to the IP of the node, you paste that configuration that you want, you say
install, and the node gets installed the way you requested.
We want to make Kairos easy to configure and maintain, but when you have a machine at the
edge, that might sometimes mean that you don't have the person that is in charge of doing
the configuration there physically.
Sometimes you have someone who is maybe not technically that savvy to do the work, or
that for some reason you don't have the trust of that.
It could be in this case the manager of the store, let's say.
For that, what we provide is that Kairos will present itself on installation with a QR code,
and then all they need to do is send a picture of that QR code, and then whoever is doing
the installation or the configuration can use a command line and then take the configuration
in YAML and also pass the image with the QR code, and Kairos will be doing the configuration
itself.
What else?
So, Kairos also performs AB upgrades.
What does this mean?
Like I was telling you, Kairos is distributed as a full image.
So whenever we are doing an upgrade, we switch the image completely.
So whatever image you have on active mode that is being run on the system right now,
we download the new image for the upgrade, and then after reboot, we do a transition
in which the new image becomes the active one.
That is helpful because if for some reason we cannot really start that new image, there
is still your old version of the OS that can still run, right?
So it's a little bit more reliable, we could say.
Of course things can go bad, and even if you have your two versions of the OS, you could
still screw things up, and for that we also provide a recovery partition, also in this
OS part which is immutable, that you can access to so that you can do manually whatever fixes
you might need to do to one of those two partitions.
Another goodie that Kairos provides is it has TPM encryption, so nowadays a lot of machines
or IoT devices come with another chip, generally, which can do TPM encryption.
This is useful because you can imagine it's just like having a UB key or I don't know,
there are other providers right now inside of your system, and that way you can trust
that only this machine, for example, is the one that is going to be able to unencrypt
the data of the user.
This is useful because, like I was saying, if you're at the edge of the network, if for
some reason someone steals your hard drive and they put it into some other machine, they
are not able to unencrypt it because they don't have that chip on their machine.
There are multiple facets in which TPM encryption can work.
I'm not going to detail all of them.
I think if you're interested into that, you should check there was a talk yesterday by
Leonard Pottering about the TPM encryption was very interesting.
But it can get a lot more complex, we could say.
You could even say, for example, that not only you need the machine, so the chip, with
your data, but you also need a Kubernetes approval in order to do that.
It will really depend on your model of security that you want to have, of course.
If you put all of these things together, we believe that Kairos makes it a great distribution
to be run at the edge.
What I wanted to tell you about is now how do we build that kind of distribution ourselves.
For that, we use something that we call the Kairos factory.
Just to give you an idea of how this works, we start by having Linux container images
that are provided by the distributions.
We base our work on the amazing work that the distributions are doing, of course.
We don't want to reinvent the wheel.
When we take that container image, we pass it through the Kairos factory, and as a result,
what is at the end is what we call the Kairos images.
Right now, we offer two different types.
One is the Kairos core, which provides the immutability.
It has an agent, which can be used for upgrades, for installation, and many other things.
It comes with the kernel and the init.rd, like I was mentioning.
We also provide what is called the Kairos standard, which brings everything from the
Kairos core, but on top, we also add K3S Kubernetes flavor, and also HBPN users for peer-to-peer
networking and VPN and other things.
The way you consume these ones is we have releases.
If you go to our GitHub page, you can download, for example, an ISO, and you can install this
in bare metal, or we also have, in our docs, we have all the different distributions that
we support and links to the OCI images, the container images, which you can use for upgrades.
Not only you can use those for upgrades, but you can also use them for customization.
This is where I guess it gets most interesting to the people who are going to be using Kairos
at the edge, because it's very simple to extend the image that we provide you.
If you're already using Docker, which is probably the case, if you're used to using Docker files,
which is probably the case, if you're using Kubernetes, well, you don't have to learn
anything new.
All you have to do is, say, the front part, you use our image, and then you do whatever
you want to do.
For example, I'm installing the application figlet, and then I just have to tag the version
of the OS that I want to distribute.
Then you can have whatever release cadence of your own that you might want to have.
Now you might tell me, okay, that sounds good, but maybe I want something a lot more complex.
Maybe I want to do a release like the way you guys do Kairos standard.
No problem.
We also have this thing called providers, in which we allow you to do a lot more complex
things on the Kairos machine that you're building.
That's exactly how we do the Kairos standard.
That's how we put K3S and HBPN in it.
You can basically just start from any of the ones that we have and build your own.
That's pretty much the process that I was explaining.
Let me talk a little bit about the challenges of starting with the different distributions
all over.
First of all, it's not so easy with the packages because some distributions come with certain
packages, some with others.
Sometimes they are named differently, sometimes they come in different versions.
Another problem is that the base configuration is not the same in all the distributions.
For example, we recently had a bug related to QR codes not being easily displayed because
the configuration in Ubuntu was a little bit different.
Another thing that is quite different is the init system.
Right now we only support system D because it's basically the most mainstream one, I
would say, on the mainstream distributions, and open RC.
That's problematic because we have to maintain two different flows of code just for these
two systems, but it is what it is.
We also have issues with the C standard library.
For most distributions that we consume right now, they come with the Glib C, but there
are distributions like Alpine that comes with Muscle, and that makes it a little bit challenging.
For those distributions, we cannot provide a kernel or init.rd from that particular
distribution.
If you check, for example, the image that we put out there for Alpine, it will have open
Susa kernel or an Ubuntu kernel because we need to be able to build it somehow.
Let's dig a little bit deeper now that we know the challenges that we have.
How do we try to address them?
Well, again, starting from a Docker file, in that first from that you see on the left,
we put the distribution image, so let's say open Susa Tumble with latest, and then after
that we install certain packages.
That way we base out all the different distributions that we have out there, so if there are some
packages that are not there in open Susa, we install them, or if it's in Ubuntu, we
install them so that they are kind of balanced out, so that we can do everything that we
need to do.
We also do a little bit of system configuration.
Mainly it is about ensuring that certain init processes get started properly.
Then the result of that, we put into container image.
One thing I want to mention is that we are agnostic.
We use any OCI-building engine.
If you're using Docker, great, if you want to use Podman, whatever works for you.
We start building that new image.
Then we install Kairos binaries.
This can be, for example, the agent.
The agent is used for installation, for upgrades, and other things.
We then install Kairos packages.
This is different from the distribution packages.
These packages that are specific for Kairos, they are mainly for tooling.
The really cool thing there is that they are completely agnostic.
The OCI-building engine, so that means that we can be building Fedora image, and we can
be using packages from open Susa and Ubuntu at the same time to do this.
I personally find that really cool.
Then after that, we do certain system configuration.
This can be about how we're going to, for example, mount the different systems in the
disk and stuff like that.
The result of it is going to be a container image, an OCI image, that you can download
for, like I was saying, because you might want to do certain configuration on top of
it.
You might also for doing upgrades, or we pass it through something that we call the OS
Builder, which will convert that OCI image into an image that you can actually boot.
You burn it into a USB, and you can put it into hardware, or net boot it.
Of course, all of these changes are prone to issues, to errors, to breaking things.
To avoid having that kind of situation, we have a CI system that is ensuring that we
can build every one of those distributions.
If something fails there, we go and fix it before it can be released.
Once every image has been built, we run a certain acceptance criteria tests, like I
don't know, we're sure that we can do an upgrade, we're sure that we can do an installation
of Kairos, et cetera, et cetera.
Putting it all together, that's how you can have a really nice secure distribution running
at the edge of your network, talking to your Kubernetes cluster.
If you're interested in testing it out, please, you can go to our website.
There's a lot of documentation there.
You can download the different releases, try it out.
You can try it out on your Raspberry Pi.
It's quite fun to do.
You can talk to us via matrix, GitHub discussions, and we have even office hours.
Every Wednesday, 5.30 p.m. European time, you can chat to us if you want.
That's all I have for you today.
If you have any questions, please let me know.
By the end, if you are interested in stickers, please come and grab one or more.