Hello.
All right. Hello.
Welcome. Thank you for joining this talk.
This is about infrastructure, code,
and mostly about...
and mostly about choices.
So that's the main point I want to make.
There will be some analogies with food.
So, yeah, I hope you don't leave the room to go find food.
So, stay with me.
Mm-hmm.
You really need to stay close to that.
Yeah, yeah, okay. Thanks.
All right, so this is me.
The one piece of information you may want to keep
is the Codeberg username I have,
because the code I'm talking about,
so some samples and things you can copy from
are in the Codeberg repo I'm going to show later.
So if there's something to keep from this slide,
it's this one.
And also that I'm working for Kyros,
which I'm going to talk about a bit more later.
That's the project I'm working on,
the App & Source project I'm working on.
So, yeah, I said this is about choices, mostly.
So I'm starting with food,
because it's a general thing in life.
I mean, when you have to judge something
with just one criterion,
sometimes choices seem obvious, right?
So let's take...
This is a popular choice, a popular Greek,
this, Musaka, and that's the well-known burger.
So if we were to choose what we eat
based on just one criterion,
that one was the amount of time you need to prepare,
the choice is obvious, right?
But obviously we don't always eat burgers for some reason.
And the reason is that we have other requirements.
But sometimes when the main criterion is there in front of us,
our mind gets stuck to certain choices.
And we had to choose otherwise,
and that's what I want to show you,
our story in Kairos and how we chose to do infrastructure
otherwise.
So this was our problem.
So what Kairos is in a sentence?
No, maybe two sentences.
It's an immutable OS, a special purpose OS,
mainly targeting Kubernetes.
So it makes Kubernetes very easy to deploy,
maintain data operations and such.
But it's this diagnostic.
What that means is it brings immutability
on top of your favorite distribution.
So you start with your distribution
and then you apply Kairos, let's say, on top,
and it makes it immutable, safe, secure, encryption and all.
But that also means for our CI
that we have to build many different images,
lots of artifacts.
So you see some numbers there, like the number of pipelines.
And one thing to keep in mind, and you all know that,
is that when you're working,
you don't generally want to think whether you push something
or if you open a pull request or something.
If you just do it, you don't want to have to think
that you're gonna pay for something.
So initially we started, it's on GitHub, by the way,
the project, so we started with GitHub Actions,
the free ones.
But because of these requirements,
like huge disk space in some cases,
like two VMs in one job,
KVM support which free runners didn't have,
we were looking for a different solution.
And initially we said, okay, let's put money into it, right?
Money solves everything, and let's just start paying
for runners.
In our case, that didn't work because,
like I said, we didn't want to have to think
whether we push, so we were controlling ourselves,
okay, let's not open a draft pull request, right?
Because that also runs pipelines, let's wait a bit.
I'll do that tomorrow and things like that.
So we very quickly reached the point where the cost,
the amount of money we were paying was double the money
for the same kind of hardware, but on bare metal.
So what did we do?
I mean, okay, we rejected that one.
Dog fooding, so as I said, Kairos makes it easy
to deploy Kubernetes, which is maybe the hardest
and the more complex part, if you want to start
with your own hardware, like how do I maintain it and such?
And that's what we do as a project.
So we said, okay, we're gonna do that.
And then some more tools, like, okay, Kairos,
we solved the Kubernetes problem,
how we provision stuff and how we maintain it.
Then we chose FlexiD for GitOps.
I can't go into details on what these are.
I hope many of you know what these tools are,
but look them up because I only have 10 minutes or so.
Shops for Secrets, so that allows you
to actually commit secrets in your repository
but being encrypted.
So it's safe to commit them
because you need keys to decrypt them.
But if you give the keys to FlexiD at runtime,
they will be decrypted and deployed.
That means you can have full GitOps,
nothing, no manual intervention for secrets or whatever.
And then there is a project.
You probably know that the Actions Runner Controller
that allows you to run GitHub runners
in a Kubernetes cluster.
So this is our toolset.
The next slide is completely relevant.
It's just that I generated this one with AI
and it reminded me of something.
And after a while, I remember what it reminds me.
That's my real dog there doing the same thing.
So we don't need the AI, just remember that real life.
Does the same.
So yeah, back to infrastructure.
Yeah, so some steps we did.
So you can start with a cluster on your laptop, right?
Very easy to get one, K3D kind, micro-kates,
whatever you prefer.
Then you read the docs for FlexiD, obviously, and Shops.
You create the keys you need.
I'm not going into details, but this is what we did.
More documentation on how you deploy
Actions Runner Controller.
And when everything is working,
that's the interesting part.
Then you can go and get some real hardware.
We went for value for money things.
We tried a couple of them.
So we got, I don't know, 10 different machines
or something and then we deployed.
And that's the interesting part.
The three commands there.
How much time do I have?
I think I can show you a demo.
At this point, where am I?
I'm there.
There.
No, no, no.
Where are you?
Where did you go?
There you are.
So what I'm doing here, not here, but here,
I will restart it, don't worry.
So I start on a project that has no runners.
On the left one is the repo.
I'm gonna show you the link in a while.
And on the right, I'm just copying and pasting three commands.
There's a timer down there.
It's three and a half minutes spoiler.
So what I'm doing is with three commands,
I'm gonna start from a cluster, from scratch.
So I use K3D to create a cluster.
And I play the secrets.
So there are just two secrets one that Fluxy needs
to pull the repository and another one
to decrypt the secrets that are encrypted with SOPS,
like up keys for GitHub and such.
This is the secrets command.
This is the one that creates the namespace in the final
and bootstraps the already existing repository.
So after three minutes, you'll see that we got runners
deployed and connected to GitHub.
And that's from a scratch cluster.
And we had to do that two or three times
because we chose some hardware that didn't work
because of some network issues in Hexen, I think.
We had, then we moved, but by moving,
we were afraid that, okay, we're gonna do that again,
but it turned out to be very simple.
So what I'm trying to say here is that the choice we made
paid off because every time we need to recreate
the whole thing, just three commands, right?
We create a cluster and we just spin it up.
So the initial choice, I mean, we weren't sure
if we had to spend the time to create all these
because yeah, I described that in 10 minutes,
but it took a sprint or two, like a week or two
to implement it, but it turns out it pays off
because now our infrastructure is cut.
It's not patch anymore.
We don't care.
We can go somewhere else if it's cheaper.
So it does pay off.
So I'm not sure where it is now.
Yeah, I'm cheating a bit here because I don't want
to wait for the reconciliation.
So I just keep the controller to force it to check.
This tool is K9s, by the way, if you don't know
that check it out, it makes it extremely easy to navigate
through Kubernetes resources and all.
So yeah, at this point, yeah, this is the action
runner's controller and this will bring up the runners.
So when we want to make changes on this thing,
we just commit.
So if you don't know what GitOps is, you probably know,
but you just make a change, you commit, yeah.
You commit that to Git and you can actually review that
and this thing will apply the diff to make your cluster look
like what you described.
So yeah, after a while, they come up.
Maybe quickly I can skip a bit.
Yeah, there we are.
And eventually they show up.
So just go back.
Yeah, that was it.
This is the repo.
So everything I showed is there.
You can't really use it because some of the secrets
are encrypted with my own keys, but you can copy everything else.
You just have to replace with your secrets.
There are instructions.
I try to write as many docs as possible,
but feel free to open issues or ask me anything.
I have my, yeah, that was a screenshot in case I didn't
have the video.
So the outcome, yes, it works.
It broke sometimes initially, but it balanced out.
And like I said, it really paid off
because it makes making changes extremely easy.
So yeah, what I wanted you to think that it is possible
and it's a good idea.
And I'm not saying it's going to work for everybody.
And I'm not certainly saying that GitHub, but others,
the paid ones or whatever is not good one,
but different teams have different needs.
So if you're thinking about it, check it out.
Check Kairos and ask us any questions.
This is my email.
This is the team's email.
We'll get that one.
And this is our matrix channel.
So whatever questions you have, we'll
be happy to talk to you.
And we're around.
We also have an ice hoodie like this, but it's large.
So the first come, first served, who is large and we have
stickers and we'll be happy to talk to you outside.
Thanks.
Do we have time for questions?
Yeah, we have.
OK.
Need a mic?
I don't know.
Microphone?
Where?
Or you can shout.
I'll try to hear.
Hold this there.
Hi.
Have you faced problems with different CPU architectures?
Because sometimes it may be hard to get some types of hosts,
like ARM x86.
Are you talking about Kairos?
Yes.
Yes, we're not trying to test.
We're not trying to specifically.
So the architecture of Kairos had problems with the...
Can you repeat the question?
So you run the test, but it's mostly based on containers
locally, but have you found problems testing in different
CPU architectures?
Yeah, got you now.
No, the tests are mainly running with QMU.
That's why we need big machines because it's a full OS.
So we do test ARM as well.
It's just a bit hard to test boards because Raspberry Pi's
are just not as easy to automate.
You need some KVM or something.
So yeah, mainly QMU would do ARM, but not the actual boards.
So sometimes things break.
But yeah, that's a Kairos question.
Thanks.
Anybody else?
We can take one more, I guess.
Yeah, there you go.
I saw that you're using the summer bind renders.
Those are the old ones, right?
Yeah, no, sorry.
I realized when I was running, yeah,
we have to update.
You saw that.
You're going to switch to the Google
Managed GitHub supported runners.
Ah, they changed the images you mean?
Yeah, I think the GitHub adopted the actions
runner controller.
Ah, I didn't even notice that.
Yeah, sorry.
Thanks.
We'll do that.
Last question.
No, OK.
Thank you.