Alright, thank you everyone who moved and gets some space.
We can now start the next talk.
Josse is going to talk about using NixOS at NLNet.
Hello everyone.
So, yes, my name is Josse van den Noeven.
I'm an employee at NLNet.
NLNet is a Dutch foundation.
And who here has heard of NLNet, by the way?
Are there any hands down?
Wow, this is amazing.
That's very cool.
Yeah, so it's an honor to work at NLNet.
And this talk is going to be about how we use NixOS there.
There were so many hands.
NLNet is the organization which here at FOSAT might be known
for, you know, spamming stickers everywhere.
We have the stand in the K building with so many stickers.
And each of these stickers here is a project we have supported.
But not all of the projects that we have supported have a sticker
because, you know, command line tools might not have a logo all the time.
As you can see, NixOS is up there as well, as well as many other projects.
I'm wondering who here has ever had funding for NLNet?
See, that's less hands.
But we have funding for open source projects.
So if you have good ideas, if you're part of a community that has
these tenacious bugs that nobody is coming around to help fund to fix,
or if you have a protocol which has not been implemented in your particular library,
or whatever good idea you have, just look on our website what other projects
we've been funding and, you know, write your own proposal.
Proposals for writing to NLNet are not difficult to do.
It's one form.
You say who you are, you say what your plan is, what the outcome is,
what it's going to cost, or what you think it's going to cost,
and then you press send.
And every two months, there's a new call.
So this is the tagline that we use since this week, actually.
We have a PR person now, and she, you know,
she says the message should be simple, it should be clear,
and it should be to the point, and so we try to, she tried to fit it into one line.
We fund those who contribute to the open internet, you know,
because that's what it's all about, why are you here at FOSDEM?
And, yeah, we're just very happy that we can help there.
So what do we mean when we talk about the open internet?
Well, we should be able to communicate directly, right?
Get rid of big tech, which is in between our communications.
No dependencies, no lock-in, just get the source, compile it yourself,
and that way we can have a good democracy, we can be independent
and not have to be living in fear that some service is going to be taken away from us,
because, you know, we can run it ourselves.
So self-hosting is a thing that we very much promote, yeah, free software, free society,
and this logo here, Next Generation Internet, is, that's the thing that makes me standing here,
because that's the fund by the European Union that is providing over 90% of the funding
that Anonite is able to give out.
We have been given out money for decades now,
but we were always very minor operation until the EC decided that, you know,
there's so much software in this world, we're running on it, we're depending on it,
we should, you know, also be the owner of it and invest in it.
So that's what the EC is doing now, and we are one of the facilitators that, you know,
seek out the right projects that are to be supported.
So we fund open software, hardware, standards, documentation.
When you submit a proposal to us, it has to be something that you can deliver,
you can, you know, get pushed or you can publish somewhere,
not, for example, server maintenance for that or having a meeting for that you have to go elsewhere.
We like to, you know, we like to check what the money is being spent on,
and that's also what we have to report to the people that give us the money,
which we will mostly be doing, so we try to keep the bureaucracy very low.
Yeah, self-hosting. So self-hosting, of course, means system administration.
Who here likes system administration?
50-50. Yeah.
So, yeah, it doesn't always go well with system administration.
You're in the basement in some organizations.
In the Netherlands, we're only small, so I get to sit with the other people.
It's not all that bad.
Once a year, you know, you have system administration appreciation day,
which is awesome, right, if people remember it,
and, you know, if they're not on holiday, because it happens to be in the middle of summer.
So, yeah. Not everything's perfect.
Okay. How do you use Nixle's in a small organization?
That's what this talks about.
And in the Netherlands, we're currently tendon police when I started, we were four,
so we're growing. Also, when we started, we were running a bunch of different systems
with backups sometimes, no commits of the configuration,
so no history of what was running.
It was a mail, for example, was running in a BSD system with set-afers,
so it had snapshots, so that was pretty good.
And our requirements are really not that crazy.
We need mail, website, telephone, you would think.
But then if you drill down, there's quite a lot of stuff that you need to keep running, actually.
So here's what we have that is open source and which is not free and open source software.
So a website, obviously, it's run by EngineX.
Our email server is self-hosted, mailing lists.
We have our own code forge.
Well, what makes us tick? Our grant management system.
That is running using open source components and chat, video,
micro-blogging since a short while.
We are also hosting it ourselves.
But not everything.
For example, our router, which we could do, of course.
We haven't gotten around to that.
Printer, open hardware for printers.
That's not worth it right now.
We have some people using Apple devices, so it's not completely open there either.
Biases and chips.
I mean, we support people designing chips.
We're not yet at the stage that we can also dog food those.
But we have quite some components that we do ourselves.
So when we choose a system to get rid of the whole collection that we had before,
what options are there?
Well, there's Nixos, there's Geeks.
We could go to a closed cloud, but obviously that would be very bad for our image.
Or we could go to an open cloud hoster, which there are more and more of those now.
But we said, well, we are funding projects.
Projects are sending us that code.
It would be great if we could also try to keep our knowledge about all these systems up.
So let's try to do it all ourselves.
And then Nixos has quite a lot of advantages, also some disadvantages.
But the declarative part is, yeah, it takes some getting used to.
But it's really useful, right?
It's just nice static files.
It's mostly reproducible, and mostly it's mean 99.99% for the stuff that we use at least.
Extremely many packages, as you've seen in the talks just before now.
You can mix versions of stuff.
I'll show you a bit later how we actually need to do that.
The Nix language, well, there's a lot of discussion always about it.
But personally, I really like it.
So you have to get it.
But then it's great.
So it's familiar to us because before we decided to switch all the systems to it,
we were already using it on our laptops.
So that's a bias over there.
The Flake lock is very important to us because we can lock down the dependencies
and we can be sure that whenever we update, it's a conscious choice to do so.
Propriety packages are packaged, but they're disabled by default.
So we don't have to worry that by accident we are starting to depend on closed software.
Yeah, there are some downsides as well from our perspective.
So the community is organized on a proprietary system.
A lot of open source projects these days are.
And we really promote self-hosting.
So if a project is self-hosting, that's a plus in our book.
Another thing, not everything is as polished as it could be.
I'll show you that we are using an officially unstable feature.
So yeah, and there's no storage handling.
And what that means, I'll get back to that as well.
So there are a lot of green flags there.
Full disclosure, Nixos is a partner with us.
When people get funded at an Lnet, they also get services.
So they get free packaging and Nixos is providing that.
So we are a bit prejudiced when choosing Nixos.
For me, Nixos, I've been using it a long time,
but I always find it very difficult to write the packages
until one day I had to explain to a colleague of mine how these files work.
And I was sitting there and suddenly it clicked that yeah, everything is a function.
I mean, it's called a purely functional package, but still somehow it didn't click.
But then I had to explain it to him what are these brackets at the top with the columns.
And yeah, that's the arguments to the function.
And the rest of the file is what comes out of the function.
There are many Nixos developers who are thinking, wow, this is a newbie here.
But I feel a bit embarrassed to say it, but once that clicks,
it's really a very nice system because it's like JSONnet or Haskell,
other functional languages.
It's very predictable in what it does once you get it to do what you want.
So is it just Nix? Is that enough?
How do you deploy it on many systems?
So there was a talk by Sir Leanne Rappen a few years ago
on all the possible options that there are to deploy Nixos to a number of systems.
So it's a whole list here, and in her talk she explained what the pros and cons
of each of these systems were, and that was very helpful to us.
So that's why I wanted to highlight it here.
That was really amazing work that she did.
And in the end, what we chose is to keep it simple and do everything with Nixos rebuild.
And that's the basic command that everybody's using when you're using Nixos.
And it turns out you can just manage your service with that.
So all of our systems are defined in one Git repository.
They're all defined in one nix.flake.nix file.
Each machine has a configuration nix, hard configuration.nix,
but there are a lot of placeholders there for stuff that we import from another directory
where most of the services are configured.
And we try to keep the simple readable for everyone.
We use a JSON file that has sort of the structure of our setup in there,
and then that's imported and readable as variables further on in the system.
So if you do a nixflake show and the flakes are the not yet completely stable part of Nix that we are using,
then you will see Nix configurations has five servers in our case.
And what we do to deploy that is we type nixos rebuild, switch,
and then we say here's the flake for the server, and it should go to that server.
So that's how our deployment system works. It's just built into nix.
And this is our machine's JSON, so it tells us what should be the IP number for the different machines,
what name servers should it talk to, where are the secrets.
Secrets management is really done by rzink by us, so we just, when the machine reboots,
we don't store the secrets in the nix store,
we just copy them into the slash run directory with rzink.
And yeah, here's the flake. So we are mixing an old version of nix packages
because we haven't completely switched that, I'll explain later why,
with nixos, the current nixos, I mean you can just do that, you can put it together,
and so these are the inputs, and then here is the function that defines the outputs
where these things are coming in.
And this is then a very simplified version of how we define each of our machines.
So we have a function called makeSystem which takes the hostname and the definition,
and we define our systems by looping that function over all the machine definitions.
It's a bit more complicated because it has to know which inputs to use on which machines,
but this is sort of the magic that makes us able to just use nixos rebuild.
Now when you're setting up your system, this is the thing I think is most important,
is the alerts. The computer has to do stuff automatically for you,
and you would like to make sure that it continues to do so even while you're sleeping
or while you're giving a talk at FOSDAN.
So I'm very happy that this box in my mail folder has not had any unread messages
for a very long time now, so that's great.
Our alert board is green very much of the time.
We have a very particular alert here which is called the nixos flake committed.
So if somebody doesn't deploy without committing first, it gets read
because then it's undocumented what our system is doing.
This was a zoom in, but I think it was good enough to read.
Yeah, so backups, that's the second most important thing for your system.
We use Borg for backups and Butterbackup,
to do snapshots every hour, and here's a small point of critique
for nixos or actually a feature which is not really there at the moment.
When you do anything with software and it also needs data,
you have to say where the data is and everything is declared in nix,
except the folders have to be written by hand
or they're set by defaults in the services.
Doing backups, there's no enforcement that there is a backup
or an easy way to do the backup.
In the setup of your backup system, you have to repeat all the directories again
or you define them at the top level and then you use the variables
for those directories at the top level.
This is a thing that might be a bit more polished,
it's an opportunity for a new system, a new extension.
So mail, who here is hosting their own mail?
Wow, that's not enough.
We need more people hosting their own mail.
It's so important, it's still the backbone of all your communication,
it's email.
We really want to self-host, we were self-hosting,
so when we're setting up a new system, it would feel like a defeat
to stop doing that, so we continue doing that.
And nixos has a project which is called Simple Nixos Mail Server,
which ties together Dovcott Postfix, LDAP and Rspundi together,
it didn't use to tie up LDAP, but we needed that,
so we paid a contractor to add this support and upstream it.
So that's what we're using right now.
However, we're announced, so we're funding a lot of projects.
We're also funding Stahl words, that's simpler,
all included Rust implementation of a mail server,
and we're also supporting Mox, that's a go implementation of a mail server.
And we're soon going to try out Stahl words
on a less important mail domain of ours.
Yeah, and then you get these wonderful 100% scores.
If you fill around long enough, well, actually we didn't have to fill that long
because the Simple Nixos Mail Server really configures your mail properly,
and this wonderful website, internet.nl,
is what you can use to check if actually your mail server is configured correctly.
One highlight of Nixos that we really value is the testing.
So to test two computers working together is made very easy in Nixos
because there are Python scripts that you can call
and you set up both computers and you tell them how to talk to each other
and what the expected outcome is,
and all of these scripts, many of these scripts are just part of Nix packages.
So you can read how these systems, how this testing is done
and for your own setup you can also write those scripts, and that's great.
And we run that in CI via Flake checks.
Well, sometimes something can go wrong.
You don't have to be a genius to see what's going wrong here.
We are sending the configuration of server one to server two,
and this is where the system that we saw earlier comes in handy,
how to fix your booting because this really killed deployment one time.
So when I say we keep our system simple and we try not to build on top of stuff,
here we decided that it would be a good idea to make a small alias script
that only takes one argument so you don't confuse the two servers with each other anymore.
We recovered from this in five minutes so it wasn't that bad,
but I did get a big fright.
How do we do updates? I'm just putting this command here.
It's not that interesting, but I just want to have it documented somewhere
because it's a bit long, but we have a number of inputs to our Flakes,
and if you want to update just one input, which is often something that we need to do,
for example, when one of our software packages updates that we write ourselves,
then you can do only that Flake input with this command.
So conclusions.
We like to keep it simple.
We just use the basic tools of NixOS, and we put most of the configuration,
or we try to move as much as we can into JSON files so that it's easier to read.
So technically, NixOS is really great for an alat.
However, for the average office, it's probably quite complicated to do this.
So I think there's an opportunity here for open cloud providers to use a system like this
and make it more user-friendly.
And in fact, there is a project currently called NGI Fidiversity
where the EU is funding us to help create a new hosting stack that will be using Nix.
And that has just started. We have the planning phase for this.
So if you're interested, look it up.
There will be this, or talk to that guy over there.
That's a raving, but this will be probably a talk next year for them.
And with that, I'm done, and I'm open for questions or tips
because there are many people who are more expert than I am.
Thank you.
Do we have any questions?
Hello. Thank you very much.
I'm just wondering, you said that you are thinking your secrets to run directory,
like why you are not using like, Agenix or SOPS for that,
which will do it for you and you don't need to do it manually.
So the reason we're not doing that is there were so many options which made me confused.
And then also some of them were putting the keys encrypted, but nevertheless in the Nix store.
And I just felt more comfortable doing it with Arsene.
That's the whole explanation for it.
Hi. You said something about Nix not being aware of storage locations.
I didn't really understand that. Could you kind of explain that a bit more of what's...
Yes, so Nix defines where all of your software is coming from and how to compile it,
and it puts it all in the Nix store.
But of course the software is interacting with data.
And there's no sort of, you know, a type or a class which defines where the storage is.
So you could say, I'm doing backup now. Just backup all of my systems.
Or if you pass a directory into a service, that directory is an object which has been defined elsewhere
as it needs to be in the file system.
It needs to be... the file system needs to have, you know, a type of file system.
It needs to be mounted. All of those things are something that you have to take care of.
And because Nix is declarative, you know, once you hammer it down, it's fine.
But it would be great if at compile time you get an error for that.
Any other question?
There's a question in the back.
I just wanted to react to the storage location thing because it's interesting.
So in NixOS there is a problem which is you want to declare things, you want to be declarative.
But when you deploy a software, software often comes with automatic migrations.
So they proceed to the operation on your state, on your files, at every new deployment.
And this breaks the rollback system.
Because if you rollback to previous version, you don't rollback the data.
You just rollback the configuration.
And what could be done here is that the NixOS modules themselves could learn about where the state is.
What does it mean to back up an application?
What is dependency in the PostgreSQL database or whatever.
And it would start to provide a solution for the problem you're mentioning.
Yes, exactly. Databases are a whole extra level of complexity and possible, you know, data corruption.
I think we can take one last question.
Hi, thanks for the talk.
I might have missed it because I joined a little bit later.
But in the configuration, do you have a way that you are happy to pass secrets?
Yes, so the way we pass secrets is we have a top level JSON file.
And there we declare all our secrets.
So for root, it needs these secrets.
So wire guard needs a private key, the mail needs a password.
So these are files that have to be under slash run slash root.
And when the Nix evaluates it, they are stored. Where do they get?
So Nix doesn't do anything with that.
It just writes in the configuration where that file is supposed to be.
And then when the machine starts, some services will say, hey, I'm missing my password.
So I copy in the password in there and then I restart that service.
But that doesn't happen very often. We are fairly small office.
So we don't have 100 machines.
So automating it more seemed like complexity and overkill for our situation.
Okay. Thanks.
Thank you very much.