Please join me in welcoming the next speaker, Roman.
Thank you.
Hi.
Today, I'll tell you a success story of adopting Nix at the workplace, all the way from reproducible
CI builds to NixOS in production, and also what we have learned in the process.
By the way, you can use the QR code to follow the slides on your devices if you want.
I have some links there as well.
So first of all, me, well, now ex-Principal Software Engineer at ProfiN, I've been using
Nix at NixOS since 2016.
Now ProfiN was a security company, and we were custodians of the NRX open source project.
We were mostly focused on, well, developing NRX and also providing network services around
NRX.
But unfortunately, just earlier this week, we actually closed the company because we
ran out of money and couldn't secure funding or acquisition.
So NRX is a confidential computing platform providing you the ability to run web assembly
within TEEs, so hardware secure enclaves.
And well, I'm not going to go into too much detail, but the relevant part here is that
NRX builds trust based upon a remote registration procedure.
And to do that, basically, how it works is that we as ProfiN, we build a portable, mostly
static NRX release binary, then we run it and we measure the memory pages within a TEE,
then we add that to an allow list in a station service, and finally, then the users are the
release binary, and they, well, they also run it on a TEE, and we in our station service
can verify that indeed they are running in a trusted system, this trusted release of
NRX within a real TEE.
So there are two questions, of course, which arise, which is what if the release version
of NRX is compromised, or what if actually our built pipeline is compromised.
So then we can also ask a question, so how can users actually verify that the source
code that we have released, we say, okay, this release corresponds to this source code,
how can I actually verify that it is indeed true, right?
So if you would try to just do plain cargo build with a Rust project, then you'll notice
that actually you get different binaries depending on the system in which you compile
your binary, and so the answer to this is to actually use the produceable builds, which
of course, Nick has a way to do.
So here is an example as well, so if I just compile it in the Docker container, right,
and if I just do it locally, I get exactly the same binary, so I get exactly the same
digest.
But so how do we actually get here?
It's all, of course, self-of-the-shell.
How do you develop anything without the next shell before in your project, right?
So I've got positive feedback on this, and, well, we infected the project with Nick's.
Just a few months later after that, something very surprising to me happens is one of our
first outside contributions was actually a NRX build package, so I just added a shell
with actual build script by Vincent from Germany, I know if you're here, but thank
you very much if you're looking at this.
And so by the time we had to ship and release, you know, like a big release of NRX, we already
had all our Linux and Mac binary builds in the flake, and we could also build OCI images
in the flake.
I did some changes in the meantime.
So yeah, it just made sense to use Nick's for this, and we quickly discovered that actually
building for ARM64 in QMU on X86 runners was just too slow, so that's why I implemented
course compilation.
It was tricky, but eventually it worked, but unfortunately it also made all functionality
parameterized by the package set, and it also made flakes very difficult to maintain a review,
especially for people who are not familiar with Nick's, because I was the only Nick's
guy in the company, so it complicates things.
And here's an example, I don't know if you can see, but before it looked like a just
normal kind of build script, you know, like if you did GEN2 packaging, you would kind
of understand what's happening here, if you just use Rust, it's kind of clear what's
happening, but when you have cross-completion, it gets difficult, right?
You have a linker, you have the CC target prefix, right?
It's not clear, like this let-in block, there are two different package sets.
So even worse is that, well, we had multiple repositories, and initially we just duplicate
our flakes, right, because we want to build everything reproducibly as well, we want to
have cross-completion, we want to have just consistent, you know, CI, and basically each
flake was branched off from a different version of the original flake, right, because we also
were keeping developing and changing things.
So they started to diverge in some supple ways, but they still largely were doing exactly
the same thing.
And another thing is that because the flake logs were actually managed independently,
so we could benefit from some CI caching, but we actually couldn't, because those potential
hits were actually becoming a miss.
So and the maintenance just became a burden over time.
So let's just take a step back and think about, okay, what do we actually want to do?
We just want to build some static Rust binaries, just like Cargo does, except it won't do it
reproducibly, right?
We want to have an OCI image for that binary, and ideally you also want to have a fast CI,
but well, if you can't, all right, fine.
But we don't really care how any of this is done, right?
All we care about is that if Cargo can do it, Nick should be able to do it as well.
So just add functionality, not remove any functionality, it doesn't make anything harder
for us.
And you could say, right, there are templates in flakes, right, you could just write it once
and then propagate across repositories and just use that.
But it's not much different from just duplicating the flake as I did before, right?
You still have to adapt your template for the actual project, and you still have to
maintain, it's going to get out of date, and there should be a better way.
So that's why I built a Mixify library, it's designed to be an easy to use, batteries include
library with opinionated but customizable defaults, and it just works well in CI.
It doesn't try to cover all use cases, but it should be good enough for most.
It just simply plugs into your existing language tooling and currently supports only Rust via
Crane back end, but it could support other back ends as well if you want to, and there
is also support for other languages just kind of designed for.
So what does it actually do?
So per each default system, it provides you with a flake check for linking and testing,
development shell of your tool chain, formatter to do Nick's FMPT, and then release and debug
builds for all targets with cross compilation and OCI images and whatnot.
Here's an example, so this is actually just a Rust Hello World application, right, that
has just simply one input, which is Mixify, and I have just outputs generated by this
make flake function, and the absolute minimum is just if I have one argument, which is a
source.
So Mixify will parse actually the Rust tool chain and cargo automobile if you're familiar
with Rust, so Rust tool chain defines basically what version of a tool chain I want, what
targets I want to have, and cargo automobile is just basically metadata about the package.
So let's look at the next flake show.
I formatted the output because it just wouldn't fit on the screen, so this is the checks,
so I get my linting, my formatter, the Rust formatter, and the testing on check.
I get also a development shell with my tool chain and all different systems.
I get Nick's FMPT already predefined, and I also have an overlay with actually the tool
chain and all the packages, so including OCI images, and finally that's the packages
generated.
So I have my native default build, I have also profiles for release builds and debug
builds.
You'll notice that there's no Darwin builds here, because again I've formatted it.
On Darwin you would see also builds for Darwin, but you cannot cross compile from Linux to
Darwin, but you could do other way around, so on Darwin you would see that as well.
In fact there was an issue with the next package set right now, there is no, like one of the
Darwin was not able to compile for the other one, unless you can check yourself.
So here's an example of NRX packaging with this tool, so I can also configure for example
some paths to ignore, to improve caching, I can configure the Clippy features, whatever
I want to do there.
So it's pretty small, pretty simple, I can add some, like for example opens a cell in
my shell, and yeah, it works.
And NRX itself is, by the way, anything but a simple project, so it's using nightly
Rust, it has seven crates in the workspace, we actually use bin depths as well, so sometimes
we build for three different targets at the same time, basically the shim and the execution
layer which are merged later in one binary.
So this what it means to actually build something in CI of this, so we just simply have a matrix
of all our hosts and targets, and we just simply do nix build, and this is consistent
in the same everywhere, right, the only difference would be like the name of the package, but
again it could be removed if I wanted to.
So next we have testing, and we have linting, it's pretty much exactly the same thing, again
it could be a shared github action workflow and just used everywhere.
So how do we actually maintain and update this, the next five changes are actually for
us was propagated automatically, so we used github action to actually do the next flag
update, so the changes were already reviewed by me, so I've audited the changes essentially
and then anyone in the team can actually, well you can see the bottom, but you can actually
review and auto merge this because the team trusts me, this actually brings me to a trust
question which is an open question I just wanted to raise, so the nixify state is essentially
a root of a miracle tree, because it includes all the dependencies and digest of those,
so I audit the state of the world, so the next package set and all the dependencies,
so team trusts me, therefore they trust the world or my audit of it, so it's a transitive
trust, but then how can the team actually verify my audit, so can I sign the nixify
a root in any way, can I maybe add my signature on this, so maybe nix could validate my signature,
maybe it could be like a parameter locate trust, only these projects, so that's just
an open question, if you have answers please let me know.
So eventually it was time to deploy our services and for NRX we require some auto tree kernel
patches until provided asmd pccs service running some use of rules for the hardware, we had
no dedicated operations engineer and all our repositories were already nixified, so we just
made, well we also had two cloud providers, which is AWS and Equinix, both had nixwire
support out of the box, before we had nix we actually used custom OCI images for pccs
and asmd and those were basically just compiled once to some binaries just put inside the
image and those images were outdated, largely amante, no one knew what to do with them,
we required custom provisioning steps like to do podman, secret, create or something
and we also needed manual udev rule setup, so once again open source come to help, so
the pccs and asmd were actually turned out to actually be supported by nixOS already,
also the Intel SGX hardware was supported, by the way again it's either Vincent or one
of his team, someone from his team who added this, so again thank you, so it was just these
six lines were enough for us to just enable the services and all the hardware support.
So next step was that I just added some simple nixOS modules for our services, I set up secret
management with SOPS with nix and deploy RS, again like simple tooling, I automated deployments
to testing environments, I set up CI to just test all PRs, so if CI fails PR doesn't go
through right, so we don't update anything and of course we have the automated updates
just like everywhere else, just for every other repository.
So to begin with we actually ended up doing this, so we actually were tracking tags of
different projects, so you can see there are three groups of, well these are the services,
different environments, so we just have three different inputs and essentially we progress,
we test things and staging, then we go to production, right.
If I were to redo this I would actually use branches, something like mixed package that
channels where you essentially just do a release rather than you merge it to say unstable,
eventually you test that, you promote it to stable, so I think that's more work upstream,
but it's easier to, well, maintain downstream, it just makes sense.
So you could ask, okay, why don't we just use OCI for this, but the thing to understand
with nix, we get source code references, not binary references, so essentially we get all
the benefits of binaries without actually sacrificing and usability, in fact we even
get things like updates and the flag.log is nothing else than really a software build
material, which is the buzzword today and everyone really cares about it, so because
in the flag.log which you get is that you can audit the whole state of the world, right,
you can audit all the service source code, you can audit all the build dependencies and
you can audit also all the tooling that you used to actually deploy this stuff, right,
and you have a consistent simple update procedure, which is nix flag update, it's super fast
to deploy once you have your module set up and you know how to use nix, it's just, boom,
you're just there and of course you get rollbacks with nixOS or really any deployment tool is
used nix.
So next step was actually providing things like AMIs and whatever else, so I used just
nixOS generators for it and like the way I did it, I just simply imported the nixOS module,
like a common module from our infrastructure and it took me a less than a day to set up
and this is for SGX and AMD SCV, again it's extremely powerful and just extremely simple
to use.
I do have to mention that we asked ourselves a question, okay, so how many enterprises
would actually use nixOS to deploy NARCs, that was probably not so many, so once we
actually got a dedicated operations engineer, we eventually moved most of our services to
Kubernetes, but there was a little win because for every service that actually required custom
kernel, custom services, custom udev rules, those things are difficult to set up and those
actually were kept in nixOS, which brings me to the next point, that nix and nixOS actually,
in my opinion at least, make difficult things simple and in this case for me it was great
for prototyping, it was great for productivity, for composability, for building literally
anything anywhere and for audibility and trust.
One particular thing I want to mention about productivity here is like if you ask me, one
killer feature of nix, at least from my perspective, this is that, so we had for example lab machines
with SGX and AMD SCV hardware and I did it countless times when I was just developing
something as a feature branch and I just needed to test it out, I just SSH and do nix run,
GitHub call, my repository, my branch and I just run it, I don't need to use git to add
a new remote, I don't need to set up my toolchain, nothing, I can just SSH and run it, so I think
that's really powerful.
So another thing is that if you introduce nix, of course it's also your responsibility
to teach it, so you are the FM in RT FM, so for my case I had the nix 101 classes on Fridays
and the real uncommon thing about nix, I think the people is a functional programming part,
so I ended up teaching a lot of functional programming principles to actually explain
nix and also just introducing teams with ecosystem because well, if you just newcomer to nix,
it may be overwhelming, it's not clear what is, why there's nix, a nix package set, nix
the way, how they're related and I also create a nix channel for questions, so whenever people
had questions I would just answer them.
So some derivations from this, it's really important to give people examples of how to
get things done, like real examples and it's very great also to give analogies with known
technologies and okay, so this is not exactly what you would do in nix, but I implement
Fibonacci in nix and in Rust, I just show them two side by side, I have a main function
here in nix, I have a main function over here, both free to sum in both print Fibonacci examples,
so it just works, you don't see but opposite here the same and yeah, so this kind of is
an example of how you can show the people that you know, it's not so scary after all
right, it's just the same function, just look at it right, the same thing.
So one thing I noticed also that nix is perceived as being pretty novel, so I ask people the
question like okay, so how long do you think nix exists and I got answers all the way from
two to ten years where ten was like a real stretch or not, it can be so long, but the
real answer is twenty years, if you look at the git log right, I don't know, maybe it
was not before, maybe it was earlier, but that's what I see from the git log.
Some frequently asked questions I've received is okay, I'm not on nixOS, how do I use nix
or I'm trying to use the flakes but I get an error, why are flakes experimental, is it
safe to depend on some other features, what if they get abandoned, for us our show.nix.
So a few things we can note here that at least in my experience nix is tightly associated
with nixOS, but we need to understand or at least explain to newcomers, I believe, that
nix is not just another package manager, so nixOS is just one possible output for nix
build but it's just that, it can build many other things and I think we should present
it as a generic build tool which is not tied to a particular OS and sometimes for people
familiar with Docker, I even presented something like Docker or Podman where you basically have
a docker file, you can build a docker file, it's put in your store and you can run it
afterwards, it's something like that, maybe.
And yeah, this thing was such a pain, so come on guys, we need to do something about this,
I mean you hear about this nix thing, it's so cool but it's so difficult, so out there
and your very first try in nix is a failure.
I understand that this error is descriptive, yes, you read this, you kind of understand
what to do, mostly, but it's just not a good developer experience, right, and I think we
should, I don't know, I don't know what to do, there could be possible solutions but
we really need to do something about this as a community.
So thank you at this, you can, yeah, as I said, profing close, I'm looking for a job
by the way, but yeah, any questions maybe, yeah, good, right.
So the question is, can nix be a generic build system like Maison or Autotools, whatever,
so I'm not a C developer, I've never was, I believe so, I believe you will be able to
do, at least for Rust, it was a breeze to do, I can definitely imagine doing that for
Go as well, and I think that was pretty, so for example, I compile, so we had a collection
of examples where we mostly compiled to Wasi, so for C it was actually very, very simple,
so I would think so.
The only thing to care about though, so it's great for releases, not so great for development
because, for example, with Rust you have this target directory with all your cache, so you
don't get that anymore, right, or, I mean, you have to, you can do things around it,
you can maybe build some libraries to actually achieve that.
Maybe you can do it and contribute.
If I may add about this, there is some RFC on how you enable nix to make replacements,
but there are much more things which are a bit complicated on how you make it fast.
Okay, okay, I'll check it out, thanks, I don't know that.
Thank you.