I'm very excited about this because this is actually the tool that I'm using to build
kernels for a while now and it's made my life a lot easier.
So thank you for that.
Dan?
Thank you.
Yeah, so let's talk about kernel hacking.
First a little bit about me.
I'm Dan.
I work at META on the Linux user space team and I'm a system demaintainer and I also maintain
the tool that I'll be talking about today which is a METCO sign.
So quick motivation for the stock.
A little while ago I started looking into running system dejournality which I work on for individual
users instead of just on a per system basis.
But to make this work I actually needed a BPF feature for Unix tokens that wasn't available
yet.
So I looked at the kernel source code and I figured this is probably doable to do myself.
So I got into kernel hacking.
One site figured out the code and were written up my first batch.
I of course had to test it but there wasn't really a clear way like this is how you test
your Linux kernel batch.
So I started looking into what I could do to do this.
The first thing of course that I needed to fix is like if you have your batch you can't
test your compiled kernel on your host machine of course because if it's broken then you
suddenly lose your system.
So you need a virtual machine or something to avoid breaking your machine.
I also wanted to make sure that this setup is quickly replicable to any different machine
because I started on my laptop because that's what I do for system dejournality and it works
great.
The kernel is quite a bit bigger than system dejournality and it also compiles a lot slower.
So I was quickly looking for a bigger machine with a lot more cores so that my kernels could
compile quicker.
So it would be very nice if that I could replicate the setup very quickly to another machine.
And ideally I'm not too reliant on whatever the host distribution of that machine is because
well I work at Meta and we can get like very big beefy servers that have a lot of cores
that we can work on but they might also be running some old version of CentOS with all
the not all the latest tools available.
So ideally I still get those but on the big beefy server with the old Linux distribution
running.
Of course I want it all to be fast so that I have a quick turnaround time so I can fix
bugs or notice bugs, fix bugs, recompile everything and boot again without waiting
too long.
Like everyone knows the XKCD with like compiling and two dudes are fighting.
So I wanted to avoid that.
And then of course when you hack on the kernel these days it's not just the kernel that you're
working on.
There's very often some user space projects involved as well.
A good example for the file system service XFS tests which is a separate project.
I also wanted to be able to compile all those things and get them available in the virtual
machine so that I can run them.
So of course because I work on system D and we use Mako aside to do all of this for system
D because system D suffers from the same problems.
You also can't really test system D on your system because if it's broken then you can't
use your system anymore.
So Mako aside is basically my hammer and like kernel hacking is just another nail that I
wanted to slam in.
So what is Mako aside specifically?
It's a tool that the Linux puttering developed to simplify its hacking on system D. So he
had all the same issues so he developed Mako aside to fix it.
What Mako aside does is it builds you a Linux image.
So it invokes a package manager and then it installs packages.
It packages that up as like in one of various formats and then it allows you to boot it
either in a virtual machine or in a container and then you can do whatever testing you want
and when you're done you just exit the virtual machine and it's like nothing ever happened.
So Mako aside supports like it has a general execution flow.
So of course like we have CLI options, configuration all that.
We install packages for the distribution.
So this is invoking DNF apps, what zipper, Pacman for all the distributions that we support.
Optionally we set up a boot loader and all that if you're doing it bootable disk image.
We run various system D tools that are helpful to configure an image.
If needed we build an NNRMFS.
This is again when you're doing bootable stuff.
We generate what's called a unified kernel image.
This is this new system D thing that allows you to combine the kernel command line, kernel
image NNRMFS all in a single file and then boot that from that in UEFI.
Then we package up the entire thing as a disk image and then optionally of course you can
boot it in QM or container with this in the N spawn.
So how do you get started with Mako aside?
This is not like the kernel hacking specific stuff but this is just like if you want to
make a side image you specify which distribution you want, you specify the packages you want
in this case is in the NLINX and we're running on ARCH.
We have an auto log an option to basically automatically get a root shell in the virtual
machine and then you say I want to boot this in QM.
That gives you something like this.
So we support this for Debian, Santos, OpenSuzi, ARCH, Fedora and Ubuntu.
And there's a few other distributions but they're all derivatives of these.
So everything can be specified via CLI, the settings as you can see here but of course
we also have configuration.
So this is like the system, the init, any files and things that we all know and love.
So we more or less do the same stuff so you can also specify it all in the configuration
file.
Using Mako aside for kernel development and development in general.
So what I showed previously just in soft packages from the distribution of course that doesn't
really help us.
We want to build stuff from source either system D or in this case the kernel.
So you can specify a build script.
The build script is responsible for building your software.
Canonically we call this Mako aside.build.
So when you define that Mako aside will pick it up and it just contains the instructions
to build your software.
So either make for the kernel or mess on for system D.
You can specify build packages which are just the packages that are needed to run the build
script.
So compiler, build system and all that.
You can specify a build directory so that everything is cached.
This is important so that your incremental builds are fast.
With the build directory we have to build cached but we don't have the image cached yet so
we have the incremental setting for that which will install all the packages once cached
result and then reuse that on the next builds so that our image builds are fast as well.
And then we have various settings that you can use to configure the image without invalidating
the cache.
So you can add extra files for testing or to configure your shell in the image or basically
anything you might want that configures the environment to your liking.
You can do with the extra trees and the post installation script so that the testing environment
is the way you want it.
Whatever customization you want you can pretty much do it.
And then we have the runtime trees to basic which we use Fertile.ufsd then to mount extra
directories into the virtual machine so you can make the XFScast source code for example
available for running XFS tests or you can make your home directory available in the
VM if you want that.
Whatever you want with runtime trees.
You can modify the kernel command line and whatever way you want.
And we want to specify the output format as a directory so that we don't have to build
a disk image but we can just boot from the directory itself also using Fertile.ufsd.
Why do we want to do that?
Because it's faster.
It takes time.
And we're looking for this quick turnaround time so we try to make everything go as fast
as possible.
So make OSI kernel is really nothing more than a make OSI configuration in the separate
repository that's specific to hacking on the kernel.
So we have a build script for the kernel and then we have various other modules that are
all just build scripts for user space projects that are related to kernel development.
So as of this moment we have of course a module for the kernel and then we have other modules
for better FES procs because well I work at Meta and Meta work some better FES.
The Linux test project which I added for Christian and then some other testing projects like
block tests and BP filter which is a Quentance project for hacking on firewalls.
So I added that as well.
So you basically specify which modules you want and then all those get included.
So getting started with MakeOSI kernel more or less looks like this.
You clone the repository.
MakeOSI is pretty easy to install.
You can also install it from your package manager of course but it's a pretty fast moving
project so in this case we install it from source so you just clone the repository.
You sim link the script to somewhere in your path and then that's all you need.
You can then run it.
We download if you want, well by default for the MakeOSI kernel we download all the other
tools we need on demand.
So the only stuff it needs is Python and bubblewrap and of course the package manager and then
that's enough to get started.
Then we clone the MakeOSI kernel repository which contains the kernel configuration, a
specific configuration and then you can write a local configuration file that basically says
what distribution do I want to use to test or to use MakeOSI kernel.
So we support Fedora, CentOS and Dabian at this point but it's easy to add more.
The only thing that's distribution specific is basically which packages you need to do
kernel development.
So you just define the list of packages to build a kernel and to boot the system and
that's sufficient to add a new distribution.
So it would be very easy to add Arch Linux here as well.
And then of course finally we specify the modules and we specify where our kernel sources
live.
So this is what the build source is setting.
So your kernel can be checked out anywhere on your system and then you use the build
source setting to specify here's the source location and then the target directory where
it should be mounted when we run the build script.
So this should always be kernel, the target directory.
Of course the directory can be anything and it will be mounted in the right place and
then we run MakeOSI and it will do its thing.
So I hope this works with the internet here but I made a video.
This is with everything cached so otherwise it would take a little bit too long for the
stock but when we run QM we see the images cached and then we start running make.
Kernel build is of course cached as well otherwise it would take forever.
So not too much happening but we get a new kernel image packaged up.
Then we make OSI does its thing and then we boot and then you're running in a VM that's
running the kernel compile from source and you can do whatever testing you want and then
we shut down again.
So of course to build the kernel we need kernel configuration.
We ship a default kernel config in MakeOSI itself.
This is just with the minimal amount of stuff enabled to do to test various things and to
the necessary drivers to be able to boot in a virtual machine.
So we keep the drivers to a minimum and the features to a maximum.
Anything like that's related to kernel development can be enabled so that it's available and
then you can use it for testing.
We also enable a few debugging things so that it's easier to figure out what's going on.
For example also the kernel command line we configure it to panic on oops and stuff like
that so that when something goes wrong when you're testing you immediately see and you
don't have to go to the message to figure out if something went wrong with stuff like
that.
We also allow configuring to build a self test if you want that and specifically which
self test so you can specify targets or you can specify to skip specific targets.
For example the BPF self test because those take absolutely forever to build.
You can specify your own K config if you want so you don't have to use MakeOSI's default
one you can specify your own and the interesting way that we basically use this minimal config
file is by using the all death config make command which basically says take the config
file that we specify with K config all config use everything from that and set every other
option K config option to its default value.
So we specify what we want and we give everything else a default value.
And then finally while I said that MakeOSI can build an inner ramifest for you building
an inner ramifest is again more work which means slower which means slower turnaround
time so in this case because we're building our own kernel anyway we simply build the
virtual aero fests driver right into the kernel and that removes the entire need for even
needing an inner ramifest so we just skip that step completely.
As I already mentioned there's a few useful settings to like runtime trees and extra trees
to customize the image.
Another one that's useful for file system development is the QMU drives in the QMU arcs
setting.
So to add extra devices block devices to a VM with QMU you need both a drive which is
the host facing side of it and then of course a device which is the guest facing side of
it so MakeOSI can allocate the drive for you using a file that it creates itself on the
file system which then removes when the VM shuts down so that's what you can do with
the QMU drives and you can specify the serial or the drive ID you can specify the size and
you can specify all the extra QMU options you might want in this case we specify that
asynchronous IO should be done using IO U-ring and then of course you need to attach the
drive to an actual QMU device so in this case we specify an NVMe device and we give it a
better RFS serial and we specify that the drive should be better RFS which is the same
as the ID we gave to drive.
Like I said we can configure the kernel command line and if you want to do bootloader stuff
you might want to hack on the EFI stuff or stuff like that you can also specify that
we should boot in UEFI environment so that you can basically hack on the EFI stuff code
or anything related to that.
Well all this stuff I mentioned works like usually what you do with QMU is you have your
dash kernel argument and your dash append and your dash init RD which you use for kernel
development but when you start doing UEFI you might not have all of that available anymore.
Now what MakeOSI does is it basically sets things up so that even if you're booting in
a UEFI environment everything really works the same even though we don't directly might
not directly use dash kernel anymore we might be booting from a disk image we can still
like append to the kernel command line and all that it's all still all supported.
You can get some extra shells in the image as well so of course you get the serial console
but you might want extra shells so you can do that with MakeOSI SSH.
You have to also enable the SSH option to make sure that the image gets configured for
this but we do that by default in MakeOSI kernel.
There's a very complicated diagram here that basically shows how we implement this in systemd
but the interesting thing about what MakeOSI SSH is that you don't need your VM to have
a configured network to be able to do this.
So for VMs there's this alternative socket family which is called AFVSoc which allows
for inter-VM communication that doesn't rely on the network interface being up and running
and configured.
So using a bunch of new systemd features what we're able to do is at runtime provision
the virtual machine with your SSH public key so we can put it in the authorized keys file
for the root user and then if there's a VSOC device attached to the VM in the next systemd
release systemd you'll basically be able to automatically detect that a VSOC device is
attached and if so it will generate a socket unit that will run SSHD on port 22 of the AFVSoc
family and this allows you to connect to the VM over VSOC from the host without needing a network.
We can also do install a drop-in file for the SSH on the host configuration which SSH
support now as you can do drop-in configuration for SSH and we can use the SSH proxy protocol
to basically take possession of the UNIX and the VSOC host name prefixes so that you can use
those to connect to VSOC enabled VMs. So with all this setup you can basically do SSH VSOC
slash the VSOC connection ID to connect to that specific virtual machine all without going over
the network. So we don't use this stuff yet and may go aside we have our own version because the
systemd thing is very recent but we'll be moving to this in the future once this is available
everywhere. Running tests manually is all good and fine but you want to move from manual testing
to automatic testing of course so we also support this when you want to do automatic testing you
want to run the test and you want to get an exit status usually this is very simple with a process
you just run the process in your shallow whatever and you get the exit status from the kernel when
you run the test in a VM this gets a bit harder there's not really an easy way to get the exit
status of a process that's run in the VM and transfer it back to the host. If you're running a
directory from a directory with FURTAOFS you can just write some files to the directory and retrieve
all the information that way if you want but if you're doing testing from a disk image then you
have to mount the disk image once the VM shuts down to access the information and of course to
mount the disk image on Linux you need root privileges so you have to start entering your
password and stuff so it all becomes a bit more complicated so what we added instead is a way again
using the VSOC stuff to have the VM when it shuts down and you use these two unit settings
success action equals exit and failure action equals exit in the systemd unit when that
unit exits the VM will also system they will also shut down so the VM will shut down but it will
use the SD notify protocol which is a some systemd thing to send notifications to send the exit
status over VSOC from the VM to the host and make a way so I can pick up on this and exit with
that exit status so seems pretty trivial to get the exit status but there's a bit of work involved
to get it out of the VM and then of course what we also want is the locks so this isn't actually
upstream yet but we're looking to have add another forwarding mode to systemd-journally so that again
using VSOC it can forward locks over an AFV socket and then we can listen on the host receive those
forwarded locks with systemd-journally remote and write all the locks to a local directory and
that means we can access the locks on the host without needing root privileges we don't have to
mount the image we just have to the locks locally we run internal cuddle on it and we can access the
locks see what went wrong with the test and debug further of course I'm not the only project in
the space we do have some competition so the latest product in this space is Furtmianji so I thought
I'd mention it as well because I don't want to claim everything for myself like there's more
tools than just to make a side kernel so definitely take a look at Furtmianji as well Furtmianji is
very focused on kernel development so it has a lot more options to for example use the kernel from
the host and various other options but it's very specific to kernel development it also has its
own in its system that runs in the VM which allows it to boot very fast but you don't get all the
you don't get a regular Linux system like you would when you well I mean I don't want to like
say that system D is regular but you don't get system D so if you wanted to start doing stuff with
devices or something like that you definitely won't be running so it gives you a bit more limited
environment so depending on what you're doing one or the other might be more useful yeah that's
more or less it on the comparison if you want to know more about this like come talk to me
afterwards or something and I can say a bit more about the differences between it too of course I'll
end with some reactions from users so of course like Christian already said he was using it so
it's very nice as well his reaction to it and then Joseph from Meta the better FS fast
system maintainer is also using it so and he's also very happy with it so I hope it can be more
useful for more than just them so please give it a try and I'm happy to answer any questions or
implement more features if needed thanks for listening
hello thanks for the thanks for talk or two quick questions so one what about cross compiling
so that works we don't like we don't have like a specific environment rival in the
build script yet that allows you to specify cross compile but we can simply add that but I
already tried it like just by hacking the build script and saying cross like changing the
architecture to compile for arm 64 and that works Christian also or I'm not sure who added it but
we also had the support for compiling with LLVM if you want to and the second small question
maybe I missed that because I was late for the talk what gets into the unit MFS so what about
from the all in the last half so make was I kernel by default doesn't boot with an around
fast when we do the third I your FAS stuff if you do a disk image then the inner MFS is built
with make or side cell so I actually have another talk about this in the distributions
there from but yeah we just install regular RPM packages or whatever into the inner MFS and
then by default we just copy all the girl modules and firmware from the host but we have a suite of
settings to basically include and exclude whatever you want and we also have like the stuff that
in the ramfs generators to to include everything that's loaded on the host if you want that so you
can configure a bit which firmware and then drivers you want we'll also make sure that we when if
you specify these drivers modules to be included we pick up all the pen the dependencies as well so
we make sure that all that is set up correctly and included I'm using the inner drum FAS stuff
like I'm building full images and I'm not using the QMU part I'm using a different virtual machine
manager for this and it works really nice because that was the biggest the biggest thing for me that
it wasn't easy to build an inner drum FS especially if you want to do it destroy independent which
was really annoying it's this also useful if you want to run a mainline on a new device
where there's only some heavy patched in the window kernel so you want to test if your drivers
work and so you need to you want to test it but don't want to touch any non-volatile memory just
started somehow without like this fast boot boot or something like this
sorry I don't think I completely here to her to question so this was all about mainly so if you
want to test if the kernel works on a new device so where there's only vendor kernels are known to
boot so you want to you don't want to destroy the user space there but first test it there before you
touch your space and you want to boot it only from ram can it also be used in that way so it's very
focused on virtual machines at the moment while make or size kernel specifically is but make or
size can build your images that you can then deploy on another device so like you can run the stuff
that is produced by make or sign on your laptop or you can flash it to your disk and it will it will
boot but specifically without destroying the user space we don't have anything specifically to make
that work you could take the kernel produced and then keep the user space the same but it's not
something I've really I've looked at before so it probably won't work
all right I think if there are no more questions and thanks for your talk thanks for the tool
you