Can you hear me?
Okay, fine.
Stefan, please.
Yes.
Good morning.
Hello and welcome to my talk.
I'm Stefan Krakowski.
I'm a genote developer since 2009.
And today I want to present you how to transplant Linux kernel
drivers into the genote OS framework much more faster
than before and hopefully precisely.
So let me start with the motivation behind this.
Of course, you might ask why we use monolithic kernel drivers
when we talk in a microkernel death room.
Of course, there are good reasons to implement drivers from scratch
and we also have several drivers which are written from scratch
but the ever increasing complexity of modern hardware
for single devices but also for the pure number of devices
inside a system one ship is not easy to handle for a small team.
That's the one reason.
And often it's poorly documented even if at all.
And we also have hardware bugs inside
and you have to find those bugs
because they typically are not documented
and then you have to find work around for it.
And all of this is mostly part of the Linux kernel
and you can reuse it because it's free and open source software.
And so to sum it up, it's simply an economic decision.
So if you want to enable a modern device
and you have limited time, then this is the way to go.
Okay, we have collected a lot of experiences in the last decade
to port drivers from Linux to the Geno.S framework
and you have in general two extreme approaches
and the reality is somewhere in between always.
So either you use just the pure driver code.
What I mean by this?
I mean code which directly interacts with the hardware
by writing to some IO registers
or by setting up a DMA transfer or something like this.
And in that case, you of course have to implement
each Linux kernel function that is called by this driver code.
But the good thing is you don't have to implement the whole semantic
which the original function is implementing.
You only have to match this single driver needs.
This leads to a more low complex function than the original ones
and in sum to a more minimal Linux kernel driver.
But of course you cannot share this emulated code
in between different code bases.
So if you have not only one driver but several,
they will have slightly different semantic needs.
And so reusing the same emulation code might be a problem
and therefore the whole effort,
if you don't only port one driver but several ones,
can increase if you ever and ever again have to implement
the whole emulation code base for the driver.
And of course you need the actual needs of that driver.
So you need a deep knowledge of the driver itself.
On the one hand, this is the one approach.
On the other hand, you can use as much as possible
from the original code base.
Thereby you might gain more or less the same runtime behavior
than the original one.
And you can of course then better share resulting emulation code
because it's already stressed by this whole bunch of code running on top.
Thereby you get less manual work to do for having more than one driver.
But of course the code base for the single driver increases
because you have much more of the original Linux kernel.
And if a problem arises, then you have to know a lot of the whole Linux kernel itself
because it might be in the timing subsystem and whatever.
You can name it.
So in the recent past we were more on this side,
on the taking the pure driver approach.
But the high effort for each driver was also leading to the situation
that you keep your old code base, that you are not that good
and maintain the code and getting a new kernel version and driver updates.
So, and at some point there was a need for action.
For me this was at the beginning of the pandemic situation
when I was trying to enable the display engine of this device,
which is M&T Reformed 2 from M&T RE,
a small company from Berlin, so a completely open hardware.
And yeah, I tried to enable the display engine
and it includes some NXP IMX8 SoC.
And we already had a driver for this
because a colleague of me, he enabled in three months
on the early evolution kit the HDMI connected display.
So this was one part.
And then another colleague of me wanted to have a touch screen
which is connected via DSi connector and not via HDMI.
And again he had to spend three months into this work
because on the one hand there are more devices involved now.
On the other hand there you had all this bureaucracy
for device tree management and it was all hard coded
for this first use case of using HDMI for the specific board.
So there was a lot of manual tweaking to do to enable the touch screen.
And then I wanted to, I thought, yeah, I don't have to do this,
someone else did it.
And now we use it for the M&T reform for the panel
because it's also connected via DSi.
But actually there's another device in between
and EDP bridge in between the DSi connector and the panel.
So yeah, I had to do work again and then I recognized,
oh no, the code base we used for porting
is a different one than the one of the M&T reform
and it's a totally different kernel version.
You have to back port stuff, you cannot correlate it.
No, I give up.
So that was a turning point for me to start a new way of porting.
And of course it was not only me,
but we had a lot of discussions formerly in the kitchen,
you and coffee breaks, what we want to change.
And so number one requirement for the new approach
was to reduce the manual work for tailoring a driver-specific environment.
And we wanted to meet as close as possible
the original semantic of the driver
so that whenever you change the context,
like with this display engine, it just works.
You don't have to do much more.
And because formally we all,
at some points we have the impression
that you cannot be deterministic in knowing
when you will finish your porting work
because when some problem arises,
you could not correlate it to the original runtime often.
So it was somehow hard and we wanted to change this.
So it should be an easy way to correlate it to the original runtime.
And last but not least, we wanted to share more of this resulting emulation code
which is more semantic complete
so that we can maintain the code better.
Okay, so this is the story beforehand
and now I come to the actual work.
So I want to introduce you to this approach
for those of you who like to port drivers to Genote
or like doing the same approach somewhere else.
So we typically start now by configuring
a minimal executable Linux kernel.
Let me just call it tiny kernel, so to say.
So you have to do some manual work here.
You have to use the Linux kernel build system itself.
It has a tiny config, some small configuration
which is at least compilable for your architecture,
but it won't run any device.
And then you just enable certain configuration options
and of course you have to find them by looking at the configurations.
And in the end, this might take some time,
but in the end you will have something which you can correlate laterally
if you run the driver in your ported environment
and you want to look at why doesn't it work,
then you really have a minimal Linux kernel
which just drives this device and this is the first thing to do.
And another aspect of this is that you gain
a minimal kernel configuration for your codebase
which just calls those kernel functions
that you really actually need to drive that device.
So you don't have to emulate that much.
Okay, kernel configuration is only one part.
If you take an ARM device today,
then you have of course these device trees
which name what kind of devices you actually have in hardware
and which also contain additional driver information.
So it's a bit of configuration is also inside of these device trees.
And this is the device tree for the M&T reform.
You see it's quite complex.
So you have to identify what kind of devices are interesting
for my tiny kernel to execute those.
And this is again some work to do, some manual work to do,
but at least you start to know more about the dependencies
of your hardware.
And we have developed some tooling for it.
So you can, this is a small tickle shell script
which pauses the device resources
and then you can name device nodes that should be extracted
and it will take them and the transitive closure
to give you something like this.
And then you can take that device tree of course
with your tiny kernel and start it
and it will just drive that device.
And we also take that for our own ported drivers
as input value.
Of course you won't implement everything which is seen here.
So powering, reset pins,
IEQ stuff like GPIO or something like this
would be part of other drivers in the system
like the platform driver or some dedicated GPIO driver
in the GenoDOS framework.
So those highlighted ones are the ones that we actually need
for porting.
And this is the starting point for you
to identify the first compilation set that you need.
So each of those device drivers
has some compatibility string
and those are used in Linux to identify
the concrete driver of the Linux kernel.
And so you can take those strings
and grab in the Linux kernel sources
and then you get something like this.
So you have your first compilation set units
and you can put them into a make file,
into a build environment
and then we combine it with the unmodified Linux kernel headers.
So we take the original include path of the Linux kernel.
Formally we always define the whole definitions
you needed by hand.
So this was a lot of work to do.
I would say initially the most work you had to do.
And now we just take the original Linux headers
and then you can just compile those compilation units
you already have seen.
So it's really a work which is done by this.
But of course there are some exceptions.
So we had to tweak some headers.
We shadow some few headers to prevent the system
from trying to enable, disable interrupts or something like this.
And especially to define init calls in the Linux kernel.
So each subsystem in the kernel including any driver
has some init call definition
and those are the order of the init calls is important.
Even if you have one init call priority level
there are dependencies in between the different compilation units
and they are solved by linking order.
So the Linux kernel uses some weird linking magic
to put them all into one order
and later when starting the kernel it takes that order.
So we didn't want it to infect our linking script with this.
Thereby we have built some tooling again
which uses this tiny kernel you built in the very beginning
and just extracts the order of the init calls
and puts it into a header
and you can just include it in this built environment
and then you run and it will,
the emulation code environment of us
will just call the init calls by the correct order.
So when we do all of this
then you of course get a lot of undefined references
for all the functions which are not implemented yet.
And this is a lot of error messages from the compiler.
So we made a small tool to identify those undefined symbols
help you to identify the original compilation unit
which implements them
and then you can try to find a correct setup for this.
And I want to show you this shortly.
Okay, so I've prepared some makefile like here.
So here you see the compilation units we identified.
There's some inclusion of the general emulation code base
and if you now use this tool
it will build the target which you name
so it will try to build the driver
and it will collect all the undefined symbols
and here it just shows you the symbols
and the overall count of the undefined symbols.
Typically you can also have what I said
the compilation unit which is responsible
but I've skipped this here
because on this machine it's a bit slow.
So we can now identify, okay, there are symbols
for DRM mode which we want to solve and we see, okay,
let's try to add the original one.
Oh, sorry.
And yeah, you just run the tool again
and it will show you in a few seconds.
So on the PC this is quite quick
but this is just one gigahertz or whatever, I don't know.
So it's a bit lame
and it has to recompile the driver of course
in the background.
Okay, and now you see it's seven symbols less
and in the end I think because of the time
we will skip this, in the end you can generate
with the tool the missing symbols
and it will give you per function
the correct declaration of the function of course
and it calls a function which gives you the backtrace
till then and just loops endlessly
so you have a no returning function
therefore you don't have to get a valid value back
or something like this.
So if you now take the driver it will link,
you can start to execute it
and you will always get the point
where something is not implemented yet.
Okay, so let me just switch back.
So, okay.
So this is the overview of the APIs involved.
I don't want to explain them in detail now
but what you should take from that is
we have a very strict layering
there is this layer where there's only C
and assembly code which is actually the Linux kernel code
and the shadow copies of the Linux kernel code.
Those are the only ones which can include Linux kernel headers
and then you have this emulation code base
which is just the C abstraction
for the Linux kernel code above
and then you have all this C++ stuff from us
which abstracts the genome services
and the concrete driver services.
And the good thing is you have those abstractions here
from the device services
and you have their Pondong here
and then if there's one, let's say for an Ethernet class
you can just reuse it.
So if you already have this in our emulation code base
you just need to implement
or port the concrete driver
but you have all this glue code
which connects with the actual APIs and services
it's always there.
Okay, so now let's see this in practice.
Okay, I just skip this here
and once you shut down the Fitching Machine Monitor
Okay, so you actually see the whole time genome in action
and what you see here on this mount reform
is actually everything the device brings with it
except audio.
So all other drivers are already in place
and this also is valid for the GPU for instance.
So here you can see the GL mark demo again.
Those of you who have seen Normans presentation yesterday
already knows it.
So I think I run out of time.
No, you have five minutes.
Okay, good.
If you have any questions about two more minutes, it's okay.
Anyway, I think this is enough.
I've had more or less everything runs on this device.
Yeah, so what are the results?
So this is the list of drivers we have ported within one year now
besides of course doing other stuff.
So we don't just port drivers all the time
but this is really a significant change.
So we have taken new drivers for our whole x86 code base
for instance.
And you see a lot of ARM drivers for the pine phone
and for the MNT reform are also,
we're also added like the Mali and the Vante GPU for instance.
And this was done by a very small team
and we also had some architecture independent porting
of wire guard.
So something which doesn't even use any device at all.
Okay, so in numbers, the initial driver porting,
so nowadays to compare to the initial approach,
we have something like 15% of the time that it takes to do this.
Of course it's a bit hard to measure
because we don't track all the times we do in spending porting work
but approximately this is the number.
And this is especially because of this tooling
which reduces the manual work.
Of course you have to find semantic backs
but here this tiny kernel correlation helps a lot.
So you can instrument the original code
then just run it on original Linux and on your ported code
and you can see the difference.
Driver updating, we also did this within that year
because the first port was done for this display engine
and then there was a new version available,
two kernel versions later and we made an update.
So it's significantly faster than the initial driver port,
of course.
And the drivers meet a better all purpose.
This is what I meant with,
for instance it took one day to enable the HDMI connector
for the M&T reform once the panel worked.
So it's much better matching the different contexts.
And of course there's something bad on the other hand,
so the code base for a single driver
explodes like two or three times more than before.
But on the other hand, the code to maintain by ourselves
decreased to the count of 20% than before.
Okay, so I think that's it.
If you want to read more about this,
I can reference this book, the second genote book
about how to enable a platform.
There's a lot of this stuff already written by Norman.
And we have also much details
in different genotian block articles.
So thank you for your attention
and I'm open for questions.
Thank you.
Thanks, Stefan.
Any questions?
Yeah, please.
Yeah, first of all, awesome work, by the way.
Thank you.
Linus is known for not having a stable driver API.
I think there's a Linux developer from Red Hat who once says
we do not do hardware extraction layers in Linux.
I did say that the initial port is the hardest
and then it's a lot less work maintaining back ports going forward
but there's still some work involved.
So I was wondering, wouldn't it be less painful
to, for instance, support drivers from BSD?
Because I'm not mistaken,
we have a more stable hardware extraction layer.
That might be probably the case
but actually we want to have this first argument,
Linux runs on all kind of hardware
and all kind of different situations.
You have, for instance, we have a BSD ported driver for audio
but for today's Intel HD audio devices,
it's somehow, yeah, that device might work
but on that device, the microphone doesn't work
and that device doesn't, this and this.
So it's more about we just want to have the functionality
and therefore we need to look at this.
And of course, we were not, yeah,
we didn't like to get kernel experts, Linux kernel experts
but now we had to do it and, yeah,
if you once dived into it then maybe we just take that advantage.
Thank you.
Another question here for Malik.
Hi, great talk and so I was just wondering,
I wonder if I might be able to use your tooling
to introduce NVMe driver into OSV
but I also wonder if maybe similar approach could be used
to also port file system drivers into operating system.
Like OSV is missing EXT to driver
and I wonder if I could do something like that.
I'm pretty sure you can.
I mean, we already used this RAM kernel approach
from Antikanta who was also in the step room in the past years
and the support BSD port of the protocol stack
and we also used the Linux IP stack in the past from Linux
and we will of course use this approach again
to renew that version.
And there's, as I said, we already used WireGuard for this
so something which is not at all connected
to any device driver code.
Yeah, it's possible.
Okay, thank you.
I have a question on licensing.
Is it okay because GPL, BSD?
It's all under GPL then only, of course.
Each driver report has to be under GPL.
And there's no problem with having the link together.
I'm not sure about the license of GNode.
No, it's not a problem because this code then is only GPL code
and GNode itself is also under GPL.
It's possible.
Thank you very much.
Okay, thank you so much, Stefan.
Let's take it again.