At our final talk for this session, we have Pierre here. He's going to discuss about
Loup, a tool that he and we have been using to measure compatibility for different OSs.
Pierre, you have the floor. Thank you, Resvan, and thanks everyone for
attending my talk. This is joint work with a bunch of colleagues and students, including
Hugo, my PhD student. He's a key player behind his work. I'm just, you know, getting all the
medialization, but he has built all this stuff, so all the credits go to him.
So in this brief talk, I want to speak a bit about application
compatibility for custom operating systems. So I guess most of you don't need to be convinced
that we still need custom operating systems today. When I say custom, I mean both, like
research operating systems and prototypes operating systems from the industry, right?
The thinking that Linux has solved everything is not true, in my opinion. We still need things
like Unicraft if you want to go fast, or if you want to specialize like crazy, we still need things
like Rusty-Armet if you want security, or SEL4, so we still need custom operating systems,
and the thing is with these operating systems, they're only as good as the application that they
can run, right? So compatibility is key. Compatibility with existing application is extremely important.
If you want to build a community, you want your user to go to your website, compile your custom OS,
and then try some of their favorite applications, or try some of the highly popular applications
in a given application domain like Enginings or Redis for cloud, if you want to attract
sponsors or investors, or even if you, like me, are scientists, you want to gather some early
numbers to make a publication, well, you need to do that on standard applications, right?
So compatibility is important, and another argument would be like how many times did you hear the
one POSIX spoken today, right? There were some slides, there was POSIX like three or four times
written in a single slide. So compatibility is important, and it can be achieved in a few different
ways as we have seen with Simon. But one important thing to note is, in my opinion, porting is not
sustainable. So porting is what many of us do. We build a custom operating system, and then we take
Redis, and obviously it doesn't work as is with our operating system, so we modify Redis a bit,
we disable some features because we know that they make our OS crash, and then we have Redis
like a version customized for our operating system. This is not sustainable because you can't
maintain like a branch of Redis for every operating system out there, right? In the long
term, it doesn't work so well. So porting also basically means that you as a OS developer,
you ask the users of your application to make some effort to the application developers,
they need to make some effort to be compatible with your operating system. This doesn't work,
nobody is ready to make that kind of effort. Maybe if you give them 10x performance speed up,
but this is unrealistic. So what you want to do is, once again, in my opinion,
as an OS developer, you want to provide compatibility as transparently as possible.
And this means you emulate a popular operating system, for example Linux, or a popular abstraction
like POSIX or the standard C library, and then you can be compatible at three different levels.
The first level is source level or API level compatibility. So you ask the users to compile
their application code against the sources of your kernel, in the case of a unique kernel.
So this is, you're still asking some effort from the users, right? In many scenarios,
you don't have access to sources, right? If you have proprietary binary or pre-compiled
binaries, well, you can't have source level compatibility. So it's not perfect and binary
compatibility is generally a more, let's say, pure version of compatibility. And there are many
two ways to achieve it. You can do that at the level of the standard C library, like OSV. So you
will dynamically link your kernel plus a standard C library against your application, compile as a
position independent executable or as a shared library itself. This is great, but if the application
is making directly system calls to the kernel without going through the standard C library,
well, once again, it doesn't work, right? And as a matter of fact, I have counted
more than 500 executables in the Debian repository that contain the C score instruction, right? So
they make C scores directly to the kernel. They don't go through the C library. Go, for example,
is making most of its C score, put directly to the kernel and not through the C library. So what
you want to do is to be compatible at the level of the system calls. So your kernel needs to
emulate the C score API that Linux is providing. This is the most transparent way of achieving
compatibility. Now, this is scary, right? Linux has more than 350 system calls. Do we need to
implement them all? Will we be and aren't we going to reimplement Linux by doing so? And some of them
are extremely scary by themselves, right? You have like hundreds of IO controls and each of
them probably require its own implementation. The Linux API even goes beyond system calls. You
have things like slash proc slash devs that are, you know, used by many applications. Like the
first thing a muscle binary does when it runs is to look in, I believe it's slash proc or slash
this to get the size of the terminal, right? So you need to have emulation for this part of the
API too. And this, because it seems like a big engineering effort, this creates, it hinders
the development of custom operating systems. So this is inspired by the keynote by Timati Roscoe
at ATC and OSDI 2021. We looked at all the papers. So these are top tier operating systems
conferences. And we look over the past 10 years, over a total of more than 1,000 papers,
how many were about proposing a new operating system as opposed to things like security or
machine learning. And among them, how many were just hacking Linux versus proposing an actual
operating system implemented from scratch. And the numbers are similar to what we saw earlier,
right? You have just a very, very, very few papers proposing a new operating system because
it's a significant engineering effort. And part of the effort is to be providing compatibility
to an application like Apache already to get a few numbers at the end of the paper, right?
So this is the problem. Now, the particular problem that I want to talk about is how
I'm sure several people in this forum have attempted to build some form of compatibility
layer for an operating systems. And we are all kind of working on the same thing in parallel
with some form of ad hoc processes that may benefit from some optimization. So I've just listed
here a few projects that have a Cisco level binary compatibility layers, but actually there are many
more. And from what I understand, it is a very organic process. So first of all, it is application
driven, right? People have a few sets of application in mind that they want to support. If you are
doing cloud, you want to support the user's suspect, ready, Apache, whatever. And the process
basically looks like that. You take an app, you try to run it on top of your operating system.
Obviously, it fails. You investigate. You're like, oh, I'm missing the implementation for this system
code. So you implement whatever operating system features are required to fix that particular
issue, rinse and repeat until the app is working. And then you go to the next app. So it's a very
intuitive and organic process. So when I built the Armitage, this is exactly what I was doing.
So something that comes to mind is, can't we have some form of generic compatibility layers
that we could plug? Like something a bit like New Lib that would provide a generic interface.
And I believe it's not really possible because most of this implementation to support the system
code is very specific to whatever operating system you are using. And it's not clear if a
generic compatibility layer can be achieved. But can we still somehow optimize that process?
Some have tried static analysis. So they take the binary of the application they want to support
and they look, okay, so what are the system codes that are made by these applications?
So this has been done in the best paper in Eurosis 2016, analyzed all the binaries from
Ubuntu, I believe it was 14 or four repositories. And they concluded that every Ubuntu installation,
including the smallest one, require more than 200 system codes and 200 IO controls,
five controls, and PRCTL codes, and hundreds of 2.0 files. So this doesn't help. It is still
quite scary. It still represents a gigantic engineering effort. But do we want full compatibility
with Ubuntu installation? In the end, especially in the early stage of the development of an
operating system, you just want to get a few applications up and running. And do you even
100% compatibility? When I write a paper, I don't really care if everything is stable. I just want
to get some numbers. So isn't there a better way? And obviously, maybe you think about, yeah,
let's do dynamic analysis. Let's run the applications that we want to support.
We send them some input that we want to support, like I'm running
engineering and I'm submitting some HTTP for something like that. And then we trace the system
codes that are done. So this is going to give us a subset of the system codes that can be identified
through static analysis that has a tendency to overestimate. So with this trace, the engineering
effort to support an application and a set of input is a bit lower. But it's still not a panacea
because it's not taking into account two things that we do when we implement compatibility layers.
So this is my code. Don't judge me. One thing that I did with Hermitux was at some point,
it was an app that was calling MNCore to check if some page of memory wasn't swapped or not.
It has actually, you know, there is no swap in most unique kernels. So it really didn't matter
to implement this. So you know this means operation not supported. So stopping a system
code is just saying, yeah, we don't support it. And you cross your finger that the application
has some kind of fallback path to do something else if the system code fails. And it works
in some cases. And then we can do something even more nasty. Don't judge me again. You can fake
the success of a system code, right? Surprisingly, in some situation, returning a success code,
even if the system code doesn't have any implementation in your operating system,
it's going to work in some cases. You know, I'll tell a bit more about why this works sometimes.
So stubbing and faking lets you implement even less system calls than what you would trace with
it trace. So in the end, you know, if you want to support an app or a set of application in your
custom operating system, the amount of system calls that you actually need to implement.
So obviously, it's smaller than the entire Linux SQL API. Static binary analysis will,
on the binaries of the applications you want to support, will identify a subset of that.
Still pretty big. It's an overestimate. Source analysis gets you more precise results.
But it is pretty hard to achieve. And it is still overestimating. S trace will give you,
once again, a subset. Things start to look better. And among these trace by S trace,
you actually don't need to implement everything. You can stub and fake some of this SQL. So
can we measure that? Yes, with loop. So loop means magnifying glass in French. It's a tool
that was built by Hugo, my student, and it's some kind of super S trace that is measuring
the system calls that are required to support an application. And that can also tell you which
one you can stub and which one you can fake. So we used it to build a database of measurements
for a relatively large set of applications. And with loop, if you give me a description
of your operating system, basically the list of system calls that you already support,
and you give me the list of applications that you would like to support, we run them through loop
and loop can derive a support plan, which basically will tell you, okay, for this set of target
application. And given the set of system calls that you already support, what is the optimized
order of system calls to implement to support as many applications as soon as possible? Okay,
so I will give you an example of support plan by the end of the presentation.
So from the user point of view, loop needs two things to perform its measurement on a given
application. You give it a Docker file that is describing how you want to build and run the
application for which you want to measure the system calls needed. And optionally, you may need
an input workload. Think about a web server. It's not going to call many system calls until you
actually start to send requests to it. Loop will instantiate the application, launch it on a,
you know, standard Linux kernel and analyze the system calls that are done and with a few tricks
we'll be able to know which ones can be faked or stubbed. The results are, it's basically just a
CSV file for each system call that is made by the application. Can it be faked? Can it be stubbed? Or
does it require a full implementation? We start that in a database and later, so, you know,
we populate the database with as many measurements as possible. And this database can
given the list of these calls that is already supported by your operating systems, give you
like some form of optimized super plan given which of the applications you want to support.
Okay, so how does it work? When loop runs the application, first it does a quick pass of S-Trace
to measure all the system calls that are done by the application and then for each system call
that we identify, we use SecComp to hook into the execution of each of the system calls and
rather than actually executing them through the Linux kernel, we emulate the fact that
the Cisco is stubbed, so we just return EnoSys without executing the Cisco. We can also emulate
the fact that the Cisco is faked, we return zero. And then we check if the application
works or not following the stubbing or the faking of this particular Cisco. And then we do that
for each system call that we have identified with S-Trace. How do we actually check for the success
of the execution of the application? So we identified two types of apps. Some we call them
run to completion. There'll be something like FIO when you know you start FIO, it runs for one
minute and then it exits outputting some kind of some stuff on the standard output. So with run to
completion apps, we run the app instrumented with loop, we check its exit code. If it's different
from zero, we consider that the run was a failure, could have been killed by a signal or things like
that. And we can also run a script optionally in addition to that after each run of the application
to check its standard output. We can grab for error values, we can grab for success printing,
something like, you know, 50 requests per second have been achieved. The files that may have been
created by the application and so on. And then another type of application is client servers.
So with client servers, we run the app instrumented by loop and in parallel we run a workload,
could be WRK, HTT path, the Redis benchmark for Redis and so on. And we check the success of both,
we check that the app doesn't crash, generally servers are not supposed to exit. So we check that
it doesn't crash and we check the success of the workload. Like, you know, if Redis benchmark returns
something different than zero, probably something went wrong. And then we are able to see, okay,
so I'm currently trying to stub the read system call, is the application succeeded or not?
So really the database, let me check the time, okay. And we analyzed the results. So these results
are made on a relatively small database of about 12 highly popular, sorry, 15 highly popular cloud
applications. So this is just a subset. So what you have on the y-axis is a number of system calls
that are identified by static analysis in purple on the binary, on the sources in yellow. And then
dynamic analysis. And we run for each of these applications, both the standard benchmarks,
that will be Redis benchmark for Redis, WRK for engineering, and so on. And we also run the
entire test suite. So the key idea with the test suite is if you, you know, support, I mean,
if you measure what's going on during the entire test suite, you get a very good idea of what are
all the possible system calls that could be done by the application. Obviously, you need to assume
that the test suite has a good coverage, but it is the case with these very popular applications.
And, and what we see is, first of all, you know, static analysis overestimates. This is not very
surprising. The amount of system calls that is identified by static analysis is relatively high
compared to what we get with dynamic analysis. And if something interesting, too, is that the
amount of system calls that can be stirred or faked, so the grain bits on the dynamic analysis
pass, it is actually quite non-negligible, right? So, so what this means is that if you want to
support Redis with a Redis benchmark, where binary level static analysis tells you that you
should implement 100 system calls, if you just want to run the Redis benchmark to get, you know,
performance numbers for your paper, you actually need to implement just 20, right? So that's what,
like, divided by five, right? And if you want to pass the entire test suite of Redis, you need
to implement about 40. It's still like half what static analysis is telling you. So it's kind
of a message of hope, right, for building compatibility layers and for developing custom
operating systems in general. So, yes, static analysis overestimates a lot of the engineering
effort to support an app. And even naive dynamic analysis does measure much more these calls
than what is actually required if you know that you can stop and fake these calls.
Another view at these results can be seen here. So for each of the system calls, you know, zero
is read, one is write, two is open, I guess, and so on, among our dataset of about 15 apps,
how many of these apps require the implementation of the system calling question, right?
And then so you have here the result for static analysis at the binary level.
At the source level, this is S trace without counting which system calls you can stop or fake.
And this is what is actually required. So if you consider that you will not implement what you
stop or fake, this is what you actually need to implement. And as you can see, you know,
it's much, much, much, much less engineering effort versus what static analysis is telling.
And why does stopping and faking work? So here you get some code snippet from Redis.
So if you stop, get our limit, the C library wrapper will return minus one.
And as you can see, Redis will actually fall back on some kind of safe value, you know,
so I'm not able to understand the maximum number of files that I can open. So I'm going to fall
back on 100, sorry, 1000. And the fact that faking works is actually that you have quite a bunch
of system calls. So this is for each system call and each app in our dataset, what is the
percentage of apps that are actually checking the return value of the system calls. And some system
calls are almost never checked the return value. It kind of makes sense, right, when you see this,
why check the return value of close. And this is why, you know, faking work in many cases.
Another question that we asked is, okay, so when you speak about providing binary
compatibility and you don't do porting anymore, basically, all the effort of supporting apps
is on you, the operating system developer. And this is how it should be, in my opinion, but how
much effort does that mean in the long term, right? So we had a look at versions of Redis
and Jennings and Apache over the last 10 years and what, you know, what are these calls that
actually needs to be implemented in purple. And we saw that this number does not change very much,
right? So once you make an app and you make it work, it actually means that you need to
keep up to date with the most recent version of this app that are coming up, but it doesn't
necessarily mean a very big engineering effort either. And these are the support plans. So we
had a look at Unicraft, Fushia, which are some operating systems that have already a
relatively good support for a good number of system calls. And we look at Kerala,
so Kerala is another Unicernel written in Rust. And it's very, I wouldn't say immature,
but it doesn't have support for a lot of system calls. And for a set of 15 apps that we had in
the database, we derive a support plan. So for Unicraft, for example, in its current state,
it's already supporting most of the apps of our data set. If you want to support an additional
app, what you need to do is to implement system call number 290 and stop these, and then you'll
get memcached. And next, if you implement this syscall, you get H2O, and then you need to implement
these two syscalls, and then you stop that, and you get MongoDB. So same thing for Fushia and
Kerala. Obviously, it's a bit more interesting because this one doesn't support many applications
out of the box. And I believe I have time to do a quick demo. I'm going to do it real quick.
So I'm going to do a test with LS, which is like the simplest test because we don't have a lot of
time. In the Docker file, I just copy a test that I'm going to show you, and then I call like the,
this is kind of the top level script of loop with a few options that don't matter that much. And I
say, okay, the binary that we are going to instrument is slash bin slash LS, and this is the
parameter. So I'm going to do LS slash, and we are going to check if it works or not with every
possible syscalls that can be invoked by LS. And the test, which should be there, the test that
we are going to run after each execution of LS to see if things have worked. So this share script
will take the standard output of LS as parameters, and to make things simple, I'm just checking that
LS actually outputs something, right? I'm doing LS slash, so something should be outputted. If
nothing is output, there is a problem. And keep in mind that loop is also checking the return
value of LS itself. So, okay, so I'm launching loop like this, so it should work. So what happens
under the hood is that we build the container that we've seen the Docker file for. We are starting
two containers in parallel. Each one is running a full set of tests trying to stop and fake all
the system calls. And we use this to check for differences between the replicas in case there
is a problem. Most of the time, there is no differences. So it takes a bit of time. And then,
okay, it's done. So, if we go to the database, so we have now much more than 12 apps. And if we go
to LS, the most interesting result is this CSV file, which contains, for HCC call, 0 being read,
1 being write. Is it called by LS or not? Can we fake it? Can we stop it? Or can we
both fake and stop it? Or it's more like, does the application works when it's fake? Does it
works when it's stubbed? And does it work when it's both fake and stubbed? And as you can see,
some CSV calls, like 11, I don't know which one it is, can be both stub and fake, same thing for
12, same thing for 16. Some CSV calls, like this is read, for example, it is called, but you can't
stop or take it, which kind of makes sense. LS wouldn't work if it can read. And yeah, that's
pretty much it. So briefly, what we are currently working on is some more fine-grained measurements.
Some system calls have kind of sub features, like a lot of programs require at least a map anonymous
for a map to allocate memory, but not really to map a file. So we are looking at, you know,
checking which flags can be stubbed or fake and things like that. And we are also looking at the
virtual file system API. That's it. So building compatibility layers is important for custom
operating system. It seems a bit scary, but actually, it's not that much engineering effort.