Our next talk is by Stefan, he's the project leader for LexD, a container manager, a former
teammate of mine as well, and he's going to talk about safe containers through system
call interception.
Hello.
It starts working well.
Thanks, sir.
All right.
So, you can edit the intro.
I'm Stefan.
I work at County Call.
I'm the project leader for LexD, LexEFS, and a bunch of other stuff that we do, effectively
system container guide.
And, yeah, we're going to be talking about system call interception today.
First, just a tiny bit of going back to the basics.
We can't need to explain what we're trying to achieve.
So there are two main kind of containers out there.
We've got privileged containers and unprivileged containers.
The ones you want are the unprivileged guide.
Privileged is bad, and just to clarify there, too, we don't mean privileged as in dash dash
privileged in Docker.
That's extra, extra bad.
Docker by default is privileged, and the definition of privileged is whether you're using a user
namespace or not.
So in the case of LexD, which is a container manager that I'm working on these days, with
default to unprivileged containers, that's great.
It means that root in the container is not root on the host.
If there's a container escape of any kind, you don't, you get as much permission as a
nobody user on the system, that's great.
Problem is, not being real root also means you don't get to do stuff that real root can
do.
A lot of stuff have been enabled now inside the user namespace that you can do yourself.
You can create network devices.
You can reconfigure.
A lot of stuff is great.
But there are still things you can't do.
You can't change your process priority to something higher than what you would be allowed
to do as a normal user on the system.
Otherwise, a user on the system can just create a new user namespace, go in it, and bump their
process priority to whatever they want, and bypass all kind of settings on the system.
So there's a lot of things that are not quite possible.
In general, we want to eradicate privileged containers because having real root is very,
very bad.
And it's kind of a game of, like, welcome all, as far as trying to prevent nasty things
from happening.
We've got Apama.
We've got SecComp.
We've got a whole bunch of things that are all trying to prevent you from doing bad things.
So that's done by us thinking about what all the bad things are and trying to block them.
And someone just needs to find another bad thing we didn't think of.
And then there goes the entire system.
So we don't want those.
We'd like to get rid of them completely.
But for that, we need to find ways to allow for unprivileged environments to do things
that are normally only allowed to be done by privileged environments in a way that's
still safe.
All right.
So all I'm going to be talking about today relies on SecComp, which is the system call
interception mechanism in Linux.
It lets you do a bunch of nice policies.
You can just put policies for, like, this system code with those arguments, just deny
them or return this return code or return, yeah, this particular error number, for example.
But it also grew the ability with Linux 5.0 in 2019.
It grew the ability to notify user space instead.
So you can put a policy in SecComp that says, if this is called and the arguments are so
insert, instead of doing, taking action right now, go and notify this file descriptor that
something happened.
And then the whole system can have a privileged demon monitoring that notification mechanism
and take actions.
There is some complexity around security that I'm going to get into very shortly because
you can do very, very bad things with that.
But if you do it correctly, it lets you run a more privileged action kind of on behalf
of a less privileged container after going through some kind of a list or that kind of
logic on the host to make sure that this is actually fine.
Now for the nasty issues, time of check, time of use is a very, very common issue in security.
And this mechanism has definitely got some issues around that.
User space gets notified that a system code was made.
The system code can have pointers to a bunch of different arguments and structures.
And there's nothing preventing the caller from technically changing the value at those
pointers.
So you need to be a bit careful when you're processing those messages.
You effectively need to start by copying everything, evaluating it.
If everything looks good, then you can take actions.
But by taking actions, we mean you can run the thing on behalf of the user with the original
arguments, never putting them again because otherwise they could change.
Or you could just say, I don't want this reject.
What you shouldn't do is say, oh, based on those arguments, it seems fine.
Let it continue.
Because there's absolutely nothing that prevents the caller process from just racing you and
immediately changing the arguments to something else before it goes back to the kernel and
then running with a value you would not have allowed.
So you need to be careful in your design so that this doesn't happen.
Otherwise you're literally allowing people to run stuff as full route privilege inside
of imperial containers, which would be very bad.
So what do we actually do with this stuff?
So far we've implemented quite a few things.
I'm going to go into more details about each of those.
The first thing we implemented, actually, I don't know if they are in the right order.
One of the first things we implemented is make node.
Then we followed that, which is useful for save device nodes creation.
I'm going to go into more detail shortly.
Set X adder, we also added pretty early on.
We've got support for eBPF, so we can allow some specific eBPF programs.
We've got support for set scheduler, which is used to change some of the process priorities.
We've got support for mount, which was a real pain in the ass to implement, but we've got
support for mount.
And we've got support for this info, which was also reasonably fun to implement.
Now kind of going over those things directly, make node, what do we use that for?
One of the things we wanted to enable is for running Docker inside of lexd containers.
As I said, lexd containers are on privilege, they are nice and safe.
Docker, by and large, not safe.
But Docker running inside of a privileged lexd container safe.
So we figured we'd try and make that work.
And we did manage to get it working.
The main driver at the time was Travis CI.
The Travis CI platform was using lexd containers on m64, IBM Z series, as well as IBM power
at the time.
And they wanted the same behavior as they were getting on Intel.
And on Intel, they were using full VMs that you could do whatever you wanted them.
So we wanted to make sure that Docker worked properly in there.
And what we noticed is that Docker layers, especially the directory white out files and
five white out files, rely on either, I think it's C00 device nodes, or they rely on specific
excellent attributes that just say that this is like a directory that was removed, effectively
in the underlay, using that as white out.
Both cases, those things were not allowed.
Device creation in a container is a big, big, big no go usually.
Because if you can create, say, the device node for dev SDA, then there's nothing preventing
your own previous container from rewriting your disk, which would be bad.
So that's usually not allowed.
But some specific device nodes are fine.
And the work we did there also allows for things like creating a new dev null device
or new dev zero device or those kind of devices which are inherently safe.
And making that possible means that you can now do things like running the bootstrap or
similar tools inside of an underprivileged container, because the few devices that are
needed to be created as part of an image creation process are safe devices and this allows it.
We generally consider this particular interception to be safe, as in you can pretty much turn
it on on any complexity container without having to think too much about who's in that
container, like do we actually trust the workloads on that kind of stuff.
The other piece to that puzzle was set XSATA.
As mentioned, same deal with Docker and the white out files.
We needed to implement that one.
Similarly, it does not allow all of the XSATA.
It just allows very safe namespaces of XSATA attributes.
It will not let you do things like setting a security XSATA attribute, because that would
let you do some really, really bad things, for example.
And this is similarly considered to be safe on our side.
Then got the pretty interesting one, mount.
Also, again, mount is a bit of a problem, because usually, well, first of all, usually
it relies on having a block device.
You kind of need to have that allowed in the container, which is already a bit fishy in
many cases.
You've got to be careful that any block device exposed to the container, you consider as
entrusted from that portal and you never mount it as real roots somewhere else, or they could
try and attack you.
And by attack you, what I mean is the kernel has a super block parser that will process
a new device as it gets mounted.
And this is not guaranteed to be bug free.
So a user that can craft a very specific block device might be able to trick something
like X4, XFS, BarFS, or any of the other file systems into either crashing the entire
system or doing arbitrary code execution in the kernel.
Both cases not very good.
But we still enable, so we still enable support for that.
If you have a container that you trust, that you don't want to give full access for everything,
but that you still trust, you can technically do this, and it will let you mount inside
of the container.
We added an extra layer on top of that, which lets us do a shift, because if you do the
amounts, the ownership information on that device are probably not landing at the container.
So we allow stacking shift effects, which is a fact-send that's hopefully dying soon,
but that we implemented for Ubuntu, that we can stack on top and that fixes the permissions.
So we support that as well.
The really cool thing with this stuff is we also support redirecting to Fuse instead,
which then becomes safe.
So what we can do is say, Ernie attempt at mounting EXT4 inside of the container called
defuse 2FS binary instead.
And yeah, that's safe.
That actually does work pretty well.
And I'll show that in just a tiny bit.
And then we worked on the BPF, not allowing all of the BPF programs, but specifically
allowing those that we need to do nested containers and doing device permission management throughout.
So we can review what the program is, if it matches what we expect, then we load it, otherwise
we just reject entirely, and that's also considered to be safe.
Then for an unsafe one, SCAD set scheduler is not super-duper safe, because it lets you
reconfigure scheduler options.
It was needed to be able to run Android inside of an entry-registerly container.
They were doing some slightly wonky stuff on startup.
That needed that.
But we know that effectively the container could make itself unkillable, for example,
or could raise its priority enough to slow down the rest of the system.
So it's something to keep in mind.
There's no way to escape that we're aware of by enabling this thing, but there's definitely
some ability to affect the entire system.
And then we had this info.
That was kind of led by Alpine deciding not to use Proc Mem Info to figure out the memory
usage in the free command.
So you would run a container with a limit of, like, a gig of RAM and run free, and you
would see 128 gigs, because it would just show you the host value directly.
And that's because you look like CFS, which is another project we run, that overlays on
top of Proc to show the right values.
They don't work, because they were not reading the file system.
They were going straight to the kernel with a system call using CS Info.
So we've implemented Interception for that, and we've filled it with the same values as
you would be getting from, like, CFS, and that gets us the right behavior.
And so just switching to the demo, I'm also just rechecking something here real quick.
Okay, I was just making sure that Christian was wrong with the time.
That's good.
I've got until 3.
You showed me at 10 minutes.
All right.
So let's just move here to the terminal.
And the first thing I'll do is play with Makenode, so, and that should be, yep, that's all right.
It's launching a new container, Ubuntu 22.04, and let's try Makenode, I'm sure I'm getting
those wrong again, because I always get them wrong, Makenode.
Depending on the kernel.
Makenode name, then type.
Depending on the kernel, this might work.
Yeah, it actually does work now.
Let me figure out DevNode, this one shouldn't work, 1, 3.
C5, 1.
Yeah, C1, 3, for example, for DevNode does not work out of the box.
But if we stop this, then set demo, makenode, security, calls, intercept, makenode, true.
In this case, we do need to restart the container, because the entire second policy needs to
change.
For smaller changes, we don't need to, but for that we do.
And now that works properly.
So that's the Makenode piece.
Then we've got Docker.
For that one, I did prepare a tiny bit, because I did not feel like downloading Docker on
the Wi-Fi here.
I mean, actually, I did, but it took an hour, so happy I didn't do it during the talk.
So for Docker, actually, let me show you the config first.
So the container here has security nesting enabled, which allows for running containers
in containers.
And it has intercept of both Makenode and setExider that are set up.
And in there, that part does use the network, so I'm just hoping that it's tiny enough.
There we go.
So that works properly.
And the issue before was that unpacking the layers would just blow up.
All right.
So that was Docker, then to the more, to the fancier one, which is mount.
All right.
So for mount, launch the container here.
I'm going to go and pass it a block device.
So I'm passing it dev loop 11 on my system as dev SDA inside of the container.
Yeah, your signs are still wrong.
I've got 15.
I mean, it's until three.
Okay.
So demo mount, make FS, EXT4 on the SDA, yes.
So just formatting, that you can always do, like there's nothing preventing you from creating
a file system.
Normally, that works just fine.
What doesn't work is this, like you're not allowed to actually mount it inside of a container.
But now we can make that work, actually.
So what we're going to be doing here is turning on mount interception.
And then we need to set an extra one, which is the one allowing specific file system.
So in this case, EXT4.
And then restarting the container here.
Okay.
Exit back in there and try mounting again, and that works.
And if we look at this, and we look at DF, it's mounted normally, it's fine.
Other than it actually did this as your route, and I could have done very nasty things by
crafting a particular device ahead of time.
It works as expected.
Now, what we can do to make things a bit more interesting here, actually, we did back in
there, let's unmount it, and then install fuse to FS.
That's a fuse implementation of EXT234.
That's pretty readily available.
You can install that.
And then what we need to do is remove the config key that allowed straight up mounting
of EXT4.
And we replace it with another config key that instead says EXT4 is fuse to FS.
So we put that in there, go back in, and then do SDA back to M&T.
And the funny thing is that you won't actually notice any difference whatsoever.
It actually looks completely identical unless you go look at proc mounts at which point
you're going to notice that the file system here is not EXT4, it's fuse.exe4.
And if you look at the process list, you're going to notice that there's an extra process
running in your container now.
So that's pretty sweet, because it means we can literally forward any file system to
fuse, because it's done at the Cisco layer, the container doesn't really need to be aware
of it.
Like it is not doing anything to the mount command.
You can just call the mount Cisco at any point, and it will just forward it to fuse
and do the right thing for you, which means no chance you work yours whatsoever.
It just works.
So that is pretty cool.
And the last thing I wanted to show in the demo is for launch a Alpine Edge container.
So that's going to be the most info.
And I'm going to set a memory limit of one gig.
So if I go in there now, and I look at the free memory, we can see I've got 16 gigs,
which is considerably more than one gig.
The enforcement is in place.
So that's where problems happen, is that the enforcement is in place.
Now if you run something that will look at the free memory output to figure out how much
memory it can claim, it's going to claim the wrong amount of memory, and it will get
killed by the out-of-memory killer.
So that's a problem, which is why we did the work to fix that shoe system call interception.
So you can do security, syscall, intercept, sys, info, sure.
And then bounce that container.
And if I go in there and I look at free, now we've got a gig.
So that actually works properly.
It also fixes a bunch of other things.
It doesn't just do the memory, but it also does CPU load, uptime, and a bunch of other
things.
So that's how StrapSysInfo is now properly handled.
It's just the easiest use case we have is free on uptime, because we know that this
command has been changed to use sysinfo instead of the five system, so it's a very easy one
to just prove the concept works.
But that particular piece of work fixed a lot of other things.
I think it also improved Java, was using the wrong interface and the wrong amount of memory
sometimes, so that fixed that.
It fixed a bunch of other stuff.
That was pretty good to have.
So looking forward, where do we want to take this?
We've got most of what we wanted covered, really.
The big items that are really problematic have been resolved.
Docker was a big one that we really wanted to solve, and it's working fine.
Alpine behaving, that's really nice.
We're happy with it.
The monitor section allows for a lot of different stuff now.
It's possible to do things like image building instead of an previous container.
You can really do a lot.
And Android seems to be happy, too, with the tiny bit of stuff we added for that.
The things we'd like to add now are kind of weird stuff, really.
The one I've got in mind mostly for the sake of it, but also because I'm sure we found
a use case for it, is to implement init module and finit module.
So the ability to do kernel module loading from inside of an previous container, which
is as terrifying as it sounds, but the idea there is that we would not actually allow
for the container to feed us the actual object file that we would then store at the counter.
What we would do is we would receive that object file, we'd look at what the kernel module
name is, look against an alias that we have of kernel modules that are finitely loaded
on the system, and if it's in that list, then we'll do the loading using the module from
the host system that we know is correct.
This might help quite a bit with things like firewalls in containers that might need to
load custom IP tables or net filter modules, and potentially some other things like file
systems and other things.
So that would be an interesting one to implement, and I'm sure we're going to have to explain
to a lot of people exactly how it is that we're doing it, because otherwise we're going
to be absolutely terrified.
Before eBPF program handling, I think it's also in our plans, as I said, we currently
only intercept the eBPF program that's used for device management, so for allowing device
creation, device mapping, that kind of stuff within containers.
That's because C Group 2 removed the device's C Groups file interface and moved to eBPF
instead, so we implemented it that way.
We seem to have some interest for other programs, like I think SystemD and some other pretty
common pieces of software now generate eBPF hook that they hook either globally or on
two specific interfaces, and some of those should be saved.
That should be things that we can effectively pull, validate, that they match the expected
pattern, and if they do, then show that to the camera, this is fine, and that should
actually make a lot more newer software that make use of a lot of eBPF stuff to just start
working.
I don't think we're anywhere near getting something like a eBPF trace working safely
inside a container.
That's absolutely terrifying, because it's got access to all of the candle constructs,
and that's not something we do, but some subsets of those interfaces should definitely be fine.
I think eBPF will solve a lot of those problems, probably, because then you can load unprivileged
programs.
And the other thing that I've had in mind for a while, and it's mostly a cool thing,
not something I've actually had the use case for yet, SecComp has an interesting property
in that it runs extremely early.
It runs in the system called entry time in the kernel, before the system call is resolved.
That means we can intercept system calls that don't exist.
So we can intercept system call numbers that have not yet been allocated, and that means
we get to actually implement new system calls purely in user space, that you can access
through the normal kernel system call API.
That's super interesting because it lets you do very easy prototype and testing of potential
system calls.
If you want to try specific interfaces, see how they look, the layout, what kind of arguments
you want, you can pretty quickly implement system calls through that, and already show
user space software added until you're happy with it, at which point you go back and you
do the actual kernel implementation of the system call.
So that might be pretty interesting.
I don't think anyone has actually done that yet, but that's a nice property of how SecComp
works, that it works before any kind of resolution, any kind of validity of the system call number.
All that SecComp tells us is actually a system call number and all of the pointers through
the arguments.
It doesn't care whether the thing exists or not.
So it means we get to actually intercept things that don't exist.
And that's it.
So we can start getting a few questions.
Also on your way out, if you're interested, we do have legacy stickers on the table over
there.
If you want to help yourself, there's a question over there.
I think this was the first one.
Yeah, there's one here, there's one over there.
When will we see the sysinfo system call being intercepted by default on LXD or other distributions?
Sorry.
When will we see sysinfo calls being default intercepted?
This is going to roll out?
Yeah, we've currently not decided to do any of that by default.
Please leave quietly while we are answering questions.
Yeah, so we've not decided to intercept anything by default yet.
We consider it to be safe.
The main problem we have is it depends on the kernel version that you're running, whether
it's going to be working or not.
And it's still recent enough, even though it's 5.1, which has been around a while now,
it's still recent enough that a bunch of distros would not work properly.
So we want to wait until we can generally assume that all of the distros that are like all
the long term support releases are still supported before we can start doing that kind of stuff
by default.
Please keep it down while we are answering questions.
Thank you.
Hello.
Thanks for the great talk.
So I have two questions.
First of all, you said there is this time of check versus time of use issue.
And so how do you solve it?
It's still trying to give a question, but I can't hear anything, hold on.
So first of all, how do you fix this time of check versus time of use issue, where you
know you call a syscall, the syscall gets notified, and you can, well, raise it with
another thread and change some arguments, right?
I didn't really get that, but if Christian did, you can answer instead, because you probably
know it.
It's extremely noisy.
Now, okay, I'm going to try this.
Stefan, how do you fix the time of check, time of use issue?
Okay, so the time of check, time of use issue, you fix it by never letting the kernel execute
after the check.
So you never continue a system call after the check, effectively.
If you want to intercept a system call, you are now in charge of running it.
And so you copy the arguments as they are, you do the check on your copy of them.
You never, ever reuse the pointer that the user gave you, and you go with your own copy
of it, and that's perfectly safe.
But if the argument is a pointer to a string, you need to copy the string, and when you
are copying the string, it may be changed under the hood.
So are you actually freezing the process with something like the C group, the freezer C
group, for example?
So technically the calling thread is frozen by the kernel, but it doesn't prevent another
parallel thread to modify it, which is why we effectively map the memory of the process
with the, we copy the entire thing that we care about.
The entire, like if there's pointer of pointers, we just travel start, we copy it.
Once we've copied that, that's what we check policy against.
And that's what the, those are the arguments we're going to be passing to the actual kernel.
And we just never look back at what came from the process, which means if they try to raise
us at that point, it doesn't matter.
We create full copies, we create full copies of everything, we never continue the system
call, although that's an ability that I added a while back, so you can even say, continue
the system call if I come to the conclusion that it's fine to do so.
But if you do that, then you need to be, the kernel needs to guarantee you that it's safe.
For example, continuing the make not system call after you inspected the arguments is
safe because the kernel will just allow the creation of any device.
So I have another question, because you said about MK not that if you MK not add like this
device that nothing protects you against reading or writing into this, right?
But there is this devices C group where you can actually protect this device from being
written to or read from.
And this is what Docker does, for example.
So are you doing this in LXC?
I'm sorry, I only get about 20% of what you're saying.
Well of times what we'll do is that I'm going to be outside and we can just talk because
you also have questions.
So just follow me and we'll chat, it's going to be easier.
Thank you very much.