Okay, then our next talk is by Frazier.
Over to you.
Okay.
Thank you.
So I'm going to talk about user namespaces and delegation of control of C groups in container
orchestration systems.
So yeah, containers and container standards.
This is the containers dev room, I think, hopefully most people should have some idea.
Talk about Kubernetes and OpenShift, user namespaces and C groups, and then a demo of
a system B based workload on Kubernetes and more specifically OpenShift.
So container is a process isolation and confinement abstraction.
Most commonly it uses operating system level virtualization where the processes running
in the container are using the same kernel as the host system.
Examples of this include FreeBSD Jails and Solaris Zones as well as Linux containers.
And the container image is not a container, the container image just defines the file
system contents for a container and some metadata suggesting what process should be run, what
user ID it should run underneath and so on.
On Linux containers use a combination of the following technologies, so namespaces,
process ID namespaces, mount namespaces, network namespaces, et cetera, to restrict
what the process running in the container can see.
The container may have restricted capabilities or a second profile that limits which system
calls or operating system features it can use.
There may be SE Linux or app armor confinement and it can use C groups for resource limits.
Not necessarily all of these are used at the same time, it depends on the implementation.
The OpenContainer initiative defines standards around containers in the free software ecosystem
and its runtime specification in particular defines a low level runtime interface for
containers that is not just for Linux containers but it defines the runtime semantics for Linux
Solaris containers, Windows containers, virtual machines and so there are a bunch of implementations
of the runtime spec including RunC, the reference implementation, CRUN and CART containers which
is a virtual machine based container runtime.
The OCI runtime spec has a JSON configuration and there's a link to an example, it defines
or lets you define the mounts, what process to be executed, its environment, life cycle
hooks so extra code to run when the container is created, destroyed, started, stopped and
the Linux specific capabilities that can be controlled via the OCI runtime spec include
the capabilities, that's the kernel feature capabilities, namespaces, the C group, the
system controls that should be set for that container, the seccomp profile and so on.
This is what an example runtime specification looks like, so it has a process structure
which includes a field for the user ID and the group ID, it has this Linux structure
which defines the Linux specific attributes for this container and one of those attributes
is the namespaces list which defines a list of the different namespaces that should be
used or should be newly created for that container.
So Kubernetes is a container orchestration platform, has a declarative configuration
system and it integrates with many different cloud providers.
Anyone not know what Kubernetes is in the room?
So the terminology in Kubernetes, the container is an isolated or confined process or process
tree, a pod is one or more related containers that together constitute an application of
some sort, so it might be an HTTP server and a database to encapsulate the entire
web application.
A namespace in Kubernetes terminology is not a namespace in Linux kernel terminology,
a namespace is just an object scope and authorization scope for a bunch of objects in the Kubernetes
data store, so if you have a particular team or project in your organization you might
deploy all of the Kubernetes applications in a single namespace.
And a node is a machine in the cluster where pods are executed, there are different kinds
of nodes, there are control pane nodes, there are worker nodes where the actual business
applications run.
Kubelet is the agent that executes pods on nodes, so there's a scheduling system, the
scheduler will, when a pod is created, decide what node that pod should run on and Kubelet
is the agent on the node that takes the pod specification and turns it into a container
running on that node.
The Kubernetes terminology uses the term sandbox, that's an isolation or confinement
mechanism, and there's one sandbox per pod, so there could be multiple containers running
in the sandbox.
And the container runtime interface defines how Kubelet actually starts and stops the
containers for the sandboxes.
So cryo is one implementation of the container runtime interface, that's what's used in
OpenShift, there's also container d, that's used in some other distributions of Kubernetes.
So visualizing this, the whole box is one Kubernetes node, the Kubelet has the gRPC
client to talk to us CRI runtime, the CRI runtime does something and containers appear.
So we could instantiate the container runtime interface implementation as cryo, and then
we can see that cryo talks to an OCI runtime, it uses exec to use the OCI runtime, and we
can go one step further and say that the OCI runtime implementation will be run seat.
And this is the setup that we use on OpenShift.
This is a pod spec in YAML format, so we have kind pod, the specification has a list of
containers, in this case there's one, the container has a name, defines the image to
use, the command to execute, environment variables that should be set, and so on.
OpenShift or OpenShift container platform is an enterprise-ready Kubernetes container
platform, it's commercially supported by Red Hat, there's a community upstream distribution
called OKD, as I mentioned before, it uses cryo and run seat, and the latest stable release
is 4.12, I think that came out just a week ago or so.
And its default way that it creates containers is it uses SE Linux and namespaces to confine
the processes, each namespace gets assigned a unique user ID range and the processes for
the pods in that namespace have to run in those host UIDs.
You can circumvent this using the run as user security context constraint, but that is not
a good idea, you don't want your containers running as root on the container host because
if they escape the sandbox, then your cluster got owned.
So this is the why of user namespaces, user namespaces as we're talking about Linux kernel,
user namespaces can be used to improve the workload isolation and the confinement of the
pods in your cluster.
They can also be used to run applications that require or assume that they're running
as specific user IDs, or to phrase this a different way, you can drag your legacy applications
kicking and screaming into the cloud and get the benefits of all of the orchestration and
networking support that these platforms like Kubernetes and OpenShift can give you while
still running that workload securely.
In other words, trick it to believe it is running as root.
And yeah, there have been a bunch of CVEs in Kubernetes and the broader container orchestration
ecosystem arising from sandbox escapes where user namespaces would have prevented the vulnerability
or severely limited or curtailed its impact.
So visualizing a user namespace, we can have two separate containers with a user namespace
mapping of UID range 0 to 65535 inside the container's user namespace to a range of unprivileged
user IDs in the host's user ID namespace.
So the processes running in the container believe that they are running as, for example,
root UID 0 or some other privileged user ID when, in fact, it's running as UID 200,000
on the host, an unprivileged user ID.
Take the sandbox and you're still an unprivileged user on the host.
So in Linux, there's some references to some man pages about user namespaces and how to
use them.
The critical thing is the unshare system call, which is how you create and use the user namespace.
In the OCI runtime specification, there are some fields.
And again, this is Linux specific, so it's inside the Linux specific part of that configuration
that you can specify that a user namespace should be created or used for that container
and you can specify the mapping.
So how do we map the containers user ID range to the host user ID range?
User namespaces were implemented before they were implemented in Kubernetes upstream.
We did it in cryo, it first shipped in OpenShift 4.7, but it required a considerable amount
of additional configuration of the cluster to use it.
And since OpenShift 4.10, you've been able to use it out of the box.
You do still have to opt in using annotations on a per pod basis.
It requires the NEU ID security context constraint or an equivalent privileged security context
constraint in order to admit the pod because the admission machinery does not yet understand
about user namespaces.
So the pod spec says, I want to run as user ID 0 and the admission machinery says, uh-uh,
no way.
We need to circumvent that for the time being, but the workload itself will run in the user
namespace.
And depending on the workload, it may still require some additional cluster configuration.
So the annotations to opt in, you can say IO.OpenShift.Builder is true.
That activates a particular cryo, what we call a workload, basically an alternative bunch
of runtime settings.
And then we use the user NS mode annotation to specify that we want an arbitrary user
namespace of size 65536.
So that'll allocate you a contiguous host UID range for that container and map it to
unprivileged host user IDs.
In the Kubernetes upstream, it took a bit longer to get user namespace support and it's
still a work in progress, but the initial support was delivered in Kubernetes 1.25.
And that version is what we've moved to in OpenShift 4.12.
So you can now use the first class user namespace support in OpenShift.
It is an alpha feature, so it's not enabled by default.
You have to turn it on with a feature gate.
And at the moment, it only supports ephemeral volume types, so empty to a config maps, secrets,
no persistent volume support yet.
You opt in by putting host users false in your pod spec and currently gives you a fixed
mapping size of 65536 that will be unique to that pod.
It is hoped that a later phase will deliver support for the additional volume types.
The reason that we didn't have them is the complexity around ID mapped mounts and how
to adapt the volume mounts to understand how to map the user IDs between the host UID
namespace and the container's username space.
There's also very simple heuristics around the ID range assignment.
As I mentioned, it's a fixed size of 65536 that limits the number of pods that you could
run in user namespaces on a given node, and there are still some other mount point and
file ownership issues, for example, with the C-group FS.
And that takes us to the C-group's topic.
So OpenShift creates a unique C-group for each container, and it also creates a C-group
namespace so that the container in the CISFS C-group mount only sees its namespace.
Inside the container, CISFS C-group actually points to CISFS C-group slash a whole bunch
of stuff specific to that container in the host's file system.
If you want to run a systemD-based workload, systemD needs right access to the C-group
FS, but by default, the C-group FS will be mounted read only.
So the solution, we modify the container runtime to chone the C-group to the container's
process UID.
So that is we chone it to the host UID corresponding to UID0 in the container's user namespace.
But first, before we did this in an ad hoc basis, we engaged with the OpenContainer initiative
to define the semantics for C-group ownership in a container, and those proposals were workshopped
and accepted, and after that, we were able to implement those semantics in RunC.
So what are the semantics?
Well, the container's C-group will be choned to the host UID matching the process UID in
the container's user namespace, if and only if the node is using C-groups V2, and the
container has its own C-group namespace and the C-group FS is mounted read-write.
So only the C-group directory itself and the files mentioned in sys-kernel C-group delegate
are choned.
These are the ones that are safe to delegate to a sub-process.
And the container runtime, if that file sys-kernel C-group delegate is defined, then it will
read that file and only chone the files mentioned there.
So it can respond to the evolution of the kernel where new C-group nodes may come and
go, some of them may be safe to delegate, some of them may not.
In OpenShift, C-groups V2 is not yet the default when you deploy a cluster, but it does work
and it is supported, and to activate the C-group choned semantics that I just explained, we
still require an annotation in the pod spec.
So let's do a demo.
Here's a cluster I prepared earlier, and we can see new project test, okay, OC create
user test, maybe test already exists, okay, we'll just use test.
So we can now, oh, well, I'll show you the pod I'm going to create, cat pod nginx, host
users false.
So this is a pod spec, let's get some syntax highlighting.
This is going to run nginx, it's a system D based workload, so it's a fedora system
that will come up and system D will run and it will start nginx.
We're setting host users false so that it will run in a user namespace.
I have already enabled the feature flag on this cluster.
There's that annotation for the C-group choned, and the name of the pod will be nginx, so
let's create that.
So OC as test, create a share.
Okay, fingers crossed.
Okay, so we'll say OC admin policy add role to user edit, okay, let's try that again.
So the pod has been created, and we'll just check it's status to see which node it is
running on, and it hasn't yet started, so we don't have a container ID for it yet, but
in the upper pane, I'll get a shell on that worker node.
Okay, pod is now running, so we can run cry control, inspect, container ID, and we'll
just pull out the PID, so this is our PID.
Now if we have a look at the user ID map for this process, okay, so here we see that
we have a user namespace with the user ID mapping of size 65536, which is mapping user
ID 0 in the container's user namespace to user ID 131072 in the host user namespace.
And now we can look at the processes that are actually running there, so we'll do pgrep,
l-ns says show me everything with the same set of namespaces as this process ID, and
then I'll just pipe that to PS, let's print the user, the PID, and the command, sorting
by PID, okay.
So we can see that this container is running, well, in it, and then bunch of systemd processes,
and then eventually nginx, and these are running under, see, 131072, yeah, 132071, yeah, so
these are running as various user IDs in the container's user namespace, and those are
being mapped to the host user ID namespace as these PIDs.
If we look at the logs, we can see that it looks like a regular systemd system has come
up, indeed it has, and let's see, I see, maybe we'll get a shell on the node, on the pod,
and yeah, if we have a look at what is the container's view of the processes that are
running, it sees that systemd is running as root or other systemd-related users inside
the container's user namespace, nginx is running as the nginx user in that container's user
namespace, but as we saw, these are all mapped to unprivileged host UIDs in the host user
namespace.
So that concludes the demo, here's a link to various resources, I have a lot of blog
posts on this and related topics, so you can hit my blog and just look at the containers
tag, a recording of a demo or a similar demo, slightly earlier version, link to KEP127,
which is where all of the discussion around how to do the upstream support for user namespaces
in Kubernetes, all of that discussion happened there, the OCI runtime spec is referenced there,
and that's it, so I think there is time for some questions.
Please stay seated until 25 so we can ask questions, okay, there's one in the back, do you want
to read the one from the chat first?
There's a question in chat, I don't see it anymore, it says, why would I want to run
systemd in a container, it's cool that it's possible with user namespaces, but I lack
an idea for use case.
So the use case is you have a complicated legacy workload that runs under systemd or
makes assumptions about the environment it runs in, the user IDs that the different components
are running as, you've got two choices, one is to spend a whole lot of upfront engineering
effort to break up that application and containerize it and make it a cloud native application,
which is expensive and typically has a long lead time, or you can just wrap that whole
application up in a container and run it securely, hopefully, in a container orchestration platform
and get the benefit of all of the scaling, networking, observability features that the
orchestration platform gives you without having to spend that effort to bust your application
into a hundred pieces.
So I would say that that is the use case, I think it's a compelling one, if you were
building applications today you certainly wouldn't do that, but there are 100 million
legacy applications out there and people don't want to break them up and change them.
Hi, thanks for the talk, I'm actually doing this right now at my company, but basically
the container is running as privileged, so that's why it can access C group, it doesn't
use user namespaces, it just runs as a privilege, so I was wondering if using this method you
could set a memory max, memory high, or other values for some processes in the C group running
in the container, I mean.
I'm sorry, I couldn't hear the question because it's rather echo-y.
Sorry, yeah, so can you set memory high, memory max, values, CPU affinities, like all these
kinds of things you would set in the C group usually, can you set them from this particular
use case of C groups in the container?
Yes, absolutely, because the container still has its own C group namespace, so all of the
standard C group confinement and resource limit capabilities can be used.
Okay, I guess I got confused by the list of values that were allowed to do a listed in
the previous slide, there was a restricted list of values.
Yeah, so those were just the particular files that need to be choned in order for a safe
delegation of control of that branch of the C group hierarchy to another process, so
you can still set on the C group directory all of the limits and the container won't
be able to change those because those will not be choned to the container's process
UID.
Okay, thank you.
So I have a question regarding the CFFC group C-H own, so you mentioned it's going to be
changed to the UID of the container, of the process container, can you do that if you
want to have several ports to run their own system D?
Is your question around can you do it in a nested way?
No, not in a nested way, you have three different ports which each port needs their own system
D.
Yes, yes, absolutely.
So if I created more pods in my demo, you would see that they would then be mapped to
different host UID ranges, so the limit is only how many of the range allocations can
you fit into the host UID range?
So the limit will be a little under 6.536 because the size of the host UID range on Linux by
default is 2 to the 32, yeah.
Okay, I think we have time for one more question and thank you all for your patience.
Thank you Fraser.
I just wanted to know if v2, secret v2 by default in OpenShift is on the road map yet
and whether or not there's any sort of estimated time scale for that.
Yes it is, yep, so there is a plan to eventually move to secret v2 as the default, I don't
know the exact time frame.
Thank you so much for your talk, thank you all for your patience and I know this sounds
weird but you're free to leave.