Thank you.
Yeah, thank you.
So welcome to my session for forensic container,
check pointing and analysis.
So my name is Adran Reber.
I worked at Red Hat since 2015.
I'm involved in process migration.
What's the basis for container check pointing now?
I guess now 13 years now.
Everything I'm talking about today is about CreeW,
Checkpoint Restore and User Space, a low level tool.
I'm involved there for a long time.
And I'm focusing on container migration since 2015
and forensic container analysis is one use case
of the overall container migration topic.
So this talk will look something like this.
I will give a bit of background about the tools,
who uses Checkpoint Restore currently, who uses CreeW,
how is it used, the use cases.
I will go through a couple of them.
Then I will talk about the title of the talk,
forensic container analysis.
This is basically just a demo.
So maybe it fails.
And then I will talk a bit about the future of Checkpoint Restore,
especially with focus on Kubernetes today.
Okay, so Checkpoint Restore and User Space CreeW,
that's the tool we're using today to do the check pointing
and create the images for the analysis.
And the reason why it's called Checkpoint Restore and User Space
is because Checkpoint Restore is a technology
which exists on operating system and Linux for a long time.
And previous approaches were either in the kernel,
that's why this one is called in User Space,
or they required some preloading.
So you would do an LD preload and then some library
would intercept everything you do.
And then later the Restore,
something would try to create the steps you did before.
CreeW is different.
CreeW is something what you would call
a completely transparent Checkpoint Restore utility.
It doesn't require any preparation of the tool.
You can just point it at any process and you can checkpoint it
if the process is not using any resources CreeW cannot handle.
And then you can restore it on the same or on another machine.
CreeW was developed with the goal to use existing kernel interfaces
as much as possible.
Over the years there were additional kernel interfaces introduced
to support CreeW.
None of these interfaces are specific to Checkpoint Restore,
so there are always multiple different users using those new interfaces.
So the most changes CreeW did to the kernel
are not Checkpoint Restore specific.
Most of the time it's just how to get more information
about the running process out of the kernel.
There are multiple integrations of Checkpoint Restore
in different projects, container run times, container engines,
container orchestrations.
And the first I have to mention here is OpenVz.
It's something I never used personally,
but that's the group behind CreeW,
so they developed CreeW to be able to live migrate their containers.
They were doing containers before it was container,
so it's something which existed for a very long time.
And at some point I'm not sure about the history exactly,
but they came up with CreeW to have a Linux tool
which works for everybody and not just for them.
Another interesting integration of CreeW is in-bored.
This is Google's container engine, what they use in-house
to run all their tasks.
And although the upstream CreeW developers
don't have direct contact with Google,
we know from conferences how Google uses it,
so basically what they do, they can migrate containers
and they mostly do low priority containers.
So if you have a node, there's something running on it.
It needs more resources than before CreeW.
They just killed the low priority container
and restarted the work somewhere else from the beginning.
And with the integration of CreeW,
now they can just move it from one host to another host.
So this is...
And as far as we know, they're using it at least since 2017.
I think that's when we saw the first presentations from Google
how they use CreeW.
Then there's an integration for a long time in CXT,
and I probably have to mention in just today.
It's also integrated there.
Also for a very long time, it's integrated in Dockoff 4.
Also, I don't know, maybe also 2016, something like this.
I've worked for a couple of years
to integrate checkpoint restore support in Portman,
so you can also, using Portman,
checkpoint restore containers migrate them from one host
to another host.
And the thing which I'm currently working on,
which I started around,
people talk to me about how they think to use container migration,
container checkpointing,
and the simplest one is maybe reboot in safe state.
So you have your system running with a container on it,
and you have a blue kernel there,
and it has some problem, and you want to update the kernel.
But your container takes a long time to start.
You're not really happy doing a reboot
because your application is down for a long time.
So with GRI, you can update the kernel,
then you can create a checkpoint,
basically an image, a stateful image of your container,
write it to disk, reboot your host,
and then it comes up with a new kernel.
This time it's green.
You restore the container,
and it's running pretty fast,
much faster than waiting for all the initialization.
So you can quickly do reboot of your systems
using checkpoint restore.
Another one, it's similar to the first one.
And also people have been talking to me about this.
So this is also used in production.
You have a container which takes a long time to start,
the one I've been told about.
It takes like 10 minutes until everything is initialized.
So, and they have a service which they want to sell to customers,
and they want to have the customers fast access to the containers.
They don't want them to wait for 10 minutes.
So what they do is they initialize the container once,
create a checkpoint, write it to disk,
and then they can immediately start in matter of seconds services
from this pre-initialized container,
and their customers don't have to wait 10 minutes.
It's just in 10, 20 seconds, something like this.
The combination of those two use cases is the container live migration.
So we have two hosts.
We have the container on one host.
And it's hopefully stateful because if the container is not stateful,
the whole migration thing of container doesn't make much sense in the end.
For the forensic use case, it can be a stateless container as well
because you can still analyze it.
So what is the same again?
We create a copy of the container, write it to disk,
and then we can create one or multiple copies on the destination system.
And the original container can keep on running or not.
So this is really up to you how you want to use checkpoint restore today.
Another interesting thing people are talking about are spot instances.
Spot instances are usually something which is cheap,
but they go away.
Those VMs, like, I don't know, you have two minutes warning,
and people are using checkpoint restore there in combination with pre-use.
So you get a signal that your VM is going down.
You create a checkpoint, write it somewhere,
and then you can continue to run your workload on another system
without losing any work or without having to do any restart
or long down times or whatever you would like to avoid.
And something which came up recently is people are interested to use it for AI training.
So you have your AI training running somewhere with a GPU,
and for some reason it's aborted, or you have to make space on the node,
and with a combination of checkpoint restore you can create a checkpoint of your container.
In this case it's less about migration.
It's just creating a copy of your state somewhere so you can continue to run it later,
or even migrate.
It really depends on what you want to do there.
The interesting thing here is I mentioned previously that the CRIU cannot handle all resources,
and GPUs are kind of the resources which CRIU cannot always handle.
We are lucky that AMD came up to us and they actually implemented support to migrate
or checkpoint restore applications which are running on the host CPU
and at the same time on the AMD GPU.
For NVIDIA, we don't know if that exists.
We have heard people talking about it.
I think Microsoft mentioned it at some point that they might have been using CRIU
in combination with NVIDIA, but nobody talked to the CRIU Upscreen Project at least.
So we are not aware that people are doing, but we kind of expect that people are using CRIU
in combination with NVIDIA GPUs.
So the next one is then forensic container analysis and my demo.
So my demo is based on a container.
I am using OpenHPC as a base.
So the container is a stateful container.
It is calculating pi and memory which we can hopefully later find in the container.
So to create a checkpoint, there is a complicated way to do it.
So currently checkpoint restoring Kubernetes is only a Q-Bled interface.
Officially, the reason is because checkpoint restore writes your container,
every memory page to disk.
There is the potential, the risk that you now have private keys, random numbers, passwords,
now all written on disk.
The checkpoint is only readable by root.
So the situation doesn't really change because if you root on a machine,
you could also extract the memory, but for now,
because it's not clear how to handle this or how we want to continue in the Kubernetes community.
With this feature, it's just a Q-Bled only interface and it looks like this.
I've also written a QCTL interface.
It looks like this.
It creates also the checkpoint archive.
It's basically doing the same.
It's just wiring all the calls completely to QCTL instead just a Q-Bled.
So now we have a checkpoint and there's a tool called checkpoint control,
which was mainly developed by Google Summer of Code students this year.
So we're very happy for this help, which they did.
And in its simplest form, checkpoint control will give you,
I'm just going to make the font a little bit smaller for a short time here,
and it gives me just some basic information about the container.
I see it's the container's name counter.
It's based on that image, ID, runtime, when it was created, the engine cryo.
Checkpoint size is basically the size of all memory pages,
and root fsdif size is the size of all files which have changed compared to the base image.
So let's unpack the checkpoint archive to see some details.
And it's just a car archive, so it's easy to unpack.
I'm just going to move this to the top again.
And there are a couple of files which now were created by the container engine.
And so we have bind mounts.
This is just some information that is necessary for restore,
because we need to restore all the mounts from the outside of the container to the inside,
and we need to know if it's a file or a directory,
because the container engine doesn't want to remember if it's a container or if it's a directory or a file,
but we need it for the restore.
Configdump has some information.
dump.loc has what cryo tells us.
In this case, it doesn't matter because it works.
Then we have the root fsdif file.
This is all the files which have changed compared to the base image we saw previously.
In the checkpoint directory is the one created by cryo.
So that has the actual process information.
So if we go there, this is the normal thing which cryo does is all,
most of them are protobuf files generated by cryo here.
And cryo comes with a tool called crit,
crit, cryo image tool, and it has a parameter show,
and we can have a look at one of those files.
Let's look at UTS namespace information here.
It basically just tells us the namespace,
the UTS namespace has hostname counters,
but we can also look at a file called pstree.
This is the process tree.
This one, so it starts to get difficult to understand what's going on.
I have a couple of commands prepared.
So I see with this one, we have four PIDs running in our container,
140, 40, 142.
It's important to know this is the view from inside of the PID namespace,
so cryo always remembers the PIDs from within the PID namespace
and tries to recreate those PIDs later.
If I looked at my process, which is maybe still running,
it should, I can see here, basically, it's not hard to read,
but those are the four, where's my mouse?
I don't know.
Oh, there it is.
You see, so this is the PID one of the container,
and this is probably 41, 42, 43, I guess,
and you can see here it has other PIDs on the outside,
because that's the view from the outside of the PID namespace.
So it's important if you ever do an analysis of your checkpoint,
it's always the PIDs from within the PID namespace.
There's also for each process, we have a file called core,
with the core information about the process.
Let's have a quick look at this one,
and it basically has the registers, the value of all the registers,
floating point and much more stuff, and the end you see,
the policies and the name of the process,
and using the name of the process,
I think I can get a list of what processes are running inside of my container
and what they do, and you see the first one is called bash login wrapper,
bash, pi and t, and if I compare this again with what's currently,
I don't know, it's the wrong command, with here, again, I see it's again,
bash login wrapper, bash, the Python code and the t command.
So looking at these files, I can find out everything about the processes here,
so there's a lot of information in here,
and if you're looking for something specific, it might be difficult,
but the information is here.
There are additional files, for example, the tempfs.dev files,
those are maybe also interesting files.
Those are basically, let's have a look at those.
Something like this is probably the right one.
And you see, this is the content of a tempfs,
so every tempfs which is not bind-mounted from the host,
which is native to the container,
KreeU kind of puts it into the image, it's basically just a tar,
so every tempfs which was in your container is now also here,
you can find all the information here.
This looks like this was slash dev.
What else do we have here? Let's have a look.
Yeah, I think that's okay.
And previously, I also wrote some, my secret data into the memory pages,
and I can actually find this memory again here,
this information again in the pages files.
The pages images are, those are not,
protobuf files are raw dumps of the memory,
this is all the memory which was written to disk,
and I can again find the information I've written to memory here.
So if I know what I'm looking for, it's easy,
if I'm looking for a password, then I have to pause it all through
and maybe find a useful string in there,
but this is just to show you, you have access to all memory pages,
and they are now all on disk,
and it can be easily analyzed, or at least look at.
So if, okay, let's, I also wrote a couple of files to my container,
I mentioned this here, the root of sdif tar,
let's unpack that one.
And so now this contains three files,
so these are all files which have changed
compared to the base image of the container,
and this is just really simple, a file which is created,
it just has the, it just contains the name of the file itself,
but it's just to show you, if you want to look at content
which has changed in the container,
you will find it here in this root of sdif tar,
which contains all the changed files.
And if you think this is all too much work,
then I already mentioned checkpoint control before,
and it's even, has even more possibilities
than what I've shown you, most of the things I've done here manually,
the tool, thanks to our Google Summer of Code students,
can at this point do.
So let's have a look at checkpoint control inspect of the,
and the $CP variable is basically pointing to the tar archive,
so the tool is now unpacking the tar archive
and giving us all the information.
And what we see here now is this information we saw before,
so it's just some basic information about the image, where it was,
how big the checkpoint size is, then we see CreeO dump statistics,
this is basically the time CreeO needs to write the checkpoint to this.
You see how many memory pages were scanned,
if they should be written to this,
how many memory pages were actually written to this,
and then we see the full process command line.
We see all the environments variables of all processes running in our container,
and next one even more variables, and more and more,
and at some point there's, I think it even contains the open files,
too many variables here.
You see, now we see the open files, you see the one has open def null,
and then two pipes, and then the working directory,
and open sockets, you even see that that's the socket I've been talking to,
and then we go to the process here,
and then we see all the mounts we need,
this is also important for restoring the process later.
So I guess that's the end of my demo,
so checkpoint control was the tool I was using,
I was using CreeO image tool to have a look at the content of the images,
and then I was using grep to find my secret key from the memory pages.
So one thing I didn't show, you can use, there's a tool in CreeO
which converts the checkpoint images to core dump files,
and then you can use gdb to look at them,
it's basically the same, you see the registers and the call stack and things like this,
might also be interesting for a couple of people,
to what's next, especially with focus on Kubernetes,
so I've shown that I have a kubectl checkpoint kind of working,
that's an open pull request, it's not being actively discussed at this point,
but it's there so if somebody needs it, it can be easily used,
maybe the next step would be to integrate checkpoint for complete parts,
I've implemented this a couple of years ago, it's pretty simple,
we just do a loop over all containers in a pod,
we just create some metadata for the pod and then we can recreate it,
so this is not a technical challenge,
it's just most things at this point are how to get it in a way into Kubernetes
which is sustainable and makes sense,
and then maybe we have something like kubectl migrate,
so we don't have to do it manually,
maybe at some point the scheduler will decide,
let's move this pod somewhere else,
and one thing, so the image format I'm using is currently just a tar file,
I came up with, but it's not a standard,
so container D uses something else,
I looked at the container D format,
it's applicable for what I was looking at,
but the problem was they were using internal protobuf structures,
I didn't thought make sense to have in a public checkpoint,
in theory, checkpointing on container D and restoring in cryo should not be a problem,
but at this point we don't have a common image standard,
I tried to start a discussion here,
but it also didn't continue unfortunately,
so with this I'm at the end,
so I showed you that cryo can checkpoint containers,
I haven't shown the restore part, but it works,
it integrated in different container run times,
it's used in production by different companies at this point,
use cases are things like reboot into new kernel and save states,
multiple copies, container migration,
spot instances, AI learning support for GPUs there,
and this is all available in Kubernetes under the forensic container checkpoint in cap 2008.
So, I'm at the end, thank you, any questions?
Thank you.
Oh, sorry.
Sorry, please be quiet, we cannot hear the questions.
You mentioned GPUs are something you can't handle,
what are the other big resources that...
So basically cryo cannot handle anything that's external to the kernel,
so InfiniBand is one which comes up in high performance computing always,
so everything where you have a state in additional hardware,
you need some way to extract it,
you need to extract the state so you can later restore it, so...
And just create a text in the process,
is that stuff that fails?
Exactly, it fails, Daniel.
So currently the people I've talked to today,
they are just interested in finding out if there has been an attack
or if there is an attack ongoing, things like this,
and then maybe at some point, maybe if you can have a couple of checkpoints
and figure out, okay, this looks like an attack pattern,
maybe detect it automatically using check pointing,
this would be maybe something in the future,
but finding a possible attack is one of the main motivations
for people for the forensic use case.
Thank you.