We want to start, and that means I need to once again ask you to quiet down, please,
so that we can hear our speaker.
Our next talk is by Adrian Reber, and he's going to talk about Kubernetes and Checkpoint
Restore.
Hello, Mike is on. So welcome everyone to my talk about Kubernetes and Checkpoint Restore.
Please quiet down.
So I've actually done the talk about container migration here in 2020.
This was using Portman in the last three years.
I was able to move it into Kubernetes.
It's not on.
It's green.
Better now? No?
Too soft?
What's too soft?
I think the only thing that you can do is move it slightly down.
Down?
Better now? No.
You can turn it up?
Oh, there was...
That's for the...
Green is good.
Better now? Better now?
Not too good?
Is it better now? No? No? No?
Okay, we've got to make do with what we have for now, but please, if you all quiet down,
we can hear our speaker a lot better.
Okay, so I'm working on process migration for at least 13 years now.
I'm involved in CRIU, which is the basis for the container migration we are doing here today.
I don't know.
Since around 2012, and I'm focusing mainly on container migration since 2015.
So the agenda for today's session here is...
Can we turn something down? I get feedback.
Okay, so the agenda is something like...
I'm going to talk a bit about background of checkpoint restore,
especially how CRIU is integrated in different things.
Then I will present use cases for container checkpoint restore, container migration.
Then I want to talk about few technical details about CRIU.
I might make this very short depending on the time.
And then I want to talk about a future of checkpoint restore,
especially in Kubernetes, what we are thinking about topic right now.
So checkpoint restore in user space is the name of the tool CRIU.
The reason for the name is that checkpointing and restoring is a thing
for over 20 years now in Linux, even longer maybe.
And there were different implementations.
There were ones using an external module.
There were ones doing LD preload.
And around 2006 or 2008 there was something called a patch set
for Linux kernel to do it in the kernel.
It was over 100 patches.
It was never merged because it was really huge and complicated.
And because the in kernel thing didn't work,
CRIU was named in user space because it's not in the kernel, it's in user space.
There are multiple integrations of CRIU in different container engines,
sometimes orchestration. The first one to mention here is OpenVizet.
They invented CRIU for their container product many years ago
to live migrate containers from one node to another.
So the thing is about CRIU, it has been developed with containers in mind.
At that time it was for different containers probably,
but it's for containers and that's why it works as well as it does today.
Then we know that Google uses CRIU in their internal container engine called Borg.
I have no details about it except the things I've heard at conferences from them.
So what they told us at CRIU upstream is that they use container migration
for low priority jobs on nodes.
And if there's not enough resources then the container will be migrated.
They said they killed the container before and restarted it somewhere else.
All the work was lost and now they can just migrate it to another node
and they say they use it for background tasks like the example they gave
is YouTube recoding of things which happens in the background,
it's not time-critical then so that's why they use a checkpoint restore for it.
There's an integration in CXD which enables you to migrate container from one host to another,
then it's integrated in Docker, it's integrated in Portman,
this is what I've been working on the last five years mainly
and the thing I've been working on in the last three years to get it into Kubernetes
is to integrate CRIU support into Cryo.
This is one of the existing container engines which Kubernetes can use.
Interestingly enough there's a ticket about container live migration
in Kubernetes open since 2015, since then nothing has happened
until now where we kind of can migrate container,
we can definitely checkpoint them and we introduce this into Kubernetes
and the label forensic container checkpointing.
This was an interesting experience for me because I was not aware
how Kubernetes processes are working to get something new in there,
so I wrote some code, I submitted the patches and then nothing happened
and at some point people told me you have to write something called a Kubernetes enhancement proposal,
it's a document where you describe what you want to do,
so I did this, so this is the links to the documents,
I wrote for this, the third link is then the pull request for the actual code changes which is marginal
and the last link is a blog post where it is described
how to today use forensic container checkpointing in combination with Kubernetes.
The reason for the name forensic container checkpointing is
we were looking at a way to introduce checkpointing into Kubernetes
with the minimal impact on Kubernetes,
the thing is it's a more or less completely new concept for containers
because Kubernetes thinks about containers, you start them, you stop them,
they're done, you don't care about anything,
and now there's something new there which tries to,
okay, but I can still move my container from one node to another node,
keep all the state and so it was a long discussion to get it into Kubernetes.
The idea behind forensic container checkpointing is
you have a container running somewhere and you suspect there might be something wrong,
you don't want to immediately stop it, maybe the attacker can detect if you stop it
and remove things from it so you can take a checkpoint from the container,
the container never knows it was checkpointed,
you can analyze it in a sandbox environment somewhere else,
you can look at all the memory pages offline without the container running
or you can restart it as many times as you want,
so that's the idea for forensic container checkpointing
and under which label it's currently available in Kubernetes.
So use cases for checkpoint and restore container migration,
I have a couple of them and one has a demo which relies on the network
so we will see if this works.
So first and maybe simplest use case for checkpoint restore
for containers is reboot and save state,
so I have a host with a blue kernel running on it,
the kernel is getting out of the state,
I have to update it, I have a stateful container
because for stateless containers it doesn't make sense,
but the container, the stateful container takes some time to start
so what can you do with checkpoint restore,
you can take a copy of the container, write it to the disk with all the state,
with all memory pages saved as they were just before,
you update the kernel, you reboot the system and it comes up with a green kernel
with all security holes fixed,
but a container you can restore it without waiting a long time,
it's immediately there on your rebooted host.
Another use case which is similar to this is quick startup use case,
I have, people were talking to me about this,
so this is what people actually use in production from what I've been told,
so they have a container it takes forever to start,
it takes like eight minutes to initialize,
and they have some software as a service thing
where they want customers to have a container immediately,
so what they do, they don't initialize it from scratch,
they take a checkpoint once it's initialized
and then they can create multiple copies really fast of the container
in matters of seconds,
and so the customers can have their containers faster
and maybe they are happier.
The next thing is the combination of those things is container live migration,
I have a source node, I have a destination node,
and I want to move my container from one system to the other system
without losing the state of the container,
so I take a checkpoint and then I can restore the container
on the destination system one or multiple times,
and this is the place where I want to do my demo,
so let's see, so I want to have a Kubernetes thing running here
and I have a small YAML file with two containers,
let's have a look at the YAML file,
so it has, it's a part with two containers,
one is called WildFly, this is a WildFly-based Java-based application
and the other one is Counter,
both are really simple, stateful containers,
if I do a request to the container I get back a number,
the number is increased and second time the number is the increased number,
so let's talk to the container, hopefully it works.
Okay, this is hard to read, but I think I need this ID,
so I'll just do a curl to the container here
and then I need to replace the ID to figure out the IP address
of my, where's my mouse here, container,
and it returns counter zero, counter one, counter two,
so it's stateful, but it's simple,
so and to use checkpoint restore in Kubernetes,
so this is currently a kubelet only interface
because we still don't know how it,
it's the best way to integrate it into Kubernetes,
so it's not straightforward yet to use it, but it's there,
so I'm also doing a curl,
now let's find my command in the history, now that's the wrong one,
oh there it was, missed,
sorry, almost have it,
okay, so this is the command,
so what I'm doing here, I'm just talking to the kubelet,
you see the HTTPS address at the end of the long line
and it says I'm using the checkpoint API endpoint
and I'm using, I'm trying to checkpoint a container
in the default Kubernetes namespace in the pod counters
and the container counters, so I'm doing this
and now it's creating the checkpoint in the back
and if I'm looking at what it says,
it's just created a file somewhere
which contains all file system changes, all memory pages,
complete state of the container
and now I want to migrate it to another host
and now I have to create an OCI,
kind of an OCI image out of it, I'm using builder here
and then I'm saying,
I'll give it an annotation so that the destination knows
this is a container image,
then I'm gonna include the checkpoint archive
into the container, into the image
and then I will say commit, it's the wrong one,
commit and I'm gonna call it checkpoint image latest
so now I have an OCI type container image locally
which contains the checkpoint
and now I will push it to a registry,
here it was, and I will call it tech39
and now it's getting pushed to a registry
so this works pretty good but this VM is not local
and now I want to restore the container on my local VM
and that's happening here,
right, ctrl-ps, so nothing is running
then I have to edit my YAML file
and so it's pretty similar to the one I had before
I have a pod called counters
and I have a container wildfly
which is started from a normal image
and the other container called counter
is started from the checkpoint image
and now I say apply
and now let's see what the network says
if it likes me
so it now says, so it's really hard to read
because it's a large font but it said
pulling the initial image so that's already there
so it doesn't need to pull, it created a container wildfly
it started a container wildfly
and now it's actually pulling the checkpoint archive
from the registry, oh and it's created a container
and started a container so now we have a restored container
hopefully running here, let's get the idea,
the idea of the container
and let's talk to the container again
and so now we shouldn't see counter zero
but counter, I don't know, three or four
I don't remember what it was last time
this is the right idea, I hope
yeah and it says, so now we have a stateful migration
of a container from one Kubernetes host
to another Kubernetes host
by creating a checkpoint, pushing it to a registry
and then tricking, kind of tricking Kubernetes
into starting a container
but in the background we kind of used a checkpoint container
so Kubernetes thinks it started a container, a normal container
but there was a checkpoint container behind it
so the checkpoint restore, the restoring of the checkpoint
all happens in the container engine below it
in cryo and for Kubernetes it's just a normal container
it has restored so back to my slides
so another use case
people are interested in a lot
which I have never thought about is spot instances
which AWS and Google has
it's cheap machines which you can get
but the deal is they can take it away anytime they want
like you have two minutes before they take it away
and so if you have checkpointing
it's now independent of Kubernetes or not
but if you have Kubernetes on your spot instances
you can checkpoint your containers right into some storage
and then restore the container on another system
and still use spot instances without losing any of your
calculation work, whatever it was doing
so something about cryo
so I mentioned everything we are doing here is using cryo
so the call stack is basically
the kubelet talks to cryo, cryo talks to runc, runc
talks to cryo, cryo does the checkpoint
and then each layer adds some metadata to it
and so that's how we have it but all the main work
of checkpointing a process is done by cryo
and some details about cryo so of course the first step
is it's checkpointing the container
cryo uses ptrace or the secret freezer to stop
all processes in the container
and then we look at proc pit
to collect information about the processes
that's also one of the reasons why it's called in user space
because we use existing user space interfaces
cryo over the years added additional interfaces
to the kernel but they've never been checkpoint only
they are usually just adding additional information
you can get from a running process
so once all the information in proc pit has been collected
by cryo another part of cryo comes
which is called the parasite code
the parasite code is injected into the process
and it's now running as a daemon in the address space
of the process and this way
cryo can talk to this parasite code
and now get information about the process
from inside the address space of the process
from memory pages to get to dump them all really fast
to this but a lot of steps are done
by the parasite code which is injected into the
target process we want to checkpoint
the parasite code is removed after usage and the process never knows
it was under the control from the parasite code
I have a diagram which tries to
show how this could look like so we have
the original process code to be checkpointed
we put something out of the code
it's not perfectly correct but we put a parasite code
in the original process now the parasite code is running
doing the things it has to do
and then we remove it and the program looks the same as it was before
and at this point all checkpointing
information has been written to disk
and the process is killed or continues running
this really depends on what you want to do
and no
we are not aware of any effect on the process
if it continues to run after checkpointing
and to migrate the process you have then the last step
is restoring and what CRIU does it reads all the checkpoint images
then it recreates the process tree of the container
by doing a clone, clone 3 for each PID
a thread ID and then the process tree is recreated
as before and then CRIU kind of morphs
all the processes in the process tree
to the original process and always a good
example is file descriptors
this is easy so what CRIU does during checkpointing
it looks at all the file descriptors
looks the ID and the file name
the path and the file pointer where it is
and during restore it's just put again
so it opens the same file with the same file ID
and then it points the file pointer to the same location
and then the process can continue to run
and then the file is the same as it was before
the file descriptor
then all the memory pages are loaded back into memory
and mapped to the right location
we load all the security settings like app-amor as ilinux.com
we do this really late in CRIU
because some of these things make
it very difficult but it's happening late
so it's working well
and then when everything
all the resources are restored, all the memory pages are back
then CRIU tells the process to continue to run
and then you have a restored process
so now to what's next
in Kubernetes so we can kind of migrate
a container like I have shown
and then we are only at the start of the whole thing
so the next thing would maybe be
kubectl checkpoint so that you don't have to talk directly to the kubelet
for kubectl checkpoint
one of the things which is currently under discussion
is if you do a checkpoint all of a sudden
you have all the memory pages on disk with all secrets
private keys, random numbers, whatever
and so what we do for the current Kubernetes setup is
it's only readable by root because if you root
then you could easily access the memory of all the processes
so if the checkpoint archive
is also only readable as root
it's the same problem you have
the thing is you can take the checkpoint archive
move it to another machine and then maybe somebody else can read it
so there's still a problem that you can leak information
you don't want to leak so the thing
about is to maybe encrypt the image
we don't know yet if we do it at the OCI
image level or at the CRIU level
we're talking about but it's not yet clear what we want to do
but at some point the goal is definitely to have something
like kubectl checkpoint to make it easy
then I've shown only how I can
checkpoint a container out of a pod and restore it into another
pod so the other thing would be to do a complete
pod checkpoint restore
I've done proof of concepts of this so this is not really a technical
challenge but you have to figure out
how can the interface in Kubernetes look to implement this
then maybe if all of this works maybe you can do a
kubectl migrate to just tell Kubernetes
please migrate this container to some other node to some other hosts
and if this works then maybe we also
could have schedule integration that if certain resources
are getting low, low priority containers
can be moved to another place
another thing which we're also discussing concerning this is
so I've shown you I've migrated a container with my
own private OCI image standard
which is the thing which I came up with
it's a tar file with some metadata in it
but we would like to have it standardized so that
other container engines can use
that information, the standard and not the thing I came up with
which just felt like the right thing to do
so this is the place where the standardization discussion is going on
it's not going on really fast
or anything like this but yeah I guess that's how
creating a standard works and with this
I'm at the end of my talk, the summary is basically
query you can checkpoint and restore containers, it's integrated
into different container engines, it's used in production
use cases are things like reboot into new kernel
without losing container states, start multiple copies
quickly, migrate running containers, the new spot instances
I've been asked about, this has all been done under the
forensic container checkpoint in Kubernetes enhancement proposal
and currently we
trick Kubernetes to restore a container
by using create and start without letting
Kubernetes know that it's a checkpoint
and with this I'm at the end, thanks for your time
and I guess questions
we have time for questions
I have two questions, the first one is
Howard
one second please, stay quiet until the talk is over
so two questions, how are network connections
handled when the containers are stored
and the other question is does CreeU support some kind of optimization
in incremental checkpoints?
so the first question is network connections
so CreeU can checkpoint and restore
TCP connections established
established is an interesting thing, if they're just open
and listening it's not really a difficult thing to do
but it can restore established TCP connections
but I'm not sure it's important
in the case of Kubernetes because
if you migrate, maybe you migrate
to some other cluster or somewhere else
maybe the network is set up differently
and you can only restore a TCP connection if the
both IP addresses of the connection are the same
and it makes sense for live migration
because at some point the TCP timers will time out anyway
but I think maybe it would make sense
if you migrate a part and keep the TCP connections
between the container and the part alive
then it would make sense, it's technically possible
I'm not sure how important it is
for external connections
but for internal connections it makes sense
the other question was about optimization
so CreeU itself supports pre-copy and post-copy
migration techniques just like VMs
so you can take a copy of the memory
move it to the destination then just do the
diff at the end or you can do page faults
if on missing pages and missing pages
are then collected during the runtime
so this is all just like QEMU does
all the technology is the same
but it's not integrated into Kubernetes at all
it's...
technically it's possible
in Portman we can do this
the only thing is you have to decide
if this is an incremental checkpoint
or not because the checkpoint looks differently
so if we know it's an incremental checkpoint
only the memory pages are dumped
and if it's the final checkpoint
we have to dump everything
and if it's the first checkpoint you say
it's the final checkpoint you cannot do an incremental checkpoint
on that one
very impressive thing
except network what else do you know
will not be possible to migrate
I'm impressed by this thing
except network you mentioned
something else that cannot be checkpoints
so the main problem is every external hardware
like infinite band, GPUs, FPGAs
because there's state in the hardware
and we cannot get it out
two years ago AMD actually provided a plugin
for KreeU to get the state out of their
GPGPUs so KreeU
should be able to checkpoint
and store processes using AMD GPUs
I never use it myself
I don't have one but they implemented it
so I assume it's working
and so everything external hardware
where you don't have the state in the kernel
that's the main limitation
Hi, thank you for this
you said there's parasite code, does that mean it changes the container hash
so how do you propose to secure them again
and make sure that's your parasite code
and it's somebody else's
I didn't get it 100%
something about container hashes and making sure it's
I think the worry is that if you inject parasite code
that the container hash has changed somehow
it doesn't
it doesn't change the container hash
the parasite code is removed afterwards so it's
okay, thank you
thank you, excellent talk
how big are the images the size of the process memory used
or the total process allocated to the system
I don't hear anything in the front
how big are the images that you restore
exactly, so the size of the checkpoint
is basically the size of all memory pages
which we dump, all the additional information
which crew is dumping is compared to it is really small
and then it depends if you do something like
an importment or docker, if you do a diff
you usually see which files changed in the container to your base image
and this comes on top of it, all files which change
we include all the files completely into the checkpoint
whether we don't include this
while I'm bringing the mic over there
has anyone changed, has anything changed in terms of how complex
process trees you can restore because we're thinking about
we discussed using it for system deservices for example
for you
one of your limitations that you usually had is as soon as you run something
fairly complex inside of the container and you try to
check point restore it with crew it would just fail
because it would use kernel features that it wouldn't support
so the biggest problem we're currently seeing
is containers using systemd, because systemd is very advanced
it uses things nobody else uses
so this is the point where crew might fail
because it seems like at least from previous point
or from what I've seen nobody uses as many
new kernel features as systemd does
so it sometimes fails
the systemd is running there
but usually I don't see often people in the OCI container
world using systemd
I guess it would be a good idea to have a real
init system even in your container but it's not something people do
so it's not something we get
complaints at all about
I also thought this talk was very interesting
so I saw that you had these
talked about having these
kubectl migrate and kubectl
checkpoints
because I'm thinking that mostly what you want to migrate
might be like a stateful application
for example like a stateful, what is called a stateful something
so I was thinking maybe you could have
something in the stateful
stateful deploy whatever it's called instead of
say you want to drain a node
actually one of the first implementations I did
I was using drain
I added an option to kubectl drain which is for checkpoints
so all containers were check pointed during drain
and then they were restored during
boot up
okay
sorry for being the buskill but we're out of time
thank you for their talk that was really interesting
and thank you everyone for attending and being so quiet during your question