So, hi, I will start right away because my talk is quite packed, so I'm focusing on
this working for Amazon in the Amazon Coretto team.
My slides and the examples are on GitHub, I will show this link one more time at the
end of the talk, so you don't have to take a copy.
I am principal engineer in the Amazon Coretto team, working in the OpenGDK since more than
15 years, been with SAP before, that's for also more than 15 years and have various duties
in the OpenGDK and JCP.
So let's get started about firecrackers, so firecracker is a minimalistic virtual machine
monitor, it's KVM backed, it only supports a limited set of devices, basically block
and network devices which are virtualized to Vortio and a VSOC and a serial device that
makes it very fast and also very secure because it doesn't support any exotic devices like
for example QMU, it has a rest-based configuration, it's completely written in Rust which also
makes it kind of safe, it's based on, it was forked from Google's CrossVM and it's nowadays
based on Rust VMM library which is like a based library for virtual machine monitors
and I think that's also used by CrossVM meanwhile.
It supports a microvia metadata service which is basically a JSON storage where you can
share data between guest and host because with full virtualization it's not easy to
exchange data between guest and host because all the guest applications run on their own
kernel and with this data service for example you don't need a network connection between
host and guest and then the firecracker process itself supports in addition to the security
provided by KVM, sandboxing, so a jailer utility which basically places the firecracker process
on the host into additional C-group, change-route and sec-comp environment and it's all open
source, Apache 2 licensed and it's the technology behind AWS Lambda.
So every Lambda runs in its own firecracker virtualized container.
So here's just a picture of what I've just told you.
So we have the kernel with KVM on the downside and then we have the firecracker process which
has a thread for each VCPU which you configure in your guest and then it has a special thread
to handle IO and an API thread which is low priority to handle the rest requests and then
it boots the guest kernel which has the VATIO devices and the VM thread handles these VATIO
queues and maps them for network to tap devices on the host and for the block devices for
either on a native block device on the host or on a file system which is exported as
block device to the guest and then you can run a bit more application on the guest and
you can run as many guests as you want, it's only limited by your amount of memory basically,
and overhead by firecracker is just about 50 megabytes per, I know it's less, we will
see, it's very small.
So let's go to a demo.
So I have to truncate the file.
So here we just start firecracker, we specify the API socket where we communicate, we have
a log file and a log info in the boot timer to see the boot time.
And now from another terminal we start to config this with JSON data as I told you before,
so we configure two VCPUs and 512 megabytes of memory.
I have here a root file system, extended X4 root file system and a freshly compiled Linux
kernel, so I will now use another REST command to configure the Linux image which will be
booted and I pass quite a lot of kernel arguments, it's mostly to switch off devices which we
don't need anyway and which unsupported and we define as init script to just run bash,
so init script will be just a shell and then we finally have to define a root file system,
that's our X4 file which I showed you before and now that we've configured everything we
can just start the virtual machine again with a JSON request and when we go back into our
window we see that now the virtual machine has been started and it took about 200 milliseconds
to start bash and it's fully configured Linux, the image was assembled from Ubuntu 22 image
and the kernel I've compiled it myself, you see we have two CPUs and about 512 megabytes
of memory, so if we exit the shell it will be able just reboot because it was our init
process, from this 200 milliseconds which you take to boot the serial device alone took
about 100 milliseconds, so if you take that away usually in production you don't need
the serial device it puts in 100 milliseconds and that's on my laptop, okay, so very quick
comparison of Firecracker and Docker, so Firecracker is fully KVM virtualized, Docker has only
C group namespace isolation, the good thing about C group namespace isolation only is
that Docker images run on the same kernel so they can do copy and write, page cache memory
sharing so if you run many of them they are denser whereas for if you run several Firecracker
images they cannot directly share memory so you have to use ballooning devices for example
in the guest to give back memory to the host, on the other side that's much more secure
because every container has its own memory, its own kernel and Firecracker has snapshot
support to a checkpoint the whole container like with the kernel everything together and
Docker can use Crewe checkpoint to store in user space to do the same thing basically serialize
Docker container with all processes to a file, I will see examples for that now, so now what
is crack and Crewe, so as was mentioned before crack is called in native to store and checkpoint
that's a new project in the OpenJDK, it has basically three points which are important,
first one is to create the standard checkpoint restore notification API because many applications
are not aware of being cloned and there is state, security, time, all this kind of stuff
which an application might want to react upon especially not only when cloning but not only
when checkpointing and restoring but especially when cloning the application, think for example
of an application which logs to a file and then you checkpoint it and restart two clones
and they both write to the same file they will corrupt the file usually so you have
to take some measures if you run many things in parallel and the application is not prepared
for that, so if you want to, a crack is currently not part of an official OpenJDK release it's still
mostly a research project in the OpenJDK but you can already now make your application ready
for crack by using the org crack API that's available on Maven Central and that basically wraps
JDK crack namespace which is currently in the crack repository in OpenJDK but if it finds
javax.crack once it should become available it will switch to that and it also offers the
possibility to pass the custom implementations to a system property and then finally what
makes crack interesting for many people to experiment with is that it basically integrates
with Creel so it has a copy of Creel packed with the crack distribution so you can easily
checkpoint your java process and restart it and then as I mentioned before Creel is checkpoint
and restore in user space that's an old java functionality which allows to serialize a single
process to the file system it uses kernel free cgroup freezer to freeze the processes or process
tree and then writes all the memory to the disk and so on. Still Creel has some issues because
it has to take to look at all the open file descriptors, shared memory segments, stuff like
that which might not be available again when you restore the image whereas firecracker as I said
before it restores the whole kernel with all the file system everything in place so it's much
much simpler from that perspective. So let's take a quick demo on crack.
So I have here open gdk.17 with crack extensions and then you simply pass the option checkpoint
to that's a file and this is just a pet clinic up a spring boot pet clinic example application
and I modified it to register with the orc crack callbacks as I said you can see here
it's registered to orc crack and now that I've started it I can use j command to checkpoint it
so I send it a checkpoint command and when you see just out of the box it didn't work it shows
some exception because it found for example that the port 8080 is open and this uses a vanilla
version of Tomcat which is implementing the crack callbacks so but that's not that bad
it has a developer option which has to ignore exceptions so for this simple case it will probably
work so let's try it started one more time prepare the checkpoint here so let's wait
until it becomes ready so and now now checkpoint it and you see we also locked the resources so
you see what they were about 10 file descriptors and most of them were okay because like the crack
modified VM already knows a lot of the file descriptors the VM is using for example for
the jar files it has opened or for the module files and it closes them by themselves without
need to register anything so and the checkpoint you work and what's interesting is here that
before checkpointing it calls the my the the listener the handler I installed in my pet clinic
application so I could do additional stuff before checkpointing and now we can just
restore this frozen process and you see it starts instantly it calls the after restore a hook
I have registered and we can send a serial request on 8080 and yeah it basically still works
so that's nice let's go further so now firecracker so that's basically combination of
initial firecracker and crack I found it somehow funny that words are so similar so
it's a play with words and my my opinion it's the best of two worlds to combine these two
currently as I said a crack project is based on crew but I think it might be interesting to
add support for firecracker as well and I'm currently working on that so with firecracker you can
basically checkpoint a plain JDK even with if it's not modified by crack because as I said no need
to worry to worry about fire descriptors so on one issue with firecracker as I said before you
cannot trigger the checkpoint from Java so the crack implementation in open JDK can checkpoint
itself because crew is running on the same kernel like the Java application so the Java just
so JNI calls crew and checkpoints itself that's obviously not possible in firecracker because
you cannot escape from the gas that's the whole thing about running it in in a in a
fully virtualized guest so we need another means of communication but that's not not that complicated
it offers maximum security and speed and I said before no copy and write memory sharing but
you can use ballooning same page merging kernel features which are also have their plus and
their drawbacks but things to investigate so let's do a firecracker demo with Java now
to not bore you more with all this
JSON request I've written a shell script which basically does all that in in one script and
instead of calling bash it just starts Java as in it process and we can now submit the request
and you see it's it's it's working it's here here is the request my I have still registered
this this callbacks although I'm running on a vanilla JDK by using the org crack library so
they are they are empty they won't do anything and I can now snapshot firecracker you see that's
also quite quite fix quite quick firecracker is not is resumed automatically so I have to kill
it manually and now if I restart from snapshot you will see it also it takes just a few milliseconds
to restart the whole image and again I can see well into it it it works you see there is no the
hooks are not being called because there is no real crack implementation in the back in this case
but like checkpointing for Java itself works and it's also easy to run a second clone now
obviously we cannot run it in the same namespace because it will use the same IP address like
the like the first version so we we started in a in a network namespace so minus and zero is just
to create a new namespace for for the clone and you see it uses IP net NS net names with exec to
execute firecracker but it restores quite as quickly and the initial IP address of the
of the of the process has now in this namespace is it's now mapped on a different IP address on
the host but you see it's it's still working so in the get the guest still has the same IP address
it has in the first place it's just running in its own namespace and inside the guest again the
Tomcat is running on the same port all no problem so we just kill the first instance and we kill the
we kill the second instance how much time do I have oh okay okay so just a few words I I realized
that talks which are rated highest are usually so some animation so I decided to do animation
because usually only so console console demos so quick introduction user fold demon is a
is a possibility to handle page faults from the user space and firecracker offers the possibility
instead of mapping the image file right into fires firecrackers memory to to use an external
user fold demon and if we write the user fold demon ourselves we have the possibility to follow
page by page which addresses get loaded at the restore and I found it interesting so
I created that kind of thing so to an animation for that and for that
we we restart our our our firecracker service native memory enabled native memory tracking
and from the guest we do now ssh into into our firecracker guest where Tomcat is running
and just call j command native memory details and and put that into a file and we do the same thing
with the pmap information this is just a shell script inside the guest which basically
prints all the virtual to physical mappings for all processes into a file
and now we can start the the visualizer and it takes the locks
oops it it takes the locks of the user fold demon and the nmt and the native mapping so what you
see here is basically the physical memory layout of the guest so it's memory page zero and in the
end it's memory page one gigabyte and every square is four kilobyte page and if you go and
that's on the java process for example you see the dark these are the pages the rss of the java
process blue ones are occupied by the java process but they are also in the page cache so that's
probably a file for example or something or uh uh class uh spare shell class for example
when you when you look at the nmt output we see that for example for the classes we use about
66 i probably cannot read it it says virtually 69 megabytes uh rss is 60 megabytes and user
fold demon loaded about 10 megabytes of it and here's the the animation i promised you
so this is how the pages got loaded when we did the first call request on a on a resumed image
and like the the yellow ones are all the pages which i've loaded and the orange one i don't know
yeah some are orange belong to the to the to the virtual memory region i have selected here so for
example all the orange pages are the the parts of the class space which got loaded for the first
request so this is a lot of space for more investigation would be nice to to compact this
more like physically because you want to prefetch the the things which get loaded especially if you
download your images from from network for example and but the problem is that all the physical
address space is continuous like the virtual uh the physical pages are are not and try to look
into uh possibility to do that so that that's it thank you thank you very much
there's about 30 seconds for questions is anyone got a question called your answer question
i have a question regarding uh when you showed uh uh crack uh implementation there was uh
implementation that put into the uh
so
yeah
yes i unfortunately there is no time in 20 minutes to show that but you can obviously use the current
crack implementation inside firecracker use j command and instead of crue there is a backend
called uh post handler that's just a small program which instead of calling crue just
dispense the whole process and then you can send in the signal to restore it so with firecracker you
basically checkpoint with the post engine then do the firecracker snapshot then restore firecracker
and then just do an ssh with a kill signal on on the process and it will will restart that's one
possibility another one is i wrote the jvmti agent which basically has the same thing even without crue
it uh it um suspends all threads it calls system gc and then waits uh on a on a port so you just
ping it with telnet or whatsoever and and it even calls uh the the the the hooks by implementing
the this custom possibility to uh with the property so i i i i say or crack to use my
crack implementation to call the hooks so that all works it's in the in the repository which is
i had a resource slide which i didn't show it has all the links so