Just two classical optimizations that will help modern but mature virtual machine where
we have that powers native images and why is it important?
Well, and who I am?
My name is Mito Chukko.
I work at a company named Bellsoft which actively participates in OpenJDK community
and we release our own JDK distribution which you probably met if you have ever built a
Spring Boot container with default build pack.
So it's in there.
And now Spring Boot, since version 3, supports containers with native images.
It can be built as a native image and if you do that, the compiler being used is the American
native image kit which is a Bellsoft distribution of GrowLVM.
So that's another project that we participate and GrowLVM itself can be seen as different
things at least two major modes that we can absorb.
It can run as a JIT where compiler is GrowLVM or we can build a native image with a static
compilation and it will utilize a special virtual machine substrate VM and here it's
different from the traditional Java, traditional way of how we run it.
Well, another interesting and peculiar point here is that it is written in Java.
So it is a complex project but the most of the code is Java and this is beautiful.
So you have a virtual machine and a compiler for JVM languages and Java in particular written
in Java.
So if you look at Java itself, why is it so beautiful?
Well, not so beautiful compared to Kotlin as we know, right?
But still, both Java and Kotlin, they share those concepts.
So from the very beginning, there is a way to write correct parallel programs.
So then the right parallel programs, we need some means of synchronization or to orchestrate
so our threads, if we share data, most typically we do that.
And also it's a managed runtime where we don't have to worry that much about pre-memory
because we have garbage collection and garbage is collected for us and our programs just,
they can't have memory leak but you have to work hard to get one.
And having that native image implementation makes our final binaries very, sometimes makes
them very performant.
Of course, we have an instant startup.
It was mentioned today several times.
But we can also have a very good peak performance.
In certain cases, that's not a rule but it can happen, like it happens here on this plot.
That's just a simple spring boot application and we just ping the same endpoint.
And here the native image works better and also it warms up instantly and it has very
good latency.
So for this small amount of memory that it takes, so this is a small service, it takes
small amount of memory, very small heap, and it also has low latency.
And under the hood, it uses, well, serial GC and we'll talk about that later.
Well, what about relationship between Graal VM and OpenJDK?
Well, we're here in a Friends of OpenJDK room and Graal has been integrated as an additional
experimental compiler in JDK9.
But while it has been removed from recent JDKs, but what's the left over?
It's an interface to plug it in.
So now it's going to be a second attempt to do that.
So here on slides it's mentioned that there is a discussion about project, new project
they all had, but last week it was already called for votes in OpenJDK to start the project
of bringing the most sweet parts of this technology into OpenJDK, back into OpenJDK.
It's something that happens right now.
So that default garbage collector that sometimes shows very good latency even compared to ParallelGC
or G1 in hotspot, well, on small heaps.
Well, it's a kind of garbage collector we can easily understand.
And it's generational stop the world collection.
So here only one survivor space, but actually it's 16 by default.
But anyway, so we stop all our application threads and we collect garbage in a single
thread, so this is a kind of a basic garbage collector, right, but from the other hand
it's reliable and it's very effective, especially if you have only a single core available.
So you see the problem.
We have some CPU which may be enough to run many threads, but we run only one at least
for garbage collection.
Now garbage collection can take significant time during our application execution, well,
that's obvious.
Well, what would we do?
Of course, we would like to do exactly the same thing, but in parallel, to decrease the
time garbage collection takes to reduce the garbage collection pause, because it still
stopped the world pause, but we reduce it because we process data with multiple threads.
So that's the idea of parallel garbage collection.
The idea is not new, but surprisingly, this modern runtime doesn't have it yet.
Well, we decided to implement it and it's still being under review and some implementation
details, well, they change, but the idea is very simple.
You just say, pass the garbage collection selection during the creation of your native
image.
For instance, if you use some Maven or Gradle configuration for your Spring Boot container,
you also can do that.
And then you have some GRIPS in runtime, which you also can twist when you run your application.
And well, you enable that implementation.
I'll show some performance results later, but basically the implementation itself, well,
it can be analyzed as a change in a big Java program, which Brawl VM is.
And there are now two GC interfaces and implementations.
And this functionality just re-use existing things in a very, I would say, smart way just
to keep what is all about the parallelization as a code.
So everything else is reused from serial GC.
Basically there's a problem of how do we synchronize and share the work?
Because parallel threads for garbage collection, they also have the same problem because they
work on the same data, so they have contention or may have contention.
So we need to share in some smart manner.
Well, it's implemented with a work divided in its volume.
So every thread operates its local memory, and it's a chunk of memory of one megabyte.
So if we need an extra memory, like we scan objects and we fulfill some set of data that
we operate on.
And then we have an extra chunk.
We can just put it aside so someone else can pick it.
So that's the stack that contains the chunks of work.
And then the work is finished, the thread just takes the next chunk of work.
There may be a situation when several threads try to copy to promote the same object.
And this is actually solved very simply.
They just reserve some space for the object and then tries to install forward pointer
using an atomic operation.
And as this is an atomic operation, only one thread succeeds, so others just roll back
and this is a lightweight operation.
Again this is Java.
This is not a strict AML, sorry, but still all existing places that manage memory were
reused without changing the architecture of Growl itself.
So there are already possibilities to add garbage collectors.
So if you want to implement one, it's not that complex.
The major problem is to be correct when you deal with memory.
When you deal with concurrency, and then you inject your code into this virtual machine
because it's all declarative magic that requires you to be careful.
Well, some performance results.
With relatively large heaps with serial GC, you can have pauses of several seconds, which
is long, of course.
And there's a big difference if you have a two or three or four second pause or if you
decrease it by one second.
So that's possible with this implementation already.
So that's the order of this improvement.
With another benchmark, hyperalogue, you see that latency here, latency of pauses can be
decreased like two times.
Those pauses are not that big, and we have frequent collections here, so x-axis is epoch,
so each point is a garbage collection, and y-axis is time in, I believe, milliseconds.
Well, that's paralogy.
So we can obviously improve many applications and many installations where we have an option
to use several CPUs.
If we use one CPU, of course, we won't see much difference.
There is some increase in memory used for service needs, but that's kind of moderate.
So other parts of this complex system.
I mentioned synchronization, and, well, synchronization is useful, but it has tradeoffs.
Because if we implement the non-synchronization, we need to save our CPU resources to put aside
threads that won't get the resource.
We need to stop them, to queue them, to manage that queues, to wake them up, to involve operating
system in that process.
So that's not cheap, but there are situations that, that's another queue, right?
And that even influences the design of standard library, because, like, we all know string
buffer and string builder, right?
One class appeared because, well, another one wasn't very pleasant in terms of performance.
Yeah, we need it sometimes, but in many cases, we need a non-synchronized implementation,
saying, like, hash table and hash map, whoever uses hash table, right?
But it's very good synchronized.
But not all classes that have any synchronization in them have their twins without synchronization.
That makes no sense, right?
So there's a well-known technology, how to deal with a case where accesses to our data
structures, to our classes, are mostly sequential than at any point in time, only a single thread
owns and operates with an object.
And it's called bus-locking or thing-locking.
Well, why is it simpler and more lightweight?
Because we don't want to manage all the complex cases.
We know that we are in a good situation.
And if we're not, yes, we can fall back, and it's called inflate our monitor.
Well, it existed in OpenJDK for ages, and it has been removed from OpenJDK.
If it was deprecated, then no one noticed, I believe, because still, are there too many
people using something newer than JDK 11?
Well, some consequences were noticed probably too late.
Well, what are the reasons, first of all?
What are the reasons to remove a bus-locking from OpenJDK from hotspot JVM?
Well, to ease the implementation of virtual threads, to deliver project loom, to decrease
the amount of work there.
So some consequences here, initials discovered.
In certain cases, things like input streams can be slowed down, like here it's 8x or something.
That's enormously slow.
And for GraVM, there is a mode that you say during static compilation, OK, this native
image doesn't try to work with many cores.
It's a single-treaded program.
So it's simple, and it works really better in these circumstances.
So there is an optimization for that.
But you have to know it in advance, then you compile your program.
Well, and there is, of course, a runtime option that supports all kinds of situations, and
it's complex.
So the missing part is in the left lower corner.
Well, to dynamically be able to process the situation of sequential access pattern.
So we've lamented quite a classical approach to this problem.
That helps to, that brings that thing locking to GraVM.
The initial idea was operating with object header.
So where it already contains a pointer to a FAT monitor object.
But it can be treated as well as some words.
We can atomically access and put some information there.
Probably close to final implementation that we have right now still, or again, uses a pointer
because it turned to be not so easy to keep correctness across the whole VM with some
memory that you treat as a pointer or as a word depending on the situation.
Well, anyway, inside that part of header or inside that special object, we can have 64
bits of information.
And we can mark it as a thin log, this is a flag, then we can do it atomically.
We can keep the ID of an owner thread, which we can obtain, then we work with threads.
And account of recursive logs that we currently hold.
That, by the way, means that after a certain amount of recursive logs, we have to inflate
the monitor because we can store more information in that part of this work.
Yeah.
So again, it's a pure Java implementation where we work with some atomic magic and we
update this information.
What we've got, and the most recent numbers are even better.
So we see that effect on exactly that example, the streams.
We can speed them up.
And even in a very kind of nano-benchmark kind of measurement, you also see the improvement.
And even in multi-threaded case, there is now no difference with the original.