Hello. I'll get started. Okay. My talk is entitled, The Next Frontier in Open Source
Java Compilers, Just in Time, Compilation as a Service. Whoops, this isn't working.
My name is Rich Agarty. I've been a software engineer for way too many years. I'm currently
a developer advocate at IBM. So, we're all Java developers. We understand what a JVM
and a JIT is. We'll do the JVM, execute your Java application during runtime. It sends the
hot methods to the JIT to be compiled. With that in mind, we're going to talk about JIT
as a service today. And we're going to break it down into three parts. First, I'm going
to talk about a problem, right, which is Java running on cloud, specifically in distributed
dynamic environments like microservices. Then we're going to talk about the reason, which
is going to take us back to the JVM and the JIT, which has some great technology. It's
great technology but does have some issues. And then the solution, which is the JIT as
a service. So, is Java a good fit on the cloud? So, for context, we'll talk about legacy
Java apps, enterprise apps running. They're all monoliths running on dedicated servers
or VMs to ensure great performance. We loaded with a lot of memory and a lot of CPUs. They
took forever to start, but it didn't matter because it never went down. We have clients
running Java applications for years. If they did upgrade, it would be every six months to
a year, do some simple refreshes. That was the world of legacy Java enterprise apps.
Now we move to the cloud. That same monolith is a bunch of microservices talking to each
other. They're all running in containers, managed by some cloud provider with a Kubernetes
implementation to orchestrate. And we have auto-scaling up and down to meet demand.
So the main motivators behind this, obviously, are flexibility and scalability. Easier to
roll out new releases. You can have teams assigned to specific microservices and never
touching other microservices. Once you're on the cloud, you can take advantage of the
latest, greatest cloud technologies like serverless coming out. Obviously, you'd have less infrastructure
to maintain and manage. And the ultimate goal is saving money.
So before we start counting all our money, we've got to think about what about performance?
So there's two variables that impact cost and performance. It's container size and
the number of instances of your application you're running. Here's a graph showing all
the ways we can get these variables wrong. Starting down here, containers are way too
small. We're not running enough instances. It's pretty cheap, but the performance is
unacceptable. On the opposite side, we have our containers are too big. Way too many instances
running. Great performance, wasting money. So we need to get over here. This is a sweet
spot. We got our container size just right. We have just enough instances for the demand.
That's what we want to get to. Very hard to do. In fact, most conferences have a lot
of talks about how to get here or their fixes for this problem. So before we can figure
out how to fix it, we've got to figure out why it's so hard. And in order to do that,
we've got to talk about the JVM in a JIT. So first of good, device-independent Java became
so popular because we write once, run anywhere, in theory. 25 years of constant improvement,
a lot of involvement from the community in it. The JIT itself, optimized code that runs
great. It uses profiler, so it can optimize a code that you can't get doing it statically.
Has very efficient garbage collection. And when the JVM collects more profile data in
the JIT, compiles more methods, your code gets better and better. So the longer your
job application runs, the better it gets. Now, the bad. So that initial execution of
your code is interpreted, so it's relatively slow. Those hotspot methods compiled by the
JIT can create CPU and memory spikes. CPU spikes cause lower quality of service, meaning performance.
And your memory spikes cause out-of-memory issues, including crashes. In fact, the number
one reason JVM, or a main reason JVM crashes because of out-of-memory issues. And we have
slow startup and slow ramp-up time. So we want to distinguish between the two. Startup
time is the time that it takes for that application to process first request, usually during an
interpretation time. And ramp-up time is the time it takes a JIT to compile everything
it wants to compile to get to that optimized version of your code. So here we have some
graphs to back that up. Here we take a Java Enterprise application, and you can see on
the left we got CPU spikes here happening initially, all because of JIT compilations.
Same thing with the memory side. We got these large spikes that we have to account for.
So let's go back to that graph I had finding that sweet spot. Now we have a little more
information, but still we need to figure out a way to right-size those provisioned containers.
And we got to make our auto-scaling efficient. So we have very little control over scaling.
We control the size of our containers, but as far as scaling goes, we just have to set
the environment enough up correctly so that auto-scaling is efficient.
So on the container size portion of it, the main issue is we need to over-provision to
handle those out-of-memory spikes, which is very hard to do, because JVMs have a non-deterministic
behavior, meaning you can run the same application over and over, and you're going to get different
spikes at different times. So you've got to run a series of tests with loading to figure
out, to get that number kind of right. And on the auto-scaling part of things, again,
we talk about the slow start-up and ramp-up times. The slower those are, the less effective
your auto-scaling is going to be. And the CPU spikes can cause other issues. A lot of auto-scalers,
the threshold for starting new instances is CPU load. So if you start a new instance
and it's spinning, doing JIT compiles, your auto-scaler may detect that as a false positive,
say, oh, you need, the demand is going up, you need more instances, when in this case,
you really didn't. So it makes it very inefficient. So the solution to this problem is we need
to minimize or eliminate those CPU spikes and memory spikes, and we've got to improve
that start-up and ramp-up time. So we are proposing here, we're going to talk about
JIT as a service, which is going to solve these issues, or help solve these issues.
So the theory behind it is we're going to decouple the JIT compiler from the JVM and
let it run as an independent process. Then we're going to offload those JIT compilations
to that remote process from the client JVMs. As you can see here, we have two client JVMs
talking to two remote JITs over here. We have the JIT still locally in the JVM that can
be used if these become unavailable for some reason. Everything since we're all in containers
is automatically managed by the orchestrator to make sure that we have their scaled correctly.
This is actually a model to microsolution, so we're taking the model, as in this case,
as a JVM. We're splitting it up into the JIT and everything left over in the other microservice.
And again, like I mentioned, the local JIT still is available if this service goes down.
So this actual technology does exist today, and it's called the JIT server, and it's a
part of the Eclipse OpenJ9 JVM. It comes with the, it's also called the SAMRU cloud compiler
when used with SAMRU runtimes, and I'll get to that in a minute. And I'm sure everyone
here knows OpenJ9 combines with OpenJDK to form a full JDK and totally open-source it
free to download. And here's a GitHub repo there. A little history of OpenJ9. It started
life as the J9 JVM by IBM over 25 years ago. And the reason IBM developed it was because
they had a whole range of devices they needed to support, and they wanted to make sure Java
ran on all of them. That's all the way from handheld scanners to mainframes. So it was
designed to go from small to large in both types of environments where you have a lot
of memory or very, very little. And about five years ago, IBM decided to open-source
it to the Eclipse Foundation. And OpenJ9 is renowned for its small footprint fast start-up
and ramp-up time, which we'll get to in a minute. And again, even though it's got a
new name, it's OpenJ9. All of IBM enterprise clients have been running their applications
on this JVM for years. So there's a lot of history of success with it. Here's some OpenJ9
performance compared to Hotspot. Again, this doesn't take into account the JIT server.
This is just the JVMs themselves going left to right here. OpenJ9's in green. Hotspot's
in orange. So in certain circumstances, we got to see 51% faster start-up time, 50% smaller
footprint after start-up. And it ramps up quicker than Hotspot. And at the very end,
after a total full load, we have a 33% smaller footprint with OpenJ9. So, several run times.
So that is IBM's OpenJDK distribution. Just like all the, someone just mentioned, there's
a ton of distributions out there. This is IBM's. And it's the only one that comes with
Eclipse OpenJ9 JVM. It's available no cost. It's stable. IBM puts their name behind it.
So it comes in two editions, open source and certified. The only difference being the licensing
and what platforms are supported. And if you're wondering what Samaru comes from, the name
comes from, Mount Samaru is the tallest mountain on the island of, anyone know? Java, there
you go. See how that makes sense? If I had a t-shirt, I would have given you that. Alright,
from the perspective of the server or the client talking to this new JIT server, this
is the advantages they're going to get. From a provisioning aspect, now it's going to be
very easy to size our containers, right? We don't have to worry about those spikes anymore.
So now we just, we level set based on the demand or the needs of the application itself.
Performance wise, we're going to see improved ramp-up time, basically because the JIT server
is going to be offloading. We're going to offload all the compiles in the CPU cycles
to the JIT server. And there's also a feature in this JIT server called AOT cache. So it's
going to store any method it compiles. So another instance of the same container application
calling it, and then they'll have that method, it'll just return it. No compilation needed.
Then from a cost standpoint, obviously any time you reduce your resource cost or your
resource amounts, you're going to get a savings in cost. And I mentioned earlier the efficient
auto scaling, you're only going to pay for what you need. Resiliency, remember the JVM
still has their local JIT. So if the JIT server goes down, it could still keep going.
So this is kind of an interesting chart. This is pretty big. So we're going to talk about
some of the examples of where we see savings. So this is an experiment where we took four
– let me see my pointer works – we took four job applications and we decided to size
them correctly for the amount of the memory and CPU they needed doing all those load tests
to figure out what this amount should be. And we have multiple instances of them. So
the color indicates the application. You can see all the different replications. The relative
size is shown with the scale of the square. And in this case, we used OpenShift to lay
it out for us and it came out to use three nodes to handle all of this, all these applications
in your instances. Then we introduced the JIT server, ran the same test. Here's our
JIT server here, the brown. It's the biggest container in the nodes. But you notice the
size of all of our containers for the applications goes way down. So we have the same number of
instances in both cases, but we've just saved 33% of the resources. And if you're
wondering how they perform – whoops, went too far – you see no difference. The orange
is the baseline, the blue is the JIT server. And from a stable state, meaning once they've
performed, they perform exactly the same. But we're, again, saving 33% of the resources.
Now we'll take a look at some of the effects on auto-scaling in Kubernetes. Here we're
running an application and we're setting our threshold, I think it's up there, at
50% of CPU. And you can see here all these plateaus are when the auto-scaler is going
to launch another pod. And you can see how the JIT server in blue responds better. Shorter
dips and they recover faster. And overall, your performance is going to be better with
a JIT server. Also, that other thing I talked about with false positives. So, again, the
auto-scaler is not going to be tricked into thinking that that CPU load from JIT compiles
is the reason for demand. So you're going to get better behavior in auto-scaling. Two
minutes. All right. When to use it? Obviously when the JVM is – we're in a memory and
CPU constrained environment. Recommendations, you always use 10 to 20 client JVMs when you're
talking to a JIT server. Because remember, that JIT server does take its own container.
And it is communication over the network, so only adding encryption if you absolutely
need it. So some final thoughts. We talked about the JIT provides great advantage that
optimize code, but compilations do add overhead. So we disaggregate JIT from the JVM and we
came up with this JIT compilation as a service. It's available in Eclipse OpenJ9, also called
the SAMRU Cloud. It's called the Eclipse OpenJ9 JIT server. That's the technology.
It's also called the SAMRU Cloud Compiler. It's available on Linux Java 8, 11, and 17.
Really good with microcontainers. In fact, that's the only reason I'm bringing it up
today. It's Kubernetes ready. You can improve your ramp-up time, auto-scaling. And here's
the key point here I'll end with. So this is a Java solution to a Java problem. Initially
I talked about that sweet spot space. So there's a lot of companies, a lot of vendors trying
to figure out how to make that work better. And a lot of them involve doing other things
besides what Java's all about, running the JVM, running the JIT. So it is a Java solution
to your Java problem. That's it for me today. That QR code will take you to a page I have
that has a bunch of articles on how to use it, also the slides and other good materials
about it. That's it for me. Thank you very much.
It sounds amazing. It's amazing. It really is amazing. Well, why wouldn't you? Open
J9 is a perfectly, I mean, it's a viable JVM. It's nothing special, right? And nothing
unique about it that makes you change your code. It's a JVM that just points to the
open JDK, the open J9 JVM. Okay, here it comes. I think so because I've seen examples
of using those apps in tests. Check that, yeah. Yeah, okay. That may be a problem. She
go out and check the latest coverage of that. Well, the way the AOT cache will work in this
case for the JIT server, it's going to keep all that information and the profile has to
match from the requesting JVM, right? So if it matches, it'll use it, right? Because
also on the clients, they also have their own cache. They'll keep it, but they go away
once they go away, right? Or when you start a new instance of that app, you have a brand
new flush cache. I'm sorry. Yeah, so that's what we were talking about. You want to go
static. You're going to get a smaller image running statically, but you lose all the benefits
of the JIT. Over time, yes. So that may be a great solution for short-lived apps, right?
But the longer your job app runs, the more you're going to benefit from that optimized
code, right? Yes? So Eclipse on the J9 is not a certain set-byte, but my main server is
also a set-byte for open edition, but today it has available binaries. But for Eclipse,
they are not able to actually release the binaries because they cannot actually access
the TCK certification process. So that whole TCK issue is a, I don't know. Well, I guess
I could say, it seems to be an issue more with IBM and Oracle, right? So our own tests
are going to be, they're going to encompass all the TCK stuff. Open J9 is managed by Eclipse,
but 99% of the contributions are from IBM. It's a big part of their business. It's not
going to go anywhere. If you have to do open source, this is like the best of the most
worlds, I think. It's available. It's open. You can see it, but you know you have a vendor
who has their business based on it that it's not going to go anywhere, and they're going
to put a lot of resources to making it better. So, you know, I'm just telling you right
now that we just came up with a JIT server. We're going into beta on Instant On. I don't
know if you've heard of that. It's based on CryU. So we're going to be able to take snapshots
of those images, and you can put those in your containers. Those are going to start
up in milliseconds. So JIT basically handles the JIT server, handles the ramp up time,
but Instant On will handle the start up time. So we're talking milliseconds. That's coming
out in the next couple of months or so. Anyway, thank you. Well, if you don't have the JIT,
then you're going to be running interpreted. That's like the worst of everything. Oh, well,
it won't be. But you still want to use the JIT remotely. Oh, you're talking about locally.
It will not be used. It will not be used. By the way, yeah. And by the way, the JIT server
is just another persona of the JVM. It's just running under a different persona. No, it
won't do that. Okay. Thank you very much. Okay. Thank you.