Hello everyone. My name is Chris Ma and I am the manager of the Java monitoring team
at Red Hat. Back in 2020, if you had come to this Java
Dev Room here at Fosdom, my colleagues talked about a project called ContainerJFR. And that
today is, Cryostat is the direct evolution of that project. Today I want to share with
you guys the progress of that, the project. Before I jump into Cryostat, I just wanted
to share a screenshot that depicts a demo of a sample pet clinic application that you've
probably all seen before. And for the purposes of this demo, that pet clinic application
is deployed in the cloud. So here's that pet clinic application. It does everything that
you would expect a sample pet application to do. Now let's just say as a developer, this
application has performance issues such as having abnormally high resource consumption
or abnormally high response latency. What would you do? Now I kind of spoiled it with
my title with Java Flight Recorder, but JFR is the first thing that pops into my mind.
I know my colleague Robert talked a little bit about JFR and his early presentation,
but as a quick refresher, for those who don't know what JFR Flight Recorder is, JFR is a
profiling and event collection framework for Java. It gathers low level application behavior
such as garbage collection metrics, memory allocation, and much more. It is built into
the JDK, is low overhead, and therefore is suited for production environments. You can
also use the low overhead recording infrastructure for your own event types. It pairs well with
JDK, JMC, which is JD Commission Control, which is an analysis tool used to visualize
JFR recordings in flame graphs, histograms, and many other visualizations. So going back
to the pet clinic application, how do I use Java Flight Recorder with this application?
Well, you might think I'll go into my trusty terminal, start JCommand, but then await. This
pet clinic application is actually deployed in the cloud. And so herein lies a number
of problems. I don't necessarily have access to the root terminal. Where and how do I access
the JFR outputs files? I don't have any place to store them. What if I have multiple applications
that I want to profile with JFR? Using JFR locally, I can only start it on one application
at a time. And then lastly, what if my application shuts down unexpectedly? There is no records
on spontaneous shutdown, and all of the JFR data would be wiped. So this is where I would
introduce CrowdStat, and how can CrowdStat help? So CrowdStat can be added to help manage
JFR recordings in the form of a web view. It is designed to work with Kubernetes pods,
and it used Kubernetes persistent volumes to backup recordings. It also has functionality
to capture JFR recordings if an application shuts down unexpectedly. Some additional benefits
is that we can also define rules to automatically create recordings for matching JVMs, view time
series metrics in Grafana dashboards. You can also export the JFR recordings for more
detailed analysis in JMC, and then it can be used for applications also developed on
frameworks such as Quarkis and Spring Boot to name a few. So I have a picture here of
the web console view when I first start CrowdStat. And in this example, as you can see on the
left-hand panel, is starting at the topology view. And this topology view gives you a sort
of a bird's-eye view of all the applications CrowdStat can connect to. Starting from the
bottom, you'll see that it is using the Kubernetes API. So it also works for Podman. So you would
see Podman in that, I guess, configuration. The second, my apps, is the namespace that
I have defined CrowdStats to be able to connect with applications within that namespace. And
then you can see the Spring PetClinic acts application there. The reason why in this
topology view you see that there's two PetClinic nodes is because I've scaled up the number
of PetClinic pods just to give you a flavor of the ability in scale deployments that CrowdStat
would be able to also access the multiple replicas. In the top view, you'll see there's
some filtering. So in a large, maybe large scale deployment, you might want to narrow
in on certain applications. These filtering capabilities allow you to specifically hone
in on our target application. One thing I would like to note is that you might be
wondering how I sort of ended up at this page and to sort of set up some context as
to how I was able to set up CrowdStats to work with these PetClinic applications. I have
a couple steps or a couple configuration that needed to happen in the background to get
a demo or this setup working. So one of the things I did was I installed the CrowdStat
operator on Operator Hub and installing the operator basically is a way of automating
the CrowdStat deployment. Another notable sort of configuration responsibility of the
operator is that it can help define the namespace boundaries in which CrowdStat can be used.
So in my particular example, I configured it in sort of the most basic simple case,
CrowdStat in a single namespace. CrowdStat does have the capabilities for multi namespace
support. So if you have applications that span across multiple namespaces, CrowdStat
can be used in those scenarios as well. And lastly, in order for CrowdStat to sort of
communicate with the target applications, there are two methods of connectivity. And
the first, I guess, methodology is using RemoteJMX. This requires setting up various
JMX environment variables, also potentially configuring TLS or SSL, and then also exposing
a port on the route of the Petcom link service. The other option is the CrowdStat agent. And
so this is the recommended approach. This would involve basically, we provide a CrowdStat
agent jar that you would build with your container image, and that would allow sort of auto
discovery of that target applications. So what we're going to do is we're going to
do a little bit of a demo. One of the next things that I wanted to show that's CrowdStat
is capable of is, of course, the sort of bread and butter of CrowdStat, which is creating,
creating JFR, well, starting JFR recordings. And so this is a sort of a simple simplistic
UI of how you would define the creating that JFR recording. You set the duration. You can
set the template. And this is synonymous to a JFR configuration file that you would use
when using JFR locally. Out of the box, it comes with the standard configuration files,
such as profiling it continuous. But you can also define your own sort of custom JFR configuration
files if there are specific events that you want to look for. Lastly, at the bottom, you'll
see there's show metadata options, and I haven't actually sort of expanded that a little bit.
There is the ability, we have added the ability to add labels to your recordings, and this
just allows users, you know, if you have a lot of recordings, you can sort of annotate
or add metadata to these recordings so that you can search for them quicker.
Once you create a recording, so when the recording is created, you can see that I've created
a recording one. It's put duration for 200 seconds. It's still running. What we've added
here is the automated analysis page from JMC. So if you've used JMC before, we basically
took sort of the guts of that automated analysis, like its library, and we've basically just
added it here for simpler access so that you don't have to switch between, you don't have
to necessarily switch between JMC and Criostat. You also see archived recordings in the top,
and that's basically if you create an active recording and you want to put it into storage,
that gives you that ability to do so.
So, as part of gathering the JFR recordings, we do have some, I guess, dashboards to have
a quick look at some of the data coming out from the JVM. Now what I would say is that
the data that you're seeing here is from the MBIN. These are MB metrics and not actual
data from the JFR recording. And so these can actually be, I guess, received in real
time or visualized in real time because the JVM is already, we're just basically visualizing
this data.
I did mention that there are some Grafana dashboards that are also available. And so here's an
example of a, I guess, a JFR snapshot with, I guess, certain metrics. We have some canned
templates that, on Grafana, that you can, when you create a JFR recording, you can open
this and you can see data specific to that, those sort of buckets. Similarly, you can
a little bit more detailed information about heaps. Heap memory, other threads as well,
are available within the Grafana dashboards. One quick thing I want to highlight here is
going back to the Polys view. As you can see, you can do bulk actions, meaning if you have
multiple, if you select multiple applications, you can start recordings for multiple pods
at the same time rather than doing them individually. That kind of simplifies things a little bit.
This is the page for automated rules. It follows the same mindset of bulk actions, but instead
you can sort of query the target application that you want to start the recording for. This
just allows a little bit more granularity when searching for a target JVM.
To kind of just wrap up where we're at with cryostat and what's next, there's a couple
things that we want to sort of focus on. One of them is smart triggers. Smart triggers,
to give you some context about that, smart triggers is a way of dynamically starting
JFR recordings. This would be done through the Java agent or the cryostat agent. Right
now we do have a preview feature where you can specify the metric and the metric value
on when you want the JFR recording to start and then using what kind of JFR configuration
file. What we would like to do with this feature is ultimately maybe make it more user-friendly
so that you don't have to add, say, arguments to the Java agent to start this.
The second thing was just cryostat agent auto configuration. We would like to have the
operator streamlined in the installation for the installation process with the cryostat
agent. I'd say those are the two main things to look out for. We have some upstream sites.
If you want to ask some questions or have any feedback for us, we're always listening
and that concludes my presentation.