Okay. Hello, everyone. Welcome to the talk on open telemetry with Grafana. Microphone
broke, so I need to do it with this microphone now. Let's see how it goes with typing and
live demo. Few words about me, so who am I, why am I here talking about Grafana and open
telemetry, so I work at Grafana Labs, I'm an engineering manager, I'm also a manager
for our open telemetry squad, and I'm also active in open source, so I'm a member of
the Prometheus team where I maintain the Java metrics library. So what are we going to do
in this talk in the next 25 minutes or so, so it will almost exclusively be a live demo,
so basically the idea is I have a little example application running on my laptop, and it is
instrumented with open telemetry, I will show you in a minute what it does and how I instrumented
it, and I also have an open source monitoring back and running, right, it consists of three
databases, one is Loki, which is a open source logs database, one is Temple, which is an
open source trace database, and one is Mimir, which is an open source metrics database, so
Mimir is compatible with Prometheus, so I could have shown the exact same demo using
Prometheus instead of Mimir, so it doesn't really matter for now. And of course I also
have Grafana, I have those databases configured as data sources, and what we are going to
do, we are going to start up Grafana, you know, have a look at metrics, have a look
at traces, have a look at logs, and basically the idea is that at the end of the talk you
kind of have seen all the signals that come out of open telemetry, you know, explore a
bit what you can do with this type of data, and so you should have a good overview how
open source monitoring with open telemetry looks like, right? So last slide before we
jump into the live demo, so this is just a quick overview of what the example application
does so that you know what we are going to look at. It's a simple hello world rest service
written in Java using Spring Boot, and so basically you can send a request to port 8080
and it will respond with hello world, and in order to make it a bit more interesting,
I made it a distributed hello world service, so it doesn't respond directly, but when it
receives a request, it reaches out to a greeting service running on port 8081, the greeting
service responds with the greeting, which is hello world, and then the response is forwarded
to the client, right? And there are random errors to have some error rates as well, so
basically a hello world microservice architecture or whatever, right? And in order to instrument
this with open telemetry, I use the Java instrumentation agent that's provided by the open telemetry
community, that's something you can download on GitHub, and the thing is this thing, you
basically attach it to the Java virtual machine at start up time with a special command line
parameter, so I didn't modify any source code, I didn't use any SDK or introduce any custom
stuff, all we are going to look at in this demo is just data produced by attaching the
open telemetry instrumentation to a standard Spring Boot application, right? Cool. So let's
get started. As said, I have my data sources configured here, so Prometheus and Mimera are
compatible, so it doesn't really matter which one we choose. There are a lot of, so I want
to start with metrics, and yeah, so... Yeah? Can we turn the lights down a bit? I don't
know. Okay. Maybe the other way around. Okay. I will just continue, come on. So there are
lots of metrics that you get from the open telemetry instrumentation, so kind of JVM-related
stuff like garbage collection activity and so forth, but the one I want to look at, oh,
no, it's getting brighter and brighter. Yeah. Okay. Great. I think there is also a light
mode in Grafana. Maybe that would have been a better choice. But no, I'm not going to
use light mode. So let's figure out how to do the demo while I have a microphone that
I should hold in my hands. Let's just put it here. Okay. Thank you. Cool. So the metric
that we are going to look at for the demo, it's a metric named HTTP server duration.
This is a metric of type histogram. So histograms have a couple of different numbers attached
to them, so there are histogram buckets with the distribution data and so forth, and there's
also a count. The count is the most simple one, so we are going to use this in our example.
I actually got it two times. I got it once for my greeting service here and once for
the hello world application. And if we are just, you know, running this query, maybe
take a little bit of a shorter time window here, then we basically see two request counters,
right? One is the green line, which is counting the request resulting in HTTP such as 200.
So the successful requests, and basically we see that since I started the application
on my laptop, I got about a little more than 400 successful requests, and the yellow line
is, you know, requests resulting in HTTP status 500, and we got around 50 of them, right?
And obviously, raw counter values are not very useful, right? Nobody is interested in
how often was my service called since I started the application, and the way, you know, metric
monitoring works with Prometheus, as probably most of you know, is that you use the Prometheus
query language to get some useful information out of that kind of data, right? And I guess
most of you have run some Prometheus queries, but they're still going to show maybe a couple
of examples. So for those of you who are not very familiar with that, does this one work
again? Hey, nice. It's even better. The lights work, the microphone works. Wow. Now let's
hope the demo works. So I'm going to run just a couple of quick, you know, Prometheus queries
so that for those of you who are not very familiar with it, so that you get an idea
of what it is, right? And the most important function in the Prometheus query language
is called the rate function. And what the rate function does, it takes a counter like
this and a time interval like five minutes, and then it calculates a per second rate,
right? So based on a five minute time interval, we now see that we have about 0.6 requests
per second resulting in HTTP status 200, and we have about 0.1 requests per second resulting
in HTTP status 500. And this is already quite some useful information, right? So typically
you want to know the total load on your system, not buy status code or something. So you basically
want to sum these two values up, and obviously there's also a sum function to sum values
up, and if you call that, you get the total load on your system, which is just one line
now and it's just, you know, around 0.7 requests per second, right? And this is, yeah, this
is basically how Prometheus queries work. If you're not familiar with the syntax, there's
also kind of a graphical query builder where you can, you know, use a bit drag and drop
and get a bit more help and so forth, right? And so eventually, you know, when you got
your queries and got your metrics, so what you want to do is you create a metrics dashboard
and for monitoring HTTP services, there is, there are a couple of best practices, what
type of data you want to visualize on a dashboard for monitoring HTTP services. And the most
simple and straightforward thing is to visualize three things. One is the request rate, so
for the current load on the system, which is exactly the query that we are seeing here.
The next thing you want to see is the error rate, so the percentage of calls that fail.
And the third thing is duration. How long does it take, right? And I created a simple
example dashboard just to show you how this looks like. So I put the name of the service
as a parameter up here so we can reuse the same dashboard for both services. Maybe let's
use a 15 minute time window, so here I started the application. The first is the request
rate, that's the exact same query that we just saw. Second thing here is the error rate,
so we have about, I don't know, around 10% errors in my example application. And then
for duration, there are a couple of different ways how to visualize that. So what we see
here is basically the raw histogram, right? The histogram buckets. And this representation
is actually quite useful because it shows you the shape of the distribution. So what
we see here is two spikes, one around 600 milliseconds and one around 1.8 seconds. And
this is a typical shape that you would see if your application uses a cache, right? Because
then you have a couple of requests that are responded quite quickly. Those are the cache
hits. A couple of requests are slow that are the cache misses. And visualizing the shape
of the histogram helps you understand kind of the latency behavior of your application,
right? The other and most popular way to visualize durations is this one here. These
are percentiles. So the green line is the 95th percentile, so it tells us 95% of the
calls have been faster than 1.7 seconds and 5% slower than that. The yellow line is the
50th, so half of the calls faster than that, half of the calls slower than that. And this
doesn't really tell you the shape of the distribution, but it shows you a development over time,
which is useful as well. So if your service becomes slower, those lines will go up, right?
And it's also a good indicator if you want to do alerting and so forth. You can define
a threshold and say it's above, if it's above a certain threshold, I want to be notified
and stuff like that. And there are other more, you know, experimental things like this heat
map showing basically development of histograms over time and stuff like that. So it's pretty
cool to play with all the different visualizations in Grafana and, you know, see what you can
get. So this is a, you know, quick example of a so-called red dashboard, Request Rates,
Error Rates duration based on open telemetry data. And the cool thing about this, about
it, is that it actually, all that we are seeing here is just based on that single histogram
metric HDP server duration. And the fact that this metric is there is not a coincidence.
The metric HDP server duration is actually defined in the open telemetry standard as
part of the semantic conventions for HDP services. So whenever you monitor an HDP server with
open telemetry, then you will find a histogram named HDP server duration. It will have the
HDP status as an attribute. It will contain the latencies in milliseconds. That's all
part of the standard. So it doesn't matter what programming language your services uses,
what framework, whatever. If it's being monitored with open telemetry and it's compatible, you
will find that metric and you can create a similar dashboard like that. And this is kind
of one of the things that make application monitoring with open telemetry a lot easier
than it used to be before these standardization. Cool. So that was a quick look at metrics,
but of course we want to look at the other signals as well. So let's switch data sources
for now and have a look at traces. So tracing, again, there's a kind of search, like graphical
search where you can create your search criteria with drag and drop. There's a relatively new
feature which is a query language for traces. So I'm going to use that for now. And one
thing you can do is to just search by labels. So I can, for example, say I'm interested
in the service name greeting service and then I could basically just open a random trace
here. Let's take this as an example. Can I, I need to zoom out a little bit to be able
to close the search window here. Okay. So this is how a distributed trace looks like.
And if you see it for the first time, it might be a bit hard to understand, but it's actually
fairly easy. So you just need like two minutes of introduction and then you will understand
traces forever. And to give you that introduction, I actually have one more slide. So just to
help you understand what we are seeing here. And the thing is distributed traces consist
of spans, right? And spans are time spans. So a span is something that has a point in
time where it starts and a point in time where it ends, right? And in open telemetry, there
are three different kinds of spans. One are server spans. The second is internal spans
and the third is client spans. Okay. So what happens when my Hello World application receives
a request? So the first thing that happens if a server receives a request, a server span
is created. So that's the first line here. It's started as soon as the request is received.
It remains open until the request is responded, right? Then I said in the introduction that
I used spring boot for implementing the example application. And the way spring boot works
is that it takes the request and passes it to the corresponding spring controller that
would handle the request. And open telemetries Java instrumentation agent is nice for Java
developers because it just creates internal spans for each spring controller that is involved,
right? And that is the second line that we are seeing here. It's basically opened as
soon as the spring controller takes over and remains open until the spring controller
is done handling the request, which might seem not too useful if I have just a single
spring controller anyway, but if you have kind of a larger, you know, application and
if you have multiple controllers involved, it gives you quite some interesting insights
into what's happening inside your application. Like you would see immediately, like which
controller do I spend most time in and so forth, right? And then eventually my Hello
Word application reaches out to the greeting service and outgoing requests are represented
by client spans. So the client span is basically opened as soon as my HTTP request goes out
and remains open until the response is received. And then in the greeting service, the same
thing starts again, you know, request is received, which creates a server span and then I have
a spring controller as well, which is an internal span and that's the end of my distributed
application here. And this is exactly what we are seeing here. And each of those span
types has a corresponding metadata attached to it. So if you look at one of the internal
spans here, we see the name of the spring controller and the name of the controller
method and a couple of JVM-related attributes, whatever. And if we look at an HTTP span,
for example, we see, of course, HTTP attributes like the status code, method and so forth,
right? So of course, you do not want to just look at random spans. So usually you're looking
for something. There are standard attributes in open telemetry that you can use for searching.
So we already had the service name greeting service, for example. But the most important
or one of the most important attributes is HTTP.status, no,.status code, this one here.
And if we, for example, search for spans with HTTP status code 500, then we should find
an example of a request that failed. So let's close the search window again. Yes, that's
an example of a failed request. You see it with the indicated by those red exclamation
marks at the bottom here. So this is where the thing failed, right? So the root cause
of the error is the internal span, something in my spring controller in the greeting service.
If I look at the metadata attached to that, I actually see that the instrumentation attached
the event that caused the error, and this even includes the stack trace. So you can
basically immediately navigate to the exact line of code that is the root cause of this
error, right? And this is quite cool. So if you have a distributed application and you
get an unexpected response from your Hello World application, without distributed tracing,
it's pretty hard to find that actually there's an exception in the greeting service that,
you know, propagated through your distributed landscape and then eventually caused the unexpected
response. And with distributed tracing, finding these kind of things becomes pretty easy because
you get all the related calls grouped together, you get the failed ones marked with an exclamation
mark, and you can pretty easily navigate to what's the root cause of your error, okay?
Cool. So that was a quick look at traces. There are a lot of interesting things about
tracing. Maybe one thing I would like to show you, because I find it particularly cool,
so if you have all your services instrumented with tracing in your back end, then basically
those traces give you metadata about all the network calls happening in your system,
and you can do something with that type of data, right? So for example, you can calculate
something that we call the service graph. So it looks like this. It's maybe not too impressive
if you just have two services calling each other, right? So, but if you imagine, you
know, a more larger, you know, dozens or hundreds of services, so it will generate a map of
all the services and indicate which service calls which other service, and this is quite
useful. For example, if you intend to deploy a breaking change in your greeting service
and you want to know who's using the greeting service, what would I break? Then looking
at the service graph, you basically get this information right away. Traditionally, if
you don't have that, you basically have a PDF with your architecture diagram, and then
you look it up there, and also traditionally, there's at least one team that deployed something
and forgot to update the diagram, and then you missed that, and there's a service graph
that won't happen, right? This is the actual truth. This is based on what's actually happening
in your backend, and this is pretty useful in these situations, right? And you can do
other things as well, like, you know, have some statistics like the most frequently called
endpoint or the endpoint with the most errors and stuff like that. So, that was a quick,
quick look at traces. So we covered metrics, we covered traces. One thing I want to show
you is that metrics and traces are actually related to each other, right? And so in order
to show that, I'm going to go back to our dashboard, because if you, let's take a 15
minute window, then we get a bit more examples. So if you look at the latency data here,
you notice these little green dots. These are called exemplars, and this is something
that's provided by the auto instrumentation of open telemetry. So whenever it generates
latency data, it basically attaches trace IDs of example traces to the latency data,
and this is visualized by these little green dots, right? And so you see some examples
of particularly fast calls, some examples of particularly slow calls and so forth.
And if you, for example, take this dot up here, which is kind of slower than anything
else, it's almost two seconds, right? Then you have the trace ID here, and you can navigate
to tempo and have a look at the trace and start figuring out why did I have an example
of such a slow call in my system, right? And in that case, you would immediately see that
most of the time spent in the greeting service. So if you're looking for the performance bottleneck,
then this is the most likely thing. Yeah, four minutes, that's fine. Cool. So if I have
four minutes, it's high time to jump to logs, the third signal that we didn't look at yet.
So let's select Loki, our open source logs database as a data source. So again, there's
a query language, there's a graphical query builder and so forth. So let's just open random
logs coming from the greeting service. It looks a bit like this. So it's even, I don't
know, I didn't even log anything explicitly. I just turned on some whatever spring request
logging so that I get some log data. And from time to time, I throw an exception, which
is an IO exception to simulate these errors. Looks a bit broken, but that's just because
of the resolution that I have here. Yeah, so what you can do, of course, you can do some
full text search, for example, can say, I'm interested in these IO exception. And then
you would basically get, well, if you spell it correctly, like that, then you would get
the list of all IO exceptions, which in my case are just the random errors I'm throwing
here. And this query language is actually quite powerful. So you can, this is kind of
filtering by a label and filtering by full text search, but you can do totally different
things as well. For example, you can have queries that, you know, derive metrics based
on log data. There's a function pretty similar to what we have seen in the metrics demo,
which is called the rate function. So the rate function, again, takes a time interval
and then calculates the per second increase rate. So it basically tells you that we have
almost 0.1 of these IO exceptions per second in our log data, which is also kind of useful
for information to have. And the last thing to show you, because that's particularly interesting,
so it is that these logs and traces and metrics are, again, not independent of each other.
They are related to each other. And so if we look at an example here, just let's open
a random log line. So what we see here, there's a trace ID. And this is interesting. So how
does a trace ID end up in my log line? So this is actually also a feature of the Java
instrumentation that's provided by the OpenTelemetry community. So the way logging in general works
in Java is that there's a global thing with key value pairs called the log context. And
applications can put arbitrary key value pairs into that context. And when you configure
your log format, you can define which of those values you want to include in your log data.
And if you have this OpenTelemetry agent attached, then as soon as a log line is written in the
context of serving an HTTP request, then the corresponding trace ID is put into that log
context. And you can configure your log format to include the trace ID in your log data.
And that's what I did. And so each of my log lines actually has a trace ID. And so if I
see something fancy and I want to know maybe somewhere down my distributed stack something
went wrong, I can just query that in tempo, navigate to the corresponding trace, close
that here, yeah, and then basically maybe get some information what happened. And then
the same navigation works the other way around as well. So of course, there's a little, you
know, log button here. So if I see something fancy going on in my greeting service thing
here, and maybe the logs have more information, I can click on that, navigate to the logs.
And then it basically just generates a query, right? I click on the greeting service with
that trace ID. So it's basically just a full text search for that trace ID. And so I will
find all my corresponding log lines. In that case, just one line. But if you have a bit
better logging, then maybe it would give you some indication what happened there. Okay.
So that was a very quick 25 minutes overview of, you know, looking a bit into metrics,
looking a bit into tracing, looking a bit into logs. I hope it gave you some impression,
you know, what's the type of data that you get out of open telemetry looks like. All
of what we did is really, you know, without even modifying the application. I didn't,
you know, even start with custom metrics, custom traces and so forth. So but it's already
quite some useful data that we get out of that. If you like the demo, if you want to
explore it a more, a bit more, want to try it at home, I pushed it on my GitHub and there's
a readme telling you how to run it. So you can do that. And yeah, next up, we have a
talk that goes a bit more in detail into the tracing part of this. And then after that,
we have a talk that goes a bit more into detail how to run open telemetry in Kubernetes. So
stay here and thanks for listening. Please remain seated during Q&A. Otherwise, we can't
do a real Q&A. So please remain seated. Order any questions. Yes.
Hi. Thank you for this. One quick question. You mentioned you just need to add some parameters
to the Java virtual machine to run the telemetry. What happens to my application if, for example,
the back end of the telemetry is down? Is my application failing or impacted in any way?
If the monitoring back end is down. Yes. Say the monitoring is down, but I started my application
with these parameters. Is it impacting the application? No. I mean, you won't see metrics,
of course, if you're monitoring back end is down, but the application would just continue
running. So typically, in like production setups, the applications wouldn't send telemetry
data directly to the monitoring back end. But what you usually have is something in
the middle. There's alternatives. There's the Grafana agent that you can use for that.
There's the open telemetry collector that you can use for that. And it's basically a
thing that runs close to the application, takes the telemetry data off the application
very quickly, and then, you know, can buffer stuff and process stuff and send it over to
the monitoring back end. And that's used for decoupling that a little bit, right? And if
you have such an architecture, the application shouldn't be affected at all by that.
Two more. Two more.
So I really like being able to link from your metrics to traces. But what I'm actually
really curious to be able to do, and as far as I know, doesn't exist, or I guess that's
my question, is like, is there any thought towards doing this, is being able to go the
other direction, where what I'd like to be able to answer is, here's all my trace data,
and this node of the trace incremented these counters by this much. So I could ask things
like how much network IO or disk IOPS did this complete request do, and where in the
tree would that occur?
Yeah, that's a good question. I mean, linking from traces to metrics, it's not so straightforward,
because I think the things you can do to relate this is to use the service name. So if you
have the service name part of your resource attributes of the metrics, and consistently
you have the same service name in your trace data, then you can at least, you know, navigate
to all traces coming, to all metrics coming from the same service. Maybe you have some
more, you know, related attributes, like in whatever instance ID and so forth. But it's
not like really a one-to-one relationship, so.
That's specific. What request, how much did this request come from the IOPS?
Yeah, no, I don't think that's possible.
So in this example, you've shown that Grafana World and Prometheus works great with server-side
applications. Have you had examples of client-side applications, mobile desktop applications that
use Prometheus metrics and then ship their trace, their metrics and traces to the metric
backend?
Did I hear it correctly? You're asking about starting your traces on the client-side and
the web browser and stuff?
You have tracing on the server-side, but what about having traces and metrics on the client-side
and, for example, for an embedded or mobile application so that you could actually see
the trace from when the customer clicked a thing and see the full customer journey?
Yeah, that's a great question. That's actually an area where there's currently a lot of research
and new projects and so forth. So there is a group called real-user monitoring, RUM,
in open telemetry that deal with client-side applications. There's also a project by Grafana.
It's called Faro. It's kind of, you know, JavaScript that you can include in your front
end, in your HTML page, and then it gives you traces and metrics from in the web browser
coming from the web browser. And this is currently a pretty active area, so lots of, you know,
movement there.
And so there are things to explore. So if you like, check out Faro. It's a nice new project
and standardization is also currently being discussed, but it's newer than the rest of
what I showed you, right? So it's not as, so there's no, you know, clear standard yet
or nothing decided yet.
Cool. Okay. Thanks, everyone, again.