Yeah. Hi, everyone. My name is Christian Simon, and I'm going to be talking about continuous
profiling. So we heard a lot about observability today already, and I'm going to want to introduce
maybe an additional signal there. So maybe quickly about me. So I'm working at Grafana
Labs. I'm a software engineer there, and worked on our databases for observability. So I worked
on Cortex slash Mamiir now. I worked on Loki, and most recently I switched to the flat team,
and I'm 50% of the flat team, and we kind of work on continuous profiling database. There's
not going to be a particular focus on flat today. So basically what I want to talk today
is kind of introduce how it's measured, what you can achieve with it, and then maybe as
I learn more, at the next first time I can go more into detail and look at very specific
languages there. So when we talk about observability, what are our common goals? Obviously we want
to ensure that the user journeys of our users are successful, that we maybe even can be
proactively find problems before a user notices it. And basically we want to be as quickly
as possible when those problems happen to disrupt less of those user journeys. And observability
provides us like an objective way of getting some insights into the state of our system
in production. And even after a certain event has happened, we found the right way, reboot,
and it's all up again. I think it might be able to help us understanding what exactly
happened when we want to figure out the root cause of something. So one of the I guess
easiest and probably oldest observability signals is logs. So like I guess it starts
with a kind of print hello somewhere in your code. And I guess you probably don't need
a show of hands who's using it. Like I guess everyone somehow uses logs or is asleep if
they don't show a hand. So basically your application doesn't need any specific SDK.
It can probably just log based on the standard library of your languages. One of the challenges,
most of the time the format is rather varied. So like even in terms of timestamps, it can
be really, really hard to get a common understanding of your log lines there. And also like when
you then want to aggregate them, they are quite costly. So like it spides, you need
to kind of convert them to in floats and so on. And also that something that can happen
is that like you have so many logs that you can't find the ones that you're actually interested
in. So like grab error, for example, could be, yeah, could be maybe helpful, but also
like there might be just too many errors. And you kind of lose the important ones. So
also like if you want to learn more about logs, my colleagues, Oven and Kavi, they're
going to speak about Loki. So definitely stay in the room. And I'm going to move on
to the next signal. So metrics, I'm also assuming pretty much everyone has come across them
and is using them. So in this case, you kind of avoid that problem I mentioned before.
You have the actual integers exposed. You maybe know about those integers that you care
about them. So to get a metric, most of the time you have to do some kind of define a
list of metrics you care about and then you can collect them. So it might be, you might
be having kind of an outage and didn't have that metric that you care about. And so you
need to kind of constantly improve on the exposure of the metrics. Obviously, like Peruvius
is the kind of main tool in that space. And very often we talk about web services, I guess,
when we think about those applications. So the red method, so like get the rates of your
requests, get the error rate of your request, and get the latency duration of the request
can already cover quite a lot of cases. And obviously, like as it's kind of just integers
or floats, you can aggregate them quite efficiently across like, yeah, a multitude of pods or
like a really huge set up of services. And then if you get more into that kind of microservices
architecture that has kind of evolved over the last couple of years, you will find yourself
kind of having a really complex composition of services being involved in answering requests.
And so like, you might even struggle to understand what's slowing you down or where the error
is coming from, why do I have this time out here? And distributed tracing can help you
a lot with kind of getting an understanding what your service is doing. It also might
be that like, maybe the service is actually doing way too much and you're calculating
things over and over again. So that is super helpful to get a bit more like the kind of
flow of the data in your system. So like the challenge there might be like, you might have
a lot of requests and that's, while it's somewhat cheap to get the tracing, you might not cover
all the requests. So for example, now production system, I think maybe someone needs to correct
me if I'm wrong, but like when we receive a Grafana cloud like logs and metric data,
we only cover 1% of those with traces while we cover 100% of our queries. So like, basically
you need to make a selective decision if it's worth investing that. Obviously logs data
looks always the same, it comes every second and so on. So like, we see more value in having
all of those queries where there's a complex kind of caching and all sorts of systems interacting
with it and so that allows us, yeah, to look a bit deeper and even then find that one service
that maybe is the bottleneck there. So maybe looking at a bit of a real problem, so I'm
having an online shop, I'm selling socks and a user is complaining about getting some time
out when wanting to check out. That's obviously not great because I'm not selling the socks,
but at least the user got some trace ID and is complaining to our customer service. Then
starting from there, I'm figuring out it's the location service that actually was the
one that cost the time out in the end. And then looking at the metrics of the location
service, I might find, oh, there's actually 5% of the requests timing out, so maybe 5%
of my users are not able to buy their socks monthly or whatever. So what are the next steps?
I guess scaling up is always good. Maybe the service is just overloaded. The person that
wrote it left years ago, so we have no idea. So we just scale it up. Obviously, it comes
with a cost and so I think always the first thing would be fixing the bleed, making sure
there are no more timeouts. So scaling up is definitely the right option here. But then
if you do that over years, you might suddenly find yourself having a lot of extra costs
attached to that location service. And so that's kind of where I think we need another
signal. And I think that signal should be profiling. So I guess most people might have
come across profiling. And it basically measures your code and how long it executes, for example,
or what kind of bytes it allocates in memory. And it basically helps you maybe understand
your program even more or someone else's program in the location service case. And that eventually
can translate in cost savings if you find out where the problem lies, like you can maybe
fix it or can get some ideas. Maybe you can also look at the fix and see if it's gotten
actually worse or better. And yeah, like that basically gives you a better understanding
of how your code behaves. And so now the question is, what is actually measured in a profile?
So I created a bit of a program. I don't know. I hope everyone can see it. So it's basically
like a program that has a main function and then calls out other functions. So you can
see there's a do a lot and there's a do little function. And both of them then call prepare.
And obviously in the comments, there's some work going on. And obviously the work could
be allocating memory, like using the CPU, something like that. So let's say we use CPU.
So when the function starts, like we are first going to do something within the main, let's
say we spend three CPU cycles, which is not a lot, but that then gets recorded like yes,
we took us three CPU cycles in main. We then go into the prepare method through do a lot.
Then we spend another five CPU cycles. And those kind of stack traces then they are recorded
in the profile. And going through the program, like we will end up with that kind of measurement
of stack traces. And while it kind of works with ten lines of codes, you can maybe spot
where the problem is. Like there's the 20 and do a lot. It definitely kind of breaks
down when you're speaking about like a lot of services or like a lot of code base that
is happened or happens to be hot and actively used. And so like there are a couple of ways
of visualizing them. I think one of the first things you would find is kind of a top table.
So like in that table, you can order it by different values. Like so this is kind of
an example from P prof, like which is kind of the go tool. And you can see kind of clearly
do a lot is the method that comes out on top. And like there are now different ways how
you can look at the value. So you have the flat count, which is the function itself only.
So you can see the 20 that we had before, 20 CPU cycles. But we also have the cumulative
measurement, which also includes the prepare that is going to get called from do a lot.
And so like we already can see we spend 52% of our program and do a lot. So maybe we already
can stop looking at the table and just look at do a lot. Because if we fix do a lot or
get rid of it, we need half of the CPU less. And that's that kind of, it's kind of represented
by the sum. So the sum will always change depending on how you order the table in that
particular example. So in this case, like we have 100% already at row number four, because
we only have four functions. And to get a bit more of a visual sense of what's going
on, there are the so-called flame graphs. And I think the most confusing thing for me
about them was the coloring. So obviously like red is always not great. Should we look
at main? No, we shouldn't. So basically like the coloring I think is random or uses some
kind of hashing. And basically it's only meant to look like a flame. So like the red here
doesn't mean anything. So like if you're colorblind, that's perfect for flame graphs. So what we
actually want to look at is this kind of like at the leaf end is basically where the program
would spend things. Like you can see the three CPU cycles main here. So there's nothing
below. So main uses 100% through methods that are called by main. And then there's nothing
beyond this here. So like here we spend something in main. And in the same way in do little,
we can see the five. And in do a lot, we can see the 20 quite wide. And then the prepares
with five each as well. And now obviously if you look across a really huge program, you
basically like can can spot kind of what's going on quite quickly. And then if you have
like similar with like route in main, like you basically can ignore that, but it helps
you maybe locate which component of your of your program you want to look at because
like maybe you're not good at naming and you call everything prepare in util and and it
would still tell you roughly where it gets called and and how the program gets there.
So how do we get that profile? And that can be kind of quite that can be quite a lot of
challenges how to get that. So I think I would say like there's like roughly two ways like
either your ecosystem supports kind of profiles fairly natively. And then you instrument the
profile, you added maybe a library and SDK. And like basically like the runtime within
your environment will maybe expose the information. So like it's not available for all languages.
There's I guess a lot of work going on that it becomes more and more available. And kind
of other approaches more like through an agent and EPPF has been quite hyped. I'm I'm not
very familiar with the EPPF myself. I have used the agents but haven't written any code.
But basically what it would use it would use an outside view of it. So you wouldn't need
to change the binary running really like you would just kind of look at the information
you get from the Linux kernel like you hook into, I don't know, often enough when the
CPU runs to then find out what is currently running. So there are different languages
like for example in a compiled language. You would be having a lot more information. The
memory addresses are the same. You can kind of use the information within the simple table
to figure out where your program is and what is currently running. In like I don't know
like an interpreted language like Ruby, Python, this might be a bit harder and that information
might not be accessible to the kernel without further work. Like it also when you compile
you might drop those simple tables like so that you really need to kind of be preparing
your application a bit for that. I want to look into the kind of prime example I'm most
familiar with. I'm mostly a go developer over the last couple of years. Go has quite a kind
of mature set of tools in that area. So basically the standard library allows you to expose
that information. It supports like CPU and memory. And especially garbage collected
languages. Memory is quite a thing to also non-garbage collected languages. But memory
is really important to understand the usage there. I have a quick example of a program
like so you basically like just expose an HTTP port where you can download the profile
whenever you want and you have that P prof tool that you can point to it. So like in that
kind of first line example you would just get a two second long profile. So CPU profile
that looks at the CPU for two seconds and basically records like whatever is running
how long on the CPU and then you get the file and P prof will allow you to visualize it through
that top table for example. So what I forgot to mention as well. So later you can maybe
go to that URL and look at that profile that I had as an example. And in the same way you
can get the memory allocations and P prof also allows you to launch an HTTP server to
be a bit more interactive and select certain code paths. So that is like quite a lot in
the go docs, go.dev about profiling. So I definitely leave it there. So you can also
look at kind of maybe if you are a go developer like use that and play around yourself. But
now I want to speak about why profiling might be actually quite difficult. So the example
I had like I had three CPU cycles and if you think about that is not very much. So and
just to record what the program was doing in those three CPU cycle probably takes I
have no idea about thousands of CPU cycles. And so you really want to be careful what
you want to record. So if you really would record all of that like your program would
probably have like a massive overhead would slow down by profiling behave completely different
and you also would have a lot more data to store to analyze. And then if you think about
micro services and replica count 500 you might get quite a lot of data that is actually
not that useful to you because are you really caring about three CPU cycles? Probably not.
And because of that to allow continuous profiling so to do that in production across like a
wide set of deployments like I think Google were the first ones to do that and they were
starting to sample those profiling. So instead of looking at really every code that runs
go for example looks 100 times a second what is currently running on the CPU and then records
it and obviously maybe like integer adder will not be on the CPU if you don't run it
all the time and so you get a really accurate representation what is really taking your
CPU time. And the way that works you also need to be kind of aware that like some things
the actual program might not be on the CPU because it might be waiting for IO and so
like when you kind of collect a profile and the profile is not having that many seconds
you really need to think about is this really what I want to optimize or maybe I'm not seeing
what I actually want to see. With that kind of statistical approach like I don't have
any kind of sources to say but like I think generally you say that it's like a two to
three percent overhead that gets added on top of your program execution so that's I guess
a lot more reasonable than the full approach with recording everything. And so what do
you generally kind of would do obviously if you first need to ship your application somewhere
and run it then you can look at the profiles and yeah think about it look at it like maybe
you are the owner of that code maybe you have a bit more understanding and those profiles
maybe can give you a bit more of an idea of what you're actually doing there or how the
system is reacting to your code. And so basically like for that green box multiple solutions
exist so I'm obviously a bit biased but I also have to say our project is fairly young
and evolving. So for example there's like CNCF Pixie, EBPF based, there's Polar Signals
Parker like people are in the room, Pyroscope and kind of our solution. I think they're
all great like you can all use them and start using them and exploring like maybe just your
benchmarks for a start and then as you get more familiar with it like you might kind
of discover more and more of the value there. So I'm still going to use Flare now for my
quick demo. So let me just see. So I guess most of you are kind of familiar with Grafana
and Explore. Why is it so huge? And so basically that's kind of the entry point you're going
to see in the Explore. You have the kind of typical time range selection so let's say
we want to see the last 15 minutes now and here we can see the profile types collected.
So that's just a Docker compose running locally on my laptop, hopefully running locally on
my laptop since I started to talk. And for example we can here look at the CPU. So that's
kind of the nanoseconds band on the CPU and you can kind of see the flame graph from earlier
and maybe some bug. I don't know. It looks a bit bigger than it usually should be. But
we can see that kind of top table. We can see the aggregation of all of the services.
So I'm running like five pods or something like that, different languages. So you can
see like for example this here is like a Python main module where it's doing some prime numbers.
So what I first want to kind of break down here is by label. And that's really the only
kind of functionality that we have in terms of querying. So here we would look at the
different instances and we kind of see the CPU time spent like, I don't know, there's
like a Rust port and they are both blue so I don't know which switch, but I guess Flare
is doing more. So that might be the Flare one. And for my example now I want to look
at just like a small program that I wrote to show like how like the aspect. So like here
we can see kind of the timeline. So this is like a profile gets collected I think every
15 seconds and that's basically a dot. And then the flame graph and the top table below
would kind of just aggregate that. So like there's no time component in here. That's
also important to understand. And so like while we were looking at memory I'm now going
to kind of switch to the allocated space. And oh no. And here we have some label selection
like that you might be familiar. And this random port here you can see like the allocation
so the amount of memory that gets allocated is like around six megabytes. But then every
couple of every five minutes roughly you can see like some peak. And so if you already
look in the flame graph there's already some kind of big red box and the colors don't matter.
But basically like you can see this this kind of piece of code is doing kind of a majority
of the allocations. And now you could even kind of zoom in here if you really want to
figure out and then it even gets bigger and you can see some more what's going on. And
so now if you actually want to look at the kind of code for this. And if flare is maybe
in version 0.6 we could even see the line of code that we should look at for now you
can. But basically like it would show us allocations in line 21. And I guess most of you can see
what this kind of program is doing so every five minutes it will kind of have some peak
of allocations. And you only see that kind of because you have the time component you
can select and then see the flame graph aggregation. Cool. Yeah that was almost my talk. Like I
have one more slide that I should just quickly want to. So in the latest version 120 there
is profile guided optimizations and I think that might be a really big topic. So what
it does it looks at kind of your profile and that can come from production from benchmarks
from whatever and tries to do the better decisions during compile time of what things to do with
your code like for example think the only thing that it does right now is making inlining
decisions. But basically like if it sees this method is called a lot and is in the hot path
it would then make the decisions to inline the method maybe if it's a bit colder it would
not do it and you can be a lot more accurate as a compiler if you have that kind of data
if you know that method is in the hot path or not. Okay that was it. Thank you.
Thanks a lot that was awesome. Questions. Thank you. Thank you for the talk. I'm just
wondering how would the profiling work with very multi-threaded code. Is there ability
to drill down into that level. Yeah so like maybe so like in terms of multi-threading
like obviously we only have the main method in that example. So you can see rude and then
mine is 100% and like if it's multi-threaded you would have kind of maybe more so it's
basically all only the stack trace that gets recorded like you would not see kind of maybe
the connections where the thread where it's threading off and things like that. You would
get the stack trace. Cold stack. Have you looked into any other profiling formats than
B prof ingestion. I know open telemetry has been doing some stuff about introducing a profiling
format that people can standardize on but I don't know if you've looked at that at all.
Yes. Can you like I haven't seen you like sorry can you repeat like I struggled to.
So I was wondering if you've looked at any other profiling ingestion formats other than
B prof. No I like I or so like right now we use P prof personally with the player. So
I think there's a lot of kind of improvements to be had over over the format there and that's
like as far as I know some active work around open telemetry to to come to I guess a better
format in the sense to not send symbols over and over again and reduce interest but not
no it's the the accurate and short answer. Okay so thank you for the talk and my question
is that looking at the flare architecture it's currently pool models so the flare agent
is scraping the profiling data from the applications that they configure it to scrape. My question
is is there an eventual plan to also add maybe a push gateway or similar API for applications
where this might be suitable. Yeah like definitely like I think I also can see kind of the push
use case for maybe if you want to get your micro benchmarks from CI CD in so like the
API in theory allows it but tooling is missing but I definitely think it's a valid like push
use case as well. I think in terms of scalability I think pool will be better but yeah I agree.
Thanks for the talk. I have a small question. Did you try to implement this tooling in the
end of the CI CD and CO continuous optimization? No like so we're not using it yet for that.
I think it's it's definitely a super useful thing because like yeah like you want to see
maybe how a pool request behaves like maybe how your application allocates more or less
in different parts and and if the trade-offs are right there but yeah I think it definitely
can and should be used for that but no no tooling right now. Yeah no I fully agree as
well yeah. Hello thank you. So if I understand correctly profiles such as traces combined
with OS metrics right so at a concrete specific time you can see how much CPU you used and
so on right. Yeah I guess it looks a bit more at the actual line of code rather than I don't
know like I don't know I haven't used like tracing where it automatically finds the function
maybe that also tells you the line of code but yeah like it definitely adds some metrics
to it like without you doing much I guess other than making sure it can read the symbol
tables and the function names. Yeah so so I just had like a dumb question or like dumb
idea couldn't you just combine for example you already have node exporter which exposes
metrics at all times so you have OS metrics and you have traces for example so couldn't
you just have some kind of integration graph on or or somewhere else that just combines
traces with metrics. Yeah so like I think that was also the like like people that work
longer at continuous profiling software that they try to kind of reuse kind of Prometheus
and I think where you end up kind of in it's just a very high cardinality it's too many
lines of codes and and that's kind of where it stops but like in theory like I guess most
promql constructs and functions are maybe something we need to implement on top of that
in a similar way because in the end you just get metrics out of it and so basically the
problem was too many lines of code too much changing over time and like you just get too
much serious turn through that. So thanks a lot. Yeah thank you for coming.