Hi, everybody. Thanks to be here for this talk. That's a lot of people. I'm Nicolas
Frankel. I've been a developer for a long time, and I would like to ask how many of
you are developers in this room? Quite a lot. Who are ops? Just as many, and who are devops,
whatever you mean by it. So this talk is intended for actually developers, because I was, or
I still think I'm a developer. So if you are an ops people, and for this, for you is not
that super interesting. At least you can direct your developer colleagues to the talk, so
that you can understand how they can ease your work. Well, perhaps you've never seen
that, but I'm old or experienced, depending on how you see it. And when I was starting
my career, monitoring was like a bunch of people sitting in front of screens the whole
day. And actually, I was lucky. Once in the south of France, I was told, hey, this is
the biggest monitoring site of all France. And actually, it really looked like this.
And of course, there were people watching it. And that was the easy way. Now, I hope that
you don't have that anymore, that it has become a bit more modern. Actually, there is a lot
of talk now about microservices, right? Who here is doing microservices? Yeah. Yeah, because
if you don't do microservices, you are not a real developer. But even if you don't do
microservices, so you are not a real developer, and I encourage you not to be a real developer,
in that case, you probably are doing some kind of distributed work. It's become increasingly
difficult to just handle everything locally. And the problem becomes, yeah, if something
bad happens, how can you locate how it works? Or even if something works as expected, how
you can understand the flow of your request across the network. I love Wikipedia. And
here is the observability definition by Wikipedia, which is long and in that case, not that interesting.
So I have a better one afterwards for tracing. So basically, tracing helps you to understand
the flow of a business request across all your components. Fabian, where is Fabian?
Fabian is here, so he talked a lot about the metrics and the logging. So in this talk,
I will really focus on tracing because my opinion is that, well, metrics is easy. We
do metrics since ages, like we take the CPU, the memory, whatever. Now we are trying to
get more business-related metrics, but it's still the same concept. Logging also. Now
we do aggregated logging. Again, nothing mind-blowing. Tracing is, I think, the hardest part.
So in the past, there were already some tracing pioneers. Perhaps you've used some of them.
And well, now we are at the stage where we want to have something more standardized.
So it starts with the trace context from the W3C. And the idea is that you start a trace
and then other components will get the trace and will append their own trace to it. So
it works very well in a web context. And it defines two important concepts that Fabian
thanks already described. So now I am done. So I have the same stupid stuff. So here you
have, oh, sorry. Yes. It reminds me of the story. I did the same to my colleagues. They
didn't care about the presentation. They only remember that. Okay. So here you have a trace
and here you have the different spans. So here the X1 is the parent one. And then the
Y and the Z1 will take this X span as their parent span. And so this is a single trace.
This is a single request across your service. Web stuff is good, but it's definitely not
enough. And so for that we have the open telemetry stuff. Open telemetry is just a big bag of
miracles all set into a specific project. So it's basically APIs, SDK, tools, whatever
under the open telemetry level. It implements the W3C trace context. If you have been doing
some kind of tracing before, you might know it because it's like the merging of open tracing
and open sensors. Good thing is a CNCF project. So basically there is some hope that it will
last for a couple of years. The architecture is pretty simple. Basically you've got sources,
you've got the open telemetry protocol, and as Fabian mentioned, you dump everything into
a collector. Collector, we should be as close as possible to your sources. And then some
tools are able to read like data from it and to display it into the way that we expect
to see it. What happens after the open telemetry collector is not a problem of open telemetry.
Just they are collectors that are compatible, and for example you can use Yeager or Zipkin
in a way that allows you to dump your data, your open telemetry data into Yeager or Zipkin
into the open telemetry format. So you can reuse, and that is very important, you can
reuse your infrastructure if you're already using the tools, but just switching to open
telemetry. And then you are like you are using a standard, and then you can switch your open
telemetry back end with less issues. Now comes the fun developer part. If you are a developer,
you probably are lazy. I know, I'm a developer. So the idea is open telemetry should make
your life as a developer as easy as possible to help your ops colleague, like diagnose
your problems. And the easiest part if you do auto instrumentation. Auto instrumentation
is only possible in cases where you have a platform, when you have a run time. Fabian
mentioned Java, Java as a run time, which is the Jivem. Python as a run time. Now if
you have rusts, it's not as easy. So in that case, you are stuck. My advice if you are
using a run time, and probably most of you are using such run times, whether Java, whatever,
use it. It's basically free. It's a low hanging fruit, and there is no coupling. So basically
you don't need extra dependencies as developers in your projects. So since it's called practical
introduction, let's do some practice. So here I have a bit better than the hello world,
so I have tried to model like an e-commerce shop with very simple stuff. It starts just
asking for products. I will go through an API gateway which will forward the product
to the catalog, and the catalog doesn't know about the prices, so it will ask the prices
from the pricing service, and it will ask the stocks from the stock service. The intra
point is the most important thing, because it gives the parent's phrase. Everything will
be from that. So in general, you have a reverse proxy or an API gateway, depending on your
use case. I work on the Apache API 6 project. It uses the NGINX reverse proxy. On top you
have an open resty, because you want to have Lua to script and to auto reload the configuration.
Then you have lots of out of the box plugins. Let's see how it works. Now I have the code
here. Is it the begin off? Good. So I might be very old, because for me it wouldn't. Okay,
here that's my architecture. I'm using Docker compose, because I'm super lazy. I don't want
to use Kubernetes, so I have Yeager. As I mentioned, I have all in one. I'm using the
all included, so I don't need to think about having the telemetry collector and the web
to check the traces. I have only one single image. Then I have API 6. Then I have the catalog,
which I showed you. Of course I have a couple of variables to configure everything. I wanted
to focus on tracing, so no metrics, no logs. I'm sending everything to Yeager, and then
I do the same for pricing, and I do the same for the stock. And normally at this point,
I already started, because in general I have issues with the Java stuff. So here I'm doing
a simple curl to the product. I've got the data, which is not that important. And I can
check on the web app how it works. So here I will go on the Yeager UI. I see all my services.
I can find the traces. Here you can find the latest one. And here is the thing. If I click
on it, it might be a bit small, right? I cannot do much better. You can already see everything
that I've shown you. So I start with the product from the API gateway. It forwards it to the
product to the catalog. Then I have the internal calls, and I will show you how it works. Then
I have the get request made from inside the application. And then I have the stocks that
responds here. Same here. And here we see something that was not mentioned on the component
diagram. From the catalog to the stock, I go directly. But from the catalog to the pricing,
I go back to the API gateway, which is also a way to do that for whatever reason. And
so this is something that was not mentioned on the PDF, but you cannot cheat with open
telemetry. It tells you exactly what happens and the flow. And the rest is the same. So
regarding the code itself, I told you that I don't want anything to trouble the developer.
So here I have nothing regarding open telemetry. If I write hotel, you see nothing. If I write
telemetry, you see nothing. I have no dependency. The only thing that I have is my Docker file,
and in my Docker file, I get the latest open telemetry agents. So you can have your developers
completely oblivious, and you just provide them with this snippet, and then when you
run the Java application, you just tell them, A, run with the Java agent. Low-hanging fruits,
zero trouble. Any Java developer here? Not that many. Python? OK, so it will be Python.
Just the same here. Here it's a bit different. I add dependencies, but actually I do nothing
on it. So here I have no dependency on anything. Here I'm using a SQL database because, again,
I'm lazy. I don't care that much. But here I have no dependency, no API call to open telemetry.
The only thing that I have is in the Docker file again. I have this. Again, I'm using
a runtime. It's super easy. I let the runtime, like, intercept the calls and everything to
open telemetry. And the last fun stuff is Rust. Any Rust developer? Please don't look
at my code too much. I'm not a Rust developer, so I hope it won't be too horrible. And Rust
is actually, well, not that standardized. So here I don't have any runtime, so I need
to make the calls by myself. The hardest part is to find which library to use, depending
on which framework to use. So in this case, I found one, and perhaps there are better options.
But I found this open telemetry OLTP stuff. And here this is because I'm using XM. I'm
using this library. And so far, it works for me. I don't need to do a lot of stuff. I just,
like, copy pasted this stuff. Copy past developer. And afterwards, in my main function, I just
need to say this and this. So I added two layers. So if you don't have any platform,
any runtime, you actually need your developers to care about open telemetry. Otherwise, it's
fine. Now, we already have pretty good, like, results, but we want to do better. So we can
also ask the developers, once they are more comfortable, to do manual instrumentation
even in the case when there is a platform. Now, I will docker compose down. And it takes
a bit of time. I will prepare this. And on the catalog sides, now I can have some additional
codes. So this is a Spring Boot application. What I can do is add annotations. Like, I
noticed there were a couple of Java developers. So it's the same with Kotlin. It's still on
the JVM. So basically, I'm adding annotations. And because Spring Boot can read the annotation
at runtime, it can add those calls. So I don't have to call the API explicitly. I just add
some annotation, and it should be done. On the Python side, I import this trace stuff,
and then I can, with the tracer, add some, again, explicit traces, so internal traces.
And from the first point of view, because I already, like, did it explicitly work.
And now you can see that I am in deep trouble, because it happened a lot of time. The Java
application doesn't start for a demo, and that's really, really fun. So I will try to
docker compose down the catalog. And docker compose, hey, what happens? Dash? Are you
sure? No, no, no, no, no, no, no. Not with the new versions. Yes. That's fine. We are
only here to learn. What? Stop. Thanks. The stress, the stress. Yeah. Honestly, if there
is any, like, person here able to tell me why this Java application sometimes has issues
starting because I've added one gig at the beginning, and it's stuck always here. So
I can tell you what you should see normally. If I'm lucky, I made a screenshot. Yes, here,
but it's the beginning, it's the rust one. So here, this is what you can have in Python.
This is what I added explicitly. I have five minutes. Well, if the demo doesn't work, it
will be much better. Then I won't have any problems with the timing. Here, you can see
that this is the trace that, yeah, this is a trace that I added manually in Python. And
here we can see that I filled the ID with the value. And on the Java sides, again, nope,
nope. I think it will be here. This is not the manual stuff that I added. Yes, it is,
you have the fetch here. You have the fetch here. So this is the span that I added manually.
I'm afraid that at this point, the demo just refused working. Yes, it's still stuck. I
will stop there. I won't humiliate myself further when it's done. It's done. Perhaps,
if you are interested, you can follow me on Twitter. You can follow me on MasterDone.
I don't know what's the ratio. More importantly, if you are interested about the GitHub repo,
to do that by yourself, perhaps with better configuration of the code compose with the
right memory, it would work. And though the talk was not about Apache API 6, well, have
a look at Apache API 6. It's an API get away, the Apache way. Great. Are there some questions
now? I never got so many uploads with a filling demo.
Please remain seated so we can have a Q&A. Who had a question?
Thank you. Very good talk. I have two questions. So one is about this.
Let's start with the first one. Right. Yes, yes, yes. How much overhead does
this bring in Python and Java or Rust? How heavy is this instrumentation?
That's a very good question. And the overheads of each request depends on your own infrastructure.
But I always have an answer to that. Is it better to go fast and you don't know where
you are going to go a bit slower and to know where you are going?
I think that whatever the cost, it's always easy to add additional resources and it doesn't
cost you that much. Whereas a debug incident across a distributed
system can cost you days or even like weeks in injuring costs. And you are very, very
expensive, right? Okay. Thank you. And the second one is have
you encountered any funny issues with multi-threading or multi-processing? Something like when your
server just now... Can you come closer to your...
Your server just now was not starting. So some software, when you have multi-threading
or multi-processing and have you encountered any issues when the instrumentation costs
you trouble? This is not production stuff. This is just
better than the hello world. So I cannot tell you about prediction issues.
You should find people who have these issues. As I mentioned, it's a developers-oriented
talk. So it's more about pushing the developers to help
up to their job. For production issues, I must admit I have no clue.
Hi. In the case of runtime, does it always work
with also badly written application? I mean, how bad can an application be before it stops
working? I'm not sure. I understood the question.
So how often do you need to do it before it stops working?
No, no. I mean, let's say I use deprecated libraries, bad clients, something that doesn't
work as it's supposed to be for the instrumentation perspective. I mean, I do request to the network
using UDP clients, something I've written myself, some custom stuff that...
I'm imagining that the instrumentation sits between some layer of the network, which is
going to the Internet, for example. And so how bad can I be before it stops recognizing
a request from junk? You cannot be banned.
OK. Well, it's a moral issue first. But then on the platform side, the Austo instrumentation,
they work with specific frameworks and tools. It's those frameworks and tools that know
how to check what happens and to send the data to open telemetry.
So if you don't play in this game, nothing will be sent.
On the manual instrumentation side, it's an explicit call. So it depends what you want
to send. Yeah. I was thinking of auto instrumentation.
So let's say I do the NS resolution by myself and then I just throw a request to an IP.
Let me show the Python stuff here. This is what I showed you in the screenshot.
This is what I write. And this is the attributes that I want to have.
So basically, if here you have something that is completely unrelated, it's up to you.
That's why it's easier to start with auto instrumentation.
And then once you get a general overview of what you have and your app starts saying,
hey, perhaps we want to have more details here, then you can come with manual instrumentation.
But start with the less expensive stuff. I didn't really answer the question.
I understand it. But that's the best I can do regarding it.
Sorry. Okay. Thanks for the talk.
For the agent you use in the Docker file, how you can configure it, for example, for
the tracing for Jagger or other stuff. Regarding the Docker file, sorry?
Yeah. How you can configure the agent to send the tracing for Jagger or other stuff.
The Docker file doesn't mention where you send it.
The Docker file just says, hey, I will use open telemetry.
And it's during configuration, it's like in the Docker Compulse file where I'm using,
like, agreed upon environment variables where I'm saying you should set it here or here
or you should use logging or tracing or metrics or whatever.
So that's very important to, like, separate those concerns.
On one side in the Docker file in the image, you say, hey, I'm ready for open telemetry.
And when you actually deploy it to say, okay, open telemetry will go there for the metrics
and there for the tracing and for logging, I will disable it or whatever.
Thank you for... Oh, sorry.
Sorry. Go ahead.
Sorry. And then you have a Docker image that can be, like, reusable.
Thank you for being good first-time citizens to remain seated.
Next question.
Thank you for your presentation.
So my question is does open telemetry support error handling like sentry?
If not, is there any plans to do that?
It's really useful to catch crashes and capture the context of the crash.
So that's it. Thank you.
If it happens, when you mean crashes of open telemetry itself or of the components that
are, like, under watch?
Yeah, of the application that's monitored, yeah.
Well, Fabian showed you how you could log and, like, bind your traces and your logs.
So you could have both here.
My focus was just on tracing, but you can reuse the same Docker, the same GitHub repo
and just, like, here, put the logs somewhere in, I don't know, Elasticsearch or whatever.
No, because it's not a sponsored room.
And then you can check and you introduce some errors and then you can check how the two are
bound and you can, like, drill down to where it failed.
Okay, thank you.