Welcome our next speakers and give them a round of applause.
Can you hear me?
I guess you can.
So yeah, hello, everyone, and welcome to the session about how we can use open telemetry
on Kubernetes to collect traces, metrics, and logs.
So my name is Pavel, I'm software engineer at Red Hat, I contribute and I'm a contributor
and maintainer of Open Telemetry Operator and Yeager project.
Yeah, my name is Bine, and I'm also working on the Open Telemetry Operator and spent most
of the time on Open Telemetry.
And so as I mentioned on today's agenda, there is the Open Telemetry Operator.
We will show how you can use it to deploy the collector, how you can as well use it
to instrument your workloads on Kubernetes.
And after this brief introduction, we will walk you three use cases, how you can use
it to collect traces, metrics, and logs.
However, I will start with the history of open source observability.
I'm doing this because I believe that if we understand the history, maybe we will better
understand where we as industry are going.
So on this slide, you essentially see a timeline of the, with the open source projects.
And it's divided into the, the upper and bottom parts.
In the bottom, you see the open source projects or platforms that you can deploy and they
provide you with a storage and visualization capabilities for, for the observability data.
Most of them work with distributed traces, however, some of them, like the Apache skywalking
hyper-tracing signals, those are more like end-to-end platforms that can show traces,
metrics, and logs.
I would like to focus on the upper part that shows you the open source data collection
kind of frameworks.
And what we see there with, especially with open sensors and open telemetry is that it's
becoming more important that these frameworks kind of work with all the signals.
For me, the, the data collection, especially for tracing started with Zipkin project.
It gave us a stable data model that we, as developers, could use to export traces into
Zipkin, but as well to many other kind of platforms that adopted Zipkin project.
As a developer, when we wanted to use Zipkin clients, because the ecosystem hosted client
libraries as well, it was a bit problematic in polyglot environments because those clients
were using kind of inconsistent APIs, there was no standardization.
And so this problem then was partially solved with open tracing.
The scope of the project was a bit wider, there was a specification, there was a document
that defines which data should be collected and as well how the API in those languages
should look like.
This enabled us to build reusable instrumentation libraries.
And then, even later, the open sensors project started with slightly different approach.
There was no specification, there was no API, but there was SDK that everybody could use
and a collector.
So with open tracing, the approach was that developers would use the API and then at the
build time provide the SDK from a vendor.
With open sensors, everybody would use the SDK and then in the collector decide where
the data should be sent.
Those two projects were kind of competing and then finally they merged into open telemetry
in 2019.
So the hotel, it adopted all the pieces from open tracing and open sensors, but kind of
the biggest innovation in hotel is the, at least in my view, is the auto instrumentation
libraries or the agents.
Those agents are production ready, most of them, because they were donated by one of
the observability vendors, so they are, you know, production tested.
So when we kind of summarize what happened is that we started with some instrumentation
libraries, you know, with Zipkin project, then since we have some kind of standardization,
we could build reusable instrumentation libraries and kind of create more sophisticated instrumentations
for runtimes.
And now we are in an age that we have available in open source agents or auto instrumentation
libraries that we can just grab, put into our platforms, and we will get telemetry data
almost for free.
And I think, you know, so where are we going?
I think we are going into an era where we, as developers, we won't have to care about
how the telemetry is created for us.
We will be, the instrumentation will become maybe the feature of the platform where we
deploy the application.
So this is one way to look at it.
The other way might be that the observability will shift left, and since we have this data,
we will start utilizing it for other use cases, probably like testing and security.
So with that, I would like to move to the open telemetry, and it's obviously open source
project hosted in the cognitive computing foundation, and its main goal is to provide
the vendor or neutral telemetry data collection.
It's the second most active project in CNCF after Kubernetes, so it's quite large.
And there are several independent components that we can use.
There is a specification that defines what data should be collected and how the API should
look like, and obviously then there is the implementation of the API, the SDK and the
standard data model called OTLP or open telemetry protocol.
These four pieces are meant to be used primarily by instrumentation authors or the people that
work on the observability systems.
And last two components, the auto instrumentation or agent and collector are meant to be used
by end users to kind of roll out observability in their organization.
To facilitate open telemetry deployment on Kubernetes, there is a Helm chart and Kubernetes
operator.
What I would like to stress is that open telemetry is only about how we collect and create telemetry
data.
It's not a platform that you can deploy, it doesn't provide any storage or query APIs.
So now let's go to the main part, the Kubernetes operator.
The operator itself, it's a Golang application, it uses QBuilder and operator SDK, and it
has three primary use cases.
It can deploy the open telemetry collector as a deployment, demon set, stateful set.
It can as well inject the collector as a side card to your workload.
The second use case is that it can instrument your workloads running on Kubernetes by using
those instrumentation libraries or agents from open telemetry.
And last but not least, it integrates with Prometheus ecosystem.
It can read the service and pod monitors, get the scraped targets, and split them across
the collector instances that you have deployed.
To enable this functionality, the operator provides two CRDs, one for the collector that
is used to deploy the collector and integrate the Prometheus.
And the second one is the instrumentation CRD, where you define how the applications
should be instrumented.
The operator itself then can be deployed through manifest files, home chart, or OLM.
So what we see here is the Kubernetes cluster.
There are three workloads, pod one, pod two, and pod three.
The first workload is instrumented with the hotel SDK directly, so when we were building
this application, we pulled in the hotel dependency and we compiled it against it and used those
APIs directly in our business code and in the middlewares that we are using.
The second pod is using the auto instrumentation libraries that were injected by the operator
through the Venetian webhook.
And the third pod is using Zipkin instrumentation and Prometheus instrumentation libraries,
and it has the collector sidecar as well injected by the operator.
So essentially the operator there, it reconciles three open telemetry CRs, two for the collector,
and one instrumentation.
And then all these workloads, they send data to the collector deployed, probably as a demon
set, and then this collector then does some data normalization and sends finally data
into platform of your choice, which can be Prometheus for metrics, Yeager for traces.
With that, I would like to move to the second part, explaining the CRDs in more detail.
Yep.
The microphone should work.
Yeah, so with the CRDs for today, we wanted to show both of them, and we start with the
collector one.
The collector CRD is a bit loaded, so therefore we picked a few things here, which I would
say are the most used or important.
So as Pawe mentioned, there are different deployment modes, different use cases for
the open telemetry collector, and in the specification, we can go to the mode and just specify it there.
There's a handy thing, which is the sidecar, we will see it afterwards.
And if we want to use it, we only go to the part definition of our deployment and inject
the annotation we see on the top right.
And if we go with the deployment mode or something like this, and we want to expose it for collecting
metrics, locks, and traces from a different system, for example, we can use the Ingress
type, we can set there a lot of more, we configure there a lot of more like also the annotations,
your Ingress class.
But yeah, mainly the operator takes care of everything, creating services, also is able
to balance your load there.
And yeah, the last thing here is then the image section, which is also important.
With the open telemetry operator, it usually ships the core distribution of open telemetry
by default.
So in open telemetry, the collector is split into two repositories when you go up and look
at GitHub.
So in core, you will find OTP, a logging exporter, so some basic stuff.
And in Contrip, you find basically everything.
So if you want to send your traces to some proprietary vendor or to Jäger, you probably
need to look there.
Okay, the next thing is then the configuration.
The configuration for the open telemetry collector is here provided like it's usually done for
the collector itself.
So it's passed directly forward.
It's split it into three parts here.
We see the receiving part there.
We specify our OTP receiver.
Here it's accepts GRPC on a specific board.
It could also be there that we specify a prometers receiver, which is then scraping something.
Then the optional part is basically the processing part.
We might want to save some resources and we batch them our telemetry data.
And yeah, there are other useful things.
And on the exporter section, here we use the logging exporter, which is part of the
core distribution, but you can configure whatever you like.
You can also have multiple exporters for one resource.
There is one thing.
On the right side, we see the extensions.
It didn't fit on the slide, so it's there in this box.
This is then used if you have, for example, an exporter, which needs some additional
headers.
Yeah, you want to set a barrier token or something else.
You can do it there.
And then finally, we go to the service section where we have different pipelines for each
signal.
And then we can then configure a processor and receiver and exporter in the way we wanted.
So then there is another CD, which is used for the auto instrumentation.
And it looks slightly different.
So here we have also the, in the specification, we have the exporter.
And the exporter only exports OTP, so which means if we want to export it to some, yeah,
back end of our choice, we usually instrument our application directly then forward this
traces to a collector instance, which is running next to it.
And yeah, we can use the power of these processors.
Yeah, then we can configure some other useful things like how the context is propagated
and the sample rate.
And to use it, it's also quite easy.
So we have our deployment.
In this case, we can, it can choose from this list of supported languages.
We might use Java and we only set this annotation on the port level and it will take care of
adding the SDK and also setting and configuring the environment variables.
If we use something like Rust, we can also use the inject SDK annotation to configure
then, yeah, just the destination because then SDK should be there.
And if we have a setup where there is, let's say, some proxy in front, like Envoy, we can
then just skip the, yeah, adding the auto instrumentation there by only configuring the container
names we want to instrument.
And we will see this in a minute, a bit more in detail.
So this is then basically what we would need to do.
So we create this instrumentation, we add this annotation on the left.
We see the pot, there is our application.
And in this gray box, you see what automatically is added.
And this is then forwarded in this example to a collector.
And yeah, how does this work?
So the operator in that mission web hook, he will add this in its container.
On the top left, we see how the container looks before.
So there are no environment variables.
It's just a plain application.
And in the command section, there is then the copy, which copies the Java agent to our
original container.
And on the right side, we see the final result.
We see the Java tool options where the container is loaded, and then we see all this environment
variables to configure our SDK.
And finally, what we have seen also in the presentation from Nicholas previously, we
have here the Yeager output.
So we can see the resource attributes and all the beautiful stuff that comes with it.
So next, we have, we can have a look on metrics.
So there is the open telemetry SDK.
So if you want to go with open telemetry metrics, but I assume a lot of people have already
some perimeter stuff in place.
And the open telemetry operator also helps us with this.
So we can, I might we look first on the receiver part on the bottom.
We see there, we configure the perimeter's receiver, which has a scrape configuration,
and there we can, for example, add some static targets.
So we assume we add there three different scrape endpoints, then afterwards, if the target
allocator is enabled, this will then take these scrape targets and divide, well, spread
these targets across our replicas, which are then responsible for getting the metrics.
And yeah, that's basically how it works.
There's also an option to enable perimeter CRs, so we can then forward to this one.
And the target allocator, which is an extra instance created by the Yeager, by the open
telemetry operator, will then, yeah, get the targets from there.
So we see this here in this graphic, quite good.
On the left side, we see part one, which is using open telemetry, and it's pushing the
information telemetry data it gets directly to a collector.
And in this gray box, we see there, we have two instances running Prometoys, running instruments
with Prometoys, and the collector one and collector two are pulling the information
from there.
So this is all managed then by the operator.
We have seen the replicas, this is basically collector one and collector two, and since
we enable the target allocator, we get the targets from there, so which is then coming
from the port monitor.
And finally, we send the information somewhere.
So the last thing here, the last signal are then locks.
So for locks, there are different options.
So the first one would be to use the open telemetry SDK, what we might don't want right
now because we need to do some work, but if we directly want to go ahead, there is the
philoc receiver, we can configure it to get the information from this, and yeah, it's
available in the conflict repository, and we have different parsers there which help
us to move the locks into the OTP format.
We will see in a minute how this looks like.
And there are other options if you want to integrate with FluentBit, so there is a forwarder,
so you can use it as a kind of a gateway then.
And yeah, the only thing we need to do then is we can configure it as a demon set.
We need to pass our information there, and the philoc receiver, for example, can then
get all the locks.
And how does this look like at the end?
So this is when we exported the locks to the logging output, so standard out.
We see that we have the resource attributes which are added automatically, and yeah, we
see then the lock information, and on the bottom the trace ID and span ID which are
not given if we read it just from disk, but that's it.
Yeah, then we are almost at the end.
Yeah, thanks a lot for the interesting talk.
Does anyone have questions?
Any questions?
Raise your hand.
Question?
Yeah, there we go.
For the logging part, would you suggest to replace any kind of cluster logging like with
Fluent Bit, or that's like sending it off to Loki or something with an open telemetry
log scraping, or is that complementary?
I'm not sure if I fully got it.
So you want to, yeah, in this case it's just another way, but the useful thing is if you
have the open telemetry SDK, it will automatically add then the trace ID to it, and then you
can correlate your signals.
Sorry.
So I'm super newbie to this, so I failed to understand how the, if the open telemetry
is trying to replace, for example, the log parsers like the telegraph, for example, which
is able to generate prometheus metrics by log scraping, or also how Zipkin, which is the
tracing thing, fits in the metric collection of all this picture.
So I'm not trying to understand how you cobble together all these sources and how open telemetry
either replaces or either makes it easier to use all these technologies together.
Thank you.
So maybe on this slide you see that the third port is using the Zipkin and prometheus, and
the collector can receive data in Zipkin format, it can scrape prometheus metrics, then transform
this data into OTLT or the Zipkin as well, and then send it to the other collector.
So the collector essentially can receive data in multiple formats, transform them to the
format of your choice, and then use that format to send it to other systems.
Hello and thanks for the talk.
I'm just wondering, what's your strategy of filtering health check requests, for example,
or the life probes request that you get in the pod?
Health checks, like to avoid generating traces for health checks?
Sorry?
To avoid generating traces for health check endpoints?
Yeah.
That's a very good question.
So you could maybe configure the collector to drop the data, but I think the best way
would be to tell the instrumentation to skip instrumenting those endpoints.
To be honest, I'm not sure if this is implemented in OTL agents, but I saw a lot of discussions
around this problem, so probably there will be some solution.
We have time for one last question, if there is any.
No?
Okay.
Oh, no.
And thanks a lot.