So, I hope it will be fun enough for you to wake up at the end of the day and very excited
to be here at FOSDEM and specifically at the CI CD Dev Room.
And today I'd like to share with you about how we gained observability into our CI CD
pipeline and how you can do too.
So let's start with a day in the life of a DoD developer on duty, at least in my company.
And it goes like that.
So the first thing the DoD does in the morning, at least it used to be before we did this
exercise, is going into the Jenkins.
We worked with Jenkins, but the takeaways, by the way, will be very applicable to any
other system you work with, so nothing too specific here.
Going into Jenkins at the beginning of the morning, we're looking at the status there,
the pipelines for the last few hours over the night, and of course checking if anything
is red, and most importantly, if there's a red master.
And if you can obviously finish your coffee or jump straight into the investigation.
And to be honest, sometimes people actually forgot to go into the Jenkins and check this,
so that's another topic we'll maybe touch upon.
So you go in, and then you need to go, let's say you see a failure, you see something red,
you need to start going one by one on the different runs, and start figuring out, understanding
what failed, where it failed, why it failed, and so on.
And it's important that you actually, you needed to go one by one on the different runs,
and we have several runs, we have the backend, we have the app, we have smoke tests, several
of these, and start getting the picture, getting the pattern across, and understanding cross
runs, across branches, what's going on.
And on top of all of that, it was very difficult to compare with historical behavior, with
the past behavior, to understand what's, and anomaly, what's the steady state for these
days, and so on.
So, and just to give you a few examples of questions that we found it difficult or time-consuming
to answer, things such as, did all runs fail on the same step, did all runs fail for the
same reason, is that on a specific branch, is that on a specific machine, if something's
taking longer, is that normal, is that anomalous, what's the benchmark?
And so these sorts of questions, it took us too long to answer, and we realized we need
to improve.
A word about myself, my name is Dotan Horvitz, I'm the Principal Developer Advocate at a company
called Logs.io, Logs.io provides a cloud-native observability platform that's built on popular
open-source tools such as you probably know, Prometheus, OpenSearch, OpenTelemetry, Yeager,
and others.
I come from a background as a developer, a solutions architect, even a product manager,
and most importantly, I'm an advocate of open-source and communities.
I run a podcast called Open Observability Talks about open-source DevOps observability,
so if you're interested in these topics and you like podcasts, do check it out.
I also run, organize, co-organize several communities, the local chapter of the CNCF,
the cloud-native computing foundation in Tel Aviv, Kubernetes Community Days, DevOps
Days, et cetera, and you can find me everywhere at Horvitz.
So if you have something interesting, you tweet, feel free to tag me.
So before I get into how we improved our CI CD pipeline or capabilities, let's first
understand what we want to improve on.
And actually, I see very often that people jump into solving before really understanding
the metric, the KPI that they want to improve, and very basically, therefore, primary metrics
for, let's say, DevOps performance, and you can see there on the screen, there's the
deployment frequency, lead time for changes, change failure rate, and MPTR, mean time to
recovery.
I don't have time to go over all of these, but very important, so if you're new to this
and if you want to read a bit more about that, I left a QR code and a short link for you
at the bottom for a 101 on the Dora metrics, do check it out, I think it's priceless.
And in our case, we needed to improve on the lead time for changes or sometimes called
cycle time, which is the amount of time it takes a commit to get into production, which
in our case was the time was too long, too high, and was holding us back.
So we are experts at observability in our engineering team.
That's what we do for a living, so it was very clear to us that what we're missing in
our case is observability into our CICD pipeline.
And to be fair with Jenkins, and there are lots of things to complain about Jenkins,
but there is some capabilities within Jenkins.
You can go into a specific pipeline run, you can see the different steps, you can see how
much time an individual step took.
Using some plugins, you can also visualize the graph and we even wired Jenkins to get
alerts on Slack, but that wasn't good enough for us.
And the reason that we wanted to find a way to monitor aggregated and filtered information
according to our own timescale, according to our own filters, obviously to see things
across branches, across runs, to compare with historical data, with our own filtering,
so that's where we aimed at.
And we launched this internal project with these requirements, four requirements.
One, first and foremost, as we need the dashboard, we need the dashboard with aggregated views
to be able to see the aggregated data across pipelines, across runs, across branches as
we talked about.
Finally we wanted to have access to historical data to be able to compare, to understand
trends, to identify patterns, anomalies, and so on.
Thirdly, we wanted reports and alerts to be able to automate as much as possible.
And lastly, we wanted some ability to view flaky tests, test performance, and to be able
to understand their impact on the pipeline.
So that was the project requirements and how we did that.
Essentially it takes four steps, collect, store, visualize, and report.
And I'll show you exactly how it's done and what each step entails.
In terms of the tech stack, we were very versed with the Elk stack, Elasticsearch, Kabbana.
Then we also switched over to OpenSearch and OpenSearch dashboards after Elastic re-licensed
and it was no longer open source.
So that was our natural point to start our observability journey.
And I'll show you how we did these four steps with this tech stack.
So the first step is collect.
And for that we instrumented the pipeline to collect all the relevant information and
put it in environment variables.
Which information, you can see some examples here on the screen, the branch, the Kamecha,
the machine IP, the run type, whether it's scheduled, triggered by merge to master or
something else, fail step, step duration, build number, anything essentially that you
find useful for investigation later.
My recommendation, collect it and persist it.
So that's the collect phase and after collect comes store.
And for that we created a new summary step at the end of the pipeline one where we ran
a command to collect all of that information that we did in the first step and created
a JSON and persisted it to Elasticsearch, as I mentioned then move to OpenSearch.
And it's important to say again for the fairness of Jenkins and for the Jenkins experts here,
Jenkins does have some built in persistency capabilities.
And we tried them out, but it wasn't good enough for us.
And the reason is that by default Jenkins essentially keeps all the bills and stores
them on the Jenkins machine, which burdens these machines of course.
And then you start needing to limit the number of bills and the duration, how many days and
so on and so forth.
So that wasn't good enough for us.
We needed a more powerful access to historical data.
We wanted to persist historical data in our own control, the duration, the retention and
most importantly off of the Jenkins servers so as not to risk and overload the critical
path.
So that's about store and after store.
Once we have all the data in Elasticsearch or OpenSearch, now it's very easy to build
command dashboards or OpenSearch dashboards and visualizations on top of that.
And then comes the question, sorry, then comes the question, okay, so which visualizations
should I build?
And for that, and that's a tip, take it with you, go back to the pains, go back to the
questions that you found it hard to answer and this would be the starting point.
So if you remember before we mentioned things such as did all runs fail on the same step,
did all runs fail for the same reason, how many fail, is that a specific branch, is that
a specific machine and so on, these are the questions that we guide you then to choose
the right visualizations for your dashboard.
And I'll give you some examples here.
So let's start with the top line view.
You want to understand the health of your house table, your pipeline is.
So visualize the success and failure rates, you can do that overall in general or at a
specific time window on a graph, very easy to see the first glance, what's the health
status of your pipeline.
You want to find problematic steps, then visualize failures segmented by pipeline steps, again
very easy to see the spiking step there.
You want to detect problematic build machines, visualize failures segmented by machine and
that by the way saved us a lot of wasted time going and checking bugs in the release code.
When we saw such a thing, we just go, you kill the machine, you let the auto scaler spin
up a new instance and you start clean and in many cases it solves the problem.
So lots of time saved, in general this aspect of code based or environmental based issues
is definitely a challenge I'm assuming, not just for me, so I'll get back to that soon.
Another example duration per step, again very easy to see where and at the time is spent.
So that's the visualize part and after visualize comes the reporting and alerting phase.
And if you remember before the DOD, the developer on duty, needed to go manually and check Jenkins
and then the health check, now the DOD gets start of day report directly to Slack and
actually as you can see the report already contains the link to the dashboard and even
a snapshot of the dashboard embedded within the Slack so that at the first glance even
without going into the dashboard you can see if you can finish your coffee or if there's
something alerting that you need to click that link and go start investigating.
And of course it doesn't have to be a schedule report, it could be also you can define triggered
alerts on any of that, the fields, the data that we collected in the first phase and the
collect phase so and you can do any complex queries or conditions that you want, you want
to do something like if the sum of failures goes above x or the average duration goes
above y trigger an alert.
So essentially anything that you can formalize as a Lucene query, you can automate as an
alert and that's some alerting layer that we built on top of elastic search and open
search for that.
One last note, I'm giving the examples from Slack because that's what we use in our environment
but you're not limited obviously to Slack, you have support for many notification endpoints
depending on your systems, pager duty, victorops, ops genie, MS themes, whatever.
We personally work with Slack so that the examples are with Slack.
So that's how we build observability into the Jenkins pipelines but as we all know especially
here in the CI CD dev room, Jenkins, CI CD is much more than just Jenkins.
So what else?
So we wanted to analyze if you remember the original requirements to analyze flaky tests
and test performance and following the same process, collecting all the relevant information
from test run and storing it in elastic search and open search and then creating a cabana
dashboard or open search dashboards and as you can see very all the relevant usual suspects
that you'd expect, the test duration, fail test, flaky test, failure count and rate moving
averages, fail test by branch over time, all of the things that you would need in order
to analyze and understand the impact of your test and the flaky tests in your system.
And similarly after visualize you can also report, we created reports to Slack, we have
a dedicated Slack channel for that, following the same pattern.
One important point is about the openness.
So once you have the data in open search or in elastic search, it's very easy for different
teams to create different visualizations on top of that same data.
So I took another extreme, a different team that didn't like the graphs and preferred
the table views and the counters to visualize, again, very similarly, test stats and so on.
And that's the beauty of it.
So just to summarize, we instrumented Jenkins pipeline to collect relevant data and put
it in environment variables, then at the end of the pipeline we created a JSON with all
this data and persisted it to elastic search open search, then we created Kibana dashboards
on top of that data and lastly we created reports and alerts on that data.
So four steps, collect, store, visualize and report.
So that was our first step in the journey but we didn't stop there.
The next step was we asked ourselves, what can we do in order to investigate performance
of specific pipeline runs?
So you have a run that takes a lot of time, you want to optimize, but where is the problem?
And that's actually what distributed tracing is ideal for.
How many people know what distributed tracing is with a show of hands?
Okay, I see that most of us, there are a few that know, so maybe I'll say a word about
that soon.
Very importantly, Jenkins has the capability to emit trace data spans, just like it does
for logs, so it's already built in.
So we decided to visualize jobs and pipeline executions as distributed tracing.
That was the next step.
And for those who don't know, distributed tracing essentially helps pinpoint where issues occur
and where latency is in production environments, in distributed systems, it's not specific
for CICD.
If you think about a microservice architecture and a request coming in and flowing through
a chain of interacting microservices, then when something goes wrong, you get an error
on that request, you want to know where the error is within this chain, or if there's a
latency, you want to know where the latency is.
That's distributed tracing in a nutshell.
And the way it works is that each step in this call chain, or in our case, each step
in the pipeline, creates and emits a span.
You can think about a span as a structured log that also contains the trace ID, the start
time, the duration, and some other context.
And then there is a back end that collects all these spans, reconstruct the trace, and
then visualizes it typically in this timeline view or gun chart that you can see on the
right-hand side.
So now that we understand the distributed tracing, let's see how we add distributed
tracing type of performance, pipeline performance into a CICD pipeline.
And same process.
For the collect step, collect.
And for the collect step, we decided to use an open telemetry collector who doesn't know
about open telemetry, who doesn't know the project, just so I have a background, okay.
I have a few, so I'll say a word about that.
And anyway, I added a link, you see a QR code and a link at the lower corner there for a
beginner's guide to open telemetry that I wrote.
I gave a talk about open telemetry at KubeCon Europe, so you'll find it useful.
But very briefly, it's an observability platform for collecting logs, metrics, and traces.
So it's not specific only to traces in an open unified standard manner.
It's an open source project under the CNCF, the Cloud Native Computing Foundation.
And at the time, it's a fairly young project by the time, the tracing piece of open telemetry
was already GA generally available, so we decided to go with that.
Today, by the way, also metrics is soon to be GA, it's already in release candidate,
and logging is still not there.
So what do you need to do if you choose open telemetry?
You need to set up the open telemetry collector, it's sort of an agent for it to send.
You need to install the Jenkins open telemetry plug-in, very easy to do that on the UI.
And then you need to configure the Jenkins open telemetry plug-in to send to the open
telemetry collector and point over OTLP over GRPC protocol.
That's the collect phase, and after collect comes store.
For the back end, we used Jega.
Jega is also a very popular open source under the CNCF, specifically for distributed tracing.
And we use Jega to monitor our own production environment, so that was our natural choice
also for this.
We also have a Jager-based service, so we just use that.
But anything that I show here, actually you can use with any Jager distro, whichever one
you use, managed or self-serve.
And if you do run your own, by the way, I added the link on how to deploy Jager on Kubernetes
in production, so you have a link there, a short link that I added, a very useful guide.
So what do you need to do?
You need to configure open telemetry collector to export in open telemetry collector terms
to export to Jager in the right format, all the aggregated information.
And once you have that, then you can visualize, the visualized part is much easier in this
case, because you have a Jager UI with predefined dashboard, you don't need to start composing
visuals.
Essentially, what you can see here on the left-hand side, you can see this indented
tree structure, and then on the right, the gun chart.
Each line here is a span, and it's very easy to see the pipeline sequence.
The text is a bit small, but you can see, for each step of the pipeline, you can see
the duration, how much it took, you see which ones ran in parallel, and which ones ran sequentially.
If you have a very long latency on the overall, you can see where most of the time is being
spent, where the critical path, where you best optimize, and so on.
And by the way, Jager also offers other views, like recently added the flame graph, and you
have trace statistics, and graph view, and so on.
But this is what people are used to, so I'm showing the timeline view.
So that's on Jager, and of course, as we said before, CICD is more than just Jenkins, so
what we can do beyond just Jenkins, and what you can do is actually to instrument additional
pieces like Maven, Ansible, and other elements to get final granularity into your traces
and steps.
For example, here, the things that you see in yellow is Maven build steps.
So what before used to be one black box span in the trace.
Suddenly, now you can click, open, and see the different build steps, each one with its
own duration, each one with its own context, and so on.
So that's in a nutshell how we added tracing to our CICD pipeline.
The next step is, as I mentioned before, many of the pipelines actually failed not because
of the released code, but because of the CICD environment.
So we decided to monitor metrics from the Jenkins servers and the environment.
It goes to the system, the containers, the JVM, essentially anything that could break
irrespective of the released code, and following the same flow.
So the first step, collect, we use the telegraph, we use that in production, so we use that
here as well, that's an open source by inflex data, and essentially you need two steps.
You need to first enable, configure, sorry, Jenkins to expose metrics in Prometheus format.
We work a lot with Prometheus for metrics, so that was our natural choice, and that's
a simple configuration in the Jenkins web UI, and then you need to install telegraph
if you don't already have that, and then make sure that it configured to scrape the
metrics off of the Jenkins server using the Prometheus input plugin.
So that's the first step.
The second step is on the store side.
As I mentioned, we use Prometheus for metrics, so we use that as well here.
We even have our own managed Prometheus, so we use that, but anything that I show here
is identical whether you use Prometheus or any Prometheus compatible backend.
And essentially you need to configure telegraph to send the metrics to Prometheus, and you
have two ways to do that.
You can do that in pull mode or in push mode.
So pull mode is the default for Prometheus, essentially when you configure a telegraph
to expose a slash metrics endpoint, and then it can be exposed for Prometheus to scrape
it from.
If you want to do that, you use the Prometheus client output plugin, or if you want to do
it in push mode, then you use the HTTP output plugin.
Just an important note, make sure that you set the data format to Prometheus remote write.
So that's the store phase, and then once you have all the data in Prometheus, then it's
very easy to create Grafana dashboards on top of that.
And I gave some examples here.
You can filter, of course, by build type, by branch, machine ID, build number, and so
on.
And you can monitor in this example, this is a system monitoring, so CPU, memory, disk
usage, load, and so on.
You can monitor the Docker container, like the CPU, IO, inbound, outbound, disk usage,
obviously the running, stopped, paused containers by Jenkins machine, everything that you'd
expect, and JVM metrics, by being a Java implementation, thread count, heap memory, garbage collection,
duration, things like that.
You can even, of course, monitor the Jenkins nodes, queues, executors themselves.
So again, you have an example dashboard here.
You can see the queue size, status breakdown, the Jenkins jobs, the count executed over
time, breakdown by job status, and so on.
So this is the types, just to, obviously, lots of other visualizations that you can
create, and you can also create alerts.
I won't show that in the lack of time, so just to summarize what we've seen.
Treat your CICD the same as you treat your production.
For your production, use whatever, elastic search, open search, Grafana to monitor to
create observability.
Do the same with your CICD pipeline, and preferably leverage the same stack, the same tool chain
for that, and don't reinvent the wheel.
That was our journey.
As I mentioned, we wanted dashboards and aggregated views to see several pipelines across different
run branches over time, and so on.
We wanted historical data and controlled persistence off of the Jenkins servers to determine the
duration, the retention of that data.
We wanted reports and alerts to automate as much as possible, and lastly, we wanted test
performance, flaky tests, and so on.
You saw how we achieved that.
Four steps.
If there's one thing to take out of that talk, take this one, collect, store, visualize,
and report an alert.
And what we gained, just to summarize, significant improvement in our lead time for changes,
in our cycle time, if you remember the Dora metrics at the beginning.
On the way, we also got an improved developer-on-duty experience, much less of a sufferer there.
It's based on open source.
Very important.
We're here on FOSDEM.
So based on open search, open telemetry, Yeager, Prometheus, Telegraph, you saw the stack.
If you want more information, you have here a QR code for a guide to CICD observability
that I wrote.
You're welcome to take a short or a bit short link and read more about this, but this was
very much in a nutshell.
Thank you very much for listening.
I'm Doton Horvitz, and enjoy the rest of the conference.
I don't know if we have time for questions.
No.
So I'm here if you have questions or if you have a sticker, and may the open source be
with you.
Thank you.
We have time for questions, if there are any.
We have time for questions, so if you want, we can just see for a few minutes.
Is that a question?
Yeah, the other question in the back.
Okay.
Which one do you want to be the first one to ask a question?
Thanks.
So have you considered persistence?
How long do you store your metrics and your traces?
Have you wondered about that?
And for how long at a time you store your metrics?
So we have.
That was part of the original challenge when we used the Jenkins persistence, because when
you persist it on the nodes themselves, and obviously you're very limited, there's the
plugin that you can configure per days or per number of bills and so on.
When you do it off of that critical path, you have much more room to maneuver, and then
it depends on the amount of data you collect.
We started small, so we collected for longer periods, but the more it came with the app,
the more the appetite grew, and people wanted more and more types of metrics and time series
data, so we needed to be a bit more conservative, but it's very much dependent on your practices
in terms of the data.
Yeah, the question was more about the process, so iterative, you explained it, so it starts
small.
Yeah, exactly.
And iterative is the best, because it really depends, you need to learn the patterns of
your data consumption, the telemetry, and then you can optimize the balance between having
the observability and not overloading and overpricing costs.
Right.
Thank you very, very interesting.
Thank you.
There was another question in the back, yeah?
Thank you.
So what was the most surprising insight that you've learned, good or bad, and how did you
react to it?
I think I was most surprised personally about the amount of failures that occur because
of the environment and what kinds of things, and how simple it is to just kill the machine,
kill the instance, let the auto-scaler spin it back up, and you save yourself a lot of
hassle and a lot of waking people up at night, so that was astonishing.
How many things are irrespective of the code and just environmental, and we took a lot
of learnings out there to make the environment more robust, to get people to clean after
them, to automate the cleanups and things like that, that's what me was insightful.
Thank you.
Any other questions?
Then I have one last one, sorry.
No, no worries.
My question is, who are usually the people looking at the dashboard, because I maintain
a lot of dashboard in the past, and sometimes I had a feeling that I was the only one looking
at those dashboards, so I'm just wondering if you identify a type of people who really
benefit from those dashboards.
So it's a very interesting question because we also learned and we changed the org structure
several times, so it moves between Dev and DevOps.
We now have a release engineering team, so they are the main stakeholders to look at that,
but this dashboard is the goal, as I said, the developer on duty, so everyone that is
now on call needs to see that, that's for sure, and the tier two, tier three, so let's
say the chain for that.
You also use that as a high level also by the team leads in the developer side of things,
so these are the main stakeholders, depending on if it's the critical part of the developer
on duty and the tiers, or if it's the overall thing the health state in general by the release
engineer.
Thank you.
Thank you very much, everyone.