Hello everyone. Thank you for having us here. It's our first time here, so please be kind.
It's like for both of us, it's the first time here, so we're a little bit nervous. And we're
here to talk about practical CI, CID observability with open telemetry. This is the abstract we
have submitted and of course we don't expect you to go through a whole thing because it's
just enormous. But like if we could abstract the abstract in a way, this talk is about enhancing
your pipeline's ability and performance by bringing observability to every stage of software
delivery. So we're going to answer two questions like how we can identify flakiness and bottlenecks
during our CI-CB process and envision a future of effortless visibility. And again, we're going to
talk about what's the role of open telemetry on that and also how, with the role it plays in
shaping CI-CB's future and explore all the challenges and opportunities ahead. So if I go to the
next slide, a little bit about us, I'm Dimitris, a software engineer at Grafana Labs at the
Platform Productivity Squad. And I'm Giordano, Gio for France. I'm a software engineer for the
X-Port Squad at Grafana Labs. So this is the agenda. As I said, we're going to start by
defining what CI really is and then talk about current issues we have with CI-CB systems. And then
we're going to do a small intro to open telemetry and how we use it really, why it is important to
own our data, and then practical use cases where we are and what's next. So all that being said,
we can proceed to the next slide with a question to you all. So what is CI? We're looking for a
definition of CI here. Anyone there to guess? It's fine if not. Okay, I'll proceed. So, sorry,
say again? Yeah, we're not looking for continuous integration. We're looking for the definition of
that. So CI, I guess, thank you very much, is continuous integration. But the definition of
that is, you know, as some experts have defined it in a couple of books, the thing is that continuous
integration definition can mean different things to different people. It's all a matter of a
perspective where you're looking that from what CI means to everyone. But one thing is for sure,
continuous is the only thing that's going to be there all the time because we're talking about a
never-ending feedback loop, you know, which keeps improving stuff and gives us like visibility
over our CI CD processes. So we can move to the next question, which is what is CI like for real
this time? And I would like again, if someone dares to guess. It's a black box. Yes, right. Yeah,
go on. Yes, running test could be. So again, CI is a list of things like it's a mechanism to,
for example, reduce repetitive manual processes or generate deployable software at any time and
any place. And of course, like, you know, scores, flaky tests and flaky builds and prevent people
from, you know, getting paid at 3am in the morning because we don't want to like spend human hours
during the night. They're really important for us. So the next slide is about, I mean, if you think
that this is complicated, I think not. I think, you know, this happens to this has happened to at
least everyone of us, at least like once, I think. So, you know, it starts from testing, building,
deploying and then waiting for changes or wait for 3am maintenance windows. You can see here
to errors and downtime and panicking. And also when we have resolved all those issues, we all go
to LinkedIn after that and say, you know, I'm a troubleshooting expert or like DevOps expert,
automation expert. So that's what we do because that's who we are. Yes, true. Exactly. So the next
slide is another question. Yeah. So another question, the next slide, what is CI like for real, real
this time? And what we're looking for is a single word. If anyone, there's to guess that single word.
Pipelines, automation. Alertings. Yes, that's what we're looking for. We're looking for
alerting. So CI and alerting serve a common person, a purpose or at least they try to serve a common
purpose. So they work closely together as essential components of continuous automated
monitoring. So you can see that both of them are practically identifying issues or, you know,
we have like continuous system monitoring or like, you know, all those things. And then CI,
we're looking at alerting as the left shift of CI basically, which means that if we have
affecting alerting within CI ensures that, you know, threshold bridges and like potential
problems are going to vanish. And we see I needs to focus on like robust build for new
releases. So together, CI and alerting serve a common goal like prompt problem identification,
fortifying system, reliability and sustainability. As you can see from the picture, they need to be
like holding hands forever because that's what they do. And we'll go to the next slide. So
a few things about continuous integration. We already talked about some of them, but like
continuous integration is the guard in early stages, like we can detect changes, maintain
bills, health and constantly monitor system signals. And like CI is used to catch issues
before they breathe really. So if we go to the next slide, we're having alerting next to that,
but we're not actually comparing those two. We just want to show you how closely, like how
tightly coupled they are together. So alerting is like, you know, is our alerting system,
like for later stages, they identify as problems as well, maintenance, allows and monitor system,
just like CI does. Just but we should see that alerting is a mechanism to be used just in case
CI has something has slipped through CI and we didn't catch it. So when we have alerting in place
and CI in place as well, we need to know that there are not two components running in parallel,
right? They are like CI lanes that lays the groundwork and then alerting response to threats. So they
are like, you know, unstoppable working together to serve the same purpose. So an important thing
to remember about alerting is that every time we need to create actionable alerts. So if something
slips through CI, we need to know, we need to get the alert, have a runbook, have some documentation,
automatically resolve some alerts if we don't think that are important enough to wake up someone in
the middle of the night and all that. So where we are now with CI-CD systems and like, what is this
whole talk amounted to? So observability so far, as you can see here, is about like, you know,
all the all-time classic concepts we know, like from printing here, we have all done this,
like in our early stages, I guess, from printing that, we're still doing that? Yeah, okay, we're
still doing that. And then from paging the platform team or from having three different
platforms, like we can have GitHub or GitHub or Atlassian or Bitbucket, whatever you use and then
find a broken test, go from there to your favorite CI vendor and then go from there to Grafana or
Data Dog or your favorite visualization tool to try and correlate those errors together. So
focusing, as you can see down there, if we focus, if like the sole focus of observability is at the
run part of things, this neglects valuable insights from earlier phases like code review or building
or testing and like incomplete observability across the CI pipeline leads to limited visibility
during, you know, earlier stages. We don't know what happened during the build phase, for example,
or the test phase or we have difficulty in root cause analysis or increased mean time to recovery.
Gio is going to talk to you more about that and how this is related to Dora metrics and
also missed optimization opportunities. Like we know that our CI pipelines take a lot to run,
but we don't actually know what to improve if we want to make them sort of make them faster.
So next question, typical, this is fine meme. You know, we know we deploy something,
everything catches fire, we are happy and what we do basically is that we try to mitigate the fire.
But when the observability part of things is so late in the deployment and development and
life cycle, I think it's too late. So there was no reason to let it last this long and get this bad.
So how we can be more proactive? If we shift our focus a little bit to the left, we can address
issues before they escalate and be proactive. We can enhance the efficiency by catching problems
early in the process. We can have, we can ensure robustness by focusing on like the integrity of
our builds and tests and also be mindful about the cost reduction because this is also a really
important topic and minimize expenses associated with post deployment troubleshooting at downtime.
So the next slide, if we assume that we have focused our shift left, the other, so you know,
things turn the other way around. So instead of having the fire everywhere and then us in the
middle like being agnostic of what's happening is the other way around. So we have a lot of time
to mitigate the fire. We can actually be proactive and as we prioritize observability earlier in
the development process, we are identifying and addressing issues actually before they become
fire. We tried many tools. We tried to find the best way to set up like such a system so we can
proactively like react to all those problems. So the tool we found easier to use and address like
all those issues in CI CD pipeline is open telemetry because it helps us create like standard
patterns and some underconventions. Jir is going to talk to you more about that in a little bit.
So in the next slide, we're going to show how we use open telemetry to get to exactly this point
where it's even if something appears, it's still too early and we can, you know, act and fix the
issue before we wake up people in the middle of the night. So stage is yours. Thank you. Can you hear
me? Okay. Thank you. So first question. What is open telemetry? Does anyone know what to work with?
One? So no one. Okay. A few people. But as a short definition of open telemetry is it's an observability
framework which is designed to manage and create telemetry data such as metrics, logs, traces,
events, whatever. There is of course a more comprehensive definition of open telemetry which
is way longer, way more complex, which you can find on the open telemetry website. For this case,
though, our, what I want to focus on here is two bits of definition which is semantic convention
and owning your own data. Now, semantic conventions, we can think about them as a standard. Like it's
a standard way of naming things, of a standard way of defining attributes for your logs, for your
metrics, for your traces. And I mean, we know, we all know this. If we think about semantic
conventions as standards, we can divide them by two different areas or we can categorize them by two
different, in two different ways. By signal type such as metrics, logs, traces, events, whatever.
And by AIA. Now, by AIA means we have telemetry, there are semantic conventions for databases,
we have semantic conventions for cloud providers, we have semantic conventions for
a lot of different things, really, for log files. Something that is not there yet, though,
is semantic conventions for continuous integration and continuous delivery.
Now, what is important in my opinion? This is important because, I mean, we use some
CI tool, I guess, everyone here uses a different CI tool. But regardless of what we use, we can see
that at the end of the day, the data that is behind its CI tool is the same. Regardless of
whether someone calls it stage, someone calls it job, someone calls it status or outcome or
whatever, the underlying data, such as the job name, the outcome of a CI system is the same.
Now, I'm not extremely familiar with every CI system of there, but at some point I was trying
to figure out why in our CI we had a test that sometimes was taking three minutes to a test,
like a pipeline, that sometimes was taking only three minutes to complete, while some others up
to nine or ten, which, I mean, without any code changes. So, like, if you talk about flakiness,
yeah, that's part of, like, test failing, failing, other parties, why sometimes they take too long.
Easy peasy, I think. I wrote some totally reliable Go code. No? No, okay. It was very good code.
So, what was code was doing was getting stuff out of the ground database and pushing it to
log a template in here for later analysis. It worked great. Worked perfectly. Now, what happened
is that we were able, at the end of the day, even if the code wasn't very good, we were able to at
least look at something outside of our CI system. Why this? Because our CI system didn't provide us
with the UI, with the tools to query for the data we were looking for. So, we were trying to
analyze why something was happening. And our UI wasn't able to do so. So, okay. I share the news
with my team. And, I mean, I guess every one of you has been there at some point in your life.
They got too excited. Ivana wanted us to have the log data on Elasticsearch. Piotr was,
which is not a colleague, wanted to get tracing data from Git action, this is a lot of drone.
And, yeah, no, that code was not good enough.
I mean, back to the drawing board. What happens now? We need to figure out a way of getting data out of
GitHub CI, drone, GitHub actions, whatever. I don't know. What else? Bit bucket. I don't know
if they have a CI system. I'm not sure about. And, we need to push it to every database out there,
from Graphite, Tempo, Elasticsearch, I don't know, whatever. Yeager.
I mean, it sounds like a very silly question. I bet there is no one of us that really uses
10 different databases to match their telemetry data. Also, because then, you know, this is what
was going to happen. We had to write code to get data out of every CI system to push it to every
other database system. And, I say no, like, I wasn't going to do that. But, this, I like some very
important point that's owning your data. When I think about, when I started thinking about
owning my data, what I thought about was mostly owning the hardware in which the data was going
to be stored. So, like, owning the drive or having it stored on one of my machines. I think that
that's not exactly the point we need to make here. I think owning your data means you being able to
decide where the data goes, where and how to store the data. We can very, very well be using a cloud
database provider to store our data. The important bit is that we own, we know, we decide where the
data is going to be stored and we decide and we have the ability to use the data however we want.
So, the reason why open telemetry is important and fits very well with the picture is that by defining
standards and by defining a specification for which data can be transferred and stored, not
stored but transferred, we are able to only take care of the first part of the equation here. We
take data out of the systems and then open telemetry is going to take care about
inverting and sending to the database we need.
What we did was we built an open telemetry collector who, does any of you know what
our collector distribution is? A few. Okay, so an open telemetry collector distribution is basically
a set of pre-built components. It's a binary that you can run, of course, and you can configure to
do things. It consists very reductively of receivers, processors and exporters. Receivers
are the components that allow you to get data in, can be like watching at log files, can be,
I don't know, even listening on Bluetooth 1.0 and check for things over the air,
can be extracting metrics from some running services. There are processors that transfer this
data in the format that you need. They add attributes, they modify attributes or remove them.
And exporters that send this data out to your database of choice. The thing here is that
for those exporters, we didn't write anything. Those are already open source exporters.
There are more for elastic search for whatever, like really Jager or you name it.
So the only bit we had to do was writing a drone receiver, which was getting data out of drone
to pipe into open telemetry and then push it to log in, tempo and prometheus.
There are some practical examples. There is a Jenkins plugin that gets traces data out of
Jenkins and sends it via the OTLP format. Irokinz brought these other
get-up functions that run commands and exports the execution of these commands as trace data.
And of course, our own experiment, which is very complete but very well free to take a look at.
Now, what is unlocked? What is unlocked for us? As I said, first of all, there were these
performance issues with our first test. Second thing, at some point we had this test here
that was a bit flaky. Failing sometimes, sometimes not. Of course, worked on my machine, worked on
my advanced machine probably. But yeah, we couldn't figure out what to do with it because we would
disable it but then when we were going to enable it, if you cannot really reproduce it locally.
Now, by getting this data out of our CI and pushing the build logs into our observability
system into our log instance, we were able to trace back from the build that you see on the right
to the logs for that build, trace back to the first time that failed test in our CI.
And from there, if you look down here, this is an attribute that we thought was valuable.
We had a build number, which is our unique ID for drone, which then pointed out to the first
pull request that introduced that test or that flakiness. With that, we were able to identify
what was causing the actual issue, which was a test which was totally unrelated,
running a different suite. But turns out that was causing the flakiness.
Something else that we were able to do was, so first of all, like one, I don't know, silly thing
that we did but we liked, was to create a custom UI within Grafana to mimic sort of like the UI
that you have when you look at the output of your system. I mean, there is some value maybe
near but the important bit here is that we own the data. We were able to do something which was
funny. We spent maybe one day on it. And yeah, it will look good. The second thing, however,
is more important. Now, in Grafana, we have a very complex release system. Very complex.
We maintain a set of different release branches that need, in theory, should need to be
released at an even time. Of course, like everyone, like for everyone, something things breaks.
Sometimes things break and you don't know why because you are not looking at it. Sometimes
a commit you make in main breaks something else somewhere else because you back ported it but
you didn't really test it. What we were able to do with this was keep getting metrics and
stats out of our system, out of our builds so that we could be the timeline of our deployment
branches. This means that at any given time, we had a single pane of glass to look at what was
the status of our release processes so that our release team could just go here and check
whether something was broken they needed to act upon before trying to do our release.
We also had visibility over the stats over the number of running pipelines or failed pipelines.
We can dig into builds. We can do a lot of different things which we didn't feel like
they were possible in our CI system UI. What is Unlocks? Really anything. The point here is that
we are trying to define standards. We are trying to get into this space. It's a very early stage
concept but what it may unlock given that you own your data, it's really up to you. We can talk
about Dora metrics so having ways of reducing mean time to restoring services, we can talk about
generating red like metrics or requests for your CI. How long did it take? Did something that
happened start at the rate of our failing test? The duration started going up because of something
we did on our environment. That's something that for us it unlocked quite well.
Some other example, caught coverage over time. There is no reason why you cannot export test
results as JUnit maybe and then graph them on Grafana and keep in track of your coverage.
You can do flakiness detection like we did before. You start seeing that the test started
failing at some point. You can detect that. You can create an alert on flakiness.
You can trace back to where the test started flaking. At that point we think it's for us
and we think it was way easier to identify what was the actual root cause of the flakiness.
Then we have security, whatever. Really, the data is yours. You decide what to do with it.
Again, all in all, what is unlocked for us at this point, I think there are three different
CI systems. We are using three different systems for different reasons. All in all,
what is unlocked for us was bringing all the data into there to work with Grafana and to have our
production metrics together with the pre-production metrics. Now, what's next? We have formed an
open telemetry working group about CI security observability. There are more stuff to come.
Join the discussion. If you have your own issue that you want to fix or your own
use case that you want to bring up to the group, please join the
Cloud Native Computing Foundation Slack channel. This is the proposal for the standard.
That's it. If you have any questions.
Any questions? Yeah, up there.
Anything you need?
Yeah, we'll try.
What kind of?
Sampling.
So far, we're not sampling anything. We are collecting a trace for every build that goes
through the CI system. For PRs, it's a bit different because we don't want to create
bad data, like useless data. It costs money. Data costs money. What we do is we generate data only
for pipelines that happen on those branches we care about. So if you make a PR and the PR is okay,
it gets merged into main. After it gets merged, we run another pipeline, the same one before the PR,
and that one we collect data from. That way, basically, we have the flakiness on our list
branch and not on the PRs because in PRs, I mean, it's not flaky. I mean, okay, we can
reflect it as in PRs, but maybe we are doing something and it breaks the build, but maybe it's
not a vital point. Yeah, if we did that for every branch, basically, we would face cardinality
explosion and it's going to be so expensive. So you have to define which branches you're interested
in. For example, in Grafana, we have the main branch, which is like the main branch of our repo,
and then some version branches for all the different versions that Grafana have, for example.
And this is what we're interested in. But again, you're on the data. You can decide to do it all
your way. Any other? Yeah.
How many flaky tests or else have you found by exporting data from this?
What's the case? Yeah, number of unnecessary... Oh, yeah, sorry. Apart from flaky tests,
what other metrics can we get, like useful metrics we can get out of that? So I think...
No, no, no. What other problems have you encountered?
Like you found out that you didn't find what you're just looking at?
For example, stack runners. Runners were stuck in unused repositories. We didn't have a way to know
that there was a runner running all the time. We're getting timeouts and all that. This one problem.
Then another problem is the number of restarts in builds, which is basically related to flaky tests.
But there was no way for us to know how many, like for Geo, for example, went and restarted his
build because it was problematic, because there was a bug, or because there was an actual issue
with runner. It doesn't have to be necessarily code related. So we needed to know how big was
the number of the restarts and then try to find the root cause of what caused this, basically.
There was also something I want to talk about, maybe, is that we are also able to improve a bit
of the performance of our pipeline. And by performance, I mean just allocating more resources.
By doing that, we were also maybe able to reduce the cost of the bit because
the runner where pipelines were running for shorter, there was less queue.
So it's also like improving performance also comes from having the data about
how long they take. And also, last thing is that we also had issues where we used extremely
powerful runners to build docs, for example. And docs builds took, I don't know, a minute
where if the docs build took like five minutes, it was not going to be the end of the world,
because there are docs, they're just small changes, really important changes, don't get me wrong,
but small. So we could move away from really powerful runners to something smaller just to
help with some cost reduction and stuff. Any other questions? Do we have time?
Do we have one up there? Do we have time? No? Come join us at the Grafana booth, please.
Thank you.