with the topic. It is a very big mouthful of a topic today, but I'm hoping that we're
going to break this down for you today and that you're actually going to learn something
that you can take home back to actually implement yourselves. I'm here just to be talking about
the open telemetry part. Sonya is actually the brains of this operation. She's basically
been planning this whole thing, set everything up and just invited me at the end because
yeah, because I'm pretty. That's basically all that I'm contributing today. So I am hopeful
that a lot of you have had any type of touch with open telemetry and observability in general,
but also that you know the basic DevOps principles and how that is going to be connected with
API Ops. Just an introduction for both myself and Sonya. I am Adnan. I do developer relations
as you obviously might have already figured out. And yeah, Sonya here is a product manager
at Tyche and I would like to hand over the microphone.
Yeah, hi. I'm a product manager at Tyche. So we do API management. We have an open source
API gateway. If you were in the session before that, you have seen it on the screen. It's
an API gateway that's written in Go. It's really fast and has lots of capabilities. So
do check it out. And now we are happy to talk about the topic.
Cool. Just a quick rundown of the agenda for today. We have four main topics for the agenda
today. First and foremost, we're going to talk about API Ops, what it is, how you can
get started. And then from there, we're going to take a closer look into how to do API Ops
hands on. So we're going to start with the Kubernetes cluster. We'll walk you through
how to use Argo CD and Tyche for your API gateway and basically just enable very fast
flows and very fast deployments and release cycles within your APIs. From there, we're
going to move into production environment. So we're going to say, okay, so what do I
need to do to get observability, to get insight into my production APIs? And from there, we're
going to shift left even more and figure out how to integrate the release cycles and make
them have integrated. I'm going to say integration testing as well. So we're shifting left even
more using the production data, so the observability data for testing as well. So that's going
to be, I'm going to say my most favorite part because I'm here from Trace Test and we do
that. But for right now, let's do the API Ops portion first.
Yes, so what is API Ops? Thank you. So you might be familiar with API management and I find that
sometimes in API management, we have too many manual operation. And as you all know, manual
operation, that's a cause for disaster, that's a cause for error, that's a cause for security
problems and we need to speed up things. So my interpretation of what is API Ops and
you might have heard about API Ops and some vendors will try to push their ideas of what
is API Ops. Some would say it's about deploying your API fast. I'd like to bring a bit back
the cultural side of DevOps and say that API Ops is the offspring of DevOps and API management.
So it's applying the culture of DevOps to your API management lifecycle. And why? Because
you want to deliver value fast without disrupting your users. So if we think back about the
DevOps culture, the DevOps principle that originally came from before we started to have
lots of vendor trying to sell off things that are DevOps applied, it's about fast flow.
I want to be able to commit and have it used by user to have feedback. So to have that
culture of having feedback loops. And it's also about enabling that culture of learning.
I want to understand what's going on. I want to learn fast, fail fast and be able to provide
value to my users. And we're here today to tell you that we think that observability is
a key enabler for all that in API management or API Ops. So let's take a look at how to
implement API Ops in modern Kubernetes environments to have fast flow. So typically you will have
a developer that's building a service. You will have things like open API specification
along the way. So we had a talk in this room earlier about open API. I'm not going to go
more into details, but it's definitely a space, a place that you have to take into your CI,
into your continuous integration, making it all automated. Today we're going to talk now
a little bit more on the deployment side. That's why we haven't added it, but of course things
like linting and generating documentation. All that should be part of your process. So
once the developer commits something, it goes to the CI, continuous integration, and the
result might be a Docker container. So it gets published. And now we want to deploy that. We
want to deploy that new version of that service. We want to deploy it with an API specification.
And for that in Kubernetes, the new way of doing continuous deployment is to use GitOps.
There are projects like AgroCD or Flux that are able to do GitOps. What does it mean? GitOps?
You're lucky you're really pretty. Okay. So the main thing about GitOps is you don't have a
continuous pipeline that pushes the things and deploy to your server. That's the Kubernetes
cluster with something like Agro. Pull the information and deploy it itself. So how does
it look like? You have then at the end of your CI pipeline, you have to make a change into your
deployment repository. You have a code artifacts for all your changes, all the configuration. And
you might have a new version that is placed into staging. And AgroCD on your Kubernetes cluster
can be configured to automatically pick it up and deploy it. So all automated. Now there's
another thing that you need is to expose an API is an API gateway. So in that example, we are using
tag API gateway to use the authentication, verification, monitoring. So we add an API
gateway, open source API gateway to that. And that's going to be interesting also for the
observability part later. So an API gateway helps you to centrally manage your APIs to use
authentication, authorization, weight limiting, all this capability that you need in operation.
How do you add that? The Kubernetes and GitHub way. Typically we focus on resource definition
like it's the way in Kubernetes. So you can add things. And that's a very, very simple where you
can say which protocol it use. You could define things like weight limiting, like security policy,
which service is proxying on your cluster. And again, it's configuration as code. So it's again
central repository. And when you make changes to it into your deployment configuration
repository, something like ArgosCD will track it and will apply it automatically. So what we see
at the end in your ArgosCD application, you see, okay, all my application definitions, all my
application are synchronized automatically with whatever I put into my Git repository. So now we
have the first step, right? We have automation for fast flow. We are preventing configuration drift.
We have enhanced security. All is automated. No manual error. We are more efficient. We also
have an audit trail. So we see exactly what was changed in the deployment of your APIs. And we
have better collaboration and visibility on what's happening.
Wonderful. And obviously, as the slide says, that is not enough. So we're getting the automation part
down. What do we do next? Step three in the whole process is to get additional feedback into your
feedback loops so you can connect both ops and dev correctly. So what this means is that the ops
team needs to enable the dev team to fix issues by exactly knowing what the issue is, so that the
dev team doesn't need to spend useless cycles trying to figure out what the problem is. And we do that
by using OpenTelemetry and using Yeager, which are observability tools within our API ops pipelines.
Now, this is what we exactly don't want. We don't want to see gears turning and hoping it's all fine
because it's not really fine. You don't know what your users are seeing. So we don't really know if our
users are happy. We just kind of know it works. And then you kind of do prayer driven development, as I
like saying, that's not really what we want. We want to use observability to infer the internal state of
our system by getting telemetry out of our system to understand what's actually happening. And then we
can figure out whether our users are happy. Because this is something that we can see by using
observability with distributed tracing. When our API is exposed telemetry, we can actually see, oh, okay,
obviously something is wrong because we have breaking APIs. So it's pretty obvious that our users are
unhappy because we can obviously see things breaking for them. And this is a particular view that you
get by using Jaeger. Now, let's get to the fun part of actually showing you how it all works and how you
can set it up yourself. Now, the way you do it is you use CNCF observability tooling. So tooling from the
CNCF tracing landscape, more specifically open telemetry and Jaeger. Open telemetry is an incubating
project. Jaeger is a graduated project. So they're all fully open source supported by the CNCF. Now, the
specifics are that you use open telemetry as the open standard, we're very focused on open standard for the
whole dev room today. So once again, it's an open standard to generate, collect and export your
telemetry. Remember that part, it's a bunch of libraries and APIs that help you generate, collect and
export telemetry. Now, where do you export it to? Well, you export it to Jaeger, which is a tracing
backend, which is just like a data store for your distributed tracing. And then you use Jaeger for all of
your production monitoring troubleshooting and whatever else you need to do in your production
environment. Now, from this, one of the bigger issues is that open telemetry is quite hard to implement if
you're new to it. So some vendors like to bake it in into their systems. One such vendor is
there was a lot of suspense, right? Yeah. Yeah. So one thing that we did in tech is to add support,
native support for open telemetry, because we know that people that works in the API space, they use
API's to proxy multiple services, and the developers might not yet have implemented open
telemetry. But we know they need one where to report the data on all the APIs have really visibility on
what's happening. And so we added support, native support for open telemetry in tech to enable our
user to export this data and to capture them automatically for older APIs. So that's need a couple
of settings. This is settings for our hand charts. So where do you need to enable it in tech? You need
to say where do you want to send the data to an open telemetry collector could be also directly to an
observability backend. And this is what you get. So for every API request, you get a distributed
trace for what's happening at the gateway and till the upstream service. So you can see, first of all,
you can see any error that's happening already at the API gateway level, authentication error,
wait limiting. We see sometimes people only monitor what's happening on the service, but they don't
realize they're already missing a lot of people having issue with the authorization, authentication,
wait limiting. And then you see what's happening in the upstream. So you can very, very quickly catch
up errors, understand not only the timing text, the HTTP response code, but really what's happening
if there's an error, if something is slow, where is it happening? Is it on the API gateway, is it on
the upstream service? What are the details of the transaction that enables a team to better troubleshoot
the issue? And with that, we have now achieved feedback from production. So we have healthy
development lifecycle with feedback loop between Dev and Ops. If there's an issue, then the Ops team
can report it, can take a look. So it's not only an error on a metric that goes up, it's really a
trace where you understand where's the problem, you know, which team needs to act on. And it enables
you to provide a better user experience, fix the issues earlier. Again, what have achieved, feedback
from production, we no longer relying on user reporting feature, no longer somebody that calls
support and say, oh, I have a problem, something is done, no, you see it, you see it all, so you can
be proactive. You understand the API performance, you understand really what's happening, where the
error is happening, and you can solve issues faster.
And with that suspendsful mic switch, again, it's not enough. So we need to introduce another layer
of, actually this one, no, we need to introduce another layer of protection. Because right now,
we want, we're only stopping bugs after our users are seeing them. So we exactly know that a user saw
problem that broke our API, and then we're now rotating back to fix it. We need to be more
proactive and figure out how to stop the bugs before they even reach our users. Now, so this is a
shift left even more approach, but actually for you guys, it's shift left even more approach.
Because we want to add observability to our release cycles as well. So not just our production
systems. So the way we're going to go through that a bit is by doing this little squiggly in between,
as well. So this basically means that you need to implement something called trace-based testing,
which is also called observability driven development. If you like honeycomb and their CTO,
it's a term that they coined. Okay. Anyway, the way that you use trace-based testing is you quite
literally using the distributed tracing that your observability, like open telemetry exposes,
and then you're running tests on those actual data points from your infrastructure. So that
means that even though we can see that we have our gears turning, that's awesome. But my initial
connection to that API gateway is returning 200. But how do I know this is not broken? How do I
know if this is on fire or not? This is an external service. I don't like I don't manage this.
So this is something that easily breaks and that you don't really have a lot of control over.
Now, let me show you how you can actually get to that state where you can do your testing against
the distributed trace itself. This is a screenshot from Trace-Test, which is also a
CNCF tracing landscape tool. You can build your test by getting the trace itself from Jaeger,
and then you're writing your test specs directly against trace data. So you're not using any mocking,
you're not using any faking or whatever the word is nowadays with kids use, I don't even know.
You're literally getting the actual data back and running your test against that data.
Now, the magical part here is that you can quite literally test against anything that's
exposing telemetry. It can be an API gateway like TIC, it can be databases like Postgres,
it can be caches like Redis, it can be pretty much anything that you have instrumented to
export traces. Now, this is a really cool use case for authentication as well, but also for
GraphQL. Now, for authentication, you have a very good example.
Yeah, something like Off-Flow where you have multiple service taking to each other and getting
the request, that's one of the really cool, useful examples. And also something that I've
noticed as well is for GraphQL. So one thing for GraphQL is that it often returns a 200,
even though it's failing because the actual error is within the response. So you don't really know,
it's very intricate to test that. One thing you can do with trace-based testing is you can drill
down to the actual middleware that handles that in your API gateway, find the exact error that
happened, and then you can run your test spec on that exact value. So with all of this, we're
getting step one, which is functional testing. So we can actually functionally validate our
behavior of the system by using all of the telemetry that you've implemented in the prior
step to make your production environment reliable. Now, but it doesn't really stop there. We also
have step two, which is performance testing, because every span has a duration. You can quite
literally go in and say, I want my duration for this span to be less than whatever value of 200
milliseconds or something, which means that if you have external services, external APIs, upstream
APIs that you're not in charge of, if their performance is bad, you can validate that and
you know exactly what part of your system is misbehaving. So this is the performance aspect as
well. So you're getting basically two things from one, I'm going to say exercise. Now the way you
do it, I'm going to walk you through quickly. You do this shifting left with trace test, which is,
as I said, open source part of the CNCF tracing landscape as well. And what it does, it is quite
literally giving you the infrastructure by actually the distributed system architecture by looking at
the trace data. And then you can both get the overview of what your system is doing,
and you can run tests against exactly what's happening in your system. So those are two
powerful things because as engineers, it's very hard to know what the system is doing if it's
highly distributed with a lot of microservices, especially if you're a new person on a team,
it's just, it's a pain to do that. But with trace test, I want to show you how you can implement
these integration tests in your Argo CD, like right here. So this is what an integration test in a
post sync hook would look like. You have a API that you're deploying, you have your integration test,
which basically runs a Kubernetes job from Argos, from the Argo CD sync hook, then it runs a few
integration tests. If they, if they're failing, awesome, you know that they're failing, if they're
passing, even better, you see that they're passing, but doesn't really stop here. The thing that you
get with this is also every test that fails, you have a URL to go to that particular test
to actually see precisely which part of that transaction failed within your API, within your
API microservices. And I really like that part because this is not just, oh, yo, this failed,
this is actually, this failed, here's exactly how, where, and what happened.
And with that, we're actually getting to a stage where we're validating our production, but we're
also using that effort we put into our production reliability to validate pre-production as well.
So you're basically getting the exact same overview graph that Sonya just showed you, but
instead of using your end users, you're running tests with trace test against the API Gateway
platform, then you're getting the traces back from your Yeager or Grafana or whatever you're using,
and then that info goes back to the API developer that can then fix the issues that were found.
Now, with this, I'm just going to wrap up everything that we learned from this last section,
which is that we got functional testing and we got performance testing. So you can both
validate your behavior, or actually the behavior of your system, so all upstream and downstream
services, API transactions, both the ones that you manage and don't manage, you can
actually test database performance, you can test cache, you can also test the size of an HTTP
response and request, but you can also do very intricate performance testing by validating
the duration of every part of your API. And with that, I have a saying where I'm from. We say
you're swatting two flies with one swing because I think that's more friendly than killing birds
with stones. So yeah, with that, I think that this is the closest we can get to be bounty
hunters because we're bug hunters. That was very lame. Anyway, so that's a CU space cowboy reference
if somebody can. Thank you for making this. So, and just before we close, I want to say if this is
a topic that's interesting for you, we're running an online API observability conference in February.
It's going to be called LEAP because it's going to be on the LEAP there. So if that's the topic
that's interesting to you, make sure to register. We have lots of people from the API space and
observability space that will be coming. We also have a GitHub project about all the screenshot
that we showed to you today. We were working on it as a GitHub example. We don't have a link for
it, but if you're interested, just reach out to us. Those are LinkedIn. Yeah, I don't like Twitter
anymore. So make sure to send a connect and we're happy to send you a link to a GitHub project. You
can try it all by yourself at this combination of open source projects. Thank you so much.
So we have some time for questions. Yeah, there is one over there.
Questions down. Questions down.
Go ahead with one customer. Yeah.
Okay, so the question is, I have to repeat for the video, the question is, if I have a service that
can be accessed by multiple customers, do I want to have one to send the data to different places
so to split them out or do I want to have just one year, one open telemetry? And as always,
it depends. And on what does it depend? It depends on do you want to give access to those data to
your customers somewhere? Do you want to have strict regulation on the data of your customer
where you may need to split them by location? But yeah.
Yeah, yeah.
Yeah.
Yeah, that's a very, very, very good question. So the question is, how do I monitor the service
level for every customer? So typically you have for every customer, they have, they are authenticated.
So you have maybe something like a token. Yeah, yeah, but in production, yeah, yeah. So they're
authenticated. So when they come to you, you can put a tag on an information on the trace,
and tag will do it automatically if you're using the authorization or authentication from
tag, tag. The API, yeah, it's tag. Tag. Yeah, no worry. And so on the traces, we put the information
on who is going to API. And with open telemetry, you can then use the data to create your own report
based on that information. Yeah. So we add that information on the API call so that you can reuse
it for your report. Yeah, it's directly exposed. Yeah. That's a very good question. It's really
important to monitor per customers because you want to, some customers have different usage,
different patterns, and you want to make sure that every one of them is happy and not just like an
average where you don't really understand whether problems.
Also, the question is whether Trace Test notifies on errors.
No, Trace Test is just a testing tool. You would then need something to automate the test,
like Argo, and then you need something to alert on failures as well. And then you can pick the
alerting tool that you want. Whatever you're using right now, you can automate within your CI,
so you can build your CI within Argo or within whatever you can use Tecton. You can do basically
whatever CI tool you're using, and then you're sending errors on that. So think just integration
testing. You just get works, doesn't work, then you do whatever else you want to do. Yeah.
Yeah. Another question.
Observability data for APS, I can take that one. So,
yeah, so the question is how do you deal with data privacy? And because in the observability
data, they can land a lot that could be considered privacy data. So first, you have to be very
aware of that, that observability data could potentially have some data that in your country,
in your own regulation could have some impact. OpenTelemetry has a lot of tool for that. In the
OpenTelemetry collector, there are kind of plugins that you can define using Yamal and say,
that arguments, that thing I want to filter out, I don't want to register it. So you're very flexible
in your observability pipeline, but that's something that you have to take care of to make
sure that your developers haven't added something that you don't want to store.
Sorry. I'll go for it. Go for it.
Jack, when I use the data to send the data to the OpenTelemetry, this data is made only on HB8.
HB8, the status only. So like a 100, 500 message, the status of the response of the
HB8 request. Yes. All on another way is to analyze the response of the request.
So the question is, what do we track or what kind of data do we expose with tech?
So, yeah, so in tag the gateway, when it's being called, you will get the answer, but the traces,
it will export using OpenTelemetry will contain all the data, all the steps, the traces that we
saw in Yeager. And you can also extend them. So we have a plugin mechanism where you could,
that you could load into there and add even more data if that's more open, extend your OpenTelemetry
traces. The question is, where is the effort? So tech make it easier for you because it captured
the starts up to the call to the upstream service and it tell you how long it took.
And but if you want to get even more details, what happens after that, then it's where you
need to instrument your services using OpenTelemetry. And then the beauty of it is when all the services
speak the same observability language, they all send the data to the same place,
then you have the full picture and that's kind of the operational dream. Thank you.
Yeah. You suggest to run that on a trade production?
It's right.
Correct. Correct. So you wouldn't use trace this in this point of view for your production,
you would use it in pre production, where you need sampling to be at 100%.
Yeah, yeah, we can also just stand. We'll just wait so you can come by and chat with us. So because
yeah, we don't have time. Don't follow up on questions. Yeah. So yeah, yeah, we'll be here. Come here. Yeah.
Cool. Thank you.