Perfect. Hello everyone. I might need to take a selfie because they're not going to believe
me when I get back. They go like, so people like testing, obviously, and they know, you
know, sounds. Yeah, I should do a video because they're freaking not going to believe me. So
apparently people know what open telemetry is and what testing is. And yeah, I was not
expecting this to happen. So you're going to go out on Twitter. That's for sure. But yeah,
anyway, let's just take a second to welcome our new guests in. Perfect, perfect. Yeah,
this went from fun to stressful really quickly. But yeah, so let me begin. For the next 20 or
so minutes, I'll be talking about observability driven development with open telemetry. So a
lot of complicated words, a lot of stuff that's going to be happening. And a lot of things I
need to explain for you as testers and how you can get started with this new thing of being ODD
instead of TDD. So first, a quick rundown of who I am. I'm running DevRel at trace test, which is
a, it's like a new tool, new open source tool that we're building for trace based testing. Obviously
explain all of that later on. But you're wondering like what am I DevRel person doing at
a open source conference when it's kind of because I successfully failed a startup that was doing
online education. So I was from there, went into education. And because we're basically educators
in DevRel, I was like, maybe, maybe, you know, I write shitty code, I can maybe be good at something
like talking. So I figured that might be a good career shift. But I've also been helping build
open source DevTools for five or so years. So it's pretty natural for me to be here. So enough
about that, you probably think that I know what I'm talking about. Let's keep it, keep it rolling.
There are four main topics. So remember these four topics that we will cover in the next 20 or so
minutes. And that's, first, we'll talk about the pain of testing microservices. It's a horrible,
horrible thing. And we'll also talk about TDD and how integration testing is really hard. We're all
doing it. It's terrible. It's hard. But we're still doing it. And then in the last two parts,
we'll talk about observability-driven development, how it can help. And then we'll show a code
example, a hands-on example of how you can do it as well. So I want you to take something home
with you after this 20-minute talk and actually start doing it yourself. So from the beginning,
from the top down, let's talk about the pain of testing microservices. So first, the biggest
issue is that you have no way of knowing where your HTTP transaction fails. You don't know. You
can test an API endpoint. You get a response back. But it might be task failed successfully. You
never really know if you have a row of microservices behind that initial service. So that's
something you can track. You can track and test how these microservices to microservice
communications happen. And of course, the hardest thing, what we all really love to hate, is mocking.
It's really hard. It's really, really hard. So the solution that we propose is that we go into
doing something called observability-driven development, which means that you're using
distributed traces as the test assertions. So you're already using your underlying trace
infrastructure to run your tests. And now, because this is a testing dev room, you might not know
what tracing is, LightStep has a very nice definition of it. And they say that distributed
tracing refers to methods of observing requests as they propagate through a distributed system,
which means that if you have a distributed system on the left, you have services that communicate
with each other. And on the right, you can see that that entire distributed trace is split into
different spans. A span is the smallest unit of a distributed test. So a span can be, it can be a
type of stamp. It can be a database interaction or database statement. It can be HTTP codes. It
can be objects that you generate in your custom instrumentation itself. So they're literally
the smallest form or part of a distributed trace. The distributed system we'll be talking about
today, so the samples we will be talking about is very simple. We have two services with a mock
database connection. Just to simplify this whole architecture, we will be using this to explain
how distributed tracing works and how you run observability-driven development on such a system.
Now, just a code sample, because this is JavaScript, the only language I really know, not that well,
this is what a trace would look like. You're setting the span, you're adding attributes,
and then you're ending the span. So this is the code representation of what we have over here.
So just remember that for now, and we'll get into more details as we progress. So the visual
queue or the visual layout of a distributed span would look like this. This is taken from the
trace test app, but this is any, like any distributed span looks like this, where you have
your distributed trace and you can see all of the spans within it. And if you drill into one
particular span, you can see, okay, so here are all of the attributes that this span has. It can
be available, book ID, the check, and the parent ID. There's a lot of different attributes you have
in every span. So the next topic I do want to cover, we have all the basics down. We know why
it's hard. We need to figure out why integration testing and TDD really need help. Everybody knows
about the red-green feedback loop. It's awesome. It's great. We like it. We don't need to change it.
But integration tests are hard. Integration tests are the kicker, where they need access to services
and infrastructure. That's the hard part. You need to set up different triggers. You need to access
databases. You need to set up environment variables. You need to set up authentication. All of those
things that everybody hates doing. And of course, you can track which part of the microservice chain
failed, which means that you're writing 90% of your code just as plumbing, just to make sure that
the test will run. 10% is actually writing the test, writing the test case, and actually getting value
from your TDD process. So here's what a traditional integration test would look like. You have all
of your setup. Again, this is JavaScript. It can be any language. You have your setup, a bunch of
modules, a bunch of setup, a bunch of plumbing. And then you have more plumbing because you need to
mock something. Then you have even more plumbing because you have to figure out how to run this
custom freaking syntax that has nothing to do with any language, really. You just have to learn it. So
it's a lot of stuff you have to know before you actually run tests. If you compare that to a trace
based test, you say, here's my URL. Here's my method. This is what I'm suring against. That's it. No
complications, no plumbing, no nothing. It just points to the trace, the trace span you want to
target. You have your assertion and it's done. So this is why I think observability driven can help
our testing process, where obviously we need to explain what ODD is. The main thing that I think is
important to know is that you need to write your code and your observability instrumentation in
parallel. So the same way you do the red-green process for TDD, in ODD you write your trace
spans and you write your code and your features in parallel. Which is good, first thing because in
production that helps your DevOps people when they have troubleshoot, but it's also helping you
write better code. And ODD is really powerful because first and foremost, of course, you're not
testing mocks. Nothing is artificial. You're not creating black boxes. You're literally testing
data from the traces in the real environment. So you can spin up your system, get traces from
the system and test on those traces. Of course, it works with all of your existing open telemetry
based distributed tracing. So if you have tracing enabled or if you want to enable it, it's really
simple nowadays, it'll just work. And then from the ODD definition, we need to figure out what
trace-based testing is here. So you basically add assertions against span values. And that's what
determines whether the test has failed or the test has passed. It's really straightforward. So you're
not just testing against the API response, you're actually testing against the whole distributed
trace your system generates. So unlike postman where you trigger a test, you get something back,
and then you're asserting on that response, you're literally testing and running assertions against
the entire distributed trace. Really, really cool. Now let's go into some practice. How do you do
observability during development? Well, you do trace tests because that's the open source tool
we're building, you know, shocker. But what's important about trace test is fully open source,
100% open source, CNCF project, and it uses open telemetry trace bands as assertions. Very
straightforward. Of course, it does work with any existing tracing solution you might have. You can
use vendors, you can use open source tools, you can use whatever. If you have tracing in your system,
it'll just work. Also, what's important is it doesn't matter if you're a QA engineer, if you're a
backend developer, if you're a DevOps person, it'll just work. You have tools for everybody,
web UI, CLI, whatever you want, whatever you need, it's there for you. And then why I think it's
powerful, you're not running artificial tests, you're testing against real data, and obviously you
have a tool belt that you're really used to. You can run test suites by chaining tests together,
have transactions where the standard way you're running integration tests is you have a setup,
you connect into a database, you're running an insert, you're checking if the insert works,
you're deleting that whole path, that environment, and that's what we provide as well. You can set
that whole transaction up through the UI. So it's literally what you're used to, but better. You
always have test environments as well, which is a very big thing because you can have one set
environment for your dev, for your QA, for your prod, for your whatever. So it's very,
very flexible in that way as well. Obviously, I'm going to stress this no mocks because I really
like that. I hate mocking. So I'm going to just shove this down your throat. Every slide is going
to be no mocking. But also, one thing that I think is massively important is that if anybody's
running serverless, I've been running serverless since it was a thing like in 2018 when everybody
wanted to run serverless, and it was horrible, it was a horrible experience. So I'd suggest nobody
really does it. But if you have to because of PMs, testing events on message queues and testing
events on distributed systems and services in AWS or whatever, like it's prayer driven
development. You never really know what's going to happen. So that's something that we provide.
You can literally see the entire trace from that ASIC message queue from other systems,
from other services, and you really know what's happening. Obviously, it's important that you
get assertions based on timing. Maybe you want all of your database requests and your database
queries to finish within 500 milliseconds. That just works. And you can also set wild color
assertions. So the same thing I was saying about the database queries, it works for wild cards
as well. So a visual demo, like a representation, what that would mean is literally like this. So
you have your test executor, which is you can think of that as a trigger. You're testing your
system. That trace data is getting written to your trace data store. It can be pretty much
anything you've all heard of. Yeager, OpenSearch, Rufana Tempo, OpenTelemetryCollector, like all
of those, even vendors like Datadog or whatever. And then what happens is that once the response
gets back, we pick up that response, but we also pick up the trace. So you can run assertions
based on both the trace and the response itself. And then, obviously, you get the result back
and then you can see if it's passed, if it's not passed, what you need to fix, et cetera. So yeah,
let's show up after all of this over 10 minutes and perfect. After all of this, just like theory
and understanding what's happening, we want to jump into actual code. So let's go back to the
sample of checking our trace-based test. So we have a URL and we're making sure that we're sending
a GET request to that URL. We're setting up a span. So we're targeting the books span in the
Books API and we're making sure that we want to have a list of books equal to three. So this is
our TDD red-green process. We have a test. We want to run the code and we see, okay, so we have a
handler here. It's getting some books. We have some books, but you can see that there's no
instrumentation. So if we do run the test, it's going to say, okay, the 200 is fine, but we're
not getting any books here. Red, let's go ahead and refactor. We're adding in our spans. So we
say, okay, so now I'll add an attribute and I want to pass in the book's length into this
attribute right here. Perfect. Now it passes. So this is the most banal simple use case that you
can see, but you're already seeing value from it because you can pass in a custom value. That's a
real data. You don't have to mess about with any marking or anything. And then obviously one thing
that I'm stressing is very important is what if you want to add a span duration? So I want this
API to finish within 500 milliseconds. Okay. Right now, if we have an issue, even though the code
works, it might be performing badly. We can add in the span duration, check for the timing, and
then obviously refactor if we need to refactor. And that's the thing in the UI as well. Once you do
refactor it, this is what you would see. You go and say, okay, so finally now I have a passing test.
This book's API is returning within 500 milliseconds. And then obviously the last and I think crucial
thing with using trace-based testing is that you can literally test on a search on every part of an
HTTP transaction. So if we go back to our books handler API, instead of calling books, we're now
calling available books. So we are calling an external API to see if the books are available or
not. So we're having this microservice to microservice communication. And if you check that,
get the available books function. So we have some promise thingamajig happening here. We're
calling an availability API and we're just checking if it's available or not. So the kicker here is
we're calling an external API. The external API is super simple. We're just running some tests
whether it's available or not and we're setting this attribute. So it's very, very simple example.
But the thing is, what if in the availability check, we have a problem? This is why I don't do
live demos. Anyway, so if in the availability check, if you're checking here, you can see,
oh, we have a problem. There are books that are out of stock. So this is that down the chain
action that would happen. You would never know what the hell is the problem. But now,
because we have this set up, we can say, okay, so I'm adding into my trace-based test. I want
to make sure the availability API is up. So I'm actually triggering this host. And I also want
to make sure that all of these is available attributes is true. If I do run that test,
I'll see that, whoopsie, I'll see that they're all passing except for this one because, oh wait,
there was actually one node, like one part of the trace, one span that was returning false,
because one book was out of stock. And if you jump in here, you can see that everything is
literally passing. Everything is passing except for that one span, which is something you would
never figure out if you're running the traditional way of running tests. And the last thing I really
want to stress before we wrap up is that this will work with any distributed system that has
open telemetry instrumentation. So any system that looks like this, you have an app with open
telemetry, you're sending to the open telemetry collector, and then you're sending that trace
data to any trace data store. Yeager open search doesn't really matter. You hook in your trace test
instance, you pick up data on every request, you pick up data from the trace data store, and you
run these tests. This is the only setup you really need to do. Install the CLI, install the server,
one command, one command, and you're ready. Set up your Docker composer or Kubernetes, all of this
works out of the box with the install. We have good engineers, like these guys really try to make
the install really simple. You set up the trace data store, you can do that in the UI or in the CLI,
doesn't really matter. Connect the data store, and you're done. It just works. So last recap,
two minutes left. What did we learn today? We learned that obviously open ODD or observability
driven development is really awesome. You don't have to mock, again with the mocking. You're testing
against real data, and you don't have any black boxes anymore. You know exactly what's happening
in every single microservice. You can assert on every step of the transaction. And as the last
recap, I mean, you wouldn't be here if you thought testing was fun or easy or something that you
really enjoy doing. It is hard, like we all know it is very hard. Testing distributed systems is
even harder. Testing microservices is even harder. So I want to help you elevate that TDD process
that you're already doing. You're already doing well that you like to doing ODD as well. That's
pretty much it. We're on point. If you have any questions, if you want to check out Trace Test,
go just go to githubcubeshop slash trace test. You can download it. You can read a blog post I
wrote about this as well. So knock yourselves out, I guess. You can also do, just to make it
easier, you can do the, like you can also jump into Discord. You can chat with me or the engineers
face-to-face. If you have any questions, if you want to try it out, check out the github. Also,
give us a star, you know, because it's kind of why I'm here. I have to earn my salary something,
in some way. So questions? Yeah, sure. So test run against the trace from the system. Yes,
the way it happens is that, imagine you're running a postman request. That would be called,
because this is trace test, that would be called response test. You get a response,
you're testing on it. For trace test, you get the response, but you're also tapping into the
trace data store and getting the traces that that request generates. So from that distributed
trace, then you're running assertions based on the spans within that trace, if that answers your
question. Yeah, for sure. The only thing is that, obviously, if you're running locally, you have a
setup where your application is sending to either an open telemetry collector or whatever. You can
also tap into that, where you configure trace test to be the pipeline endpoint of your open
telemetry collector. So you can just run it as a dev tool as well. So also we might, I'm not sure
if I'm good saying this on camera, but we might be building a desktop app very soon, because we're
like half a year into this, so we're still kind of figuring out what you guys need. So that's why
I'm here as well. But yeah, let's see what happens. It's a great question, by the way. It's good
to finish early. We have time for questions. This is great. So yeah, the question was measuring
SLOs for user journeys. That's actually something we're working on now. I'm not sure if you know
about the captain project. So we have an integration with the captain project as of last week, quite
literally. So if you want to check that out, you just jump into trace test integrates with captain
and you'll get a lot of documentation and sample apps examples and whatnot to set that up as well.
So that's an excellent use case and something that we actively have been working on. So
100% like the thing is that whatever you have implemented, if you have hotel traces coming in
from that system, it works. So it's language agnostic, setup agnostic, it's literally like just
the traces are important. So if you're running hotel, if you're running the data dog agent,
elastic agent, literally anything that generates traces, it'll work. Obviously works best with
hotel because open source, you know. But yeah, it just works.
So if I understand the question correctly, it is, do I run synthetic tests with trace test? Yes.
You can, if you have a CI pipeline or like you can have a cron job somewhere running,
doesn't really matter. Every five minutes, I want to trigger this test and make sure that all of
the assertions are true. That's perfectly fine. Oh, yeah, 100%. It works. You can think of it
as testing in production and making sure that the production environment is healthy. That works as
well. Hi. Yeah. So the test, trace test test depends on your instrumentation. Yes. Your
instrumentation is in your production code. Yes. Do you have any advice on how do you prevent your
production code from bloating with the instrumentation to beat these tests? Hmm. I'm going to say,
that's a great question, but I think I'm not even close to being good enough of an engineer
to answer that question, to be honest. 100%. 100%. 100%. Also, yeah. Go ahead. Yeah. Yeah,
You really had to pick black boxes, but now the other hand,
all right, writing tasks like this,
and then spending might be unnecessarily
tackled into the intricate details
of the infrastructure right now.
So is this necessarily good?
So the text actually knows what's under,
what's in that black box.
So when you're in factory infrastructure,
you might have to throw out all your tasks
because you don't have the database
that's a good question as well.
I think the logical solution would be,
trace test is just mapping out your infra.
So if you're using it,
you can also use it just to gain visibility.
So it doesn't have to be that it's only focused
on the testing.
If you're using it to map out your infra,
even if you have changes,
if you're running the test again,
you'll exactly know what changed.
So if you're running assertions based on one database table,
so to say, and then running an API on one endpoint
that has one particular host name,
if you change those up, you'll see what fails
and you can figure out, oh, okay,
so we changed that last week because of XYZ
and you can know exactly what changed.
So I think the overview, the visibility into your system,
because when you're running microservices,
when you're running a bunch of stuff,
distributed systems, whatever,
it's just hard to have a mental model,
a mind map, so to say, of everything that's happening.
So I think that's a good part of the value there as well.
Thank you, no more time.
No more time, yeah.
Thank you.
No more time.
No more time, yeah.
Thank you.
Thank you, no more time, yeah, no more time, yeah, no more time.