[00:00.000 --> 00:10.840]  Perfect. Hello everyone. I might need to take a selfie because they're not going to believe
[00:10.840 --> 00:15.200]  me when I get back. They go like, so people like testing, obviously, and they know, you
[00:15.200 --> 00:22.320]  know, sounds. Yeah, I should do a video because they're freaking not going to believe me. So
[00:22.320 --> 00:27.800]  apparently people know what open telemetry is and what testing is. And yeah, I was not
[00:27.800 --> 00:33.520]  expecting this to happen. So you're going to go out on Twitter. That's for sure. But yeah,
[00:33.520 --> 00:42.680]  anyway, let's just take a second to welcome our new guests in. Perfect, perfect. Yeah,
[00:42.680 --> 00:50.240]  this went from fun to stressful really quickly. But yeah, so let me begin. For the next 20 or
[00:50.240 --> 00:54.800]  so minutes, I'll be talking about observability driven development with open telemetry. So a
[00:54.800 --> 00:59.000]  lot of complicated words, a lot of stuff that's going to be happening. And a lot of things I
[00:59.000 --> 01:05.240]  need to explain for you as testers and how you can get started with this new thing of being ODD
[01:05.240 --> 01:12.600]  instead of TDD. So first, a quick rundown of who I am. I'm running DevRel at trace test, which is
[01:12.600 --> 01:18.160]  a, it's like a new tool, new open source tool that we're building for trace based testing. Obviously
[01:18.160 --> 01:24.600]  explain all of that later on. But you're wondering like what am I DevRel person doing at
[01:24.600 --> 01:31.280]  a open source conference when it's kind of because I successfully failed a startup that was doing
[01:31.280 --> 01:36.720]  online education. So I was from there, went into education. And because we're basically educators
[01:36.720 --> 01:42.680]  in DevRel, I was like, maybe, maybe, you know, I write shitty code, I can maybe be good at something
[01:42.680 --> 01:48.000]  like talking. So I figured that might be a good career shift. But I've also been helping build
[01:48.000 --> 01:52.800]  open source DevTools for five or so years. So it's pretty natural for me to be here. So enough
[01:52.800 --> 01:56.280]  about that, you probably think that I know what I'm talking about. Let's keep it, keep it rolling.
[01:56.280 --> 02:02.120]  There are four main topics. So remember these four topics that we will cover in the next 20 or so
[02:02.120 --> 02:07.440]  minutes. And that's, first, we'll talk about the pain of testing microservices. It's a horrible,
[02:07.440 --> 02:13.920]  horrible thing. And we'll also talk about TDD and how integration testing is really hard. We're all
[02:13.920 --> 02:19.400]  doing it. It's terrible. It's hard. But we're still doing it. And then in the last two parts,
[02:19.400 --> 02:24.320]  we'll talk about observability-driven development, how it can help. And then we'll show a code
[02:24.320 --> 02:28.120]  example, a hands-on example of how you can do it as well. So I want you to take something home
[02:28.120 --> 02:33.680]  with you after this 20-minute talk and actually start doing it yourself. So from the beginning,
[02:33.680 --> 02:39.080]  from the top down, let's talk about the pain of testing microservices. So first, the biggest
[02:39.080 --> 02:45.560]  issue is that you have no way of knowing where your HTTP transaction fails. You don't know. You
[02:45.560 --> 02:52.160]  can test an API endpoint. You get a response back. But it might be task failed successfully. You
[02:52.160 --> 02:58.840]  never really know if you have a row of microservices behind that initial service. So that's
[02:58.840 --> 03:03.880]  something you can track. You can track and test how these microservices to microservice
[03:03.880 --> 03:10.720]  communications happen. And of course, the hardest thing, what we all really love to hate, is mocking.
[03:10.720 --> 03:17.800]  It's really hard. It's really, really hard. So the solution that we propose is that we go into
[03:17.800 --> 03:22.440]  doing something called observability-driven development, which means that you're using
[03:22.440 --> 03:28.960]  distributed traces as the test assertions. So you're already using your underlying trace
[03:28.960 --> 03:34.720]  infrastructure to run your tests. And now, because this is a testing dev room, you might not know
[03:34.720 --> 03:40.920]  what tracing is, LightStep has a very nice definition of it. And they say that distributed
[03:40.920 --> 03:46.120]  tracing refers to methods of observing requests as they propagate through a distributed system,
[03:46.120 --> 03:53.200]  which means that if you have a distributed system on the left, you have services that communicate
[03:53.200 --> 03:59.400]  with each other. And on the right, you can see that that entire distributed trace is split into
[03:59.400 --> 04:07.080]  different spans. A span is the smallest unit of a distributed test. So a span can be, it can be a
[04:07.080 --> 04:12.600]  type of stamp. It can be a database interaction or database statement. It can be HTTP codes. It
[04:12.600 --> 04:18.720]  can be objects that you generate in your custom instrumentation itself. So they're literally
[04:18.720 --> 04:26.120]  the smallest form or part of a distributed trace. The distributed system we'll be talking about
[04:26.120 --> 04:31.200]  today, so the samples we will be talking about is very simple. We have two services with a mock
[04:31.200 --> 04:36.720]  database connection. Just to simplify this whole architecture, we will be using this to explain
[04:36.720 --> 04:41.360]  how distributed tracing works and how you run observability-driven development on such a system.
[04:41.360 --> 04:48.760]  Now, just a code sample, because this is JavaScript, the only language I really know, not that well,
[04:48.760 --> 04:53.960]  this is what a trace would look like. You're setting the span, you're adding attributes,
[04:53.960 --> 05:00.920]  and then you're ending the span. So this is the code representation of what we have over here.
[05:00.920 --> 05:08.040]  So just remember that for now, and we'll get into more details as we progress. So the visual
[05:08.040 --> 05:12.080]  queue or the visual layout of a distributed span would look like this. This is taken from the
[05:12.080 --> 05:17.000]  trace test app, but this is any, like any distributed span looks like this, where you have
[05:17.000 --> 05:22.600]  your distributed trace and you can see all of the spans within it. And if you drill into one
[05:22.600 --> 05:29.040]  particular span, you can see, okay, so here are all of the attributes that this span has. It can
[05:29.040 --> 05:34.800]  be available, book ID, the check, and the parent ID. There's a lot of different attributes you have
[05:34.800 --> 05:40.080]  in every span. So the next topic I do want to cover, we have all the basics down. We know why
[05:40.080 --> 05:46.880]  it's hard. We need to figure out why integration testing and TDD really need help. Everybody knows
[05:46.880 --> 05:52.000]  about the red-green feedback loop. It's awesome. It's great. We like it. We don't need to change it.
[05:52.000 --> 06:00.000]  But integration tests are hard. Integration tests are the kicker, where they need access to services
[06:00.000 --> 06:04.160]  and infrastructure. That's the hard part. You need to set up different triggers. You need to access
[06:04.160 --> 06:07.960]  databases. You need to set up environment variables. You need to set up authentication. All of those
[06:07.960 --> 06:14.000]  things that everybody hates doing. And of course, you can track which part of the microservice chain
[06:14.000 --> 06:19.720]  failed, which means that you're writing 90% of your code just as plumbing, just to make sure that
[06:19.720 --> 06:26.560]  the test will run. 10% is actually writing the test, writing the test case, and actually getting value
[06:26.560 --> 06:32.280]  from your TDD process. So here's what a traditional integration test would look like. You have all
[06:32.280 --> 06:37.400]  of your setup. Again, this is JavaScript. It can be any language. You have your setup, a bunch of
[06:37.400 --> 06:41.040]  modules, a bunch of setup, a bunch of plumbing. And then you have more plumbing because you need to
[06:41.040 --> 06:44.800]  mock something. Then you have even more plumbing because you have to figure out how to run this
[06:44.800 --> 06:50.320]  custom freaking syntax that has nothing to do with any language, really. You just have to learn it. So
[06:50.320 --> 06:55.720]  it's a lot of stuff you have to know before you actually run tests. If you compare that to a trace
[06:55.720 --> 07:07.720]  based test, you say, here's my URL. Here's my method. This is what I'm suring against. That's it. No
[07:07.720 --> 07:12.560]  complications, no plumbing, no nothing. It just points to the trace, the trace span you want to
[07:12.560 --> 07:20.160]  target. You have your assertion and it's done. So this is why I think observability driven can help
[07:20.160 --> 07:27.200]  our testing process, where obviously we need to explain what ODD is. The main thing that I think is
[07:27.200 --> 07:31.880]  important to know is that you need to write your code and your observability instrumentation in
[07:31.880 --> 07:38.480]  parallel. So the same way you do the red-green process for TDD, in ODD you write your trace
[07:38.480 --> 07:44.160]  spans and you write your code and your features in parallel. Which is good, first thing because in
[07:44.160 --> 07:49.280]  production that helps your DevOps people when they have troubleshoot, but it's also helping you
[07:49.280 --> 07:54.040]  write better code. And ODD is really powerful because first and foremost, of course, you're not
[07:54.040 --> 08:00.520]  testing mocks. Nothing is artificial. You're not creating black boxes. You're literally testing
[08:00.520 --> 08:05.840]  data from the traces in the real environment. So you can spin up your system, get traces from
[08:05.840 --> 08:11.400]  the system and test on those traces. Of course, it works with all of your existing open telemetry
[08:11.400 --> 08:17.440]  based distributed tracing. So if you have tracing enabled or if you want to enable it, it's really
[08:17.440 --> 08:24.280]  simple nowadays, it'll just work. And then from the ODD definition, we need to figure out what
[08:24.280 --> 08:31.160]  trace-based testing is here. So you basically add assertions against span values. And that's what
[08:31.160 --> 08:37.000]  determines whether the test has failed or the test has passed. It's really straightforward. So you're
[08:37.000 --> 08:41.960]  not just testing against the API response, you're actually testing against the whole distributed
[08:41.960 --> 08:47.240]  trace your system generates. So unlike postman where you trigger a test, you get something back,
[08:47.240 --> 08:52.880]  and then you're asserting on that response, you're literally testing and running assertions against
[08:52.880 --> 08:59.040]  the entire distributed trace. Really, really cool. Now let's go into some practice. How do you do
[08:59.040 --> 09:02.800]  observability during development? Well, you do trace tests because that's the open source tool
[09:02.800 --> 09:08.560]  we're building, you know, shocker. But what's important about trace test is fully open source,
[09:08.560 --> 09:15.600]  100% open source, CNCF project, and it uses open telemetry trace bands as assertions. Very
[09:15.600 --> 09:21.840]  straightforward. Of course, it does work with any existing tracing solution you might have. You can
[09:21.840 --> 09:25.320]  use vendors, you can use open source tools, you can use whatever. If you have tracing in your system,
[09:25.320 --> 09:30.000]  it'll just work. Also, what's important is it doesn't matter if you're a QA engineer, if you're a
[09:30.000 --> 09:36.480]  backend developer, if you're a DevOps person, it'll just work. You have tools for everybody,
[09:36.480 --> 09:43.200]  web UI, CLI, whatever you want, whatever you need, it's there for you. And then why I think it's
[09:43.200 --> 09:48.920]  powerful, you're not running artificial tests, you're testing against real data, and obviously you
[09:48.920 --> 09:53.680]  have a tool belt that you're really used to. You can run test suites by chaining tests together,
[09:53.680 --> 09:59.200]  have transactions where the standard way you're running integration tests is you have a setup,
[09:59.200 --> 10:02.920]  you connect into a database, you're running an insert, you're checking if the insert works,
[10:02.920 --> 10:08.600]  you're deleting that whole path, that environment, and that's what we provide as well. You can set
[10:08.600 --> 10:16.640]  that whole transaction up through the UI. So it's literally what you're used to, but better. You
[10:16.640 --> 10:20.720]  always have test environments as well, which is a very big thing because you can have one set
[10:20.720 --> 10:24.360]  environment for your dev, for your QA, for your prod, for your whatever. So it's very,
[10:24.360 --> 10:29.960]  very flexible in that way as well. Obviously, I'm going to stress this no mocks because I really
[10:29.960 --> 10:36.480]  like that. I hate mocking. So I'm going to just shove this down your throat. Every slide is going
[10:36.480 --> 10:41.840]  to be no mocking. But also, one thing that I think is massively important is that if anybody's
[10:41.840 --> 10:46.360]  running serverless, I've been running serverless since it was a thing like in 2018 when everybody
[10:46.360 --> 10:50.680]  wanted to run serverless, and it was horrible, it was a horrible experience. So I'd suggest nobody
[10:50.680 --> 10:56.960]  really does it. But if you have to because of PMs, testing events on message queues and testing
[10:56.960 --> 11:02.680]  events on distributed systems and services in AWS or whatever, like it's prayer driven
[11:02.680 --> 11:06.800]  development. You never really know what's going to happen. So that's something that we provide.
[11:06.800 --> 11:11.160]  You can literally see the entire trace from that ASIC message queue from other systems,
[11:11.160 --> 11:15.280]  from other services, and you really know what's happening. Obviously, it's important that you
[11:15.280 --> 11:19.760]  get assertions based on timing. Maybe you want all of your database requests and your database
[11:19.760 --> 11:25.120]  queries to finish within 500 milliseconds. That just works. And you can also set wild color
[11:25.120 --> 11:30.400]  assertions. So the same thing I was saying about the database queries, it works for wild cards
[11:30.400 --> 11:36.280]  as well. So a visual demo, like a representation, what that would mean is literally like this. So
[11:36.280 --> 11:40.680]  you have your test executor, which is you can think of that as a trigger. You're testing your
[11:40.680 --> 11:45.440]  system. That trace data is getting written to your trace data store. It can be pretty much
[11:45.440 --> 11:51.880]  anything you've all heard of. Yeager, OpenSearch, Rufana Tempo, OpenTelemetryCollector, like all
[11:51.880 --> 11:56.800]  of those, even vendors like Datadog or whatever. And then what happens is that once the response
[11:56.800 --> 12:02.160]  gets back, we pick up that response, but we also pick up the trace. So you can run assertions
[12:02.160 --> 12:09.040]  based on both the trace and the response itself. And then, obviously, you get the result back
[12:09.040 --> 12:14.160]  and then you can see if it's passed, if it's not passed, what you need to fix, et cetera. So yeah,
[12:14.160 --> 12:19.760]  let's show up after all of this over 10 minutes and perfect. After all of this, just like theory
[12:19.760 --> 12:23.440]  and understanding what's happening, we want to jump into actual code. So let's go back to the
[12:23.440 --> 12:31.000]  sample of checking our trace-based test. So we have a URL and we're making sure that we're sending
[12:31.000 --> 12:37.480]  a GET request to that URL. We're setting up a span. So we're targeting the books span in the
[12:37.480 --> 12:43.440]  Books API and we're making sure that we want to have a list of books equal to three. So this is
[12:43.440 --> 12:48.560]  our TDD red-green process. We have a test. We want to run the code and we see, okay, so we have a
[12:48.560 --> 12:52.920]  handler here. It's getting some books. We have some books, but you can see that there's no
[12:52.920 --> 12:58.600]  instrumentation. So if we do run the test, it's going to say, okay, the 200 is fine, but we're
[12:58.600 --> 13:04.640]  not getting any books here. Red, let's go ahead and refactor. We're adding in our spans. So we
[13:04.640 --> 13:09.640]  say, okay, so now I'll add an attribute and I want to pass in the book's length into this
[13:09.640 --> 13:18.800]  attribute right here. Perfect. Now it passes. So this is the most banal simple use case that you
[13:18.800 --> 13:23.320]  can see, but you're already seeing value from it because you can pass in a custom value. That's a
[13:23.320 --> 13:29.160]  real data. You don't have to mess about with any marking or anything. And then obviously one thing
[13:29.160 --> 13:35.880]  that I'm stressing is very important is what if you want to add a span duration? So I want this
[13:35.880 --> 13:44.880]  API to finish within 500 milliseconds. Okay. Right now, if we have an issue, even though the code
[13:44.880 --> 13:52.080]  works, it might be performing badly. We can add in the span duration, check for the timing, and
[13:52.080 --> 13:57.560]  then obviously refactor if we need to refactor. And that's the thing in the UI as well. Once you do
[13:57.560 --> 14:02.560]  refactor it, this is what you would see. You go and say, okay, so finally now I have a passing test.
[14:02.560 --> 14:09.960]  This book's API is returning within 500 milliseconds. And then obviously the last and I think crucial
[14:09.960 --> 14:15.960]  thing with using trace-based testing is that you can literally test on a search on every part of an
[14:15.960 --> 14:22.440]  HTTP transaction. So if we go back to our books handler API, instead of calling books, we're now
[14:22.440 --> 14:28.520]  calling available books. So we are calling an external API to see if the books are available or
[14:28.520 --> 14:33.640]  not. So we're having this microservice to microservice communication. And if you check that,
[14:33.640 --> 14:39.080]  get the available books function. So we have some promise thingamajig happening here. We're
[14:39.080 --> 14:44.200]  calling an availability API and we're just checking if it's available or not. So the kicker here is
[14:44.200 --> 14:49.520]  we're calling an external API. The external API is super simple. We're just running some tests
[14:49.520 --> 14:54.240]  whether it's available or not and we're setting this attribute. So it's very, very simple example.
[14:54.240 --> 15:08.200]  But the thing is, what if in the availability check, we have a problem? This is why I don't do
[15:08.200 --> 15:12.600]  live demos. Anyway, so if in the availability check, if you're checking here, you can see,
[15:12.600 --> 15:19.560]  oh, we have a problem. There are books that are out of stock. So this is that down the chain
[15:19.560 --> 15:23.920]  action that would happen. You would never know what the hell is the problem. But now,
[15:23.920 --> 15:29.040]  because we have this set up, we can say, okay, so I'm adding into my trace-based test. I want
[15:29.040 --> 15:33.920]  to make sure the availability API is up. So I'm actually triggering this host. And I also want
[15:33.920 --> 15:38.800]  to make sure that all of these is available attributes is true. If I do run that test,
[15:38.800 --> 15:45.440]  I'll see that, whoopsie, I'll see that they're all passing except for this one because, oh wait,
[15:45.440 --> 15:51.360]  there was actually one node, like one part of the trace, one span that was returning false,
[15:51.360 --> 15:56.360]  because one book was out of stock. And if you jump in here, you can see that everything is
[15:56.360 --> 16:03.120]  literally passing. Everything is passing except for that one span, which is something you would
[16:03.120 --> 16:10.560]  never figure out if you're running the traditional way of running tests. And the last thing I really
[16:10.560 --> 16:15.560]  want to stress before we wrap up is that this will work with any distributed system that has
[16:15.560 --> 16:20.760]  open telemetry instrumentation. So any system that looks like this, you have an app with open
[16:20.760 --> 16:24.440]  telemetry, you're sending to the open telemetry collector, and then you're sending that trace
[16:24.440 --> 16:30.560]  data to any trace data store. Yeager open search doesn't really matter. You hook in your trace test
[16:30.560 --> 16:36.800]  instance, you pick up data on every request, you pick up data from the trace data store, and you
[16:36.800 --> 16:43.600]  run these tests. This is the only setup you really need to do. Install the CLI, install the server,
[16:43.600 --> 16:49.120]  one command, one command, and you're ready. Set up your Docker composer or Kubernetes, all of this
[16:49.120 --> 16:54.440]  works out of the box with the install. We have good engineers, like these guys really try to make
[16:54.440 --> 16:59.880]  the install really simple. You set up the trace data store, you can do that in the UI or in the CLI,
[16:59.880 --> 17:08.320]  doesn't really matter. Connect the data store, and you're done. It just works. So last recap,
[17:08.320 --> 17:14.040]  two minutes left. What did we learn today? We learned that obviously open ODD or observability
[17:14.040 --> 17:19.040]  driven development is really awesome. You don't have to mock, again with the mocking. You're testing
[17:19.040 --> 17:23.720]  against real data, and you don't have any black boxes anymore. You know exactly what's happening
[17:23.720 --> 17:30.560]  in every single microservice. You can assert on every step of the transaction. And as the last
[17:30.560 --> 17:36.440]  recap, I mean, you wouldn't be here if you thought testing was fun or easy or something that you
[17:36.440 --> 17:41.400]  really enjoy doing. It is hard, like we all know it is very hard. Testing distributed systems is
[17:41.400 --> 17:48.120]  even harder. Testing microservices is even harder. So I want to help you elevate that TDD process
[17:48.120 --> 17:54.360]  that you're already doing. You're already doing well that you like to doing ODD as well. That's
[17:54.360 --> 17:59.640]  pretty much it. We're on point. If you have any questions, if you want to check out Trace Test,
[17:59.640 --> 18:04.920]  go just go to githubcubeshop slash trace test. You can download it. You can read a blog post I
[18:04.920 --> 18:10.440]  wrote about this as well. So knock yourselves out, I guess. You can also do, just to make it
[18:10.440 --> 18:15.040]  easier, you can do the, like you can also jump into Discord. You can chat with me or the engineers
[18:15.040 --> 18:19.680]  face-to-face. If you have any questions, if you want to try it out, check out the github. Also,
[18:19.680 --> 18:24.920]  give us a star, you know, because it's kind of why I'm here. I have to earn my salary something,
[18:24.920 --> 18:52.200]  in some way. So questions? Yeah, sure. So test run against the trace from the system. Yes,
[18:52.200 --> 18:58.120]  the way it happens is that, imagine you're running a postman request. That would be called,
[18:58.120 --> 19:01.800]  because this is trace test, that would be called response test. You get a response,
[19:01.800 --> 19:07.440]  you're testing on it. For trace test, you get the response, but you're also tapping into the
[19:07.440 --> 19:13.200]  trace data store and getting the traces that that request generates. So from that distributed
[19:13.200 --> 19:18.240]  trace, then you're running assertions based on the spans within that trace, if that answers your
[19:18.240 --> 19:35.440]  question. Yeah, for sure. The only thing is that, obviously, if you're running locally, you have a
[19:35.440 --> 19:41.400]  setup where your application is sending to either an open telemetry collector or whatever. You can
[19:41.400 --> 19:46.520]  also tap into that, where you configure trace test to be the pipeline endpoint of your open
[19:46.520 --> 19:53.760]  telemetry collector. So you can just run it as a dev tool as well. So also we might, I'm not sure
[19:53.760 --> 19:58.200]  if I'm good saying this on camera, but we might be building a desktop app very soon, because we're
[19:58.200 --> 20:02.880]  like half a year into this, so we're still kind of figuring out what you guys need. So that's why
[20:02.880 --> 20:11.840]  I'm here as well. But yeah, let's see what happens. It's a great question, by the way. It's good
[20:11.840 --> 20:28.840]  to finish early. We have time for questions. This is great. So yeah, the question was measuring
[20:28.840 --> 20:33.840]  SLOs for user journeys. That's actually something we're working on now. I'm not sure if you know
[20:33.840 --> 20:39.400]  about the captain project. So we have an integration with the captain project as of last week, quite
[20:39.400 --> 20:44.240]  literally. So if you want to check that out, you just jump into trace test integrates with captain
[20:44.240 --> 20:49.840]  and you'll get a lot of documentation and sample apps examples and whatnot to set that up as well.
[20:49.840 --> 20:53.520]  So that's an excellent use case and something that we actively have been working on. So
[20:53.520 --> 21:07.880]  100% like the thing is that whatever you have implemented, if you have hotel traces coming in
[21:07.880 --> 21:15.200]  from that system, it works. So it's language agnostic, setup agnostic, it's literally like just
[21:15.200 --> 21:19.280]  the traces are important. So if you're running hotel, if you're running the data dog agent,
[21:19.280 --> 21:23.680]  elastic agent, literally anything that generates traces, it'll work. Obviously works best with
[21:23.680 --> 21:28.560]  hotel because open source, you know. But yeah, it just works.
[21:28.560 --> 21:57.240]  So if I understand the question correctly, it is, do I run synthetic tests with trace test? Yes.
[21:57.240 --> 22:01.440]  You can, if you have a CI pipeline or like you can have a cron job somewhere running,
[22:01.440 --> 22:06.040]  doesn't really matter. Every five minutes, I want to trigger this test and make sure that all of
[22:06.040 --> 22:20.360]  the assertions are true. That's perfectly fine. Oh, yeah, 100%. It works. You can think of it
[22:20.360 --> 22:25.840]  as testing in production and making sure that the production environment is healthy. That works as
[22:25.840 --> 22:36.560]  well. Hi. Yeah. So the test, trace test test depends on your instrumentation. Yes. Your
[22:36.560 --> 22:41.400]  instrumentation is in your production code. Yes. Do you have any advice on how do you prevent your
[22:41.400 --> 22:51.960]  production code from bloating with the instrumentation to beat these tests? Hmm. I'm going to say,
[22:51.960 --> 22:57.960]  that's a great question, but I think I'm not even close to being good enough of an engineer
[22:57.960 --> 23:24.040]  to answer that question, to be honest. 100%. 100%. 100%. Also, yeah. Go ahead. Yeah. Yeah,
[23:24.040 --> 23:27.880]  You really had to pick black boxes, but now the other hand,
[23:27.880 --> 23:29.720]  all right, writing tasks like this,
[23:29.720 --> 23:33.200]  and then spending might be unnecessarily
[23:33.200 --> 23:35.600]  tackled into the intricate details
[23:35.600 --> 23:37.240]  of the infrastructure right now.
[23:37.240 --> 23:39.200]  So is this necessarily good?
[23:39.200 --> 23:42.680]  So the text actually knows what's under,
[23:42.680 --> 23:44.360]  what's in that black box.
[23:44.360 --> 23:46.880]  So when you're in factory infrastructure,
[23:46.880 --> 23:50.480]  you might have to throw out all your tasks
[23:50.480 --> 23:52.440]  because you don't have the database
[23:52.440 --> 23:56.520]  that's a good question as well.
[23:56.520 --> 24:01.120]  I think the logical solution would be,
[24:01.120 --> 24:04.520]  trace test is just mapping out your infra.
[24:04.520 --> 24:05.960]  So if you're using it,
[24:05.960 --> 24:08.720]  you can also use it just to gain visibility.
[24:08.720 --> 24:11.640]  So it doesn't have to be that it's only focused
[24:11.640 --> 24:12.680]  on the testing.
[24:12.680 --> 24:14.560]  If you're using it to map out your infra,
[24:14.560 --> 24:15.960]  even if you have changes,
[24:15.960 --> 24:17.280]  if you're running the test again,
[24:17.280 --> 24:19.280]  you'll exactly know what changed.
[24:19.280 --> 24:23.480]  So if you're running assertions based on one database table,
[24:23.480 --> 24:26.280]  so to say, and then running an API on one endpoint
[24:26.280 --> 24:28.880]  that has one particular host name,
[24:28.880 --> 24:31.320]  if you change those up, you'll see what fails
[24:31.320 --> 24:32.320]  and you can figure out, oh, okay,
[24:32.320 --> 24:35.760]  so we changed that last week because of XYZ
[24:35.760 --> 24:37.520]  and you can know exactly what changed.
[24:37.520 --> 24:41.840]  So I think the overview, the visibility into your system,
[24:41.840 --> 24:43.240]  because when you're running microservices,
[24:43.240 --> 24:45.520]  when you're running a bunch of stuff,
[24:45.520 --> 24:47.160]  distributed systems, whatever,
[24:47.160 --> 24:50.480]  it's just hard to have a mental model,
[24:50.480 --> 24:52.960]  a mind map, so to say, of everything that's happening.
[24:52.960 --> 24:56.080]  So I think that's a good part of the value there as well.
[24:57.400 --> 24:59.200]  Thank you, no more time.
[24:59.200 --> 25:00.040]  No more time, yeah.
[25:00.040 --> 25:00.880]  Thank you.
[25:00.880 --> 25:01.720]  Thank you.
[25:01.720 --> 25:02.560]  No more time.
[25:02.560 --> 25:03.400]  No more time, yeah.
[25:03.400 --> 25:04.240]  Thank you.
[25:04.240 --> 25:19.240]  Thank you, no more time, yeah, no more time, yeah, no more time.