Hello. How's it going, everybody? A lot of people in this room didn't really expect
so many. This is wonderful. Thank you for coming to see us. I just want to say that
we want to talk today about mutation testing. That's what we're here for. If you like this
penguin, does anyone not like this penguin? Just you. Okay. Personal vendetta noted.
This is a penguin generated by Dolly. Hopefully, it's friendly enough because this is going
to be part of our talk. We're going to see a lot of penguins in this talk. If anyone
has a personal objection to penguins, please speak now. Otherwise, if you like penguins,
can I get a hand up just to see if we cool with that? Awesome. I've never seen so many
people want to put their hands up but not really be sure. I absolutely love the energy
in this room. My name's Max. I'm this guy. As you can tell, I'm also this guy. I'm here
to talk to you about mutation testing. I work for a company called Vonage and I'm a Python
developer advocate there. Now, what that means is that I maintain our Python tooling. I'm
here to talk about mutation testing because I've just kind of went through this process
myself of understanding all this stuff and applying it to my own work. I want to show
you kind of how that went. But with me, not only do I have the tallest person in the room.
Stand up straight. Stand up straight. This person is 196 centimeters tall. I'm like 177.
I'm not sure. I promise. I'm average in Britain. In this place, right? This person knows a
lot more about mutation testing than me. I'm really not the expert here but I just want
to say this is Paco. Yes. I'm Paco. I work for Opavailu, a small consultancy company
in the Netherlands. I got into mutation testing via my thesis. When I wrote my thesis on test
effectiveness, I wanted to learn more about mutation. Also, after that, I got into speaking
at conferences and more spreading the word about this. Quite awesome, too. I hope that
at the end of the talk, you have another cool tool in your toolbox to write better code.
Awesome. If we're cool with that, we do have to do the obligatory. These companies paid
for us to come here and paid for our flights and stuff. What my company does, I'll just
quickly tell you. We do communications APIs as a service, basically. Things like SMS,
like voice calls, like video chats, like two-factor authentication, all via API. That's kind
of what we do. That's really just what I want to say. It is relevant because I will show
you what I actually applied this to, which was one of our SD case.
For me, we don't actually have a product to sell. Also, definitely didn't fly here from
the Netherlands just to make that sure. It's just a two-hour car drive. No, so we're here
just a consultancy company, and we really like to share knowledge. That's mostly the
reason why I'm here to tell you more and teach you more. It's quite simple.
Yeah. He doesn't have the funding crush that I do, unfortunately. Luckily, we're all good.
There's two of us on this talk. There's two of us here, and actually, there is a third
person in this talk. We've seen a hint about this person already, but this person's really
the thing that's going to tie this whole talk together, and it's going to get us all feeling
good about mutation testing. This person's very important, so say hello to Henry. This
is Henry. Look at his little face.
Thank you. Hands up if you think Henry's a cute AF penguin.
As for it, thank you very much. Yes, I'm glad we agree. I'm glad we're on the same page.
Now, just some quick audience participation, because if you can't tell, we're quite big
on audience participation. So, quick question here. Who has heard of this stock photo, but
more importantly, who's heard of testing? This is just a check to see if we found the
room. Thank you very much. Great stuff. Okay. Who's heard about code coverage? A lot of
people maybe not everybody, and that's okay if you haven't. We're going to talk about
code coverage, so please don't worry if you haven't. But yeah, it's awesome to know that
some people have. That's a good starting point too. Okay, final one. I'm going to say other
than Vaia knowing about this talk, who's heard of mutation testing? Oh, quite a few. Yeah.
And now, quick break. Who actually was already using mutase testing? Ah, nice. There are
enough quick wins here, and hopefully you have some good experiences.
Yeah. So, really nice to see that people are familiar with the concept, but if you're not,
it's also okay, because we're going to go through this like you don't know anything
at all, because when I started doing this, you know, a few months ago, I didn't know
anything at all, and so I want to take you through that journey as well, and that's what
we're going to do. But before that, what I want to do first is give us some background,
and what I actually really want to do is pass to Paco, who knows a lot more about this than
me, so I'm going to pass to you right now. Yes. This is going to be some improvising.
Good work. Good luck. I'm going to drink water with this. I'll feed you. Yeah. Nice. Great.
So, yeah, we're first going to talk a bit about testing in general, and then we're going
to more specifically talk about unit testing. So, just a quick check. Does anybody know
what a unit test is? That's great. I don't have to explain that part. For those who don't
know, it's the smallest possible test you can write in your code base, just in one method,
and you write one test for it to test the outcome of that method. Now, there are many
different reasons why we're writing unit tests, and I think one of them, my favorite or the
most used one is for maintenance. We write tests because we want to be confident in the
changes we make to our code base. So, whenever we make a small change, we add a new field
to some endpoint that we know that we didn't completely break the database integration
because it can happen at times. So, yeah, that's very important maintenance regression
testing, but there are more reasons. One I like also a lot is tests can actually serve
documentation purposes as documentation. You can use tests to describe certain scenarios
in your code base that when you have a specific test for that, it already makes clear this
is intended behavior. I have an example for this, which is I worked for a company where
we had an endpoint that returned warehouses, and these warehouses, just a domain object,
had a soft delete. So, there was a flag in there that indicated whether it was deleted
or not. At some point, so this endpoint returned both deleted and non-deleted warehouses, and
at some point over time, as we were working on it, a new guy came in and looked at it
and said, hmm, that's strange. Why are we returning deleted warehouses? Why would you
want that? It was a fair question because we also forgot, and there was only one test
which tested the success flow, and you can already kind of guess here a bit. So, the
success flow in this case meant they only returned non-deleted warehouses in the test.
So he made the changes, and we all thought, oh, this makes sense. It looks broken. Of
course, they didn't check with product management, the product team deployed it, and then you
can guess, of course, this was broken, so we had to revert it. And the whole lesson here
was just one test which also included a negative scenario with tests of warehouses that were
deleted could have already been a trigger to think like, hey, this behavior is intended,
and that's where documentation, so where tests can serve as sort of a documentation purpose.
Also very useful in getting to learn a new code base. So, whenever you're on a new code
base, you have this very complicated method. A test can help you step through the method
to sort of explain what's going on, for example, while debugging it.
Now, another one, and this one is here for the consultant. So, who here works as a consultant?
Oh, not that many. Wow. Because we're sort of the root of all evil always. We tend to
run to the next project, and we don't have to maintain our own code often, not always.
So I have this nice quote that's mostly also for us. Keep in mind that you're not doing
this only for yourself. I had a colleague who once told me, keep in mind that you always
have this point in your development process where you think, okay, should I write a unit
test for this? It's going to be a painful unit test. I know that it works. I do really
have to document it. We all know how it works. Yeah, sure, we all know how it works, but
we also leave the project and then go on and go to another project. We as consultants.
And I will speak to myself, what would I do if I would be the next person? So what would
I do if I were the next John or Jane Doe working on this project? So tests are not there just
for you, but also for the next person working. I would actually like to jump in here, because
I've been that person. Thank you. I've been the person who works on a project after someone's
left that. And honestly, if you have good documentation or if you don't have that, if you have good
testing, thank you, you do your water break. So if you have good testing, it can really
help you understand what a project does. And so when I came to a certain project recently,
I didn't have necessarily the kind of testing that I would have liked to really document
my code that well. And so like, honestly, if I'd had someone like Parker, who actually
was a bit more conscientious with what they tested, that would have really helped me get
on board with the project quickly. But as it was, this was a real problem for me. And
it was something that we want to hopefully avoid other people having to deal with as well.
Like, quick question, actually. Has anybody ever taken over a code base that they may be
looking at and go, what the heck is this? Okay, so you know what the point of this slide,
right? You know why we're saying this. We know this is important. Now, let's stop that
from happening to the next generation of very pain developers, right? Let's stop that happening.
Yes, so write tests. And so if all these reasons haven't convinced you, there's often maybe
a team lead or a boss or somebody else who's telling you to write tests. In most cases,
there's always, of course, exceptions. Ah, okay. Wow. This is annoying. So at the end
of the day, we're all writing tests, if it's not for ourselves and it's for someone else.
And as we're, even though we're now sort of happily all adding tests, we also have to
sort of sketch a problem scenario here. And this problem is that as projects evolve and
grow, our tests also evolve and grow. But the problem is that we do reflect there a lot
and we spend a lot of time on keeping our production code clean and well monitored. We
have lots of metrics where on the other hand for tests, what you can see on long living
projects is that sometimes you just get tests where nothing more than a blank setup and
tear down and some mocking going on because the functionality already moved long ago,
which means to the point that test code is often not monitored. Test code is sort of
our, the kid that didn't get all the attention that needed. So there is still one metric
for testing. What's one, what do you think is the most used metric for test code? Yes,
yeah, we have sort of gave it away already in the intro, but yes, yes, code coverage.
Code coverage tells you how much of the code is executed when you run the test suite. And
I personally really like code coverage because it already helps you write more and better
tests. And I want to go through a simple example here to show you how it can already help you.
So here we have a submit method. So this is the Python guy. I'm the Java guy. Yeah, he
said simple example, but I don't, I don't think so. Yeah. So the context is you are at
the conference and you have a service where you can submit proposals. You can only have,
you can't have more than three, three or more over proposals and you can submit after the
deadline. If you do that, there will be a failure and otherwise you will get success. So quite
a simple method with everything as a parameter just to make it easy to explain. So if you
would take method coverage, method coverage is the simplest coverage metric we can get
which checks is this method coverage as or no, we can add one simple test called a test
X which submits a proposal. There are no open proposals, which is good. And we have a deadline
that's 999 seconds in the future. So great. Now we can get a step further. We can get
into statement coverage and with statement coverage, we check, well, if each statement
was executed and now we see, hey, we didn't cover our unhappy flow. So we need to add
another test. In this case, we add another test which has five over proposals, which
means this check evaluates the true and we have a negative scenario. Now we can even
go one step further through, for example, condition coverage. And with condition coverage,
we check if each Boolean sub-expression has been evaluated to both true and false because
what we don't know now is whether our deadline check is actually working. We just know that
it returns false, but we haven't seen it return true yet. So we add one more test now with
a deadline that is 999 seconds in the past. And now we have three tests. And this is already
why I like code coverage so much because it really helps you write proper tests. Proper,
it helps you write tests because let me get on to the good part here. As I said, writes
better and more tests. Code coverage is really easy and cheap to measure. In most, I think
most of the languages, it's just a matter of instrumenting the code. You run the test
suite and you get a nice report out of it that everybody can quickly see and you can
quickly see the pain points of where you're lacking in testing. But to get a bit further,
so it guarantees, as I mentioned, it shows you what you didn't test. But the only guarantee
I'm going to get to the bad parts next is that the only thing that shows you that what
you did test didn't crash. It doesn't guarantee anything actually about functionality because
code coverage can actually be quite misleading. It doesn't guarantee any test quality. So
if I take this method, for example, this is a unit test, a valid unit test, this test
generates coverage. It calls a method, but there is no assertion on the result, which
makes this test, for example, generate 80% coverage, yet the test actually only guarantees
it said the method doesn't crash. It doesn't tell us whether it's returned true, false
or anything. And this is the pain point of code coverage, which brings us to the something
nice which Max told me about, which is called the good horse law. So can you maybe explain
a bit about that? Can I grab your clicker? Can I explain about
good horse law? No, sorry. I can't. Just kidding. Okay, so when a metric becomes a target, it
seems it's to be a good metric. So quick question, has anyone ever written a unit test just to
get coverage up rather than because the test was useful? Come on, let's be honest. This
is the safe space. Okay. Microphone, okay. Hello, everybody. Welcome to the live stream.
This is our radio announcer voice. Right. So this is something, I'll be honest, I've
done this. We now know a lot of people in the room have done this. But what we don't
want to have is with code coverage. It's supposed to tell us something about our code. But if
instead we turn that into a target, that can really limit what we actually, you know,
what the kind of useful tests that we actually create. And that leads to a few quite big questions
that we do genuinely care about. So I'll wait for that photo if you. Cool. Sorry, I'm very
audience participation. I'm very sorry. So the next question that we ask there is how
do we know if our tests are high quality? How do we know if these tests are actually
good quality tests? We test them. We test them. Great, great answer. I've got a further
follow up question for you. How can we understand what our tests are really doing? Same answer,
if anyone, I see a hand. I literally had a code base where I could delete half the tests
and nothing changed. And they all, yeah. So I'm in delete or half the kid. Hello.
Yes. So just for the live stream, I'll just repeat that because that's a really good
point. I won't repeat this wearing, but I do understand and appreciate the, you know,
the emotion behind it. If you, if you end up, you know, shipping some code that does,
does not do what it's supposed to do. You end up with users getting very angry at you.
And yeah, that's a problem, right? That's going to be an issue. And that is a way of
finding out, but I guess the real question we're asking here is how do we know if we
can trust our tests? That's really the crux of this, this problem, right? And so as it
turns out, the very, the very famous Roman poet juvenile, actually in 100 AD after he'd
had a few drinks, he was able to summarize this in such a beautiful way. And this was
something that maybe wasn't appreciated at the time because, you know, obviously he
was talking about mutation testing 2,000 years before it was relevant. But I will mention
it here. It's who watches the watchers, right? And this is the question. Who's testing our
tests? Who cares about that? Who's, how do we actually gain trustworthiness for our tests?
And I see there's, there's people in production who's having bugs. There's people who understand
here that this is a really big deal. Luckily, we have a two-word answer for you, which is
the reason we're all in this room. Mutation testing. So, spot the odd one out. You might
see here, that's, that's Henry. He's having a great time, but maybe he shouldn't be still
in a row of pigeons. But more importantly right now, I'll just explain the basic premise
and then Paco here will explain in a little more detail how it's actually kind of done.
So first of all, mutation testing, this is a really quick summary. What you do is you
introduce some faults in your code, so just a few little things that you change. And for
each of those little changes, that's a mutant version of your code. Once you've got that,
you run your test suite against those mutant versions of your code. And if they fail, awesome,
because that means that, awesome, because that means that your, your tests have actually
picked up that change. And that's a good thing, right? That's, that's good. We want those tests
to fail if our code changes, right? But if they don't fail, that's a bad time because
that means those tests didn't test that change. It didn't test for that. And so that's something
that could have made it to production. So what mutation testing kind of gives you is
a way to evaluate that test quality. But this is very abstract. So let's look at penguins.
I like penguins. So Henry here, he's a great example and he's going to, he's going to bring
all this home. So I was kind of unfamiliar to the topic. So I kind of created some analogies
with penguins that really helped me. So I'll share those with you. So the way I kind of
imagine my software is we do lots of stuff with messaging. And so I imagine software
that works properly to be like a pigeon or a dove, like a bird that can fly. I've used
a dove here because Paco has a deadly fear of pigeons. He's terrified of them.
Not fear. Vendetta.
He has a personal vendetta against pigeons. Sorry. He doesn't like them. So I've used
a dove here. But ideally we want something that I can tie a message to the bird's leg
and it can go and deal with that message for me, right? So it can go, it can go do something
like that. So one of the key features of penguins is that they're not very good at flying,
right? I think we, can we all agree that that's probably not the best. If you want to tie
a message to a bird's leg and get it to deliver it, a penguin might not be the bird you choose
unless you may be delivering something underwater. So this is the kind of example here where
we've got an a bird, but it's not the kind of thing that performs the way we expect
it to. And this would cause some serious problems if we try to use this kind of thing in production.
If we wanted to send a message via a penguin, we're going to have a tough time, right?
So Paco, I'd like you, if possible, to explain this in a way that makes more sense than what
I just did.
Good luck.
We have one mic. It's a bit, it's a bit, yeah. So let's get into the positive mutation testing.
The first step of mutation testing, so we've, what Max just taught you is about introducing
faults. So you can do, introduce faults manually, but this is a process that's, well, manually,
and that means it's a lot of work and it's usually also not that reproducible. You don't
want to do it manually. We want to do this in an automated manner. And this is where
mutation testing comes in. In the first step of mutation testing, we're going to generate
mutants. And each mutant is just a very tiny version of the production code. Mutate testing
works with the concept of mutators. And mutators are the ones that are making these very small
changes. So what we have in this case, we have a perfectly fine dove, which is the production
code. And then at the end of it, we have a mutator, which generates, makes a tiny change,
which kind of transforms this into Henry, our penguin who can't fly and we want our
software to fly. So this would be a bad thing. So how does it look? Because this is still
a bit abstract. And I'm going to give you some examples. This would be an example here.
So for the Dutch, and I think for other countries as well, you have to be 17 years or older
to apply for driving license. This could be code that's in your code base, which will
fly, which is good. Now, the mutant would be the entire code base stays the same. And
just this, this little piece change. So here we inverted the logic. This is, of course,
the bug. This is something we don't want to manage and get into production. And actually
just from this, this single line, we can already generate quite some mutants because we can
not only invert the conditional operator, we can also change the conditional boundaries.
So this means that we now have age larger than 17, which is a very nice bug that would
force us to test the edge cases, the, the famous off by one errors, whether we forgot
our equal operation in our conditional check. This, this will help you find that one. But
it can also just return always true and false. We can generate quite some mutants for this
this and we can do the same for, for example, mathematical operations. We can make each
plus into a minus each multiplication into a division, etc. And therefore, we also have
the ability to remove statements. So in this case, we have a method that adds a published
date to some object. And we can also just remove the whole setter. And now this means
that we have a bug in which we don't set this attribute anymore, which is something that,
of course, we don't want to make the production. What's important to note here is that we
mutate testing is always important that the code actually compiles because we're not testing
the compiler. We're testing the code. The compiler is definitely out of scope here. Now at the
end of step one, we have a lot of Henry's. We have a lot of mutants. And now Henry is
going to try to fly. So he already got his wings ready to try to fly. And now for each
Henry, we're going to run the test suite. And if this test suite fails, as Mike's already
mentioned, then we have, then we, then it's good because then we expose Henry 40 is, which
is just a penguin, something that can't fly. So this is great. The not so happy scenario
is where the test passed, which means that Henry made it into production. And as we know,
well, assuming that it also got through the PR, of course, we have more than just tests.
Is that a problem? Because Henry is not supposed to fly. And then we have a bug into production.
So this is something that you don't want. So this is the theory of mutation testing.
And now, Max, you can tell a bit more about the frameworks.
Sure. It works for me. Alrighty. So first of all, I just want to say I'm so proud of
this prompt. I don't know why Dali chose this, but I'm really happy. Like, I think I typed
in penguin trying to be a pigeon. And it came up with this. And I'm very happy. Okay. So
moving on, yeah, frameworks. So this is going to get a little bit more specific to, you
know, to actually implementing this stuff. So anyone here is a Python developer? Heck
yeah. All right. Awesome. So I'm going to show you what I did in Python. So as you
can see, you know, Parker's Java developer, he'll explain Java in a sec. But I'll just
show you the kind of basic concepts, but using using my code and using what I did. So there's
two kind of main supported packages that you can use in Python. It's not like, you know,
in Java, there's like an enterprise thing you can get in Python, it's very community
supported. So you're not, you know, you're not going to get big products. But what we
do have are these kind of like nice and supported repose for mutation testing, which have just
these packages. So I am not a professional, you know, in this, I'm not a doctor, I'm not
a lawyer, I'm not a professional financial advisor. I'm just a person who, you know, has
a certain opinion. And so my opinion of those two frameworks I showed you, there's Muttmutt
and Cosmic Ray. And personally, I prefer Muttmutt. Because it's easy to get going. Oh, angry,
angry face, shaking hairs. You don't like Muttmutt. We will talk later.
So if we have time, we'll have a third presenter very shortly. So for now, while I've still
got the mic, while I'm still, you know, while I'm still here, we'll talk about Muttmutt.
And so this framework is quite simple to use. You know, it's the reason I kind of like it
is because it's very much you install it and you run it. You know, there's a bit of config
you can do. But really, it's quite simple just to get an idea of your code base and
what's going on. So I want to show you this slide. This is just, this is the SDK that
I maintain. And I'm showing you this because it's what I've applied my mutation testing
to. So it's what I'm showing my examples. But basically, what we do is when we go here,
I had this locally, first of all. So I installed Muttmutt with pip install. It's that simple.
It's a Python package. It's what we do. If you went to my talk on malware earlier, you
know why that's a bad idea, but I did it. So after we do that, we've got Muttmutt run,
which just runs those tests for you. So when we do that, I'll show you what my output was.
So when I ran this myself, I actually got a whole lot of this output. But really what's
important here is that, first of all, it ran my entire test suite. And the reason it ran
my entire test suite is just to check how long that's supposed to take and just to make
sure everything does work as expected because there's various types of mutants to do with
timeouts as well that we might want to consider. After it's done that, what it will do is it
will generate mutants based on lines of code in my code base. That's what it will do. And
once it's done that, it will run my tests against those. So there's a few different types and
it can characterize them like this. So the first type is mutants that we've caught, not
killed. We never kill a penguin. We love penguins. We catch them. We've caught them and put them
back into the zoo. In this case, in this case, we've managed to say, yep, our test failed.
That's great. But it could be the case where the mutant's timed out. So it's taken way
too long for this code to run or it's taken enough time that we feel like we're not so
great feeling great about that code. Alternatively, we might end up in a situation where the mutant
survived and made it through our test code. In that case, it corresponds to a bug that
might make it to production. So when I run this on my particular SDK, what I saw was
that we checked the stuff. I created 682 mutants, versions of my code, with changes in them.
And of that, it managed to catch 512 of those, but it managed to miss 170 of them. Now, if
that's a good number or a bad number, we'll talk about later. But what's important now
is let's just look at some of those mutants. So first of all, the ones that we actually
did catch. Here's a couple of examples. So here's a line where basically we say, here
are some valid message channels. So for our messages API, here are some valid messages
ways you can send, right? But what's important here is that this basically removed the ability
to send an SMS. And so when I tried to test that, it failed, which is what we want to
see. Here's another one. Again, this is Python. So if you're a Java dev, don't worry, we'll
look after you soon. And here's another one. We've got a decorator here, which basically
runs this method. And we can see when we remove that, that will never happen. This is actually
through Pydantic, if anyone has used that before. But basically, it means that we're
not going to round a number anymore. And so when we test for that, a number doesn't get
rounded and we catch that. But that is not really very interesting. That doesn't tell
us anything. That tells us about this much, right? It doesn't tell us much at all. And
the reason for that is that we kind of know that our tests work. We kind of know that
our tests work for that. Thank you very much. I'll do the M&M thing. So we kind of know
that our tests work for that. And so what's kind of useful is to see if we do what much
show, we can see the mutants that we didn't catch. We can also do what more HTML, which
shows us essentially an HTML coverage output as well. So we can see in a list all of the
mutants that we didn't catch. So with more and more show, on that code base that I just
showed you, we can see the 170 mutants that survived. It shows you the indices of these.
And then we can manually specify the ones we want to look at. So here we can see, for
example, that we changed the authentication method to fail. And we can see in this case
we caught that because we did a test for authentication and it failed, so that's great.
But more importantly though is you get this HTML output, which you can then explore. You
can explore every method, every sort of module that you have. You can explore all the methods
inside of there and which ones were and were caught. And you do that with the HTML command.
So to do that, I'll just show you this is a mutant that we did not catch. And I want
to show you why we didn't catch it and what it's going to do. And I'll just do that for
a few just so you get some context if that's cool. So first of all, what this mutant did
was it renamed the logger. Now I think logging is out of scope of my test code, so personally
I don't care too much about anything related to logging. So I don't mind if I don't get
a pass here. Here's another one. In this case, what we do is we've slightly changed the value
of a constant. This is just part of a function signature. And again, we don't care about
this that much. It isn't something that I really mind about. What's more important though
is this mutant here. Because this is from our client class where we instantiate all
of our different API classes. And you can see we actually set voice to non, so we completely
remove that instantiation. And our tests are still passing. So the reason that actually
still works, our code base still works even though this isn't testing that case, is because
our tests actually test the voice API separately. They call it manually. But if our clients
are calling it like this, maybe we should have a test for this as well. So this tells
me, hey, maybe my test suite does need to be expanded. Does that make sense? I'm seeing
some very, very like, yeah, yeah, that makes sense. I like it. Awesome. Okay, so if you
are a Python Dev, this isn't the end of the talk, by the way, this is, you know, we've
got some more context and we'll show you about CI. But if you are interested, then, you know,
feel free to scan this. You've got like four seconds before I move slides. And as I move
slides, I'll very slow motion, I'll be passing over this microphone. Because this was just
Python, of course. And I think there are more non-Python Devs here. Just not Python.
Let's see. We, of course, have more frameworks. I think they're from more languages out there.
But I think they're the most important ones that I like personally. And pretty much the
only really good one for Java is PyTest. And we also have Striker. And Striker is one that
supports quite some languages. It supports JavaScript, C sharp, Scala. Of course, it doesn't
do this in one two. Each one has their own dependencies because you can't have one solution
for all. And what you particularly like about is that it supports JavaScript. And this brings
this kind of back end heavy tool. Testing is usually mostly, I think, in front end can
use some law when it comes to testing often. And this also brings the testing frameworks
and the testing quality more to the front end. So that's what I really, really like.
But we wanted to discuss a bit more. Mike's already sort of introduced it. So what is
a good mutation score? We had the good hearts law where we sort of saw that code coverage
can also lead to people implementing tests just to improve coverage, not just sort of
defeats the purpose. You're doing it just for the metric, not for the actual purpose.
And so how does this work with mutation score? Now, first, here's a picture of how PyTest
report looks. So not to bash on Python, but much prettier and much clearer. Because now
what, particularly what is interesting about this one, it shows you both the line coverage
and the mutation coverage. We can ignore the test strain. And this shows us the sweet spots
in a report. Because at the end, we have generated a lot of mutants. We have a lot of classes.
And we only have very little time. So where are we going to look and investigate this
report to see where the strains are? And the one that's the least interesting here is the
notification service. The notification service also doesn't have any coverage. And if there
is no coverage, then the mutants are also not interesting because you have a bigger problem
here, which is you don't have tests at all for this. Then you have a choice. You have
the proposal service and proposed service too. Now, the fact that they are named equally
is because they're from another example. But proposed service too is the one that has 100%
coverage and yet it didn't kill a single mutant. And this is the sweet spot because this means
that we have code that is well tested. Or at least there's tests that covering this
piece of code, but there is no single bug that was caught. So this deserves some attention
because it means that we didn't fully test this. So these are the hotspots where you
open a report. The ones with high line coverage and low mutation coverage, those that are
the ones you really want to go through. Those are the ones that give you the findings to
go to your team and say, hey, see, we need mutation testing because here, just these
two classes alone already, it showed me that we need to improve our quality. Now, back
to the score. So the example we had, we managed to kill 512 out of 682 mutants, which is about
a 75% score. Now, the question is, is this a good score? Yes, yes, the golden answer.
It depends. I love that answer. We already saw that 100% doesn't make sense. Things like
logging and there are more things like generated code, et cetera, things that you don't necessarily
want to test, even though there are mutants generated for it. Now, there are a couple
of things you can, of course, do. You can also, depending on the language and the framework
you use, you can tweak the mutation testing framework quite a bit. For example, the PyTest
version actually, out of the box, already ignores and doesn't mutate any logging lines.
And all the big frameworks are known to the tool. So anything that goes to SLF4J, it doesn't
mutate it. So it also doesn't appear in your report, which is quite nice. And you can easily
add things, like if you have a custom metrics facade somewhere, also typically something
you don't want to cover unit tests, you can add that as well. So the thing here is that
mutate testing is not really a score you want to achieve. It's more that the report can
be interesting to look at and gives you sort of the nice spots. And once you completely
set it up nice and you're familiar with the report, you can maybe start looking at the
score, but definitely it shouldn't become an 80% goal or something like it was of code
coverage. It just goes through the report instead. So now we've sort of discussed all
the tools you need. We have discussed the frameworks. We have discussed the technology
technology. And now it's time, of course, for you to fly. So we need to, how would you
get started on this? And the thing I think that's important here is if you want to start,
so you now think, oh, this is a great talk. I want to start with mutate testing. Depending
on the size of your project, it might be wise to just start with just a single package.
I've done this on projects that are a couple of, say, a thousand lines big. And even though
in Max's example, we had 682 mutants, this can also, depending on the kind of code you
have there, easily grow to tens of thousands of mutants, which can be quite slow. It can
also be that there's something weird in your code base that doesn't really work well with
mutate testing or something that's just extremely slow. An example that I had was that we hadn't
thought it's good to keep in mind. It's actually just to take a sidestep. The mutate test framework
also measures in the beginning for each individual test, which code it covers. So there's a nice
graph from code, production code, to the tests. This helps us optimize because if we want
to run the entire test suite, all the tests for every single mutant, it's going to take
endless. Instead, because we know the coverage, we can also see if we mutate this one line,
we know which test is covered. So we only need to execute those few tests. But what
if you have tests that actually cover half your code base? So, for example, one of the
things you can do in Java, if you're doing things with Spring, is you can actually boot
up the entire Spring application and start doing acceptance tests from your unit tests,
which is typically also quite, not necessarily the worst thing to do, but you now have a
very slow test that does cover half your code base that will be executed for each single
mutant. So these are things you want to get rid of. You want to exclude this acceptance
test because otherwise you're going to be waiting endlessly. So my point about starting
locally and starting small was start just with one package. Start with the utility package
to see if it works, see if the report works for you. And then from there, you can expand
at more packages and also you can see, oh, now it's taking 10 times as long. Why is this?
And you can find the painful packages there. So as I mentioned, you can exclude some tests
and also there are often candidates, certain pieces of code you might want to exclude.
For example, there's no use in testing generated code, but also it might be that you have certain
domain packages that contain just all your domain objects, your pojos, which is just
setters and getters, something that you also typically want to exclude in your coverage
report. You might also want to exclude this from code mutation, from mutation testing.
And now that's done. We need to, so we talked about running it on your machine. We also
can do this in the cloud, of course. Thank you. So as you can see, there's a pigeon
on the slide and Paco, as we've said, has a personal vendetta, so I've taken over the
section. So here we can see that we're going to run off our machine. So why would you want
to run off your machine rather than on your machine? Any questions? Any ideas?
What happens in the background?
Yes. So what happens in the background is what was said there. Any other reason you
might want to run non-locally? No. I got a couple. Oh, oh, hand.
CI.
CI. Yeah, you might want to end your CI system. In fact, that's what we'll be showing you.
So foreshadowing. I like it. So yeah, it takes some time. And if you're using a CI
system, you get to use those cloud resources. And also what's important is that you can,
if you've got code which is maybe dependent on different OSes, might behave differently,
you can specify different versions and platforms to run on as well.
So stop talking. I hear you cry. Well, I'm afraid this is what we're here for. But unfortunately,
I will be keeping talking. But what I will do is show it a bit of an example. So I applied
this to my code base, my own code base myself, into my CI system. So you can see here, this
is GitHub Actions. And I've got a piece of YAML, essentially. I've got this mutation
test.yaml file. And what that does is set up an action for me to use. So this is something
that I manually run. And I can do this here. So I manually run that. What it will do is
do the mutation test non-locally, and it will produce some HTML output for me to look at.
Now that seems, I'll go a little bit into what YAML does, but it seems like something
that should be able for everyone to do themselves if they want to. So GitHub Actions, the reason
I show that partly is because what we use, but also it's free for open source projects.
So, you know, it's been useful for me because I've not had to pay for it. So, you know,
just a heads up. See, I'll be showing you this with GitHub Actions really quickly. And
I'll show you the YAML, I'll show you what I did. Hopefully by the end of this, the next
couple of slides, you will see how easy it is actually to do this and why actually this
is all good and maybe you want to try this yourself when you get home.
So here's some YAML. First of all, this is our mutation test YAML. It's got one job.
It's pretty simple. All we're doing, we're running on Ubuntu. We're running one specific
Python version to do this. Depending on what your test base is. Oh, they're running a great
time in there. Oh, there is thunder. So basically, we have, yeah, we're testing on one version
for me because I just, my code doesn't vary enough between versions and OSes. So for me,
it's not relevant to do that. But if we look at this next slide, I'll actually show you
the workflow that goes through when I actually run this action. So first of all, we check
out the code. Then we set up a version of Python with it. Once we've done that, we actually
install our dependencies, including now MutMut as well as our regular dependencies. So now
we've got the new mutation testing framework installed here as well on this kind of test
runner. Then what we do is we run a mutation test. So we do that with MutMut run. But because
we're running in a CI system, we don't want insanely long logs and due to how it's outputted,
we want a no-progress flag there just to show that we're not seeing every line of output.
We just see the important parts. We also have the CI flag, which is one of my only contributions
to watch a open source, but I added that and I'm kind of proud of myself. So that basically
means that you get a good, sensible output, like return code when you run in a CI system.
Because the default for MutMut is, depending on the type of mutants that we call, it will
give you a different exit code that is non-zero. So you kind of need to consider that or to
suppress that with some scary, scary bash. That's why I did it first. That's why I wrote
the flag. So once we've done that, we save it as HTML and we upload it so that you can
access that yourself as well. So that's it. That's the whole piece of YAML. It's 35 lines
and that set up the entire mutation test for my suite. So you can see, hopefully, does
this seem kind of easy? I think it seems pretty gentle to do, at least in this sort of scope.
If you're a Java dev with a 20,000 line project, you might want to be a bit more careful, but
if you've got like a Python hobby thing, try it out, right? Try it out. What I would say,
there are some more concerns. So first of all, I chose to run this manually when I want
to run it. I chose not to run this on push or PR. I chose to run this manually. And the
reason for that is that I don't expect my code base to sufficiently change between
like small commits. And what I want to do is really not use like mutation test as a,
you know, that kind of score, that 75%. I don't want that to be a metric for me that
I've just turned into a target. I want it to stay as just a good idea, an indicator
of what my tests are doing and what I could be doing better. So for me, I don't want to
run every time, partly because it takes a blum in a long time, especially if I'm using
multiple versions, which we also have to factor in. So you might want to do that. So I didn't.
I just ran on a bun two and that was fine for me. But yeah, depends on what your code
is, you might want to run on different platforms, right? So do factor that in and that will
help you a lot if you're in a CI system. So the other question there is just should
we run on push or PR? My opinion is no. I think there'll be people in this room who disagree
with me, maybe say on a PR, you should run that or maybe there's some kind of metric
you want to associate with score that you then want to look at in some way. For me,
that's not how I use mutation testing. And I think what I want to get out of this is
we don't want, we don't want a situation where mutation testing becomes a new target. We've
got to get a certain score because then we're just kind of abstracting that problem of code
coverage targets. We're just doing that all over again, right? So we're trying to avoid
that. So the final question here is one I'll ask of Paco to explain is Paco, do you think
I should use mutation testing as, you know, in my role as an audience member right now?
What do you reckon? Yes. Well, so with stair already, it depends. There's some things you
can ask yourself because needed is a question. So mutation testing is of course definitely
not a silver bullet. It's something that the reports take quite some time to go through
and of course it's quite computationally expensive to run the process. So the couple of questions
that you can ask yourself that are quite obvious are for projects which have a really high
quality goal. When people die or when a lot of money is lost or a combination of those
two. So just to check, how many of you are working on a project that fits in these three?
Okay, then you need this yesterday. Yes. But for the rest of the room, including me, there
are some other questions we can ask yourself. And I think one of the important ones is are
you using code coverage? Because if you're not using code coverage, let's start with
that and let's first get coverage and get to see how many tests you have. Then the next
question is, is how much value do you put into this? How much value do you get out of
this code coverage? And what I mean with that is, do you make decisions based on it? It's
like a definition of done on your sprint or is it with a build fail if there's 80% coverage
or also in the case of due diligence, you're selling a company, not something we also would
do. But then you would also want to know how well is the software I'm buying or how well
is the software I'm working on. So here I would say, if you're using code coverage
and you're making decisions based on that code coverage, then yes, you should at least
have a look at mutations testing to see what the state is. You don't have to do this always.
You don't have to put it in CI just once a year or go home, run it on your computer
once just to see what the current state of your team is. Because it can very well be
that you're on a high-performing team, which already has their PRs and everything so well
and set up that it's not worth the time maybe. Because apparently the mutation testing report
might even confirm that, the fact that you killed all the mutants. So that would be great.
And there's another question that I like. What's the cost of fixing a bug? And I have
two stories for this. My first example is, and that's the first company I worked for.
This was an enterprise company that built software that was running on-premise at the
customer and the customer was government. And then you're in the line with all these
big integrators, which means you have feature freezes and moments where you can actually
go to the customer and deploy your software, which is quite expensive, which also means
that if you get a bug after this feature freeze or after this upgrade window, you have a serious
issue because you need to go to the customer, you need to explain what went wrong. It's
a very costly thing, a very costly issue. So here definitely, again, mutated testing
can be quite interesting because a lot of money can be involved with the reputation.
The other example that I had was more of a Greenfield project, which had more of the
startup vibes where it was really of a feel fast and fix fast mentality. So this was a
project where rather than focusing on getting our quality monitoring up to speed, we were
mostly focusing on making sure that we could very quickly fix bugs as well. It was of course
running on-premises in the cloud, so we could control it. And most important goal was there
to just click a button and be in production again in 10 minutes and have active monitoring
to see if anything goes wrong. Here the cost of fixing a bug is already a lot lower, which
means that the reason to consider it might be a bit less, especially if you're again
in a, for example, a high-performing team, which are all worked into each other, you
know what you're doing, and you know you can trust each other because you're really,
you're all professionals. Then maybe it's not worth to also spend half a day going through
mutated testing report if you already know what the outcome is probably going to be.
Then again, still do it once. These are two things you could consider and want to use it.
So those are the things I want to give away with you is don't go into it blindly, just
ask yourself, should I really use it? And then, yeah, for the last part, I'd just like
to sum up. So I think hopefully if we've gotten here, we've kind of shown you what
mutation testing is, why you might want to consider using it, and how you could possibly
get going starting with running that, and also why you should. So if we're here, I just
want to summarize, first of all, I'm sorry I used this penguin as an evil penguin earlier.
It is adorable. I just like the Dali. When I asked it to give it some fake wings, it
gave it three. It gave it this extra flipper here. I'm not sure what that was for. But
what I'd like to do is just quickly summarize what we talked about today. First of all,
mutation testing is a way to test your tests. It helps you to beat the problem where you're
using Goodhart's law for coverage, right? So it saves you from just trying to turn
coverage into a metric that you don't have as a target, right? You don't want to have
code coverage. It's got to be above this threshold or we don't merge. That's not where we want
to be. What we want to do is write good tests. So if you are going to do this yourself, an
important part is to start small. So start locally on your machine. If you've got a big
code base, then what you need to do really is run on a subset of that code base. If you've
got a smaller code base like me, you're probably okay. But either way, start locally on your
machine. You also want to run if you can. If you want asynchronous reports, if you want
to use the resources available on a CI system, you can run mutation testing there. So do
consider that if your stuff is in CI. And finally, I just want to say that mutants, hopefully
we've demonstrated that mutants are like adorable penguins, right? They're valuable and they
are wonderful, right? They're really great to use. They can tell you so much about your
code. They're extremely useful. So don't fear them because you should love them. Thank you
very much.
If there are any questions, comments, objections, love mail, hate mail, anything, shout at me.
So the question there was just if we can give some more examples of the kind of range of
things that are possible to mutate. So essentially, the short answer is anything that will still
make the code run. So in the Java case, the code compiled, in my case, the code run. So
in this situation, things like, I'll give you some Python examples. For example, changing
a variable from a certain type to another. So you might typecast something. You might,
with a mathematical expression, you might add extra terms to that expression. You might
change return types, error types. You might set things to non at any given time. You might
call something and, yeah, remove parts of it, set things to zero. There's other stuff.
Paco, can you think of any mutation testing Java examples?
Yeah. So I think that the examples you gave for that, it sort of depends on the mutators
you use. So you often, you can also, of each framework, you can also go through the list
of mutators to see what kind of mutators are out there. What's good to keep in mind is that
it does use some basic, fundamental strategies to determine if it can be mutated. Because
for example, if you have a stream, and in this stream, you do some operations, which
you could, in theory, cut out, you're still using the return value, which means that the
mutation testing framework thinks, okay, let's keep that intact. The same goes for if you're
using the spring reactor framework. You could do lots and lots of smart mutations in there,
but it's not really there yet. It's really the rudimental things, the conditional logic,
the mathematical logic, I think, are the two main things you'll see. And actually also,
the count for often the most typical programming errors, I would say, awesome. I mean, anything
you'd like to mutate, you know, because I guess a lot of these things are open source,
you know, anything that you might be good if it did exist. Any other question, answer?
So, two questions. The first one, could you comment on some framework for C and C++? And
the second, what do you think about the idea to force developers or to require developers
to use that code, which they have actually changed, just to save computational power
on the entire machine and the server side is yet.
Okay. So, the question there just for the live stream was two things. One is, are there
any mutation testing frameworks for C or C++? I will say personally, I don't know. You haven't
used C++ since my physics degree, so I couldn't tell you. I don't know if you know anything
about that, Parker.
I just did a quick Google search. That's all. So, I see there are some frames available
for you.
There is a project by the University of Luxembourg, which is called FAQAS.
FAQAS.
And it's not there quite yet, but it's something for C and also for a bit more for C++. Regarding
your other question, by the way, so should you do it as a get hook? I'm given that it's,
right, that was the question.
Yeah, the idea was basically to require developers to run those mutation tests, but not the full
set, but only mutation tests which are touching the machine, which are testing the unit tests,
which are testing the code, which was modified in this one.
Yeah. Yeah. So, actually there are some, depending on the framework, some have features which
are incremental reports, so where they can just store the last state, then you can do
a diff and use the results from your last execution to not execute all mutants and not
generate all mutants, because it knows I only changed these production lines, so I only
need to generate mutants for these, and I only changed these tests, so I only need to rerun
the tests for these mutants, which can tremendously speed it up. But still using it as a get
hook, I'm not sure. You can, by the way, use the same logic in CI as well to use the incremental
reporting that saves a bit, despite and also supports a thing.
Yeah. So, with what you have, you have caching, so you can cache those tests that you've done
already, and if those cases aren't touched, then you're sort of good if that changes to
your code don't affect that. So, that is an option. I would say, yeah, thank you. My
opinion is, again, that maybe you don't want to explicitly mandate this on every run, and
the reason for that is it's kind of like, it can then become kind of a metric that you
try and optimize for or something to look at, whereas really, I think the nice way to
use it is every now and then is how I would say so. I think if you've got a super critical
project where that's really important, you may want to run it like that. For me, I don't
need to, but I think that's really up to you, up to you as an implementer, what you want
to do, and I think there's definitely a use case to do it in that way if that was important
to you.
Hand over here, hello. Yes, yes. Short answer is yes. Long answer is, depending on the actual
framework, it might be that you add a comment to ignore it. Alternatively, there is a config
file set up as well in Python where you can say, only mutate these paths, only do these
things. What language do you use? That would be Striker. I would say yes. I haven't looked
that much into Striker, but I think they make quite some nice stuff. It's quite generic
for all frameworks. Exclude code for mutation, definitely yes. Depending on the framework,
some even have nice things like exclude, do not mutate any calls to these classes, which
is interesting for the logging, for example. Do not mutate any calls to this logging class,
but same you can do for packages, class paths, et cetera.
I'd say with Striker as well. One of my colleagues uses Striker because he maintains our.NET SDK,
and he's actually also got mutation testing there in Striker. It does seem very perform
and it seems like it does have a lot of those features as well. Honestly, if you're interested
in TypeScript, I think there is something for you there.
Cool.
I think it might be free on open source repos. Sorry, another question.
Yeah. Are specific mutated runs that are reproducible for debugging purposes? If you have a run
and you see something you don't expect, can you reproduce that specific run with a given
SC or something?
Well, so the question is, how reproducible are the mutants? If you find one, then next
run, is it still there? As far as I know, there shouldn't be any randomness in these
mutant generations. It just goes over the code. Any condition that it finds that it
can mutate, it will mutate. So the next time you run it, the same mutant should be there
at the same place. So you could also see whether you killed it the next time. So yes, it's
reproducible.
I think that's the person who was first, sorry.
That's a good question. I'll repeat that one. That's a good one. So the question there
was, so mutation testing, we've talked all the big game, we've come up here and been
like, hey, look, this is important, right? That's what we've talked about. And the question,
which is a very valid question is, hey, if it's so important, why is no one supporting
this in Python? Why is this all open source stuff, right? And you know what? I agree.
That's a really good question. It's one I asked as well, to be honest. So no, I totally
support the question. And the question I'll properly say is, yeah, why aren't employers
supporting this? The short answer I think is to do with ROI, unfortunately. And that
sucks, honestly, because I would like us to invest more time in certain things. And I
think it's just to do with company priorities, right? So I would like to spend more time.
Honestly, I had quite a lot of fun adding the one feature I did get to add. I'd quite
like to do some more. But again, I've got this API to implement, so do I have time? Well,
no one's funding me to do it. So unfortunately, it really is like, unless there's an obvious
ROI, this just seems to be the way things go. Unfortunately, that's the way we've kind
of structured our platforms and so on. So I gave a talk earlier on a PyPy and malware.
And there was actually, the reason that that kind of is so prevalent and so possible on
PyPy is because PyPy haven't really implemented many ways to actually protect against malware
being uploaded. So currently, I've uploaded some malware to PyPy that you can get yourself.
And actually, the reason that they, it's not real malware to be clear, it's a rick roll.
But you saw that. But basically, what I'm trying to say here is that that project kind
of didn't really get off the ground in terms of protecting users just because I think originally
Facebook were funding it and they stopped funding and that just didn't then continue.
So unfortunately, yeah, this is just kind of the way that things are in open source right
now. And yeah, I do feel your pain. I do understand. But that's all I can really say. I'm afraid.
Yeah, to quickly add to this, by the way, a Striker, for example, is actually funded,
is backed by a company who, for example, let's work, interns work on it as well. So some
frameworks actually are backed and there are people already investing it. So it's not always
bad. But sorry, Nick, let's go to that side.
So you showed some HTML reports for the results of the mutual test.
Yes.
We all know all managers and PDT teams love your KPIs. So I'm wondering, is there any
integration or plugins to export the mutual test results in Sonar Cloud or other platforms?
That's a really good question. So I'll answer quickly for Python and then I'll pass it over.
Because in Python, the answer is quite short. The answer is unfortunately no. So the maintainers,
the maintainer movement is not really a big fan of the CI system stuff and the report stuff.
I think it's like, I think the premise there is, you know, I like running this locally
and, you know, that's fair. And that is really how you can get started and get an idea. So
in Python, unfortunately, the answer is no. But I think that Paco might have a more positive
answer for you.
Yeah. So let's also ask the, you were the maintainer of the other framework. So how
does it go for the other part of the framework?
So, okay. So I talked about not having that facility, that feature in Cosmic Ray. Is that
a bit un-maintained? I don't want to say names, but it is a very, very large, 450 maybe vendor
that uses it. And we asked them, can you fund development? They said, you know, no. And
yeah.
So they have shown this around at large events, like in front of thousands, thousands of people.
But yeah. They're like, okay, we keep all the data stores there for whatever we find
as it is.
Yeah. So for the Python frameworks.
Yeah.
So for the Python frameworks, there's not really CI plug-in support. I do know that, for
example, for PyTest, there is support for Jenkins and Sonar. And I'm not sure about Stryker,
but I know it's there. And usually these things are relatively easy to build yourself here
as well, because all you have to do is, if there is a report in some JSON file, you can
quite easily parse it and make a nice HTML form about this. Because again, they're all
open for contributions. Do we have time for one last?
I want to just add to that a little bit.
Okay.
Okay. Really quickly. First of all, with your question, yeah, I, when I originally implemented
my mutt mutt thing, I did do it on PR. And in that case, I got a, you know, an action
that would comment my coverage in a nice metricy way. And so you can, it's quite simple to
do.
So about, about cosmic, very first of all, that sucks. And I'm sorry. That's, that's
blumming awful. Like, yeah, sadly, it does seem that a lot of, a lot of what we've kind
of been discussing on the side of the room is just like, man, it would be good if some,
you know, we all agree this is important, right? And it's useful for a lot of things.
It'd be great if someone funded it. So I think, unfortunately with Python, that is the state
of play. And it does suck. But yes, I, I get you. Any other questions? Finally, I think
one, yes, hello.
Can you write custom mutation to mutate your code in a custom logic?
That's a really good question. So, sorry. That's, I will now repeat your really good
question. The question was, so the question was, if I, if I have a certain type of mutant
that I want to make, can I do that? So I would say with the stuff that I used in Python,
the answer is you, you'd need to actually, you know, use the version you've downloaded,
edit it yourself and add that stuff. So sadly, there's not an easy customizable way. That
will be an awesome enhancement, though, that I would, I would like to see, you know, that
would be cool. In other platforms, Paco, any other?
I do know that I think Python did have some extension points. So it really depends. I
know that the company I work for currently called Picnic, they're also working on extending
it, for example, for reactive code. So there are some extension points often. So in short,
it depends on the framework and how easy it is.
Are we, we're done. Okay. We're at time. Thank you so much. This has been a really nice discussion
as well. So thank you for sharing this.
Thank you.