Hello, everyone. Do you hear me well? Thanks, pretty large audience. If I may ask a quick
show of hands, who among you have some experience, just any level of experience with machine
learning? Okay, cool. Awesome. So, I'll be talking today about how to run testing on
machine learning systems. So, there are different keywords, CICD, quality assurance. A few words
about us. So, I'm one of the founders of Giscard. We are building a collaborative and open
source software platform to precisely ensure the quality of AI models. And I'll be explaining
in this presentation a bit how it works. In terms of agenda, I prepared kind of two sections
on the why, like why a project on testing machine learning systems is needed, why we
personally, I personally decided to work on that problem. Some of the risks and why classical
software testing methods don't quite work on AI. And then I'll do some more concrete
examples on two important quality criteria that you might want to test for machine learning.
One is robustness and the other is furnace. And if we have the time, it's just 30 minutes.
I hope that we can do a quick demo of an example use case where we run the full CICD pipeline
on a machine learning model. So, to kind of start off easy, I put together a series of
memes to explain my personal story of why I came to create a company around, and a project
around this machine learning testing thing. So, about 10 years ago, I started in machine
learning, statistics, data science, and you know, you had this, you start using the scikit
learn API, and you're like, yeah, it's super easy, right? Anybody can be a data scientist,
you just dot fit, dot predict, and that's it. You're a data scientist. And probably if
you're here today, you're like, yeah, have you tested your model? Yeah, sure. Train test,
yeah. Reality, if you've deployed in production, is quite different. So, if you've deployed
through production, often you'll have this painful discovery where you have your product
manager, business stakeholders to whom you said, look, I worked really hard on the fine
tuning and the grid search to get to 85% accuracy, and you push your first version to production,
and things don't quite work out. You don't reproduce these good accuracy numbers.
So, well, this was me. I hope it's not you. It was one of these, my first experience deploying
machine learning through production was on a fraud detection system. So, frauds are notoriously
difficult as a use case for machine learning because what you're trying to detect doesn't
quite want to be detected. There are people behind it who have a vested interest not to
have your machine learning system detect them. So, often in terms of performance, that's
at least what I ended up doing a lot of hot fixes in production. It's bad. So, kind of
like five years ago, this was my stance on machine learning in production, a very painful
grueling experience where you never know when you're going to be a complain, where you're
going to be on call to solve something in production. So, that's when I decided to buff
up and switch roles to join a software engineering team. I was a data crew back then, so I moved
internally from data science to the product team, and here are some of the things to summarize
that as someone with a machine learning background, but no real software engineering experience
that these were kind of like what I was told, like you must learn the ways of a CI CD, otherwise
your project will not come to production. And for context, so I was specifically at
that time in charge of creating like an open source layer to federate all the NLP and computer
vision APIs that vendors in the cloud provide, and then to do the same for pre-trained NLP
and time series models. So, what was difficult in this context is I was not even the one
in charge of the models, and the models will be retrained fine-tuned, so the guarantees
into the properties of that system as an engineer, that's more difficult. There are some elements
in the stack that you don't have control of. So, yeah, this is a bit of a repeat of a previous
meme, and I really wanted to say one does not simply ship an ML product without tests.
The challenge I had then is that from an engineering management standpoint, I was told, yeah, but
you know, it's easy, no engineers, they all write their test cases, so you do machine
learning, just write them all, just write all the test cases. So, this was me being
kind of a square one. It's like, okay, so you're telling me, I just need to write unit
tests, okay, that will not really solve the issue, and that's kind of the beginning of
a quest that set me on to build something to solve that gap between, okay, I want to
test my models, I need to test my models, and how do, how can I do that? Because clearly,
and I'll explain why, unit testing, your model is really not enough. So, a different angle
on the Y, I'll try to take a step back and talk about quality in general. I think in
this track, we all agree that quality matters, and if you look at AI, it's an industry that's
an engineering practice that is far younger than software engineering or civil engineering,
and it's just riddled with incidents. I encourage you, if you don't know that resource already,
it's an open source database, it's incidentdatabase.ai, and it's a public collection of reports,
mostly in the media, of AI models that have not worked properly, and it's a really great
work that has been going on for about two years and a half. It's a global initiative,
and just in this time, they collected more than 2,000 incidents. Since these are public
reports, think of it as the tip of the iceberg, of course. There are a lot of incidents internal
to companies that are not necessarily spoken out in the media. The incident database has
a very interesting taxonomy of the different types of incidents. It's very multifaceted.
I took the liberty to simplify it in three big categories of incidents. One is FX, the
other is business economic impact, and the third one is on security. We're talking about
really, if they happen at a global company scale, incidents that are very, very severe.
In FX, you can have a sexist credit scoring algorithm that exposes the company to lawsuits,
to brand image damages, et cetera. These are notoriously hard to identify. In a way, machine
learning is precisely about discrimination. It's hard to tell a machine that is learning
to discriminate, not to discriminate against certain sensitive groups. I'll speak on some
methods that can be used precisely on this problem, but Apple was working with at the
time Goldman Sachs on deploying this algorithm and probably some tests and safeguards were
unfortunately skipped. It was actually discovered on Twitter that in a simple case, a male loan
applicant would get 10x their loan limit compared to his wife. That sparked a huge controversy
that probably exposed Apple to some lawsuits. In another area, that is not with sensitive
features such as gender. There was a huge catastrophe a year and a half ago that happened
to Zillow, a real estate company, where there was a small bias that was overestimating the
prices of homes. They decided to put this algorithm live to buy and sell houses. It
turned out that this tiny bias, which was left unchecked, was exploited by the real
estate agents in the US. Literally, this created a loss of nearly half a billion dollars. Again,
maybe if going back to testing, this could have been unseapated and avoided. Now on a
more cybersecurity spectrum, there's a lot of good research from cybersecurity labs showing
that you can hack, for example, a computer vision system in an autonomous driving context.
Here you put a special tape on the road and you can crash a Tesla. We don't quite know
if these types of vulnerabilities have been exploited in real life yet, but as AI becomes
super ubiquitous, and obviously there are some bad actors out there that might want
to hack these systems and introduces a new type of attack vectors. That's also something
we need to care about. Both from the practitioners of AI and from a regulatory standpoint, testing
just makes sense. Yanlequin, chief AI scientist at META, was actually taking a stance at the
beginning of last year on Twitter saying that if you won't trust in a system, you need tests.
Also making a slight criticism towards some of the explainability methods, because two
years ago, if you've followed that realm, people were saying, oh, you just need explainability
and then your problems will go away. Well, that's just part of the answer. Lastly, and
this was covered in some of the talks this morning on the big auditorium, there's a
growing regulatory requirement to put some checks and balances in place. That also says
that you need specifically in case your AI system is high risk, you need to put quality
measures in place. The definition of high risk AI systems is pretty large. Obviously,
you have anything related to infrastructure, like critical infrastructure, defense, et cetera,
but you also have all AI systems that are involved in human resources and public service
and financial services, because these are considered, obviously, critical components
of society. Now that we kind of agreed that it's an important problem, these are some
of the challenges, because if you've encountered some of these issues, you probably looked
at some easy solutions, taking some analogies on what you might do to do this. There are
three points that make this problem of testing machine learning a bit special, meaning it's
still a big work in progress. Point one is that it is really not enough to check the
data quality to guarantee the quality of a machine learning system. One of my co-founders
doing his PhD proved experimentally, you can run experiments, you can have really clean
data in a bad model. So you cannot just say it's an upstream problem, it's technically
like systems engineering, you have to take the data, the machine learning model, and
the user into context to analyze its properties. Moreover, the errors of the machine learning
systems are often caused by pieces of data that did not exist when the model was created,
they were clean, but they did not exist. Second point, it's pretty hard just to copy-paste
some of the testing methods from software into AI. One is like, yes, you can do some
unit tests on machine learning models, but they won't prove much. Because the principle
is that it's a transactional system and things are moving quite a lot. So that's a good baseline.
If you have a machine learning system and you have some unit tests, that's really like step
one. It's better to have that than to have nothing. But you have to embrace the fact
that there has got to be a large number of test cases. So you cannot just test on three,
five, hundred, even a thousand cases will not be enough. The models themselves are probabilistic.
So you have to take into account statistical methods of tests. And lastly, and I think
this is specific to, because there has been some systems that were heavily dependent on
data, but with AI, AI also came with a fact that you increase the number of data inputs
compared to traditional systems. So you very quickly come into issues of, well, it's a
combinatorial problem, and it's factually impossible to generate all the combinations.
Very simple example of that. How can you test an NLP system? Lastly, like AI touching a lot
of different points. If you want to have a complete test coverage, you really need to
take into account multiple criteria. So performance of a system, but also robustness, robustness
to errors, fairness, privacy, security, reliability. And also, and that's becoming an increasingly
important topic with green AI, it's like what is the carbon impact of this AI? Do you really
need that many GPUs? Can you make your system a bit more energy efficient? So today I'll
focus on the, because I see we have 10 more minutes, I'll focus on two aspects, the robustness
and the effects. So I'll start with robustness. Who has read or heard about this paper? Quick
show of hands. Okay, one. So who has heard of behavioral testing? Because that's not
machine learning specific. Yeah, cool. So Ribeiro three years ago, along with other co-writers
of this paper, did I think a fantastic job to see how to adapt behavioral testing, which
is a really good practice from software engineering, to the context of machine learning. And specifically
wrote something for NLP models. The main problem that this research paper aimed to solve was
test case generation. Because really NLP is by a sense a problem, NLP, a natural language
processing. So you have an input text, it's just raw text. So you need to test this. But
what you can do is to generate test cases that rely on mapping the input and the input
changing changes in the text to expectations. I'll give three examples from very, very simple
to a bit more complex. One is like the principle of minimum functionality. For example, if
you are building a sentiment prediction machine learning system, you could just have a test
that says if you have extraordinary in the sentence, you should always predict that the
model will say it's a positive message. Now you will probably tell me, yeah, but what
about if the user has written it's not extraordinary or absolutely not extraordinary? And that
actually brings me to the concept of test template. And the fact that probably for NLP,
what you need to do, and this is obviously language specific, is start to have templates
where you change the text by, for example, adding negations. And then so you might want
to test if your system, if you're adding negation, if you have a certain direction. Because normally
if the machine learning model has understood, it should, if it's about sentiment, understand
that putting not an extraordinary or not good, you have then synonyms, will not affect the
system too much. Or actually either your system, you want it to move to a certain direction
or there are cases where you want actually the opposite behavior. You want robustness.
So that's called invariance. So for instance, you will want a system that is robust to typos
to just changing like a location name, just putting synonyms, et cetera, et cetera.
So we've created this diagram to explain it. And it's a really thriving field in the research.
There is a lot of research going on these days about testing machine learning systems.
And metamorphic testing is one of the leading methods to do that. The principle is, if I
take an analogy, is very similar to if you've worked in finance or if you have some friends
who work there, the principle of backtesting an investment strategy. You simulate different
changes in the market conditions and you see how your strategy, your algorithm behaves,
what is the variance of that strategy. This concept applies very well to machine learning.
So you need two things. You need one to define a perturbation. So what I was explaining earlier
in NLP, perturbation might be adding typos, adding negation. In another context, like
let's say it's more in an industrial case, it might be about doubling the values of some
sensors or adding noise to an image. And then, pretty simply, you define a test expectation
in terms of the metamorphic relation between the output of a machine learning model and
the distribution of the output after perturbation. And once you have that, and if you have enough
data, then you can actually have, like you can do actual statistical tests, see there's
a difference in distribution, et cetera. So I won't have too much time to dive into all
the details of this, but we have wrote a technical guide on this topic and you have a link in
QR code up there. Next, I'll talk a bit about a really tricky topic, which is AI fairness.
And I want to emphasize that it's, at least our recommendation, is not to come at the
problem of AI ethics with a closed mind or a top-down definition of this is an ethical
system or no, this is an unethical system. My co-founder did his PhD on precisely on
this topic and wrote a full paper on this, looking at the philosophical and sociological
implications of this. And the gist of it is that, yes, to a certain extent, you can adopt
a top-down approach to AI fairness, saying, well, for instance, as an organization, we
want to test the fairness on explicitly free, sensitive categories. You can say, well, we
want to check for gender balance. We want to check for race balance. That means if the
country where you deploy a machine learning allows to collect this data, this is not always
the case. But the challenge with these approaches is that, A, you might not have the data to
measure this, and B, you may miss out because often when this exercise of defining the quality
criteria for fairness and for balance are done, you only have a limited sample. So it's,
in taking some sociological analysis, it's really important to have this kind of top-down
definition of AI ethics, meet the reality on the ground, and confront the actual users
and the makers of the systems to get them to define the definition of ethics, rather
than a big organization, if I put a bit of a caricature that says, AI ethics, yeah, we
wrote a charter about this. You follow, you read this, you sign, and then, oops, you're
ethical. Having said that, so there are some good top-down metrics to adopt that are kind
of a baseline, and I'll explain one of them, which is disparate impact. Disparate impact
is actually a metric from the human resources management industry from at least 40 years
ago, so it's not new. That says, so it's probabilities, but essentially it's about setting a rule
of 80%, where you measure the probability of, you define a positive outcome with respect
to a given protected population, and you say, well, I want to the proportion of the probability
of a positive outcome relative to the probability of a positive outcome in the unprotected context
to be above 80%. So, for instance, so if you want to apply that to a, oops, to put more
concrete, yeah, so if, say, you're building a model to predict the churn of customers,
and you want to check whether your model is biased or not for each class, this formula
allows you to really define this metric and write a concrete test case. Right, so I just
have three minutes, so I'll highlight what one of the features of our project enables
is putting human feedback, so really having an interface where users and not only data
scientists can change the parameters, so there's a link to metamorphic testing, and actually
give human feedback to a point art where the biases may be, and the benefit of this approach
is that it allows for the community to precisely define what they think are the risks. So sadly,
we won't have time to do a demo, but this phase, in our project, we call that the inspection
phase, and it's about before you test, and this is super important, and again, one of
the things where it's different from traditional software testing, before you even test, you
need to confront yourself with the data and the model, so that's where actually we think
explainability methods really shine, it's because they allow to debug and to identify
the zones of risks, and this is precisely what helps once you have qualified feedback
to know where you should put your effort in test, so in a nutshell what I'm saying for
testing machine learning systems is it's not a matter of creating hundreds of tests, of
automating everything, but rather to have a good idea of, from a fairness standpoint
and for a performance standpoint, of what are the 10, 15, maybe max 20 tests that you
want in your platform. If you want to get started actually on it, this is our GitHub,
and if you have a machine learning system to test, we're interested in your feedback.