Hello, everyone. I'm Marco. Thank you for being here to listen to my talk. I'm an engineer
manager at Mozilla. I've been at Mozilla for almost 10 years now. I started as a contributor,
then an intern, then I was hired, and I've been here for almost 10 years. I started working
on some funny projects, like writing a Java VM in JavaScript, and then more recently I
started focusing on using machine learning and data mining techniques to improve developer
efficiency, which has also been the subject of my PhD. During this talk, I will show you
how we will all be out of a job in a few years, joking. I will just thank you through our journey
of how we incrementally built features based on machine learning for improving software
engineering, one on top of each other. I'm the father of two, Luna and Nika. Before we start
with the presentation, I wanted to explain a little why we need to do all these complex
machine learning things on top of bugs, CI, patches, et cetera, et cetera. Firefox is a very
complex software. It's a browser. We have hundreds of bug reports and feature requests
open per day. We have 108 million bug reports at this time, which is almost the price of
one bedroom apartment in London. We release every four weeks with thousands of changes,
and during 2022 we had 13 major releases and 45 million minor releases. As you can see,
we even sometimes party when we reach a certain number of bugs. As I said, Firefox is one
of the biggest software in the world. We have a lot of legacy. Netscape was open sourced
25 years ago. A few days ago we celebrated the 25 birthday. Over time we had 800,000
commits made by 9,000 unique contributors representing 25 million lines of code. We had 37,000 commits
only last year by 1,000 unique contributors. Not all of them are paid. Many of them are
volunteers. And this is a list of the languages that we use. As you can see, we use many of
them. We have C++ and Rust for low-level things. Rust is gaining ground and is probably going
to overcome C soon. We use JavaScript for front-end and for tests. And we use Python for CI and
build system. But we have many more. So if anybody is interested in contributing, you
have many options to choose from. But let's see. So as I said, the complexity is really
large. We have thousands and thousands of bugs. And we need some way to control the
quality, to increase the visibility into the quality of the software. And we cannot do
that if the bugs are left uncontrolled. One of the first problems that we had was that
there is no way to differentiate between defects and feature requests. We call them bugs on
bugzilla. But they are actually just reports. Many of them are defects. Many of them are
actually just feature requests. And so at the time, we had no way to measure quality.
We had no way to tell in this release we have 100 bugs. In this other release, we had 50.
So this release is better than the previous. And so we need a way to make this differentiation
in order to measure the quality. And it was also hard to improve workflows if we had no
way to differentiate between them. So we thought of introducing a new type field. This might
seem simple. It's just choice between defect, enhancement and task. But in practice, when
you have 9,000 unique contributors, some of them not paid. It's not easy to enforce a
change like this. And we also had another problem. We have 100 million bugs. If we just
introduce this type, it's not going to help us at all until we reach a mass of bugs that
we change. So if we just introduce it at this time, it will only start to be useful six
months from now. So we thought, how do we set the field for existing bugs so that this
actually becomes useful from day one? And we thought of using machine learning. So we
collected a dataset. I'm not sure it can be considered large nowadays. With 2,000 manually
labelled bugs, few of us labelled independently. And then we shared the labelling so that we
were consistent. And we had 9,000 labelled with some heuristics based on fields that
were already present in bugzilla. Then we, using the fields from bugzilla and the title
and comment through an NLP pipeline, we trained an XGB boost model. And we achieved accuracy
that we deemed good enough to be used on production. And this is how the bug, bug project started.
It was just a way to differentiate between defects and non-defects on bugzilla. We saw
it worked and then we decided, we thought, what if we extend this to something else?
What is the next big problem that we have on bugzilla? And it was assigning components.
Again, we have lots of bugs, millions of, hundreds of thousands of bugs. We need a way
to split them in groups so that the right team sees them, so that the right people see them.
And the faster we do it, the faster we can fix them. At the time, it was manually done
by volunteers and developers. So you can see a screenshot here, product and component,
PDF viewer. In this case, we didn't need to manually create a data set because all of
the 1 million bugs were already manually split into groups by volunteers and developers
in the past. So we had, in this case, a very large data set, two decades worth of bugs.
The problem here was that we had to roll back the bug to the initial state because otherwise,
by training the model on the final state of the bug, we would have used future data to
predict the past. And it was not possible, of course. So we rolled back the history of
the bug to the beginning. We also reduced the number of components because, again, with
the Firefox scale, we have hundreds of thousands of components. Many of them are no longer
actually maintained and no longer relevant. So we reduced them to a smaller subset. And
again, we had the same kind of architecture to train the model. With a small tweak, we
didn't have perfect accuracy. And so we needed a way to choose confidence and recall. So
pay the price of lower quality but catching more bugs or catching fewer bugs but be precise
more time. So we can control this easily with a confidence level that is output by the model,
which allows us to sometimes be more aggressive, sometimes be less aggressive. But at least
we can have a minimum level of quality that we enforce. The average time to assign a bug
then went from one week to a few seconds. Over time, we auto-classified 20,000 bugs.
And since it worked, we also extended it to webcompad.com, which is yet another bug reporting
system that we have at Mozilla, which if you find web compatibility bugs, please go there
and file them because it's pretty important. And you can see here an action of the bot
moving the bug to, again, the Firefox PDF viewer component. Maybe I should have used
another example just for fun. Now we had something working. And it was starting to become promising.
But we needed to make it better. We needed to have a better architecture for the machine
learning side of things. We needed to retrain the models. We needed to collect new data.
We needed to make sure that whenever a new component comes in, we retrain the model with
the new components. If a component stops being used, we need to remove it from the dataset
and things like that. So we built, over time, a very complex architecture. I won't go into
too many details because it will take too long. But maybe if somebody has questions later,
we can go into that. And then with the architecture now, it was easier to build new models. So
we even had contributors building models just all by themselves. In particular, there was
a contributor, Ayush, which helped us build a model to root out spam from bugzilla. So
it seems weird, but yes, we do have spam on bugzilla as well. People are trying to get
links to their websites into bugzilla because they think the search engine will index them.
It's not actually the case. We tell them all the time, but they keep doing it anyway. We
have university students. Bugzilla is probably the most studied bug tracking system in research.
And we have many university students from many countries that use bugzilla as a playing field.
Many times we even contact the universities and professors asking them if we can help
them give more relevant topics to students, et cetera, but they keep filing bugs. And
this contributor maybe was from one of these schools, was tired of it and helped us build
a model. And the results were pretty good. I'll show you a few examples of bugs that
were caught by the model. So this one was, if you look just at the first comment of the
bug, it looks like a legit bug. But then the person created a second comment with a link
to the website. And it was pretty clear that it was spam. This one is another example. This
is actually a legit bug. It's not spam. Maybe it's not so usable as a bug report, but it
was not spam. And then somebody else, a spammer, took exactly the same contents, created a
new bug injecting the link to their website in the bug report. And somehow, I don't know
how the model was able to detect that it was spam. It's funny because you can see that,
so when you file a bug on bugzilla, bugzilla will automatically insert a user agent so
that we have more information as possible to fix bugs. But in this case, he was filing
the bug, copying the contents of the other bug, so we have two user agents. And they're
even on different platforms, one on Mac and one on Chrome, actually. Okay. So we were
done with bugs. We are not done with the bugs. We will have plenty of things to do in the
future forever. But we were happy enough with bugs and we thought, what can we improve next?
One of the topics that we were focusing on at the time was testing and cost associated
to testing. We were experimenting with code coverage, trying to collect coverage to select
relevant tests to run on a given patch. But it was pretty complex for various reasons.
So we thought maybe we can apply machine learning here as well. But before we go into that,
let me explain a bit about RCI because it's a little complex. So we have three branches,
three repositories, which all kind of share the same code, Firefox. We have Try, which
is on demand CI. We have AutoLand, which is the repository where patches land after they've
been reviewed and approved. And we have Mozilla Central, which is the actual repository where
Firefox source code lives and from which we build Firefox nightly. On Try, we run whatever
the user wants. On AutoLand, we run a subset of tests. At the time, it was kind of random,
what we decided to run. And on Mozilla Central, we run everything. To give you an idea on
Try, we will have hundreds of pushes per day. On AutoLand, the same. And on Mozilla Central,
we have only three or four. And it's restricted only to certain people that have the necessary
permissions since you can build Firefox nightly from there. And it's going to be shipped to
everyone. The scale here is similar to the bug case. We have 100,000 unique test files.
We have around 150 unique test configurations. So combinations of operating systems, high
level Firefox configurations. So old style engine versus new style engine, certain graphics
engine versus another graphics engine, et cetera, et cetera. We have debug builds versus optimized
builds. We have asan, code coverage, et cetera, et cetera. Of course, the matrix is huge and
you get to 150 configurations. We have more than 300 pushes per day by developers. And
the average push takes 1,500 hours if you were to run it all one after the other. It
takes 300 machine years per month and we run around 100 million machines per month to run
these tests. If you were to run all of the tests, you would need to run all of the tests
in all of the configurations. You would need to run around 2.3 billion test files per day.
Which is, of course, unfeasible. And this is a view of our tree herder, which is the user
interface for Mozilla test results. You can see that it is almost unreadable. The green
stuff is good. The orange stuff is probably not good. You can see that we have lots of
tests and we spend a lot of money to run these tests. So what we wanted to do, we wanted
to reduce the machine time, spend to run the tests. We wanted to reduce the end-to-end
time so that developers, when they push, they get a result, yes or no, your patch is good
or not, quickly. And we also wanted to reduce the cognitive overload for developers. Looking
at a page like this, what is it? It's impossible to understand. Also, to give you an obvious
example, if you're changing the Linux version of Firefox, I don't know, you're touching
GTK, you don't need to run Windows tests. At the time, we were doing that. At the time,
if you touched GTK code, we were running Android, Windows, Mac, that was totally useless. And
the traditional way of running tests on browsers doesn't really work. You cannot run everything
on all of the pushes. Otherwise, you will have a huge bill from the cloud provider.
So we couldn't use coverage because of some technical reasons. We thought, what if we
use machine learning? What if we extend bug, bug to also learn patches and tests? So the
first part was to use machines to try to parse this information and try to understand what
exactly failed. It might seem like an easy task if you have 100 tests or 10 tests, but
when you have two billion tests, you have lots of intermittently failing tests. These
tests fail randomly. They are not always the same. Every week, we see 150 new intermittent
tests coming in. It's impossible to, it's not easy to automatically say if a failure
is actually a failure or if it is an intermittent. Not even developers are able to do that sometimes.
So not all of the tests are run on all of the pushes. So if I push my patch and a test
doesn't run, but runs later on another patch and it fails, I don't know if it was my fault
or somebody else's fault. And so we have sheriffs, people that are only focused, whose only focus,
whose main focus is watching the CI, and they are pretty experienced at doing that, probably
better than most developers. But human errors still exist. Even if we have their annotations,
it's pretty hard to be sure about the results. You can see a meme that some sheriff created.
Lucky tests are the infamous intermittently failing tests. So the first step, the second
step after we implemented some heuristics to try to understand the failures due to a
given patch was to analyze patches. We didn't have readily available tools, at least not
fast enough for the amount of data that we are talking about. We just used Mercurial
for authorship info. So who's the author of the push? Who's the reviewer? When was it
pushed? Et cetera, et cetera. And we created a couple of projects written in Rust to parse
patches efficiently and to analyze source code. The second one was actually a research partnership
with the Politecnico di Torino. And the machine learning model itself, it's not a multi-label
model as one might think, where each test is a label. It would be too large with the
number of tests that we have. The model is simplified. The input is the table, test,
and patch. And the label is just fail, not fail. So the features actually come from both
the test, the patch, and the link between the test and the patch. So, for example, the
past failures, when the same files were touched, the distance from the source files to the
test files in the tree. How often source files were modified together with test files? Of
course, if they're modified together, probably they are somehow linked. Maybe you need to
fix the test. And so when you push your patch, you also fix the test. This is a clear link.
But even then, we have lots of test redundancies. So we used frequent item set mining to try
to understand which tests are redundant and remove them from the set of tests that are
selected to run. And this was pretty successful as well. So now we had architecture to train
models on bugs, to train models on patches and tests. The next step was to reuse what
we built for patches to also try to predict defects. This is actually still an experimental
phase. It's kind of a research project. So if anybody is interested in collaborating
with us on this topic, we will be happy to do so. I will just show you a few things that
we have done in the space for now. So the goals are to reduce the regressions by detecting
the patches that reviewers should focus on more than others, to reduce the time spent
by reviewers on less risky patches, and to when we detect that the patch is risky, trigger
some risk control operations. For example, I don't know, running phasing tests more comprehensively
in these patches and things like this. Of course, the model is just an evaluation of
the risk. It's not actually going to tell us if there is a bug or not. And it will never
replace a real reviewer who can actually review the patch more precisely.
The first step was, again, build a data set. It is not easy to know which patches cause
regressions. It's actually impossible at this time. There are some algorithms that are used
in research. The most famous one is SZZ. But we had some answers that it was not so good.
So we started here, again, introducing a change in the process that we have. We introduced
a new field, which is called regressed by, so that developers, QA users, can specify
what caused a given regression. So when they file a bug, if they know what caused it, they
can specify it here. If they don't know what caused it, we have a few tools that we built
over time to automatically download builds from RCI that we showed earlier. Automatically
download builds from the past and run a bisection to try to find what the cause is for the given
bug. With this, we managed to build a pretty large data set, 5,000 links between bug introducing
and bug fixing commits. Actually, commit sets. And then this amounts to 24,000 commits. And
then we were able, with this data set, to evaluate the current algorithms that are presented
in the literature. And as we thought, they are not working well at all. So this is one
of the areas of improvement for research. One of the improvements that we tried to apply
and to SZZ was to improve the blame algorithm. If you're more familiar with Mercurial annotate
algorithm, to try to, instead of looking at lines, splitting changes by words and tokens,
so you can see changes, past changes by token instead of by line. This is a visualization
from the Linux kernel. This is going to give you a much more precise view of what changed
in the past. For example, it will skip over tab only changes, white space only changes
and things like that. If you add an if, your code will be intended more, but you're not
actually changing everything inside. You're changing only the if. This actually improved
the results, but it was not enough to get to an acceptable level of accuracy. But it's
nice and we can actually use it in the IDE. We're not doing it yet, but we will to give
more information to users because developers use annotate and get blame a lot. And this
is a UI that is work in progress for analyzing the risk of a patch. This is a screenshot
from our code review tool. So we are showing the result of the algorithm with the confidence.
So in this case, it was a risky patch with 79% confidence. And we give a few explanations
to the developers. This is one of the most important things. Developers do not always
trust developers like any other user. Do not always trust results from machine learning.
And so you need to give them an explanation. And this is another part of the output of
our tool. This is again on our code review tool. We're showing on the functions that
are being changed by the patch if the function is risky or not. And which bugs in the past
were involved in this function. So developers can try to see if the patch is reintroducing
a previously fixed bug. And they can also know what kind of side effects there are when
you make changes to a given area of the code.
Now we did a lot of stuff for developers. We trained models for bugs. We trained models
for patches. We trained models for tests. We trained models to predict the facts. Now
I'm going to go to a slightly different topic even though it's connected. Privacy-friendly
translations. So we're working on introducing translations in Firefox. The subtitle was
actually translated automatically using Firefox translate, which you can use nowadays. The
idea is that translation models improved a lot in recent times. Current cloud-based
services do not offer the privacy guarantees that we like to offer in Firefox. They are
closed source. They are not privacy-preserving. So we started a project. It was funded by
the European Union to investigate client-side private translation capabilities in Firefox
itself. It is currently available as an add-on that you can install in Firefox. We support
many European languages and we're working on supporting even more. We're going to also
work on supporting non-European languages like Chinese, Korean, Japanese, etc.
And in this case, we use machine learning on the client side to perform the translation.
So your data never leaves your Firefox. The models are downloaded from our servers, but
they run locally on your machine. So the contents of the web page that you're looking at will
never go to Google Bing or whatever. They will be translated locally on your machine.
We use a few open data sets. Luckily, we have lots of them from past research. Not all of
them have good quality, but many of them have. But we are looking for more. So if you have
suggestions for data sets that we can use, please let us know.
On the data sets, we perform some basic data cleaning. And we use machine learning-based
techniques to clean the data, to remove bad sentence pairs that we believe are bad. Of
course, the data set that I showed before are open, but sometimes they are just crawled
from the web, so they contain all sorts of bad sentences. Also, HTML tags and stuff
like that, we need to clean them up. Otherwise, the translations will learn to translate HTML
tags. And we use some techniques to increase the size of the data set automatically, like
back translations, translating sentences from one language to the other, and back translating
it in order to increase the size of the data sets.
So we trained a large model on cloud machines, which is pretty large. You can see it's around
800 megabytes, so every language pair, you would need to download 800 megabytes, and
it is super slow, so we can only use that on cloud.
So we use some techniques in order to reduce the size of these models and to make them
faster. We use knowledge distillation, basically using the model, the large model that we trained
as a trainer for a student model, which is much smaller, so you can see that from 800
megabytes we got to 16, I think now we're around 5, 6, something like that, so it's
much smaller and you can actually download it on demand from our servers. And we use
quantization for further compression and perf improvements, so moving the data from the
model from float 32 to int 8.
Then we compiled the machine translation engine to WebAssembly in order to be able to use it
inside Firefox. We introduced some SIMD extensions into WebAssembly and into Firefox in order
to be able to be even faster when translating, even though we translate a bit at a time,
so it's pretty fast. And the engines are downloaded and updated on demand.
Let me show you a demo.
So, you can see my Firefox is in Italian, but you can see that it automatically detected
that the page is in French and it is suggesting me to translate it to Italian. I will change
it to English. Oh, fuck.
So it is downloading the model. Now it's translating. So while it was translating, you already
saw the contents of the first part of the page was already translated, so it's super
quick in the end. And the translation seems to be pretty good. I don't speak French, but
I think it makes sense. You can also use it from the toolbar, so you can choose a language
and translate it to another. Let's do Italian to French. It works.
All right. So if you know any data set that we can use, in addition to the ones that we
already use, or if you're interested in building a new great feature in Firefox, or if you
want to add support for your language or improving support for your language, come and talk to
us at our booth. We would be really happy if you could help us. And before we come to
an end, let me show you how far we've come. The dogs have grown, and we have learned that
it is possible to tame the complexity of a large-scale software. It is possible to use
the past history of development to support the future development, and it is possible
to use machine learning in a privacy-friendly way and in the open. What else could we do
with the data and the tools that we have at our disposal? I don't know. I'm looking forward
to know. I'm looking forward to see what other wild ideas you and us at Mozilla can come
up with. Thank you.
Thank you very much, Marco, for the amazing talk. Now we're open for questions. If anyone
would like to make a question, please raise your hand so I can take the microphone. Questions,
questions, hands up. There. Okay, okay. I'm sorry, I'm learning. I'm new to this. I'm
coming up. Hello. I have actually two questions. First question is, have you actually think
about the idea to use this mechanism to automatically translate interface of Mozilla products?
Sorry? This thing? Yes. Yeah. So the question is, have you think about mechanism of automatically
translating the interface of Mozilla Firefox products, or maybe documentation you already
have like MDN, because it's still a demand to translate this stuff?
I'm sorry. I'm not hearing well. Can you maybe come closer?
From here? Okay. Now it's better? Yes. Okay. So my question is, have you trying to use
this mechanism of automatic translation to use this translation for existing interface
you have in the products, and especially also documentation part? Because it's kind of vital
part when you need to translate new functionality, or you have to translate something new in
the interface, you need the help of translator. But if you already know how to translate in
doing this stuff, so that means like you already have a data set, you can actually automatically
translate new parts of interface without translator? Yes. So it is definitely something
that could be used to help translators do their job. We could translate parts of the
interface automatically. And of course, there will always be some review from actual translator
to make sure that the translation makes sense in the context, especially because Firefox
UI sometimes you have very short text and it needs to make sense. But yeah, it's definitely
something that we have considered. And actually one of the data sets that we use from the
list, it's not possible to see from the slide, but it's called Mozilla L10N and they are
sentence pairs from our browser UI. People are actually using it in research for automating
translations. Does anyone have any other question? Please raise your hands. If you have any
other questions, Marco? Okay. If not, thank you very much again, Marco.
Thank you.