Welcome to this session about LKFT, the Linux Can Help Functional Testing Project.
My name is Rémi Durafor, I'm a Principal Technic at Linao.
I've been working on open source projects since 2007 and I've been the Lava Architect
and Main Developer since for eight years now, so quite some time now.
So I will speak today about LKFT because it's a project I'm working with.
So what is LKFT?
So the goal of LKFT is to improve the Linux kernel quality on the ARM architecture by
performing regression testing and reporting on selective Linux kernel branches and the
Android common kernel in real time.
That's what is written on the website.
So it's a project that is led by Linao.
The goal is to build and test a set of Linux kernel trees.
So we care mainly about LTS trees, mainline and next.
For LTS in particular, we have a 48 hour SLA, which means that we have to provide a full
report in less than 48 hours for any change on LTS.
If you look at the numbers for 2023, we tested 465 RCs.
As we test mainline and next, we also built and tested 2,628 different commit versions,
which means that we built 1.6 million kernels and ran 200 million tests in a year.
That's only for Linux.
If you look at Android common kernel, only for the test, that's 58 million tests, 580
million tests, so VTS and CTS mainly.
And this is all done by only three people.
So the question is how do we do to build that many kernels and test that many kernels with
only three people, obviously automation.
So my goal today is to show you the architecture of LKFT and to also show you the different
tools that we created and maintained to make that possible.
Because I'm sure that you can go back home with some of these tools and might be useful
for you.
So let's look at the architecture now.
So this is a really simple view.
We have a set of trees in GitLab that are just simple mirrors in GitLab of the official
trees.
We just use GitLab for a scheduling mechanism.
So it will pull the new changes and it will run a GitLab CI pipeline.
But we won't do anything specific in GitLab CI pipeline.
We won't do build or test inside it.
It's too slow and costly.
So we just use it for submitting a plan to our system that will do the build and test
and reporting.
And at the end, we will just get a report that three engineers will look at and decide
if we have to report something to the main developers or if we can just find a commit
ourselves and send a patch.
Let's dig in a bit now.
So as I said, we don't use GitLab CI for building.
We submit only from GitLab CI a build request to our system.
So for building, we created a tool which is called text make.
I will explain the different tools later on.
I'm just showing the architecture right now.
So we use a tool called text make that allows for building the system with different combinations
of options.
And we created a software as a service that allows to use text make at a large scale in
the cloud.
So we can build something like 5,000 kernels in parallel in the cloud in some minutes.
When one build is finished, so when text make finishes build, they are sent to a storage.
It's an S-free like bucket somewhere.
And a result is sent to Squad, which is a second project that we also maintain.
That would be what that I like where everything is stored.
As we send results really early, if there is a build failure, a build regression, you
will notice that in some minutes or hours depending on how long the build takes.
Because for example, if you do an old mod config build with Clang, it will take up to
one or two hours easily.
But this way we can have early regression that we can send immediately to the main
list saying that it's failing to build on this architecture for this tool chain.
That's for building.
I will explain text make a bit later on.
So as I said, when a text make build finish, we send the result to Squad, we store in the
storage and we also submit multiple run test runs that will be done in the cloud.
So we do a test in the cloud and on physical devices.
For the cloud, we have a product called text run that will allow to test on virtual devices,
so QMU and a VP.
And the same, we have a system that allows to scale in the cloud the text run processes.
So you can spawn the same thousands of processes of text run processes in parallel in the cloud.
And they will send the results to Squad also.
Testing in virtualization is nice.
You find a lot of bugs because you can test a lot of different combinations.
But that's not enough.
So I have to test on real devices.
That's where a second software come in, which is Lava, that will allow to test on real devices.
So the same when text make finishes to build, it will submit a set of test requests to Lava
that will run on real hardware, this case.
So obviously, we run less test on real devices and on virtual devices because we don't have
enough board.
It's always the single point that you're missing.
The same results are sent to Squad and when everything is finished, we have a full report
that we can provide to the developers that we run something like thousands of tests,
thousands of builds, and everything is working or we find some regressions.
That's the overall architecture.
I will now look at the different projects so you can know if something can be useful
for you.
So let's look at the build parts.
So as I said before, we use text make.
It's a project that we created to make building easy and reproducible.
So it's an open source command application.
It allows for portable and repeatable Linux kind of builds.
So for that, we use containers.
We provide a set of containers with all the tools you need inside and everything is done
inside a container.
So it can be reproducible from one machine to another.
So because that's often a problem when you report a build failure, it's always a nightmare
to know the exact toolchain that you're using, everything.
So as everything is inside a container, you can just reproduce it in another machine.
So we support multiple toolchains from GCC from A to 12, client from 10 to 15.
In fact, 16 has been added this week.
We also have a Clang Android version and a Clang Nightly.
Clang Nightly is specific because we rebuild the nightly Clang toolchain every night and
we push it to our system so we can just test with the latest Clang.
We also support multiple target architectures, all the ARM versions, Intel EMDs, and then
some MIPS, PowerPC, RISV5, and some exotic one like S390, SH4, things like that.
So building is really simple.
You just specify the target architectures, so X8664 in this case.
You specify the toolchain, so I want to use GCC12.
You just need to have text-making installed on your computer because everything will then
be done inside a container where you will have GCC12 to chain for X8664.
If you want to build with GCC13, just change toolchain to GCC13 and it will use another
container to build it.
As I said before, we have a private software that allows to run text-making at a large
scale in the cloud, but I'm not presenting that it's a close-up software.
So just to explain how it's working, text-making will pull the right container for you.
So for this specific target-arched toolchain couple, it will be X8664 GCC12 container.
We have thousands of containers, hundreds of containers.
It will create a unique build directory, so it's reproducible from one build to another.
And then we just start a podman container, jump into it, and just build.
We advise to use podman, obviously, and not docker because it will be a rootless container,
so you can at least don't run asboot your build.
And then it will invoke a set of different make comments depending on what you want to
build.
And then it will move everything to a specific directory that will be kept on the machine.
And you will have all the artifacts, kernel, headers, et cetera.
And you also have metadata.json file that will include a lot of metadata about your
build, like version of your toolchain, of different utilities on the machine, the time
taken by different steps, the size of everything, et cetera.
And it will be useful for debugging also what's going on, if something breaks.
And yeah, we provide multiple containers that you can reuse.
And it's an open source project, so you can contribute to it a few months, and you can
just use it right now.
And some kind of developer use it for reproducing builds, build failures.
And in fact, as I said, we have a client-nightly toolchain that is rebuilt nightly.
It's in fact because the client project asked us to do that because they use Tuxmake with
client-nightly for validating their client version against different kernel versions
to see if clang is not regression.
That's for building.
So now, how do we test?
So as I said, we test on virtual devices with Tuxrun and on physical devices with Lava.
So for Tuxrun, it's the same.
It's an open source common line application.
It's the same for Tuxmake, but for running.
It allows for portable and repeatable kernel tests.
We support multiple devices, MVP MVA, which is an ARM V9.3 emulator, a simulator.
That's the latest version that you can try for ARM.
And then multiple ARM versions with multiple QEMU devices.
Many ARM Intel MIPS in many different versions and PPC, et cetera, and multiple tests with
LTP, K-Unit, K-Self tests, et cetera, et cetera.
Adding one is not quite easy to do.
The same, the common line is quite simple.
We also use Sponman for containerizing everything.
You specify the device that you want to use, the kernel that you want.
It can be your URL, obviously, and a root file system also if you want.
And again, we have a SAS that allows to run that at large scale in the cloud.
When you call that, that common line, Tuxrun will download all the artifacts that you need.
So kernel, DTB, root file system modules.
It will inject the modules inside the root file system for you, so that it will be used
at a good time.
And start the container, start QEMU system, so R64 in this case.
Look at the outputs, et cetera, et cetera, all the classical things, and store the results.
As I said, we provide a lot of root file systems because we know it's painful to build your
root file system for multiple architectures.
So we do the work for that.
We use billroot and debian.
Billroot allows us to have the 19 supported architectures, one root file system for each.
And for the main one, the one supported by debian, we do provide the debian root file
system that we build.
And obviously, if you build your own one, you can use it if you want.
And we will do the job of rebuilding the billroot and debian file systems regularly.
And in fact, it's a fun thing, we actually found bugs in QEMU before pushing the new
file systems.
We test in our system with the new root file systems.
And the last time we did that, we found issues in QEMU 7.2 that are currently being fixed
by QEMU developers.
Something fun because Tux-Mech and Tux-Run has been done by the same team.
So we make the work to combine the two tools together.
So obviously, you can, doing a bisection of a build failure is quite easy.
You just need a lot of CPU time.
Same for a runtime issue, which is you find a regression where a test fail on a specific
architecture.
For example, when you run a LTP test suite on QEMU ARM64, it's failing.
And you want to bisect that.
So find the faulty commit.
You have a good commit and a bad commit.
And you want to find the faulty commit.
Git allows you to help you on that.
But thanks to Tux-Mech and Tux-Run, we can automate all that job of testing.
So with this common line, Git will call Tux-Mech on different commits to try to find the 41.
And Tux-Mech will just build.
And at the end of the build, thanks to minus minus result hook, it will exec the command
that is behind that will run Tux-Run with the kernel that has been just built.
So it will build with Tux-Mech, and at the end, run with Tux-Run, the exact LTP test
suite that fails.
And if it's passing, it will return zero.
If it's failing, it will return one.
So based on that, Git will be able to find the faulty commit for you, which is quite...
We find a lot of regression or test regression and find the faulty commit thanks to just
that command line, which is really cool.
Thanks to Anders for the idea.
So that was all virtual build, building containers, test on virtual devices, but as I said before,
we have to test on physical devices because multiple bugs are only found on physical devices
because they are based on drivers failing and things like that.
So for that, we use Lava, like many, many, some people in this room.
So Lava stands for linear automated validation architecture.
It's a text execution system.
So it will allow for testing software on real hardware automatically for you.
So it will automatically deploy, boot, and test your software on your hardware.
So it's used by Canon CI a lot, by LKFT, obviously.
And for...
You can do system level testing, boot level testing.
You can do boot loader also testing.
You can test directly, directly, your boot loader and the firmware.
And it currently supports 356 different device types.
So from IoT to phones, Raspberry Pi-like boards, and servers.
So multiple different device types.
So for example, if you want to test on a Raspberry Pi, without Lava, you will have to pour on
the board, download the artifacts, so kernel, rootFS, files, DTBs, place them on a specific
directory, like NFS or TFT directory, connect to the serial, type a lot of commands, boot
the board, watch the boot outputs, type the logging prompt, et cetera, et cetera.
So it's really painful to do that manually.
Lava will just do exactly what I just listed, automatically for you.
It will just provide a job definition, which is a YAML file, with links to all the artifacts
that you want to test.
You specify the kind of board that you have.
So it's a Raspberry Pi 4B, and Lava will know then how to interact with that board.
And you will say that you boot install on it, and you have a TFTP server.
Just use that, and test what I want to test on it.
And Lava will do that automatically for you.
Obviously, you can have multiple boards attached to the same worker, and you can have multiple
workers on a Lava instance.
So as a user, it's really an abstraction of the hardware, and you just send a YAML file
and you get results, and all the hardware part is done automatically by Lava for you.
So as I said, maybe you remember the first LKFT diagram.
I'm sure you don't.
That was a small box called KeysCache.
So when we submit jobs to Lava, we submit multiple jobs for the same artifacts at the
same time.
We have multiple devices.
So the scheduler will start the job for the same artifacts all at the same time.
So it will download multiple times the same artifact at the same time, so we just should
be able to catch that and decrease network usage.
So we tried squid, and the short answer is squid is not working for that use case for
different reasons.
The first one is that, as I said before, all the artifacts are stored in an S3 like bucket.
So it's somewhere on internet.
So obviously we use SSL, HTTPS, to download it.
And squid and HTTPS are not really working well together.
You have to fake SSL certificates.
It's all creepy things to do.
And also a thing that, as I said, with download, Lava will start all the jobs at the same time.
So they will more or less download all the same artifacts at exactly the same time.
And if you do that with squid, squid will download, if you ask for n times the same
file to squid, if it's not already cached, squid will download it n times.
And only when one is finished, or when download is finished, the next one will use a cache
version.
So it's just pointless for us, just not working.
So we created a tool called keyscache, the keys is for keep it simple, stupid.
It's a simple and stupid caching server.
It's not a proxy, it's a service, which means that it can handle HTTPS, and it will only
download once when you have multiple clients, and it will stream to the clients while downloading.
It's not transparent because it's not a proxy, and because it's not transparent, it can do
HTTPS, because you will have to prefix your URL by the keyscache instance that you have.
And you will talk to keyscache directly.
It also automatically retries on failures, because we've found multiple failures that
all the HTTP code that you can have when you request on an S3 like bucket, just insane.
And sometimes also you will get, the connection will finish like if everything was done correctly.
And in fact, the file is not complete, it's a partial download, and you don't get any
errors.
So keyscache will detect that for you.
It will detect that it's a partial download, and it will retry and download only the remaining
things for you.
And it's fully transparent as a user.
It will do that in the background and still stream your data to you.
So thanks to that, we've been using it for 2.5 years now.
In the graph, in green is what we serve locally from keyscache, and in red is what we download
from Internet.
So we downloaded 25 terabytes of data from Internet, and we serve 1.3 petabytes of data
in the local network, which is the 52 times expansion ratio.
So it's quite useful, and it improves stability also.
So it's really cool.
It's a good tool for your CI if you don't use it already.
And last but not the least, we store all the job results in Squad.
So it's software quality dashboard.
It will store, it's a data lake.
It will store all the results for you in different categories, and it will allow you to create
reports, so failures, regressions, et cetera.
Everything is stored in this one, and then we extract data and make report based on Squad.
And that's all.
That's what I just explained.
If you have any questions, I have some time for questions.
Five minutes.
Perfect.
Oh, yeah, that's good.
Testing methods?
We use LTP, KUNIT, KSELF-SES, all the kernel test suites that we don't, we are not creating
new test suites.
We are using test suites that does exist, and we build for the community, and we test
for the community, and then we provide reports.
We obviously interact a lot with the test suite maintainers, because we found bugs in
the test suite, too.
We have to report to them, and there's reporting a lot to them.
And one of our projects is to test KSELF-SES in advance, test KSELF-SES master, to find
bugs in KSELF-SES before they are actually running in production after.
If you find any problems and report them, are current developers actually looking at
them, or do you have to ping them and make sure they take care of the problem?
Okay, so we have an SLA with Greg Croatman, so he's waiting for our results.
So they will look at it for LTS.
And for Mainline and Next, we are used to reports.
We report a lot of issues, so they know us.
If you look at LWN articles, about they classify the different contributions to the kernel,
and Linaro is in the tested-by top in the tested-by, so they know us a lot, so they
know that we provide good results.
And when we provide a mail, there is everything that, every tool they need for reproducible.
They are reproducing a build, so we provide all the binaries that they need for reproducing
it.
If it's a big failure, we provide a tux-make command line that they can use, and they
are now used to use tux-make for rebuilding things.
And if it's a test failure, we provide the logs, obviously, the job definition, and all
the binaries they need for reproducing it.
Do you actually check that every problem you found is actually fixed?
And those are all the bugs that we found fixed?
Not all of them?
Yeah, if you found some bugs on SH4, no one will care, for example.
The QMU 7.2 has been released recently, just not working on SH4.
I couldn't answer that.
We use the WS.
No, it's not that bad.
We build a dynamic system, which means that we do not rent 5,000 machines in parallel.
Obviously not.
It's just impossible for us.
We are a small company.
Everything is dynamic, so from one second to another, if you look at the graph of usage,
when Anders submits a plan for testing, in one minute, we'll book 5,000 machines for
building it, likely more 1.5,000 machines to build it.
They will build and they will just stop at the end.
So no, we don't have 5,000 machines.
How many devices do you have in your lava test brick?
So for the LKFT, we have multiple lava instances in Linauro, in LKFT, how many devices?
About 20.
20, yeah.
And about 5 different device types, like Rolls-Royce, Dragon Balls, Junos, X8, X15.
But yeah, you can have really large labs in lava.
We have another one for just Linauro usage, where we have something like 100 balls, I
think, the main one.
Thanks.
Thank you.