Okay, hello everyone, thanks for being here so early on a Sunday morning.
My name is Laura, I work at Glabra and today I'd like to share with you like a war story
about how we built and grew our laboratory for upstream testing.
I'm going to share with you a little bit about our infrastructure as well as some of
the challenges that we had to face while scaling up.
So our main goal was to build a big test bed for open source projects to use.
So of course we're going to need a diverse ecosystem of devices, so many different devices
of different architectures and from different vendors.
Of course we're going to need a software to automate the tests on the actual devices.
We need a monitoring system, so a way to monitor and assess the health of the devices that
we have in the lab and we also need some recovery strategies.
So mainly when devices start to misbehave or don't behave as expected, we need some
way of recovering them automatically or putting them offline automatically if they're not
reliable to run tests.
So it all starts with a commit, the developer pushes the changes into a development branch,
the artifacts for the tests all built automatically, a test job is submitted and run, the results
are gathered and parsed and finally a report is generated and sent back to the developer.
So from the lab perspective we're interested in the part that runs the test jobs and makes
the results available and what we chose for our lab is Lava as we saw earlier, this is
the linear automation and validation architecture.
So it automates the boot and deploy phases of the operating system on the device.
It has a really scalable scheduler, it allows to run thousands of jobs on hundreds of devices
on a single instance, so that's really convenient for big labs.
It handles the power on the devices, so it switches the power on and off on the devices
when needed and it also helps monitoring the serial output.
And finally it also makes the results of the tests available in many different formats,
which is again pretty convenient.
Lava again just takes care of this part of the CI loop while all the other phases needs
to be implemented with different tools.
So in order to run devices in Lava we need to fulfill a set of base requirements, of
course we're going to need to be able to turn on and off the power on the devices remotely,
we need access to a reliable serial console remotely and finally we need some way of booting
an arbitrary combination of kernel, device tree and Vultafast remotely.
For all the devices that we have in the lab we rely on TFTP, so we need network connectivity
at the bootloader level and that means that we often have to build our custom bootloaders
and enable all the features that we need for debugging.
So there are a few steps to prepare the devices before they enter the lab.
As far as the configuration of the devices itself in Lava we have a couple of, you only
need to define some JINJA 2 and YAML templates, so the device type template basically defines
the characteristic of the device type, so for example which kind of bootloader run on
a certain device, which kind of command line options are needed for booting, while the
device dictionary defines device specific characteristics, so for example what command
do we need to run to turn on and off the power or to access the serial console and finally
we have the health check, which is a special kind of job associated to each device type
and the aim of the health check is to assess the health status of each device, it's supposed
to be run on a fairly regular basis, we run a health check on every device that we have
in the lab every day and the examples of tests that you can fit in a health check are for
example a battery test or you can check the temperature on the device to make sure it's
not overheating, you can check the network connectivity, basically all the tests that
you need to make sure that the device is functional, you can fit them in health check and whenever
a device fills its health check, lava automatically puts it offline, so it's really useful just
to shut down all the devices that are not reliable at the moment, so Colabora maintains
a laboratory running lava and we have as of a couple of days ago 217 devices of 38 different
types, spread across 16 racks, each rack is controlled by its own server and that's also
where the lava dispatcher runs and of course besides all the device types, devices we also
have a bunch of hardware equipment that we need to automate the boot and test phases
on our devices and this is what the device distribution looked like in January, so the
vast majority of our devices are X8664 and ARM64 platforms and we also have some QM instances
that are mainly used by KernelCI and the very vast majority of our devices are actually
Chromebook laptops but we also have some embedded SBC devices as well, so what kind of hardware
do we have in the lab, so different devices as usually different requirements, so for
embedded SBCs what we use to control the power on them remotely are Ethernet control relays
and PDUs, I left there some examples of the actual models that we currently have in the
lab, Chromebooks are kind of a different beast, they have their own hardware debug interface
which is the Servo V4 and the Susie cables, so Servo V4 allows you to control the power
on the device to access the serial consoles on the device and also provides network connectivity
to an Ethernet port, so you can fit everything you need to automate the boot and test phases
on a Chromebook that fit inside just one hardware box, as an alternative you also have Susie
cables which pretty much have the same functionality except for the network connectivity that you
have to provide usually to a USB to Ethernet adapter.
We have a couple of servers as well in the lab and for those we use the IPMI standard
protocol just to control the power and access the serial consoles and for all the devices
of course we're going to need a bunch of USB cables with their fragilities and also we
use USB hubs, we find especially useful the switchable hubs such as the Y-Cush, especially
for those devices that are controlled by just one USB connection such as the Chromebooks,
so that's really convenient just not to having to manually intervene every time you need
to re-plug the USB connection.
As for the software, I left here a couple of links, you want to check them out.
We use PDU Demon to execute comments on the PDUs, we use Conserver to access the serial
consoles and monitor the output and the HDC tools are just for the Chromebooks, these
are the software tools that allows you to interact with the server v4 and with the Susie
cable as well just to control the power and serial on the Chromebooks.
For the interaction with Lava, we use Lava CLI, it's a command line interface and that's
useful to run the tests on the device and also configure and push the templates.
Finally we also have a Lava GitLab runner that serves as a bridge between GitLab and
Lava.
That's pretty much it for the software side.
In our lab we have two major users, one is KernelCI which is focused on continuous testing
of the Linux kernel, it's not only boot tests, we have a bunch of other test suites running
on them and the type of testing that KernelCI does is post merge, so changes are tested
after they landed on a set of monitored trees.
After the tests have run, KernelCI will generate some build reports as well as some regression
reports for every regression that is found.
The other major player in our lab is MesaCI, that's DCI for Mesa3D and it does conformance
testing and also performance tracking.
There are a bunch of test suites that are currently run by MesaCI, I left the list here, bunch
of APIs and drivers are tested and while KernelCI only does post merge testing, MesaCI also
does pre merge conformance tests, so that's a little bit of both.
In this diagram you can see what's the usage of KernelCI and MesaCI in our laboratory,
as you can see both projects keep our lab pretty busy, KernelCI uses almost all the
architectures that we have in the lab, while MesaCI is focused more on X8664 and ARM64,
and with so many jobs running every day in our lab, of course the impact of any error
or unreliability in the infrastructure can be quite big, so for pre-merge tests you
have the merge requests from users can get blocked, and definitely if a certain device
type is not available, and also, yeah, there's a risk of merge requests getting rejected
if there are many errors in the lab, so what we need to make sure from the lab perspective
is that the merge requests from users do get rejected only because the changes introduced
made the test fail, and not because of any infrastructure error, while for post merge
tests we have a risk of reporting false regressions, in this case we want to make sure again that
the infrastructure errors are reported as such by Lava, and Lava defines different types
of exceptions that you can raise based on the type of error that occurs, we need just
to make sure that the devices and Lava itself is configured properly to do so, so yeah,
this is just a minimal list of the common issues that we have seen over the years, of
course there can be other degradation, you can have faulty cables at any time, or batteries
just failing, power chargers not working properly, all kind of network issues can happen at any
time and they can have quite a big impact, we also saw some issues related to the rack
setup, so for example we had some laptops where the lid was likely too closed because
of how it was set up in the rack and it was closing the device to enter suspense unexpectedly,
so we have all kind of different errors that can happen, of course we can have firmware
bugs either in the firmware running or the actual devices, or also firmware running in
the hardware debug interface, so that's a lot of errors that can happen, I gather a few
of my favorite pitfalls, these are issues, tricky ones that we have found recently and
we're still dealing with some of those, so one of the things that we saw is that sometimes
it happens that the serial console will just stop outputting anything on the serial console
and if this happens during the test phase it's kind of hard to understand in an automated
way, whether the kernel is hanging or whether your USB cable connection has dropped or if
it's just like an unreliable serial connection, so that's usually a tricky one to deal with.
Another serial related one is caused by interference, so not all devices can have multiple UART
connections for debug, most of our devices in the lab don't, so we have to share the
same serial connection between the kernel and the test shell and this sometimes can
cause some interference and of course it will confuse lava about the outcome, so one way
that we are thinking of many solutions to deal with this kind of serial issue stuff
and one approach that we are looking into is actually using a docker container, so running
a docker container connecting to the device over SSH and run the tests on the SSH console,
this way we can probably work around some of these serial issues.
So as I said there are also network connectivity issues from time to time and if the network
drops during the bootloader phase, that's usually something we can easily catch because
lava of course monitors the serial output and if our bootloader is nice enough to print
error messages we can just catch the right patterns at the right time and just raise
an infrastructure error so that won't initiate like the outcome of the test.
When this happens we can also configure lava to retry the job if needed, so when this happens
it's useful to catch error patterns.
If network decides to drop during the test phase that's usually worse, especially for
devices that rely on a network phase system, so it's usually pretty hard to recover from
this.
We have seen occasional USB disconnection for whatever reason and yet it's hard to recover
from these kind of issues usually.
So these are some of the best practices we came upon while working on these issues.
So the first one is about writing robust health checks, so as I said devices will be put offline
by lava if the health check fails so we need to make sure that the health checks catch
as many issues as possible automatically.
We found very useful to monitor the lava infrastructure error exceptions and this is mainly to catch
issues with specific racks or specific device types.
We usually try to monitor also the devices health and the job queue as well and this
is to make sure that we have enough devices of a certain device type to feed all the pipelines
for the projects and also to minimize like if a certain device type goes offline and
if we have redone this we're able to kind of recover from that and last but not least
as I said what best practice is to try and isolate the test shell output and kernel
messages whenever possible.
If not we're trying to work around some of these issues.
So next steps for our lab is of course increase the lab capacity and try to cover even more
platform and different vendors as soon as they come out.
While doing this we are continuing to improve our infrastructure and monitoring tools.
I haven't included in this presentation how we actually monitor things but yeah lava just
has some APIs that you can use to monitor the status of each device and also of the server
and yeah of course while keeping to keep adding new lab devices we also want to increase
the coverage of test suites so we're working on adding even more test suites on kernel
CI and meso CI as well and that's it.
If you have any questions I think I have time right.
Pretty often I'd say yeah I don't have data at hand with the actual failures but yeah it
happens pretty often.
I mean we have so many jobs running every day and we rely heavily on USB which is kind
of not great like it breaks pretty often I'd say the most common issues that we have is
usually due to the serial consoles being not too reliable.
The vast majority of devices that we have are Chromebooks and we're using like these
hardware debugging interfaces that were meant for debugging so sometimes like the serial
connection is not great and that cause all kind of issues so we try at least to retry
the jobs when possible and catch the infrastructure errors as they come out.
I'd say we don't have to manually intervene every day I'd go like every couple of days
we need to maybe re-plug some of the devices because we as I said we use the switchable
hubs to try and avoid having to reset the connection manually but we don't have this
setup on each and every device that we have we're working on it but yeah I'd say like
at least a couple of times a week I haven't really checked the frequency of it but yeah
of course there's people in the lab actually taking care of all the devices.
So from the lab perspective we don't really care about the test switch running it's more
the responsibility of kernel CI and mesoCi you can check out the links that I left like
everything is of course open source so you can check out all the test suites and how
they work yeah some tests you just need a RAM disk some other tests rely on the most
heavier ones rely on natural pack system.
I mean it depends on the type of tests that you need to run.
We use a lot of Chromebooks because we need Chromebooks so you cannot really emulate one.
Thank you.