I know everyone's excited for Tessa.
We keep the good talks for the end to make sure you all stay the entire day.
All right.
So we have Tessa in the box.
So please give a warm welcome.
Thank you very much.
Because the end is always a bit chaotic, I was just going to say, could we have a round
of applause for all our volunteers here?
So without further ado, this talk is testing in a box.
It's going to be given by my colleague.
Hi, everyone.
My name is Munti Chirma.
I'm a software engineer at Coltsing.
I've been with them for almost a year and a half, and I specialize in embedded hardware
and software.
And myself, William Salmon, I've been at Coltsing for about six and a half years, and I've
specialized in embedded Linux and Linux integration and testing of those things.
So I'm going to give us a little bit of context as to how this project was motivated and a
bit of a very high level view.
And if you guys are wanting to do similar things, maybe you can take inspiration from
this for persuading your colleagues of the value.
And then that's going to give us some very top level requirements, and then it's going
to talk through how those top level requirements become hardware requirements, and then our
implementation.
So Code Think has been offering software services for over 15 years.
We work with open source software a great amount.
All of our developers have Linux laptops, and all of our infrastructure runs on open
source services.
So we really believe in open source software, and this is kind of our journey into open
source hardware as well, and we're trying to take all of the good things that we've
learned from open source software to open source hardware, but we're a little bit earlier
in that journey at the moment.
So we work with a lot of embedded projects.
We have an awful lot of automotive clients, and we also have some in the financial industries,
and what they all share is that they want to bring lots of different bits of software
together, so they always work in exactly the way that they want, and they want to make
sure that what goes into their things is exactly what they want, but also that it behaves exactly
the way that they want it, so how do they verify that it behaves how they want testing.
So one thing that we found across most of our clients is that they all share some kind
of testing transition, some kind of testing progression, so on your left hand side you've
got your final production, and the reality is that some bugs only ever turn up all the
way out there, but they're very difficult to deal with, and we really want to avoid that,
and then we've got things on the right hand side of this that are very convenient for
us to run our tests with, but maybe don't provide as much value.
So no developer wants to come and contribute bad code.
We have these really creative developers in our industry, many of you have sat here among
us, and we want to make their lives as easy as possible, so we want to provide as much
value on this side of this transition, but also make it as easy as possible to get information
back from the later steps.
So what sorts of things are we battling?
So some things that we find quite problematic, so slow automated tests are problematic because
that's taking a long time to get information back to our developers.
Manual tests are never going to be particularly reproducible, so we really want to squash
those. There are some tests that are just expensive, maybe they involve some kind of
consumable that's expensive, maybe they involve some huge AWS server that we just want to
touch as little as possible, and sometimes maybe there is only one lab car because it's
a brand new car, and so we are really just resource constrained.
It doesn't matter how much money you spend, you've only got one.
So kind of our drivers are that we want to make the early steps as accurate as possible,
so we want to use as representative hardware as possible, we want to have as representative
peripherals as possible. A lot of cars these days have surprisingly complicated networks,
so there's a lot of different bits of hardware to go into making it really accurate, and
so as our code moves along this transition it gets more and more expensive, more and
more difficult to deal with, so another thing, and we get longer and longer feedback times
for our developers, so a key thing is to make the left hand side of this graph as easy as
possible. For our developers to interact with, so making it easy for our developers to trigger
things for later steps in the transition, but also to get information back.
So testing on hardware is what we're all about in this talk, and some things that, so I talked
about this is kind of a requirement for the next step, for Muddit, and so some of the
things that we definitely want to be able to do with this project is completely flash
a rig, because if you're not completely flashing a rig, you're going to end up with the last
MR affecting the next MR, whatever it is, we need to make sure we've got known starting
points, so we need to be able to do that. OTA, over the air updates are a thing that
goes wrong all the time, and have massive consequences, so we definitely need to test
them. As we went through our testing transition, we tend to find the more and more complicated
bugs. One of the reasons for that is that the actual tests that people do kind of get
more and more complicated and more and more integrated, so you need to be able to have
tests that in a single test can interact with the UI, and the various buses involved,
so in automotive often can, and also get down to some peripherals. So one test that we were
working with quite recently involved changing the set point of a car's climate control,
making sure that from the UI, not just some API call, but actually have the UI set the
point, then have that command go down through a bus into a real or a mocked peripheral,
and then have data come back up through the bus of the value slowly changing and make sure
that it all behaves together. By doing that, automatically we can also improve the reproducibility
of that test. Some very top level requirements are we need CI rigs, so we need some rigs
that are only for CI. We mustn't be merging without our automatic tests. We need some
kind of coordinator so we can make the most of those assets that we have. If we have five
rigs and we've got 50 developers, we want those rigs to be absolutely hammered so that
those developers are waiting as short as time as possible. We need to be able to interact
with the UI, and we need to be able to interact with the hardware through various buses through
IO, and we need to be able to control those peripherals. So now that I have gone through
all of those, we have a pretty good set of requirements for our testing infrastructure.
Thanks, Bill. Now that we've talked about different types of tests and requirements for them,
let's go through the basic requirements for testing infrastructure.
Let's see if it's a slide.
All right. I don't know. Anyway, I'll go about it anyways. So, I mean, talking about testing
infrastructure, we need a computer of sorts to talk to your device on the test, and you
need hardware to simulate the actual production application. So that would be, I mean, talking
about car rigs, we're talking about canvases, we're talking about serial in most of the
test images. These ECUs will have serial enabled, so you should be able to probe the ECUs and
get out of it. So you'll have multiple serial dongles. In some cases, the testing application
or the testing requirement is going to be so niche that you'll have to make custom hardware,
and sometimes making custom hardware is not feasible, so the first thing you'll do is
have your own reason of the shelf hardware, and you'll get a working example from there,
but following which, you'll have to make your own piece of hardware. We've done that before.
We've had clients who wanted to test device discovery for Android automotive and CarPlay,
and their ways of testing were quite manual, so they had engineers with spreadsheets and
iPhones, so the engineers would go to a rig, plug in a device, check the output, mark the
representation in a spreadsheet, and based on that, they would make a decision that if
it's behaving as it should be or not. So based on the description, we can make out that that's
a solution to a problem that will not and cannot scale well. So we had to automate that problem
for them, and we made a custom hardware for that and called it the USB switch. So the
USB switch is a bi-directional USB-C switch, which can help you programmatically control
one host between two peripherals and vice versa. It's a completely open source hardware
made with QICAD and the respective firmware and cases are also available with the project.
It supports USB super speed. That's something we've tested internally, not certified, but
that's something we've tested, and recently we've gone through the effort of EMC getting
the hardware EMC tested. It has cleared the EMC tests, and we are in progress of a process
of getting the documentation for it in the open as well. The QR code is going to take
you to the project, so if you want to learn more, come. So now that you've gathered all
the pieces of hardware for your basic testing infrastructure, you put it all together, and
the resulting setup looks something like that. So one good quality of this setup is it's
functional, it works, but it will confuse you when you're setting it up for the first
time and setting it up multiple times and tens of hundreds of rigs maybe sometimes is
going to be quite a challenge. You have to maintain a bit of material, you have to maintain
the documentation for it, and sometimes you'll buy hundreds of field long-books and you can't
buy them anymore, so you'll have variability in your piece of hardware. Again, a trouble.
So that gives us a question, what are the key requirements for a robust testing infrastructure?
So when we talk about long-term support of software, we cannot discount the maintainability
of the supporting testing infrastructure. That means we want to be able to maintain the
inventory of spares. So when you put in the effort to set up a rig, you buy a certain
piece of hardware that comes with its own tooling, and then you put in the effort to
set up that tooling with your testing rig, and then you can't buy it in the next two
years, that's added effort, and that added effort means time, and that time means money.
So you want to be able to buy and manage that piece of hardware that you're putting on your
rigs. And you want to be able to have the testing setup as easy as possible so that
the developers can actually get to use those tests, get value out of those tests rather
than spend time on setting up those tests. And we want to be able to have the setup
as consistent as possible because consistency removes variability from your setup, and that
means that you'll be pulling out less hair when you're debugging problems.
So keeping all these requirements in mind, we kind of made our own solution, and quite
descriptively called it testing in a box. So testing in a box can be seen as a multiple
for testing. It's supposed to be a kind of I.O. for testing and facilitating remote access
to your rigs, and it's designed with modularity in mind. When I talk about modularity, I think
this image depicts the modularity a bit better. So each section in that image is something
that we call as a slice. So at the base of the slice, we have a SBC, an unbased SBC.
On top of that, we have the USB switch slice. On top of which we have a slice which can
hold can dongles. And on top of that, we have our in-house built I.O. board. I'll be talking
about that in the upcoming slides. And the USB switch slices and the can device slices
are stackable, so you can add more USB switches if you want or not have them if you don't
need them. And same goes for the can devices. So I have the hardware with me here. So in
its most basic setup, this small contraption gives you the ability to have a GitLab runner.
It gives you the ability to do UI validation tests with OpenQA. You can do device discovery
tests by programmatically plugging in and out USB devices. And you can check states
with GPIO. And you can also mock or monitor can devices. So essentially checking out all
the boxes that we put on our list collectively in the beginning of the talk. So now to the
fun part. That's the custom built I.O. board that we have for testing the box. Starting
from the left hand side, we have three FT232H, which gives us serial SPI square C and JTAG.
Then we have an R24T for HID emulation. And we have three USB hub, USB two hub inputs.
And all of that then gets connected to your host, which can be an SVC or a laptop with
a single USB C cable. We also have an ESP32C3, which sends serial to the R24T for HID emulation.
And a new pixel, because everybody likes LEDs. And yep, if you want to electronically isolate
your GPIO, you have an optocouples on top. So this is the version one, region B of the
board. And these are the changes we got to make to version two of the testing in the
box I.O. board as a work in progress project. So the idea behind the version two is reducing
the cost, getting the bill of material smaller and with the space that we're going to make
with that, we want to add more functionality to the board. So we're going to be getting
rid of the three FTDI, FT32H and replace it with a single FT4232H, which will give us
4X serial and 2X MPSSE, which is multi-protocol synchronous serial engine. So basically that
will give you your SPI, I2C and JTAG. And we'll be getting rid of the ESP32C3, which
because we are not using it that much, the version one on the board was more of an experiment.
So we saw after you refused that we were not using it that much, so we're getting rid of
that and moving the functionality to the IP2040. And because we'll be getting rid of so many
USB devices, we can actually get rid of one of the USB hubs on the board, which was re-chained
initially, so that again gets the bomb cost lower. And with the space that we're going
to make, we are looking at putting in USB PD on the board. So what essentially you'll
be able to do is that you can put in a battle jack on this board and have USB PD out, so
you can basically power other SBCs from the board itself. I've said a lot of words in
the previous slide, so here's a quick overview of what we're going to be changing and the
benefits. So here is an example of the use case that we have internally for testing the
box. So in this example, we have a laptop, we have an X86 requirement, so we have a laptop
connecting to testing the box, which shows modularity because currently in this case
we're only using the IO board, which is connected to a Jetson Orin, and the output of a Jetson
Orin is being monitored with the capture card. Now I won't be going into much detail about
the actual example, but I'll be talking about the developer workflow. So every time a developer
has to make a change, they can test those changes on their own machines, which runs on
QMU, but when they're very confident that they can push these changes to the main, so
they raise a merge request, and that merge request will trigger a set of pre-much tests.
Those pre-much tests will flush the Orin, it will run a set of tests and check that it
meets the set of requirements, and based on that, once those pipelines are green, the
developer can be sure that, well, that the code hasn't broken the system in a way that
was not intended, and the reviewers also have a metric to basically check the code against.
So any piece of hardware is only as good as the software that comes with it, and we package
together with testing in a box various tools. So we have Ansible scripts that let you easily
set up good cloud runners, easily set up open QA workers, CI templates, we also have CI
templates that you can use, and we also have a self-servicing script that you can use to
set up the UDL rules for the IO board. We also provide an example, which, in a sense,
for that, you need a Raspberry Pi with the AGL, and all you do is put the QAD binary on it,
I'll be talking about that in a bit, and that essentially gives you a fully functional test
in which you are doing UI validation, you are able to run your GitLab CI pipelines, and
test and monitor can signals passing through the entire network. So I've got a couple of
things to talk about. I've mentioned open QA, QAD, and Canvas before, so open QA is a tool
that we use in-house to do UI validation tests, we have found it quite useful. So we have
also contributed to it to make it better, and QAD is a tool that we've written to do user input
simulation in CI, and how you make that work is, it's a small binary which you put in the
right place in the root of S, and that helps your GitLab runner to send it HTTP requests,
and mock the touch inputs that a user would actually, like the user would do. And Canvas
is a tool that we worked on as well, it's something we've made in-house, which helps you
build those tests and make it easier, essentially. So what's next? As I've already mentioned,
we're already working on the version two of the board, it's still a work in progress,
so if you guys want to join the project or have any suggestions, we're willing to, and
we're very happy for some much requests or suggestions. And we're also looking forward to
actually making a can of the expansion board, given that we set up these boxes on various
automotive rigs, and we do use various canned dongles, it would be nice to have a single one
that helps us basically transfer the knowledge between different projects, so we'll be looking
at making one of that as well. And there's many more to come, so stay tuned, and that
will be it for the day. Thank you very much for listening.
So this is a great initiative. So what I'm wondering, like the single board computer or some other
component, can it act as a USB device using the USB gadget subsystem, and can I, with the existing
setup that you have, act both like switching between acting as a USB device and acting as a USB host,
depending on the use case, using the same USB connection to the device of the test?
I mean, I don't see why it's not possible. Like it depends on the single board computer that you're
using, it needs to have a USB, and what single board computer are you using? That could also be a question.
Yeah, I mean, again, for our cases, we got the Rock5Bs, for instance, in this board, which actually does support
USB gadget. But the actual setup of testing the box is not limited to the Rock5B. You can put any system as you want,
as I say, it's modular, so you can actually change it for Raspberry Pi, or a much more beefier compute if you want.
Okay, so it's okay to ask, like how does it connect to the board? How do they connect with the interface?
Just a USB cable. Okay, perfect. Single USB cable.
So, really cool project. It's really interesting thinking about testing hardware, so I have no idea, so this is probably a
question from ignorance, but when it comes to sort of what you call a test rig, how close do you get to, like,
some mechanical hardware where something's moving, and at which point do you have to worry about state control?
You say you have to flash the whole thing, but as soon as you've got hardware in the loop, you've got a state of not just
the electronics, but you've got a state of the hardware, and that's where we've thought about this and decided to let somebody else
think about it, and I hope that's you.
It depends. I've got a mic, so you're off.
Sure, so essentially in this case, like, we try to get as close to the production hardware as possible.
Of course, it is hard to kind of test a moving car, per se, you know, so the idea behind the Cinebox is, as we'll explain
in the testing progression, that we want to get as close as possible to a testing environment, so when you actually go to
test and deploy to the production environment, you find as less bugs as you can, but yes, certain bugs will slip through,
but the more closer you can get in your testing rig, the better it will be when you actually deploy to the end car.
Okay, yeah, very cool. Do you support or have plans to support acting as a SD card for system booting from SD card?
So that is, again, implemented in detail of your tests.
Testing in a box is a multi-tool, so if your actual device supports booting from that, that shouldn't be a problem.
Again, that's implementation in detail of how you want to do your tests.
Booting from SD card. When you say, booting from SD card, do you mean, does any of our hardware boot from an SD card, or?
The device and the test moves from the SD card that image has.
Yeah, I mean, again, as I say, that's an implementation thing, so.
The question was, can the device and the test boot from an SD card?
Yeah, I think my answer is the same. It's an information in detail to the actual thing.
Yeah, so there are various bits of hardware out there for mocking an SD card.
We don't have any examples with that, but if you had something like that, you could potentially use the USB hub to facilitate something like that.
All of the devices that we have allow you to flash the root file system through some kind of GPIO manipulation to put it into a reset mode, and then you can flash it like that.
So the more modern Raspberry Pi's allow you to flash them by putting them into a reset mode, so you don't have to mock the SD card and the more sophisticated rigs we have also support that.
However, if you did really need to do that, then there is hardware out there that we don't currently use in this example.
But if you were doing that, then please send us some patches and we will add documentation for that for the system.
So thank you.
So I noticed that this is extremely neat and modular and self-contained and it's very easy to see what it's doing.
Why is something like this not standard in history?
Because in consumer electronics and in medical, big mess of wires is the standard testing rig and apparently automotive as well.
So why is everyone tolerating this?
Do you want to?
That's a good question. I think some of it comes down to the fact that everyone thinks this is secret source and they spend huge amounts of money on things that are very niche and custom.
So the way that we're developing this is that we think these are really important things to have and so we're working on them.
And like I said, these are tools, so we think that tools should be in the open, so we're doing it in the open and that means we can collaborate and we can all work together on them.
But when things are in little silos, then often the tools are the things that get hurt the most.
So I would say that I think there's a lot of secret source or perception, someone else was talking about secret source before and it was a perception of secret source being really valuable but maybe it's not.
So yeah, I think it's a really good question and maybe it would be better pointed at the OEMs to get themselves together.
Did you want to say something?
I was just saying lines.
Okay, so it's a cool project but if I want to participate in it, could I just buy ready made boards or do I have to make them myself?
So we publish all of the files that you need to go and ask a standard FAB to make them.
So you can take the files from our project, go to a fab of your choice dot com, upload the files and in a few weeks you'll have them.
We are in the process of hopefully CE marking this, so that gives us potential for selling them directly to people.
But in order to make that economically successful we need to move a lot of the items and so I'm not sure if that will happen but we want to make it as easy as possible for everyone to collaborate.
Yeah, if you want to buy them then not from us but we make it very easy for you to do that.
Well, making a small batch production is expensive.
So I was thinking that maybe you make thousands of them just sell to...
In theory but some of the cheap hobby fabs, so this is all chi-cad, if you've got a little chi-cad project and you want it made, you send it off to China or somewhere like that and you get it back not actually that expensive.
I think those companies that are set up for those small batches, if you wanted... how do I word this?
If you wanted a dongle for this that was custom to automotive you might have to pay a few hundred pounds for that custom thing whereas you can get five of these for like a hundred quid or something like that.
So yeah, it's not nothing but actually if you're interested in this niche stuff I hope it's not too high a barrier to entry.
And if you're really interested in helping us out, I don't know. Some of these have made their way out of CoatThink into the wild so I don't know.
Okay, thank you very much for a very interesting talk.
So that concludes the open hardware dev room for 2024. As you make your way out, if you want mine looking down at the floor and if there's any paper or anything that moves, if you want mine picking it up, bringing it with you to one of the garbage cans, we also have a bag up here.
So anything that you find on the way out the door, I also wanted to very definitely thank the other folks who were critical in making this dev room happen. Ian and Clement and John and Wayne.
This is not an individual effort so thank you very, very much for all your help.