Next, we have the Jump Starter project with Miguel and Ricardo.
This is going to be a rather interesting bit of open harbor for those of you who don't
know.
Miguel has a long time contributed to the KECAD project, so an alumnus if you will.
So welcome.
Thank you.
Hello.
Hello.
Yep.
Thank you very much for attending this session.
My name is Ricardo Nariega.
I work in the office of the CTO at Red Hat.
And I'm Miguel.
Yeah, I worked with him since one year and a half ago.
And we are going to talk about the Jump Starter project.
We will go through these slides and hopefully a live demo as well.
So let me introduce you to PNAT.
This is PNAT developer.
PNAT works developing applications for embedded systems and edge computing use cases.
And he uses all the more tools of development that we know.
He develops locally in his laptop, pushing code to a give repository to have version
control.
He uses IDs, testing frameworks, virtualization containers.
So basically all the other tools that we use for developing services in the cloud.
The problem is that PNAT, after some hours of coding, he really needs to test a release
candidate or some code that he thinks is ready to test in the real target platform.
Let's say he uses an NVIDIA Jetson device, for example.
The problem with this is that he needs to take the power adapter connected to the Jetson
device to the plug, then an Ethernet cable maybe to get some connection, then an HDMI
cable or a serial cable.
Then at the end he goes, takes a USB stick, puts some operating system image in the USB,
install it in the device, and at the end he needs to take the application to the device
somehow via SSH or whatever.
So by the end of the day, the poor PNAT is completely exhausted.
And this is when we start to think, okay, we need to tackle this.
We cannot afford every day doing the same thing.
And this is where Jumpstart came to life.
We see that developing applications for embedded devices comes with unique challenges.
There's a huge lack of standardization.
Every device is different.
We see in big companies that enrolling these devices into CI systems is rare or sometimes
is very expensive.
We want to keep high quality in our code, in our applications.
And testing, especially automated testing, is a key aspect of it.
So we thought, okay, what are our testing goals?
We would like to test our application in those target platforms at every pull request that
we push into the repository or for every merge request.
We would like to test a release candidate in all the models of the platform that we
are going to run in production.
Let's say a point of sales, I have five or ten different models, so I want to test my
application in all of them.
So we need some kind of automated testing and if possible, something that is hands-free,
so no manual intervention.
And this is why we created Jumpstart.
It's not a device management system, it's basically a testing tool.
So what is it?
I know this is the open hardware room, but Jumpstart is basically a software project.
It's written in Golan and it has the concept of devices, which are the devices under test,
embedded systems that we want to test our software on, and the concept of driver.
Driver basically exposes the capabilities of a hardware connector.
We will explain more later, but a hardware connector is a piece of hardware that allows
you to enroll these embedded systems into CI platforms.
We have built a script language based in Yaml and allows you to automate some of the
onboarding process.
And Jumpstart allows you to remotely control these systems and it has the following functionality
like power management, control signal management, storage, and console management.
It works with the major CI platforms like GitLab runners, GitHub Actions, Jenkins, TecTon
pipelines.
We are developing as well a Kubernetes device plugin to be able to schedule these TecTon
pipelines in the Kubernetes nodes.
And at the bottom of the slide, you can see when you use the Jumpstart CLI how you can
list the devices that are connected.
So as an example of GitHub Actions, if you want to enroll your embedded devices into
GitHub, you just need to run a self-hosted runner service per available device, per the
device that you want to run.
Then you can add a tag like Jumpstart.vrasperepy for.
And whenever you want to run a job, you just select which tag or which platform you want
to run it on and it should work.
We have created a reference design for a driver, for this hardware connector that I mentioned
before.
We call it Datlink.
And Miguel will explain more later.
But if you see Datlink, you just need to connect it via USB to the GitHub runner and then create
your workflow, your GitHub Actions workflow, and run it.
This is an example of the GitHub Action workflow.
For example, list a device, download an operating system image, prepare the image, mount it,
change some configuration, inject some application and ready to use.
And then we can use the scripting system language that we have created to automate the onboarding
of the device.
Just a disclaimer, if you use, for example, GitHub Actions, it's better to change the default
settings because for first, Jumpstart requires full root access to the runner.
So if someone has privileges to run, to push a PR, it can compromise the system.
So this is how the script language looks like.
You can put a name to the script, a selector for the target platform, and then a set of
steps that would automate the onboarding.
Power off on, write image to disk, and then we can control also the console.
We'll see that in action later.
So as I said, we have designed Jumpstart with modularity in mind, with a driver-based model.
The Datlink is our reference design, but we have also developed other kind of hardware
connectors just to show you how easy it could be.
And if there are other hardware that you can use to enable this, please write the driver
and you can leverage all the benefits from Jumpstart.
So we have the Datlink driver.
Driver B could be done the same with an SD card multiplexer, plus a smart plug, plus
a serial cable.
So, yes, as Ricardo explained, when we started the project, we didn't find a proper test
harness.
Along the way, we found some others, and we will be adding drivers for those.
And if you have something and you want to add the drivers, I'm super happy to help.
This is what, it's very obvious, at least now it's not pink, like in this morning we
had an issue.
So what our test harness is doing is switching a storage device between the testing host
and the device.
So you can access the storage device, the iOSV, from the testing host and write your image
very, very, very fast.
And then you can connect it back to your device and power it on, and then you can talk to
the device via the console and we have some control pins.
So far it's very basic control.
There is no analog interfaces.
But we have the next revisions.
We have taken a lot of feedback to add extensibility to the platform.
So yeah, this is how the version 1.1 looks.
We did it in a mini ITX form factor, so you could put it in racks in a data center or
in boxes, like in this case, the one we brought.
You can control power via a barrel connector.
So in the back plane, you have the inputs for power, and here down you have the outputs.
You can put your storage device in here and you can mount your device under test on top
if it fits.
So you can control up to five amps, and you can provide the power via USB PD.
Yeah, so we have, yeah, as I said, the USB storage multiplexing, and this is how it looks
if you mount something on top.
This is star 5 vision 2, and yeah, we are running some tests with that.
And then, one of the best features of this is the speed.
So you can get five gigabits per second, so you can go to a little bit of speed.
And it makes it very interactive.
When you are working, you get feedback very quickly on if things are working or not.
So that's really nice.
About the hardware, so the design is made with kick-up.
You have the repository in here, and here the pollution of our prototypes.
So we made 100 before, I mean, last summer, and yeah, it was around $80 per device, and
we just made five, I think.
Then we made 1.1, and we added some additional EMC filtering to the power.
We moved the storage device inside because initially it was outside, and it was okay,
yeah.
And we added some connectors for the expansibility, so we have an SPI and H2C connector, so you
need like a doter board to talk something specific to your device.
You can do that.
We have version 2.00, which we could not produce yet because of company policies.
If I want to make an order to make the prototypes, it's going to be beyond the maximum without
the purchase order, so I need to register the vendor and so on, and it's complicated, so
eventually.
But that one, instead of requiring two USB connections, which this one needs one for
control and one for the storage, we'll only have one.
I think I have a picture here, yes.
So in this one, so the connection from the testing host comes here.
It comes to this USB 3.1.5, and you can connect additional devices.
Maybe you need to put a camera, or maybe you need to put a logic analyzer, or a Canvas
adapter, so you can do it via USB, and with the software we could detect via the USB topology
where those devices belong.
So the idea is that you could have a testing host, but you can have 10 jump starters.
That links, so we changed the name at some point to make it clear what the software and
the hardware was.
And yeah, we also added a connector for APX.
Yeah, I will run a little bit more so I can make the demo.
That link board has a controller chip, and the firmware is written in Rust.
It has a nice console that you can talk to if you want to do it manually, but that's
handled by the driver in Jump Starter.
For people who make hardware with USB, this project is super interesting.
You have it in almost every Linux distribution, and it allows you to update your firmware
on the field.
So you can publish your firmware to firmware update, and then you can, I mean, you create
the descriptors, and so on.
Fingert update in the Linux systems will realize that you have a device that is on
the database of firmware, and you can get updates through the network.
This is how it looks.
Suddenly, I couldn't take one of Jump Starter, but this is how it looks, for example, if
you're running on desktop, or if you do it on the console, you see something like this.
So, yeah, we're releasing every version in GitHub with all the production files, so you
can just download the production files and take them somewhere, and hopefully, I mean,
probably you will need to adapt to the vendor, but you can get that.
And, yeah, Cid asked me if I could talk about them.
We are talking with them to see if they can pre-make an amount of devices and make them
available in their co-create program.
Normally that is meant for, I mean, if you are a creator and you want to make money on
your device, there are other programs like this.
They will handle the production, and you just take care about your design.
But what we did in the meanwhile, I don't know if that will work or not, we just provided
the links to the, how do they call it, the Fusion Gallery.
So, when they made the prototypes, they give you a link that you can share and they will
repeat the prototypes for others.
And now, hopefully, small demo time.
So, okay.
So, this is, we prepare this demo repository that is actually connected to this.
This is registered as a runner on that GitHub.
I don't know, hopefully, it's connected, we'll see.
And this is the device under test that we have.
This is Raspberry Pi 4.
And we are building an image and testing it on the device with two different distributions.
So, yeah, this is some of the previous runs that all passed.
We can look at them and we can see, for example, how do I see that?
Checks.
We can see that they were tested with Raspberry Unlight and Fedora Rock Height.
So, in the process, it will download the latest version of the image, prepare the image and
test it in the hardware.
So, we go to one of those.
You can see the steps of what happened.
Okay, this one was a simpler test.
Yeah, we can see previous runs.
And we can see an example here of, okay, what happens if I break the construction of my
image?
In this case, we are testing a DPM module that is connected to the Raspberry Pi.
So, if I remove the DTV overlay in the config for the Raspberry Pi, it should not work.
So, when we go to the checks, we see, okay, Raspberry Unlight stopped working and we can
see that it failed at the DPM interactions.
We can see, yeah, when the image was being flashed, it was not working.
Into the device.
And I could show you if this is all working.
Maybe I need to make a bigger font size.
In this case, this is what the runner is calling.
I can list the devices or I can run stuff on them.
I can do things manually.
For example, I can power on the device.
Hopefully, I need to tell which device I want to power on.
So you can, and if it's working, yeah, power on.
You can power it off.
You can request a console or you can run the scripts that we run in CI.
For example, if I go with this other device, which is an SD wire, I don't know if any of
you are familiar with this.
It has an SD card and then a connector that looks like an SD card so you can plug it into
a device that boots up the SD card.
So if you connect these to the jump starter to the software, you can see that it's working.
And I provide, it also needs like a serial console to talk to the device, otherwise it's
going to complain.
So if I list the devices, I should see, okay, I have the SD wire with this serial number.
In this case, I cannot make all the associations with the tags and so on, which are stored
in the hardware.
So I have a config file that matches the serial number and then at this point I can just
flash one or the other and it's set disk image.
For example, if I set the Raspberry Pi 4.
So this is the same process that you can do in the scripts, but you can also do it manually.
For this, I need privileges.
We want to split this part of the executable in a separate one only for that purpose with
lots of filters to make sure that it will not break anybody's server.
Yeah, this is the nicest part.
How quickly those.
Yeah, we need to be a little bit cautious with the data.
Linux, because even if you request the system to eject the device, sometimes it tells you,
okay, everything is all right, but the cache is still finishing in some part of the subsystem.
And yeah, I think we're, okay.
Thank you.
Thank you.
Thank you.
Thank you.
From the software side of things, have you looked into LabGrid?
No, I have not.
But, sorry.
So yeah, the question is if I have looked at LabGrid and not, but I will.
Yeah, because it does something also similar.
Is it open source and open?
Maybe we should work with them.
Hi, thanks for the talk.
I would have asked for LabGrid too, but maybe one other thing.
Do you know about the automated testing conference call, the monthly one, that's coming out of the
Elinux project?
There are already record people there talking about the Coliseum stuff.
Maybe that's interesting to share there too.
Yeah, so what's the name of the?
It's the automated testing conference call around the Elinux project.
Yeah, I can come to the front end.
Okay, yeah, thank you.
That's great.
Yeah, that happens sometimes.
Big companies, you have people working on things from different places.
Thanks for the talk.
I wanted to ask how do you actually specify tests?
How does the test work with this?
Do you have the device?
Yeah, so.
Is that the YAML syntax thing?
Yeah, exactly.
That is the YAML syntax.
So far, it's rather simple.
So for example, if I go, and this is available on the repository, if I go here to the demo, and I go to the
Raspberry Unlight, for example, you can see this test, TPM, or latest raw image.
So we assume that the image is already built, and we just tested on the image.
And it's just a series of steps so far, steps, interactions with the CDL console.
So we expect something like that for control.
And yeah, we want to add also integration maybe for other types of devices that are not Linux based,
maybe within where you want to flash them.
Something that I did not explain is that we also do power metering.
So one of the things that we want to do is provide that report, maybe from this point to this point.
Okay, how many millibars, what's our I consume?
So you can check if your software is consuming more or less in your hardware.
So really cool project.
I'm really excited.
So you said there's an external USB for connecting modules.
So is that for so I can test something externally like, let's say a very simple system where I've got,
I'm turning on a light switch, I can plug in something with a Luxe beta and check that the light actually came on rather than the board went,
yeah, that came on and I have no idea.
Yeah, yeah, that is the idea of the version to that.
Yeah, you could connect anything and have a way to associate the device under test with those devices because you know where they are in the bus.
My question is a similar question because there's, you know, literally thousands of pieces of various test equipment out there.
Most of them these days probably run on either a network interface or a USB interface and they're like, yeah, they're classic.
They've been translated from the old HP IB GP IB, skipping commands.
But you know, things that are like really powerful test tools is, is there any plan to integrate be able to integrate stuff like that because maybe I have an analog board that's really highly precision.
And I need like a really highly precise meter, you know, digital multimeter to measure that I'm not going to measure that with a simple weight bit a to D.
Is there any any plans on being able to integrate stuff like that so you can.
We want, I mean, we want to be able to enable that but every tool is different.
So we want to make it as modular as possible.
We haven't still thought exactly how to how to do that.
But the first thing that I guess we need to figure out which which USB devices are related to that or maybe other config file in the system.
So we have a lot of consistency saying, OK, this serial number has these two and these two and these two associated.
And then when you call the software that talks to that tool at some point in your script, you can you can talk talk to that.
There is even more interesting stuff like sometimes you need to test different parts of your system in parallel.
Maybe it's not one hardware piece. You have them and they need to talk together.
So at some point we want to be able to to run in parallel multiple devices and the rest.
But yeah, let's see how far can can we get.
OK, and thank you so much for your presentation.
My question was around the jammer files and the specification about although it is not available now the canvas communications and the flex ray and other protocols.
If do you plan or you have a roadmap in order to put it into your specification and the way of sending standard ones in order that OK, I want to get profit on my job and of the opens of the tool.
But I want to create a new hardware, but I want to use the same protocols as yours in order to change for your second version that you are developing into one year.
But I need to I don't know if you have planned into your role in order to something.
It's not on the roadmap. But yeah, one of the things that is doing and probably one of the reasons why it's getting into the embedded space is automakers.
And yeah, they I mean they use can buses or automotive other med.
Yeah, so at some point, yeah, we will need to figure out how to do that is we haven't thought about it, but yeah.
OK, thank you very much for the presentation.
Thank you for having us.
You