Yeah, thanks. Thanks for introducing me. Hello, a warm welcome also from my side to
first them to the embedded froom. And we're going to hear something today about Delta
like streaming of encrypted over the air updates for Rao. So luckily I managed to put the entire
abstract of the presentation already in the title. So what we hear about is the changes and
developments during the roughly two or three years that happened in Rao, the Rao updating
framework. And so it's basically the development that happened since we've last met here, I guess.
So short notes about me. My name is Sandvik. I'm an embedded software developer. I work at
Pangotronics. I'm the team lead of the integration team at Pangotronics, and I'm the co-maintenor
of the update framework Rao that we will hear more about soon. Pangotronics for those who don't
know, it is a company based in Germany and we provide professional embedded Linux consulting
and support and work closely together with the community. And with since the beginning, I think
more than 7,000 patches in the Linux kernel. So a short overview of what we hear today. So the
first thing is a short introduction into what Rao is for those who are not that familiar with,
but very, very short. Then we talk about the bundle format because this is crucial development for
or the base for all the further features that are listed here. So the first thing I will talk
about then is bundle streaming. Then we will hear about adaptive or delta-like updates, how to
encrypt our bundle, give a short outlook on recent development about app updates, and at the end,
we have a short look into what's coming next on features and what's in the ecosystem. So, yeah,
a typical over-the-air field update scenario could look like this. We have here our server. The
server builds the image that we want to deploy to the target. We create an update artifact from
it, sign it, upload it to our deployment infrastructure, and then we have the individual
targets, update targets here that download the update and install it. And there's also still
this conventional not-so-over-the-air use case for, for example, using a USB stick. So what Rao
handles is basically two parts. The first one is the creation of the update artifacts, the signing,
verification and so on, and the actual installation, the failsafe installation of the updates on the
target. So, yeah, basically Raoq is an embedded Linux update framework, so it handles the failsafe
and atomic update of AB systems, so redundant system where you have one partition, where you're
running from an inactive partition, and when you update, you write your update into the inactive
partition. Once you're done, you switch in the bootloader to the inactive partition reboot,
and everything is fine. Raoq is basically two parts on the target. It's the service that handles
the update that runs and installed there, and it gets its view on the system from the system
configuration file. And, yeah, the artifact for updating we call in Raoq a bundle. A bundle
consists of the images that should be installed. It consists of additional hooks or something like
this, and a manifest that holds the description, yeah, what these images are for, basically. So,
it's written in C with some utility libraries to not reinvent the wheel for everything. It's
licensed on the LGPL and hosted on GitHub. It was started in 2015, and I think the first
release was in 2017. So, yeah, as I already mentioned, the bundle format is quite essential
for the next things that we talk about here. So, let's first of all have a short look at the
initial bundle format, because this was a motivation for changing the bundle format then. The initial
bundle format was quite straightforward. It was just all the artifacts and the manifest packed
together in a squash file system. We signed this squash file system and append the signature to the
end of the bundle. So, the verification is also quite easy. We just have to read the entire
bundle and have to read the signature to be able to verify the bundle. So, yeah, this is also the
downside. Even if we don't also just want to access the manifest, we have to always authenticate
or read the entire bundle. So, this is quite slow, and if it comes to over-the-air updating,
it requires us to always download the full bundle before we can access any data in this. So, yeah,
this is bad if we want to use it for streaming. So, this is why we have introduced in 2020 a new
bundle format. And this bundle format is basically, it's called the variety format, and it uses the
DM variety. So, short intro, a device mapper system in Linux is a generic abstraction of, yeah,
manipulating block devices. So, a device mapper has the same API as the block device has. So,
for the upper layer, it looks like it's just talking to a block device. But below, it can
manipulate the block device, authenticate data, it can merge data together like we know from
DM linear, and there are several device mappers in the kernel. So, the one that we talk about here
is DM variety. It is basically an integrity protection method for read-only block devices. So,
the rough concept is that you split the block image into several chunks and generate a hash for
each. And of these, you recursively do this again and again until you have a single root hash, and
then you can verify each single block until the root hash recursively. So, yeah, you have the data
protection or integrity protection for read-only files. The variety table is just appended to the
image. So, let's see how we use this in the RAUK bundle. So, we take our images here and first
create the variety hash and the root hash. The variety hash is simply appended to the bundle,
and the root hash is now placed in the manifest. And then, we just sign the manifest with an
enveloping signature, which means that the manifest is a payload of the signature. And what this
gives us is now the verification of the manifest is quite easy. We just have to verify the manifest
or the signature and get the manifest content. And inside the manifest, there's also the root
hash. This is then automatically trusted if we have verified the authentication of the manifest.
And then, we can set up the invariity and use the hash tree appended to the manifest and the
authenticated root hash. And then, for each access to each chunk or block on the block device,
this is authenticated to the invariity in the kernel. And this allows you to have fully authenticated
random access to your bundle. And you also, you only need to verify by the time of using the
data. So, the next logical consequence is to implement streaming. So, up to now, RAUK was not
so over the air. So, downloading means that we assume there is an external service like
Hockbird or an application or an SCP that downloads the RAUK bundle to the target device. And then,
with RAUK, we start installing it from the local storage. Well, the disadvantage of this is obvious.
We have to have some extra space on the target where we can store the bundle. And the artifacts can,
yeah, in a modern system become quite huge. And so, the approach is that we implement streaming
or downloading in RAUK itself. And if RAUK is able to do this and directly download it to the
target device that we update, then no intermediate storage would be required. So, let's have a look
how this is realized in RAUK. So, first of all, what we do in RAUK is that we spawn an, or fork
an unprivileged helper process because RAUK, yeah, runs as root as it has to update the system.
And you really don't want to use a root service to download data from the internet. So, it spawns
an unprivileged helper. And this helper acts as a translation. It plays a block device on one
side and talks to the update server via HTTPS range request on the other side. And, yeah,
HTTPS range request should be supported by all common web service, also light TTP services,
and it's also supported by many delivery networks. And if we combine this now with what we've seen
with the access to a variety bundle, then we have fully authenticated random access to the
remote bundle. And, yeah, we can randomly access so no intermediate storage is required. So,
the next need when we are able to download things is normally that we want to save download
bandwidth because bandwidth is limited, expansive, or something. And the normal approach for this is
to do conventional data updates. It means you have two versions of your image on your host
system, calculate a delta, and then you perform the update with this delta image on the target. So,
if you have the exact version that you have to calculate the delta for on your target,
this works very well. You can go here from version two to the target to version three. But,
if you now have a system that is on a different version, yeah, this fails because it simply
doesn't apply. So, it's an optimal diff. It allows very small updates. But, yeah, you require to
have access to the different image versions on the host, and you only can update step-by-step. So,
from version one to version two to version three. So, in route we've chosen a different approach,
a more generic approach for optimizing download. This is called adaptive updates. The concept
behind this is that the bundle or the manifest itself provides a number of optimization options.
So, with each option, there's normally an additional data connected that is stored in the
device for optimizing the download. But, since we are able to stream the bundle, we don't have to
download these additional data that is stored in the manifest. And then, it's a responsibility of
the route service on the target to see, okay, which of these capabilities do I support and which
can I use and which is the best one. And, there's always a fallback to use a full bundle download.
So, you're always able to download the image you want to install. One method, adaptive method,
generic one is the hash index. The idea behind this is that you split your image into several
chunks and hash each chunk and generate a hash list from this. And, for installation, you just
basically do the same on the target. You take your target device, block device, for example,
you hash it with the same algorithm, create the same hash index. And then, for the optimization,
yeah, you just download, first of all, the hash index that is stored in the bundle. And then,
you compare it line by line with the hashes that you've calculated on the target. And then,
you can download or just need to download the hashes that differ between what's on your target
and what's in the bundle. And this works both for the intended target version, but also,
if you come from a fully different image, then you just have to download a bit more,
because the hashes that differ are a bit more. For block devices, this is already implemented
in the current drug version. And there are also plans to support this for file-based updates
using R-Sync and offline generated checksum files. The next topic is bundle encryption. So,
the motivation is, I think, quite clear. You will have some sensitive data in your bundle,
and you want to protect it, because you have it on an unsafe cloud storage or an unsafe
communication channel. So, in Raoq, we have implemented this in two-stage approach. So,
the first one is a symmetric encryption of only the payload. This is this part. This is what
normally already the build server does. And this does not yet require access to the key material.
And the second part is the individual encryption. Then you can take the symmetrically encrypted
image and encrypted per recipient. You can just take one key and encrypt it for all your devices
by using a shared key, or if you really want to do security, then you can also use per device
or per recipient keys and encrypt the bundle for many individual recipients, many thousands.
So, this again uses a device mapper, a different device mapper. Now we use DMCrypt. It's also
quite simple. For the generation of the DMCrypt image or the image we use for DMCrypt, we just
split up the original image into equal sized chunks, generate random symmetric key, and encrypt
each block individual. And the DMCrypt device mapper then just provides a transparent description
of the images. So, if we access a chunk there, then, yeah, DMCrypt just decrypts this chunk
we just selected with the key, with the symmetric key, which is the same used for encrypting.
And if we combine this now in the bundle, so we have here the image encrypted and combined
it with DMVarity, then we have a blockwise authenticated description. And since we have
random access to the device mapper and the variety format, we also have the possibility
to stream an encrypted update. So, short on time, a few notes about app updates. So far
in route we assumed, okay, the application is normally the application. So, we assumed
a bit a monotolic system where the application is the one thing that the device should do.
And so we said, okay, the application is normally either part of the root file system
or you can have it in a separate slot. But it actually, anyway, linked against the libraries
that are contained in the root file system. So, it's fine to install it always together
with the updated root file system. The reality showed it's a bit different and there are
more and more demands of having the capability of doing container updates, doing app store-like
updates and where you also have one vendor which provides a base system, which is rarely
updated and other vendors provide the applications, which are much more frequently updated and
additional data should be added there. And up to now we had no solution for this in route
and said, okay, then use route for the base system and use another updater or update approach
for this application or file updates. What we are working on, and this is in a quite premature
state, actually, is route artifact updates. The basic concept behind this is that you
have a slot for artifacts and inside the slots we don't do image-based updates, what we do
directly are file-based updates. And then we provide the same as we do for image-based
updates. We ensure that the update is atomically and we also support both the case where we
don't have any dependency of the app of the container to the base system, so this is what
you basically see here. But we also support the use case of having a dependency on the
root file system but the need to more frequently and independently from the root file system
update your application. And together with our checks and files, the idea is that this
again also supports streaming and delta-like updates.
So just a very quick rest through the other features and community things. We've switched
to Mason build system recently. This is already merged. It wasn't when I started the slides.
So a new feature we also have is adding custom metadata in the manifest that you can then
access via route info or the deepest API for custom application. And an ongoing development
is also about providing more fine-grained process because currently we just have a per
slot progress and if you have a large tar then you wait very long until the progress gets
to the next step. And a contribution that came or was started by the community is the
Rooke Hockpit update. This is basically an interface between the Hockpit deployment server
and Rooke on the other side. It talks via the deepest API with Rooke and this is a good
example where the community started things and they moved then to the Rooke organization
and are now maintained by the Rooke community. And with the latest version of Rooke Hockpit
update we are also compatible with using streaming updates for Hockpit. And shout out
to Leon who is sitting somewhere here in the room. The Meteorite community is a layer or
layer collection started by Leon which provides some example integration of Rooke into, for
example, QEMO or for Raspberry Pi and it's a very good starting point if you want to
check out how to use Rooke, how to use all the features in Rooke. And yeah, I really
recommend you to use this as a starting point. A final slide. For an open source project
it's always hard to know which are the users of your project and where it's actually used.
So it's always interesting for us to know this. One example where we came aware of that
Rooke has used is a very famous one. It's a well-steamed deck that uses Rooke together
with the async. Another example is the home-assisted operating system that uses Rooke for updating
the PANY system and the Ornero Eclipse project. And one thing that I also find very interesting
is that the infotainment or information panels on the German ICE trains have a custom distribution
they call Linux for ICEs and they also use Rooke for updating the systems. So this was
very quick. Thank you for attending. I think we still have two or three more questions.
Yeah, I think we have a time for one or two questions.
Yeah.
Hi there. Thank you for that. That's absolutely intriguing, really interesting. So one of
the questions was how do I plug this into BitBake and you've answered that. That's great. I
know what to do when I get home. The other was what's the granularity of this? I saw
a sort of a 4K block size in there somewhere. In terms of your hashes and then downloading
blocks through the streaming process, is that 4K increments? How does that change? And what's
the overhead in verifying those hashes as you download? What's the impact on performance
and have you looked at any figures for that?
Getting quite low. So the question was if the 4K is fine-brained enough for normal downloads,
so it's currently fixed, but it could also be changed if that's not sufficient. But in
the current approach, the 4K is a fixed size there.
Okay, so it's getting late, so unfortunately we don't have time for any more questions,
but don't hesitate to ask them in matrix chat or try to catch our speaker in the corridor.
I'll be in front of the room. You can ask questions and we can discuss there.
Thank you for a great talk.
Thank you very much.