Yeah, hi, I'm Ludwig from SUSE. I'm a research engineer there working in the so-called future
technologies team and today I'm presenting some crazy idea. Not because we're going
to build a product with that, but just because we can. So, first let's take a look what's
the difference between package-based and image-based. I mean, I'm from the package-based world,
so I don't have too many insights into actual image-based systems like I'm embedded world,
so that's just my view. Anyway, packages are known by all of you, I guess, from your desktop
Linux installation, so you have individual components pre-built, the vendor ships your
packages and the client side decides which one to install, so there's some kind of dependency
resolver that takes the components, puts them on your local system. Usually you don't need
to reboot for that. This has advantages and disadvantages, so the advantages, if you need
a new VIM version, you just get it and it works right away. The disadvantages, if it breaks,
it's broken. On image-based systems, on the other hand, you get a full Linux system pre-built by
the operating system vendor. That means it's a ready-made file system that is typically downloaded
and DD to some partition, for example, or it's a table that gets extracted somewhere. Installing
that one or activating the install requires a reboot, so that's an advantage because it wouldn't
break into intermediate states, but it just either works or doesn't. The disadvantage is that it's
typically not extendable, at least not the original image, so you would have to have some other
mechanism like system desizics nowadays that plays some tricks to get extra layers on top. So,
let's first take a look at the file-stem layout of a typical image-based system, or one, I think,
our system, the envisions it. So, we have the operating system in slash user for Linux systems
after user merge. Then the EDC partition at the EDC volume is on slash, which is writable. The boot
partition nowadays should be the ESP, no matter where it's mounted actually, and your data is in
var on a separate volume again. So, how do you do updates? Usually you have separate partitions
for user, so the standard case is AP, so two versions, and then you DD your operating system
into one, while the other one runs, and on reboot you just switch. That's quite an easy
technology because it's just a regular partition table. It's actually read-only if the file-stem
you use is a read-only one. You could use our single CAsync to download deltas and getting it
from the server, and the signing is also pretty easy because it's one image, so you can put some
GPG signature on it, and that's it, and you can even verify it afterwards because the image is
unmodified on your partition. Disadvantages, there's no deduplication, so you always consume
twice the space basically, or however big your user partition is. Even if your updates are small,
you still need all that space. The amount is limited, so usually if you have only two partitions,
you have two versions of the operating system, and the space is pre-allocated. Again, that could
be an advantage because there's no surprises, the space is just there, and if it's there,
you can put the image, period. The disadvantages, your image can't grow, and the updates are always
of a fixed size basically. So I can be optimized that. You could actually use butterfaces to our
operating system, that's how the micro-S works, some more details on that later in Ignite's talk.
Anyway, we use a sub-volume for the operating system. That means you get the copy and write
semantics automatically, so deduplication means your updated system does not automatically need
twice the space, but only the changed amount. You can still use rsync or casync to only apply
deltas. The amount of versions you can store is basically infinite, only depends on how big your
updates are. So if you only have updates on text files, it could be a lot of versions. Disadvantages,
it's not really read-only, it's just a butterfaces flag, and that can be changed of course. Also,
put a question mark on verification. So previously with image, we could just run GPG for example,
and you can verify whether the image was modified or not. Here we have to take a look how to solve
that later. But how do distributions actually build those images? At least an operating system
vendor like OpenSUSE would use packages to build the image, just on server side. We learned that
we can even use this technology for building any of these, and the way the image would be shipped
would be just install packages somewhere on a server system and then throw away everything that is
not in user. That means all the scriptlets that run in packages are just modifying something that's
not relevant. Like in EDC and VAR, it doesn't make any sense to have a scriptlet doing something
there. So when, for example, a package needs to add a user, it can't just call user add. It needs to
use SystemDISUS users. Same if you don't enable a service, you can't just call SystemControlEnable,
you need to ship presets. And that way packages also can't just put the kernel in slash boot
anymore because that's not in user. That means there needs to be some extra tool that somehow
makes a system bootable when there's a new kernel, for example. So back to verification. Actually
packages, at least RPM packages, have signed headers, and the headers have check sums for each
file. So in the end, an image is a list of RPMs with signed headers, and by verifying each header,
you can also verify each file. So in the end, you have a tree that could be verified, you can
check that there's no file added, no file removed, and no file modified, just by looking at the
RPM headers. Disadvantages and images that you ship, you typically remove the RPM database
because it's this ugly binary blob. Even if it's a SQLite database, it's still an ugly binary
blob with more binary blobs in there. So that's why people really hate having the RPM database
there. This is something that nobody wants to see in an image. So how can we fix that? We could
actually store the RPM headers as files. So we just dump the header part of the package into a
directory. That means the directory is the RPM database that looks much less ugly. Different
two file directory listings to actually see which packages could update it. That is quite useful
if you already use Microsoft, for example, and do some snapper compare, then it tells you this
RPM database and this RPM database change, but you don't know what. If the database is a listing
of files, it's naturally visible what changed. And still we have the RPM header, so users fully
verifiable because they're signed. So the question now is, what do we actually need an image for?
So you don't need to take those RPMs, put them into some file system, and then download the whole
file system or this image or table. You could actually define an image as a text file that lists
RPMs. And then you download those RPMs, which are bits of your image, and just put them into this
file system or partition that is user. Disadvantage of this method again would be that with nowadays
RPMs you would lose the ability to do easy deltas because the payload is actually a CPO that is
compressed with some compression. So if you don't want to use delta RPMs or other fancy things,
you need to find a solution for the payload of RPMs. So the payload doesn't actually have to be a
CPO that is compressed. It could be actually completely uncompressed. So you just concatenate
all the file contents at the end of the RPM header. And because RPM header contains also the file
sizes, you know which file is where. Now if we do another trick and align those file datas to
page size, you actually get reflinkable packages. That means you could download this uncompressed
RPM, for example by means of CAsync, which would compress it actually on the server side. Then
you have the RPM on disk. And then instead of copying the payload to some other location,
you just use reflinks as a file system feature that reuses the blocks. So you have this one big
chunk of data as the RPM. And to create your actual files, you just link the data into there.
That means user bin bash is not actually a copy, it's just sharing the same data that is in this
RPM that is stored there, which conveniently is at the same time your RPM database. So it's not
just the headers that are in this directory, it is the full package. So in the end, in this example,
I put userLibsusImagePackages. UserLibsusImagesPackages would be your image. That means users just
a view. And if you would map this into ButterFS, like in this example, that means you have several
versions of this image directory as a snapshot. And then you could create other ButterFS volumes
that just link into those RPM headers. Or you could even omit this user completely, a colleague
of mine Fabian, even wrote a fuse plugin that just creates a file system from looking at RPM,
RPM's in the directory. Quite crazy. So to summarize, we could build an image-like system by
leveraging ButterFS. So instead of using AP partitions, we just use snapshots. The behavior
would be exactly the same because you prepare this new snapshot, put all your data in there,
and then you have to reboot to activate it. But since it's still packages, you retain the
flexibility to actually change it on client side. You could ship the image as a list of RPM's on
text file, but you could also add RPM's locally in this directory, and you have them installed.
So best of both worlds. And this is not just completely crazy in my head. I actually built
a prototype that kind of works. So it uses busybox because it was easy to modify the RPM
implementation in there, to work with those raffle link in the packages. This is our sync for
updates, and it uses SystemD's kernel installed to make this system actually bootable. So you
can try it out. There's also pull request open for RPM, I think, to have this raffle linkable
stuff and send a patch to busybox, but I don't expect it to be accepted. It's just a proof of
concept. Of course, to make this work in practice, there's lots of to-do's. So in existing distros
like OpenSUSE, we have to fix all the packages to no longer use scriptlets. We need to talk about
the butterfly sub volumes. The naming should be standardized. There was actually a discussion
like two years ago already on the list. RPM raffle link payload would be nice to have upstream.
There are other stakeholders that also would like to have that. I'm already working on SystemD's
kernel installed to make it usable for this use case. In case of micro-as-like systems,
we want to have roll-baggles of slash and just the operating system. For installing deltas, I would
like to use the async, which we revisited. And last but not least, all of that should be native
in RPM or SIP, and in my opinion, not just some extra tool. And that's it already. So any questions
for this crazy stuff? Okay, me first. So how did you handle the timestamp AC? Because if you're
using BTRFS and doing an RPM minus I, you get the A time, M time, and C time in the i-node. And
an image-based system is supposed to hash end to end the same. And if it has different timestamps,
it doesn't hash end to end. Yeah, well, you can't modify the timestamps. You can't modify. You can
only look at the ones that are under your control. But then doesn't that mean that effectively we
can't use this to reconstruct image-based systems? Well, it's not a bit-to-bit identical. What is
on disk is not bit-to-bit identical, of course. Only the actual RPM payload. But you know that
the payload that is linked is actually the same. So I don't think, I mean, maybe there's a use case
why the timestamps need to be exact. But in my view, it's not important because the, like,
user bin bash is user bin bash, no matter what the timestamp is. Well, the use case is just for
image-based systems. The end-to-end hash tells you that you've done the right thing. And it's
simple to compute. With your system, you basically have to do a tree hash down all the packages
to prove that this is what you're supposed to have installed. It's, semantically, they are
equivalent. It's just the latter is more difficult to do than the former, which is why people like
image-based. I mean, you only need to hash the directory with all the RPMs in them, right? So
if the RPMs, you have to check some of all the RPMs, and then you can verify the RPMs. Yeah,
I'm not disputing. Exactly. You're just saying it's more complicated. Yeah, it's, of course,
there's always a trade-off. Of course, yeah, yeah. And depends on how you construct this user view.
I mean, if it's really a butterfly, it's a real file system tree, then you have to walk it. But if
it's only a view, like with a view stingy, then it doesn't actually exist on disk. It's just,
you know, looking at the RPMs. So it cannot be modified. You need to walk, don't need to walk
the tree. You just hash the RPMs. Okay. Yeah. So how would you integrate that into what we have
heard in the previous talks? So during boot, I specify something similar to my dm-varity root
hash, so that I know that I'm actually booting from an unmodified root of s. That's a good question.
That problem is not solved yet. Yeah. So far, the challenges are already at the point, how do you
actually specify which version of the user tree do you want to boot. But now all the models assume
that if you ship a new image, like a new user version, it also comes with a new kernel and a
new init.d. So this init.d knows what disk to boot. But in the Butterfest use case, there is
a kernel that can have a number of init.d.s and those init.d.s match with a number of snapshots
that they can actually boot. So then I'm already struggling with this part. So the verification
comes later. So dm-varity gives you authenticity of the blocks at runtime. So the device cannot
switch them underneath. And I guess that if you verify the kernel headers, the RPM headers,
I don't know, when loading the image, this wouldn't give you the same runtime properties.
I haven't played with those technologies yet, to be honest. So another interesting thing would be
this FA policy daemon. Only with a bit about it, it uses the audit subsystem to actually block access
to modified files by comparing them with the information in the RPM header. So
it would be another area to just explore how to integrate some verification technologies into
this model. Any more questions? If not, then I guess we'll wrap up this talk. Thank you very much.