Thank you.
So welcome everyone.
We are going to talk about fast disk images checksums.
So who am I? I'm a long time contributor to this of the project, and I worked for Red Hat
more than nine years on over storage. Who knows, obviously.
And I focused on incremental backup, image transfer, and NBD tools.
And this project is continuous, the work on fast disk checksum in Ovid that is available
for Ovid. So we're going to talk about why do we need disk image checksums, and why we
cannot use the standard tools for disk images.
And we'll introduce the blockchain command, which is optimized for disk images.
And the block cache library, which is used by the blockchain command.
And you can also use it in your program. And a good example of this usage is the new
checksum command in QMMA image, which is work in progress using the block cache library.
Then we see a live demo. We'll play with the tools.
And finally, the most important stuff, you can contribute to this project.
So what is the issue with disk images? Why the additional than standard images?
Let's say we want to upload a disk to, we have this QGAP2 image, usually it's compressed
because it's the best format to publish images.
And we want to upload it to Ovid, or maybe to OKD.
Maybe Simon wants to upload it to OKD, so what we have to do is get the data from the
disk image, and we need to upload it to some server on the virtualization system, and this
server we write in with some disk. Now the disk can be many things.
It can be just NFS server with, and we'll have there look a file similar to the file
we uploaded, but not the same. It can be raw sparse image, and it will not be compressed
while we have QGAP2 compressed image.
Or it can be a small block device if it's Ovid, it can be a small block device just large
enough to fit the guest data that we uploaded on some escasi server, or it can be SAS image
stored in many nodes, in a cluster, in many disks.
So we have very different shapes on the two sides, like disk image on one side, something
completely different on the other side, different format, different way of storage.
One thing must be the same. The bits must be the same. The guest data must be the same.
So if we start the guest with the disk image, or with the virtual disk, we must get the
same bits.
So we can verify this operation by creating a checksum of the disk image, the guest data
inside the disk image, and the guest data inside the virtual disk, whatever the format
and shape, and the checksum must be the same.
The logic image is mostly the same problem. We have some shape of disk on one side, some
shape of disk on the other side, different formats, but the guest data must be the same,
and the checksum must be the same.
Another interesting problem is incremental backup. In this case, the backup system will
only copy, only change blocks on each backup if it wants to be efficient.
So let's say two days ago we did a full backup, and we stored it on this full QR2, and this
is just one way that we can store the backups. It can be many other things.
And yesterday we did another backup, and we stored it in one file, which is sitting on
top of the full QR2, and this is the backing file of this file.
So we created a chain, and today we did another backup. We copied more data from the virtual
disk. We created another layer.
So also in this case, the guest data inside the virtual disk must be the same as the guest
data inside this entire chain. So if we know how to read the guest data from the entire
chain, like a guest does, we can create a checksum, and we can compare it to the checksum
of the virtual disk at the time of the backup, and we know if our backup is correct.
So if we will restore this chain to another virtual disk, we will get the same data.
So what is the issue with the standard tools? Can we use SHA-SUM to create a checksum of
this chain?
So first we have the issue of image format. Standard tools do not understand image format.
So if we have the raw image, everything is fine. But if we have QCAP2 image, which is
identical, and here I compare the images, QM image compare, which access the guest data
and compare it bit by bit, so the images are identical, but we get different checksum from
SHA-SUM. Image compression, even with the same format, both images are QCAP, but one
of the compressed will get different checksum, because the host clusters are compressed, and
SHA-SUM is looking at the host data, not at the guest data. Even if we have the same image
format without compression, everything is the same, right? I just converted one image
to the other image, and the images are identical, but we get different checksum. Why? I use
the dash w flag, and this allows an order rights. So the order of the cluster on the
host can be different. The guest data is the same. Finally, the issue of sparseness. Standard
tools do not understand sparseness, so here we have six gigabyte image, but only little
more than one gigabyte of data. But SHA-SUM must read the entire image, so it will read
this one gigabyte of data, complete a hash from this one gigabyte, and then read almost
five gigabyte of zeros, because anything which is not allocated is read the zeros. So it
must do a lot of work, which is pretty slow. For example, if we take a bigger image, here
I created 100 gigabyte image, but there is no data in this image. It's completely empty,
just a big role, and completing your checksum took more than three minutes. So do we really
want to use this for one terabyte image? It's not the best tool for this job. And let's
introduce the blockchain command, which is optimized for this case. So first it looks
just like the standard tools. If you know to use standard tools, you know to use it.
Just run it and you get the checksum. It understands your image format, so if you use identical
image, you will get the same checksum. Although the images are different. The size is different.
They do not look the same on the host. Of course, it supports compressed QCaul, because
it reads the QCaul image, which the compressor data, and it gets the actual guest data, so
we get the same checksum. And it also supports snapshot. So if I create a snapshot, here
I created a snapshot, this snapshot QCaul, on top of the Fedora35 image. Fedora35 is
the backing file of your snapshot. And if I compute a checksum of the snapshot, I actually
compute a snapshot of the guest data inside the entire chain, not of the tiny snapshot
file, which has no data yet. And we also support NVD URL. For example, if I start NVD server,
this QMNVD is NVD server. Here I started it, exposing this QCaul2 file using this NVD
URL. And if you compute a checksum with the URL, we access QMNVD. I will get the guest
data and compute a checksum. And actually, this is the way we always access images. Under
the hood, we always run QMNVD and use NVD URL internally to access images. This is the
reason it works. We also support reading for pipe, like the standard tools, but in this
case, we cannot support any format, just raw format. And this is less efficient, because
we must read the entire image. In other cases, we can read only the data. But it works. So
it's not enough to create tools to get correct snapshot checksums. We want it to be much
faster than a standard tool, because we are dealing with huge images, which are usually
empty. Usually, when you start to use an image, it's completely empty. Then you install operating
system, add some files. Everything starts really empty, and then goes. So here we tested
this 6-gigabyte image with about 1 gigabit of data. And in this case, Blocksum was about
16 times faster. And another example, can we compute a checksum for 8 terabyte image?
Is it practical? It is. It took only 2.5 or 2.6 seconds. And if we do it with checksum,
it's not practical to actually do it. So I tested 100 gigabyte image. It took about 200
seconds. So the estimated time is 4 hours and 29 minutes. It means, in this case, we
have 6,000 sign faster. And of course, we create a different checksum. It's probably
obvious, but any tool exists on checksum because they use different algorithms. So Blocksum
is using, under the hood, some cryptographic hash function, but it's a different construction.
So we don't get the same checksum. Now, it's not available everywhere, anywhere. But I
build it in copper. So if you have Fedora or CentOS or L-System, you can enable the copper
repo, and then you can install the package and play with the tool. So the block hash
library. Basically, Blocksum is just using the library to compute the hash. So you can
also use the library to integrate this functionality in your program. The library is very simple.
This is the entire API. It gives you the standard interface to create a new hash, to update it,
and to get the result, the final interface, and of course, to free the resources used.
So if you use any cryptographic hash libraries, maybe Hashlib or OpenSSL, you know these interfaces.
Now the important difference is that when you update, when you give it a buffer with
data, this API will detect zeros in the data. And if you find the zero block, we will hash
it much, much faster. So this increases the throughput by one order of magnitude or something
like this. Even if you read from the file, you give it a buffer with zeros, we can hash
it much faster. But the most important API is the zero API, because new API, that no
other library supports. So if you know that a range in a file is not allocated, let's
say empty 8TB image, you check the file, you see that you have an 8TB hole. So you can
just input this range to the library, and it will hash it really, really quickly, much,
much faster than any other way. And you don't have to read anything for this. So how fast
it is? For data, we can process about two gigabytes of data. If you give it a buffer
with zeros, we can process almost 50 gigabytes per second. And the BLCache zero API can process
almost three terabytes per second. And this is on this laptop. If you try on a Peler one,
and this is the first one, like from two years ago, it's almost three times faster for data,
and almost five times faster for zero, up to 13 gigabytes per second. And I didn't try
the newer M1s or M2s. So if you want to use this library, you install the developer package,
you install the headers, and the library package, the Libs package, and your application will
depend on the Libs package. And everything should be automatic using RPM. Now, the most
important step is integrating this into your image, because this is the natural tool to
consume this functionality. So I have patches in review for adding this new command. It's
pretty simple. You give it the fine name. You have bogus. You can control caching, and
you can enforce the image format. And with this, you can compute a checksum of anything
that your image can access. You get the same checksum. It uses the block hash library using
the same configuration. So both tools are compatible. You can check my QMFork if you
want to see the details. So what is the difference if you compare to a block sum? Usually, it
runs almost the same. A little faster, maybe five percent faster. In some cases, it can
be 45 percent faster, like in this special case, the image full of zeros. I think it,
because QMFork image is closer to the data, it does not have to copy the data to QMFBD
and then read it over the socket. So this is really the best place to use this technology.
So let's see a small live demo. So we have several images. Let's try to look at the third
of 35 images. So we have a 60-gigabit image, a little more than 1 gigabit of data, and
we have this Q2 image, again similar size, and we can compare them. And of course they
are identical, and we can create checksum with bilkash, bilkasum, and we'll get the
same checksum pretty quickly. So let's try a bigger image, and this time we'll use the
progress flag to make it more fun. This is a 60-gigabit file with 24 gigabit of data,
so it will take some time to compute it. You can see that the progress jumps really quickly
when we find a big hole. So it took 12 seconds, and we will not try to use Chassum now. And
let's say the 8-terabyte image, which is really fast, and let's say the same is QMFBD
with the new checksum command. Okay, takes more time to type than to compute it. Okay,
so this is all for the demo, and the last part is how you can contribute to the project.
So the easiest stuff is just install it, and play with it, and test it, and if you find
an issue, report it. If you have interesting machines, benchmark it, and send the results
to the project. We have some results in the readme now, probably should be in a better
place. To run the benchmark, you need to build it on your system. It should be easy enough.
The most important stuff is packaging and hardware. Packaging for Fedora and Centro
Centrail is mostly done. I just need to get this into Fedora. There is some review process
that I probably need some help on this. Other Linux resources, if you want to package it for
Debian or whatever arch, please do, and contribute to the project. I like to keep all packaging
upstream, so I will be very happy to add whatever you need to make it transparent. So it will
not break when I change something upstream. Mac OS and 3BSD, I tested it on this platform
without LibNBD, because we don't have LibNBD there, and we can also cannot port it before
we port LibNBD and package it. But we can package only the library part, which will be useful
for QML. Missing features, there are a lot of stuff that can be added. You can look at
the site, we have an issue tracker, a lot of issues, a lot of stuff that we can add,
like supporting any image format, because we can't support only QCAP 2.0, checksum, multiple
image, use the file to verify checksum using what we recorded before, and stuff like this,
improve the CI. And even more important, but of course much more work integrates into your
system. So Ovid already supports checksum API using older implementation, Ovid's REST
project uses this API to verify criminal backups. It can be upgraded to the new tool.
Ovid can use it. For example, when you import an image to using CDI importer, or mainly
for storage operation, Ovid can verify operation with this tool. For example, running a port
connected to the disk and reading the image and verifying it. Maybe other systems, I don't
know, if you like other systems and think that it can be useful there, you can contact
me to discuss it. If I want to see the project links, we have the project in GitLab, also
the issue tracker in the project, and the copy that I showed before. So that's all.
How much time? How much time do we have? Five minutes per question. Okay. Yes. I have a question about the special case with block hash, blocks, I'm sorry, zero. Yes. How do you handle it under the hood? I see you specify the chat with the six, but how's that done?
Okay. Okay. So how do we handle zeros? How do we do it efficiently? So I have a bonus section at the end of the slides that you can get from the site, and basically the idea is that we split the image to blocks using fixed size.
The last block can be smaller, but it doesn't matter. And then we compute the hash for each block, but for zero blocks, we don't need to compute anything because we know that, like, we compute basically when you start operation, we compute the hash of the zero block based on the
configuration. And then each time we see a zero block, we can use the pre-computed hash, so it's cost nothing. And then we compute a hash from the list of hashes. This is called a hash list. It's not really new.
And now this costs something. You need to pay something for computing the hashes, but they are much, much smaller, like one of those magnitude smaller than the actual data. And to make it even faster, we also use multiple threads.
So this, what I, the previous slide show actually what's done in each worker. So we map the blocks, when you write something, we split the blocks and we map them to multiple workers at the same time and send them to the worker queue and the
others go and compute this combined hash, this block hash. And finally, we create a hash from the worker hashes. Yes.
So, oh, how hard is it for you to add a new checksum algorithm instead of Shah? How hard is it to use? So how, can we use another checksum algorithm? Yes, in Blocksum, you have the parameter, you can specify another
algorithm, you can use Shah1 or MD5 or whatever, but anything that OpenSSL supports, it also supports. I'm not sure if this is the best option, maybe we limit it because Shah1 is considered broken.
But currently, this is what we support. Anything that OpenSSL provides. Yes.
How do you identify zero blocks? What? How do you identify zero blocks? We, how do we identify zero blocks? So we use the very popular method that somebody ought to block about it, someone from the kernel a few
years ago, you can use MenComp to, with an offset to compare the file against itself. So you check the first 16 bytes and then you can just check the two pointers, the start of the image and the start of the image plus 16.
And it, it can process up to 50 gigabytes per second on this machine. Very efficient. Yes.
Okay. I did get the question. Yes.
Did we try on cryptographic algorithm? We didn't try, because we try to use something secure, like we try to get something which is secure as the original hash function, but we can try other algorithms.
It's interesting stuff that we can try. Thank you.
We're going to use the mic.