Hello, our next talk is going to be by Ahmad about having something to hide.
Thank you.
Yes, so my name is Ahmad Fatoum, I'm an embedded Linux engineer with Pingotronics and thanks
for attending my talk on having something to hide trusted key storage in Linux.
So Pingotronics, a company I work for is a German Linux consulting company.
We specialize in embedded systems.
So all around embedded Linux consulting, around drivers, bootloaders, kernel porting.
And in the course of one project, I had occasion to get more familiar with kernel's trusted
key subsystem, which I will talk about today.
But first I will talk about what we need to store these keys for.
This is usually disk encryption.
So if you install a new Linux distribution on many systems, you already have whole disk
encryption out of the box.
And it's just really one click affair.
But what are the mechanisms underlying that?
That's usually the M-crypt.
So the M-crypt is device mapper with the cryptarget.
And what that does is that it maps physical devices to a virtual device and applies some
transformation to it.
In this case, it's cryptography.
And you see how that looks like in code at the end of the slide.
You specify a range.
You start from the first block, the number of blocks.
You specify that you want to use crypt.
You specify your crypto parameters.
For example, here it's AES.
And then you reference crypto key that you want to use for this symmetric encryption.
Here is 32 byte long key with the name key, and it's of type lock on.
And in the line after that, you see this key being added.
And that's all you need to do.
So to initialize your Dm-crypt, then there is a Dm setup tool you can call.
And then you have the M-script running.
You can use this virtual device, just write to it.
And the physical device, everything that will be written there will be encrypted with this
parameters that you have set.
Most people don't do this manually via Dm setup, but they have a wrapper around that.
That's usually crypt setup with looks.
So looks is desk encryption specification.
You see at the end how the header is laid out.
You have this binary header that's still there for compatibility.
And you have a JSON area that can describe these parameters that we had in our Dm setup
table, like what sort of algorithm is used or what HMAC is used.
And then there is this key slots area.
And in this key slots area, you have this volume key that was at 32 byte long keys that
we had.
That key is what's actually used for crypto.
But if that leaks, yeah, you have all your data encrypted with it.
So the idea with the M-crypt is what the crypt setup and looks do is that you can have multiple
key phrases.
For example, your normal key phrase that you always enter or a recovery key.
And then in turn, you encrypt that volume key with each time with a different key.
And that's stored in these key slots area.
And that way, you can have multiple passphrases for the same volume.
And yeah, where does that passphrase come from?
So it's usually entered by the user.
So in the init.rd, you are asked what's the passphrase that you want.
And then you enter it.
You could be a bit more sophisticated and insert a USB stick that has a file.
That's the same code pass, basically.
You could insert a Fido security key or a smart card, but what all of these have in common
is that the user is inserting or writing or you need user involvement.
And in my project, it was an embedded system.
And we don't have really a user powering up the devices.
And yeah, we need some sort of automated solution for unattended boots.
And here is where trusted storage comes in.
So in the regular case, the trusted storage is like the memory of the user or his USB
stick.
But for an unattended boot, you need some on-chip or off-chip device that's appropriately
secure that can hold the key.
Such device is in many systems, the TPM or the trusted platform with yours.
This is an industry-wide standard.
It's also an international standard and it's mandated by Windows 11, which helps its adoption
in a lot of modern systems because you couldn't boot Linux otherwise.
They are available as discrete devices, as chips, sometimes on like a breakout board
for your motherboard, but they can also be implemented in firmware.
And TPMs have this standardized interface where you can talk to them and they provide
you a lot of services.
What's interesting for us is that it has a random number generator built-in, so it has
its own entropy source and gives you access to it.
And it holds a unique never-disclosed key.
And with this unique never-disclosed key, you can encrypt arbitrary data.
So instead of having a passphrase that you need to remember, you could have an encrypted
passphrase and then you pass it to the TPM and the TPM will decrypt it with this unique
never-disclosed key that it has inside and then pass you the data in a decrypted form,
which you can then pass into the M-crypt or into the crypto setup or whatever.
And you can make this even dependent on having reached a state that's an unintegrity measurement.
So each boot state could verify the boot stage after it and then tell the TPM this is a measurement
value.
And these measurement values are concatenated and hashed and kept in the TPM.
And you can configure the TPM to only release and only to decrypt data when it reaches that
state.
And then you can be, yeah, and when you configure it correctly, the TPM would only decrypt your
encrypted blob when you are indeed in that secure, in that measured boot state that you
want to be.
You can even bind it to a time.
So after a given time has elapsed, you can't access it anymore.
Yeah, how does it look like in practice?
The kernel has drivers for that that abstract away the different modes of communication.
It can be I squared C, it can be SPI.
You don't need to worry about that in user space.
You have these device files that provide your access.
There are user space libraries that wrap that and there is even a system D support since
I think a year and a half or so, where you can enroll looks keys into TPMs.
It's very easy to set up.
But whatever you do, the common way of using this with looks has the common, you could
call it issue that privileged user space has access to this key material.
So if you, you have seen there is this JSON area where you could store stuff.
So you could store your encrypted key there.
And what would happen on boot is that prep setup or system decrypt setup would go there,
it would get this encrypted key, encrypted key, it would send it to the TPM.
The TPM would do its checks and see, okay, I'm in the correct state.
It will decrypt this data and then send it back to your user space.
And then your user space now has this passphrase, which it could use to decrypt the M-crypt
key and then it would pass it into the kernel again.
So it's a real roundabout way to get the M-crypt key into the kernel key ring.
So the idea behind trusted key was why not directly decrypt the TPM secured key into
the kernel key ring and reference it from there without involving user space at all.
And yeah, so it has been implemented.
It was first added in 2010.
The first kernel was released in 2011.
It was originally TPM specific, but the naming was held generic enough, I think, in hopes
that it can be extended in future.
So the same patch series that added it added also encrypted keys.
So encrypted keys are keys that you can only observe from user space in encrypted form.
That's how it should be.
So you will tell the kernel, generate a key for me.
And then when you try to export the key, you only get it in encrypted form.
And then when you want to load it, you give it a kernel in encrypted form and it will
decrypt it, but it will stay in kernel memory in decrypted form.
And that's encrypted keys.
And trusted key additionally have hardware root of trust.
So they use a TPM for doing the encryption and decryption.
So in theory, you shouldn't be able to decrypt a trusted key to load it and have it decrypted
on another system than the one where you generate it on.
Because on the other system, you would have another trust source with its own unique key
which is used for the encryption.
How does it look like in code?
So it's basically the same line as we have seen before, but instead of having a 32-byte
long login key, we have a 32-byte trusted key here.
It's called KMK.
And to create it, you can use the key CTL command, you add a trusted key.
You don't specify the key material like we did with the logon key because you can do
that.
You can just ask the kernel to generate you a 32-byte key.
And then when you try to pipe it, which is the command to pipe the key contents out,
unlike a user key which would just output the key material in plain text, it would output
the encrypted key and set you can store wherever and use it on subsequent boots.
So what the rest does is it sets up a loop device and does the encrypt on it and write
it works and then it reboots.
And then on the second boot, if you were to create a new trusted key, it would be completely
different.
It would be generated randomly.
And you want to use the key that you have stored already, which is what the blue line
is doing.
It does add trusted KMK, but instead of creating a new key, it loads the key blobs that we
have stored.
And with that, you should be able to read back what you have written before.
Yeah, so that's how it works.
We have a way to do it in user space already, and that's how it's usually done.
And not everyone agrees that sets strict advantages by doing it in the kernel.
But what was interesting to me is that it is a very useful interface to represent much
more than just TPMs.
Because on modern system, you can have off-ship secure enclaves, basically a TPM that doesn't
speak to TPM protocol and doesn't implement everything, but it implements part of it.
You can have an on-ship trusted execution environment.
You can have crypto units inside everyday socks.
Very often you have a crypto accelerator that also has access to a key that it could use
for wrapping and unwrapping data.
And indeed, in 2019, work started from Sumit Garg at Linaro to generalize trusted keys
and add T support in the first instance.
So what is T?
T is also an API standard.
And what it's about, it's having a hardware isolated environment where you can run trusted
applications on the same CPU where you execute your Linux.
But thanks to this hardware isolation, normally armed trust zone, if you do everything right
and have firewalls in place and all that stuff, you shouldn't be able to read the secure memory
from your normal world, which is Linux.
And these trusted applications can do basically everything.
You can have a trusted application that offers you a TPM.
And in that case, you could just use trusted keys with TPMs.
But you can do basically anything.
It's software.
You can just do random number generation in T. You can do key sealing and unsealing
with a hardware unique key.
So that's available on some processors that when you are in the secure mode, you have
access to a key that you can never see from Linux, which is unique and fused in.
And there are even people doing clock reset power domain support stuff in it because they
don't want Linux to have access to these things.
So if you are interested, you can just grab the kernel tree for a T client driver and see
all the stuff that's there.
And what was interesting to me was the crypto unit inside the IMX SOCs.
It's called CAM by free scale.
And we already have a CAM driver in Linux.
It does random number generation.
It does crypto acceleration.
It works a bit like a network card.
So I have these shared TMA rings where you push the jobs you want the CAM to do.
And then the CAM replies to you.
And you can do, as I said, the crypto acceleration RNG.
And it also has access to a one-time programmable master key that's fused by NXP in the factory.
And that's unique between devices.
That's the selling point.
And the CAM can use it for red blob generation, which means it seals and unseals user supplied
data using it.
Basically the same we have seen with the TPM and with T. And it has black blob generation.
So TPMs are very slow.
And I don't know if they support crypto offloading, but you probably don't want to do that if
you want to do something quickly.
But the CAM can do it much quicker.
And you can have this key never exit the CAM and use it for crypto inside the CAM.
You are, of course, limited to the crypto algorithm.
The CAM supports.
But the possibility is there if you don't want your key to even enter the kernel.
It should be all the time in the CAM itself.
And yeah, so why do we need that for?
The common use case is certificate storage.
So you are a vendor and you need to call into your own cloud.
And you have client certificates for that.
And you don't want someone to be able to desolder this EMMC and read it out and get access to
your certificates.
And thus you decrypt the certificates and at runtime encrypt it into memory, maybe normal
memory, maybe unshipped memory, however, whatever.
And yeah, we had many customers that needed something like that.
And we had been carrying out of three patches for it in 2015.
We send it out the first time to get some feedback.
Back then it was using the standard thing, a custom CSS interface.
In the following years, NXP tried to upstream their own new key types to represent, to rep
this hardware functionality.
And finally in 2019, work began on generalizing trusted keys.
And yeah, it was finally merged in 2021.
In 2021, I also started then with implementing it for CAM.
And that support is now available since 5.19.
And it's usable exactly the same way as with TPMs.
You can't do this measurement stuff because a CAM doesn't have support for that.
But on NXP SOCs, you would rather use their form of verified boot.
So this unique key that's inside the CAM, it's only released when the SOC believes it's
in a high assurance boot state.
It means that the boot ROM has verified the boot loader.
And then you are supposed to keep that chain of verification going.
And boot loader verifies the kernel, verifies the init.rd and so on.
Yeah, some interesting tidbits.
While I upstreamed the series, T and TPM both don't use a kernel entropy pool for TPMs.
They always have a random number generator for T. It was specified that they need to
provide random number generation.
That's not something that I wanted to do for CAM because we have a perfectly fine CAM
RNG driver.
Not everyone was fine with it, but eventually, stubbornness prevailed.
And yeah, you can now choose it for existing backends as well.
You can specify trusted RNG equals kernel, and then you can, even for T or TPM, use the
kernel entropy pool if you want to use that.
The default is leaving it to the trust source to decide what it wants to do.
And that's also useful for devices like on the IMX6 ultra-light light, you can guess
from the name.
It's supposed to be very lightweight.
And their crypto unit doesn't support an RNG as is, and yeah.
So you rather want to use the kernel driver that's available, that does this differently
than you have to do it in your own driver.
And what was also interesting, hardware feature bits were broken on some variants, so you
can ask the CAM what features it supports, and the R-CAMs that support, say they have
a blob support, but they lack AES support, so they fail with an internal exception when
you try to use it, because it's, yeah.
Because the ceiling and unsealing is AES based.
But yeah, that's one more thing the kernel needs to take into account to work on these
systems.
And yeah, that's also something I only learned about while getting review feedback was not
something I anticipated.
As you have seen, NXP had different, okay, NXP had different attempts on getting into
the kernel, and they applied that to their vendor tree.
They called it secure keys, and during the upstreaming feedback I was asked if I wouldn't
want to change my modifier key to be compatible with the NXP kernel, so people have an easier
time migrating to it, because it was no problem for me.
It broke my SysS interface, but I needed a migration step anyway, and yeah, this makes
stuff easier for most of the users that want to switch, and yeah, so I did that.
Why did I need a migration step?
Because I was using looks before, but looks doesn't have trusted key support.
So what I did is I used the M-Crip directly.
I basically did the same things that looks would be doing, but only on the M-Crip part,
and I would exclude the header you had seen in the first, one of the first slides.
You can specify the range of blocks that it should work on, and then you can just cut
out the looks area and do the M-Crip directly.
And yeah, and you need a one-time import step, because the first time you don't want to generate
the trusted key randomly, but you want to take the ones that you have already been using
for years.
Of course, in a new product, you don't want that non-upstream patch I linked there, but
in an existing product, yeah, that's how you could do it.
Old key blob, put into CISFS, gets a plain text key out, keysetlmports, and you have
the new key blob.
We store both alongside, so if the update fails for whatever reason, you can fall back
to the old system and use the old key blob and both work.
Yeah, finally, what more is there to do?
So there's encrypted key support for the M-Crip, eCryptFS, eFAM, and VDM.
There's direct key support, trusted key support, without involving encrypted key for the M-Crip,
and yeah, you can use encrypted keys.
Future candidates would be FS-Cript, there has been attempts, one for the old key set-up
scheme, the second by me for the new key set-up scheme, UBFS authentication also currently
uses a logon key that could be changed to be a trusted or encrypted key, but yeah, these
patches have died down.
Look support would be awesome, because yeah, with looks it just works out of the box, with
the M-Crip, we still need to do it manually, but that enables us to do it completely in
the kernel without involving user space, and yeah, you don't really want user space missing
with a DMA-capable device that could just overwrite the kernel if you give it access,
so trusted keys was the correct solution for us there.
And that concludes my talk, and I would accept your questions if you have any.
Thank you, and we have some time for a few questions.
I have a question, are you aware of any way to kind of get this step of getting the secret
from the hardware to automate that into the kernel as well, so you don't need user space
interaction, user space utilities, my use case is mainly like the root file system,
and to forego using an NDRAMFS that needs to run a lot of commands, so you could, from
the kernel command line, similar, like with DMInit, also get the key.
Personally, if I had that requirement, I would consider doing it from the boot loader and
then have the kernel read it off the kernel command line, because the encrypted key blobs
there is nothing confidential about it, so yeah, in theory the kernel could accept it
over the kernel command line, but there is nothing like that currently.
I can repeat the question, if it's to.
Is there a way to also combine these hardware keys with some pin and looks, so you have
to authorize yourself to the device?
That's not really how it's meant to be used, because, well, yeah, the key material shouldn't
exit the kernel, and you directly reference the DMCrypt key, insert the key in the kernel
key ring and directly reference it, so I don't know how to do it to easily factor in a user
pin.
There's a passphrase option, apparently there is a passphrase option that I need to look
up when using trusted keys.
So thanks for the talk, would it be possible to add a manual step before communicating
with the TPM, for example, a fingerprint scanner or anything like that?
Is there a hardware and software option to combine the two verification steps?
You could.
So currently you need to have an init RD, so in my case you have an init RD, or I don't
even have an init RD, I don't use it for the root file system, but if you were to use
it for the root file system, for example, you could in the init RD first check that you
have that fingerprint is there, but there is no way to wire it in the kernel, first
this needs to happen, that's more of a policy thing that you would do in user space.