We're going to start here, we have a presentation about installation, almost presentation for
NADIS, how to manage Kubernetes, we think Kubernetes.
The presenters are Mohit and Malta, Malta, so big shout out to them.
Thank you. Yeah, we'd like to pick up where I think Nick in his initial presentation of
saying we need to have like a let's encrypt movement, we need to make confidential computing
a commodity, where he started off and then Mark knows I think had a great talk showing
all those bits and pieces that we need to have to bring together like the use cases,
the cloud native world, the way we develop applications for the cloud and the advantages
that the confidential computing technology gives us, how to bring them together and where
are those little gaps and historically seen or for different kind of use cases, I would
roughly split from a use case perspective of if I want to develop an application, where
can CC help me, how can I apply confidential computing, I can roughly split that in three
tiers if you will, I think one is definitely managing keys, having enclaves that hold your
cryptographic certificates, your keys that process the crypto operations for you, very
small TCB, very small kind of application, right, and then the second one is where I
package my entire application inside a enclave, inside a confidential container and I think
that's what we've been doing lately a lot and then I think the third thing is what Mark
has described is how can we bring that together, making this orchestratable, making this manageable,
deployable and I think there are different ways of getting from the tier two or the way
we are to here, one is I guess what Mark has described with taking containers, making them
confidential containers and then having the problems with orchestration and an orthogonal
approach that we like to present now is more of the idea of having confidential clusters,
so instead of isolating individual containers, isolating the nodes, the downside probably
is a little bit larger TCB and the advantage is being more closely to where we are right
now with deploying and developing cloud native applications.
Talking about challenges for level three, definitely I think one of the biggest ones
is UI UX, right, there's little hope that people will go ahead and drastically adjust
the way they deploy and develop applications for the cloud just because they want to use
this new type of technology, so we need to get very close to where they are and bring
those worlds together and then of course there are the challenges Mark has described
with orchestration, orchestration, attesting, how can we attest all those different containers
that are running in my cluster and don't necessarily want to verify each and individual instance
of it, right, that could be a thousand and more of the same.
And then once I have a cluster, all those day two operations of updating, upgrading
and doing that in a sensitive way where I can always verify what's currently being running
and what are the changes and yeah, big part of what we are going to present today is the
right one here where a big benefit of the cloud is actually that I can give away some
of this orchestrational work and I consume managed services that are operated by someone
or autonomously and I just consume them through an API or any other kind of interface.
So as Nick has said, infrastructure is rolling out, we see all those confidential technologies
in the cloud, AMD, SCB, we have heard so many, many today, IBM, RISC-5, most of them give
us a confidential VM, which is, as we've seen, not necessarily the abstraction we want,
but still we can already consume managed Kubernetes that runs on confidential VMs, at least for
the worker nodes.
I think Azure has it, GCP has it, yeah.
So this exists, but it's not really solving the problem, I mean, it gives us runtime encryption
for the stuff that works on, lives on that nodes, but all the edges, all the IO is not
protected, right?
The API server is not protected, we've seen that in Magnus Talk, the metadata problem,
the problem of the trusted control plane, the way if you want to consume persistent volumes,
is that automatically encrypted or do you need to adjust my application to encrypt before
writing to storage?
So the idea of a confidential cluster is that I have somebody or something that fills in
those gaps, so that I have, instead of those individual confidential VMs, I have one big
context that I can verify through attestation, that I can establish a secret channel, and
then if I'm in that context, if I'm in that cluster, I can just use Kubernetes as I used
to, and from inside there, essentially everything is trusted, right?
It's a different type of approach, it just creates an envelope around my Kubernetes and
isolates that as a whole.
As I said, I think UX and UI and the way we use this is super important, it's not gonna
work that we need a lot of adjustments, a lot of additional steps in the development
workflows, so having, this is just an example of constellation here, but having a simple
way of creating such a confidential cluster and then using it is important, and all those
things that I showed, all the challenges we need to solve below, we need to make this
more or less invisible, right?
In terms of constellation, we try to make the node operating system as verifiable as
possible, strip it down as much as possible, harden it, then strip them together for a
cluster, we need to think about supply chain, we need to think about how we can automatically
encrypt all the stuff that goes over the network, that goes to the storage.
Ideally this is all open source, so constellation, if I didn't have mentioned it's open source,
and it's cloud agnostic, so it can run everywhere, and then for most of confidential computing
stuff, I need some way of recovery, should things go, should things go south and everything
is down and need to get back into running mode.
So yeah, the big problem with the confidential cluster concept is now I can create a cluster
and we will see in a bit of what that means, but if I can create a cluster, I have everything
verified, now I have to maintain and run it on my own, and this is I guess the biggest
problem with that concept, right?
People want to consume managed stuff, when they have managed Kubernetes, don't want
to run their own, orchestrate their own Kubernetes, but this is a big trade-off that people are
facing, and yeah, we try to work on concepts of making that as easy as possible, and yeah,
Malte is going to show you how.
Yeah, so thanks for the introduction.
So the basic idea that we had was how can we manage Kubernetes from inside Kubernetes
itself, and to kind of draft this idea, I will start by explaining what typically you
can do today, so on the left side you really have the traditional on-prem model, which
is you have the whole cluster in your own hand, the control plane, the worker nodes,
it runs on your own hardware, which is great for security, right?
Because you have full control, but it also means you are responsible for every single
interaction, like scaling up the cluster, joining the nodes, performing upgrades, both
on the OS level and also Kubernetes upgrades, and then on the other side you have something
that is super popular, it is just let the cloud provider deal with it, and it means
the cloud provider can scale your cluster up and down, just if you have a burst of traffic
coming in, you get new nodes, it is all super easy, you can set it up so that the cloud
provider will automatically patch your operating system, it will automatically upgrade your
Kubernetes, and this is great from a DevOps perspective, it is super simple, it scales,
it takes away work from the developer and the operator of the cluster, so what we thought
is why don't we meet in the middle, and that is kind of like we have to run our own control
plane in the confidential context, but if we do this, we lose all of these smart features
from the cloud provider, so we will just reinvent them but inside the cluster, that means we
can still do auto scaling, we can still join the nodes by themselves without any human interaction,
we can still roll out OS updates and we can even roll out Kubernetes upgrades inside a
running Kubernetes cluster, so to explain how this works, I will first go on to how Kubernetes
nodes and constellation can actually join the cluster without any outside interaction,
and what you have to understand here is these are all confidential VMs and they make heavy
use of the measured boot chain, I think we already had some good introductions on this,
but I will still show you how this works in an example, so first in the confidential
VM we have the firmware, and the firmware is basically just the first part that starts
up, and the main task here is to load up the first stage boot loader and to measure it,
so we measure it on AMD SEV in the approach we are currently doing, it is measured into
a virtual TPM, and then we load this boot loader and then we will start executing it,
and the boot loader just has the task of, in our case, loading the next stage and measuring
it, which is a unified kernel image, and this is a very neat trick, it is basically just
one blob that contains the Linux kernel and in it RAMFS and also the kernel command line,
so the nice property here is we can measure all of these in one blob and don't have to
take care of the individual components, which can be quite hard to do correctly, and inside
of this, in the RAMFS, we will use the kernel command line to extract the hash that we expect
for the root file system, and for this we use the Emverity, which I will not go into
too much detail about this, it just allows us to have a read-only root file system and
we know in advance that it has not been tampered with, and we can efficiently check every block
while it is read and before it is actually given to the user land, so that's how we get
to the root file system, and inside of this root file system we have a small application
and the task of this application is to join this node into the Kubernetes cluster.
So next to the completely unmodifiable OS, we have a state disk and the only task of
the state disk is to have the data for Kubernetes itself, like container images and state at
runtime that has to be stored on disk, and this is initialized to be completely clean,
it's encrypted, and yeah, this is like a component we need to operate.
So the next question is how do we make these things possible, and for this we deploy some
microservices inside of Constellation, and these are the node operator, this is responsible
for actually rolling out updates, it's the join service that attests nodes that are joining
the cluster and decides if they are allowed to join or not, we also have a key service
that is handling encryption keys, and yeah, some more that are not really important right
now.
So how does a node actually join the cluster?
So I mentioned there's the bootstrapper that is started inside of the confidential virtual
machine and it will autonomously search for the existing Kubernetes control plan and it
will perform remote attestation using attested TLS, and it will basically use the attestation
statement for example from AMD SCV, SNP, and this way, so the join service already knows
what measurements to expect from a correct node that is running the expected software,
so it can decide at this point if the booted node is running what you wanted to run and
decide if it is allowed to join the cluster.
So based on this, the join service can then offer the node a join token which allows it
to join the cluster and it will also hand out a permanent encryption key for the state
disk.
So next we will have a quick look at how updates work and on a high level, we want the administrator
to be in control, we don't want to give up complete control over the update process,
but we want the actual execution to be completely automatic and seamless and we do this by basically
just telling the cluster what to do and the rest is done by a Kubernetes operator which
is a way to give in a desired state and let the cluster handle moving towards the state.
And an important thing to think about here is we are running in the cloud environment
and we don't want you to depend on individual nodes, this is also what GKE and EKS and others
are doing, we are saying if you want to upgrade, we will give you a new node that has the desired
configuration and we will never try to do updates in place.
So how does the actual update work?
We basically give in custom resources that describe the desired state, so the Kubernetes
version and the OS image that we want to run on and some properties to actually verify
like the expected measurements for the new image and tashes for the individual Kubernetes
components and the operator reads this information and basically checks if the desired state matches
reality and if it detects a mismatch, it will first stop any auto-scaling operations that
are happening in the cluster and then it will start replacing the nodes one by one and for
this we use the different APIs by the cloud providers.
So in this case, we will just spawn a new node in the correct node group that has the
desired configuration.
We wait for the node to autonomously join the cluster and we wait for it to become ready.
Next we will cordon and drain the node which just means we will safely move over your running
workloads from this node to other nodes in the cluster and only if we are sure that your
running workloads moved over, we will then remove the old node from the cluster and this
is basically how you can get from having outdated nodes to having updated nodes and this will
just go on until your whole cluster is up to date.
You can also parallelize this and when this is done, you can just restart the auto-scaler
and move on with your day.
All right, quick conclusion.
So in summary, the fundamental ideas, we take this confidential cluster concept, enveloping
the entire Kubernetes cluster instead of protecting single containers or parts, where
we gain is we basically get all the orchestration for free, we need to protect the edges, all
the Ion and so forth.
The downside is we can't isolate inside that cluster so it's one big envelope, of course.
This works already, it's an open source tool, you can check out Constellation on GitHub
and try it locally or on one of the big clouds.
From a Kubernetes perspective, it's just vanilla Kubernetes so not surprising that it's certified.
To give out some more references, if you're interested in this whole image part, there
was the image-based Linux and TPM death room, there's a lot of talks on these topics, also
very interesting.
There's a, so this is the last talk here, but if you're interested in more confidential
computing, sneak a little advertisement here for the OC3 Open Confidential Computing Conference
that's going to happen in March, it's virtual free, you can just sign up and listen to the
talks if you're interested.
A bunch of the folks that were here think also have a talk there.
Yeah, so yeah, if you have any questions, please feel free to get in touch and that's
it.
Thank you.
Oh, so yeah.
So the question was about the Attesta TLS, when we join nodes, we establish a seek connection
based on Attesta TLS.
Yes, so first of all, our implementation is open source, it's part of the consolation
source on GitHub.
I think it's nothing fancy, we use the AMD SCV or Intel TDX and so forth, remote attestation
statement to exchange a key as part of the data that's sent over.
And we bind the TLS session to that attested key.
I guess there are a couple of implementations for Attesta TLS, they work more or less the
same.
Yeah.
I think to the most that I'm familiar with, there is this vulnerability in remote attestation
that can be faked by a machine's whole spread and now I wonder if it is possible to fight
out from the remote attestation of the whole cluster, any single machine in the cluster
may make an attestation and it goes unnoticed or not, all the others are for example useful.
Okay, so repeat the question.
The question was there is a known vulnerability for attestation in confidential computing
and if given this confidential cluster, if from the whole cluster attestation I can refer
to if one of the nodes is faking attestation.
I have to say there were several vulnerabilities in several of the CC technologies over time,
I'm not aware of, no with what vulnerability you're referring to.
Okay, so the way the cluster attestation works is you give the, let's say the first
node, it has a known configuration, it will attest all other nodes based on this known
attestation.
If one node would be able to perfectly fake that attestation, you would not know from
an outside, from a cluster attestation perspective which node this would be.
But yeah, I guess that's what you can say.
It is super simple but it is big TCB, do you have any plans to reduce the TCB?
Yeah, we try to, as I said, this is a trade-off, yes, it's a much larger TCB than SGX, much
larger TCB even than confidential containers.
We of course will try to make it as minimal as possible.
Biggest leverage is of course the node OS and everything we can do inside there, yeah,
we'll definitely try to improve there.
Yes?
So you mentioned that there's some firmware at the beginning of the food processes that
firmware provided by you or by the provider?
Very good question.
Oh, sorry.
Yeah, the question is, part of the confidential VMs is the first component that's booted
is the firmware.
Do we have control of the firmware?
Ideally we would have, but what's provided by the cloud providers right now is Azure has
something in preview that allows you to do that.
It's not general available and GCP does not allow you that.
So the firmware for at least GCP or Azure is completely controlled by them.
On OpenStack with QM or KVM, you can potentially fully control the firmware, yeah.
Yes, next question.
That doesn't create a huge trust problem because you have to trust the firmware to be secure.
I mean, this is, of course, does this create a trust problem is the question.
Yeah.
I mean, this is a controversy, I fully agree with you.
This is not how we would like it.
This is just the best we can have.