Okay. Our next talk is going to start right now. Mark's already on stage. He's going to
talk about automating secrets, rotation, and Kubernetes, and please quiet down so we can
understand him. Okay. Hello. Can you hear me? All right. So thank you for joining here today. My name is Mark. I'm an engineer
tech lead at Cisco. For the last couple of years, or maybe the better part of the decade, my primary job was helping
engineering teams around their business applications and Kubernetes and helping them succeed without having to get into too
much details about Kubernetes. Let me start with the story. I'm pretty sure this will sound familiar to a lot of us here.
A couple of years ago, I was in the middle of debugging session. It was already the middle of the night. Everyone was tired. And finally, we
found a problem. I committed the change, pushed the code, and then suddenly all the buzz went off. We received an e-mail from
AWS that a pair of credentials was committed in a public repository. Who did something like that before? Come on. I'm pretty
sure it's more than that. There's no shame in that. Everyone has to go through that once. So we obviously had to revoke the
credentials, generate a new pair, and deploy it to production. And we were able to do that because we had, like, good
secret management pipeline in place. And this kind of hints at why rotating secrets or being able to rotate secrets is
important, because if you have an incident like this, you have to be able to act quickly and rotate those secrets and make
sure that, well, in a first-case scenario, people may steal your data in a better scenario than AWS. Someone might start
mining Bitcoin. But you have to be able to react quickly. Another reason why this is a very important topic is we often have to
meet certain compliance requirements that require us to rotate every secret we have, like, every 90 days. I'm pretty sure many of
us have to deal with that. But the worst of all, the worst situation of all is when you don't even know that a secret has
been leaked. Or maybe an angry ex-employee took something with home. And you don't even know that happened. And they are
stealing your data. They are stealing your customer's data. Or they are mining Bitcoin in a better situation.
All right. So probably nobody disputes that secret rotation is important. But unfortunately, it comes with its own
self-challenges, which often turns people away from actually caring about this. And obviously, secret rotation or managing
secrets or configuration is a very complex problem, especially in a Kubernetes environment where you may have multiple
different clusters, multiple different in-spaces where you have to deploy these secrets, many different secrets and
integration, which means it takes a lot of time to do it right. And it's still an error-prone process. And in an idea
scenario, if you screw something up, it may not result in an actual outage or incident. But it may, which is obviously, it
would affect the business, which is what we wanted to avoid in the first place by making these secret rotations. So all
right. So I'm going to talk about some of the key challenges and why it's important points to that secret rotation
should be possible. I mean, it's probably always possible. But I've seen situations where rotating certain secrets would
have been very, very hard. Like it would have taken like hours, which is a problem. But so it should be possible. And
you should be able to do it relatively quickly. Secret rotation should also be as much automated as
possible. Like we are not really trustworthy, like we make mistakes, exhibit A. So it should be ultimately as much as
possible. And humans should interact with secrets and secret rotation as little as possible. And finally, secret
rotation should happen periodically. Like you shouldn't have a secret that you use for years, because as I mentioned, you
don't know if it's been leaked. And if you don't know if it's been leaked, how do you know if your system is secure or not?
So how does secret rotation look like in general? We are not even talking about Kubernetes here. First, you need to
have a secret store. If you don't have a secret store, then the whole thing is a lot more complex than it should be. You
have a secret store where you store your secrets, and then you have some solution to deploy those secrets to your production
environment or production environments. Now, when you need to change a secret, depending on what type of secret that is,
you have to go to the secret provider, which may be a third-party provider like AWS or GitHub or anything like that. You
have to issue a new pair of credentials or generate a new secret, change that in the secret store, and then you need some
sort of mechanism to deploy the new secret. That probably should be an automatic process that notices the secret
change, and it should deploy the secrets for you in your production environment. Now, in some cases, if you have a secret
store that supports that, for example, Hashicorp's vault, your secret store may be able to
automatically issue credentials for you, for example, for AWS, your database, or whatever else Hashicorp's vault supports, so you
don't even need to do that manually. Hashicorp's vault takes care of that, and that's like the best case scenario. Now, how does
this look like in Kubernetes? First of all, you have to decide whether you want to use Kubernetes secrets at all or not. There
are options when you don't have to use Kubernetes secrets, but that's probably the easiest way to many secrets in
Kubernetes, and the reason why generally people don't like using Kubernetes secrets is because they have this notion that
Kubernetes secrets are not secure because they are base 64 encoded, and that's not secure. So that's an entirely
different conversation. The bottom line is if you have envelope encryption enabled, which is disabled by default, then you're
probably safe using Kubernetes secrets. Now, if you decided to use Kubernetes secrets, then you need something that
deploys the secrets from your secret store to Kubernetes, and this could be, for example, the external secrets
operator. There are other solutions, but this is probably the one that the community organizes around a lot lately. So
external secrets operator is able to synchronize your secrets from an external store, external being to Kubernetes in this
case. For example, Hashicorp's vault or AWS secret manager or whatever else you have, external secrets operator is able to
synchronize secrets to Kubernetes secrets, and it's also able to pick up changes. It doesn't actively monitor changes, but
periodically it takes a look at the secrets, and if something changes, then it synchronizes the changes to Kubernetes. So
we have that part covered, and then you can use the Kubernetes secrets, either as environment variables or mount them as
files, however you want to use them. Now, the secrets change. What then? So if you mount secrets as files, and your
application is able to pick up that change, then you don't have anything to do. Your application will already reload the
configuration, and you have the whole thing covered. Now, if your application can't do that, or if your application
uses environment variables, you mount secrets as environment variables, but that's a more difficult problem, and for years we
didn't really have a solution for that other than manual restarts. A couple of years ago, this component called reloader
appeared on the market, which basically watches workloads that have, that references secrets, and it also watches the
secrets, obviously, and when it detects a change, it triggers a standard workload rollout, similarly to how you would do
that with kubectl rollout, for example. So it may change the annotation of the workload, and that would result in the
workload being rolled out, which means that it would run with the new environment variables, and it would remount the
secret with the changed file. And if we take a look at the whole process from the previous diagram, we don't have one component that
takes care of the deployment, in this case, but we have two, one that synchronizes the secrets from the secrets store to
Kubernetes, and the other one that takes care of the rollouts, or making sure that the workloads notice the secret
change. Well, let's take a look at a very quick demo, how that looks like in action, and I have a repository
prepared, you can go ahead and try it if you want to, and I have a Kubernetes cluster running here with both external
secrets and reloader installed, and in addition to that, we have like a simple echo server, which just, I believe it's,
yeah, we just output something. So let's take a look at how we configure external secrets first. So as I mentioned,
you configure external secrets, or maybe I don't need to mention, I don't know, but you configure external secrets
via custom resources, which means you create, can you see it from the back? Okay, cool. So you configure external
secrets via custom resource called external secret, and you tell external secrets to, you tell external secrets how to,
and from which external store should it synchronize secrets from, and where it should put it. So in this case, we are
telling external secrets to synchronize secrets from a store I created and called as fake. This is basically a static
secret store in this case. It synchronizes secrets into a secret called full bar, and it's going to synchronize from the
fake secret store under the key, from under the key full slash bar to a key under hello in the Kubernetes secret.
So let's take a look at, if we do, in fact, have that secret there. So we have a full bar secret. That's good so far.
And we have a hello key here. I'm sure if you can see that. Now, if I change this secret right now, this,
this is just a command that patches the external or the fake store to change the secret value. If I go back and check the
secret value, it should be changed to everyone. Now, if I try to curdle the service again, there are no changes here. So if I
manually restart the pod, let's see, do I have the command here? Yeah, I have a rollout command. If I manually restart the pod
and restart the port forward as well, then I should see that the secret value is in fact changed. Maybe I haven't shown
you, but I do have the application deployment here that references the full bar secret. All right. So now we have the secret
synchronization part covered. Now, let's see how it works if I want the workload to be automatically rolled out when the
secret changes. So I can annotate the echo server with this reloader annotation, which will make reloader start
watching this workload and the secrets mounted in it. So nothing changed yet. I should still see everyone. That's fine. And now
let's change the secret again to fuzz them. So if I, yeah, the secret is changed to fuzz them. And if we take a look at the,
I probably have to restart this. If we take a look at the service, it should now say hello fuzz them. So in this case, I
didn't have to restart the virtual manually because reloader did that for me when I changed the secret. When I changed the
secret in the store, that external secret synchronized into the Kubernetes secret and reloader noticed that
change, so it rolled out the deployment. So that's what I wanted to show you today. If you have any questions, I'm happy to
answer them.
Hi. Thanks for your presentation. Can we use a reloader? Can you speak up, please, because I can't hear you. Please stay
quiet. Thank you. Can we use reloader without Kubernetes secrets? Because we're one of, can we use reloader without
syncing to Kubernetes secrets? I mean, you absolutely can. So with reloader, you can watch either secrets or
config maps or both if you want to. But you need to use Kubernetes secrets and config maps. How do you change secrets
is up to you. If you don't want to automatically synchronize, you don't have to. You can use reloader just to trigger
a reload without using external secrets or synchronized secrets. So if you want to do that manually, you can
absolutely do that. Does it answer your question? No.
I would like to do something like synchronize secrets right into volumes, for example, like skipping Kubernetes secrets
totally, because we don't want to, like, resist that in that CD.
So no, probably reloader is not really useful in that case. But I see what you mean. So if you, for example, if you
use something like bolt-amp and you grab the secrets directly from within the pod and you want to trigger a reload,
then no, reloader can't be used that way. But we are actually working, so I'm from Cisco and before that I was
working for Banzai Cloud and we are working on a solution right now exactly for that so we can have, like,
a component that watches secrets that have external bolt references and reloads a component or
trigger reloads for workloads based on those changes. But none of these tools support that at the moment.
So are there some risks of using this method instead of using, for example, a secret vault? I mean, with a secret
vault, if you watch for a file and if you watch for a secret that should be written in a file or somewhere,
if the secret change vault usually emits a signal like a sig up to reload the process.
So what when the secret changes? Usually vault emits a signal, an up signal to reload the process and load the
configuration. In this way you are reloading the whole container so there are some risks.
The problem is that only works if you talk to a vault directly from your workloads and with the solution you
don't have to integrate vault directly, like you can use whatever secret story you want to. And the problem is that
vault doesn't actually know where it should set its signal. So in this case you may deploy the secrets to a
number of different clusters and the logists wouldn't know where to send those signals.
So the minor advantage is that it's fully transparent to the solution. I don't know.
We have time for one more question.
Any advice about some tools to do the rotation on the other part, like, for example, rotate the standard database credentials,
something like that, that will automatically update in the secret store then trigger the workshop?
The problem with that is that secret providers, like, there are many different secret providers.
So it's really hard to build a central solution for that. But hashicorp vault is one.
Hashicorp vault has a bunch of, I think it's called old backends or something like that, that you can use to issue
credentials, for example, to a Postgres database. And that credential can actually have a TTL, a deadline.
And then after a certain time, hashicorp's vault would issue a new pair of credentials and then external secrets would be
able to synchronize those credentials. We actually use that with AWS back end.
And that's how we rotate database credentials every 90 days.
Okay. Thank you so much for the talk. Thank you for all the questions. Thank you for staying quiet.
Thank you.