Hello. Thank you for coming to my talk. It's not a TED talk, but it's just my talk. Continuous
delivery to many Kubernetes clusters. My name is Carlos Sanchez and I'm here to talk to
you about our live experience, real world. I'm not here to sell you anything. So at least
I'll try to tell you if I have time some of the mistakes we made too. She's not all beautiful
and wonderful. So I'm a principal scientist at Adobe Experience Manager Cloud Service.
I'll talk a little bit about the product. On the open source side, I started the Jenkins
Kubernetes plug-in. Anybody heard about Jenkins? Yes, some people probably, yeah. Okay. And I'm
a Kubernetes. Anybody heard about Kubernetes? Yeah? Okay. Anybody using Kubernetes in production?
So I'm a long time contributor to open source. There are multiple projects on Jenkins, Apache
Foundation and all that. So a quick intro to what Adobe Experience Manager is because people,
every time I say Adobe, people say Photoshop and PDF and Flash, yeah. So that's not any
of those, right? So this is a content management system that you probably never heard of, but
it's powering 80% of the 4100 and it's very, very enterprise. I'm not expecting people
to know, but this is widely used because it's based in a lot of open source. It's a distributed
OSGI application that was started many years ago and uses a lot of components of open source
from the Apache Foundation and we contribute back to those components like Felix, Apache
Felix, Sling and a few things about content management there. And it has a huge market
of extension developers, people that are writing their own Java code that then runs on Adobe
Experience Manager and AM. So when I joined Adobe, the goal was, let's move this into
a cloud service and this is running AM on Kubernetes. We're running currently on Azure
and we have 35 clusters and growing very quickly because this is a content management. We run
it in multiple regions, right now 11, so multiple ones in the US, Europe, Australia, Singapore,
Japan, whatever, because people want to have low latency between their users and the content.
And then another interesting fact is that we have the Kubernetes clusters. We don't
run them directly. We build stuff on top of them and we have a different team at Adobe
that manages Kubernetes for us. Some curiosities is like customers can run their own code,
so we are running this for them and we take their code and run it inside our processes.
So we have to limit clusters permissions for security and we have several security concerns
because this is a very multi-tenant setup. Each customer can have multiple environments,
multiple copies and they can self-service, so they can deploy new environments whenever
they want, they can update them and do a few things, so it's not just us controlling what
is running, it's also the customers. Each customer can have three or more Kubernetes
namespaces where these environments run and this, I like to call this a micromanolith.
So we don't run a big service that spans like thousands of instances, we run slightly
different versions of the same service over a thousand, ten thousand times. So micromanolith
defines it very well. And then we use namespace Kubernetes namespaces to provide the scope
on network isolation, quotas, permissions and so on.
Now internally we have multiple teams building services, so different services have different
requirements, they have people can use different languages and we are more in a philosophy
of you build it, you run it. And we are basically doing each services post as an API or we follow
the Kubernetes operator patterns.
We also use to split the monolith, we use a lot of init containers and sidecars, if
you know in Kubernetes you can run multiple containers at the same time, so the main application
runs in one container and then we have to apply division of concern, many sidecars that
do different things. And it's an easy way to split separate concerns without having
to rewrite your whole architecture to go to a fully network-based, micro-service oriented
architecture.
So on the continuous delivery side, which is probably what you are interested in here,
we are running, we are moving to a, from a generally release to, I mean we are pushing
changes daily multiple times, right? Not only, not just the application, the application
may be slower to move, but on the operational side and all the services and operators, micro-service,
all these things, all of them together, any of them at any point in time, any day can
receive changes.
So we use Jenkins for CI CD in some places, we have Tecton, you heard about that in one
of the talks before, it's another open source project to do workflows on Kubernetes, to
orchestrate some pipelines and we also started using Argo CD for some new micro-services.
We follow a GitOps process, so where most of the configuration is a storing Git and
it's reconciling each commit, right? And we use a pull versus push model to scale. And
I'll go through this in a bit.
We have a combination of multiple things being deployed to the clusters. We have the AM application
that is deployed with a Helm chart. We have operation services that are on operators and
services and all the other things that are not the application. These are deployed using
Kubernetes files but templatized. And we are also using customized and Argo CD for some
new micro-services.
On the Helm side, we use the Helm operator. So in each namespace, we use the Helm operator
CRDs to do a more state-based installation of Helm. So we create the CRD and the Helm
operator is going to install the application based on the parameters on the CRD. A word
of advice is don't mix application and infrastructure, infrastructural configuration on the same package
because if you cannot enforce the same Helm chart for old tenants. For example, as I mentioned
before, customers can decide when to update things, right? So we have some customers in
older releases and some once in newer releases. This is something that we want to change.
But in the meantime, if we want to update a specific version of something in an old
release, it's hard when this is already packaged on Helm. So we built a solution for this.
So from the platform level, we can go and manipulate this Helm chart. So we can have
overrides and this is easy to do when you have the Helm operator. So you can inject,
whenever there's a request to install a new Helm chart, we change parameters. So we change
both Helm values. This is easy. Instead of passing some values, you pass different ones.
Or you can use customized patches. And this is also support from the Helm operator. This
is also support for customized patches. And customized patches are very interesting because
they allow you to patch any Kubernetes resource. So even if there was no previous Helm value
defined for it. So if we want to change a sidecar container image version across the
whole fleet, we just have to change the patch. And this patch is going to be applied to all
the clusters, all the namespaces. And all the Helm charts that were installed are going
to get reinstalled with the right version that we want. So we do this combination of both
Helm chart and then operational values on the other hand.
Very important for us was the shift left mentality, right? Detecting problems as soon as possible.
Not waiting for developers to push things to production because the cost increases.
So we run checks as soon as we can on pull requests. So this is still fresh in your memory
when you make a change and something is broken. You want to catch it as soon as possible.
And we do this by generating all these templates. We have some tests that generate these templates
and then apply tests, multiple tests on them.
The most basic check that you can run is the apply QCTL, apply the right run. This will
tell you if the manifest is wrong in some very obvious way. So if it's valid or it's
not valid. Cube conform is a tool that will allow you to validate the Kubernetes schemas.
So this is the successor of Kubeval. Anybody heard about Kubeval or Kubeconform? Okay.
So this is very useful for if you have custom CRDs or just to make sure typical problems
are you, you miss the jammer indentation and now it's not valid anymore and then you catch
this on a PR. You just run this and it will tell you, you know, this property is missing
or this is property is in the wrong place because everybody loves jammer, right?
Conf test is another tool for open policy agents. Any people familiar with open policy
agents? Open policy, OPA. So OPA allows you to write policies where you can go and pretty
much check anything in any structure file. In the case of Kubernetes, you could say,
I don't know, don't mount, don't run the pod as root. Make sure you don't mount secrets
as environment variables or with files. Make sure, enforce that all the pods have some
labels. Any random thing that you can think of, you can do it. And like, don't pull from
Docker Hub, pull from the internal registry. You can do that with Conf test and OPA policies.
The only problem is that it uses the regular language that if you haven't heard of, it's
very painful to work with, but it works great once you try to figure out.
We added another tool which is called Pluto. Pluto is just a CLI that will tell you what
API versions have been deprecated or removed. So if you are running, if you are thinking
about upgrading Kubernetes, you run Pluto and it will tell you, you know, this is deprecated,
it's going to be removed in this version and so on. So you can enforce that.
We built a tool that we call Git init, which is our own version of a GitOps pool. So we
have the Kubernetes definitions storing Git and we deploy these to blob stores across
regions. So they are pulled in each cluster. And Git init is a deployment that runs continuously
on each namespace. We have around 10,000 namespaces in our fleet. So it basically pulls the blob,
applies the changes and does this thing every so often. And an example of why we do a pool
versus pool, because pushing to all the clusters, we have a job that does this and it runs in
parallel, like in 20 threads or something, and still takes like five hours to run. So
we cannot push things when we want. On Argo CD, we have a newcast platform that allows
you to do Argo CD-based microservices. Argo CD, basically, this would create a new Git
repo, it would come with some templates and that would get deployed with Argo CD to the
cluster. And this is for us, we are thinking about moving this way and each team will have
their own Git repo, because right now we have mostly centralized operators and everything.
And this is good for the, okay, you go on your own direction, you do whatever you want,
you build it, you run it. On the other hand, it's a bit tricky because when we decide or
figure out something is problematic, we cannot just centrally say, you know, on this Git
repo tell me who is doing this and let's change it. But we are moving towards that direction.
Let me skip this and talk a bit about progressive delivery. So progressive delivery is a way,
well, it's something that, it's a name for something that you've probably heard of, which
is canary rollouts and doing percentage-based rollouts, feature flags, blue ring, so basically
don't update everybody at the same time because you can break everybody.
So we can do rollouts to different customer groups in separate waves and we can also do
rollouts to percentage of customers. By default, we have a time-based rollout that goes from
dev to stage to prod candidate after a period of time. And this is running on Jenkins and
ensures that things have been running on dev on stage before we merge them to prod. I mean,
this is very basic. What we built was feature flags at the namespace level. We have 10,000
namespaces and then the Kubernetes definition templates. So what we allow developers to
do is for each namespace, they can decide, I want to roll out this change to this environment,
dev, stage, or prod, or I want to deploy this change to a specific cluster or by template
namespace type of, yeah, type of namespace or a percentage. And this is just using templates
on Kubernetes objects. So an example is, in this case, a rule, sorry, a Kubernetes definition
where you can have a template that is as full version or bar version, or you can enable
a container, a sidecar container, or disable it. And then at the bottom, you can see the
rules. So by default, we want full version to be 1.0, but for the namespace, all the
namespaces on the dev environment, we want that to be 1.1. So this allows us to quickly
roll out changes, but progressively. We can also do it for percentiles. So in this case,
we could say, I want all the namespaces in dev and all the namespaces in a stage to have
this full version 1.1 and enable matter rule true, but for prod, I only want 5%. So I roll
out a change to 5% of prod, and then I can continue after that. So this has proven really
useful for developers to test in safely, increases development in speed, PRs are much faster,
so it's all great. And we are thinking about, well, we're thinking, we are working on getting
ARGOR rollouts also at the deployment level. ARGOR rollouts allows you to do blue-green
and canary rollouts, where you can say, progress the number of pods over a period of time,
so instead of changing, I don't know, 10 pods at the same time, because one by one, and
if you have a service mesh, you can go even more fine-grained and say, I want 5% of the
traffic to go to the old version, to the new version, everything else to the old version,
and keep progressing that and do automatic rollbacks. So, yes. So, yeah. With the service
mesh, you can fine-tune the traffic percentages, but with Kubernetes services, you can still
do it. It's just that we are limited with the number of pods.
So to sum up, Shift left on Garrail, so keeping people safe on what they are doing, this increases
development speed, reduces the issues that you are going to have in production, and you're
never going to prevent having issues in production. What you can prevent is how many customers
are affected and how fast you can fix them, right? So for us, what was very useful is
the progressive delivery techniques, like canaries, percent of rollouts, or automated
rollbacks, and the automation to do this, control and progressive rollout, pays off
over time. So I think we have one minute for questions. Or you can find me afterwards.
Thank you.