Hi everyone, I'm here today to talk about delivering a cross-plane based platform.
A few words about myself. My name is Maximilian Blatt. I'm a Kubernetes and
cross-plane developer and consultant at Accenture in Germany. I'm using or working
with cross-plane for almost two years or yeah it's two years now and I'm the
maintainer of several cross-plane related open source projects including the
provider for AWS, the provider Styra, provider AguCD and I've contributed to
many more including cross-plane itself. Now since this is the CI CD dev room I
don't know if everyone is familiar with cross-plane so I just want to spend a
minute or two explaining what it is. So cross-plane essentially is an extension
to the Kubernetes API and it allows you to create cloud resources the way you
would create resources in Kubernetes. So the thing on the left is something most
of you probably have seen once or twice which is a Kubernetes pod and it's a
very common resource that you have in Kubernetes and it basically just schedules
and container where you can run an application. And on the right you see a
bucket as you would create it with cross-plane and it represents an actual
bucket on AWS S3. And if you look at both of these objects then you see that
they are very very similar because they are both inside the Kubernetes cluster
and you have both very common or the same kind of structure. You have your API
version and your kind. You have the metadata that comes with every cross-plane
with every Kubernetes object. You have a declarative spec so where you
describe the state of the resources the resource that you want and then you have
the status information about the resource itself. And that is one of the
features that cross-plane does for you so it connects external APIs any kind of
external APIs with Kubernetes and lets you manage your whole cloud
infrastructure through one Kubernetes cluster. And the second very powerful
feature of cross-plane is that it allows you to create your custom
Kubernetes APIs by using something that is called compositions and then it's the
thing that you can see in the middle. It's a very rough and simplified graph
to show the way cross-plane works and it essentially is always works that you
have the user claim for a resource for your API that you have to find using a
so-called XID or a composite resource definition and that is then passed to a
composition and then the composition spawns a number of managed resources.
Managed resources are something that you have seen in this slide before which is
in a bucket or any other kind of external resource on any other kind of
external API. Today I want to talk mostly about XIDs and compositions because
that is what you do most of the time when you are working with cross-plane.
Now developing a platform with cross-plane. If you look at simple CI CD
pipeline then you have usually build, test and then deploy and that is that is
very easy and for most software projects that is also very easy to
understand but because cross-plane is a bit different and you have different
things that you do inside these steps. So what you do with cross-plane is you
are first building and pushing a package and you are you're not writing code
but you are just writing YAML objects which are then applied on the cluster
and then they are handled and treated like data by cross-plane and then when
you are testing your cross-plane platform then you are applying all your
compositions and your XIDs to a test cluster and then you are claiming them
and then you see if they work and then if that is okay then you are deploying
them and you're just doing the same but on a production cluster. I don't want to
talk about the deployment today because that is very simple that is basically
just like doing a Kubernetes deployment you are building an OCI image and then
pushing that and then you are installing that on a cluster using
cross-plane and that's it there's not much to tell about but I want to talk
about the building and the testing. Let's start with the building. If you have
worked with cross-plane before then that is probably very familiar for you. On
the left you see an XID as you would write it and on the right you see a
composition. So an XID I said it basically just defines the API that your
user has applied to and it's very similar to custom resource definitions that
you are writing in plain Kubernetes. So you have your API schema in the spec of
your XID and then in the composition what you do is you define the resource
that should be created when the user claims this API and that can be an
arbitrary number of resources so you don't have to create just one resource
but you can create dozens of them so I've written compositions where you are
creating 30 or more resources at once but that is essentially how it how it
is done you are specifying a base resource and then you can modify this
resource by copying information from the user claim into the resource that you
want to create. That is what you do the whole time you are working with
prospering you are writing an XID and then you are writing a composition or
multiple compositions and then the user can claim it and then choose the
composition that he he wants. That now looks easy at first but when you are
doing this on an enterprise level then you are very easily you end up with
compositions that can be thousands of lines of code where you are creating
dozens of objects and then because you are just dealing with pure YAML then
you really starting to get at the limit because you have a lot of things that
are very repetitive inside compositions you have very similar structures let's
say if you are spawning a lot of similar objects on your cluster but in
different compositions then you sometimes you have the same patches that
you are reusing for example if you just want to patch the name of a resource by
what the user has given to you then you are repeating this patch over and over
for every resource for every file you are writing and sometimes you then have
compositions who only vary in details if you have different environments for
example you are in different AWS accounts and you only want resources to
appear in specific accounts or you have different values like the region or
static resources that you are that you want to connect like the account ID and
then you have to to write the same composition over and over but just with
different values and then you see that you are ending up with something that
gets really really complicated because you're just doing a lot of copy and
paste and so you need something to generate the YAML dynamically and in
these two years I spent a lot of thoughts how to simplify this process and
I have experimented with a bunch of stuff and we've tried out Q which is some
form of JSON like framework that allows you to build structures and have them
validated but it's very complex and not very easy for newcomers so if you have
new developers and teams then it's a bit hard to to onboard them on it on it
because the error messages are not very helpful in many cases and the tool that
we ended up establishing was Helm and not the biggest fan of Helm because it's
a bit quirky to use and sometimes if you have error messages or if you have
errors then it's sometimes hard to detect where the error actually is because it
just tells you all there's something wrong with your YAML but you don't know
where exactly happened but the good thing with Helm is that it can do
everything that we need you can replace common code blocks such as constants
with things that you have written out in your values YAML you can use templates
to parameterize patches and to save lines of code and you can even replace the
the API schemas of XRDs by something that you can generate and that is a
really really cool thing so I just checked the code in our repository and
we have about a hundred lines of code for for Helm I'm sorry 10,000 lines of
code for Helm and we are generating 200,000 lines of code of compositions
that are then applied on our API clusters if you are doing this if you
are generating code for for crossplane with Helm or any other kind of code
generation tool then I recommend you to check these generated YAML bits into
your Git because as it turned out it's very hard to detect unintended changes
that you are doing in Helm with your bare eyes if you are changing one value or
a template somewhere and then it might have some side effects that you're not
seeing so easily and so I really recommend you to check these generated
codes YAML code into your Git and do not treat it as artifacts and then if you
are in your CI then you should what we are doing and that is really helpful is
that you should regenerate all your package and your generated YAML and
see if any diff appears and if that is the case then you should just treat this
as an error and abort and if there is no diff then it's okay and then you can
continue on push your package to the OCI repository. Now so much for the
building now let's look at the testing. The first things that you are doing
probably when you are starting working with crossplane is that you are
writing your composition and then you are applying it on a cluster and then you
are claiming it and then you see if it works if all the resources get
ready and if you can use them and then it's done and that is all manual and
that is very easy to do because it requires no additional setup and you
can just use the cluster that you have but when you are really want to do
automatic testing or enterprise level testing then that is not enough and
because you have manual steps you have an outcome that is not reproducible
because you are doing the things all by yourself then also you don't have to
find what is actually expected outcome because sometimes even if a resource
gets healthy it doesn't mean that the resource is configured the way you want
it. So we also tried and tested a few things and we started with go testing
but it turned out to be much more complicated because you have to write
a lot of Bola plate code stuff and so we ended up using Cuddle. I don't know
if some people know it. It's basically a Kubernetes testing toolkit and that
allows you to define all your test cases in YAML and then just let Cuddle do
all the work all the application of the YAML on the server and then you
can define the resources that you expect afterwards and if you're imagining the
graph that I showed you before where you have the composition and then you
claim it and then you have a number of managed resources that are then spawned
and so you can have the claim as an input and then you can just define the
resources that you want to have created as an output and then you can handle
let Cuddle handle all the rest for you and then it can do things in
parallel and such and this is a really really great thing. So I recommend
Cuddle just to show you an example how these tests look like so you have your
small bucket claim if we are sticking to this simple bucket example then you
have your bucket claim on the left which is your test case and then on the right
you are defining all the objects that you want. You have the bucket claim
itself which has a resource status that should become ready and then you have
composite resource which is an internal resource that gets created by
crossplane where it stores some reconciling information which should
also become ready and then you have your actual bucket managed resource which
also has properties that you are expecting it to have and it also
has a status and so that is all you need to do testing with Cuddle for
crossplane and one thing I want to highlight is because in crossplane the
names of the composite resource are always generated by the Qube API
server so every time you are claiming an API the name is different it's always
different and you cannot influence it so what you can do with Cuddle is
you can let Cuddle identify the objects that you are expecting via the
labels you don't have to pass the name but instead just tell Yammer that you
just want an object with certain properties and label set and then Cuddle
will look for one object for any object on the server and if there is one that
satisfies this constraint then you are good to go.
One other thing that we've experienced is very good is you should run your
tests in separate clusters for every pipeline that you are running so we are
using virtual clusters or B clusters for that that they run inside a physical
cluster of course you can create your your own physical cluster all the time
but if you are spinning up physical clusters at least on EKS it can take
up to 30 minutes and that is not something that you want for every test
and it also costs a lot of money and so you're just spinning up virtual clusters
which are Kubernetes control planes that are running as POTS inside a cluster
where you can then install cross-plane its providers apply the compositions and
then run all the tests with Cuddle and then once you are done with the tests
then you can just delete the cluster and everything is fine and also you don't
have any intervention between two different pipelines because compositions
are cluster scope and they are most likely overriding each other. Now I've
been talking a lot about end-to-end tests and they are really good and I
recommend you to write end-to-end tests when you are building a cross-plane
platform but end-to-end tests also take a lot of time to run if you're
considering that you have an API where you are creating real physical cloud
resources and then you always have to wait for your resource to actually start
and then after some time maybe it says it says that something is misconfigured
and then you have to look for an error and if you're really just doing
development that it really slows you down because you have always this 10, 15, 20
minutes gaps between something happening and there are a lot of
mistakes that you can make when you are writing compositions and so I just want
to highlight a few things so you have these composite type rest that
reference the composition with the XRD they have to match and they are only
validated at runtime then you have the group names which have to match with the
XRD name you have an unstructured open API schema because XRD is because
Kubernetes does not support recursive API schemers yet maybe it will come in the
future but as of now it's not supported the same goes for the resource base
which can also have any kind of field and then you have the resource patches by
default the behavior in cross-plane is if you have if you want to patch from a
field to another field and the path of your source does not exist then cross-plane
cross-plane default behavior is that it will just ignore the patch and it will
not throw an error or anything and if that is the case and you you might
easily swallow any any errors and then it you're wondering why things don't
work but but you just have a typo in your patch and it's really hard to find
these if you have two thousand signs of YAML code and then you have types that
must match so if the user is inputting a string then you have to make sure that
the string is actually expected and not an integer on the on the actually
bucket API for example and then you have the indentation the big thing that if
when you are writing YAML files that is my big problem if I'm writing YAML
files I always mess up the indentation and then things get all messy so we
need something to detect these errors sooner because the sooner you detect an
error the easier it is to fix so what we have done because there is nothing out
there at least we couldn't find anything we've developed a linter for
cross-plane compositions where we are loading actual XID and CRD schemas and
then comparing them with the compositions and then applying a set of
rules like ensuring that the composition actually supports a valid XID type
that you don't have duplicate objects which can sometimes happen especially if
you are generating things with helm and then the most important thing is that
it actually validates the patches that you are running against the CRD and the
XID schemas and that is really really helpful that the first time when we ran
this against our production code it turned out to have I think 800 errors
that nobody noticed but somehow our our platform still worked yeah and other
cool thing about our linter is that it's pure CLI and you don't need a
Kubernetes cluster or a cross-plane installation you can just run this
locally without setting anything else up and you can it really takes maybe one
minute or two and then you have all your your your compositions linter and that
is really really really great you're wondering where to get it and there will
be a link on the last slide where you can find the code yeah summing things up
and so this is our CD CI CD pipeline that we have developed after a couple of
years of testing and failing so we use helm to write and build our compositions
to generate the YAML code dynamically we use our self-written linter to lint
our compositions and we use Cuddle to run all the end-to-end tests and then we
are just pushing things with train or any other kind of OCI tool that that comes
handy yeah so so much here's a QR code for the linter we are actually making
this open source today so you are the first one to actually see the code
except us yeah thank you do we have time for questions okay any questions
so my question is more about crossplane then crossplane this looks really good
though and how does crossplane compare to things like cluster API and the CRDs
that that introduces like where's the distinction between the two of them just
you know if you're familiar with cluster API so crossplane makes use of CRDs
under the hood so if you are if you are applying your XIDs on the cluster then
crossplane will generate CRDs which are then used as the API that can be
the user can claim
if there are no more questions then thank you we're going to make a five
minutes break