Okay. Thank you, everyone. Our next speaker has some interesting news for your CI. There
are better solutions than the YAML you're used to. Mark is going to talk to us about
building a CI pipeline with Dagger in Go. Thank you. Thank you. Can you hear me? Okay.
So very important information before we get started. I have some Dagger stickers here
if you want to pick them up. I don't know. Maybe I can just leave them after the talk
or you can come to me and pick them up. I'll leave the stickers over here. Perfect. People
can grab them. Thank you. An important thing. Stickers are for your laptop, not for the
room. Every sticker you put inside a room involves them we have to pay for. So keep
them for yourself. Yeah. Well, that's why we are going to conferences for it. So thank
you again for joining me today. My name is Mark. And for the better part of the decade
I've been focusing on helping engineering teams focus build, helping them focus on
their business applications, building their best business applications instead of worrying
about things like CI and how they are being deployed. And I have this fake title at Cisco
technical lead that I decided that I would come clean here today that I'm really nothing
more than just a YAML engineer. Ooh, that feels good. Anyone else want to onboard them
themselves? Any other YAML engineers here? All right. So let's talk about CI-CD a bit.
And CI or CI systems improved a lot in the last couple of years. We have new and more
evolved CI solutions today. But we still have some challenges that we face every day. Like
the one I've been already hinting at, YAML. Obviously, YAML is one of the biggest problems
with CI systems today. Admittedly, sometimes, like using YAML to build the declarative pipeline
can be fine. But, man, you miss a space. The whole thing just broke. The whole thing just
breaks. And you might not even know where to start debugging it. So YAML makes it often
really hard for people to even just touch CI. And the other thing is CI tends to break for
no obvious reasons. Like, the pipeline that worked yesterday may not work today and you
don't really know why. And obviously, as developers, when something breaks in production, we can
just tell the ops people to worry about it. But with CI, that's not really the case. Like,
we have to interact with CI. And if something goes wrong, we might have to be the ones who
fix it. And with the currently available CI solutions today, you can't really, like, everything
was running in the cloud or in the remote system. You can't really have or you don't
really have tools that you can use to debug effectively. You have to start gassing and
start changing some YAML config. And you have to push that to a repository and then wait
for the CI to get triggered. And you have to go through this whole and long feedback loop
to be able to debug what's going wrong and to be able to fix that. And that's a pain.
Like, it takes a lot of time. It's really a huge waste of time. And it's really painful
to do that. Now, sometimes it's not the CI that's wrong. Sometimes it's you, like, pushing
something that you shouldn't be pushing to the repository like tests are not passing
or the linters are not passing or something else goes wrong. And again, you may have the
tools locally in your machine, but you may not have the same versions. You may not have
the same setup as in the CI. And it may just break in the CI even though you ran the test
locally and everything was green, it may still fail in the CI. And you still have to go through
the same long feedback loop again and again, trying to fix that, instead of being able
to run the whole thing locally and being confident that it will just work in the CI as well.
And probably there are other challenges with CI, but these are the ones that wasted hours
from my life in the last couple years. So how can Dagger provide an answer to this problem?
So first of all, Dagger is a programmable and deportable CI solution, which means you
can run your CI pipelines basically anywhere. We will get to how it does that. But the important
thing is that you can run your CI pipelines anywhere using the same environment, which
means if it runs on your machine, then you can be confident that it will run the same
way in your own CI system. Now, that's a great thing for a number of reasons, because when
you start building a pipeline, for example, using any of the CI systems today, you still
have to go through that feedback loop, like adding some config and then pushing it to the
GitHub OO and trying to figure out if it works or not, and then changing until it works. Now,
the ability to run this whole thing locally, it's much shorter feedback loop, so you can
build your own CI pipelines much more quickly than using some remote system. The other thing
is that if something goes wrong, you have the whole thing running locally. So again,
shorter feedback loop, you have more tools to debug, so it's much easier to figure out
what goes wrong, even if it's either the CI pipeline or your code. The other thing about
Dagger is that you can actually write your pipelines in your own preferred language.
Now, not any language, obviously. Some of the languages that Dagger supports, but that's
already much better than YAML. You can write your pipelines in Go, Python or TypeScript,
ThinkQ, EvenQ, but that's already much better than YAML. You can write your own pipelines
in code, and you don't have to invent or use some weird syntax, for example, to represent
dependencies between steps or between different pipelines. You can just do that in plain code,
so that's great. And all those, so the possibility of writing pipelines in your own language points
to the fact that you can avoid Pandor locking entirely. You would still have a CI solution
like Jenkins or GitHub actions or whatever, and you would still run Dagger on those systems,
but you would have to write a very thin integration layer just to run the Dagger pipelines. You
can, you would be much more confident that the pipelines would run the same way on the
CI system as on your computer, and yeah, you can avoid Pandor locking entirely. You can
move to another CI system if you want to, and you may say that it doesn't happen often,
but when it does, man, it's really painful, like converting from one YAML to another or
one YAML to, I don't know, Groovy or JenkinsFly or something, that hurts. And lastly, costly
caching. So every CI system or most CI systems have their own caching solutions where you
can cache the dependencies of your language or dependency manager, but that requires configuration.
You have to make sure that you configure it right, otherwise, well, it could either like
grow the cache endlessly and then you will be paying a lot of money for that, or it would
just be non-functional at all and it wouldn't cache anything properly. Now with Dagger,
everything is cached by default, like every step is cached. You can think about it like
a Docker file. Every instruction or the result of it is basically cached in a separate layer
in the Docker file, and if nothing changed between the steps, then when you run it again,
it will basically run the same way and it will come from the cache. That's really how
Dagger works. Obviously, you have some control over what you want to cache and how you want
to do it, but by default, Dagger got that covered for you. Now how does all this work
behind the scenes? If I had to describe it in one word, it's obviously containers. Now
we can do it ourselves today, right? We could just run everything in a container and it
would be reasonable to say that it will run on the CEI the same way. What Dagger adds
to the mix here is that you can actually build pipelines with code and that would be translated
into build pipelines. So you would use the Dagger SDK, the language SDK that Dagger provides.
Again, today, I believe it's for Go, TypeScript, Python, maybe Q as well. But the underlying
API is actually the GraphQL. So if you have a language client for GraphQL, you can actually
build your own SDK if you want to, or you can just write GraphQL queries and send those
directly to the Dagger engine. But basically, you write your own pipelines using this SDK
in your own language, and then the SDK will basically send GraphQL queries to the Dagger
engine. Now, when you run the whole thing locally first, then the Dagger SDK will actually
launch the Dagger engine for you. All it needs is really a Docker-compatible container runtime.
So if you have Docker on your computer or in your CEI, then you can run your Dagger pipeline
basically. So that's once more the portability of this whole thing. If you have Docker on
your machine and Docker basically runs anywhere these days, then you can run the Dagger pipeline
there. So locally, when you launch this for the first time, the Dagger SDK will launch
the Dagger engine for you and you send these GraphQL queries. You'll see a couple examples
how that looks like in the SDK, and the Dagger engine basically builds a DAG, directed basically
graph of all those steps, and then sends it through, well, it says an OCI runtime, I believe
currently Docker is the only supported runtime, but sends through an OCI runtime and runs the
whole thing in containers for you. And then when, obviously, when a pipeline is finished,
you get back the results, and you can use the results in another pipeline if you want
to. For example, the result of your build pipeline would be used in your deployed pipeline,
and you could deploy or project or whatever you have. So that's how Dagger works under
the hood. And let's take a look at an actual example. Let's see. So the example will be
go because this is the go-to-room. Can you see it from the back? Okay, cool. So the first
thing you need to use the Dagger SDK and go is importing this module from Dagger. It's
the Dagger SDK for go that you can use to interface with the Dagger engine. And once
you have that, you can basically start writing your own program. Now, in this case, I'm using
mage. I'm not sure if you're familiar with that, but it's basically like a make file
like solution for go. You can write these functions, and mage will basically compile
the binary from that and execute it like it would work with make. Now, you can absolutely
import this Dagger package in your own application if you want to. In case of applications, it's
probably not a huge deal if you have an additional dependency in your go modules. If you're writing
the library, though, you might want to create a separate module, for example, called CI,
and import the Dagger SDK in that separate module so you don't import Dagger as a development
dependency in your libraries go that modified. I know it still won't be built or still won't
be in the final binary if you import that library, but some people get to know it if
they see dependencies that is actually not necessary for the library. So make the life
easier for your peers, and if you develop a library and use Dagger, just create a separate
module and put all the Dagger-related code there. The first thing you need to do when
you want to write pipeline with Dagger is call this DaggerConnect function, which will
basically connect to your Docker runtime, and it will launch the Dagger engine for you and
start the so-called session. Now, within that session, you can start building your actual
pipelines using these containers. Now, it's pretty similar to how a Docker file would
look like for good reason, but what you can do here is basically use some of the same
instructions as you would do in a Docker file. You can obviously go from a base image, which
will be going in a Go project, for example. You can mount your source code. That's how
you would have access to the source code within the container, and then you can run a bunch
of commands like test or you can do the same with the linter, for example. And the other
two here, these are the mounted caches. You can do that with built-kit, actually. I believe
that's a built-kit functionality, so you can mount a cache directory to the container
that will not actually be part of the container, but it will be a mounted cache directory from
your host. Now, let's see if I can run this. So, I'm using the mage miner here. I'm telling
it to change to the CI directory because it's a separate module, and then I'm telling it
to use the current jet. Can you hear me? Okay. I don't know what happened there. And then
I'm basically just telling it to run the test function here again, similarly how a mage
file would look like. Now, let's see what happens. Kind of hope that I don't have to
download all those container images. Let's see. Let's get some locks here. I swear this
worked like a couple of hours ago. Oh, you know what? I think I don't have Docker running.
Yeah. That's a problem. Yeah, maybe we'll, yeah. So, I don't have the Docker engine running
at the moment. Let's see. This should start a new container. I mean, this should start
a new container. Let's see what's going on. All right. This is not great.
Can we all pray to the demo gods, please? Thank you.
Okay. You all just have to believe me that this actually works. Okay. So, here are the
locks from the previous run. So, this actually worked before. Yeah, it says, let's see. Okay,
let's try that. Let's see. Do we have internet connection here? Yeah, we do have internet
connection. Okay. Well, we'll have to work with the locks from here. So, well, basically
what happens here is when it works is it just runs the whole thing within this goal and
the image mounts the source code and then runs the goal that test command and just gives
back the results. Obviously, this is the build log, like this is the debug log, but normally
it would just output the, output of the go test command. Or the go lxci command. Let's
see. It's still not working. Let's try from hotspot. Maybe that works better. Anyway,
if, well, if someone wants to get back their money, sorry, folks, this is a free conference.
Anyway, yeah, you will just have to believe me that this works, but I will try to make
this work after the presentation. Now, if you take a look at the code here, this is
still not very user friendly. If you don't know how dagger works or if you don't know
what happens here, then it's not really useful to you. You will have to go to the documentation
and understand how this whole thing works, when it works. So, but the good thing is that
this is like, this is not an arbitrary YAML interface you have to use, so we can actually
make this a bit better if we want to. And what I did in the last couple of weeks is that
I built a higher-level library over the dagger SDK. So, instead of writing all that container
mount nonsense stuff, you can just use this go link package. It's actually called the
OCI. You can find it here if you want to give it a try. And instead of, you still have to
connect to dagger, obviously, but instead of writing that whole container code, you can
just use this much more friendly interface to run your tests or run the go link CILin,
for example. And it's much easier for developers to interact with this. Like, if they want
to change the cover mode, for example, it's pretty obvious how you would want to do that.
In this case, compared to how you would want to do that, it would be the lower-level dagger
SDK stuff. Let's give this another try. Now, from the... Oh, still on. Let's go to the...
Well, it doesn't work. Anyway, if you want to give this a try at home, you're absolutely
welcome to do that. The documentation is getting better by the day. It has a bunch of different
examples. You will find these examples on my GitHub as well. I promise it works. They've
actually just released a brand-new kickstart guide. So far, the documentation... They had
documentation for the different SDKs in different places. Now, they have a kickstart guide that
is basically the same for all of the languages. Regardless which one you want to choose or
if you want to try all three supported SDKs, you can do that with the kickstart guide.
You can actually go ahead and run the code from there. And finally, they have a playground
that works with their low-level GraphQL API. So if you want to give that a try, it's fairly
similar to the SDK, actually. If you want to give that a try, then you can absolutely
do that and see how dagger works without actually installing it on your computer. That's all
I had for today. If you have questions, feel free. There is a sticker. Thank you very much
for your attention.
If there are any questions, raise your hand. I'll try to give you a microphone. I have
one over here that's closer.
Can you use it with something other than Docker? I'm sorry. Can you use it with something other
than Docker underneath?
I think you can use it with Docker compatible runtimes. So I think you can use it with Podman
at the moment. I think, technically, you can use it at runtime. But I don't know if that's
currently available as an option. But we have someone from the Dagger team here who can
actually answer that question.
Hi. So how does the portability work when parts of your deployment depend on publishing
a Docker image to a repo that is external or AWS or Terraform or things that require secrets?
How does that fit in running it locally?
So that's a great question, actually. So the code itself should be completely portable.
So the pipeline itself can run anywhere. What you would need to do in that case is you need
some sort of either a central secret store that you can connect to from your own computer
or you need to be able to load some sort of secrets or credentials from your environment
variables, for example. You can absolutely do that with the Dagger pipeline. So from
that perspective, if the processor is here, you can push to another registry or push to
a development environment, for example. So you can parametrize pipelines based on where
you run them. You would still run the same code, but you could deploy to different environments
from locally.
We have one more question there. Okay. Thank you. One last applause, please.