First thing first, the dagger team was kind enough to send me some stickers.
This is really the reason why I go to conferences.
So if you want to pick some up, I will leave it somewhere here.
Okay, cool.
So, good evening, everyone.
Thank you for joining my presentation today.
My name is Mark, and for the last couple years or maybe for the better part of the decade,
I've been helping engineering teams focus on their business, building their business
applications instead of worrying about things like deployments or CI CD or stuff like that.
And I'm currently, my current title at Cisco is Tech Lead, but by the way, is there anyone
here who saw my presentation in the morning in the go-to room?
Okay.
Okay, so I'm going to make it up to you guys.
So, and please laugh at my jokes as well again, even though you already heard them.
So I decided that I would come clean here today, that this is a completely fake title, and
that in fact my real job is a YAML engineer.
So anyone else want to come clean, unburden themselves?
Okay, okay, cool.
Oh, yeah, yeah, that's engineer probably, that's an overstatement.
So let's talk about CI CD, and we do have a bunch of CI CD services available today,
and it's still evolving continuously, but we do have a couple of challenges that causes
pain to developers and others, people every day, and I've already kind of hinted at one
of them, YAML, like, man, you put a space in the wrong place and it just breaks over,
and you don't even know why, because the CI tool may not even tell you where that extra
space is.
So YAML is really one of the core pains of all the CI solutions today, and yeah, I know
there is Jenkins and Groovy, which is inverse, but YAML is really the standard of CI CD languages
these days.
There are some places, like, there are a couple of solutions where it works kind of okay for
simple pipelines, but for more complex cases it's just a nightmare.
Then CI has this tendency to break for no obvious reason, like, one day the pipeline
works and the other way it just doesn't, and, well, for operations, for deploying your application
to a production environment, you can always say, okay, OAPS problem, let them solve, but
for CI that's not really a case, like, developers have to interface with interactive CI, and
if the CI is breaking, then it's probably the developers who have to fix it.
And the problem with current CI solutions today is that we don't really have, like,
an easy way to debug CI issues, like, if there is something wrong, you probably have to guess
where the problem is, maybe add a few echo lines to the YAML file, push the whole thing
to the repository, wait for the CI to get triggered by the repository, and then go through
this whole long feedback loop over and over again, you see a lot of people nodding.
So when something goes wrong, it's your job to fix it, and it takes a lot of time, and
it's just painful.
And sometimes, sometimes, it's actually not the CI's fault, but your fault.
The code doesn't, or the test doesn't pass, the linter doesn't pass, and that's often
caused by things like having different versions in the CI and different versions in your development
environment, and there are tools, and there are, like, ways to make those as close to
each other as possible, but still this is happening very often, like, I don't know, I had this
problem like a week ago.
So sometimes it's just your code that's not working with the CI, and you have to go through
the same feedback loop trying to push a change, hoping that it will fix your problem, and
of course, it doesn't work for the first time, so you do it over and over again, until after
an hour, maybe, maybe if you sacrifice something to the CI gods, it works.
So I'm pretty sure there are other challenges with CI, but let's see how Dagger may be able
to solve some of these challenges here.
So first of all, who has heard about Dagger?
Who knows what Dagger is, oh, cool.
So Dagger is a program about portable CI-CD solution, and portable is a pretty great
feature here, because instead of going through the total feedback loop I was talking about,
you can run your CI-CD pipeline on your own local machine and figure out what's wrong
much sooner than by pushing to the Git repository and waiting for the CI over and over again.
So it's much quicker that way to debug issues either related to the CI or your code, and
it's also much easier to build the pipeline in the first place, like when you build a
new CI pipeline for a new project, you have to go through the same feedback loop, because
you have to add new steps and figure out if it works or not, and if it doesn't, then you
have to figure out how to add the right parameters.
So even building new pipelines is way easier, because the whole thing is portable.
The other thing that makes Dagger great is that you can basically write your pipelines
in any language.
Dagger officially supports a couple of languages, like Go, Python, TypeScript, and Q, but basically
any language that can talk to a GraphQL API, because that's what's under the hood, any
language that can talk to a GraphQL API basically can be used to build your own pipelines with
Dagger.
And if you combine these two traits, like being portable and being able to write pipelines
in your own language, it also points to the fact that you can completely avoid vendor
locking, like you can, obviously you would still need some sort of CI service and you
would need like a thin layer of integration that would run Dagger itself, but once you
have a portable pipeline written in your own format in your own language, not in a proprietary
or CI-specific general format, you are not logged into the CI vendor you are using right
now.
And that you don't really switch like CI providers often, but that happens, like when they buy
your company and then you have to move from one provider to another and then you have
to move again because reasons.
And the fourth reason or the fourth thing that makes Dagger great is caching.
Now, most CI services already have some sort of caching solution that you can use to cache
like intermediary artifacts or dependencies or whatever you want to store in a cache that
you don't want to download or compile every single time when your CI pipeline runs.
But you still have to configure it properly and you have to make sure that you have the
right caching keys, you have to add the right paths to the caching configuration.
And you may either end up with a huge cache at the end of the day or you may not use cache
at all.
So if you don't configure it properly, it may not work.
With Dagger, you get caching by default.
And by default, I mean every single step in your pipeline, the result of that run will
be cached similarly how a Docker file works, like every single instruction, the result
of that instruction will be cached if there are no changes before that step, actually.
So similarly to that, Dagger caches every step in your CI pipeline.
Now how does it do that?
How does it work?
How is this portable?
Any guesses?
One word?
Yeah, Docker, yeah, that's right.
So containers, of course, containers.
So in order to be portable and to do all this magic that Dagger does, it needs to have like
a reasonable level of isolation so that you can be confident that it will run on your local
machine and on your CI the same way.
So it runs your builds in containers.
And I already mentioned that Dagger has a few official SDKs that you can use to build
the pipeline in your own code.
Using that Dagger SDK, you can talk to the so-called Dagger engine, which is the API
that implements the GraphQL specification.
And the Dagger SDK will call this API with the steps in your pipeline.
And the Dagger engine will build a DAG from these steps.
And then we'll pass that to basically to run through a container runtime.
And that's how your pipeline will run.
And the good thing about this is that you can actually change this pipeline so the output
of one pipeline can be the input of another.
And this whole thing goes through a single thing called session.
So in a single session, you can have multiple, like, these container executions.
And you can change the results into each other if you want to.
Now let's actually take a look at how these things run.
And the reason why I asked if there is anyone here who was in my presentation in the morning
because I completely botched the demo and it didn't work at all.
So let's hope it works this time.
So the example is in Go.
But again, it could be, like, TypeScript and Python or even Q. And I'm not going to go
into that much detail about the Go specific here.
But basically, you need to import this Dagger SDK in order to, by the way, can you see the
screen or the code?
Make it bigger?
OK.
Better?
Cool.
So you have to import this Dagger SDK if you want to use Go.
And then the first thing you need to do is connect to the Dagger Engine.
Now if the Dagger Engine doesn't run locally, then the SDK will actually run it using, as
a simple Docker container.
So the first thing you need to do is connect to this Dagger Engine.
And then you can start launching these containers and start building your pipelines.
And if it looks very similar, it's because it's basically the same, uses the same language
and it looks very similar to Docker files.
And it works basically the same way.
So you have, like, a base image.
You have a bunch of mounted volumes for caching.
And then you mount the source code and you run some sort of command.
And that's it.
That's your pipeline.
Now let's see if it actually runs.
So I use this make file authority for Go called mage.
So this is how I have this whole code implemented in a test function.
Let's see if it runs.
Okay.
So it did run.
Cool.
Let's try just for, let's debug to see what's happening in the background.
So it pulls an image, the goal and image.
It mounts the code.
It mounts the volumes.
And then runs my Go test on the mounted code.
And then basically exits and outputs the result of the test.
So, well, that's it.
If you want to get started with Dagger, check out the documentation.
It's getting better by the day.
They actually released, well, either today or yesterday, a new quick start guide, which
is pretty awesome.
It has all the three or four supported languages in a single document.
So you can switch between languages if you want to.
There is even a playground for the lower level GraphQL API.
So if you don't want to start a new project, you can play directly with the GraphQL API
with the hosted version of the Dagger engine.
So thank you for your attention.
If you have any questions, I'm happy to answer if you have time for that.
Thank you so much.
Awesome.
I have a question with regards to implementation, so let's say that you roll your pipeline,
you commit that you want to run that somewhere like in a CI environment.
GitHub Actions, or Gaffer Bids, Jenkins, or whatever.
How do you go about that?
I can imagine that you need to expose the Docker socket to the pipeline, or how does
it work?
Yeah, so basically, if you have Docker running in your environment, you can run this pipeline.
And you can run Docker anywhere, basically, today.
You can run it in Jenkins or GitHub.
You have it in GitHub Actions, actually.
And you probably have it on your machine, as well.
So wherever you have Docker running today, this pipeline will run.
So you just invoke the Dagger command, that command that you just showed us.
Yeah, it's not even a Dagger command.
This is entirely my code.
This is my go binary, basically.
Right, okay.
And it will find the Docker API socket, and if you just start containers there, yeah.
Before I, very cool stuff.
So before I switch all my CI to Dagger, let's frame it like this.
What would be the two things that you would really love to see an improved implementation
of in the next version?
Can you repeat the question?
What are the two things that really need to be improved about the current state of Dagger?
What improves?
To improve Dagger, in your opinion, what are the two things that need the most improvement?
Okay, so one thing is secret management.
Right now, Dagger, it's not that easy to work with secrets, so that needs to be improved,
and they are actually working on it, so that's great.
The other thing is that right now, if you build something in one language, for example,
if I build a reusable library in Go to run my pipelines, I can't reuse it in TypeScript,
for example, today.
And for that, there is actually a feature called extensions, so they are working on a
feature so you can build extensions to the Dagger engine, so you can build these reliable
or reusable pipeline PCs, like running liters and stuff like that, so you don't have to
build that in your own code, you just have to build the extensions, and you can call
it from whatever language you want to call them.
Basically GraphQL API extensions.
Thank you.
Last question.
Hi, does Dagger support spinning up Service A concurrently with Service B, because the
tests need something else to run while the test is running, and then afterward you can
continue to other stuff?
Right now, I don't think it does.
Again, this is something that they are thinking about, but it's not a trivial thing to do,
so no.
Currently.
Okay, someone is working on it.
Thank you.
Of course.
Thank you.
Thank you very much.