Okay, this speaker claims that in 20 minutes, Robert is going to build an event-driven application.
Well to be kind, I gave him 25 minutes.
So start your countdown clocks.
Hello.
So my name is Robert, and yes, I would like to show you today that we can build an event-driven
application in Go, and it can be as simple as building a simple HTTP server.
And I actually decided to put the bar a bit higher.
I think that I can do it within 15 minutes.
All right, at the beginning, a couple of words about myself.
So during the day, I work in a company named SlashID, so I work there as a principal engineer,
and we are creating some identity and we're onboarding from a solution that is a bit more
frictionless than a solution available now on the market.
And during the night, I'm blogging at 3.0.tech blog, where we are writing some blog posts
that are covering how to create Go applications that are business applications, but are also
maintainable in the long term.
I know maybe some of you had a chance to read at least one article there, there are some
people.
Nice.
I will have something special for you later.
You can find me on Twitter, GitHub, Mastodon, there's also my email if you would like to
write to me and ask about something, but what's the most important for today?
I'm Oselton of Watermill Library, and how everything started with Watermill, because
I think that this is pretty important context.
So a couple of years ago, I worked in a company where we are creating products that were not
doing something super unusual, but the idea was that each user was able to add some content
and he should be able to, we were storing it to MySQL, plus we wanted to have some more
advanced search, plus have ability to create fit for other users with some magic machine
learning models that they were doing personalization.
And usually if you are building such kind of system in a synchronous way, there's one
problem.
So this part may be sometimes slow, because elastic search is under high load, or magic
machine learning model so that this day it will not work.
Not nice, but it happens in work, unfortunately.
And yeah, or even worse, for example, some part is not working, and it's not best user
experience if it's working slowly, or it's, so for example, you can imagine that you're
adding some tweet and you're waiting for 10 seconds, because I know elastic search need
to index something or machine learning model is working slowly, or even you are not able
to add this content.
And it doesn't make sense, because everything what is done on this other part of the diagram
could be done asynchronously, because okay, it's not a problem if, for example, the search,
some content that was added cannot be searched, for example, for one minute after it's added,
if something is done.
It's much better than not allowing people to add anything.
So by the book, the default solution for such problems is using some kind of pop-up and
doing it asynchronously.
So in this case, we decided to use Kafka, because it's scalable, it's nice, but as usually
with some concepts that you're reading in the books or listening on the conferences,
it's not that simple in practice.
And it was also the case here.
The first problem was that the big part of the team wasn't actually working in asynchronous
architectures earlier.
That kind of makes sense, because if you're starting to learn to code, you're not starting
with building some event-driven application, you're rather creating some REST API or website.
So it makes sense that it was a big entry point for people that didn't use that.
And it was not the only problem, because event-driven architecture has a lot of concepts that you
need to know, like customer groups, partitioning, message ordering, at least one's delivery,
acknowledge negative, acknowledge poison queue.
And with all of that, you need to be sure that you didn't miss an event.
And it's pretty important in some domains.
In some cases, okay, it's fine, you're missing some event and okay.
But for example, I used to work in the financial domain, and losing one event may, for example,
mean that somebody will be not paid out.
Not nice.
In general, I believe that as engineers, we should be responsible, because sometimes the
code that we are building has a really big impact to the real life.
And after thinking for a while, I actually started to wonder, is it maybe something that
I can do to making, to building some kind of applications in Go simpler?
And here we are.
This is how WaterMill was created.
So far, we have more than 5,000 stars in the Github.
We have more than 50 contributors across multiple WaterMill repositories.
We are supporting 12 different PubSupp implementations, like Kafka, like Google Cloud PubSupp, like
NATS JetStream, Rabbit and Q, but we have also some more strange implementations, like
MySQL, for example, if you don't have infrastructure for some real PubSupp, or for example, would
like to avoid to face commits problem.
If you are doing some more fun projects, you can have just Go channel implementation
or BoldDB, for example.
But there is one more important thing than that, WaterMill has logo, and it is a logo
with Go for Vomiting to Gobernati's logo, not as Muai.
And you can think about WaterMill, like, so let's go back to this HTTP server example.
So you can think about WaterMill, like something that makes your life simpler, like standard
library for HTTP.
So, for example, if you are implementing an HTTP server, you don't care about TLS, layers
of network, you can start connection pooling and all this stuff, you are just implementing
the logic in most cases.
Sometimes, of course, you may have some specific scenarios that you care about that, but in
most cases, you should just implement your handlers and don't care about everything around.
And as you already, some of you shown, so I sometimes wrote the article that I think
that frameworks are probably not working best in Go, and WaterMill is also, for example,
that's the case why WaterMill is actually a library.
And it's pretty good to upside.
So the first one is that if you already have some system and you would like to migrate
to WaterMill, it's kind of simple, because WaterMill doesn't add anything super custom
and it can be integrated with any existing system, and vice versa.
See, for example, for some reason, you decide that you don't like WaterMill, but you will
not.
So you can migrate from WaterMill to some different library.
So this is the good thing.
And I think what's pretty important, so how everything is done, because, okay, in theory
it may sound nice, but it's helping, but how WaterMill is built.
And in the heart of WaterMill, I would say that you can see in multiple places something
that is named UNIX philosophy.
And it's kind of old philosophy, because it's from 1978.
And it's saying us to write programs that do one thing and do it well, write programs
to work together, and write programs to handle, in our case, message.
Because that is a universal interface.
And some small question now.
Do you know who's that?
So it's Ken Thompson.
So he's the author of this philosophy.
And what's also interesting, he's one of the authors of Go programming language.
Actually it makes sense, because if you look on the Go, for example, to IO Reader or our
writer, this is pretty nicely visible there.
And I know that for a lot of people didn't know about UNIX philosophy.
And sometimes when I have too much time to think, I have some impression that, no, sometimes
we forgot about some good old ideas and we're trying to reinvent the wheel, even if some
problems were already solved.
And you know, it's maybe something like in Dark Ages that it was some old nice ideas,
but it was a bit forgotten.
And OK, maybe I'm thinking too much.
Let's go back to the watermill.
So there are a couple important times in watermill.
So the first one is message.
So if you compare it to HTTP server, so it's something similar to HTTP request.
So in message we have UID, that is pretty useful for debugging.
We have metadata.
So metadata is something like headers request plus payload.
So this is the place where you are storing your event, for example.
The two next important parts of watermill are publisher and subscriber.
So publisher, you can publish those messages.
And with subscriber, you're right.
You can subscribe for those messages from the provided topic and receive that by the
channel.
You usually are not using these interfaces because it's used somewhere internally in
watermill.
But for example, if you would like to add a new implementation of PubSub, this is something
that you're implementing.
And each PubSub implementation is implementing this interface.
That's why I actually pretty like this interface.
Because it's making some constraint on the implementers that, OK, they need to implement
that in that way.
But it's also not good because it's making each of them pretty compatible with themselves.
And the last but not least type is hender function.
Hender function is something like HTTP handler that you are implementing in your HTTP server
with the small difference that instead of receiving HTTP request, you are receiving
a message.
And optionally, you can receive the message.
So the idea is that you can react on some message, do something, and emit some other
messages so you can do some kind of changing later.
I will show shortly an example.
And everything is magically connected, sorry, it may be small, but you need to trust me
that in the middle there is a router here.
And this is connecting everything.
So the message is going from some publisher, it doesn't need to be WaterMill, it's going
to the queue by subscriber, the router.
Router is passing it through middleware.
Middleware works in WaterMill like HTTP, so another thing that is pretty similar.
And it's processed by handlers.
And later, if we want, we can publish some other messages.
Not super complex.
So do you know the first rule of live coding?
Don't do live coding.
So do live coding.
What can go wrong?
All right.
Like to change sharing settings, so on second, it's probably not this one.
This is why you are not doing live coding.
Yes.
Okay.
So something does work, that's good, but I'm not really like, I want it.
This is something that I wanted to have.
So I prepared a simple application here.
And what does application does?
So if you're not from Brussels, so this may be something familiar to you.
So it allows you to book a room in hotel.
So you can provide room ID, pass guest counts, and let's see if it works.
Okay.
It seems that it's not working sometimes.
Sometimes it's working.
Sometimes it's not working.
Sometimes it's working slowly, slowly, slowly, slowly.
Sometimes it's even not working slowly.
So it's even worse.
So let's check the source code of that application.
So okay.
So here we are running HTTP, so boring, signals handling boring, but this is probably not
boring.
This is usually when the most interesting part of the application lives.
Let's check our handler.
So okay, so we are unmartialing stuff, to book room request, we have some advanced algorithm
of calculation of room price, and we are taking payment.
What can go wrong here?
And okay, as we can see, our payment provider, it's not super stable, but okay, I don't know,
let's imagine that it's our boss colleague and we cannot change that, no, politics.
It happens.
It's okay.
What we can do?
We can do like that, go, fang, okay, done, it works now, but it's one problem with that.
So if our server will die, there is a chance that we'll not take payment, and it doesn't
like that as the best idea.
So what will be my idea?
So instead of doing it synchronously with this HTTP handler, I would like to emit some
event, listen to that event, and take payment asynchronously.
So let's do that, and let's do that with watermill, of course.
So at the beginning, we need to get rid of that, and we need to have our publisher here.
Message publisher, so this is the interface that you should remember, all right.
And I also can prepare some code snippets to not lose time on some boring stuff like
room booked.
Well, we have our event, so room booked, all right, guest count, and price, room, price.
All right, now we need to marshal that, because we are sending bytes between our processes
through our PAPS app, so JSON, because JSON is kind of common and it's pretty easy to
debug.
So let's marshal that, payload error, room booked.
Don't do such error handling at home, please.
And now let's publish that.
The H publisher, publish topic, so let's use bookings, and we need our message.
Let's remember we need to have UID, so it doesn't matter actually what format of UID
it can be, I know, it can be even empty for some plantations, but good luck with debugging,
and room booked payloads.
All right, and it returns error, so we need to handle that in not a nice way, but it's
live coding, so it's fine.
All right, so we have the first part.
So we have our room booked event, we're publishing that to the topic bookings, and, okay, so
we just need to inject now the publisher.
So let's check where it's created, okay, we no longer need payments.
I heard that Kafka is nice and scalable, so let's use Kafka.
I have also snippet for that, it's nothing magical here, it's just this and the water
mid-documentation, and let's use this publisher.
We don't need subscriber yet, but probably we'll need it later.
All right, by the way, I'm running some nice Docker Compos under the hood that is recompiling
the project each time when I'm putting changes there.
At the end of the presentation, I will give you materials with all the source code, and
with the description of how it's done, that it's automatically reloading after each change.
All right, so we have our publisher, we are publishing our event, so let's check if it
works.
Hopefully it will work, okay, so you can see that our API is pretty stable, and let's check
if our event is really published.
So we'll use mule tool, so mule is part of water mule, as you can guess, and we'll consume
from bookings from Kafka.
Mule is allowing you to consume messages from multiple Pub-Sub types that are supported
in water mule.
I know that there is tool for that in Kafka, but it's not mine, so.
And yeah, with mule, you can use multiple Pub-Sub types, and okay, as you can see, now
we have event here, so it seems to work.
Okay, so done, thank you.
Not really.
We are not taking payments, so probably if our company will go bankrupt pretty quickly,
so we'll need to start to take payments.
So for that, we already have our subscriber, that's good, so let's uncomment that, okay.
We need to have water mule router, so message router error, router config, water mule logger,
router handling, and now we need to add a handler.
So we'll use addHander, so we'll need to provide handler name, so it will be payments.
It doesn't matter really what is the handler name, but again, pretty useful for debugging.
Subscribe topic.
So we're subscribing to the topic that we published this message, so this is bookings.
Bookings, we need to use subscriber, and we need to publish the topic.
So we'll publish event when we succeed to take payments, so payments, publisher, and
handler function.
So hopefully you remember handler function signature, so yeah, we are receiving message
and we are returning message, but we'll do it in a bit more fancy way, payments handler,
because we can inject some dependencies earlier, I need to fix that, and that, all right.
So we have our payments handler, so we'll receive message, and we'll take payment and
emit some event.
So we need to have our payment provider, and what?
We need to have room booked, we need to have our shoulder, so message payload to room booked.
And compared to standard library HTTP handler, you can return errors from a water new handler,
so I don't need to panic.
And all right, so we should have the payload that we published here, so that's good, so
we can now use that to take payment for room booked price, great, great.
And as I said, so I would like to also, I need some event, so it may be useful, so if
you're an intimate event that we took the payment, we can have some BI or we can, I
don't know, do something else, I mean, I don't know, we can send beer to this person
after he booked room, because why not?
And, okay, so we need the second event, payment taken, payment taken, filled, filled, room
booked, room booked as well as price, and we need to marshal it again to JSON.
Error.
Cool, okay, and the last thing that we need to do is returning message, message as new,
message new, UID new string, and payment taken payload.
I hope that I'm not writing too fast or too slow, all right, so in there, there is a chance
that it may work, so what we are doing, so we are receiving our room booked event, we
are marshaling that, we are taking payment, and when we succeed, we are emitting another
event.
Sounds like a done, so the only thing that we need to do is to reuse that handler, so
we have that one, and handler, cool, let's check if it compiles, it even compiles, so
let's check if it's working, so let's book a couple rooms, and the idea is that by default
WaterMe handler will try if the payment provider failed, so in there we should see some information
that payment was taken, and we don't see that, I don't, I know why we don't see that, because
we didn't start at router, run, context, error, it's a bit naive implementation because it's
not really graceful shutdown, but what in the documentation, as I remember, we have
examples with real graceful shutdown, so, okay, and let's see, okay, so we have some
random error, and you can see payment taken, hooray, our company is saved, all right, so
this is working, but there's one problem with that, so now we figure out that, okay, actually
Kafka is a bit hard to run, and we are on GCP, so maybe we can just Google it, so I think
that I can change Kafka implementation to Google, it pops up in one minute, I'm rewriting
the bar today, hi, but I think that I can do that, let's start the timer, one, two,
three, okay, let's check, I think I did that, so let's book, and okay, payment taken, we
can double check, so let's use meal, and let's consume bookings, you see, it works, all right,
so it will be that from live coding, one last thing that I would like to show you, because
you may notice that, okay, it's a lot of boring JSON there, et cetera, et cetera, you may
notice that I don't like boring stuff, because probably there are more interesting things
to do than marshalling to JSON, so that's because of that we created a component that
is named CQRS component, and the idea is that instead of doing this JSON-marshall and all
that stuff, you can provide configuration to which format you would like to marshall everything,
and under the hood it would be done, so you can use JSON, you can use Protobuf, Avro,
I don't know, even something custom if you really want, the idea is that you're only
implementing this interface, so you're providing the name of the handler, you are providing
the event that you are expecting to receive, so in that case it will be room-booked, and
you may notice that it was pre-generic, so we have the interface here, but we are working
on the newer version, and you are just receiving this event, zero, un-marshalling, or whatever,
and the same is going when you are publishing an event, so you are just providing the struct
and watermill under the hood is doing all the marshalling stuff.
Okay, so I think that will be all for live coding, it looks that I was lucky this time
that everything worked, and yeah, of course it's still not production-grade implementation,
I mean it's even hard to create a production-grade implementation of HTTP server, so it's more
kind of inspiration to look deeper and see that, okay, it's not that scary, but you need
to take into consideration that there are things like Kafka and Google Cloud pops-up
internals, what is once delivery, actually shown the secure component, but I didn't
call that, but it's helping a bit, so where you should start, because okay, it may be
a lot of sources for you, and a lot of stuff to check, so I heard that we have pretty nice
documentation, so we don't have any consulting or whatever for watermill, so we kind of don't
care to have bad documentation, so yeah, I heard that we have pretty good documentation,
so at the end of the presentation it will be in the link, what else, we have also a
lot of examples in watermill, so I will encourage you to, it's black, oh, live coding, okay,
it's not live coding, not only live coding, it's risky, so yeah, we have a lot of examples
that probably you cannot see because it's on the black, but you need to believe me that
this is on the watermill repository, at this point I wanted to say a big thank you to all
watermill contributors, because without you it wouldn't be like it's now, and it's not
an announcement that we actually released watermill 1.2 after having too many release
candidates, so yeah, finally it's released, and you are all invited to an online release
party, and we will say what are the new features, and it will be on March 1st, on the last link
it will be also linked for that, and I think that will be also, this is the, again it's
not working, oh, yeah, so this is the link that I promised to give you, the bonus that
I have, I have super fancy holographic sticker notes, I'm sure that you don't have sticker
notes, laptop stickers, so I'm sure that you don't have holographic ones, so if you don't
have, so I have a lot of them, and yeah, I think that would be all, so thank you very
much for your attention.
Thank you.