Okay, awesome.
The mic is on, hopefully.
All right, good afternoon everyone.
So I'm going to talk to you about a worker story,
which is something we did at work recently.
And for once, it was like not using Rails.
That's awesome.
Not using web at all.
That's what motivated me to tell you this story.
So before we start, I would like to know who here is the Rails developer?
Who would like?
Yeah, awesome.
Who would say that they are Ruby, but not Rails developer?
Okay, awesome.
That's great.
Love it.
I didn't expect that.
Awesome.
All right, first of all, who am I?
Because if you don't know who am I, you might not rely on whatever I'm going to say.
So I've been Ruby and mostly Rails developer for 10 years.
I've been working with Kevin for almost that period.
More recently, I have become a lead dev, then a manager, then a CTO.
So I'm doing a lot of new responsibilities now, which also gives me a new perspective
on a lot of programming topic, actually getting new perspective when you start making a decision
about people and processes and stuff like that.
And finally, I've been a teacher for more than six years.
I've given a lecture at EPL and Le Wagon.
Hopefully, we'll do that again.
I feel like a deep-footed lover for teaching and sharing knowledge.
And this is also why I'm here today.
So I was saying the point of this talk is talking about Ruby, but not about Rails, not about
web.
And this was the first time for me.
I was like a new experience.
And it's strange to see how much changes when you start doing that, how much you realize
Rails was giving to you once you don't have that anymore.
I have some notes.
By the way, all my slides are going to be minimalistic.
I'm not going to show you a single line of code.
I'm also going to forget a lot of stuff, which is why everything I intend to tell you is written
in notes available directly in this.
So hopefully, you will get everything I intend to say because I'm going to forget part of that.
So the main message of this talk is like it's doable.
It sounds strange that this is my message, but as most Rails developers sometimes, when we think
about Ruby program, we're not even sure we can do it.
We're not even sure how would we approach that.
So the main message is like, yes, it's doable.
There's a lot of tools.
There's a lot of process.
There's a lot of help along the way.
And you can possibly, you can very likely, sorry, get most of your tools and knowledge
used in a normal, not web Ruby application.
The second news is you can also get all of your Rails knowledge useful in Ruby application
if you get things right.
So the story I'm going to tell is about like a worker.
What is a worker in our case, in my case?
It's like microservice.
The specificity, why do we call it a worker?
Because it's not a web microservice.
It's a microservice which is consuming messages from a queue and very likely is going to process
files, so it's going to get files from a bucket, process them locally, put them on another bucket.
We have, we are using the word worker because we have like lots of them.
That's simple definition.
We have lots of them.
So I'm going to talk about like one of them, but it could be any of them.
So we think start with a loop.
The whole story starts with a loop because when I started this, I really like I opened
my editor and I saw something which I hadn't seen since school.
It's an empty directory.
It's very strange.
Like really first as a Rails developer, I'm really used to Rails new and then you get
like everything.
You get folders, tree, substructure.
You get the config directory, you get the app directory.
You, like there's drawers everywhere about what you expect to put things.
In this case, I just like create a new folder and it was empty.
I'm a firm believer in emergent design.
So I started immediately like new file, worker.rb, make a loop, while true, read, perform, delete
message.
I'm done.
It was nice.
Like it was, I knew it was not the end, but it was capturing whatever I was, I knew about
my process.
It was a single level of abstraction.
So I knew it was a good start, but it wasn't.
It wasn't a good start because I was already forgetting like my main tool when doing Rails
apps, which was going to be my main tool when doing any app.
It's tests.
Anybody who knows me know that I'm a firm believer in tests.
And it's a policy.
It's not a religion, but it's a policy.
This is how I write code.
I do believe in it, but you mileage may vary.
But for me, it was the beginning.
And it's funny because I knew I was going to write loop, the loop of my program, but I
was also starting another loop, the loop of my process.
And this is what tests are for me.
Test first does not mean you do test, then you do code, then you're done.
Test first means your first step in the journey is test.
Then code, then test, then code, then test, then code, then test, then code.
That's what it means to me to do test.
But I did it wrong.
I started with code.
So I tried again.
I deleted my file.
I created a spec directory.
I created a spec file explaining what I knew about it.
And I was happier because test is the file that depicts my best understanding of what
I currently believe is the success.
And I need that because I'm going to write code right after the word.
And once you're deep in the code, you're super focused.
You forget about landscape.
You don't know what comes next.
You might have a story.
You might have specification requirements.
You name it.
But I do believe that a story or specification is like coordinates of where you're supposed
to land.
The whole puzzle, the whole activity of development, of programming, is like playing golf in the
fog by night.
You know where you are at the beginning.
You sort of know when you want to land.
But after your first shot, you're going to be lost.
It doesn't matter even anymore what you're supposed to land because you've given your first
shot and you don't even know where you are anymore.
I'm using tests as torches in the night.
So I read my specs.
I write some tests.
This is my belief.
I'm going to follow that path.
And then I shoot my first shot.
Hopefully I'm going to reach my first torch in the night.
When I have reached that one, I'm going to go to my second torch again and again.
But my loop is that my test is only my best understanding of my success.
So my test is going to evolve.
I'm going to move my torches and I'm going to move my ball.
And this is how they make sense together.
Back to the story.
I wrote my test, was happy with my understanding, run it, and it failed.
It was a catastrophe.
And why did it fail?
Well, because it couldn't find our spec.
Because I didn't bundle it.
Because it couldn't find bundler.
Like, that is how empty the whole story was.
Like, I didn't even have bundler.
Okay, so bundling is always easy.
Bringing my dependency, starting my gem file.
I need to run my spec.
Run it again.
Well, it still fails.
But for a better reason.
And that's the whole point of TG, right?
You have to fail, but for a better reason than the previous failure.
So now it's failing because it doesn't know about what is a queue,
what is the method we see in the queue, what is a message,
what is a processor, what does perform even means.
Well, that makes me happy.
Because now I can actually write more tests about what do I believe is a queue at this stage.
Why do I believe is a processor, what do I believe should do the receive method.
And this was really the starting of both my loops.
I got my main loop back, but I got my working loop as well.
I got a lot of tests.
I knew that trying to make them go green would just generate more tests.
Trying to make them go green.
I got my actual work loop.
Right.
So, test code, test code, test code.
I was in the middle of it.
And every single of the code file was starting with like probably five to ten require or require relative.
And I wasn't happy with that.
First of all, because it is boilerplate, it's noise.
I don't like noise.
Also, because I want my code files to be about the responsibility they're supposed to hold.
And knowing what files contains the dependency that this file depends upon,
it's not the responsibility of each file to know where do I store the other responsibilities.
That was wrong.
And this is not something we have with Rails.
I realized that we actually get something super nice from Rails is put any file in any sub directory of app folder,
and you get it.
It's like magic.
Once you have to start all your require by hand, it felt wrong.
So, I Googled.
I got a few options.
And the best one, which is actually the one which is currently adopted by Rails, was using SideFerq.
Hopefully, I'm pronouncing it right.
It's written in my speaker mode.
And that stuff helped me, like, auto load the constants I was looking for by looking them up in my lib directory.
Default config.
I'm happy as far as I know this is what I need.
But reading the rest of SideFerq, I also realized that this enables you to use short names.
So, if you are in the same namespace, you can just mention a constant by a short name.
Well, obviously, I want that.
I'm doing that in Rails, so I want that again.
It's also handling multithread code loading.
I have no idea if I'm going to need that, but I certainly don't want to handle that myself.
It sounds like something I really don't want to handle myself.
And it also handle code reloading, which is not something I'm going to use because of TDD.
But again, this is my approach.
I know that most people don't do that.
And code reloading is a very important part of code loading.
So, SideFerq was like my first take, my first really great companion that I found along the way.
The second one was dry container.
Now, small disclaimer, I knew from the stuff that I was going to use dry gems because I wanted to.
And as Kevin said, it's also a little bit about finding joy.
So, I wanted to heavily rely on dry gems, but I wanted to wait until the use case was there.
I wanted, because I did not only want to skip the requires, I wanted to not know the classes.
I wanted to not call new in the middle of my code.
My code is about business logic.
Most of the code is about business logic.
I wanted to separate, sorry, the logic about creating objects and the logic about like, I need something.
And most of the time, when you're in a controller, in a Rails controller,
you don't even care like where does the request object comes from.
You're just like, okay, I want a request object.
Just make it happen.
If you're in a view, you don't care about what the view context comes from.
You just have it.
You just want it.
And it's really comfortable to write code with just focusing on like using the stuff you need,
not focusing on how you get them.
So, this is what dry container brings.
I've been using dry system, which is like dry container for handling all of that, and dry injector.
And dry injector basically works hand in hand with dry container and allows you to call your services,
call your dependencies by the small name, by the first name.
You give a name to an object and then you can basically say, okay, I want this object.
I don't want this class.
I don't want to instantiate that class.
I want specifically that object.
And I'm going to use it.
And I don't even care what its class is for.
I want that object by name.
Interestingly, this had almost no effect on the test.
Even though it's a very different approach, I still had most of my tests instantiate object by themselves.
Why?
Because unit tests actually give a lot of fake dependencies.
That's the point of unit, right?
You want to test a single unit.
So I was still building my subject into tests manually.
And for the larger, the broader tests, I actually wanted to use the container set up correctly
because I wanted to test that things were correctly wired together.
So even though dry containers is like, oh, some you can stub and fake and change whatever you want.
I didn't stub it because I was either using it and testing it or not using it at all in my test.
And...
Sorry.
Yep.
Yeah, I'm still in time.
Dry container also brings something else, which is quite interesting.
It's a settings, a settings object.
And I realized very soon that the settings object was the object that I was injecting everywhere.
Almost every part of my system needed to access settings.
So I was injecting it everywhere.
It was awesome.
And dry settings provide some really interesting value.
First of all, it allows any of the settings to be overridden by environment variable,
which is quite important.
If you know about 12 factors, it is one of the aspects you want for your config to be overridden
by the environment that your program runs in.
So that was the first part.
And the second part is that you can coerce, you can define the type of your settings.
Because if you work with environment variables, everything is a string.
But when you work in your system, not everything is a string.
We do have a lot of strings, but we have dates, we have integers.
We have a lot of system.
And usually what we do is we just parse them.
Dry types allows you to create all types, name them for starters.
Also naming things is probably the most important stuff we do in our work, I believe.
You can name your type and get them correctly and get your settings in the proper types,
which brings me to my next slide about dry types.
So dry type creates a contract.
It says, okay, this value, this settings, it has to be a phone number.
And I'm going to explain exactly what is a phone number.
And I'm also going to coerce like a string into a phone number, which means at the end of the day,
I either have an error or I do have a phone number, which is exactly the object I want.
And it makes a big difference.
I don't know if any of you have ever created a class like phone number, like age, like bucket name.
If you read correctly the literature about object-oriented design, we are supposed to do that.
We are sort of supposed to do that, like subclass string when we want to make a first name.
To be honest with you, I've never done that in my life.
I've always used string and it's not a first name.
It's a string.
I know it's a first name.
I know I'm not going to use all the methods of string, but the variable is name, first name, that's enough.
Using types allows us to actually have proper types, more meaningful types,
without creating full-blown classes for everything.
Well, settings is one thing.
But this contract, it can really be used for something else.
It can be used for app input.
When you are working in a web application, app input is a request.
This is where most of our payload comes from.
In our case, the app input was messaged from a queue, but the concept was very similar.
As soon as we got one message, we treated it in a very similar fashion as we would have treated a request.
When working with app input as a web, there's a very known pattern for handling that input,
for validating that input, for correcting that input to everything that you wanted.
These are form objects.
We basically reused the same.
I realized that I'm doing my slide in the wrong order, but you don't care because you don't have the order.
But that's okay.
We used kind of form object in the form of a dry contract.
It comes from dry validation, that is the gem we have been using.
Dry validation is really about two pillars.
The first one is about typing.
Eventually, it leverages dry types.
It ensures that you get the keys of your payload that you expect, that you get the values that you expect,
that basically your data is of the type you expect, that's the schema, that's the structure.
Once you have the proper types, you still have business logic to handle.
This is the second pillar of dry validation.
A typical example would be if you have to handle a deadline.
Imagine that somewhere in your payload there's a deadline.
The first pillar would ensure that the deadline is actually a date because you get a string.
Hopefully it's an ISO 8601 string, but it could be anything else.
You want to coerce that in a string, you want to ensure that you have a string.
If it's not coerceable into a string, you want the first error.
But now that you have a string, you also need to validate that this actual date is in the future.
This is what the second pillar is.
You can create rules, business rules.
That means that once your payload goes through the dry validation mechanism,
you actually get a very valid, very reliable payload from a typing perspective,
but also from a business perspective.
Once we have that payload, what do we want to do with that?
We actually want to process it.
For that, we are using a pattern which is named Interactor.
At least we used to use a gem which is named Interactor.
You can think of an Interactor a little bit like an operation in Trailblazer.
I don't know if anybody has used Trailblazer previously.
No? Okay.
All right. I'm going to go back.
The idea of an Interactor is that this is the entry point to your business layer.
Because the entry point to your application to most of the web application are the controllers.
This is how... I'm not talking about the rules.
Let's consider that the entry point is the controller.
But that's not true because sometimes your entry point is your test.
Sometimes your entry point is a rate task.
Sometimes your entry point is an active job.
Sometimes your entry point is a channel.
So you actually get a lot of entry points into your app.
But at the business level, you don't really care if you want to delete a user
because of a GraphQL request, of a REST request, of an active job.
You want to delete a user. It's the same business unit.
And this is how we encapsulate things we are using in Interactor.
One Interactor is responsible for one business unit.
And well, very fortunately, DRI has a solution for us.
It's a name DRI Transaction.
So their name for it is a transaction.
It allows you to create a series of steps.
It relies on DRI Modad because each step can give you a result.
And if the result is a success, then the next step is going to happen.
If the result is a failure, then the next step is not going to be done.
You're going to keep your failure.
This is known as the railway oriented programming.
Nothing related to rails.
It's just because you either stay on your success track, like train track,
or at each step you have a junction to your failure track.
Well, the thing is we didn't use DRI Transaction.
So I wanted to let you know because I would really recommend that you use it.
I wanted to use it, but also we have a team of several developers
who are used to our Interactors.
And it sounded like a better idea to use what everybody knew
than trying to reinvent the wheel.
We had something, it's working well, everybody knows it well.
So this is like my manager voice talking.
If it's end broken, broken, don't fix it.
But if you're doing it from the start, give a chance to DRI Transaction and DRI Modad.
At this point in the talk, I hoped to try my own definition of DRI Modad,
of Monad, what is a Monad, which is probably going to take the next two hours.
So let's keep it.
So the end of this slide is about why do we want to do all that validation early?
And this was also something a bit new.
First of all, like failing early is a good idea.
But it was not enough because doing the business validation at each step would have made more sense.
It's just easier to keep the business steps together.
It makes more sense if you want to check some permission, then delete a record, then send an email.
It makes sense that you do everything related to sending the email at the sending email step.
It doesn't really make sense to already check stuff from the start.
But the thing is, in Rails, we are very much used to a highly rollback-able environment
because most of what we do, well, sending email doesn't count,
but most of what we do is manipulate the database.
And this is a huge comfort being able to say, my record.transaction do blah, blah, blah, blah, blah, blah, blah, blah, blah, blah.
If anything goes wrong, just roll back and done, nothing has happened.
When you're doing a microservice, at least what we are doing, nothing is rollback-able.
Everything you do, if you send an API request to something, if you delete a file, download a file, create a file, there's no rollback to that.
And this is why it was so important to check as much as we could right from the start.
All right, next step.
Next step, next challenge.
The next challenge was an interesting one, as every challenge, because it was about design and design opinion.
And there's no truth, there's no strong truth in design opinion.
So what was the challenge exactly?
The challenge was that we realized we were not using dry containers properly.
It felt like we were supposed to use it in a new way.
Why was that?
The reason was that we are very used to object-oriented design, object-oriented programming,
which means we are putting together state and behavior in small objects, and they are responsible for doing that stuff.
And the dry system, the dry container, was pushing us to use stateless objects,
because that's what you could enjoy if you want to inject something everywhere.
It better be stateless.
But the code we wanted to write, because we have a lot of experience with that, was stateful.
We don't want a command wrapper.
We want a command execution specifically about this option.
We want to ask a specific invocation.
We don't want the full program.
So it was very important to be able to write the code that we wanted to write,
but it was also important to use the tools properly.
And initially what we did is we had that big interactor, or big entry point,
get injected with a ton of stuff from the container.
It was getting all the services that it would eventually use,
and that interactor was instantiating all the small objects,
the small life cycle objects that it was going to use,
and it was instantiating those objects, giving them their state,
so maybe the current date, the current user, the current payload,
and all the dependencies that the objects needed.
So maybe there's a command service, maybe there's an API client,
so the interactor was instantiating all of that,
which means the interactor knew about almost everything.
There's a name for that.
GodObject.
And it's a bad name.
So we knew we were doing something wrong.
We had a small discussion, and we realized that actually the literature
again had a solution made for that.
There's a pattern made for that.
The pattern is factory.
So what we eventually did is that we created new services, factories,
very shallow services.
Each factory was injected with the services that it needed,
and the interactor was simply injected with the factories,
and the interactor was just asking the factory,
well, give me a command invocation specifically about this file,
about this API, about this payload.
And it's not a fun because it was so difficult to realize at first that we needed that,
but at the same time it was so obvious what was the solution.
It also raised an interesting comparison with a former colleague of mine
who told me he was like a functional programmer.
He, I'm not going to say despised, but he despised object-oriented programming.
Well, I said it.
And he told me, you know, an object is just a set of partially applied function.
He was very like this day in for like, oh, it's just a set of partially applied function.
We have like, we have object at home.
Well, it's not the same.
But to be honest, like, introducing those factories gave me that feeling
because we had like those functions.
We were partially applying all the dependencies.
That's like first partial application,
and then we were partially applying the state.
It also opened our mind about what is stateless, what is stateful.
Usually state is like all your instance variables.
It's not really true.
You don't see things like this anymore.
Like your dependencies might still make you stateless.
And your state is really what makes an object throwable.
So if it's a reusable object, it's stateless.
If it's like a one-use object, it's stateful.
That's sort of our new definition of that.
And factories helps us creating one-use object
because factories are all stateless object.
Well, I felt bad creating the slide without mentioning a single dry gem.
So I also want to bring one here, is the dry initializer gem.
And to be honest, this is my favorite, and it's so small.
The thing is this is so small that it's crazy that this is my favorite gem.
It creates contractors.
It just creates an initialized method.
But why does it matter?
Because if you are very strict about it,
all your initializer very probably look the same.
It's like you pass them arguments,
and then you store them into instance variable.
Nothing more because doing business in initializer is a bad idea.
So you always get the same initializer times and again,
and it makes no sense, and it creates noise.
And if you're used to more style guides,
it has to be at the top of the file,
and it also takes a very important part, focus,
because top of the file is very important.
So dry initializer, just do that.
It says you can create one line for each dependency or state that you want.
You can give it a type. You don't have to,
but if you have a drive type, you might want to.
You can give it a default value,
and you automatically get an initializer that accepts them,
and you automatically create an ATTR reader for each of the dependencies.
You don't want the reader. You don't have to, but by default, you get that.
And that's it, and you just transform something very long and noisy
into a series of lines.
We used to have ATTR reader anyways.
That's most of our classes have ATTR, one line for ATTR reader anyways.
So it changed nothing in terms of noise.
It changed everything in terms of clarity, intention,
and anyone reading a file now gets something directly by reading those lines.
And yeah, I'm still on time.
Well, we were done with the code application.
Of course, we had additional challenges,
but eventually using those tools and approaches,
we reached up the end of the application, and we were done, right?
Well, no, we still had to package it.
We still had to deploy it, because as long as we were actually solving problems,
we had nothing.
This is again a time when we realized how rich is the Rails ecosystem,
because for deployment, you either get services like Heroku or similar services,
or using Capistrano, which does everything for you.
You write one CAP file and everything is magic.
When we had to deploy, we were like, yeah, we have files with code,
but we still have no application.
So we get some help from partners about that.
We use Docker Compose locally for creating containers.
We use Kubernetes remotely for deploying them.
We use Helm for actually doing the deployment.
And this led us to realize that we still had problems,
because we had no observability.
We had very difficult access to the log files,
so there was still a lot of stuff we didn't have.
So what we did is we introduced Yabbaida from Evil Martian.
I don't know if anybody is from Evil Martian here,
but if you are and if you watch us, like, thank you, you're awesome Evil Martian.
So we used Yabbaida, which is an observability framework.
It allows you to mention what you want to observe, create metrics,
without having to care like where you intend to put those metrics,
what we intend to do with those metrics.
And then another part of Yabbaida, you can mention, like,
actually what you want to do.
You can separate the two.
So your business logic is not riddled with, like, technical details about monitoring.
So this observability allows us to expose some metrics,
which in turn enabled us to create autoscaling to measure health.
So these are typically stuff that you get for free in Rails
if you're using your relic or data.
But we had to do it by hand.
And we finally reached our latest challenge,
because we are not experts in Helm or Kubernetes.
We are actually very noob in that.
So we had partners helping us.
But those partners are also responsible for, like, running and ensuring that
our app is working properly.
So the way the agreement we had with them is they handled their own repo
with everything they do about us.
And we have our own repo with our code base.
And the problem we realized, and we still haven't solved,
is that part of the application is actually in the infrastructure.
And this is something we are not used to do in Rails.
But typically the queue we use have a dead-letter queue.
If you try to read something and it fails, so you release,
you retry to receive, it fails.
After sometimes that message, you put it into a dead-letter
because you don't want to lose waste more time trying to handle that.
Another aspect is buckets have life cycle.
If a file is forgotten there after 24 hours, you want to delete that file.
You don't want to pay fees for that file for the rest of your life.
And this is application logic.
Even though it fits in infrastructure, like it is application logic.
And this bothers me because application logic, anyone who clones a repo
should be able to see everything, to know everything.
They don't have to be master at everything.
They don't have to change everything.
But cloning a single repo should explain everything there is to know about this app.
So at the moment we still have those two repo.
One is like focusing on the infrastructure.
One is focusing on the code base.
Hopefully we will solve that very soon.
But with that done, we actually had the app deployed,
monitors scaled, we learned quite a lot.
We actually made a blueprint out of that,
so we are creating several other workers right out of that.
And we feel much more confident actually using Ruby for something else
than web application.
So thank you, everyone, for your time.
Thank you.
Any questions? We have two minutes of questions, hopefully.
You've talked a lot about...
I mean, first you never talked about Rails,
but you actually miss it a lot.
It's pretty funny that it was not about Rails, but actually...
Anyway, you talked a lot about types.
Is that something more to bring to the rest of the ecosystem?
Yeah, that's a very good question.
So the question is, I talked a lot about types.
Do I want to bring that into Rails?
Actually, the interactor is something we do in Rails already,
which means we are using dry validation already,
which means we are using dry types already.
To be fully honest, we don't use it enough.
We sort of use it when we realize that we should have used it before.
So it's like not good enough,
but it is something we are using,
and types have been very helpful in the past already.
And there's a lot of other tools that we have discovered here,
because we had to, and I very much hope that we are going to use them.
But also, my first slide means that I'm no CTO, I'm no manager,
which means I don't get to make those calls anymore.
And it's very important to me that the one who writes the app
are responsible for writing it, maintaining it, running it,
so I can influence, I can give my opinion,
but I don't make those calls anymore.
Yes?
You said that you use dry monot.
What has been, can you tell me more about your experience,
because I used it quite extensively in the past
before they introduced these two notations.
And it was very sticky to the code as in,
it made Ruby not look like Ruby, like something else.
So, if there is something changed there, how's your experience?
All right, so the question is, do I use dry monot?
What do I think of the do notation,
and how Ruby-esque does it feel?
Is that right?
Yes.
Okay.
So I am not using dry monot, except for like toy projects.
So we are not using dry monot in this,
so our own take is using our own interactors.
So whatever I'm going to say is out of my experience on toy projects.
I've learned initially about monads in Haskell.
This is still very painful to me 10 years later.
So my take on monads is like, most of the time,
it's like not the right tool.
And it's something that people,
the learning curve for understanding what is a monad
is so high that once you've earned the right to understand what it is,
you want to put it everywhere.
A little bit like meta-programming.
So this is my take on monads.
I wouldn't force them into anyone who is not very comfortable using them.
I do believe that it is a very elegant solution,
but I also do believe that sometimes a bunch of if-else
makes the team happier than using the best tool for the occasion.
And I don't have any opinion about the do notation and how Ruby-esque it feels.
All right, thank you.