Okay, now we have Nelson Vides with the actor model as a load testing framework.
Give it up.
Thank you very much.
Thank you for coming.
Let's get started.
As you heard, I'm Nelson Vides.
We only have so many minutes, so I'm not going to go deep into an introduction of who I am.
Just ask me on the corridors.
I love talking.
I'm Senior Erlang Consultant for Erlang Solutions and Core Mongolian Developer, messaging back-end,
different questions.
Again, ask me on the corridors.
I would love to talk about it.
Let's start with an analogy, an intro, a catchy intro.
Now let's see how the internet works.
While this loads, and I hope it loads, otherwise I have it downloaded,
I had a fantastic teacher in high school, a fantastic physics teacher,
kudos to him, whatever he is, hello.
When we were studying aerodynamics and the Newton laws,
we studied this bridge that is not loading.
I think I will just save time and reproduce it here.
Don't ask me how to make it bigger.
Back in the 40s, they built a bridge in the Tacoma Narrows in Washington State,
crossing from Tacoma to the peninsula, the other side of the Narrows,
and the bridge had that problem.
It had a very spectacular build.
Through the build, they already realized that this is happening,
that the bridge is not really stable, and very shortly after the grand opening,
they had to evacuate.
They left one car with the only casualty.
Unfortunately, a dog was left inside of that car, the only casualty of this accident.
This spectacular happening, if you check it on Wikipedia,
it will be written that something like this left a mark in the history of engineering,
engineers went all mad and crazy, what happened here, what mistake have we made?
Eventually, the bridge fell in 1940, so then there was World War II,
they didn't have a chance to build it.
In the 50s, they built a new one.
The old one, these pieces that fell are now a fantastic house for fishes in the bottom of the river.
Let's go back to the presentation.
Yeah, this never loaded, good that I don't load it.
Why am I talking about this bridge?
Back in the days, bridges were like this.
In the Roman times, it was a solid piece of stone,
you could just hammer it in all directions, it was just solid.
What is the load that this bridge was having?
A few Roman centurions walking, a hundred of them at a time.
How heavy that is?
Some armory, what armors they had anyway?
It was not like big modern missiles and things that weighed tons.
But one day, we went from bridges like that to bridges like this,
that are very lightweight.
Even if they are much bigger and they spawn way longer distances,
they are way lighter than the previous one and they are not as solid.
So there are forces that didn't used to matter in the previous bridge,
that now make a really big difference.
For example, wind.
The previous bridge put it through a hurricane,
probably like what kind of hurricane you need to do something.
But that bridge, not this one in the picture, this is a model,
but the Tacoma bridge fell under a wind of 40 miles per hour.
It's not that I like miles, sorry, I'm supporter of the international system,
but the Wikipedia article was written by an American, so it's in miles.
How many kilometers per hour that is, I don't know how to convert it.
But it's not a lot, it's not a hurricane.
So my analogy, in the previous bridge,
there was just a few people with a small load and forces that were there,
didn't play any difference whatsoever.
But in the new bridge, there is hundreds of cars with lots of loads,
probably transports of goods and much bigger weapons than in the past,
and forces that were always there really make a huge difference.
Let's have an analogy that matters to us here, we are not bridge engineers.
Not long ago we had these huge computers,
but you can probably just punch them and nothing would ever happen.
If I punch this one, the presentation is over.
That were used by just a few people with a few use cases.
And then we went to this magic infrastructure of God knows what is going on,
of lots of things put somewhere, used by millions of people,
God knows what use case people are finding out.
You know, you probably, you design your service with one or two use cases in mind,
and then people surprise you.
So, the questions again.
What are all the interactions?
There was one or two use cases, but one or two people now is the limit.
What is the traffic capacity?
In the Roman bridge, there was Centurion, an army, a small army,
a division with a few weapons.
Now, just imagine a modern bridge.
What about the amplifying factors?
The problem with the wind asked me in the Q&A or in the questions like the details
of why this bridge fell.
I love that story.
There was a little bit of wind that amplified the movement to more than the bridge would support.
This can happen also to us.
Imagine a client sends a packet that is compressed.
We decompress it and, you know, he sends half a kilobyte,
but we decompress it and it's five gigas.
And, you know, you run out of memory.
What about amplifying factors?
And what about all forces that didn't make any difference?
For example, punching a computer.
That now they really do.
All right.
Let's get with a little bit of terminology.
I'm coming back to the title of my presentation.
What is a framework?
Here you have a bunch of copy-pasted definitions from different dictionaries.
And Wikipedia is the first, which is not the best dictionary, but we all love it.
Basically, probably you have an idea like Phoenix is a web framework, for example.
It's a set of tools that gives you a way to build a system to solve a problem.
In turn, what is a model?
You can have a model of a bridge, but you cannot have a framework of a bridge.
You have a framework to build a bridge and a model that represents the bridge.
Again, some copy-pasted definitions from diverse dictionaries for you to enjoy.
And ask me later.
This model, in particular, is the inverted model of the catenarius of the Sagrada Familia.
Again, ask me.
I love this topic, but we are here to talk about Erlang.
This is how Gaudí designed the Sagrada Familia.
That is just about to finish any day now.
Let's some data.
We'll finish it.
So we have a framework, a set of tools to solve a problem, and a model, a representation,
a theoretical representation of your problem set.
Testing and load.
Testing, like kids go to school and they get a test, just to prove that they know what they're supposed to know.
It's a process of making sure that things are doing what they're supposed to do, that they know their knowledge,
that the software does what it's supposed to do, et cetera.
And load.
This is what Newton would probably love to call work.
Again, thank you, physics teacher.
Probably what Newton would love to call work is a mass of quantity of something that has to be worked on.
Like, moved, or supported, or resisted against gravity, or wind, or transported in these virtual bridges of cables that we have under the ocean, et cetera.
So load testing is testing that the software, a service, can handle the load that we are giving it.
And how it behaves under different such quantities.
So we have this roughly scheme of, like, three points of performance testing, of load testing, that you have to test.
Performance is basically how fast your algorithm is, like, executed once.
It takes 10 seconds, or 10 nanoseconds.
It's the theoretical performance, but what happens when you make a lot of requests at the point where you expect your service to still be able, but not more than that.
It depends on the hardware you deploy, your architecture.
You expect that this should behave like this, and then you test it.
And then you put more load and see how it dies.
We have this luxury in IT that we can destroy our software, because we can just replicate it, build infinite copies.
You know, the bridge guy would be very yellow.
He cannot build two bridges to break one.
He has no second chance.
There is one bridge.
Don't break it.
It's very expensive.
Make sure it works.
How do you test what happens when it dies?
So a load testing framework is going to be, of course, a set of tools that gives you a way to test these different kinds of loads.
And for these kinds of loads, you need some units of measurement.
What is a load?
In the case of the bridge, Newton would love to call that the forces.
And you need the interactions.
How are these possible loads applied?
You know, in the case of the bridge, we would usually think of gravity.
There is just one interaction.
It goes down, but wind and turbulence and your users can be very crazy.
Forces can be applied in any way.
So we need to think about the unit of measurement and the interactions.
So as I said, there is the forces.
Newton would love this.
And the equivalent.
You have a service, some backend that has users.
And as I said before, you would never imagine the ways they find to use your service.
You're usually designed with three or four things in mind, but you know.
So I would say that the equivalent of the forces that can be applied in different directions are, like,
self-independent programs.
Imagine that each one of those users is a program that decides how to apply his force,
decides how to interact.
Like, each one of those many arrows that you can draw in this bridge,
and this is infinite if you get involved with differential equations and, you know,
complicated mathematics, everything moves like crazy.
All those moving arrows can be represented with an independent program on its own.
And those programs interact with each other.
This is the model of the actor that I can imagine that most of you, more or less,
would be familiar with, like, what we do in Erlang and Elixir.
For those of you that are not, the idea, basically, by the way, before I go to the next slide,
this is Karl Hewitt, the guy that named the actor model that put it into paper.
He died a month ago, or almost two, maybe, somewhere in mid-December.
So, a bit of a tribute to him.
Thank you for the theory.
For those of you that may not be familiar with the concept of the actor,
basically, it's the universal primitive.
In a language like Ruby, for example, everything is an object.
You can do whatever, dot something, and maybe it will crash because it's not valid.
The compiler may tell you, but you can.
That's how you design your program.
In a language like Lisp, everything is a function.
Absolutely everything.
You can do whatever parentheses.
And maybe it's not valid.
Maybe it will crash.
Maybe the compiler will tell you before compiling.
In a language like Erlang, everything is an actor.
You can do whatever exclamation marks send a message.
And it's almost never valid.
It's only by a process identifier, or if it has a name, a proper name.
So, this is the model of your program.
This is how you structure the program.
How are we going to load test a service?
Light thickens, and the crawl makes wing to the rocky wood.
This has lots of background.
It's a very personal thing.
First of all, of course, I love Shakespeare, but that's not the point.
I work, as I said, at the beginning in MongoSIM service.
That is an XMPP implementation.
And in XMPP, I don't know why, but I'm very happy about it.
All the examples in the RFC are given with Shakespeare quotes.
So, when it comes to messages, you know, there is Alice writing to, not Alice.
Juliet writing to Romeo from the balcony, and then all the examples are like this.
So, we made a piece of service based on a quote from Shakespeare, the name.
That is called a murder of crows.
I also love Hitchcock.
If you haven't watched it, please watch this movie.
So, there is this library that we created in my team to test MongoSIM on the load.
That is called a murder of crows, because crows are dangerous and are there to kill you and eat your corpse.
So, this is what we try to do, to just kill MongoSIM, see dying, and then try to make it stronger next time.
And with this project, we reflect about the interactions, the traffic capacity,
amplifying factor, all new forces.
So, in the case of a messaging system, there is this vulnerability that happens to everyone back in the day.
You know, there is compression.
Somebody sends you a small packet, you decompress it, and boom, your run out of memory.
These kind of things, you have to look for these amplifying forces, the traffic capacity,
how much traffic each client can send, how many clients can you have, all new forces.
Something that may not be a surprise for old schoolers, Erlang developers.
This new world of cloud, that is someone else's computer, really.
If all your microservices connection are a lot less stable, distribution is not as cool and easy
as it was when Ericsson made it and hardware was indestructible, you know, the punching theory.
Nothing happens.
Now it dies.
So, all new forces that now make a difference in the new way of building a system.
In the case of MongoSIM, we have these usual use cases, session establishment.
So, you know, somebody logs in, authentication, password, password less, make up your mind.
Send messages.
Obviously, it's all about sending messages.
Fetch in your archive.
You reconnect after a while, you are on holidays, and then you fetch all the messages you lost.
This is stored somewhere.
It has to be stored as you send it.
What is the impact that it has on sending, on receiving?
Joining and leaving group chats.
This is something, and in all classic XMPP, it's a problem to scale, but all classic
with the time happened.
We had solutions for that.
So, I had these problems.
We need to test them.
And we think how to test them.
So, you start your scenario, and at testing time, you need a init, a startup.
Like, start the metrics, start the functionality that is going to coordinate all your actors
when they have some interaction between them.
For example, in a group chat, you are going to create so many actors that then they will
join the same group chat and talk to each other.
Or in a multi-user game, you are going to have millions of users, but they will cluster
in groups.
So, you need to coordinate them.
So, you will start logic to capture users and to coordinate them and join the same group,
et cetera, et cetera.
So, you start all the actors.
After all your init, then you spawn all the process, you know, and each one executes the
program they are supposed to, that they have been coded to do.
And then you run it.
Locally or distributed.
At some point, the load that you can generate doesn't fit in a single computer, so it has
to be distributed, so you need your service to handle the distribution for you.
The purpose of the load testing is checking how your software is going to survive or die
and not implementing the load testing idea.
We want a load testing library that will just give me all the users, give me a way to coordinate
them when I have to, to throttle them when I have to, and the rate that I have to, to
handle whatever place I need to start this load testing.
And I don't want to think about all of that.
I just want to describe the scenario that I'm going to use to kill my service.
So, we build a library that does all that other stuff.
Very important thing is the throttle idea.
In the case of the chat service, imagine that a million users connect exactly at the same
time and looking at the same time.
It's probably not a real use case.
You can test for that, but that is the stress part when you want to kill the service.
That later, you would usually see what happens when you connect 100 per second, and then
you increment 200 per second, 500 per second, 1,000 per second, and you want to have a
functionality that will throttle and progressively increment the rate.
And then seeing your metrics, both load testing library will output to Grafana, your service
that you're testing will output to Grafana, and then see the correlations.
You want actors to wait for the permission.
Am I allowed to do this already?
And the cases, the session establishment, but also joining a group chat, how many messages
are you going to send.
There is this, you know, you have an arrow that is going to be applied in one place.
How big do you want the arrow to be?
You want that arrow to grow incrementally.
And you may want to ask another actor to wait for the approval.
You can tell the throttle logic to tell that actor to wait for something.
And then that actor, which is not yourself, will wait for the action.
For example, in the case of joining a group chat, first you have to create it.
So there is a first user that says to everyone, wait, don't join because I need to create the group chat first.
Voila is created, come here, et cetera.
And another very piece of important functionality is the coordination idea.
So as actors are appearing in your load test, one thing that you will want to do, as I said before,
is to coordinate sets of them.
For example, who is going to write to whom?
So you want an actor to know about another one, so it can send him a message.
You want a functionality that will pick up actors as they are starting in a configurable way,
either all of them that are started or sets of pairs or a list of them.
And once the configurable amount of actors has started, then make them do something.
There is a callback that will get the list of actors that they identify as and will coordinate how they interact with each other.
And, yeah, the actor, as they join the coordinator, they will be given the function that they have to do.
So to us, this is what my load testing framework is supposed to help me do.
We use it for XMPP.
So then we have scenarios and functionality written that knows how to do the authentication for the protocol,
that knows the functionality of Mongoose IM.
But we don't believe that the load testing library is the one that decides your scenario.
I have seen different load testing frameworks that give you functionality to run HTTP requests.
So what if you are not testing something HTTP related?
We believe that the best way to write what you want to test is to write the code that you know how to write anyway.
So the idea is that you write Erlang, Elixir is on the way.
This library is not integrated with Elixir, but we will pull requests accepted.
The library, as I say, is called AMOC, an acronym for an order of crowds because you want to see your service dying.
There is the repo, you can look it up.
We have this other repo that we call AMOC Arsenal where we have all the scenarios for XMPP where you can take inspiration on how they work.
And I'm about to finish here. I propose to myself that I would make this presentation without showing a single line of code.
So I actually cutted the screenshot before the code starts.
Let's see how it works.
In previous presentation I have shown a lot and it's a bit more complicated to explain.
So the library has documentation. Another thing that I have pending is to use the new XDOP documentation.
It doesn't have it yet, but it has a beautiful markdown that you can read in GitHub pages.
And the scenarios library for inspiration.
That is all I will have for you.
This is my handle.
That is to repos links for MongoSIM and for AMOC.
And this is a picture that I have everywhere if you see some Nelson videos and you don't know if it's me.
It's going to be that one if it has that picture.
That's all from me. Thank you very much.
Thank you, Nelson.
So is there any questions?
Yeah.
I know that there's also a library called Zang.
It's a low testing library written in early. So how is this one different?
In that one you write the scenario in XML.
And it has a, how do you call it?
Like a domain specific language, XML base to describe what you want to do.
And the library has to offer you the protocol.
So that library actually has HTTP and XMPP helper functionality.
But if you want a different protocol, the library doesn't give it.
So we thought that we just want to write the airline code.
It's way more pleasant to write and also less limited.
Okay, thank you.
Other questions?
So by using the murder of Kraus, did you already find any like bugs in Mongoose I am that you've been able to fix based on the...
Every single time.
Fair.
Useful bottlenecks sometimes are database interactions.
And all fours that didn't used to matter in the computer you could punch.
But now, so as you write messages, you need to make sure that they are recoverable.
But the amount of messages that you can send might not be as scalable as the amount of inserts a database can have.
So this is something that we test a lot.
And another functionality that we do is the time to delivery.
So the sender puts a timestamp and the receiver just measures the difference.
And that's something that we also test continuously when we change something to see that we didn't introduce a computation that would make the time to delivery longer.
So those are the two most common tests that we test almost all the time and then there are each case that we don't test as regularly.
But we have all the scenarios for them.
Any other question?
I wanted to mention another library I saw that is called MZBench, I think.
I don't know that one.
Yeah, I think it was... I know it because it was used by VernMQ to do its load testing, I think.
And I think it's in Erlang, too, and you write scenarios.
But is Emoq able to... If I have an actor that has to perform some action and then pass the state to another actor, is that possible?
Or is that... You have to write your own code, basically, to do that.
I have. The coordinator would help.
Okay.
So in the coordinator you can say to pick up pairs of actors and then they say...
Okay, you had the first one.
Okay.
Yeah.
Any other question?
We have something similar for changing the owner of a room and then actors have to pass the state, the knowledge to another one.
Okay.
Okay, thank you again.
We'll see if there are any other questions.