[00:00.000 --> 00:11.000] Hi, everyone. I would like you to give a big welcome to Yaniv, who is coming from Israel [00:11.000 --> 00:15.000] to speak about building event-driven applications. [00:15.000 --> 00:21.000] Thank you. The floor is yours. [00:21.000 --> 00:26.000] Thank you very much for this warm hospitality and for the clapping. [00:26.000 --> 00:30.000] First time in Amsterdam, first time in Belgium, so I'm not sure how many locals from here, [00:30.000 --> 00:34.000] but one of the most beautiful places I've ever seen. [00:34.000 --> 00:40.000] I'm really happy that I got the chance to visit both, both Amsterdam and Belgium. [00:40.000 --> 00:44.000] My name is Yaniv. I'm coming from Israel and Memphis, Dev. [00:44.000 --> 00:50.000] I prepared a short and sweet tutorial about how to build an event-driven application, [00:50.000 --> 00:52.000] the main building blocks. [00:52.000 --> 00:59.000] And I also prepared a short code tutorial, not sure how much time we would have [00:59.000 --> 01:05.000] for this part of the talk, but we will try to reach there. [01:05.000 --> 01:08.000] So just a quick intro about Memphis, Dev. [01:08.000 --> 01:14.000] Basically, simple as rabbit, robust as Kafka, and perfect for busy developers. [01:14.000 --> 01:19.000] So it's a next generation of messaging brokers and messaging queues, [01:19.000 --> 01:22.000] and that's what we try to build. [01:22.000 --> 01:27.000] A bit about our open source. We are an open source project under Apache 2.0 license, [01:27.000 --> 01:34.000] so 2.2K stars on GitHub, 500 community members, contributors, production users. [01:34.000 --> 01:39.000] And if anyone is looking for an open source project to participate and maintain, [01:39.000 --> 01:44.000] we are always open and welcome new members. [01:44.000 --> 01:46.000] And that's it. [01:46.000 --> 01:51.000] What is an event? I mean, probably I'll go over some very basic stuff [01:51.000 --> 01:58.000] that you probably all know all about, but I find event-driven architecture a bit frightening. [01:58.000 --> 02:01.000] I mean, the terminology itself can be a bit frightening, [02:01.000 --> 02:07.000] but probably most of us are already participating or doing some event-driven architecture application. [02:07.000 --> 02:13.000] So just to create a baseline about what an event is, so it's a short update about something. [02:13.000 --> 02:19.000] It can be an order that took place. It can be a Kubernetes resource issue. [02:19.000 --> 02:25.000] Unfamiliar login. We see a lot of use cases around cybersecurity regarding cloud events [02:25.000 --> 02:32.000] and things that are not supposed to happen around our environment or premise or VPC. [02:32.000 --> 02:39.000] So we do a lot of event-driven application or event-driven workflows around that area. [02:40.000 --> 02:45.000] A network endpoint that got disconnected. All of that are basically events. [02:45.000 --> 02:52.000] And it would most probably look like that structure of day description and status. [02:52.000 --> 02:58.000] And it's important to save the status because we will see in the tutorial overview [02:58.000 --> 03:05.000] that we want to keep the state of the event because usually that event will go through [03:05.000 --> 03:10.000] several consumers, several processors, and we want to make sure that we are completely aware [03:10.000 --> 03:16.000] to the different stages of that event as it goes along through the pipeline. [03:16.000 --> 03:24.000] Event-driven architecture, EDA. All of a sudden, I think in the past two years, [03:24.000 --> 03:32.000] it's sort of turned into a buzzword. We see a lot of EDA articles and tutorials [03:32.000 --> 03:39.000] and features and companies that goes around EDA, so just to create a baseline on that part. [03:39.000 --> 03:46.000] So event-driven architecture is basically you have an event, something happened in our environment, [03:46.000 --> 03:52.000] and we trigger an action based on that. And when I say event, it's not that. [03:52.000 --> 03:57.000] So change the background to red and then all of a sudden the background is turned into red. [03:57.000 --> 04:02.000] It's most like order received and then we need to prepare the order. [04:02.000 --> 04:08.000] If, for example, we're talking about Deliveroo or DoorDash or Uber, [04:08.000 --> 04:12.000] chain of reaction happens based on that event. [04:12.000 --> 04:18.000] So it's not just like creating a trigger or a meet that does something on that event. [04:18.000 --> 04:26.000] It's basically an entire pipeline, an entire flow that goes, the trigger because of that event. [04:26.000 --> 04:31.000] So act on events on real time and usually more than one action per event. [04:31.000 --> 04:34.000] So it's not the changing background to red. [04:34.000 --> 04:41.000] It usually will be preparing the order notification for the customer, for the client, BI delivery. [04:41.000 --> 04:47.000] Again, according to the use case, but more than one action per event. [04:50.000 --> 04:52.000] I thought this one would be funny. [04:53.000 --> 04:57.000] Anyway, I managed to get some. [04:57.000 --> 04:59.000] So main building blocks of EDA. [04:59.000 --> 05:01.000] So first of all, defining the events, right? [05:01.000 --> 05:05.000] We need to understand what are the events that require certain actions. [05:05.000 --> 05:09.000] So we're defining the events and then we're defining the actions, right? [05:09.000 --> 05:16.000] So preparing the order, push to delivery, report to some executive about the BI or revenue, [05:16.000 --> 05:18.000] the change or something like that. [05:18.000 --> 05:20.000] Update databases. [05:20.000 --> 05:26.000] If it's, for example, the Kubernetes example that I gave, we need to trigger more resources. [05:26.000 --> 05:30.000] We need to create more EBSs, more EC2s, auto scaling of resources. [05:30.000 --> 05:32.000] So that would be the actions per event. [05:32.000 --> 05:36.000] So we usually start with mapping the events that are important to our use case, [05:36.000 --> 05:40.000] to our organization, to our platform, and then define the actions. [05:40.000 --> 05:43.000] And then we go for coding. [05:43.000 --> 05:52.000] And the building blocks or the infrastructure that supports that architecture is basically microservices and queue. [05:52.000 --> 05:57.000] We'll go in a second, why microservices and why queue. [05:57.000 --> 06:01.000] I know there is a long debate about microservices. [06:01.000 --> 06:05.000] I thought that we got over it about like two years ago. [06:05.000 --> 06:11.000] And microservices won the big debate of monolith or microservices. [06:11.000 --> 06:13.000] But apparently not. [06:13.000 --> 06:16.000] I mean, there are many, many debates in the social about it. [06:16.000 --> 06:19.000] I'm completely with microservices. [06:19.000 --> 06:26.000] So definitely doing a monolith would get you faster to what you or where you want to get. [06:26.000 --> 06:33.000] But definitely regarding scale, decoupling, event-driven architecture, microservices is your answer. [06:33.000 --> 06:42.000] And when we have microservices, we need a queue to create the asynchronous or the buffering layer between the microservices [06:42.000 --> 06:47.000] that will be responsible for the async transportation between the different services. [06:47.000 --> 06:54.000] So decoupling services, queue will basically enable us to decouple different services. [06:54.000 --> 07:01.000] So service A, not really aware if there is something on the other side that is waiting to consume. [07:01.000 --> 07:08.000] The data is just push it and then acknowledge back to the client or to the front-end or whatever [07:08.000 --> 07:11.000] that event started to be handled. [07:11.000 --> 07:15.000] Thread pool, if we have, for example, the need for scale. [07:15.000 --> 07:25.000] So if our restaurant or Uber app or food delivery app all of a sudden from one order per day [07:25.000 --> 07:29.000] got to scale into one million orders per day. [07:29.000 --> 07:34.000] So we need to scale the different microservices and a queue is perfect for that part. [07:34.000 --> 07:36.000] And real-time pipelines. [07:36.000 --> 07:44.000] So queues started as a buffering layer between services, between applications. [07:44.000 --> 07:51.000] But in the past three years, I would say, it also started to enable different parts. [07:51.000 --> 07:56.000] You probably read about them in event streaming or event processing in Kafka. [07:56.000 --> 08:06.000] But these days, you also can use queues as a temporary database for real-time pipelines, for example. [08:06.000 --> 08:11.000] What do I mean by real-time, by temporary databases, basically that your queue in the middle [08:11.000 --> 08:17.000] or actually a message broker, because the queue most of the time implements one-to-one pattern [08:17.000 --> 08:19.000] and message broker is one-to-many. [08:19.000 --> 08:25.000] So if, for example, a queue or a broker in the middle, if we have it [08:25.000 --> 08:30.000] and we have a bunch of or chain of reactions that needs to take place [08:30.000 --> 08:33.000] and process the data as it flows between services. [08:33.000 --> 08:38.000] So we basically push the data or the raw event into the queue [08:38.000 --> 08:45.000] and then Microservice B will take it, process the data, throw it to Microservice C [08:45.000 --> 08:52.000] that will process the data and then at the end will preserve it in some database [08:52.000 --> 08:58.000] or do finalize action to end the workflow [08:58.000 --> 09:01.000] and it's all enabled by the queue in the middle [09:01.000 --> 09:06.000] and the entire transportation and orchestration of the data between the different microservices. [09:06.000 --> 09:12.000] So yeah, tutorial, how much time do we have left? [09:12.000 --> 09:14.000] 10 minutes, something like that? [09:14.000 --> 09:17.000] 15 minutes, okay, awesome, thank you. [09:17.000 --> 09:22.000] So quickly about the tutorial that I've prepared, nothing fancy [09:22.000 --> 09:29.000] just to explain the real-time pipeline and how it implemented using a queue or a broker. [09:29.000 --> 09:36.000] So we have the customer on the left side, it's sort of like an Uber Eats sort of application. [09:36.000 --> 09:39.000] The customer creates an order through the front end [09:39.000 --> 09:43.000] and the front end preserve the data onto a MongoDB [09:43.000 --> 09:48.000] and push the events of that order into Memphis queue orders [09:48.000 --> 09:52.000] and then we have another service that creates the processing. [09:52.000 --> 09:57.000] Again, preserving the state, if you remember at the first slide [09:57.000 --> 10:03.000] I talked about preserving the state as it goes along through the different services [10:03.000 --> 10:07.000] because if, for example, all of a sudden something happened to the processing service [10:07.000 --> 10:12.000] or the delivery service, we know when or from where to process [10:12.000 --> 10:18.000] or to reprocess that order because we don't want to create and duplicate orders [10:18.000 --> 10:22.000] because processing or delivery all of a sudden crashed or something like that [10:22.000 --> 10:25.000] which happens a lot in real-time pipelines. [10:25.000 --> 10:30.000] So that's the general idea, so it goes through processing [10:30.000 --> 10:34.000] saving the state or updating the state on MongoDB [10:34.000 --> 10:39.000] again, push that order to Memphis queue deliveries [10:39.000 --> 10:42.000] where we have some delivery person that takes that order [10:42.000 --> 10:46.000] and create a delivery to the happy customer on the left. [10:46.000 --> 10:53.000] So let's see some code and how it looks like in reality. [10:53.000 --> 10:55.000] So again, nothing fancy. [10:55.000 --> 11:00.000] This is what it looks like if we had like a Flask API [11:00.000 --> 11:04.000] using Flask framework to create that API endpoints. [11:04.000 --> 11:09.000] It starts with an index which gives the rendered frontend [11:09.000 --> 11:12.000] and then we have the order food API endpoint [11:12.000 --> 11:16.000] and that would be like the MVP to create the ball of prey. [11:16.000 --> 11:17.000] Sorry. [11:17.000 --> 11:22.000] Oh, and to increase, like most zoom. [11:22.000 --> 11:24.000] Okay, thanks. [11:26.000 --> 11:29.000] One second. [11:29.000 --> 11:31.000] Is it big enough? [11:31.000 --> 11:32.000] Yes. [11:32.000 --> 11:34.000] Okay, thanks. [11:34.000 --> 11:36.000] So, sorry. [11:36.000 --> 11:37.000] One more. [11:37.000 --> 11:39.000] One more? [11:39.000 --> 11:43.000] One more. [11:43.000 --> 11:45.000] Okay. [11:45.000 --> 11:47.000] Big enough. [11:47.000 --> 11:49.000] Thanks. [11:49.000 --> 11:50.000] Okay. [11:50.000 --> 11:54.000] So that would be our boilerplate, [11:54.000 --> 11:57.000] but if we want to take it to the next level, [11:57.000 --> 12:01.000] let me just remove that part, [12:01.000 --> 12:03.000] but if we want to take it to the next level [12:03.000 --> 12:05.000] and add a queue into it. [12:05.000 --> 12:08.000] So let's quickly go over the different parts. [12:08.000 --> 12:09.000] Let's start from the main. [12:09.000 --> 12:12.000] So we basically have the order food endpoint. [12:12.000 --> 12:16.000] So a trigger will go to the order food endpoint [12:16.000 --> 12:20.000] and then we basically print some output for that [12:20.000 --> 12:22.000] new order received, we'll print it, [12:22.000 --> 12:24.000] then we'll create a current time, [12:24.000 --> 12:27.000] we'll stamp the order with the current time, [12:27.000 --> 12:29.000] order date with that current time, [12:29.000 --> 12:31.000] status means the state of it, [12:31.000 --> 12:34.000] so it's a new order, generate ID, [12:34.000 --> 12:41.000] and then create a new document inside our NoSQL database [12:41.000 --> 12:44.000] and then push the order towards processing, [12:44.000 --> 12:48.000] so meaning the part that the event goes through Memphis [12:48.000 --> 12:51.000] for further processing, [12:51.000 --> 12:53.000] that would be the part that handles that. [12:53.000 --> 12:57.000] Let's see, for one second, it goes all the way up, [12:57.000 --> 12:59.000] so we basically connect to Memphis, [12:59.000 --> 13:02.000] connect to the cluster itself, create a producer, [13:02.000 --> 13:06.000] and then produce that event to the station. [13:06.000 --> 13:09.000] We'll see it in a minute, like in a more visual way. [13:09.000 --> 13:12.000] And then afterwards comes the part three. [13:12.000 --> 13:15.000] I mean, part two, actually, the process itself. [13:15.000 --> 13:19.000] So the process itself, basically there is a component [13:19.000 --> 13:22.000] that listens to events coming from Memphis. [13:22.000 --> 13:26.000] So Memphis implements the paradigm of produce and consume, [13:26.000 --> 13:31.000] so it's not pushing the events into the processing part, [13:31.000 --> 13:33.000] but actually the processing service [13:33.000 --> 13:37.000] consume that event and start acting based on that. [13:37.000 --> 13:41.000] So it's an event-driven architecture that we describe. [13:41.000 --> 13:44.000] It's idle until something happened [13:44.000 --> 13:48.000] or some order got into the queue. [13:48.000 --> 13:55.000] So new order received, just short parsing into JSON, [13:55.000 --> 13:58.000] also creating a quarantine for that part, [13:58.000 --> 14:00.000] preparing the dish. [14:00.000 --> 14:02.000] It's a pretty fast restaurant. [14:02.000 --> 14:04.000] Order is ready for delivery. [14:04.000 --> 14:07.000] And we're changing the state to delivery [14:07.000 --> 14:09.000] and hack the message. [14:09.000 --> 14:12.000] The acknowledging message in every queue or broker [14:12.000 --> 14:14.000] means that I received the message, [14:14.000 --> 14:17.000] I did my processing, my handling, [14:17.000 --> 14:20.000] and it's okay to send me more messages. [14:20.000 --> 14:22.000] So that's the asynchronous part. [14:22.000 --> 14:26.000] So we do some filtering, new values, [14:26.000 --> 14:30.000] according to what MongoDB asked us to do, [14:30.000 --> 14:35.000] and we update that document, the state of it, [14:35.000 --> 14:38.000] so it goes from new order to delivery, [14:38.000 --> 14:40.000] so it's ready for delivery, [14:40.000 --> 14:42.000] and then we send the delivery, [14:42.000 --> 14:45.000] or we send the order to the delivery part. [14:45.000 --> 14:47.000] And that's, I would say, [14:47.000 --> 14:51.000] that's the main idea of doing asynchronous movement, [14:51.000 --> 14:55.000] because at the moment we are a young startup, [14:55.000 --> 14:57.000] we have only one delivery person, [14:57.000 --> 14:59.000] and all of a sudden we got scaled. [14:59.000 --> 15:01.000] And we need more delivery person. [15:01.000 --> 15:06.000] We don't want to add, like, more power into that compute, [15:06.000 --> 15:10.000] but we want to add more, to scale out our resources [15:10.000 --> 15:13.000] and add more threads, add more workers. [15:13.000 --> 15:15.000] So we would scale the delivery part, [15:15.000 --> 15:18.000] so instead of using just one service for delivery, [15:18.000 --> 15:21.000] all of a sudden we can just scale out [15:21.000 --> 15:24.000] to 10 or 100 different delivery persons, [15:24.000 --> 15:27.000] which is a delivery services. [15:28.000 --> 15:33.000] So we send the delivery event into the delivery queue, [15:33.000 --> 15:36.000] which looks pretty the same as the processing, [15:36.000 --> 15:39.000] so we will start with consuming, [15:41.000 --> 15:44.000] and then hacking the message, [15:44.000 --> 15:48.000] actually on that part right in the beginning. [15:48.000 --> 15:54.000] I just wanted to show two different ways to doing so. [15:54.000 --> 15:56.000] So in streaming, or in data streaming, [15:56.000 --> 15:59.000] basically we want to acknowledge the message really, really fast, [15:59.000 --> 16:02.000] because streaming is made for scale [16:02.000 --> 16:05.000] when we got to the place that we want to queue, [16:05.000 --> 16:07.000] or a message worker would probably handle [16:07.000 --> 16:10.000] or handling large scale of data, [16:10.000 --> 16:13.000] so we want to quickly acknowledge the message [16:13.000 --> 16:16.000] to free up our buffer and receive more messages. [16:16.000 --> 16:21.000] So at the previous service, at the processing service, [16:21.000 --> 16:24.000] we acknowledge the message only after the preparing, [16:24.000 --> 16:26.000] which is perfectly fine, [16:26.000 --> 16:28.000] but in this part, again, [16:28.000 --> 16:32.000] if all of a sudden we have massive scale that we need to handle, [16:32.000 --> 16:36.000] in order to avoid back pressure on our broker, [16:36.000 --> 16:38.000] we absorb that back pressure [16:38.000 --> 16:41.000] and doing a fast acknowledging on our client, [16:41.000 --> 16:43.000] it's just another way to doing things, [16:43.000 --> 16:49.000] and it based on your use case and how much your use case is sensitive [16:49.000 --> 16:53.000] to avoid, for example, a message or an event. [16:53.000 --> 16:55.000] Thanks. [16:55.000 --> 16:58.000] A message or event, because if, for example, [16:58.000 --> 17:00.000] we acknowledge the message, [17:00.000 --> 17:02.000] and all of a sudden, on that part, [17:02.000 --> 17:06.000] our service will go down for some reason, [17:06.000 --> 17:09.000] we basically will lost that event [17:09.000 --> 17:11.000] and we will not deliver it, [17:11.000 --> 17:14.000] and the customer will not be so happy [17:14.000 --> 17:17.000] and we will need to re-trigger the entire workflow. [17:17.000 --> 17:19.000] So just another way to doing stuff, [17:19.000 --> 17:22.000] when we work with massive scale like Uber, [17:22.000 --> 17:24.000] like Netflix, [17:24.000 --> 17:27.000] like Deliver or something like that, [17:27.000 --> 17:30.000] it definitely needs to be in that way [17:30.000 --> 17:34.000] and here will maybe come another action [17:34.000 --> 17:38.000] or a usage of a cache database like Redis [17:38.000 --> 17:42.000] that will preserve that event just for that time, [17:42.000 --> 17:44.000] just for the processing time. [17:44.000 --> 17:48.000] So again, as we did in the previous services, [17:48.000 --> 17:50.000] we print some output, [17:50.000 --> 17:54.000] we update the database with the status delivered, [17:54.000 --> 17:56.000] that's that part, [17:56.000 --> 18:02.000] and we basically print everything to our own logging and auditing [18:02.000 --> 18:07.000] and everything will be also updated inside the database, [18:07.000 --> 18:09.000] so we will be able to observe that order [18:09.000 --> 18:14.000] and the entire life cycle of ordering, processing and delivery. [18:14.000 --> 18:17.000] So I will do, [18:17.000 --> 18:20.000] I will just run it really, really quickly [18:20.000 --> 18:23.000] because I also, and by the way, everything is written in medium [18:23.000 --> 18:27.000] and you have a GitHub repo if you want to just check out the code a bit [18:27.000 --> 18:29.000] and play with it, so you have a Docker Compose, [18:29.000 --> 18:31.000] you just need to run it. [18:31.000 --> 18:36.000] It's a live demo, [18:36.000 --> 18:44.000] so I hope that it will be all right right now. [18:44.000 --> 18:50.000] Okay, so we started everything, all the services are up. [18:50.000 --> 18:55.000] Let's see, we have the front end, the process and the delivery are up [18:55.000 --> 19:01.000] and we have a local community Mongo that we used just for the preparation [19:01.000 --> 19:07.000] but we could use Atlas or any other Mongo that we wanted [19:07.000 --> 19:16.000] and just shoot an event or order some food and see what happened. [19:16.000 --> 19:20.000] So I have Memphis Sandbox here, Hope End, [19:20.000 --> 19:24.000] which emphasized the queues within it, [19:24.000 --> 19:28.000] so if I'll go to orders new, I should see our new order here. [19:28.000 --> 19:33.000] Let's see, it's a bit small, it's small here as well. [19:33.000 --> 19:38.000] So we have the status of the order, we see the order itself, [19:38.000 --> 19:43.000] we see on the other side the processing service that takes the data, [19:43.000 --> 19:49.000] do something, sorry, something that does something [19:49.000 --> 19:53.000] and push it towards the delivery queue [19:53.000 --> 19:58.000] and we should see the same event here, [19:58.000 --> 20:02.000] so ready date, order date, exactly. [20:02.000 --> 20:10.000] So we'll go to the MongoDB and refresh it a bit. [20:10.000 --> 20:13.000] We should see our order here as well, [20:13.000 --> 20:19.000] so we can see the entire state as it goes or as it updated through the different services. [20:19.000 --> 20:24.000] We have the order date and three seconds after the dish is ready [20:24.000 --> 20:29.000] and four seconds after it's already sent to the customer. [20:29.000 --> 20:35.000] So that's the main event and it happened entirely as synchronously [20:35.000 --> 20:38.000] and it's a great hamburger and yeah. [20:38.000 --> 20:42.000] So that's what I wanted to show [20:42.000 --> 20:48.000] and again as we talked about the debate of microservices versus monoliths [20:48.000 --> 20:50.000] so I wrote a lot of articles about it. [20:50.000 --> 20:56.000] I started my journey with data many years ago, [20:56.000 --> 21:01.000] four years ago, something like that, especially with data streaming. [21:01.000 --> 21:06.000] We built a data streaming based AI project [21:06.000 --> 21:12.000] that basically analyzed using LDA, Twitter and Facebook [21:12.000 --> 21:15.000] to get the general conversation of the public [21:15.000 --> 21:23.000] and the need for scale and agility really takes place very, very, very fast [21:23.000 --> 21:29.000] so I really recommend everyone to start even not to build with microservices [21:29.000 --> 21:33.000] and queues and asynchronous patterns [21:33.000 --> 21:38.000] but at least try or think about the refactor that you will need to do [21:38.000 --> 21:43.000] if you decided to go with the monolith and not even driven architecture. [21:43.000 --> 21:46.000] So that's it. Thank you very much for this opportunity. [21:46.000 --> 21:48.000] Really happy to be here.