Thanks very much, thank you, so welcome everyone and I'm glad that you're here on Saturday
early morning in this first session, so I'd like to make it as easy as possible, thanks
for the organizers, Jerez and Yamur, for inviting me to talk today about stream processing.
The fact is I don't know your background, so I'm not sure exactly how much experience
you'll have with stream processing, so if you see some concepts are easy, just get everyone
up and into this concept.
So I'll be talking today about stream processing on adaptive and there's a lot, so that's what
the main focus will be.
Obviously this title as it is could be a startup company, so you would expect to have some ideas
today where you can use some of these ideas in your work or in your experience or in your
case of study whatever you want or whether you are a Java developer or data scientist
or MLO, so it doesn't matter.
So there is something for everyone here today, so that's the main focus for this session.
So, anyone recognize these guys on the screen here?
Right, so that's where I came from, I'm based in Liverpool in the UK and on the right side
is the Liverpool football club, which is basically one of the top football teams in the UK, so
I wanted just to highlight this screen here, just to tell you that stream processing is
not in specific domain, it could be in any domain.
And if you look at it, do you know how long it takes, for example, for an eye to blink?
Come again?
Yeah, so it takes over half a second, so that's pretty fast.
So if you think about maybe minutes or hours, probably this is not the right discussion
room for you.
We're talking about some milliseconds today, so whether it's for example using it in finance,
whether you use it in IoT devices, smart devices, whether you use it in sports, hospitals or
machine learning or what we're trying to do today for stream processing.
So that's the main idea.
And obviously, if you're working with real-time stream processing, you focus on the real-time
data, right?
And I've seen it so many times where platforms and tools focus on how much data you can process
and you see these benchmarks everywhere on the internet and this is pretty cool, I think,
but the key source and the secret source for this is to use something in combination between
real-time data and historical data.
So the main reason for this is to look at context.
So without knowing what's going on, you probably don't benefit much from the real-time data
you're processing.
So what you want is always to go back and check what's going on with the context of these
data.
There is a problem in this secret source, obviously, because what you're looking at is kind of
like two different data types and you want to make sure that you process it at the same
speed or very close to the same speed.
Obviously, it becomes really a problem when you try to scale it.
So if you have, I don't know, maybe a few cases of data that you want to process, probably
it's not too much trouble for you, but when you start to scale it up, it becomes really
a problem to understand how you want to scale it.
So do you scale your data or do you scale your compute or do you scale both at what speed?
So we will discuss all these concepts today and if I ask you now how much data you process,
obviously, because in this room I would assume over a million transactions per second or
a few millions or, I don't know, some of you might be processing millions of transactions
per second.
So that's pretty good.
And what we want today is to focus this domain into a very specific area.
And this area essentially what we're trying to do today is to analyze traces.
So it doesn't matter if it's like writing system traces or platform traces or it's like programming
language traces.
What we want is to make sure that we have environment and within this environment you
can scale your loads, basically, scale your processing and at the same time we'll provide
some kind of analytics, right?
So again, if you look at how much data you're trying to process, the number by itself doesn't
give you much what's going on here.
So what you want is to find this specific information you're looking for.
Kind of like looking at, you know, finding the needles or finding the hidden areas within
your data.
So if you look at, you know, how much loads you process per day or per week and you'll
store it somewhere on, you know, crystal hard drive or you store it in Mori or you store
it in the cloud.
So what you want is to, you know, make sense of it.
And some companies do this process manually, which means they run software and they go
through their loads and this is kind of a patch service and they try to understand what's
going on within the load.
So obviously this is a problem when you want to scale it and with the scaling you have different
loads stored in different places and you want to make sure basically to have a platform
where in this platform we kind of like looking at some kind of results.
So for the sake of this discussion today, we'll focus on two different solutions.
So one of them is trying to provide some kind of alerts and the other is to provide some
kind of trends within your data.
Obviously I work for a company called Hazardcast.
So Hazardcast as a platform, I love you to do so but obviously you might have heard of
some companies or, you know, they do some kind of stream processing.
So this is kind of like, you know, overview what's going on with this domain at this time.
Obviously you can split it depending on if you're looking for open source solution or,
I don't know, hardware solution or, you know, some kind of management service.
And what you look at is kind of which domain you work so are you looking to capture your
data or some kind of, you know, streaming your data or you want to do some kind of
transformation on your data or do some kind of electrical machine learning.
So you can see that you split it into 12 squares and within these like tools and
platforms are, you know, spread it over.
Some tools not exist on this screen for whatever reason but obviously this might give
you some ideas but it's hard to decide which tool you want to go for.
Simply because I think the distribution is not clear here so it tells you basically
which tool is open source for example and where in process you can use it but it doesn't
give you full picture on, you know, how to do it in practical terms.
And so this is where it might be easier to understand what we're talking about.
So if you remember from my slide where I discussed the historical data and the new data.
So today we're kind of like, you know, trying to split everything into two categories.
So on one side you get like stream processing engines.
So these engines are pretty fast in, you know, streaming events.
And on this far right side you have some kind of fast data stores which are, you know,
are pretty fast in handling data at speed.
So again the solution for lead time stream processing is kind of a combination and you
want to process data in this moment and at the same time you want to actually also
access data storage somewhere. So that's where Hazardcast fits into this area here.
So the platform itself obviously for those who don't know, by the way we have one of
the masterminds of Hazardcast sitting in this room.
So this is the platform.
So it's open source platform.
It doesn't matter where your source is coming from, whether it's Apache Cloud or Apache
IoT devices, for example, I don't know, some kind of device applications or even like within
Hazardcast or even you can write your own connector and you feed it into the platform.
So platform historically used to be two different components.
So the IOMTG and the Jet Engine.
And essentially now it's all back in one, one jar file.
As you see here, it allows you to load your data from hard disks into memory.
So you have access to historical data and pretty much like instantaneously and this
will, well, you know, you can provide context, what's going on with your data.
At the same time, you can actually do stream processing.
So that's what Jet Engine is.
So from here you can do some kind, I don't know, maybe like data transformation or do
some kind of stream processing as we will do today or even like defined machine learning
if you want to.
You can connect it to some clients.
So these are some clients here, so written into various languages.
If you're from data science background, which means your programming languages in general
are not preferable for you, so you might be considering using SQL to do what I'm planning
today.
So this is another option you can do.
And once you process this data where you load it into memory for historical data and
at the same time you have some kind of data coming in.
For example, and you do the combination or even you do transformation, you can proceed
to do some kind of visualization.
So the good thing about Hesicas in general where it comes to scaling is it's partition
aware.
So which means basically your compute, your Jet Engine or your process essentially can
be or can detect where your data is stored.
So this is like, you know, we're trying to have as low latency as possible when it comes
to processing this data.
So this is very important to understand because latency is your enemy when it comes to stream
processing.
So what you want is kind of like having a platform where you avoid network folks.
For example, you avoid IO to your hard disk.
You will try to also avoid every time or, sorry, context switching between threads.
So you want to avoid all of these, but at the same time you want your process to be as
close as possible to your data.
You can avoid some kind of, you know, machine learning on this.
And the scaling itself could be done in various ways.
So the main thing to take away from here is there's no master-worker relationship.
So all nodes basically are peers.
And we've done this study.
It's a bit dated, but it's kind of like one million transactions per second on 45 nodes.
So what we're trying to do now is to add one zero into this number here.
And even though it's pretty impressive, what is nice about it is the linear scaling, which
means more data you can add, you know, more nodes into it.
So that's the historical bit of this talk.
So let's just move to the technical part.
So for this demo, what I wanted is kind of like, you know, show you some ideas, right?
So you should be able to take these ideas and apply it, you know, anywhere.
Obviously the solution as itself could be like, you know, project by itself.
So feel free to edit and change it.
All source code is available on GitHub and the documentation as well.
So you can go through it.
So the main idea when it comes to analyzing or, you know, making sense out of your traces
and logs is to store it somewhere close to, you know, your compute, first of all,
and shouldn't be stored locally, right?
So you want to store it first of all.
So the first thing is to store it on the cloud.
So for this demo, what I'm doing is I'm storing everything onto the cloud.
There is a solution called Hazelgast-Virginian, which is kind of like service.
So you don't need to download GR, run your project.
You can simply plug in and play.
So you can create an account.
You'll run everything I'm discussing today.
So you create an instance of Hazelgast.
And from there, you can pretty much proceed to what I'm planning to do.
So the first option we were talking about is kind of like storing everything into the cloud.
So we're going to import the data. Obviously, we need some kind of trace message,
which makes sense.
So this trace message could be, you know, changed based on how you want to approach it, right?
So for example, if you're working with machine learning,
you probably look for some kind of, I don't know, classification solution for your, you know,
for your tests, or you could be looking for NLE.
If you don't want to work with machine learning, you probably want to look for some kind of trends.
So you look for processing your data.
It doesn't matter if you're using machine learning.
In this case, you want to have some kind of data stored somewhere.
So it could be in JSON format, or it could be like bar charts, strings.
So it depends again how much the speed is important to you.
So the option, first option is to go through the alerts.
So in alerts, what we're trying to do here is to take everything and store it in the cloud.
So obviously we don't store it in the cloud on our disks.
What we try to do is to store it in memory.
My preference in this case is to use some kind of map structure.
So map structure allows you to essentially random access and rebalance between various nodes within your cluster.
And at the same time, you want to have some key value, so in order to know where this is coming from.
So in this case, it could be like ID address, for example, so this is where, and support number, so as key.
And the value could be anything that makes sense to you.
So in this case, for example, you can track level of this error, sorry, of this loop,
and message, for example, if you want to do some kind of NLP processing on it,
and some kind of, you know, process or thread name on this.
Obviously once you have your key and value, what you can do is proceed and store it into memory.
So this is where you get this set to hazard cast.
And what we're trying to do is create the IMAP.
And once you have the IMAP, it means you should be able to store it, you know, access it and do same processing as I will show you.
So first message is to store it in the cloud, store it in memory.
In this case, I'm using hazard cast gradient, and I'm using IMAP.
And second stage is to do the same processing, right?
So there are a couple of options here for you.
So first option is to use SQL.
So SQL is built within hazard cast, which means, or on top of hazard cast,
which means you should be able to query your data, so if you provide some kind of specific messages
that you're looking for, obviously depends on your input, you can do some kind of SQL.
So whether it's an inner joy, for example, or sales and so on.
Or the other option is to do some kind of prediction.
So you're getting some logs, you don't know exactly what's going on to happen next,
and you try to predict to provide some kind of, you know, alerts or trends.
So we need to build the trends.
So in order to do this, what I did is kind of like use the same key,
but for my value, what I'm using is some kind of log score.
So log score is not important.
What I'm saying here is I want to give value for every single message,
or every single log message.
So this could be, for example, how important this specific message is for you,
or it could be, for example, how serious or how dangerous the message is.
So as levels in logs, you can define scores, so instead of having four levels, for example,
you can spread it, I don't know, from one to 100.
So this should give you some kind of predictions.
Why? Because if you have, for example, 10 messages, 10 local messages,
or, for example, warning, you don't know exactly if the event matrix will be warning or not.
If you want to predict it, obviously it doesn't give you how much will be warning or not.
Whereas if you use some kind of numerical value, you can get as close as possible to this.
So we get key from there, we get score from here,
and what we do next is to do some kind of predictions on the logs.
So in this process, what we have is exactly my key and the value,
which is like the score on each log message, and I import it into Hezakas.
So Hezakas allows you to basically input and output from two different maps,
and do stream processing, so we'll build the train based on previous logs,
based on previous log scores, and we'll use the prediction on top of it
to provide some kind of alert.
So zero means don't alert, one means alert.
And as you can see here, the actual workflow, kind of like you build, you read it from math,
then you define trend map, so which is like normal map,
and from there you can use it to predict what's going to happen next.
Obviously you do some kind of visualization, so how does it look like?
So this is kind of the prediction part of it.
So we take the logs map and we build trend map out of it.
So the trend map would start reading messages and the scores and build train for you.
And from this trend I can use some kind of machine learning,
it doesn't have to be machine learning.
In this case it's linear regression, but it could be anything to be honest.
And we check the values and we try to use some kind of prediction based on the previous values
to decide if you want to send an alert or not.
And obviously this is kind of like describing the exact thing,
so when there is a one on your values, it's alert, send alert when it's zero, don't.
And here there are three ways to do it.
So this is where the same processing comes into place.
So you could simply use SQL to read from the map and do some query if you want.
Obviously this is batch, which means it's not real time same processing,
or you can even create a pipeline or create a process.
And from this process you can read the logs and do some same thing.
So you can use either SQL or Java to do it.
But first two options are batch, which means you can process the data in real time,
you want to do changes in real time.
So the third option is the journal map.
So journal map will track all changes, so it is continuous.
So you have the logs stored and you have logs coming into Kafka topic
and you can basically store both into journal map.
So we're on 5.2 version within Hazegas, 5.3 will have the SQL features on top of it,
which allows you, for example, data scientists to just do the queries and change the data.
And obviously it's ring covered, so this is very important to understand,
so you can start processing your data from start or from the end.
And what you want is kind of like, you know, using this kind of alerts to it.
So the first part is to read it.
So this is the actual map we built in the first option.
And from here you can define the key, for example, and the value.
And you start, for example, to do some kind of filtering.
So this happens in real time on continuous and the map itself will allow you to basically track changes.
So to give you some takeaways and best practices,
so we just try to summarize everything we discussed today.
Obviously there are more to discuss, but this should give you something to go out and try.
So first of all, you need to store your logs into some kind of data platform.
So in this case, I'm using Hazegas, but obviously you don't have to use Hazegas.
The idea here is to do some kind of compute on your circle data to provide some context,
as well as real time data.
And from there, you need to store it on the cloud.
So you need to store it somewhere where you can access logs from multiple places.
Obviously it has to be stored in memory.
And from there, you need to choose the format.
If you, for example, looking to provide some, you know, I don't know,
some predictions you probably need to use JSON format.
Or for example, if you want just to do something that you can't sing and it's faster,
if you want to speed the unit, it is some kind of map structure.
Obviously when you store it in memory, because this will allow some kind of random access.
And also you need to consider how you empty your map.
So because you are limited on size, obviously.
And finally, you need to consider security.
So whatever you send to the cloud, you need to make sure that you don't include some, you know,
personal entities or whatever. So if you're interested in this topic,
we're running a conference next month.
So feel free to scan this code.
We provide training for this all free, obviously.
And everything I mentioned today is open source,
so you should be able to do everything I mentioned today.
I'll be steering around here if you want to have a chat
or if you want to discuss it a little bit more.
Obviously within half an hour, there's not much to give,
but hopefully you've got some ideas from this talk.
And hopefully it will be also useful for you.
So with that being said, thanks very much for listening.
I'll open for questions.
Thank you.