Okay, good morning. This time it's a turn to talk about something different. And top
ng. What is in top ng? It's an open source application, of course. We are here. And you
can download the code on GitHub. We'll see the link at the end of the talk. What is in
top ng doing? It is, first of all, a real-time network and traffic monitoring application.
So it means that it displays you on a web interface what is happening in your network
live. Okay, no delay. This is unless, of course, we are receiving flows coming from a router
that are somehow a little bit delayed because by nature they are on average. So they have
a certain lifetime. And it is designed for network monitoring and cybersecurity. It means
that there are some behavioral checks. So we are not bound to rules. You have seen Suricata
representation before. You see there are some rules in case this happens then. So this is
not our case. So we work based on behavior. So it means that if you have a host that is
misbehaving, more or less similar to what you have seen before, that suddenly start
to send too much traffic with respect to the past, or starting to, you know, fire up a
new application. So accept connection on a certain port for TLS that was not open before.
This is a typical example. So therefore it means that the application simply starts up
and learns what is happening on the network. There are some levels of learning. So sometimes
it is an immediate learning because, you know, you specify some sort of configuration. But
usually this is not the case. The case is that the application learns what is happening
and in case something goes wrong, goes different, fires up and alert. This is the idea. And
the architecture is actually divided in two parts. Okay. First of all, the pocket processing
part that is based on more or less PF ring or lead pickup. So this means that you can
run on Windows, Linux, Mac OS, FreeBSD, whatever. Instead, PF ring is something that we have
co-developed that is a Linux technology for accelerating pocket capture, but not only
for that, but also for merging traffic for multiple adapters, for distributing traffic.
So it's much more than simply RX acceleration. And top of this, there is an open source
library that we still maintain at N-top called NDPI. So this is the only open source library
that is doing the pocket inspection. But for us, it means that we try to understand from
the traffic what is the application protocol. So if it's TLS, if it's a generic protocol,
if it's Google mail, but it's a very specific protocol. And out of the traffic, we extract
the metadata. So for instance, we extract certificate information and we generate something
we call RISC. So looking at the traffic, we see if there is something wrong, okay, such
as for instance, an expired certificate just to give you an idea. And we trigger an alert.
On top of this, there is a TopNG because this is the first part that is basically provided
by the operating system. And TopNG has a C++ engine that is processing packets, that is
in essence doing traffic analysis, creates internally, okay, the representation of the
data based on the concept of network interfaces because we can have a multiple network interfaces
from which the traffic is received. It can be a virtual interface such as, you know,
a NetFlow collection or a physical interface, ETH0. And then we have something we call
behavioral checks, where we check flows and hosts. Flows means that each independent communication,
such as a TCP connection, is checked. Instead, a host, we take the host as a whole. So in
essence, if a host is doing a port scan, each individual communication is okay, or more
or less okay. But the fact that this host is doing this, you know, in a sequence, in
a network or in a host, it's a problem. So this is called behavioral checks. And on top
of this, we trigger alerts that can be consumed locally or sent elsewhere. This is the fast
part. On top of this, we have the Lua interface. Why Lua? Because we like C++. But C++ is
something not for everybody, okay? So we need to simplify, for instance, the development
of the web interface. So for instance, the REST API is written in Lua, sitting on top
of C++. So we have created an API that allows us to avoid typical problems of C++ at the
same time we simplify the way the application is working. So therefore, we use Lua for operations
that are not critical, such as the web GUI, or for checking interfaces that are not necessarily
real-time, so for the SNMP. For SNMP, we fetch the data every five minutes and do the checks.
So traffic ingestion, as I said, is done in multiple ways. Sometime is serial traffic,
so packets. Sometime it is not. It's a flow. And this is handled by the C++ engine. So
the C++ engine is doing it efficiently. And then we have other type of ingestions based
on events. So something that we don't really control completely, but that are relevant
for us. So we have seen Surikata, the presentation before, some minutes ago. This is a typical
example of input. Why this? Because, as I said at the beginning, we don't have rules.
We don't want to have rules. So we don't want to say if the payload contains this and this
and this and this then, because we don't believe that this is what we should do. Instead, there
are wonderful tools such as Surikata that are doing that very well. So therefore, the
idea is to combine network monitoring and behavior analysis with these type of tools.
So therefore, indirectly, through tools such as Surikata that is optional, of course, you
don't have to run it with N-top-ng monitoring, you can have this type of information that
can be combined directly by N-top-ng. Of course, we have firewall logs and syslog. Why this
is important? Because we can have a look at information that is not visible from the traffic.
So we always play with packets, me and Alfredo. But we understand that packets have limitations,
okay, especially for encryption. So we have seen before rules saying if you are downloading
a buyer application, that is fine if it's plain text. But if TLS, you will never see
that happen, okay. So you have to use things like rules on top of this, on top of this,
but they are just guesses. So instead, if through syslog or other means, we know that. So for
example, we see an attack or a wordpress saying that this host is trying to guess the password
of administrator user. This is much relevant information. And from the network standpoint,
it looks simply nice. Everything is okay. The problem is from the application. That's
why we believe in network. By the same time, we need to have some other information that
is injected into the application. And of course, we have historical data. We use a database
called Clickhouse. So we can put a billion of records. Everything is working very fast.
This is also an open source database. And for time series, we use round robin database
or influxdp. And as I said before, we have checks that are divided in two parts. C++
for efficiency. So the fast part, in essence, where you have to process traffic in line,
such as when you have a pocket, an incoming pocket, you have to check if this pocket belonging
to a flow is relevant. And then we have other types of checks that are not so real time.
So for a check on an SNMP interface, that need to be easy to be developed. But at the same
time, that don't need to be fast. Because as I said, if we pull SNMP out in five minutes,
we have plenty of time for doing that. And of course, we have notifications that we send
out. So for instance, we trigger a shell script, a webhook, syslog, you know, telegram, you
know, usual thing. Nothing new here. Okay, let's now start the talk after this introduction
of NTOC and GIM. The problem is the following. So we have added over 150 checks, behavioral
checks on the traffic. But there is always somebody that comes and says, I want to do
something different. How can we support these people? How can we enable new programmers or
let's say people that used to use Python, shell script, you know, this type of programming
language or that don't want to learn the internals of our application? How can we do that? And
many times this happens when you are in a harsh. So that is an attack. That is something
happening on your network that you want to check. And we have, you know, two levels of
the problem. First of all, we have to extend the behavioral checks in order to have some
behavioral detection in a different way. And in the second part that Alfredo will describe
later. So how can we use NTOC and G as a data lake from languages such as Python, for instance,
that is very popular. So that you can use NTOC and G as a source of data for your own
application. Of course, you have time series. As I said, we save data in influxDB if you
want. So therefore, you can use Grafana for creating your own dashboard. But these are
simple dashboards. So if we want to do something more complicated, if we want to go beyond
that, in addition to that, how can we do that? So this is the idea today. So we like C++.
C++ is super efficient. We like it. Okay. We are used to play with it since many years.
But we understand that it's not what everybody wants. Okay. We need something easier. And
we would like to understand also how it was possible to develop checks in minutes for
people who are saying, okay, if I see this specific certificate or if I see this specific
behavior, then there is a problem. Something very peculiar to an organization. So not general
for everybody, but for specific people. So for instance, how do I trigger an alert if
there is traffic, TLS traffic within host A and B? So for instance, a printer should
not make any TLS traffic just to make an example. So if this happens, and how can you trigger
an alert? Another problem is the following. If I have a certificate signed by a certain
organization, or for instance, if I have a BitTorrent connection that is going above
one gigabit, or notifying me if there is a Zoom call with bad quality, things like this.
Things like very, very peculiar, very specific checks that people want to do. Maybe on an
operating, sorry, on an autonomous system, and not on another, or on an actor, and not
on another. So things that are not general that we can implement for everybody. How can
we do that? So let me talk how it works in top NG internally. Let's have a look at the
flow, also communication. So in top NG creates a data structure inside itself as soon as
we see the first packet of the flow. So we see episodes of the destination, source parts
of destination, protocol, VLAN, whatever. And then this is the first event that is relevant
for us. And then, as I said, everything sits on top of NDPI, so the yellow part. So we
have another event when the application protocol is detected. Actually, this one is divided
in two parts. First of all, as soon as the main protocol is detected, such as TLS, okay,
and then we can refine this information with metadata saying, okay, this is TLS that is
going to Google mail, and not Google search or Google something else, okay. So second
event and NDPI. And then we have, for long-standing flows, some periodic activities. So in essence,
every minute, we do something different, something like, you know, I want to trigger,
you know, an action. And then at the end, the flow end notification, so as soon as the
flow is over. So what we wanted to do, we wanted to create a low API that allows people
to create the simple checks that are efficient. Efficient enough for most people, because
not everybody needs one and a gigabit, but many people have one gigabit networks or,
you know, two, five gigabit networks. So they need some sort of efficiency, but they are
not super extreme. So let's say use Lua for prototype on a check for some people who need
speed, or use Lua for people who have, let's say, an industrial network or a network that
is, you know, running at one gigabit or two. So in essence, we have created an API that
allows from Lua to see internally, in N-top and G, properties of the flow. For instance,
the number of bytes, multicast, layers, seven information, these type of things. And the
API calls are very small. So in essence, we don't want, you know, the application to
be inefficient simply because we download to Lua the representation of the host, the representation
of the flow. Well, simply the method that we are interested in. So in the left side,
you will see the C++ code, how it implements the stuff. On the right side, you will see
an example of the Lua code. So in this case, just to give you an idea of how it works.
So whenever there is one of the events, so for instance, we have to check the flow because,
you know, NDPI is over, so the protocol has been detected. So if you want to block, let's
say, Google Mail, okay? So what you need to do is to execute a Lua check after this happened.
So in essence, the C++ code, we have put the code to the Lua VM that executes a script,
okay? A script that can be, you know, applied to many flows, not just for one. So this is
where, you know, this happens. And this is an example of a check. So we have a simple
example. If you have a flow that is either TLS or quick from, started from host or anything
in 192.68, 178.2, 1.1. And if it's TLS, and if the protocol issue is, so a very simple
check that, for instance, a friend of mine has asked because it's monitoring IoT networks
and they have found a vulnerability on a specific type of rule and the client was a specific
device. So something that is not general. Okay. So this is the way it works. Very simple
to write. The problem is the following. That the overhead introduced, this is a very slow
Intel I3. So just to give you an idea of the super worst case, is 30 microseconds for everything,
okay, in average. Whereas with C++, we can do it in one microsecond. Now, you say, this
is bad. In a way, it is bad. I agree because we are 30 times lower. But you have to think,
first of all, on one gigabit networks, that this is not the problem. Also, you have to
think that most of these checks are asynchronous. This is one of the few ones that are synchronous.
So in essence, as soon as the protocol has been detected, we call this method. But it
is not why the packets are coming. So in essence, we have another threat that is calling this
while the traffic is coming. But we don't stop the execution tree. So in essence, just
to make it short. So if you take this overhead that you have introduced and you sum to everything
and you stay below certain boundaries, so if you want for every minute to execute the
flow checks on all the flows, you are good, okay. And of course, we trigger an alert.
And the result of the alert is a notification on the GUI that can be sent, for instance,
through Microsoft Teams, just to give you an idea. Or we can trigger a shell script
for something or can send an alert to my friend on Telegram. So this is the way it works.
Okay, now I have this. Okay, so we have seen how to extend the N-top-ng engine with Lua
scripts to access traffic information and use those information to check the traffic
and trigger alerts, for instance. Now, recently released also a Python package that you can
install with pip install N-top-ng that allows you to, you can use it as a library to create
a Python script which is able to access traffic information in N-top-ng. And this happens
through the REST API. This means that you can run your script even on a remote location.
For example, you can access live data in N-top-ng. In this case, we are importing the N-top-ng
class. We are connecting to N-top-ng using the N-top-ng class. We get an instance of
the, of an interface in N-top-ng, for instance, eth0. We use this method to get all the hosts
which are active in my network with all the metadata. And there are plenty of methods
in this class or another class in this library that allows you to get traffic information.
So you can get alerts, flows, hosts, whatever. And you can also get historical data. So the
same way, so you connect to N-top-ng, you get an interface. From this interface, you
get the, an instance of the historical class. And you can run queries in the database. For
instance, you can get alerts statistics from this time to this time, for instance, for
the last 24 hours. And just print the, of the alerts that you have.
Now, those are two examples of querying the, the engine to get the data. Of course, we
have seen that N-top-ng is able to, when a check or an external event detects something,
an event, we can trigger an alert. And we have seen that N-top-ng supports several endpoints.
So we can send this alert using mail, a messaging system, like telegrams, LAC. We can run a
shell script. We can call a web book. So we can run a shell script. For instance, in
this script, it can be a Python script. So let's try to put all the pieces together.
So we receive an event from, which is generated by an internal check or an external check.
This event can call a Python script. This Python script can get information from the
alert itself or can query the engine through this API that we created to get more data,
to fetch more data and argument the alert information. And this can have some logic
and trigger some action. So you can write your actions here to react to this event.
In order to implement this, what you have to do in N-top-ng is, first of all, you have
to enable the check that you want to use to analyze the traffic. For instance, in this
case, we are using a custom check that the user creates in Lua as Luca showed you before.
Then if you want to write your Python script that reacts to this event, you have to write
an alert tender, which is a script that you place under N-top-ng script shell. And this
case is a simple script, which is just getting, as in the standard input, the traffic information,
the metadata. And, for instance, if the alert type is our user script, I want to do something.
In this case, I'm just logging the IP address related to the host that triggered the alert
and a message from our custom script. Then you have to go inside N-top-ng. You go under
notifications. You set that you want to send alerts to the shell script. Here you have
all the options, like email, whatever. And you select your handler. And then you specify
for your handler that you want to receive just critical alerts. So you specify the severity.
You specify the category that you want to, of alerts that you want to handle, from this
case, cybersecurity, and the entity. In this case, I want to handle alerts about hosts.
And then we can extend our handler. We have seen how to print just the alert information,
but we can, again, we can use our Python library and N-top-ng to access more information about
the host. So we receive this alert, which has been triggered on a specific host in our
network. For instance, this host has been infected by malware. It's generating unexpected
traffic, whatever. We want to get more information about this host to build a report, for instance.
In fact, in our library, we also have the ability to generate a report, or you can generate
your own report using the API that we have. So we build this report and send an email.
So this is a simple script that you can use. It's a few lines of code to handle alerts
and generate reports and get, for instance, an email or your mobile phone with the alert.
So this is the big picture of the example that we're seeing right now. So we have defined
a user script that triggers an alert, or we receive, again, events from any other source
or internal checks. We are calling our script, which is getting more information from the
engine to build a report and send this report by email. So the result is this. So the system
is checking your traffic, is building a report when something happens, and we'll send you
an email with the report of the traffic for the host with the top alerts sorted by severity
or by count, the top contacts for the host, the chart of the traffic generated by the
host, where you can add more, like the top applications used by the host, et cetera.
Do you want to wrap up? Okay. So we have seen that within top ng, you can collect traffic
information from traffic, flows, events, events from Suricada, for instance, et cetera. And
we started with the top, actually Luca started with the top, then we moved to the top ng.
It was mainly a traffic monitoring tool. Today is also a cybersecurity tool able to do behavioral
checks, not just for providing visibility, but also providing cybersecurity monitoring.
You are now able to extend this engine, both with new scripts integrated in top ng or even
with C++ plugins, let's say checks, if you need to scale with performance, or you can
use our Python library to write Python tools that can run externally, even remote boxes,
to access traffic information in the top ng engine, and be, for instance, a PDF as we
have seen with reporting what's going on in your network. Of course, all the code is available
on GitHub, so if you want to contribute, you are welcome. Especially now, you don't have
excuses. We have a lot of libraries, scripting languages for interacting with the engines,
or something else to add. No. The only thing I want to say is that this
is an efficient way from our point of view to do network monitoring and cybersecurity,
and at the same time to extract information in a way that does not interfere with the
main engine, so that allows, I believe, most of the people sitting in this room to do whatever
they like to create a monitoring tool that is tailored for their own needs, and that's
the first set that is open source. That's all. Thank you very much.
Any questions? Any questions?
Wait, wait, wait. It's just a simple question. How does it compare with CN tools? It looks
like it does everything CN could do. CN tools. Yeah.
I don't know the tools. No problem. I am not familiar with them.
Any other questions? The scripts can be compiled to be more
performance, or do you not have this task in your developer timeline? To compile script,
to have more performance. Loa script, or like CCC, we saw that CCC script
takes one millisecond, but the Loa script takes 30 milliseconds. Yes, of course, you
can compile them, but you have to code it in C++ at the moment. So we used Loa just
in time to compile the one seen before by a stamp switch, but it is not available everywhere
for its own arm, and we want to support it as various. So yes, it is possible, but again,
these are microseconds, not milliseconds. So one million of them per second.
Any questions? Anybody else?
Hi, thank you. Do you have some figures about performance you are able to achieve on typical
server, about flow per second? Some figures to share on Loa scripting, and also some example
on Python, which should be less efficient? Okay. We are, when you process packets with
Ntop.ng itself, you are able to process like a few gigabits per second, depending on the
drivers you use, how you tune Ntop.ng, let's say, to scale with the performance, you can
get 10 gigabits, for instance, but more or less, we range from a few gigabits to 10 gigabits
in Ntop.ng itself. You can use it in combination with our probes, which is Nprobe, or we have
other probes like Cento. In that case, you can scale with the performance up to 100 gigabits
per second, but the architecture changes a bit. It's one 100 gigabit in plus. As of
the checks, it depends on the checks that you enable, of course. Okay. I think we are
running out of time. Many thanks for being here now.
Thank you.