Yeah, so let's go right into the topic.
So I'm Alex.
I work for Switch.
Switch is the national research and education network in Switzerland.
Like most countries have something like us, like in Belgium, it's BellNet, and in Germany
it's DFN, and in France it's Renataire.
So we connect to Swiss universities and universities of applied sciences via the ISP of those institutions.
So NetFlow, I'm not sure if everyone is familiar with NetFlow, so I just recaptured the like
a central thing about what the network flow actually is.
So when you look at an IP packet, you extract the source and destination addresses, the
IP protocol, and if the protocol is UDP or TCP, also the source and destination ports,
and those five numbers identify a flow.
So every packet with the same values is said to belong to the same flow.
And then in the simplest possible way, you basically just aggregate, you count the bytes
and packets of all the packets that belong to the flow, and then you export this information
to a collector where you can then analyze the data.
So this is an old thing, like these days people talk about network telemetry, and back in
the day when this was developed, that name didn't exist yet, and I'm not sure when exactly
Cisco came up with this, but it must have been the early 90s or mid 90s, and it used
to be a de facto standard for a long time, but people just figured out what Cisco did
and then did the same thing, and then finally got properly standardized with the IPfix IETF
standard.
And you can do this either in sampled mode or unsampled mode, so unsampled means you look
at every single packet and account for it in the flow, and with sampling you just look
at every nth packet, and then you have to make certain assumptions to then reconstruct
the actual values.
So we at Switch, we've been using NetFlow for a very long time as the basic, as the most
important metric or means to analyze our data, our network data, since the mid 1990s.
It used to be that this was provided by the Rogers themselves, which is reasonable, and
the packets passed through that device, and so the device has immediately access to the
packets and then can construct the flow data itself.
So initially that was done in software, then it was done in hardware.
It used to be basically always unsampled, but with the advent of more powerful networking
gear, and especially with the arrival of the 100 gig ports, it became basically unfeasible
to do this on the Rogers themselves because of typically software restrictions, also hardware
restrictions.
If you want to do this in software, you usually can because the Rogers are not very powerful
in terms of CPU, and in hardware it becomes very expensive.
So the vendors started to basically only implement sampled NetFlow, so these days if you buy a
Cisco or a Juniper box and you do NetFlow, you get sampling.
And sampling is fine, of course, if you're only interested in aggregate data anyway,
so big aggregated network flows between networks, for instance, sampling is perfectly fine,
you make certain assumptions about the traffic, and then you just upscale it, and you get
fairly reasonable numbers.
So why would you even want to do unsampled NetFlow?
Well, there are some couple of use cases that are really useful.
So for instance, in terms of security, one thing that sampling is fine is detect DDoS,
for instance, that's volumetric DDoS, that's very simple, so you basically have a constant
packet rate, and if you just look at every end packet, it's easy to scale this up.
But if you want to detect a bot, for instance, in your network, then it's more difficult.
So maybe you want to do this by looking at the communication with the command and control
channels, those are short lift flows, and if you do sampling, you're probably going
to miss them.
But with unsampled NetFlow, you see every single flow, so you can identify these things.
And we as a network operator, we use this fairly often to troubleshoot network problems,
so if a customer says complaints, you can't reach a certain IP address in the internet,
we can actually go look in our flows for the outgoing TCP SYN packet and see whether there's
a TCP SYN coming back in.
You can do this because we see every single flow, so this is extremely useful.
But as I said, so we cannot longer do that on our big new core routers, we can't do that,
they only give us sampled NetFlow, so we started to do this with an external box, and that's
where this SnapFlow software implementation comes in.
Because I mean, there are always ways to do that, but they might be very expensive if
you have to buy dedicated hardware, for instance.
So just to give an idea of what type of traffic we're dealing with, Switzerland is a small
country, we are a small network, and we only do NetFlow on our borders, so when the traffic
that we exchange with neighboring networks, and the peak values are these days, it's roughly
maybe 180 gigabits per second, something like that, and 20 million packets per second,
and roughly 350,000 flows per second, unsampled.
And this can actually be even much more.
The flow rate, because of the aggressive scanning that's going on for the past couple of years,
has started to perform very aggressive network scans, like plain TCP SIN scans, as fast as
they can, so sometimes a single host can easily generate 100,000 flows per second.
So the actual IPv6 traffic that the export is done in the order of 200 to 300 megabits
per second, so the flow records themselves, so this is all for the unsampled flow.
The average flow rate is maybe just around 200,000 per second, and the data it generates,
the actual NetFlow data is like roughly 1.5 terabits per day, so the actual scaling problem
is more on the collector side than.
We have 10 gig, 100 gig, and 400 gig ports, so that's what our solution needs to support.
So we used to do this historically on the routers themselves until a couple of years
ago, then we moved to a commercial NetFlow generator that did that in hardware, which
was pretty expensive, maybe the whole solution for one pop was 100,000 euros, something like
that, and then we finally moved to SnapFlow and Pure Software.
So how do we do this?
On the borders, these are all fiber connections, so we have optical splitters, we create a
copy of all the traffic flow, and then we have a second device, or the primary device
that these tabs are connected to is what we call a packet broker, it's basically a switch
that aggregates all the packets and sends it out on 200 gig ports to our actual exporter
box.
So it uses VLAN tags to identify, so in NetFlow we also want to keep track of the router ports
where the traffic was sent or received from, so because then that information gets lost
and you aggregate them, so we use VLANs to tag them.
The box we use are white box switches based on the Tofino ASIC, the ones that Intel just
decided they stopped developing, unfortunately, these are very nice boxes, like there's one
with 3,200 gig ports for 5,000 euros, and the other one has 3,200, 400 gig ports and
costs about 20,000 euros.
The thing is you have to program them yourself and you buy them, they're just plain hardware,
and so you can use the P4 language to do this.
I link here to another project of mine where I actually developed the P4 program to do that,
so that's also part of this entire architecture.
And then the traffic gets to the NetFlow exporter box, which is currently just one rack unit,
that's basic rack mount server, we use AMD Epics, mainly these days with a fairly large
number of cores, that's the way we scale, with the number of cores, NetFlow always scales
very well the cores because you just have to make sure that you keep the packets to
a flow together.
They use, the exporter has a Melonox 2.4, 100 gig card, that's connected to the packet
broker, that's where it receives the packets.
So in a picture that's what it looks like, on the upper left that would be our border
router, on the upper right that would be the bordering router of our neighboring networks,
in the middle you have this optical spitter, which is completely on passive box, just as
an optical splitter, and then you have this packet broker switch in between that aggregates
all the packets and distributes them by flow on these two links currently.
So these are now 200 gig ports between the broker and the exporter, we can easily add
more ports if that's not sufficient, and on the SnapFlow exporter we can basically just
add more cores to be able to scale.
So now let's hear Max talk about the actual software.
Hello, hello, does this work, good, all right.
All right, how do we know how SnapFlow is deployed, I want to talk about how it's built, how
it scales, how you configure it, how you monitor your running application, etc.
So SnapFlow as the name suggests is built using Snap, Snap is a toolkit for writing
high performance networking applications, Snap is written in Lua, using the amazing
Lua JIT compiler, and it does packet IO without going through the kernel, like generally the
Linux kernel packet networking stack is slow from an ISP perspective, so a Snap bypass
is that, uses its own device drivers, and this is also often called kernel bypass networking,
I think nowadays it's fairly common, and Snap is open source and independent, we're not
sponsored by any vendor in particular.
So Snap is built with these three core values in mind, we prefer simple designs over complex
designs, we prefer our software to be small rather than large, and we are open, you can
read the source, you can understand it, you can modify it, you can rewrite it, etc.
So here I have a snippet of code taken directly from SnapFlow, unedited, so this is how the
Lua code that powers the usual Snap application sort of looks like, just to give you an idea.
In this particular example we read a batch of packets from an incoming link, we extract
some metadata that tells us which flow this packet belongs to, then we look up a matching
flow in the flow table that we maintain, if we already have a flow we count that packet
towards that flow, if not we create a new entry in the flow table.
Got one more snippet, this function is called every now and then to actually export the
flows, so we walk over a section of the flow table here, and add flow aggregates from that
flow table into a next data export record, and if it's time to export the data record
we send it off to an IPfix collector, which is a separate program.
So from a bird's eye view, SnapFlow works sort of like this, we read packets from a 100
gigabits nick, the garden hole so to speak, we process those packets to extract flow information
in a snap process, and then we send off data records over a ton-tap interface to the IPfix
collector.
So on the right side here you have a device driver written, like that is part of Snap
written in Lua, that actually happens like the actual traffic, the bulk of it, and on
the left side you have an interface to the Linux network stack, so since the flow export
data is rather small in comparison, you can just do that over the regular Linux network
stack, and that works.
On the very left side you have the IPfix collector, that's a different application,
like a separate program that we send the flow data to in the end.
So sadly, or I mean I guess just obviously, single CPU core is not enough to handle 100
gigabits of traffic, so instead what we do is we do receive side scaling provided by
the network device, this way we can process n different sets of flows on n different processes
running on n different CPU cores, so every circle here is a CPU core.
And we also support to repeat basically the same trick in software, so we can do another
round of received side scaling after filtering the traffic by protocol, and this way we can
process for example DNS traffic on different set of cores than IP traffic, like non-DNS
IP traffic, and that way we can sort of like segregate the server resources into the workloads
that we actually care about.
We might for example care more about that we have an accurate general IP flow profile
to send to the collectors, and maybe if we still have some time left we will also do
some DNS analysis, but we don't want one to slow down the other necessarily.
So SNAP programs are organized into independent apps, so an app is like a logical packet processing
component, could be for example a device driver or an app that implements the address resolution
protocol, and these apps are combined into implications like SNAP flow using links.
Links are unidirectional, they really just ring buffers, and any app can have like any
number of them to use as input or output for packet data.
And you communicate with like you use those links like shown here, that's basically the
API that you call link receive on a link to receive a packet, and you call link transmit
on an output link to send a packet.
So now to forward packets from one CPU core to another CPU core we have this thing called
live interlink, these are really just like regular links except that they span process
and CPU core boundaries, and you can also use them just like any link, you have the
same interface if you want to operate with them, and we use those to implement the software
based receive set scaling that I talked about earlier, right?
We also have libp3, so libp3 implements a very strict control plane data plane segregation,
I think for most networking folks the concept of control plane data plane is pretty common,
but just to recap it, control plane is something that basically is fancy and elaborate, you
expect it to be really nice, you want to have a nice interface to configure your application
and monitor it, the data plane on the other hand you really just want it to work, it should
like preferably run at line rate, and you don't really have any time to mess around.
So since these like two different parts of the application have very different requirements
nice to keep them separate, and that's what we do.
We also have libyang, so you see both the configuration and the application state of
snap flow are actually managed by described in the yang schema.
So for example you can tell the control plane to load a new configuration of snap flow or
you can query it for some state counters while it's running, and on this slide I have some
examples how you will use the snap command line interface to do those things.
So here we have a snippet of the snap flow yang schema, and yang is one of these things
where at the beginning you wonder if you're really going to need it, but once that you
have it you are usually really happy that you do have it.
So what I like specifically about yang is it's very expressive.
If a configuration passes the control plane and it doesn't reject it because it says hey
this is invalid, I'm pretty confident that this configuration will do something useful
in the data plane and it will not just like crash.
For example here we have a list of interfaces and one of the fields is a device which is
a PCI address and the PCI address in this case this type is attached to some regular
expression that makes sure that it actually looks like a PCI address and we kind of just
pass any string in there and validate it somewhere way down the line.
Like if you don't put a thing that at least looks like a PCI address then this won't even
be loaded.
So sadly any piece of software has bugs and in our case even suboptimal performance often
considered a bug right and we deal with the second issue here with the performance by
shipping snap with a flight recorder.
So this flight recorder has minimal overhead, it's always on you even run in production
preferably and it stores useful data, part of that data is really useful to profile your
application after the fact or while it's running.
To analyze the collected data we have built a little UI that we used to do that, it's
usually running on one of our development servers so we test stuff but you can really
run it anywhere.
Did I mention snap?
I did right?
So we're dealing with a JIT compiler here.
So the UI shows you stuff that you would expect from a profiler like basically where does
my program spend its time but also some JIT related stuff like did the compiler have issues
generating efficient code for particular parts of my program.
So for example here there's like a JGC column that's like when the injected code the garbage
collector is invoked and that's for example something to look out for.
And another part of the flight recorder is a high resolution event log.
It can give you accurate latency measurements of the pieces that make up your software.
And you can see here on the slide that the OUI has or it shows latency histograms for
individual events.
These events are, some of these events are like already defined in snap but you can also
use a defined new event.
And here for example I could tell that processing a batch of packets and extracting the flow
data so this is like the main IP fix app main loop takes us about 35 microseconds per iteration
per process.
And this is really useful if you want to debug tail-latencies, right?
And tail-latencies translate basically to drop packets in our world so that's something
that's really valuable.
So to close things, if you were to write a new application based on snap today you would
have all these things and more ready at your disposal.
And also it is possible to purchase consultancy services like commercial support for snap
and developing snap applications from your friendly open source consultancy Igalia, which
is my current employer.
So yeah, that's all for now, thanks for your attention.
On the right there are some pointers if you have some contacts, if you have questions
or inquiries about snap or snap flow you can email us there after the conference or for
now.
If you have any questions, please ask them.
Thank you.
Please come down.
There are some seats available here in the middle.
The next speaker is Peter Manev, that is one of the key guys of Suricata, a very popular
open source ideas.
And today is going to talk about this open source platform.
Please have a seat here in the middle.