Hi. Welcome to my talk. So you want to build a deterministic networking system, a gentle
introduction to time-sensitive networking just out of interest. How many of you have
heard of TSN or time-sensitive networking so far? That's quite a few for a networking
session. That's great. How many of you have already worked with that? Not so many. Okay.
You will after that talk. Yeah. Who am I? I think I'm a former system engineer. I worked
a lot with time-sensitive networking and its predecessors. I also took part in standardization.
So I also did some of that. And since last summer, I worked at a kernel developer at
Pengatronics. That's a German Linux consulting and support company. We have roughly 7,600
patches in the kernel. And we also do consulting for real-time networking amongst many other
stuff. And by the way, we're hiring, of course. Now, to what we will look into today, we will
look into applications. I will give you some examples why you would probably want to do
networking over or real-time data transport over networking and what the implications
of that is, what the requirements of these applications are. We will look into the basic
building blocks. So sorry for the folks who already know about that. And we will talk
a bit about which Linux user space and kernel components are used in building these applications.
And I will sum up the state of the union a bit. And then, just as an announcement in
advance, there are some bonus slides where I will give some more details and some references
to open-source projects already working with TSM. So if you're interested in that, just
download the slides from the penta and, well, check out the links. And I also gave an example
of how to basically glue together a stage box, so a transport system for audio data
over the network. I won't make that into the talk because it has been shortened to half
an hour. So the example I will focus on today is audio video bridging. So if you want to
transport real-time data over a network for an application just as this talk, you want
to have as low jitter buff or as small jitter buff as possible to reduce latency in the
system because if you transport data over a traditional network, packets could get dropped.
So you have to resend them or you have to make sure that somehow, magically, interfering
traffic doesn't do you any harm. And that usually involves quite large jitter buffers
up to several seconds. And if I talk now and you hear me from stage and you hear me from
the PA four seconds after that, that would be quite annoying. So you want to cut that
down to as low as possible transmission latency, overall end-to-end latency. Of course, for
TSN, which started as audio video bridging or AVB as a standard, they came across the
fact that this technology could also be useful for quite some other applications. Most of
the customers do like machine control stuff with that. So if you have a large production
line and you want to transmit data between your PLC and your server drives or your robot
arms and stuff, you also want to make sure that your control data arrives in time at
the actor or your sensor data is read in within a certain point in time. And that's quite
important to keep that timing. Same holds, of course, for aerospace and automotive and
railways and stuff. I won't go into these applications today because we're, as I said,
short on time. The first requirement of said applications is that you need to establish
a common time base in the network. That's due to the fact that while measuring time
in computers, it's basically hooking up a hardware counter to a crystal oscillator.
These crystal oscillators tend to have frequency drift over time, especially with temperature.
And due to the different switch on points in time, you also have quite large offsets.
So if you start one device, say at 12 o'clock and the other at 1 p.m., they have one hour
of offset in there. So you want to make sure that all your network devices have a common
meaning or a common sense of time passing and a common sense of what time it is. Because
lots of scheduling decisions for networking traffic may depend on timing. Also, for some
applications as the audio example, you also would like to regenerate your audio sampling
clocks. So basically in order not to introduce any additional degradation in audio quality,
you want to make sure that your sampling clocks of your ADC and DAC run basically in lockstep.
And that is why you want to make sure that your time is distributed evenly. And the way
that this is done usually in networks is just shown basically in this old style picture.
You elect a so-called master clock. So basically that's the best clock reference in your network
or the most stable clock reference in your network. And then basically you compare all
other clocks to that clock reference and they have to adjust their local time for that reference
time. It's basically just as those three gentlemen do in that picture. I like that comparison
because you find a lot of analogies and the standards to just the way that works with
like pocket watches. And if you look into that, you will find that basic idea quite
useful to keep in mind. Now the other thing we want to have guaranteed is as I already
said bound and transmission latency. So if we go across the transmission of a data stream
in the network, so that's what the standard calls a talker at the left. And that's what
the standard calls bridges. Usually as we're dealing with layer two, that's ethnic switches.
And in the right, that's what the standard calls a listener. You also call it a source
and a sync. But the standard talks about talkers and listeners. And the packet goes from bridge
to bridge to along its pass across the network. And each switch basically a bridge has an
ingress queue and a switch fabric and an egress queue. That's due to the fact that you can
only transmit one packet out of a certain network port at a time. You can't just if
another packet at another port arrives for that destination port, you have to store it.
And you have to wait until the last transmission is done. And then you can transmit the next
packet. And this introduces what's called the residence time in each switch. So even
if you have a perfect pass through through network without any additional interfering
traffic, you add a little time at each step, your payload packet travels through the network.
So if our audio starts here, it's a bit later when it arrives here, and a bit later when
it arrives there, and so on so forth. So that's fine, as long as you have no interfering traffic
because if you have additional interfering traffic, and that might be because we of course
want to use our audio on converged networks. So we want to use the same network for say
our live PA system and for our network internet connection. And we want to download large file
because we want to download a presentation recording from FOSTA. And basically that's
where this entity arrives and it's introduced or it creates a large amount of traffic here.
This will cause the packet here to be delayed until it's sent out of the egress port. And
basically it won't arrive in time. And if we go for a small jitter buffers as possible,
that's a problem because we have a buffer underrun at the listener side. And basically
we have audio dropouts in the audio case, or we have stalling motors in the industrial
control case. That's something we have to avoid under any circumstances. So basically
something we want to have is quality of service. And so the picture, of course, your professional
networking engineer, so you don't need that picture, but the picture I like to use for
that is a bus lane in the street because also the bus runs in a more or less isochronous
way. So you send those bus or packets down the lane and the way not to be hindered by
the interfering traffic there is just basically to introduce a priority lane. And that is
what we also use in networks basically when we introduce quality of service measures.
Another thing we need for at least some of these applications is link layer redundancy.
So imagine if there's a mixing desk right in the back and we run a network link back
there and someone just trips over that link, rips out the cable, or maybe it's a fiber
link and someone stomps on the fiber link, bad things happen. And basically if our stem
is over, we don't want to have that. So we want to introduce means of having redundancy
schemes there. Basically you can't think of it as a real-time capable, real-time healing
with no waiting time like spanning tree-ish thing you want to have. The standard spanning
trees quite don't cut it for these kinds of applications. So we have to introduce other
stuff there. We have some other application requirements there. They're not so important
so I leave them out for now. Now what does the or what kernel and user space components
do we have to implement that? We will look into what the TSM components are later or
what the TSM standards are because that's basically just numbers and letters. So for
time synchronization, especially TSM, we use GPTP. That's a flavor of the precision
time protocol, generalised precision time protocol, of which you can think of PTP standard
PTP, IEEE 1588 boils down to layer 2. So of course we're dealing with raw, ethnic frames
so we can't use UDP for transport and it also has some other quirks but they're not
too important right there. And the way we do that with Linux kernel, we have the hardware
time sampling units and the PTP hardware clocks. That's basically the interface to
hardware clocks in your FNMAC or FI. And the user space component to run all the remaining
stuff is PTP for Linux. That's basically the way it works and it works quite well. You
can achieve down to several nanoseconds precision from point to point with that. For traffic
shaping, that's the quality of service measure we want to employ. The kernel has the TC
subsystem and usually if you configure that manually you use IPv2 or netlink if you want
to do that programmatically and that's basically the way it works and we will look into a bit
of detail later. For network management, so basically if you have to reserve a data flow
from a talker to a listener, that's where it gets a bit sketchy because that's of course
user space demons and there aren't much. There's also a problem because there's several ways
of doing that, the traditional way or ABB style, the initial implementation used the
so-called stream reservation protocol. Modern ways for especially pre-calculated or pre-engineered
networks is using young NETCON extensions and there are some demons for that but support
for the TSN extensions is not too great. So if you're into that, that's quite a nice
thing to work on. For the real-time data packetization, that's mostly user space. Of course you want
to use some kernel features like ETF, Qdisk and XDP to have as low overhead as possible
and to make sure that your transmission is sent out as asynchronously as possible and
you want to use offloading for that and then there's some very application-specific user
space components. So for audio-video stuff, you can use the G-streamer plugins and for
industrial control, I'd recommend to use a 2G Open 6651 implementation. That's not quite
finished yet but it's a good starting point at least. And for the link layer redundancy,
that's what PCR and FRER is, basically the standards are finished since one or two years.
There's not much hardware supporting that yet and you really want to have hardware offloading
for that. So you're basically down to proprietary vendor stacks at the moment. There are efforts
to put stuff mainline but there are not quite there yet. But stuff is coming and that's
the good thing with that. So I think one slide is missing there, which is not a too big problem.
Yes, one slide is missing. So basically the stuff, how to put stuff together with TSN,
I will summarize it without a slide. With TSN we have GPTP, that's IEEE 802.1AS for
the IEEE standard fetishists here in the room. And traffic shaping, the basic standard stuff
is the credit-based shaper but there are more time-aware shapers available right now. They
are basically making more efficient use of your network and the way that works is basically
a reserving bandwidth along your data flow path in your network. Network management,
again, that's a bit, that's a bit application-specific. So the audio video and professional audio
video stuff is still using the stream reservation protocols and for the payload, as I already
told, that's really, really application-specific. And for redundancy we use PCR and FRER. Usually
there are some exceptions to that, especially for professional audio video. PCR and FRER
were unstandardized when those standards were written so there are some proprietary
or not proprietary but some other redundancy schemes where you basically send two different
streams and try to separate your networks via means of VLANs usually and try to force
different data paths through network. Basically nowadays you want to go PCR and FRER whenever
your hardware supports that. So state of the union, the hard stuff is already done. So
there's already implementations in the kernel, there are user space demons available. That's
again the stuff that's difficult to get right. So if you want to implement those standards,
first of all you have to read tons of paper. I did that for an employer, took me two years.
So that's really hard to get right. And the good thing is that that is already implemented,
you just have to use it and you have to use the right knobs. For some stuff like GPTP and
traffic shaping you want to really, really use, for GPTP you have to use, for traffic
shaping you want to use, hardware offloading. You have to bear in mind that your network
gear has to support explicitly GPTP and traffic shaping. So about the preservation and basically
making sure that your traffic shaping is applied properly. That's not true for every hardware,
especially not for commodity hardware. And bear in mind that sometimes configuration
especially for traffic shaping can be quite tricky. As I said, I have added bonus slides
to the presentation. I will check that they have the right slides in there later on or
just contact me. And the point is especially credit based shapers can be really, really
tricky to set up properly and to make sure that you reserve the bandwidth you want because
you want to have the remaining bandwidth to be available for best effort traffic. So
the idea is that you can use like say 70% of your link for your audio video stuff and
still have like 30% of your gigabit link, which is what we're usually dealing with for
like audio video available for just best effort network management traffic and what so ever.
So you really want to make sure your shapers are configured the right way TM. And it's
quite hard to treat the right knobs and IP route too. So there are good examples and
I'd strongly recommend to read the docs on that. There's also a link to the TSN read
the docs for Linux. It's quite a good starting point for getting into that whole topic. And
yeah, basically I think that's it. Do you have any questions? Any questions here?
Thanks for this. What's the highest speed Ethernet implementation of this you've seen?
Have you seen anything beyond like 10 gig E for example? I have seen a 10 gig implementation
for that. As far as I recall the standards and have some limitations with respect to
how you communicate your bandwidth requirements and they're a bit capped. I'm sure and I know
that they are working on that for future revisions of the standards because of course now faster
links are becoming available more and more. Most applications for TSN like the control
stuff or the AV stuff are running on 100 megabit links still. You want to go to gigabit links
because you can achieve quite a bit lower end to end latencies on faster links. But
I haven't seen, personally haven't seen faster stuff than 10 gigs so far. But I'd be interested
to do so. Do you have happy stories or really users
that have put this in production and can you tell more about this? Yeah, so if you want
to check that out you can just Google for Milan and TSN which is the professional audio
video stuff and they just before Covid started, shortly before Covid started they ran the
Rammstein concert in Munich over a TSN system. It's a really large system with several video
walls and several like hundreds or thousands of audio streams and pyrotechnics and light
control and stuff all in the same network converged. So that's the largest installation
for live audio I know of and I think that's quite a good story to tell. I was curious
if you had the chance to play around with synchronous ethernet as well. I haven't looked
into that too deep yet so I can't tell you too much about that.
You mentioned XTP. Are you aware of any applications of XTP in that area? To be honest I haven't
seen them and I will start working on some of them for a customer project in just a few
weeks probably. The idea is that basically because it's layer 2 you don't have much
network stack above the hardware layer. So if you can cut some of the Linux networking
stack because you don't use it anyway, you work on raw sockets anyway, you could just
cut some of that out and try to achieve lower latencies in your basically Linux stack there.
Probably on the next Fostum I can probably give you a talk on that.
This is probably a big question but how do you go about debugging this sort of stuff
so like setting it up or if you think there's a problem, how do you go about finding problems?
That's actually a bit of a pain point and you have to know at least a bit what same
values for like path delays for the PTP and stuff are and one of the most useful debugging
tools I've found so far is a good ethernet switch because it will give you like output
for your stream reservations, it will give you output for your PTP or GPTP. You can also
like sniff traffic with wiretaps basically and analyze it in Wireshark or Skypie or whatever
your tool of choice is. That works best to be honest for 100 megabit links because you
can use passive tabs. It doesn't work that great for gigabit links because it violates
some of the sound it's a bit. You can also use like mirror ports and switches to exfiltrate
traffic but basically it's a more manual approach of debugging and I'd like to get in touch
with if anyone is interested in just write me an email to start a community-based project
of automated analysis of TSN networks basically because I think it's something we really really
need especially for people who aren't that deep into the standards and we need to make
sure that we can basically have a one-click check and setup and can tell from a tool that
at least if that looks okay-ish or not what you're doing but I'm not aware of any project
so far so I'd like to start but I'm not too experienced in how to start such a project
so if you're experienced in that or are interested in that just write me an email, get in touch
and maybe we can set up something. Any more questions? That's all the last one.
You mentioned some protocols for link redundancy. Can they also be used for node redundancy?
I'm not entirely sure. I would have to look something up. I think basically it should
work because it's about the data path so if one node drops out basically that would work
as well but it won't work for the endpoints so for the talk of the listener of course
it won't work but for nodes in the middle of your graph that would probably work.
Okay thank you very much again for your presentation.
Thank you.