Last but not least, it's Christoph Masio, but first of all, I would like to thank him
for his work organizing the Dev Room all day.
Thank you.
Yes, Vostem is a volunteer event.
And Christoph will be talking about distributing multicast channels to third parties, a case
study with open source software and virtualization slash SROV.
Thank you Kiran and also thank you Kiran for co-organizing the Dev Room with me.
And if next year there are more volunteers, I think we would be happy to have that.
So I'm Christoph Masio.
I've been in the broadcast business for quite some time now and I run a company called Easy
Tools.
One of the purpose of this company is to distribute linear channels.
So either we create them or we get them from satellites or from abroad or we get them in
data centers and we deliver them somewhere in the world or to network operators and so
on.
So one of the critical parts of doing that is knowing how to distribute multicast to
other people.
Usually how does it work?
So first I must clarify a linear channel from my point of view at least.
It's basically transport stream.
So it's MPEG-TS over UDP or RTP, depends on your religion, both exist and generally multicast.
So usually you put seven TS packets inside one UDP frame and you're done.
And it's a continuous stream all the time.
How do you exchange these kind of streams?
Usually you take a rack in a well-known point of exchange.
In Paris there is one called Ted House, it's very popular.
He knows that in Paris.
And so you have your rack, you put your switch and you buy cross-connects from your data
center to your peers.
So your peers can be your sources, people provide your streams or could be where you
distribute them.
So the distributors, the operators and so on.
So for safety reasons you will want to have each source or each operator in a different
VLAN so that they don't see each other from confidentiality and also some content is a
little bit critical.
So the question will be how do you copy basically a multicationals from one source VLAN to a
destination VLAN.
It sounds stupid and perfectly easy to answer, but it's not so easy.
There is a pure network solution.
With some switches you have a feature called multicast service reflection, that's what
Cisco provides.
I'm pretty sure there is the equivalent on Juniper and that kind of thing.
But it's not so widely available, lots of switches don't handle it, only a small range
of that.
And so basically you can type a command there to say that you copy this multicast address
to this destination and so you copy from a VLAN to another VLAN.
So it's not widely available, there are good chances that your switches do not support it.
And also you cannot handle complex use cases like some operators want RTP, some don't want
RTP, you cannot remove RTP with that.
And some operators also want you to have specific PID, specific service ID, different
service name and so on, that you cannot do of course.
So generally you will end up using devices on top of your switch, you will have to plug
a device that will do that kind of job.
The broadcast solution, the most popular one is what we call DCM, it's a very popular
brand.
Initially by Cisco, now it's a company called CineMedia, it's a hardware service with electronics
inside and it does all this work of switching, it can even transcode actually with some cards,
it does a lot of things, but it's also very expensive of course.
But we have an open source alternative, it's actually something I wrote maybe 15 years
ago, it's called DVBLAST, I've been doing a lot of talks about that, but in this use
case it also helps.
Originally it was written as a DVB DMAX, so you have a satellite card or DTT card and
you want to get a transponder and split each channel into a multicast address, that was
the original goal of DVBLAST.
But actually with the dash D option you can also read from a multicast channel.
And in the arguments you can also say which is the IP address of the interface you want
to read it from, so basically which VLAN, that means which VLAN you will read it from.
So basically that reads a multicast stream from a specific VLAN.
There is a configuration file associated with DVBLAST, and you can put as many lines
as you want, so as many distributors as you want, and for each line you will just put
the multicast address you want to send it to.
And you can also optionally give the address of the interface, again you want to send it
to.
The VLANs you have on your switch, they will appear as network interfaces on your server,
so you can decide to which VLAN you want to send and which multicast address.
You have a number of options to turn on or off RTP, you can remap PID, SID, channel service
name and you can even spoof the source address, which is very useful in case your peer wants
to do IGMPv3, in that case you ask you to put a specific source address and that's the
easiest way to spoof it.
So problem solved, end of talk, thank you very much.
Now I wanted to add a little something.
You may want to run hundreds of DVBLAST on one server, but for some reason I wanted to
do some virtualization.
Why virtualize?
Because I have different customers and each of the customer has different channels, some
of them are doing adult content, some of them doing children content, maybe I don't want
to mix that in the same server.
And I don't have the money to have multiple servers at 10 hours because it's expensive.
So for client isolation it's required.
Also some of my clients have direct access to the VM, because that's the service I sell.
So again I don't want them to see the streams of the competitors potentially.
So I used Proxmox, that's a very nice distribution based on DBN and it's a very big front end
over KVM, very useful.
And in Proxmox what you will do is each of the VLAN you get from the Twitch, you will
bridge that and on your guest, guest virtual machine, it will appear as a network interface
with a VIRT-Io driver.
The VIRT-Io driver is the most optimized one.
You can also emulate a networker, but it's much slower.
So that's why everybody uses a VIRT-Io driver.
So everything works fine, end of talk and thank you very much for coming.
There is just one little problem.
The morning when you get called by one of your big customers and he says, I have discontinuities
on what you send me.
Discontinuities, that's what every people in broadcast feels.
And you start another DV Blast on the IP address that you put in, you see nothing.
Everything is fine, and they insist and another customer complains.
So what you do is you rack another server, that's a use case by the way, that's why I'm
telling my life.
So you rack another server and you listen to what the other server is sending.
And then you see, a lot of discontinuities indeed.
And you dig it a little and you just see that the VIRT-Io driver, I don't know if it's
guest side or host side, re-orders some packets.
I mean, probably there are several queues inside the drivers.
And if this packet does not have any luck and goes into a queue that for some reason
doesn't run for some time, it will be pushed there.
So I hear now the network guy will tell me that well, there is no guarantee on the order
of UDP packets, that's the specification, that's true.
But we are in an industry that relies on it.
Because if you don't have RTP, there is absolutely no way you can reorder your UDP transport string.
So I had to find a solution.
I cannot tell my customer, no, you have to use RTP and I don't care, I cannot, you cannot.
So I found the first workaround is by using another driver.
It's also a driver designed for virtual system, the VMware.
But it's supported by Poxmox, KVM and so on.
So it's called VMXNet and probably this one only has one queue and it solves the problem.
The only problem is that it uses 30% more CPU and this server is only doing that.
And I already have 64 core IMD epic servers that are not full, but more than 50% already.
So there is some kind of need to optimize.
So that's why I started to look at other alternatives and many big, some clever ideas that I found
on the net.
And one of them that you probably heard of maybe is called SRRUV.
So what is SRRUV?
So it's a feature of some network cards, not all network cards, some network card of that
feature.
In a normal installation, so the network card is owned by the host.
You have some kind of software switch that is handled by the virtualizer.
And here you have virtual interfaces to each of the virtual machine.
In an SRRUV setup, what happens is that the network card will create new PCI devices.
So it's a different PCI device.
And this PCI device, there is a feature called VTD on Intel that you have the current on
AMD that will allow you to dedicate a PCI device to a virtual machine.
And doing that, the virtual machine directly talks to the PCI device without anything going
through the host.
So that looks quite interesting.
These new devices are called virtual functions, VF.
And so on the VM, you just need to have a VF driver, which is included in Linux anyway.
So in my use case, I used Intel cards.
They're not the only one doing SRRUV, but it may be a little bit different if you use
different cards.
Not all cards have the same features.
So using SRRUV, it's a little bit tricky.
First it requires support from the motherboard, the CPU, the BIOS itself, and of course the
network card, as I was just saying.
You have to enable a number of features in the BIOS, IOMMU, but also ACS, ARI, which
is some kind of PCI Express routing protocol, lots of things.
And basically, we spent a couple hours, I think, just setting it up.
My personal advice would be for you to upgrade the drivers, the inside drivers, to the latest
version and the card firmware, because there is a firmware running on the card, to the
latest version supported by the driver.
Once I did not do that when I started my test, and I ended up having half of my VLANs working
and the other half didn't work, and there is absolutely no way to know what's going
on, I just had to reboot.
And it was in production, so I had to imagine what happened.
Creating the virtual function is actually quite easy, it's just echoed into a slash
CIS device, and with my Intel card, I can create up to 64 virtual functions, so that's
up to typically 64 VMs.
So the path through is very easy on the Proxmox, just a menu, and potentially, each VM has
access to other VLANs, so that is also, that may be also drawback because it sees all the
traffic that it can send to any VLAN, can send packets on any VLAN, so you have to trust
your clients a little bit, so it may not be adapted to all situations.
But it's quite useful, because you don't have to create a bridge every time you want
to create a new VLAN, you just create a network interface, an U821.Q network interface into
your VM, and that's all, so it's actually easier to manage.
Problem is, so everything looks perfectly fine, so that's the end of the talk, not yet.
I talked about sending to VLANs, but how to receive multicast from the VLAN, that's another
problem.
But initially, it looks like it works, so you put it in production, maybe a bit early.
So how does it work, how does a network card work with multicast, first I should maybe
remind you how it works.
So you have your multicast IP address, you translate that to the multicast MAC address,
so the end is the same and the beginning is just dropped.
And then you tell the card, every packet that arrives on this MAC address, I want it.
So that's how it works on any normal PC, that's quite a standard, it's called a MAC filter,
it's quite a standard feature.
The trick is, on these cards, the number of MAC filters is limited, and the way they
limit it is a little bit stupid, they take the whole buffer of the card, the whole number
of MAC filters they have, and they divide it by the number of VF you have, so 64.
And in the end, according to my calculation, your limit is around 100 multicast addresses.
So 100 may be a lot, but I have hundreds of multicast streams in my network.
So you may reach it, so you may think about segmenting your virtual machines not to go
above the threshold info, but it's still a dangerous game because there is a feature
of the intro driver, that if you reach that limit, it's a scientific phase, of course,
while you have a message in the message, but nobody reads that.
So you will try again, and try again, and try again, and after just a few trials, like
five, the kernel decides that your VM is crazy, and it won't talk to it anymore.
So you will still receive your multicast, but if you have any other command to send
to the card, like creating a new VLAN, which could happen, you have a new distributor,
so you can't, you have to reboot, you have to reboot your VM, fortunately, not the host.
So it's not that practical, and to be honest, I have a patch in all my installation that
disables the doing dead feature, and disables using the MAC filter at all, actually, because
I have found it not practical, in reality.
Yeah.
Is there a need for the MAC filtering at all, given that, like, modern switches, like,
you're not going to get any multicast traffic, unless you do an IGNB billing?
Yeah, but the thing is, so the question is, do we need the MAC filter at all?
The thing is, you will receive approximately two or three gigabit of traffic, and maybe
your VM does not need all of that.
So you have to decide which multicast you want, actually, that's my next line.
My first idea for a workaround was to put everything in promiscuous mode.
So promiscuous mode means what it means, it means your VM will receive all the traffic
that is received by the network card.
It looks like a good idea, but it dramatically increases if you use it, because from maybe
two gigabit per second of data, you only need 200 megabit per second of data, and the rest,
the kernel, we have to filter it.
So your kernel will do a lot of job.
And from what I've calculated, basically, the gain you had from going from VertiO to
SRIOV, you lose it, right there.
So that's first.
One problem is that, imagine you have two gigabit per second on your network, and you
have 20 virtual machines.
The network card will send 20 times two gigabit per second to your virtual machines.
And that means 40.
And 40 is the limit of the card.
And at that point, you will start losing packets randomly.
Again, silently, you will not know what happens.
And of course, obviously, you only know that in production, because when you first started
with one, two, three VM, it works perfectly.
So you say, yes, I have my solution.
And then you put all your load on it, and then one day, it just stops working.
So while activating promiscuous mode, it's actually quite easy.
Again, it's an echo in such a file.
So I have found a second workaround, which is a little bit better.
I'm using it in production.
It's called, maybe it's only an Intel card.
I don't know if it exists on other brands, but it's a feature called VLAN mirror.
And basically, it tells the card to send all the traffic belonging to a VLAN to a particular
virtual function, to a particular VM.
So that kind of promiscuous feature, but only for one VLAN, which is kind of a good practice
because it means that I think most people who have ever done multicast, when you have
a backbone, you put all of your multicast addresses in the same VLAN, maybe with different
address ranges, or maybe not.
And you expect at the other end that the receiver will pick up which multicast address you want.
This approach forces you to have different VLANs per customer.
So it's actually not a bad idea, but there is one ruleback.
One specific VLAN can only be sent to one VM.
So if you have, let's say, a big broadcaster that is sending you channels and several VMs
needs those channels, you cannot use that solution because only one will be able to read
from that VLAN.
So I have a third workaround, it's just to go back to the good old vertio.
After all, why not?
The packets in version were only on TX, not on RX.
So it works.
Actually, it has also some additional features because the bridge in Proxmox actually is
but most of the time it has IGNP snooping.
So we'll only receive the multicast addresses that you subscribe to.
So it's actually a good solution, but then it means you have basically interfaces to
read the packets from and interfaces to send the packets to, which is a bit of a mess,
but still it's a good compromise.
That's my compromise currently, VLAN mirror, all this one depending on the nature of the
VLAN I have to read from.
So all is good in the best of words.
At the end of the talk, thank you very much.
Not quite.
There is another topic I haven't mentioned yet.
What if I want to read a multicast stream coming from another VM on the same server?
That doesn't work because when you write through SIOV, it just outputs to the switch.
As far as I know, maybe some of you have a solution, there is no way to get the traffic
back to the network card and use it in another virtual machine.
You could do that, but if you don't want to have different VLANs, if you want to read
from the same VLAN, then you cannot read.
Okay.
Well, there is another solution with the intercard again.
It's called egress mirror.
You can make it so that everything that's output on virtual function number one will
be mirrored to virtual function number seven.
So virtual function number seven will be on your receiving side and virtual function
number one will be on the transmitting side.
So that actually works and also I use that in production.
So conclusions.
While multicast on virtualized environment is no picnic actually, and I'm surprised.
You don't find many papers about that.
I've struggled literally for years on this problem with a number of problems in production
because you only see the problem in production because you only see them on the load.
And so this has been a little bit tiring.
Thank you very much for listening to this.
And if you have any questions.
I'm guessing you tried that, but in Vert.io there are ways to actually ask it to be dumber
by not doing some of the things it does and I'm not sure but probably you tried that.
But you could probably have asked them to say, hey, I just keep them in order.
You say in Vert.io there are ways to ask it to be dumber.
I'm not sure.
Yeah, I'm not sure I've actually tested it.
I'm pretty sure it would have the same effect as using VMX net and probably you would see
an increase in CPU consumption anywhere I wanted to go to SRIOV because for other reasons
because I wanted to have my VM talk directly to the network card.
But I think it was better practice than Vert.io.
Yeah, James.
Did you just back the 3D slide, please?
Yeah.
Okay, there was no question.
Yeah?
Just AWS provide the DC to make sense with the SRIOV.
With SRIOV?
Yeah.
Okay.
So he says AWS provides instances with SRIOV.
I guess in the cloud you don't have that kind of problem because usually you do SRT.
Usually.
And SRT doesn't care if you need to roll those packets because it will.
Yeah.
Did you consider forcing, well, either disable multi-Q on the host or forcing each VM to
a different queue?
So that by definition you will not be reordering?
Forcing each VM to a different queue, I'm not sure it works.
Disable queuing probably it will work but with additional CPU usage.
Yeah.
I wanted to, as I said, I have 60 core APIC servers which are pretty big and they're not full,
but...
It should be possible to pin, it's possible to pin a process to give, if you do XPS CPUs
you can pin a process.
At the end I don't know if you can do that.
I'm not sure if the inversion happens on the host side or on the guest side.
Disable multi-Q in your host.
Yeah.
Enjoying your guest.
Yeah.
But yes.
Mm-mm.
eBay?
Why do you want to use SRIOV for, like, what seems differently is, like, what are the other
use cases you're thinking about with SRIOV?
The question is why did I want to use SRIOV?
Speed.
Speed is the first argument, actually, but also some kind of clean design.
Maybe I'm a little bit of a purist and I wanted my VM to have direct access, physical access,
I mean, to the network.
Also, so if you really want the list of argument, the traditional way of doing VLAN bridging
works, works, but probably other environments, you create a bridge for each VLAN.
And so you have a network interface for each VLAN.
Maybe there is a way to avoid that, but I didn't look too much into that.
That means you have a limit because I think you're limited to 32 network interfaces.
The limit is low enough.
Otherwise you have to switch to a VLAN switch, but considering the amount of problems I had
without it, I thought maybe I will not go into that.
I haven't tried, to be honest.
These are probably people, you don't see papers about the most people who are provided
to use HTTP.
Exactly.
As you say, SRIOV is mainly used by people who do HTTP or some TCP based.
And using SRIOV for non-networking things?
Ah, you better ask if I'm considering using SRIOV for non-networking things.
I suppose you imply GPU maybe?
Maybe.
But my question is, do you envision other places where multimedial creatures and short
GPU is almost an obvious one, but have you ever thought of using that to do other things?
GPU, we've actually tried to.
I'm not sure of the one we had supported SRIOV, but that...
On the really expensive ones.
Yeah, they have features like VTG, I think.
But in our experience, it didn't work well.
But we would like to have the ability to share a GPU among several instances so that you
could decode or encode on different VMs and so on.
These virtual machines that we have at Tellhouse, they don't do that because Tellhouse is not
the place to do transcoding because it's too expensive.
Fundamentally, yes.
I would be interested in having that for GPUs as well.
At least GPUs, if you have any other ideas, tell me.
That's my whole thought behind the question.
Just mentioning, because of Proxmox, I mean, it's support.
Sometimes it's support as well for SRIOV for decode, and they are expensive.
And it's just landing in Proxmox and QMU, so you can actually play with that if you have
to.
Could you use that to SDI or NDI pass through to get directly processing?
Yeah, you don't need the SRIOV for that.
The question was, can you use that for SDI or NDI pass through?
Well, NDI is network.
But SDI, I think we tried it and you can pass through the Blackmagic, for instance, device.
Yeah, you pass it through a VTD.
VTD is one thing.
VTD, I already passed through DVB tuners.
You can probably pass through DVB ASI, probably SDI.
Even though we don't do that on a regular basis.
But have you tried a different connector on a different VM?
Yes, you have four inputs.
A different connector on a different VM that would not be possible with a Blackmagic driver,
because it's seen as one device, one PCI device, so it's not possible.
If you design your own card, maybe.
Siwan, you may want to answer that question.
No?
Yes, we could do it, yes.
There's no technical reason.
Yes, Gaff?
Any container technology?
We could use container.
The thing is, as you know, we have our own software.
And our own software is actually delivered as a disk image, so it can only run as a virtual machine.
The idea to use container is not a bad idea.
There is less isolation, though, than what you have with virtual machine.
But this is something I would like to try in the middle term, yes.
Because there has been a lot of improvements in the past years, regarding the network and the space.
And you can probably have some direct MagValan interface,
so I think some kind of physical interface that you can have directly.
Yes, so it says we can probably have a direct MagValan in the container and do a direct physical interface.
I think we are running out of time.
Wow, oh, so...