So good morning.
I am Flor Arlé and I am here with my colleague Jean Monnier to present you a way to announce
the video call experience by using the flexible forward error correction.
We work in the Bell Dawn Communication Company.
This company has developed the LIN phone product that is a soft phone open source to make some
video and audio calls.
And it works on several platforms.
So today I will explain to you how we implement the forward error correction for our video
calls.
The first way we choose to use the flexible FEC scheme, I will explain to you how it is
described in the RFC 8627, how it has been developed in our products and I will show
you some results.
So at first let's talk about the forward error correction.
If you see this real schematic way to represent a video call, you have at first two people
who share a video call.
On the center side you have a device that captures the video.
The signal is encoded here by a video encoder that transforms the signal into frames.
And those frames are split into packets that will be sent to the receiver through video
stream on the network.
On the receiver side the packets are collected and decoded and the frames are recovered back
and the signal is displayed and the receiver can see the video.
In the case of the video call we are in a real time context.
So we work on the RTP retight transport protocol that describes how you can send video or audio
with the internet.
This protocol describes the format of the RTP packets and it is more regalable than
the UDP because for example you don't have latency problem that is adapted for real time
communication.
Unfortunately in the real world you have problems, you can experiment problems with your network
so the traffic may be high, you can have a low bandwidth and sometimes you lose packets
during the transmission.
So the receiver doesn't collect all the packets, the signal is not complete and you see that
your video can freeze and it is really annoying for everyone.
To overcome that you can use a strategy to recover your lost packets.
For the forward error correction you will recover the lost packets by using the next
packet and also a run-down-see information that is sent at the same time than the video
stream.
With this way you can recover your full video and have a nice video call.
We choose to use the flexible scheme for our project.
So when you detect loss in your receiver, the receiver side, you have several strategies
that you can apply.
You can ask to send the packet again but you will have to wait to get the packet.
You can primitively decide to send twice the video stream but it is really costly.
Or you can try to recover the packets with what you have at the time.
The forward error corrections allow you to recover the lost packets by using the run-down-see
information and the other packets.
There are several algorithms, for example the low density parity check codes and there
is the flexible forward error correction.
That is the method that we choose because it is really simple.
It is based on the combination of the packets with an exclusive XOR operation and it is
free.
There is no patterns.
It is a recent standard.
It has been for example developed in the web RTC.
So we can be interoperable with it.
The standard is described in the RFC 8627.
It describes, this document describes fully how an FTP stream can be protected with flexible
FEC.
It gives the format for the repair packet that will be sent to carry the run-down-see
information and it proposes the codes to generate those packets and to decode to reconstruct
the lost packets.
This RFC is applicable to all media, not only video but also video, text and application.
So now we will explain you how it works as described in this RFC document.
So at first when you have a video stream you send the packets within an FTP session with
a source RTP stream.
Your packets are here represented by the squares.
They have a unique sequence number that increase with time.
And when you want to make a flexible FEC you add another run-down-see RTP stream and you
don't change the source stream.
So it is backward compatible.
The principle is simply to take a set of source packets, combine them with your XO with a
priority FEC encoder and generate a repair packet here.
So for example this one is called R5, R4.
Why using XO operation?
It is because of a nice property of this operation that makes you able to recover one of the
packets if you have all the other.
So you can encode a packet and decode a missing packet.
At the receiver side when you detect a loss, for example here the packet S4 has been lost,
you can get it back by applying your exclusive OR over S6, S5 and the repair packet R4.
And then this new packet can be sent to the stream here.
To operate your flexible FEC you can choose several parameters.
You have to decide what is the length of your repair window.
It is a duration interval that allows you to buffer your source packets to be sure that
you have enough source packets to make the recovery.
And you have to decide which packet you will combine with which one within a protection
pattern.
So now we present you several protection patterns.
If you represent your source packets like a block here from S1 to SdL with L columns
and D lines you can make a first way to protect that is a row protection, a one-dimensional
non-interleave protection where the XOR is applied on the rows.
So here you generate D repair packets that protects a set of source packets of length
L. Another way is to combine them by columns.
So here you have L repair packets that protects the source packets with depth D.
So now I will show you how you can recover the source packets with this combination.
So here you have an example with row protection and here with column protection.
Because you have random losses in your transmission you can apply the XOR to recover the lost
packets here with the row application of the XOR and here with the columns.
But in some cases it will be more difficult because if you have burst in your transmission
it means that you will lose a consecutive source packet.
You won't be able to recover because you can't have both packets here and here you
will recover the columns that have only one loss but not the columns that have more than
one loss.
To overcome this problem you can make a two-dimensional protection.
Here you have simply the combination of row protection and column protection and it generates
L plus D repair packets.
In this case you have in the RFC an iterative algorithm to recover the lost packets.
So here I show you two examples with long burst here and here with random loss.
The algorithm starts like this.
You repair at first all the rows that can be repaired.
Then you apply the XOR on the columns and you repeat so the rows, the columns until you
can't repair any more any packets.
Here you can see that the burst has been fully resolved.
All the packets have been recovered but sometimes you don't have chains and you can't recover
some patterns that are connected like a cycle here.
So in this case you can do much more with the flexible efficiency.
But this two-dimensional protection is really efficient for bursts.
But sadly it has a cost because you have to send a lot of repair packets.
You can measure the impact on the width that you will need with this term the overhead.
It is a ratio between the number of bytes of the repair packets that you sent over the
number of bytes of the protected source packets.
Usually the repair packets are bigger than the source packets but if you suppose that
all the source packets are approximately the same size, the overhead will be 1 over L
for the row protection, 1 over D for the column protection and 1 over L plus 1 over D for
the two-dimensional protection.
For example here are the values of the overhead with increasing values of L and D on increasing
protection level.
You see that the overhead increases very fast.
The RFC describes also what are the formats of the packets.
So first you have your source packets with the LTP convention with an LTP header and
a LTP payload.
And you will generate your repair packets that are also LTP packets with a header and
a payload.
But within this payload you will carry two kinds of information.
The first one is written in the FERC header.
It's information about how to identify which source packets are protected.
And in the repair payload you will have the result of the XOR operation between the payloads
of the source packets.
When you apply the XOR between the payload here you have to be sure that your source
packets have the same length.
So sometimes you will need to add zeros at the end of your payload in order to have the
same length for all packets.
A single repair packet will carry all the information needed to recover the source packets.
It says the size of the source packet protected and which is the configuration of the protection.
For example, here when you have R1 and you see that in the FEC header you read L is positive
and D is zero.
You know that you have a raw protection and the sequence number of the source packets
that are protected comes from S1 to SN plus L minus 1 with the consecutive values.
If L is positive but D is equal to 1 you also have a raw protection but you are inside
a two-dimensional pattern.
So you know that you will collect several repair packets that protect rows and then
you will have a set of repair packets that protect the columns.
And when L is positive and D is more than 1 you have the column FEC protection and the
repair packets that are protected are interleaved.
So from SN to SN plus D minus 1 times L and it can be the column of the two-dimensional
FEC block protection but it also can be a single column protection in one dimension.
This method has been implemented in our project NINFON.
We decided to use four sets of L and D parameters.
It comes from one-dimensional very low protection to high protection with three and three.
Ideally we want to have two-dimensional parity protection but it has a cost because you have
to send a lot of data.
So we decided to adapt our protection to the loss rate that is measured in the transmission
and also to the network capabilities.
The repair window is 200 milliseconds.
It is long enough to collect all the repair packets for any values of L and D and it doesn't
cause any delay in the video.
The RFC has been implemented in C and C++ in our LINFON SDK.
All the elements of the FEC stream are written in the library ORTP and in our streaming engine
for video and audio we added a way to manage the video quality with the flexible FEC.
For many months about our strategy for the video quality, our rule is to make the best
possible use to use the bandwidth but sometimes you don't know the bandwidth at the beginning
of the call.
It can change during the call and you have all events to manage.
We want an optimal video setting so the best definition betrayed and firm rates but most
important we don't want freeze in the video.
So we decided to prioritize the packet protection before having high encoding setting.
To have an adaptability to the network events we make periodic control of several values
so we measure regularly the available bandwidth, the loss rate and the bandwidth that is dedicated
to FEC.
For example in this graph you can see that we propose to have low FEC protection when
you have low bandwidth and to enable high level of FEC only when the loss rate is very
high but if you have a lot of bandwidth you can have full FEC protection it is not a problem.
And finally when you have congestions it means that you have too much packets and the transmission
stops.
You disable immediately the FEC because it is not your tool and it will make the things
worse.
So now we will show you some video with flexible FEC activated.
So here you have a video, so we simulate a video call with a moving pattern.
In the first window here there is 6% of packet losses and we do no protection so you can
see that the video is really bad.
It is a very very bad case, 6% is really a lot of losses.
In this window we have enabled the FEC with a one dimensional row protection with L is
equal to 5.
You see that the video moves a little more but they are still freezing.
In the last window it is a two dimensional FEC protection with a high level, 3 L is equal
to 3 and D is equal to 3.
And you can see that here the video is perfectly fluent so we have recovered all the lost information
here.
We have measured the recovery rate here with several values of FEC protection and you see
that it increases very fast.
So the flexible FEC is really interesting to recover the lost packets and the effects
are really obvious.
Another example here, this time we have simulated a transmission with loss and burst so we
lose consecutive packets so it is a very bad situation.
This time you can see that the performance of the FEC reconstruction decreases a little
but they are still interesting.
In the two dimensional parity protection you can see now some phrases but it is still
much more fluent than the initial video.
So we can make some conclusions about flexible FEC.
It is a simple and resilient way to improve the resiliency to the packet loss in video
transmission.
It is based on the fact that you send redundant information on a dedicated stream.
It is adaptable to the level and the event of your network and it works with a short
delay because you don't have to wait that the sender sends you back the missing information.
And the exclusive operation is really efficient and rapid.
But you have to keep in mind that you will need a significant bandwidth so in some cases
it is not indicated.
The RFC 8627 gives a complete description of the flexible FEC scheme and it is clever
because it is also backward compatible with the RTP protocol.
And we show that it gives a real improvement in the video quality.
So we decided to release it this year in the video calls of the Linfuan project and we
want to in future work add it to the video conference and the audio stream.
So thank you for your attention and we will be happy to answer any questions.
Thank you.
The question is about the size of the source packet.
And in fact you are right, it is an issue that we have to deal with.
The source packets doesn't have the same size.
And for the encoding you have to pad the payload to make the XOR operation.
And the thing is that when you combine them to build your repair packet you will have
very high, very big repair packets and your overheads will be increasing a lot because
of few big source packets.
So that is a problem that you have to deal with.
So you can change the size of the source packet if possible to make the more equal sizes.
But you have to adapt the overheads to decide to have, you have to measure your overheads
to check that the repair packets are not too big compared to the source packets and to
decide to reduce the FEC protection in order to keep an overhead reasonable.
But yes you have to take care of the real size of your source packets.
I don't know if it answers your question.
Thank you.
Yes?
Then you always have this fixed delay of 200 milliseconds, right?
On the repair window?
Yes.
So we have a fixed value here.
The question is do we have a fixed duration for the repair window of 200 milliseconds?
Or it can be changed?
The fixed delay.
The question is the video output is put on the screen 200 milliseconds after the respective
video break has arrived, right?
Yes, the 200 milliseconds is a delay that you had before displaying your video.
Yes?
I'm sorry.
Okay.
Yes, that's the, in fact yes.
Yes?
So when you assemble in the stream and rows and columns, I know the second one is reversed.
Is that right?
No, it's not reversed.
In fact, sorry, it was maybe not clear in the representation.
You have, okay.
This one?
Yes.
The draw comes back to here.
So you read those one, then those one, and then those one.
The second question, do you have any examples of an STP line describing how this is expected?
Those ones.
You have an example of an STP packet that contains a line that describes how this is established.
The question is, is it mandatory to signal?
So when the stream is a setup on the signaling layer, you have no flyer, I'm guessing, you
still use STP and this would exist as a line in the STP to describe how it's established.
Yes.
The question is to know if during the call exchange, we signal the use of this protocol
into the STP right.
I'm not sure.
There is a.
Yes, okay.
Signaling.
That's the answer.
Okay.
Yes.
So what you described seems very similar to RAID 5 with disk drives.
So when you join drives in RAID, you have an eight blocks and then you have one drive
block which contains a pad bit for each of the blocks.
But there's also RAID 6 which has not one but two pad blocks.
Could that be applicable to your skin here?
So you have a line of five packets and then you have not one but two redundancy packets
which could help you recover the line between two packets of lost.
Okay.
So the question is about what happens if we lose repair packets for example or if we.
Could that skin be improved with having two pallet packets but one?
Not one packet but two.
Yes, it may be it's always a trade-off between what you what bond with you have and what
you decide to send to improve the protection.
There is a other way to other protection pattern described in the RFC.
For example, you can decide to protect small very specific source packets by using for
example a flexible mask.
So you can have maybe here in this example decide to protect some packets twice and some
other once or not at all.
Yes, it can be an improvement to prioritize the most important packets in your stream.
And there is other schemes.
One pair is one pair two block one.
Yes, there is other parity codes.
Honestly, I have to try to tell you which one can be better.
I don't know.
Probably one of the problems if you apply too much protection is that you're also going
to a lot of overhead.
So at one point if you're in a lossy network, you send more data to try to recover from
more loss, you end up in this spinning spiral that doesn't make things better.
So finding the balances is where the black magic is usually.
Well, thanks Flo.
Well, no, it's okay.
Oh, there's one more question.
Please go ahead, we have some time.
Maybe regarding exactly what you said, how do you know that you don't make it worse?
Yes, in fact, we had the problem.
So at some point we sent more information in the written on stream than simply sending
the video stream twice.
In that case, we control the overhead periodically.
And when it goes above, for example, 1.9, you reduce the FEC protection.
It's not always indicated.
So it's a decision that you have to make.
We have established empirical rules to manage that.
Yes?
I want to ask you about the masking of your gold right now.
Yes?
The slide is right now there.
You have said that you can protect the specific package.
Yes, like you protect a group of packets.
Yes?
That's for example in video conversations for you.
For example, the push and the step or that X2, 6, 4, and you protect, for example, the
key frames do be interpolated instead of, isn't it?
Yes, so the question is if you can protect, for example, the key frames of the video conference.
Yes, it's a way to choose which packet you want to protect.
If you don't want to protect everything, but mainly the key frame, it's a good approach.
Or you can make the one-dimensional, two-dimensional only protection, only when you have the packets
of the key frame.
Okay.
So the receiver size.
Is it right like all your key frames are on one column and you just protect them?
Yes, but you, so the key frames are not necessarily in the same rows or the same columns, but you
can change the values of D and L whenever you want.
On the receiver side, the receiver just read what it has in the FEC header.
You see the value of D, the value of L, and it adapts the configuration to recover the lost packets.
Okay, so you can modify that value dynamically during the...
Yes, you can dynamically modify the protection configuration.
And it's very powerful.
Yes.
How do you measure the network's bandwidth, for example?
Because without provoking the network with high load, right?
Yes, how do we measure the available bandwidth?
We have estimator in our program that tries to measure the...
If I remember, the time delay between the reception of packets and try to establish the bit rate.
And we see if there is congestion, if there is congestion occurs or not.
But it's based on estimation.
We have to deal with that.
Yes, the idea of the algorithm that we use is to measure the regularity of the packet at the receiver side.
And when it changes, we can deduce that the bandwidth is more...
is close to be saturated.
This is more or less the way that we use.
So do you use RTCP for this configuration?
Yes, and we also use RTCP feedback as well in order to measure...
packet losses from the receiver side.
But it's a bit different than just bandwidth.
For the bandwidth, it's really the regularity of the receiver side, which is a measure.
Thanks, Bois.
Thank you.