Okay, we can start. Thank you for attempting to this meeting, which is about Linfone and
video conferencing. My name is Jean Monnier. I'm involved in the Linfone project since
2010 and I'm also part of the company which is backing the Linfone project since ten years
almost. So first, I'm going to provide you with a quick introduction on Linfone and then
have a couple of words around video conferencing with SIP followed by an introduction on the
selecting forwarding unit, which is the art of the modern video conferencing systems. And later to
talk about what is required on the SIP client part to be able to interrupt with this kind of
video conferencing system. And finally, the conclusion. Okay, so just a couple of words
about Linfone. Linfone is a voice-over IP client implementing the SIP protocol which started
in early 2000. It's available on Linux, Android, iOS, Windows and Mac. It uses SIP as the base
standard for almost everything including audio, video call and instant messaging presence. Everything
which is required for real-time communication. And it also provides some end-to-end encryption
for messaging based on the signal protocol more or less. The Linfone team developed the
Linfone software but also SIPTrover, which is basically a SIP proxy. And if you want to use
SIP account, it's possible to create them on our website for testing purpose mainly. Okay,
video conferencing with SIP in a couple of words. It's around a couple of standards. The first one
is SIP, a basic SIP with an invite referring by to create a conference, join the conference to be
able to invite other participants to the conference. And it's almost based on the RFC 4579, which
defines how to create a conference and how to join the conference. And there is also some
interesting standard, which is the RFC 5366, which is about defining the list of the participants
of the conference. So it's for the establishment of the conference. And the next important standard
is what we call the conferencing events package, which is based on the subcribe notify RFC. And
the idea is that when a participant joined the conference, it initiates SIP subscribe to the
server. And the server then notify to every participant of the conference, which are the
states of the conference within who is going to join is their audio video, everything which is
related to the status of the conference. On the media port, it's regular RTP. And for this video
conferencing project, we added the support of two important RFC, which are about bundling all the
media stream into the same socket in order to avoid to have too many RTP sockets, RTP streams
per SIP client. And regarding the security, it's a regular SIP TLS for the for the signaling. And
for the media pass, it's either SDS where the symmetric key is set in the SDP or the RTP or
even SRTP DTLS. And for the RTP itself, it's a standard AES. Okay, now let's introduce the what's
the selective forwarding units. And I'm going to start with the description on what we used to
have for conferencing server. So in the past, the media mixer received the video from every
re-user decode the video stream, perform the mixing, it could be mosaic or any layout, and then
re-encode all the stream to be sent to every participant. This kind of software exists since
30 years, maybe 20. Here, I just want to show you a page that I saw in the RFC 7667, which is
about the RTP topology of former legacy conferencing system. So for each client, A, B, C, here it's
audio, but it could be the same for video. You have one RTP stream going to the media server and
one RTP stream, which come from the media server to each client. And it's server side that everything
is decoded, mixed, and sent back to the client. The advantage of this approach was that it was
very simple from the client side, as the conferencing server was almost the same as calling
regular user agent. The drawbacks of this approach was that video layout was defined
server side, so you could have one or two different layouts. It requires a lot of CPU
resourced server side, as every video stream has to be decoded and then re-encoded. And
to end encryption was not possible due to the fact that video was decoded. Now, if we go to
the selecting forwarding unit, the idea is that the media server is no more decoding and then
encoding every video stream, but it's more switching video coming from every device to every
other devices. And it could be done depending on several policies, like ActiveSpeaker or Mosaic.
And for that, we also need some information coming from each client, like the volume of the
audio stream in order to be able to know who is talking without having to decode the audio stream
as well. If I go to the same schema, still from the RFC 7667, now you can see that from the RTP
standpoint, you still have one RTP stream for each client coming, going to the media server. But
now you have also one incoming video stream per participant of the of the conference. So if we
follow the audio, the video stream from the client A, you can see that it is copied to client B,
but as well to client F. So it's no more a media mixer, but a switching matrix, more or less.
What are the advantages of this architecture is that video layout is no more defined through
the server side, but the client can decide where to display every participant of the conference.
It's an application choice, no more a server choice. It scales very well as there is no
resources which are used through the server side to decode or encode the video stream.
And finally, it's open the door for end-to-end encryption as the media server no more
has to know the content of a video stream. The drawback of this approach is that it requires
the Cpliant to be able to manage mostly stream, which was not the case for a standard one-to-one
call. So for the Cpliant agent, what we had to change are the following mainly. It's mainly
about multi-stream requirement. In the past, the Cpliant was able to send one audio stream
plus one video stream. Now, it requires the client to be able to send one, but most of the time,
two video streams, one for high resolution video and another, a second one for thumbnail,
as well as being able to receive one video stream per participant of the video conference.
Just an example of the SDP to show what is involved. So bundle mode, as I said,
which is required, RTP MOOCs as well, it's to limit the number of sockets used for the media.
This extension is related to audio level in order to be able to display who is talking and also for
the server to be able to select which video stream is talking. It still uses IC to be able to limit
the usage of media release. And from the video part, what you can see is that there are two
video streams in receive only, one for the high resolution of the camera and another for the
thumbnail. So it means that we encode two times the video. It could be replaced by some video
encoder like H264, AVC, which supports a multi-layer functionality. But if you want to be able to do
that with a simple VP8, it's better to encode two times the video. And for the receiving side,
there is one video stream because in this example, there is only one participant in the video
conference. But this part would be multiplied by the number of participants of the conference.
Okay. So this is for what we have done on the Linfone project for this feature.
It could be tested with the FlexiSIP server, which is currently running on our infrastructure.
So you can create a video conference thanks to this conference factory CPURI. And using Linfone
client with version above 5.0, it's possible to join a video conference. Okay. Thank you.
Conclusion. Okay. So now Linfone is capable of joining video
conference in two modes, mosaic and active speaker using a selective forwarding unit,
which allows to scale. Possible evolution that we have in mind is to implement the Xcon
conferencing server in order to be able to manage conference from a website or to have
something more advanced. We are also thinking about adding end-to-end encryption to this
video conferencing server and why not to provide the compatibility with WebRTC,
knowing that the media protocol that we use are very close than WebRTC.
Useful link. If you want to have more information about this work,
you can go to the Linfone website and to have a look on our GitHub.
Okay. That's it. If you have a question. Thank you.
Are you aware of any other CIP client that implements multi-party video with someone?
Not yet because the work to move from a regular CIP phone with only supporting one audio stream
and one video stream to support this multi-stream is very significant and I'm not aware of any
work in progress so far. So if you want to use it, you have to go with Linfone.
Even if it's fully standardized, if we are following standard.
Thank you.
Not yet. Not yet, but we are quite confident that it's going to scale as we have removed
all the needs for audio or video encoding server signs. So it's really about switching of
packets. Maybe the question might be around the network on the client side.
Around network, on the client side, as we are using, we are sending two resolutions
from the client, a high resolution and a low resolution. And in the case of active speaker,
we only send back to every client the high resolution of the people which is currently
talking and low resolution of every other participant. So it highly limits the needs of bandwidth.
Yes.
On the client side, you now decode more than one stream.
Correct.
It's almost the same answer. We decode one high resolution and many low resolution
and the CPU resources is depend on the resolution of the video that you have to decode.
Just one question about the STP that you showed before. So were two receive only streams for
the client? Was that from the client? It was from the server.
Okay, because that was my question. Because it looked like the client.
The server received two videos from the client, one in high resolution and one in low resolution
and sent one video to this client. There is only one participant in this comparison.
From the client perspective, when you switch from big resolution to low resolution,
you still use the same M line that you have to send to the client.
It's, yes, exactly.
Okay, thank you very much.