Well, all right. I'll get going since we're here. My name is Saul and today I'd like to
talk to you about our little project P10K or how to get 10,000 participants into a
GC meeting.
No, it doesn't go on the loudspeakers, it's just for the recording. It is what it is.
Sorry, I lost my voice. I'll try. I suppose most of you know what it is, but for those
who don't, it's a way about to see compatible video conferencing application. I like to
say that I can think of it in three ways. A set of open source projects that allow you
to either deploy it or, you know, piecemeal it and build something with it. It's also
a set of APIs and mobile SDK so you can embed it into your existing application and fully
open source Apache to license and we have a pretty vibrant community that helps us build
some stuff.
So I've talked about scaling GC meets a couple of years ago here at FOSDOM with what we did
during the pandemic. Also at Comcom about how we reached 500 participants. Then of course
somebody will ask, yeah, how do you do more, right? So that's what I'm about to go on today.
A quick TLDR on what the trick is to scale up is mostly to cheat because it turns out
that you never see 10,000 participants at the same time. So you need to paginate and
not show all of them at the same time, not load them at the same time. Also on the back
end, you don't want to be, you know, taking care of 10,000 things at once. You want to
be really careful avoiding re-renders on the react side of things. So on your front end,
you definitely don't want to have 10,000 things. And very importantly, reducing signaling.
And this is kind of the crux of the thing. So with all of those things, we ended up getting
500 participants in a single meeting. All of them are fully functional, bidirectional audio
video participants. They will never all have video on. So that's sort of fine. I'm going
to go a quick run through our architecture. So when we dive into XMPP, we know what we're
talking about. XMPP is our course signal protocol. You heard it from Matt for chat. So all the
participants join an XMPP mock, so a group chat. And then our focus, you call for negotiates
a session with each participant. And then they all end up mixed in the JVB, which is
where we allocate the media. So this is like a back of an app design level, but it's pretty
accurate. Prosody is our XMPP server of choice. And you call for is the one that will allocate
sessions here and then establish sessions with the users. So they all end up, you know,
having this connection. Now, how do you go about solving 10,000 participants? Well, first
of all, we do some research. And what we knew is that presence is stanza. So XMPP presence
was our Achilles heel. So we needed to sort that out. And intuitively, when you need to
support many of something, you think of, well, I'll partition it in smaller chunks. And maybe
that's how I do it. So there is federated mark for that. So we thought maybe that's where
it goes. And turns out the military had sort of researched this problem as well. And there
is this cool white paper called federated multi-user chat for military deployments.
And one of the things they got there is how to avoid these presence flooding. And they
do that with the visitor role. And that's where we got the idea from. So the idea is
that we're going to have two types of users, the active users and like passive users. So
we don't need to know about all these passive users, like all these audience, we just need
to know the number. We don't need to draw a tile for them. They don't need to be as
apparently they're participating in the meeting. They're just viewers, right? And this is what
the visitor role in XMPP Mach-Lingo means. So a passive participant can then become an
active participant by switching the role. Because we're not building live streaming.
So what we want to build is a way to actually actively be able to participate. Anybody of
those 10,000 participants should be able to take the mic anytime. Scenarios for this,
earnings calls on public to traded companies. Just because we can, you name it.
So step two, how do we test it? Because if we build it, we need to be able to know we
have a complete store goal. And in order to test 10,000 participants, you need, well,
10,000 participants. So we use a big ass linear grid and we created some lightweight clients
so that we could have a lot of chunks that join the call. They've got no UI. We spawn
multiple browser windows with multiple tabs, with multiple of these clients. And a recent
trick is we use insertable streams to drop all media. One thing you can do is modify.
Another thing you can do is drop it. So it's nothing. And then there are a lot more lightweight
in our Selenium grid. Otherwise, it would take millions just to test what you're doing
is right. There's a PR by Philip Hankey actually to do something like Chrome would said, Black
Franks, very tiny ones. So maybe that's where we go in the future as well. And we also delay
track creation so that we don't create tracks. If you join muted, we don't need to do the
whole create a video track that is useless and things like this.
The next thing is we scale the signaling. And the way we do it is we ended up having
multiple processes servers. This is one node, but it could be spread to multiple nodes.
So we have a main process server, which is where the active participants join the meeting.
And then we have up to five extra nodes, which we call visitor nodes, where people join in
this visitor role. So the presence is not broadcasted. Jigofa will decide which one
you join, usually depending on the capacity. And the trick to actually become an active
participant is to just join this one, join the main one afterwards. And we can do that
very fast because you don't need to recreate the XMPP connection.
So now, in order to establish this sort of mesh, we ended up using Federation, even though
it's like within a single server, but still. So there's server to server bidirectional
connections to avoid having duplicated connections. So custom modules that's where process shines
because it allows us to do all these customizations to mirror like chat messages that have been
typed in a visitor node to the main node and back. So to kind of fake it that they are
in separate instances, actually. And as I said, becoming active is fast because you
don't need to recreate the XMPP connection. You just need to join a different mock.
Our step number four is to have an improved topology for media routing. Currently, we have
Octo, which allows us to spread the load across multiple bridges. But this doesn't work very
well for such a large load. You need a tree-style topology where some people are just receiving
and a full mesh for those who are actively participating. So both loads can be spread.
And last, we need to fix up the UI, let's say. So we don't need to render the visitors.
We just need to know that there is 100 people and then 9,000 visitors. And that's it. So
we want to refine the UI a little bit. We're thinking of using the raised-hand functionality
to become an active participant. So you raise your hand, you are approved and then you become
active. That's how we're thinking about it. Now, some of it is in the present, some of
it is in the future. So how is it going? We got there with 51 bridges. We got 10,009
participants. So it worked out. There's still some work to do. So the UI is not yet final.
We're polishing up a little bit. And we're still going to add some more modules to mirror
all the data we want, like the polls and other stuff. And we're thinking that maybe we don't
really need to support 500 active participants because that's a weird conference, really.
So that number could actually be lower or pretty much configurable. So you can say,
I want these very many active participants and the rest, it will be visitors. And that's
that. And of course, we need to make it easy to deploy for everyone. Right now, this is
a bit held together with that tip. Before I go, I'd like to give a shout out to the
heroes that worked on the guts of this. You may know their names from our community, Boy
is Domenico and Jonathan, incredible characters. And I'm relaying the message. I know they
knew words, but they did the work. And I like to share the love we have for Prosody. We
wouldn't have been able to do it, I think, without such a flexible piece of software.
They help us. We help them. It's a very nice relationship we have with the project. We
love Matt and team. So shout out to them. And since that's all I got, you can follow
the progress there. We have documentation actually how to deploy the existing way of
doing things. Again, early stages, but it's there. And if you have any questions, well,
I'm around here. Or find me online. Thank you very much.