So, it's been a while since I've been in FOSDEM and one of the things that, well, I
usually talk about technical topics, but this is really me talking about sort of using open
source instead of producing it, which is quite an unusual position for me to be in.
I work most often in things like the Linux kernel, I also work in open source politics,
so I did a lot of kernel interaction with the Linux foundation and one of those pieces
of interaction was actually being a founding member of Linux Plumbers conference, which
is the conference we'll actually be talking about for this presentation.
Plumbers was basically founded in 2008 as an in-person conference and it's always been
an in-person conference up to the pandemic years.
It's actually organized somewhat similarly to FOSDEM in that it has tracks and what are
called microconferences.
The microconferences are similar to dev rooms except they're really a bit more interactive,
so they're really expected to do participant conversations and usually you get a cluster
of people at the front talking to each other.
The tracks themselves when we have them mostly tend to be presentations, so very like this,
but the microconferences tend to be very discussion based.
If you think about trying to do that sort of hybrid and online, you realize you need
some sort of highly interactive video system, which is what we went through.
One of the good things we had for Plumbers is that the conference actually produces quite
a lot of money.
We have quite a lot of sponsorship, so it's quite easy to pay people to do things in the
conference.
One of the things we paid for in 2018 was to actually make the conference live streaming
for all of those who couldn't attend and also to try and put all of the videos of every
presentation and every discussion we had up on the website.
Obviously to do that, we needed to pay somebody to run AV in each of our rooms and Plumbers
tends to run as about somewhere between four and eight simultaneous tracks.
We needed about 16 cameras to do this and about between four and eight people to stay
in the room to also do it.
Obviously this then included provision of cameras in the rooms.
The way we set up cameras at Plumbers is slightly different from what you have here because in
your setup, the camera is mostly looking down at me because you think I should be the
focus of attention, which is true because I'm giving the talk.
When you're actually trying to record an ongoing live discussion that mostly happens
between people in the audience, you also need the people who are live streaming to be able
to see it.
One of the other fortuitous things is Plumbers is always set up with two cameras, one at
the back looking down on the speaker and then one at the front looking out at the audience.
The AV guys are actually responsible for switching between the front view and the back view so
that when we start to get conversations within the audience, the camera can actually catch
them and you can watch it.
This actually turned out to be really fortuitous as well.
Obviously, when you try and record your conference, the first thing you realize is that in a highly
interactive conference, most people don't use the microphone and it's really annoying
for everybody on the streaming when you get a load of silence in the room.
Getting people to use microphones is one of the key things that you have to do when
you start recording your conference.
This is still nothing to do with trying to get it live or hybrid.
This is just recording the conference.
You have to persuade all participants in the conversation to actually use a microphone.
However much money your conference makes, you will only have a few microphones.
This one, we have two microphones at the front.
I have one Le Valier microphone for a room that could possibly seat over a thousand people.
Trying to run around with microphones when you're doing an interactive conversation is
really difficult as I assume the guys with orange t-shirts will be able to tell you over
a beer after the event.
The solution we took was borrowed from another conference in Paris called Colonel Recipes,
which are these things called catch boxes and effectively they're square plastic polystyrene
blobs that have a microphone inside that you can throw from one end of a conference room
to another.
This gives you a way of actually getting the microphone between the participants very
fast and it's definitely fun to use and they do encourage participation.
So far we've had no injuries.
I should actually qualify.
We've actually had no serious injuries so far with people throwing microphones at each
other.
I assume there's always a first time.
The idea was really to try and make the conference that we were running available in real time
to people who can't participate because it's a current development conference.
We get people who want to come from all over the globe.
We always have corners of the globe, especially when we run the conference in America, that
even if they have the best will in the world and they can actually scrape up the money
to come, they're often denied visas by the US authority.
So it can be a really difficult problem.
And for our first attempts at doing this, the communication was really only outbound.
So the remote participants could watch what was going on in the room but they couldn't
really participate.
And obviously this all changed in 2000 when we got to the pandemic years.
Like all other conferences, plumbers moved to being fully online.
We were actually in the year 2000, we already booked a venue up in Halifax, Nova Scotia.
We had a huge amount of fun trying to get out of that venue contract when it emerged
that we couldn't actually hold the conference up there.
And then we all had to scramble to try and actually do this fully online.
So this is where the story becomes quite interesting because there are 10 people on Plumbers Committee.
Mike Rappaport over here is one of them.
Because of the origin, we are primarily kernel developers and because we're moving fully
online, effectively, kernel developers are becoming responsible for web programming.
Because what we're really doing is setting up a distributed set of web services.
And if you know kernel developers, we mostly tend to theoretically look down on web developers
because obviously the kernel development is at the hardcore of everything.
So this is somewhat a story of how a bunch of kernel developers tried to do web and
failed a bit, screwed up quite a lot and eventually got it working by the seat of their pants.
We evaluated a load of options for actually going fully online as every conference did.
They scrambled to, we discussed it a lot with other conferences.
People had different ideas.
We were actually under the umbrella of the Linux Foundation and there are much more commercial
entity than we are.
We're still a volunteer run, even pulling a lot of money.
That idea was just to pay somebody to do the solutions.
And frankly, in the early days, that looked like a good idea to us.
If you can just pay money to get rid of the problem, why wouldn't you?
And we have enough money.
We've got almost a million dollars in the bank now for Plumbers.
We could afford to spend a lot of that money to actually pay for somebody to sort out these
problems.
And the Linux Foundation tried to do it for several conferences and we looked at their
solution and went, you know, this is terrible.
I mean, why is it so difficult and so bad trying to run a conference?
And so the only way we could actually foresee getting this working is if we just abandoned
all of the commercial solutions and did it ourselves, which is much more difficult than
we imagined to actually do.
But the goal of this was that we definitely needed, in our online conference, to have
two-way video interaction in real time between the participants.
That was the goal.
And there was no real conference system that did that.
If you've tried holding conferences over Zoom, you can just about do it.
But things like BBB actually do an awful lot better than this.
And the other thing we discovered is that when you're running a collection of web services,
it really is all about the integrations.
Open Source gives you a lot of excellent pieces.
So we use Big Blue Button, we use Indico for the conference scheduling, eventually we use
Matrix for the Trap, we didn't use it in the first year.
But there's also a downside to that.
Whenever you try and integrate web services, you always write a set of glue around them
that needs to be maintained.
And if there are a load of disparate services that aren't used to talking to each other,
it does the impedance matching between the services.
And as the years go by, and remember this is a conference, so you put it on once a year,
every year you upgrade to the new versions and suddenly your glue doesn't work again.
So it always becomes a maintenance nightmare to get the glue working.
So if you ever came to the Plumber's Conference, the web front end that you first see, if you
go to meet.lpc.events, is a piece of code that John Corbett wrote in Python and Django
called the Linux Plumber's Front End, LPCFE.
Oh, and I should add that these things here are clickable URLs.
So if you go to the presentation and you want to know where it is, you can just hover over
it and it will tell you.
But obviously the green light is actually the way you log people onto BBB.
So by replacing green light and big blue button, we've effectively taken on the job
ourselves of doing integration, authentication, user management, and pretty much everything
else.
This is actually a major replacement of services on big blue button.
And the reason we did this is because we needed an integration with our conference timetable.
We just wanted the conference timetable to display on the first screen with a little
button that said, click here to go to this session.
It's very like you have in FOSDAM with the matrix chat.
If you just click on the integration of the matrix chat, it brings up the live stream
and you can see what's going on in the conference.
We wanted it to work pretty much as it does here.
And green light just wouldn't do that for us because green light was designed to be
a teaching front end for a single classroom.
It wasn't really thinking about multiple tracks, multiple track rooms, and timetable integration.
So we also, by getting rid of all of the attendee credentials, we actually had to integrate
with some sort of authentication server that could be distributed.
So being kernel developers, we chose LDAT because it's the oldest one and it's the one
everybody has the most trouble with.
So we should be able to use it, right?
The credentials were the email address that you used to register with the conference,
which is easy.
And the password was just the confirmation number because fortunately the Linux Foundation
CVent system spits out these long confirmation numbers and they're all guaranteed to be
unique.
So that's obviously really easy.
Problem, again, integration.
The CVent system will not speak to anything outside CVent.
So if you register for the conference and you don't automatically show up in the LDAT
database, what has to happen is I have to go to the CVent website and export a spreadsheet
of all of the users, figure out which are new ones, and then feed the details into the
LDAT server, which, I mean, as an offline activity, it sounds really simple.
It becomes really complicated when you keep getting people who don't register into the
last minute of the conference, turn up halfway through a session, want to register, and then
immediately be able to see what the hell's going on.
And they have to wait, I don't know how long, for somebody to actually put their credentials
in the database.
And when you're running a conference, pretty much all of your attendees are running around
– or all of your organizers are running around full-time.
Nobody has time to tend the back end of the database.
And that became a bottleneck for certain people.
So my request of you is if you ever get into this situation, register early, please, don't
turn up at the last minute, it's much easier.
Then the next problem is that nobody is actually really used to interacting online.
I mean, you think it's easy, you've seen Zoom calls, you will just turn up and you've
got, you know, pictures of each other on it.
If you try and do that with 800 people, Zoom doesn't really do autofocus very well.
You can't see who's talking.
And if you're just observing the conference, it just becomes a mass of faces and it's
completely useless.
So we obviously had standard issues like good webcam, good microphone.
The number of people probably among you sitting in this room who think that the standard set
of speakers on your Dell system when you just open the laptop will do when you're chatting
to each other, believe me, it won't.
The thing that audiences hate the most in online conferences is static background noise,
indistinctness, being unable to get the point across.
If you don't have in the attendees who are going to interact good AV equipment, this
is the thing which actually makes the conference experience go down the fastest.
This is about the only thing we got right because we gave a speaker gifts designed to
be delivered to anywhere in the world a month before the conference, a complete gift pack
of headset lights so you could actually see the face and a webcam you could put on the
top of your camera, a 1080p webcam so we were sure that we'd actually have a good view of
the speakers.
We made sure that they got to all of our nearly 100 speakers in the conference and most of
them didn't use them, which was a bit of a problem.
We also found, as I've explained with Zoom, that raised hands don't work and having a
load of people on video who you have no idea what's going on doesn't work either.
The thing we pioneered for plumbers is called video muting, which is where we try and instruct
the attendees to, if you're just watching, don't turn your camera on.
Pretty much it's a blank screen.
The only people who turn their camera on are people who are talking or people who are waiting
to talk.
It actually gives a good visual clue that somebody wants to interject at this point.
We found it was actually a reasonably good way of facilitating interaction in an online
conference and it certainly solves the Zoom confusion where you just can't figure out
who's talking to whom on your screen of, and for the online conference we did have about
900 attendees.
You could imagine on a Zoom screen everybody would probably get about five pixels.
In 2020, Big Blue Button comes with an internal chat system, which means anybody logged into
Big Blue Button can pop out a chat tab at the side and they'll be able to chat with
anybody who's logged into the conference.
This is why, if you were doing streaming of the conference, you couldn't interact because
that is a completely enclosed chat system.
We realized that people wanted to have conversations outside the discussion rooms as well, things
like hallway tracks.
We actually adopted a, it's a sort of open source project called Rocket Chat.
The core of the project is fully open sourced, but Rocket Chat themselves runs a more open
core model where there are certain enterprise upsells and everything, but this was just
the one we came across.
The good thing for us is it doesn't require any integration because we'd already started
to realize the more integrations you do, the more difficulty you have, the more strings
and bailing wire stuff there is to go wrong at the crunch time when you put the conference
on.
So no integration looked like a good thing until we tried to run the conference.
And then the problems were that chat was hard to read on the live stream because we're
streaming it out pretty much through YouTube.
We actually, Gilou Nadi wrote a streaming module for BVB and we just plugged it in as
an additional attendee and what it saw is what we streamed to YouTube.
And the problem is that YouTube does dynamic compression of most of the stuff while it's
streaming.
And if you have a low bandwidth connection, it will quite happily greek all of the chat
that's going on and you just can't see what is going on on the chat, whereas if it's sort
of having a bandwidth problem with faces and you're just looking at lips moving, you can
stand losing quite a few pixels, but if you're trying to follow chat, it's just impossible.
And obviously streamers didn't have any credentials to the Rocket Chat anyway because we based
on our LDAP, if you're not registered to the conference, you have no chat credentials.
So they couldn't join the hallway track either, which they didn't like.
This was the problem.
And the third problem was we had to buy an enterprise license from Rocket Chat actually
to get some of the notifications that we used for integration to work.
And the two chat systems were completely and fully disconnected.
So you couldn't use your Rocket Chat login to look at the BBB chat and vice versa, which
also was a bit balkanizing and a bit annoying for people.
And so you can see that the thing I initially portrayed as a huge advantage because we didn't
have to worry about the integration was the thing that everybody found the biggest disadvantage
about the conference because it wasn't integrated.
So you always get this pressure of, and you know as conference organizers, the more integration
you do, the more chance there is that something will go wrong.
But if you don't do enough integration, your audience don't like it.
So you have to get the amount of integration right.
And for several conferences, we actually struggled to do this.
Every registered attendee got a matrix ID, and since we used email and confirmation number
to log in, we figured we'd just use email address matrix as well.
But there's a slight problem here.
Emails contain an at sign.
Matrix does not like additional at signs in your ID.
So we had to replace the at sign with a full stop, which I thought, oh, it could be easy
for our attendees.
They're all computer geeks.
They'll get this.
They'll know email address for one server, email address but with at sign substituted
with dot for another server, complete disaster.
You know, they just couldn't figure out which ID correctly to use for which.
Eventually we got the BBB server to use either form of the address.
And we actually got the matrix server to translate using a JavaScript widget.
If they typed in the email address, it would just fill in the dot and everything would
just work.
We tried it without doing that for a couple of days and got a bucketful of complaints.
I should also add that when we chose Rocket Chat in 2020, I wasn't very happy about the
decision because I was a fledgling matrix user, and I was sure that matrix could handle
the conference, and I thought it would actually be a good idea to do this.
So I advocated even in 2020 that we should be using matrix.
And finally, after all the difficulty with Rocket Chat in 2021, I got my wish.
So they told me, we're using matrix and you're in charge, which was fun until the first day
came along and the matrix server with 900 people on it just fell over.
And we had actually done scaling testing.
I mean, I'm not naive.
I know things can go wrong at scale that you don't see in your own lab.
So we'd invited everybody to come along for a town hall, but only 100 people showed up.
And it turned out that most of the scaling problems we had were somewhere between 100
and 900.
So we hadn't hit any of them in our little test.
And I'd also tried to scale up the matrix server.
So our massive 16 CPU system that we were paying about $100 a month for, when you ran
top on it was only using one CPU because I hadn't realized that matrix Synapse, which
was the server we chose, was single-threaded.
I mean, who in 2021 writes a single-threaded web application?
But the matrix people thread matrix by using the back-end web server through a sequence
of proxies to actually do this, but that is not the default configuration.
And so on the Monday of the conference, I was left there in my own little basement sweating
away going, how the hell do I scale this thing?
But fortunately, several people who attended FOSDEM, which did use matrix before we did,
also ran into this problem.
And they were able to send me email saying, yeah, we've got this guy you can talk to.
He'll teach you how to do it.
So fortunately, within about four hours of discovering the problem and realizing it didn't
scale, we had a roughly scaling matrix server working.
And I managed to actually complete the scaling by the end of the first day.
So for the second day, matrix was actually fully functional, fortunately, since it was
my little baby.
And I actually did a little blog post about how we got this to work, just so if anybody
else ever runs into the problems, there are online resources to go to to get yourselves
out of it.
One of the good things about using matrix was that even the free tier attendees, the
streaming tier attendees, could now actually interact by chat.
So anybody with their own matrix ID could log on to any of our matrix rooms because they're
all public, in the same way that all the FOSDEM rooms are public.
Free attendees still got their own separated matrix ID, but that you only got logged on
to the matrix ID if you actually popped up the side chat panel.
You could also use your own separated matrix thing and use your own matrix ID as well if
you wanted to.
And it just depended how you wanted to interact with the conference.
Cee Lunadi was the one who actually modified the BVB server to, instead of opening its
own chat tab, we just stole the iframe concept it uses for the blackboard and we did a riot
embedded which is done by some, the URL to the GitHub account is there.
The integration, the riot embedded is not, it's not as good as running the electron type
application, but trying to embed an electron application into a web form is just a nightmare
and doesn't work.
So it's a very cut down version, all it really does is text pictures and reactions.
It doesn't do any of these sort of complicated threading or replies or anything, but that's
pretty much good enough for conference attendees.
But you can see the primrose path we're threading down because this is yet another integration
into matrix, in BVB and matrix that is not done by either upstream.
But it was really useful, so the streamers could see the chat, if it was greaked on their
streaming screen, they could actually use a matrix client and still view the chat in
real time without having to worry about video compression and reduction.
And so we thought after we'd done this successfully for two years now, especially for one year
with rocket chat, one year with matrix, it wouldn't be too difficult to move over to
hybrid, which is probably really the point of this talk.
We already had all the remote infrastructure.
By the way, to run this style of remote infrastructure for a conference size of plumbers, so we're
talking about 900 people and we assume that for the in-person one, we'd probably be about
400 in-person, 500 online, it will cost you somewhere between $3,000 and $5,000 to actually
set up and run.
That's not cheap to do the hosting of all of this stuff, especially because we had to
host at scale, we needed one BVB server per each track room, we had about six track rooms,
we had a few spare servers left over for hack rooms and we had one for the hallway track.
But the way we looked through this, we already had all of the AV necessary for actually just,
this is the naive thought, which is where kernel developers go, we got AV in the room,
it's got two cameras and it's got a load of microphones, the room is just another single
person attending the BVB chat, it'll be easy to do it like that, the naive way of thinking.
But it really, really is not that simple trying to get a hybrid conference running.
The significant problem is that you have to integrate with in-room AV and in-room AV,
well, okay, so at ULB, you control it, but we were doing it in hotels.
ULBs have their own local AV teams, if you touch a piece of their equipment, they tend
to go screaming to management.
In order to get the integration done, you can't do it yourself as the conference organizers,
the AV people, the audio people have to do it and it's surprising the number of audio
people who sort of sit behind those switcher boards at the back of conferences who've never
even seen a sort of online hybrid conference and have no idea how to set this up.
So just plonking this sort of video display down in front of them and saying, hey, now
you're in charge of audio and video and we just want the conference to work, it wasn't
actually a winning strategy.
So we discovered rapidly and unfortunately it was pretty much on the day of the conference
that we had to train the AV people and have to use all of this stuff.
And we were actually quite fortunate because we'd ordered the video-based AV, we actually
had two sets of AV people.
So the hotel usually supplies audio people to plug in your projectors and your microphones
into the hotel's audio.
And then we had a separate AV company whose job was to manage the cameras and sort of
cut the whole thing together and do the live streaming.
So we actually had two AV teams effectively and we were assuming that we could just co-opt
the AV team who was doing the live streaming because live streaming, running live streaming
is not much different than running an online conference, right?
Wrong.
So we held meetings beforehand with them and we talked to them about it.
We couldn't hold meetings with the hotel audio guys but we assumed it would just be a plug
for them, which it wasn't.
And nobody really anticipated the issues that we actually had.
The AV setup of remote attendees is not obvious to somebody who's used to dealing with throwing
microphones around a room because there is one extra wire that's coming off the internet
over which a lot of attendee voices can come.
And the first time they set up their AV patch panels, this wire just wasn't plugged in.
The second time it was bypassing all of the audio routing which meant that the sound quality
and the volume was chopping at high strength because it wasn't being attenuated through
the usual filters.
I mean, this is the 101, you know, how many screw-ups can you have?
We basically did them all on the first day.
So let's go through a list of the problems.
Oh, I even forgot to mention that between 2001 and 2002, we decided to upgrade the whole
of the infrastructure.
We had specific things in BBB we wanted.
After I told you we have two cameras, we have one at the back facing the speaker, one at
the front facing the audience, the new version of BBB could actually handle multiple cameras
with different views.
So we figured that wouldn't it be really great if, you know, you want to see the audience
so you just select the audience camera as a remote participant because you want to talk
to them, or let's say you're a remote speaker, you want to see the audience, but the rest
of the people just want to see you, we'll do this multiple camera selection.
It was going to be the cornerstone of our implementation of this.
Problem was, nobody told the AV guys.
So the AV guys had come set up with a switcher for the two cameras, which is how they ran
the streaming because obviously you can select one camera view or the other because the live
streaming pretty much has one camera view out on it.
And by the time we all sat together in the room, there was no way of unplugging this
setup and we doing it such that we could plug in two video cameras because it all has to
be done from the AV control deck and they physically didn't have another input because
they were expecting to use a camera switch.
So this is yet another screw up in planning.
Third screw up was because we'd upgraded the new BBB, they'd redone the, and this user's
node react as the frame backbone, they'd redone the way it was working.
Riot embedded no longer worked as a pop out panel.
We discovered this fortunately four days before the conference.
So this was Guilinardi doing a very quick back end retrofit, trying to get the matrix
chat that everybody was expecting to work to work.
And fortunately by the time we did the conference, it was actually working, but this was yet
another squeaky bum issue where we tried to do everything at the last minute.
The first day we had a lot of rooms where the room couldn't hear the remote attendees.
This is the wire problem from the remote or the remote attendees couldn't hear the room
because it's a two wire problem.
It's not just sound from the remote coming into the room, it's sound from the room going
out to the remotes as well.
So we had to have the sound correctly plugged into BBB.
The streaming guys thought they were just plugging the sound into the streaming.
So if you watch the live stream, it had the correct sound, but if you were on the BBB
thing, it was completely silent because they hadn't plugged the sound in.
It's just yet another screw up that we didn't actually think to test out.
So this is how it looks in the sort of conference hall as we were going around it.
So this is the panel you can see.
This is what BBB looks like pretty much in a classroom situation.
This is the presentation, which you can actually minimize if you click this button, but obviously
we thought that the AV people might be able to click the minimize button when the conversation
is going on and then click it back again.
No, it didn't really work.
They didn't really want to do this.
So we just kept everything up.
This is the attendees panel, which is all scrolling, which is nice, but if you think
about it, that's not really what you want to see in the conference.
The chat panel, if you click this, will pop out and come here, and the rest of this will
really shrink, which is also not really what the attendees did.
So what we should have done is have the chat panel down here and get rid of the attendee
list or make it pop out only.
So we put this on our to-do list for next time because it's sort of a nice to have,
and it wasn't essential to getting the conference working.
This is actually an excerpt from the RISC-5 microconference.
So Atish Patra is actually asking a remote question.
You can see that he's asking it of a member of the audience, so we've switched to the
audience camera view, because ideally there would have been two cameras here, one of the
audience, one of the speaker, and he could have switched between them, but we screwed
up and didn't get that working in time.
The tab also includes shared notes, which goes to a back end on ether pads, so the remote
audience can also see the shared notes, and they can also click on the pop-up, and these
buttons here actually allow you to mute people who are not supposed to be speaking when they
are, because in a remote conference you get lots of people who forget to mute their microphone
and then have a conversation with their wife or their significant other.
This happens quite often.
The way BBB is set up is that anybody can mute anybody, but we thought that might be
a bad idea, because it's cantankerous kernel developers.
You never quite know who they'll mute.
Once you've muted somebody, you can't unmute them.
They can only unmute themselves, because this is a sort of, you can't be unmuting people
while they're going to the bathroom or something.
It wouldn't be good on the conference.
We figured this might be a problem, so we restricted this functionality to administrators
only, and then nobody could figure out who the administrators were, so it became a bit
of a problem.
The video overlays work quite well.
This is pretty much a blow-up of a 1080p thing, and you can see the greaking happening in
the room.
It's very difficult to make out individual attendees, but the sort of remote participants
were reasonably visible to most people.
BBB, and especially, you can actually see the way it works with the video-muting idea.
It's really all you see are the people participating.
One of the other things that we'd actually tried to do in the room is the camera at the
front facing the back actually had a pan and zoom facility, which the AV operators assured
us would allow them to zoom in on whoever was speaking, but in the event, what we discovered
is that only two of the AV operators out of our six rooms knew how to use this.
We did get panning and zooming in some of the rooms, but not in all of them.
Again, this goes back to preparation and training.
We will do better next time.
It actually took us quite a while to get the audio routing right.
We think that plugging two wires into an AV system wouldn't be that difficult, but in
fact, it really was.
We finally figured out that what you had to say to the audio guy is, here's a wire from
what's effectively a microphone on the computer.
The video guy will give it to you.
You just plug it into your fader panel and remember you have it.
You give him a wire for the AV stream that also has to go into the BBB thing.
Obviously we got that to work, but obviously we had other problems.
One of the other things that we'd really thought about was, when we get into the room, we'd
actually like to have the chat and all the remote attendees on one monitor, and then
we'd like to have the presentations on another.
We thought we'll have a two projector setup, which sounds fine until you realize that we
have rooms about as big as this, and the two projectors were right over there and right
over there.
Somebody sitting over there can't see what's on that projector.
The only way we could make it work was to display the same image on both projectors,
which meant that two projectors was a complete waste of money, and we should have gone for
a single central one as well.
Next year we will either try with one or three, just to see if we can actually get this right.
Obviously as I said, the two-camera handling, we thought would be the centerpiece of the
technology.
The reason why we'd upgraded BBB and caused all of the React JScript failures on the Matrix
back end was to get this facility that we couldn't actually use.
That was not only sleepless nights for one person for a week to try and get it all to
work, but complete screw-up on the day when everything we thought we'd bought with that
upgrade didn't function.
The other problem we had was, because the AV people are actually doing the live streaming
and because of the way BBB works, there is no choice other than to actually run the Master
BB session that is showing up on both projectors on their AVMux console.
I had thought this might be a problem as well, so BBB is very heavily React JavaScript
based and certain browsers handle it better than other browsers.
It works with Firefox quite well now, but it is mostly optimized for Chromium.
So I had actually already asked them, God, if we have to do this, what is the built-in
browser of this Tmux thing?
They said, oh, it's Chrome, and I thought, oh, God, it's going to work.
But the problem was that the Mux platform might have used Chromium as an internal thing,
but Chromium is open source.
So the guys who built the Mux platform had just basically taken WebKit, gutted some
of the JavaScripts and hoped that that would be OK.
What it really meant was the threading functions of JavaScript didn't work very well.
The result was that it kept dropping out of the conference, and obviously, since this
is running the AV conference, when it drops out, the in-room chat and the remote participants
fully separate.
The problem is there's no indication that this has happened from the console unless you
happen to be watching very carefully, because all you see is that a microphone disappears
from that little icon in the top left that says, hey, you're the room, and suddenly your
microphone's gone.
That's the only indication the AV people had they dropped out.
Trying to train them to notice this and reconnect was a bit of a problem.
So we do have automated reconnections on the back end, but we will try and integrate into
BVV, but when we raised it on the list, they looked at us and said, well, how come you
have this problem because nobody else has this problem?
If BVV kept on dropping people out of the conversation as a matter of course, people
would have complained.
This was a specific interaction due to the cut-down browser in the T-Mux interface that
nobody had anticipated.
Again, another screw up from not actually trying it beforehand.
So as I said, rejoin was manual, assuming the AV guy noticed.
One of the good things we did have was interim shepherds who were committee members.
So they could actually watch the on-screen panel.
Every time the icon disappeared, I or one of the other shepherds could run to the back
of the room and pat the AV guy on the shoulder and say, you need to rejoin, please do this.
So we could make it work with a lot of manual fussing behind the scenes, but it wasn't easy.
So one of the other things that we realized is that the hindsight is 20-20.
They might want to run it from the AV console, but they don't have to run the whole thing.
What we could have done is have a separate laptop that actually ran all of the AV connections
under the charge of, I won't say somebody competent because it would have been one of
me, the kernel developers or somebody, but somebody who would pay more attention.
And then what we could have done for the V-Mux guys was to actually give them a VNC session
or something like that into the panels that they were projecting over the streaming and
everything else.
And that way we would have had much more control in the room and we'd have been running a
full-fledged browser that wouldn't have had most of the connection problems that we had.
But obviously trying to sort that out on the day was pretty much impossible.
Particularly unfortunate is this AV-Mux platform is sort of Windows-based and Windows does
not do VNC very well and trying to get it to work with RDSP, which is the Windows native
thing, didn't work very well.
And we just, it was, we were in Dublin, we didn't have any equipment, we had to organize
the parties and deal with online conferences, we just couldn't get it to work.
So we do know that for the future we are going to try and do a shareable console.
And I need to leave time for questions.
I have 10 minutes left.
We had a load of other problems as well.
As you can see, this, so I'm, in order to try and tell you all the difficulty about
doing this, I'm talking up the problems.
From the audience's point of view, this conference was actually very successful.
We got plaudits from most of the attendees that the interaction worked as they expected.
They could have their one-on-one conversations, including between remote and local people,
remote and remote people.
So everything appeared to just work for them, modulo, you know, the problems of not being
able to hear the room and having to rejoin and having to tell people they lost AV and
they needed to rejoin.
They forgave us for all of that.
So it was actually highly rated as an incredibly successful conference, but it was a bit like
a swan with that, you know, serene sort of thing swimming over the surface and a load
of committee members frantically paddling underneath to try and get all of this to stay
up and running.
Our other big screw up was that we had a committee of 10 people, but because this was pandemic,
not all of the committee could turn up in Dublin, and by the time we got to Dublin,
we only had two local people who actually understood the system and could deal with
it.
Surprisingly enough, the two people who are actually standing up on the platform now taking
the blame for all of this.
But believe me, in a conference with six tracks and AV guys who don't know what they're doing,
if you want a lot of exercise, please do it with two people.
But if you want to actually make it work seamlessly and vaguely competently, you need one person
who understands the setup in each room.
So next time we will try and do it with a minimum number of people as we have same number
of track rooms.
Again, it's not rocket science.
It's perhaps something we should have seen ahead of time, but we thought it would be
easy and it wasn't.
So to leave a bit of time for questions, if you have them, running conferences is actually
hard.
So just remember that when you go home from here.
And volunteering for a conference is actually an act of public self-sacrifice that more
of you should consider, because honestly, you will find out that it is really hard.
Running online conferences is harder, especially if you try and do it yourselves, because there
is actually a lot more to do.
It is actually more difficult to run an online conference than it is to run an in-person
conference paradoxically.
And this is mainly to do with the integration issues.
Everything that doesn't integrate well is either your fault or you're responsible for
integrating it.
And integrating web stuff, even in this magic day and age of sort of, you know, web scripting
and operators and everything else, really doesn't work that well.
And hybrid is really only for masochists.
Only do a hybrid conference if you want to give yourself an untold amount of pain that
you can then come and complain to an audience like you about later.
But if you really want to get it right, the key is training your AV people.
Do not wait like we did to turn up on the day and say it will be easy.
You need a couple of training sessions a few months beforehand to try and get all of the
kinks sorted out.
And having meetings and discussing it doesn't cut it.
You need the AV people to set it all up and preferably have a few people fly out to see
the setup, work with a few remote people so you actually check it out in all of its situations.
And obviously, you need to do this in advance.
You don't do it on the day of the conference, which is what we tried to do.
Oh, well, lesson learned.
So with that, I think I have about five minutes for questions.
This was actually all done in open source using HTML and CSS.
That does make me a web developer, proud of it now, finally.
Obviously you can see that current developers don't quite do web as well as web developers.
So if any web developers would like to join our program committee and become jointly responsible
for next year's disaster, please come forward.
The presentation is all online, and I've also put it up on a link in PentaBath.
So you should just be able to click on it.
All of the links in the presentation that are highlighted are clickable if you want to
see the URLs.
And with that, I'll just say thank you.
And Mike and I can now answer questions.
Oops.
You have Mike.
Okay, we'll share this, Mike.
Any questions?
Please speak loudly into the microphone.
Yes, thank you.
From what you said about the AV setups in the hotels, were there any discussion to just
ask the hotels to, you know, move their stuff and have your own stuff brought in?
Because to me, that seems after your talk to have been an easier solution to just say,
you know, your AV guys and your AV is probably good, but we have really specialized needs.
Could we just bring our own equipment in?
Okay.
So the question is, could we bring our own equipment in?
And the answer is pretty much every hotel, no.
Because remember they do the AV routing as a very specialized sort of adaption for the
conference rooms.
They have a lot of really expensive speakers, sometimes in the ceilings, sometimes in the
back of the room, and they don't want any non-experts touching it.
So what hotels usually have as a clause in their contract is a requirement that you use
their own sound guys.
And this clause is non-negotiable.
We did try to negotiate with a couple of hotels where we said, because we had a few screw-ups
even in the live streaming days where the hotel didn't do very well and we thought we
could do better.
But it's always come back to us as an absolute no.
It's actually very lucky for the ULB guys are used to having students handle their equipment
and don't seem to have as many hang-ups about it as hotels.
So there are people here who can be trained on this equipment who can use it.
But pretty much when you're running a conference in a hotel or in a professional conference
hall that we've also been to, so a community convention center or something, you actually
have to use their AV staff.
You don't get a choice.
And that means that you always have this extra component of people that you have to deal
with.
And they're usually people who don't really understand fully what your requirements are
because it's very difficult to communicate to the head of time.
And these people are usually the people that you don't get to talk to until you turn up
at the conference.
So when you have your intermediate AV guys like we had and you have your meeting with
them, you always have to make sure that they have enough knowledge and insight that they
can do the knowledge transfer to the sound guys who are going to be coming in on the
day from the hotel.
So yeah, it's a huge problem and there's almost nothing you can do to get out of it
besides sort of having a very friendly university who will let you play with their equipment
because pretty much nobody in the professional circuit will let you play with their equipment.
I have one question from the chat.
Was it worth it to record the audience?
So the question was, was it worth it to record the audience?
And the answer to that is actually very much yes, because remember in a micro conference
we're stimulating discussions between arbitrary groups of people.
So these would be people who are sitting in the audience physically versus people who
are sort of remote.
And in order to have a correct two-way conversation, you actually have to be able to see the person
on the other end.
And so the whole point of the audience camera, at least when it worked with the panning and
zooming, was that if you were having a remote to local conversation with the audience, the
remote people could actually see the local participant.
And if we only had a camera facing the speaker, we'd either have to call every participant
up to stand on the stage, which wastes time, or we'd just have to not have them be able
to see the person they're talking to.
And being able to see the person you're talking to is very important for interactions.
There are lots of visual cues that people get wrong.
So I think having an audience camera is actually one of the good benefits of, or at least one
of the benefits the plumbers did and one of the good insights we had to try and get an
interactive conference running.
So I wouldn't go back and do no audience camera, no.
Just wanted to get back to what you said about the ULB being willing to allow us to use their
audio equipment.
I just want to point out that this has a lot to do with the fact that we've been doing
this for 20 years and they know us.
It wasn't always the case, and if you're not supposed to, or you're not working with
the ULB, this probably isn't the case.
So could you repeat the, I got half the question, no.
Yeah, it wasn't the question really so much as to point out that us being allowed to touch
the audio equipment here, by the ULB has a lot to do with the fact that we've been doing
this for 20 years and they know us.
Okay, so the point being made is that ULB only allows the staff here to touch the equipment
because they've been doing it a long time and they've built up trust.
The way we work, we tend to go to a different hotel every time we hold plumbers in a different
location.
So pretty much even if we could get a trust to build up with a hotel chain, we wouldn't
be in the hotel often enough to do it.
But the important point to emphasize is that pretty much 95% of the time you will be forced
to work with AV people who are supplied to you by whoever owns the equipment.
You won't get a choice of who does it.
Okay.
Is that it?
Okay.
Well, thank you very much indeed.