[00:00.000 --> 00:12.920]  So, it's been a while since I've been in FOSDEM and one of the things that, well, I
[00:12.920 --> 00:19.000]  usually talk about technical topics, but this is really me talking about sort of using open
[00:19.000 --> 00:25.160]  source instead of producing it, which is quite an unusual position for me to be in.
[00:25.160 --> 00:33.200]  I work most often in things like the Linux kernel, I also work in open source politics,
[00:33.200 --> 00:37.960]  so I did a lot of kernel interaction with the Linux foundation and one of those pieces
[00:37.960 --> 00:43.040]  of interaction was actually being a founding member of Linux Plumbers conference, which
[00:43.040 --> 00:50.040]  is the conference we'll actually be talking about for this presentation.
[00:50.040 --> 00:54.800]  Plumbers was basically founded in 2008 as an in-person conference and it's always been
[00:54.800 --> 00:59.040]  an in-person conference up to the pandemic years.
[00:59.040 --> 01:03.840]  It's actually organized somewhat similarly to FOSDEM in that it has tracks and what are
[01:03.840 --> 01:05.760]  called microconferences.
[01:05.760 --> 01:10.520]  The microconferences are similar to dev rooms except they're really a bit more interactive,
[01:10.520 --> 01:14.840]  so they're really expected to do participant conversations and usually you get a cluster
[01:14.840 --> 01:20.600]  of people at the front talking to each other.
[01:20.600 --> 01:26.080]  The tracks themselves when we have them mostly tend to be presentations, so very like this,
[01:26.080 --> 01:30.280]  but the microconferences tend to be very discussion based.
[01:30.280 --> 01:34.080]  If you think about trying to do that sort of hybrid and online, you realize you need
[01:34.080 --> 01:39.440]  some sort of highly interactive video system, which is what we went through.
[01:39.440 --> 01:44.160]  One of the good things we had for Plumbers is that the conference actually produces quite
[01:44.160 --> 01:45.480]  a lot of money.
[01:45.480 --> 01:52.000]  We have quite a lot of sponsorship, so it's quite easy to pay people to do things in the
[01:52.000 --> 01:53.000]  conference.
[01:53.000 --> 01:57.960]  One of the things we paid for in 2018 was to actually make the conference live streaming
[01:57.960 --> 02:03.440]  for all of those who couldn't attend and also to try and put all of the videos of every
[02:03.440 --> 02:07.880]  presentation and every discussion we had up on the website.
[02:07.880 --> 02:13.360]  Obviously to do that, we needed to pay somebody to run AV in each of our rooms and Plumbers
[02:13.360 --> 02:18.960]  tends to run as about somewhere between four and eight simultaneous tracks.
[02:18.960 --> 02:24.800]  We needed about 16 cameras to do this and about between four and eight people to stay
[02:24.800 --> 02:30.160]  in the room to also do it.
[02:30.160 --> 02:33.680]  Obviously this then included provision of cameras in the rooms.
[02:33.680 --> 02:38.720]  The way we set up cameras at Plumbers is slightly different from what you have here because in
[02:38.720 --> 02:43.280]  your setup, the camera is mostly looking down at me because you think I should be the
[02:43.280 --> 02:48.120]  focus of attention, which is true because I'm giving the talk.
[02:48.120 --> 02:52.920]  When you're actually trying to record an ongoing live discussion that mostly happens
[02:52.920 --> 02:57.360]  between people in the audience, you also need the people who are live streaming to be able
[02:57.360 --> 02:58.360]  to see it.
[02:58.360 --> 03:02.720]  One of the other fortuitous things is Plumbers is always set up with two cameras, one at
[03:02.720 --> 03:08.440]  the back looking down on the speaker and then one at the front looking out at the audience.
[03:08.440 --> 03:13.040]  The AV guys are actually responsible for switching between the front view and the back view so
[03:13.040 --> 03:17.400]  that when we start to get conversations within the audience, the camera can actually catch
[03:17.400 --> 03:19.960]  them and you can watch it.
[03:19.960 --> 03:23.480]  This actually turned out to be really fortuitous as well.
[03:23.480 --> 03:29.080]  Obviously, when you try and record your conference, the first thing you realize is that in a highly
[03:29.080 --> 03:34.360]  interactive conference, most people don't use the microphone and it's really annoying
[03:34.360 --> 03:39.520]  for everybody on the streaming when you get a load of silence in the room.
[03:39.520 --> 03:43.960]  Getting people to use microphones is one of the key things that you have to do when
[03:43.960 --> 03:45.640]  you start recording your conference.
[03:45.640 --> 03:49.440]  This is still nothing to do with trying to get it live or hybrid.
[03:49.440 --> 03:53.320]  This is just recording the conference.
[03:53.320 --> 03:59.480]  You have to persuade all participants in the conversation to actually use a microphone.
[03:59.480 --> 04:03.400]  However much money your conference makes, you will only have a few microphones.
[04:03.400 --> 04:05.560]  This one, we have two microphones at the front.
[04:05.560 --> 04:11.280]  I have one Le Valier microphone for a room that could possibly seat over a thousand people.
[04:11.280 --> 04:14.760]  Trying to run around with microphones when you're doing an interactive conversation is
[04:14.760 --> 04:18.840]  really difficult as I assume the guys with orange t-shirts will be able to tell you over
[04:18.840 --> 04:21.720]  a beer after the event.
[04:21.720 --> 04:27.160]  The solution we took was borrowed from another conference in Paris called Colonel Recipes,
[04:27.160 --> 04:33.200]  which are these things called catch boxes and effectively they're square plastic polystyrene
[04:33.200 --> 04:38.080]  blobs that have a microphone inside that you can throw from one end of a conference room
[04:38.080 --> 04:39.200]  to another.
[04:39.200 --> 04:42.920]  This gives you a way of actually getting the microphone between the participants very
[04:42.920 --> 04:50.640]  fast and it's definitely fun to use and they do encourage participation.
[04:50.640 --> 04:52.680]  So far we've had no injuries.
[04:52.680 --> 04:54.520]  I should actually qualify.
[04:54.520 --> 04:58.240]  We've actually had no serious injuries so far with people throwing microphones at each
[04:58.240 --> 04:59.240]  other.
[04:59.240 --> 05:04.760]  I assume there's always a first time.
[05:04.760 --> 05:09.920]  The idea was really to try and make the conference that we were running available in real time
[05:09.920 --> 05:14.680]  to people who can't participate because it's a current development conference.
[05:14.680 --> 05:17.840]  We get people who want to come from all over the globe.
[05:17.840 --> 05:22.160]  We always have corners of the globe, especially when we run the conference in America, that
[05:22.160 --> 05:25.280]  even if they have the best will in the world and they can actually scrape up the money
[05:25.280 --> 05:28.880]  to come, they're often denied visas by the US authority.
[05:28.880 --> 05:33.600]  So it can be a really difficult problem.
[05:33.600 --> 05:38.280]  And for our first attempts at doing this, the communication was really only outbound.
[05:38.280 --> 05:42.440]  So the remote participants could watch what was going on in the room but they couldn't
[05:42.440 --> 05:45.800]  really participate.
[05:45.800 --> 05:53.240]  And obviously this all changed in 2000 when we got to the pandemic years.
[05:53.240 --> 05:58.200]  Like all other conferences, plumbers moved to being fully online.
[05:58.200 --> 06:03.400]  We were actually in the year 2000, we already booked a venue up in Halifax, Nova Scotia.
[06:03.400 --> 06:07.720]  We had a huge amount of fun trying to get out of that venue contract when it emerged
[06:07.720 --> 06:10.240]  that we couldn't actually hold the conference up there.
[06:10.240 --> 06:14.680]  And then we all had to scramble to try and actually do this fully online.
[06:14.680 --> 06:20.360]  So this is where the story becomes quite interesting because there are 10 people on Plumbers Committee.
[06:20.360 --> 06:23.120]  Mike Rappaport over here is one of them.
[06:23.120 --> 06:29.040]  Because of the origin, we are primarily kernel developers and because we're moving fully
[06:29.040 --> 06:34.120]  online, effectively, kernel developers are becoming responsible for web programming.
[06:34.120 --> 06:39.280]  Because what we're really doing is setting up a distributed set of web services.
[06:39.280 --> 06:46.000]  And if you know kernel developers, we mostly tend to theoretically look down on web developers
[06:46.000 --> 06:50.280]  because obviously the kernel development is at the hardcore of everything.
[06:50.280 --> 06:56.040]  So this is somewhat a story of how a bunch of kernel developers tried to do web and
[06:56.040 --> 07:05.320]  failed a bit, screwed up quite a lot and eventually got it working by the seat of their pants.
[07:05.320 --> 07:10.760]  We evaluated a load of options for actually going fully online as every conference did.
[07:10.760 --> 07:15.240]  They scrambled to, we discussed it a lot with other conferences.
[07:15.240 --> 07:16.840]  People had different ideas.
[07:16.840 --> 07:21.720]  We were actually under the umbrella of the Linux Foundation and there are much more commercial
[07:21.720 --> 07:22.840]  entity than we are.
[07:22.840 --> 07:26.680]  We're still a volunteer run, even pulling a lot of money.
[07:26.680 --> 07:30.000]  That idea was just to pay somebody to do the solutions.
[07:30.000 --> 07:34.320]  And frankly, in the early days, that looked like a good idea to us.
[07:34.320 --> 07:38.360]  If you can just pay money to get rid of the problem, why wouldn't you?
[07:38.360 --> 07:40.480]  And we have enough money.
[07:40.480 --> 07:44.480]  We've got almost a million dollars in the bank now for Plumbers.
[07:44.480 --> 07:48.560]  We could afford to spend a lot of that money to actually pay for somebody to sort out these
[07:48.560 --> 07:49.960]  problems.
[07:49.960 --> 07:53.600]  And the Linux Foundation tried to do it for several conferences and we looked at their
[07:53.600 --> 07:55.880]  solution and went, you know, this is terrible.
[07:55.880 --> 08:01.680]  I mean, why is it so difficult and so bad trying to run a conference?
[08:01.680 --> 08:08.080]  And so the only way we could actually foresee getting this working is if we just abandoned
[08:08.080 --> 08:13.680]  all of the commercial solutions and did it ourselves, which is much more difficult than
[08:13.680 --> 08:16.600]  we imagined to actually do.
[08:16.600 --> 08:21.800]  But the goal of this was that we definitely needed, in our online conference, to have
[08:21.800 --> 08:25.800]  two-way video interaction in real time between the participants.
[08:25.800 --> 08:26.800]  That was the goal.
[08:26.800 --> 08:30.040]  And there was no real conference system that did that.
[08:30.040 --> 08:34.120]  If you've tried holding conferences over Zoom, you can just about do it.
[08:34.120 --> 08:39.040]  But things like BBB actually do an awful lot better than this.
[08:39.040 --> 08:44.360]  And the other thing we discovered is that when you're running a collection of web services,
[08:44.360 --> 08:47.680]  it really is all about the integrations.
[08:47.680 --> 08:50.360]  Open Source gives you a lot of excellent pieces.
[08:50.360 --> 08:55.600]  So we use Big Blue Button, we use Indico for the conference scheduling, eventually we use
[08:55.600 --> 08:59.560]  Matrix for the Trap, we didn't use it in the first year.
[08:59.560 --> 09:01.760]  But there's also a downside to that.
[09:01.760 --> 09:06.800]  Whenever you try and integrate web services, you always write a set of glue around them
[09:06.800 --> 09:09.680]  that needs to be maintained.
[09:09.680 --> 09:13.560]  And if there are a load of disparate services that aren't used to talking to each other,
[09:13.560 --> 09:16.640]  it does the impedance matching between the services.
[09:16.640 --> 09:21.360]  And as the years go by, and remember this is a conference, so you put it on once a year,
[09:21.360 --> 09:25.400]  every year you upgrade to the new versions and suddenly your glue doesn't work again.
[09:25.400 --> 09:32.480]  So it always becomes a maintenance nightmare to get the glue working.
[09:32.480 --> 09:38.600]  So if you ever came to the Plumber's Conference, the web front end that you first see, if you
[09:38.600 --> 09:44.960]  go to meet.lpc.events, is a piece of code that John Corbett wrote in Python and Django
[09:44.960 --> 09:49.680]  called the Linux Plumber's Front End, LPCFE.
[09:49.680 --> 09:54.640]  Oh, and I should add that these things here are clickable URLs.
[09:54.640 --> 09:59.320]  So if you go to the presentation and you want to know where it is, you can just hover over
[09:59.320 --> 10:01.800]  it and it will tell you.
[10:01.800 --> 10:07.480]  But obviously the green light is actually the way you log people onto BBB.
[10:07.480 --> 10:13.720]  So by replacing green light and big blue button, we've effectively taken on the job
[10:13.720 --> 10:20.160]  ourselves of doing integration, authentication, user management, and pretty much everything
[10:20.160 --> 10:21.160]  else.
[10:21.160 --> 10:26.600]  This is actually a major replacement of services on big blue button.
[10:26.600 --> 10:31.280]  And the reason we did this is because we needed an integration with our conference timetable.
[10:31.280 --> 10:35.080]  We just wanted the conference timetable to display on the first screen with a little
[10:35.080 --> 10:37.720]  button that said, click here to go to this session.
[10:37.720 --> 10:41.120]  It's very like you have in FOSDAM with the matrix chat.
[10:41.120 --> 10:45.280]  If you just click on the integration of the matrix chat, it brings up the live stream
[10:45.280 --> 10:47.440]  and you can see what's going on in the conference.
[10:47.440 --> 10:51.120]  We wanted it to work pretty much as it does here.
[10:51.120 --> 10:55.520]  And green light just wouldn't do that for us because green light was designed to be
[10:55.520 --> 10:58.240]  a teaching front end for a single classroom.
[10:58.240 --> 11:08.720]  It wasn't really thinking about multiple tracks, multiple track rooms, and timetable integration.
[11:08.720 --> 11:16.680]  So we also, by getting rid of all of the attendee credentials, we actually had to integrate
[11:16.680 --> 11:20.440]  with some sort of authentication server that could be distributed.
[11:20.440 --> 11:25.040]  So being kernel developers, we chose LDAT because it's the oldest one and it's the one
[11:25.040 --> 11:26.860]  everybody has the most trouble with.
[11:26.860 --> 11:30.920]  So we should be able to use it, right?
[11:30.920 --> 11:34.760]  The credentials were the email address that you used to register with the conference,
[11:34.760 --> 11:36.480]  which is easy.
[11:36.480 --> 11:41.280]  And the password was just the confirmation number because fortunately the Linux Foundation
[11:41.280 --> 11:45.600]  CVent system spits out these long confirmation numbers and they're all guaranteed to be
[11:45.600 --> 11:46.600]  unique.
[11:46.600 --> 11:49.160]  So that's obviously really easy.
[11:49.160 --> 11:51.680]  Problem, again, integration.
[11:51.680 --> 11:55.720]  The CVent system will not speak to anything outside CVent.
[11:55.720 --> 12:03.240]  So if you register for the conference and you don't automatically show up in the LDAT
[12:03.240 --> 12:08.320]  database, what has to happen is I have to go to the CVent website and export a spreadsheet
[12:08.320 --> 12:13.720]  of all of the users, figure out which are new ones, and then feed the details into the
[12:13.720 --> 12:18.680]  LDAT server, which, I mean, as an offline activity, it sounds really simple.
[12:18.680 --> 12:23.480]  It becomes really complicated when you keep getting people who don't register into the
[12:23.480 --> 12:28.640]  last minute of the conference, turn up halfway through a session, want to register, and then
[12:28.640 --> 12:31.720]  immediately be able to see what the hell's going on.
[12:31.720 --> 12:36.480]  And they have to wait, I don't know how long, for somebody to actually put their credentials
[12:36.480 --> 12:37.480]  in the database.
[12:37.480 --> 12:41.520]  And when you're running a conference, pretty much all of your attendees are running around
[12:41.520 --> 12:44.880]  – or all of your organizers are running around full-time.
[12:44.880 --> 12:47.480]  Nobody has time to tend the back end of the database.
[12:47.480 --> 12:51.960]  And that became a bottleneck for certain people.
[12:51.960 --> 12:58.640]  So my request of you is if you ever get into this situation, register early, please, don't
[12:58.640 --> 13:01.160]  turn up at the last minute, it's much easier.
[13:01.160 --> 13:08.120]  Then the next problem is that nobody is actually really used to interacting online.
[13:08.120 --> 13:11.520]  I mean, you think it's easy, you've seen Zoom calls, you will just turn up and you've
[13:11.520 --> 13:14.000]  got, you know, pictures of each other on it.
[13:14.000 --> 13:19.240]  If you try and do that with 800 people, Zoom doesn't really do autofocus very well.
[13:19.240 --> 13:21.760]  You can't see who's talking.
[13:21.760 --> 13:25.640]  And if you're just observing the conference, it just becomes a mass of faces and it's
[13:25.640 --> 13:28.680]  completely useless.
[13:28.680 --> 13:33.560]  So we obviously had standard issues like good webcam, good microphone.
[13:33.560 --> 13:39.200]  The number of people probably among you sitting in this room who think that the standard set
[13:39.200 --> 13:43.360]  of speakers on your Dell system when you just open the laptop will do when you're chatting
[13:43.360 --> 13:46.000]  to each other, believe me, it won't.
[13:46.000 --> 13:52.280]  The thing that audiences hate the most in online conferences is static background noise,
[13:52.280 --> 13:55.280]  indistinctness, being unable to get the point across.
[13:55.280 --> 14:00.680]  If you don't have in the attendees who are going to interact good AV equipment, this
[14:00.680 --> 14:05.560]  is the thing which actually makes the conference experience go down the fastest.
[14:05.560 --> 14:11.160]  This is about the only thing we got right because we gave a speaker gifts designed to
[14:11.160 --> 14:16.520]  be delivered to anywhere in the world a month before the conference, a complete gift pack
[14:16.520 --> 14:22.040]  of headset lights so you could actually see the face and a webcam you could put on the
[14:22.040 --> 14:26.480]  top of your camera, a 1080p webcam so we were sure that we'd actually have a good view of
[14:26.480 --> 14:27.640]  the speakers.
[14:27.640 --> 14:33.240]  We made sure that they got to all of our nearly 100 speakers in the conference and most of
[14:33.240 --> 14:37.680]  them didn't use them, which was a bit of a problem.
[14:37.680 --> 14:41.640]  We also found, as I've explained with Zoom, that raised hands don't work and having a
[14:41.640 --> 14:46.800]  load of people on video who you have no idea what's going on doesn't work either.
[14:46.800 --> 14:51.600]  The thing we pioneered for plumbers is called video muting, which is where we try and instruct
[14:51.600 --> 14:56.480]  the attendees to, if you're just watching, don't turn your camera on.
[14:56.480 --> 14:58.120]  Pretty much it's a blank screen.
[14:58.120 --> 15:02.000]  The only people who turn their camera on are people who are talking or people who are waiting
[15:02.000 --> 15:03.000]  to talk.
[15:03.000 --> 15:07.640]  It actually gives a good visual clue that somebody wants to interject at this point.
[15:07.640 --> 15:12.480]  We found it was actually a reasonably good way of facilitating interaction in an online
[15:12.480 --> 15:17.280]  conference and it certainly solves the Zoom confusion where you just can't figure out
[15:17.280 --> 15:22.520]  who's talking to whom on your screen of, and for the online conference we did have about
[15:22.520 --> 15:24.080]  900 attendees.
[15:24.080 --> 15:31.080]  You could imagine on a Zoom screen everybody would probably get about five pixels.
[15:31.080 --> 15:36.600]  In 2020, Big Blue Button comes with an internal chat system, which means anybody logged into
[15:36.600 --> 15:41.480]  Big Blue Button can pop out a chat tab at the side and they'll be able to chat with
[15:41.480 --> 15:44.400]  anybody who's logged into the conference.
[15:44.400 --> 15:49.080]  This is why, if you were doing streaming of the conference, you couldn't interact because
[15:49.080 --> 15:53.680]  that is a completely enclosed chat system.
[15:53.680 --> 15:58.760]  We realized that people wanted to have conversations outside the discussion rooms as well, things
[15:58.760 --> 16:00.400]  like hallway tracks.
[16:00.400 --> 16:05.040]  We actually adopted a, it's a sort of open source project called Rocket Chat.
[16:05.040 --> 16:09.800]  The core of the project is fully open sourced, but Rocket Chat themselves runs a more open
[16:09.800 --> 16:14.320]  core model where there are certain enterprise upsells and everything, but this was just
[16:14.320 --> 16:16.240]  the one we came across.
[16:16.240 --> 16:20.480]  The good thing for us is it doesn't require any integration because we'd already started
[16:20.480 --> 16:25.440]  to realize the more integrations you do, the more difficulty you have, the more strings
[16:25.440 --> 16:30.920]  and bailing wire stuff there is to go wrong at the crunch time when you put the conference
[16:30.920 --> 16:31.920]  on.
[16:31.920 --> 16:36.200]  So no integration looked like a good thing until we tried to run the conference.
[16:36.200 --> 16:41.280]  And then the problems were that chat was hard to read on the live stream because we're
[16:41.280 --> 16:44.640]  streaming it out pretty much through YouTube.
[16:44.640 --> 16:49.240]  We actually, Gilou Nadi wrote a streaming module for BVB and we just plugged it in as
[16:49.240 --> 16:53.400]  an additional attendee and what it saw is what we streamed to YouTube.
[16:53.400 --> 16:58.840]  And the problem is that YouTube does dynamic compression of most of the stuff while it's
[16:58.840 --> 16:59.840]  streaming.
[16:59.840 --> 17:04.560]  And if you have a low bandwidth connection, it will quite happily greek all of the chat
[17:04.560 --> 17:11.120]  that's going on and you just can't see what is going on on the chat, whereas if it's sort
[17:11.120 --> 17:16.480]  of having a bandwidth problem with faces and you're just looking at lips moving, you can
[17:16.480 --> 17:23.080]  stand losing quite a few pixels, but if you're trying to follow chat, it's just impossible.
[17:23.080 --> 17:27.800]  And obviously streamers didn't have any credentials to the Rocket Chat anyway because we based
[17:27.800 --> 17:31.840]  on our LDAP, if you're not registered to the conference, you have no chat credentials.
[17:31.840 --> 17:35.160]  So they couldn't join the hallway track either, which they didn't like.
[17:35.160 --> 17:36.800]  This was the problem.
[17:36.800 --> 17:40.560]  And the third problem was we had to buy an enterprise license from Rocket Chat actually
[17:40.560 --> 17:46.400]  to get some of the notifications that we used for integration to work.
[17:46.400 --> 17:50.400]  And the two chat systems were completely and fully disconnected.
[17:50.400 --> 17:55.440]  So you couldn't use your Rocket Chat login to look at the BBB chat and vice versa, which
[17:55.440 --> 18:00.000]  also was a bit balkanizing and a bit annoying for people.
[18:00.000 --> 18:04.320]  And so you can see that the thing I initially portrayed as a huge advantage because we didn't
[18:04.320 --> 18:08.920]  have to worry about the integration was the thing that everybody found the biggest disadvantage
[18:08.920 --> 18:11.680]  about the conference because it wasn't integrated.
[18:11.680 --> 18:16.480]  So you always get this pressure of, and you know as conference organizers, the more integration
[18:16.480 --> 18:20.200]  you do, the more chance there is that something will go wrong.
[18:20.200 --> 18:24.120]  But if you don't do enough integration, your audience don't like it.
[18:24.120 --> 18:28.560]  So you have to get the amount of integration right.
[18:28.560 --> 18:33.520]  And for several conferences, we actually struggled to do this.
[18:33.520 --> 18:40.800]  Every registered attendee got a matrix ID, and since we used email and confirmation number
[18:40.800 --> 18:45.080]  to log in, we figured we'd just use email address matrix as well.
[18:45.080 --> 18:46.800]  But there's a slight problem here.
[18:46.800 --> 18:48.760]  Emails contain an at sign.
[18:48.760 --> 18:51.920]  Matrix does not like additional at signs in your ID.
[18:51.920 --> 18:56.120]  So we had to replace the at sign with a full stop, which I thought, oh, it could be easy
[18:56.120 --> 18:57.280]  for our attendees.
[18:57.280 --> 18:58.280]  They're all computer geeks.
[18:58.280 --> 18:59.280]  They'll get this.
[18:59.280 --> 19:03.600]  They'll know email address for one server, email address but with at sign substituted
[19:03.600 --> 19:07.320]  with dot for another server, complete disaster.
[19:07.320 --> 19:11.480]  You know, they just couldn't figure out which ID correctly to use for which.
[19:11.480 --> 19:17.200]  Eventually we got the BBB server to use either form of the address.
[19:17.200 --> 19:21.440]  And we actually got the matrix server to translate using a JavaScript widget.
[19:21.440 --> 19:24.520]  If they typed in the email address, it would just fill in the dot and everything would
[19:24.520 --> 19:25.520]  just work.
[19:25.520 --> 19:31.920]  We tried it without doing that for a couple of days and got a bucketful of complaints.
[19:31.920 --> 19:40.160]  I should also add that when we chose Rocket Chat in 2020, I wasn't very happy about the
[19:40.160 --> 19:45.080]  decision because I was a fledgling matrix user, and I was sure that matrix could handle
[19:45.080 --> 19:49.780]  the conference, and I thought it would actually be a good idea to do this.
[19:49.780 --> 19:54.200]  So I advocated even in 2020 that we should be using matrix.
[19:54.200 --> 20:00.480]  And finally, after all the difficulty with Rocket Chat in 2021, I got my wish.
[20:00.480 --> 20:06.320]  So they told me, we're using matrix and you're in charge, which was fun until the first day
[20:06.320 --> 20:13.000]  came along and the matrix server with 900 people on it just fell over.
[20:13.000 --> 20:14.680]  And we had actually done scaling testing.
[20:14.680 --> 20:15.880]  I mean, I'm not naive.
[20:15.880 --> 20:21.360]  I know things can go wrong at scale that you don't see in your own lab.
[20:21.360 --> 20:26.920]  So we'd invited everybody to come along for a town hall, but only 100 people showed up.
[20:26.920 --> 20:30.560]  And it turned out that most of the scaling problems we had were somewhere between 100
[20:30.560 --> 20:31.640]  and 900.
[20:31.640 --> 20:37.960]  So we hadn't hit any of them in our little test.
[20:37.960 --> 20:41.040]  And I'd also tried to scale up the matrix server.
[20:41.040 --> 20:48.520]  So our massive 16 CPU system that we were paying about $100 a month for, when you ran
[20:48.520 --> 20:53.320]  top on it was only using one CPU because I hadn't realized that matrix Synapse, which
[20:53.320 --> 20:56.360]  was the server we chose, was single-threaded.
[20:56.360 --> 21:01.600]  I mean, who in 2021 writes a single-threaded web application?
[21:01.600 --> 21:07.360]  But the matrix people thread matrix by using the back-end web server through a sequence
[21:07.360 --> 21:11.800]  of proxies to actually do this, but that is not the default configuration.
[21:11.800 --> 21:17.520]  And so on the Monday of the conference, I was left there in my own little basement sweating
[21:17.520 --> 21:20.240]  away going, how the hell do I scale this thing?
[21:20.240 --> 21:25.640]  But fortunately, several people who attended FOSDEM, which did use matrix before we did,
[21:25.640 --> 21:26.640]  also ran into this problem.
[21:26.640 --> 21:31.200]  And they were able to send me email saying, yeah, we've got this guy you can talk to.
[21:31.200 --> 21:32.520]  He'll teach you how to do it.
[21:32.520 --> 21:39.440]  So fortunately, within about four hours of discovering the problem and realizing it didn't
[21:39.440 --> 21:43.160]  scale, we had a roughly scaling matrix server working.
[21:43.160 --> 21:47.240]  And I managed to actually complete the scaling by the end of the first day.
[21:47.240 --> 21:51.560]  So for the second day, matrix was actually fully functional, fortunately, since it was
[21:51.560 --> 21:54.240]  my little baby.
[21:54.240 --> 21:58.760]  And I actually did a little blog post about how we got this to work, just so if anybody
[21:58.760 --> 22:02.840]  else ever runs into the problems, there are online resources to go to to get yourselves
[22:02.840 --> 22:05.560]  out of it.
[22:05.560 --> 22:10.200]  One of the good things about using matrix was that even the free tier attendees, the
[22:10.200 --> 22:14.240]  streaming tier attendees, could now actually interact by chat.
[22:14.240 --> 22:19.120]  So anybody with their own matrix ID could log on to any of our matrix rooms because they're
[22:19.120 --> 22:23.280]  all public, in the same way that all the FOSDEM rooms are public.
[22:23.280 --> 22:28.880]  Free attendees still got their own separated matrix ID, but that you only got logged on
[22:28.880 --> 22:34.280]  to the matrix ID if you actually popped up the side chat panel.
[22:34.280 --> 22:39.280]  You could also use your own separated matrix thing and use your own matrix ID as well if
[22:39.280 --> 22:41.480]  you wanted to.
[22:41.480 --> 22:47.760]  And it just depended how you wanted to interact with the conference.
[22:47.760 --> 22:53.640]  Cee Lunadi was the one who actually modified the BVB server to, instead of opening its
[22:53.640 --> 22:59.520]  own chat tab, we just stole the iframe concept it uses for the blackboard and we did a riot
[22:59.520 --> 23:04.240]  embedded which is done by some, the URL to the GitHub account is there.
[23:04.240 --> 23:10.560]  The integration, the riot embedded is not, it's not as good as running the electron type
[23:10.560 --> 23:16.480]  application, but trying to embed an electron application into a web form is just a nightmare
[23:16.480 --> 23:17.880]  and doesn't work.
[23:17.880 --> 23:25.400]  So it's a very cut down version, all it really does is text pictures and reactions.
[23:25.400 --> 23:29.440]  It doesn't do any of these sort of complicated threading or replies or anything, but that's
[23:29.440 --> 23:32.840]  pretty much good enough for conference attendees.
[23:32.840 --> 23:37.280]  But you can see the primrose path we're threading down because this is yet another integration
[23:37.280 --> 23:45.520]  into matrix, in BVB and matrix that is not done by either upstream.
[23:45.520 --> 23:53.880]  But it was really useful, so the streamers could see the chat, if it was greaked on their
[23:53.880 --> 23:58.880]  streaming screen, they could actually use a matrix client and still view the chat in
[23:58.880 --> 24:04.040]  real time without having to worry about video compression and reduction.
[24:04.040 --> 24:10.120]  And so we thought after we'd done this successfully for two years now, especially for one year
[24:10.120 --> 24:14.440]  with rocket chat, one year with matrix, it wouldn't be too difficult to move over to
[24:14.440 --> 24:21.760]  hybrid, which is probably really the point of this talk.
[24:21.760 --> 24:24.040]  We already had all the remote infrastructure.
[24:24.040 --> 24:28.680]  By the way, to run this style of remote infrastructure for a conference size of plumbers, so we're
[24:28.680 --> 24:34.000]  talking about 900 people and we assume that for the in-person one, we'd probably be about
[24:34.000 --> 24:41.040]  400 in-person, 500 online, it will cost you somewhere between $3,000 and $5,000 to actually
[24:41.040 --> 24:42.040]  set up and run.
[24:42.040 --> 24:46.240]  That's not cheap to do the hosting of all of this stuff, especially because we had to
[24:46.240 --> 24:52.480]  host at scale, we needed one BVB server per each track room, we had about six track rooms,
[24:52.480 --> 24:59.840]  we had a few spare servers left over for hack rooms and we had one for the hallway track.
[24:59.840 --> 25:05.480]  But the way we looked through this, we already had all of the AV necessary for actually just,
[25:05.480 --> 25:10.240]  this is the naive thought, which is where kernel developers go, we got AV in the room,
[25:10.240 --> 25:14.640]  it's got two cameras and it's got a load of microphones, the room is just another single
[25:14.640 --> 25:21.240]  person attending the BVB chat, it'll be easy to do it like that, the naive way of thinking.
[25:21.240 --> 25:27.120]  But it really, really is not that simple trying to get a hybrid conference running.
[25:27.120 --> 25:33.880]  The significant problem is that you have to integrate with in-room AV and in-room AV,
[25:33.880 --> 25:39.480]  well, okay, so at ULB, you control it, but we were doing it in hotels.
[25:39.480 --> 25:43.760]  ULBs have their own local AV teams, if you touch a piece of their equipment, they tend
[25:43.760 --> 25:45.920]  to go screaming to management.
[25:45.920 --> 25:51.000]  In order to get the integration done, you can't do it yourself as the conference organizers,
[25:51.000 --> 25:57.640]  the AV people, the audio people have to do it and it's surprising the number of audio
[25:57.640 --> 26:02.520]  people who sort of sit behind those switcher boards at the back of conferences who've never
[26:02.520 --> 26:08.440]  even seen a sort of online hybrid conference and have no idea how to set this up.
[26:08.440 --> 26:13.000]  So just plonking this sort of video display down in front of them and saying, hey, now
[26:13.000 --> 26:18.520]  you're in charge of audio and video and we just want the conference to work, it wasn't
[26:18.520 --> 26:22.680]  actually a winning strategy.
[26:22.680 --> 26:27.880]  So we discovered rapidly and unfortunately it was pretty much on the day of the conference
[26:27.880 --> 26:32.840]  that we had to train the AV people and have to use all of this stuff.
[26:32.840 --> 26:39.160]  And we were actually quite fortunate because we'd ordered the video-based AV, we actually
[26:39.160 --> 26:41.360]  had two sets of AV people.
[26:41.360 --> 26:46.240]  So the hotel usually supplies audio people to plug in your projectors and your microphones
[26:46.240 --> 26:47.960]  into the hotel's audio.
[26:47.960 --> 26:52.840]  And then we had a separate AV company whose job was to manage the cameras and sort of
[26:52.840 --> 26:55.720]  cut the whole thing together and do the live streaming.
[26:55.720 --> 27:01.960]  So we actually had two AV teams effectively and we were assuming that we could just co-opt
[27:01.960 --> 27:06.560]  the AV team who was doing the live streaming because live streaming, running live streaming
[27:06.560 --> 27:10.200]  is not much different than running an online conference, right?
[27:10.200 --> 27:12.200]  Wrong.
[27:12.200 --> 27:18.080]  So we held meetings beforehand with them and we talked to them about it.
[27:18.080 --> 27:22.400]  We couldn't hold meetings with the hotel audio guys but we assumed it would just be a plug
[27:22.400 --> 27:25.160]  for them, which it wasn't.
[27:25.160 --> 27:31.320]  And nobody really anticipated the issues that we actually had.
[27:31.320 --> 27:37.040]  The AV setup of remote attendees is not obvious to somebody who's used to dealing with throwing
[27:37.040 --> 27:42.560]  microphones around a room because there is one extra wire that's coming off the internet
[27:42.560 --> 27:45.720]  over which a lot of attendee voices can come.
[27:45.720 --> 27:51.160]  And the first time they set up their AV patch panels, this wire just wasn't plugged in.
[27:51.160 --> 27:57.280]  The second time it was bypassing all of the audio routing which meant that the sound quality
[27:57.280 --> 28:01.560]  and the volume was chopping at high strength because it wasn't being attenuated through
[28:01.560 --> 28:02.560]  the usual filters.
[28:02.560 --> 28:07.920]  I mean, this is the 101, you know, how many screw-ups can you have?
[28:07.920 --> 28:12.200]  We basically did them all on the first day.
[28:12.200 --> 28:14.720]  So let's go through a list of the problems.
[28:14.720 --> 28:22.560]  Oh, I even forgot to mention that between 2001 and 2002, we decided to upgrade the whole
[28:22.560 --> 28:23.600]  of the infrastructure.
[28:23.600 --> 28:26.920]  We had specific things in BBB we wanted.
[28:26.920 --> 28:31.360]  After I told you we have two cameras, we have one at the back facing the speaker, one at
[28:31.360 --> 28:36.560]  the front facing the audience, the new version of BBB could actually handle multiple cameras
[28:36.560 --> 28:38.000]  with different views.
[28:38.000 --> 28:42.680]  So we figured that wouldn't it be really great if, you know, you want to see the audience
[28:42.680 --> 28:46.600]  so you just select the audience camera as a remote participant because you want to talk
[28:46.600 --> 28:50.600]  to them, or let's say you're a remote speaker, you want to see the audience, but the rest
[28:50.600 --> 28:54.040]  of the people just want to see you, we'll do this multiple camera selection.
[28:54.040 --> 28:58.400]  It was going to be the cornerstone of our implementation of this.
[28:58.400 --> 29:01.120]  Problem was, nobody told the AV guys.
[29:01.120 --> 29:06.000]  So the AV guys had come set up with a switcher for the two cameras, which is how they ran
[29:06.000 --> 29:10.640]  the streaming because obviously you can select one camera view or the other because the live
[29:10.640 --> 29:14.520]  streaming pretty much has one camera view out on it.
[29:14.520 --> 29:18.640]  And by the time we all sat together in the room, there was no way of unplugging this
[29:18.640 --> 29:23.440]  setup and we doing it such that we could plug in two video cameras because it all has to
[29:23.440 --> 29:27.560]  be done from the AV control deck and they physically didn't have another input because
[29:27.560 --> 29:29.920]  they were expecting to use a camera switch.
[29:29.920 --> 29:33.000]  So this is yet another screw up in planning.
[29:33.000 --> 29:39.520]  Third screw up was because we'd upgraded the new BBB, they'd redone the, and this user's
[29:39.520 --> 29:44.480]  node react as the frame backbone, they'd redone the way it was working.
[29:44.480 --> 29:47.360]  Riot embedded no longer worked as a pop out panel.
[29:47.360 --> 29:50.900]  We discovered this fortunately four days before the conference.
[29:50.900 --> 29:55.280]  So this was Guilinardi doing a very quick back end retrofit, trying to get the matrix
[29:55.280 --> 29:58.880]  chat that everybody was expecting to work to work.
[29:58.880 --> 30:02.960]  And fortunately by the time we did the conference, it was actually working, but this was yet
[30:02.960 --> 30:10.360]  another squeaky bum issue where we tried to do everything at the last minute.
[30:10.360 --> 30:15.000]  The first day we had a lot of rooms where the room couldn't hear the remote attendees.
[30:15.000 --> 30:20.800]  This is the wire problem from the remote or the remote attendees couldn't hear the room
[30:20.800 --> 30:22.680]  because it's a two wire problem.
[30:22.680 --> 30:27.040]  It's not just sound from the remote coming into the room, it's sound from the room going
[30:27.040 --> 30:28.840]  out to the remotes as well.
[30:28.840 --> 30:32.360]  So we had to have the sound correctly plugged into BBB.
[30:32.360 --> 30:35.320]  The streaming guys thought they were just plugging the sound into the streaming.
[30:35.320 --> 30:39.720]  So if you watch the live stream, it had the correct sound, but if you were on the BBB
[30:39.720 --> 30:43.480]  thing, it was completely silent because they hadn't plugged the sound in.
[30:43.480 --> 30:48.240]  It's just yet another screw up that we didn't actually think to test out.
[30:48.240 --> 30:54.080]  So this is how it looks in the sort of conference hall as we were going around it.
[30:54.080 --> 30:56.160]  So this is the panel you can see.
[30:56.160 --> 31:00.600]  This is what BBB looks like pretty much in a classroom situation.
[31:00.600 --> 31:06.580]  This is the presentation, which you can actually minimize if you click this button, but obviously
[31:06.580 --> 31:10.800]  we thought that the AV people might be able to click the minimize button when the conversation
[31:10.800 --> 31:12.760]  is going on and then click it back again.
[31:12.760 --> 31:14.360]  No, it didn't really work.
[31:14.360 --> 31:15.960]  They didn't really want to do this.
[31:15.960 --> 31:18.040]  So we just kept everything up.
[31:18.040 --> 31:22.360]  This is the attendees panel, which is all scrolling, which is nice, but if you think
[31:22.360 --> 31:25.720]  about it, that's not really what you want to see in the conference.
[31:25.720 --> 31:30.960]  The chat panel, if you click this, will pop out and come here, and the rest of this will
[31:30.960 --> 31:34.380]  really shrink, which is also not really what the attendees did.
[31:34.380 --> 31:38.120]  So what we should have done is have the chat panel down here and get rid of the attendee
[31:38.120 --> 31:41.120]  list or make it pop out only.
[31:41.120 --> 31:44.920]  So we put this on our to-do list for next time because it's sort of a nice to have,
[31:44.920 --> 31:48.640]  and it wasn't essential to getting the conference working.
[31:48.640 --> 31:52.200]  This is actually an excerpt from the RISC-5 microconference.
[31:52.200 --> 31:55.600]  So Atish Patra is actually asking a remote question.
[31:55.600 --> 31:59.520]  You can see that he's asking it of a member of the audience, so we've switched to the
[31:59.520 --> 32:03.320]  audience camera view, because ideally there would have been two cameras here, one of the
[32:03.320 --> 32:06.560]  audience, one of the speaker, and he could have switched between them, but we screwed
[32:06.560 --> 32:11.160]  up and didn't get that working in time.
[32:11.160 --> 32:18.840]  The tab also includes shared notes, which goes to a back end on ether pads, so the remote
[32:18.840 --> 32:24.480]  audience can also see the shared notes, and they can also click on the pop-up, and these
[32:24.480 --> 32:30.120]  buttons here actually allow you to mute people who are not supposed to be speaking when they
[32:30.120 --> 32:34.360]  are, because in a remote conference you get lots of people who forget to mute their microphone
[32:34.360 --> 32:39.560]  and then have a conversation with their wife or their significant other.
[32:39.560 --> 32:41.560]  This happens quite often.
[32:41.560 --> 32:46.120]  The way BBB is set up is that anybody can mute anybody, but we thought that might be
[32:46.120 --> 32:50.040]  a bad idea, because it's cantankerous kernel developers.
[32:50.040 --> 32:52.080]  You never quite know who they'll mute.
[32:52.080 --> 32:54.840]  Once you've muted somebody, you can't unmute them.
[32:54.840 --> 32:59.360]  They can only unmute themselves, because this is a sort of, you can't be unmuting people
[32:59.360 --> 33:00.960]  while they're going to the bathroom or something.
[33:00.960 --> 33:04.560]  It wouldn't be good on the conference.
[33:04.560 --> 33:10.000]  We figured this might be a problem, so we restricted this functionality to administrators
[33:10.000 --> 33:14.640]  only, and then nobody could figure out who the administrators were, so it became a bit
[33:14.640 --> 33:17.160]  of a problem.
[33:17.160 --> 33:20.280]  The video overlays work quite well.
[33:20.280 --> 33:26.600]  This is pretty much a blow-up of a 1080p thing, and you can see the greaking happening in
[33:26.600 --> 33:27.600]  the room.
[33:27.600 --> 33:32.080]  It's very difficult to make out individual attendees, but the sort of remote participants
[33:32.080 --> 33:34.760]  were reasonably visible to most people.
[33:34.760 --> 33:41.160]  BBB, and especially, you can actually see the way it works with the video-muting idea.
[33:41.160 --> 33:44.960]  It's really all you see are the people participating.
[33:44.960 --> 33:52.560]  One of the other things that we'd actually tried to do in the room is the camera at the
[33:52.560 --> 33:59.040]  front facing the back actually had a pan and zoom facility, which the AV operators assured
[33:59.040 --> 34:04.440]  us would allow them to zoom in on whoever was speaking, but in the event, what we discovered
[34:04.440 --> 34:10.040]  is that only two of the AV operators out of our six rooms knew how to use this.
[34:10.040 --> 34:13.360]  We did get panning and zooming in some of the rooms, but not in all of them.
[34:13.360 --> 34:15.720]  Again, this goes back to preparation and training.
[34:15.720 --> 34:24.240]  We will do better next time.
[34:24.240 --> 34:27.160]  It actually took us quite a while to get the audio routing right.
[34:27.160 --> 34:32.240]  We think that plugging two wires into an AV system wouldn't be that difficult, but in
[34:32.240 --> 34:35.720]  fact, it really was.
[34:35.720 --> 34:41.800]  We finally figured out that what you had to say to the audio guy is, here's a wire from
[34:41.800 --> 34:44.200]  what's effectively a microphone on the computer.
[34:44.200 --> 34:46.120]  The video guy will give it to you.
[34:46.120 --> 34:49.560]  You just plug it into your fader panel and remember you have it.
[34:49.560 --> 34:56.840]  You give him a wire for the AV stream that also has to go into the BBB thing.
[34:56.840 --> 35:02.440]  Obviously we got that to work, but obviously we had other problems.
[35:02.440 --> 35:06.640]  One of the other things that we'd really thought about was, when we get into the room, we'd
[35:06.640 --> 35:11.160]  actually like to have the chat and all the remote attendees on one monitor, and then
[35:11.160 --> 35:13.600]  we'd like to have the presentations on another.
[35:13.600 --> 35:18.800]  We thought we'll have a two projector setup, which sounds fine until you realize that we
[35:18.800 --> 35:25.200]  have rooms about as big as this, and the two projectors were right over there and right
[35:25.200 --> 35:26.800]  over there.
[35:26.800 --> 35:30.480]  Somebody sitting over there can't see what's on that projector.
[35:30.480 --> 35:34.320]  Somebody sitting over there can't see what's on that projector.
[35:34.320 --> 35:38.040]  The only way we could make it work was to display the same image on both projectors,
[35:38.040 --> 35:41.800]  which meant that two projectors was a complete waste of money, and we should have gone for
[35:41.800 --> 35:44.400]  a single central one as well.
[35:44.400 --> 35:53.680]  Next year we will either try with one or three, just to see if we can actually get this right.
[35:53.680 --> 35:57.240]  Obviously as I said, the two-camera handling, we thought would be the centerpiece of the
[35:57.240 --> 35:58.240]  technology.
[35:58.240 --> 36:03.560]  The reason why we'd upgraded BBB and caused all of the React JScript failures on the Matrix
[36:03.560 --> 36:07.280]  back end was to get this facility that we couldn't actually use.
[36:07.280 --> 36:11.680]  That was not only sleepless nights for one person for a week to try and get it all to
[36:11.680 --> 36:16.040]  work, but complete screw-up on the day when everything we thought we'd bought with that
[36:16.040 --> 36:21.800]  upgrade didn't function.
[36:21.800 --> 36:28.720]  The other problem we had was, because the AV people are actually doing the live streaming
[36:28.720 --> 36:34.400]  and because of the way BBB works, there is no choice other than to actually run the Master
[36:34.400 --> 36:42.320]  BB session that is showing up on both projectors on their AVMux console.
[36:42.320 --> 36:51.720]  I had thought this might be a problem as well, so BBB is very heavily React JavaScript
[36:51.720 --> 36:57.520]  based and certain browsers handle it better than other browsers.
[36:57.520 --> 37:01.880]  It works with Firefox quite well now, but it is mostly optimized for Chromium.
[37:01.880 --> 37:05.480]  So I had actually already asked them, God, if we have to do this, what is the built-in
[37:05.480 --> 37:07.440]  browser of this Tmux thing?
[37:07.440 --> 37:12.360]  They said, oh, it's Chrome, and I thought, oh, God, it's going to work.
[37:12.360 --> 37:20.960]  But the problem was that the Mux platform might have used Chromium as an internal thing,
[37:20.960 --> 37:22.720]  but Chromium is open source.
[37:22.720 --> 37:27.240]  So the guys who built the Mux platform had just basically taken WebKit, gutted some
[37:27.240 --> 37:30.120]  of the JavaScripts and hoped that that would be OK.
[37:30.120 --> 37:34.920]  What it really meant was the threading functions of JavaScript didn't work very well.
[37:34.920 --> 37:39.880]  The result was that it kept dropping out of the conference, and obviously, since this
[37:39.880 --> 37:46.400]  is running the AV conference, when it drops out, the in-room chat and the remote participants
[37:46.400 --> 37:48.080]  fully separate.
[37:48.080 --> 37:52.160]  The problem is there's no indication that this has happened from the console unless you
[37:52.160 --> 37:56.880]  happen to be watching very carefully, because all you see is that a microphone disappears
[37:56.880 --> 38:01.280]  from that little icon in the top left that says, hey, you're the room, and suddenly your
[38:01.280 --> 38:02.680]  microphone's gone.
[38:02.680 --> 38:06.120]  That's the only indication the AV people had they dropped out.
[38:06.120 --> 38:10.320]  Trying to train them to notice this and reconnect was a bit of a problem.
[38:10.320 --> 38:15.960]  So we do have automated reconnections on the back end, but we will try and integrate into
[38:15.960 --> 38:20.240]  BVV, but when we raised it on the list, they looked at us and said, well, how come you
[38:20.240 --> 38:23.000]  have this problem because nobody else has this problem?
[38:23.000 --> 38:27.920]  If BVV kept on dropping people out of the conversation as a matter of course, people
[38:27.920 --> 38:29.420]  would have complained.
[38:29.420 --> 38:34.320]  This was a specific interaction due to the cut-down browser in the T-Mux interface that
[38:34.320 --> 38:35.920]  nobody had anticipated.
[38:35.920 --> 38:41.600]  Again, another screw up from not actually trying it beforehand.
[38:41.600 --> 38:45.480]  So as I said, rejoin was manual, assuming the AV guy noticed.
[38:45.480 --> 38:49.440]  One of the good things we did have was interim shepherds who were committee members.
[38:49.440 --> 38:52.960]  So they could actually watch the on-screen panel.
[38:52.960 --> 38:57.560]  Every time the icon disappeared, I or one of the other shepherds could run to the back
[38:57.560 --> 39:02.240]  of the room and pat the AV guy on the shoulder and say, you need to rejoin, please do this.
[39:02.240 --> 39:10.000]  So we could make it work with a lot of manual fussing behind the scenes, but it wasn't easy.
[39:10.000 --> 39:16.760]  So one of the other things that we realized is that the hindsight is 20-20.
[39:16.760 --> 39:21.000]  They might want to run it from the AV console, but they don't have to run the whole thing.
[39:21.000 --> 39:26.040]  What we could have done is have a separate laptop that actually ran all of the AV connections
[39:26.040 --> 39:29.520]  under the charge of, I won't say somebody competent because it would have been one of
[39:29.520 --> 39:33.800]  me, the kernel developers or somebody, but somebody who would pay more attention.
[39:33.800 --> 39:37.800]  And then what we could have done for the V-Mux guys was to actually give them a VNC session
[39:37.800 --> 39:43.960]  or something like that into the panels that they were projecting over the streaming and
[39:43.960 --> 39:45.200]  everything else.
[39:45.200 --> 39:49.080]  And that way we would have had much more control in the room and we'd have been running a
[39:49.080 --> 39:53.320]  full-fledged browser that wouldn't have had most of the connection problems that we had.
[39:53.320 --> 39:58.360]  But obviously trying to sort that out on the day was pretty much impossible.
[39:58.360 --> 40:03.600]  Particularly unfortunate is this AV-Mux platform is sort of Windows-based and Windows does
[40:03.600 --> 40:08.840]  not do VNC very well and trying to get it to work with RDSP, which is the Windows native
[40:08.840 --> 40:11.000]  thing, didn't work very well.
[40:11.000 --> 40:15.440]  And we just, it was, we were in Dublin, we didn't have any equipment, we had to organize
[40:15.440 --> 40:19.720]  the parties and deal with online conferences, we just couldn't get it to work.
[40:19.720 --> 40:26.160]  So we do know that for the future we are going to try and do a shareable console.
[40:26.160 --> 40:29.240]  And I need to leave time for questions.
[40:29.240 --> 40:32.280]  I have 10 minutes left.
[40:32.280 --> 40:34.120]  We had a load of other problems as well.
[40:34.120 --> 40:41.360]  As you can see, this, so I'm, in order to try and tell you all the difficulty about
[40:41.360 --> 40:44.560]  doing this, I'm talking up the problems.
[40:44.560 --> 40:48.980]  From the audience's point of view, this conference was actually very successful.
[40:48.980 --> 40:54.080]  We got plaudits from most of the attendees that the interaction worked as they expected.
[40:54.080 --> 40:58.800]  They could have their one-on-one conversations, including between remote and local people,
[40:58.800 --> 41:00.160]  remote and remote people.
[41:00.160 --> 41:04.840]  So everything appeared to just work for them, modulo, you know, the problems of not being
[41:04.840 --> 41:09.000]  able to hear the room and having to rejoin and having to tell people they lost AV and
[41:09.000 --> 41:10.280]  they needed to rejoin.
[41:10.280 --> 41:11.800]  They forgave us for all of that.
[41:11.800 --> 41:16.520]  So it was actually highly rated as an incredibly successful conference, but it was a bit like
[41:16.520 --> 41:21.240]  a swan with that, you know, serene sort of thing swimming over the surface and a load
[41:21.240 --> 41:25.080]  of committee members frantically paddling underneath to try and get all of this to stay
[41:25.080 --> 41:28.880]  up and running.
[41:28.880 --> 41:34.480]  Our other big screw up was that we had a committee of 10 people, but because this was pandemic,
[41:34.480 --> 41:38.960]  not all of the committee could turn up in Dublin, and by the time we got to Dublin,
[41:38.960 --> 41:43.680]  we only had two local people who actually understood the system and could deal with
[41:43.680 --> 41:44.680]  it.
[41:44.680 --> 41:48.320]  Surprisingly enough, the two people who are actually standing up on the platform now taking
[41:48.320 --> 41:50.280]  the blame for all of this.
[41:50.280 --> 41:57.480]  But believe me, in a conference with six tracks and AV guys who don't know what they're doing,
[41:57.480 --> 42:01.000]  if you want a lot of exercise, please do it with two people.
[42:01.000 --> 42:06.760]  But if you want to actually make it work seamlessly and vaguely competently, you need one person
[42:06.760 --> 42:09.160]  who understands the setup in each room.
[42:09.160 --> 42:13.840]  So next time we will try and do it with a minimum number of people as we have same number
[42:13.840 --> 42:14.840]  of track rooms.
[42:14.840 --> 42:16.480]  Again, it's not rocket science.
[42:16.480 --> 42:19.760]  It's perhaps something we should have seen ahead of time, but we thought it would be
[42:19.760 --> 42:23.760]  easy and it wasn't.
[42:23.760 --> 42:29.360]  So to leave a bit of time for questions, if you have them, running conferences is actually
[42:29.360 --> 42:30.360]  hard.
[42:30.360 --> 42:33.240]  So just remember that when you go home from here.
[42:33.240 --> 42:38.360]  And volunteering for a conference is actually an act of public self-sacrifice that more
[42:38.360 --> 42:44.640]  of you should consider, because honestly, you will find out that it is really hard.
[42:44.640 --> 42:49.480]  Running online conferences is harder, especially if you try and do it yourselves, because there
[42:49.480 --> 42:51.400]  is actually a lot more to do.
[42:51.400 --> 42:55.440]  It is actually more difficult to run an online conference than it is to run an in-person
[42:55.440 --> 42:59.760]  conference paradoxically.
[42:59.760 --> 43:02.400]  And this is mainly to do with the integration issues.
[43:02.400 --> 43:05.720]  Everything that doesn't integrate well is either your fault or you're responsible for
[43:05.720 --> 43:07.080]  integrating it.
[43:07.080 --> 43:11.600]  And integrating web stuff, even in this magic day and age of sort of, you know, web scripting
[43:11.600 --> 43:16.360]  and operators and everything else, really doesn't work that well.
[43:16.360 --> 43:19.880]  And hybrid is really only for masochists.
[43:19.880 --> 43:24.360]  Only do a hybrid conference if you want to give yourself an untold amount of pain that
[43:24.360 --> 43:29.640]  you can then come and complain to an audience like you about later.
[43:29.640 --> 43:34.120]  But if you really want to get it right, the key is training your AV people.
[43:34.120 --> 43:38.600]  Do not wait like we did to turn up on the day and say it will be easy.
[43:38.600 --> 43:42.800]  You need a couple of training sessions a few months beforehand to try and get all of the
[43:42.800 --> 43:45.120]  kinks sorted out.
[43:45.120 --> 43:48.600]  And having meetings and discussing it doesn't cut it.
[43:48.600 --> 43:54.320]  You need the AV people to set it all up and preferably have a few people fly out to see
[43:54.320 --> 44:01.880]  the setup, work with a few remote people so you actually check it out in all of its situations.
[44:01.880 --> 44:04.280]  And obviously, you need to do this in advance.
[44:04.280 --> 44:07.360]  You don't do it on the day of the conference, which is what we tried to do.
[44:07.360 --> 44:09.440]  Oh, well, lesson learned.
[44:09.440 --> 44:15.360]  So with that, I think I have about five minutes for questions.
[44:15.360 --> 44:18.880]  This was actually all done in open source using HTML and CSS.
[44:18.880 --> 44:23.440]  That does make me a web developer, proud of it now, finally.
[44:23.440 --> 44:28.400]  Obviously you can see that current developers don't quite do web as well as web developers.
[44:28.400 --> 44:32.640]  So if any web developers would like to join our program committee and become jointly responsible
[44:32.640 --> 44:35.920]  for next year's disaster, please come forward.
[44:35.920 --> 44:40.640]  The presentation is all online, and I've also put it up on a link in PentaBath.
[44:40.640 --> 44:42.680]  So you should just be able to click on it.
[44:42.680 --> 44:45.760]  All of the links in the presentation that are highlighted are clickable if you want to
[44:45.760 --> 44:47.440]  see the URLs.
[44:47.440 --> 44:49.200]  And with that, I'll just say thank you.
[44:49.200 --> 44:51.120]  And Mike and I can now answer questions.
[44:51.120 --> 44:52.120]  Oops.
[44:52.120 --> 44:53.120]  You have Mike.
[44:53.120 --> 45:01.120]  Okay, we'll share this, Mike.
[45:01.120 --> 45:06.080]  Any questions?
[45:06.080 --> 45:14.080]  Please speak loudly into the microphone.
[45:14.080 --> 45:20.240]  Yes, thank you.
[45:20.240 --> 45:25.720]  From what you said about the AV setups in the hotels, were there any discussion to just
[45:25.720 --> 45:29.880]  ask the hotels to, you know, move their stuff and have your own stuff brought in?
[45:29.880 --> 45:35.440]  Because to me, that seems after your talk to have been an easier solution to just say,
[45:35.440 --> 45:39.760]  you know, your AV guys and your AV is probably good, but we have really specialized needs.
[45:39.760 --> 45:41.680]  Could we just bring our own equipment in?
[45:41.680 --> 45:42.680]  Okay.
[45:42.680 --> 45:45.840]  So the question is, could we bring our own equipment in?
[45:45.840 --> 45:50.440]  And the answer is pretty much every hotel, no.
[45:50.440 --> 45:55.200]  Because remember they do the AV routing as a very specialized sort of adaption for the
[45:55.200 --> 45:56.400]  conference rooms.
[45:56.400 --> 46:00.360]  They have a lot of really expensive speakers, sometimes in the ceilings, sometimes in the
[46:00.360 --> 46:04.820]  back of the room, and they don't want any non-experts touching it.
[46:04.820 --> 46:09.840]  So what hotels usually have as a clause in their contract is a requirement that you use
[46:09.840 --> 46:11.440]  their own sound guys.
[46:11.440 --> 46:13.560]  And this clause is non-negotiable.
[46:13.560 --> 46:18.080]  We did try to negotiate with a couple of hotels where we said, because we had a few screw-ups
[46:18.080 --> 46:21.800]  even in the live streaming days where the hotel didn't do very well and we thought we
[46:21.800 --> 46:23.160]  could do better.
[46:23.160 --> 46:26.200]  But it's always come back to us as an absolute no.
[46:26.200 --> 46:31.400]  It's actually very lucky for the ULB guys are used to having students handle their equipment
[46:31.400 --> 46:34.960]  and don't seem to have as many hang-ups about it as hotels.
[46:34.960 --> 46:39.720]  So there are people here who can be trained on this equipment who can use it.
[46:39.720 --> 46:44.360]  But pretty much when you're running a conference in a hotel or in a professional conference
[46:44.360 --> 46:48.720]  hall that we've also been to, so a community convention center or something, you actually
[46:48.720 --> 46:50.480]  have to use their AV staff.
[46:50.480 --> 46:51.800]  You don't get a choice.
[46:51.800 --> 46:56.040]  And that means that you always have this extra component of people that you have to deal
[46:56.040 --> 46:57.040]  with.
[46:57.040 --> 47:00.680]  And they're usually people who don't really understand fully what your requirements are
[47:00.680 --> 47:03.920]  because it's very difficult to communicate to the head of time.
[47:03.920 --> 47:07.520]  And these people are usually the people that you don't get to talk to until you turn up
[47:07.520 --> 47:08.920]  at the conference.
[47:08.920 --> 47:13.120]  So when you have your intermediate AV guys like we had and you have your meeting with
[47:13.120 --> 47:17.600]  them, you always have to make sure that they have enough knowledge and insight that they
[47:17.600 --> 47:22.040]  can do the knowledge transfer to the sound guys who are going to be coming in on the
[47:22.040 --> 47:24.000]  day from the hotel.
[47:24.000 --> 47:28.160]  So yeah, it's a huge problem and there's almost nothing you can do to get out of it
[47:28.160 --> 47:33.160]  besides sort of having a very friendly university who will let you play with their equipment
[47:33.160 --> 47:39.320]  because pretty much nobody in the professional circuit will let you play with their equipment.
[47:39.320 --> 47:42.440]  I have one question from the chat.
[47:42.440 --> 47:46.160]  Was it worth it to record the audience?
[47:46.160 --> 47:50.080]  So the question was, was it worth it to record the audience?
[47:50.080 --> 47:55.280]  And the answer to that is actually very much yes, because remember in a micro conference
[47:55.280 --> 47:59.160]  we're stimulating discussions between arbitrary groups of people.
[47:59.160 --> 48:03.040]  So these would be people who are sitting in the audience physically versus people who
[48:03.040 --> 48:05.080]  are sort of remote.
[48:05.080 --> 48:10.160]  And in order to have a correct two-way conversation, you actually have to be able to see the person
[48:10.160 --> 48:11.440]  on the other end.
[48:11.440 --> 48:15.640]  And so the whole point of the audience camera, at least when it worked with the panning and
[48:15.640 --> 48:21.120]  zooming, was that if you were having a remote to local conversation with the audience, the
[48:21.120 --> 48:24.600]  remote people could actually see the local participant.
[48:24.600 --> 48:29.760]  And if we only had a camera facing the speaker, we'd either have to call every participant
[48:29.760 --> 48:35.880]  up to stand on the stage, which wastes time, or we'd just have to not have them be able
[48:35.880 --> 48:38.120]  to see the person they're talking to.
[48:38.120 --> 48:42.320]  And being able to see the person you're talking to is very important for interactions.
[48:42.320 --> 48:44.880]  There are lots of visual cues that people get wrong.
[48:44.880 --> 48:52.200]  So I think having an audience camera is actually one of the good benefits of, or at least one
[48:52.200 --> 48:56.160]  of the benefits the plumbers did and one of the good insights we had to try and get an
[48:56.160 --> 48:58.000]  interactive conference running.
[48:58.000 --> 49:02.640]  So I wouldn't go back and do no audience camera, no.
[49:02.640 --> 49:10.360]  Just wanted to get back to what you said about the ULB being willing to allow us to use their
[49:10.360 --> 49:11.360]  audio equipment.
[49:11.360 --> 49:14.480]  I just want to point out that this has a lot to do with the fact that we've been doing
[49:14.480 --> 49:17.160]  this for 20 years and they know us.
[49:17.160 --> 49:23.080]  It wasn't always the case, and if you're not supposed to, or you're not working with
[49:23.080 --> 49:26.080]  the ULB, this probably isn't the case.
[49:26.080 --> 49:29.600]  So could you repeat the, I got half the question, no.
[49:29.600 --> 49:35.440]  Yeah, it wasn't the question really so much as to point out that us being allowed to touch
[49:35.440 --> 49:39.680]  the audio equipment here, by the ULB has a lot to do with the fact that we've been doing
[49:39.680 --> 49:41.280]  this for 20 years and they know us.
[49:41.280 --> 49:45.840]  Okay, so the point being made is that ULB only allows the staff here to touch the equipment
[49:45.840 --> 49:48.920]  because they've been doing it a long time and they've built up trust.
[49:48.920 --> 49:53.840]  The way we work, we tend to go to a different hotel every time we hold plumbers in a different
[49:53.840 --> 49:55.000]  location.
[49:55.000 --> 50:01.760]  So pretty much even if we could get a trust to build up with a hotel chain, we wouldn't
[50:01.760 --> 50:04.240]  be in the hotel often enough to do it.
[50:04.240 --> 50:10.920]  But the important point to emphasize is that pretty much 95% of the time you will be forced
[50:10.920 --> 50:15.720]  to work with AV people who are supplied to you by whoever owns the equipment.
[50:15.720 --> 50:20.720]  You won't get a choice of who does it.
[50:20.720 --> 50:21.720]  Okay.
[50:21.720 --> 50:22.720]  Is that it?
[50:22.720 --> 50:23.720]  Okay.
[50:23.720 --> 50:48.280]  Well, thank you very much indeed.