Okay. Hello, everyone, and welcome to the last lightning talk of the conference. I hope
you've enjoyed yourselves. This will be the FOSDEM infrastructure review, same as every
year presented by Richard and Basti. So normally I do this thing, but Basti has been helping
a ton and the ball of spaghetti and spit and duct tape, which I left him. He turned it
to into something usable. So I'm just going to sit here on the side and I'm here on for
the Q&A. But for the rest, it's Basti and it's his first public talk for Realty, so
give him a big round of applause. Well, thank you. I hope I will not screw this one up.
Okay. So we'll have about 15 minutes and 10 minutes of talk and five minutes for Q&A,
and I hope it's somewhat interesting to you. First, the facts. The core infrastructure
hasn't changed that much since the last FOSDEM 2020. We're still running on its Cisco ASR
10K for routing, ACLs, NAT64, and DHCP. We have already reused several switches that
were already here from the last FOSDEM. They're owned by FOSDEM. These are Cisco C73750 switches.
We had our old servers, which are now turning 10 this year. They were still here and they
will be replaced next year. We have done, like all the years before, everything with
Prometheus, Loki, and Grafana for monitoring our infrastructure, because that's what helps
us running all the conference here. And we've built some public dashboards and we just put
it out to a VM outside of ULB because we were running out of bandwidth, like the years before,
and I'll come to that back later. We have a quite beefy video infrastructure. You might
have seen this one here. It's a video capturing device. It's called a video box here at FOSDEM.
It's all public. It's all open source, except one piece that's in there. You can find it
on GitHub if you try to build it yourself. Go ahead, just grab the GitHub repo and clone
it. These devices, there's two of them, one at the camera, one here for the presenters'
laptop. They sent their streams to a big render farm that we have over in the K building, where
every year our render farm is running on some laptops. So laptops sent the streams off to
the cloud from Hatsner and from there we just distribute it to the world. So everyone at
home can see the talks. We have some sort of semi-automated reverb on cutting process.
Those of you who have been talking here maybe have known S-Review for years. This is the
first time it's running on Kubernetes. So we are trying to go cloud native as well with
our infrastructure. Just to show how all has been held together. This is our video boxes.
I don't know if you can see it. We got those black magic encoders here that are turning
the signals that we get, like SDI, HDMI, into a useful signal that we can process with our
banana pie that we have in there. Everything is wired up to a dump switch here and then
we go out like here and have our own switching infrastructure inside those boxes. There's
some SSD below here where we just, in case of network failure, dump everything to the
SSD as well. So hopefully everything that was been talked about at the conference is
still captured and available in case of a network breakdown. Those boxes also have a
nice display for the speaker so we can see if everything is running or it's not running,
which makes it easy for people to operate these boxes. You don't have to be a video
pro. You just have to wire yourself up to the box. You see a nice Fostum logo and see
okay, everything is working and you're done and everything gets sound set out. This is
like how the video system is actually working. All this can be found on the GitHub. You don't
have to take screenshots for that. If you would like to see it, we will tear down this
room afterwards so everyone can have a look at the infrastructure we're using because
it's not being used after this talk. You see it's quite some interesting things to do.
This is the instructions that all our volunteers get when they wire up the whole buildings
here on one day on Friday. So they are not here but they should be given some round of
applause because they are volunteers that are doing really the hard work and building
up on one day the complete Fostum. So maybe it's time for round of applause for them.
So here we have the other thing. This is also on the GitHub where you can see where is something
coming from. We have the room sound system. This is what you're hearing me through and
we have a camera with audio gear and speaker laptops and it's all getting pushed down until
someone reaches your device down here. There's a ton of services processing it in between
and this is almost all done with open source software. Expect for the encoder that's running
in there which is from Blackmagic Design. So how is it processed? We have a rendering
farm. This is the laptops. It's 27 this year. For those of you who don't know those laptops
are being sold after Fostum. So if you want one you can grab one. This year they are already
gone but for next year maybe you want to have a cheap device. You can have them with everything
that's on them because we literally don't care for that. You can have it because after
things have been processed after the Fostum you can see it. We have some racks where we
just put them four wise and we have 27 of them. We have some switch infrastructure that
is used for processing all that stuff and this one's not running out of bandwidth but
we're coming back to what's running out of bandwidth. You might see this mess over here.
This is our internet and looks like every common internet on the planet and this is like our
safety net. We have a big box here where all the streams go and this will be sent out to
Bulgaria to the video team right after Fostum so we have a really off-site copy of everything.
So the challenges for this year. DNS64. All the years we've been running on bind 9 since
ages and we switched to Core DNS just like testing it on Sunday of Fostum 2020. We really
saw a significant reduction in CPU usage and that's why we stuck to Core DNS since then
and this year we also replaced the remaining bind installations that we handled for all
the internal DNS and all other recursive stuff that's been used here to provide you internet
access. Richie always used to give you some timelines and that's what I'm trying to do
as well. There were times when it was mentally challenging for people building up Fostum.
We got better by year by year by year by doing some sort of automation and getting people
used to know what to do and have everything set up before that. We installed routers.
You see that there's a slide. It's getting better year of the year. This year we had
like a very, we thought it would be okay from what we know. We just set it up in January
and everything worked. We came here on the 5th of January I think, put everything up
and it just worked which is great which gives you some sort of things not to care about
because there were other things to care about. The network to have it up and running here
took us a bit longer this year than the last years because we were playing around with
the second uplink that we got. We used to have one gigabit uplink. Last week we got
a 10 gigabit uplink and we thought okay, just enable that and play with it and it turned
out to be not that easy to getting up both of the BGP sessions running and doing it properly.
That's why it took us a bit longer this year. The monitoring was also one thing which really
helps us to understand if FOSDM is ready to go or if someone has to stay very, very late
here. The last years we've been very, very good at that. Basically in January everything
was done like the last of January but it's January. This time it was in the first half
of January everything was set up and was running and it worked and yeah, it was really great
because some people actually got some sleep at FOSDM, didn't need to stay here very long
because everything was pre-made and it just go and look at the dashboard, okay, this is
missing, this is missing and just okay, just have them all checked. The video build up
took a bit longer this year because of we getting old and rusty at that. Also, very
many new faces that have never built up such great conference. This is why we took us a
bit longer and the video team also, yeah, I think they got the least amount of sleep
of all of the stuff that was running the conference. This was the story so far. We closed FOSDM
2020, I was also there at 2020. 2020 was really one of the best ones we ever had from a technical
perspective. We had everything running via Ansible, just like one command and then wait
an hour till everything is deployed and you're gone, cool, have some beer, some mate in between
and everything was cool. Then we had this pandemic, just for me like a week after FOSDM
everything went down. You had FOSDM 2021 and 2022, there were no conference here at ULB
so we had no infrastructure to manage, it was quite okay. We had to do some other things
like most of you have learned that we have a big metrics installation to run and the
FOSDM conference and help you with communicating during the conference. Then there was this
bad thing that the maintainer of the infrastructure left FOSDM in between these years and so Richie
searched for someone who was dumb enough to do that, yeah, that's me. So this year we're
back again in persona, sorry, yeah thanks. So after two years we came looking for the
two machines after almost two years, like no one touched them, they rebooted one or two
times due to power outages in the server cabinet, but we had a working SSH key, we had tons
of updates to install after literally three years, I wonder nobody broke into the machines
because they were publicly exposed on the internet, but only SSH and I think a three
year old or three and a half year old Prometheus installation which was full of bugs, yeah.
We noticed that the battery pack of the rate controllers have been depleted, so this was
the only thing that actually happened in the three years, the batteries went to zero and
didn't set themselves on fire, so everything was okay, the machines worked, just a bit
of performance degradation, but everything seemed to be okay. Then we tried to run this
Ansible thing from the last years and three years later Ansible has done a lot of things
in the time and you want to use a current version of Ansible with that old stuff, you
end up like this, this is me, yeah, start from scratch or fix all the Ansible roads,
you can have a look at them, they're also on GitHub, so when we saw okay, how do we
do this and said okay, Ansible be gone, we just fix it after the FOSDM because we will
have to renew the service anyway and everything will change.
So the service timeline, we have them service life at the eighth of January, services, DNSX4
although at the mid of January, we had centralized all our locks, this was something Richie was
looking for since ages, that we had easy accessible lock files for everything that's running here
at the FOSDM, which is good that we had them because we could see things like oh, the internet
line that was proposed to be there actually came, we did, nobody told us but it came up,
you see that, we, thanks to the centralized logging, we were aware of things like that
and then we could go and fire up our BGP sessions.
Then two days later, we noticed okay, firing up the BGP sessions wasn't that a good idea
because we lost almost all connectivity, stop it says but I don't care, yeah, I just keep
talking, yeah, we lost all our connectivity and said okay, damn it, we're in some sort
of panic mode because the reason for looking at the service was like this bind security
issue that was been, I read the mail at the morning of January 28th and said okay, we
have to fix the bind installations and then you suddenly can't reach the service anymore
and say okay, are they already hacked or what's going on and doing some back and forth with
our centralized logging, you see that, this is Grafana Loki that we leveraged for that,
we were kind of like, yeah, it's been really nice to debug things like that, we also noticed
that there was an interface constantly flapping to our backbone which we also could fix within
that session and after that we said okay, there are some MTU problems, we have to restart
our BGP and so on and back and forth and then we finally agreed to just throw away the BGP
sessions, go with the 1 gigabit line and yesterday evening we switched to the 10 gigabit line
because we had the congested uplink like since 11 in the morning, so many people using too
much bandwidth and since yesterday evening everything is okay, it's better and we are
on the 10 gigabit link, due to the fact that there are not so many people here today, yesterday
there were quite a bit more, the link was not fully saturated but you can tell this
is the place where we could use some more bandwidth, it was like, I don't know, this
is usually time for something to eat but at 3.30 we could actually use something of the
new bandwidth that we had available, so if you want to look at all of the things, we
have a dashboard put out there publicly, if you want to have a look at the infrastructure
and the Ansible repo, that will be fixed to work with current Ansible versions within
the next few days, just clone our infrastructure, clone everything and if you have any questions
I'll be glad to take them, yeah, fire away.
As I don't see any questions then, we're about to tear down this room after this, so please
don't leave anything in here because it will be cleaned and everything will be torn out.
If anyone else has a question, just, there's one, we use, the question is, why do you use
laptops for rendering because they have a built-in USB called battery, so in place of the power
outage we can easily run with them, also they're very cheap for us, we can just use the computing
power and sell it at the same price that we bought it to the people here, you get a cheap
laptop, we get some computing time on them before and that's the main reason for running
it on laptops.
Well actually the question was why you were using banana pie, that's a good question,
the thing is that the capabilities of the banana pie were a bit better than the Raspberry
pie, the times the decision was made, if you see there's a big LCD screen in front of the
boxes where you can see that thing, I think it was with driving those LCD panels and also
the computing power available on the banana pie that wasn't, yeah, but actually we have
to look that up in the repo, there's everything documented, okay, yeah, there's another one
in the front.
So the question was if there are any public dashboards out there, yeah, we've put some
public dashboards on dashboard.graphana.org, oh, dashboard.phasdom.org, sorry, which you
can have a look at the infrastructure, we used to have some more dashboards like the
t-shirts that have been sold, but due to the fact that we changed the shop, we converted
to something that we bought to an open-source solution and the thing is we totally forgot
to monitor that, so that's, but there are some dashboards out there to monitor it and
if you want to see something more, just come to me after the talk and I'll show you something
more here at the laptop, okay, yeah, another one.
The biggest one standing here, no, actually the biggest issues we had was like running
all that stuff after three years and not having set up everything properly was quite challenging,
like on Saturday morning we had to run and redo the whole video installation on the K
building because of, you see those transmitters here, they were not plugged properly and so
we had no audio on the stream, this was one thing and then another very challenging thing
was like when we played around and as I play, we did not engineer anything properly, when
we played around with the BGP sessions, it was not clear how long it would take till
things distributed to the whole net and we were literally just trying to get information,
is it working, is it working not and till this BGP information propagates from here
to the rest of the planet like Brazil, it takes quite some time and so you can't be
sure that you're setting up the BGP session, everything works because shit will hit the
fan after ten, twenty, thirty minutes and not instantly and so it's quite a problem
to have instant recognition if things are going well or not.
So the question was if the problems with the Wi-Fi that we had here on site were due
to our BGP playing or was it due to something else, solar flares and so, the thing is that
we had some issues, we've been given access to the WLC, the wireless controllers, you
see these boxes over there, they're centrally controlled and we have to dig in that.
We have some visibility of the infrastructure that's owned by the ULB, they've given us
access to that so we can engineer that but we're not quite sure why it was that.
Most of the time FOSDEM, which is an IPv6 only, was working quite good except for some
Apple devices that do tend to just set up an IPv4 address even if there's no proper
IPv4 and things get complicated and FOSDEM dual stack, which is dual stack, usually worked
for most of the Apple devices, but we're not very certain of it.
Yeah, yeah, you will see that.
There's another one.
So the question is if the live streams will be made, yeah, this week, rewindable or not,
I honestly can't tell you that, I don't know, I can ask the video guys if they're planning
that for next year, but there's no plan of that as far as I know.
The biggest challenge was to redo things with HDMI over VGA, which we had the last years.
But there's another one, yeah.
So the question is that we're planning to use service, do we know what and what's planned
for next year?
We'll have a talk about that next week, I think, and then we go through the post-mortem,
which is usually a week after FOSDEM, and then we decide on things to be bought for
next year because switches are old and the route is also old, I think, and we have one
more year on the route to go that should be fine for next year, but what after that?
We have to make some decisions and some investments for next year to run this stuff, and this
will be done next week when we're all bit cooled down and refreshed after this FOSDEM.
Anyone else?
Yeah, come.
So the question was what part of the infrastructure are being reused and what do we bring for
the event?
Well in numbers, I think it was three truckloads of stuff, not three because the video arrived
in a second.
We bring mainly cameras and those boxes here, switches stay at the ULB, most of them stay
here but the one that didn't stay here, they won't be here next year because ULB is planning
to do some tidying up and giving us here some video ports for our VLANs.
They're very, very good at working with us.
We get access to most of the infrastructure, we just say tell them what you like to use
and they just throw it on their controllers and bridge it to our service and we can use
it and make fun with it, and they will be replacing part of the network infrastructure
next year and we then will have to bring even less gear here.
Which one first?
Yeah.
What about the rest of the year?
So the question was what's about all the other stuff that FOSDEM is doing through the year?
We hosted on our own hardware, is it in the cloud or somewhere?
We used, yeah we have another company called Tegron here, it's a Belgian provider and most
of the stuff is running at Tegron during the year.
During FOSDEM we also spin up some VMs at Hetzner in Germany and they are only for during
the event and short time after the event, so like cutting videos and so on in the cloud
and they will be turned off like two or three weeks and then everything is running on Tegron
on our own hardware there as well.
So there was another question.
So the question was what is being used for the communication between volunteers?
We have that matrix set up, I don't know who's aware of matrix, it's a real time communication
tool like Slack or something like that.
We use matrix since 2020 internal for our video team for communicating and then we expanded
that for 2020 and then with the pandemic we opened it up for all of the people and now
the volunteers are being coordinated to that.
And we also have our own trunk tier steel that we have here especially for this event
set up and volunteers are also can be reached via those radios.
Am I correct volunteers?
Yes, yes, okay.
We have two volunteers here.
So yeah, we'll get you.
Is there anything else you want to know or any other?
Where's the money?
The question is where's the money Lebowski, that's the real phrase from the film.
I don't actually know, I'm not yet the member of FOSDOM stuff so you have to ask someone
in a yellow shirt, there happens to be one next to me just from the microphone.
We have a money box and a bank account.
Anyone else?
Three to one.
Thank you very much.