All right, well, welcome back everybody. Up next, the one and only Dan Jenkins is going
to tell us all about G-Streamer and Golang. Take it away, please.
Thank you. Hello, everyone. Can everyone hear me okay? Yeah? Good. Great. Cool. Okay.
I forgot my clicker. Number one rookie thing to do. No, no, I've got my phone. So I'm good.
But yeah, that's why I've got my phone. And it's going to look a little bit weird.
I also forgot to buy I brought two European plugs with me. But one wasn't European. One was American.
So my day did not start off well. So yes, G-Streamer and Golang. So a little bit about me.
Very, oh, that's just going to get really annoying. I'm just going to click. Cool.
Okay, so a little bit about me. So yes, I'm Dan Jenkins. I run a couple of companies.
One called Everycast Labs, one called Nimbleape, and another one called Comcom.
So Everycast Labs does broadcast stuff, bringing in remote talent into broadcast workflows.
Nimbleape is a consultancy service, consultancy company based in the UK. And then Comcom is an
event that we put on for open source people, our way of kind of giving back to the ecosystem that
we build from. I was the very first Google developer expert in the world when it comes to WebRTC.
I'm not saying I'm the best at WebRTC, but I'm the first that actually got accredited by Google's
developer program. I love Lego, and I love real-time media. So yeah, Nimbleape, we're a consultancy,
and if you've got hard problems that you want solved, come talk to us. And Everycast Labs,
we've got that product that I was just talking about called Broadcast Bridge. And then Comcom.
So Comcom is dear to my heart. Historically, it's been a residential event where we bring
everyone, everyone stays in the same place. And then we've got three days of awesome real-time and
open media content. And then we're back in 2024. Dates are still up in the air because of contracts,
but it's not going to be residential this year. We're going to go on tour, so we're not just
going to be in the UK. And that's quite exciting. So to the actual topic, GStreamer building real-time
applications with Golang. So what are we actually going to talk about? We're going to talk about
GStreamer, obviously. We're going to talk about Golang, obviously. But I want to introduce you to
something called GoGST. GoGST has been around for a long time now, but kind of got itself into a
bad state where it was not un-maintained, but there were lots of little forks and lots of little
patches everywhere. And so we've kind of changed how that project's being managed now. And then
also I want to introduce you to something called Pion. So let's take a look at GStreamer first.
Who in the room has heard about GStreamer? Good. That's the answer I was looking for.
So open source multimedia framework basically does everything that you chuck at it in some form.
And I absolutely love GStreamer. So a lot of you might know GStreamer as something like this.
I'm not going to ask you to tell me what that is, because I know that it's kind of taking in an RTSP
source and then doing something with it and then outputting something at the end,
via UDP, but all the stuff in the middle now. But GStreamer is actually super powerful and
ultimately lets you do ingress, do something with it, and then egress. And it kind of boils down to
something that's simple, right? GStreamer can do it all and can do a lot of things. So for us at
everycast labs with our broadcast bridge product, we care about certain things. So GStreamer can do
NDI, GStreamer can do WebRTC, GStreamer can do SRT, can do RTP, it can do HLS, it can do RTMP and
RTSP, right? I'm not telling you anything that you don't know at this point. But for us, at least
with broadcast bridge, GStreamer has a superpower and that superpower is app source and app sync.
How many people in the room know about app source and app sync? Okay, good. That means like 60%
of you are going to learn something now. The rest of you just sit and be happy. So yeah, this is
what we use at broadcast bridge, in our broadcast bridge product. And that's because we don't write
C. And so ultimately kind of adding code to plugins within GStreamer is really difficult for us.
I know that's changing as time is going on. There are more and more Rust plugins, but at its core
there are a load of stuff that we don't feel able to kind of contribute to if we find a problem.
And so a lot of the time we don't like writing C like this, but we do like writing a lot of Go.
And so we end up writing something like this. And this is Go GST.
It was originally created by a guy with the GitHub handle, Tinyzimmer. I love the name.
But now it's in its own GitHub organization. So it's under github.com.
So it's under the new GitHub org and there's three main contributors. I think there's something like
17 total, but there's three main ones. Tinyzimmer me and R.S. Willy. So Lesfawkes is better for
everyone. So this other one, Big, Little Ben. That's from the LiveKit team. And the LiveKit
team had their own fork of Go GST. And they had put a load of work into fixing bugs, but
they were never getting merged back into the project as it was under the Tinyzimmer GitHub.
So now it's forked out. Well, it's not actually forked. We forked it into its own organization
and then did the GitHub magic where we unforked it and then the Tinyzimmer one is now a fork of us.
So there's a lot of GitHub kind of organization going on to make it easier for everyone. Did you
know that GitHub forks don't turn up in Google SEO results and they don't turn up in GitHub
search results either? And search doesn't work in the repo. So yeah, and search doesn't work in the
repo. So basically forks are dumb. I mean, they're not dumb. But forks are bad. We should not be
relying on forks for a long-term thing whatsoever. So yeah, this is actually really great for everyone
now. So less forks is better for everyone. And like I said earlier, BroadcastBridge uses a mixture
of SRT, NDI, WebRTC, among a load of other things as well. And so why, you're probably asking,
why would we even need to use AppSync and AppSync when the modules, the plugins are already in Gstreamer?
Like Gstreamer already knows how to take in an SRT feed. It already knows how to output an NDI
feed and it knows how to do WebRTC stuff. So why are we building on top of AppSync and AppSync?
And it comes down to greater control like I was kind of alluding to earlier.
We use Pion to do WebRTC. And that's not because the Gstreamer implementation isn't good. It's just
that if we want to be able to do anything that isn't implemented into the Gstreamer implementation,
then we'd need to get someone to actually go and change that code. And that's something that we're
able to do. My team aren't capable of doing, but we don't go really, really, really well.
And so we can definitely kind of go and take that greater control.
Like I said, this means we're handling WebRTC in something that we really know. Like, ultimately,
very few people in this room know about transcoding something from one codec to another. And we just
rely on FFMpeg or Gstreamer or whatever to do it for us. It's the same with WebRTC for us. We
really know what we're doing with WebRTC and we want to be able to kind of tweak things that we
can't necessarily tweak with the Gstreamer implementation. But Pion is hugely, hugely
powerful. And this is the other key thing and it's easily upgradeable. So when we actually find a bug
in Pion, we can
a Gstreamer pipeline and never leaving the C level. But cost isn't just measured in terms of
compute. Cost is everything from building the feature all the way through to deploying the
feature and running the feature. And you've got to look at the whole picture. Pion gives us huge,
huge flexibility and we can move fast and we can add new features and ultimately that means
that we win business. So let's take a quick look at AppSource. How many people are actually familiar
with AppSource? Right. So AppSource is just another plugin, module, whatever they're called.
And ultimately, you can put it inside of your pipeline and you can push data into Gstreamer
using AppSource. You set a load of capabilities on that element, that AppSource element,
telling it, oh, well, this media that I'm just about to push into you is this format and this
frame rate and whatever else. And you can push in data or you can make, so you have to push
data in obviously, but you can also make Gstreamer ask you for the data. So instead of just going,
oh, I've got data, data, data, data, data. And then Gstreamer goes, oh, hold on. I can't do anything
with this. Why are you sending me so much data? You can, Gstreamer can actually ask for it. Now,
that's not hugely helpful when it comes to real-time applications because real-time applications,
in the case of Pion, sending us web, getting RTP data from Pion, for example, that's real-time.
And so we want to get that data from Pion and we want to pass it into Gstreamer straight away.
Because we're getting it in this constant flow from Pion. Whereas if you were reading a,
if you were reading a file and then you were passing those chunks into Gstreamer,
well, you've got control over how fast you push those chunks in. And so why not let Gstreamer go,
ah, I want a bit more data. I want a bit more data. I want a bit more data.
Right. App sync is absolutely no different. It's a, it's a plug-in, it's a module. And,
and when you put it into the pipeline, it becomes an element.
And ultimately you get push data out of AppSync. And so imagine you've got AppSource and then you've
got something in the middle, whether or not that's transforming it or transcoding it. And then you've
got AppSync and you're connecting all these bits together. And so you're pushing data in. Gstreamer
is then doing something with it. And then it's pushing it, it's passing it over to AppSync.
And then AppSync sends it out to your application as data. Not as UDP, not, not via report or anything.
It's giving you the, the, the raw buffer of data. So you get pushed your data from AppSync via the,
the, the, the new sample signal and event. I've got some data here you go.
Notice how this is all go lang. So, yeah, let's take a very quick look. So we've got our sync.
So that's an AppSource, AppSync element that I've made. And I'm setting some callbacks on it.
And then we've got new sample funk. And then that gives me, that gives me my, my sync.
And then I'm going to tell it as a return. I'm going to tell it what the return, what the flow
state is. And so I pull the sample. And then if the sample isn't end of, isn't nil, then,
then we carry on. If it is nil, then I'm returning that we are at the end of the stream.
And then buffer. So we get this, our sample. So we're pulling the sample.
And then, and then we're getting the buffer out of that. And then ultimately reading some, some,
some information from that, from that buffer map, changing it from big Indian to little Indian,
I think, or something. And then, and then doing some stuff on it, doing some maths on it.
Not a lot of like useful information there. Like in terms of like, what am I actually then going to
go and do with it? Well, at the moment, it's just printing out RMS. But then you can go off and do
whatever you want with it. For us, that means getting a video and audio data out of G streamer
and chucking it into NDI. Oh, Dan, why are you not using NDI within G streamer? Well, I tell you
number one, when we did our NDI integration, G streamer didn't have NDI. It was, it was completely
separate. It was, it was a different repo. And it wasn't part of the G streamer rust plugins.
And then B, we do extra stuff that G streamer doesn't know how to do yet. So we, we grab tally
information from, from NDI. And to be able to do that, you need access to the underlying NDI sender.
And, and so there's stuff that G streamer can't do yet. Something that we actually want to add in to
G streamer. So that we can stop sending stuff via the NDI SDK directly and we can just let G streamer
deal with it for us. But again, goes back to that cost analysis, right? At the moment, we can get that
data out of G streamer using app sync and chuck it out via NDI. We can do that. And it's relatively
cheap. But then there's a load of extra work for us to be able to kind of go in and figure out the
right way of doing it in G streamer so that like tally information becomes available as a signal.
So yeah, for us, this means that we have to handle RTP and RTCP from Pion. Because Pion,
within WebRTC, WebRTC is made up of lots of standards. But ultimately the media is RTP.
And the bit that tells you what the quality is and everything else that goes with it along with it
is RTCP. So it's very easy to forget about things that are very important when you don't deal with
them. Like RTCP. SFU people in the room will go, ah, you could never forget about RTCP. But as a
web developer, the browser deals with all of this for us. And so it's very easy for us to go, ah,
RTP, I'm going to get my media. I'm going to get my media. And then everything works really,
really well when you're in a really nice network environment. But then you chuck in real life
scenario and the audio in the video goes terrible. Why did the audio and video go terrible? Because
there's no RTCP feedback mechanism to go, ah, something's going wrong. But yeah, GStreamer
makes all of this easy. And very quickly on this very specific thing, we use RTP bin within GStreamer.
So that's that middle bit for us. We use app source, chuck it into RTP bin, and then we do a
load of transcoding and stuff as well. And then we get app sync. RTP bin is magical. If you deal
with RTP at all with GStreamer, then you need to be using RTP bin. There's a lot of text there.
But ultimately, it implements everything you need to be able to handle RTP and RTCP
and demuxing of payloads. And it's just a very nice all in all thing that deals with
everything using all of the separate, all the separate plugins. But it forces it all together
nicely for you. So for us, that's connecting the app source sync pads to RTP bin. And you'll notice
I say pads. So for us, you can see up the top there RTP bin. So we're requesting a pad from RTP bin
in that format. So it's a receive RTCP sync. And then we're also requesting a pad of send RTCP
sync source as well. We then go and make a new app sync and a new app source. And you can see
they're labeled RTCP app sync and RTCP app source. We then add those to our pipeline because otherwise
nothing works. All of your elements have got to be in a pipeline. And then we link our RTCP app source
pad RTCP app source, get static pad source, link RTCP sync pad. Yes. So I'm getting the app,
sorry. I'm grabbing the RTCP sync pad from the RTP bin. And I'm linking it over to the RTCP app source.
So that's basically just saying RTP bin is going to give me some information up to RTCP
information via a pad. And I'm connecting to that pad so that I can then grab that information and
send it over, send it back via Pion up to my web RTCP. So you'll get RTP in, in this case, you'll get
RTP in into RTP bin, but you'll get RTCP in and out. So you'll get told RTCP and you'll also send
it back out as well. And like I say, don't forget about the RTCP. As you can tell, I forgot about the
RTCP and ended up doing certain demos and going, ah, look, it's really great. And then someone went
and tried it on a really crappy internet connection and went, no, Dan, it doesn't work. And, and made
me look rather foolish. So you end up looking something like this. So does everyone know about
the dot graphs that you can generate from GStreamer?
A couple of nods, not that many. So you can, within GStreamer, you can tell it, I want you to export
a dot graph file on anything, on, on a state change or whatever. You, you've got control over when it
generates it. And so for, for me, we, when we've got debugging enabled, we enable a dot graph
generation whenever state changes. And so ultimately, this looks really small and dumb. It's a PDF. So
you can go in and, and look at it in high quality detail. Um, because it's not a PNG. So you've got
lots of options. You can, the dot graph can be converted into lots of different formats.
But the really cool thing about dot graphs is it tells you what's connected to what. And so it's
really great for debugging. And so for us, we've got our app source, um, our app source and our,
our two app sources. So one is, um, one is RTP, which is this one. And then this one is RTCP.
And you can see, I'm coming off the camera. I'm sorry. Um, so you can see that this one's set
with, um, with capabilities to say that this is RTCP. And this one is set with capabilities to say
this is RTP. And so you can see those are linked to a pad within a GST bin, a GST RTP bin. And so
those pads are then connected to an RTP session. The RTP session is then, um, connected to a
demuxer. The demuxer is then connected to a jitter buffer. And the jitter buffer is then able to go.
Oh, well, in this, in this RTP stream that I'm receiving, that's both audio and video,
where it's demuxed it and then it automatically goes, ah, here's the video and here's the audio.
Right. And then it chucks it back out,
chucks it back out, creates some pads for me, which I then connect over to, well,
there's an app sync up there and that's my RTCP app sync. But then you could see here
that it's then connecting out Opus and VPA into my pipeline.
And then this is like the rest of the pipeline, which we don't care about, but like, I get told
it's Opus and I get told it's VPA. And so I'm able to decode it and do stuff with it,
whether or not that's outputting to NDI or whatever. At the end of the, um, at the end of it is, um,
is an app source, uh, sorry, an app, an app sync for sending out via NDI.
So we, we got into go purely because of Pion and Pion gives us loads of control.
It's basically WebRTC in pure Golang. If you ignore the fact that WebRTC does lots of like
actual media stuff, but when you look at, say, the, just the, the network portion of it of sending,
sending data from here and sending it there, then it's pure Golang.
So yeah, you can do any of this with any of the G streamer bindings or you can just, you know,
do it with actual G streamer C. I mean, who actually want to do that? I don't know.
But you can go and use whatever bindings you want. And so there's really nice bindings for Python,
Rust, um, and I haven't used any of the others. Um, I've definitely used the Python one and the,
and the Rust one myself. Um, and the Golang one, I went on there this morning to take the screenshot
and I was like, Oh, where's the Golang one? Um, so here's the pull request to add it to the list.
So if you've got a problem and G streamer doesn't quite solve that problem,
that's what this talks about. This talk is about the fact that you can make G streamer do what
you want it to do using app source and app sync. You can build it yourself with app source and
app sync. So why G streamer? Why not FFM peg? Whatever. G streamer does everything that we
need it to do. It has a fantastic community, super friendly community. And ultimately it's just
super flexible and does exactly what we need it to do. Um, which is not something that we felt as
a team. FFM peg would give us, for example, G streamer has a lot of scaffolding, let's say,
um, and, and gives us an awful lot, um, for free. Whereas G, uh, FFM pegs a little bit more, more
work, right? So my last message is G streamer for the win. Um, so yeah, don't wait for others.
Don't wait for others to build your plugin for you. You can go and build with G streamer,
app source and sync. And that's me. Thank you very much.