So, we start with the first presentation by Olivier Caix from the G-Streamer of the State of the Union for this year.
So, hi everyone. I'm Olivier Caix. I've been a short developer now for 15 years.
And I'm going to tell you a bit of what has happened in the G-Streamer community in the last two years that we last met.
Here. So, we have two major releases, 1.20, 1.22.
Quite a lot of merge requests as you can see.
And an interesting fact, in 1.22, a third of the commits were in the Rust modules.
So, we've been investing a lot in Rust in the community. There's a lot of excitement there.
And that's been happening. One of the kind of big things for us as developers that we did was to merge all of the various
git repositories into one big giant repository. So, now everything is together, except for Rust.
Rust modules are all on the local corner because they're released with the GTK-RS infrastructure.
So, the rest of these three ones. But that's kind of our big change for the developers,
but it doesn't change much for the end-user because we still release all the packages in separate car balls, just like these.
Another thing that we did that's between infrastructure is we've improved our build system,
and now we can select the elements one by one. Not just the plug-ins, but you can select exactly which element you want in the plug-in.
And then we can also link all the plug-ins, all the elements, all the dependencies into one giant library.
This makes it actually easier to make a smaller build because we can build only exactly the functionality that you need for a specific application.
Create a big library that only has the exact functions that are being used, right, and nothing else.
So, we did that for embedded systems mostly so that you can have something a little smaller.
Another area, and there has been quite a lot of excitement in this terminal the last couple of years, is WebRTC.
So, as probably all of you are familiar with, WebRTC is a way to send video and load it in SQL from browsers,
and Distributor has one of the most complete implementation outside of the Google implementation that's used by the browsers.
We were missing one big bit, and that was the congestion control, and that's been added in the last releases.
So, now we have a module that is compatible with what's called Google congestion control, which is what Chrome and Firefox and Safari use.
And this is in Rust.
And to make that work, so we have a WebRTC implementation, but that did not do any of the actual encoding that was left separate on purpose.
Now we have a module in Rust plug-in that will plug the encoder and the WebRTC and do the congestion control so that you can adapt the bitrate of the encoder to the encoding,
and this is all automatic if you use this plug-in.
We also have Web and Web, so these are also within Rust there. Web and Web are a way to replace RTMP, but based on WebRTC, so it's a single request,
a single HTTP request way to set up a WebRTC stream.
It's mostly meant to stream to and from a server, so it's really a replacement for RTMP for a low latency video transmission.
Speaking of WebRTC, so WebRTC is based on RTMP, and this is an area where there's also been quite a bit of development.
So, starting with 222-1 order correction, that's a system used mostly for legacy broadcast transmission, and we have the 2D order correction.
So what does it mean 2D? It means that we do 4D order correction, which is basically, you absorb multiple packets, then you generate a new one,
and if you have any of these packets except one, then you can regenerate the missing one, right? That's what the parallel error correction is.
Traditionally, you would do like packets one, two, and three, and four, and then you would generate a fifth packet.
What losses tend to come in bursts in networks? So with 2D error correction, we have this kind of traditional version, and also a version where you do packet one, and five, and ten, right?
So if you have a burst, you can recover more of the more packets.
The other thing that we've added, so just remember for a long time, we've had the API to add RTP-Ether extensions.
That's a way to each packet to add a little header with something else. So for a long time, we had libraries to actually write these,
but we didn't have something in the system to easily plug in something to insert this header in every packet without having to write application code.
So this is something that we've added, and we've added a bunch of different modules.
The multiple is this client-to-mixer audio level. This makes it possible for a sender of audio to say the volume that I'm sending is this,
so that a mixer or some kind of service can select the person who speaks loudest without having to decode all of the audio.
So it can know from this level who's speaking louder and just forward that one.
Then, color space. So this is for VP9. If you want to send HDR over VP9, we now have this RTP-Ether extension to make it work.
It's compatible with Chrome again, so this is a phase-on-article experiment.
And we have an AD1 builder and deep builder, so this is, I think, probably the first thing where we've decided that the official implementation of a major feature
is only available in Rust. So this is something that we're pretty happy about.
Another thing, so H264, H265, they have two kinds of timestamp.
Presentation timestamp, decoding timestamp. When you send RTP, normally only send a presentation timestamp.
You need to apply an algorithm to recover the decoding timestamp. We now have modules that generate that.
We also support RxC6651, so that's a way to synchronize streams immediately.
So traditionally with RTP, we send two streams, audio, video, separate timestamps,
and then sometimes later you get a packet telling you what the correspondence is.
With RxC6651, we add the RTP-Ether extension in every packet so that we can be synchronized from the first packet.
And we also improve our base class for video decoders a bit.
So now it can recognize that there's a corruption and use that to request a retransmission previously.
We kind of applied the error, but we let the application do the decision.
Now we've added something to the base class.
Another big feature that was worked on is that we basically rewrote the HLS and dash base class.
So the previous one was over 10 years old and had been written largely without the specs.
And even when we had the specs, HLS has changed quite a lot over the last 10 years.
So now we have almost a state-of-the-art implementation based on 10 years more knowledge.
It's much more simple, has fewer trends, much better control on how we download things, on the buffering.
We do a little bit of the parsing in there because sadly for many of these for us, you have to parse the base stream to handle it properly.
So this is all implemented as one in this decade.
We've put a few things around decoding, mostly video decoding.
One thing I'm quite excited about is the subframe decoding that has been quite a few years coming.
And this means that we now have infrastructure in our base classes to start decoding a frame before you receive the entire frame.
Some format, issue 6.46.5, we can split the frame in slices.
And from this first now, you can actually start doing the decoding.
We have two implementations of this.
One is based on HPEG, which can do this only for issue 6.46.4.
And the other one is for the exiling hardware because they have the hardware features to do that.
So they can do super low latency decoding.
We have WebM Alpha.
So the WebM format, so HPEG tonight don't have support for transparency built in.
But there's a WebM extension where we basically store two separate video streams.
One with the colors and one with the transparency.
And now we have an element that will spin up two decoders and then recombine them into a A1B stream.
We have a DirectX Elevent library.
So make it easier to integrate Direct 3D Elevent applications in the streamer to do zero copy and coding.
For example, from a Windows application.
And also speaking of Windows, our Direct 3D Elevent decoders are now default.
So they're becoming the choice that will get auto plugged if you have them.
So you get hardware accelerator decoding on Windows. That works.
What about MacBooks?
Yes, there's also...
We've got a question already.
So we have a hardware decoder from Mac and IOS, which is the same idea.
CUDA, so some people use proprietary software and proprietary drivers.
So we have now also a CUDA library so that you can insert more easily CUDA data into the streamer for encoding,
decoding all these things.
We have some more CUDA native elements, one that is a converter and a scaler, so using CUDA itself.
We have CUDA and Direct 3D integration for Windows again.
And this whole thing basically gives you zero copy and coding on NVIDIA hardware,
especially when you match with some other CUDA-based software.
But some people prefer free software. So we also have BAPI, we have a new plugin for BAPI.
So we've had G-SUMOR BAPI for a long, long time.
It was getting quite freaky. It was not based on any of the base classes that we have improved since then.
So it's been completely rewritten from scratch with a new plugin that we call VA.
It supports all the major codecs now that we've implemented, all the ones with VA.
It supports AVY, it supports just 5, VPA, between 9 and 8, MPEG-2.
Encoding, we only have the issue 6, 4, 5 codecs for now, but the rest are being worked on, as we speak.
And using live VA, we have a bit more than VA.
We have a bit more features. We have a compositor.
It's an element that will take two or more streams and composite them into the same video.
We have a D-interlacer.
And we have a post-processor element with scaling and color space conversion,
using the video functionality instead of the GPU.
And open work has happened around 8 to 1 or something in the last two years.
So we have quite a lot now.
We have support for AV1, both in the legacy VA plugin and in the new VA plugin.
We have it for AMD, the coders.
We have it for Direct3D on Windows, using the NVDI-PI's.
For Intel, using their multiple libraries, either Quixin, QSV, or the new VSDK.
So we have pretty comprehensive AV1 support, in addition to the RTB plugin that I mentioned earlier.
Another new thing that we've done is, this is our first official machine learning integration that is in G-streamer itself.
So it's the first step.
And we've written a plugin to use the Onyx runtime from Microsoft to basically take some video frames,
some model and recognize objects, put little boxes in the metadata,
and then another element that can take these boxes and draw them on the image.
So this is the first step. A lot of work is happening right now to have a better video on the fixed framework as part of G-streamer.
All these things I like about sometimes you want to have a UI.
And a few pictures were recently added there.
We have a GTK4 paintable, so that's an object.
I would say that you can use the GTK4 to actually draw something on the screen.
So now you can easily integrate G-streamer with GTK4 to your old copy playback.
This one is in Rust, which is kind of cool.
Qt6 as well as that thing, so that we have something that is very similar to what we have for Qt5, which is a QML item,
so that you can integrate a G-streamer sync with the output with Qt and draw in your Qt application.
And the last one is a niche case.
So we had a Wayland sync for a long time, and what this Wayland sync allows you to do is to basically take the video buffer
and give it to the Wayland compositor directly without going through the toolkit.
So you can use the 2D hardware planes of the platform.
This is multi-use useful and embedded. It allows you to do things like greater performance, not use the GPU on embedded systems
where the GPU might be too slow to jail.
Up to now, this all works fine, but you have to write a low level Wayland code, and that's non-trivial.
So we've written a GTK widget that wraps up.
So now we can write your application GTK, just add the widget, and you get all of these performance benefits for free.
Last but not least, in this category, we added touch event navigation.
So previously we had navigation, we could send letters, we could send mouse clicks,
but now we can also send touch events so that you can have elements in your pipeline that are controlled by the user,
such as we have a Webbar app, a web view, a webkit-based source.
And we have some new tracers, so these are tools for developers to know actually what's going on in a pipeline live.
We have a bunch of tracers already, four more were added.
Some of them are quite useful, like we have one to generate buffer,
to read the buffer lateness and one to trace the queue level, and these will output the information in a CSV file
that you can then load and make nice graphs and understand what's the live performance of your pipeline, what's going on.
We have one to draw pretty pipeline snapshots, so for a long time we had a feature where we could draw a dot file
to draw a picture of the pipeline, but this required it.
I added some code to the application to actually trigger it. With this tracer, now you can just listen to a unique signal
and trigger it on the spot.
The last one is the factory tracer. This is the very first feature that I mentioned.
So it's nice to say, oh, I'm going to build a G-streamer, build specific for my application with only the elements I use.
But if you use PlayVin, there's a lot of automated things, and you might not know exactly which plugins you've been using.
So with this tracer, we can actually trace all the plugins that get loaded, all the elements that are used,
and print, when you exit your application, print the list of what was actually used.
A question about that. PlayVin sometimes tries to use PlayVin, but this got it later because it just worked.
It's going to be listed anyway.
So yes, right. PlayVin sometimes tries to use something, but it doesn't work because the hardware is not there or something.
So in this case, the tracer will still list it.
It's everything that is loaded, right? When they're tried or loaded, you call a function in it and it says no.
So this is really at the loading stage at this place.
Thank you. Any questions? Yes.
You mentioned V1 and the RTP support in there. So can you also add the SPC extensions?
I have no idea. Anyone knows? No.
So RTP, any one extension, SVC extension in, I don't know if it's there.
So we do layer selection of the highest quality here, but it's not there.
So there is an external, there is a dependency description of RTP extension where you can get information about the SVC layers in there,
which is basically something like that. So you encode the SPC layers into one screen,
then you use these external extensions to vary information about what's in there and what isn't, so that you can do the connection.
So that's what I got to introduce from the other video.
None of the RTP and SPC stuff, including the VT1, which is quickly required because there's no SVC inside the screen.
Yes, so the question was RTP, AV1, SVC, there's an extension that can make it really useful.
It's being standardized and it's not implemented yet, but it will be at some point.
I forgot to mention we have an online question, online question at number 43401.
So if people at home want to ask questions, it's possible, but since we don't have any questions, there are any more questions on the floor.
Q6, does support different rendering backends, like DirectX on Windows and Vulkan on Pino or something,
with a QML event?
And then Q6 supports post-interference that you can directly pass the DirectX buffer to the QML event?
So does Q6 support other backends than OpenGL? And I think the answer right now is, won't it support OpenGL?
Any more questions? Yes.
Can you do a statically linked binary?
Can you do a statically linked binary? Yes, you can.
That's kind of one of the use cases as we create this statically linked library and then your application can link to it and only link the required bits.
That's kind of the part of the trick to make it a little smaller.
Yeah, my question.
So you said that there is congestion control in WebRTC now?
Yes.
Is it like the same feature set as in Google's implementation?
So, yeah.
The question is about congestion control in WebRTC. Is it the same feature set as the Google implementation?
As far as I know, yes, because it's basically a copy of the implementation in Rust.
So they basically re-implement the same algorithm.
So that is compatible.
But it's pluggable. So you could write your own.
There's a plug-in mechanism and the core version is in C, but this one is in Rust with a separate plug-in.
One could write a different implementation because there's a bunch of heuristics in there.
There's no line.
Perfect answer.
Thank you so much.
Do you have a question?
Yes.
If I have an application that does WebRTC signaling of a matrix, for example, would I benefit from switching to WebRTC sync or would I sit with WebRTC?
So the question is, if you have an implementation of WebRTC that does signaling something custom, for example, other matrix, would you benefit from using WebRTC sync?
And the answer to that is yes, because it will do all of the encoding and congestion control hooked up for you.
And there's an interface that you can implement for your own signaling.
So the signaling is separate from this WebRTC sync.
There's still a module that we can implement.
We have one implemented for like a custom signaling mechanism where there's a server that's implemented, but you could write your own.
Can I ask the last question?
No, just a comment from the question before.
The QT6 direct 3D integration.
There is a merge request for it.
Okay.
So Tim says that for the QT, there's a direct 3D merge request open to integrate that.
It's not merged yet, but do test it at home and complain when it doesn't work.
Last question.
Okay, thank you.