[00:00.000 --> 00:11.000]  So, we start with the first presentation by Olivier Caix from the G-Streamer of the State of the Union for this year.
[00:11.000 --> 00:20.000]  So, hi everyone. I'm Olivier Caix. I've been a short developer now for 15 years.
[00:20.000 --> 00:27.000]  And I'm going to tell you a bit of what has happened in the G-Streamer community in the last two years that we last met.
[00:27.000 --> 00:35.000]  Here. So, we have two major releases, 1.20, 1.22.
[00:35.000 --> 00:39.000]  Quite a lot of merge requests as you can see.
[00:39.000 --> 00:46.000]  And an interesting fact, in 1.22, a third of the commits were in the Rust modules.
[00:46.000 --> 00:52.000]  So, we've been investing a lot in Rust in the community. There's a lot of excitement there.
[00:52.000 --> 01:01.000]  And that's been happening. One of the kind of big things for us as developers that we did was to merge all of the various
[01:01.000 --> 01:09.000]  git repositories into one big giant repository. So, now everything is together, except for Rust.
[01:09.000 --> 01:15.000]  Rust modules are all on the local corner because they're released with the GTK-RS infrastructure.
[01:15.000 --> 01:23.000]  So, the rest of these three ones. But that's kind of our big change for the developers,
[01:23.000 --> 01:33.000]  but it doesn't change much for the end-user because we still release all the packages in separate car balls, just like these.
[01:33.000 --> 01:39.000]  Another thing that we did that's between infrastructure is we've improved our build system,
[01:39.000 --> 01:47.000]  and now we can select the elements one by one. Not just the plug-ins, but you can select exactly which element you want in the plug-in.
[01:47.000 --> 01:54.000]  And then we can also link all the plug-ins, all the elements, all the dependencies into one giant library.
[01:54.000 --> 02:01.000]  This makes it actually easier to make a smaller build because we can build only exactly the functionality that you need for a specific application.
[02:01.000 --> 02:10.000]  Create a big library that only has the exact functions that are being used, right, and nothing else.
[02:10.000 --> 02:19.000]  So, we did that for embedded systems mostly so that you can have something a little smaller.
[02:19.000 --> 02:25.000]  Another area, and there has been quite a lot of excitement in this terminal the last couple of years, is WebRTC.
[02:25.000 --> 02:33.000]  So, as probably all of you are familiar with, WebRTC is a way to send video and load it in SQL from browsers,
[02:33.000 --> 02:41.000]  and Distributor has one of the most complete implementation outside of the Google implementation that's used by the browsers.
[02:41.000 --> 02:50.000]  We were missing one big bit, and that was the congestion control, and that's been added in the last releases.
[02:50.000 --> 02:58.000]  So, now we have a module that is compatible with what's called Google congestion control, which is what Chrome and Firefox and Safari use.
[02:58.000 --> 03:00.000]  And this is in Rust.
[03:00.000 --> 03:10.000]  And to make that work, so we have a WebRTC implementation, but that did not do any of the actual encoding that was left separate on purpose.
[03:10.000 --> 03:23.000]  Now we have a module in Rust plug-in that will plug the encoder and the WebRTC and do the congestion control so that you can adapt the bitrate of the encoder to the encoding,
[03:23.000 --> 03:28.000]  and this is all automatic if you use this plug-in.
[03:28.000 --> 03:40.000]  We also have Web and Web, so these are also within Rust there. Web and Web are a way to replace RTMP, but based on WebRTC, so it's a single request,
[03:40.000 --> 03:45.000]  a single HTTP request way to set up a WebRTC stream.
[03:45.000 --> 03:57.000]  It's mostly meant to stream to and from a server, so it's really a replacement for RTMP for a low latency video transmission.
[03:57.000 --> 04:05.000]  Speaking of WebRTC, so WebRTC is based on RTMP, and this is an area where there's also been quite a bit of development.
[04:05.000 --> 04:21.000]  So, starting with 222-1 order correction, that's a system used mostly for legacy broadcast transmission, and we have the 2D order correction.
[04:21.000 --> 04:29.000]  So what does it mean 2D? It means that we do 4D order correction, which is basically, you absorb multiple packets, then you generate a new one,
[04:29.000 --> 04:36.000]  and if you have any of these packets except one, then you can regenerate the missing one, right? That's what the parallel error correction is.
[04:36.000 --> 04:41.000]  Traditionally, you would do like packets one, two, and three, and four, and then you would generate a fifth packet.
[04:41.000 --> 04:52.000]  What losses tend to come in bursts in networks? So with 2D error correction, we have this kind of traditional version, and also a version where you do packet one, and five, and ten, right?
[04:52.000 --> 04:59.000]  So if you have a burst, you can recover more of the more packets.
[04:59.000 --> 05:05.000]  The other thing that we've added, so just remember for a long time, we've had the API to add RTP-Ether extensions.
[05:05.000 --> 05:13.000]  That's a way to each packet to add a little header with something else. So for a long time, we had libraries to actually write these,
[05:13.000 --> 05:24.000]  but we didn't have something in the system to easily plug in something to insert this header in every packet without having to write application code.
[05:24.000 --> 05:29.000]  So this is something that we've added, and we've added a bunch of different modules.
[05:29.000 --> 05:39.000]  The multiple is this client-to-mixer audio level. This makes it possible for a sender of audio to say the volume that I'm sending is this,
[05:39.000 --> 05:47.000]  so that a mixer or some kind of service can select the person who speaks loudest without having to decode all of the audio.
[05:47.000 --> 05:54.000]  So it can know from this level who's speaking louder and just forward that one.
[05:54.000 --> 06:08.000]  Then, color space. So this is for VP9. If you want to send HDR over VP9, we now have this RTP-Ether extension to make it work.
[06:08.000 --> 06:12.000]  It's compatible with Chrome again, so this is a phase-on-article experiment.
[06:12.000 --> 06:23.000]  And we have an AD1 builder and deep builder, so this is, I think, probably the first thing where we've decided that the official implementation of a major feature
[06:23.000 --> 06:29.000]  is only available in Rust. So this is something that we're pretty happy about.
[06:29.000 --> 06:34.000]  Another thing, so H264, H265, they have two kinds of timestamp.
[06:34.000 --> 06:41.000]  Presentation timestamp, decoding timestamp. When you send RTP, normally only send a presentation timestamp.
[06:41.000 --> 06:49.000]  You need to apply an algorithm to recover the decoding timestamp. We now have modules that generate that.
[06:49.000 --> 06:59.000]  We also support RxC6651, so that's a way to synchronize streams immediately.
[06:59.000 --> 07:04.000]  So traditionally with RTP, we send two streams, audio, video, separate timestamps,
[07:04.000 --> 07:08.000]  and then sometimes later you get a packet telling you what the correspondence is.
[07:08.000 --> 07:16.000]  With RxC6651, we add the RTP-Ether extension in every packet so that we can be synchronized from the first packet.
[07:16.000 --> 07:21.000]  And we also improve our base class for video decoders a bit.
[07:21.000 --> 07:32.000]  So now it can recognize that there's a corruption and use that to request a retransmission previously.
[07:32.000 --> 07:38.000]  We kind of applied the error, but we let the application do the decision.
[07:38.000 --> 07:46.000]  Now we've added something to the base class.
[07:46.000 --> 07:54.000]  Another big feature that was worked on is that we basically rewrote the HLS and dash base class.
[07:54.000 --> 08:02.000]  So the previous one was over 10 years old and had been written largely without the specs.
[08:02.000 --> 08:08.000]  And even when we had the specs, HLS has changed quite a lot over the last 10 years.
[08:08.000 --> 08:17.000]  So now we have almost a state-of-the-art implementation based on 10 years more knowledge.
[08:17.000 --> 08:25.000]  It's much more simple, has fewer trends, much better control on how we download things, on the buffering.
[08:25.000 --> 08:33.000]  We do a little bit of the parsing in there because sadly for many of these for us, you have to parse the base stream to handle it properly.
[08:33.000 --> 08:41.000]  So this is all implemented as one in this decade.
[08:41.000 --> 08:45.000]  We've put a few things around decoding, mostly video decoding.
[08:45.000 --> 08:50.000]  One thing I'm quite excited about is the subframe decoding that has been quite a few years coming.
[08:50.000 --> 08:59.000]  And this means that we now have infrastructure in our base classes to start decoding a frame before you receive the entire frame.
[08:59.000 --> 09:05.000]  Some format, issue 6.46.5, we can split the frame in slices.
[09:05.000 --> 09:10.000]  And from this first now, you can actually start doing the decoding.
[09:10.000 --> 09:12.000]  We have two implementations of this.
[09:12.000 --> 09:17.000]  One is based on HPEG, which can do this only for issue 6.46.4.
[09:17.000 --> 09:22.000]  And the other one is for the exiling hardware because they have the hardware features to do that.
[09:22.000 --> 09:27.000]  So they can do super low latency decoding.
[09:27.000 --> 09:30.000]  We have WebM Alpha.
[09:30.000 --> 09:36.000]  So the WebM format, so HPEG tonight don't have support for transparency built in.
[09:36.000 --> 09:42.000]  But there's a WebM extension where we basically store two separate video streams.
[09:42.000 --> 09:45.000]  One with the colors and one with the transparency.
[09:45.000 --> 09:54.000]  And now we have an element that will spin up two decoders and then recombine them into a A1B stream.
[09:54.000 --> 09:57.000]  We have a DirectX Elevent library.
[09:57.000 --> 10:05.000]  So make it easier to integrate Direct 3D Elevent applications in the streamer to do zero copy and coding.
[10:05.000 --> 10:09.000]  For example, from a Windows application.
[10:09.000 --> 10:16.000]  And also speaking of Windows, our Direct 3D Elevent decoders are now default.
[10:16.000 --> 10:21.000]  So they're becoming the choice that will get auto plugged if you have them.
[10:21.000 --> 10:27.000]  So you get hardware accelerator decoding on Windows. That works.
[10:27.000 --> 10:30.000]  What about MacBooks?
[10:30.000 --> 10:32.000]  Yes, there's also...
[10:32.000 --> 10:34.000]  We've got a question already.
[10:34.000 --> 10:42.000]  So we have a hardware decoder from Mac and IOS, which is the same idea.
[10:42.000 --> 10:48.000]  CUDA, so some people use proprietary software and proprietary drivers.
[10:48.000 --> 10:57.000]  So we have now also a CUDA library so that you can insert more easily CUDA data into the streamer for encoding,
[10:57.000 --> 11:00.000]  decoding all these things.
[11:00.000 --> 11:08.000]  We have some more CUDA native elements, one that is a converter and a scaler, so using CUDA itself.
[11:08.000 --> 11:12.000]  We have CUDA and Direct 3D integration for Windows again.
[11:12.000 --> 11:19.000]  And this whole thing basically gives you zero copy and coding on NVIDIA hardware,
[11:19.000 --> 11:25.000]  especially when you match with some other CUDA-based software.
[11:25.000 --> 11:32.000]  But some people prefer free software. So we also have BAPI, we have a new plugin for BAPI.
[11:32.000 --> 11:36.000]  So we've had G-SUMOR BAPI for a long, long time.
[11:36.000 --> 11:44.000]  It was getting quite freaky. It was not based on any of the base classes that we have improved since then.
[11:44.000 --> 11:51.000]  So it's been completely rewritten from scratch with a new plugin that we call VA.
[11:51.000 --> 11:55.000]  It supports all the major codecs now that we've implemented, all the ones with VA.
[11:55.000 --> 12:00.000]  It supports AVY, it supports just 5, VPA, between 9 and 8, MPEG-2.
[12:00.000 --> 12:09.000]  Encoding, we only have the issue 6, 4, 5 codecs for now, but the rest are being worked on, as we speak.
[12:09.000 --> 12:14.000]  And using live VA, we have a bit more than VA.
[12:14.000 --> 12:18.000]  We have a bit more features. We have a compositor.
[12:18.000 --> 12:24.000]  It's an element that will take two or more streams and composite them into the same video.
[12:24.000 --> 12:27.000]  We have a D-interlacer.
[12:27.000 --> 12:33.000]  And we have a post-processor element with scaling and color space conversion,
[12:33.000 --> 12:42.000]  using the video functionality instead of the GPU.
[12:42.000 --> 12:47.000]  And open work has happened around 8 to 1 or something in the last two years.
[12:47.000 --> 12:49.000]  So we have quite a lot now.
[12:49.000 --> 12:56.000]  We have support for AV1, both in the legacy VA plugin and in the new VA plugin.
[12:56.000 --> 12:59.000]  We have it for AMD, the coders.
[12:59.000 --> 13:06.000]  We have it for Direct3D on Windows, using the NVDI-PI's.
[13:06.000 --> 13:12.000]  For Intel, using their multiple libraries, either Quixin, QSV, or the new VSDK.
[13:12.000 --> 13:20.000]  So we have pretty comprehensive AV1 support, in addition to the RTB plugin that I mentioned earlier.
[13:20.000 --> 13:29.000]  Another new thing that we've done is, this is our first official machine learning integration that is in G-streamer itself.
[13:29.000 --> 13:31.000]  So it's the first step.
[13:31.000 --> 13:44.000]  And we've written a plugin to use the Onyx runtime from Microsoft to basically take some video frames,
[13:44.000 --> 13:53.000]  some model and recognize objects, put little boxes in the metadata,
[13:53.000 --> 13:58.000]  and then another element that can take these boxes and draw them on the image.
[13:58.000 --> 14:08.000]  So this is the first step. A lot of work is happening right now to have a better video on the fixed framework as part of G-streamer.
[14:08.000 --> 14:14.000]  All these things I like about sometimes you want to have a UI.
[14:14.000 --> 14:18.000]  And a few pictures were recently added there.
[14:18.000 --> 14:23.000]  We have a GTK4 paintable, so that's an object.
[14:23.000 --> 14:28.000]  I would say that you can use the GTK4 to actually draw something on the screen.
[14:28.000 --> 14:35.000]  So now you can easily integrate G-streamer with GTK4 to your old copy playback.
[14:35.000 --> 14:38.000]  This one is in Rust, which is kind of cool.
[14:38.000 --> 14:44.000]  Qt6 as well as that thing, so that we have something that is very similar to what we have for Qt5, which is a QML item,
[14:44.000 --> 14:52.000]  so that you can integrate a G-streamer sync with the output with Qt and draw in your Qt application.
[14:52.000 --> 14:57.000]  And the last one is a niche case.
[14:57.000 --> 15:04.000]  So we had a Wayland sync for a long time, and what this Wayland sync allows you to do is to basically take the video buffer
[15:04.000 --> 15:09.000]  and give it to the Wayland compositor directly without going through the toolkit.
[15:09.000 --> 15:14.000]  So you can use the 2D hardware planes of the platform.
[15:14.000 --> 15:22.000]  This is multi-use useful and embedded. It allows you to do things like greater performance, not use the GPU on embedded systems
[15:22.000 --> 15:25.000]  where the GPU might be too slow to jail.
[15:25.000 --> 15:35.000]  Up to now, this all works fine, but you have to write a low level Wayland code, and that's non-trivial.
[15:35.000 --> 15:39.000]  So we've written a GTK widget that wraps up.
[15:39.000 --> 15:47.000]  So now we can write your application GTK, just add the widget, and you get all of these performance benefits for free.
[15:47.000 --> 15:54.000]  Last but not least, in this category, we added touch event navigation.
[15:54.000 --> 15:59.000]  So previously we had navigation, we could send letters, we could send mouse clicks,
[15:59.000 --> 16:06.000]  but now we can also send touch events so that you can have elements in your pipeline that are controlled by the user,
[16:06.000 --> 16:15.000]  such as we have a Webbar app, a web view, a webkit-based source.
[16:15.000 --> 16:22.000]  And we have some new tracers, so these are tools for developers to know actually what's going on in a pipeline live.
[16:22.000 --> 16:27.000]  We have a bunch of tracers already, four more were added.
[16:27.000 --> 16:31.000]  Some of them are quite useful, like we have one to generate buffer,
[16:31.000 --> 16:38.000]  to read the buffer lateness and one to trace the queue level, and these will output the information in a CSV file
[16:38.000 --> 16:45.000]  that you can then load and make nice graphs and understand what's the live performance of your pipeline, what's going on.
[16:45.000 --> 16:52.000]  We have one to draw pretty pipeline snapshots, so for a long time we had a feature where we could draw a dot file
[16:52.000 --> 16:56.000]  to draw a picture of the pipeline, but this required it.
[16:56.000 --> 17:03.000]  I added some code to the application to actually trigger it. With this tracer, now you can just listen to a unique signal
[17:03.000 --> 17:05.000]  and trigger it on the spot.
[17:05.000 --> 17:11.000]  The last one is the factory tracer. This is the very first feature that I mentioned.
[17:11.000 --> 17:18.000]  So it's nice to say, oh, I'm going to build a G-streamer, build specific for my application with only the elements I use.
[17:18.000 --> 17:25.000]  But if you use PlayVin, there's a lot of automated things, and you might not know exactly which plugins you've been using.
[17:25.000 --> 17:30.000]  So with this tracer, we can actually trace all the plugins that get loaded, all the elements that are used,
[17:30.000 --> 17:37.000]  and print, when you exit your application, print the list of what was actually used.
[17:37.000 --> 17:45.000]  A question about that. PlayVin sometimes tries to use PlayVin, but this got it later because it just worked.
[17:45.000 --> 17:47.000]  It's going to be listed anyway.
[17:47.000 --> 17:54.000]  So yes, right. PlayVin sometimes tries to use something, but it doesn't work because the hardware is not there or something.
[17:54.000 --> 17:58.000]  So in this case, the tracer will still list it.
[17:58.000 --> 18:04.000]  It's everything that is loaded, right? When they're tried or loaded, you call a function in it and it says no.
[18:04.000 --> 18:10.000]  So this is really at the loading stage at this place.
[18:10.000 --> 18:13.000]  Thank you. Any questions? Yes.
[18:13.000 --> 18:21.000]  You mentioned V1 and the RTP support in there. So can you also add the SPC extensions?
[18:21.000 --> 18:28.000]  I have no idea. Anyone knows? No.
[18:28.000 --> 18:33.000]  So RTP, any one extension, SVC extension in, I don't know if it's there.
[18:33.000 --> 18:41.000]  So we do layer selection of the highest quality here, but it's not there.
[18:41.000 --> 18:48.000]  So there is an external, there is a dependency description of RTP extension where you can get information about the SVC layers in there,
[18:48.000 --> 18:53.000]  which is basically something like that. So you encode the SPC layers into one screen,
[18:53.000 --> 19:01.000]  then you use these external extensions to vary information about what's in there and what isn't, so that you can do the connection.
[19:01.000 --> 19:04.000]  So that's what I got to introduce from the other video.
[19:04.000 --> 19:11.000]  None of the RTP and SPC stuff, including the VT1, which is quickly required because there's no SVC inside the screen.
[19:11.000 --> 19:21.000]  Yes, so the question was RTP, AV1, SVC, there's an extension that can make it really useful.
[19:21.000 --> 19:27.000]  It's being standardized and it's not implemented yet, but it will be at some point.
[19:27.000 --> 19:35.000]  I forgot to mention we have an online question, online question at number 43401.
[19:35.000 --> 19:43.000]  So if people at home want to ask questions, it's possible, but since we don't have any questions, there are any more questions on the floor.
[19:43.000 --> 19:51.000]  Q6, does support different rendering backends, like DirectX on Windows and Vulkan on Pino or something,
[19:51.000 --> 19:53.000]  with a QML event?
[19:53.000 --> 19:59.000]  And then Q6 supports post-interference that you can directly pass the DirectX buffer to the QML event?
[19:59.000 --> 20:11.000]  So does Q6 support other backends than OpenGL? And I think the answer right now is, won't it support OpenGL?
[20:11.000 --> 20:13.000]  Any more questions? Yes.
[20:13.000 --> 20:15.000]  Can you do a statically linked binary?
[20:15.000 --> 20:17.000]  Can you do a statically linked binary? Yes, you can.
[20:17.000 --> 20:25.000]  That's kind of one of the use cases as we create this statically linked library and then your application can link to it and only link the required bits.
[20:25.000 --> 20:29.000]  That's kind of the part of the trick to make it a little smaller.
[20:33.000 --> 20:35.000]  Yeah, my question.
[20:35.000 --> 20:39.000]  So you said that there is congestion control in WebRTC now?
[20:39.000 --> 20:41.000]  Yes.
[20:41.000 --> 20:47.000]  Is it like the same feature set as in Google's implementation?
[20:47.000 --> 20:49.000]  So, yeah.
[20:49.000 --> 20:55.000]  The question is about congestion control in WebRTC. Is it the same feature set as the Google implementation?
[20:55.000 --> 21:01.000]  As far as I know, yes, because it's basically a copy of the implementation in Rust.
[21:01.000 --> 21:05.000]  So they basically re-implement the same algorithm.
[21:05.000 --> 21:09.000]  So that is compatible.
[21:09.000 --> 21:11.000]  But it's pluggable. So you could write your own.
[21:11.000 --> 21:19.000]  There's a plug-in mechanism and the core version is in C, but this one is in Rust with a separate plug-in.
[21:19.000 --> 21:25.000]  One could write a different implementation because there's a bunch of heuristics in there.
[21:25.000 --> 21:27.000]  There's no line.
[21:27.000 --> 21:29.000]  Perfect answer.
[21:29.000 --> 21:31.000]  Thank you so much.
[21:31.000 --> 21:33.000]  Do you have a question?
[21:33.000 --> 21:35.000]  Yes.
[21:35.000 --> 21:47.000]  If I have an application that does WebRTC signaling of a matrix, for example, would I benefit from switching to WebRTC sync or would I sit with WebRTC?
[21:47.000 --> 21:57.000]  So the question is, if you have an implementation of WebRTC that does signaling something custom, for example, other matrix, would you benefit from using WebRTC sync?
[21:57.000 --> 22:08.000]  And the answer to that is yes, because it will do all of the encoding and congestion control hooked up for you.
[22:08.000 --> 22:12.000]  And there's an interface that you can implement for your own signaling.
[22:12.000 --> 22:15.000]  So the signaling is separate from this WebRTC sync.
[22:15.000 --> 22:18.000]  There's still a module that we can implement.
[22:18.000 --> 22:25.000]  We have one implemented for like a custom signaling mechanism where there's a server that's implemented, but you could write your own.
[22:25.000 --> 22:29.000]  Can I ask the last question?
[22:29.000 --> 22:31.000]  No, just a comment from the question before.
[22:31.000 --> 22:35.000]  The QT6 direct 3D integration.
[22:35.000 --> 22:37.000]  There is a merge request for it.
[22:37.000 --> 22:39.000]  Okay.
[22:39.000 --> 22:50.000]  So Tim says that for the QT, there's a direct 3D merge request open to integrate that.
[22:50.000 --> 22:55.000]  It's not merged yet, but do test it at home and complain when it doesn't work.
[22:55.000 --> 22:58.000]  Last question.
[22:58.000 --> 23:21.000]  Okay, thank you.