I'm going to talk about WebAssembly, WebComponents, and MediaFilters all at once.
I propose to open the Web to a variety of formats.
That's it. So thank you very much.
I'm really excited to present this project today, since it's the first time I'm presenting the open source aspect of the Bevar project.
So I'm Jerome Gorin. I'm a lecturer and researcher at an engineering school, which is called Gunilla Salamiya.
I've been quite active in many open source projects at the Kniele and in Telecom on a JPEG.
But the work I'm presenting today is the fruits of the research I have conducted many years ago, ten years ago.
At this time, the proposal of my PhD was more theoretical and not practical.
But since then, there is many technologies which have been integrated inside the browser, which let us deliver all types of contents on the web.
So with my associate, Maya Bistrom, we created a company which is called Bebara.
That means in Swedish Preserve.
And with this company, we want to promote this technology and we want to speed up the adoption.
So the talk I'm going to present today is to show you the open source aspect of this project and show you how you can contribute to it.
But to dive into this project, let's start by thinking back a decade ago.
So at this time, when browser didn't include the ability to embed multimedia playback.
So at this time, you have to use plugins, web extension like Flash from Adobe and later Civilite from Microsoft to add this ability to play contents.
But this kind of extension are now used.
I mean, they face many issues like post-sale, HTML and CSS integration, security issues, accessibility issues.
So to fix this, HTML integrate new tags, which are the video tags, the audio tags to allow the browser to natively support multimedia contents.
So these new tags allow a wide variety of usages like rich internet application, like social media, video sharing.
But they also restrict the media format and content to an handful of codecs.
So there is no guarantee that a format will be supported across all the browsers.
So the formats which are supported, you can count it on the end.
It's like mp4, mp3, Flac and so on.
So on the next slide, what I show is for instance, org and Fulera are not supported across all the browsers.
And what is also concerning is that the JPEG Excel has recently been dropped by Chrome.
So what this realistically means is that a lot of people put a lot of efforts to develop useful formats, useful codecs,
but they are restricted from a wide adoption due to this gate keeping placed by the browser.
So what we are proposing is kind of the construction of this by-bill tower of an entire container and format
in order to let people freely use and develop their own formats.
So we are only using W3C standards and open source technologies to fix some of the problems of this gate keeping
and to make everyone, it make it easier to integrate like new formats to innovate, to deploy stuff
and also give the ability to support legacy formats like AC3, DNG, MKV, EPS and so on.
So let me now turn into details of our solution.
So there are two parts, there are three parts, but there are mainly two parts which are the web component and the web assembly
and we also use media filters.
So for the first part, for the web component part, we instead of creating like new scripts, creating new tag,
we just extend the usual tag, the audio tag, the video tag and so on.
We are using the attribute IS which is something standardized by the web component
and we add internal logic, we add logic to this video tag.
So this is where web components are used.
Then for the web assembly part, we are creating a new attribute which is the using attribute
on which you point to an external library which is compiled with web assembly.
So for instance, on this one, on this example, we will point to the library, I mean the libraries from the XIV foundation
which include OGV, FLAC and VORBIS to decode the input source which is in OGV and maybe in AFERA.
So if we think about it, about this solution, it can be like a bit overwhelming
because I mean we know the source, the format of the source and we include quite a lot of code on it.
So what we added as syntax is the web attribute.
So we developed what we call a solver which is based on the open source project JPEG
which has been presented many times at FOSDEM.
And this solver will be able to create a media pipeline to adapt from an input source to an expected output source.
So for instance, in this example, we have as a source an MPEG1 program stream that embed MPEG1 videos
which is mostly not supported by browser and we will transcode it to an H264 format file
which is supported by all the browser.
So what the solver will do is by itself the solver is able to check amongst all the libraries
which has been provided with the web attributes to do the transcoding, so to adapt the video to the user browser.
So in this example, we take a portion of code of LIM and MPEG1 and LIM X264 to do the transcoding dynamically.
So with this same principle, we are supporting audio and we are supporting images.
So now let's do a demo.
So I have delivered on a website so you can test it with your desktop or with your mobile phone if you just flash the code.
I have created two web pages, one with raw contents delivered on the website, on the left part.
So you will have a raw G2K image, a JPEG-G-cell image, a Dolby Digital AC-free sound and an MPEG1 video.
And on the left, we will not use the universal extension that I have presented and on the left,
which is the main page of BVRA, we will show the results with this universal extension.
So I will do the demo in live so I hope that it will work.
So if we go on Safari on the left, so this is the page without the universal extension,
what we can notice is that Safari itself is quite supporting a lot of format and container.
So it supports G2K images, it supports GXL images, it supports MPEG1 video,
but it doesn't support AC-free.
So now let's see the situation with Firefox.
So in Firefox, you can see that G2K is not supported, GXL is not supported, AC-free is not supported,
and MPEG1 is not supported.
And with Chrome, that's the same solution.
So now let's see the real main page of BVRA.com that includes this universal extension.
And so G2K is supported and if you just dive into the source code, this is the semantics I have shown to you.
So you have the ease to say that we are using the universal, we add the internal logic,
we have the solver and we are using the open source library which is OpenJPEG.
Same for GXL and on AC-free and MPEG1.
Now if I switch to Firefox, you see that you will have exactly the same result.
And on Safari, everything is as it is, but you have now the AC-free audio which is supported.
So on all the browsers, you have the same results.
So we also release an SDK in open source.
We release an IDE based on Visual Studio Code that you can use.
And the goal of this IDE is to help web developers to find the right combination of filters for a given content.
So I'm going to show you the IDE.
So to install it, it's quite simple.
You go in the store of Visual Studio Code, you type BVRA.
So on my computer, it's already installed.
And then you can open any content in the file explorer.
So for instance, let's use the GXL file.
So you have first the preview.
So it means that a library is existing to decode this content.
On the graph part, you will see the mega filter pipeline that has been used to decode this content.
You can also view the source of each filter.
So on this one, the filter is based on libgxl.
So I think the connection is quite slow.
And on this part, on the Accessor tab, then you have the script to be integrated inside your HTML.
To support this given format.
So now, I mean the connection is quite slow today.
This is the source that has been used for the graph.
So this is this one.
So I use libgxl.
And this is the code from one filter.
So as you can see, it's like using GXL.
And then you have a semantics just to adapt the open source libraries with the input format and the output format to help the server finding the right combination.
Yeah.
So we're to find us.
So this is like the end of my presentation.
At the moment, we are starting.
We are like only supported and full of media credits and container.
However, we are like adapting new libraries.
We are working on support of new formats, new type of documents, 360 videos, 3D object.
I mean, everything which is multimedia can be like constructed with the server, you know.
Everything is open source.
I mean, what I've shown to you is open source.
So you can see check the code from the editor on GitHub.
Also test interface and the SDK is on better slash filters.
Everything is in LGPL.
So you can contribute to it.
You can take this code.
You can, yeah, I mean, I've kept this decision short to have the time to let the time to you to ask a question or you can also find me later on the audience.
So that's how for me.
Any question?
Yeah.
I mean, it depends on which.
Sorry.
Okay.
So our expensive it is to transcode on the browser.
So it will really depend on the type of filter you will use.
I mean, we have some filters which are called WebCodec.
So it will use the acceleration of the browser to do the transcoating.
So by using the WebCodec, if it's supported by the browser, then it's very fast.
The other thing is that you don't always need to transcode if you are using a canvas, then you will just decode and display it on the canvas.
So it means that there is like no delay when you open a file, when you are using the canvas.
So it really depends on the complexity of the encoder.
It depends on the technology which you are using and depends on how you want to integrate your video.
But we are using WebAssembly, so there is no overhead which is imposed by WebAssembly itself.
You have quite a native performance, you know, on your browser.
Yeah.
These are static files, why do you want to transcode ahead of time?
Because you can, if it's like delivered, why don't you use static files? That's it.
So that's a good question.
I mean, why?
Because first you can adapt depending on your browser, because for instance some browser does support the native file like Jigsail, you know.
And the other thing is there is a lot of files which has like functionality in it,
functionality that are embedded in the container.
Lex takes for instance DNG files like raw files that are used by your camera, by professional photographer.
Then if you use the DNG itself, you will be able to like view the raw format.
You can view the preview and you can play with, you know, everything that you can do for instance with Photoshop, you know,
like having this high HDR range of color and so on.
And if you're on 360 videos, then you will have for instance to use the interaction.
If you're on document, you have to add this interaction.
So by playing with native file, then you will not lose any functionalities that the container has initially, you know.
Yeah.
Do you want to make a browser plugin so you can view the websites that don't have the text?
Yeah, there is also a browser.
Yes.
Do you want to do a browser plugin for that?
There is a browser plugin already that is able to detect the, if the format is supported or not for a given content.
For, I think that the best functionality yet is to trust the web developer itself because he knows like he wants to use a specific file.
He wants to use a native content so he will integrate, I mean, the functionality that he requires for his website.
If the web developer asks the user to install something on his browser, then I think that he will lose a big part of his audience.
It's better to prepare everything for the end user than ask him to install something.
You know, I don't want to come back to the situation where we were with Flash and Civil Light and have all this kind of issues.
So, I guess the web pages that still have JPEG Excel will make sense if you want to do that.
Yeah.
Can you repeat again?
Yeah, yeah, yeah, yeah, of course.
But this extension is existing, I mean, but it's like less useful than this first presentation that I'm doing, you know.
Yeah.
Is the MKB file format not supported?
It's working on it, actually.
Ah, yes.
Sorry.
Is MKB supported?
I'm currently working on it, you know, because what I'm one of the break we can have is that a lot of format has patent on it.
So what we can distribute freely is license free patents.
So MKB is one of the license free patent is you have the open source on it, the open source code to be used.
So the application is quite easy and I think that that will be my next work, you know.
Something that I forgot to present, which is quite really important, is that the plugin extension already have a store on it.
So if you want to try a combination, then you just have to add a library on it.
So for the moment we have PNG, JPEG, JXL, OpenJPEG.
We have the full FFFPEG, like a Kika.t.pmp, and then we extract some part of the FFFPEG just to reduce the size of the big project of this project.
But MKB is really like the MKB inside the XIV decoder, so I will extract it and I will work on it.
And then if you like click on add, for instance, let's say that on this one it was JXL, then I'm adding OpenJPEG on this one.
So you will see that it will be a candidate.
And on the preview it will check amongst the JXL and JPEG and see that JPEG itself is not just for me, you know.
So it's like unused.