Hi everyone, my name is Marco. I'm one of the maintainers of
MISSERVER and today I want to talk just a little bit about the
streamcrafter which is a broadcasting studio which runs from
any modern web browser. Any questions about that?
Well, it looks like there's still a few minutes left so I'll go
ahead and give a bit more context about the streamcrafter.
First of all, this is developed by the MISSERVER team so I want to
talk just a little bit about what MISSERVER is. Then I'll move on to the
streamcrafter itself. I'll do a quick demo and talk about what will be next on the
real map. First of all, what is MISSERVER? Well, it's a media server.
It's completely open source and public domain right now and it has a very
broad support in terms of which ingest protocols, delivery protocols,
codecs, containers, remixes on the fly and it's very efficient in memory and
CPU usage. We think it's a fairly cutting edge media server so to say.
Hopefully we'll get some more contributors in the long term to work on the streamcrafter
with us because we're all backend engineers and especially making the
interface nice and snappy is not our strongest part but we're working on it.
So what's the streamcrafter? Well, let's say you are developing a
social media platform and I have to deal with a whole bunch of users who do not
know anything about codecs or configuring OBS to get a lower latency or a higher
quality. Basically, they just want to drag and drop their inputs and have a
go live button and that should be it for them. So what we want to create here is
something which is very intuitive to use and the system integrator can then set
up which kind of delivery protocol they want to use and just drop it into their
platform. It's a drop-in react component and it's also a composter of course so
the user can add cameras or screen shares and in the future we also want to add
the ability to pull in streams from this server or pull in the video and audio
feeds from a web-articity conference call so that the streamer can just
composite all of these video feeds and use the audio mixer to their liking and
just broadcast that. Cool. So this is a slightly outdated overview of how the
streamcrafter started. As you can see, it's not that complex. You have a way to
add inputs and then you have a way to mix all these inputs together and process them
at an overlay or a sound effect or whatever and then you need ways to broadcast this.
Now one thing which you don't see on this image is how do you get this media data
from the input to the compositor. So right now the default way this works is it all
happens in the main thread which is not ideal because if you have a whole bunch of
inputs it kind of slows down a bit. So we're moving to a web worker mode which is
already implemented but web APIs aren't really there just yet to make it work
really well. So at the moment what it does is the web worker will ask the main
thread for new frames because during the broadcast and then the main thread will
send back individual frames to the compositor and it works. It's not ideal
but in the future we hope to make use of the media stream track processor API which
isn't available in modern web browsers yet but it would allow you to just transfer the
entire video buffer into the compositor and then it can do all its work in a separate
thread and then broadcast that directly. So let's move on to the most exciting part
which is going to be the demo.
Let's move this a bit to the side. So as you can see the interface is not our
strongest part but it's usable so we're going to just add a scene and add a couple
of inputs. It's a bit difficult to use from... oh, that's after a good start when...
oh shit. Looks like my mouse isn't working on the big screen.
Cool, so it looks like we won't have a demo of this but feel free to visit the
website video.strong.rocks and play around with the broadcaster.
But basically you can just drop sources into the canvas and it will also have a way
larger canvas screen on your own monitor than this little screen over here and it should
stream in low latency and you can just share that link to a viewer anywhere else in the
world and they will be able to view it.
There it is.
Yeah.
Here's the inspiration here.
Oh, yeah. Thank you.
Cool. Can you add a second window and we can also put the player side by side.
I'll let you.
That's green. So second screen.
Well, let's just start streaming first then. That's fine.
Yeah, so we've added a scene. Now let's add... just add a tab screen share.
And then you just drag that on top of there.
I don't know. The scaling is a bit off but it should be better on your own monitor.
You can crop layers if you want.
Like if you only want to share this part of the screen and it will automatically fit the layer inside the...
Automatically crop the input to fit inside the layer so that people don't have to worry about stretching the input and it looks off.
Yeah, and then you just click start, share the link with your viewers and then they can view that instantly.
Cool.
Cool.
Well, it looks like it's not responding anymore.
So what's next? Well, as you can see the UI needs a bit of an oval.
It needs to scale better for mobile devices and for low resolution screens.
Secondly, a code refactor because currently it's a bit of a prototype grade code.
We want to make it extensible and easily maintainable.
We want to have a plugin feature so people can add their own processing to the video layers, for example.
And lastly, integration. Currently you can broadcast in web RTC which is fine for low latency workflows.
But maybe you want something with a bit higher quality at the cost of a bit of latency.
And so we're thinking of a tight integration with Miss Server so you can stream media data.
For example, in Matroska format using HTTP readable streams, streaming directly to Miss Server.
And that way you will get a bit higher quality without the low latency of web RTC, of course.
But it should look a bit better.
Cool. Are there any questions?
What format is the video from the RZTAP to Miss Server in this case?
If you're using web RTC it will be, sorry.
So what format is being streamed from the stream crafter to the media server?
Well, if you're using web RTC it will be Opus Audio.
So you might have to do a bit of audio transcoding to get to AAC.
If you're streaming with the other options which we'll be adding in the later dates,
you will be able to transmit in any other audio format which is supported by the browser.
It will be VP9 if you...
What's the video codec being transmitted? It will be VP9.
But this is also something which you can modify if you're using a different caption.
Regarding the video and audio, what are the limits of the codec?
What you can do with it? Is it machine specific, browser specific?
Where do the limits come from for the codecs?
So what limitations are there on the video and audio codecs?
Yeah, I think you're doing it inside a browser.
What are the limitations of that program?
Well, I think the biggest limitation is...
It depends on which browser they're using, if they're using a modern browser or an old browser,
for example, modern browsers have very wide support for basically anything you want.
Of course, anything that's happening on the main thread can get a bit slower over time.
That's why you want to move all the compositing to a separate web worker thread
to keep the main thread nice and snappy.
Because if you add lots and lots of inputs, you can notice the UI starts to slow down a little bit.
That's what we're trying to prevent there.
So in a new browser, if it has AV1 supporting the future, you'll be able to just...
Yeah.
...not AV1?
AV1 wouldn't be supported right now. I don't think it's supported at the moment, but maybe in the long term.
You're saying you will roll back-end to engineer, but this is all done in the browser, right?
This is a direct, yeah.
Well, I consider it a bit of back-end engineering because you have to transfer media data from to the web worker.
You have to overlay them a little bit of math there.
It is a little bit of back-end work, but you know, of course, the UI presenting it is all front-end work.
Is there anything you should be running on a server?
So this is all running inside the browser.
It will be cool, of course, to offload it to, for example, a mis-server and do auto-compositing in the background
because then you can maybe do a bit more fun stuff with it.
But the idea is that you can just drop this into your existing platform and your users can start going live
without any other setup required.
Is the rendering hardware accelerated or does it need to be?
This is currently not hardware accelerated now.
What framework are you using for the UI?
So what framework am I using?
Currently, it's all written in React.
We do want to put some of the processing into native JavaScript, maybe,
but currently it's all hooks and components.
I'm assuming it's open-source, but I'm just taking it out.
Can I find it?
Yes, sorry about that.
So the question is, is it open-source?
Yes, but we're still working out which exact lines we want to put out.
We do have a repository.
It's not public yet.
I was supposed to do that before the talk.
So, yeah, probably later today we'll have the GitHub repo up with the full roadmap and demo link.
How many people are working on this?
So how many people are working on this?
Currently, it's just mostly me.
It's products by the mis-server team.
And hopefully, of course, in the long term, we'd like to have more contributors
because it is an open-source project.
It would be cool if we can have other people work on some nice plugins
for more video processing or audio processing.
But this was all written by me at the moment.
Sorry, what would be the next steps to this project?
We'll be the next steps.
So we do have a full roadmap of other features we want to add.
So, of course, the UI is the most important thing to fix
because that's what the end user will be interacting with the most.
We also want to have more publishing options
because WebRTC does degrade the quality a little bit.
We want that option for the integrator to say,
you want to have the full quality and maybe at a higher latency.
And also, more input options like currently,
it only has screen shares and adding video and audio devices
accessible by the browser.
But it would be fun if you can add way more inputs than that.
And we're not really focused on having advanced editor controls
because that would be maybe too daunting for a normal person to use.
It's more about having the flexibility for the system integrator
to choose these kind of inputs, these kind of outputs.
So the question is, how is this being uploaded?
Is it being recorded or not?
This is a very broad question.
It's very broad because I do this solo, those jam sessions.
And if I can just use my phone to do it and people can watch it,
that's fine, but also I want to get the video afterwards.
Yeah. So at the moment, it does not do recording.
Of course, if you send it to a media server
because it does support any web-based media server
as well as the server-special signalling protocol.
But if you send it to a media server,
that can do the recording on the server side, of course.
But adding recording from the browser directly,
that wouldn't be too much of a feature to add, I think.
It would be nice once you add to the web map.
So thank you.
I have a couple of questions.
I don't know, should we have to...
Yeah, no, that's fine. I think we have some time.
One more question.
What is the commercial app that is similar to this?
Sorry.
What is the reason that it is similar to this?
Sorry, can you repeat the question?
The commercial app is very commercial.
But it is similar to this?
Well, of course, so is there any commercial application
currently already integrated or in the...
Is it similar to this presentation in the content?
You're talking about existing applications, which...
Yeah, so, Restream, of course, they have web-based broadcasting suite.
But they don't have the composing in the browser.
The user can choose a layout and add a few, like the cameras and stuff.
But it does look really nice.
We want to get some ideas there from the user interface.
I don't think there are many competitors in terms of what we do exactly.
Of course, you have OBS, that's a client application
which users to install and configure themselves.
Sorry.
Restream it. I have to check it out, but...
Cool.
Thank you.