I'm here thanks yes hello good morning and nice day for foster them hope you
had a nice evening yes let's start with the torque matrix RTC matrix beyond
instant messaging let's get that so first of all what is matrix maybe we
have some redundancy a little bit here but for the sake of the recording and
I'll repeat some of what Matthew already showed us so basically the idea is an
open and federated secure and decentralized real-time communication
network and the use cases are many for it so the obvious one is of course the
interoperable chat but we are also aiming for voice of IP applications or
even VR applications but you can also think of something like IOT data and
this is so most of those use cases you're quite familiar with and what's
the well-known event layer so for chat for instance you have a store and
forward semantics however the cool thing about matrix is that it is yeah
constructed such a way that it is de-central it is federated and what's
for me pretty cool and important is this replicated and end-to-end encrypted so
for instance if you're a client on one of those hosts here and you're attaching
to one home server and you're joining a room from another home server at that
moment the rooms replicated and everything is end-to-end encrypted it
is pretty cool it's like like a git git clone something like that similar with
an automatically your synchronization mechanism on top of it I'm quite sure
that you're very familiar with it you could also summarize it matrix is a
de-distributed real-time database and I can recommend this talk here from my
colleague Andy it's a very nice overview of what the specifics here on the
other side this talk today is about matrix RTC and now what is the
definition of matrix RTC what was it so there's basically it's the world first
de-central federated real-time channel or communication platform and it's
packed in msc341 and msc3898 and it's depicted here what basically the idea
is from a client perspective you have a peer-to-peer semantics and we don't have
a storage or persistency in it but you can exchange data with a low latency and
a low jitter from one client to the other and another interesting thing here
is that the business logic is owned by the clients though in most recent RTC
platforms or video conferencing platforms you have something like an
SFU and an application server and here the idea really is the business logic
is in the clients let's have a look at some use cases so the most obvious of
course is video conferencing right that this is what everybody would you know
assume when you talk about RTC but we can also have the embedded version here
so let's just take a normal matrix client and put a widget into it and then
have audio video conferencing or go to the VR world or the the matrix
interpretation of the metaverse which is called third room everything can be
realized using matrix RTC and the cool thing here is that you can also think
about a hybrid use case so for instance imagine we have a whiteboard like we
have this here so we can use for fast UX the matrix RTC layer such when I make
a stroke now that it's more or less immediately at the other whiteboards but
for persistency we can just use the room the distributed database so what are
the base building blocks which we use to create matrix RTC so first of all of
course we need something like a real-time communication framework and the
obvious choice at the current point in time is web RTC because it has a quite
good adoption it's in all the web browsers and there are a lot of a lot of
SDKs around we can use but going forward maybe you can also sing something
like web transport or web codex to to replace it so we are not mandating web
RTC but for the time being that is of course the framework to use then for
the signaling that's quite obvious that we use matrix for it rates and then to
make it scale you might optionally also want to use a back-end component a
focus this could either be an MCU or an SFU
so now as I stated the back-end component is optional let's have a look on
start matrix RTC without an back-end component so having a look at the
connection models so the obvious and the simplest one to start with is peer-to-peer
right that's just easier create a peer connection and then you have a they have
a data layer and then you can play around with matrix RTC it's getting a
little bit more complicated if you want to to have an RTC session with more than
two people so then we spin up a full mesh and then it really depends on the
use case how many people can join if you have a look maybe at the use case of
video conference then you would need to distribute n-1 media up links and
down links which is then limited by your internet connection and if you're
having at least in Germany an average DSL connection it would scale to up to
five to eight participants and this is something we're currently using in
an element call to element call to the current point in time is based on this
full mesh setup or was we have some news today the the vision which also really
scales and would allow a large scale RTC sessions is what we call cascaded
selective forwarding units and it's depicted here so what you would have
you would have a selective forwarding unit as long side with your home server
it's optional but you can have it and by this we then would allow a further
scaling and can really yeah using the cascading concept to dynamically crawl
to any size we have to test it but that's the theory so far the signaling
here of course is also carried out over matrix and for the for the specifics
for the SFU we have this MSE 3898 which handles then all the the magic of the
cascading and so on and so forth from a setup perspective if you want to scale
large network and large RTC network the idea is that you place the SFUs as close
as you can to your customers or to your clients and to ensure a proper internet
so this ensures a proper internet connection with low jitter low packet
loss and then you have the SFUs placed in a strong network center and have a
good interconnection and by this setup you have yeah that's that's that's the
best you can do in terms of media quality
so what we did we started with an SFU which is capable to speak the matrix
flavor of of RTC it's a prototype which was handed over from Sean from the
inventor of the pine stack and it's a Golang based web RTC implementation we
added the matrix bits to it we wrote a lot of it it's early stage but we have
support for audio channels we have support for video channels and screen
sharing and on top of that recently we also added so-called similar cast
support similar cast is you can imagine that maybe have to go step back so in
the full mesh mode you have a literally a peer-to-peer connection to each of
those clients and by adding some signaling you can the receiver can tell
the sender oh I'm struggling with a network connection can you please
adapt your your network bandwidth from from the encoding side and this is a
little bit harder if you're having a central point the SFU so here the
trick is that the sender will provide several media quality levels and then
on the SFU the client can decide which one to choose low quality mid quality and
high quality and this is called similar cast and we all already have basic
support for similar cost in our selective forwarding unit matrix is known
for privacy and end-to-end encryption and in the full mesh set up of matrix
RTC that's quite easy because you have the transport layer which ensures end-to-end
encryption but if you terminate a peer connection on an SFU the transport
layer is of course terminated and hence we need media encryption and this is the
missing part here so using insertable streams you need going forward to
implement end-to-end encryption on the client side that such that the media is
encrypted on the client send over and then it's not an issue on the SFU for
this specific topic matrix RTC and cascaded foci or a selective forwarding
units we have later a dedicated talk for my colleague Shimon it's the in-depth
talk cascaded foci it will be in this no not in this room it will be online at 2
p.m. this day
so now we have an idea of what the vision of matrix RTC is and what we can do
with it we have seen so far the use cases let's come back to element call I
think we demonstrated it last year right yes very early very early and after one
year you could imagine something happened so first recap what's what's
element call it so initially it was developed on a green field as a single
page application in the cool story here it's just a matrix client right it's not
for chat it's a matrix client and the implementation was using the full mesh so
without a back-end component so what's new so after this year first of all in
our site or not site project our partner project hydrogen we also have call
support right now and it's also working interoperability so you can start with
hydrogen the call and join this element call and the other way around we added
the SFU bits to element call and we integrated it into element web so in
element there we now have two flavors of element call we have a crew call
experience we are just press the crew call button and we have dedicated video
rooms which is pretty cool so the semantic of this room is that when you
press or click on this room you're asked to join the conference immediately
the question is how to embed element call into a matrix client in general so in
theory you could just implement the msc's but that would be very expensive
because then you need to implement in each platform expensive in terms of
engineering bandwidth so all requirements are one implementation which fits
all and the idea is yeah that we embedded element call or the embedded
element call needs to share the same underlying matrix client and room in
order to not waste resources or to have device proliferation so the idea is
quite obvious let's use a widget for it also short recap on widgets what is a
widget a widget is basically an application living in a matrix room
it's simply an embedded iframe and it's yeah a small form factor web
application html JavaScript the widget is embedded within a room and can
communicate with matrix clients and therefore from the matrix client through
the widget API and this widget API is a defined post message API basically a
widget is able to request permissions and to perform on actions on the user's
behalf something like posting into a room receiving specific event types and so
on and so forth to have a more easy a way of approaching widgets we have also
have a widget SDK which is written in JavaScript and TypeScript it's basically a
web app and now here we have an overview a few about element call in the
various flavors so in the single page application the mode it's just the
matrix client using the client server API and in the widget mode here we are
going through the widget API over the underlying matrix client and then to the
home server the abstraction layer client server API versus widget is a really
thin so from a development perspective that was really easy to implement yeah
so the solution is it's just a widget it's web RTC in a web view and this is
the nice thing so the the whole web RTC stack to implement on various platforms
is quite painful but if you can just a web view where this is included for free
that's that's a thing we needed to extend the widget API to add some missing bits
specifically where the 2d rice messages and to access the turn server also
spec in the msc's and this this whole concept we call matroshka embedding
where we have a web view hosting widgets in the various clients and that could be
a web client but it could also be the native clients like the iOS and the
intro lines and hence we have the solution one implementation that fits
all let's have a demo right
so what you see here basically is just the desktop application of element
element nightly on the left hand side you can see the various rooms with
various flavors so here this is a general room and he has already a
conference started so if you press it you will join it and at the top here we
have a so-called video room and you're now if you join it directly prompted to
to join a conference and what you can also see here is the chat so let's join
it
hello I raised the volume here so hi Enrico hi Simone yeah welcome to
foster them what we can do here right now is that we have various flavors so
we have a new layout here which we call large crits design and you can yeah
change the tile sizes and I also added here some debug information and you can
see that the small video on the top left receives the low quality stream
whereas the large screen or a large tile on the bottom receives the mid
quality stream hey Dave hello so and so and now we want to carry out an
experiment I have a QR code here hopefully I find it yeah and I encourage
everybody to join the call and to crash our SFU let's load test it sorry with
chrome
yeah yeah I will do but that's the resolution the projects are good enough
we'll put the URL in the bedroom yeah so let's see
so of course if it's not working then we can blame the Wi-Fi here right
yeah it's slightly unhappy now
no no no it's recovering it's recovering so we plan a second demo later
the day with the talk from shimon so yeah but so how many do we have it depends
okay let's say I leave it in the background play around with it and we go
back to me talk okay so what's what's next in the matrix RTC ecosystem of
course we want to implement the whiteboard at least the hybrid version
of the high pot then the very important thing really to is to yeah to start with
the insertable streams to also have the end-to-end encryption in case of using
the SFU with respect to the selective forwarding unit we need to implement the
focus selection logic so if you remember the craft where we had the other picture
where we had the the home servers alongside the SFU's alongside with the
home servers we sketched out a nice algorithm where you can automatically
choose the right one and this is something we need to implement right
now yeah and then obviously the cascading this has not be started the
implementation but yeah we need to carry out on the similar cost layer we
also have some optimizations in mind for instance I think it's only a good idea
to upload your video to the SFU if on the other side someone consumes this
video stream so this is an optimization we can add and I think also the the
switching point where when you change the layer from high quality to low
quality there's also some room for improvement on top of that we need to
care a little bit more about network bandwidth rate control this is an
important thing we need here such that the SFU also feeds back information
back to the client such that the client is then in a good position to adapt the
local yeah video audio encoders and finally we want to extend the call
support in hydrogen and that the SFU bits to it and now maybe some of you are
wondering a little bit why Matthew is sitting here with his classes yeah and
now let's head over so what we have so far seen was the
obvious use case for matrix RTC which is a video conferencing solution but I
told you earlier that we also have something like the metaverse
interpretation of matrix and we call it third room yeah and in a half minute
yeah any questions though on the floor answer yeah it's hard Q&A part
apologies for being the idiot with a VR headset on
it's gonna start streaming in a second so wanted to talk about third room which
is the social collaboration platform that we have built on top of matrix and
I'm gonna slightly mess around with trying to get this to work at the
right resolution because it's going to look crap if it's not over right
whereas how do you change the resolution these days in macOS but you know how it's actually
changes refresh rate that's not helpful that's not me yeah and is that your fault
that it's like streaming it's the wrong one thank you too many death rooms you
should play be using the same one and 2001 online death room no it's not the
online death room normal death room yes what is it
I can't spell matrix this is going well that one right thank you okay we sure
that looks totally squashed that's literally why I'm trying to fix right
now I'm gonna be to actually set a resolution on this thing these days how
about I press that button where we're gonna go to run yeah so it's gone to
four by three but it wants to do four by three everywhere but it's not four by
three it's 16 by nine let's go to that well I look thank coming in on the
stream all right just check this it'll be worth it don't worry on the plus side
the Oculus thing is cakes in good right yeah this is looking good so let me
actually bring up third room so the point of third room is that it's a tiny
team it's sorry okay hi everybody welcome to my talk on third room which is a
project tiny project done by three people Robert and a an AJ AJ also famous
for doing Sydney as matrix client which is trying to show people that matrix is
way more than chat and VoIP I know that it's called to look at 3D stuff these
days and go don't like 3D honestly I think this is incredibly interesting in
showing the potential of what we have to build on top of matrix today now the way
it works is that you've got hydrogen SDK going and basically providing a plain
old matrix client and if I jump into this room here which is hash
presentation on third room.io if people want to play along at home feel free to
come and jump in and heckle my presentation and you can see that this
is a virtual world going and sitting in browser if I pull up the frame rate which
is obviously control shift s you can see it's actually going at 60 frames a
second I'm indeed you're stuck in the floor it's running at 60 frames a
second in browser at well 1080p as we all just saw which is pretty impressive
for a fairly complicated scene that we have going on here and the way that
third room works is quite unusual in the it's properly multi-freaded in
browser it's using an entirely new game engine that the team basically put
together and I should hasten to add I've basically been encouraging people rather
than actually working on this but Robertson and Sanfran and so it'll be
cruel and unusual to get him to do this and talk and I've even got some slides
here and it's showing the scripting that is built in that I'll talk about it in
a minute now the interesting thing is that we're using shared arrayed buffers
to go and share data between the main thread and a bunch of worker threads
using post message between these and then the atomics API's and the browser so
that you can actually have properable multiple threads in order to have the
rendering thread running completely independently from the gaming thread that
does physics and the main thread that does react and does hydrogen because
with embedded hydrogen in a react app here as well as matrix so if I go to
the next slide here are so on the main threads we've got react matrix and
WebRTC happening and we have spatial audio in here so if I actually unmute
myself oh I've got first for myself on my own talk that's annoying let me pause
that amundini still out there somewhere you want to come over and say something
to me it's anybody else I'll go over to you say something yeah okay so if we had
headphones on at this point and I turn this way and you say something it's
coming out the left speaker and you have to believe me and look the other way and
it's coming out honestly it helps the immersive experience massively that we're
going using spatial audio to go and position where things are here
whilst we're wandering around here you can see that we've got at the moment
generic avatars but if you walk around a bit and you can see her moon walking
backwards for whatever reason I'm sure you can also go forwards there we go and
fly for that matter and so the B button lets you fly in this so you can go and
jump around like so and so you're spoiling my talk so if we go down here
then on the game thread we've got a bunch of rust we have the ability to run
arbitrary WebAssembly scripts which is sitting in a sandbox which allows you
to basically add any arbitrary functionality into the world from a
pure matrix perspective this is probably the most exciting thing here now if
you remember IRC and Merck scripting the ability to run arbitrary scripts on
your IRC client this is effectively allowing you to define bots and
arbitrary functionality in matrix which run inside your client inside the
sandbox and the actual data is stored in your room now this whole thing is a
matrix room right if I go and hit enter then well you can see a bunch of users
there I can say hello world and if I go back to my element client and if I
literally join presentation on thirdroom.io for matrix.org then I can you
can see I'm Dean saying yeah and I can say yeah to you too and hopefully hang
it off well I've got traffic running one way interesting well we should be
seeing messages coming into the room as well because it is there we go it's
just a plain old hydrogen overlay that is being rented in react for the
contents of the room now the actual geometry of the room if we start flying
around some more looks like this is actually a big GLTF or a single GLTF
asset this thing is just sitting in the media repository in the room it's just a
file that is GLTF the transfer format for OpenGL that has been uploaded there
and also any scripts in the room like the one which is executing the let me
press on the buttons here again there's a bit of I think JavaScript using the
quick.js engine that has gone and compiled down the JavaScript to web
assembly in real-time it's pretty cool that you literally write it in JavaScript
and then the engine sucks it up turns it into wasm and runs it within that
sandbox so you could argue it's a little bit perverse to be taking JavaScript
compiling it to web assembly and then running it from within a JavaScript
environment but it gives you a hell of a lot more safety than you would if we were
just I know having random blobs of JavaScript running here on the render
thread we are using WebGL2 and we're using Freejs to manage the actual
driving of WebGL but the scene itself is the scene itself is managed using a
really cool technology called Bit ECS that was actually created by Nate one
of the developers before he started working on they before they started
working on third room and Bit ECS is a entity component system where you
basically trap the state of the world the objects that exist within it their
transformations and it's done with arrays in JavaScript and it turns out that if
you stretch your arrays intelligently enough in JavaScript you can get as
good as web assembly performance and it's one of the other secrets to the
crazy and performance that we have here so this isn't a scene graph API under
the hood like a frame if anybody ever played with a frame instead it's using
the Bit ECS then another thing which is interesting here is that everything is
some triple buffered so and a kind of traditional game engine you just have
one sort of buffer that you write data into and the renderer reads it out and
you have some kind of locking system to make sure that it doesn't collide
whereas here we have lot free data structures letting things go as rapidly
as possible with the various different bits of the engine writing into these
shared this shared triple buffer as a shared array and buffer which is then
juggled effectively between the various different threads and it means that the
render thread can run at the native speed of whatever device which is
particularly useful if it's a less powered device than my MacBook Pro here
and then the game engine that is actually rendering what's going on can
go at its own speed so you totally decouple the two and you get as high a
frame rate as you can and I think that oh yeah and finally lots of fun stuff
going on with asset optimization pipeline particularly the textures have
been highly compressed using these fun codecs I think it's called universal
basis format from binomial and KTX and one of the things we've done to cheat
to bootstrap third room is to build a pipeline from Unity where you can take
existing Unity assets like this scene here is one that we bought off a Unity
assets store and then export it as proper open standardized JLTF somewhat
liberating the contents from the slightly proprietary world of Unity in
order to get content in more rapidly and then compress it down and there are
lots of fun things like the it has instancing support built in so if you
start generating lots of objects like the physics engine here I know going
create a whole bunch of objects and attack the various people who are
wandering around in here I'm sure they love me for it and then this is
basically just the same JLTF asset going and being created multiple times all
the textures are sprighted so there's just one great big thing there's also
some really interesting extensions to GL that we've contributed by the
Kronos group and particularly if you look at if we grab one of these
mirror balls which are mainly used for debugging purposes let's grab that one
and if I run around with it while I fly around with it you should see that the
reflection changes there we go like if I go between zones there it's not they
need to be tuned a bit but basically rather than ray tracing which would be
incredibly time-consuming instead we have lots of different probes hanging
around the scene that allow you to so I'm hitting myself in the face with the
ball a common problem but as you run around you can see the reflection
changes it's pretty nasty if you do it rapidly but if you're doing it more
slowly like this and it's a quite a subtle but nice effect and it's even
better when it's on not perfect maribles if you look at say Dave if you walk
backwards if you can hear me or go into the light or out of the light then you
actually see a fairly subtle shadowing effect as it's gone and figured out where
the shadows are there right guys cool to see all the people running around in
here so what else can I show you one of the so we're gonna launch some tech
preview 2 of this next week and this is the first time anybody has seen tech
preview 2 and I have 10 minutes yet left on and some tech preview one is sort of
what we've been looking at here except it didn't have scripting we've already
shown some of the scripting here but one of the big things sort of being added
and let's pray that this thing works is web VR so hopefully if I go to the
Oculus streaming thing which I had a second ago should have possibly cleaned
this up first of course it stopped working well I get this thing back in
again and everything is going wrong apparently I've got to recalibrate the
entire thing so I apologize for using proprietary technology but
unfortunately there aren't any open source headsets which do the trick yet
there we go and let me try to cast this up and unfortunately it takes ages for
the screen casting to kick in for some reason but I'll go as quick as I can
bump and cost and computer go right so WebXR is a really cool technology it's
been there for ages now since about 2017 built into browsers like Firefox and
Chrome obviously and also interestingly the browser that your Oculus and Quest
like this or Quest Pro has built into it which is based on Chromium it has
awful screen casting support as you can see and that I started at screen
casting and something is happening in the depths of Facebook trying to figure
out how to actually get this onto the screen here but hopefully it will come
through assuming I've got internet connectivity here it is thank God and I
can start talking and I apologize for I'm going to focus on very interesting
so I'm going to focus on Flora in rather than embarrassing everybody else but
let's just use a stationary boundary confirm right so the browser here sits
there I'm not going to update this right now but here is some third room and if I
continue into third room as guests you can see this is just a static boring
old web browser just sitting here worth noting that third room uses OIDC entirely
so this thing here is actually a skinned key cloak I'm gonna say I'm not a bot
I'm not gonna bother giving myself a name and then capture failed brilliant
thanks Google I'm gonna have to type in third room except caffeine and stress
means my ability to use a stupid keyboard like this is gonna be fun okay
about to third room here is a streaming okay okay brilliant let's go to login go
about third room continuous guests this time hope that it's not gonna make me
pick stupid things right good continue accept the T's and C's honestly the
using a key cloak for this is really really fun and very anticlimatically we
end up with eventually monthly connectivity a 2d version of third
room just sitting right here so isn't it amazing by the way that just loaded
from index DB local storage but the fun thing is hopefully come on you can do
it you can see it's actually struggling quite a lot in this but if I press the
old X button there that's I have to close the welcome to third room dialogue
here it is come on and to XR thank God for that then I can see Florian
hello Florian but more excitingly hopefully if I stay in the right place
there we go you can see that I'm actually in the third room environment
right now and this is genuinely cool this is running at 90 frames a second for
me right now and if I go and press some buttons to crates and oh god crates like
that massive crate let me get rid that you can see it's actually hooked up to
the normal physics engine so I can go and pull that and I can confusingly throw
it into the audience which is no way surreal to be going and flipping back
and forth and then back in the normal world again at the moment we've just
got basic things like the joystick and hung up hooked up to it it's got a
kinematic controller and I'm I running out of oh no thank you Florian what else
can I show you if you can jump we can spawn more objects I can go after that
glitter globe and sorry Mirable except is running faster than I can run after it
that's awkward and theoretically if I was a little bit close to the bloody thing
I'd be able to grab it and pick it up etc so this is pretty cool honestly it's
as good as the native non-web VR and closed stuff that Facebook or
meta horizons does and the entire thing is open and they built on the surgery
mentioned how am I doing for time three minutes in which case I very quickly go
and sorry thanks for I will stop looking at random emissives go back into this
and just look at some other things we've done in fact this one is really cool
let's just flip into this one because this is a really complicated bit of
wasm it's actually an audio reactive widget which is sitting here as you can
see as I yell at it it goes and changes size and this is a whole bunch of C code
that has been compiled down to wasm to show how you can have interactive things
sitting inside the scene another example is slightly less excitingly a chat box
echo service so if I go into here and say hello and then do echo hello with a
slash command it says hello back to me now the echo that's happening down there
is actually being done from the widget API of matrix going into web assembly
talking to I think a JavaScript service and then echoes back and you can see we
have a slight bug sometimes with scripting where it loads two worlds at
the same time and that's pretty surreal and that's what happens if London got
combined with Mars final thing here I was going to show you oh yeah is this guy
which is a bit silly but fun anyway and this time I'll remember to refresh as
fun as it is to have the scenes combined so you might recognize this from a
certain film and if we actually look at the script for this particular room if
I could figure out how to get out of fullscreen mode the script here is again
just sitting in the media repository it's a little bit of JavaScript and to use
the web scene graph API which is a new API that we've created we hope it will
become a W3 standard for manipulating scene graphs it looks a lot like Dom you
basically get a node by name the TV and then every frame you'd see if the TV is
being pressed and if it is then you enable the matrix material and the end
result is if I go here and I click on the TV then predictably enough you end up
in a matrix style world and I've got it in third-person view which is also new
with tech preview 2 and clay back and forth this is super early but you can
imagine this is basically a platform for doing any kind of real-time
collaborative app could be figma on this it could be multiplayer blender it
could be a game it could be digital twins it could be I know smart cities GIS
applications it's as powerful and inslexible as the web but for real-time
thank you very much and we have no time for questions
one question
oh
it will be coming in the next release after tech preview 2 I mean the hub bit
of actually rigging up the engine and doing all the inverse kinematics and if
you run around with the current avatars you can now trip over things which is
very important but suffice to say we're focusing on the engine rather than the
assets and also this is at a point where people can start contributing things so
if you've got amazing assets if they support the maximum rig then they
should just drop straight in cool