[00:00.000 --> 00:20.440]  I'm here thanks yes hello good morning and nice day for foster them hope you
[00:20.440 --> 00:28.640]  had a nice evening yes let's start with the torque matrix RTC matrix beyond
[00:28.640 --> 00:36.240]  instant messaging let's get that so first of all what is matrix maybe we
[00:36.240 --> 00:39.840]  have some redundancy a little bit here but for the sake of the recording and
[00:39.840 --> 00:45.760]  I'll repeat some of what Matthew already showed us so basically the idea is an
[00:45.760 --> 00:50.840]  open and federated secure and decentralized real-time communication
[00:50.840 --> 00:56.040]  network and the use cases are many for it so the obvious one is of course the
[00:56.040 --> 01:00.840]  interoperable chat but we are also aiming for voice of IP applications or
[01:00.840 --> 01:08.520]  even VR applications but you can also think of something like IOT data and
[01:08.520 --> 01:12.840]  this is so most of those use cases you're quite familiar with and what's
[01:12.840 --> 01:17.200]  the well-known event layer so for chat for instance you have a store and
[01:17.200 --> 01:23.480]  forward semantics however the cool thing about matrix is that it is yeah
[01:23.480 --> 01:29.440]  constructed such a way that it is de-central it is federated and what's
[01:29.440 --> 01:34.600]  for me pretty cool and important is this replicated and end-to-end encrypted so
[01:34.600 --> 01:39.160]  for instance if you're a client on one of those hosts here and you're attaching
[01:39.160 --> 01:42.760]  to one home server and you're joining a room from another home server at that
[01:42.760 --> 01:46.360]  moment the rooms replicated and everything is end-to-end encrypted it
[01:46.360 --> 01:50.360]  is pretty cool it's like like a git git clone something like that similar with
[01:50.360 --> 01:55.800]  an automatically your synchronization mechanism on top of it I'm quite sure
[01:55.800 --> 01:59.120]  that you're very familiar with it you could also summarize it matrix is a
[01:59.120 --> 02:03.800]  de-distributed real-time database and I can recommend this talk here from my
[02:03.800 --> 02:11.120]  colleague Andy it's a very nice overview of what the specifics here on the
[02:11.120 --> 02:15.320]  other side this talk today is about matrix RTC and now what is the
[02:15.320 --> 02:21.520]  definition of matrix RTC what was it so there's basically it's the world first
[02:21.520 --> 02:26.360]  de-central federated real-time channel or communication platform and it's
[02:26.360 --> 02:32.600]  packed in msc341 and msc3898 and it's depicted here what basically the idea
[02:32.600 --> 02:37.720]  is from a client perspective you have a peer-to-peer semantics and we don't have
[02:37.720 --> 02:44.280]  a storage or persistency in it but you can exchange data with a low latency and
[02:44.280 --> 02:50.160]  a low jitter from one client to the other and another interesting thing here
[02:50.160 --> 02:56.960]  is that the business logic is owned by the clients though in most recent RTC
[02:56.960 --> 03:01.080]  platforms or video conferencing platforms you have something like an
[03:01.080 --> 03:05.200]  SFU and an application server and here the idea really is the business logic
[03:05.200 --> 03:13.280]  is in the clients let's have a look at some use cases so the most obvious of
[03:13.280 --> 03:17.600]  course is video conferencing right that this is what everybody would you know
[03:17.600 --> 03:22.480]  assume when you talk about RTC but we can also have the embedded version here
[03:22.480 --> 03:26.920]  so let's just take a normal matrix client and put a widget into it and then
[03:26.920 --> 03:32.200]  have audio video conferencing or go to the VR world or the the matrix
[03:32.200 --> 03:36.440]  interpretation of the metaverse which is called third room everything can be
[03:36.440 --> 03:43.520]  realized using matrix RTC and the cool thing here is that you can also think
[03:43.520 --> 03:47.040]  about a hybrid use case so for instance imagine we have a whiteboard like we
[03:47.040 --> 03:53.600]  have this here so we can use for fast UX the matrix RTC layer such when I make
[03:53.600 --> 03:57.440]  a stroke now that it's more or less immediately at the other whiteboards but
[03:57.440 --> 04:06.320]  for persistency we can just use the room the distributed database so what are
[04:06.320 --> 04:11.200]  the base building blocks which we use to create matrix RTC so first of all of
[04:11.200 --> 04:15.720]  course we need something like a real-time communication framework and the
[04:15.720 --> 04:19.880]  obvious choice at the current point in time is web RTC because it has a quite
[04:19.880 --> 04:23.360]  good adoption it's in all the web browsers and there are a lot of a lot of
[04:23.360 --> 04:27.800]  SDKs around we can use but going forward maybe you can also sing something
[04:27.800 --> 04:33.040]  like web transport or web codex to to replace it so we are not mandating web
[04:33.040 --> 04:38.200]  RTC but for the time being that is of course the framework to use then for
[04:38.200 --> 04:43.200]  the signaling that's quite obvious that we use matrix for it rates and then to
[04:43.200 --> 04:47.560]  make it scale you might optionally also want to use a back-end component a
[04:47.560 --> 04:52.840]  focus this could either be an MCU or an SFU
[04:56.320 --> 05:01.800]  so now as I stated the back-end component is optional let's have a look on
[05:01.800 --> 05:06.840]  start matrix RTC without an back-end component so having a look at the
[05:06.840 --> 05:11.160]  connection models so the obvious and the simplest one to start with is peer-to-peer
[05:11.160 --> 05:15.000]  right that's just easier create a peer connection and then you have a they have
[05:15.000 --> 05:19.800]  a data layer and then you can play around with matrix RTC it's getting a
[05:19.800 --> 05:25.320]  little bit more complicated if you want to to have an RTC session with more than
[05:25.320 --> 05:29.920]  two people so then we spin up a full mesh and then it really depends on the
[05:29.920 --> 05:36.160]  use case how many people can join if you have a look maybe at the use case of
[05:36.160 --> 05:41.480]  video conference then you would need to distribute n-1 media up links and
[05:41.480 --> 05:46.320]  down links which is then limited by your internet connection and if you're
[05:46.320 --> 05:50.960]  having at least in Germany an average DSL connection it would scale to up to
[05:50.960 --> 05:55.480]  five to eight participants and this is something we're currently using in
[05:55.480 --> 06:00.480]  an element call to element call to the current point in time is based on this
[06:00.480 --> 06:08.000]  full mesh setup or was we have some news today the the vision which also really
[06:08.000 --> 06:14.800]  scales and would allow a large scale RTC sessions is what we call cascaded
[06:14.800 --> 06:18.960]  selective forwarding units and it's depicted here so what you would have
[06:18.960 --> 06:23.840]  you would have a selective forwarding unit as long side with your home server
[06:23.840 --> 06:32.480]  it's optional but you can have it and by this we then would allow a further
[06:32.480 --> 06:38.720]  scaling and can really yeah using the cascading concept to dynamically crawl
[06:38.720 --> 06:46.760]  to any size we have to test it but that's the theory so far the signaling
[06:46.760 --> 06:52.040]  here of course is also carried out over matrix and for the for the specifics
[06:52.040 --> 06:59.240]  for the SFU we have this MSE 3898 which handles then all the the magic of the
[06:59.240 --> 07:05.920]  cascading and so on and so forth from a setup perspective if you want to scale
[07:05.920 --> 07:11.000]  large network and large RTC network the idea is that you place the SFUs as close
[07:11.000 --> 07:16.520]  as you can to your customers or to your clients and to ensure a proper internet
[07:16.520 --> 07:20.960]  so this ensures a proper internet connection with low jitter low packet
[07:20.960 --> 07:26.320]  loss and then you have the SFUs placed in a strong network center and have a
[07:26.320 --> 07:31.240]  good interconnection and by this setup you have yeah that's that's that's the
[07:31.240 --> 07:36.400]  best you can do in terms of media quality
[07:38.480 --> 07:46.560]  so what we did we started with an SFU which is capable to speak the matrix
[07:46.560 --> 07:55.080]  flavor of of RTC it's a prototype which was handed over from Sean from the
[07:55.080 --> 08:00.760]  inventor of the pine stack and it's a Golang based web RTC implementation we
[08:00.760 --> 08:06.320]  added the matrix bits to it we wrote a lot of it it's early stage but we have
[08:06.320 --> 08:10.240]  support for audio channels we have support for video channels and screen
[08:10.240 --> 08:14.760]  sharing and on top of that recently we also added so-called similar cast
[08:14.760 --> 08:21.840]  support similar cast is you can imagine that maybe have to go step back so in
[08:21.840 --> 08:25.680]  the full mesh mode you have a literally a peer-to-peer connection to each of
[08:25.680 --> 08:31.800]  those clients and by adding some signaling you can the receiver can tell
[08:31.800 --> 08:35.520]  the sender oh I'm struggling with a network connection can you please
[08:35.520 --> 08:40.040]  adapt your your network bandwidth from from the encoding side and this is a
[08:40.040 --> 08:45.200]  little bit harder if you're having a central point the SFU so here the
[08:45.200 --> 08:50.720]  trick is that the sender will provide several media quality levels and then
[08:50.720 --> 08:55.520]  on the SFU the client can decide which one to choose low quality mid quality and
[08:55.520 --> 09:00.000]  high quality and this is called similar cast and we all already have basic
[09:00.000 --> 09:09.120]  support for similar cost in our selective forwarding unit matrix is known
[09:09.120 --> 09:14.320]  for privacy and end-to-end encryption and in the full mesh set up of matrix
[09:14.320 --> 09:18.040]  RTC that's quite easy because you have the transport layer which ensures end-to-end
[09:18.040 --> 09:24.440]  encryption but if you terminate a peer connection on an SFU the transport
[09:24.440 --> 09:30.400]  layer is of course terminated and hence we need media encryption and this is the
[09:30.400 --> 09:35.120]  missing part here so using insertable streams you need going forward to
[09:35.120 --> 09:39.880]  implement end-to-end encryption on the client side that such that the media is
[09:39.880 --> 09:47.440]  encrypted on the client send over and then it's not an issue on the SFU for
[09:47.440 --> 09:53.200]  this specific topic matrix RTC and cascaded foci or a selective forwarding
[09:53.200 --> 09:59.840]  units we have later a dedicated talk for my colleague Shimon it's the in-depth
[09:59.840 --> 10:06.440]  talk cascaded foci it will be in this no not in this room it will be online at 2
[10:06.440 --> 10:09.400]  p.m. this day
[10:12.040 --> 10:17.440]  so now we have an idea of what the vision of matrix RTC is and what we can do
[10:17.440 --> 10:23.160]  with it we have seen so far the use cases let's come back to element call I
[10:23.160 --> 10:32.280]  think we demonstrated it last year right yes very early very early and after one
[10:32.280 --> 10:38.240]  year you could imagine something happened so first recap what's what's
[10:38.240 --> 10:42.680]  element call it so initially it was developed on a green field as a single
[10:42.680 --> 10:47.440]  page application in the cool story here it's just a matrix client right it's not
[10:47.440 --> 10:52.080]  for chat it's a matrix client and the implementation was using the full mesh so
[10:52.080 --> 10:56.600]  without a back-end component so what's new so after this year first of all in
[10:56.600 --> 11:01.400]  our site or not site project our partner project hydrogen we also have call
[11:01.400 --> 11:05.080]  support right now and it's also working interoperability so you can start with
[11:05.080 --> 11:09.480]  hydrogen the call and join this element call and the other way around we added
[11:09.480 --> 11:15.800]  the SFU bits to element call and we integrated it into element web so in
[11:15.800 --> 11:20.240]  element there we now have two flavors of element call we have a crew call
[11:20.240 --> 11:24.280]  experience we are just press the crew call button and we have dedicated video
[11:24.280 --> 11:28.240]  rooms which is pretty cool so the semantic of this room is that when you
[11:28.240 --> 11:34.880]  press or click on this room you're asked to join the conference immediately
[11:37.640 --> 11:44.720]  the question is how to embed element call into a matrix client in general so in
[11:44.720 --> 11:49.080]  theory you could just implement the msc's but that would be very expensive
[11:49.080 --> 11:53.520]  because then you need to implement in each platform expensive in terms of
[11:53.520 --> 11:59.800]  engineering bandwidth so all requirements are one implementation which fits
[11:59.800 --> 12:07.000]  all and the idea is yeah that we embedded element call or the embedded
[12:07.000 --> 12:11.160]  element call needs to share the same underlying matrix client and room in
[12:11.160 --> 12:17.400]  order to not waste resources or to have device proliferation so the idea is
[12:17.400 --> 12:23.960]  quite obvious let's use a widget for it also short recap on widgets what is a
[12:23.960 --> 12:28.360]  widget a widget is basically an application living in a matrix room
[12:28.360 --> 12:35.120]  it's simply an embedded iframe and it's yeah a small form factor web
[12:35.120 --> 12:44.920]  application html JavaScript the widget is embedded within a room and can
[12:44.920 --> 12:48.520]  communicate with matrix clients and therefore from the matrix client through
[12:48.520 --> 12:56.920]  the widget API and this widget API is a defined post message API basically a
[12:56.920 --> 13:04.760]  widget is able to request permissions and to perform on actions on the user's
[13:04.760 --> 13:10.080]  behalf something like posting into a room receiving specific event types and so
[13:10.080 --> 13:17.520]  on and so forth to have a more easy a way of approaching widgets we have also
[13:17.520 --> 13:24.400]  have a widget SDK which is written in JavaScript and TypeScript it's basically a
[13:24.400 --> 13:30.000]  web app and now here we have an overview a few about element call in the
[13:30.000 --> 13:34.280]  various flavors so in the single page application the mode it's just the
[13:34.280 --> 13:38.920]  matrix client using the client server API and in the widget mode here we are
[13:38.920 --> 13:42.520]  going through the widget API over the underlying matrix client and then to the
[13:42.520 --> 13:48.640]  home server the abstraction layer client server API versus widget is a really
[13:48.640 --> 13:56.760]  thin so from a development perspective that was really easy to implement yeah
[13:56.760 --> 14:01.280]  so the solution is it's just a widget it's web RTC in a web view and this is
[14:01.280 --> 14:06.200]  the nice thing so the the whole web RTC stack to implement on various platforms
[14:06.200 --> 14:10.280]  is quite painful but if you can just a web view where this is included for free
[14:10.280 --> 14:16.040]  that's that's a thing we needed to extend the widget API to add some missing bits
[14:16.040 --> 14:21.440]  specifically where the 2d rice messages and to access the turn server also
[14:21.440 --> 14:26.840]  spec in the msc's and this this whole concept we call matroshka embedding
[14:26.840 --> 14:30.760]  where we have a web view hosting widgets in the various clients and that could be
[14:30.760 --> 14:34.640]  a web client but it could also be the native clients like the iOS and the
[14:34.640 --> 14:39.240]  intro lines and hence we have the solution one implementation that fits
[14:39.240 --> 14:48.120]  all let's have a demo right
[14:54.720 --> 15:00.560]  so what you see here basically is just the desktop application of element
[15:00.560 --> 15:05.200]  element nightly on the left hand side you can see the various rooms with
[15:05.200 --> 15:09.600]  various flavors so here this is a general room and he has already a
[15:09.600 --> 15:14.360]  conference started so if you press it you will join it and at the top here we
[15:14.360 --> 15:19.040]  have a so-called video room and you're now if you join it directly prompted to
[15:19.040 --> 15:27.480]  to join a conference and what you can also see here is the chat so let's join
[15:27.480 --> 15:33.200]  it
[15:33.200 --> 15:42.400]  hello I raised the volume here so hi Enrico hi Simone yeah welcome to
[15:42.400 --> 15:50.960]  foster them what we can do here right now is that we have various flavors so
[15:50.960 --> 15:58.160]  we have a new layout here which we call large crits design and you can yeah
[15:58.160 --> 16:03.440]  change the tile sizes and I also added here some debug information and you can
[16:03.440 --> 16:07.880]  see that the small video on the top left receives the low quality stream
[16:07.880 --> 16:16.120]  whereas the large screen or a large tile on the bottom receives the mid
[16:16.120 --> 16:31.160]  quality stream hey Dave hello so and so and now we want to carry out an
[16:31.160 --> 16:42.840]  experiment I have a QR code here hopefully I find it yeah and I encourage
[16:42.840 --> 16:52.280]  everybody to join the call and to crash our SFU let's load test it sorry with
[16:52.280 --> 16:56.600]  chrome
[16:56.600 --> 17:10.960]  yeah yeah I will do but that's the resolution the projects are good enough
[17:10.960 --> 17:21.080]  we'll put the URL in the bedroom yeah so let's see
[17:21.080 --> 17:41.880]  so of course if it's not working then we can blame the Wi-Fi here right
[17:41.880 --> 17:53.760]  yeah it's slightly unhappy now
[17:53.760 --> 18:22.400]  no no no it's recovering it's recovering so we plan a second demo later
[18:22.400 --> 18:39.680]  the day with the talk from shimon so yeah but so how many do we have it depends
[18:39.680 --> 18:56.080]  okay let's say I leave it in the background play around with it and we go
[18:56.080 --> 19:13.640]  back to me talk okay so what's what's next in the matrix RTC ecosystem of
[19:13.640 --> 19:16.320]  course we want to implement the whiteboard at least the hybrid version
[19:16.320 --> 19:22.320]  of the high pot then the very important thing really to is to yeah to start with
[19:22.320 --> 19:28.040]  the insertable streams to also have the end-to-end encryption in case of using
[19:28.040 --> 19:35.360]  the SFU with respect to the selective forwarding unit we need to implement the
[19:35.360 --> 19:40.960]  focus selection logic so if you remember the craft where we had the other picture
[19:40.960 --> 19:46.080]  where we had the the home servers alongside the SFU's alongside with the
[19:46.080 --> 19:49.800]  home servers we sketched out a nice algorithm where you can automatically
[19:49.800 --> 19:53.520]  choose the right one and this is something we need to implement right
[19:53.520 --> 19:57.840]  now yeah and then obviously the cascading this has not be started the
[19:57.840 --> 20:03.720]  implementation but yeah we need to carry out on the similar cost layer we
[20:03.720 --> 20:10.120]  also have some optimizations in mind for instance I think it's only a good idea
[20:10.120 --> 20:14.720]  to upload your video to the SFU if on the other side someone consumes this
[20:14.720 --> 20:20.240]  video stream so this is an optimization we can add and I think also the the
[20:20.240 --> 20:23.960]  switching point where when you change the layer from high quality to low
[20:23.960 --> 20:29.680]  quality there's also some room for improvement on top of that we need to
[20:29.680 --> 20:33.760]  care a little bit more about network bandwidth rate control this is an
[20:33.760 --> 20:39.560]  important thing we need here such that the SFU also feeds back information
[20:39.560 --> 20:44.400]  back to the client such that the client is then in a good position to adapt the
[20:44.400 --> 20:51.360]  local yeah video audio encoders and finally we want to extend the call
[20:51.360 --> 20:59.640]  support in hydrogen and that the SFU bits to it and now maybe some of you are
[20:59.640 --> 21:04.200]  wondering a little bit why Matthew is sitting here with his classes yeah and
[21:04.200 --> 21:14.080]  now let's head over so what we have so far seen was the
[21:14.080 --> 21:18.720]  obvious use case for matrix RTC which is a video conferencing solution but I
[21:18.720 --> 21:22.480]  told you earlier that we also have something like the metaverse
[21:22.480 --> 21:30.040]  interpretation of matrix and we call it third room yeah and in a half minute
[21:30.040 --> 21:45.680]  yeah any questions though on the floor answer yeah it's hard Q&A part
[21:45.680 --> 21:50.640]  apologies for being the idiot with a VR headset on
[21:50.640 --> 21:56.360]  it's gonna start streaming in a second so wanted to talk about third room which
[21:56.360 --> 22:02.200]  is the social collaboration platform that we have built on top of matrix and
[22:02.200 --> 22:08.840]  I'm gonna slightly mess around with trying to get this to work at the
[22:08.840 --> 22:13.000]  right resolution because it's going to look crap if it's not over right
[22:13.000 --> 22:24.520]  whereas how do you change the resolution these days in macOS but you know how it's actually
[22:24.520 --> 22:52.120]  changes refresh rate that's not helpful that's not me yeah and is that your fault
[22:52.120 --> 22:58.000]  that it's like streaming it's the wrong one thank you too many death rooms you
[22:58.000 --> 23:05.440]  should play be using the same one and 2001 online death room no it's not the
[23:05.440 --> 23:11.880]  online death room normal death room yes what is it
[23:11.880 --> 23:20.920]  I can't spell matrix this is going well that one right thank you okay we sure
[23:20.920 --> 23:24.360]  that looks totally squashed that's literally why I'm trying to fix right
[23:24.360 --> 23:36.040]  now I'm gonna be to actually set a resolution on this thing these days how
[23:36.040 --> 23:44.160]  about I press that button where we're gonna go to run yeah so it's gone to
[23:44.160 --> 23:47.240]  four by three but it wants to do four by three everywhere but it's not four by
[23:47.240 --> 23:54.520]  three it's 16 by nine let's go to that well I look thank coming in on the
[23:54.520 --> 24:04.120]  stream all right just check this it'll be worth it don't worry on the plus side
[24:04.120 --> 24:14.320]  the Oculus thing is cakes in good right yeah this is looking good so let me
[24:14.320 --> 24:20.480]  actually bring up third room so the point of third room is that it's a tiny
[24:20.480 --> 24:36.280]  team it's sorry okay hi everybody welcome to my talk on third room which is a
[24:36.280 --> 24:42.520]  project tiny project done by three people Robert and a an AJ AJ also famous
[24:42.520 --> 24:48.000]  for doing Sydney as matrix client which is trying to show people that matrix is
[24:48.000 --> 24:52.720]  way more than chat and VoIP I know that it's called to look at 3D stuff these
[24:52.720 --> 24:57.920]  days and go don't like 3D honestly I think this is incredibly interesting in
[24:57.920 --> 25:02.200]  showing the potential of what we have to build on top of matrix today now the way
[25:02.200 --> 25:07.760]  it works is that you've got hydrogen SDK going and basically providing a plain
[25:07.760 --> 25:14.040]  old matrix client and if I jump into this room here which is hash
[25:14.040 --> 25:19.440]  presentation on third room.io if people want to play along at home feel free to
[25:19.440 --> 25:24.480]  come and jump in and heckle my presentation and you can see that this
[25:24.480 --> 25:30.080]  is a virtual world going and sitting in browser if I pull up the frame rate which
[25:30.080 --> 25:33.560]  is obviously control shift s you can see it's actually going at 60 frames a
[25:33.560 --> 25:39.240]  second I'm indeed you're stuck in the floor it's running at 60 frames a
[25:39.240 --> 25:45.440]  second in browser at well 1080p as we all just saw which is pretty impressive
[25:45.440 --> 25:51.680]  for a fairly complicated scene that we have going on here and the way that
[25:51.680 --> 25:57.760]  third room works is quite unusual in the it's properly multi-freaded in
[25:57.760 --> 26:02.680]  browser it's using an entirely new game engine that the team basically put
[26:02.680 --> 26:06.640]  together and I should hasten to add I've basically been encouraging people rather
[26:06.640 --> 26:11.200]  than actually working on this but Robertson and Sanfran and so it'll be
[26:11.200 --> 26:17.280]  cruel and unusual to get him to do this and talk and I've even got some slides
[26:17.280 --> 26:20.400]  here and it's showing the scripting that is built in that I'll talk about it in
[26:20.400 --> 26:24.960]  a minute now the interesting thing is that we're using shared arrayed buffers
[26:24.960 --> 26:29.920]  to go and share data between the main thread and a bunch of worker threads
[26:29.920 --> 26:34.160]  using post message between these and then the atomics API's and the browser so
[26:34.160 --> 26:37.600]  that you can actually have properable multiple threads in order to have the
[26:37.600 --> 26:41.840]  rendering thread running completely independently from the gaming thread that
[26:41.840 --> 26:45.920]  does physics and the main thread that does react and does hydrogen because
[26:45.920 --> 26:53.840]  with embedded hydrogen in a react app here as well as matrix so if I go to
[26:53.840 --> 26:59.200]  the next slide here are so on the main threads we've got react matrix and
[26:59.200 --> 27:03.880]  WebRTC happening and we have spatial audio in here so if I actually unmute
[27:03.880 --> 27:10.240]  myself oh I've got first for myself on my own talk that's annoying let me pause
[27:10.240 --> 27:14.480]  that amundini still out there somewhere you want to come over and say something
[27:14.480 --> 27:20.560]  to me it's anybody else I'll go over to you say something yeah okay so if we had
[27:20.560 --> 27:25.520]  headphones on at this point and I turn this way and you say something it's
[27:25.520 --> 27:28.720]  coming out the left speaker and you have to believe me and look the other way and
[27:28.720 --> 27:32.520]  it's coming out honestly it helps the immersive experience massively that we're
[27:32.520 --> 27:36.400]  going using spatial audio to go and position where things are here
[27:36.400 --> 27:40.000]  whilst we're wandering around here you can see that we've got at the moment
[27:40.000 --> 27:44.920]  generic avatars but if you walk around a bit and you can see her moon walking
[27:44.920 --> 27:49.800]  backwards for whatever reason I'm sure you can also go forwards there we go and
[27:49.800 --> 27:56.080]  fly for that matter and so the B button lets you fly in this so you can go and
[27:56.080 --> 28:03.360]  jump around like so and so you're spoiling my talk so if we go down here
[28:03.360 --> 28:07.840]  then on the game thread we've got a bunch of rust we have the ability to run
[28:07.840 --> 28:12.320]  arbitrary WebAssembly scripts which is sitting in a sandbox which allows you
[28:12.320 --> 28:15.760]  to basically add any arbitrary functionality into the world from a
[28:15.760 --> 28:20.000]  pure matrix perspective this is probably the most exciting thing here now if
[28:20.000 --> 28:25.360]  you remember IRC and Merck scripting the ability to run arbitrary scripts on
[28:25.360 --> 28:29.680]  your IRC client this is effectively allowing you to define bots and
[28:29.680 --> 28:34.600]  arbitrary functionality in matrix which run inside your client inside the
[28:34.600 --> 28:40.320]  sandbox and the actual data is stored in your room now this whole thing is a
[28:40.320 --> 28:46.960]  matrix room right if I go and hit enter then well you can see a bunch of users
[28:46.960 --> 28:53.160]  there I can say hello world and if I go back to my element client and if I
[28:53.160 --> 28:59.800]  literally join presentation on thirdroom.io for matrix.org then I can you
[28:59.800 --> 29:08.760]  can see I'm Dean saying yeah and I can say yeah to you too and hopefully hang
[29:08.760 --> 29:15.920]  it off well I've got traffic running one way interesting well we should be
[29:15.920 --> 29:20.800]  seeing messages coming into the room as well because it is there we go it's
[29:20.800 --> 29:25.560]  just a plain old hydrogen overlay that is being rented in react for the
[29:25.560 --> 29:31.080]  contents of the room now the actual geometry of the room if we start flying
[29:31.080 --> 29:37.040]  around some more looks like this is actually a big GLTF or a single GLTF
[29:37.040 --> 29:40.560]  asset this thing is just sitting in the media repository in the room it's just a
[29:40.560 --> 29:45.280]  file that is GLTF the transfer format for OpenGL that has been uploaded there
[29:45.280 --> 29:51.400]  and also any scripts in the room like the one which is executing the let me
[29:51.400 --> 29:55.440]  press on the buttons here again there's a bit of I think JavaScript using the
[29:55.440 --> 29:59.680]  quick.js engine that has gone and compiled down the JavaScript to web
[29:59.680 --> 30:03.680]  assembly in real-time it's pretty cool that you literally write it in JavaScript
[30:03.680 --> 30:07.280]  and then the engine sucks it up turns it into wasm and runs it within that
[30:07.280 --> 30:11.440]  sandbox so you could argue it's a little bit perverse to be taking JavaScript
[30:11.440 --> 30:14.640]  compiling it to web assembly and then running it from within a JavaScript
[30:14.640 --> 30:19.480]  environment but it gives you a hell of a lot more safety than you would if we were
[30:19.480 --> 30:25.840]  just I know having random blobs of JavaScript running here on the render
[30:25.840 --> 30:31.520]  thread we are using WebGL2 and we're using Freejs to manage the actual
[30:31.520 --> 30:40.600]  driving of WebGL but the scene itself is the scene itself is managed using a
[30:40.600 --> 30:45.040]  really cool technology called Bit ECS that was actually created by Nate one
[30:45.040 --> 30:48.640]  of the developers before he started working on they before they started
[30:48.640 --> 30:55.720]  working on third room and Bit ECS is a entity component system where you
[30:55.720 --> 30:59.760]  basically trap the state of the world the objects that exist within it their
[30:59.760 --> 31:06.040]  transformations and it's done with arrays in JavaScript and it turns out that if
[31:06.040 --> 31:09.440]  you stretch your arrays intelligently enough in JavaScript you can get as
[31:09.440 --> 31:12.960]  good as web assembly performance and it's one of the other secrets to the
[31:12.960 --> 31:18.760]  crazy and performance that we have here so this isn't a scene graph API under
[31:18.760 --> 31:23.800]  the hood like a frame if anybody ever played with a frame instead it's using
[31:23.800 --> 31:31.680]  the Bit ECS then another thing which is interesting here is that everything is
[31:31.680 --> 31:36.400]  some triple buffered so and a kind of traditional game engine you just have
[31:36.400 --> 31:41.040]  one sort of buffer that you write data into and the renderer reads it out and
[31:41.040 --> 31:44.400]  you have some kind of locking system to make sure that it doesn't collide
[31:44.400 --> 31:50.320]  whereas here we have lot free data structures letting things go as rapidly
[31:50.320 --> 31:54.480]  as possible with the various different bits of the engine writing into these
[31:54.480 --> 31:59.120]  shared this shared triple buffer as a shared array and buffer which is then
[31:59.120 --> 32:02.840]  juggled effectively between the various different threads and it means that the
[32:02.840 --> 32:07.040]  render thread can run at the native speed of whatever device which is
[32:07.040 --> 32:11.240]  particularly useful if it's a less powered device than my MacBook Pro here
[32:11.240 --> 32:15.520]  and then the game engine that is actually rendering what's going on can
[32:15.520 --> 32:19.640]  go at its own speed so you totally decouple the two and you get as high a
[32:19.640 --> 32:24.760]  frame rate as you can and I think that oh yeah and finally lots of fun stuff
[32:24.760 --> 32:31.120]  going on with asset optimization pipeline particularly the textures have
[32:31.120 --> 32:35.480]  been highly compressed using these fun codecs I think it's called universal
[32:35.480 --> 32:41.040]  basis format from binomial and KTX and one of the things we've done to cheat
[32:41.040 --> 32:47.000]  to bootstrap third room is to build a pipeline from Unity where you can take
[32:47.000 --> 32:50.920]  existing Unity assets like this scene here is one that we bought off a Unity
[32:50.920 --> 32:56.400]  assets store and then export it as proper open standardized JLTF somewhat
[32:56.400 --> 33:00.680]  liberating the contents from the slightly proprietary world of Unity in
[33:00.680 --> 33:06.040]  order to get content in more rapidly and then compress it down and there are
[33:06.040 --> 33:11.240]  lots of fun things like the it has instancing support built in so if you
[33:11.240 --> 33:15.560]  start generating lots of objects like the physics engine here I know going
[33:15.560 --> 33:19.040]  create a whole bunch of objects and attack the various people who are
[33:19.040 --> 33:22.640]  wandering around in here I'm sure they love me for it and then this is
[33:22.640 --> 33:33.360]  basically just the same JLTF asset going and being created multiple times all
[33:33.360 --> 33:37.760]  the textures are sprighted so there's just one great big thing there's also
[33:37.760 --> 33:43.920]  some really interesting extensions to GL that we've contributed by the
[33:43.920 --> 33:48.280]  Kronos group and particularly if you look at if we grab one of these
[33:48.280 --> 33:53.480]  mirror balls which are mainly used for debugging purposes let's grab that one
[33:53.480 --> 33:58.000]  and if I run around with it while I fly around with it you should see that the
[33:58.000 --> 34:03.680]  reflection changes there we go like if I go between zones there it's not they
[34:03.680 --> 34:06.920]  need to be tuned a bit but basically rather than ray tracing which would be
[34:06.920 --> 34:11.480]  incredibly time-consuming instead we have lots of different probes hanging
[34:11.480 --> 34:15.280]  around the scene that allow you to so I'm hitting myself in the face with the
[34:15.280 --> 34:22.000]  ball a common problem but as you run around you can see the reflection
[34:22.000 --> 34:25.400]  changes it's pretty nasty if you do it rapidly but if you're doing it more
[34:25.400 --> 34:29.680]  slowly like this and it's a quite a subtle but nice effect and it's even
[34:29.680 --> 34:34.840]  better when it's on not perfect maribles if you look at say Dave if you walk
[34:34.840 --> 34:39.120]  backwards if you can hear me or go into the light or out of the light then you
[34:39.120 --> 34:42.800]  actually see a fairly subtle shadowing effect as it's gone and figured out where
[34:42.800 --> 34:46.520]  the shadows are there right guys cool to see all the people running around in
[34:46.520 --> 34:52.400]  here so what else can I show you one of the so we're gonna launch some tech
[34:52.400 --> 34:56.360]  preview 2 of this next week and this is the first time anybody has seen tech
[34:56.360 --> 35:01.480]  preview 2 and I have 10 minutes yet left on and some tech preview one is sort of
[35:01.480 --> 35:04.280]  what we've been looking at here except it didn't have scripting we've already
[35:04.280 --> 35:08.080]  shown some of the scripting here but one of the big things sort of being added
[35:08.080 --> 35:20.960]  and let's pray that this thing works is web VR so hopefully if I go to the
[35:20.960 --> 35:24.440]  Oculus streaming thing which I had a second ago should have possibly cleaned
[35:24.440 --> 35:31.200]  this up first of course it stopped working well I get this thing back in
[35:31.200 --> 35:35.960]  again and everything is going wrong apparently I've got to recalibrate the
[35:35.960 --> 35:41.280]  entire thing so I apologize for using proprietary technology but
[35:41.280 --> 35:48.880]  unfortunately there aren't any open source headsets which do the trick yet
[35:48.880 --> 35:56.360]  there we go and let me try to cast this up and unfortunately it takes ages for
[35:56.360 --> 36:04.080]  the screen casting to kick in for some reason but I'll go as quick as I can
[36:04.080 --> 36:15.200]  bump and cost and computer go right so WebXR is a really cool technology it's
[36:15.200 --> 36:24.520]  been there for ages now since about 2017 built into browsers like Firefox and
[36:24.520 --> 36:29.560]  Chrome obviously and also interestingly the browser that your Oculus and Quest
[36:29.560 --> 36:34.960]  like this or Quest Pro has built into it which is based on Chromium it has
[36:34.960 --> 36:38.120]  awful screen casting support as you can see and that I started at screen
[36:38.120 --> 36:41.200]  casting and something is happening in the depths of Facebook trying to figure
[36:41.200 --> 36:44.680]  out how to actually get this onto the screen here but hopefully it will come
[36:44.680 --> 36:48.280]  through assuming I've got internet connectivity here it is thank God and I
[36:48.280 --> 36:55.120]  can start talking and I apologize for I'm going to focus on very interesting
[36:55.120 --> 37:00.000]  so I'm going to focus on Flora in rather than embarrassing everybody else but
[37:00.000 --> 37:05.720]  let's just use a stationary boundary confirm right so the browser here sits
[37:05.720 --> 37:12.080]  there I'm not going to update this right now but here is some third room and if I
[37:12.080 --> 37:17.360]  continue into third room as guests you can see this is just a static boring
[37:17.360 --> 37:22.360]  old web browser just sitting here worth noting that third room uses OIDC entirely
[37:22.360 --> 37:28.120]  so this thing here is actually a skinned key cloak I'm gonna say I'm not a bot
[37:28.120 --> 37:33.560]  I'm not gonna bother giving myself a name and then capture failed brilliant
[37:33.560 --> 37:42.480]  thanks Google I'm gonna have to type in third room except caffeine and stress
[37:42.480 --> 37:46.640]  means my ability to use a stupid keyboard like this is gonna be fun okay
[37:46.640 --> 37:53.320]  about to third room here is a streaming okay okay brilliant let's go to login go
[37:53.320 --> 37:58.520]  about third room continuous guests this time hope that it's not gonna make me
[37:58.520 --> 38:04.600]  pick stupid things right good continue accept the T's and C's honestly the
[38:04.600 --> 38:12.920]  using a key cloak for this is really really fun and very anticlimatically we
[38:12.920 --> 38:19.560]  end up with eventually monthly connectivity a 2d version of third
[38:19.560 --> 38:25.800]  room just sitting right here so isn't it amazing by the way that just loaded
[38:25.800 --> 38:33.240]  from index DB local storage but the fun thing is hopefully come on you can do
[38:33.240 --> 38:37.280]  it you can see it's actually struggling quite a lot in this but if I press the
[38:37.280 --> 38:43.400]  old X button there that's I have to close the welcome to third room dialogue
[38:43.400 --> 38:51.520]  here it is come on and to XR thank God for that then I can see Florian
[38:51.520 --> 38:56.360]  hello Florian but more excitingly hopefully if I stay in the right place
[38:56.360 --> 39:00.960]  there we go you can see that I'm actually in the third room environment
[39:00.960 --> 39:05.520]  right now and this is genuinely cool this is running at 90 frames a second for
[39:05.520 --> 39:15.600]  me right now and if I go and press some buttons to crates and oh god crates like
[39:15.600 --> 39:19.160]  that massive crate let me get rid that you can see it's actually hooked up to
[39:19.160 --> 39:23.760]  the normal physics engine so I can go and pull that and I can confusingly throw
[39:23.760 --> 39:28.360]  it into the audience which is no way surreal to be going and flipping back
[39:28.360 --> 39:34.560]  and forth and then back in the normal world again at the moment we've just
[39:34.560 --> 39:40.680]  got basic things like the joystick and hung up hooked up to it it's got a
[39:40.680 --> 39:48.280]  kinematic controller and I'm I running out of oh no thank you Florian what else
[39:48.280 --> 39:53.600]  can I show you if you can jump we can spawn more objects I can go after that
[39:53.600 --> 39:59.640]  glitter globe and sorry Mirable except is running faster than I can run after it
[39:59.640 --> 40:04.000]  that's awkward and theoretically if I was a little bit close to the bloody thing
[40:04.000 --> 40:09.080]  I'd be able to grab it and pick it up etc so this is pretty cool honestly it's
[40:09.080 --> 40:14.000]  as good as the native non-web VR and closed stuff that Facebook or
[40:14.000 --> 40:20.280]  meta horizons does and the entire thing is open and they built on the surgery
[40:20.280 --> 40:24.840]  mentioned how am I doing for time three minutes in which case I very quickly go
[40:24.840 --> 40:32.040]  and sorry thanks for I will stop looking at random emissives go back into this
[40:32.040 --> 40:37.200]  and just look at some other things we've done in fact this one is really cool
[40:37.200 --> 40:40.840]  let's just flip into this one because this is a really complicated bit of
[40:40.840 --> 40:45.480]  wasm it's actually an audio reactive widget which is sitting here as you can
[40:45.480 --> 40:50.920]  see as I yell at it it goes and changes size and this is a whole bunch of C code
[40:50.920 --> 40:54.480]  that has been compiled down to wasm to show how you can have interactive things
[40:54.480 --> 41:01.440]  sitting inside the scene another example is slightly less excitingly a chat box
[41:01.440 --> 41:09.040]  echo service so if I go into here and say hello and then do echo hello with a
[41:09.040 --> 41:13.120]  slash command it says hello back to me now the echo that's happening down there
[41:13.120 --> 41:17.840]  is actually being done from the widget API of matrix going into web assembly
[41:17.840 --> 41:23.640]  talking to I think a JavaScript service and then echoes back and you can see we
[41:23.640 --> 41:26.360]  have a slight bug sometimes with scripting where it loads two worlds at
[41:26.360 --> 41:30.760]  the same time and that's pretty surreal and that's what happens if London got
[41:30.760 --> 41:40.760]  combined with Mars final thing here I was going to show you oh yeah is this guy
[41:40.760 --> 41:46.240]  which is a bit silly but fun anyway and this time I'll remember to refresh as
[41:46.240 --> 41:49.840]  fun as it is to have the scenes combined so you might recognize this from a
[41:49.840 --> 41:55.880]  certain film and if we actually look at the script for this particular room if
[41:55.880 --> 42:00.640]  I could figure out how to get out of fullscreen mode the script here is again
[42:00.640 --> 42:05.080]  just sitting in the media repository it's a little bit of JavaScript and to use
[42:05.080 --> 42:08.840]  the web scene graph API which is a new API that we've created we hope it will
[42:08.840 --> 42:13.480]  become a W3 standard for manipulating scene graphs it looks a lot like Dom you
[42:13.480 --> 42:17.560]  basically get a node by name the TV and then every frame you'd see if the TV is
[42:17.560 --> 42:20.840]  being pressed and if it is then you enable the matrix material and the end
[42:20.840 --> 42:25.320]  result is if I go here and I click on the TV then predictably enough you end up
[42:25.320 --> 42:30.000]  in a matrix style world and I've got it in third-person view which is also new
[42:30.000 --> 42:33.880]  with tech preview 2 and clay back and forth this is super early but you can
[42:33.880 --> 42:36.760]  imagine this is basically a platform for doing any kind of real-time
[42:36.760 --> 42:40.440]  collaborative app could be figma on this it could be multiplayer blender it
[42:40.440 --> 42:46.720]  could be a game it could be digital twins it could be I know smart cities GIS
[42:46.720 --> 42:51.320]  applications it's as powerful and inslexible as the web but for real-time
[42:51.320 --> 43:02.840]  thank you very much and we have no time for questions
[43:02.840 --> 43:06.840]  one question
[43:06.840 --> 43:10.840]  oh
[43:10.840 --> 43:18.680]  it will be coming in the next release after tech preview 2 I mean the hub bit
[43:18.680 --> 43:22.360]  of actually rigging up the engine and doing all the inverse kinematics and if
[43:22.360 --> 43:25.760]  you run around with the current avatars you can now trip over things which is
[43:25.760 --> 43:30.920]  very important but suffice to say we're focusing on the engine rather than the
[43:30.920 --> 43:35.200]  assets and also this is at a point where people can start contributing things so
[43:35.200 --> 43:39.880]  if you've got amazing assets if they support the maximum rig then they
[43:39.880 --> 43:52.880]  should just drop straight in cool