[00:00.000 --> 00:11.840]  Welcome everybody. Good morning to our talk. All the people on stream on video on demand
[00:11.840 --> 00:16.480]  and of course here in the deaf room. My name is Kim and I'm here together with my colleague
[00:16.480 --> 00:23.360]  Oliver. We are from Nordic and we are going to talk to you about the matrix widgets and
[00:23.360 --> 00:28.960]  particularly those we develop for the sovereign workplace project in the German public sector.
[00:32.960 --> 00:40.160]  So at first, quickly about us, Nordic is an IT consulting company based in Hamburg, Germany.
[00:40.800 --> 00:49.120]  We are about 40 IT professionals and we, among other things, we develop first matrix integrations
[00:49.120 --> 00:55.360]  for productivity software in the public sector in the so-called sovereign workplace project.
[00:56.160 --> 01:03.920]  And we are here today to provide you some insights from that we gained in the last couple years.
[01:06.080 --> 01:12.480]  So what's the sovereign workplace? Let's take a step back and introduce you all to it.
[01:12.480 --> 01:18.880]  It's a project with the goal of providing IT services to public administration in Germany
[01:18.880 --> 01:24.400]  and potentially also Europe. It's founded by the German Federal Ministry of the Interior
[01:24.400 --> 01:32.640]  and Community. And one of the core aspects is to gain independence from US cloud services
[01:32.640 --> 01:40.720]  and to retain full control over your data. And there's this suit we call sovereign workplace
[01:41.280 --> 01:47.760]  which covers many use cases for productivity and it's achieved by combining the products from
[01:47.760 --> 01:53.440]  many different vendors which you can see on the right here. But all of this is open source software
[01:53.440 --> 01:59.760]  so yeah you could say this whole project also supports and maybe funds some of the open source
[01:59.760 --> 02:06.320]  projects here. We could talk a lot more about the whole sovereign workplace project and that
[02:06.320 --> 02:10.560]  could be a whole talk for itself on another track but of course we are here to talk about
[02:10.560 --> 02:18.560]  matrix and the matrix death room. So yeah quick overview here we can see the Univention
[02:18.560 --> 02:24.560]  Corporate Server and Keycloak which implement identity management in this stack. There's
[02:24.560 --> 02:32.880]  Groupware which is done by OpenExchange. There's cloud storage by Nextcloud. There is an office
[02:32.880 --> 02:40.240]  suit which integrates with Nextcloud very tightly which is Kolobora online and of course
[02:40.240 --> 02:46.960]  there's video conferencing which is done with jitzy right now and perhaps in the future using
[02:46.960 --> 02:55.840]  matrix natively and of course there's messaging which is done natively on matrix and particularly
[02:55.840 --> 03:04.240]  using element as a client. How are we involved in this? Well together with element we provide
[03:04.240 --> 03:11.440]  this real-time communications component which is messaging and also video conferencing so there's
[03:11.440 --> 03:21.280]  chat and video and in particular we are extending this chat use case using widgets for some specific
[03:21.280 --> 03:27.440]  use cases so and this is the crucial part of this talk. The idea is to provide better integration
[03:27.440 --> 03:35.600]  with other components in this stack and build new work solutions. So you might be familiar
[03:35.600 --> 03:40.400]  with the concept of a widget and we've seen it in the previous talks but still I want to summarize
[03:40.400 --> 03:48.560]  a bit. As I said we can use widgets to extend the regular chat client functionality for specific
[03:48.560 --> 03:54.880]  use cases and I brought you some examples here so essentially it's an option and also this was
[03:54.880 --> 04:03.520]  said before but it's an option to embed some kind of web app into your existing matrix client and
[04:03.520 --> 04:09.360]  display more functions right there. So for example you can put it in the right bar like Hookshot does
[04:09.360 --> 04:17.440]  here or you can pin it to the top of the room and it will even adjust to your theme like this
[04:17.440 --> 04:25.120]  countdown and there's more there you can add a lot of widgets at once and see all of them at once
[04:25.120 --> 04:34.640]  in multiple places. Of course there's video conferencing which you can also maximize to
[04:34.640 --> 04:41.520]  view all of the people in the video conference rather than having chat and yeah there's also
[04:41.520 --> 04:47.440]  this half full screen mode where the chat moves to the right side and you can see the widget in a
[04:47.440 --> 04:55.680]  full screen-ish style and there's more so actually if you're following FOSTA remotely you might
[04:56.800 --> 05:03.920]  find this a familiar site so you are actually using widgets right now to watch this and there's
[05:03.920 --> 05:09.440]  this schedule there's the live stream and also from the editor side there are widgets to supporters.
[05:09.440 --> 05:18.160]  So if you followed closely or you are very familiar you might have noticed wait a minute
[05:18.800 --> 05:26.240]  this is a bit more than just static webpages so actually there's a thing called the widget API
[05:26.240 --> 05:34.320]  and that allows your embedded page to interact with the client but also with the matrix.
[05:34.320 --> 05:41.360]  So for example from our widgets we can also send messages read messages to the room.
[05:42.480 --> 05:52.240]  So one way to explain widgets would be you could say these are some kind of limited matrix line
[05:52.240 --> 05:59.440]  for a specific use case and you can build a lot of things really there are endless possibilities
[05:59.440 --> 06:07.680]  and now Oliver is going to show you a bit more about how that works. Thanks Kim so let's
[06:07.680 --> 06:14.320]  ahead a step back we're now back at the concept that it's a website that's embedded
[06:15.760 --> 06:24.240]  so you have an iframe we'll see it soon and the iframe is just any static website and that
[06:24.240 --> 06:30.720]  website has an API that can be used to communicate with the client so for example in this case
[06:30.720 --> 06:38.880]  element and then you have the other option that the element client could be connected to your home
[06:38.880 --> 06:47.360]  server so it's like I would say a pass-through API which gives the commons of the widget to the
[06:47.360 --> 06:54.400]  home server via the widget API and with this API you can do a lot of stuff so one thing that is
[06:54.400 --> 07:00.320]  important is that this iframe also allows to have like an isolation so even though you embed
[07:00.320 --> 07:07.440]  a website there it's not like having full access to what the element does everything that you do
[07:07.440 --> 07:18.320]  has to be done via the widget API so from the feature perspective of the widget API you can
[07:18.320 --> 07:24.080]  separate it into two parts so on the one side you have everything that is related to displaying
[07:24.080 --> 07:30.880]  widget interacting with the client for example Kim showed you that you can have this different
[07:30.880 --> 07:37.440]  display mode there are also even more for example you can model widgets that then displayed in a full
[07:37.440 --> 07:45.680]  screen view you can set things like always in top where the widget is displayed in the right bottom
[07:45.680 --> 07:52.320]  corner and it's always there even if you switch to a different room and you can do capability
[07:52.320 --> 07:59.520]  requests I will come to a second what that means and then you have the second group of features
[07:59.520 --> 08:07.120]  which is like matrix or room interactions so that is everything about sending events receiving events
[08:07.120 --> 08:15.680]  state events room events or to device messages as Florian also mentioned today yeah some special
[08:15.680 --> 08:21.840]  things like reading relations and requesting ODC identity token that you can use at another
[08:21.840 --> 08:31.360]  service to identify yourself so if you go now back to this reading and sending messages you
[08:31.360 --> 08:37.680]  notice that actually you could use matrix for non-chat application where you store your data
[08:37.680 --> 08:46.560]  in the room you can see it is like a real-time database we had the term today and store your
[08:46.560 --> 08:55.120]  stuff there and have all the benefits of the matrix protocol that you would have for chat too
[08:55.120 --> 08:59.760]  for example you can store your data and have it enter and encrypt it you can collaborate with
[08:59.760 --> 09:06.720]  others via federation and all these features are already there and you can see it as like a nice
[09:06.720 --> 09:16.560]  backend for building collaboration apps so I already mentioned this word widget capabilities
[09:17.280 --> 09:25.680]  so one issue that you would have if you like provide all these features just to an app without
[09:25.680 --> 09:32.240]  like having any security they could do a lot so actually I could build a widget that reads all
[09:32.240 --> 09:39.360]  your data and post it somewhere like which chats you write or does stuff in your name for example
[09:39.360 --> 09:45.200]  it could send messages in your name and therefore there is a security mechanism with this widget
[09:45.200 --> 09:53.040]  capabilities and it's actually quite similar to what android is doing with permissions so your
[09:53.040 --> 09:58.720]  web app once it is embedded and once you started it so every administrator can embed
[09:58.720 --> 10:03.440]  a widget into your room once it's embedded you have the chance to
[10:06.080 --> 10:12.880]  request specific permissions for your app and then the user gets a screen like this it's quite
[10:12.880 --> 10:19.600]  a long list and can explicitly allow data to be shared with the widget and that way you have
[10:19.600 --> 10:24.880]  your control over your data you don't share anything that you don't want to share with a
[10:24.880 --> 10:33.760]  third party site and only then the widget has access to it so
[10:55.520 --> 11:06.640]  but maybe grouping and writing best attacks can help here to avoid this situation where users
[11:06.640 --> 11:10.160]  just click accept without thinking about what they are sharing
[11:14.080 --> 11:18.720]  so now I would talk a little bit about with you about the status of widget API
[11:18.720 --> 11:25.760]  so right now it's only supported completely in element web and desktop so
[11:27.760 --> 11:35.440]  there is support for widgets in android and ios but it's more like static web pages without
[11:35.440 --> 11:43.840]  the widget API and the widget API is what it gives is this interactivity stuff that you
[11:43.840 --> 11:50.880]  wouldn't have with just a static web page yeah why is it only supported in that two clients
[11:50.880 --> 11:58.080]  I think there are at least two reasons one reason is that it's not part of the magic spec yet so
[11:58.080 --> 12:05.520]  there is like a draft here which is the spec but the draft is a bit outdated it collects
[12:05.520 --> 12:12.160]  some of the mscs around it but a lot of the mscs are not part of the spec yet so it makes it
[12:12.160 --> 12:17.200]  I guess quite hard to develop based on it so it makes it hard for a consumer of the widget
[12:17.200 --> 12:22.080]  API to develop it but it would also make it hard as in a better because you always have to go
[12:22.080 --> 12:30.320]  through the mscs look for stuff that you need and build it but I think it's not the only reason
[12:30.320 --> 12:35.920]  why it's not yet fully supported by every client it's I think it's also the situation that it's
[12:35.920 --> 12:44.480]  maybe not the perfect implementation yet um and what I mean with it I want to talk about it in
[12:44.480 --> 12:51.600]  the next slide so if right now I want to extend the widget API with a new feature it's not feature
[12:51.600 --> 12:57.840]  complete so it doesn't support everything you have in the matrix js SDK it has a subset of features
[12:57.840 --> 13:03.520]  but if you want to bring a new feature into it um right now we have to follow this process so let's
[13:03.520 --> 13:09.520]  assume that I want to support uploading content into the media repository so it would be quite
[13:09.520 --> 13:15.920]  nice for some use cases um then I would create an msc which I think itself is not a problem
[13:17.360 --> 13:22.400]  do you look into the current spec of the client server API look at the behavior for example
[13:22.400 --> 13:29.200]  uploading requires like thinking about quota thinking about limits size limits and stuff like
[13:29.200 --> 13:40.880]  that and then how the API responds these information back to you and uh so you look at the spec copy
[13:40.880 --> 13:46.080]  that behavior so it's it's the same and think about how you integrate into the widget API
[13:46.800 --> 13:53.440]  then the next step you think about like capabilities uh how can I prevent users from doing stuff that
[13:53.440 --> 14:00.480]  they shouldn't do with it or how can I keep control about it and then you have like a base
[14:00.480 --> 14:05.360]  to start the implementation you would probably implement it in the matrix widget API repository
[14:05.360 --> 14:13.040]  which is the API that both embedders offer widgets of widgets use but also the widget itself
[14:14.080 --> 14:19.280]  and the next step you would implement it in element and if there would be more clients that support
[14:19.280 --> 14:26.720]  it in all other clients that you want to support it and that's actually a lot of work and just for
[14:26.720 --> 14:33.200]  something that was already there right so if you wouldn't use widgets you could upload files with
[14:33.200 --> 14:40.800]  the client server API and then you notice that you can upload files how to download them starts
[14:40.800 --> 14:47.680]  the process over again so I think it's not the problem with the msc process but the problem is
[14:47.680 --> 14:54.400]  that you have to copy stuff that is already there so behaviors that is already there specs that is
[14:54.400 --> 14:58.800]  already there it would be much better if I could just use the client server API from my widget
[15:02.400 --> 15:08.720]  and there's already like the idea of doing that so there's this msc that thinks about how can I
[15:08.720 --> 15:20.000]  share like the client server API with my widget and it's done in this example or in this specific msc
[15:20.000 --> 15:27.920]  by sharing the access token with the widget which brings like new challenges for example you have
[15:27.920 --> 15:34.720]  to think about capabilities again but because if you share the whole access token then you can do
[15:34.720 --> 15:41.120]  everything so you would have to think about how can I restrict the access again and the one idea
[15:41.120 --> 15:47.600]  is there to use a scoped access totems that have there's actually an msc and I've missed it here
[15:49.040 --> 15:57.120]  you use scoped access totem to create a totem that is like only possible one can only access
[15:57.120 --> 16:02.800]  the stuff that the user previously allowed the widget so that's something where you can mirror
[16:02.800 --> 16:09.680]  this capabilities that you've previously had with the widget API with this approach so that would
[16:09.680 --> 16:15.120]  bring us a lot of benefits so we would actually directly have this feature parity with the client
[16:15.120 --> 16:23.760]  server API we can do uh yeah it may be also more performant because now we rely on element loading
[16:23.760 --> 16:30.800]  all the data uh and also relaying all our requests to the home server we could directly talk to the
[16:30.800 --> 16:37.440]  home server and optimize our challenge or our tools and we would also assume that's a lot easier
[16:37.440 --> 16:45.200]  to implement because the widget and better so a client like element only has to implement the
[16:45.920 --> 16:53.280]  exchange of the credentials to the widget but not all the API calls that are available that
[16:53.280 --> 16:57.920]  have to be relayed and capabilities have to be checked and all that stuff so it would
[16:57.920 --> 17:02.800]  uh make the implementation a lot easier and hopefully also bring it to more clients
[17:03.760 --> 17:07.920]  there are some challenges for example it's actually quite good that element does all the
[17:07.920 --> 17:13.600]  stuff because as a widget author you don't have to do have to think about sync you don't have to
[17:13.600 --> 17:23.040]  think about e2e uh e2e uh so end to end encryption so it actually makes it quite simple but probably
[17:23.040 --> 17:32.640]  that's some challenges we have to solve then and uh there's also like this msc proposes to
[17:33.440 --> 17:38.560]  bring the access token via the url into the widget which might also not be the best way
[17:39.440 --> 17:46.640]  maybe or i'd see once it's there can help us here to delegate the identity and access into the widget
[17:46.640 --> 17:57.360]  in the morning florian already talked about uh this matoska matoska mode for element call
[17:58.160 --> 18:04.320]  and there they're using the i think it's called room with widget room client from the matrix
[18:04.320 --> 18:11.280]  js sdk which is quite cool because it allows you to already start using the matrix js sdk
[18:11.280 --> 18:18.640]  in your widget and it feels like it's a matrix js sdk but it's relayed over the widget api and uh
[18:20.160 --> 18:27.200]  providing you later on maybe a better way to migrate to this state or to this style of api
[18:32.320 --> 18:37.200]  so what would these features bring us or what can you already do with existing widget api
[18:37.200 --> 18:43.840]  so you could be built really cool collaborative tools because you have like a real-time communication
[18:43.840 --> 18:52.960]  channel you can build you will give some examples later so i don't give them um you can build stuff
[18:54.960 --> 19:00.000]  where you would actually normally have to build a backend a communication layer and all this stuff
[19:00.000 --> 19:06.160]  and think about a lot of stuff that matrix already has and brings you can use the rooms
[19:06.160 --> 19:13.520]  for data storage um they actually some tricks needed to do that efficiently that would be like
[19:13.520 --> 19:24.880]  the talk for itself um but you have the idea that all these um applications that you build
[19:24.880 --> 19:33.760]  could just use it um i talked about real-time communication before so actually if you use
[19:33.760 --> 19:41.040]  matrix you have some kind of very slow real-time communication it's not suitable for building
[19:42.320 --> 19:52.240]  more complex or more more quicker stuff like for example a game or a whiteboard or stuff like that
[19:52.240 --> 19:58.160]  that's where for example matrix fdc comes into play where you have direct p2p connections or
[19:58.160 --> 20:05.200]  via sf use and if you have access to that in your widget that would be would allow you to build
[20:05.200 --> 20:12.480]  really great stuff and actually if if like you reach your limits which with widgets you always
[20:12.480 --> 20:19.440]  have the options to switch to uh yeah more components for example like a bot said is invited
[20:19.440 --> 20:25.360]  into your room and helps you to um do stuff that you could actually not do with just the widget
[20:25.360 --> 20:35.520]  it wants on the user side um yeah kim will give now some examples for widgets that we build
[20:37.040 --> 20:40.160]  and also do does a quick demo thank you all over
[20:44.960 --> 20:52.160]  so uh here are our use cases the widget that we build um so yeah as you can see there are four of
[20:52.160 --> 20:58.160]  them and so the first one we built is the pulse widget as you might know there are
[20:59.600 --> 21:05.600]  pulse in element now and i believe they are coming to the spec i think there's an msc
[21:07.760 --> 21:13.600]  but yeah these allow you to do some simple pulse but you might have some cases where
[21:13.600 --> 21:20.160]  you actually need to do some more fancy things like you want to even use parties for example
[21:20.160 --> 21:27.680]  and so we built this pulse widget that allows you to do to cover many more advanced use cases
[21:28.480 --> 21:34.640]  and in fact this is already open sourced the end of last year i believe in november december
[21:34.640 --> 21:42.640]  sometime and you can find it online under this link on our github and so the next one is the
[21:42.640 --> 21:50.880]  barcam widget if you are unfamiliar with the barcam the idea is to um meet in a group um collect
[21:50.880 --> 21:56.240]  some topics and then build a schedule right there and then have an event based on that schedule
[21:56.960 --> 22:02.480]  i'm going to show it to you in a minute and uh yeah so this is our second widget that is also
[22:02.480 --> 22:10.880]  now open source and further we are also developing a meetings widget which allows you to create um
[22:10.880 --> 22:17.600]  well appointments with within the widget and it will set up rooms for you and set up the possibility
[22:17.600 --> 22:24.880]  to have a video call right there and this is working progress but will also be open sourced
[22:24.880 --> 22:33.040]  and then there's also the whiteboard widget we are building and again this is going to be open
[22:33.040 --> 22:46.720]  source at some point when it's when it's finished or in beta state um right so as i said i want to
[22:46.720 --> 22:52.080]  show you the barcam widget for a bit and we actually got the chance to use it productively
[22:52.080 --> 23:05.280]  on friday at our matrix community meetup and um yeah we have prepared a quick video for you
[23:06.960 --> 23:12.800]  so right here on the left hand side you have the grid and you can for example add tracks you can
[23:12.800 --> 23:19.680]  edit the track names you can choose some icons uh on that axis and of course you can also um
[23:19.680 --> 23:25.920]  modify the other axis which are the different time slots you can move stuff around change the
[23:25.920 --> 23:33.120]  length of stuff and then once that's set up you can go to you can enable the topic submission
[23:33.120 --> 23:40.880]  and you and all the other users in the room can create this kind of posted cards where you enter
[23:40.880 --> 23:47.680]  your topic and maybe a short description and then once you send it it will appear here on the right
[23:47.680 --> 23:57.600]  in the parking lot but yeah for this because this is not yet quite supported on other platforms
[23:57.600 --> 24:06.640]  than element web desktop and we also built a bot as a compatibility layer which allows you to also
[24:06.640 --> 24:13.840]  submit topics through the chat right here you write a bot command the bot will convert it to the
[24:13.840 --> 24:20.400]  event that's read by the widget and you see a tick and it also appears right here in the
[24:21.360 --> 24:27.680]  select your topic button here's the first one we created at that point you can even edit
[24:28.880 --> 24:35.280]  as a moderator and then you just move it into the schedule and then you can select the next
[24:35.280 --> 24:44.240]  topic and also review it maybe you edit it maybe you don't and put it on your schedule
[24:45.760 --> 24:51.040]  yeah you have the feature of locking and unlocking submissions for the non-moderator users in your
[24:51.040 --> 25:02.800]  room and I believe that's it all right so yeah thank you very much everybody and it's now time
[25:02.800 --> 25:11.280]  for qa we also have this qr code if you want to find us on matrix you can use the qr code you
[25:11.280 --> 25:18.560]  can use this room alias and come talk to us of course you can also find us in the deaf room
[25:18.560 --> 25:34.480]  online
[25:48.560 --> 25:54.400]  right do you want to answer yeah okay so I think it was kind of two questions so one question was
[25:55.200 --> 26:00.080]  how do users find the widgets that are installed in a room and then I think this second question
[26:00.080 --> 26:04.720]  goes more into the direction of integration managers with dimensions so there's something
[26:04.720 --> 26:11.760]  like that available so um widgets are like state events in a room um once you edit it to the room
[26:11.760 --> 26:18.560]  you have like an event within url so it could be hosted on any web server that's then embedding it
[26:20.000 --> 26:26.880]  and uh you have the question about discoverability so yeah there are integration managers for example
[26:26.880 --> 26:32.960]  dimensions that you can use to add widgets but um at least dimension doesn't have set
[26:32.960 --> 26:38.080]  good support for the widget that use the widget api so you would uh
[26:38.080 --> 26:44.480]  uh probably need something else I don't know about any one or any integration
[26:44.480 --> 26:48.400]  managers that support them very well
[26:59.440 --> 27:05.040]  so right now you have to uh have the url of the widget and then you can just use the add
[27:05.040 --> 27:11.920]  widget command and add it to your room um but an integration manager would be great so I could just
[27:11.920 --> 27:15.600]  click in the bottom right of element this is already this integration for it
[27:17.600 --> 27:25.520]  and then as a room admin or moderator you even have the ability to to pin the widget to the top
[27:25.520 --> 27:33.040]  of the room for example or to maximize it and then to save that that view state for the room
[27:33.040 --> 27:38.880]  and then everybody else in the room will automatically also have the widget open for them
[27:42.000 --> 27:47.200]  so just added additional info so in the sovereign rack place we have the meeting widget which is
[27:47.200 --> 27:51.920]  like the start point for creating meetings with widget and there we have some kind of
[27:51.920 --> 27:56.560]  integration manager built in doesn't help for other rooms but there the user already has
[27:56.560 --> 28:03.840]  option to just tick the widget that they want for the meeting
[28:03.840 --> 28:09.680]  in this uh virtual presentation of the death room the widget which is showing me all the
[28:09.680 --> 28:16.080]  questions where the people ask the question in there which are most upvoted and there's a question
[28:16.080 --> 28:22.800]  they are asking how do widgets manage the anti-encryption so does widgets have access to the
[28:23.840 --> 28:30.800]  encrypted rules so the question was how do widgets manage anti-encryption and right now it's actually
[28:30.800 --> 28:36.800]  quite transparent to widgets because they don't know about anti-encryption so the client itself
[28:36.800 --> 28:43.360]  does everything and just returns the already decrypted events to the widget and the other
[28:43.360 --> 28:49.280]  way around so as a widget I just sent a message over the widget API and element for example
[28:49.280 --> 29:04.240]  does the heavy lifting of encrypting it and sending it to the room
[29:11.120 --> 29:18.560]  yes and no so element call itself is a widget too so it uses also the widget API and implements
[29:18.560 --> 29:25.040]  all its stuff also the matrix RTC stuff via the widget API so it's currently I think the code for
[29:25.040 --> 29:31.520]  that is mainly in the matrix JS SDK and then they use this room widget client to communicate
[29:32.480 --> 29:36.400]  via the widget API with the client and then the room
[29:36.400 --> 29:46.960]  um is there a place to discover widgets or is there a collection of widgets where to find them
[29:48.560 --> 29:54.000]  yeah actually it's quite hard right now so the question was how to discover widgets is a place
[29:54.000 --> 29:59.920]  a central place where you can find them I would say yes and no so there is a widget integration
[29:59.920 --> 30:06.320]  manage like managers like dimensions but it only has a subset of all the widgets that are available
[30:06.320 --> 30:12.560]  built in and there's right now not something like a store or collection where you can easily choose
[30:12.560 --> 30:22.560]  all of them I believe the matrix.org website either is already or is going to collect a list
[30:22.560 --> 30:35.200]  of all the available widgets so you can browse them there or yeah of course if you create a
[30:35.200 --> 30:42.560]  widget or any other app yourself you can make a pull request against the matrix website so please
[30:42.560 --> 30:57.120]  let everybody know about anything you build okay thank you very much