Welcome everybody. Good morning to our talk. All the people on stream on video on demand
and of course here in the deaf room. My name is Kim and I'm here together with my colleague
Oliver. We are from Nordic and we are going to talk to you about the matrix widgets and
particularly those we develop for the sovereign workplace project in the German public sector.
So at first, quickly about us, Nordic is an IT consulting company based in Hamburg, Germany.
We are about 40 IT professionals and we, among other things, we develop first matrix integrations
for productivity software in the public sector in the so-called sovereign workplace project.
And we are here today to provide you some insights from that we gained in the last couple years.
So what's the sovereign workplace? Let's take a step back and introduce you all to it.
It's a project with the goal of providing IT services to public administration in Germany
and potentially also Europe. It's founded by the German Federal Ministry of the Interior
and Community. And one of the core aspects is to gain independence from US cloud services
and to retain full control over your data. And there's this suit we call sovereign workplace
which covers many use cases for productivity and it's achieved by combining the products from
many different vendors which you can see on the right here. But all of this is open source software
so yeah you could say this whole project also supports and maybe funds some of the open source
projects here. We could talk a lot more about the whole sovereign workplace project and that
could be a whole talk for itself on another track but of course we are here to talk about
matrix and the matrix death room. So yeah quick overview here we can see the Univention
Corporate Server and Keycloak which implement identity management in this stack. There's
Groupware which is done by OpenExchange. There's cloud storage by Nextcloud. There is an office
suit which integrates with Nextcloud very tightly which is Kolobora online and of course
there's video conferencing which is done with jitzy right now and perhaps in the future using
matrix natively and of course there's messaging which is done natively on matrix and particularly
using element as a client. How are we involved in this? Well together with element we provide
this real-time communications component which is messaging and also video conferencing so there's
chat and video and in particular we are extending this chat use case using widgets for some specific
use cases so and this is the crucial part of this talk. The idea is to provide better integration
with other components in this stack and build new work solutions. So you might be familiar
with the concept of a widget and we've seen it in the previous talks but still I want to summarize
a bit. As I said we can use widgets to extend the regular chat client functionality for specific
use cases and I brought you some examples here so essentially it's an option and also this was
said before but it's an option to embed some kind of web app into your existing matrix client and
display more functions right there. So for example you can put it in the right bar like Hookshot does
here or you can pin it to the top of the room and it will even adjust to your theme like this
countdown and there's more there you can add a lot of widgets at once and see all of them at once
in multiple places. Of course there's video conferencing which you can also maximize to
view all of the people in the video conference rather than having chat and yeah there's also
this half full screen mode where the chat moves to the right side and you can see the widget in a
full screen-ish style and there's more so actually if you're following FOSTA remotely you might
find this a familiar site so you are actually using widgets right now to watch this and there's
this schedule there's the live stream and also from the editor side there are widgets to supporters.
So if you followed closely or you are very familiar you might have noticed wait a minute
this is a bit more than just static webpages so actually there's a thing called the widget API
and that allows your embedded page to interact with the client but also with the matrix.
So for example from our widgets we can also send messages read messages to the room.
So one way to explain widgets would be you could say these are some kind of limited matrix line
for a specific use case and you can build a lot of things really there are endless possibilities
and now Oliver is going to show you a bit more about how that works. Thanks Kim so let's
ahead a step back we're now back at the concept that it's a website that's embedded
so you have an iframe we'll see it soon and the iframe is just any static website and that
website has an API that can be used to communicate with the client so for example in this case
element and then you have the other option that the element client could be connected to your home
server so it's like I would say a pass-through API which gives the commons of the widget to the
home server via the widget API and with this API you can do a lot of stuff so one thing that is
important is that this iframe also allows to have like an isolation so even though you embed
a website there it's not like having full access to what the element does everything that you do
has to be done via the widget API so from the feature perspective of the widget API you can
separate it into two parts so on the one side you have everything that is related to displaying
widget interacting with the client for example Kim showed you that you can have this different
display mode there are also even more for example you can model widgets that then displayed in a full
screen view you can set things like always in top where the widget is displayed in the right bottom
corner and it's always there even if you switch to a different room and you can do capability
requests I will come to a second what that means and then you have the second group of features
which is like matrix or room interactions so that is everything about sending events receiving events
state events room events or to device messages as Florian also mentioned today yeah some special
things like reading relations and requesting ODC identity token that you can use at another
service to identify yourself so if you go now back to this reading and sending messages you
notice that actually you could use matrix for non-chat application where you store your data
in the room you can see it is like a real-time database we had the term today and store your
stuff there and have all the benefits of the matrix protocol that you would have for chat too
for example you can store your data and have it enter and encrypt it you can collaborate with
others via federation and all these features are already there and you can see it as like a nice
backend for building collaboration apps so I already mentioned this word widget capabilities
so one issue that you would have if you like provide all these features just to an app without
like having any security they could do a lot so actually I could build a widget that reads all
your data and post it somewhere like which chats you write or does stuff in your name for example
it could send messages in your name and therefore there is a security mechanism with this widget
capabilities and it's actually quite similar to what android is doing with permissions so your
web app once it is embedded and once you started it so every administrator can embed
a widget into your room once it's embedded you have the chance to
request specific permissions for your app and then the user gets a screen like this it's quite
a long list and can explicitly allow data to be shared with the widget and that way you have
your control over your data you don't share anything that you don't want to share with a
third party site and only then the widget has access to it so
but maybe grouping and writing best attacks can help here to avoid this situation where users
just click accept without thinking about what they are sharing
so now I would talk a little bit about with you about the status of widget API
so right now it's only supported completely in element web and desktop so
there is support for widgets in android and ios but it's more like static web pages without
the widget API and the widget API is what it gives is this interactivity stuff that you
wouldn't have with just a static web page yeah why is it only supported in that two clients
I think there are at least two reasons one reason is that it's not part of the magic spec yet so
there is like a draft here which is the spec but the draft is a bit outdated it collects
some of the mscs around it but a lot of the mscs are not part of the spec yet so it makes it
I guess quite hard to develop based on it so it makes it hard for a consumer of the widget
API to develop it but it would also make it hard as in a better because you always have to go
through the mscs look for stuff that you need and build it but I think it's not the only reason
why it's not yet fully supported by every client it's I think it's also the situation that it's
maybe not the perfect implementation yet um and what I mean with it I want to talk about it in
the next slide so if right now I want to extend the widget API with a new feature it's not feature
complete so it doesn't support everything you have in the matrix js SDK it has a subset of features
but if you want to bring a new feature into it um right now we have to follow this process so let's
assume that I want to support uploading content into the media repository so it would be quite
nice for some use cases um then I would create an msc which I think itself is not a problem
do you look into the current spec of the client server API look at the behavior for example
uploading requires like thinking about quota thinking about limits size limits and stuff like
that and then how the API responds these information back to you and uh so you look at the spec copy
that behavior so it's it's the same and think about how you integrate into the widget API
then the next step you think about like capabilities uh how can I prevent users from doing stuff that
they shouldn't do with it or how can I keep control about it and then you have like a base
to start the implementation you would probably implement it in the matrix widget API repository
which is the API that both embedders offer widgets of widgets use but also the widget itself
and the next step you would implement it in element and if there would be more clients that support
it in all other clients that you want to support it and that's actually a lot of work and just for
something that was already there right so if you wouldn't use widgets you could upload files with
the client server API and then you notice that you can upload files how to download them starts
the process over again so I think it's not the problem with the msc process but the problem is
that you have to copy stuff that is already there so behaviors that is already there specs that is
already there it would be much better if I could just use the client server API from my widget
and there's already like the idea of doing that so there's this msc that thinks about how can I
share like the client server API with my widget and it's done in this example or in this specific msc
by sharing the access token with the widget which brings like new challenges for example you have
to think about capabilities again but because if you share the whole access token then you can do
everything so you would have to think about how can I restrict the access again and the one idea
is there to use a scoped access totems that have there's actually an msc and I've missed it here
you use scoped access totem to create a totem that is like only possible one can only access
the stuff that the user previously allowed the widget so that's something where you can mirror
this capabilities that you've previously had with the widget API with this approach so that would
bring us a lot of benefits so we would actually directly have this feature parity with the client
server API we can do uh yeah it may be also more performant because now we rely on element loading
all the data uh and also relaying all our requests to the home server we could directly talk to the
home server and optimize our challenge or our tools and we would also assume that's a lot easier
to implement because the widget and better so a client like element only has to implement the
exchange of the credentials to the widget but not all the API calls that are available that
have to be relayed and capabilities have to be checked and all that stuff so it would
uh make the implementation a lot easier and hopefully also bring it to more clients
there are some challenges for example it's actually quite good that element does all the
stuff because as a widget author you don't have to do have to think about sync you don't have to
think about e2e uh e2e uh so end to end encryption so it actually makes it quite simple but probably
that's some challenges we have to solve then and uh there's also like this msc proposes to
bring the access token via the url into the widget which might also not be the best way
maybe or i'd see once it's there can help us here to delegate the identity and access into the widget
in the morning florian already talked about uh this matoska matoska mode for element call
and there they're using the i think it's called room with widget room client from the matrix
js sdk which is quite cool because it allows you to already start using the matrix js sdk
in your widget and it feels like it's a matrix js sdk but it's relayed over the widget api and uh
providing you later on maybe a better way to migrate to this state or to this style of api
so what would these features bring us or what can you already do with existing widget api
so you could be built really cool collaborative tools because you have like a real-time communication
channel you can build you will give some examples later so i don't give them um you can build stuff
where you would actually normally have to build a backend a communication layer and all this stuff
and think about a lot of stuff that matrix already has and brings you can use the rooms
for data storage um they actually some tricks needed to do that efficiently that would be like
the talk for itself um but you have the idea that all these um applications that you build
could just use it um i talked about real-time communication before so actually if you use
matrix you have some kind of very slow real-time communication it's not suitable for building
more complex or more more quicker stuff like for example a game or a whiteboard or stuff like that
that's where for example matrix fdc comes into play where you have direct p2p connections or
via sf use and if you have access to that in your widget that would be would allow you to build
really great stuff and actually if if like you reach your limits which with widgets you always
have the options to switch to uh yeah more components for example like a bot said is invited
into your room and helps you to um do stuff that you could actually not do with just the widget
it wants on the user side um yeah kim will give now some examples for widgets that we build
and also do does a quick demo thank you all over
so uh here are our use cases the widget that we build um so yeah as you can see there are four of
them and so the first one we built is the pulse widget as you might know there are
pulse in element now and i believe they are coming to the spec i think there's an msc
but yeah these allow you to do some simple pulse but you might have some cases where
you actually need to do some more fancy things like you want to even use parties for example
and so we built this pulse widget that allows you to do to cover many more advanced use cases
and in fact this is already open sourced the end of last year i believe in november december
sometime and you can find it online under this link on our github and so the next one is the
barcam widget if you are unfamiliar with the barcam the idea is to um meet in a group um collect
some topics and then build a schedule right there and then have an event based on that schedule
i'm going to show it to you in a minute and uh yeah so this is our second widget that is also
now open source and further we are also developing a meetings widget which allows you to create um
well appointments with within the widget and it will set up rooms for you and set up the possibility
to have a video call right there and this is working progress but will also be open sourced
and then there's also the whiteboard widget we are building and again this is going to be open
source at some point when it's when it's finished or in beta state um right so as i said i want to
show you the barcam widget for a bit and we actually got the chance to use it productively
on friday at our matrix community meetup and um yeah we have prepared a quick video for you
so right here on the left hand side you have the grid and you can for example add tracks you can
edit the track names you can choose some icons uh on that axis and of course you can also um
modify the other axis which are the different time slots you can move stuff around change the
length of stuff and then once that's set up you can go to you can enable the topic submission
and you and all the other users in the room can create this kind of posted cards where you enter
your topic and maybe a short description and then once you send it it will appear here on the right
in the parking lot but yeah for this because this is not yet quite supported on other platforms
than element web desktop and we also built a bot as a compatibility layer which allows you to also
submit topics through the chat right here you write a bot command the bot will convert it to the
event that's read by the widget and you see a tick and it also appears right here in the
select your topic button here's the first one we created at that point you can even edit
as a moderator and then you just move it into the schedule and then you can select the next
topic and also review it maybe you edit it maybe you don't and put it on your schedule
yeah you have the feature of locking and unlocking submissions for the non-moderator users in your
room and I believe that's it all right so yeah thank you very much everybody and it's now time
for qa we also have this qr code if you want to find us on matrix you can use the qr code you
can use this room alias and come talk to us of course you can also find us in the deaf room
online
right do you want to answer yeah okay so I think it was kind of two questions so one question was
how do users find the widgets that are installed in a room and then I think this second question
goes more into the direction of integration managers with dimensions so there's something
like that available so um widgets are like state events in a room um once you edit it to the room
you have like an event within url so it could be hosted on any web server that's then embedding it
and uh you have the question about discoverability so yeah there are integration managers for example
dimensions that you can use to add widgets but um at least dimension doesn't have set
good support for the widget that use the widget api so you would uh
uh probably need something else I don't know about any one or any integration
managers that support them very well
so right now you have to uh have the url of the widget and then you can just use the add
widget command and add it to your room um but an integration manager would be great so I could just
click in the bottom right of element this is already this integration for it
and then as a room admin or moderator you even have the ability to to pin the widget to the top
of the room for example or to maximize it and then to save that that view state for the room
and then everybody else in the room will automatically also have the widget open for them
so just added additional info so in the sovereign rack place we have the meeting widget which is
like the start point for creating meetings with widget and there we have some kind of
integration manager built in doesn't help for other rooms but there the user already has
option to just tick the widget that they want for the meeting
in this uh virtual presentation of the death room the widget which is showing me all the
questions where the people ask the question in there which are most upvoted and there's a question
they are asking how do widgets manage the anti-encryption so does widgets have access to the
encrypted rules so the question was how do widgets manage anti-encryption and right now it's actually
quite transparent to widgets because they don't know about anti-encryption so the client itself
does everything and just returns the already decrypted events to the widget and the other
way around so as a widget I just sent a message over the widget API and element for example
does the heavy lifting of encrypting it and sending it to the room
yes and no so element call itself is a widget too so it uses also the widget API and implements
all its stuff also the matrix RTC stuff via the widget API so it's currently I think the code for
that is mainly in the matrix JS SDK and then they use this room widget client to communicate
via the widget API with the client and then the room
um is there a place to discover widgets or is there a collection of widgets where to find them
yeah actually it's quite hard right now so the question was how to discover widgets is a place
a central place where you can find them I would say yes and no so there is a widget integration
manage like managers like dimensions but it only has a subset of all the widgets that are available
built in and there's right now not something like a store or collection where you can easily choose
all of them I believe the matrix.org website either is already or is going to collect a list
of all the available widgets so you can browse them there or yeah of course if you create a
widget or any other app yourself you can make a pull request against the matrix website so please
let everybody know about anything you build okay thank you very much