Hi, everyone. So today I'm going to talk about the state of the rest SDK in 2023. So all
the things that we've accomplished as of last year and some of our future plans as well.
So first of all, who am I and how did I get into the rest SDK? Well, I'm Benjamin Bouvié.
I'm a software engineer in the Rust team at Element. Prior to that, I worked in a game
engine, well, a game dev company on a game engine that was written in Rust and WebAssembly.
And prior to that, I was a compiler engineer in the SpiderMonkey team, which is the JavaScript
engine, powering Firefox, where I did Rust and WebAssembly. So you can sense that there is a
common theme here. And back in the days at Mozilla, we were using IRC. And so I wrote a few
bots that were just pulling out jokes from the internet and posting them on the channels. And
then at some point, we decided to use this new cool thing called Matrix. And so I rewrote my
bots so that they could also run on the Matrix using JavaScript at the time, because when you
work at Mozilla, you have to bet on JavaScript all the time. And then a few years later, I decided to
rewrite it in Rust because I like Rust. And I made this framework system called Trinity that would
use Rust for interacting with the Matrix system. And then you can actually write the bot
comments themselves using WebAssembly, which is pretty sweet. And I experimented to
neutralize it in production. It's mostly a fun project. And that's how I started to use the
Rust SDK. So what is the Rust SDK? Very good question. So it's a Rust library implementing the
client server API to allow you to implement clients easily if you want to use Rust in your
project. So the code is available on GitHub under the FHT2 license. And it does all the things
that you would expect from a Matrix client. Logging in, logging out, sending messages,
receiving messages. But I guess the most interesting thing is that you get into an encryption for
free. And you don't have to worry about the, excuse my French, gory details in the sense that you
don't have to learn about Olm, Meg Olm, like sending, uploading your keys, claiming keys,
querying keys and all of that stuff, which is very fine. And that we handle for you. Some
history for this Rust SDK. So there was in the past one project that was called Ruma for Rust
Matrix, which modeled all the events that are happening in a Matrix room timeline. And also all
the requests and responses to the endpoints. And the goal at the time, I think, was to try to create
a home server in Rust. Eventually that didn't happen for the Ruma project itself. But people
realized that it was a good idea to actually model all those events, request and responses, and
reuse them across other projects. And there was another Rust home server that started to be
written and that is conduit. And like in another timeline in the world, so there was Damir, who is
now the team leader at the Rust SDK team at Element. He was doing Rust on his part like free
time. And he maintained a small plugin so that you can use WeChat with Matrix. And that was written
in Python. And so as he was trying to learn Rust, he decided to rewrite it in Rust. And the thing
is, well, he did so. He searched for library written in Rust to do that. And there was none. So he
decided to start one. And that's how the Matrix Rust SDK started. And from the outset, it started
to use Ruma because it made sense. And that allowed to reuse massive amounts of code, which was very
nice. And Damir, being a crypto engineer, he also implemented all the crypto stack, which was very
sweet. And then that was first in the Matrix Rust SDK. And all of that code was pulled out and
extracted as an independent library called Vodose Mac, which apparently in question means
amphibian. And it's like a big pun across languages like Olm, Megolm, and like all of these just
refers to amphibians, it seems. And yeah, so that's how it goes. All right. So why Rust, you would
ask, well, this is my minute for the Rust evangelizing taskforce. So I mean, you're probably convinced
if you're here already, but it's like at the same time high level and super fast. It allows you to
write code in a very fast fashion without having to worry about lots of low level details and
issues. It is secure and memory safe, which is very nice for a library because you want to have
something very robust. It has an amazing tooling and ecosystem, like all the packages, the crates that
are published on craze.io give you all the things that you want to have. And like the cargo, the
tool that does it all is just wonderful. You can run tests and, you know, the documentation and all
of that. You just want to also very important for the rest of this talk. It is compatible with
foreign function interfaces. So you can call into other native languages that speak the CABI. So
it's quite important to see. And one of the things that is maybe a bit undervalued in the Rust
community is that it's actually also in trying to empower you to write a multithreaded code without
you having to know too much about it, trying to make it very accessible. And it's a value that was in
the community first, and you can find it in all the places. It transpires in translates to all the
places in Rust from the error messages that just hold your hands and try to explain you what you did
wrong and try to tell you how to fix the problem that you run into, et cetera, et cetera. So it's very
sweet to use. And yeah, being a former C++ programmer, so there was this notice in one of the
offices where I worked before that read, you must be this tall to write multithreaded code. And it's
apparently at three meters high on the wall. So this is something of the past. Like with Rust, you
can just be fearless when you're writing multithreaded code because there is this thing called the
ownership model. And that makes it really easy to also model concurrent implementation of anything
really. So that's really, really nice. So why the Rust SDK? Well, there was this story where we had
three apps, basically Android apps, the iOS apps and the web version that is also powering the desktop
version. And they all were using a different SDK and a different crypto stack. So that means that if
you are serious about your security, and you want to, for instance, audit your cryptography, now you
have to do it in three places and make sure that every single implementation actually does what it's
supposed to do, which is a bit of a nightmare. And now you have also per platform issues. You can have
a bug in one stack, and then you need to check whether the other stacks also have it, et cetera,
et cetera. Well, now we are saying, no, we have only a single stack for the element apps, and it's
written in Rust. In particular, it's a single crypto stack. You have very high test coverage. As I'm
speaking, it's like more than 83% of test coverage in the Rust SDK. The VodoZemac library, the
crypto stack is being first as well, which is very important in terms of finding issues, security
issues. So that means it's a single place where you can add features, you can code once, where you use
everywhere the old Java Dream that everybody knows and loves about. All right, who's using it? So there
is Fractal, the GTK-based Matrix client, which is using it. There is IMB, terminal UI client, if you
like, Veeam bindings and all of that. There's the new generation of element apps. The element X apps
are only using that, which is pretty sweet. And also the crypto stack, as it could be extracted, and it's
also like there are specific bindings just for the crypto stack. And so it could be used in the current
generation of element apps. And it's another codename element R. And I guess that you can imagine what
the R stands for at this point. Rust. All right. So what happened since the last first time? Well,
the previous release of the Rust SDK was in October 22. So we made a new release this year. Yay! At the
beginning of this month. Thank you. So it's still not 1.0, still quite experimental. We're breaking
APIs all the time, but trying to do a better job at writing, changing logs and all of that. And we'll
see how it goes. So new features. So you probably heard about sliding sync last year. And this year,
the new kind of sync synchronization that makes it so that logging into a new device and retrieving
events is always instant, even if you haven't opened the app for months or years. So we entirely
support that. There is the basic feature that you can subscribe to specific rooms and list of rooms
of which we get a sliding window that is computed by the server. But we're getting rid of that, as
Matthew said. And it also implements a modular design in the sense that you have opt-in extensions for
read receipts and typing notices and many other things. And all of that is supported in the SDK. As you
can see on the right, it's quite verbose because, well, it's a very versatile and general like API to
give you the most control so that you can build higher level primitives on top of that. We'll get back
to that. And it's vitrugated behind the experimental sliding sync cargo feature. And you can, we basically
use it in production in element X. So it's quite stable, actually. There's also support for OIDC,
so OpenID Connect. It's a cross stack effort moving from the custom metrics authentication to
OpenID Connect. If you have a metrics authentication service running, so it's another service running
on your server alongside the Synapse or your own server, it can act as an actual OIDC provider or
specialized proxy to an upstream provider. So if you have a GitLab instance, for instance, you can
connect it to the metrics authentication system and then have your GitLab users log into matrix for
free, like that. And so that's the server side part. It's also written in REST, which is pretty
sweet because that means that the request and responses can be actually reused in the client, the
REST metrics SDK. And the SDK implements all of that already. And we are also using it in production
in element X. So it gives you all the things that you would like to do with OIDC, create, reload,
metadata, register on your OIDC client, do the login flow in all the steps and all of that. And
it's also behind the cargo feature at this point. Among the big news, we have a new default storage
backend. So the storage backend are implemented using traits, which are REST for interfaces. The
previous defaults when you wanted to persist things on disk was sled. And now it's been replaced to
SQLite because, well, pretty much everybody knows about SQL. And it's also much faster for our
use case. We still have an in memory backend if you don't care about losing states and an index DB
backend that is used when you're compiling for the web to WebAssembly. Some new cryptography
features. So there is this new thing called secret storage. And it's mostly an implementation
detail, but it gives you an encrypted key valley store that is backed in the user account data. And
where you can put any information that you would like to share across all your devices in a secure
way. Like the server doesn't know about this information. It cannot peek into it and know what
is in there because it's also encrypted. On top of that, we implemented key backup and restoration. So
that means that when you have a new device, well, when you're using elementics, for instance, it
will store all the room keys that are used for decrypting room messages in encrypted rooms in the
secret storage. And then another device can restore them so that you can actually see the
history of events before you joined with that new device. Also, in addition to that, we made it so
that the cross-signing automatically happens and you don't have to worry about this at all. That's
what's used to verify your own devices and other people's devices. And it's also like some of the
private keys are stored in that secret storage as well. And speaking of high-level primitives, so
we made a new crate, new package called the Matrix SDK UI. It is highly experimental and also highly
opinionated in the sense that we're enabling a few cargo features by default. And we are trying to
make it so that we implement the best practices in terms of user experience and performance. And
it's also as robust and tested as the rest of the SDK, which is very sweet. And we use sliding sync
as a foundation for all these new high-level features. One of these features is the room list
service, which, as its name suggests, gives you a list of the rooms. Yes, it does it so in a way
that we try to make it to show something to the user as soon as possible. So that's how you feel
that the app is kind of instant when you open the app, because it will try to load just one event
for all the rooms you were in, or for no, a few rooms you were in. So you have something to display.
And then in the background, once that's done, it will try to fetch more events. And also you can
configure it to say, this is a set of visible rooms in my apps. So because when you have an app,
you cannot show like a thousand rooms, you will only show a subset, right? So you can configure it
to say, this is the ones that are actually rendered on the screen. And those are prioritized so that
you get more events for these rooms. Another thing we added was the encryption service. So it's
basically a sliding thing that is just running encryption on the side, and it gives you access
to more concurrency with the other one. So think of it this way, the room list service, the one I
just talked about, when you're scrolling on a mobile app, it will change the list of rooms that are
shown on the screen, right? So that now means that it's sending new requests to ask for things. And
if we did the encryption in the same request, but it's getting a bit technical, but that would
mean that we would need to abort those requests and delay encryption. So now we have basically more
concurrency and more performance. And we can do the encryption task in the background while you're
still scrolling on the room list using this encryption service. And we also have a notification
service. So that's very specialized client that just handles push notifications. So if you're
given an event and a room identifier, we want to retrieve the event and maybe a bit of context,
like what's your name, what's the name of the person who sent the message to you and all of that.
It's also using a sliding thing for that. And it makes use of the encryption service because on
an encrypted room, of course, you would get a push notification for an encrypted event and the
server cannot know if it's a meaningful event, right? Maybe it's just a reaction, putting a
thumbs up on one of your messages. So we decrypt the event in the client itself and then we decide
whether it's worth sharing as a notification. The one fun thing, if you can call it fun, is on
iOS, if you want to modify the notification in case it's encrypted, it's running a separate
process. And that makes our life very hard because even if you're just decrypting data, the state of
the cryptography keys is mutably changed, right? So now we have multiple states that is global
across two processes that are sharing the same database. So we had to be a bit creative to solve
that issue and we are basically enabling the writer head log in SQLite and using some data in
the database to indicate who's the process that currently tries to read and write to the database.
So basically implementing a text like that. All right. And since we added those two services,
the encryption service and the room service, we wanted to make it very simple to just fire
synchronization and forget about it. So we made some nice high-level service that just wraps
the other two and you can just build it and start it and it will do all those things for you and
implement all the best practices and you don't have to worry about any of this. And then you can
just take listeners on that service and get information that is meaningful to do when we're
rendering for a client. Now that we have a list of rooms and decrypted events, what do we do? Well,
we want to display them and we have an API for that called the Timeline API. It's basically a
room view MVC, so model view components on steroids. The thing is that in the matrix protocol,
events are actually like atomic. It's an app and only database. So let's say you have a thumbs up
reaction to a message that is a response to something else that would be two events, like the
reaction itself and the message itself. So the timeline will aggregate all those different
events into a single timeline item that is much more what you want to render as a client on the
screen. So it makes it much simpler to render a single timeline like that. And it does a lot of
things for you too. It can enter local echoes. So basically when you're sending a message to a
room, you want to show it even before the server has returned that it received it. So it will do
that and then reconcile the response from the server with the local state and all of that. So
it's pretty sweet. And it's all observable, very reactive. So that's nice. You just get, like,
as a user of that API, you get notification that one item has been added or removed or updated.
And you can just, like, react according to that. So how is this all used in ElementX? We're using a
Mozilla project called Unify, Unify FFI. It will automatically create bindings for you for calling
interest from other languages. So at this point, we generate bindings for Swift on iOS and Kotlin
and Android. It can also generate bindings for other languages. And we use that for Go, for testing
purposes, I think. It requires a bit of integration with the foreign languages runtime. And over the
years, we've contributed a few PRs to this project. So we made it so that you can just use
procedural macros for exporting your types and your input blocks to other languages. And we also
added this year's support for async code. So you don't have to block when calling into an async
function on the Rust side. It will just look as an async function on the Kotlin or Swift side. And
you have actual concurrency and background processing happening, which is pretty sweet for
performance. And reactive programming in Rust. How do we do it? Well, the principle of reactive
programming is you have some data and you want to make it observable so people can subscribe to it.
And then they will get notifications. And I mentioned the Timeline API that will notify you
when there is a new Timeline item that has been added, removed, et cetera. So we're using
crates that we created ourselves, IBOL. And there's also an extension that is
div-based for collections because when you have a vector with a thousand entries in it,
you don't want to say, oh, now there's a new thing that has been pushed into the vector.
I hear all the 1,001 entries for that vector. No, you just want to hear that
there's a new entry and that's its position, right? It also has some extra querying facilities.
So you can batch all these updates, div updates. So you don't have to cross the FFI language
boundary too often. That has an inherent cost that we want to avoid, some overhead that we want to
avoid. And for your batch transaction, well, for your batch to be quite precise, you need to have
also transactions to say, this is the beginning of the batch, this is the end of the batch.
And also you can do some filtering on these stream of events, limiting, sorting. So it's kind of
mapping to things that you would do on SQL in general. It's pretty sweet and that's what we're
using, for instance, to filter the rooms in the room list immediately on the client side.
All right. So some of the future work that we're going to do, well, I intentionally remain a bit
vague here, but we're going to eventually support all the major features a matrix client would expect.
We are already working on Scrantum cryptography. And as of today, I think there has been a PR
against Voters and Mac to have something that is compatible with Libsignal and with what they do. So
that's pretty exciting. And there is a general theme of doing more things client side. When you have
end to end encryption, your server kind of becomes dumb sometimes because it cannot peak into the
encrypted event. And so you have to resolve a lot of things on the client side. If you get a
new event in a room, does that trigger a notification for an encrypted room? Well,
you have to push a notification and it's the clients that will decide whether or not it
resolves into an actual notification. And even for sorting the room list, you have to do it
client side because if there is some room activity, you want to sort by room activity, just show me
the room that have some activity. Well, it's the same thing. If the event was encrypted, you don't
know if it was just a thumbs up reaction. Maybe that doesn't justify putting the room at the top.
If it was something meaningful like an actual message. So that means that this task has to be
done on the client now. And yeah, we're also computing the other badges client side in the
rest SDK. So we are trying to be very careful to not get into static notifications situations
because it's a pain for everyone, us included. And yeah, that's pretty much it. All right.
Just a few things. Well, first to all the contributors of the rest SDK special shout out
to Kevin Komei from the fractal community. It's done like a bunch of work in the rest SDK,
including most of the support for OIDC on the client side, which was MaceFPR. And if you want
to be on this slide next year, you can contribute to we have a few issues that are tagged as good
first issues or help pointed if you want. And I would like to take this opportunity also to
thank elements for donating all of my work to the matrix organization. You can also be a supporter
of matrix if you want by following one of these two links. Thank you for listening.
And I would be happy to answer any questions if you have any.
To the internet asking why have you moved away from sled?
Why have we moved away from sled? That's a good question. So I think in terms of performance,
so slay this, if I recall correctly, I wasn't there when that happened. So it's kind of hard to
answer this precisely, but I think that it's a key value store embedded key value store. And
the performance was not great, especially on mobile devices. And we just figured that using
SQLite that has been like performance tested and improved and tuned over the years was
a right thing to do. And also the way you structure your data using a SQL database is quite
different from the way you would structure it with a key value store. So it's just like
slightly easier to perform requests when you have a SQL database because you know all of that.
Yeah. Any other question? The internet also asks how is your developer experience when using
UniFi, UniFFI in general? Are there any hard edges? That's a good question. So there's, oh yes,
so when using UniFi across, for calling rest across other languages, have there been hard edges?
Yes. It's been a few cases where we have a memory leak that is identified. Well, Kotlin uses the
GVM and GVM as a garbage collector. And so we accidentally, and when I say we, I think it's
like the UniFi group in general, introduced some leaks by having the equivalent of premises or
futures leak sometimes. So that was a problem, but usually it's, I would say it's 90% of the time
it's stable. And the 10% of the time where there is an issue, it's high priority for us because
obviously it breaks our apps. So we fix it, we try to fix it as quick as possible and we
contribute back. But most of the time it works fine for Kotlin and Swift. So the support and
stability is also per language, I suppose, since you have to create bindings for each language.
So yeah, I cannot speak for the Python or Go generation on the UniFi side. But usually,
since Mozilla also use UniFi, they have to provide high stability guarantees as well. So they are
pretty reactive and also fixing bugs. So it's working well. Yes.
I was wondering about the startup times.
Yeah, so the question was, what about startup times for the rest of the time? I think there were
two questions. The first one was just starting the SDK itself and then when you're syncing a list
of rooms, do you get instant and response and all of that? And well, it's native code, so you don't
have to boot up an entire VM for the SDK itself. So it's pretty fast. It will restore the state
from the disk. So that can be a slow step. But even like for users who have thousands and thousands
of rooms open and I'm looking at Matthew on the side of the room, our general benchmark runner,
it's pretty fast. And for receiving a room list, we are also tracking these performance of our time.
Pretty much instant. And every time there is an improvement that needs to be done, we'll do it.
Yeah. I mean, we are in sync with the synchronization times where about
between five to 20 minutes, if you are a very heavy weight user of Matrix,
now it's really up to three seconds. So consider that an improvement.
Any other questions? Yes?
What's the state of supporting extensible events in the rest SDK? So I think that's a question for
Ruma. And since we're using Ruma for passing the events, and I'm pretty sure that so the rest type
system is quite extensive in the sense that you can have union types. And for each event that can
be extended, I suppose that you can have there is a variant in that unit tab that says it's a custom
event. If you're referring to a specific MSC, I don't know what it is. And I'm sorry about that.
Was that a custom MSC or? No. No. Okay. Just events in general. So yes, you will end up in
these case where we have, you will match on this union type or the event and it will say,
well, it's something I don't know about. So I'm just ending it over to you and you do something with this.
Yes?
Can we also provide a question for the rest of the SDK?
I'll rephrase this question as are there plans to use the rest SDK for web?
Because it's not used. So right now, as we are speaking, as of last week, people have started,
well, have enabled by default for new logins on Element Web, I think. So that may be the nightly
version using the rest cryptography for Element Web. We have a separate repository for bindings
that are for WebAssembly because there's no meaning in using Unify for that. We can directly compile
the rest to WebAssembly. So no need to have an intermediary in the middle. And I think the
long-term goal is to use the rest SDK everywhere for the Element apps at least. So don't, don't
take my word as granted, but I think that this is going to happen. Yeah.
Any other question? Yes?
That's a very good question. So the question is, is search in scope for the rest SDK and what
kind of features would be out of scope for the rest SDK? So to respond for search, that depends
if you mean room search or message search, full text search. And well, actually it doesn't depend
because the answer for both is yes. We're going to try to take care of that. For full text, we,
there was a previous client made by Element called Hydrogen. That was a web client and that could do
that and had the fancy system to actually index the messages on your client and then share
parts of the index with your other clients devices. So we're probably going to reuse and
reimplement some of that in the rest SDK at some point. Yeah. In terms of what features are out
of scope for the rest SDK, it's kind of hard to tell, but I think that everything that is like
high level UI related like rendering widgets, but not in the sense of widget API, but actually
UI widgets and stuff like that is not something that we want to implement or provide.
And then I think that, well, the MSCs that have proven to be features that have been
proven to be not very useful will probably not be implemented. It's not clear what's not in the
roadmap at this point. Sorry, it's not a very satisfying answer, but yes, question here.
So the question was using the rest SDK can store a lot of data if you're listening to lots of events
and is there any way to limit that amount of data that is stored on this? Well, as I was saying,
the storage is implemented as a trait. So one could always implement a different version of the
Scallot backend and decide to drop items at some point. One thing that we wanted to make
it is the ability to store events locally. And that's connected to the previous question. If you
want to be able to do full text search, well, you have no other choice, but decrypting all the events
and storing them locally, at least in memory for some time to do the indexing. And then the indexes
have to go to the disk. And that means that, yeah, the size of the index can grow a lot.
And so we would probably have to implement some kind of garbage collection and say, well, we kind
of forget about like old data, older than like a month, year or something like that. And we only care
about the most recent data. All right, thank you very much.