So now we are really getting a little bit after we dove a little bit in the old specs
and standards and you know like details of IMAP, we are now going to hear a lot about
Jmap, sorry, yeah, sorry.
We'll be talking about Jmap which is a new set of standards that has been engineered in
the last couple of years by some very engaged people that also in parallel have been contributing
a lot to IMAP and still do and we are very happy that we have one of these persons here
or representative from Fastmail which has been a company very instrumental and putting
a lot of effort in this new set of standards and yeah Rick your stage is yours, we have
learned about Jmap.
All right, applaud fast because I got a lot of slides and I got a little bit of time.
So we are going to talk about Jmap, funny story, I was pitched this talk where I was
going to talk about Jmap and I'm like what is it, how does it work, why is it so great,
how can you use it, how does Fastmail use it, all this stuff I covered everything, it was
a really good talk, it was like an hour long and I looked at my email that I was coming
here and it said you get 15 minutes.
So and it had to be in PDF so this is just like the absolute minimum dot PDF of my slides
and if you want to hear the whole thing and see all the builds and all the animations
and everything about IMAP and Jmap that can be arranged very easily, talk to me later.
That's me, I work at Fastmail, I'm not going to talk about myself, we don't have a lot
of time.
Let's talk about IMAP, sorry.
Who is here for the IMAP, what I wanted to know before I wrote IMAP library earlier.
You, okay well you missed a lot of horror stories but I'm going to give you some now,
this is IMAP and I'm going to be real brief about it.
What you're seeing here is the server in white, the client in yellow, we log in, we
says yeah you logged in and now we select an inbox, okay this is the IMAP protocol,
very basic but here's all the like beginning of the parts of the grammar you need to parse
and it's a bunch and if you were here earlier you're going to see lots and lots and lots
of more stuff, weird literals, like weird ways the interaction with the server changes
how you parse the response, synchronizing and non-synchronizing literals, it's a complicated
protocol and it's not like other protocols that you're using and there's a really simple
reason for that which I'll get to, oh yeah, right this is the protocol to do stuff and
then this is the payload of the message is MIME which is like another thing nobody wants
to deal with, it works great and it pays my salary but I mean, oh, see what you want about
HTTP and JSON but at least it's not this stuff, right?
Like you probably all know how to use HTTP even if you don't know how it works under
the hood and you probably know how it works because it's really nice and simple.
So I had lots and lots and lots of slides talking about how weird IMAP is and I would
love to tell you about it but I'm just going to tell you about this one thing and this
was touched on earlier, blah blah blah blah, server and client are talking and eventually
the client says I want to mark message 12 deleted, store the flag deleted onto that
message and the server says great, you have fetched this information and this is where
people get really confused and it comes down to something I didn't said earlier, the only
way to understand IMAP is that IMAP is a cache invalidation protocol, it's a protocol that
tells you what to do with your cache. So you've got a server and you've got a client
and the client can send basically the commands you expect like I want to fetch or update
or create or delete messages and the server's response is in response to that here is how
you should update your cache and if you don't think about IMAP that way you're going to
have a bad time. Everything, yeah everything works this way. If the client says I want
to work with the inbox it says select inbox and the server says there are 172 emails and
these flags exist which is a way of saying here's how to initialize your cache. When
you say I want to look at my new mail the client says fetch these things and the server says
you fetched these things which means put these in your cache. When you say I want to mark
this mail red you say store this flag and the client says put this in your cache. That's
how it all works and you have to start by understanding that even to understand IMAP.
You want to talk much more about IMAP. Okay there is one more thing though. This is another
fairly basic IMAP conversation where we're saying we want to come up to date and come
up to date is really important. See at the beginning we say queue resync that means we
want to quickly resynchronize our IMAP storage offline. So we say queue resync and that our
client state is 123. Just tells what the state was the last time we synced. And we get told
great your next sync is going to be 130. Here's all the changes to apply and when you're
done you'll be at state 130. Without this IMAP kind of sucks. I mean it's better than
pop but one of the great things about it is you can synchronize, go offline, come back
later and quickly get up to date no matter what else has been going on. Okay now you
understand IMAP. Good job everybody. Yep. Who wants to go implement it? Yeah these four
freaks. Okay the good stuff is good but the bad stuff sucks and there's so much bad stuff.
So like good stuff. You can resynchronize from previous session. Great. You've got a
domain specific model IMAP is built around email. Really nice. How about the bad stuff?
Okay the data format sucks, the transport layer sucks. The code that's out there is not great
mostly. The key features of IMAP aren't in the core protocol sometimes so you need to
make sure you've got the right extensions loaded, the right capabilities available or you're
implementing to the worst common denominator and there's way, way too many parentheses.
Okay so this is why we built JMAP. JMAP is the JSON Meta Access Protocol. It's just
JMAP, it's IMAP plus a thousand right. This is what it looks like. So already I hope people
are feeling better, you know what this stuff is right. We're posting a request, we're
posting a request to the JMAP endpoint and we say I want to get these emails. Great right
so just like everything else it's a restful protocol kind of. Here's what you get back
in response. You said you wanted to get emails one two three four, here's one of the ones
you might get. You did an email get, you're getting a list of messages, this one has
ID one and there's its subject and there's more stuff. But it looks like this, you can
parse this. Anybody knows what this means. Here's a bigger context of it so you can see
like there's an ID and there's parts of the body and the subject but the thing I want
to call your special attention to is it's got one simple date format. Yeah I mean you
can stop there and it'd be a pretty good improvement on IMAP and IM. But we're going
to keep going. Here's another thing, you can say I want to get, when the server responds
to you, it can say yeah you did just get these messages and by the way your email collection
is at state 616. It's just like that Q-resync thing. It's going to let you say later I've
got mail and it's got it all up to state 616. Hey server tell me what changed since then.
And the server replies and what it says is here are the changes. You were at 616, you
will be at 717. These two IDs were created and this one has changed in some way. And
then you can decide to do what? Update your cache. What do you do? Maybe you refatch those
messages. Maybe you just invalidate the local storage but you know how to change your cache.
It's just like IMAP. JMAP is a cache management protocol. It's just easier to use. Here's
another example. Email query is basically what we call search when you search your email
on IMAP. So we're going to search for mail that's been flagged and that's from me. Really
simple. And the response to that will look like this. You did an email query. Here are
the IDs that result from that. And the reason that it gives back IDs, it's about managing
your cache. You should have messages cached. If you don't have these, well now you can
fetch them. But if you did have them, why send you the messages back? You should have
a cache with these messages. If you didn't, you would go ahead and say great, email get
these messages. I didn't have them but I want them so you get them now. And it works great.
It makes sense. You can think about this really easily. But we should talk about IMAP again.
So in IMAP it works the same way. You say I'm going to search flagged messages and it
says here they are and then you say I'm going to fetch those. Right? Makes sense. Same thing.
IMAP and JMAP look the same in a lot of ways. This is what you don't always see in these
diagrams. Where the round trips come in. Right? First we search, goes to the server.
Server computes the answer, sends it back. They only say I need those messages. We say
give me those messages. Send it to the server. Server finds the answer. Server sends it back.
You're waiting for the speed of light back and forth twice. That's what happens here
too. Right? You say I want to do a query. I get the answer. I look for those messages.
It goes to the server again and it comes back. So the same waits sit here. But you don't
have to let them sit here with JMAP. Because when you write your query you can write this.
I want to do a query and a get. And what is the get going to fetch? I don't know the answer
yet. That's okay. You tell the server which IDs do you get? They came from another thing
I asked you to do. So get the IDs by looking at A. It should be an email query. Get the
IDs out of the response that you compute before you send anything back to me and do the method
call with those. It's called a back reference. And you can have a whole bunch of method calls
that back reference to one another to let the server do all the work and only do a round
trip back to you once. So you get one wait state. Really good. Okay. Couple more things.
This is a larger section of a JMAP query. I've put in some more things. I've been skipping
on these slides. Mostly you've been seeing this stuff, actual method calls. But what
up here is good too. This is called the using block. It tells you what capabilities you
want to use. This one's really simple. If you squint you can see we're using core which
is like, yeah, I'm speaking JMAP. And mail, again, I'm looking at mail. But you didn't
have to squint apparently. I had to build. But you can have lots of other capabilities.
At fast mail we have contacts and calendars over JMAP. And those are going through the
ITF now. They'll be RFCs and we have lots of other stuff too. What that means is if your
server supports mail and contacts and calendars and other stuff, when you come back from offline,
you can synchronize everything with the same request. Not just the same protocol, but hello,
I'm back online. Please get all the changes since my offline state and fetch the updates
to me all at once. You can also write your own custom data types for whatever appeals
to you, whatever your business needs to use, add it to your implementation. Because even
though the data types in JMAP are domain specific, we let you build your own. Anybody can build
their own just by describing how those methods will work. I'll talk about it just a little
bit. Fast mail uses for mail filters, your preferences, your credentials, your DNS, your
files and billing, all kinds of stuff. We just do over JMAP because it's great. Okay.
Getting close to the last things. We also give you event source. Event source is a long-running
connection. I'm old enough that I still call it combat, right? Like you connect to the web
server and you say, tell me when things change and you stay connected. And every once in
a while, the server sends you a little blob like this saying, oh, there's an update to
your email state. Oh, email and contacts have changed. And when that happens, what does
your client do sitting there connected? It invalidates the cache. It can refresh things.
It can update the screen immediately. So I'm at Paz this with something called idle, but
CalDav doesn't, CardDav doesn't. And when you do this on your mobile phone, idle is
not going to help you much because Apple sure as hell is not letting your phone sit there
with a connected TCP stream live to your IMAP server all the time. So people build these
interstitial servers instead of getting a web push which would just directly send your
phone a message. And JMAP supports web push. So you could just get real-time updates from
all these protocols. So this is our IMAP. We get rid of all the bad stuff just about
and add all this good stuff. JMAP and HTTP, anybody can use. Avoiding around chips by
combining these requests. Putting lots of data types in one place and real-time synchronization
and the cost is that not everybody's using JMAP yet. It's growing, but it's still pretty
early and there's way too many squiggly braces and double quotes. But like that's the price
I'll pay. Okay. So what now? You want to know how this works? The first thing you should
do is go look at this repository, fastmail slash JMAP samples. It's code that just does
some real basic stuff with JMAP and you won't understand it yet, but it's going to give
you an idea of what JMAP use looks like. Simplest form. Then it's time to read RFCs. Yes. Don't
worry. They're actually pretty good RFCs. You should look at these if you want to play
with JMAP. The first one is 8620, which is going to tell you what the basic methods are.
And then 8621, which tells you the data types. So 20 is going to tell you things like how
do you get, how do you set, how do you do changes, just what those are that work on
any data type. 8621 is going to tell you the specific data types that we use like mailbox,
thread, email and so on. Everything else, you just learn more data types in more, in
calendars and contacts, basically how the protocol works. You learn the data types on top of
the core methods. Some highlights from RFCs. Yeah, okay, I got it. A minute and 18 before
questions. Email is the most complicated data type in JMAP for obvious reasons. Emails are
big and weird and complicated. JMAP does a great job of making them easy to deal with.
Here's an email get. When you do a get, you can also say which parts of the thing do I
want to get. Don't get every property, just get pieces. So I might say I want the from,
to, subject, preview, like a little snippet you see in your mail client, and it's mailbox
IDs. So what do you get back? This. You have a build. Great. The to and from come back as
structured objects that have parsed the email headers for you. Nice. The subject comes back
decoded. That's ASCII, so that was a poor choice of string, right? But it comes back
decoded. The preview is decoded and mailbox ID is this weird set thing. Why is it an object
instead of just the one mailbox ID? Because it can be in multiple mailboxes ID. And if
you hit me up later, I can tell you about labels mode, which is what we use this for.
It's really nice. So the headers, you could fetch the subject, but you could also fetch
the header called subject. And when that happens, you get back the quoted printable, the literal
thing. But if you want instead, you could say, give me the subject, the literal bytes,
or give me all the headers, because maybe there's multiple subjects, or all the headers, but
decode the text. You can get anything like that. You've got no time left. I'll show you
this. When you fetch the body, you can get the blob ID. Don't do that. That's how you
have to mine parse it. Instead, you say you want to fetch the text bodies and all their
values, and you get something like this. Here's all the bodies you need to display the full
text of the message. Like there's no mind parsing, there's no remembering. What do you
do with, you have multi-part alternative and multi-part related? How does that, no. Just,
just do that. Okay. Yep. Time for Q and A. The first thing I will say is you can ask
me for more later, use fast mail, blah, blah, blah. How about questions?
All right. Same here.
Hi. Thank you very much. So one quick question about adoption. Did you reach out to, because
when looking at this protocol, and I've been playing around with it for some time now,
it looks fairly similar to whatever Google and Microsoft do.
I'm not familiar with those companies.
Yeah, yeah. Yeah. So is there any chance that these guys would be interested in adopting this?
Yeah. Yes. I mean, I think I can just say that. It's, you can imagine like Microsoft, Apple
and Google are all standing around a well in like a spaghetti Western with their guns at
each other, like who's going to change first?
Right.
Apple's client is by far the most popular mail client in use. Google's servers are the most
popular servers. If either one breaks, we're in. And I've spoken with people, these companies,
and they're interested, but of course, it's a huge amount of work on something that even
though it's clearly technically superior and a big win is a gamble. It hasn't won yet.
I'm pretty optimistic that we're going to see things happen, but I don't have any secret
knowledge.
Yeah. Thanks.
Hi, thanks for the talk. What's about JMTP?
Yes, JMTP. Yeah. So replacing, replacing server to server communication is a much more fraught
problem than replacing what your client does.
I would.
Yeah, yeah. So submission. Are you asking about submission?
Okay. So mail, MTA to MTA, right? Full exchange of mail between different, the Fediverse of
email, if you will. That's going to be SMTP as far as I know forever. I'd love to see
JMTP replace it, whatever the hell that is. But submission where your mail client says
I want to give this message to be sent, JMAP supports that and it's really, really good.
It has lots of really nice features. It has the ability to tell you, oh, by the way, that
mail you sent bounced. It has the ability to tell you how many people has it been sent to.
And the way that you create messages as a client author is much, much, much simpler. You don't
have to think about constructing mind bodies yourself. You can just say, here's some attachments.
Here's the text in the HTML and the server can do everything for you. So it does replace
that. It also, because it's one protocol, you're never like, I can fetch mail, but I can't send
mail because one server's up and one server's down. It just always works.
What do you do about encrypted messages?
So you're like open PGP or SMIME sort of things? Yeah. So what do we do about encrypted
messages? Punt. Well, so there are some RFCs about SMIME and handling SMIME messages, I
think all by Alexi, if not mostly by Alexi, that I would say are optimized for the server
having access to your key material, right? Is that a fair way to describe it? Yes. Yeah. And
there have been discussions about how we would deal with encrypted messages when the server
doesn't have your key material and only the client does. We've talked about it's complicated
and I think there's interesting things we can do. But generally, Jmap is built around
the idea that whatever the server can see, you can see. And encryption, as usual, makes
things less convenient. All right. Thank you again very much, Rick. I think it will be around.