which is, as you introduced, a complete groupware suit.
And I will start by pointing out the corner piece of all of this, which is map-ed map,
the messaging application programming interface, something that has started around in the early
90s, let's say, by Microsoft. And it can actually refer to a multitude of things, including
the data models. You have inboxes, folders, messages, attachments, and a relation between
one another. And then you also have the programming interface. So, for example, this is what Outlook
uses a C API that is in some DLL. But as we may see in the specifications, map is also
used to refer to network protocols, which makes this a bit confusing. Because a program
like Outlook could use the programming interface without ever talking map on the network. Or
we could have the inverse where there is no map programming interface, but there's map
token at the network. And so the leading map-ed product is, well, there's no question about
it, is just Microsoft, so exchange and exchange online, which is the, the first is the on-premise
server, which you can install in your own hardware. And exchange online is just this magic cloud
service that you have no more site into what happens. There's an extra interface provided
by the online services, which is Graph API, another rehash on how to access your own mailbox.
And maybe that will, in the future, be added to the on-premise version as well. We don't
know. And the coverage of the on-premise version is, well, it's not a problem. But it's a problem
of exchange is quite broad. There's over 100 documents, over 8,000 pages of documentation
that they've written in the past 10 plus years on the matter. And of course, because we have
our Unix mailbox formats and internet mail formats that were defined by, well, let's say
the internet community, the text-based formats that we all know and love. You also have to
support them in such a mappy world, because that's what everybody else talks in the transport.
And so coming to Grimunio itself, for the audience at Foster, it's probably easiest to say Grimunio
is a product and it's also something of a Linux distribution with various components.
We have our core central server, so what we call the information store, which holds the
mail and provides a basic API, like give me this message, give me that message. And then
we have added components such as the postfix that's well known for just handling most of
the SMTP delivery part. For file sharing, for example, we have also reused more software
because why invent your own? If you can make something else work rather well, and we can
use either own cloud or next cloud, they are pretty much the same still.
For chat, for chat we've used Mattermost and all of the integrated with, for example,
the user database that we utilize. It may be hard to see on the screen right now, so feel
free to click on the PDFs in the schedule. So our information server is called Gromux.
It wasn't really meant as an acronym, but if you find something that fits, feel free
to just make whatever you want of it. And it implements, so the first slide alluded to
27 protocols, it's a bit more, but it's still not all of them, all of the 100 documents.
And we just used and implemented the bits of the specifications that we needed to get
email clients running in the expected functionality, which is mail, calendar, contacts, meetings,
which is a part of calendaring. And so at the end of the list, you can see, for example,
PST and CFP, so we can import from most, so like 80% of the Microsoft's very own formats.
I'm still working on something more, but I need to see where that leads me, because
that's not documented. So when we do reimplementation of the protocols, what you more or less do
is you set up an SSL interceptor on either the local machine or, for example, a Linux
machine, use the same SSL certificate or redirect. The tools for this are Fiddler, which is something
of a wireshark in its own right, but more web-oriented. And then you just compare to
the specs if it matches them or not. And you've write your usual marshalling code, which is
serialization and deserialization to network protocols, because the Microsoft ecosystem
is very binary format heavy. So, yeah. Using MFCMAP and OutlookP1 can trigger the data
and the various actions on the mailbox and then look at what the requests look like. Again,
this screenshot doesn't do justice on this laptop, so please excuse that. So you see
the individual requests and can analyze them. So people have already written large amounts
of dissectors in the past, and one can use that. Furthermore, once you get the decoding
of the network bytes right, one can look at the logical structure at which point. It's
beneficial to just turn on request logging at whatever level there might be one issue
remaining. For example, when you just return the wrong data or unlocks are broken once
again. That's then where you can also step in with the buggers, for example.
We also implemented EWS. It's the exchange web services. It's more of an XML kind of
access protocol once again for the mailbox. Again, we start Outlook or other mail clients.
Look what the requests are, trying to make sense of it using specifications. And then
possibly replay those requests to an exchange server, because when you start out, you let
the Outlook connect to an exchange and see what is it going over the wire, and then it's
re-implemented in our own. So that's a bit of, it's a laborious task, but eventually
that too was finished. The tool we used here is called Postman. It's something like a
graphical lip curl if you want. And so we now have a little bit of a problem.
With the help of EWS, connectivity for Mac clients, for example. So this is Outlook for
Mac with my sample inbox here, just one attachment. And it also works with Mac mail, it's called,
which is Apple's own implementation of a group web client, I guess.
And so you have now access to exchange-ish calendars, but using of course our implementation
instead of Microsoft's.
A very small technical interrupt, sorry for that. I got some feedback from the back of
the room that it's quite hard to understand speakers in general. Is that right? Can probably
raise your hand if it's a little bit hard to listen. Okay, that's nice. I was with a
technical team at the sets that cannot amplify the speakers in that room because the acoustics
in these rooms generally is a problem. So what can be done is that we keep the doors
closed first of all, that people keep a little bit silent because it will reduce surrounding
noise at all and probably also as a speaker, you can try to speak up a little bit because
that would work.
Bring the mic.
If you might. Maybe just use one.
Turn it off.
Yes. Thank you.
So this works.
Seems to work better.
So in the course of doing that, I have and my team have found a number.
Like so. Directional microphones.
So in the course of all that work, we have identified multiple problems and specifications.
Things being underspecified or omitted outright. And so we just send a bunch of pull requests
to Microsoft in that regard and they've been accepted so far.
And so Dromot is the information store that we have. It uses SQLite. So it's also quite
snappy. I would have a demonstration, but this is a presenter laptop, so I didn't really
depend on it. So we can do all these various protocols, including the traditional RPC.
So Samba was a great help in that. On top of that is MAPI HTTP and it's EMS MDB.
These are all the binary formats that the classic outlook speak that you can run on
Windows. And as said, the Mac ecosystem uses EWS and your mobile phone. This is then on
the right hand side. It uses actually EAS, the active sync. So yet another protocol for
mailboxes. I think I'll be here implementing even more protocols for the next 10 years.
As I said, as I alluded to earlier, Graph API is the next hot thing on Microsoft. Yeah,
let's see where that goes. Yeah, some components are implemented in PHP 8. If you want, you
can also run them on 7, but who would want that? This comes from that there was existing
software that could be reused once again. And so that's our main binding if you don't want to interact with C++.
What does the feature hold? We would like to work on better utilization of accessing one
mailbox, like carrot mailboxes. You've never really seen any until you have a lot of
one store where a thousand users at once try to access it. Better improved support for
the internet formats would be very well done. We have some old internet mail pauses in there
for the RFC 2822 stuff and so on. I've started working on that using more of LibVmime, also
already properly realized by some other open source projects of the past. And of course,
reporting more errors in the specifications as I move along and find some time to deal
with the Microsoft paper. Because not all of the specifications are actually published
on GitHub, so all you can really do is make a normal issue here. I'd like to have that
in that paragraph edited and see where that goes. Thanks, that's my time. If you have
any questions, now's the time.
Maybe we'll change again. Thank you. Is there any questions? I see one here. Wait a second.
The current Outlook clients. Which one? Yes, the new one in the web one and the one right
before that was still the C implementations, the two for Windows. Which protocols do they
use? They're unused right now.
The Windows Outlook client as of today, so 2019, 2021, offers the classic binary protocols,
EMS, MDB and NSP. It can run them over DCR obviously or MAP HTTP depending on which features
the server advertises. So it can be proxied these days since 2013. And you can also configure
the Windows Outlook to use ActiveSync instead if you so want. But that way you don't really
have access to public folders or shared mailboxes right now.
A question about these various protocols that Microsoft made, how old are they and do you
think some of them are well designed or not?
What's your feeling about this? I would like to say at first I said, oh, well, this is
horrible, but then there's a bit of a storm, where you get into, oh, so that's what they
must have for that time. Certainly, so the DCE stuff comes from OSF as some of you might
know. So the people that invented X.400, the very terrible version of LDAP. So that's
the worst I guess. Rest is kind of sensible at some point. There's a bit of legacy baggage
and times when you think, but why? You got to do what you got to do. It's specified and
if you want to make it work with the various clients, you just have to do it and then move
on. And do you have client interoperability issue that different clients do it differently
or are the clients somewhat consistent? So under Windows at least, everyone uses mappi32.dll,
that's the C interface. That's, oh, how long has it been here? So mappi started, I believe,
in 92 in its very first incantation and then they, at some point, it was what it was today
about 96, I guess, outlook 97, maybe everyone remembers. That's where it, it still looks
the same like back then. So that's the approximate age of it. As said, you have the C interface
for programming or even visual basic interfaces. And then there's the network parts, how the
network protocols are, I don't really know because Microsoft only started specifying them
after the EU legislation, antitrust things and so on and so on. So you can see in the
documentation, the documentation was an afterthought as always, which isn't all that wrong in practice.
I mean, Perl is not really specified by its interpreter. Python is kind of specified by
its C Python interpreter until someone said, we need to improve that. So all good. But specification
is really like manual pages. You can't read it like a book. But if you try to find something
out, you use the index or just search function for a keyword and then go to all those places
where the keyword appears and then just read and then maybe magically you get a bright idea
what it was that it's supposed to be. It seems to me like in any large business, the people
go in and go out of jobs over the years and then, okay, so this paragraph has almost nothing
to do with the next one, but it's correct what's in there.
All right, we still have time for questions. Is there any? There's one.
I see here on the table of the C++ 70 protocols, there's also IMAP and POP3. Are they actually
re-implemented on the project or you're using another backend server to actually implement
over them, exchange and so on?
We have an IMAP POP3 gateway on top of the information stores because I would love to
throw the one out that we've inherited over the years and use something like Dov and write
a plugin for Dovcott, for example. Unfortunately, Dovcott is not documented at all.
So right now I'd prefer, well, we have what we have, so that's that.
All right, thank you very much for your talk again.
Thanks.