We're ready for our next talk.
Mathieu is going to talk about MPTCP in the upstream kernel.
Thanks.
Yes, hello, everybody.
So welcome to this short presentation about MPTCP in the Linux kernel.
So it was a long road that started almost 15 years ago.
I'm indeed Mathieu Bartz, working at Tesserace in Nouvelle Anneur, so it's 30 kilometers
from here.
And let's start by a quick overview of the agenda.
So today I suggest to begin with a short introduction of MPTCP and its main use cases.
I will try to be quick for those who already know about that, but still trying to make
the concepts clear for everybody, hopefully.
Then I will explain what we can do today and what's expected for later.
I will finish by giving some explanation about why it took so long to have it included in
the official versions.
So MPTCP is short for multi-pass TCP.
This is an extension to TCP that breaks the assumption that a connection is linked to
a fixed pair of IP addresses and ports.
In one sentence, it allows to exchange data for a single connection over different paths
simultaneously.
Now that you can have multiple paths for the same connection, you can then have more redundancies,
more bandwidth, and many more things.
But enough with the nice definitions, let's have a look at a typical use case.
Here is a classical MPTCP use case with smartphone.
So a smartphone can typically connect to both Wi-Fi and cellular networks.
That's a completely different view from the 70s when TCP was designed and where everything
was fixed and clearly not transportable.
Let's take a typical scenario.
So you are here in the room connected to the Wi-Fi access point.
Quickly you realize that, A, you have enough and don't want to listen to me anymore, and
B, you got called by the smell of the fries outside.
You then decide to watch a video stream about the history of fries.
Why not?
On your smartphone and leave the building to get real once, much better.
Slowly, the Wi-Fi signal will become weaker and weaker and likely the video will stop.
It will only restart when the system detected the issue and each app will then have to handle
that by reconnecting to the server and then continue where it was if it can.
It's clearly not a smooth experience for both devs and users, of course.
In other words, do not leave the building if you don't have MPTCP on your phone.
Of course there are fries for everybody.
So I guess you already got that MPTCP is going to improve this situation.
And yes, it will because it helps supporting seamless handover scenarios.
MPTCP allows to create multiple paths for the same connection.
So these paths are called subflows and they look like TCP connection when you look at
packet traces.
Except that these packet content are the channel TCP option to let the client and server attach
new subflows.
They can also announce available IP address.
Of course they need to have some numbers to reassemble the data and more things.
Multiple paths can be used at the same times like here on the slide.
So with the same workout scenario, the frustration of being disconnected from one network goes
away.
Indeed, MPTCP can quickly take the decision to continue the communication on another path
and even use multiple paths at the same time when one can no longer cope with the demand.
This kind of use case is already supported by Apple with apps like Siri, Maps, Music
and others but also by Samsung and LG in some countries like South Korea and Turkey.
Another use case which is one that kept us busy for a bit of time at my company is the
hybrid access network case.
Many people are stuck at home with a not so great internet connection.
That's usually because they are using a couple line and are far away from the street cabinet.
Improving the situation is costly but also take time, especially if it is needed to deep
new and long trenches to bring fibre to home.
On the other hand, different assets of the network operator can be used, like the available
capacity on the mobile network, so I mean 4G and 5G.
With the help of a transparent proxy installed in the residential gateways for the client
side and the telco cloud of the operator for the server side, MPTCP is used in the middle
to offer more bandwidth to the end users.
One last use case that can be quite interesting is that MPTCP can also play a key role in
managing data between cellular networks like 5G and fixed one like Wi-Fi.
So the 3GPP, which is the organisation in charge of defining the 5G technologies, suggests
operator to set up an AT-SSS core function.
The goal is to use MPTCP to have a seamless handover between networks, so 4G, 5G and Wi-Fi
not to break connection when you go from one to another, but also to reduce the utilisation
of the mobile network and avoid the situation of these mobile networks in the future.
MPTCP is then part of 5G, but I cannot tell you if this is the same 5G as the one they
put in the COVID vaccine.
Anyway, enough with the theory, how do we use it and what can we do with it today?
So MPTCP in the upstream kernel is fairly new, a recent kind of kernel is required.
An application can create an MPTCP socket and use it like it would do with a TCP socket,
so it's just one line change.
You can see on the slide that IP Proto MPTCP is used instead of TCP.
So yes, the application needs to explicitly pick MPTCP, but it is also possible to change
the behaviour of existing applications by forcing them to create an MPTCP socket thanks
to LD preload.
It is also required to configure the network side to tell the kernel that multiple interfaces
can be used.
So tools like Network Manager and MPCPD can help doing that automatically, but it is also
possible to do it manually with the IP tool.
So it's probably better with an example.
So just install a recent GNU Linux distribution, so Fedora Ubuntu and you name it, then you
set up the network configuration.
So here in this example, you can see that we need to declare which other IP addresses
can be used to create new set flows.
That's for the client side, the top, and also to signal the IP addresses to the other side.
It is also needed to tell the kernel that the traffic generated from one IP should go
through the right interface.
So here we do that manually, but this can be done, of course, by a network manager and
others.
Finally, at the end, you can see that we need to run the application and here we use IPRF3
and we use it with MPTCP I just to force it to create an MPTCP socket.
So the last table Linux kernel support most of the protocol features.
So using multiple subflows, announcing IP addresses, priority, fast close, which is the equivalent
of TCP reset and many other things.
It also supports many socket options used by many apps.
So for example, TCP fast open can be used with MPTCP, for those who know what it is.
And it's also important to support these options because some existing application depends
on them and would fail if they are not supported.
It is also possible to retrieve information from the user space thanks to MIP counters,
so also an INET-DIG interface and MPTCP socket option, which is the equivalent of TCP info.
It's also important to mention that two pass managers are available and one packet scheduler,
but maybe better if I explain what it is.
So quickly just about the MPTCP path manager, so it's a component that is in charge of creating
additional subflows, removing them if needed, announcing addresses, priority, etc.
It is needed on both hands, but serve different purposes.
So for example here, it is traditionally the client who create new paths and the server
which announce additional addresses.
There are two paths manager available, one where the user can define global settings
to get the same behavior for all the MPTCP connection, that's the net name space, and
also another one where the KNL notifies MPTCP events to user space via net link and accept
commands to create, for example, new subflow, announce IP addresses, etc.
So in short, the user space can control the path manager and take decision per connection.
The other important component that I mentioned before is the MPTCP packet scheduler.
Its role is to decide on which available paths the next packet will be sent to.
So it can also decide to retransmit one packet to another path if needed, and that's what
we call a reinjection.
The packet scheduler relies on the TCP congestion control algorithm used on each subflow to
know if more data can be pushed.
But additionally, to better use all available resources, and sometimes limited buffers,
it has also to send packet in a way to reduce packet reordering on one side, but also on
top of that, it might decide to penalize some subflow that could impact the MPTCP connection,
because some networks are quite bad with losses, buffer loads, and others.
So the packet scheduler, in this case, might also be able to trigger a reinjection of data
from one subflow to another, like if a failure has been detected.
So there is an internal packet scheduler for the moment, and only one, but other ones will
be able to be built with EBPF.
So yes, we need EBPF for the packet scheduler, and not just to look cool, or to be accepted
to conferences.
In fact, EBPF here will avoid us to maintain all sorts of different packet scheduler in
the kernel.
It's a bit similar to TCP congestion control, there are few in the kernel, but sometimes
no longer maintained.
So quite a bit of work has already been done, and it is already possible to do some experimentation
if you use a development version in our Git tree.
But this work is currently on hold, because we ended up discussing the behavior of the
current in-canner scheduler and its API, and yes, some work is still needed here.
But there is also a system socket option that needs to be supported, but most likely they
are specific to some very specific use cases.
So it should be fine, but feel free to report them if some are missing.
And one last thing that is worth mentioning is the support of Golang.
As you may know, Golang does not depend on a C runtime library, or libc, and it is then
not possible to use the LD preload technique with mpcp is to use mpcp.
So the default net package doesn't allow application to create mpcp socket, only UDP or TCP, and
a feature request has been sent to let apps easily create mpcp socket.
But quickly the question Golang developers asked was, then why not using mpcp by default
when a stream connection is requested, so when asking for TCP.
And the proposition has been accepted, so we hope that stream application using the net
package will be able to create mpcp connection, and maybe later that will become the new default
behavior.
So I will now finish this presentation with a bit of history.
I think it is worth telling you that because it was not easy to get mpcp in the official
Linux kernel, it could be good to say a few words about that.
So still it was not as long and intense as having the full real-time support, and I see
that some people here really know what I am talking about.
The development of multi-pass TCP in the Linux kernel started in Belgium, at the university
in Luven and Ev, something like 15 years ago.
Surprisingly it didn't involve BS, no of course it did.
The legend says that the ID popped up when the young authors were drinking bees at a
crowd pub where the bartender was able to cope with the high demand by using multiple
bee pumps at the same time.
More seriously it started as a fork, but more to do some experimentation and to validate
the concept.
So at the beginning of his PhD, Sebastian just wanted to prove it could work.
He started to modify TCP by adding more conditions, so just if it is multi-pass TCP, do that
if not do something else.
Later, more people, mostly Christophe and Gregory, joined the project to help Sebastian.
They then took over his work to make it, let's call it, production ready, but also to be
able to reach high performances.
In other words, to get there, the modification in the Linux kernel were consequent and
optimized for the mpcp use case.
In parallel, mpcp v0 RFC has been published in 2013 and the same year, a big company with
a logo looking like an apple, if you see, announced its support for the client side.
And of course they needed to have the support for the backend side and I will let you imagine
what they used.
So if we concentrate on the very beginning of the project, we can say that it was easy
to fork, but you will pay for it.
Yeah, please don't read the two lines above out of the context.
But anyway, there are different utilization of a fork.
You can pick your level.
So I let you guess which one has been picked here, probably ultraviolence.
Maybe because the Linux kernel is big, it's also complex and the development is very active.
So small modifications should not be difficult to maintain in a fork, but here we are talking
about quite a lot of code and an important part is modifying the network stack, which
still has many adaptations specific to mpcp.
And in fact, from those that are even duplicated function that were adapted for mpcp case.
So imagine that the code is modified on TCP side, we don't see it directly and then we
need to adapt it later to mpcp.
But still that was not the nightmare level.
This is the nightmare level.
So imagine that you have to deploy it on various embedded system with different LTS kernels
from very old version like 3.4.
So that's what we had to do at Tesserace and my explain why some of my colleague here
look like the avatar just by mentioning kernel back ports.
In the meantime, very old version have been deprecated, but thanks to the embedded system
wall, this took time.
So of course, this back port brought the drought of having to deal with many conflicts.
But good tools like git re re re and topgit help a lot for that.
So also add to that a bunch of batch script and it was possible to automate most of this
laborious task.
Topgit allows us to create a tree with dependency, that's what we can not really clearly see
on the side, but it is also very handy if a fork has to be maintained by a team where
regular sync with the upstream have to be done as well.
So at the end for us, what we were doing is that we were applying the patch likely at
the bottom and then propagated to all the kernel versions and then we had to resolve
a few conflicts.
But likely we were not doing that too much.
At the end, the fork is still quite well used today despite all the work that has been done
on the upstream code.
I even published new releases last Friday and probably one of the last one.
But on the bright side, the migration process has started, wait, just take time.
The MPTCP support in the upstream kernel has started in 2020.
Why a so long delay?
Was it an homage to the Belgium Rideway company?
No, it was not in fact a new idea.
A few discussions and attempts have been made in the past, but were not successful.
In all case, it was not an easy task to upstream MPTCP.
Also because the Linux TCP stack is highly optimized, but also because the net dev maintainers
have been clear on that topic.
It is okay to include MPTCP in the official Linux kernel, but the new implementation cannot
affect the existing TCP stack, which means no performance regression maintainable and
possible to disable it can be extended via user space.
Now with what I said earlier, you might already understand that we are not allowed to take
the initial fork as it was.
So it was built to support experiments and rapid changes, but not generic enough.
Also at the end, it was and still used on environment where the majority of the connection
are using MPTCP and not the opposite.
So what were the solutions?
A rewrite almost from scratch was needed.
That's probably why it took so long to say, okay, we need to do it.
A key difference with the upstream kernel is that a new circuit type is used.
So there is no clean separation.
The user space interacts with the MPTCP circuit, which controls the different TCP sub-flows.
Thanks to the TCP upper layer protocol, ULP, that was introduced in 2017 for KTLS, it was
possible to minimize the modification in TCP code while still avoiding duplicating code.
An SKB extension mechanism has also been initially developed for MPTCP, not to include the socket
buffer size for the generic case.
This is also used now by other components today.
Also we had to be very careful when modifying the TCP stacks.
So any ID to avoid that were good to take.
One last point is that the APIs have been defined not to have to maintain multiple version of
pass manager and packet scheduler in the kernel, even if for the last one is still ongoing.
But also one thing that we needed to do a lot of work.
Here I just want to say a special thanks to our ex-maintenor, Matt Martino and other
fellows at Intel who had to step out very recently.
In conclusion, it was a long road and it's not over.
Thank you.
Thank you, we have time for a couple of questions.
Thank you.
Just two quick questions.
One, when you have multiple connections, can you kind of do it RAID 1 sort of style, like
where traffic goes on both simultaneously so that you don't have to resend something
if something gets dropped?
And can you speak also about SCTP and what's going on if it's dead or if, you know, because
it's sort of in a similar space and I never understood why people focused more on MPTCP
than SCTP.
Thank you.
I will maybe start just with the SCTP aspect because I don't know much about it.
From what I remember is that here with multi-pass TCP we do an extension to TCP.
So most likely where TCP was working before MPTCP can work.
There are some exceptions with some nasty middle boxes, but I think that's the main reason
why we can't see multi-pass TCP in the field and maybe not the SCTP.
I think it is not dead and still used for data centers, but I don't know exactly about
it.
For the other question, I might have not understood everything, you said that you wanted to aggregate
multiple paths.
You have your two paths, can you send the same data simultaneously?
Yes, you can.
So there is even a packet scheduler called redundant packet scheduler.
There is one small bit that is important to mention is that each path is still a TCP connection,
which means that if you have some losses on one path, you still need to retransmit it
on the same path.
So at some point it might be okay to say that, okay, the other side received it via the other
side, via the other path, so if you got a loss on one path.
So the end host doesn't need it, but because there are middle boxes and others on the path,
you need to retransmit it at the TCP level.
I don't know if it's clear, but so you can do re-injection, but you need to continue
retransmit on the same path too.
You can't just when you're trying to receive that request, just drop it.
No, if you want to do that, the best is probably to stop the connection, like if you want to
have a low latency thing, or if you want a low latency, maybe don't use TCP, but that's
another question, not the topic.
But if you want to do that, it's probably best to stop the pass and recreate it.
So I looked at the SysCityLs for MP TCP, and I found one called DSS checksum, and reading
the patch notes, it's something to do with middle boxes.
So is that giving you issues?
And last question, depending on that, why is it not on by default?
Yes, no good question.
So in short, middle boxes are not nasty.
They like to modify everything, and I will not comment too much about that because at
my company, we do a transparent proxy, so we are kind of middle box.
But what can happen is that middle boxes can change a lot of things in TCP.
For example, you have all protocols like FTP, where the IP address is sent on the by-screen,
but in clear text.
Which means that if you have a NAT, you probably have a NAT that starts to look at the connection,
identify it is FTP, and modify the text in the by-stream, like the IP addresses.
But because it does that, the size can change, and if they don't update MP TCP header, because
we need to add some information to be able to reassemble the data on the other hand, they
can mess up with MP TCP.
So there is this checksum mechanism.
But there is one big inconvenience is that for the moment, there is no hardware acceleration,
so it's quite costly.
And the other thing is that at the end, it's quite rare that you have some middle boxes
modifying the by-stream like that.
I know that in the past, you had some, if you were going on some website, for example,
for AT without HTTPS, it's possible that some by-stream were injected.
And probably when they do the injection, they don't modify MP TCP.
Sorry, we need to move on.
Yeah, sorry.
Otherwise, we won't be unscheduled.
So that's why we don't have checksum.
But thank you.
Thank you so much for the talk.
Thank you for the questions.
Thank you.