[00:00.000 --> 00:08.560] We're ready for our next talk. [00:08.560 --> 00:12.640] Mathieu is going to talk about MPTCP in the upstream kernel. [00:12.640 --> 00:13.640] Thanks. [00:13.640 --> 00:18.720] Yes, hello, everybody. [00:18.720 --> 00:24.320] So welcome to this short presentation about MPTCP in the Linux kernel. [00:24.320 --> 00:28.040] So it was a long road that started almost 15 years ago. [00:28.040 --> 00:33.480] I'm indeed Mathieu Bartz, working at Tesserace in Nouvelle Anneur, so it's 30 kilometers [00:33.480 --> 00:35.000] from here. [00:35.000 --> 00:39.480] And let's start by a quick overview of the agenda. [00:39.480 --> 00:45.560] So today I suggest to begin with a short introduction of MPTCP and its main use cases. [00:45.560 --> 00:51.080] I will try to be quick for those who already know about that, but still trying to make [00:51.080 --> 00:55.440] the concepts clear for everybody, hopefully. [00:55.440 --> 01:00.440] Then I will explain what we can do today and what's expected for later. [01:00.440 --> 01:05.760] I will finish by giving some explanation about why it took so long to have it included in [01:05.760 --> 01:08.120] the official versions. [01:08.120 --> 01:10.960] So MPTCP is short for multi-pass TCP. [01:10.960 --> 01:16.000] This is an extension to TCP that breaks the assumption that a connection is linked to [01:16.000 --> 01:19.640] a fixed pair of IP addresses and ports. [01:19.640 --> 01:24.360] In one sentence, it allows to exchange data for a single connection over different paths [01:24.360 --> 01:26.600] simultaneously. [01:26.600 --> 01:32.160] Now that you can have multiple paths for the same connection, you can then have more redundancies, [01:32.160 --> 01:34.800] more bandwidth, and many more things. [01:34.800 --> 01:40.360] But enough with the nice definitions, let's have a look at a typical use case. [01:40.360 --> 01:45.360] Here is a classical MPTCP use case with smartphone. [01:45.360 --> 01:49.880] So a smartphone can typically connect to both Wi-Fi and cellular networks. [01:49.880 --> 01:55.640] That's a completely different view from the 70s when TCP was designed and where everything [01:55.640 --> 02:00.080] was fixed and clearly not transportable. [02:00.080 --> 02:02.080] Let's take a typical scenario. [02:02.080 --> 02:07.280] So you are here in the room connected to the Wi-Fi access point. [02:07.280 --> 02:12.560] Quickly you realize that, A, you have enough and don't want to listen to me anymore, and [02:12.560 --> 02:16.160] B, you got called by the smell of the fries outside. [02:16.160 --> 02:19.640] You then decide to watch a video stream about the history of fries. [02:19.640 --> 02:20.640] Why not? [02:20.640 --> 02:25.480] On your smartphone and leave the building to get real once, much better. [02:25.480 --> 02:32.640] Slowly, the Wi-Fi signal will become weaker and weaker and likely the video will stop. [02:32.640 --> 02:40.080] It will only restart when the system detected the issue and each app will then have to handle [02:40.080 --> 02:45.000] that by reconnecting to the server and then continue where it was if it can. [02:45.000 --> 02:51.120] It's clearly not a smooth experience for both devs and users, of course. [02:51.120 --> 02:56.560] In other words, do not leave the building if you don't have MPTCP on your phone. [02:56.560 --> 02:59.480] Of course there are fries for everybody. [02:59.480 --> 03:04.560] So I guess you already got that MPTCP is going to improve this situation. [03:04.560 --> 03:10.040] And yes, it will because it helps supporting seamless handover scenarios. [03:10.040 --> 03:14.800] MPTCP allows to create multiple paths for the same connection. [03:14.800 --> 03:20.680] So these paths are called subflows and they look like TCP connection when you look at [03:20.680 --> 03:23.840] packet traces. [03:23.840 --> 03:29.520] Except that these packet content are the channel TCP option to let the client and server attach [03:29.520 --> 03:31.440] new subflows. [03:31.440 --> 03:33.960] They can also announce available IP address. [03:33.960 --> 03:41.200] Of course they need to have some numbers to reassemble the data and more things. [03:41.200 --> 03:46.480] Multiple paths can be used at the same times like here on the slide. [03:46.480 --> 03:52.160] So with the same workout scenario, the frustration of being disconnected from one network goes [03:52.160 --> 03:53.160] away. [03:53.160 --> 03:59.280] Indeed, MPTCP can quickly take the decision to continue the communication on another path [03:59.280 --> 04:08.120] and even use multiple paths at the same time when one can no longer cope with the demand. [04:08.120 --> 04:14.600] This kind of use case is already supported by Apple with apps like Siri, Maps, Music [04:14.600 --> 04:21.760] and others but also by Samsung and LG in some countries like South Korea and Turkey. [04:21.760 --> 04:27.240] Another use case which is one that kept us busy for a bit of time at my company is the [04:27.240 --> 04:30.800] hybrid access network case. [04:30.800 --> 04:35.400] Many people are stuck at home with a not so great internet connection. [04:35.400 --> 04:41.080] That's usually because they are using a couple line and are far away from the street cabinet. [04:41.080 --> 04:46.560] Improving the situation is costly but also take time, especially if it is needed to deep [04:46.560 --> 04:52.000] new and long trenches to bring fibre to home. [04:52.000 --> 04:58.160] On the other hand, different assets of the network operator can be used, like the available [04:58.160 --> 05:02.880] capacity on the mobile network, so I mean 4G and 5G. [05:02.880 --> 05:09.040] With the help of a transparent proxy installed in the residential gateways for the client [05:09.040 --> 05:16.080] side and the telco cloud of the operator for the server side, MPTCP is used in the middle [05:16.080 --> 05:20.640] to offer more bandwidth to the end users. [05:20.640 --> 05:26.440] One last use case that can be quite interesting is that MPTCP can also play a key role in [05:26.440 --> 05:34.080] managing data between cellular networks like 5G and fixed one like Wi-Fi. [05:34.080 --> 05:40.520] So the 3GPP, which is the organisation in charge of defining the 5G technologies, suggests [05:40.520 --> 05:45.760] operator to set up an AT-SSS core function. [05:45.760 --> 05:52.560] The goal is to use MPTCP to have a seamless handover between networks, so 4G, 5G and Wi-Fi [05:52.560 --> 05:59.200] not to break connection when you go from one to another, but also to reduce the utilisation [05:59.200 --> 06:05.880] of the mobile network and avoid the situation of these mobile networks in the future. [06:05.880 --> 06:12.960] MPTCP is then part of 5G, but I cannot tell you if this is the same 5G as the one they [06:12.960 --> 06:15.080] put in the COVID vaccine. [06:15.080 --> 06:24.200] Anyway, enough with the theory, how do we use it and what can we do with it today? [06:24.200 --> 06:31.200] So MPTCP in the upstream kernel is fairly new, a recent kind of kernel is required. [06:31.200 --> 06:37.880] An application can create an MPTCP socket and use it like it would do with a TCP socket, [06:37.880 --> 06:42.000] so it's just one line change. [06:42.000 --> 06:48.320] You can see on the slide that IP Proto MPTCP is used instead of TCP. [06:48.320 --> 06:55.120] So yes, the application needs to explicitly pick MPTCP, but it is also possible to change [06:55.120 --> 07:01.680] the behaviour of existing applications by forcing them to create an MPTCP socket thanks [07:01.680 --> 07:05.440] to LD preload. [07:05.440 --> 07:11.320] It is also required to configure the network side to tell the kernel that multiple interfaces [07:11.320 --> 07:12.320] can be used. [07:12.320 --> 07:19.480] So tools like Network Manager and MPCPD can help doing that automatically, but it is also [07:19.480 --> 07:25.080] possible to do it manually with the IP tool. [07:25.080 --> 07:27.840] So it's probably better with an example. [07:27.840 --> 07:34.560] So just install a recent GNU Linux distribution, so Fedora Ubuntu and you name it, then you [07:34.560 --> 07:36.840] set up the network configuration. [07:36.840 --> 07:43.080] So here in this example, you can see that we need to declare which other IP addresses [07:43.080 --> 07:46.880] can be used to create new set flows. [07:46.880 --> 07:56.080] That's for the client side, the top, and also to signal the IP addresses to the other side. [07:56.080 --> 08:01.000] It is also needed to tell the kernel that the traffic generated from one IP should go [08:01.000 --> 08:03.040] through the right interface. [08:03.040 --> 08:08.680] So here we do that manually, but this can be done, of course, by a network manager and [08:08.680 --> 08:09.680] others. [08:09.680 --> 08:18.600] Finally, at the end, you can see that we need to run the application and here we use IPRF3 [08:18.600 --> 08:25.360] and we use it with MPTCP I just to force it to create an MPTCP socket. [08:25.360 --> 08:29.840] So the last table Linux kernel support most of the protocol features. [08:29.840 --> 08:36.360] So using multiple subflows, announcing IP addresses, priority, fast close, which is the equivalent [08:36.360 --> 08:39.840] of TCP reset and many other things. [08:39.840 --> 08:44.080] It also supports many socket options used by many apps. [08:44.080 --> 08:49.240] So for example, TCP fast open can be used with MPTCP, for those who know what it is. [08:49.240 --> 08:54.640] And it's also important to support these options because some existing application depends [08:54.640 --> 08:58.760] on them and would fail if they are not supported. [08:58.760 --> 09:04.960] It is also possible to retrieve information from the user space thanks to MIP counters, [09:04.960 --> 09:14.720] so also an INET-DIG interface and MPTCP socket option, which is the equivalent of TCP info. [09:14.720 --> 09:20.360] It's also important to mention that two pass managers are available and one packet scheduler, [09:20.360 --> 09:23.560] but maybe better if I explain what it is. [09:23.560 --> 09:32.400] So quickly just about the MPTCP path manager, so it's a component that is in charge of creating [09:32.400 --> 09:40.080] additional subflows, removing them if needed, announcing addresses, priority, etc. [09:40.080 --> 09:43.680] It is needed on both hands, but serve different purposes. [09:43.680 --> 09:48.760] So for example here, it is traditionally the client who create new paths and the server [09:48.760 --> 09:53.760] which announce additional addresses. [09:53.760 --> 09:59.000] There are two paths manager available, one where the user can define global settings [09:59.000 --> 10:05.200] to get the same behavior for all the MPTCP connection, that's the net name space, and [10:05.200 --> 10:12.240] also another one where the KNL notifies MPTCP events to user space via net link and accept [10:12.240 --> 10:16.800] commands to create, for example, new subflow, announce IP addresses, etc. [10:16.800 --> 10:23.920] So in short, the user space can control the path manager and take decision per connection. [10:23.920 --> 10:29.680] The other important component that I mentioned before is the MPTCP packet scheduler. [10:29.680 --> 10:36.880] Its role is to decide on which available paths the next packet will be sent to. [10:36.880 --> 10:42.200] So it can also decide to retransmit one packet to another path if needed, and that's what [10:42.200 --> 10:44.920] we call a reinjection. [10:44.920 --> 10:51.600] The packet scheduler relies on the TCP congestion control algorithm used on each subflow to [10:51.600 --> 10:55.280] know if more data can be pushed. [10:55.280 --> 11:02.880] But additionally, to better use all available resources, and sometimes limited buffers, [11:02.880 --> 11:07.840] it has also to send packet in a way to reduce packet reordering on one side, but also on [11:07.840 --> 11:14.480] top of that, it might decide to penalize some subflow that could impact the MPTCP connection, [11:14.480 --> 11:19.960] because some networks are quite bad with losses, buffer loads, and others. [11:19.960 --> 11:26.600] So the packet scheduler, in this case, might also be able to trigger a reinjection of data [11:26.600 --> 11:33.600] from one subflow to another, like if a failure has been detected. [11:33.600 --> 11:39.720] So there is an internal packet scheduler for the moment, and only one, but other ones will [11:39.720 --> 11:43.800] be able to be built with EBPF. [11:43.800 --> 11:49.120] So yes, we need EBPF for the packet scheduler, and not just to look cool, or to be accepted [11:49.120 --> 11:52.600] to conferences. [11:52.600 --> 11:57.960] In fact, EBPF here will avoid us to maintain all sorts of different packet scheduler in [11:57.960 --> 11:58.960] the kernel. [11:58.960 --> 12:05.280] It's a bit similar to TCP congestion control, there are few in the kernel, but sometimes [12:05.280 --> 12:07.200] no longer maintained. [12:07.200 --> 12:12.760] So quite a bit of work has already been done, and it is already possible to do some experimentation [12:12.760 --> 12:16.800] if you use a development version in our Git tree. [12:16.800 --> 12:21.080] But this work is currently on hold, because we ended up discussing the behavior of the [12:21.080 --> 12:28.960] current in-canner scheduler and its API, and yes, some work is still needed here. [12:28.960 --> 12:36.000] But there is also a system socket option that needs to be supported, but most likely they [12:36.000 --> 12:40.840] are specific to some very specific use cases. [12:40.840 --> 12:45.800] So it should be fine, but feel free to report them if some are missing. [12:45.800 --> 12:50.000] And one last thing that is worth mentioning is the support of Golang. [12:50.000 --> 12:57.240] As you may know, Golang does not depend on a C runtime library, or libc, and it is then [12:57.240 --> 13:04.240] not possible to use the LD preload technique with mpcp is to use mpcp. [13:04.240 --> 13:12.520] So the default net package doesn't allow application to create mpcp socket, only UDP or TCP, and [13:12.520 --> 13:18.400] a feature request has been sent to let apps easily create mpcp socket. [13:18.400 --> 13:24.960] But quickly the question Golang developers asked was, then why not using mpcp by default [13:24.960 --> 13:30.160] when a stream connection is requested, so when asking for TCP. [13:30.160 --> 13:35.760] And the proposition has been accepted, so we hope that stream application using the net [13:35.760 --> 13:42.520] package will be able to create mpcp connection, and maybe later that will become the new default [13:42.520 --> 13:45.040] behavior. [13:45.040 --> 13:49.440] So I will now finish this presentation with a bit of history. [13:49.440 --> 13:55.560] I think it is worth telling you that because it was not easy to get mpcp in the official [13:55.560 --> 14:00.120] Linux kernel, it could be good to say a few words about that. [14:00.120 --> 14:05.600] So still it was not as long and intense as having the full real-time support, and I see [14:05.600 --> 14:11.920] that some people here really know what I am talking about. [14:11.920 --> 14:17.360] The development of multi-pass TCP in the Linux kernel started in Belgium, at the university [14:17.360 --> 14:21.400] in Luven and Ev, something like 15 years ago. [14:21.400 --> 14:25.720] Surprisingly it didn't involve BS, no of course it did. [14:25.720 --> 14:31.480] The legend says that the ID popped up when the young authors were drinking bees at a [14:31.480 --> 14:37.160] crowd pub where the bartender was able to cope with the high demand by using multiple [14:37.160 --> 14:42.640] bee pumps at the same time. [14:42.640 --> 14:47.920] More seriously it started as a fork, but more to do some experimentation and to validate [14:47.920 --> 14:49.520] the concept. [14:49.520 --> 14:54.600] So at the beginning of his PhD, Sebastian just wanted to prove it could work. [14:54.600 --> 15:01.560] He started to modify TCP by adding more conditions, so just if it is multi-pass TCP, do that [15:01.560 --> 15:03.640] if not do something else. [15:03.640 --> 15:10.520] Later, more people, mostly Christophe and Gregory, joined the project to help Sebastian. [15:10.520 --> 15:16.440] They then took over his work to make it, let's call it, production ready, but also to be [15:16.440 --> 15:19.520] able to reach high performances. [15:19.520 --> 15:24.240] In other words, to get there, the modification in the Linux kernel were consequent and [15:24.240 --> 15:29.600] optimized for the mpcp use case. [15:29.600 --> 15:38.360] In parallel, mpcp v0 RFC has been published in 2013 and the same year, a big company with [15:38.360 --> 15:44.320] a logo looking like an apple, if you see, announced its support for the client side. [15:44.320 --> 15:48.880] And of course they needed to have the support for the backend side and I will let you imagine [15:48.880 --> 15:51.160] what they used. [15:51.160 --> 15:55.920] So if we concentrate on the very beginning of the project, we can say that it was easy [15:55.920 --> 15:59.440] to fork, but you will pay for it. [15:59.440 --> 16:05.400] Yeah, please don't read the two lines above out of the context. [16:05.400 --> 16:09.680] But anyway, there are different utilization of a fork. [16:09.680 --> 16:11.920] You can pick your level. [16:11.920 --> 16:19.880] So I let you guess which one has been picked here, probably ultraviolence. [16:19.880 --> 16:25.600] Maybe because the Linux kernel is big, it's also complex and the development is very active. [16:25.600 --> 16:31.520] So small modifications should not be difficult to maintain in a fork, but here we are talking [16:31.520 --> 16:39.080] about quite a lot of code and an important part is modifying the network stack, which [16:39.080 --> 16:44.080] still has many adaptations specific to mpcp. [16:44.080 --> 16:51.080] And in fact, from those that are even duplicated function that were adapted for mpcp case. [16:51.080 --> 16:57.560] So imagine that the code is modified on TCP side, we don't see it directly and then we [16:57.560 --> 17:00.920] need to adapt it later to mpcp. [17:00.920 --> 17:04.160] But still that was not the nightmare level. [17:04.160 --> 17:05.960] This is the nightmare level. [17:05.960 --> 17:12.600] So imagine that you have to deploy it on various embedded system with different LTS kernels [17:12.600 --> 17:17.280] from very old version like 3.4. [17:17.280 --> 17:22.120] So that's what we had to do at Tesserace and my explain why some of my colleague here [17:22.120 --> 17:28.240] look like the avatar just by mentioning kernel back ports. [17:28.240 --> 17:33.560] In the meantime, very old version have been deprecated, but thanks to the embedded system [17:33.560 --> 17:36.240] wall, this took time. [17:36.240 --> 17:44.000] So of course, this back port brought the drought of having to deal with many conflicts. [17:44.000 --> 17:49.280] But good tools like git re re re and topgit help a lot for that. [17:49.280 --> 17:55.440] So also add to that a bunch of batch script and it was possible to automate most of this [17:55.440 --> 17:58.600] laborious task. [17:58.600 --> 18:04.080] Topgit allows us to create a tree with dependency, that's what we can not really clearly see [18:04.080 --> 18:11.760] on the side, but it is also very handy if a fork has to be maintained by a team where [18:11.760 --> 18:16.360] regular sync with the upstream have to be done as well. [18:16.360 --> 18:21.520] So at the end for us, what we were doing is that we were applying the patch likely at [18:21.520 --> 18:29.280] the bottom and then propagated to all the kernel versions and then we had to resolve [18:29.280 --> 18:32.920] a few conflicts. [18:32.920 --> 18:36.920] But likely we were not doing that too much. [18:36.920 --> 18:42.320] At the end, the fork is still quite well used today despite all the work that has been done [18:42.320 --> 18:44.960] on the upstream code. [18:44.960 --> 18:51.840] I even published new releases last Friday and probably one of the last one. [18:51.840 --> 18:57.760] But on the bright side, the migration process has started, wait, just take time. [18:57.760 --> 19:03.200] The MPTCP support in the upstream kernel has started in 2020. [19:03.200 --> 19:05.000] Why a so long delay? [19:05.000 --> 19:08.600] Was it an homage to the Belgium Rideway company? [19:08.600 --> 19:13.200] No, it was not in fact a new idea. [19:13.200 --> 19:20.360] A few discussions and attempts have been made in the past, but were not successful. [19:20.360 --> 19:25.320] In all case, it was not an easy task to upstream MPTCP. [19:25.320 --> 19:32.240] Also because the Linux TCP stack is highly optimized, but also because the net dev maintainers [19:32.240 --> 19:34.080] have been clear on that topic. [19:34.080 --> 19:39.240] It is okay to include MPTCP in the official Linux kernel, but the new implementation cannot [19:39.240 --> 19:45.760] affect the existing TCP stack, which means no performance regression maintainable and [19:45.760 --> 19:52.440] possible to disable it can be extended via user space. [19:52.440 --> 19:57.080] Now with what I said earlier, you might already understand that we are not allowed to take [19:57.080 --> 19:59.600] the initial fork as it was. [19:59.600 --> 20:06.320] So it was built to support experiments and rapid changes, but not generic enough. [20:06.320 --> 20:12.560] Also at the end, it was and still used on environment where the majority of the connection [20:12.560 --> 20:16.880] are using MPTCP and not the opposite. [20:16.880 --> 20:18.960] So what were the solutions? [20:18.960 --> 20:22.040] A rewrite almost from scratch was needed. [20:22.040 --> 20:26.440] That's probably why it took so long to say, okay, we need to do it. [20:26.440 --> 20:32.320] A key difference with the upstream kernel is that a new circuit type is used. [20:32.320 --> 20:34.520] So there is no clean separation. [20:34.520 --> 20:41.480] The user space interacts with the MPTCP circuit, which controls the different TCP sub-flows. [20:41.480 --> 20:51.120] Thanks to the TCP upper layer protocol, ULP, that was introduced in 2017 for KTLS, it was [20:51.120 --> 20:58.360] possible to minimize the modification in TCP code while still avoiding duplicating code. [20:58.360 --> 21:06.360] An SKB extension mechanism has also been initially developed for MPTCP, not to include the socket [21:06.360 --> 21:08.960] buffer size for the generic case. [21:08.960 --> 21:13.400] This is also used now by other components today. [21:13.400 --> 21:17.640] Also we had to be very careful when modifying the TCP stacks. [21:17.640 --> 21:22.400] So any ID to avoid that were good to take. [21:22.400 --> 21:28.080] One last point is that the APIs have been defined not to have to maintain multiple version of [21:28.080 --> 21:34.240] pass manager and packet scheduler in the kernel, even if for the last one is still ongoing. [21:34.240 --> 21:38.800] But also one thing that we needed to do a lot of work. [21:38.800 --> 21:44.120] Here I just want to say a special thanks to our ex-maintenor, Matt Martino and other [21:44.120 --> 21:49.000] fellows at Intel who had to step out very recently. [21:49.000 --> 21:53.280] In conclusion, it was a long road and it's not over. [21:53.280 --> 22:02.600] Thank you. [22:02.600 --> 22:16.000] Thank you, we have time for a couple of questions. [22:16.000 --> 22:17.000] Thank you. [22:17.000 --> 22:18.600] Just two quick questions. [22:18.600 --> 22:24.120] One, when you have multiple connections, can you kind of do it RAID 1 sort of style, like [22:24.120 --> 22:29.720] where traffic goes on both simultaneously so that you don't have to resend something [22:29.720 --> 22:31.920] if something gets dropped? [22:31.920 --> 22:38.480] And can you speak also about SCTP and what's going on if it's dead or if, you know, because [22:38.480 --> 22:45.040] it's sort of in a similar space and I never understood why people focused more on MPTCP [22:45.040 --> 22:46.280] than SCTP. [22:46.280 --> 22:47.280] Thank you. [22:47.280 --> 22:56.960] I will maybe start just with the SCTP aspect because I don't know much about it. [22:56.960 --> 23:02.840] From what I remember is that here with multi-pass TCP we do an extension to TCP. [23:02.840 --> 23:07.720] So most likely where TCP was working before MPTCP can work. [23:07.720 --> 23:13.000] There are some exceptions with some nasty middle boxes, but I think that's the main reason [23:13.000 --> 23:18.360] why we can't see multi-pass TCP in the field and maybe not the SCTP. [23:18.360 --> 23:24.840] I think it is not dead and still used for data centers, but I don't know exactly about [23:24.840 --> 23:26.840] it. [23:26.840 --> 23:34.000] For the other question, I might have not understood everything, you said that you wanted to aggregate [23:34.000 --> 23:35.000] multiple paths. [23:35.000 --> 23:40.000] You have your two paths, can you send the same data simultaneously? [23:40.000 --> 23:41.000] Yes, you can. [23:41.000 --> 23:47.120] So there is even a packet scheduler called redundant packet scheduler. [23:47.120 --> 23:55.720] There is one small bit that is important to mention is that each path is still a TCP connection, [23:55.720 --> 23:59.920] which means that if you have some losses on one path, you still need to retransmit it [23:59.920 --> 24:01.240] on the same path. [24:01.240 --> 24:05.840] So at some point it might be okay to say that, okay, the other side received it via the other [24:05.840 --> 24:11.720] side, via the other path, so if you got a loss on one path. [24:11.720 --> 24:19.680] So the end host doesn't need it, but because there are middle boxes and others on the path, [24:19.680 --> 24:23.360] you need to retransmit it at the TCP level. [24:23.360 --> 24:28.160] I don't know if it's clear, but so you can do re-injection, but you need to continue [24:28.160 --> 24:30.200] retransmit on the same path too. [24:30.200 --> 24:33.920] You can't just when you're trying to receive that request, just drop it. [24:33.920 --> 24:37.960] No, if you want to do that, the best is probably to stop the connection, like if you want to [24:37.960 --> 24:43.160] have a low latency thing, or if you want a low latency, maybe don't use TCP, but that's [24:43.160 --> 24:46.320] another question, not the topic. [24:46.320 --> 24:52.360] But if you want to do that, it's probably best to stop the pass and recreate it. [24:52.360 --> 25:00.920] So I looked at the SysCityLs for MP TCP, and I found one called DSS checksum, and reading [25:00.920 --> 25:05.440] the patch notes, it's something to do with middle boxes. [25:05.440 --> 25:08.760] So is that giving you issues? [25:08.760 --> 25:13.160] And last question, depending on that, why is it not on by default? [25:13.160 --> 25:15.120] Yes, no good question. [25:15.120 --> 25:19.880] So in short, middle boxes are not nasty. [25:19.880 --> 25:24.360] They like to modify everything, and I will not comment too much about that because at [25:24.360 --> 25:30.200] my company, we do a transparent proxy, so we are kind of middle box. [25:30.200 --> 25:36.000] But what can happen is that middle boxes can change a lot of things in TCP. [25:36.000 --> 25:46.000] For example, you have all protocols like FTP, where the IP address is sent on the by-screen, [25:46.000 --> 25:47.840] but in clear text. [25:47.840 --> 25:53.240] Which means that if you have a NAT, you probably have a NAT that starts to look at the connection, [25:53.240 --> 26:00.200] identify it is FTP, and modify the text in the by-stream, like the IP addresses. [26:00.200 --> 26:06.200] But because it does that, the size can change, and if they don't update MP TCP header, because [26:06.200 --> 26:12.920] we need to add some information to be able to reassemble the data on the other hand, they [26:12.920 --> 26:16.160] can mess up with MP TCP. [26:16.160 --> 26:19.920] So there is this checksum mechanism. [26:19.920 --> 26:26.400] But there is one big inconvenience is that for the moment, there is no hardware acceleration, [26:26.400 --> 26:28.440] so it's quite costly. [26:28.440 --> 26:35.320] And the other thing is that at the end, it's quite rare that you have some middle boxes [26:35.320 --> 26:37.520] modifying the by-stream like that. [26:37.520 --> 26:42.600] I know that in the past, you had some, if you were going on some website, for example, [26:42.600 --> 26:48.040] for AT without HTTPS, it's possible that some by-stream were injected. [26:48.040 --> 26:52.920] And probably when they do the injection, they don't modify MP TCP. [26:52.920 --> 26:55.480] Sorry, we need to move on. [26:55.480 --> 26:56.480] Yeah, sorry. [26:56.480 --> 26:57.480] Otherwise, we won't be unscheduled. [26:57.480 --> 26:58.480] So that's why we don't have checksum. [26:58.480 --> 26:59.480] But thank you. [26:59.480 --> 27:00.480] Thank you so much for the talk. [27:00.480 --> 27:01.480] Thank you for the questions. [27:01.480 --> 27:12.480] Thank you.