[00:00.000 --> 00:05.920] Good morning, wow. [00:05.920 --> 00:10.760] I was not expecting this much of an audience at 9am on Sunday at a FOSSTEM, so thank you [00:10.760 --> 00:11.760] all for coming. [00:11.760 --> 00:16.840] Yeah, I'm here to talk about how I was at FOSSTEM five years ago. [00:16.840 --> 00:20.240] I told you all a whole bunch of things and I was utterly wrong. [00:20.240 --> 00:23.520] So many ways, it's actually kind of amusing. [00:23.520 --> 00:25.160] But who am I? [00:25.160 --> 00:26.160] My name's Richard. [00:26.160 --> 00:32.040] I've been working on OpenSUSA since it began, I've been a customer of SUSAs, I've been [00:32.040 --> 00:39.240] a contributor, I've been a Q&A engineer, I've been working there for almost 10 years. [00:39.240 --> 00:43.200] These days I am a ridiculous advocate of rolling releases, it's what everybody should [00:43.200 --> 00:45.280] be using. [00:45.280 --> 00:50.360] I created the micro-S desktop, my day job is being one of the release engineers for [00:50.360 --> 00:57.480] tumbleweed and micro-S, I also do a bit of consulting and I also do a bit of photography. [00:57.480 --> 01:02.360] But a long time ago in a room, actually just on the other side of this campus, I was here [01:02.360 --> 01:11.080] at FOSSTEM telling everybody that containerized applications, so things like flat-back snap [01:11.080 --> 01:17.720] app images, the idea that graphical apps in some portable format are absolutely utterly [01:17.720 --> 01:20.800] terrible and nobody should be ever using them ever and they were going to eat all of our [01:20.800 --> 01:26.280] users and yeah, it's just going to be horribly, horribly wrong. [01:26.280 --> 01:31.160] And I even started the presentation with quickie comments, like those who don't remember [01:31.160 --> 01:38.040] the past are condemned to repeat it and I even made really unflattering comparisons. [01:38.040 --> 01:45.160] Like doing diagrams from Windows architecture and pointing out, Windows has all these wonderful [01:45.160 --> 01:50.160] runtimes where you can have different environments and run your application on top and it was [01:50.160 --> 01:53.560] absolutely terrible in Windows, it's going to be absolutely terrible when we do the [01:53.560 --> 01:56.920] same thing in Linux. [01:56.920 --> 02:02.920] Giving the examples of all of the security issues that you see in Windows in this kind [02:02.920 --> 02:04.000] of approach. [02:04.000 --> 02:11.240] Things like security relevant DLLs lurking in some folder in your Windows machine, being [02:11.240 --> 02:17.000] an absolute nightmare to patch, an absolute nightmare to fix when it goes wrong, all these [02:17.000 --> 02:21.240] horrible update issues, how do you end up getting an update on your Windows or your [02:21.240 --> 02:22.240] Mac machine? [02:22.240 --> 02:26.920] Well, you download some EXE or some bundle and then there's some updater and it does [02:26.920 --> 02:30.600] whatever the heck it wants on its machine. [02:30.600 --> 02:34.120] Licensing issues, especially with open source, how do you mix and match all these different [02:34.120 --> 02:39.360] licenses together in one cohesive thing and it's just going to eat up all of your disk [02:39.360 --> 02:42.360] space. [02:42.360 --> 02:48.840] And then I went back to this slide again and then started talking about the various technologies [02:48.840 --> 02:54.800] at the time, 2017, we're out there doing this containerized runtime stuff and I would compare [02:54.800 --> 03:01.200] this lovely Windows diagram to this lovely canonical diagram which looks very, very similar [03:01.200 --> 03:02.800] because actually it is. [03:02.800 --> 03:08.400] The idea is similar, the concept is similar, but as you'll see just because the concept [03:08.400 --> 03:13.800] is similar doesn't necessarily mean the whole idea is bad, execution doesn't matter. [03:13.800 --> 03:19.320] And it wasn't just Snap, I wasn't just shifting on Ubuntu because I don't like Ubuntu, I was [03:19.320 --> 03:27.560] doing the same with Flatback and I was basically pointing out that this whole containerized [03:27.560 --> 03:31.440] application idea was repeating the same issue. [03:31.440 --> 03:36.840] We were going to be going down this road of security relevant libraries, lurking in all [03:36.840 --> 03:39.960] of these snaps in Flatback. [03:39.960 --> 03:44.840] Back then we didn't necessarily have a good story about how are we going to update these [03:44.840 --> 03:49.200] things, how are we going to keep them maintained, who was going to look after all of these base [03:49.200 --> 03:54.920] snaps and run times in Flatback and the like. [03:54.920 --> 03:58.520] Who was going to look at all of the legal issues and review the possible licensing issues [03:58.520 --> 04:04.760] of bundling these things together and who was going to buy everybody bigger hard disks. [04:04.760 --> 04:09.440] And the kind of main conclusion that I left with which despite the fact you'll see I was [04:09.440 --> 04:15.760] wrong about a lot of what I said, I still actually hold true is at the heart of it when [04:15.760 --> 04:22.080] distributing software doesn't matter if you're doing it as a container or as a full-blown [04:22.080 --> 04:28.760] fat OS distributor or anything in between with any kind of fancy technology, the responsibilities [04:28.760 --> 04:35.880] are the same. Happy image, Flatback snap might make it easier to be the upstream than giving [04:35.880 --> 04:40.080] out your application to the users, that's great, but the responsibilities are still [04:40.080 --> 04:44.120] the same that distributors have been doing in distributions for years. [04:44.120 --> 04:48.880] You have to worry about maintainability, you have to worry about the security, you have [04:48.880 --> 04:51.880] to worry about licensing and all this wonderful stuff. [04:51.880 --> 04:55.160] So they're going to have to borrow all of the same stuff. [04:55.160 --> 04:59.640] So five years ago I gave this presentation, there was lots of people in the audience from [04:59.640 --> 05:04.040] App Image, Snap and Flatback, some of them said very nice things to me, some of them [05:04.040 --> 05:12.680] said very un-nice things to me. Starting with App Image, they took a lot of what I said [05:12.680 --> 05:22.360] surprisingly on board and really ran with it. I said all this stuff in February 2017 [05:22.360 --> 05:31.000] and by June 2017 I was saying stuff like this on stage, this was taken at the OpenSUSA conference, [05:31.000 --> 05:37.760] this was on the App Image website for, well, longer than I wish it was. [05:37.760 --> 05:44.800] But the reason it was because in that short window, App Images thought they could address [05:44.800 --> 05:51.280] most of my concerns by actually obviously running to the OpenSUSA build service and [05:51.280 --> 05:56.600] working with the OpenSUSA build service guys and integrating App Image really quite nicely [05:56.600 --> 06:02.880] with it at the time. So the idea being the problem, App Image wasn't the problem, maybe [06:02.880 --> 06:08.840] the way you build App Image is a problem. If you build them in a nice auditing build [06:08.840 --> 06:13.920] system and have the whole thing tracked with dependencies in a build system and you build [06:13.920 --> 06:19.480] it reproducibly and you do all the licensing reviews there, then OBS could be the solution [06:19.480 --> 06:26.680] to all of the App Images problem. And yeah, they worked really nicely with it and they [06:26.680 --> 06:32.480] gave all these promises, they'd be encouraging people to be using OBS as the main App Image [06:32.480 --> 06:40.240] building tool and we'd all move on happy in a nice unified way forward. And I even said [06:40.240 --> 06:45.880] things to Snappy and Flatback like you're falling behind App Image at this point, saying [06:45.880 --> 06:51.720] App Image had a better build story and they were working with other people and telling [06:51.720 --> 06:58.200] people to be more like App Image. And I still was badgering on, by the way, you can tell [06:58.200 --> 07:02.840] all my old slides because they have this thing at the bottom so you can see old me compared [07:02.840 --> 07:08.440] to new me. I was still worrying a little bit about dependencies because as you'll see, [07:08.440 --> 07:17.320] App Image makes some really interesting assumptions but I was, you know, tuned 2017 kind of hopeful [07:17.320 --> 07:21.880] that, you know, we'd get to a point where everybody would be working together and we'd [07:21.880 --> 07:28.080] have sort of maybe a new consistent run time and things could move forward. I was also [07:28.080 --> 07:33.040] hopeful that we might have sandboxing finally because, you know, Snapp kind of had some [07:33.040 --> 07:40.160] with App Armor. Flatback has Bubblewrap. You know, maybe App Armor would be the way forward. [07:40.160 --> 07:47.760] How wrong I was. So now, five years later, where are we? And I don't want to go deep [07:47.760 --> 07:53.200] down in technical issues too much because a lot of this isn't just technical. You know, [07:53.200 --> 07:57.760] we're an open source project. Any technical issue can be fixed, right? It is a lot about [07:57.760 --> 08:01.840] what are people actually doing? What do they actually care about? Where are they actually [08:01.840 --> 08:08.320] taking things? What are we actually doing? So let's judge people by their own standards. [08:08.320 --> 08:13.720] This is a screenshot from the current App Image website. And it says, use this to make [08:13.720 --> 08:21.280] Linux apps that run everywhere. But they don't run everywhere. And they say, as a user, it [08:21.280 --> 08:29.960] should be as easy to install as it is on a Mac or Windows machine. But they're not. And [08:29.960 --> 08:33.240] they say, you don't have to learn all these distributions with all these different distros [08:33.240 --> 08:38.520] doing things different ways. Technically, that's true. You just need to learn all these [08:38.520 --> 08:42.080] different distributions and doing all the different things. And you have to build your [08:42.080 --> 08:49.120] own to put in your App Image. And I'm not just, you know, saying this to, you know, [08:49.120 --> 08:54.840] core shade on them. You know, these, we have, I have users on micro OS who are trying to [08:54.840 --> 09:01.440] run App Images. And they can't because App Images require Fuse 2. I'm a rolling release. [09:01.440 --> 09:07.360] I haven't shipped Fuse 2 for like a year. I've been using Fuse 3. And you can't get [09:07.360 --> 09:13.000] an App Image to work with Fuse 3. It has to be Fuse 2. The portable image format that [09:13.000 --> 09:18.280] isn't portable because it makes assumptions about stuff that's on the base OS. And not [09:18.280 --> 09:26.200] just, you know, not just weird stuff like Fuse, but even down and dirty in the kernel. [09:26.200 --> 09:29.760] If you're running Debian and you're trying on an electron app, it's not going to work [09:29.760 --> 09:34.880] properly because the kernel in Debian isn't built the way that App Image is expecting [09:34.880 --> 09:41.880] the kernel to be running. So this is great promise. And it's going to work in some places, [09:41.880 --> 09:45.960] but only if you're lucky enough that your distro has the same assumptions baked into [09:45.960 --> 09:52.520] it that App Image has. And this is a recurring issue. Even reading the App Image documentation [09:52.520 --> 10:00.040] for building App Images, you know, it tells you, as a developer, think about all of the [10:00.040 --> 10:03.960] distros where you want your App Image to run on. So the whole promise of, you know, [10:03.960 --> 10:06.520] not worrying about distros goes away. You have to worry about more of them than you [10:06.520 --> 10:12.640] normally would. And put every single dependency which might not be fulfilled by that distro [10:12.640 --> 10:18.480] in your App Image. So, yeah, avoid distros by building a huge one and putting it in a [10:18.480 --> 10:25.400] big table. It's a lot of work. It's way too much work. I utterly respect anybody using [10:25.400 --> 10:31.360] it because they're probably doing more work than I am doing a rolling release. Especially [10:31.360 --> 10:37.760] when the recommendations for what you put in that giant App Image is the oldest, crustiest [10:37.760 --> 10:44.240] stuff you can find. They recommend avoiding using anything new because anything new is [10:44.240 --> 10:50.040] more likely to have compatibility issues with older distros. So literally find the oldest [10:50.040 --> 10:55.080] distro that's still supported and use that as your base for building App Image. Which, [10:55.080 --> 10:58.360] you know, also seems a bit of a problem to me because, you know, if you're always picking [10:58.360 --> 11:03.200] the oldest, the oldest is always the first one to not get maintenance updates. So you [11:03.200 --> 11:09.040] are always going to be rebasing on some crusty old almost out of day LTS to do what you want [11:09.040 --> 11:15.640] to do with App Image. It doesn't make any sense by their own standards. And they tell [11:15.640 --> 11:21.720] everybody that it's installing just like on a Mac. You know, just download the binary, [11:21.720 --> 11:27.880] put it on your desktop, right click it, make it executable and it will run. Which, you [11:27.880 --> 11:34.440] know, 15 years ago, that's true. That's how you run something on a Mac. I own a Mac now. [11:34.440 --> 11:40.240] That's not how you run stuff on a Mac. There's not a single Mac application I've ever installed [11:40.240 --> 11:45.560] that works that way. Even the Apple documentation makes it very, very clear that if you're downloading [11:45.560 --> 11:50.480] something from the Internet and you double-clicking it on a Mac, it's going to run an installer. [11:50.480 --> 11:54.920] Which is a terrible thing anyway. But it needs to run an installer. When you're downloading [11:54.920 --> 11:58.080] random stuff from the Internet, there needs to be checks for dependencies. There needs [11:58.080 --> 12:05.160] to be some, yeah, modification to what's on the host. So every random downloaded Mac [12:05.160 --> 12:11.520] application has an installer just like Windows. Or it's done in an app store where, you know, [12:11.520 --> 12:17.240] Apple are controlling all that kind of things and helping that along. So yes, I was wrong [12:17.240 --> 12:23.880] about App Image. First thing, it was terrible. Because they did try and make an effort. But [12:23.880 --> 12:29.640] then I was wrong again because it's actually even worse than I said five years ago. You [12:29.640 --> 12:36.120] know, they failed to do everything that they set out to do. They don't do anything to address [12:36.120 --> 12:44.120] the actual problems with software releasing. Dependency problems are just hand-wavy worse [12:44.120 --> 12:49.240] than anyone else could possibly do. Licensing issues, security, maintenance, good luck. [12:49.240 --> 12:55.800] Build a new distro and ship it again. And this is worse than we do in distros with all [12:55.800 --> 13:01.840] of the faults I will admit distros have on this. So please, do not use App Images. And [13:01.840 --> 13:05.960] also they're not nice people. Because they kept publishing this for like four years after [13:05.960 --> 13:13.000] I told them to take it down and I had to threaten to sue them. So they're just not nice. [13:13.000 --> 13:20.520] Now, SNAP. Despite my reservations back in 2017, actually SNAP was at the time the one [13:20.520 --> 13:26.560] I was most optimistic about. You know, at the time Canonical were actively collaborating [13:26.560 --> 13:32.120] with other distributions. They even invited me to a SNAP workshop trying to get SNAP supported [13:32.120 --> 13:37.960] in as many Linux distributions as possible. They had an approach of upstream first. They [13:37.960 --> 13:41.560] were promising that all of their app armor patches and all of the enablement they had [13:41.560 --> 13:46.160] to do was going to end up in the kernel and going to end up being upstream. At the time [13:46.160 --> 13:50.640] in 2017, you could run your own SNAP store. So you could have your own repository for [13:50.640 --> 13:57.680] downloading SNAPs. And unlike Flatpak where, you know, it's much more just graphical. They [13:57.680 --> 14:02.640] also had a story for, you know, non-graphical apps. And, you know, it's only five years [14:02.640 --> 14:06.840] ago. But back then, you know, everybody wasn't necessarily using containers for server stuff [14:06.840 --> 14:14.040] the way we are now. So, you know, it was interesting on all those levels. But it's five years [14:14.040 --> 14:23.120] later. And all of the promises of SNAP confinement working everywhere so you can have your nicer [14:23.120 --> 14:30.000] sandboxed SNAP application hasn't come true. You know, SNAPD does not support confinement [14:30.000 --> 14:37.320] on most non-Obuntu distributions. And even some Ubuntu distributions. And, you know, [14:37.320 --> 14:44.120] this was posted on their forums three years ago now. That was the case three years ago. [14:44.120 --> 14:51.000] You know, users still waiting to get any kind of proper sandboxing insecurity with SNAPs. [14:51.000 --> 14:57.960] Still not there. And then this was posted this month. Still promising. It might happen. [14:57.960 --> 15:02.160] But it's been five years. None of the app armor stuff is in the kernel yet. None of [15:02.160 --> 15:08.440] the enablement we need is in the kernel yet. Distros can't easily or really at all, you [15:08.440 --> 15:17.440] know, keep with an upstream kernel and get SNAP running in the way SNAP should be running. [15:17.440 --> 15:21.880] So if you run a SNAP on a non-Obuntu distribution, you're probably running it in an incredibly [15:21.880 --> 15:26.360] secure and insecure way. You know, do you trust that random software deliverable with, [15:26.360 --> 15:34.940] you know, access to everything on your machine? Probably not. At least that random software [15:34.940 --> 15:39.680] developer using SNAP isn't using their own SNAP store because they can't anymore. You [15:39.680 --> 15:44.800] know, 2017 you could. Then they released a new version of SNAPD. So now the only version [15:44.800 --> 15:52.240] of the SNAP store that works with SNAPD is Kmonocos. So, you know, it's an open source [15:52.240 --> 15:57.120] package format, but it's a closed source delivery format. You're only going to get that software [15:57.120 --> 16:03.040] from Kmonocos and if you read up on it, you know, there's lots of examples where Kmonocos [16:03.040 --> 16:07.800] have done the right thing and, you know, handled SNAPs that were, you know, malicious and got [16:07.800 --> 16:12.040] them off quickly. But it's like, how do you know, you know, you're just trusting Kmonocos [16:12.040 --> 16:15.640] that they're always doing the right thing because you can't see. You can't see what [16:15.640 --> 16:21.440] they're putting on there. You can't see how they get there. You can't do it yourself. [16:21.440 --> 16:27.960] You know, if you trust Kmonocos, that's fine. But, you know, I'm much more open source orientated [16:27.960 --> 16:32.000] myself. I'd rather, you know, even if I am trusting somebody else, I'd rather be able [16:32.000 --> 16:36.160] to have a look and see what's going on in there, maybe run my own, maybe compare something [16:36.160 --> 16:42.280] alongside, you know, and yes, for most developers or at least most small developers, this is [16:42.280 --> 16:48.960] free. So you can build your SNAP and publish it to the Kmonocos SNAP store, you know, with [16:48.960 --> 16:52.960] no effort. But as soon as you start getting bigger, as soon as you start becoming a bit [16:52.960 --> 16:57.880] of an ISV or doing stuff with IoT with lots of devices, then Kmonocos want you to have [16:57.880 --> 17:03.880] a brand store. And this isn't a documentation for SNAPcraft where it comes to building. [17:03.880 --> 17:08.280] When you actually have a look at the price list for having a SNAP store, you know, the [17:08.280 --> 17:14.680] price list is kind of dear. You know, do you really want to be spending at least 5000 euros [17:14.680 --> 17:22.200] just to be able to publish your application on somebody else's server under your name? [17:22.200 --> 17:25.680] But I can understand if people are buying into this, you know, I can definitely understand [17:25.680 --> 17:31.560] why Canonical Antenna Rush to change it. It's probably making them a good bit of money. [17:31.560 --> 17:37.800] On open-suited, like I said, at the time in 2017, they were working with us. Now, not [17:37.800 --> 17:43.920] going so well. SNAP is the only bit of software in all of my years doing anything, police [17:43.920 --> 17:49.480] manageried open-suizer where it's felt more than one security order. It's the only bit [17:49.480 --> 17:55.480] of software I've had to project in multiple times. And there was good collaboration going [17:55.480 --> 18:02.360] on to get those issues fixed, but since 2019, that's kind of fizzled out. Haven't seen anything [18:02.360 --> 18:09.960] since. So when it comes to SNAP, you know, I was wrong. I was really kind of keen on [18:09.960 --> 18:17.760] SNAP back in 2017. And these days, I can't really say that much nice about it. The upstream [18:17.760 --> 18:23.840] first promises have all stalled. It doesn't seem to be an effort to get it really moving [18:23.840 --> 18:27.760] again on other distributions. So, you know, it's not a portable format by any stretch [18:27.760 --> 18:34.120] of any imagination. There's no open-source delivery option, you know, even if SNAP, you [18:34.120 --> 18:37.960] know, the SNAP store may always be the best way of doing it anyway. There's a case to [18:37.960 --> 18:44.800] be made for that, even if there was an open-source way. And, you know, it's not really a viable [18:44.800 --> 18:48.840] alternative for something like Flatpak until, you know, unless you use Ubuntu, unless you [18:48.840 --> 18:56.720] trust Canonical, unless you're willing to give them money to distribute your stuff. [18:56.720 --> 19:04.840] And so, Flatpak. Now, I need to kind of do a little bit of a detour on this, because when [19:04.840 --> 19:11.600] I was talking five years ago about all of this stuff, one of the things that I was trying [19:11.600 --> 19:16.600] to pitch in the side thing there was this idea that, you know, well, everybody should [19:16.600 --> 19:21.440] be using rolling releases. I really, really believe that. And I still believe that now. [19:21.440 --> 19:31.080] And I really think, you know, in this modern age, to get applications in the hands of users, [19:31.080 --> 19:36.080] you know, a rolling-based operating system is the absolute key. You know, you need to [19:36.080 --> 19:42.800] have it all built together, you need to have everything, yeah, integrated, built consistently, [19:42.800 --> 19:46.960] tested consistently, and, you know, taking the fair share of the maintenance and security [19:46.960 --> 19:50.320] burden, and then shipping it all in a way that the users don't really care that everything [19:50.320 --> 19:55.840] is churning around underneath, you know, it just works. And at SUSE we've still been working [19:55.840 --> 20:03.080] on this. We have an operating system called Open SUSE MicroRS. Vanilla MicroRS is much [20:03.080 --> 20:11.280] more server-orientated. It's immutable, like CoreOS and other similar immutable platforms. [20:11.280 --> 20:17.080] Can't be modified during runtime at all. It's rolling, so changing snapshots, it's actually [20:17.080 --> 20:23.000] using the same code base as Tumbleweed, so every day, almost. It's small, but small enough [20:23.000 --> 20:29.120] to do the job that it's meant to do. And the assumption is, you know, that server is going [20:29.120 --> 20:35.080] to do just one job in a data center, so, you know, a VM running one RPM or a VM running [20:35.080 --> 20:40.400] containers, and then, you know, as many containers on top, but, you know, the job is a container [20:40.400 --> 20:45.680] from the operating system point of view. And this is working really quite well. In fact, [20:45.680 --> 20:50.080] SUSE also has commercial products based on this. SLEE Micro is based directly off Open [20:50.080 --> 20:55.640] SUSE MicroRS, the new SUSE Alp you might have heard of, where we're thinking of doing like [20:55.640 --> 21:01.760] a whole new ecosystem of enterprise distros. You know, that's building off what we did [21:01.760 --> 21:08.760] with SLEE Micro and Open SUSE MicroRS. But me, you know, I'm still a desktop guy. So, [21:08.760 --> 21:13.200] you know, doing this with my day job, I found myself asking, yeah, I found myself asking, [21:13.200 --> 21:19.080] was like, okay, so, I've got this nice small OS and it can run just one thing. You know, [21:19.080 --> 21:26.640] what if that just one thing with a desktop? And so, I started the MicroS desktop project, [21:26.640 --> 21:34.640] sort of alongside regular MicroS. And, yeah, basically it's a modern Chromebook-like, silver [21:34.640 --> 21:40.720] blue-like environment where you have a nice minimal base system. My recommendation will [21:40.720 --> 21:44.400] be running the GNOME one. That's the one that's most maintained with a desktop environment [21:44.400 --> 21:51.360] on top. And the basic configuration tools are, yeah, the in there, but everything else [21:51.360 --> 21:58.280] is provided by somewhere else. In fact, everything else is provided by Flaphack. So, this is [21:58.280 --> 22:01.600] one of the reasons why I'm doing this presentation. I kind of have to explain how in five years [22:01.600 --> 22:05.680] I went from Flaphack is the devil to Flaphack is the only thing you should be running on [22:05.680 --> 22:10.520] your desktop. Because I talked to some of the people that I was talking to back then and [22:10.520 --> 22:19.440] this is kind of their expression. Because five years ago, when I was talking about this stuff, [22:19.440 --> 22:25.360] I was meanest about Flaphack than all the other ones. I was even invited to Gwadek and [22:25.360 --> 22:29.880] I gave the meanest talk I have ever given to anybody right to the people who were actually [22:29.880 --> 22:38.520] developing the thing. And the guys from GNOME, they listened. I wasn't right. I'm not right [22:38.520 --> 22:43.600] about everything. That's the recurring theme of this presentation. But they challenged [22:43.600 --> 22:51.480] some of my opinions, but they accepted at least the cool ones that actually mattered. [22:51.480 --> 22:58.400] And Flaphack has changed. Like I was talking about earlier, responsibility is the key issue [22:58.400 --> 23:04.560] when you're talking about delivering software. No matter how you're distributing it. You [23:04.560 --> 23:10.400] need to be thinking about dependencies and licenses and maintenance and security. And [23:10.400 --> 23:15.080] one thing that Flaphack does very, very well is basically take all of that away from the [23:15.080 --> 23:21.960] distribution and make it the packages problem. Not great if you're a package, but they do [23:21.960 --> 23:26.520] it in a way that actually probably lowers the burden for everybody. So that's nice. [23:26.520 --> 23:30.960] Automation and technology is great. But really, dependencies become the issue of the person [23:30.960 --> 23:36.520] making the Flaphack. Licenses become the issue there. Maintenance, security, etc. So distros [23:36.520 --> 23:44.080] can stop worrying about it. And Flaphack does this very well with their runtime concept [23:44.080 --> 23:48.040] where if you're building an application for GNOME, you have a GNOME runtime. If you're [23:48.040 --> 23:53.760] building an application for KDE, you have a KDE runtime. Elementary have their runtime [23:53.760 --> 23:59.560] as well. And then for everything else, there's the generic free desktop runtime, which is [23:59.560 --> 24:06.880] a little bit heavier and clunkier, but gets the job done. And back in 2017, this terrified [24:06.880 --> 24:12.040] me. Not because there was competing distributions, because I'm used to competing distributions. [24:12.040 --> 24:19.240] The question was really, are these mini-distributions going to be maintained anything like every other [24:19.240 --> 24:25.760] distro out there? Are these going to handle CDEs well? Are they going to not have horrific [24:25.760 --> 24:31.240] licensing issues, etc., etc.? Well, they've been doing this for five years now. These [24:31.240 --> 24:38.520] runtimes are very well maintained. These are snapshots from their various git trees. They're [24:38.520 --> 24:44.000] all updating very, very quickly, keeping up with their respected upstreams of GTK and [24:44.000 --> 24:50.600] QT and what have you. Handling CDEs very, very well. I don't know more about that later. [24:50.600 --> 24:55.840] So basically, they're handling this just as well as any other distribution does. Maybe [24:55.840 --> 24:59.400] even better in some cases, because they're narrow in scope. They've actually got less [24:59.400 --> 25:07.560] work to do themselves than a full-blown distribution with tens of thousands of packages. [25:07.560 --> 25:12.000] So you've got your runtimes and you've got your Flatpak application on top of that. But [25:12.000 --> 25:15.040] what about the Flatpak client? Especially if you think about what I was just talking [25:15.040 --> 25:20.440] about with Snap earlier, with all of the issues with app armor and custom patches and what [25:20.440 --> 25:27.720] have you. Well, as a distribution guy, getting Flatpak in my distribution is really not that [25:27.720 --> 25:33.000] hard at all. You need to have the client on that. But you're not having to worry about [25:33.000 --> 25:37.360] a huge chain of dependencies and a whole bunch of plumbing to get it running. I don't need [25:37.360 --> 25:43.440] to have fuse to on my distro. All I need to have is bubble wrap, OS tree, and a couple [25:43.440 --> 25:49.040] of XTG packages. And they themselves don't really pull that much in as well. So it's [25:49.040 --> 25:54.480] small, it's simple, it's relatively easy, self-contained. Doesn't cause me huge build [25:54.480 --> 25:59.560] chains when I have to rebuild the whole thing in tumbleweed. It's a really nice ecosystem [25:59.560 --> 26:08.000] to just plop on top of my distro and then all of the applications come from Flatpak. [26:08.000 --> 26:14.040] From a licensing perspective, all the Flatpaks on FlatHub are checked. They all have to have [26:14.040 --> 26:22.160] some kind of license that allows open redistribution or legal redistribution. Or they do also support [26:22.160 --> 26:29.080] proprietary stuff. You can get a Spotify Flatpak. But obviously, you can't have the source code [26:29.080 --> 26:37.800] for the Spotify binary in their Git tree. So all of the proprietary stuff has to be pulled [26:37.800 --> 26:46.200] through by discrete declared links. And the Flatpak, specifically the FlatHub team, are [26:46.200 --> 26:51.880] checking that, verifying that things aren't changing there, not letting nasty things happen [26:51.880 --> 26:58.240] and binaries flip around. So at the very least, you may not know exactly what horrible thing [26:58.240 --> 27:04.040] is in this sandbox, but it's sandboxed. It's not much of a threat to your machine anyway. [27:04.040 --> 27:07.040] And you know it's the one that was sent at the submission time. You know it was the one [27:07.040 --> 27:11.240] that was reviewed. You know it isn't changing unexpectedly. So basically, it's as good [27:11.240 --> 27:18.080] or as better as any other distribution out there with their native packages. When it [27:18.080 --> 27:24.400] comes to maintenance, basically the same story. You know, just like open SUSE, FlatHub doesn't [27:24.400 --> 27:29.120] like Flatpaks to have distro-specific packages or Flatpak-specific packages. You know, they [27:29.120 --> 27:36.040] want everything upstream as possible. They have an incredibly robust build, test, publish [27:36.040 --> 27:41.120] workflow. They're not using OBS, I wish there was. They're not using OpenQA, I wish there [27:41.120 --> 27:46.720] were. But you know, what they're using is just as good, maybe in some way that's better. [27:46.720 --> 27:50.280] They can actually like give everyone nice test channels for testing their application, [27:50.280 --> 27:56.480] which I really think I want to copy sometime. But yeah, it's maintained. It's easy there, [27:56.480 --> 28:01.320] you know, easy for maintainers to keep their app maintained, and that is all ticking over [28:01.320 --> 28:09.080] nicely. From a security point of view, well, Flatpak is the only one that works everywhere. [28:09.080 --> 28:15.400] It's the only one that those applications are sandboxed. The portal concept where, you [28:15.400 --> 28:22.120] know, basically holes are pegged through the sandbox to give you things like access to [28:22.120 --> 28:26.240] the file picker and other parts of the file system, and the like, you know, has proven [28:26.240 --> 28:31.360] to be secure enough and, you know, expandable enough, you know, it's not great, it's not [28:31.360 --> 28:35.480] perfect, nothing ever is. But it's doing the job, and it's doing the job well, and these [28:35.480 --> 28:43.640] applications are working very well. And Flatpak CVEs, you know, happen very, very rarely. [28:43.640 --> 28:47.240] And when they do happen, they're not these terrifying, scary things, because the thing [28:47.240 --> 28:52.840] is architected very, very well. So, you know, the last CVE that I could find was in February [28:52.840 --> 28:56.920] 2002, you know, it was a medium score, it was fixed incredibly quickly, I think every [28:56.920 --> 29:01.800] distribution had no problem adding that, because, again, like I mentioned earlier, given the [29:01.800 --> 29:06.800] client is very well structured, you know, you don't have a huge dependency chain, even [29:06.800 --> 29:12.880] the most ancient of LTSS distros can then just happily get the patch in, get the thing [29:12.880 --> 29:20.640] running. So, when I started the microS desktop, I adopted Flatpak, some Flab, actually November [29:20.640 --> 29:24.520] 2017, so if you put the timeline in that, you know, I did change my opinion quite a bit [29:24.520 --> 29:30.240] from the beginning of February 2017 to the end. But I was using Flatpak as it was the [29:30.240 --> 29:36.640] one that I could work with, you know, I couldn't use Snap, couldn't use App Image. And I didn't [29:36.640 --> 29:40.640] trust it that much at the time, you know, I was thinking like you've seen with other [29:40.640 --> 29:45.400] distributions of building my own Flatpaks and using them rather than trusting Flatup, [29:45.400 --> 29:48.760] or doing like Fedora does with, you know, they build their own and then they also give [29:48.760 --> 29:53.320] Flatup with some kind of filtering. But I didn't really want to mess with that at the [29:53.320 --> 29:59.280] beginning of my project doing all of this, so I just opted for trusting Flatup first [29:59.280 --> 30:04.840] and then waiting for the problems to surface. And it's five years later and I'm still waiting, [30:04.840 --> 30:10.720] like we haven't had a single issue with the microS desktop where a Flatup application [30:10.720 --> 30:14.960] really got in the way and needed us to think, okay, you know, we can't trust these guys, [30:14.960 --> 30:18.880] we should start doing that. It just hasn't happened. You know, the few times an application [30:18.880 --> 30:23.400] hasn't worked right, well, we send a patch. We work with them because that's how open [30:23.400 --> 30:30.880] source is meant to work, right? So as a distribution guy, I've realized, you know, we don't need [30:30.880 --> 30:37.160] to be building these giant, humongous, huge code bases, you know, even though that's still [30:37.160 --> 30:42.640] what we do with Tumbleweed, you know, I don't meet myself. I'm purely a microS person now. [30:42.640 --> 30:49.120] All of my servers are microS. My desktop here is microS. I'm using a tiny 1,000 package [30:49.120 --> 30:54.760] fraction of my Tumbleweed code base. And everything else is coming from containers, some of which [30:54.760 --> 31:01.840] are built from that much bigger code base. All my graphical stuff is coming from Flatup. [31:01.840 --> 31:08.880] And my life is good and I'm happy. And this presentation is Libre Office from Flatup. [31:08.880 --> 31:12.880] So my final thoughts, which I realized I'm actually finishing a little bit early, but [31:12.880 --> 31:19.560] that's good. More time for Q&A. Flatpacks are ready for primetime. The other ones aren't. [31:19.560 --> 31:25.040] You know, don't use app image. Only use Snap if you trust Canonical. But, you know, we're [31:25.040 --> 31:29.280] here at Fostem. Flatpacks are the better way to go for people like you who are here at [31:29.280 --> 31:39.200] Fostem. And my system automatically updated in the background. Yeah. Desktop Linux distros [31:39.200 --> 31:42.760] do not need to package the whole world. If you're a distro builder, think about following [31:42.760 --> 31:47.160] the model we are doing with microS desktop. Think about, if not narrowing your scope [31:47.160 --> 31:50.880] because you're building the packages and you don't want to tell maintainers to go away, [31:50.880 --> 31:56.040] then at least just, you know, start drawing your focus more on just what you need to be [31:56.040 --> 32:00.480] doing. Start testing that part more. Start telling your users, you know, that's the [32:00.480 --> 32:05.440] bit you can really, really trust and, you know, give some secondary class to the old [32:05.440 --> 32:13.400] fashioned way of doing things. Yes. [32:13.400 --> 32:20.080] So you're telling us that Flatpacks run everywhere. Is that also true for different architectures? [32:20.080 --> 32:25.440] That is true at least for ARM. For Z probably not, but do you really have that many desktops [32:25.440 --> 32:28.320] in the main frame? Yes, of course. [32:28.320 --> 32:32.200] Yeah. Well, then that's something I'm sure the Flathop team wouldn't mind. Well, I'm [32:32.200 --> 32:36.960] sure we could get that working on Flatpack. Like if there's a need there, then also thinking [32:36.960 --> 32:43.280] about a risk drive, of course, and stuff like that. Yeah. But then that kind of, you know, [32:43.280 --> 32:47.680] point, actually nicely draws me to my sort of finishing point really, you know, none [32:47.680 --> 32:52.360] of this stuff is ever going to be perfect. You know, no technology ever is. That's why [32:52.360 --> 32:58.360] we do this stuff in the open. That's why we do this stuff open source. So when things [32:58.360 --> 33:02.240] aren't perfect and aren't the way they are, aren't covering a architecture that you want [33:02.240 --> 33:08.040] or whatever, you know, isn't it better to go to a project that is already going in that [33:08.040 --> 33:13.120] direction, that is trying to be available to everybody that is open to, you know, open [33:13.120 --> 33:19.640] to me yelling at them for months about how terrible they are and then work with them [33:19.640 --> 33:24.440] to get it all done rather than sticking in your own tiny little sandbox, doing it all [33:24.440 --> 33:29.440] on your own and then being burdened with it for decades. Like if you're doing graphical [33:29.440 --> 33:34.240] applications, this is the way we should be going. It's easier for package maintainers. [33:34.240 --> 33:38.760] It's easier for distros. It's easier for everyone to keep up. It's easier for users [33:38.760 --> 33:42.560] too. I mean, you just, you know, nice little web store. They click on what they want. You [33:42.560 --> 33:46.040] know, they can have the beta version if they publish in the beta version. It's, yeah, it's [33:46.040 --> 33:52.400] a nice way of getting stuff done. So, yeah, please, if you're doing anything with graphical [33:52.400 --> 33:58.840] apps, please get it on FlatHub. Please contribute to Flatpak. Please put Flatpak in your distro. [33:58.840 --> 34:12.200] And is there any other questions? Because, yes, right at the back there. [34:12.200 --> 34:17.960] You've addressed the outstanding question about CPU architecture, which is a great question. [34:17.960 --> 34:22.480] How do you feel about the fact, and I realize I'm asking a Linux question of a Linux distro [34:22.480 --> 34:27.280] maintainer, but how do you feel about the fact that containers tie everyone in the world [34:27.280 --> 34:34.720] to the Linux kernel interface as their interface shutting out other open kernel options like [34:34.720 --> 34:42.160] the BSDs from participating in that ecosystem and that the overall drive towards containers [34:42.160 --> 34:49.160] is further orphaning these already minimally represented, but very, very strong options [34:49.160 --> 34:59.840] in other kernels? They're strong. But I mean, I guess the recurring point I get to with [34:59.840 --> 35:06.480] all of this kind of thing is, you know, niche players are great for playing in niches. You [35:06.480 --> 35:10.880] know, when you're talking about something that needs to have widespread adoption and [35:10.880 --> 35:16.360] or widespread contribution, you know, some degree of centralization does make sense. [35:16.360 --> 35:19.200] It doesn't make sense for everybody to go make their own kernel. It doesn't make sense [35:19.200 --> 35:24.560] for everybody to make their own distribution. I would say it doesn't make sense for everybody [35:24.560 --> 35:29.960] to go packaging their own graphical applications 20 times over. So as hard as it is to say [35:29.960 --> 35:35.880] to somebody who's clearly passionate about other kernels and BSDs and what have you, [35:35.880 --> 35:42.240] I'm fine with containerization and these technologies dragging everybody to the Linux kernel because [35:42.240 --> 35:48.840] that's where the contributions are. So, you know, and as long as the Linux kernel is open [35:48.840 --> 35:52.920] to contributions and everybody can steer it in, you know, a good direction, I'm kind of [35:52.920 --> 36:01.960] okay with that. [36:01.960 --> 36:08.080] Thank you for your talk. I was with the presentation of 2017, so I think it's very nice that you [36:08.080 --> 36:13.480] changed the views. That year I also watched the presentation about Atomic from Fedora, [36:13.480 --> 36:18.280] so it was funny how those things interlapsed. I have a question about how you feel about [36:18.280 --> 36:26.320] the base system. You see currently there are trends like Nix and like SteamOS, which use [36:26.320 --> 36:31.960] like an immutable image as a base. How do you feel about that? [36:31.960 --> 36:35.680] I think immutable distributions are the way to go. I think if you're running Linux, it [36:35.680 --> 36:44.640] should be immutable. Immutability does bring with it a bunch of extra questions. For us [36:44.640 --> 36:51.600] as geeks, I think I can say that without insulting anybody in the room, we are keen to tinker [36:51.600 --> 36:55.600] with our machines and of course immutability quite often can get in the way of that. If [36:55.600 --> 36:59.200] you can't change your running system, how are you going to install that one little thing [36:59.200 --> 37:08.120] that you want? I think there's a sweet spot and I don't think some of the other distributions [37:08.120 --> 37:13.400] get it. You know, image-based deployments, you know, you've got a frozen image, you can't [37:13.400 --> 37:17.960] really modify that image or you have to build a whole new one. That's too much work. I don't [37:17.960 --> 37:23.240] like image-based immutable systems that much. Nix has an interesting way with everything [37:23.240 --> 37:28.160] being declarative, but it's a lot of hassle. Declaring everything, it kind of swings the [37:28.160 --> 37:36.120] other way for me, so I don't necessarily like the Nix way. OSTree has an interesting take [37:36.120 --> 37:43.120] on the whole thing, both from a user's perspective and the fact that it's immutable. It's nice, [37:43.120 --> 37:46.560] but then you end up with a million different layers of OSTree and that kind of just gets [37:46.560 --> 37:51.880] technically burdensome. Obviously, I work on micro-OS. I think we found that sweet spot. [37:51.880 --> 37:58.080] In our case, we're using BTFF snapshots to do all the magic underneath the hood where [37:58.080 --> 38:03.040] your running system never gets touched, but you can still do traditional package management [38:03.040 --> 38:07.040] against a new snapshot and that becomes your Nix Boot target. You never affect the running [38:07.040 --> 38:11.040] system, but you can do whatever the heck you want with your Nix Boot. Then if that Nix [38:11.040 --> 38:16.760] Boot goes horribly wrong, we just throw the whole snapshot away. It's super fast, super [38:16.760 --> 38:22.360] easy. It avoids all of that. You can still tinker with it, but unfortunately, the downside [38:22.360 --> 38:26.640] of that is I do sometimes have to tell people, don't tinker too much. The more you do crazy [38:26.640 --> 38:31.360] stuff, the more likely you're going to throw that snapshot away, but I think that sweet [38:31.360 --> 38:36.680] spot is better than super lockdown images or complete freedom of having to declare everything [38:36.680 --> 38:50.240] in a config file. Okay. Thank you for your presentation. I had never heard of Flatpak. [38:50.240 --> 38:57.120] On my Ubuntu, I'm using a Snap to install application. On my Mac, I'm using Homebrew. [38:57.120 --> 39:04.560] What do you think of Homebrew on Linux? I don't see the point of Homebrew on Linux. [39:04.560 --> 39:11.480] Yeah. Why? I get it on Mac. I've installed a few things on my Mac that I desperately [39:11.480 --> 39:20.720] need there, but my Mac, I use for photography. I don't do anything technical on it. Don't [39:20.720 --> 39:33.200] see the point. Okay. Thank you. How likely is it for the files stored in the home directory, [39:33.200 --> 39:39.080] especially the user files, to be affected if I roll back a snapshot after a failed upgrade? [39:39.080 --> 39:46.800] So yeah, that's a really micro-specific question that's called that. The way we do it on micro-OS [39:46.800 --> 39:51.400] is when we talk about the root file system, we're not talking about the root partition [39:51.400 --> 39:58.040] because we're using BTRFS. So BTRFS, you have this concept of subvolumes. We have a subvolume [39:58.040 --> 40:04.320] for literally everything where the data should be changing. So Home opt because that's third [40:04.320 --> 40:10.840] party, so it's not us, user local because, again, that's not us. Anything that isn't [40:10.840 --> 40:16.040] the distro is in a subvolume, and then the distro's root file system is just that last [40:16.040 --> 40:19.480] bit that's left. So that bit's read-only. That's the bit that's managed by the package [40:19.480 --> 40:25.880] manager. All the subvolumes are freely available in read-write. That doesn't make ETC a little [40:25.880 --> 40:31.320] bit interesting because that's the one folder where it's both. Distros put stuff in there. [40:31.320 --> 40:37.040] In micro-OS, we handle that with overlayFS right now, where we're basically taking copies [40:37.040 --> 40:41.600] of that, knowing what we put there, knowing what the user put there, or at least trying [40:41.600 --> 40:46.520] to, and then merging everything together so the thing works. Ideally, what we would like [40:46.520 --> 40:54.200] is everybody to start using, like most people already are, user for putting in distribution [40:54.200 --> 41:00.320] configs at USR. It should be in userlib or useretc or whatever. Just like you see with [41:00.320 --> 41:07.800] systemd, where distros put their distro config in userlib systemd, and then users put their [41:07.800 --> 41:15.080] local config in ETC systemd. That way works very, very nicely, but meanwhile, ETC is a [41:15.080 --> 41:28.360] bit of a mess, but a mess that we can manage. [41:28.360 --> 41:37.240] Thank you for the presentation. Why isn't FlatBag suitable for CLIs? [41:37.240 --> 41:52.440] It is suitable for CLIs. There's actually guides now for how to do that. The assumption [41:52.440 --> 41:58.320] is always probably going to be that it's graphical, but there's no reason why a graphical application [41:58.320 --> 42:04.480] can't start an X-term and run a CLI app. There's actually examples in the FlatBag documentation [42:04.480 --> 42:13.360] of how to do that. Generally speaking, for apps that might not fit that kind of model, [42:13.360 --> 42:21.360] I think a lot of that CLI or more service-based command-line-y stuff, that's handled so well [42:21.360 --> 42:28.720] by OCI containers, Podman, Docker, and the like. Why mess with that? You've got all those [42:28.720 --> 42:34.720] containers already out there. You've got everyone building the command-line tooling and server [42:34.720 --> 42:41.120] tooling in containers. That does very, very well in that context. It just sucks on the [42:41.120 --> 42:46.120] desktop, have FlatBag that just handles the desktop issue. You don't necessarily have [42:46.120 --> 42:53.360] to have one thing to do everything. I think FlatBag draws that line quite nicely, where [42:53.360 --> 43:02.040] it naturally starts getting painful when you head down that road. [43:02.040 --> 43:09.560] Any more questions? No? Well, hopefully I will see you all in a couple of years when [43:09.560 --> 43:26.200] I'm back in. Thank you very much.