[00:00.000 --> 00:15.560] All right. Well, welcome to the next talk. I'm going to be talking about open QA testing [00:15.560 --> 00:25.840] of a pretty complex graphical desktop environment. So, I'm an operating systems developer. I've [00:25.840 --> 00:31.120] been involved in GNOME for a long time, possibly too long. And I've also been involved in Code [00:31.120 --> 00:37.480] Think for maybe 10 years off and on. We're like a consultancy firm based in Manchester. [00:37.480 --> 00:42.200] And we work a lot with the automotive industry, helping them with testing. So, that's how [00:42.200 --> 00:48.680] we got an interest in open QA. And that led on to research of trying to set it up for [00:48.680 --> 00:58.240] the NOMO-S as well. It should be, but maybe it's not. It is green, but yeah, can nobody [00:58.240 --> 01:06.680] hear me? Okay, so there's no room speakers. All right, I'll try and talk a bit louder. [01:06.680 --> 01:11.560] So GNOME is a desktop environment. How many GNOME users do we have in the room? Quite [01:11.560 --> 01:20.280] a few. KDE users. Nice. Other desktop environments. Tiling window managers, et cetera. Quite an [01:20.280 --> 01:27.020] even mix, actually. Everybody welcome. So GNOME is quite an old project, right? GNOME predates [01:27.020 --> 01:34.120] FOSDEM, Git, and predate. It's older than Greta Thunberg. It's older than some of its [01:34.120 --> 01:39.120] contributors. It's older than MySpace. And this leads to some sort of technical challenges [01:39.160 --> 01:46.160] that have built up over the years. So, the GNOME designers design a cohesive experience [01:46.160 --> 01:51.800] of everything working together. But then we release more than 200 individual modules as [01:51.800 --> 01:56.480] tar balls. And distributions get to integrate those back together to produce something that [01:56.480 --> 02:01.280] hopefully works. So, it's difficult to test those 200 modules. It's difficult to test [02:01.280 --> 02:08.280] what we release. Maybe you've heard of Conway's law, the rule that a project source code will [02:09.460 --> 02:15.400] mirror the structure of the organization that makes it. So, this is a rough diagram of how [02:15.400 --> 02:20.520] GNOME development works. Most of the work is done by module teams who focus kind of on [02:20.520 --> 02:27.440] individual modules or one or two modules. So, things are tested well in isolation. And then [02:27.440 --> 02:30.880] the release team tries to build everything and they get to the point of like, okay, everything [02:30.920 --> 02:37.920] builds. So, we'll release it. And they give this to packages who give it to users. So, [02:39.040 --> 02:44.840] the question is, which of these groups are responsible for integration testing? The maintainers [02:44.840 --> 02:50.800] are working on isolated components. The release team are very busy. And distro developers [02:50.800 --> 02:57.320] are also very busy. So, certainly when the project started, the users were responsible [02:57.360 --> 03:02.080] for integration testing. You got to use Linux for free and you got to report bugs if it [03:02.080 --> 03:09.080] broke. And we would give the works on my machine certificate. And you get a lot of crazy bugs [03:10.240 --> 03:14.120] at integration time. Like, oh, this feature doesn't work because it turns out the code [03:14.120 --> 03:17.840] isn't broken, but you pass the wrong configure flags. So, you don't get the feature that [03:17.840 --> 03:24.840] you wanted. Time has passed since then. There have been lots of development. GitLab. GitLab [03:25.840 --> 03:32.840] was a huge help for GNOME. And it means that we can now do CI quite easily. So, the situation [03:34.560 --> 03:39.040] now looks kind of like this. Module maintainers generally have unit tests. And we'll check [03:39.040 --> 03:46.040] that the module works in isolation. The release team have an integration repo that says these [03:46.160 --> 03:50.080] are the versions of the components that make up this GNOME release. So, we know what we're [03:50.080 --> 03:54.360] releasing. And distributions have started doing some downstream testing as well. At least [03:54.400 --> 03:58.280] some distributions have. There's a lot of good work going on testing that the released [03:58.280 --> 04:04.440] software is good. But there's still a gap because from landing a commit into the main [04:04.440 --> 04:09.840] branch of your module to actually having integration testing run by a distribution, there could [04:09.840 --> 04:14.160] be months. There could be months from you making that change to someone cutting a beta [04:14.160 --> 04:19.840] release and actually testing it. So, there's still a lot of time for problems to appear. [04:19.880 --> 04:25.560] So, the question we tried to answer over the last sort of 10 years within the GNOME project [04:25.560 --> 04:32.560] is what if we built our own distro just for testing? And so, we did. It was a long job, [04:33.000 --> 04:40.000] but GNOME OS exists. Lots of people worked on this over the 10 years. And it exists specifically [04:40.480 --> 04:44.280] for testing. So, some people say, can I use it? And, well, you can, but it's designed [04:44.280 --> 04:49.160] to be broken, right? So, don't use it unless you want something that breaks every day, has [04:49.160 --> 04:55.080] no security updates and doesn't support most hardware. But what it is good for is testing [04:55.080 --> 05:02.080] the up-to-date latest in development version and for seeing how new designs might work [05:02.720 --> 05:08.360] as they're being developed. And a goal was always automated regression testing, but that's [05:08.360 --> 05:11.480] kind of been the last piece of the puzzle. And that's the thing I'm showing off today [05:11.480 --> 05:17.980] is that we now have automated regression testing of GNOME OS. You can get it from here if you [05:17.980 --> 05:24.980] want, like I say, only use it for testing. And it works for manual testing, but it's [05:25.260 --> 05:28.260] quite boring, right? People don't spend their weekends going, oh, I think I'll download [05:28.260 --> 05:35.260] this image and, you know, just test and report bugs. And it's not quite suitable for pre-merge [05:35.460 --> 05:41.380] gating yet because it takes hours to build the image. So, we can't gait every merge request [05:41.380 --> 05:47.500] and say, well, the OpenQA tests have to pass because it can take hours before the new OS [05:47.500 --> 05:54.500] image is built, right? So, what we're doing at the moment is we've set up an OpenQA instance. [05:55.740 --> 06:02.500] So, OpenQA, I haven't introduced OpenQA yet. OpenQA is a test tool developed by SUSE. How [06:02.500 --> 06:07.000] many people are familiar with OpenQA, actually? We're in the testing room, so hopefully some [06:07.000 --> 06:12.460] people are. Maybe half the room. Okay. Well, I'm going to do kind of a deep dive into how [06:12.460 --> 06:17.980] it's set up for Nome and how it works. There are three components. The web interface is [06:17.980 --> 06:24.580] the thing you look at, and this is called OpenQA. The thing that actually does the work [06:24.580 --> 06:31.580] is a test driver called OS auto-inst. It's a less catchy name, and that has, it supports [06:32.200 --> 06:38.100] multiple backends, right? So, in Nome, we use the QMU backend, but you can also use [06:38.100 --> 06:42.620] backends that run on real hardware. I think some distros are doing this. Some of the code [06:42.620 --> 06:48.400] think projects use this. In Nome, we only use emulation at the moment because it's kind [06:48.400 --> 06:54.140] of the simplest option. And then, we have a library of tests. So, actually, most of [06:54.140 --> 07:00.060] the fun stuff in OpenQA lives in Open SUSE's test repo, and when we want to do more advanced [07:00.060 --> 07:05.220] stuff for the Nome test, we go in there and copy stuff out of it and use it sort of like [07:05.260 --> 07:10.460] a library in the traditional sense of something that we copy from. There are some built-in [07:10.460 --> 07:17.460] utilities as well, but a lot of the good stuff is in the Open SUSE tests. Lots of people [07:18.300 --> 07:23.460] are using it these days. SUSE, of course, having invented it. Fedora is using it. I found an [07:23.460 --> 07:29.460] article about Eurolinux, which are using it. Various car companies are using it. Maybe [07:29.540 --> 07:35.180] you were using it. Hopefully, you will do after this talk. CodeThing is also using it [07:35.180 --> 07:42.180] to test Linux kernel master branches on some ARM hardware using Lava as well. That's a [07:42.860 --> 07:46.420] whole separate talk, which I'm not going to go into, but if you're interested, find [07:46.420 --> 07:50.580] someone with this t-shirt on and they can talk about it. [07:50.580 --> 07:54.780] So, let's be adventurous, right? Here's a screenshot of OpenQA, but hopefully, the [07:54.820 --> 07:59.900] Wi-Fi is going to work and I can show you the real thing. Here's the front page of the [07:59.900 --> 08:05.820] GNOME OpenQA website. It doesn't tell you much, right? Actually, we don't use this front [08:05.820 --> 08:12.820] page. We go via GitLab. Here's the GNOME BuildMeta repo. This defines what goes into GNOME. [08:13.820 --> 08:20.820] This defines what goes into GNOME. This has a CI pipeline set up. Ah, the internet's not [08:23.740 --> 08:30.740] working. Let's see. Aha, you got me, FOSDM Wi-Fi. Let's go back to the screenshots. I did [08:32.900 --> 08:38.180] anticipate this happening. Here's some CI pipelines that I prepared earlier. These are [08:38.220 --> 08:43.780] the tests running on master. These do various things. You know, they build all the components [08:43.780 --> 08:48.180] using a build tool called build stream. The interesting one for our purposes is a job [08:48.180 --> 08:55.180] called S3Image. This builds an ISO installer and pushes it to Amazon S3, which is a good [08:56.420 --> 09:03.420] place to store these kind of five gigabyte ISO images. Then, we have another job called [09:03.560 --> 09:10.560] testS3Image. That's the fun one, right? That goes to S3, downloads the image and runs the [09:12.560 --> 09:19.060] OpenQA tests. There's a long explanation of how it works, but actually, I'm going to [09:19.060 --> 09:26.060] see if I can show you the job log. I did load one earlier. No, I can't load the job log [09:26.380 --> 09:33.380] either, so I'm going to show you the long explanation of how it works. In brief, this [09:33.440 --> 09:38.820] design of OpenQA initially was that you'd have a separate machine or a farm of machines, [09:38.820 --> 09:45.320] and the tests would run on one of those machines. That's a perfectly fine model, but it involves [09:45.320 --> 09:49.740] maintaining quite a lot of infrastructure. We're trying to do this in the easiest possible [09:49.740 --> 09:56.580] way, because we don't have a big team working on this. We kind of inverted the design, and [09:56.580 --> 10:03.580] we use the GitLab runner as the worker. The GitLab runner uses the OpenQA worker container [10:04.180 --> 10:11.180] image. It calls the OpenQA web UI and says, hi, I'm a new machine. Send me a job, and then [10:12.140 --> 10:16.820] it queues a job and adds a flag saying, oh, by the way, this can only run on the machine [10:16.820 --> 10:23.820] that I just created. The effect is then the GitLab runner becomes the OpenQA worker, runs [10:24.220 --> 10:31.300] OS auto-inst and runs the tests and communicates the results back to the OpenQA web UI. It's [10:31.300 --> 10:35.060] maybe a little unsupported, but actually is working quite nicely, and there's just a [10:35.060 --> 10:40.420] couple of caveats in the web UI from doing things that way. It means we only have one [10:40.420 --> 10:45.300] big build server, which is configured as a GitLab runner, and we don't have any other [10:45.300 --> 10:51.500] infrastructure apart from the web UI, which is fairly simple to maintain. That's why we [10:51.540 --> 10:58.540] do it that way. Now I'm going to go through what you can see in the web UI. First, I'm [10:58.540 --> 11:05.540] going to drink some water, actually. I've got a lot of talking to do today. Each test [11:09.260 --> 11:16.260] run gets an ID. I can't see the ID, but it'll be some long number. We have one long test [11:17.040 --> 11:24.040] job, which tests everything we care about. Actually, I think this one I loaded. Here's [11:24.180 --> 11:31.180] the real thing. We test all the way from taking the OS image on a bare machine, running the [11:33.020 --> 11:39.780] installer. Can I open that? No, I can't open that. You'll have to look at the tiny screenshots. [11:39.780 --> 11:46.140] Using the initial setup, this is the GNOME initial setup, we poke at the serial console [11:46.140 --> 11:51.980] a little bit once we've created a user. Once we've created a user account, we can log in [11:51.980 --> 11:56.700] over serial, and we just enable journal logging to the serial console to make things a bit [11:56.700 --> 12:03.700] easier to debug. Then the fun starts. We start poking around at the desktop, and we run each [12:04.060 --> 12:10.820] of the GNOME core apps. At the moment, we just check that it starts, and then it looks [12:10.820 --> 12:17.940] the same as it did the day before. The core of OpenQA is doing screenshot matching, and [12:17.940 --> 12:24.580] it has some tools for making that a little bit nicer than it would be if it was just [12:24.580 --> 12:29.820] pixel by pixel comparisons. The core of it is screenshot matching. We have a screenshot [12:29.820 --> 12:34.580] of each app, and we say this is how it should look, and as long as it looks the same, or [12:34.580 --> 12:41.420] within 95% the same, then the test passes. If it looks different, then the test fails. [12:41.420 --> 12:45.260] This one, I guess you can't see, but this one has failed because a pop-up has appeared [12:45.260 --> 12:52.260] over the top, which is pretty annoying. One of the things that we still need to sort out. [12:52.780 --> 12:59.780] Most of these have passed. This one has failed because the icons change size slightly. Again, [12:59.860 --> 13:06.860] the image matches maybe 95%, and the threshold is 96%, so it hasn't quite passed. In most [13:08.500 --> 13:12.220] cases, the solution to these failures is just update the screenshot, and that's quite an [13:12.220 --> 13:19.220] easy process. Let me show you how. This is a gallery of some tests viewed closer up. [13:19.640 --> 13:27.300] When you click on one of the screenshots, you get to see the before and after, or rather [13:27.300 --> 13:32.180] the golden screenshot and the real screenshot. You can drag this slider across and go, okay, [13:32.180 --> 13:39.180] this is, you know, here's the difference. These areas are the actual match zones which [13:39.740 --> 13:45.100] are defined in the screenshot. OpenQA calls these needles. A needle is like a screenshot [13:45.140 --> 13:51.980] plus some metadata, and we define zones that we want to match against, and it uses OpenCV [13:51.980 --> 13:57.740] to do the matching. That's what this percentage means. It's saying, you know, it's 99% the [13:57.740 --> 14:04.740] same. A cool thing about using OpenCV is that it can move around the screen, right? This [14:04.780 --> 14:11.780] window might have popped up in a different place, but an OpenQA would, you know, if the [14:11.820 --> 14:16.260] match was over here 20 pixels to the right, it would detect that, and the test would still [14:16.260 --> 14:22.760] pass. So that's pretty useful. And you can also lower the threshold. The manual says [14:22.760 --> 14:28.260] don't lower the threshold below 90%. I guess because maybe anything will pass at that point. [14:28.260 --> 14:35.260] I haven't played with it too much. I tend to go with between, you know, 95% and 100%. [14:36.020 --> 14:42.620] Your tests can input text. So here we're creating a user. All this is done via the QMU APIs, [14:42.620 --> 14:46.860] so it's simulating a real mouse and a real keyboard to do this. It's going really through [14:46.860 --> 14:51.100] the whole, you know, stack, the whole software stack from the kernel through the graphics [14:51.100 --> 14:56.860] drivers right into everything in user space into GNOME. So the ultimate in integration [14:56.860 --> 15:03.860] testing. Here's some more screenshots of needles. This is an exclusion. So I excluded [15:04.340 --> 15:11.460] the version number so that when we bump the version number, the tests don't fail. This [15:11.460 --> 15:16.260] is the needle editor, right? So the WebUI lets you edit these needles. They're stored [15:16.260 --> 15:20.980] in a Git repo, but not everyone wants to dig around in Git repos. So there's also a WebUI [15:20.980 --> 15:26.820] to edit them. And you can drag and drop. This is a screenshot, but the green is like let's [15:26.820 --> 15:32.500] match this area. And the brown is an exclusion because this is a process list, right? So [15:32.540 --> 15:37.580] it's going to be different every time. So we exclude that from the match. When a test [15:37.580 --> 15:41.620] fails because the screenshots changed and you want to update the screenshot, which is [15:41.620 --> 15:47.740] a really common case, you can go in here, change the screenshot, use the existing matches, [15:47.740 --> 15:53.820] and then you can commit to your changes under a new name to the needles Git repo. So it's [15:53.820 --> 16:00.820] like a two-click process. It's pretty straightforward. And here's the actual needles repo. And [16:02.820 --> 16:08.820] it's nothing too complicated. Each needle is a PNG file and some JSON metadata. And [16:08.820 --> 16:14.820] here's a really simple example of what the JSON metadata looks like. This has one match [16:14.820 --> 16:20.580] area. And it has a tag. So the important thing here is the tag. In your tests, you would [16:20.580 --> 16:27.580] say assert screenshot app, bow, bow, bow, home. And it will validate any needle that [16:28.580 --> 16:35.580] has that tag. So you build up this collection, like maybe version 40, it looks like this, [16:35.860 --> 16:40.860] and then maybe version 42, the design changes. So you make a new needle with the same tag, [16:40.860 --> 16:47.860] same tag. An open QA will now accept either of those needles. So the old design would [16:48.380 --> 16:53.620] still pass. And if your application randomly regresses to the old design, actually it wouldn't [16:53.620 --> 16:58.180] catch that case unless you've deleted the old test. Seems kind of limiting, but actually [16:58.180 --> 17:02.540] open sues have built an enormous library of tests of using this method. So I trust that [17:02.540 --> 17:09.540] it works well. I think some people have actually improved on that on one of the code thing [17:09.940 --> 17:15.460] projects, but I don't know the details. Ah, and the last thing I wanted to mention was [17:15.460 --> 17:21.220] the tests. So this is the fun part. You get to write your tests in Perl. It's like a trip [17:21.260 --> 17:26.740] back in time. But it's not super complicated. I don't know much about Perl, but I can figure [17:26.740 --> 17:33.740] out most of what's going on. This is the main entry point. So we import a couple of helpers. [17:34.980 --> 17:39.340] We set some constants. Please don't steal my password, but this is a constant that we [17:39.340 --> 17:46.340] reuse. And then we load each of these is a specific test, and it's own Perl module. So [17:46.340 --> 17:53.340] then if we go look at one of those, here's a test case and the meat of the tests is calling [17:55.500 --> 18:02.500] these functions. So assert that this needle matched with a time out and then we click [18:03.260 --> 18:08.820] on an area that's defined in the needle. So in this case, it's the next button and that [18:08.820 --> 18:15.820] area is defined in the needle. We can also, we eject the CD-ROM after the install, and [18:16.900 --> 18:22.940] then we reset the machine. So a lot of this is building up libraries of useful functions [18:22.940 --> 18:29.100] and calling them. So your tests end up being quite readable. All right, I've got a couple [18:29.100 --> 18:32.700] more things, and then I'm going to open up to questions. Things that I've learned, Open [18:32.700 --> 18:37.140] QA is very good. It's probably the best thing to do this that's open source and available. [18:37.140 --> 18:43.060] You use it, contribute to it. On the other hand, don't go crazy with it because you will, [18:43.060 --> 18:50.060] you know, go slow and try and test integration. You know, don't do unit tests in there. The [18:52.700 --> 18:58.340] main documentation doesn't list the actual API that the tests use. So look here, look [18:58.340 --> 19:02.540] for the test API docs. And also look at open source of tests, which have loads of examples [19:02.580 --> 19:08.260] of good things that you can copy from. You don't have to run the tests in CI. When you're [19:08.260 --> 19:12.020] developing tests, you can run them locally. It's a little bit of a faff. You have to work [19:12.020 --> 19:17.380] out the right container command line. But you can run the container locally with OS auto [19:17.380 --> 19:21.820] inst and you can then see the logs locally and iterate much faster than having to push [19:21.820 --> 19:27.820] your changes to CI every time. That's a great help. I've messed up the numbering slightly. [19:28.260 --> 19:32.620] I guess you can follow it. There's a couple of errors that only appear in the logs. So [19:32.620 --> 19:36.780] the web UI is great, but occasionally you see like a 0% match and you're like, but these [19:36.780 --> 19:41.740] are the same. But it turns out there was something invalid in the needle and you get a 0% match, [19:41.740 --> 19:45.580] but it's only reported in the logs. So don't lose time to that. I've lost probably hours [19:45.580 --> 19:51.420] collectively to forgetting this and then remembering it again. Also, the upstream containers [19:51.420 --> 19:55.500] are useful and they're usable, but it's a very rolling release process. So you probably [19:55.540 --> 20:00.180] want to pin a specific version and update it when you're ready to deal with the new [20:00.180 --> 20:07.180] changes. Don't just pull the latest one every time. So within GNOME, this is kind of ready. [20:07.220 --> 20:11.740] It's working, but it has a bus factor of one at the moment. So before we can declare it [20:11.740 --> 20:16.460] stable, we need more people to get involved, both maintaining the infrastructure, maintaining [20:16.460 --> 20:23.460] the tests. And these are some credits of people that have worked on this over the last 10 [20:24.460 --> 20:28.500] years. Apologies if I missed anyone, but I wanted to make it clear this is not something [20:28.500 --> 20:34.700] that I've done myself in my free time. I'm really adding the finishing touches to a huge [20:34.700 --> 20:41.700] amount of work that's taken a decade to get to this point. On the topic of OpenQA, separately [20:42.180 --> 20:47.060] to GNOME, we're quite interested in it and Code Think, we're continuing doing the OpenQA [20:47.060 --> 20:54.060] lava testing of the Linux kernel. We've written a small tool to control hardware from within [20:54.300 --> 21:00.380] OpenQA tests and we've built this USB switcher, which if you have lots of test rigs and lots [21:00.380 --> 21:04.900] of USB hardware connected to them, it's very useful. If you want to see it, then they have [21:04.900 --> 21:11.900] some real ones over there. Find these people and they can tell you about how great it is. [21:12.500 --> 21:18.700] So, that's everything from me. Please, if you want to get involved with the GNOME initiative, [21:18.700 --> 21:25.700] it's a good time. I'll help you to learn everything about OpenQA you ever wanted to know and more. [21:25.820 --> 21:30.660] You can get involved on the discourse or on Matrix and I'm going to leave it there. I [21:30.660 --> 21:37.660] think we have a few minutes for questions. Thank you. [21:42.580 --> 21:47.580] We've got one here already. [21:47.580 --> 21:52.620] Okay, the question is how many bugs has this caught and how many false positives? It's [21:52.620 --> 21:58.660] caught some real bugs. We're not testing a huge amount of stuff really at the moment. [21:58.660 --> 22:02.900] We're just testing that every app starts. That can already catch some interesting breakages [22:02.900 --> 22:07.580] because in GNOME, we use 12 different programming languages. If any of the bindings break, then [22:07.580 --> 22:12.700] one of the apps will stop working. But it's found at least two known real bugs where I [22:12.700 --> 22:18.700] reported it to the app maintainer and they said, oh, wow, yeah, that's broken. Probably [22:18.700 --> 22:24.260] 20 or 30 false positives in the sense of the test has failed, I've had to go in, update [22:24.260 --> 22:31.260] the screenshot and the test pass again. But that's quite an easy process. At the moment, [22:31.740 --> 22:35.140] I'd say it's worth the effort. We're still going to evaluate over time if it's really [22:35.160 --> 22:41.500] worth the effort. But I think it's promising that as long as we keep the test suite small [22:41.500 --> 22:45.780] and we don't have to keep updating all of the like a million screenshots every time they [22:45.780 --> 22:50.780] change the desktop background, then I think it's going to be useful. [22:50.780 --> 22:55.780] One at the back first. [22:55.780 --> 23:02.780] That's a really good idea. Yeah, so the question was testing other locales like Chinese or German. [23:08.260 --> 23:11.900] I hear Turkish is always a fun one. You put in the Turkish translation and things always [23:11.900 --> 23:16.900] break. It's not an immediate plan, but it definitely would be good to do that. Another [23:16.900 --> 23:22.860] thing is testing themes. We have a high contrast theme and a large text theme for accessibility. [23:22.880 --> 23:26.420] These often also break because they're not widely used, but they're very important. So [23:26.420 --> 23:31.420] yeah, in the distant future, we want to do that. One here. [23:31.420 --> 23:38.420] Have you thought about some process where usually when maintainers of apps do visual [23:41.220 --> 23:47.220] changes, they usually know that they're doing visual changes. Some process that they can [23:47.580 --> 23:53.580] add some comment in the CI or some process which that information can be injected into [23:53.580 --> 23:58.580] OpenQI and reduce the fast postage. Yeah, so the question is if we can get app developers [23:58.580 --> 24:04.780] to notify the tests of changes somehow. Yeah, I think the solution is to get app developers [24:04.780 --> 24:11.540] interested in actually using this. So my goal is to have the developers of these apps actually [24:11.540 --> 24:16.740] finding the test useful, maintaining their own small set of tests, and then yeah, they [24:16.780 --> 24:20.340] will know at that point, okay, I just changed this. So obviously we're going to have to [24:20.340 --> 24:25.340] update the tests. It's certainly not going to scale if it's just always me doing it. [24:25.340 --> 24:30.340] So hopefully at the GNOME conference, this year, I can get everyone excited about this. [24:30.340 --> 24:36.340] One more question over here somewhere. How is that for you? [24:36.340 --> 24:43.340] So if you find an actual bug in an app, that's only happening in the OpenQI environment. [24:44.340 --> 24:51.340] How do you give them the reproducer? How easy is it for a developer to reproduce the environment? [24:51.340 --> 24:56.340] Okay, yeah, good question. So the question is how easy is it to reproduce this environment? [24:56.340 --> 25:03.340] It's actually quite easy because it's GNOME OS. So the developer can go to os.gnome.org, [25:03.340 --> 25:09.260] they can download a VM image and run it in GNOME boxes, and so they can boot the virtual [25:09.260 --> 25:14.260] machine and it's exactly the same code, like right down to the kernel and systemd and everything. [25:14.260 --> 25:19.260] So it's an effort, they have to download this image, but they get exactly the same environment [25:19.260 --> 25:26.260] and they can reproduce exactly the bug. Most of them don't, they just install into [25:26.260 --> 25:33.260] slash user and try and reproduce it there, but it's possible to do it this way. [25:33.260 --> 25:40.260] Yeah, so from what we saw, OpenQI is focused on visual testing and comparing screenshots. [25:40.260 --> 25:48.260] Is there a way to mix that or perhaps only do the headless testing settings with the [25:48.260 --> 25:51.260] commands from the CLI or from the console? [25:51.260 --> 25:55.260] Yeah, that's a good question. So the question is if it can do more than just screenshot testing. [25:55.260 --> 26:00.260] It can. I mean, we got a serial console, so we can run arbitrary commands. [26:00.260 --> 26:05.260] I think if we were going to do that, we wouldn't write the tests in Perl in OpenQI. [26:05.260 --> 26:11.260] What we'd probably do is write the tests in Python or in C, inject them into the OS image, [26:11.260 --> 26:16.260] and then we just run the test program over the serial console and check that it outputs pass. [26:16.260 --> 26:21.260] So it's definitely possible, and I think that's how we would do it. [26:21.260 --> 26:24.260] So how are we for time? [26:24.260 --> 26:25.260] No more time. [26:25.260 --> 26:27.260] No more time? OK, well thanks everyone for watching. [26:27.260 --> 26:28.260] Thank you very much.