Detecting language using up to the first 30 seconds. Use `--language` to specify the language Detected language: English [00:00.000 --> 00:13.920] It's good to learn at the errors of someone else, I would say. [00:13.920 --> 00:22.160] We all do errors, but if you can avoid doing all of them on our own, that's a little bit [00:22.160 --> 00:24.000] better. [00:24.000 --> 00:32.160] That's why I'm going to share with you today a set of my favorite errors I have seen in [00:32.160 --> 00:35.520] embedded products. [00:35.520 --> 00:41.240] And if you have worked with me in the past, don't worry, I'm changing the details of [00:41.240 --> 00:47.960] all of the examples that you cannot figure out which project it exactly was, okay? [00:47.960 --> 00:51.520] So no panic. [00:51.520 --> 01:03.280] But before we start, a disclaimer, I'm a security person, so I have my bias, okay? [01:03.280 --> 01:08.960] And now a task for you, an important task for you. [01:08.960 --> 01:11.400] Concentrate. [01:11.400 --> 01:18.320] Concentrate and think about an embedded product or project you have been working on. [01:18.320 --> 01:24.800] It may be something that you are working on right now or it may be something you have been [01:24.800 --> 01:28.560] working with in the past. [01:28.560 --> 01:30.440] Concentrate. [01:30.440 --> 01:33.960] You have one? [01:33.960 --> 01:36.000] Keep it. [01:36.000 --> 01:43.360] We are staying honest with ourselves because count one point for every single error on [01:43.360 --> 01:47.800] my list that was in your project, okay? [01:47.800 --> 01:52.880] You stay honest with yourself. [01:52.880 --> 01:53.880] First one. [01:53.880 --> 01:54.880] Easy. [01:54.880 --> 01:55.880] Binary ingit. [01:55.880 --> 02:09.640] When we are thinking about this, we probably would say- [02:09.640 --> 02:12.080] Your microphone is muted. [02:12.080 --> 02:13.080] So- [02:13.080 --> 02:14.080] I think- [02:14.080 --> 02:17.520] No, it's not muted. [02:17.520 --> 02:18.520] It's- [02:18.520 --> 02:19.520] Okay. [02:19.520 --> 02:20.520] It's great. [02:20.520 --> 02:21.520] Okay. [02:21.520 --> 02:22.520] Okay. [02:22.520 --> 02:28.960] When we get to binary ingit, what you think about at the beginning is there's some beginner [02:28.960 --> 02:37.040] developer that got the application, compiled it, and then everything to git, right? [02:37.040 --> 02:40.160] But it's not the whole truth. [02:40.160 --> 02:45.560] I have seen binary ingit for different reasons too. [02:45.560 --> 02:52.000] One important example, firmware, for a big project. [02:52.000 --> 03:00.720] And whilst I started talking to the team about why do we have that binary ingit, but it's [03:00.720 --> 03:01.720] hard to compile. [03:01.720 --> 03:04.040] You need that toolchain, that distribution version. [03:04.040 --> 03:07.560] Then you have to patch this and that. [03:07.560 --> 03:08.800] So it was too complicated. [03:08.800 --> 03:11.560] They just put it in git. [03:11.560 --> 03:15.120] And in the to-do list, we are going to compile it later. [03:15.120 --> 03:24.960] But later, it took some time to arrive, right? [03:24.960 --> 03:33.600] My suggestion, if you are thinking about putting binaries in git, first think. [03:33.600 --> 03:43.320] And then what you can do is, at the minimum, put a script that compiles that binary. [03:43.320 --> 03:52.360] At the maximum, in your CI, that, of course, you have one, in your CI, put a different [03:52.360 --> 04:01.480] job that is doing all the complicated work to compile that firmware binary, whatever. [04:01.480 --> 04:11.920] This binary, if it can be compiled from source, would make sure that Alberto, who is here, [04:11.920 --> 04:19.320] for people who don't know Alberto, you should know him, Alberto won't be crying when he [04:19.320 --> 04:26.880] audits your repository for license compliance. [04:26.880 --> 04:33.800] And for me, as a security person, when I see binaries in git, I tell myself, what do we [04:33.800 --> 04:34.800] have here? [04:34.800 --> 04:41.400] Probably five-year versions of everything with all the CVs from the last five years. [04:41.400 --> 04:45.240] Great. [04:45.240 --> 04:49.320] Try to avoid binaries in git, except if you really know what you are doing. [04:49.320 --> 04:51.600] But really, know what you are doing. [04:51.600 --> 04:57.560] Okay, forgotten independence is number four. [04:57.560 --> 04:59.680] Do you know what you have in your project? [04:59.680 --> 05:00.680] Really? [05:00.680 --> 05:01.680] No? [05:01.680 --> 05:06.840] Yeah, yeah. [05:06.840 --> 05:12.320] Not knowing what you have in your project that quite often happens for embedded projects [05:12.320 --> 05:23.520] that use one git repo and they copy everything in this library's configuration files. [05:23.520 --> 05:30.840] And then after 10 or 15 years, nobody knows what is in there. [05:30.840 --> 05:39.880] But it may also happen when you are using more advanced systems like Yocto, because [05:39.880 --> 05:45.840] there are quite few people looking into the Yocto dependency list to figure out what they [05:45.840 --> 05:49.200] have in their build. [05:49.200 --> 05:56.160] And when they do, they look for the first time they start shouting and running away. [05:56.160 --> 06:01.680] A test for you, in your project, the same project that you are honestly counting points [06:01.680 --> 06:07.600] for, how many open SSL versions are there? [06:07.600 --> 06:11.480] Zero? [06:11.480 --> 06:13.520] Are you really sure there's zero? [06:13.520 --> 06:16.280] Okay, we are going to add it. [06:16.280 --> 06:18.000] That could be fun. [06:18.000 --> 06:19.000] One copy. [06:19.000 --> 06:24.240] Yeah, there are some people that may be this one. [06:24.240 --> 06:26.920] Okay, let's go forward. [06:26.920 --> 06:30.960] Less than three, more than three. [06:30.960 --> 06:33.000] Some people think that maybe more than three. [06:33.000 --> 06:35.080] And I think most of the people are not really sure. [06:35.080 --> 06:38.520] Okay, how many people are not sure? [06:38.520 --> 06:41.280] Yeah. [06:41.280 --> 06:44.440] And it's not only open SSL. [06:44.440 --> 06:52.720] For a security searcher, open SSL think that you need to update frequently. [06:52.720 --> 06:56.560] But there are other libraries like that. [06:56.560 --> 07:04.400] If you do not know what you have as dependencies, have a look and think how you can improve [07:04.400 --> 07:09.080] yourself here. [07:09.080 --> 07:16.480] And for those who have managers who do not understand why looking to dependencies is [07:16.480 --> 07:20.920] important, use the word SBOM. [07:20.920 --> 07:24.920] We are generating an SBOM. [07:24.920 --> 07:31.560] For those who do not know what this SBOM is yet, I assume that in 24 months you are going [07:31.560 --> 07:33.120] to learn that. [07:33.120 --> 07:37.280] The hard way. [07:37.280 --> 07:39.480] Number three. [07:39.480 --> 07:46.600] Number three is not considering vendor support for everything you use in your project from [07:46.600 --> 07:49.960] the beginning. [07:49.960 --> 07:58.520] The classical example is not very open source friendly support for a processor or not completely [07:58.520 --> 07:59.600] up to date. [07:59.600 --> 08:04.760] But this is going and getting better. [08:04.760 --> 08:12.440] What I would like to give you an example is an embedded product I was working with. [08:12.440 --> 08:20.520] They were using some quite specialized devices, good quality, the product itself was very [08:20.520 --> 08:31.120] good quality, with one asterisk. [08:31.120 --> 08:38.880] The chip itself was done by a company of three, including people doing drivers. [08:38.880 --> 08:43.120] So of course the driver wasn't upstreamed when I looked into it, it wasn't in the state [08:43.120 --> 08:52.760] to be upstreamed any time soon, with devs all around the place in the code. [08:52.760 --> 09:01.240] They were very welcome to accept patches, but you had to write all of them and test yourself. [09:01.240 --> 09:06.120] I recommend everyone starting an embedded product. [09:06.120 --> 09:10.040] Then you have the first list of components that you want to use. [09:10.040 --> 09:17.440] Have a look of them and figure out how much it's going to cost to put that chip. [09:17.440 --> 09:24.800] Maybe choosing a different chip, even if the chip is a little bit more expensive or harder [09:24.800 --> 09:33.120] to get, it's going to be less expensive at the end. [09:33.120 --> 09:42.480] Okay, number two, update added last minute. [09:42.480 --> 09:49.160] That is one of my favorites. [09:49.160 --> 09:55.920] Update has a pretty important impact on the embedded system quite usually. [09:55.920 --> 10:04.240] It means quite often that the flash size is too small, that the partitioning scheme has [10:04.240 --> 10:12.400] to be changed, that you need to change the whole boot process, and you need to retest [10:12.400 --> 10:20.600] all that from the beginning. [10:20.600 --> 10:27.640] If the legislation is lurking behind the scenes, if you are starting working on an embedded [10:27.640 --> 10:40.240] project, and update system is not yet on the requirement list, it's good to have a look, [10:40.240 --> 10:47.960] because for some of you, what's going to happen just before the release, the management comes. [10:47.960 --> 10:58.040] We have a checklist here for you before we release, and on that checklist, update SBOM. [10:58.040 --> 11:10.240] If you are not prepared, it may be a good idea to get vacations before that. [11:10.240 --> 11:21.800] Now my favorite, developing an embedded system on the life system. [11:21.800 --> 11:31.520] My real example of that were people working on a system with a very expensive FPGA, and [11:31.520 --> 11:40.040] very expensive peripherals, so they basically had one piece. [11:40.040 --> 11:46.760] As the team was small, so they were working all on the same system, in addition, it was [11:46.760 --> 11:54.760] based on Ubuntu, so what they did, they were installing packages, creating sim links because [11:54.760 --> 11:59.400] something they didn't want to compile, changing configuration files, and of course there was [11:59.400 --> 12:04.360] no single place when they documented it all. [12:04.360 --> 12:10.280] Then what happened when they started building the second prototype? [12:10.280 --> 12:15.440] That was a little bit complex. [12:15.440 --> 12:19.720] Why not developing on the life system when you are prototyping, you do not know how it's [12:19.720 --> 12:28.320] going to work during later on, if you are not going to change the approach you are going [12:28.320 --> 12:29.320] to take. [12:29.320 --> 12:30.320] Why not? [12:30.320 --> 12:39.440] In this case, DevOps, but it's not a catchy word to get more views of the video, it's [12:39.440 --> 12:45.440] really something that you can use, use the DevOps tools as ansible, for example, in this [12:45.440 --> 12:52.960] case, so that you have a script that exactly deploys the system as it needs to be, and [12:52.960 --> 13:00.200] the right moment, and keep the script in a version control system, so then you can [13:00.200 --> 13:07.760] work on it and update during the system life. [13:07.760 --> 13:16.560] We are getting to the end of my favorite list, and now I would like to make a check. [13:16.560 --> 13:21.680] How many of you have projects with five points? [13:21.680 --> 13:23.000] All five points? [13:23.000 --> 13:24.000] We have some. [13:24.000 --> 13:25.000] Okay. [13:25.000 --> 13:32.640] Congratulations for your honesty. [13:32.640 --> 13:36.840] Congratulations for your honesty to yourself. [13:36.840 --> 13:47.400] Yeah, that's the decision of our managers. [13:47.400 --> 13:52.880] I could do another, yeah, I expected to do a little bit of explanation on how to explain [13:52.880 --> 14:00.760] to managers, but I think that would be another talk of how to explain that to managers. [14:00.760 --> 14:13.480] What I would recommend you today, in a new project you are working on, take the list, [14:13.480 --> 14:22.560] choose one of the subjects that's one of the problems that happens in this project, [14:22.560 --> 14:30.240] and remove that single one for now. [14:30.240 --> 14:40.160] For quite many of them, talking about legislations, IP compliance, S-bombs, stuff like that works [14:40.160 --> 14:44.200] with the management. [14:44.200 --> 14:51.280] If you are sure, talk to Albert again. [14:51.280 --> 14:57.400] For some other cases, it may be a little bit more complicated, but in my experience, talking [14:57.400 --> 15:07.360] about legal, talking about cost, maintenance has cost. [15:07.360 --> 15:16.440] If you choose something that is hard to maintain, it's going to cost expenses, but for company [15:16.440 --> 15:29.640] finances and rated expressions, that helps. [15:29.640 --> 15:36.240] I hope that was helpful for you, that you have learned something, you learned some techniques, [15:36.240 --> 15:45.560] and now I have planned some time to get a little bit of a feedback from the audience. [15:45.560 --> 15:48.200] We have a question here. [15:48.200 --> 16:00.480] Chris is on the other side. [16:00.480 --> 16:07.640] In the front row. [16:07.640 --> 16:08.640] Thank you. [16:08.640 --> 16:11.840] Thanks for the talk. [16:11.840 --> 16:18.320] Our first point was binaries in Git. [16:18.320 --> 16:29.480] When we are developing an embedded system and compiling a firmware, what's a good solution [16:29.480 --> 16:37.120] when we are not making releases, but in between, if we need to have access to the binary file [16:37.120 --> 16:48.320] and make sure that it's the last version, what's the solution about putting just the [16:48.320 --> 16:51.120] binary in Git? [16:51.120 --> 16:56.440] If I understand the question correctly, your question was, when you have a firmware in [16:56.440 --> 17:03.600] your product, you want to know that you always have the latest version? [17:03.600 --> 17:05.600] Yes. [17:05.600 --> 17:14.520] That's not a release, so it's not, I don't think it's doable with the ICD. [17:14.520 --> 17:19.520] I can see two cases in such a situation. [17:19.520 --> 17:26.840] Either you are compiling the firmware yourself, or you are getting from the vendor. [17:26.840 --> 17:30.640] If you are getting from the vendor because there's some feature they have added that's [17:30.640 --> 17:32.640] a little bit more complex. [17:32.640 --> 17:36.480] In this case, you don't really have an option. [17:36.480 --> 17:48.280] If you are compiling yourself, and it's hard to compile, I prefer to have a separate build [17:48.280 --> 17:51.280] stage for the firmware itself. [17:51.280 --> 17:57.040] You may have different branches for the firmware, and you are using every single dependency [17:57.040 --> 18:07.360] from a different build system, when you are using multi-stage CR. [18:07.360 --> 18:08.360] We can chat. [18:08.360 --> 18:10.680] Maybe I'm not as advanced as you are. [18:10.680 --> 18:16.440] Maybe you can chat about the details of setting it up later. [18:16.440 --> 18:19.440] Any other questions? [18:19.440 --> 18:25.440] We have someone in the middle, in the front. [18:25.440 --> 18:35.960] Yes, thank you very much for the presentation. [18:35.960 --> 18:44.080] I wanted to ask, if you have a product which is really long running, like several years, [18:44.080 --> 18:51.480] and then regarding this vendor support for hardware components, sometimes on our project [18:51.480 --> 18:58.680] it is like some of these components are running into end of life, and is there a strategy [18:58.680 --> 19:03.480] or something like that where you can anticipate this kind of scenario, where your product [19:03.480 --> 19:10.360] really has a long life cycle, and then you have to really think about what is if some [19:10.360 --> 19:15.280] of our hardware components having end of life or something like that. [19:15.280 --> 19:21.040] Unfortunately, the mic level wasn't great, so I'm not sure I cached everything. [19:21.040 --> 19:30.920] If I do a summary of what you have said, you have an example of a project using components [19:30.920 --> 19:43.360] that may be reaching end of life, and you want to support it for a very long time. [19:43.360 --> 19:50.560] So what to do in this case? [19:50.560 --> 19:54.920] It depends if it's about drivers, about all the components. [19:54.920 --> 20:02.720] If you are about drivers, drivers in Linux get removed really, really late, so normally [20:02.720 --> 20:06.160] the driver should still be there in the latest system. [20:06.160 --> 20:14.120] There may be some changes that are not exactly compatible with what you are using. [20:14.120 --> 20:15.120] That's true. [20:15.120 --> 20:22.520] You may have vendor BSP that they stopped upgrading, and that's when that happens, that's a big [20:22.520 --> 20:23.520] problem. [20:23.520 --> 20:31.440] One solution is talk with the vendor, but if they do not want to understand what you [20:31.440 --> 20:48.360] need, I would probably try to create some abstraction layers and keep some parts on [20:48.360 --> 20:57.080] the older versions and migrate the newer parts, things that you can maintain actually. [20:57.080 --> 21:04.000] Then in this case, it will depend exactly on the case, on the situation, which component [21:04.000 --> 21:05.000] it is. [21:05.000 --> 21:06.000] It will really depend. [21:06.000 --> 21:07.000] Yeah. [21:07.000 --> 21:08.000] Complicated. [21:08.000 --> 21:09.000] Okay. [21:09.000 --> 21:28.040] I'm going to second. [21:28.040 --> 21:29.640] Thank you for your talk. [21:29.640 --> 21:34.240] What if you had to convince your colleagues to follow these practices? [21:34.240 --> 21:38.520] You put them in place, but management doesn't really care much about them. [21:38.520 --> 21:39.520] It doesn't enforce them. [21:39.520 --> 21:40.520] Okay. [21:40.520 --> 21:48.360] The question was how to convince the colleagues, even if the management is quite okay with [21:48.360 --> 21:49.360] those graphics. [21:49.360 --> 21:58.560] What I use is a set of horror stories from my past. [21:58.560 --> 22:06.520] When people did like that, six months later, what happened? [22:06.520 --> 22:08.760] It was like developing new stuff. [22:08.760 --> 22:16.880] They do not like fixing old bugs, looking into history, so using the argument of if [22:16.880 --> 22:22.800] we do it messy this time, then we'll have to maintain it, and this is you who is going [22:22.800 --> 22:25.240] to maintain that stuff. [22:25.240 --> 22:27.240] They have to get burned at least one time. [22:27.240 --> 22:29.240] That can help. [22:29.240 --> 22:30.240] Thank you. [22:30.240 --> 22:31.240] Okay. [22:31.240 --> 22:34.080] I think we'll be done now. [22:34.080 --> 22:46.040] Just one comment on one of the earlier questions. [22:46.040 --> 22:51.720] I think a good approach would be to look at the vendor, how they support Linux. [22:51.720 --> 22:55.040] Some vendors provide, I mean, look at one processor. [22:55.040 --> 23:02.440] It had 500 patches to a five-year-old kernel, and another vendor, they push everything to [23:02.440 --> 23:07.480] the mainstream, and you might want to think who you want to choose. [23:07.480 --> 23:08.840] Absolutely agree with that. [23:08.840 --> 23:15.760] When I'm looking into the chip to use, I'm looking at the vendor's mainstream support, [23:15.760 --> 23:19.880] and that's one of the criteria to start with, basically. [23:19.880 --> 23:21.080] I think it's a great point. [23:21.080 --> 23:25.120] I think we should all boycott vendors who don't have upstream drivers. [23:25.120 --> 23:26.360] Yeah, that's a separate. [23:26.360 --> 23:30.040] Just say no, okay? [23:30.040 --> 23:31.040] Thank you all. [23:31.040 --> 23:32.720] Thank you, thank you very much.