It's good to learn at the errors of someone else, I would say.
We all do errors, but if you can avoid doing all of them on our own, that's a little bit
better.
That's why I'm going to share with you today a set of my favorite errors I have seen in
embedded products.
And if you have worked with me in the past, don't worry, I'm changing the details of
all of the examples that you cannot figure out which project it exactly was, okay?
So no panic.
But before we start, a disclaimer, I'm a security person, so I have my bias, okay?
And now a task for you, an important task for you.
Concentrate.
Concentrate and think about an embedded product or project you have been working on.
It may be something that you are working on right now or it may be something you have been
working with in the past.
Concentrate.
You have one?
Keep it.
We are staying honest with ourselves because count one point for every single error on
my list that was in your project, okay?
You stay honest with yourself.
First one.
Easy.
Binary ingit.
When we are thinking about this, we probably would say-
Your microphone is muted.
So-
I think-
No, it's not muted.
It's-
Okay.
It's great.
Okay.
When we get to binary ingit, what you think about at the beginning is there's some beginner
developer that got the application, compiled it, and then everything to git, right?
But it's not the whole truth.
I have seen binary ingit for different reasons too.
One important example, firmware, for a big project.
And whilst I started talking to the team about why do we have that binary ingit, but it's
hard to compile.
You need that toolchain, that distribution version.
Then you have to patch this and that.
So it was too complicated.
They just put it in git.
And in the to-do list, we are going to compile it later.
But later, it took some time to arrive, right?
My suggestion, if you are thinking about putting binaries in git, first think.
And then what you can do is, at the minimum, put a script that compiles that binary.
At the maximum, in your CI, that, of course, you have one, in your CI, put a different
job that is doing all the complicated work to compile that firmware binary, whatever.
This binary, if it can be compiled from source, would make sure that Alberto, who is here,
for people who don't know Alberto, you should know him, Alberto won't be crying when he
audits your repository for license compliance.
And for me, as a security person, when I see binaries in git, I tell myself, what do we
have here?
Probably five-year versions of everything with all the CVs from the last five years.
Great.
Try to avoid binaries in git, except if you really know what you are doing.
But really, know what you are doing.
Okay, forgotten independence is number four.
Do you know what you have in your project?
Really?
No?
Yeah, yeah.
Not knowing what you have in your project that quite often happens for embedded projects
that use one git repo and they copy everything in this library's configuration files.
And then after 10 or 15 years, nobody knows what is in there.
But it may also happen when you are using more advanced systems like Yocto, because
there are quite few people looking into the Yocto dependency list to figure out what they
have in their build.
And when they do, they look for the first time they start shouting and running away.
A test for you, in your project, the same project that you are honestly counting points
for, how many open SSL versions are there?
Zero?
Are you really sure there's zero?
Okay, we are going to add it.
That could be fun.
One copy.
Yeah, there are some people that may be this one.
Okay, let's go forward.
Less than three, more than three.
Some people think that maybe more than three.
And I think most of the people are not really sure.
Okay, how many people are not sure?
Yeah.
And it's not only open SSL.
For a security searcher, open SSL think that you need to update frequently.
But there are other libraries like that.
If you do not know what you have as dependencies, have a look and think how you can improve
yourself here.
And for those who have managers who do not understand why looking to dependencies is
important, use the word SBOM.
We are generating an SBOM.
For those who do not know what this SBOM is yet, I assume that in 24 months you are going
to learn that.
The hard way.
Number three.
Number three is not considering vendor support for everything you use in your project from
the beginning.
The classical example is not very open source friendly support for a processor or not completely
up to date.
But this is going and getting better.
What I would like to give you an example is an embedded product I was working with.
They were using some quite specialized devices, good quality, the product itself was very
good quality, with one asterisk.
The chip itself was done by a company of three, including people doing drivers.
So of course the driver wasn't upstreamed when I looked into it, it wasn't in the state
to be upstreamed any time soon, with devs all around the place in the code.
They were very welcome to accept patches, but you had to write all of them and test yourself.
I recommend everyone starting an embedded product.
Then you have the first list of components that you want to use.
Have a look of them and figure out how much it's going to cost to put that chip.
Maybe choosing a different chip, even if the chip is a little bit more expensive or harder
to get, it's going to be less expensive at the end.
Okay, number two, update added last minute.
That is one of my favorites.
Update has a pretty important impact on the embedded system quite usually.
It means quite often that the flash size is too small, that the partitioning scheme has
to be changed, that you need to change the whole boot process, and you need to retest
all that from the beginning.
If the legislation is lurking behind the scenes, if you are starting working on an embedded
project, and update system is not yet on the requirement list, it's good to have a look,
because for some of you, what's going to happen just before the release, the management comes.
We have a checklist here for you before we release, and on that checklist, update SBOM.
If you are not prepared, it may be a good idea to get vacations before that.
Now my favorite, developing an embedded system on the life system.
My real example of that were people working on a system with a very expensive FPGA, and
very expensive peripherals, so they basically had one piece.
As the team was small, so they were working all on the same system, in addition, it was
based on Ubuntu, so what they did, they were installing packages, creating sim links because
something they didn't want to compile, changing configuration files, and of course there was
no single place when they documented it all.
Then what happened when they started building the second prototype?
That was a little bit complex.
Why not developing on the life system when you are prototyping, you do not know how it's
going to work during later on, if you are not going to change the approach you are going
to take.
Why not?
In this case, DevOps, but it's not a catchy word to get more views of the video, it's
really something that you can use, use the DevOps tools as ansible, for example, in this
case, so that you have a script that exactly deploys the system as it needs to be, and
the right moment, and keep the script in a version control system, so then you can
work on it and update during the system life.
We are getting to the end of my favorite list, and now I would like to make a check.
How many of you have projects with five points?
All five points?
We have some.
Okay.
Congratulations for your honesty.
Congratulations for your honesty to yourself.
Yeah, that's the decision of our managers.
I could do another, yeah, I expected to do a little bit of explanation on how to explain
to managers, but I think that would be another talk of how to explain that to managers.
What I would recommend you today, in a new project you are working on, take the list,
choose one of the subjects that's one of the problems that happens in this project,
and remove that single one for now.
For quite many of them, talking about legislations, IP compliance, S-bombs, stuff like that works
with the management.
If you are sure, talk to Albert again.
For some other cases, it may be a little bit more complicated, but in my experience, talking
about legal, talking about cost, maintenance has cost.
If you choose something that is hard to maintain, it's going to cost expenses, but for company
finances and rated expressions, that helps.
I hope that was helpful for you, that you have learned something, you learned some techniques,
and now I have planned some time to get a little bit of a feedback from the audience.
We have a question here.
Chris is on the other side.
In the front row.
Thank you.
Thanks for the talk.
Our first point was binaries in Git.
When we are developing an embedded system and compiling a firmware, what's a good solution
when we are not making releases, but in between, if we need to have access to the binary file
and make sure that it's the last version, what's the solution about putting just the
binary in Git?
If I understand the question correctly, your question was, when you have a firmware in
your product, you want to know that you always have the latest version?
Yes.
That's not a release, so it's not, I don't think it's doable with the ICD.
I can see two cases in such a situation.
Either you are compiling the firmware yourself, or you are getting from the vendor.
If you are getting from the vendor because there's some feature they have added that's
a little bit more complex.
In this case, you don't really have an option.
If you are compiling yourself, and it's hard to compile, I prefer to have a separate build
stage for the firmware itself.
You may have different branches for the firmware, and you are using every single dependency
from a different build system, when you are using multi-stage CR.
We can chat.
Maybe I'm not as advanced as you are.
Maybe you can chat about the details of setting it up later.
Any other questions?
We have someone in the middle, in the front.
Yes, thank you very much for the presentation.
I wanted to ask, if you have a product which is really long running, like several years,
and then regarding this vendor support for hardware components, sometimes on our project
it is like some of these components are running into end of life, and is there a strategy
or something like that where you can anticipate this kind of scenario, where your product
really has a long life cycle, and then you have to really think about what is if some
of our hardware components having end of life or something like that.
Unfortunately, the mic level wasn't great, so I'm not sure I cached everything.
If I do a summary of what you have said, you have an example of a project using components
that may be reaching end of life, and you want to support it for a very long time.
So what to do in this case?
It depends if it's about drivers, about all the components.
If you are about drivers, drivers in Linux get removed really, really late, so normally
the driver should still be there in the latest system.
There may be some changes that are not exactly compatible with what you are using.
That's true.
You may have vendor BSP that they stopped upgrading, and that's when that happens, that's a big
problem.
One solution is talk with the vendor, but if they do not want to understand what you
need, I would probably try to create some abstraction layers and keep some parts on
the older versions and migrate the newer parts, things that you can maintain actually.
Then in this case, it will depend exactly on the case, on the situation, which component
it is.
It will really depend.
Yeah.
Complicated.
Okay.
I'm going to second.
Thank you for your talk.
What if you had to convince your colleagues to follow these practices?
You put them in place, but management doesn't really care much about them.
It doesn't enforce them.
Okay.
The question was how to convince the colleagues, even if the management is quite okay with
those graphics.
What I use is a set of horror stories from my past.
When people did like that, six months later, what happened?
It was like developing new stuff.
They do not like fixing old bugs, looking into history, so using the argument of if
we do it messy this time, then we'll have to maintain it, and this is you who is going
to maintain that stuff.
They have to get burned at least one time.
That can help.
Thank you.
Okay.
I think we'll be done now.
Just one comment on one of the earlier questions.
I think a good approach would be to look at the vendor, how they support Linux.
Some vendors provide, I mean, look at one processor.
It had 500 patches to a five-year-old kernel, and another vendor, they push everything to
the mainstream, and you might want to think who you want to choose.
Absolutely agree with that.
When I'm looking into the chip to use, I'm looking at the vendor's mainstream support,
and that's one of the criteria to start with, basically.
I think it's a great point.
I think we should all boycott vendors who don't have upstream drivers.
Yeah, that's a separate.
Just say no, okay?
Thank you all.
Thank you, thank you very much.