Hi, my name is Joshua Lawton, I'm here to talk to you today about automated S-bomb generation
using a case study of the way that we generate S-bombs in open embedded.
A little bit about me, I've been working at Garments since 2009 and we've been using
open embedded in the active project to do embedded system development since 2016.
I'm a member of the open embedded technical steering committee and there's all the ways
you can contact me if you're interested later, I'll post my slides after my talk.
So we're all hopefully familiar with what an S-bomb is, we use it to describe what
software components we have in our system, what we know about them, what we don't know
about them and importantly what the relationship between them is.
So why are S-bombs important?
If we're using software ourselves or allowing other people to use it or shipping it to customers
or whatever we're doing with it, we want to know what's in our software and we want
to know where it came from, what versions those things are at a very minimum.
If they're software licenses, we want to know if we need to do anything to comply with them
or things like that or make sure they're not being used improperly.
We don't want to expose ourselves or people using our software or customers or whatever
it is to risk by having software that's been tampered with either maliciously or unintentionally
and we also want to know if any vulnerabilities come up after it's shipped so that we can
fix them if necessary or if it's vulnerable to exploit.
And really the question that we want to know is can we trace the binary things that we
have given to people back to the source code that produced it?
Often when we talk about S-bombs, we talk about them as being nutrition information
for software and I really do like this analogy.
I think it easily encapsulates something that everyone is familiar with which is a standardized
way of encoding something.
For S-bombs, we're trying to standardize the way that we encode information about software
just like nutrition labels try to standardize the way that we communicate what's in our
food.
So most people can look at a nutrition label and have an understanding of how it works
so we want S-bombs to be the same way.
You can look at the S-bomb and it's a way of encoding what we have.
I think this is a great analogy but it is missing a few key pieces and the pieces that
it's missing are really the supply chain part of the analysis.
So it can tell us what's in our software.
Just like a nutrition label tells us what's in our food but it doesn't tell us how it
got there.
A nutrition label isn't saying this grain came from here or whatever and that's the
part that we're sort of missing with S-bombs that we would like to know and that's what
this talk is about.
So I don't have a nice analogy for how to communicate a supply chain that's like the
nutrition label but I do come from a consumer manufacturing background so I do understand
supply chains.
So we can relate software supply chains to physical supply chains and when you have physical
supply chains so you're making some consumer electronics you've got all these steps along
the path of getting the completed product and you need to know where every component
comes from to make sure that all the right components are in the right place at the right
time to be manufactured.
You need to know what's being combined in every step for the same reason and you need
to know where this combination takes place because in modern supply chains these steps
can be spread out geographically all over the world and they can also be spread out
over time so if you produce 10,000 of one thing and then put it in storage and then
you pull those out you might need to know like are these ten years old or these five
years old?
Like how old are these parts?
When were they manufactured?
And when we talk about software supply chains we have basically the same questions.
We need to know where all the components that are in our supply chain came from however
in this case we're usually talking about things like source code that we've compiled and then
the tools that we use to compile it instead of physical components.
We need to know what has been combined to each stage.
Did we take this library from this other project and put it into what we're currently
working on, does it pull in some dependencies from somewhere, things like that.
We need to know where this combination takes place although we're probably less concerned
with the physical location as much as the build host that's doing the combination and
who did it.
Potentially we would like to know who did this step of our supply chain and then when
did it occur?
Was the software compiled ten years ago?
That's probably got vulnerabilities that we should take a closer look at.
To help answer some of these questions, SPDX has a build working group that's been working
on the build profile and it will hopefully be releasing with SPDX 3 in a couple months
or whenever that is soon.
And it's designed to answer the questions of when a build was done so it can require
time stamps for when builds happen, who wanted a build done.
So this is going to be the person who initiated the build or wanted the build done or did
the build themselves depending on the circumstances.
And that's distinct from who actually performed the build which might be, could be a person
if they're manually typing in the command to do the build or it could be a service like
GitHub actions or something like that and that's why we have the two different who elements
in there that distinguish between the person who clicked the button and GitHub actions
and GitHub actions that actually did the build or whatever your service is.
So how the build was done, so this is going to be tool specific information about how
the build was performed, like the command line arguments or things like that.
It's important to note that the build and run time dependencies are already actually
captured by the SPDX core specification so we don't include those explicitly here but
they're already included.
Where the build was done, so this is going to be the build host, the computer on which
the build was performed.
So this might be as complicated as an entire another software build of materials if you
have one that describes the system you're building on, you could link into that and
know all the information about the build host also.
But also would capture the tool use like if you have compilers or host tools or things
like that.
And then the what you're building is already covered by the SPDX core profile because it
can describe packages and files and things like that.
So one of the key points is that it's really important to try to generate build sbombs
at actual build time.
And to try to explain that a little further, I'm going to kind of compare generating sbombs
at build time versus two other ways that sbombs are commonly generated, although these aren't
the only other ways they're generated.
So there's source sbombs which are generally, this is like reuse or something that's just
included with the source code, which is really cool.
And then you got post mortem sbomb analysis, so this would be the tools that run after
you have the final artifact to try to scan it and say you're vulnerable to these vulnerabilities
and things like that.
Try to determine information after from the final artifact.
And obviously I'm trying to say that we should generate build sbomb information at build
time.
So I'm not trying to say that the other two things are just terrible and never use them.
They all have their strengths and weaknesses.
I'm just trying to explain why I think we should build them at build time.
So we talk about when something can be built.
Source sbombs obviously can't know this because they're not worried about when something is
actually built.
Build sbombs should be able to figure this out from when the thing is built.
You can record time stamps pretty easily.
And post mortem analysis may or may not be able to figure this out.
It just depends on if that information happens to be encoded in whatever you've produced.
We talk about how, so build time dependencies.
Source sbombs might be able to capture this if you're talking about something like a cargo
or NPM that explicitly encodes specific versions of dependencies in the source code.
You could very easily figure out and know what the build time dependencies are.
Otherwise, if you're talking about shared libraries or something, it might be able to
know those, but it wouldn't know them concretely.
So you'd know, like, I need open SSL, but you wouldn't necessarily know the specific
version of open SSL that it built against.
Build time, you should know all of this.
You should be able to know all of this at build time.
You basically have to in order to correctly build the software.
So you kind of need to know that.
For post mortem analysis, you might be able to figure it out, probably with some sort
of heuristically, and static libraries are always very problematic with this.
It can be very difficult to tell if a given executable has a static library in it or not,
because it's not recorded anywhere in the executable.
So those can always be very tricky to trace back to their origin.
Run time dependencies are a somewhat similar story, so source sbombs probably, you could
know what they are, but probably not concretely again.
Build sbombs, you should be, you could know this if you're doing complete packaging.
So if you're generating final packages like Debian packages, or Fedora packages, or OPK
packages, or whatever, you could know this, know what these run time dependencies are
even concretely.
And for post mortem analysis, for shared libraries, you can actually figure this out pretty easily
because it's in the elf header, but for anything that's run time dynamically loaded, like if
you do DL open or something like that, you probably can't figure that out very easily
with post mortem analysis.
In your build environment, I believe source sbombs don't care about this.
Build sbombs, you should be able to know this information, and for post mortem analysis,
maybe you could figure that out.
I don't know, if it was encoded in the executables, maybe some of that information could be known.
So there's a couple of advantages for generating supply chains from your build tools at build
time.
I like to say that they're authoritative because they have first hand knowledge because they're
the ones actually doing the build, so they should know what's actually happening at each
step.
And likewise, they're very accurate.
There shouldn't need to be a lot of guessing from your build tools about what's going on
at each step, unlike the post mortem analysis, which tends to, I think, use a lot of heuristics
or things like that.
In a comprehensive, they can analyze a lot of different steps in your build, especially
if your software supply chain is very deep, which I think it is for a lot of things.
And so they can generate a lot of information about your builds, as we'll see later.
And they're also able to analyze things that are difficult, if not impossible, to analyze
at other steps, like particularly static libraries, can be very difficult to trace back, at least
as far as I know.
So what kind of things could generate this information?
So kind of starting from the top down, it's sort of the highest level, you'd have things
like container build systems.
So this would be like Docker build or builder or something like that.
As you move down, you kind of get into what I call the meta or distro build systems.
This would be like open embedded, which is what I'm going to give an example of in just
a few minutes.
Debian, Fedora could generate this every time they generate packages.
It would be a good time to do that.
And then if you go down even a further step, you've got the package build systems.
It's not a great name for them.
But this would be things like Mason or CMake or Auto Tools.
They could all generate this information also with what they know about builds.
And you could go down an even further step and say, well, maybe GCC should spit out this
information and maybe it should.
That's also something that could happen and then it could sort of flow up the build stack
as you go.
So I'm going to give an example of what we do in open embedded.
I have to generate S-bombs.
And if you are unfamiliar with open embedded and the octa project, so open embedded is
a community driven project that provides the open embedded core layer and the build system,
which is called BitBake.
And the octa project is a Linux foundation project that provides the pocky reference
distribution and runs a whole bunch of QA tests to make sure everything stays high quality.
Here's some release schedules.
They provide funding for personnel to work on the project full time and servers and things
like that.
And they provide very good, excellent documentation.
You should go check out our documentation.
And the purpose of these projects is to build primarily but not exclusively embedded systems.
So we do have our traditional image you could flash on a Raspberry Pi up there, colloquially.
We call these target images.
So we actually can produce images for a whole bunch of different things that I'm not going
to go into in great detail here.
I've got a bunch of other presentations on this that I have links to later.
So when people want to build stuff with open embedded, what they do start with is they
have some source code and they have some metadata and they have some policy information.
And they chuck all of this into this magical tool called BitBake.
And it spits out this target image that we talked about and then you flash that on your
widget and profit, right?
It's great.
A little deeper under the hood, the way that this works is that we start off with some
host tools.
So this is like the minimal set of things that you need to build with BitBakes.
This is going to be like your host GCC, Python, and a couple other like fairly standard dependencies
that run on your host.
And we're going to take those host tools and we're going to parse some recipe metadata
that says how to build some source code.
And that source code is going to be used to build what we call the native tools and the
cross-compiler.
So the native tools are still tools that are designed to run on your host system.
And then we also build the cross-compiler at the same time.
So something like an example of this might be like the protobuf compiler, right?
We actually build that ourselves and don't require you to provide it, provide your own.
We also build our own cross-compiler, so you don't even need a cross-compiler on your
host system.
We then use those native tools and cross-compiler to process more recipe metadata that's going
to take some more source code in.
And this is actually going to cross-compile and build what we call your target packages
that are designed to run your final system, be it x86 or ARM or MIPS or PowerPC or RISC-5
or whatever it is.
And then we process yet some more metadata, and this one says how to combine all these
target packages to make your root file system and your kernel and all of these other things
that you need to actually have your target image.
The way that Bay keeps all of this sane and tracks the dependencies is it uses a sophisticated
method of hashing where each step along the way, called a task, has a hash that is the
encapsulation of all of the dependencies of that task, all of the variables that affect
that task's execution, and all of the code that it's actually executing.
And then that gets combined into a single hash, and then that hash then is the input
as a dependency to every task that depends on that one.
So you get this chain of hashes all the way down following from your recipes that you
start with to your target image.
So what happens is if, for example, something about the protobuf recipe changes, that's
going to change the task hash for that recipe, that's going to cause that protobuf tool to
be rebuilt, and that's also going to change all of the downstream hashes that depend on
that all the way through any native tool that depends on that, any target packages that
depend on that, and all the way to the root file system that indirectly depends on that.
And so all of those things will be rebuilt by BitBake when you change that particular
thing.
And so just because of this hashing mechanism, OpenEmbedded and BitBake start out with a
very strong software supply chain, because we have these very strict rules about how
these hashes change, and this causes everything to be rebuilt, and so you can actually trace
it back from your target image to the target source code that produced all your target
packages, and you can even trace that back to your cross-compiler and your native tools
that we built, and there are ways in other presentations I've done that you can see,
you can even trace this back to your host tools if you really wanted to do that, and
have that really deep supply chain.
So basically what we do in OpenEmbedded is at each step along the way here where we're
building something, while we're building it we also spit out this SPDX document that says
this is what we did here at this step, and then at the very end we take all of the SPDX
documents that went into our target image, or native tools that were used to build a
target image, and we put them all into one big archive.
And we have a rich set of dependencies that we actually report when we do this.
I'm not going to get into too much detail here, again there's other talks I've given
that you can see that describe all of this in more detail if you're interested, and these
are those talks if you want to see those, and these talk a lot more specifically about
OpenEmbedded and S-bombs.
So when you do this, we can currently generate SPDX 2.2 JSON format, and I did this for a
minimal QMU AR64 system, so the root file system was 14 megabytes uncompressed, the Linux
kernel was 20 megabytes, and we had 158 megabytes of SPDX document.
So yeah, it's a lot.
I was actually going to post up the archive, but yeah, so it's a lot of data, and we're
not even reporting on everything yet, like you know, some of that is the JSON encoding
and things like that, but it's just a ton of data.
So the question is, do we really need all of this, like there's a lot of stuff to lug
around.
And I think you can harken it back to that nutrition information, like as a consumer
of a given food product, wheat is wheat, right, like I don't necessarily care how the wheat
got into my crackers or whatever it is that I'm eating, but if I'm a manufacturer of that
food and I need to track something down that went wrong somewhere to do a recall or something,
then I really care where that came from.
And so I think the same analogy could probably be made, like it's possible your end consumers
don't really care about your software supply chain, but if you're manufacturing something
or building something, you probably definitely do, so you can trace down problems and things
like that.
And you know, there's always the possibility that there could be regulatory requirements
for this in the future that seems to be a thing that's happening now.
So yeah, so if you're trying to track down a supply chain attack or something's gone
wrong somewhere, then you probably will definitely want this information.
So if you work on a tool that does something that looks like building, please consider
adding build profile support to your tool, because it's actually really not that hard.
Like for open embedded, we already had all of this information, it was just a matter
of encoding it, as hopefully was somewhat clear from what we were doing here.
Like we already had all this information, it was just a matter of writing out the document
that had it and then combining it at the end.
And with SPDX3, the combination at the end is going to be a lot better than it was with
SPDX2, so.
And that's all I had.
Are there any questions?
Yes, you have many megabytes of SPDX, so I suppose you don't have a big file, but we
have multiple files related to SPDX relationships?
Yeah, it uses...
Can you repeat the question for me?
Oh, sorry, yeah, so we have a whole bunch of documents, so we're not generating one
big SPDX document, we've got a whole bunch of small documents, yes.
So yes, we do that, we use a whole bunch of external document references, and then they're
in SPDX2, and this will be better in SPDX3, but in SPDX2 there isn't a standardized way
to combine documents together, so we just throw them all into one big tarball.
It's not the greatest, but it does put them all in one file for consumption.
They do exist, at the point of build, they do exist on the file system as individual
documents that you could package up however you want, it's just for ease of our end users,
it's easiest if we just put them all in one big tarball, and they can extract it and do
whatever they want with it.
But yeah, a whole bunch of external document references in our output, yes.
A few questions, one, slides, I'm not there on the presentation, will you be able to
do that?
Yeah, I will post the slides, yes, I will do that.
And as part of an official release, you will give this tarball to the release of your project.
Yeah, so there's an option you can turn on in your build that will, I don't remember
if we turn it on, but it's very easy to turn on, so you turn it on and then you just get
this tarball as part of your build.
So what happens in Open Embedded is you generate a file system like my file system, and that's
your root file system, and then alongside that there'll be my filesystem.spdx.tar.gz
or whatever it is, I forget off the top of my head, but yeah, it's that simple.
I think another answer given the size of it, so you don't deliver the spdx as part of
the image?
No, yeah, we do not deliver the spdx as part of the image.
So is spdx not providing some integrity that you could say, so to an end consumer to say,
yes, I've got this spdx and I've got the image, the two things are aligned.
Right, so how do you trace the spdx back to the image?
So there's extensive checksumming in the spdx itself, so every file in that root file system
is going to be expressed in the spdx, and the spdx will have its checksum, so you can
say like, you know, yeah, at the file level, so you can say like, you know, userlibfoo.so,
and then I go look that and the spdx are the checksums the same, then they're, you know,
they're valid, right?
So who's using the spdx?
Have you created this image?
Uh-huh.
Is anybody using the spdx?
The image?
Are you using it?
No, I'm not.
Yeah, sorry, so he's asking who's using it, and the short answer is I don't know.
A lot of people ask questions about how to generate it, so I assume they're doing something
with it.
Um, but I don't personally generate this yet, but that's just because of where I work,
not because I'm...
Can you say in relation that there is a list of general consumption tools that are available?
Yeah, there are a list of, sorry, there is a list of consumption tools available, yeah,
sorry.
Go ahead, I think you're next.
Um, the build profile stuff, the supply chain part, the B looks like the salsa provenance
definition, which, do you also look at that solution, because what I like about that solution
is that, that it's separate from the supplementary material, because, uh, and it can be consumed
in a different way.
Did you ever look into that, uh, first, do you want to do a specification?
Yeah, so the question is, did we look at salsa?
Yeah, we had people that were from salsa on the build profile working group, so a lot
of what salsa did fed into what we're doing here.
Um, I think the, that we wanted it to be more closely integrated with the SPDX core profile,
so that you could say, like, this is all the licensing information and the build information
and the supply chain information, so that's why we're including a build profile.
Um, I, I think tooling could come along to, if you wanted to later on, like, if you wanted
to, you know, strip out all the build profile stuff, because you don't want to ship that,
you know, ship gigabytes of data to your customers, like, sure, I think tooling can come along
to do that, and that should be fairly trivial, um, but yeah, that, that, that's why we chose
to do that that way.
What's the main relationship with the generated SPDX, is it the reset from the big bake itself,
or is the component that's generated at every site?
Sorry, I didn't, uh, I, I, sorry, I didn't quite understand the question.
So basically, we, you, you have one of the standard initial source of information that
goes to the SPDX there, the, the initial source will be just the big, big recipe, to meet
the component itself.
Uh, the initial, right, so the question is where does the initial information come from,
and the, the recipe describes how to build, we report on both, actually, so we report
on the, the source code, and the recipe, and the thing it built, uh, currently, so, uh,
we can do all of those things, um, and I'm, I'm done, I'm sorry, I can answer more questions,
if you want.
But I, I gotta, I gotta, thank you very much.
For people living, and living at this basis, the rest of you know what to do.
As a reminder, we have, like, chocolate, snacks, and things here, if anybody wants some.