Good afternoon, everybody. Thanks for coming to our session.
Yeah, I don't know. It's on.
All right. Okay, so mine, I try to speak loud so that the people even in the rows in the back can understand me.
My name is Henrik. That's my colleague Joseph. Both of us are publishing and working on open source dependency management security topics for a couple of years already.
And so, yeah, recently we started looking into S-bombs more maybe compliance and topics that we are not so familiar with.
So we are looking forward to your questions. All right, so the agenda of our session will be like follows.
I think the first session agenda item can be very quick, seeing that you already spent like four hours on that topic.
We will then present a small case study where we basically tried to, where we ran different S-bomb generators at different points in time on an open source solution in order to see whether those tools agree on the S-bombs generated,
whether the results produced are comparable, whether the results change over time, and to also pinpoint a couple of what we believe are deficiencies.
And then Joseph will explain why it is beneficial and helpful to go from the granularity of entire components to the code level to look at functions and methods and call graphs.
All right, so that is a software bill of material in the Cyclone DX format, which is one of the, let's say, prominent standards in this space.
Cyclone DX has a little bit of a security background. That's why we kind of choose this.
It seemed more natural to our previous works.
At the very top is kind of the S-bomb format and version.
Here in the middle you have the software product for which the S-bomb is generated.
Here you have the S-bomb generator tool.
We anonymized those solutions. We didn't think it is necessary to point out problems in individual open source solutions,
but we wanted to more raise awareness for the general problem of comparability and so forth.
And then at the bottom here you have an array of components that the S-bomb generator found in the software analyzed.
And of course there will be many, and I highlighted a couple of fields that we will be looking at in much more detail.
So there is the name, there is a CPE, there is a Perl, a package URL, or a group and a version.
So these are all different fields belonging to naming schemes in order to describe or identify the component that was found.
And then they have properties that can be all kind of proprietary properties the S-bomb generator decided to include.
Why do we need S-bombs?
I think in the interest of time I just skip this altogether and go to the case study right away.
The idea of this, the motivation for this came basically by reading through a couple of documents.
The first, and I have cited them here at the bottom, is a research paper that was done recently.
There is I think a couple of interviews and surveys.
And there were some statements from the survey participants who said that an S-bomb is not something that is static, that you create at a given point in time,
and then you assume it to be stable, but that is something that is evolving throughout the software development lifecycle.
And then a corresponding information is also provided in this guidance from the NTIA,
the minimum elements for a software bill of materials on pages 6 to 7 they say,
you can generate an S-bomb at different points on the sources, after the build, maybe at the docker image,
and the S-bomb should actually contain this information when it has been created.
That would be important for the consumer.
So we looked at different S-bomb generators and then of course, but in order to do a proper comparison,
we need to kind of a software to be analyzed.
And so as sample software, we have chosen Eclipse Steady, which is a security solution that I've been contributing to over many years,
so one that I know very well because I thought that would be helpful to understand the quality of what is generated.
In particular, we looked at one of the modules of that solution.
It's a spring boot REST service that is developed with Java and Maven.
It will be deployed using a docker image that is downloadable from Docker Hub.
And the ground truth, which is the information that we will take later on to say whether the S-bomb generators perform good or bad,
is what you see here.
So there are 114 compile time dependencies.
So they are required for compiling the Java sources to runtime dependencies.
They should be present where the production software runs.
There are 41 test dependencies, JUnit and other stuff.
Good.
Before showing the results or walking through the results, a little bit of background because that is very important.
How do we name those components, right?
And it is important to understand there are context-specific component identifiers.
So, for example, Maven.
So Maven coordinates.
This is what is used by the Java developers.
It is consisting of a group identifier, artifact identifier and a version.
This is typically the graph of the coordinates of a Maven artifact.
You would download from Maven Central.
There are some optional identifier elements.
An example here is org DOM4J, DOM4J version 213.
Another context-specific component identifier is the common platform enumeration,
which is comprised of a part, a vendor, a product, a version and a couple of other fields.
And they are used by the NVD in order to say what are the components affected by a given vulnerability.
So, for example, CVE 2020 something is affecting a component CPE 2.3,
which is the version of the CPE naming scheme.
The vendor is DOM4J project.
It's not exactly the same than before, a pity.
DOM4J is the product name.
And then, besides, there are universal component identifiers.
One that got a lot of traction in recent years is the Perl package URL.
It has seven elements that I put here.
And, you know, using S-bombs in order to use it for understanding whether there are known vulnerabilities,
what is the quality of the projects used and so forth,
requires to map all those names.
Names that are given by people somehow.
And that this can go wrong becomes evident by picking one example that I generated later on.
So, this is a copy paste from one of the S-bombs, the Cyclone DX S-bombs.
And you see here that the Perl here, this universal component identifier
that was put in by the S-bomb generator says it's DOM4J.
Too bad it doesn't map to org DOM4J, which is the identifier on Maven Central.
So, if you want to look up a new version of that component, well, bad luck you don't have the right identifier.
If you want to compare this CPE to search for known vulnerabilities,
well, it's not the same identifier.
They found DOM4J, but it's DOM4J project.
Too bad.
So, the approach we have taken is we selected three open source S-bomb generators.
A and B are generic solutions. You can basically throw anything at them.
A directory, an image, a tarball, a single, whatever.
And then C is a Maven plug-in that hooks into Maven's build process.
And we ran those three tools at three different points in time.
After cloning the open source project, after getting cloned,
after creating the Maven package, or after running Maven package to create the self-contained spring boot application that you can run,
and after finally on the Docker image.
On the Docker image, we could just run A and B because C is dedicated to be integrated into the Maven build tool.
And so we collected basically eight different S-bombs from those eight runs.
And the color coding will stay the same for the Venn diagrams I will be showing on the next slides.
And then we did three things.
We computed precision and recall of those tools.
So, which means we compared what they found with the ground truth.
And so precision says basically how many false positives are there in.
False positives is the thing tells me there is a component which I know is not there.
Not so useful.
Recall is for measuring false negatives, which is the tool didn't generate a component even though it is there.
Also not so helpful, especially for vulnerable dependencies.
And then with those S-bombs, so this is kind of the quality, the accuracy of the tools judged independently against the ground truth.
And then we created a couple of Venn diagrams to see how much do they actually agree.
So how much, what is the overlap of S-bomb A and B and C in those different times?
And then we looked at some additional properties.
All right, so this is the first, let's say, result running the three tools right after having cloned the open source project.
And let me start from the bottom of the slide with the tool C, which is easy, because that is actually perfect.
That integrated in the Maven dependency life cycle in this built tool.
Perfect precision, perfect recall, no false positives, no false negatives, right? Very good.
And it has a couple of additional properties such as SHA-1, SHA-250-6, digest, license, information, descriptions, a lot of useful stuff.
Now then let's look at tool A. You see the blue bubble is much smaller, because it basically failed identifying many, many, many components.
And the reason, I think, I mean, I need to speculate a little bit how the internals work.
But the reason, I guess, is that it looked at the POM.xml, which is where the developers declare dependencies, but it didn't resolve any dependencies.
So meaning it doesn't build a complete dependency tree.
So it lacked a lot of transitive dependencies on top of that.
For the direct ones that are in the POM, it didn't have any version information, because that was specified elsewhere.
So we have components like with this Perl, org spring framework boot, spring boot starter, without a version.
They included test dependencies, which is also interesting.
The other tools didn't include test dependencies.
But the funny thing is, they included it, but if you looked at the S-Bomb, you wouldn't know that it's a test dependency.
You can't tell, is this really something only used for developing, or is it really in my production system?
Also not so helpful.
And they had a couple of CPE combinations supposedly for mapping known vulnerabilities.
I think I need to speed up a little bit, right?
Okay, now we, this is, this Venn diagram I was mentioning.
So here, so the Venn diagram I didn't explain. So here, in fact, this is the overlap of those Perls.
So we looked at those Perls and tried, do they match to each other?
And you see that, to see, even though they had all perfectly identified, Tool B had a good number, but they still don't overlap.
And this is because those Perls contain additional elements, qualifiers, like the type, it's a Java archive,
or for open source, for operating system components, could be the platform, the target platform, which makes they don't overlap.
Now, if we only look at one of the naming elements, then the overlap is much bigger, because the fact that A is lacking versions,
B has wrong version identifiers, and the fact that C adds additional details, it all vanishes and looks like it's all converging.
But it is, again, important to understand the name alone is not so useful for looking at vulnerabilities or new versions.
Good, so let me hurry up a little bit.
This is the same thing, run after Maven package. Tool A improved.
They were finding more, but still the precision and recall are not as good as for the other solutions.
The other tools didn't change at all, so for them it didn't make a difference that Maven package ran or not.
Here, again, is the difference in terms of Perls, which is resulting in the lack of overlap.
Here, this is the same component. Tool A has it as DOM4J, DOM4J, Tool B has it at ORC DOM4J, DOM4J,
and Type equal to JAR, so they added this additional information, which made that they don't overlap.
Good, and then last, after running, now we ran it also on the Docker image.
The first two tools, and maybe one finding here is what we observed in terms of lack of overlap on Maven components also happened for operating system components.
So here we have Dibyan Udbuntu, the package dash, but again they, one tool added a little bit of more information, the target architecture.
For the consumer of the bomb, who knows whether this is important information in terms of security? I don't know.
And then again, if you only look at the name, the overlap is much bigger, but even though it looks like they only disagree on very few components,
too bad for Tool B, I think, had a big miss, it was lacking the complete Java runtime, and those people being in security, they know how many security issues there are in the Java runtime.
Good, lessons learned. The reason for getting different S-bombs is a big one, tools integrated into the dependency manager seem to work much better, at least on the result of that small case study,
because generic tools that are supposed to judge the bill of material from the outside, they will need to apply some heuristics, and they don't have the same level of detailed knowledge about the dependency graph.
Production versus test components are sometimes included, sometimes not, there are different defaults, and in the S-bomb generated you don't see the difference any longer.
And of course, there is also this difference depending on when you run it, there will be different components included.
There is a standard format, but the tools include different fields, some include license and digest, others not.
And even if they all include a Perl, Perl itself is a complex naming scheme with seven elements, and the tools decide differently on what to include in a Perl.
And other reasons that we didn't discuss here, it also depends on the time of the dependency resolution, in case your version ranges,
and some tools also generate platform-specific S-bombs, so if you create an S-bomb on a Mac and on a Windows machine, maybe with different hardware architectures beneath, you would have different S-bombs.
Right, and I think I don't have so much time to look into this.
What I wanted to say, identifying vulnerabilities only on names is rather flawed, because names keep on changing, projects are renamed, there are rebundles, there are forks,
and so which is why we advocate for enriching such information with call graph information and code-level information.
And with that, I hand over to Joseph.
Yeah, thank you, Henrik.
So this will be a bit shorter version. We're running out of time here.
I guess that's all it.
All right, so why do we want to go for more like a call graph view? So with the current S-bomb format, so in general with dependency trees,
if you view from that perspective, we typically have the application and the list of dependencies of how it is dependent, right?
And if you instead try to think from a call graph perspective of looking into the source code, you could have something like this instead.
If we see, for example, like those small sort of like, almost like Lego pieces, if each of them are function calls from like the application to the API,
we can see, for example, at lib4, and if lib4 would have a vulnerability or some other type of problem,
we see that there are actually no function calls to it from the application down to lib4 via lib2.
So I mean, the interesting part here is that if you start looking from like a code perspective,
we can quickly see whether we can pinpoint or like see how we're utilizing source code.
And another interesting part is that when, so I looked for example like into the Rust ecosystem.
So if we have a couple of dependencies, so for example, maplit here,
if you run like a grep over here, you can see that only like this one is let's say like, I mean important,
but we don't see any usage of it in the package.
So this is like a case where there's no code reuse.
And I was generally like interested to know like in the Rust ecosystem to see like how we are like doing,
like how many dependency we're actually calling or not calling in general.
And when I did the study and looked into like how many dependencies are declared and resolved
versus how many are actually being reused in the code,
I found that for using only package information, it would for example report around 17 dependencies.
Whereas in the case when you looked into the like full graph information,
we found that only six dependencies are used.
And that was quite interesting why there was such a big difference.
And the reason why there is such a big difference is that if you look into this example over here,
we see that main calls full and then from app to lib1, full calls bar.
And then further down we see that from bar it goes calls to intern in lib2.
But then we see that there are actually no calls from lib2 to lib3.
And this shows that why it is quite important to think about context in general
because depending on how app is using its direct dependency,
it also directly impacts what transit dependencies are also being called.
And the assumption that we usually have when we are building an SPOM
or we are looking into a dependency tree is that we are assuming that all direct,
I mean all APIs of direct dependencies are being utilized.
And then we are also assuming that all APIs of transit dependencies are being used as well.
So we need to also start thinking a bit about what kind of context is being used in general.
And so a little bit what would be the lessons here with trying to integrate something
like call graphs or other levels apart from just using package information
is that if you start having information around source code,
we can directly try to pinpoint and understand, for example,
if there is a vulnerability in one function,
we can see that AOK is being either quite utilized in my source code
or not utilized at all.
And another thing, so this is a problem that we also see as well
that we might declare dependencies on 20 components
where you get an SPOM from like a vendor or someone else.
They have five different components, but which one is the most critically used one in that project?
That's also not very clear.
And if you know, for example, usage of APIs, you can kind of get an idea around that.
And this was also a little bit the second point that I was like highlighting on that.
We need a few more layers of information that serve different uses of SPOM.
For developers, if I have access to SPOM, I would rather not look into metadata information.
I want to go look into call phrases and call information in general.
Whereas for security management people or other layers,
they probably don't want to look into the source code.
They rather want to look to get like an overview of seeing which packages
that are being used rather than the source code.
And so this sort of wraps up or talks.
We have like a couple of takeaways here.
And we see that going towards having some type of standard around SPOM formats
is, I mean, it's being necessary, but not fully sufficient.
Based on the previous slides, like we need a bit more context
so that we can have better actionable insights.
One way of doing that could be, for example, including call graph information.
As a consumer of SPOM, it is very difficult to verify the correctness of them
because as Henrik was showing earlier, if you are using different tools
and we are getting different results, which one is the correct one
and how can we even validate that they are correct in what they are doing.
And the last point, and this is something that both me and Henrik think is extremely important,
is that we need to create some form of independent SPOM benchmark
where different SPOM generators or others could evaluate on how accurate
the generated SPOM are against a set of manually validated projects in general.
This concludes our talk.
We are happy to take questions.
Thank you.
It shows Java and that's got an ecosystem.
So I presume you've probably found something similar with Python or Rust.
What about language applications that don't have an ecosystem?
I'm thinking about C++.
What would you say about that?
Basically there are standard languages there and I like the core graph there.
How would you tackle that?
You can probably take it.
The question is how would we do this core graph analysis for C languages
and that of course is a very different game and I don't think there is an easy...
It will just not be possible to be honest.
Because with all the core graph generator, you don't agree?
We need to do this for safety.
Building a core graph for C code is more difficult than it is for languages like Java and Python.
So maybe the amount of information that is contained in such core graphs is less helpful
for taking any actions or it's less actionable.
We were using this originally for those core graphs where it's for reachability of vulnerable methods.
The idea was there is method ABC affected by log4j is this really callable from my application context.
But this required kind of...
You could map the source code where the vulnerable function is identified to what is in the bytecode
where this identifier is basically the same.
So the core graph generated from bytecode could be used for this purpose.
And I don't think this is possible in at least this application in C.
Is this possible at some point?