Okay. Hello, everybody. So, my name is Doji. I work in the tools group at Red Hat. And
so we're here today to, okay, first of all, thank you for staying. So, yeah, I wanted
to talk about application binary interface analysis today. And, okay, first of all, who
doesn't know about Lubabigel and ABI stuff? So, okay, so I think we'll have something
for you guys. So, what are we going to talk about? So, first of all, I'll introduce what
ABIgel is, and we'll look at how it works, what the project brought recently, and what
we're looking for as far as the future goes. So, ABIgel is about doing analysis of application
binary interfaces. So, it's a set of tools that can do things like compare the ABI of
two binaries, or store the ABI of a binary onto a, you know, disk format. It can do comparison
of binaries, you know, that are in packages, like Debian packages, RPMs, star files, etc.
And it is also a shared library that you can use to write more tools if you want. So, that's
all well and nice as far as marketing goes, but then let's look at, you know, what we
mean by ABI. So, suppose you have a simple program, well, a simple binary which has,
okay, simple, no, very complicated, let's say, which has, you know, three functions
that are here. The types of the functions are defined here in a simple hierarchy. Here
you have the first type that inherits, you know, S0 which inherits base type, and let's
say another type here that inherits S0. Okay. So, that's the first version of it. Let's
see if, I don't know if it compiles. Yes, it does. Then I have a second version of it
which looks quite the same, but, okay, what does it do? Okay, what's the difference between
the two? Very simple. I just inserted a, you know, data memba in the base class, and we
want to know what the impact of this is on the ABI, you know, as far as the binary goes.
So, where am I here? I'm in the source code of the project, and so I've built a version
of it. And so, here we have one of the tools which name is ABI diff, which does what you
think it does. And so, if I run it, what does it say? Basically, there are two changes
as far as ABI goes in that battery due to the change of sin. So, the first change is
about, you know, the first function which is here. And so, it is telling us basically
that that function has a parameter type that changed. And the change is about this, you
know, structure, remember? Something is interesting. The size hasn't changed, even though I've
added, you know, a data member in there. So, yeah, you know the drill, right? If you don't,
I can, you know, explain it more. But size hasn't changed. The base class has changed,
and the change here is a data member insertion at a certain offset, blah, blah, blah. So,
this is the impact of the change of the first type on the first interface, right? And so,
there is another interface that got impacted, right? And the parameter of that function,
which was struct s1, changed as well. The base class changed. The base class was struct
s0, right? And the details of s0 change were reported earlier. So, we don't have to, you
know, repeat it again, right? So, here you see that we compute the changes, right? And
we also analyze those changes so that we can detect if things have been, you know, reported
earlier or not. And also, we mess up with more stuff, because here we say, for instance,
that there were two changes, for instance. But one got filtered out. What does that mean?
So, let's see, for instance, if I recall the... Okay, I'll add a special... So, I've asked
ABIDF to show me what, you know, to show me redundant changes, because by default it removes
redundant changes. And we see that we have the third function that was impacted as well
by, you know, the change we created. And so, well, all the changes that, you know, impact
function three were already reported. So, this is why it was suppressed. That change
was suppressed by default, because it was redundant. So, it's not just... We're not just
diffing things. We're analyzing the diffs, and we're trying to, you know, massage those
diffs so that they can be consumed by human beings. So, this is what we mean by analyzing
ABI's, basically. So, how it works. The library used to implement the tools has a front-end,
which is kind of backward. The front-end reads the binary. Usually, it is back-ends that
writes binaries, but, okay, here, backward. So, we read the binary, which has to be in
the ELF format right now, and we build an internal representation of it. We look at
the publicly defined and exported symbols of, you know, declarations, basically functions
and variables. We build a representation of them and their types. And then, we construct
the graph of the types like that and their subtypes, and we pull all that together, and
we call that an ABI corpus. A corpus is an artifact for us that represents the ABI of
the binary we were looking at. And so, there is a middle-end that acts on that internal
representation. Said otherwise, it acts on ABI corpora. Corpora being the plural of corpus
in Latin, right? Let's be pedantic. So, we can, as you've seen, compare two instances
of ABI corpus. Then, we build an internal representation of the result of the comparison.
We call that an diff IR. So, it's a different IR. And then, we perform transformations on
that diff IR, like categorization. So, we would walk the graph and say, okay, this
diff node, we've seen it before. So, we'll mark it as being redundant to this other one.
And then, they can be, you know, transformations that are suppression as well. Well, suppression.
We will mark the nodes as being suppressed. For instance, because the user wrote something
that we call a suppression specification file requiring that some types changes might not
be reported. So, once we have that well-massaged diff IR, we have backends that walk that diff
IR, obviously, or the initial IR and do useful stuff, like writing, you know, emitting reports,
for instance, or emitting, you know, the representation of the ABI corpus in a disk-saved format that
we called ABI XML. So, what we've done recently, so I'm going, you know, a bit fast because,
you know, to let time for questions and stuff, and we can go on and, let's say, not very
structured discussion afterwards, if you like. So, yeah, in the recent times, what we've
done is, well, you know Dwarf, you know that it changes all the time with new versions
of Dwarf producers. So, with GCC-11 and LLVN-14, the default Dwarf version was bumped to the
version 5, which is quite ancient, actually. I think it was released in 2017 or something.
So, yeah, we support most of that right now. And another major thing that happened recently
was that thanks to folks in this room that I won't, don't worry, I won't give your name.
New debug info format were added because, yeah, we started with Dwarf only. And so, the
CTF debug info format was, support was added to the Babigel. So, basically now, if you
have a binary having CTF and or Dwarf, you can choose whatever you want to use as a source
of type information. So, things being how they are, the code got changed a bit to, you
know, to be turned into a multi front-end architecture. We also have a multi backend
architecture, basically, because we have different types of reports. The one I've shown you
is the default one, which is quite verbose. So, some people like it more terse. So, yeah.
And who knows whatever weird request users might come with in the future. So, yeah. Different
report backends. And, well, it's not, doesn't stop there. We are still working on, you know,
on new stuff while coming from user request. So, yeah, the, apparently the new kids on
the block, well, new kids in town now, cool stuff is BPF, right? And with BPF comes BTF,
which is the type description format of BPF. And so, there were some requests to support
that. So, it is now in mainline, even though it's not in Babigel mainline, but it's not
released yet. It should be released in the next version. So, what do we do with that?
What's that thing? Basically, because BTF describes the C types, basically, we are using
that to compare the interface exposed by the kernel to its modules. So, we're doing that
with CTF already, with BTF now, and also with DWARF. With DWARF, it is much less fast,
shall we say, than with the CTF support and BTF. So, people can, people are using that
feature to, you know, analyze the KABI, basically, kernel ABI, that thing that doesn't exist.
And then we've had, you know, weird project specific request over the year. And the last
one that, you know, came in last month, I say, or yeah, yeah, last month, yeah, in January,
was to have a, I call that the library set ABI analysis. So, basically, it's a project
that has a huge library, a huge library, and they're planning to split it in different
libraries, right? But then they keep ABI compatibility, they're supposed to. And so, they would like
to ensure that the set of, you know, broken down libraries has an ABI that is equivalent
or compatible with the first initial one. This is what I, you know, call the library
set ABI analysis. So, we're going to add support for that in, I don't know if it's going to
be in the next version or not. So, yeah, these are the kinds of things we are, we're working
on. So, yeah. And now, I'll let you ask questions if you, if you have any, yeah.
Does the library have any support for language specific ABI? So, languages are good on top
of C, for example, but they have language schemes?
Yeah, exactly. So, yes. So, there, of course, DWARF is multi-language. So, if the compiler
of that language emits DWARF, then we're good to go. There is a small layer of language
specific stuff we add, you know, for reporting so that we can talk, report stuff in the native
language of the programmer, you know, who wrote the thing. So, to give you a concrete
example, right now, we support C++, C, Fortran. Someone asked me for Rust support. So, we
had that, basically. We have some crashes on OCaml. So, I thought we were supporting
it, too, but I need to do some stuff. So, yeah. Basically, yeah. It needs work, but...
The new language, I just have to define a small layer for the mangling logic.
For the mangling logic. So, okay. I can show you, let me show you an example. So, yeah.
I was writing. So, yeah. Let's see. So, you see, for instance, in C++, we'll compare,
so here, you see this function, the function 3. I'll change it in the second version here,
function 3, and add, and I'll add an integer here, right? Yes. Let's, whoops. We compile
that, whoops. And, whoops. Weird stuff happened. So, look at what it is saying here. So, you
see here, because we're in C++, I changed function 3 in the source code. Yeah. Let me
just, yeah. See? I changed function 3 here, and I added a parameter, you know? That's
what the programmer would say. But then, from the binary standpoint, what we're seeing
is that the first function was removed, and then another one got added. This is because
in C++, the name of the symbols of the two functions, the two versions of the functions,
are different. They have a different mangling, okay? So, we go to, we go from the name of
the symbol to the name of the declaration, right? So, but if I do the same in C, then,
like, yeah, I knew you would ask that question. I don't know you, but, so, and I have second
version here. Boom, boom, boom. And, so, here, some, oh, sorry. I changed the name of, sorry,
I changed the parameter of the function there, but this is in C, okay? And so, if I go in
the, ah, sorry, if I go in the shell and I look at, boom, at the two, so, this is, the
first one was hello, and this one is bye, of course, because I think this is going to
be the last C here, because in C, the name of the two symbols are the same. Now, we say
that the function has changed. So, these are the kind of things that we'll have to adapt,
basically, but there is not much to do. In some cases, you have mangling, and in the
others, other cases, you don't. So, you don't have anything to do with the, for the mangling,
the, you know, does that answer your question? Roughly. Roughly, yeah. You have this code,
part of the code, which decodes the mangled name to an unreadable name. No, no, because
the, the, the, the matching is done by dwarf. So, we know that this symbol is for this declaration.
So, we don't have to do the mangling, the mangling or demangling. We, you know, we'll
look at the addresses and we know that this symbol is for that one. So, yeah, we don't
really care about, yeah. Another, yeah, please go ahead.
Oh, there is none. No, no, no, no, no, no, no, no, it's, so, yeah, I, just to, to refresh
the question, to repeat the question for the, yeah. What are the performance issues, you
know, when we analyze, like, big libraries, like, you know, he said, I love VM, but, you
know, there is WebKit, Gecko, etc., etc. So, we have a, when we're looking at, when we're
looking at dwarf, we have a fundamental problem, which is the duplication of types. Here we
are in the business of comparing things, right? And so, when we compare types, basically,
we are in a, the, the land of quadratic algorithms, right? So, things are inherently slow if we
do them naively, right? And so, the thing is, in dwarf, every single type unit is represented.
But then, when you have a, the final binary, the final shell library, for instance, and
you have, I don't know, you know, 1,000 translation units, and in every single translation unit,
you had the string type, for instance, that was used. Then, you will have the string,
the string type represented 1,000 times, at least, you know, in the, in the, in the dwarf.
And so, we must be sure that those 100 occurrences of string are the one and the same. We can't
just look at the name and say they're the same, because they could be otherwise, right?
And so, we have to compare them and make sure they're the same, and then we'll say, okay,
I'll just keep one and throw away the others. This is the duplication of type, it is called.
And so, this process takes a huge amount of time, which is, well, for, for huge libraries,
it can take, you know, it can take forever. So, we have heuristics to make this thing,
you know, be faster, but then, you know, it takes time. So, we have some of the heuristics
that we're using now are, is in the land of partitioning, like we will do things, you
know, like piecewise, and, and try it so that we can do things in parallel, right? It is
not mainline yet, but this is the, you know, the, the future we're, we're thinking about.
Another approach is to have the types be de-duplicated before we intervene. This is what, for instance,
the CTF guys do with C. So, they will do the de-duplication at debug info production time,
and then in that, in that case, we're golden. There is another, another case where we're
doing that is when we're building distribution packages, like, for instance, I don't know,
RPM or Debian package or whatever, there is a tool which is called DWZ, which does the
de-duplication to one extent. Well, when it works, it works. It does the de-duplication,
but the problem is DWZ has the same issue as us, and sometimes when the binary is too
big, DWZ will just give up, and in that case, well, we have to use our little hands and
do the de-duplication in line, and then, well, we'll spend time.
But this because someone should get DWZ, turn it into a library and put it in the linker?
Yes, and, yes.
Do it in link time?
Yeah, we can, yeah, that's, that's something that, that's one of the things that we need
to do to improve the entire ecosystem of these things, and, yeah, that's definitely, yeah.
So, yeah. So, as, do we have other questions?
Yes, are there any other formats that are on your road map?
Right now, no, but, you know, like, three months ago, BTF was not on my road map, so, you know,
the future is not what it used to be, so, I don't know, yeah.
Anyway, so, yeah, we are on, hosted on Sourceware, we still use mailing lists, you send us patches,
and yeah, you can find us on IRC, on the, on the, of this network, and, well, thank you very much.