Why are you all here?
This is a boring topic.
I was not expecting so many folks,
but I'm glad to see you.
My name is Frank Heigler,
I'm a Red Hat engineer.
I don't have a bio site because I'm not that interesting,
but I've been in free software for a couple of decades,
almost three, quite a while.
So this talk is about debugging information and
another type of information that we hope to
popularize storing online for occasional uses.
Now, many of you guys know debuggers already, all good.
The other subject is a little bit more esoteric,
but we can still talk about it.
Do you know if Mike is coming in okay?
The mic is just for the recording.
I know. By the way,
who's the next speaker after me?
Is that person in the room?
Good. I might be able to sell to you sometime.
I'll keep this pretty short.
Well, I offered you 10 bucks but too late now.
So this is boring.
How are we all right software?
Binary has come out.
Some packages up the binaries into distribution.
Distribution goes out to people.
People who run binaries,
everything is happy ever after.
That's all that ever happens.
Right? Right?
So debugging is near and dear to me.
I worked on the GB debugger a little bit here and there,
back in the prehistoric times,
and I've worked on debugger like tools ever since.
Despite all my efforts to try and make
debuggers irrelevant,
we still have bugs in our software and
you still need to silly things.
So unfortunately, here we are and debugging is not so easy.
So I have two parts of my presentation.
This first part is about debugging information that's online.
The second part will be about something else that's online,
that's tangential,
but you'll see the connection pretty shortly.
So many debuggers.
You know how debugger,
is everyone familiar with how
debuggers work roughly speaking?
Not you, Pedro.
So one of the main challenges of debuggers is that it has to
operate at the machine level,
at the register level,
at the memory bits and bytes level in order to understand
the operation of the program you're trying to debug.
And this is despite the compiler doing its darnedest
to erase any remnant of how the original source code looked.
It's trying to do its very best in nuking every unnecessary,
very variable access,
maybe tightening up the data structures,
shuffling in and out of registers all the time
just to make things go damn fast.
Compilers are done that way.
Well, I mean they're great,
but if you need to debug,
you need a sophisticated way of telling the debugger
where to find all that stuff.
So long introduction to that,
but we need a good high quality debug info,
which basically gives metadata about where the,
every piece of the source level constructs are at runtime
in the actual machine.
So where, where to registers,
which memory spots,
how each complicated data structure laid out,
all those things have to be saved by a compiler,
put somewhere,
and then made ultimately available to debugger.
So does the word dwarf mean,
you guys know what that stuff is?
Yeah.
So, okay.
Can you give me a few adjectives about dwarf?
From the heart.
Okay.
Say again.
Did you say short?
Liar.
Yeah, so dwarf is a very compact,
amazing little,
most graph database kind of thing.
It is absolutely nothing but,
it is not short.
It can be order of magnitude larger than the actual binary.
And because it is that large,
distributions then not to ship the thing,
not to trip it to normal users because,
you know, it's like,
like I said here,
users just run things, right?
You never debug.
So they don't get the debugging phone normally.
But say you do run into a problem,
you don't want to debug,
well then you need this information, right?
So either you can be the developer who already had this,
or for the last 20-ish years,
various distros made available,
the original debug data that the compiler generated,
but it's not installed.
It's somewhere off in a separate repository
you have to sometimes enable and change the route
and download and if you're lucky,
you get the corresponding debug data
that for the binder that you're trying to work on.
So the brand new 2019 thing,
which I remember when my team talked about here,
two years ago,
when it was younger,
is this gadget that we,
a community built called Debug Info D,
which automates the distribution of the Debug Info
and other precious such things.
And the whole idea is to make it as easy as possible
for people, not just developers,
but ordinary users to automatically,
without special privilege,
get all this stuff for as much of the system as possible,
you know, without having to go into route,
without having to do,
you know, activate channel,
rel-debug-blah blah blah blah.
Okay.
So that's our little baby there,
the first URL points you toward a website
that describes the current situation.
As I said, the project is now getting to its third
or fourth year, so it is,
I cannot call it a prototype in any sort of honest way,
but work is still ongoing quite a bit.
When we built this, it's a small server,
part of the ship with the AlfvTills tool set,
which is related to Alf and Dwarf decoding
and processing and such,
and there are a lot of low level,
machine level tools in there.
So Debug Info D is shipped with AlfvTills
and it's shipped on all the major distros that I know of.
All right.
It, I forgot to mention this,
but I mean, all it is,
is it allows a debugger type tool to request
Debug Info as well as source code for any binary
based on the Hexadecimal unique build ID
that's inside binary.
So this is a kind of a hash code
that's been in binaries for almost 20 years,
thanks to Roland McGrath and a bunch of other people
who made it happen back way back in the early odds.
So it's an HTTP server,
it's just an ordinary boring HTTP server
as far as the clients are concerned.
It's very cashable, it's very lightweight,
very, very simple, no like XML, API, blah, blah, blah.
It's just HTTP.
Because it is trying to be really simple,
it, we found it over a course of a couple of months to year,
most major debugging type tools grew
capability to use this API,
use this web of system to fetch this stuff.
So obviously, GDB is one of them, it was one of the first,
but system type is another tool of this kind,
it's close to me.
Practically all the debuggers and tracing tools
and profiling tools we know of is able to do this now.
So the clients are well dispersed in a source code system.
The servers are also in really good shape.
Over the last few years, the whole bunch of distros
came online with running their own debugging for the server.
So Fedora was one of the first,
and Bacento is up there,
Debbie and Ubuntu and other smaller distros,
they're all running this server now,
whereby their own distro is fully debuggable
through this system.
So that's cool.
The, we're not quite finished with it.
There's a piece of extension stuff
that we're still working on of one of them
is that's particularly cool is cryptographic signature
preservation of individual files.
As you may know,
archives as a whole can be signed by distros
and then a client can verify that the archives
have been modified, and that's cool.
But if you don't want to download the whole RPM
because it's too large or for various reasons,
you just want to extract the actual source file
that you want or an actual little debug door file.
You still want to be assured somehow
that that file was what the initial distro packaged, right?
You don't want to make sure it wasn't
adulterated somewhere in the middle.
It's kind of security critical.
So we're bringing into this web protocol
the propagation of the signatures
that may have been applied by the distro
at the build system level.
And that's, it's not easy and not many distros
do that level of signature stuff yet,
but Fedora and very modern rail do,
and we hope others come online too.
But what's nice is that each individual file
has its own crypto signature,
which can pass it down through debugging for the,
all the way to the clients.
So they can be assured they get the correct
100% grade A certified file.
Alrighty.
Psh.
I couldn't bring myself to try a demo here.
I was just too chicken, but,
but the whole idea with this debugging for the clients stuff
is that it is really automated and integrated
and you don't have to do anything special.
On the distros that this is enabled on,
you just don't have to even do the first line.
It'll be done for you in the ATC profile
for all your shells.
And you just run GDB on any random binary
or your own binary and it'll pull in the debugging for
for any shared libraries that you're using,
any source files you're stepping into,
it'll just pull each piece down one by one as necessary.
And it's just, it becomes a non-problem.
So there's almost nothing to see
because it's just so smooth and automated.
Parts of it can be slow for hilarious reasons,
but I'll explain why if someone asks me that question.
Anyway, it is nice.
It is out there in many of the distros.
I hope you guys enjoy it and I hope it makes your lives
a little easier.
That is the thing you ever encounter bugs.
All right, all right, all right.
So switching over to the other sort of health topic.
All right, does everyone know what ABI means?
Can there be someone person who does not know?
So I can justify talking about it?
No, thank you.
I'll just be brief.
As brief as possible.
So it's interesting.
There's a lot of interest, especially for ISVs
who want to build a piece of software
and then distribute it, let people run it
on multiple distros.
But even for normal projects who might want to build
a binary of their own releases
and then shift that to various other distributions.
You got that?
To shift it on various distributions
so that it can be used on modified.
Sometimes they've other problems like
wanting to match different generations of shared libraries
which might have had little evolutions of their own ABI
whether a function signature got changed
or type got changed.
Something that's not the same at the binary level
than it used to be.
Which means that shared linking between them
is no longer safe.
Some projects, some shared library projects
are exquisitely careful about this
and they do incredible measures to prevent
this kind of breakage.
When they update their shared library
it becomes backward compatible to decades ago
by a lot of the works.
Like Jalipsi is one of the best in this regard.
But some libraries are less good at that.
So if you want to write a binary
that will work with multiple shared libraries
you may need to either kind of ignore the problem
hope it doesn't happen
or you need to find a tool to check
whether this will work with that.
It is a bit esoteric.
But there are lots of solutions
or several solutions which try to work around
this whole problem by just giving you, bundling you
the one random version of a shared library
from some random distro, package it together
into a container image or a flat pack or whatever
and just plop the whole thing on your system
and then they've done the integration checking
and then they know it'll work.
It's legitimate.
It's just it's very space obnoxious
and some of them still kind of intermingle
the bundle libraries and the host libraries
and they kind of do version checking
and they hope that the host's libgl
will work with their version of libxt or whatever.
So even this is a bit fuzzy.
So anyway, what we're proposing is that projects
that deal with multiple versions of shared libraries
that they're concerned about
compatibility checking for ABI's
consider the gadget I'm going to talk about.
Okay.
You know, maybe we'll just skip this one.
Everyone, but everyone knew.
What's there's still a person
who I didn't tell what the ABI was?
It's just, it turns out to be exactly the same metadata
the debugger uses to find variables at runtime.
This is exactly the same data.
It's just that it happens to be useful
to examine even at compile time.
So even with just the dead libraries on disk
by parsing and processing the exact same debug
info, you can tell whether that shared library
has the same binary guarantees
as a normal program might require.
Sorry if I belabor the obvious guys.
Okay.
So our team at Red Hat, one of the tools they work on
is this gadget called libabigale.
I'm not sure who works on that, that guy there.
Yeah.
And it's awesome.
It's a suite of tools, binary tools
that compare shared libraries versus shared library
by extracting their debugging info basically
and just parsing it piece by piece,
function by function, type by type,
make sure they're all compatible with each other.
It can also do match a binary to a variety
of shared libraries and see whether there's still
meet each other's needs.
Like a good marriage, maybe.
One thing that it's limited though is that to do this work,
it needs to have all the files that you want to compare
right there on your local disk.
So if you want to compare your binary to a
RAL6 version of libc or libxgl
and you bunch a version, you need to somehow
get hold of those files first
and you can't really just do it otherwise.
So our gadget, the new gadget we're adding to libabigale
is a way of not requiring you to download
all these shared libraries and all their corresponding
debugging info for all these versions of distros
that you might not even have or not even want.
Just curious about.
And a key to that is to realize that abigale also
can take not just dwarf files, but also an XML
representation of the dwarf.
And XML is just conversion.
It's a subset and conversion.
So that's my four minute warning.
We're doing okay.
And because it's XML, it's large, but it's textual
and it's compressible.
And with the one track mind that I have,
how can we store a large amount of XML
for all these shared library data for a large distro?
It's text, it's large, you want to share it.
Well, how,
oh, no, that's not the next time.
I'm gonna leave that in mystery for 20 seconds.
Oops, one moment.
Two moments.
It's pretty soft for it, don't worry, it's good.
Ha ha ha.
Okay.
Yeah, we just skip over here.
So writing a little tool is really just a thin wrapper
around the existing, the abigale tooling
to extract this XML version of the ABI.
Jamitin, Git, because we love Git.
It's a great way to store text files.
And it's a great way to ship them,
great way to compress the heck out of them
and let them coexist in some nice way.
So we can extract XML from a large corpus of files.
We can give it a whole boatload of RPMs or Debian's
or whatever, it will automatically extract
all the shared libraries.
It'll download all the debugging files
and automatically via debugging 4D, if necessary.
And it will generate a Git tree,
which has all this XML stuff nicely structured
that then can be used by the tool itself
to later do a compatibility check.
That way you don't have to install
the foreign distributions anymore.
Anyone can do you the honor or the favor
of collecting this ABI XML stuff,
sharing it in Git, put it up publicly,
and then anyone who wants to compatibility check
against that version of the OS
that no longer has to worry about this.
This is a crowdsourceable enterprise.
So I tried this at home, no demo because no demo.
But it is really not hard to use.
All the prep work is just getting the software.
But the thing you think is that the data is in,
when one crowdsource version of data is now a couple
of gigabytes, it has a big section of Rela8,
all of Rela8 in there.
As RBI stuff, a few other Ubuntu releases just randomly
in there to plan to expand it,
to have as many distros in there
as people are willing to give us.
To submit new information, it looks like these command lines,
this is just to demonstrate you can build your own
share library at your own institution
and generate your own database.
This version tells you that it can just mass import
whole RPMs and they'll do the right thing,
decompress and aggregate all the information.
And at the top here is how you check a random binary
against the entire set of shared libraries
that that binary needs.
There are a few bits of cleverness in there, small.
It's not very clever, just a little clever.
For example, as you know, libraries get updated every now
and then we wanna make sure we can store more than one
version of the same shared library in the database.
There's not just one G-Lib C but 10 per update.
So they all have to have a naming convention
that lets them coexist.
So we do that.
But those are internal details.
The basic thing is you can submit to the database
this way and you can check with that
and it tries to be that simple.
And that's my conclusion page right there.
That all the code is open source obviously
and all the servers are extremely low tech on purpose.
The first one is a very thin HTTP server
and the second one is just literally a Git server
that happens to have structured data inside it.
So easy and even I can do it.
Very, very straightforward, baby technology.
And thank goodness, that's it.
Can we have entertainment for questions?
Yeah, we have minus five minutes.
Minus five minutes, my God.
Okay, any zero questions?
Thank you.