Hi, everybody. Yes. So, talk about debug packages and
distributing debug packages today. So, my name is Morten Lindrö. I go by the nickname
of Fox Brown on the Internet. I have been a contributor to the Arch Linux distribution
since 2016. I'm doing sort of open source development since 2013. I do sort of security
teamwork, re-use the builds. My care is sort of about usable security, supply chain security
and all of that stuff and a lot of secure boots. But today I'm going to talk about
what I've been spending sort of two years of my life working on, which is debug packages
in Arch Linux. So, in the skills correctly. So, one of the sort of, normally when you
sort of get some crashes at some point, you will see this fancy little stack trace. And
if you use systemd, you will at some point have the crash handlers getting you the seg
faults, which happens. And then you can sort of just debug this with GDB. And if you do
look at the backtrace, you just see nonsense. There's nothing here that makes sense at all.
You can't figure out what happened. You don't know what crashed. And you have no idea. So,
if you actually do this on an Arch Linux system today, what you'll actually see is not that
nonsense backtrace. You'll instead see, no, let's cross Y. And you'll instead get this,
which has a lot more information. You'll see what happened, what crashed it. You'll get
all the symbols. And you did nothing. You did not download any debug packages. You didn't
think about it. You just happened behind the scenes. And if we ask what actually happened,
you'll see that there's some internal syscall that crashed it. So, this is super nice. This
is a lot better than sort of what the debugging experience has been on Arch Linux previously.
And it took me, I don't know, three years, two and a half years implementing a little
bit on and off. So, why do we care about debug packages? So, if we, for instance, have Pacman,
which is a fairly sort of simple and small binary, it's like half a meg of size if you
build it. But if you strip away all the debug information, you can almost half the size,
which is nice. So, if you don't need all of that information on your disk, it's nice to
sort of have some space savings. And in more like extreme cases, like in KeyCAD, had some
sole name inside of Python, it's like half a gig. And if you strip away the debug information,
it's 33 megabytes. It's sort of nice to sort of have the opportunities to sort of debug
all of this. And this can all be sort of very large. So, what people do instead is that
GDB implements what we call detached debug symbols. And that allows us to sort of separate
out the debug symbols from the binaries and sort of re-link it together on the system.
And one of the key elements for this is this fancy little build ID, which gets stamped
into every binary on your system. And we use that to sort of link. We define the build
ID. We can make some standard directory on your system. We can split out the debug symbols
from the binary, move it to that directory, add some debug link to the binary, and everything
just works. It will be as if the binary was, as the debug sections were still on the binaries.
This is nice. And this is sort of what Debian, Ubuntu, Fedora, all of them do to make those
debug packages. And that's nice. But one of the things that you saw in the demonstration
is that we also have the source code of the binaries. And that's more of a hack which
some distributions have support for and some distribution doesn't support. So Debian, Ubuntu
does not have source listings, I believe, while Fedora, SUSE, and now Arch as well has
source listings. And the way this sort of works is sort of you do a little bit of hacking.
So if we build Pacman just normally and we run GDB on it and we ask what the sources
were, you'll have your embedded project path in those binaries. So what you can do then
instead is to use debug edit. Historically, this has been part of the RPM upstream. So
Pacman didn't want to have a dependency on RPM to support debug packages, which is a
bit weird. But this was split out now into a separate project in back in 2001, no, yeah,
2021, which is now a separate project, which is quite nice, and it makes more sort of accessible
for other package managers. So instead of sort of using the current working directory
to embed stuff, we can rewrite all of those paths inside the binary to some standard path
on the file system. So in Arch, we use source debug and then we do name spacing so we can
have sources from multiple versions of Pacman. And if you sort of do these DOMs, you'll have
rewritten all of those source listings, which is part of the binary, which is super nice.
And then you can sort of get all the source code associated with binary. So before debug
edit was available as a sort of normal thing, Pacman also had support for source listings,
but he didn't use debug edit. He decided to use awk instead. So he then tried to parse
out all of the file paths, I don't know, from read-off, try to figure out whatever was there
and sort of try to get it out. And this worked for, like, simple C programs, but if you threw
like a rush to go at it, it had no clue what that was at all. So it was a hack. It worked.
It was in the source code for, I don't know, six years maybe. So I ripped that out last
year. So this, yes. So when these packages get built and you have the debug symbols and
have all of the source listings, we can then sort of compile all of this to some package
and then distribute it to our distributions. So all our packages in Arch Linux goes to
this repo.archin.org, which is a tier zero mirror. That's where all the packages gets
distributed from to all our mirrors. And on this, there's two package pools. There is
from corn extra. There's a package. Just flash debug pool. And for community, there's, okay,
there's a big community's dashboard debug, not packages. But these can be fetched and
distributed to all mirrors, but it's a huge amount of packages. So what we do instead
is that we are synced over this to something called a debug info instance we have, which
allows us to do fetch everything over HTTP instead. So debug info is a very cool microservice
which is capable of getting you the source code and the symbols from binaries over HTTP.
So you don't have to think about which debug packages do you need, which one do you have
to download to get full backtrace. We can just point GDB at this instance and it will
just fetch everything for us, which is quite nice.
So it's written, maintained by the ELF maintenance. It's a web server in C in the year 2020.
So it's running on, like, I think a few distributions, like, I think Boyd Linux has one, Debian has
one, Debian and Ubuntu got one past six months. And there's Fedora and SUSE also has several
of these. So it's super simple. We can just use the
debug info. We can give it that this is some tar archives that you want to parse and give
it a package pool. And we just set the debug info URLs variable and then we can run GDB
on the binaries and it works. That's all you have to do to sort of make GDB read those
files instead of having to distribute them. So, yes. And then you can have this debug
info find command line thing to fetch stuff for you or you can use it as a library instead.
But yeah. So running a web server in C in 2020 is, you know, a little bit iffy. So we
sort of wrote this, distributed this in sort of this hardware system file. So if something
gets exploited or something happening in that C code, you never know. It's still sort of
only really contained to some fairly restrictive set of policies. So you can't ask your privileges,
you can't really write anything to the system. But you can sort of just read stuff, which
is quite nice. So the only really two paths this has access to on our sort of in production
system is just these two package pools and some cache directory and sort of that's everything
it sees. So that's fairly quite nice. Been planning to upstream it. And I think you bumped
into and Debian uses this as well, but it's an extremely properly yet sadly. So, you know,
we have debug packages, we distribute it, people can use them, but we can also parse
metrics from people accessing this server. So I spent a little bit of time. Look what
you're how this vendors. Yeah. Okay. It does not like that. I don't know. I can't zoom
out. I hate this. So, so what you sort of see here is just some basic statistics. So
what people have been doing on it, we enabled debug packages for all our packages fairly
recently this year. So that's why you see the biggest corpus spike going straight up
because we have more symbols now. But you also see that we reached two terabytes of
data being sent out to different users the past month. So that's the last 30 days with
two terabytes out. And you can see some statistics on how much data people are fetching the errors
from through but statistics is sort of quite nice. And you sort of get this from free from
hosting it. Yes. So all of this infrastructure that's been put up in Arch, of course, is
all open source. There's no proprietary infrastructure. There's no hidden files. So all the stuff
we use to distribute debug info is all in our infrastructure repository under the roles
of debug info. That's sort of how we fetch all of the packages, how we do the service
management stuff, and all of those things. Yes. So I'll probably have more time. Yes.
So one of the things I also did because, you know, debug packages is usually done on C
applications and stuff, but I don't actually know C. I do Python and Go instead. So what
I also spent a lot of time on doing is to sort of try to get better debug info support
in Go because that's cool. So here, just to sort of give an example, here we're going
to crash the tail scale SSH client because that's a nice example, I think. So this instructed
the Go compiler to actually give us a core dump. And then we can use the delve debugger
in Go. And it actually, with a few patches, is able to read out all the debug symbols,
all of the source code, which is fetched from the debug info server as well, which is quite
nice as it will give us the more opportunities to sort of debug Go applications. It also
works on Rust. It also works on Julia and whatever sort of programming languages you
want. Which is quite nice. So it's sort of an improvement for the entire ecosystem as
well. Yes. That was it. I'll have a lot of time for questions if anybody has anything.
So I'm wondering what you actually store for the source. Is it the build tree or are you
trying to remove some things to save storage? Because, I mean, you have like a package,
you have an upstream source, you have patches on top of the upstream source, and then maybe
even the build process might generate sources itself. Yes. So I don't quite know how, but
this is just a binary, which sort of dwarf generates the source listing as part of the
dwarf metadata, I think. So this is all the, there's some generated optimized out sources,
I think, and there's some sort of things that points around to different sources, but it
will mostly just be sort of the patched up, generated, done sources, which gets embedded
there. So it's, the source listing is a nice bonus, but it's not necessarily some would
normally be distributing with the binary. That answers the question. Yes. Any more questions?
Thanks for using it. Could you upstream the system deservers files? Yes, it's been a
moment to do this for a long time. It's a little bit problematic though, because you
don't need to figure out sort of how the paths and stuff needs to get into the service file
with some configuration file, but it can probably be done. And I think that there will be people
use it as well. Yes, it should be upstreamed. Yes.
Yeah. So, and we normally hide the HB server behind the proxy. Yes. It's written in C++
if that helps. Yeah, no, yes. It's actually C++ is not C. It's all the elf stuff that's
mostly written in C, I think. Yeah, so it's a C++ program that uses Lib, micro, HBD,
and SQLite to store our other data. Yeah. So we have it behind the reverse proxy to
sort of get the TLS configuration going and outside, but we also just warranted the hardening
there because it's just, it's easy with system D to just get the hardening there. So it's
no reason to sort of not do it. So it's quite nice, but I'll try to upstream it.
Thanks. Yes.
Are those statistics on your dashboard pulled from the HTTP server he was describing? Are
those from like your Nginx or whatever proxy you're using? What? Sorry. Are the statistics
you had on your dashboard earlier? Yes. Are those pulled from the back end? Or are they
from like a proxy in front? So the debug info has a slash metrics, which
is all Promtail. So it just exports a bunch of metrics and you just point Promtail from
it and it will just parse it. So that dashboard is something we made internally, which I just
spent two weeks making, and that's also open source. So you can just fetch the JSON file
for the dashboard on the Grafana and everything there is all sort of open. So you can go look
at it. But it's all, it's just this sort of slash metrics endpoint of debug info. So
the Red Hat people actually watches this for all the debug info servers that has been
employed and they can like look at the statistics and errors from all of different servers and
see how the traffic between all of those are sort of how much is Fedora distributing compared
to Arch and stuff, which is quite nice. That was not public, I think. But yeah, it's cool.
Can you tell us a bit about the requirements in terms of storage? Because I recently looked
at another distribution and they didn't build all the packages because of lack of storage.
So that's what I'm trying to figure out now because we enabled debug symbols for all the
packages, but they're not currently distributing it to our mirrors. So Arch, the total mirror
size for Arch is like 60, 70, 80 gigabytes, I think, of data. But I assume like that would
be several hundreds if we actually upload all the debug packages. But I think Fedora
in total is like three, four terabytes or something. So I assume it will go inside three,
four times and stuff. I know like the LLVMD stuff is like a two gigabyte package, I think,
with symbols and people try to optimize it a little bit so you get a better, faster
to upload. So it's, yeah, one sort of main issue with debug edit and sort of debug info
and stuff is that you have, Dwarf 5 has support for compressed sections, but debug edit does
not understand the compressed sections. So you have to decompress the sections before
you leave out the paths and there's no good way to sort of recompress everything again.
So getting better support for sort of compressed Dwarf info would sort of help fix a few of
those sort of space requirements, I think, on the mirrors.
Can I ask another question? Is there work on the duplication instead of compression?
Because you have different version of packages as well.
So it's not that relevant for ours because we don't keep those versions and we don't
really do delta files on the packages. So on the arch side of things I don't think that's
really relevant for us, but I don't know. It could probably be done at some level, at
least for like Fedora or Debian that keeps multiple versions of the same package.
A small question, for which architectors are you generating those debug info binaries?
So arch only really supports x8664. We don't really have any other architectures. But because
we have the 32-bit port and we have the ARM people and I think they're just pulling our
packages and probably building debug in full for them, but arch itself is not really distributing
anything else on x8664 currently.
So you mentioned different architectures. Do you know if there's plan to upstream the
booking for D and in general risk five because I know Felix Yan is working on this?
Yes, I know Felix is working on it. We want to, this is more an arch thing, but we don't
have traditional build farm server setup. So it's a bit hard for us to do multiple architectures
because one package maintainer has to build that package for each architecture. So currently
we want to have support for more architectures and better support like V2, V3, V4 versions
of X that you see now supporting. But you currently haven't really solved that in a good way currently.
Okay, thanks.
Thanks.
Thank you.