Good morning everyone, thank you for being here.
My name is Gabriel Osama, I'm going to try to get the introductions over with quickly.
I work for Carnegie Mellon's cert, which is the sort of OG cert the US government started
back in the 80s after the Morris worm because they suddenly realized computers were going
to be a thing they were going to have to care about.
The cool thing about that is I get to indulge in my paranoia and OCD in a professional capacity,
which is much, much better than it sounds, probably the way I make it sound.
I'm going to probably sit down during this presentation every once in a while when I
need to work the demo a little bit, so don't think that's weird.
With all of that out of the way, we're going to talk about self-hosting and why that's
important and how it impacts things like hardware and the ability to trust it, and then further
into that sort of distinctions between ASIC's application-specific integrated circuits,
dedicated silicon versus programmable FPGAs and what the threat models are in the trade-offs
and how much you can trust each one of those, and what you're gaining and losing when you're
switching between them.
And then next will be a demo of what probably is the slowest, most memory constrained computer
that's capable of running Fedora Linux that you've seen recently.
It will be on a 50 megahertz rocket chip CPU, soft core CPU running on an FPGA.
It's going to have 512 megs of RAM in this particular incarnation.
It is using, like I said, rocket chip and Litex on an FPGA with free open tool chains,
EOSUS, Trellis, and XPNR being used to build a bit stream for the FPGA.
And then this computer, when it runs Fedora, you can install EOSUS, Trellis, and XPNR on
the computer that was built using those tools and run those tools on the computer to rebuild
bit stream for its own motherboard.
So it's basically like a self-contained, self-hosting thing, which is really exciting.
So let's start with this whole idea of self-hosting.
Most of you are probably familiar with what that means.
The joke is, well, no, it's not me hosting my own content on Google Drive or somewhere
in the cloud, but rather it's a term of art in the field of compiler design, and it means
a compiler is written in its own language and it can compile its own sources.
And then there's a related concept of bootstrapping, which is basically kind of, well, you have
a self-hosting compiler that built its own sources, well, chicken and egg, which one
was there first.
There had to be a third-party trusted compiler that was originally used to build the first
binary of our own compiler before we could rebuild it, and at some point we reached stability
where the next iteration of the binary we build out of the sources isn't significantly
different from the one we already used, and that basically means we've achieved self-hosting
and the process for that is called bootstrapping.
And one interesting thing about self-hosting compilers is that they suffer from this attack
that Ken Thompson pointed out, Ken Thompson being one of the designers of the Unix operating
system among many other glorious achievements.
He pointed out that a compromised binary of a self-hosting compiler could be created
that attacks clean otherwise sort of benign trustworthy source code and builds malicious
binaries, one being of like the scenario he described was the attack against the login
program, which if you build a login program, it'll have a backdoor root password that will
allow somebody to log in without knowing the actual system root password.
The other thing this malicious behavior does is it inserts itself into any subsequent iterations
when it detects that the compiler's own sources are being built using it.
So it's a self-perpetuating attack that isn't actually present in the source code, and the
only way to get rid of that would be to reboot strap the entire compiler because presumably
we do trust the sources and the sources are clean and there's no like malicious behavior
specified in the code itself.
And one way to not necessarily get rid of the problem but to point out or to test whether
we have been subjected to one of these attacks is David A. Wheeler's PhD dissertation called
Diverse Double Compilation and in the example here we'll be using CC as our suspect compiler
and TC as the third-party compiler and it's not necessarily T for trusted, it's T for
third-party and the heuristic here is that we pick the third-party compiler in a way
that gives us a high degree of confidence that it is not in collusion with the suspect
compiler.
People who put it out aren't the same group think maybe GCC on one hand and MSBC, Microsoft
on the other or something like very diverse, that's where the diversity comes from.
The way this works is that if we compile the sources of CC with both our own sort of suspect
binary and with the third-party binary, if everyone's innocent and no one's trying to
screw us over what should happen is we should be obtaining binaries reflecting the sources
of CC that are functionally identical because these are diverse different compilers, they
would produce different code like the code generation would be different so the binaries
aren't like bit by bit identical but they should be doing the same thing because they're
implementing the same source code.
And then if that is true then the next move would be to take the sources to CC again and
rebuild them with our two intermediate compilers that we obtained and if you control for the
initial conditions, if you have the same initial conditions, same random number generator seed
and everything and identical input pumped into functionally identical binaries the result
should be bit by bit identical and if that's true then we can breathe a sigh of relief
and say okay we are very unlikely to be subject to a trusting trust attack and that degree
of confidence is sort of equivalent to our heuristic ability to pick a third-party compiler
that isn't in collusion with our suspect compiler and by the way the highlighted box on the
bottom here is basically the process of bootstrapping CC using the third-party CC compiler.
So back to self-hosting, if you have a self-hosting compiler and source code to everything, the
binary of the compiler when it operates, when it runs, it runs on top of I don't know a
C library and the kernel and basically a software stack, it's an application on top
of that but it's an application that can compile all of the things it needs to run itself.
And if you have source to everything and you've compiled everything from sources that you
otherwise trust then you have a self-hosting software stack built around your compiler
and sort of the applications are bonus, you know, all the stuff you actually want to use
the computer for.
If you build that from source but the stack of software, you know, with the C compiler
at the top, systems, libraries, kernel and whatever you have underneath that for the
software is a self-hosting software stack and examples of that we have in the wild,
there's the Linux ecosystem, there's the BSD ecosystem, those are all sort of compliant
to this with this idea.
Now there's kind of a holy war going on with, you know, whether hardware will respect your
freedom or not and some people are claiming that, you know, hardware should be completely
immutable and never, you know, upgradeable with firmware or anything like that in order
for it to be completely, you know, respecting of your freedom and no binary blobs and different
people say, well, I mean, you may actually be able to, you know, put free firmware blobs
on your proprietary firmware blob-enabled hardware of today if you just reverse engineer it and
so on.
Anyway, the idea is in order to, you know, trust the computer, it's not enough to just
have a self-hosting software stack, we need to understand what hardware does and hardware,
as we've learned in recent years, isn't really hard at all, it's very, very mushy, very complicated.
It does all sorts of things that scare us and we need to kind of take a closer look
at it and so software talks to an instruction set architecture and a bunch of registers
that are mapped somehow and that's basically where software talks to the hardware, that's
the D-mark here and then there's all sorts of layers underneath, micro-architecture,
whatever, it all ends up with this register transfer level which is combinational and
sequential logic, basically a bunch of gates, a bunch of flip-flops, a clock and so on.
And it's not my word for it, it's just a word I picked up from the wild, I don't know
exactly who to attribute this to, but these layers of the hardware stack are typically
referred to as gateware and it's the stuff you write in something like Verilog or VHDL
or Mejen or Chisel and so on and then obviously all of this has to run on actual physical
hardware which could be dedicated circuits, application-specific integrated circuits or
optimized silicon or programmable FPGAs.
And so we can, if we have free software tool chains for HDL compilers, for making gateware
out of sources, which we do thanks to the group who put out YOSIS, Claire Wolf and the
gate cat who made the trellis and the next PNR place and route software, so anyway,
if we have those things, those are software that can be built by the self-hosting C compiler
center, which can compile the software stack, now this thing can take source code, HDL sources
and build all the layers of gateware which then support all the operation of the software
stack so you have a self-hosting software plus gateware stack, unfortunately, that
leaves for now out the actual physical layer, the silicon versus the FPGA, so this is as
far down the layers of abstraction we can go with self-hosting that I'm personally
currently aware of.
And so being a relative late comer to developing hardware, I'm a software person, have been
my entire career, took a couple classes at the university where I work, learned verilog,
learned a bit of digital design and it surprised me to realize that essentially designing gateware
is sitting down in front of a development environment and writing a program in some
kind of functional slash declarative syntax like verilog and VHDL, you basically write
a program and then hit the compile button and it compiles your code into ever more elaborate
basically graph net list of building blocks and eventually gates and then you have a choice
of building a binary blob which is bit stream for the FPGA and it's basically a binary blob
just like a binary blob comes out of an actual program you write for software, the difference
being software will tell some CPU a sequence of steps of what to do whereas bit stream
will tell an FPGA what to be, it sits there and it acts out the configuration that is
being compiled into a binary blob but other than that it kind of looks like software development
to me and I probably am passing off a bunch of people for saying that.
Now the interesting thing is if you don't want the FPGA bit stream but rather would
like optimized silicon then you're compiling, further elaborating your gates and your RTL
into a bunch of graph, a very complicated graph of transistors which then get laid out
and made into masks and there is an entire very very expensive very very involved process
of actually etching this into carving it into stone so to speak and we have the saying
of well is it the dog that wags the tail or is it the tail that wags the dog, well in
terms of that making actual silicon is one stage in a compilation pipeline you know like
a software development compilation pipeline just like a five megaton tail is wagging a
tiny little chihuahua dog basically but if you look at it from a software guys perspective
you know it's just one stage of the compilation pipeline just kind of figured out I'll share
that with you you know.
So now we have the option of doing a CP under this slide is specifically from the perspective
of we're going to make a CPU and the choices are putting a CPU in dedicated silicon versus
putting a CPU in an FPGA with the dedicated silicon obviously you have high performance,
lower area, you know high clock speeds.
The problem with that is there is you know from the perspective of the hardware attack
surface one thing we don't control is the foundry the chip foundry where we're sending
those masks to be made right and documented attacks that have been done so the University
of Michigan group had this A2 Trojanette and IEEE Secure and Privacy like three, four years
ago and what they did was if you have access to these masks then you can tell where things
are and you can add maybe these things have like billions of transistors but if you carefully
understand how this whole thing works you can put in 20 transistors in the capacitor
and the transistors are wired such that when the CPU because this is a CPU remember is
executing a sequence of unprivileged instructions depending on how you wired those transistors
in they incrementally charge the capacitor a little bit at a time until at the end of
the sequence the charge capacitor will flip a bit in the register and if that register
is your CPU privilege flag as in ring or whatever you know your kernel mode versus user mode
then you have a baked into silicon privilege escalation vulnerability that relies not at
all on any vulnerabilities in software so if you theoretically had perfect software you'd
still be able to basically do a buffer I mean not a buffer of a privilege escalation attack
on a CPU that's been compromised like that.
As opposed to FPGAs which you're asking the foundry the manufacturing facility to make
you a regular grid of basic configurable blocks it kind of looks like snap circuits
for grown-up engineers you know and most importantly the founder has absolutely no idea what this
FPGA will be used for and if it's ever going to be used for a CPU where on this regular
grid of identical blocks will the register be that holds the crown jewels to like the
privilege ring flags or anything like that so pre-gaming an attack in this scenario is
qualitatively harder for the hardware manufacturing facility because they don't know what you're
going to be using it for and where your things are going to be put on it by the place and
drought software so the price you pay for not letting them know where your privilege
flag is going to be by using software CPUs is basically performance a huge performance
loss but that's essentially the trade-off.
So if we've decided to use FPGAs because we're paranoid and we're trying to deny the
silicon foundry knowledge of what we're going to be doing the rest of the attack surface
is you know if we don't trust our HDL tool chain but we do because it's you know part
of the self-hosting stack and we have source code to it and then there could be design
defects like bugs in the soft in the sources to the to the CPU kind of like I don't know
spectra and meltdown and you'll never know whether those are intentional or just somebody
getting away with like trying to optimize things plausible deniability all the way down
but if you have source code to everything you can always just edit the source code and
rebuild things and you have a self-hosting environment which will allow you to rebuild
every part of it as necessary which is what brings me to this slide you know freedom and
independence from any sort of black box closed non-free dependencies and you can trust the
computer that runs as a self-hosting gateway plus software stack to the same extent you
can trust the cumulative set of source code now a lot of people are going to say well
no one ever reads that much source code and it's impossible to understand I agree I don't
want to read any of those sources myself but the cool thing about it is if I ever down
the road have a question about hey this computer did something weird I could do a vertical dive
into the software layers the RTL the source code to the gate where the source code to
the what whatever it does weird I can actually have enough brain power to do one sort of
debugging session through it but in order for me to be able to do that I need to have
source code to everything and with the knowledge that I'm not going to read most of it so that's
kind of my perspective on this my ability to trust my own computer I hope I'm doing
okay with time are we talking about 15 minutes here perfect all right so I am going to now
show you a Fedora capable computer built on this lambda concept board so if you download
the PDF from the conference site the links are clickable so it'll take you to the place
where I ordered it from it's a commercially available board hopefully they'll make more
because it was sold out the last time I checked it uses light ex and the rocket chip CPU it
uses joseph trellis and next pnr for the tool chain open sbi for the firmware and then I
downloaded the latest incarnation based on on rawhide 37 of fedora's risk 564 bit port
thank you david abdrakmanov he's the guy you know like the one-man show behind building
most of the stuff and it's really really appreciated if you have bit stream well if you have if
you have light ex and all its dependency installed and there's going to be a link in in the slide
back to more detailed build instructions for this but it's pretty much a stock light ex
build you install light ex according to all the recipes that are available online and then
you run this command line which says we're going to build it with the rocket chip can't
highlight with the rocket chip CPU 50 megahertz I want ethernet I want SD card support I want
to use flow 3 optimization to the osis component of the tool chain I want strict timing and
I want the register map saved to a CSV file now this is all a little bit clunky still
at this point because you're going to have to kind of manually build the device tree
table for it light ex doesn't build a device tree table for rocket chip based designs automatically
and it's one of the things on my to-do list teach you how to do that but one but once
you have once you have the the generic sort of register map and and you know where to
you know what the addresses are for all the devices we have to add like a chosen boot
args line which contains the kernel command line for the booting fedora and the black
font is sort of the the standard cotton paste from from what fedora already uses modulo
this route which is going to be on the SD card the other thing we need to do is set enforcing
to zero because once we have r-synced stuff from one image to the SD card the labels are
all wrong and you know se linux is going to scream at us so we're we're set enforcing
to to zero and then the default is to boot into graphical mode so we have to tell it
to use like run level three equivalent which is the multi-user target than system d and
then last but not least system d is really really impatient because it's used to running
on I don't know five gigahertz you know 20 core systems this thing's 50 megahertz so
system d will give up on starting services way before the thing actually has a chance
to actually start all that stuff so we need to increase the system d timeout now enforcing
we could get rid of like it takes about a day to relabel the whole SD card on the on
the 50 megahertz system but then you can get rid of this part of the command line because
it'll actually work properly with with the se linux and you can set to this as the default
so then you can get rid of the multi-user target but this should stay because it affects
both the init rd version of system d and the one that actually boots from the real route
now once we have a device tree blob ready to go we make a binary out of it and then
we build open sbi that's another sort of stock you get open sbi to just build itself using
a built in dtb right now so the other the other thing that light tax should eventually
be made to do is to build a dtb into the actual bit stream and then have open sbi just kind
of take that like it does for most normal computers for now we just kind of have to
build a binary bit stream thing into the open sbi blob and then we put that on the first
you know the first partition of the sd card has to be vfat and there has to be a boot.json
run file which lists the open sbi blob and its load address which is the very first address
in memory and then the linux kernel image and the init rd image and the linux kernel
image and the init rd image just come from fedora or would normally but I had to do
some customizations the stock fedora kernel has two problems that I'm dealing with right
now one of them is it lacks uart irq support for the the light tax uart and so that stuff
is kind of making its way right now it's somewhere in Greg Hage's tty next tree and
it's been accepted for upstream but hasn't made hasn't made it into mainline yet.
The other thing is between the port the risk five fedora port based on fedora 33 which
was the previous kind of major release of the of the risk five port of fedora and the
current one a bunch of config flags have been turned on additionally in the stock fedora
kernel configuration and I found two but I'm working on finding a third one which if enabled
will cause the kernel to crash when it boots on this machine on this computer and either
David will tell me well we can get rid of not actually enabling this one because it
was enabled by mistake or if it has to be enabled then I either have to find a percolating
patch for some kernel bug that already been found or I have to find it and submit a patch
myself but anyway that's kind of work in progress right now I'm building a custom kernel and
I'm doing that on a risk five fedora machine running on QMU for reasons of speed and you
know I need something to actually build the kernel before I can boot this machine for
the first time and then down here at the bottom there's a URL clickable link with all of this
but in much more detail that you can actually reproduce alright well perfect because now
I'm going to sit down and actually try to work this demo for you guys I recorded an
ASCII cast of my terminal let me try to maximize this
letting my screen and I'm sending the bitstream with open OCD so this is the ECP five bitstream
that I built I'm sending it to the ECP IX five lambda concept board this is litex you
know this is basically where I type SD card boot I'm going to try to zoom in so you can
so you can actually read the screen and see what it's doing so you know it's starting
to boot loading look loaded boot JSON is loading the RAM disk and if I fast forward it's going
to load the actual kernel image and then it's going to start booting and this is what it
looks like it takes a very long time this whole video if you have time to watch it at
normal speed this four hours long well if it's a 50 megahertz computer what do you want
so anyway let's see if I fast forward to this creatively you'll see system D actually booting
here and a bunch of okay services being started let's see at some point we failed to mount
what var lib nfs whatever but we don't care about nfs on this computer oh it also failed
to start firewall D but other than that it seems to be pretty happy and at some point
it starts the console and let me pause this again let's zoom in properly so you guys can
see what it looks like so this is a boot prompt for fedora in text mode if you don't have
irq support in your uart it would trip over itself get tty would basically just kind of
interrupt itself before it's serviced its own soft interrupt so it needs actual rq support
in order to not crash when it starts the other cool thing about this is it actually does
work on the network if I end map my you know from my normal machine for fedora it'll actually
find this so 192.168.2.229 on my home network was where this fedora machine actually grabbed
the dhcp lease and talked to dns mask and everything the my attempt to log into this
machine talk about 20 30 minutes because here's the cool thing right you type log in
and it starts you know the login program and then it starts bash right so in order for
all of that to work it needs to be loaded into ram and linked against glibc and all
that stuff and that takes a little longer than the timeout the first couple of times
until it's actually managed to pull enough of it into ram and actually let you log in
so there's a couple of attempts and I'm trying to log in both at the console in this window
and over ssh and see here I actually just succeeded because it says hey last login something
something that means I'm actually going to get a shell eventually and once I do get a
shell I can start you know exploring cat proc cpu info looks like this proc interrupts looks
like that I have the uart I have ethernet zero I have my sd card and this is part of
the cpu the slash boot dot boot.json this is the file that told the litex bios what
to load into memory what else do I have here this is the actual source I mean I just copied
it over to the sd card so this is the this is the source to the device tree file and
there's my boot args console all the stuff we talked about on the previous slides this
is the cpu node okay let's see what else is going on here my devices but all of this
I had to edit by hand and I'm going to I promise I'm going to teach litex I'm going to submit
a patch to make it build generate this programmatically so that I don't have to like modify the device
tree file every time I rebuild my my thing.
So long story short I'm going to fast forward over a lot of this stuff once I'm able to
log in from everywhere the next thing is system d network or resolve d is not enjoying itself
on this machine so I had to disable it and stop it and add 8888 and 8844 to an actual
hard coded at cresolve.conf at that point my my dns resolution started working crony
also started working because it could resolve like fedora ntp whatever the alias thing it
has in its config file and once I have all of this ready to go I type dnf-y install
python 3 media and osis trellis next bnr and it's doing it really slowly but if you have
patience or can fast forward which is really cool it's a cool feature of ASCII how do you
pronounce that ASCII N E M A ASCII cinema I don't know but you guys know what I'm talking
about right like you can record your anyway this is basically what I used here and so
I don't know we're like a hundred and forty two minutes into this entire thing and it's
installing rpms well we'll get to that we're we are going to address that elephant in this
room definitely well so here's here's the thing right so basically it takes about an
hour and plus to install all the rpms but then let me pause this thing at some point
what I did to demonstrate the fact that it can actually self host is I had a very simple
verilog blinky which just makes a counter out of the ECP five boards LEDs and and if
use if I zoom in here essentially this is what it does it has a counter and the red
the you know led zero red led one green led to blue and led three red that's basically
bits twenty seven twenty six twenty five and twenty four of the counter and it kind of
goes at like you know couple of seconds you can actually see it blink and that's the
verilog and I'm running actually here we go like the build of this here's the show I
have a shell script I am running it manually so your sis is the first thing that creates
a JSON file next PNR will do the place and route and then ECP pack will actually take
what next PNR produces and spit out an SDF file which can shove at the actual board on
which this computer currently runs and so I did that and so in this other window I have
top running and so your sis is using oh I don't know let's see if I keep going percent
CPU where is percent CPU right here so it uses about 80 percent of the CPU if I don't
know man DB whatever cron starts some process that drops down to 50 percent because now
it's splitting it with like whatever you know thing so I had to kill those while this was
running just to keep it you know on on task and making progress run your sis run next
PNR and this is pretty much if you've run next PNR ever before you'll recognize the
output it succeeds then we do ECP pack to generate the SVF file and once that is over
I did a MD5 sum of the top SVF file so that when we run the following demo or when I show
you pushing this thing to the actual board when it starts to blink here's the check sum
of the SVF file BAE0 yada yada yada 618 all right so at this point the job is done it
took I don't know 50 minutes to build the bit stream and if I pause this perfect if
I pause this creatively here I am doing an MD5 sub sum of the top SVF file and you'll
see BAE0 D yada yada 618 that is actually the thing I built on fedora running on this
board and if I let it run and it goes then you'll see right now it's like kind of okay
so here it started blinking and it blinks exactly like the verilog I showed you earlier
says it should so I was capable of building bit stream for this board on fedora running
on this board and with that we're going back to the tail end of the slide deck and we're
talking about right so building the blinky on my Intel laptop takes what 10 seconds or
less so 10 seconds dot dot dot 90 minutes building bit stream for actual risk 5 rocket
chip light text takes half an hour dot dot dot whatever that translates into you'd be
here a very very very long time if you waited for this thing to really self host itself
and rebuild its own bit stream it can do it we've we've established that it's the qualitative
leap has been done it's just a quantitative problem now to make this thing faster right
and so the immediate thing is to figure out the linux config stuff teach light x how to
be more you know civilized about booting and generating device trees and working maybe
with u-boot or something and actually have a standardized boot process light sata is
like they have a sata core which works on some FPGAs but currently not yet on the on
the ECP on the ECP five in the medium term right in order to make this thing a little
faster right on my VC 707 board I can get eight cores running at the hundred and a hundred
fifty megahertz so basically twice or three times as fast and eight times as many cores
as I can fit on a on the lattice chip a problem is if I do that it's not self hosting because
I need Vivado to pull that off I need the Xilinx proprietary tools so whatever I can
do to encourage or join or whatever in the future the effort to target the Xilinx large
Xilinx not just any Xilinx chips large Xilinx chips with complete free tool chains count
me in let me know tell me what I need to do you know I don't have a lot of money but I
have a lot of determination I'm a very stubborn individual all right well thank you I'll take
that as a compliment no for real I got that we could put in fancier IP blocks if it's
a larger FPGA maybe we can get away with some kind of video card like thing or maybe be
a PCI master so we can plug video cards or other cards into this computer and then in
the long science fiction term I mean what I'm doing right now is I'm taking a class or
a sequence of classes that that culminate in taping out an actual ASIC at Carnegie Mellon
which I'm doing in my spare time I want to understand how ASIC fabrication works because
I want to have something useful that I can say about it right now it's just all high
level oh yeah you know you can't trust the fab but I have no idea what goes on in one
of those things and I want to know what goes on in one of those things and then there's
been a kid who probably just graduated from their electrical computer engineering department
Sam Zilouf was his name but he was famous before he joined CMU because he did some silicon
transistor stuff on integrated circuits in his own garage probably with like 70s technology
or whatnot but it's a start right and then maybe in the future it would be really cool
if I lived long enough to see some kind of you know nano assembler kind of like a 3D printer
in my house that maybe costs as much or less than then sort of like a the average American
single family detached home you know or something because right now the way chip fabrication
works there like you can't you can count them on one hand how many actual places are that
make these things right and then obviously they have the attention of important people
and nation state actors and all this stuff it would be nice if we could democratize that
a little bit more so that's kind of you know if I live long enough to see that I've either
lived a very long life or you know something cool happened in my lifetime either way I
win and with that thank you.
Time for questions or do we do that off okay awesome.
There's a good talk just okay use the entire 40 bits sweet thank you.