Next up, we have Peter Smith, who will talk about using all the empty-created LLVM tool
chains.
Hello, my name's Peter, and thank you all very much for staying up so late. It's almost
bedtime. So I'll be here to talk about embedded tool chains using LLVM. So the first thing
I want to clarify is what do I actually mean by an embedded tool chain? Now, some of the
people here earlier on were talking about, as I say, sanitizers, and we mentioned Yachto
and embedded Linux. That's way too high-level. This is basically for bare-metal embedded
systems. So, yeah, so typical, for those of you who already know this, I'm sorry, but
for those of you who aren't necessarily familiar about what some of the differences are. So
typically when you're developing on, say, Linux or Mac or Windows, whatever, you're
developing with the knowledge of an operating system. So when you implement your C-Library,
you already know you can use system calls. You know, if you want to get some more memory,
you ask the operating system, that type of thing. So by contrast on the embedded system,
you don't have an operating system you can ask for memory. So you basically have to roll
part of that into the C-Library, that type of thing. So also, when you're actually programming,
you're programming on the device, you're actually running the program on, embedded systems,
you're cross-compiling. That is one thing that is likely shared with Yachto and embedded
Linux because quite often you're cross-compiled for speed on that one there. Typically, you're
be static linking only because either your RTOS probably doesn't have a dynamic link
at that particular point and your RTOS might actually just be a library that you link into
your program, that type of thing. So yeah, so platform, if you're on Linux, you might
be using G-Lib C, that type of thing and that will be platform and then you just use, when
you have a tool chain, you might just need to provide a compiler and everything's there
for you. Embedded systems, everything's just, you have to do everything all yourself. I
will mention one word there, freestanding. So there is a definition in the C-plus standard
of what freestanding means. It's a little loose. It kind of says this is basically what the
minimum you have to supply, but that's practically useless unless you want to write a full C-plus
plus standard implementation yourself. So in effect, what happens is that most embedded
C-Libraries tend to roll half of an operating system into themselves, at least basically,
yeah, basically minimum from there. So that's what we're sort of talking about by an embedded
tool chain. Okay. So this is the thing we already have embedded tool chains. Why do we
need LLVM essentially at this particular point? So this is some of the reasons why you might
actually want to use LLVM over say something like GCC. So first of all, clang is kind of
a natural cross compiler. So you don't actually have to say, gather GCC for ARM, GCC for S5,
GCC for AL64, you just have one clang. Now that is quite useful if you're a team where
you don't want to use different compilers, different installations, I guess more administrative
more than anything, but it can be a benefit on some places. So code generation can also
be more mature and I will say be safe of fairness, sometimes less mature than GCC, for example.
So my, obviously for somebody who works for ARM, all my examples are for ARM just because
that's what I know, but I'm sure there are similar sort of things on their architectures
as well. So an example here, V8.1M, which is one of ARM's most recent sort of CPUs for
embedded systems, it's got a vector extension and basically clang has got better support
for auto vectorization for this than GCC, just simply because the work was done earlier,
that type of thing. But that's just one of the examples why if you've got that particular
target, you might want to use that, whereas if you've got a different target, GCC might
be better at the moment. Other thing is taking advantage of some of the tooling that clang
provides. So I'm going to go into in the next few slides how you might be able to use some
of the sanitizers. I know we kind of said in the earlier bit this morning that we were
talking particularly about MSAN and ASAN, that type of thing, and those typically have
quite a high runtime component, but there are sanitizers that you can use without that,
and I'll just go through a few of those here. And finally, you've got diversity of implementation
running more compilers, it's almost always good. Compilers find different sets of bugs,
and sometimes programs find different sets of compilers. Sorry?
I was working recently on a safety critical application for train, and you actually have
to implement several processes doing different things, so having two different compilers
is a good thing in that application. Yes, definitely, yes, and certainly different
programs find different compiler bugs as well, that sort of thing. So yeah, okay. So do you
think sort of sanitize the embedded system? So we kind of run through some of this earlier
on today. So the main restriction for sanitizers is that it's not actually the code generation,
it's actually the run times. So if you look at the run time for ASAN, it's basically using
a dynamic shared object to intercept the C library. It's got all sorts of bits that
sort of kind of are operating system dependent, but of course in embedded you don't have an
operating system, so it's very hard as a toolchain vendor to provide a kind of bare metal
thing that doesn't depend on one very specific example. But some of the sanitizers have a
very minimal run time, and some of these things you can use here. So I'm just going to go
through some of these right now. So the first one to use is the undefined behavior sanitizer.
So by default that does have a run time, but all that run time effectively doing is pretty
printing a nice error. But if you don't care about pretty printing a nice error, you might
not even have a printer. So at this particular case, then you can just say, okay, well, if
there's undefined behavior in my program and someone's trying to attack me, maybe that's
a bad thing. So maybe I just want to abort, say for example, if I've got an out of range
run time. This particular example is just using a very standard integer overflow detection.
And basically look on there, all it's really doing is just saying, check for overflow. If
I overflow branch to an undefined instruction that just happens to cause an abort on the
processor, that type of thing. So yes, crush your program. There's also a minimal run time.
So there is a default implementation of the minimal run time in compiler RT. You can't
use that directly on an embedded system, but you can basically write your own. So instead
of actually calling, well, going branching to an undefined instruction, it just calls
a user defined function. And you can basically make that do whatever you want. There are
ones for log and continue, and there's ones for log and terminate, that type of thing.
But basically the choice is yours. But those functions have got extremely trivial implementations
that you can make work from an embedded system. Okay. Next one here is the kernel control
flow integrity. And it's called KFCI. And I keep calling it KFC. I've even got this
to write right around it. Actually, I think I've even got it wrong on the slide, which
is embarrassing. I should actually be KCFI at that particular point. So there is a control
flow sanitizer that can work with embedded systems right now. That's the sort of the
full fat, I call it sanitizer. But that requires link time optimization. So the advantage of
the kernel control flow integrity sanitizers is it doesn't need LTO, which makes, if anyone
to try to use LTO on embedded systems, it works until you've got a linker script. Certainly
what a linker script that depends on placing things in different places. So yeah, so here's
just a very trivial example of something that's just calling a floating point. And this just
shows some of the code that's generated. So what we essentially have is this function
pointer has a type. And you can basically make that into a signature. So what happens
is we prefix the top of the function with the signature. And then we basically load
when we're sort of saying, oh, let's load from this arbitrary function pointer. Well,
let's check its signature. And then we'll check to see if it matches what we want. And
if it doesn't, boom. So this doesn't, as far as I know, work on C++ V tables at the moment.
As obviously this is implemented for the Linux kernel. So they don't care about C++. So,
but if you're using C with function pointers, this is a way, a relatively low cost way to
get control flow integrity checking. Okay. So this is just some of the things that the
components of an embedded tool chain. I'm kind of jumping around here at the moment.
So these are sort of the things you would expect in a GCC embedded tool chain. And as
you can see, Clang's actually, well, LLVM project, we've got pretty much all that we
need in one place. We're only really missing a C library at the moment. So yes, we can
go through some of the, I won't go through each individual thing in those titles, but
you've got, you know, Clang, the compiler, you've got LLV, the linker, you've got implementations
of obstump, read-elf, you've got implementations of the C++ runtime library. Yes, say, what
we're missing is a C library. So technically, GCC doesn't have a C library either, but there
are hooks in the build system to basically build new lib in sort of multi-lib configurations
at that point. LLVM is developing a C library. I would say at the moment that currently it's
sort of focused on what you would probably call desktop use cases, but they are planning
to have sort of scalable implementations. So I think the end goal is that it will be
able to cope with embedded systems, but I expect that to be some, some years down the
line at the moment. Okay. So how would you actually assemble one of these building? Well,
basically assemble an LLVM toolchain from the LLVM project. And the honest answer is
it's not as easy as it could be. Certainly when you're building a sort of a hosted toolchain,
it's just, you know, it's fairly easy. You just go to LLVM, Cmate, Ninja, done. So actually
building the tools is not difficult because they're all cross-compilers. They're just
all part of the default build. So if you want to get all the tools, very, very simple. Building
the run times is a bit more difficult because you've got to cross-compile the run times.
And you've got to do them in a particular order, not all of them build in all of the
things. So one of the big problems, if you say try and buy compiler, sorry, if you try
and compile compiler RT, it'll fail because you've not got all of the, you know, it's
kind of, if you try, it'll end up building the sanitizers. And the sanitizers obviously
have got dependencies on POSIX operating systems, which of course won't work. But you
can say, for example, build the built-ins, which are kind of like the LibGCC equivalent.
So what we've done at ARM is to put together an embedded toolchain for Cortex-M, which is
the sort of ARM's microcontroller range. And this is essentially a set of build scripts.
It's all open source. And we're using the Pico Lib-C at the moment as our C library.
And we did start with New Lib, but we sort of moved on to Pico Lib at that point. But
you can make it work with New Lib if you want to. So yeah, so we've got, it's primarily
just build scripts. It's not like got an LLVN project embedded on that. It will just go fetch
LLVN from the actual source code. And yes, it's got a few samples for, you know, for
building some programs, that type of thing. So as I say, it's by ARM, for ARM. But I'm
sure if anybody wanted to apply it to a different microprocessor, they pretty much could because
it's essentially just a bit of C make and that you can adapt.
So what's the usability of an LLVN toolchain like next to say the GNU embedded toolchain,
that type of thing. So one of the main things we're missing at the moment is multi-lib support.
Now there are some multi-lib support for certain targets. So for example, I think there are
some RISC-5 multi-libs that are already in the bare metal driver. But that's not the
case for ARM at the moment. I'll go on to what we're doing about that in a few slides
time. Clang also doesn't have a direct equivalent of GCC specs files. So specs files are basically
just fragments of command line, but they're not just raw command lines. They have got
some intelligence and they can talk to each other and override sort of defaults. So as
an example here, that nano.specs and RDImon.specs, that says give me newlib nano, which is the
really small version of newlib. And RDImon is the semi-hosted version, which is easier
to run on emulators, that type of thing. So for the LLVN embedded toolchain, we basically,
because we don't have the information for the specs file to say, ah, someone else has
someone added this other specs file, so I'm going to modify my behaviors, we have to basically
have multiple config files that just blow up for all of the possible combinations. So
as you see there, we've got an ARMv6m, which ideally would be handled by multi-lib in the
DMD semi-host version. And yeah, there's just more configuration files than you really
ought to have. And I would say there's probably a small, well, there's a long tail of small
incompatibilities. You might find that LLD doesn't do orphan placement exactly the same
way as GNU-LD does that type of thing. But normally these sort of small incompatibilities,
you can kind of code around it. There's normally a way you can make it work. So that's what
we've found so far anyway. Okay. So this is just, again, another jumping around, just
showing you how Clang might do some of this sort of stuff. So if any of you have played
around with Clang drivers, whenever you give them the target triple. So normally if you're
using Clang on your Linux, your target triple is, you know, native, I guess at this particular
point. Or you're using the default triple that's there. But if you're doing cross compilation,
you have to give it a sort of architecture environment. So you've got the Linux-GNU there.
So this is actually one, if you were targeting something like the Octo, that type of thing.
And that will Clang driver will then tell you where all of your header files are, what
your target features are. So it's like a much low level using sort of private command line
options at that particular one. So for what we find for embedded systems is that Clang
has added something, well, probably a few years ago, but it's sort of only recently
sort of getting a bit more development onto it. In particular, the multi-lib support for
RISC-5 came in fairly recently. And that's when you have a target that the bare metal
handles. So far, that's only ARMA ART64 and RISC-5 at the moment. In theory, it could
be added for any other target, that type of thing. If you happen to be doing bare metal
development on an X86 and you don't match, say, a Linux operating system or BSD or whatever,
you end up getting forwarded to the generic GCC driver, which basically throws everything
which generally knows what to do about things. So as long as you've got a GCC tool chain,
if you give an object file to GCC, GCC will say, oh, I'll just fire that at the linker,
that type of thing. So it will work itself out. Okay. So I've just basically repeated
what I've just said there. It will default to the LLVM tools at that particular point.
So as the last part of the talk, I just want to go to some of the ongoing work that's happening
in Clang and some of the community involvement that's going on here. So one of the first
and probably the major bit of work that we're doing at the moment is what I'm going to call
data-driven multi-lib at the moment. So currently, multi-lib support in Clang is hard-coded.
It's basically a C++ class where you basically describe what the multi-lib will do for that.
Now, that works pretty well if you're doing things like 32 or 64-bit x86 in, say, Debian
or Red Hat because the structures are well known at that particular point and they are
stable. Whereas there's no way every possible embedded tool chain with every possible library
variant that you might want to do could get that hard-coded in upstream Clang. So typically,
what you find is that every tool chain based on LLVM has its own downstream patch if it
wants to support multi-lib. So GCC allows you to set this up at configure time and the
GCC way basically maps command line options onto directories. So for Clang, we can do
a bit better because the Clang driver has a bit more scope to say, do things like target
parser and find out more about what the CPU can do. So at the moment, we're kind of proposing
that you kind of have a stacked tower of multi-libs where you can kind of almost like a Docker
container file where you get each sort of can override the next so that you can basically
describe what your multi-lib configuration is and then Clang will be able to take this
configuration file. So it will basically allow people to have multi-lib tool chains without
having to hard code them in downstream patches, that type of thing. So this is still in active
development. There's an RFC that went up probably a few weeks ago. Recently, there's some links
to the patches and that sort of thing. So please do, if you're interested in data-driven multi-lib
and how it develops, please do comment on those patches and the RFC.
So future work. So we'd ideally like to get some upstream build bots for some of the compiler
RT runtimes. So whilst there are build bots for ART64 and ARM Linux, we haven't got build
bots for, say, the built-ins for, say, the V6M, V7M, the sort of the very low-level embedded
sort of targets. And we think that would be good to, you know, well, obviously more build
bots the better, I think, at that particular point. There is some work going on at TI.
So that link to YouTube is to a presentation at the last LLVM developer meeting, basically
adding attributes from the linker script so that you can basically say things like, this
section must go in this place, this output section, this one must go in this other one.
Please do not cross-module inline across these boundaries because these things might not
be in memory at the same time, that type of thing. And also, I need this section to have
this particular name, so please don't give it a different name or merge it, that type
of thing. So that should be able to make LTO much more usable with linker scripts. And
what we tend to find with Clang is that if you get it right, LTO is very aggressive at
removing code that's not needed. So that's actually very good for code size if you can
make it work. Certainly we've seen, you know, for benchmarks, LTO is great, but then we
say to customers, hey, use LTO and it goes, ah, but we can't because of the linker script,
that type of thing. Next one is not strictly embedded, but it is very important for the
safety critical industry, which often is, you know, by definition embedded because you're
controlling some kind of hardware. And this is something called MCDC code coverage. And
that is kind of a special form of code coverage where you're kind of, if you can imagine something
like if, and then A, B, C, D, E, and E, it's a way of sort of deriving test cases so that,
so it's not quite exhaustive, but it covers more than just did this branch go this way
or this way. It's like, did it go this way because this condition held that type of thing.
Hopefully that's not going to show up too much there. And yeah, so there's some, there's
a patch in for generating that in the code coverage thing. And it obviously, a lot of
VM libc developing. And we would like that to support embedded systems. Okay, I'll skip
through this very quickly. There's some patches up for Big Endian support. If anyone actually
uses Big Endian, I don't know. I'll be rude there. There's an armed person. We're almost
our arms a little Indian. And then there's the Cortex-M security extensions, which are,
you know, that's very useful if you're trying to sort of have secure state, non-secure state.
So that supports in LLD. Again, if anyone wants to comment on those patches, please
do. Okay. Okay, so fine. Finally, if you do want to contribute to this, and this is not
just as a developer, we're perfectly happy to have contributions from users as well or
just in some ways just telling us what's important. So Clang has pretty much come out of what
I call the hosted community. You know, it's generally, at least as now I would say there's
a lot fewer people in the embedded system, embedded systems area than there is on GCC.
So if you, you know, there are certain features that are useful in embedded tool chains, but
not necessarily in say, hosted tool chains. So just telling the community that you need
these features is often helpful because quite often it will say, why do we need all this
complexity for this thing? No one's going to use it. And it's like, well, and you can
only get people only get used to features if they're there, but then you can't get them
in, you know, chicken and egg situation there. So yeah, so there is a four weekly call that
goes on, unfortunately, at a time slot that's not great for Europeans, but this is the only
sort of time slot you can kind of get across US and Europe together at that particular
point. So that's probably about, I'd say about 20 people turn up. And that's really just
about the various people who are working on embedded systems and if they want to sort
of highlight patches that want to be reviewed, discuss new features. Last time we were talking
about how we might improve LLDs, observability of diagnostics, that type of thing. Obviously
bug reports, welcome at those links. And obviously if you attend the developer meetings, there's
often a round table on embedded systems at that point. And with that, that's my last
slide. So hopefully we've got a few minutes for questions.
I'm trying to understand something. I know of some people who say that they're using
LLVM for embedded already. Does this mean that they're using the other definition of
embedded?
So there's two, well, you can do it. There'll be three ways they can do it. One of them is
they're kind of using an LLVM based tool chain from a vendor. So that vendor will have done
all of that packaging up. Or it will be like, for example, ARM will sell you a commercial
tool chain that is a derivative of Clang, that type of thing. That's one way of doing
it, that sort of thing. Or they might be using embedded Linux, that type of thing. The question
was, sorry, I've been holding up a picture all day saying, please repeat the question.
I didn't. And the question was, some people say they're already using LLVM. Does that
mean they were using a hosted system or not? Okay. Just the first, sorry, go out the back
there.
Yes. So one of the things I noticed is LLVM ships its own assembler. We've noticed for
some LLVM projects that they have trouble with some of the new assembler macros. So we have
to go back to the truth and just put some of the targets. Is there some plans for work
on this?
So with the latest LLVM, I know, sorry, sorry, repeat the question. So the question was,
the LLVM has an integrated assembler. GNU has GNU AS. And there are some directives or macro
support that might be in the GNU assembler, but not LLVM. So I think it's generally done
on demand. So there was a big effort to get the Linux kernel compiled with Clang. And that
added quite a lot of features that were basically needed for the Linux kernel. So the best thing
to do is have a really important project that's a big company wants to get compiled with the
integrated assembler. Yes. Yes. Yes. Yes. That is a very good way of doing it.
As macros were in the Linux kernel and they asked us to support them and say, no, screw
this, the kernel changed away from macros. And they changed away from macros.
Yeah. But there certainly was support. There was, I think there is a directive where you
can switch the GNU assembler into advanced macro mode or something like that. Or I can't
remember. No, that's not that. That's the inline assembly thing. There is an F. Yes,
there is a high GNU extensions option. But no, there was a patch that probably landed
a few years ago. So depending on how long ago you tried, then there was some support
done for more macros. But whether it's got all of it or not, I don't know.
Well, I actually don't. I've got a lot of problems. It was a while ago.
Right. Yes. You may find that someone has already fixed that already. Yes. Thank you.
Yeah. So my question is about, like, we, for example, tried to deploy machine learning
models on tiny bare metal devices. Yeah. And there we are also looking into, for example,
TVM as a tool chain, but also MLIR now, or like the EV project from Google.
Yeah. And they're basically what they do is they use this entire tool chain, and then
they use the MITC dialect, for example, in EV to MITC code again. Okay. To then put it
into an embedded tool chain to actually do the final compilation stuff. Do you think
that there is, or what is basically needed to omit this last going back to C, or is this
a good idea or not? Or... Oh, well, I mean, I suppose... I'm just trying to think how,
not very familiar. So the question was about people deploying machine learning models on
small devices, and they're currently outputting to a C back end, and then recompiling that
C back end. And do I think this is a good idea or not? I mean, I guess the C back ends
are often the, how do I get this up and running as quickly as possible? I do know that there
are, I guess, machine learning compilers that have got, I guess, code generation out. I
mean, I guess if you're using LLVM itself, it's probably not too difficult to just lower
to LLVM and get most, and you then get the code generation for free. I guess the bit
that you might not get is, have you got all of the run time and intrinsics that you might
have that the C compiler might insert, but I don't, you might find someone else knows
why. So maybe just in addition, it does not compile the whole machine learning models
to C, so they are still a static library linked in which it's generated via LLVM. It's just
some parts around, so it's not that they are as pure C in the end. That's not done in
the approach. Okay. Yes, sir, yes. I was one of those unfortunate uses of big ambient
arm. We were running several compilers in this safety-critical application. Everyone
had a problem with one thing, and that's the linker. We're trying to generate a header.
And the linker, you can't insert a text string in the linker. It's very difficult to insert
static data. We want to insert information about how large is the section. That was basically
possible to do with the linker. How does LLVM handle all these other things?
With great difficulty, I think, I think there isn't really a, there isn't, I think the,
yeah, I think Christoph is nailed it in that, what I would probably do myself is reserve
some space in the binary, name a section out of it, and then use Obstump to poke it in
at that particular point. Can you see my problem?
No, yes. I mean, there are some...
Putting a string in the link command file.
Well, it's a bit more difficult. It's the, I think the linker needs to know the length.
I mean, I suppose you could do it with horrifying things. You could use data statements in the
linker script, but that sounds a bit, it's really what you would want.
I want it.
Yes, really, I think, yeah, because you get a number, but really, yeah, I suppose, yeah,
find the extension request of a linker script.
I mean, specify the size of the section to start with, then I know the size.
Yeah, I think we do have the problem, we do have a problem with things like build ID,
I think at that particular point where you're generating the build ID string, which needs
to know everything all at once, but yeah, I get, yeah. Unfortunately, there's nothing
in the LLD linker that's different.
If someone should try to generate the header for a binary, contain the interest in information,
then you quickly realize all the problems.
Oh, yep, sure.
The new assembler has the ink bin, doesn't it?
Yes, the assembler does have ink bin, but I think the idea is, for the header, you want
the linker to generate something based on the properties of something, but...
You want to generate the information about the link time, not when you assembled it two
weeks ago.
You can assemble, just before the link is done, assemble something that's generated.
Even.ovmile, and then link that in using the link script.
I know there are workarounds, but a good workaround would have to have a good link.
Yeah, I mean, I think, I mean, a lot of the times with linkers, it's the, because one
of the, one of the perennial things you could ask is, how do I embed some kind of custom
checksum that I've written at link time, you know, that type of thing?
And it's just which one, and do you then have a linker Python script extension or plug-in?
It just tends to build.
Yeah.
Check some afterwards, and that's also something that should be supported in linker.
Yeah.
And it would be done, just say, run this application afterwards on this section, something like
that.
Yeah.
So it's...
Do you do a partial link, and then analyze the object file, then do a final link?
Yeah.
I mean, I guess...
More workarounds.
Yeah.
I mean, to paraphrase, I guess, is the tools are supposed to make users' life easier, I
suppose, at that particular point.
So if it's a common enough thing to do, then it should be, they should be able to find
a way of doing it.
And if this is security, then you don't want to generate the checksum in one process, and
then use it in the other process.
You want to use all in one, because I know who generated this, and it didn't come from
outside.
We actually have to have two different programs, and is that the case?
Yeah.
So, I wait, so I better go for...
Yeah, so I just have a follow-up to that discussion, so I'm not an LFM developer, but I deal with
a lot of built-ins.
Does anyone working on stuff to make things a bit better, this sort of thing?
Because essentially, like, you're talking about communicating between the file stage
and the link stage, and introducing dependencies to what you have, and that you have to link
with after that.
Right.
And now, like, it seems like you need, like, a schema, a data format, and a dependency specification
for that, so the built systems could actually use it, and spare the users who have to deal
with that.
Nice.
I mean, I think that the big...
I say it's mostly a, what I would call, almost a coordination problem between getting the
right people on board at that particular point, and it's...
So it's quite...
Can you repeat the question?
Sorry.
Yes.
Okay.
So the question was about, is anybody working on build systems and things that they will
be able to communicate the...
Of the linker to be able to communicate to the build system and automate things like the
checksome sort of handling and that type of thing.
I mean, I think the major difficulty is just an LVM is an open source project, and there's
often, as soon as you open something like that up, it ends up in lots and lots of discussions
about what the right way, and you can easily find a way that works for one, a small number
of people, but completely doesn't work for someone else, so it's one of those...
It first of all needs someone brave enough to actually try it rather than just implementing
it downstream.
So I think it's...
What it really needs in this case is, because this is sort of things that...
This is not...
It really needs people to go on the LVM mailing list and say, yes, we really need this, because
typically this sort of thing is to silent people who say, oh, this stuff's all rubbish,
but we don't...
As developers, we don't get to hear about it, or at least we don't get to hear it loud
enough for the people who pay our wages to say go and work on it.
Yeah.
Right, yeah.
Okay, I probably ought to hold it there to let everyone go, I think, at that point.
Thank you very much for staying and I'll see you all next time.