I'm sorry for the wrong title, but you know, naming is hard, so I had to put something.
But today, I want to talk to you about my experience with dealing with out-of-tree
plug-in and tools for LLVM.
Parkoach is just one example, and I will give some others.
So first of all, I will try to explain to you why and for whom am I doing this talk.
So you know the audience.
I'm going to talk about three different things.
The first one is keeping up with LLVM.
So we will see some code.
She makes C++ and stuff.
The second point is usability, both from a developer, a tool developer point of view
and a user point of view.
And the final point will be dealing with packaging when you're actually targeting some system.
So why am I doing this talk?
First of all, it's to provide some feedback and maybe provide you with some stuff I wish
I knew beforehand before coming into this.
I'm doing this also because I've learned a couple of out-of-tree projects, and I've
faced the same issues.
So maybe you've faced them too, and it will be helpful.
So it's not so much about the tool parkoach itself, it's rather about the approach.
And for whom, it's basically anyone who is involved in an out-of-tree project for LLVM.
This is my own point of view on this topic.
If you have ideas, comments, like improvements that you think may be helpful to me, don't
hesitate.
I will welcome them.
So parkoach is a tool for HPC application.
It's basically an instrumentation, analysis and instrumentation tool for OpenMP and MPI
application.
It basically checks that the user is using the APIs appropriately, that there are no
deadlock or data racism.
The developers.
This is where it gets interesting, because they are not like LLVM engineers, right?
They are interns, students, PhD students, researchers.
They have a whole job, which is not LLVM.
The users of the tool, they are scientific application developers.
So you cannot ask them to compile LLVM from the source.
It's not going to work.
They are not going to use your tool.
And the last part, which is interesting with this project, it started a long time ago when
it was LLVM 3.7, and now it's based on LLVM 15.
So there has been a lot of history in the tool.
And I'm working on it right now.
It's my main job, so they have an LLVM engineer now, and I can do stuff.
I provided the link for reference if you want to take a look.
There are two other motivating projects that I can talk about.
One, I'm actually not going to talk about much, because it's not free.
It's a commercial compiler, which is based on LLVM, and basically the developers are
LLVM engineers, so we have more flexibility when doing developments.
And the users are clients who are paying for the compiler, so we need it to provide something
good.
And the other point is student LLVM exercises.
I do courses, LLVM courses for security students, and I want them to be able to do some code
transformation with LLVM.
So the developer is just a friend of mine and me, and the users are students.
We are expecting them to code into the project, so we need to make it easy for them to get
into.
And we have 16 hours to do this project, and they cannot spend two hours like installing
LLVM.
It's not going to work either.
So in all this project, I am considered pretty much the same issues, so I'm going to talk
about them now, and the first one is keeping up with LLVM.
So I'm not sure if it was intentional in the schedule, but having a talk with you about
CMake and stuff, it's quite helpful because I don't have to go too deep in the details.
You already know them.
So let's go back maybe eight years ago.
You wanted to do some LLVM tools, there was no CMake integration.
The first approach that you had as a developer was either do stuff manually, maybe using
LLVM config to get the flags and the libraries and so on.
But basically, you had no easy way to integrate with LLVM.
It was quite manual.
Then came CMake, and you could use the standard ad library, target link libraries, but you
had to know what to feed these macros with.
And some stuff I've encountered in this project is some kind of people who were developing
this project were not comfortable with CMake, and they would perform some changes where
they would actually do CMake integration, but with R-coded passes in the CMake, so it
would be awkward.
So basically now it's, I think, at least from the examples we have, it's way better.
So using the LLVM CMake integration simplifies a lot of stuff.
You just have basically to know which component of LLVM you want to use, how you want to build
your library, like is it static or shared basically.
And you have dedicated macros to just construct whatever stuff you want to construct for LLVM.
So let's take an example code.
So you don't have to understand everything, just to give an example of how it works.
You basically say, okay, I want to find LLVM, provide a version, sometimes, include the
LLVM CMake helper, and include some definition, and then this is the interesting part.
Because you can say, okay, I want these components in my tool.
Call the CMake helper with your plug-in source, and that's it.
I mean, the CMake helper will take care of saying, okay, depending on how LLVM is installed,
like is it just the big dialy, but are there like individual libraries?
It will set up the target link libraries appropriately, and you don't have to think about it.
It's just automatic.
If you want to like do some tools or pass plug-in, there are macros to do these two.
So basically, you just have to figure out which kind of build you want, and CMake LLVM
will configure everything for you.
There are some useful examples.
For pass plug-ins, there is the buy example, which is basically a new pass plug-in, very
simple.
It's a kind of a hello world.
And LLVM tutor has some out-of-three passes to get you started with, and it's actually
quite helpful if you are looking into this.
No, let's talk about some code.
So let's say you're new to LLVM, pretty new to C++, your student, for instance, and you
want to perform some LLVM transformation.
So you go on your search engine, and you look for how do I iterate over instructions of
LLVM function?
And pretty much all the resources like Stack Overflow or even some presentations, they will
give you the code on the left.
So it's fine.
It works.
I mean, you are iterating over all the iteration instruction of the function.
But if you know a bit better C++, you know that you can put range instead of row iterators.
And if you know the instruction iterators from LLVM, you know that you can use instruction
of F to just get all the instruction of F.
All the code works, but arguably the codes on the right are easier to read, and in the
end easier to maintain, especially if you consider that there are a lot of examples like this
in the code.
It adds up, and so simplifying stuff is nice sometimes.
So it's not a problem of Stack Overflow or anything.
It's just that the answer in Stack Overflow or in the slides are old, like from 2015.
Like if you would update the answer, it would just be the option on the right.
Another thing I want to talk about, and that I've seen a lot in ParCoach, is iterating
over something, but putting a predicate.
Like I want to iterate through this stuff, but only if this stuff is true for some predicate.
So you can do stuff like that, early, continuous, or nested if.
But if you know the STL extra from LLVM, you know that you can create filtered range for
any range, actually.
So you pass a range, you pass a predicate, and inside the loop, you just get the object
you're looking for.
Again, it's a simple predicate, so it doesn't matter much as is, but if you add some more
stuff, it starts like growing up, and maintenance became a bit harder, readability is impacted
too.
So this is something to consider.
Now, something more like critical is advanced data types.
There are a lot of data types in LLVM, and if you are not familiar with LLVM, and I've
seen a lot of code like this, you will just use whatever data types is available in the
STL, and you will get a map, for instance, use some helper.
And the actual issue starts when input, the map of instruction, you want to map an instruction
to something.
If you go through the input and change an instruction, like if you delete it, or if
you replace all the uses with some other value, what happens to the instruction in the map?
So with raw map from the STL, like there is no mechanism, so nothing happens, and you
end up iterating or trying to find something which is not valid anymore, whereas if you
are aware of the data types from LLVM, you are able to use some kind of value map which
has specific handle to remove the value or update the value if it is changed during the
life of the value.
So some other helper that are quite nice, I mean, it's not a big deal, but for instance,
instead of using std-finif, you can use LLVM-finif and just put a range, instead of just like
the individual iterator, in this case, it's not a big deal, but it's actually quite nice.
But basically, every stuff like that, I've encountered this for a lot of kind of code
where you would be able to replace most of the occurrences with any vector from the LLVM
array advanced data types, or like a array or string array, like there are a lot of stuff
in LLVM that you may not be aware of and that makes your code quite nicer if you use them.
So yeah, dealing with it, so you may think, okay, this guy is just being picky with people
who are writing the code.
It may be true.
I will argue that it depends on actually who makes the contribution, because you cannot
expect the same level of contribution from a student or from a LLVM engineer, and like
especially when you're a PhD student, you have, I don't know, a deadline, you just want
a tool who does something, like you're not going to spend times and times on how you
do stuff as long as it works, at least that's my experience dealing with that.
But in my opinion, the accumulation of small details matters, and it was very explicit in
the case of Barcoach, because I came after like maybe five, six years where the accumulation
of researchers and PhD students like led to a lot of technical depths, and if there was
some advice that were given to the PhD students or the researcher, it would have been a way
nicer code to read or to maintain.
And so obviously, it's quite obvious, but doing code reviews helps a lot.
Sometimes you cannot do them if there is no one able to actually provide some useful feedback
on this, like in the case when people don't know LLVM, you cannot expect them to review
code and provide some, a lot of feedback.
But what I do know is I redirect every time to the LLVM programmers manual, it's not like
the first thing you usually just go to a search engine and search for what you want.
But I will argue that actually reading the program manual is more helpful in that, in
this specific case.
And something that I know people don't want to do when they are starting LLVM is just
read the code from the passes in LLVM, there are a lot of good stuff in there.
Obviously, if you're not familiar with C++ and LLVM, it's not the easiest, but I think
it's still worth it.
So the next topic is updating the LLVM versions.
So far, when I've developed out of three tools, I've always set the version to one
specific number, right?
And like let's say LLVM 9.
And then when LLVM 10 comes out, you rebase your plug-in and check if any API broke, if
there was like some changes in the IR.
Most recently, I am thinking about Opak pointers, it was quite a big change when updating the
LLVM version.
And something to consider when doing this is that it may be time-consuming, like a lot
of time can be spent in, it may be like just a day if there were no changes in the API,
but it could also be very time-consuming for instance if you have to change all your passes
because it's been three years that the new pass manager was out and you still didn't
do the migration and now suddenly it's deprecating and it's going to be removed, so you need
to migrate your passes, so you have to do it.
And in my experience, it's quite obvious too, but skipping versions makes it worse.
And some things that I've seen, and I know sometimes it cannot be avoided, but in that
case it was avoidable, but basically trying to support multiple LLVM versions at once,
like say support from LLVM 9 through 12, it's actually what was done, and yeah, don't
do it.
Just pick a version and stay like this, because otherwise it's just multiple if-deaf and everyone
in the code and it's unmaintainable, I think.
So now let's talk about passes.
If you look for a hello world pass on the internet, you will get a hello world pass,
which is a transformation pass.
So in LLVM, you have two kind of passes, the first kind is analysis, and basically they
don't touch the IR, you just look at the IR and maybe provide some result, which is
a result of the analysis and that can be used by transformation passes or other analysis.
And there are the transformation passes, which may or may not change the IR.
And obviously, when you get your hello world pass, you want to do everything in it.
Like, I mean, I'm not talking about LLVM developers, I'm talking about students and
researchers that have the pass and they put everything in it.
And so it's fine when it's just like one shot or something like that, but in the time,
at some point, both the analysis and the transformation are semantically different,
and LLVM has some mechanism to make it easy for you to have the analysis run only when
it's needed, right?
There is a caching mechanism, you can say, okay, I want this analysis for this object
and if it exists, it will give it back to you.
And also, it avoids passing structure around because when you are in a transformation pass,
you can request any analysis from basically anywhere as long as you have the analysis manager.
And so this is something that has costed me quite some time, like just untangling the
analysis code from the transformation code, and overall, it improved the performances
because some analyses were requested more than once for the same object.
And yes, it leads me to investigating performance issues because it was something too.
So what happens when you don't know LLVM and you want to debug your code?
You put LLVM errors everywhere and you command them out when your code is ready, okay?
So it's a nightmare, I mean, it works, but you're not supposed to do it like this.
So specifically for like printf like debug stuff, you have some LLVM helper, it's actually
quite handy.
You just put a debug type somewhere and you dot CPP, you wrap everything in LLVM debug
because it does all the things for you if you don't include debug information, it doesn't
even appear in the binary.
And when you're running your pass without, you can say, okay, I want to show debug information
for this kind of pass and it basically provides the same feature and you don't have to command
out LLVM errors.
The other thing is timing your code, being able to tell, okay, this part of the transformation
is costing me time.
And so what I've seen was some manual attempt to do timers and basically declare all the
timers, you start them manually and it starts being a mess really quick.
Hopefully, thankfully, now we have a time trace scope.
It was, I think it's what's used when you use a F time trace when starting clung.
And so basically it's just one line, you put one variable and when it's constricted, it
starts a scope and it starts a timer and when it's destructed, it stops the timer.
And LLVM has a whole system for this and it emits a JSON and you get, if you put this
in this JSON speed scope, you get something like that.
And you can see basically everything in your code without having to do anything.
You get the entry point, you get the analysis and here it was quite obvious for us was the
changes, what the changes were because this analysis, for instance, was called multiple
times but it was for the same object.
So like for instance, it would appear here too but because of the caching mechanism and
the untangling, it basically just, it was just called once.
So this is something nice that you get basically for free.
So now let's, okay, some conclusion on the tool development, so it's a fairly basic conclusion.
Try to invest in maintenance.
I know it's not always possible, especially in a scientific project but it's worth it.
Don't remember the wheel.
If you want to do something in LLVM, it likely has already something in LLVM for this.
And keep the this minimal.
One of the main weakness of Parkour right now is that we use some passes which exist
already in LLVM.
I'm thinking about memory SSA for instance, we use some copies of this and from a maintenance
point of view, it's not quite nice so we need to migrate this away.
And if your passes can be useful to others, just try to upstream them, I mean if you don't
use them, you don't have to pay for them.
Then let's talk a bit about usability because it's quite a big deal for a tool because you
want it to be usable.
So first, from a developer point of view, if your developers are going to be non-LLVM
folks, you don't want them to go into the LLVM install and stuff so I've had good experience
with using Docker and basically provide a Docker image with the LLVM compiled installed somewhere
or just installed using the APT repositories and have some clear CI, like how to build
your tool, like just looking at the CI should be enough to know how to build your tool from
a developer point of view.
And the other great thing is when you use LLVM, you get LLVM tools with it.
So you get a lead and five check and so instead of going through some manual testing and stuff,
you can just use them and it's actually quite nice.
And yes, of course, I could talk about coding standards but basically since you're making
a plugin or a tool for LLVM, it makes sense to follow the same standard and you have already
clonk format and clonk tidy configuration for this.
Now, as a user, you obviously don't want a scientific application developer to compile
your code from source, you want them to just have the plugin and use it or have the tool
and use it.
If you look at Hello World passes, you see a lot of times that you have to first get the
IR.
So in our case, it's either from clonk or from clonk and you have to call out, load the
path manually and call the path manually.
So I would argue this is not nice enough for researchers and students and since Spark
Coach is a verification tool, we cannot expect users to call it on every single file.
So we actually had to do some more tooling to create some wrapper which takes the original
compiler invocation, runs the original compiler invocation, generates temporary IR and then
runs the tool over it.
It makes it much more easy for the users to just integrate with auto tools or CMake.
So that makes the tool more user friendly than I would say unusual.
And the other part is how do you get the tool?
So again, I've had good experience with Docker especially for students because it's easy
for them.
And sometimes obviously we also provide some package for major distributions but you actually
have to worry about how is LLVM packaged on the target system because depending on what
is available, how is it shared libraries, Dalib and stuff, it's not the same thing.
And yeah, Docker, it's not something you can quite use on shared HPC clusters, you're more
looking at stuff like geeks for instance when targeting such platforms.
So for this, you need some packaging.
And packaging is my last point.
So obviously we used to use do-it-yourself approach, basically just create a shared library
and hope for the best, it doesn't work.
Because you depend on how opt is installed and compiled because you're loading dynamically
a library into opt.
So if you have not used the same like C++ libraries, you're going to run into issues.
You don't know for sure which pass manager is enabled by default in opt.
So there is also this.
So we've moved to doing some proper packages for APT.deb and for geeks and for Red Hat 2
because we have some users using some custom version of Red Hat.
And for this, we actually have quite an interesting issue because we are sure that the LL version
in their image is not available.
So we made the choice of shipping just one single static tool.
And for this, it was actually quite easy because as I said, when I talked about CMake, you
just say, OK, I want this to be linked statically or as a shared library and CMake, LLVM CMake
handles it for you.
And it was quite a nice experience for us to package for so many distribution without
having to worry too much about CMake option and stuff.
So some takeaways for the whole talk.
In my opinion, the LLVM integration has evolved a lot and in a good direction.
It's way easier to integrate with LLVM now than it was 10 years ago.
It's nice, but it's nice to say it because when nice stuff happens, you have to say it
too.
Be prepared for maintenance.
If you want to create a not-off-tree tool, you have to invest in maintenance both for
LLVM rebases and basically reviews and make sure that your contributors, if you are able
to provide some LLVM guidance to your contributors, do it and it's worth it.
Investing in CI is worth it, obviously.
And LLVM documentation, I would definitely, every day, recommend going to the LLVM documentation
rather than Google for understanding what is available in LLVM.
And I want to encourage my students to read LLVM source code, but it's sometimes a bit
hard.
So if you have questions or comments, feel free and I will be happy to answer them.
Yeah.
So the question is for the wrapper we created, what do we use to create this wrapper, right?
So basically, it's a very, very small LLVM tool, maybe you are familiar with not in LLVM.
There is a very small utility in LLVM which just does not on the return of a program.
And it's a very small LLVM tool based on LLVM.
And we use a similar approach.
Basically we say, okay, I created basically an empty main where I just use the LLVM support
library to get the benefit from like argument parsing and the data types and so on.
And I just parse the command line and call successively clang the original compiler line.
And then I just generate the intermediate representation for it by adding the appropriate
flag and filtering out the other object generation flags.
And then I just run the tool over it.
Yes, yes, because you can just, for instance, with CMake, you can use the CMake C launcher
basically just like Ccache work for LLVM, you just change the launcher and you can use
the tool to launch the compiler.
And for other tools, you can actually, actually in our project we use MPICC but we are able
to change if compiler used for MPICC and say, okay, use instead of GCC for instance.
So the question is when you ship your tool, do you link statically or dynamically?
So actually both.
When shipping for Red Hat because we don't have a control over what package are in their
custom image, we ship statically because we are not sure about which LLVM we are going
to have.
So we just, the binary is 100 megabyte but we don't have much choice.
And when shipping for system like Ubuntu or Debian, we just ship the dependence on the
shared libraries.
So the question is, when we're basing the tool from one LL version to the next one,
do you use the changelog developers put their love into and if yes, is it helpful?
Unfortunately the answer is no.
But that's because I look at the LLVM weeklies so I kind of know what happens.
This is just my way of doing stuff, yeah, so no, but if I would look into the changelog,
I would find helpful information I'm sure.
So the question is, am I trying to rebase as LLVM progresses or am I just rebasing every
version when it's released and it's only when a release came out, comes out, I do the rebase.
It's easier because otherwise, you know, depending on what kind of target you ship for, it's
hard and it's just simpler to say, okay, we know and then, we know we need to rebase
the version and it's fine.
I think we're out of time now, thank you Philippe.