So good evening, everyone, and welcome to the last talk in this session.
I hope you still have some energy left.
So my name is Sascha Roloff.
I'm working at the Huawei Intelligent Cloud Technology Lab of the Huawei Munich Research
Center, and today we are going to take a look under the hood of build systems and what common
practices are currently used in basically all of the build systems and why many of
them are suboptimal in certain regards and how they can be improved by a concept called
staging.
So in order to explain you the issues with current build systems, I directly jump into
an example and I guess many of you have used make once or twice in your open source developments.
So let's start with this classic build system.
So we want to create a build description for a very simple hello world application composed
of a hello binary and a greet library, and the greeting phrase is hard coded inside the
hello binary and a greet e can be injected at compile time at the greet library.
So this is a make file.
We have our rules which describe which artifacts are generated by actions based on a set of
input artifacts, and so we have different actions actually involved to generate the
final binary, and for example we have compile actions to generate the object files like
the hello.o or the greet.o, we have archive action for the greet library, and the final
linking action to actually generate the binary.
At the end we also want to create some sample output, so we just take the output of the
hello world and store it in a text file.
So nothing spectacular right now, each artifact is associated with a file on the file system
so the actions can directly operate on it.
If we execute the build, we just see all the actions are executed, everything fine,
and the output is generated, and yes now the boss comes into our office and he's unhappy
with our result, he wants to put it basically on a poster and it should be more readable.
Okay so yeah then let's add some post processing to the task, and we just take the output of
the hello binary, store it in the intermediate file and put this intermediate file into the
post processing and translate all letters into, capitalize all letters basically and
store it into post processed text.
And then finally put this text into the target sample output, and we execute this, we see
new actions are executed and the result is fine, looks much better now, hello world and
capital letters great, but the boss is still unhappy, he wants to have some localization,
he doesn't want to greet the whole world, he just wants to greet Munich and Brussels,
and he wants to have it both in a single make file, so what do we have to do now, okay we
have to basically we have two program variants now, and what should we do in order to reuse
most of our rules that we already have, we can use a for loop over the location dependent
targets and interpolate the city name into the artifact names as you can see here, so
we have now not only a single hello binary, but two hello binaries with dot and the name
of the city, and these are our two program variants, and as you can see there is a lot
of string interpolation coming into our make file and it doesn't make it really readable,
but we have to do it, because each artifact is associated with a file on the file system
and this needs to be a unique name, so we have to do it basically, and when we execute
it, okay now we get a bit more actions, but it's working, and we see now, okay the output
is as required and we greet Munich and Brussels, but the boss is still, I mean he's happy now
with our output, but now he's unhappy with our implementation, he says that's not maintainable,
why do we use a build system from the 70s, use a modern one, they are supposed to do
much better now, well okay, then let's use Bazel, and this is what it looks like in Bazel,
so the same application, and as it turns out they are better, but not in all regards, so
they introduce high level concepts like the CC binary and CC library, we don't have to
manually write object file creation and linking, but it's, everything is wrapped now inside
these high level concept calls, and also our bash calls are wrapped in these general targets,
but I mean it looks a bit more readable now, but still we have this string interpolation
here, and the for loops over the city names, and yeah why is it actually like that, why
do we need this, I mean it's a modern build system, and the reason is because Bazel also
associates each artifact with a file on the file system, and yeah, so that's why this
basically brings us to an important observation, and this means even modern build systems,
it's required that you have unique names for your artifacts, and because they basically
follow a design decision implemented by make in the 70s, and the design decision by make
was that each artifact needs to have a fixed location in the file system, well for make
it was perfectly fine at that time, because there was nothing else or not much different
to do in order to determine which part of a program needs to be recomputed, basically
to compare timestamps, and for this you need files, so for make this was totally fine,
but there is actually no reason anymore to do this in modern build systems, because they
anyway isolate their actions in order to avoid getting unwanted dependencies into their builds,
so their actions are executed either in a separate directory or in a container in order
to better control the dependencies, so when they anyway execute their actions, why don't
we allow the targets to specify where to put the artifacts, and this is exactly the idea
of staging, so basically there is no technical reason for modern build systems for restriction
of to associate each artifact with a file, and instead we propose that we should stop
following this common practice and apply staging instead, and the idea of staging is that an
action can freely select the location of input and output artifacts within its working directory,
and this basically introduces a separation between physical and logical paths, inside
an action, you only work on the logical paths, and the action can freely decide where to
put a generated artifact, or where it wants to read an input artifact, and so this is
basically our proposal to apply staging, and how could it be look like if it's implemented
in a build system, so this is basically our project, it's called just build, and this
is a build description that we propose, so we also have the definitions of our targets
here, we also use the high level concepts like binaries and libraries, and in this JSON
syntax the type just selects which kind of artifact or which kind of target it basically
is, and what we can see inside the target definitions our artifacts are named without
string interpolation, so we don't need to artificially invent unique names for our artifacts,
they are just like they are, and also for example here this use target we just access
the hello binary, even though we will have two of these binaries but we just write hello,
and we don't care, I mean it's staged, and what is created from the depending dependency,
it's just staging the final result at that location where we need it, so, but still we
have the for loop, this is something what we of course still need to, which basically
creates two configurations, which is then propagated, I mean this variable that is created
here is propagated to all the depending targets, and it propagates until the greed library,
which then reads this configuration variable and injects it into the compile command, so
this is how a description could look like with staging, and from this description we
can also generate a so called target graph, which shows the dependencies of the targets,
so main depends on all, all depends on two post process because we have two configured
targets, so the greed library basically is duplicated and this propagates until the post
process target, and these target graph or targets are basically high level concepts,
if you want to take a look into which actions are actually executed, you can also generate
an action graph, which shows a data flow, that's why the errors are inverted, and it's
a bipartite graph, which means so, the ellipses are the artifacts and the rectangles are
the actions, and yeah, so you can really see the artifact names are basically the same,
so post process dot txt and post process dot txt are the same names in both branches,
and since they are staged in logical paths, there's no problem, there's no conflict actually,
this would not work in make, you would have to use unique names, okay, so and what happens
when we execute it, so we just select the target that we want to build, and there is
some output coming here, and it says okay we have 12 actions, zero cache hits, of course
we execute, built at first time, so you can count it's 12 actions, and it's just built,
the artifact is somewhere, I mean it could be in a remote execution, and then it's just
existing in a remote cus, if you want to have the artifact in your local folder, then you
have to install it, and when we execute the installation, we now see okay, again 12 actions,
and also 12 cache hits, because everything is known already, and then the file is in
your local directory actually, and we see the content is fine, and we even don't need
to store it into our local directory, we can just print the content of an artifact by the
minus p option, if we take staging seriously, we have also very nice implications, and one
is for example, assume that you have an external source code that you want to use in your project,
and you want to apply some patches on it, and yeah how do you do it, normally you would
copy it, apply the patch, because you don't want to modify the original source code, and
yeah this results in a lot of maintenance problems, but with staging this can be done
much easier with logical in place patching, you just apply the patch on the logical path,
and yeah, then let's take a look how this could look like, so we have now put our example
files in a third party directory outside of our project, and a directory with the patches,
and the patch just modifies the hello greeting phrase with a bonjour greeting phrase, and
we just have to add a single block, a single block into our build description, which points
to our patch and to the file that needs to be patched, and that's it, and the resulting
target graph just shows, okay we have now one more target here, the hellocpp source
target, and the other, the binaries are depending on this extra target now, and also in the
action graph you can see that there is just a single new action actually added here to
the action graph, where earlier was hellocpp is now the patched version of hellocpp, and
it's just another input, and if something is changed in the patch all dependent targets
are executed, okay if we execute it we see bonjour Munich, bonjour Brussels, works well,
okay so quickly to summarize my talk, as we have seen there are some inconvenient habits
in modern build systems, and yeah we propose to apply staging instead to make build systems
better, and you will have a couple of advantages if you apply staging, and yeah which are written
here, and this is not only a concept it's already implemented, so if you want to take
a look into our project please come by, and yeah now the stage is yours, thanks for your
attention, are there are any questions, there is a question, no I think it will repeat the
question, you heard the question, yeah exactly we do actually content addressable storage,
so repeat the question please, okay the question was how do we identify which source code we
actually need for the staging, and yeah we apply content addressable storage, so we
determine a hash basically from all of our source codes, and then also we know what has
been changed or not, any other questions, yeah so the question was whether we
use the jason syntax, yeah no we decided for jason, and yeah it's jason is used as our
build description syntax, okay so how many developers are working on it, and is it widely
used, a very good question actually we recently got open sourced, and we are in total five
developers currently working on it, and yeah but we really try to implement the new concepts
into this build system, and make it a really sound build system compared to other modern
build systems, and yeah so please just take a look at our project, and there is a nice
tutorial also, which well everything explains nicely, and hope to see you there, okay thank
you for the talk, thank you, goodbye, thank you for coming.