So next speaker is Todd Gamblin and as I think a lot of people here know I'm very much involved
in the EasyBuild project which was actually the excuse we used to start the HPC Dev Room
but we're also very open to other projects which are very similar to what we work on
every day so in some sense back as our mortal enemy but we do allow them to give talks in
the Dev Room as well. Yeah, with that, thanks. Okay, who's heard of SPAC? Okay, cool. People
have heard of SPAC. We don't need to do too many introductions for this talk. This is
less of a talk about SPAC and more of a talk about the CI that we've started doing since
introducing binary packages in SPAC. I don't think I need to tell people why they need
SPAC for HPC. I think lots of folks have talked about that already today. Harman said
that, so I'm supposed to talk about a little bit about deployment. To deploy SPAC, if you
want to try it on a new system, just clone it from the Git repo and run it. Like all
you need is Python and a few other tools on your system to do that. So you can just run
it straight out of the repo if you want to play around with it and build stuff right
there. SPAC is designed to install lots of different versions of things like others
have said. This is sort of a snapshot of the syntax. Some of the things that you can add,
you can install HDF5 at lots of different versions, you can inject flags in the build,
you can pick a compiler, you can do all that on a fly and it will build you sort of a custom
version of that software and let you use it. You can get it into your environment a lot
of different ways. What we're trying to do with SPAC is provide the ease of use of mainstream
tools that people are used to, but with the flexibility for HPC, whether we fully accomplish
that is a whole other question because there's a lot of complexity still in this because
it is intended for HPC. Originally it was designed to build from source because it
was trying to automate people's common workflow. The Fermi lab and CERN folks added a first
implementation of binary packaging to SPAC and I talked about some of that in a past
FOSDM. Since then we've actually started relying on the build caches a lot more. SPAC has relocatable
build caches that you can build in either a build farm or you can make one right out
of your SPAC build. You may not want to do one yourself that way because then you won't
have padding on the path. Like you said, the patch off relocation is dangerous. Generally
if we build binaries for wide use, we pad the paths pretty extensively so that we can just
poke values in instead of having to do all the patch off stuff. Anyway, you can install
SPAC binaries from a build cache in S3 to your home directory. You can make a common
build cache in the file system. You can use a build cache to accelerate CI. It's very
handy because it eliminates the need to rebuild lots of stuff all the time. If you look at
the SPAC project as a whole, I think people know most of this. There's a community. We
maintain the core tool. There's package recipes but the part that you don't see is all the
infrastructure behind the scenes that keeps the thing working. Originally, we did not
have CI for SPAC or at least not for the package builds. We've always had CI for the tool itself
and we've done unit tests and checked a bunch of things about concretization and so on,
but we weren't building all the packages. We're still not building all the packages,
but we're building quite a few of them. With the infrastructure that we have, we have a
system where essentially you can build lots of software stacks on top of SPAC. You can
write an ML description of what you want in the software stack. You can have E4S, AWS
stack, Livermore's, Math stack. There's a Viz SDK within the Exascale Computing Project.
Every application is its own software stack these days. Our production codes have upwards
of 100 dependencies that they are using for multi-physics. Each of them is essentially
maintaining their own little private software distribution in some sense or another. We'd
like to be able to build all of this stuff and ensure that these things keep working.
That's hard to do given that the GitHub for SPAC is a pretty busy place. There's almost
7,000 packages in SPAC now. Over the whole life of the project, there's been over 1,100
contributors. You can see down there, this month, there's been 122 people active on the
GitHub repo. Over 400 commits and 300 to 500 PRs per month that we have to merge. Ensuring
that everything stays working without many changes is pretty hard and you'd be nuts
to do it without CI. One of the problems that we have, though, is that CI for HPC is hard.
If you want to test in the HPC environments that you actually care about, you can't just
take an HPC node and hook it up to random pull requests on GitHub. They don't like that
when the machine might have export-controlled software on it because you're effectively
allowing some random person in a pull request to run software on your HPC machine. This
is the model for SPAC. We have a bunch of external contributors on GitHub constantly
contributing to this develop branch. We have stable release branches where we freeze the
packages to reduce the churn that some people rely on. Most users are actually on develop,
at least according to our surveys, which is a little surprising to me, but that's where
we are.
Then off of the release branches, there is a software distribution within the XSCL project
in the U.S. called E4S. There's a few others that sort of freeze a commit from SPAC and
do their own integration after that. That's really supposed to be the deployment mechanism
for the 100 or so packages, and it's like 600 with dependencies, that are in ECP. What
happens with this is that that gets deployed to the facilities, and the E4S team goes and
ensures that everything works, but we're not able to run on these systems on pull requests
in CI, and it's very frustrating. Essentially, this is a bunch of downstream work that we
would really like to get rid of. Moreover, the applications are also doing downstream
integration. They may have their own CI, which may be good. They're essentially pulling
from all these places. They may pull a facility deployment. They may pull from develop. They
may pull from a release. They're essentially integrating from all these places, and so
there's a lot of downstream porting there.
What we would ultimately like to do in SPAC is take all of that work that's going on downstream
and move it upstream so that we're actually doing CI testing on develop along with everything
else. This is progress towards that, but we're not doing that yet. Essentially, the main
obstacle for us to build stuff that looks like the HPC environments right now is a
licensing issue, which is that we can't take the CrayPE container and run it in the cloud,
because that's just not something you can do with HPC's license. We are pushing them
real hard on this and trying to get an exception for us to build things, in which case we would
be able to do work upstream and, ideally, deploy at the facilities from the binary cache,
which I think would be way more stable and less error-prone than what we do right now.
We set out to make this CI system to enable this with a bunch of different goals. One
of the goals is that we want to be sustainable. We don't want to change the maintainer workflow,
and we already have few enough maintainers for the amount of work that there is that
we don't want to change what they have to do. They're used to going out on GitHub and
approving PRs, getting them merged, checking if they build and so on. We don't want them
to have to do something different, so we don't want them to both have to maintain PRs and
think about how the integration branch is doing, like some distros do. We'd like that
to just happen. In that vein, we want a rolling release where, on develop, we're constantly
building binaries for the develop branch, and that we basically snapshot develop for
every release that we do and say, okay, it's stable. Everything built. We are ready to
do the release. We'll just cut one, and then we will backport bug fixes to this back tool
on that release if we need to. We want it to eventually support all 6,900 packages.
It's not something we're doing now, and we want source builds to still work with these
binaries effectively once it's done. We want to make sure that the recipes are still versatile
enough to do all those combinations of builds that I showed on the first slide.
Then finally, and this is a big one, we wanted to ensure that the binaries that we have in
SPAC are just as trustworthy as the sources. If you feel like you can trust our maintainers
and rely on the sources that are in SPAC packages with checksums, that you feel just as comfortable
with the binaries that we're putting in the build cache for you.
If you think about how this works, if you look at traditional package managers, like
say APT or YUM, you have a recipe per package configuration that's getting thrown into a
build farm. For each of those package configurations, think like easy configs if easy build had
binaries. Throw that into a build farm and then you get these portable unoptimized binaries
for theoretical binary having easy build, where there's one of those per package configuration
or per spec file or whatever. This is more like an APT or whatever. You're managing one
software stack that's meant to be upgraded over time, and there's a consistent ABI across
the distribution so that you can swap one package in for another. The solver in those
distributions really operates on the binaries. In SPAC, you have parameterized package recipes
that we are designing to be portable, and we want the maintainers to work on them together
so that they remain portable, so you can use them in different environments. Throw those
into the build farm and effectively test the parameterized package recipe in lots of different
configurations and spit out different stacks. Lots of different stacks optimized for different
environments from the same portable recipes for different systems, OSs, compilers and
MPIs and so on. We also want, at any time, for you to be able to choose to build something
from source along with that if you want to customize some aspect of the pipeline. To
enable that, we came up with this architecture. We have a bunch of AWS resources, because
AWS has been nice enough to donate some cycles to the project. They are interested in using
SPAC in their parallel cluster product, and so that's the motivation for them is they
want binaries ready to go if someone spins up a cluster in the cloud. They don't want
to spin up a cluster and have it sit there and build software for hours and then run
after having charged a bunch of money, which is nice to them, because they would make a
lot of money. But then no one would use their service. They want binaries. In there, we
use S3 and CloudFront to distribute the binaries around the world. EC2 is really the main build
resource, and RDS is in there, but it's not that important. We've got a Kubernetes cluster
in there that we have autoscaling runners in, and so we're building mostly in containers
inside of Kube, and there's a GitLab instance in there, too. We have a high-availability
GitLab instance. We chose GitLab because the HPC centers actually have GitLab CI themselves.
The same CI logic that we run in the cloud, you could take that and run it internally
and have these pipelines generated for you at your own site, too. You could slap another
back end on this and have it generate build grass for some other system, but that's the
one that we're using. We're using runner pools with something called Carpenter to basically
get just-in-time instances and allocate the containers on them efficiently, and then we
have some bare-metal runners at the University of Oregon with some fairly exotic architectures
on them. So if we need to build or if we need to specifically run on something that has
an AMD GPU or A64FX and so on, we can do that. And we could add more runners to this eventually.
And there's a bot that coordinates all this work. So it's a lot of stuff. Every time I
look at this, I am amazed at how complicated CI is and how it's one of those things that
seems like it should just work, but there is a lot to maintaining a reliable service
for doing this many builds. And I suspect other distro maintainers have realized that,
too, and I'm just late to the game.
The way that contributing a stack in SPAC works is we have this directory in the repo
that has all of the cloud pipelines in it. And so you can see some of them are for AWS.
Some of them are different variations on E4S. Each of those directories contains just a
SPAC.yml that defines the stuff that is to be built. And so if you look inside of there,
it's basically just, it's a list of packages. So here's the ML CUDA one that has the build
of, I think, PyTorch and TensorFlow, Keras, Jax and friends for CUDA. It's just a list
of packages plus there's a target up there, a target setting for all the packages. You
could have a matrix of targets if you wanted. And then there's disable, rock them and enable
CUDA on everything except for LLVM because there's that bug that's linked there. And
I'm not entirely sure about the specifics of that. But the configuration part is up here
and it's fairly minimal for the stack. There's currently, if you look at these, a bunch of
other boilerplate stuff for things like mapping runners. I'll get to that in a minute. But
this is, there's a PR that's going to go in where this is basically all that's going
to be in your stack. And you might include some stuff from elsewhere. But this is essentially
a stack definition. And we take that and, you know, this makes it very easy to change
low-level parameters in the stack. So we had a working E4S stack with something like
6 or 700 packages building. We wanted to get better testing for one API because that's
what they're going to use on Aurora. And so we wanted to use the one API compilers.
We added some compiler config and we said everything should use one API. And then, you know, at
the very least we got a pipeline generated with some errors for one API. And it made
it really easy to iterate on this with Intel where we would basically say, okay, this package
is broken. Here's the bug. Go fix it. And then it would come back with another version
of one API and we would iterate with them until it was done. I think this is probably
more open-source than anyone has recently run through a vendor compiler. And so just being
able to do this, I think, is big because it might make those compilers like actually viable
things to use for real programs that have lots of dependencies. At the moment, you
know, you have to sort of piece your program together and build parts of it with like,
I don't know, PGI was the infamous one that broke on everything. But, you know, I think
this could help with the vendor compiler being a viable second option and, you know, maybe
instill some competition among the vendors because they can do this frequently and show,
you know, benchmarks against these packages. So this was, I think, a win. Yeah, thank you.
Each of those stacks gets concretized. And so people know, in SPAC, you take that abstract
description of the things that you want to install, which is basically the requirements.
You run it through our dependency solver. You get, essentially, a concrete description
of what you're going to build, which is the whole concrete graph. And then we generate
a GitLab CI YAML from that that describes the jobs that need to be run to build the
whole thing. This is the part that we could swap out for something else. So, like, we've
looked at, like, Tecton pipelines. We've looked at other options. I don't know, some
people use Jenkins. There's all sorts of things out there that you could potentially map the
jobs to. And I think we could generate a description like that from the representation
that we have. For mapping those jobs, we have a section in the CI YAML right now, or in
the SPAC.YAML, that basically tells you how to generate the GitLab piece. And so you
see this mapping section here. There's a match section. If you match any of those specs there,
and the first three are just a couple, just some names, then we have special tags that
we put on the runners that say, you know, get me a special resource for these things.
And so that first block is basically so that I don't run out of memory building LLVM TensorFlow
or Torch, get me something with a lot of memory and a big CPU to build that one. It has to
run on a big instance, because those are sort of the long poles in our tent in CI. And then
down at the bottom, there's just a mapping from everything else gets something that supports
X8664, V4, and it's a little smaller than the other one for builds. And you could do
this for lots of different architecture combinations and so on. And you can ask for images and
things like that. I said that we needed to ensure that the source is, or that the binaries
are as reliable as the source. And so we sat down and we asked ourselves, you know, what
is it that people trust about the SPAC project? And it's really the maintainers. If you use
any open source project, you're trusting the maintainers, or you really shouldn't be using
that open source project. And so I don't see where we can do better than that. And so what
we've done is we've said the place where bad things could get into a build, at least from
SPAC, is in the build environment. And so if you give people control of the PR environment
where they're submitting things there, they could push a commit that puts something in
a binary that gets cached. And then, you know, somehow, I don't know, they could do bad things
and end up caching a binary. And if we took that binary and stuck it out there for anyone
to use, you know, there could be bad things in it. And so we have this separate set of
untrusted S3 buckets where we only build PR things. Each PR gets its own build cache.
That enables the maintainers to see if things work. And then they come along and review
the code. And then once things are actually merged to develop, we don't trust any of the
binaries that we built on PRs. And we go and rebuild everything in sign, specifically
from the, you know, the sources that got approved, just, you know, so that we know that we didn't
cache anything from that environment. So that's where the development and release caches are
coming from, where they're entirely separate from the PR environment. And the signature
here is, you know, it's ephemeral. They have, like, a signing key locked up somewhere in
a secret server. And we generate, you know, we have subkeys and then we generate ephemeral
keys for the signing in the pipelines. So whatever it is that you got signed with doesn't
actually exist anymore by the time the user consumes the binary. We could look at sick
store for this. It wasn't quite ready for arbitrary binary signing when we did this.
But that's an option to reduce some of the custom GPG stuff we had to do here. So the
pull request integration, I think, makes it easy for at least for most of the contributors.
They get status updates on PRs. And it's fairly easy for users because they can just add one
of these binary mirrors and then start using the build cache. And I'm not going to get
into the details here, but in SPAC, for a very long time, it was easy to get a lot of
cache misses, like we would just look up hashes. And I have another presentation about our
reusing Concretizer. The summary is, if you add one of these build caches and you have
those binaries available, SPAC will prefer to use them. And so before it tries to rebuild
something. And so with the reusing Concretizer, this is actually quite powerful. And so, yeah,
what could go wrong? Well, there is a burden to doing this. And a build cache distribution
like SPACnix or Geeks is different from an RPM distribution because every node has a
hash. And the deployment model is really that you have to deploy with what you built
with. And so you can't just swap in a new version of Zlib in a stack. If something has
a particular hash, that implies all of its dependencies' hashes. And so you need to
deploy the build cache with everything that it was built with. So if, for example, you
modify XZ, right? Yep. And then you're going to need to rebuild all of these things, too.
And you're going to need to do that all the way up to the roots of your environment every
once in a while so that there's a consistent build cache for people to deploy. And that
can be bad if your stack is this big. This is E4S, right? And someone comes in and submits
a PR, which you can do, by the way, that, you know, modifies package comp. And then
all of a sudden, you know, this is what happens to your CI system, right? Your whole graph
is rebuilding again. And it can take a long time for develop to catch up with a change
like this. And right now we are rebuilding all that stuff on PRs. So your pipelines can
get long. You dig in there and you see that, like, visit is still building. And you're
like, this is the fifth time I've built visit today. I think Harman once commented that
he was worried that SPAC would eventually cause the heat death of the universe because
of pair of view builds. Or no, the pair of view builds would eventually bring on climate
change in the U.S. So we worry about that. We don't want to do that all the time.
The other thing that can happen is there's a delicate balance between redundant builds
and, you know, holding back PRs. I didn't think about this before we really got into
CI. But it matters what commit you pick to merge with when you're doing a build-cache
build. And so if you have a pipeline like this where you've built B and develop has
now picked up on D and that one's building up there. And you get a PR like this. So PR1
comes in. You can merge that with B and get a lot of reuse there and get a pretty good
testing on PR1. If instead you get a PR up here that is based beyond your last developed
build and you try to merge that with D or even C, I guess it's already based on C, so
you can't really merge with C. But if you merge that with D, you're going to be duplicating
the work that's already being done on develop. And so if you get a bunch of PRs like this
at the same time, you can get a whole bunch of builds at the same time that are effectively
already being done on develop. And so this is a difficulty of navigating these PR-based
CI systems. If you had a server that had shared that one patch was built all the time once,
then you could get around this. So you have to be picky about this, hold up PR2 until
the next thing is built and then merge with that commit and send it to GitLab to be merged
or to be built. And this can annoy contributors because they have to wait for that to happen
for their PR in order to keep the CI system sane.
We actually did bring down GitLab once with a bunch of PRs like this. Essentially something
got broken in develop, develop, got held up, people started submitting a bunch of PRs,
they were all doing redundant builds and GitLab fell over. So that was fun.
CI does keep things stable. And so we have had, at least anecdotally, that our package
maintainers at the lab are much more happy with how reliable their builds are for packages
on the machines since we've had CI. But like I said, the committers get frustrated. And
the other thing that happens here if you're doing so many builds on PRs is that if your
CI system has occasional system errors, if you're building a thousand things on a PR
pipeline, it's very likely that you're going to get a system error on there.
And so what ends up happening is that you end up having to babysit PRs a bit. And that
can be painful. The other thing that happens is it's hard to stay correct. So testing on
PRs doesn't really ensure that you have a working develop branch. If you have a setup
like this with an initial package state, you get a pull request at update B. You get another
pull request in there that updates C. You test both of those configurations on your PRs
and they work. And you merge them. The thing that you now have in develop is actually updated
B and updated C. And you never tested that. And so keeping that state consistent is rather
difficult. And we're thinking we're going to, we didn't, you know, before we had CI,
I think we just didn't see these kinds of issues. They would just get manifested on
users, which is not great. But now we run into them in CI because we can see that things
are broken undeveloped. So we're looking into using merge queues, which actually solved
this problem and a couple others that we have pretty effectively. So you can do faster
iteration on PRs with merge queues because you're merging in sequence, testing in parallel.
I'll describe what that looks like in a minute. It's a good balance of CI versus responsiveness
because you can do sort of sparse tests on the PRs and queue them and then do the heavy
tests. And it actually does preserve the security model because anything queued in a merge
queue is actually approved by containers, by maintainers, and you can take the builds
and move them straight into develop. And so what that looks like is this, where you might
have the same initial packet state, you get two pull requests, you do some small testing
on the pull request, and then you set up this merge queue where effectively you're doing
heavy testing on things that are basically staged exactly as they will be merged if they
are successful. Okay. So that gets committed, that gets committed, and now you've tested
the final configuration on develop and you're not in an inconsistent state. So we're going
to stage the work that we do in CI. On PRs, we're probably going to build just the package
or just the package and its dependence, which is similar to what Nix does. On most merge
queue pipelines, we may build a bit more than that, and then every once in a while we'll
build everything on develop, and we'll see how it goes. We can probe, you know, what
the balance is here. So that's where we're at. Thanks.
Okay, I think we have time for one or two questions. Any questions for Todd?
And off the wall question, we have, for example, software bill of materials, dev room. You
mentioned export controlled software and also being able to trust binaries. I work with
classified customers who have isolated networks, probably Shopify, MI6, if I told you who
they were. But could SPAC help with providing, they're now asking for what software is running
on these systems. I mean, what does that question mean, really? Can you help with producing
a report on exactly what software is? Yeah, we have a PR right now for so that
every SPAC build would produce an SBOM in some standard format. There's a whole dev
room on SBOMs today, which gets into that. And so I think, yeah, I mean, we know everything
in the graph, and so do Nixon Geeks and the other systems that do this. We don't expose
it in a standard format that auditing systems can scan right now, but that's what we'd like
to do. So very briefly, Debbie and I, a while ago,
did something on reproducible builds, which were much more difficult. So if you haven't
worked with her a bit, that might be interesting for you.
Yeah, so we would like to have fully reproducible builds. It's a lot of upstream patching, right?
And even Debian isn't fully reproducible right now. I think that that would be like something
we could consider after we get down to, like, libc even, because at the moment, because
we have to run on things like craze where there's so much dependent on, like, the module
environment, we have to include the external environment to get some of these builds done.
But yeah, I would like to have a much more isolated build environment with that. It's
a good practice. Okay, one more question here, and then need
to switch. Hi, so you were talking about padding your
header files for rallies of pathing. Given that you don't have a static path or a pre-defined
destination as in FHS-type locations, are you in serious danger of running out of space
in that header? Well, we're not building in a static path.
We might be building in a home directory, right? And so you can put padding in your
install tree prefix, it's like the next store, and you can say build with 256 long paths.
And you wouldn't want to have a user actually deploy in a path like that, but you can build
that way, create the binary, and then redeploy in a short path.
You've got potentially a space where there's, where you can have an arbitrary length path
as you're running. A lot of stuff doesn't build with overly
long paths. So if you get to 5.12, auto-tools starts breaking down and not supporting that
length of a path, and the packages actually don't support it. And so the sweet spot seems
to be like 256. Okay, thanks.