All right, so I'm Todd Gamble, and I'm from Lawrence Livermore National Laboratory.
Normally I would give an intro of what Livermore is, but who's been hearing about Livermore
in the news lately?
The people heard about the fusion ignition over in the US, that's our lab.
So I'm from there.
I work in the HPC area at Livermore, and so we have a big supercomputing center.
And the HPC ecosystem is a pretty complex place.
People distribute software, mostly as source.
You build lots of different variants of the package.
Users typically don't have root on the machine when they install software, and so they're
building from source in their home directory or installing something in their home directory.
And you want the code to be optimized for fancy machines like these ones over here.
So you're trying to build software that supports a really broad set of environments, including
like Power, ARM, AMD, Intel, and then also GPU architectures.
So things like NVIDIA and now AMD GPUs are showing up, and we've even got a machine coming
all out at Argonne.
This is near Chicago with Intel Panaveco GPUs.
On top of all that, the ecosystem has C, C++, Fortran, Python, other languages, Lua, all
linked together in the same app.
And so we want a distribution that can support this type of environment.
And so SPAC is a package manager that enables software distribution for HPC, given that
set of constraints.
Packages are not quite like the build specs that you would see in your standard RPM or
Deb-based distribution.
They're really parameterized Python recipes for how to build that package on lots of different
architectures.
And it has a DSL for doing that.
I'm not going to get into that today.
But the end user can essentially take one package and install it lots of different ways.
So you could say, I want to install HDF5 at a particular version.
I want to install it with Clang, not GCC.
I want to have the thread safe option on, or I want to inject some flags in the build
and have an entirely different version of it that's built with a different set of flags,
or that's targeted at a particular micro-architecture, or that maybe uses a particular dependency.
So you can build the same package with two versions of MPI.
So we're trying to provide the ease of use of mainstream tools with the flexibility needed
for HPC so that we can get the performance everyone.
And it builds from source, but you can also install relocatable build caches in SPAC, much
like you would with, say, Nix or Geeks.
They're not relocatable because they're not really targeting the sort of home directory
use case, but it's the same sort of build cache model.
It's not a typical binary distribution.
The whole project has a fairly large community of contributors, or at least maybe not large
by some of the other distribution standards, but we have 1100-plus contributors.
We maintain the core tool, and then there's a whole bunch of people who work on package
recipes.
So in some ways, it looks a lot like Homebrew or a project like that.
And then there's a whole bunch of infrastructure behind the scenes to keep all this working,
and all these things together enable people to build lots of different software stacks.
And so there's like an extreme-scale software stack that's maintained by the US Exascale
Project.
AWS has a stack that they use on their parallel cluster product internally, and also for users.
Livermore has its internal software deployment.
There are some math library stacks, VizTools, things like that.
And every application, really, in HPC is kind of its own software stack.
So you heard about flat packs and snaps in the last session, well, really, making apps
more mindful of how their software is actually a distribution is something that we've been
pushing for a long time within HPC.
The GitHub is a pretty busy place.
We merge 300 to 500 PRs per month, and it's like something like 411 commits or more.
And so managing that is kind of painful.
And we're trying very hard to reduce downstream work, which is actually difficult for a source-based
distribution.
If you think about how SPAC is structured, there's this mainline develop branch that
actually most people use.
They'll just clone it straight from the repo, build from that, kind of like you do with
mixed packages or something.
External contributors contribute there.
And we cut a release every once in a while where we stabilize the packages and keep them
sort of fixed so that you don't have a lot of version churn in the repo.
And then to actually integrate with the HPC facilities, all the places that are deploying
supercomputers, we have this E4S software distribution where they end up doing a whole
bunch of downstream integration at the site, where they're basically building the whole
thing from source, essentially in a new environment.
And there's a whole lot of debugging that takes place there that we would really like
to be able to move upstream.
The applications, likewise, they are not necessarily using what the facility deploys.
Some of them do.
Some of them don't.
They pull from basically all of these places.
They might get a math solver library from the facility.
They might get something else installed from SPAC mainline built the way that they want.
And they may pull stuff off of release branches too, all to assemble an application and have
it built.
And so this is a lot of porting at the lowest end, and what we'd really like to do is take
that software integration and move it upstream and get to a point where we can have these
types of environments building NCI all the time in sort of a rolling release and do binary
deploys on the supercomputers with actual optimized binaries.
So that's what we're trying to get to.
So we set out to make a binary distribution with a bunch of different goals.
The main one, and the one that's pretty key to our whole ecosystem, is it has to be sustainable.
We don't have that many maintainers.
And they currently, their workflow is basically to work with people who are making contributions,
on pull requests, help them get them merged, and then move on to the next one.
And we don't want them to have to sit around and babysit builds on, say, a release integration
branch all the time.
We want a rolling release because people do tend to use the develop branch.
And so we want that to be up to date with pretty current binaries all the time.
But some people do fix themselves to releases, and so we want sort of snapshots for those
releases as well.
We need to be able to support, at least eventually, all the packages that are in SPAC.
And it still has to be source-buildable around those binaries.
So if you want to build a component and rely on binaries for some other component, we want
to support that.
And then finally, people trust sources.
They can check some of them.
You can download the tarball.
You can usually check some of them, except for when GitHub changes the hashes.
But we want to ensure that the binaries that we're generating are just as trustworthy as
the sources.
So we've taken some steps to ensure that.
So SPAC is a little different from your standard distro if you haven't gathered already.
If you think about a traditional package manager, you have a sort of a recipe per configuration.
And so that's like your RPM build spec or dev spec or whatever.
It goes into a build farm, and you produce packages, at least for one platform, in sort
of a one-to-one relationship with those specs, actually.
There's templating and things that goes on to reduce that.
But you're typically maintaining one software stack that gets updated over time.
In SPAC, what we're trying to do is we have these parameterized package recipes that go
into build farm, but it's really the same recipe that's being used across different
architectures.
We force the contributors to work on the same package so that essentially you're modeling
all the different ways the software can be used, and we try to get a lot of reuse out
of the recipes across platforms.
Those go into the build farm, and you can use the same recipes to produce optimized binaries
for lots of different platforms.
So you could get a graviton, arm build, you could get a Skylake binary, you could get
a GPU build, and so on.
And then you could do that for many different software stacks for different use cases.
And then we want you to be able to build from source on top of that.
So that's what we're trying to do.
We put a CI architecture together that is sort of based around this.
Like I said, we want to be sustainable, we want to maintain the workflow that we already
have on the project, and so we want people, we want basically GitHub to be the center
of the distribution.
What goes into develop is really maintaining the distribution as well as contributing to
the project.
And so we have a bunch of infrastructure currently stood up in AWS to support this.
So the binaries themselves and the sources are all distributed through S3 and CloudFront.
We set up a big Kubernetes cluster to support autoscaling runners, and we're using high
availability GitLab in there to drive the CI.
GitLab may seem like a strange choice for maintaining a distribution, but the motivation
behind that is really that all of the HPC centers also have internal GitLabs, and so
do a lot of universities and other sites.
And so the goal is really for all of this automation and tooling to be usable not just
in the cloud for the large distribution of SPAC, but also for people's personal software
stacks locally.
And so the idea is that we're generating GitLab CI configuration, and you can use that either
for this or internally or in an air gap network somewhere.
So we're leveraging Carpenter on the backend for just-in-time instances for runner pools.
That's a tool for AWS, it's open source, you can find it on GitHub.
It essentially lets you make requests for nodes with certain amounts of memory, certain target
architectures, and so on, and it manages containers on the instances for you on the
backend and sort of moves work around so that you can have an efficient build pool in Kubernetes.
We also have some bare-metal runners at the University of Oregon with more exotic architectures
than you can maybe find in the cloud.
So like there's an AMD MI 200 GPU builder in there, there's A64FX, which is what runs
on Sugaku, it's the ARM architecture with vector instructions, Power9, and so on.
And so we are able to do runs there for architectures that aren't supported in the cloud.
There's some monitoring thrown in.
We haven't really leveraged it in a smart way yet, but we are collecting a lot of data
about our builds.
And then there's a bot that helps sort of coordinate between GitHub and GitLab.
And so we have sort of a sync script that allows us to build off of forks and things
like that in GitLab over this whole setup.
So it's fairly custom, but at least the GitLab component is recyclable internally.
And we would like to be able to support more runners in the future, like if maybe we want
to work with Azure on their HPC setup and they want to provide runners for the project
or if other universities and places want to provide runners, we want to leave that open.
For maintaining the stacks themselves, we made it possible to sort of instantiate a new
stack in a pull request.
And so we have this directory full of the sort of 16 stacks that we currently build
in CI.
You can see them there.
Each one of those is some targeted software stack for some type of machine or some group.
Each of those contains sort of a YAML file with configuration for the stack in it.
And so the YAML file itself is fairly simple.
It has a list of packages that you want to build, and so this is the machine learning
one for CUDA.
Those are all the names of the stack recipes that you're building here.
And then some configuration up here.
And so for this particular stack, you're saying, I want to build for x8664v3, which
is AVX2.
And I want to disable Rockum and enable CUDA, except on LLVM because there's some weird
bug with the CUDA support there, at least in our stack.
And so you can see it's fairly concise.
You make a list of packages.
You say, here's the configuration I want, and you can go and take this thing and build
a bunch of packages.
We make it easy to change sort of low-level stack-wide parameters.
So the parameterized packages in stack, you can tell it to build with a different compiler.
And so we had essentially this large E4S stack with maybe 600 packages working in standard
environments.
We wanted to support the one API compilers from Intel.
And so that's Intel's new optimizing compilers.
It is unlikely that anyone has ever run this much open source through a proprietary vendor
compiler like that, but it is client-based.
And so we were able to throw one API into the config by just saying, here's where one
API lives, and make all packages require one API.
And so the build system swaps in the one API compiler through some wrappers that are at
the lower level.
And we were able to get that stack working in a week or two, despite the fact that we've
never built a lot of these packages with one API before.
So I think that's actually pretty cool.
In a lot of cases, it's not worth it to use a vendor compiler because there's so many
bugs and issues with software that's never been built.
But here, we're just really throwing sort of a bunch of open source packages through,
and it helped us communicate with Intel.
We were able to say, hey, here are bugs that we're seeing with your compiler.
We can link you directly to the build log for the build that failed.
And that helps them patch up the compiler, and it continues to help them ensure that
it can build everything it needs to.
In SPAC, you don't.
So like I said, the recipes are these parameterized things, and so there's actually a solving
step to these stacks.
You saw sort of the requirements in the YAML file that said what I want to build.
We run that through our packet solver to get sort of a fully resolved graph of all the
things that need to be built in a stack.
And then that is used to generate a GitLab CI YAML.
And then for one of the problems that we have to solve there is mapping builds to runners.
So once the whole thing is concrete, and we've said here's all the dependencies, these are
all the exact build configurations we want to make, we have to say how that should be
mapped to particular runners.
And so we don't currently support things like cross builds.
So if you want to build for AVX 512 or the more fancy vector instructions on newer Intel
CPUs, you need to make sure that you get one of those CPUs in the build environment.
And so we say, if you match AVX 512, give me an AVX 512 runner.
If you match one of these somewhat atrocious, hard to build packages up here like LLVM and
PyTorch, give me a gigantic runner with lots of memory, things like that.
And essentially what this is doing is it's just saying, here's the package properties
up at the top, here are the tags that should be on the runner, make sure that I get a runner
with those capabilities.
And we haven't got a schema for all the tags yet, but I think we could standardize this
and make it easy for someone to plug in runners at their own site for this sort of thing.
All right.
So one of the things that we did here to ensure trust is we have essentially a build environment
going on in pull requests.
If you trust back, you're basically trusting the maintainers.
We want to ensure that the binaries are things that are approved by the maintainers.
And so we can't just distribute binaries that got built in pull requests.
So when contributors submit package changes, we go and we have private buckets for every
PR that we're supporting where we're doing the builds.
The maintainers come along and say, oh, it worked.
They review the code.
And then they say, okay, we can merge that and rebuild everything on develop and sign.
So essentially everything in the main release is getting built from only approved recipes.
It's not using any binaries that were built in the PR.
All right.
The pull request integration, yeah, definitely makes things easy for contributors.
And we were able to take the system and announce our public binary cache last June with something
like 4600 builds in CI.
And so it's mostly easy for contributors.
They get a status update on their pull request.
And mostly easy for users.
They can just say, hey, use the binary mirror.
So there are some problems.
One issue is that build caches are a lot different from RPMs and devs.
In most distributions, you would have sort of a stable ABI for your build cache.
Your rebuild package, you can throw it in the mix with the others.
Here if you modify one package, you really do have to rebuild all the dependents.
And so if you modify XZ here, then you have to build everything that depends on it again
in the build cache.
And so what that can mean is if you have a gigantic software stack like this one and
you modify, say, package conf at the bottom of it, it can trigger a massive rebuild of
everything in the stack.
And so that's one of the scalability problems that I think we're going to have to deal with
in the long term is that you can get these really long-running pipelines.
Caches like Visit and PyTorch and so on will build forever, and it frustrates contributors.
The other sort of thing that happens is if you think about how the release works on develop,
you're picking a commit every once in a while and building it.
And if you have a PR that is sort of based behind the last develop build, that's OK.
Although GitHub typically wants to merge that with head, which means that you'll build a
lot of redundant things in your build environment.
We can be picky and merge it with the last develop build to ensure that we get a lot
of cache reuse in the build environment.
But what that means is if we get a PR that's out ahead of the last develop build and say
D up there is in progress, if you merge that second PR with D, you're basically going to
be doing the same builds that D is doing but in a PR environment.
And so if you have a bunch of those, we've brought GitLab down before by accidentally
building all of those PRs that are not caught up with the latest or for which develop has
not caught up with them.
And so we have to be picky and hold back these guys until there's a build ahead of them so
that we get enough reuse out of the cache to support this.
So the other problem with long pipelines is that they, depending on how reliable your
infrastructure is, the more things that you build in a pipeline, the more likely you already
get a build failure somewhere.
And so because we're building this cone of destruction in our pipelines, we are sort
of subject to system failures happening in the pipeline somewhere.
And so users have to kind of babysit and restart builds that have nothing to do with what they're
contributing.
So we're looking for ways that we could make that better.
One issue that we have is consistency.
So when you test on PRs, it's not always sufficient to ensure that your develop branch is working.
So you may have this initial package state, a PR gets submitted, you test with new B.
Another PR gets submitted, you test with new package C.
If you take those and you don't require your PRs to be up to date with develop, when they
both get merged, the state that's in develop is something that you've never tested because
you have basically new versions of those two packages together now.
And so there are ways to get around this.
One of them is merge queues.
So we're looking at merge queues as a way to scale this pipeline out.
They essentially allow you to have pull requests with a small amount of testing where you then
enqueue them in your sort of merge queue up there, that's the gray stuff.
And they are sort of serialized for commit to develop.
If they succeed, then they're merged directly in a fast forward fashion.
And then basically the full testing is only done on the merge queue.
And you always are assured that the thing that you tested is the thing that gets merged
into develop.
So we're looking very much forward to GitHub making merge queue available in the next couple
of weeks.
The other thing we think that could do is allow us to sort of stage the work on PRs.
So we're looking at ways we could scale this out.
Right now, for a relatively small number of packages, 4,600, we're able to build this,
these massive rebuilds on PRs.
But we need the stage to see how to scale it out further, so that's what we're looking
at now.
We might build only the package or only the package and direct dependence on PRs and maybe
phase how much work we do on the develop builds as well.
But we do need to do a full build every once in a while so that there's a consistent state
in the build cache.
So that's where we're at.
Thanks.
Thank you very much for the presentation.
You mentioned quite a bit of other technologies, like Nix, Gwix, Dab, RPM.
You could have mentioned Ombru as well, or maybe you did.
And Docker.
And it feels like all these tools could help you.
Yeah.
And it feels like you are building everything on your own.
So is there a reason not to leverage any of these technologies?
Which technologies do you mean?
Yeah.
So we are leveraging a lot of technologies, right?
I guess which ones do you think we should?
Nix, for example.
So we don't.
So Nix has essentially one version of everything in the mainline, right?
And in the HPC environment, what we want you to be able to do is not build that one thing
that's in the mainline, but to be able to build a one-off very easily.
So the whole point of SPAC is think of it as Nix with a solver, right?
It's Nix where you can say, actually, no, build this version of this thing with this
build option for that GPU, and it will take the recipe and reuse it for that purpose.
Whereas in Nix, it's much harder to have package variants like that.
So that's really the power of SPAC.
And so we're combinatorial Nix.
You can think of it that way.
Well, wouldn't you be able to leverage Nix and describe all these differences instead
of redoing it?
No.
The Nix packages don't do that.