I hope you can hear me online, well, if not complain on the chat.
We are going to talk about HPC, High Performance Computing, and Nix, and how we kind of deal
with that.
My name is Rodrigo and my co-worker Raul here.
A bit of what we do, essentially we work on a parallel, concurrent task-based runtime,
like similar to OpenMP, if you are familiar.
We also need to work with a compiler based on LBM to read this pragmas in the code and
transform them to function calls to the runtime.
In our job performance it's critical, so we really need to take care, and in general
we execute workloads on several hundreds or even thousands of CPUs.
Here's a little example of something that we have observed.
We have a program here that runs, and here you can see the CPUs, and this is the time
of execution, and we are examining this little point here, because the time here is slightly
bigger than what is normal, and we can see that the problem is that the allocator took
a bit longer, so this is just an example.
In general, in HPC or this high performance computing, it's just a lot of machines connected
by fiber optic, they are managed by this SLUR diamond, which allows you to request a certain
number of nodes.
We don't have fruit in any node, and in general it's very old kernels and very old software
stack, so yeah, we are stuck with that, and in general the state of the art now is to
use LD library path to load other software and change versions.
Problem with this technique is not very easy to reproduce, so the question is, can we benefit
from using NICs?
In general, we will get up-to-date packages, configuration options for every package, no
more LD library path, and we can track everything that we use for an experiment.
The problem is we don't have fruit, so we cannot install the NICs diamond as we would
like to do.
So let's take a closer look at what we do and how we do it.
In general, we work in these three hats, so to say, in the development side, we take a
program and we compile it several times until it actually compiles, and we kind of need
to do this cycle quickly, so we want the compilation time to be very low, so we need to reuse
the already built tree to rerun the build command.
When we are finished, we switch to the experimentation side, and we run this program in the machine,
and we maybe need to tickle with the arguments or the configuration file of the program to
get some results that we want to examine.
And then we also do some visualization of the results, but we are not going to talk in
this talk about this.
So we will focus first on the experimentation and later on in the development side.
So a bit of what we did.
We tried with individual installation of the NICS store by using username spaces.
The problem is that the number of packages grows, so we would like to share the store
with several users.
So we use an auxiliary machine where we actually have a NICS demo, and then we can perform
the build in that machine, and then use the post-build-hoc to execute some script that
copies the output derivation to the actual cluster.
Problem is inside the cluster, NICS store doesn't work, so we wrap the command NICS store
in a shell script.
And when it's involved by the auxiliary machine, it creates a name space where it mounts the
NICS store there, and then it runs the NICS store and receives the derivation, so we can
actually copy it over SSH.
We also try to patch the NICS diamond to run inside the machine, but it's a bit complicated
because we cannot even run a user diamond there.
Okay, so let's focus on the experimentation cycle.
The first requirement, most important thing, well, assuming that you already have a program
that somehow you built in a sandbox, we want to execute this program in the machine, and
we want to make sure that this program doesn't load anything that is outside the NICS store.
So, especially the LD library path may have some path that actually has libraries for
your program, so we don't want that, and also it may use the deal open to load other libraries.
So ideally we want something like the NICS booth with a sandbox that prevents accesses
to slash user or a slash opt, and it needs to work in a slurm too.
Another requirement that we need is for MPI, the communication mechanism, to use this syscall
process being read by, that only works if the process are inside the same name space.
So we solve this by running a check, that checks if the name space is already created,
and if it is so, we enter it, otherwise we create another one.
So let's take an overview of how this works in the cluster.
We have here the login node and two compute nodes that were given to us for running our
program.
In general, we have to wait a bit after requesting the nodes, that is fine.
After this moment, we take a shell that is connected to one of the allocated nodes.
These are the nodes, and each node in your case has two sockets.
So we usually run one process per socket, and we talk to one of them only.
Inside this process, we don't have NICS.
So we first load this name space by using our script, and then we can run other programs
like srun, which is the client that will launch the workload, but is inside the NICS store.
So we can compile programs linked to this specific version of Slurm, sorry.
After that, it requests the Slurm demo to execute something in parallel, and the Slurm
demo forks in every process, one process that will run something, but it's outside the name
space because it's not controlled by us.
So we execute our script again to join the name space if it's found, or otherwise we
create another one, like in the second compute node.
And here we can see that we can communicate in the same node because they are both in
the same name space, and they're one too, and here we use fiber optic communications.
Another requirement that we need is that we need custom packages.
We use that with this technique where we define a call package function that takes priority
over our attribute set.
So we can change software that is provided in upstream with NICS packages, and we use
our version first, so we can hack on those without disturbing the whole package set.
Another thing that we need is to define packages with compilers.
In general, we use LBM with a custom runtime, so we use the wrapcc width and inject this
little environment bar so we can load our runtime without needing to recompile the compiler.
We also need, unfortunately, proprietary compilers, and we use this RPM extract and the AutoPatch
Elf hook to fix the headers so we can run them on NICS2 and compile derivations for
them.
Okay, let's move on to the development cycle.
In general, the development process consists in getting an application, adding some new
on-cube features to it, breaking things, testing and retesting the tiles, okay?
And this interactive workflow requires frequent changes in the source and compilation steps.
For this reason, NICS build is not good to work with because every change in the source
will trigger a full copy of this source to the NICS store and a full compilation.
With big repositories, this is a problem because, for example, in the slide we can see the time
about how much time it takes to build LLVM in a 32-core machine with hardware string,
so it's a big machine.
And we can see that although we use Ccatch, we talk about different orders of magnitude
comparing it with simply reducing the previous build.
Another alternative could be using NICSel to get our tools to build the application,
but this environment is not isolated from the system.
And we can find software that includes hard-code paths directly to the system, like in this
case with a Sigma model file of ROKM that is CUDA4AMD for those who don't know what
is it.
And if we take an application that uses ROKM and configure it and check the lock output,
we can see that, at the end, the installation selected is the system one instead of the
NICS package we want to.
An isolated environment will prevent us from this situation, avoiding the necessity of
patching the source to solve this problem.
Our solution for these two requirements is to first build an isolated environment with
a tool we named NICSwrap.
NICSwrap is a script that uses bubble wrap to enter a username space where the NICS store
is available, but not the system directories, like in this case, slash user.
And in this environment, we can launch our NICS tools, like, for example, NICSBuild.
This works because inside the name space, NICSBuild creates a new sandbox in an instant
name space so the environment is not affected.
And the most powerful feature of it is running NICShell inside this isolated environment
to get your tools to build your application in an isolated environment so you don't have
to worry about accessing to the system.
And in this case, it's the previous example, LLBM, and reusing the build.
And finally, if you are using, like, as a slum, you can execute your application by running
NICSwrap after the slum step-forward process and your application.
Another requirement for us is, since we are in an HP environment, we want to get the best
performance of the applications, and for this reason, we need to build the critical performance
software with CPU optimization flags.
Our solution for this situation is to override the compiler wrapper injected flags by overriding
the host platform attribute, specifying the architecture and other stuff to the compiler
in a standard environment in OCC, and finally, we create the standard environment we will
use to build our software with this compiler wrapper.
I will talk about conclusions.
In general, we can actually benefit from using NICS, but obviously we have some drawbacks.
The cycles that I was talking about, we can still do it very fast, so yeah, it's very
nice for us.
And also, if we have the chance to get something like a NICS demo without the root requirement
and still be able to share the NICS store, that would be awesome.
Thank you very much.
We have five minutes left for questions, if there are questions.
Yeah, so question is how can we manage to NICS store where users can install if that
can be an issue for this space?
So in general, right now we have about 300 gigabytes of storage for our particular group.
We have around 2,000, 3,000 gigabytes of space available.
In general, in HPC, people use a lot of space, but if we share the store, that will be the
best solution instead of every user to have their own installation.
And we also, when we kind of analyze someone that says to us, please use less space, we
run their garbage collector.
Thank you.
Yeah.
Did you consider using, or can NICS use our path instead of using run path, because that
would get rid of your problems there.
And then the other thing you can do is, there's a talk in HPC about free binding paths to
SO.
Okay.
But that's a little bigger hammer.
Okay.
I see your SPAC T-shirt from here.
Okay, so question is about using our path.
Our path, yeah, because I mean it takes precedence over all the library paths, and you don't have
to worry about the user being stupid.
Yeah.
So the problem is that you can see programs using DL Open to load their own, so they don't,
sorry?
DL Open respects our path.
Ah.
Okay.
I didn't know that.
We're doing something to be okay with that.
So.
You can do, like, if you find the different namespace for DL Open, they won't respect that, but
most of them are not using that.
Okay.
So DL Open is not only the only problem, because we also see software trying to read each C
slash configuration file somewhere, and we also want to prevent that.
Yeah.
In general, we saw that it's safer to avoid the programs from accessing any path than
trying to find every single option that the program can use to, because.
Okay.
There was still one eager question.
Can we find the next wrapped script online?
Yeah.
I will, I think I will upload it to the fourth page.
Any other questions?
Yeah.
Not so much of a question, but a bit of a shameless plug.
The main blocker for having a rootless NixDemon was merged last week on the week before.
So hopefully that's going to eventually solve the third of your points.
Perfect.
Thank you.
So what about in-care libraries on the system?
Are you only envisioning that you would install the library to things like MBI through Nix?
Okay.
Because that's not possible on something.
Yeah.
It's a very good question.
For now, we have been very lucky to be able to work with only proprietary packages that
can be put inside Nix.
But it may happen that the proprietary something, it doesn't work.
So we don't have a solution for now.
Thank you.
Just switch it over again.