Okay, next talk is Pablo.
Who is going to explain us how to set up slurm client environments more easily.
My name is Pablo and I have been running the HBC Clusters since I was 9 years.
I was running the HBC Clusters at CERN and got involved mostly in slurm, running slurm.
That's when I came up with the idea for this tool since about 8 or 9 months ago I started
the HBC Clusters and I'm also participating in the SKA project, hence the pretty background,
where we do also things related to the HBC infrastructure.
So just a brief introduction to slurm in case anybody is not familiar with it.
Slurm is basically both a resource manager and a job scheduler, meaning slurm will manage
their allocations, it will track which machines are used to which jobs and which users own,
which CPUs and which nodes, etc.
And it's also the job scheduler, meaning it will, when users submit jobs, you have your
happy users over there, or hopefully it will be happy users.
And they can be one-on-one on your cluster, so they make a job submission, usually writing
a script that launches some workloads.
And they will basically interact with slurm and slurm will manage all of these job submissions.
You won't just have one by one, you will have hundreds or even thousands of jobs that are
scheduled to run on your infrastructure and slurm will manage the views and the priorities
and the accounting, etc.
So basically it's a batch manager, but there's both resource managing and the scheduling
of the jobs.
Building a bit deeper into how slurm works, because this is relevant for this talk, there's
basically two main components, two units that are the most relevant, and those are the
controller, which is called the slurm CTLD, and then the deals that run on the worker nodes
at the bottom, which is the slurm VDU.
And then you have other demons like the slurm VD, slurm RST, slurm RST.
Those are not relevant for this talk, I will mostly focus on the part on the left here.
So users and client tools, they basically interact with the controller over a slurm protocol.
There's nowadays a slurm RST, so you can also interact with the rest with some scripts,
but mostly all the user lab tools, and mostly almost everything in the slurm ecosystem just
talks to the slurm CTLD, and this controller handles the source of truth for slurm, so it
knows which resources are allocated where, it knows which jobs exist, and knows who the
users are, etc.
The controller talks to the slurm units, and talking to the nodes and the slurm units are
in charge of launching the jobs, so you do the cleanup, setting up the seed routes for
the jobs, whatever you have.
Now, what's important here is to know that for all of this to work, you need at least
the same thing. You need the slurm conflict files, and they need to be instinct between
the whole cluster, so you may have some difference, but mostly it should be the same.
There was no audio online? Okay.
So as I was saying, the slurm CTLD handles the source of truth.
The slurm units are in charge of launching the jobs, and the two important things are
that you need the slurm configuration files.
It's mostly the slurm.conf file, but there's other files as well.
Those need to be in sync in the whole cluster, and they need to be basically the same.
They should have the same hash, ideally.
And then you should also have a shared secret so that nobody can, a rogue client cannot just
add a worker node to the cluster and start doing malicious things.
So you have usually it's a munch secret.
It's called the demon called munch, and you have a shared secret as well for the whole cluster.
And this fact is important, is very relevant for this talk.
Now, up to containers.
So containers are increasingly becoming a super popular tool to run infrastructure for
reproducibility, for automating deployments.
And just in general, they're becoming super ubiquitous in our industry.
I think for good reasons.
And there are, I think, very good use cases for using containers with slurm.
In this talk, I will focus on the use case where you use containers on the user and client
side of things.
So those tools that will talk to slurm, to the controller mostly, to do things on the cluster.
So this could be some automation that you have run to do whatever.
For instance, you could use it for monitoring purposes.
You could write a tool that does health checks on the cluster for accounting.
I've used it extensively for accounting as well.
But also integration with other services, right?
Or if you want to connect the Jupyter notebook with slurm, you will end up with some tools
that talk to the controller.
Now, there are basically two scenarios in which you can use containers with slurm.
On the left, we have the local use case.
That means imagine you have a frontend mode, you have a machine that's configured where it uses SSH2.
And from there, they can run the slurm commands to launch jobs, to track their job usage, et cetera.
It's conventionally called frontend mode for the cluster.
So if you just add the slurm client container on that node, it's very simple.
Because you can just, as I said, you need a secret with munch, and you need the config files.
And that scenario is very simple because you can just do bind mounts,
and you can access the munch socket to talk to slurm.
And you might bind mount the slurm config directory, and you're done, basically.
So that's sort of easy.
However, what if you have, for the use case on the right, you have the distributed or remote use case.
And in that case, you may run your slurm client container in a different service.
That's a different network, or you may run it on Kubernetes or somewhere else.
In that case, you obviously can't just do the bind mounts because you need to give it all those things.
So you would have to give it all the slurm config files and somehow the munch shared key
so that your external service can talk to your cluster, right, specifically to the slurm controller.
Now, this is an extraction from a Docker file.
This is the naive approach.
This is how I started trying things.
Easy, right?
You just take the slurm config, and you just copy it to the destination, right?
And this will absolutely work.
But I was not happy with this approach because then you end up managing two copies of your slurm config.
And I really like having a single source of truth for when you do configuration management and automation of your infrastructure,
I really like having a single source of truth.
And managing this in this way with containers is very fiddly because it's very easy that you will forget to update it
or something that will fail to update it automatically.
It's just not ideal.
I didn't like this approach, but it will work.
It will work.
And some of you who know slurm may say, oh, but Pablo, why wouldn't you just use slurm's config less feature?
So slurm config less is a new feature since slurm 20 or so that will basically allow a client to just pull the config files from slurm.
So the slurm ddemons that run on the worker nodes, when they start, they will just grab the slurm config files.
So you can just remove the needs to even copy the slurm config, right?
Well, that's a trick question.
Not necessarily because then you need to run a slurm ddemon in your container.
And you also need the munch demon.
And it sounds easy, but it's really not.
You will need to do a lot of hacks. This is an instruction from a container that I was creating.
And you run in lots of awful things.
Like the slurm ddemon expects this release agent file to exist in the C group and the containers, they just don't create it.
I tried it on Docker. I tried it on different Kubernetes versions. It just doesn't exist.
I don't know why. I couldn't find out why. If anybody knows, please tell me.
I googled around a found that could have been related to some privilege escalation issues.
However, if you just remount the C groups, the file appears.
So I'm not sure what's going on there.
Another fun story is that, for instance, if you're using Kubernetes,
Kubernetes likes to give a sim link to your secrets, and munch refuses to take the secret from a sim link for security reasons.
It makes sense. So there's no more.
So you will need to put in hacks.
And it's hacks on top of hacks on top of hacks just to run these two demons.
And yeah, I was not very happy with this approach either.
So basically I was faced with two options.
We arrived at this situation. You're faced with two options.
Either you basically do the first naive approach where you just copy all the stuff into your slurm container.
You manage a copy of your slurm config files.
But as I said, if you want a single source of truth, this might not be ideal.
You also need, of course, in the case of use case, unless you need munch,
and you need to supply the munch key.
Or you can try the configless approach, but then you need to add slurm d to your container
so it can pull via configless your config files.
But then anyway, you also need munch.
And you need to add the munch key to your container somehow and managing secrets.
I mean, if you're running Kubernetes, it might not be a big issue or some other container manager.
But you will still need to maintain all these extra demons with nasty hacks.
And we don't always like all these having lots of hacks in our infrastructure.
There's a third option, by the way, which is trying to go secret less.
It doesn't work in combination with configless, where you try to use JSON web tokens.
But it gives a lot of issues. It doesn't really work. I tried it.
So I didn't include it here.
Just mentioning it in case somebody thought about it.
So Pablo, you talked about the bad and the ugly. What about the good?
Is there any good part to this?
I'm glad you asked.
Yes. What if we had a single shot CLI tool,
that just a very simple tool that just was able to authenticate to the controller,
either using munch or JSON web tokens, which Slurm also supports,
and just fetch the config files, and then it's done.
That's all you really want to do, right?
Because then your tools, the Slurm tools can work,
because they have the Slurm config files, and just by having the JSON web token in your environment,
you can just talk to the Slurm controller.
And yeah, that's the tool that I wrote.
It's a very simple tool. It just does exactly what I described there.
And it's open source. You can find it on GitHub.
I uploaded it in the past month.
Fun story about this.
As I said, I had the idea for this when I was back at CERN.
I worked on this a year ago already.
But then I somehow lost the source.
I don't know what happened.
Just before I left CERN, the source was just lost.
I don't know why. I must have deleted it by accident.
I don't know what happened.
So after I left CERN, I kept in contact with my ex-colleagues,
and they were telling me that they wanted to do this integration between the swan,
which is the who here knows swan? Anybody?
Okay, one, two, three.
Yeah, so it's the Jupyter Notebook Service for CERN,
which also does analytics.
And we wanted to connect it to Slurm,
and we run into all these issues, because this is a service that's exposed to the whole internet.
So we didn't want to have the munchkey for the Slurm cluster in the container, et cetera.
Anyway, so then I left CERN, and then, yeah, my colleagues were telling me,
oh, it would have been so useful to have this at Watapiti.
And then a few months ago, I just didn't like the fact that I had lost the source and all these days.
I spent a couple of days reverse-engineering the Slurm protocol,
and I just didn't like losing it, so I just rewrote it more properly in Python and just made it public.
So if you're interested in making client containers like this,
feel free to give it a try.
It looks a bit like this.
It's very simple.
You can choose between munch or JWT, JSON WebToken's authentication.
If you choose JWT, which is the most simple one,
you just need an environment variable with a token,
and you can tell it where you want to store the config files,
and then you have verbosity as an option.
So it's very simple.
It has very little dependencies.
So the tool talks several Slurm protocol versions,
because with every major release, Slurm changes the protocol versions.
So you can list them with minus L,
and it will show you basically all the versions that it supports.
So imagine you have a Slurm WebToken in this variable.
You can just tell it to do JSON WebToken authentication with the server.
It supports multiple controllers in case you have high availability set up in your Slurm cluster,
so you can specify a list of servers that it will retry until it succeeds,
and then you tell it the protocol version of the Slurm CTLD,
because it needs to know what protocol it should talk.
The protocol version negotiation, I think it doesn't exist in the Slurm protocol,
so you have to tell it which version you want it to talk,
and that's it, and then it will just download the Slurm config files
and happy days for your containers.
Conclusions, I think I'm ahead of time.
So this tool called straw, it can simplify the cost of creating and maintaining
your Slurm client containers.
It can also increase the security,
because you don't need to put the Munch key everywhere,
where you're running your client containers.
JSON WebToken's surface.
Caveats, caveats.
I think this tool should not exist,
because ideally this would be supported upstream.
So, you know, if anybody has any influence on SCADMD Slurm development,
yeah, I think it would be nice if we had this built-in into Slurm.
And then the second caveat is that the JSON WebToken,
the token needs to be associated with a Slurm user, basically.
So ideally, you would be able to just generate a JSON WebToken for a user
that's going to run on the Slurm cluster,
and then if the secret for some reason is exposed, you've only exposed
the JSON WebToken of a single user.
However, this is a limitation built into the Slurm, into Slurm, basically.
You cannot pull over the protocol the Slurm config file
unless the token belongs to the Slurm user, or to root.
Still, I think it's an improvement over having your Munch key available everywhere.
If you're free to try it out, that was it.
I'm happy to answer any questions you might have.
APPLAUSE
Thank you very much, Pablo.
Time for questions.
So what kind of clients do need the config file?
Could you do everything over REST nowadays?
Is it still necessary to use the config file?
Yes, so anything that wants to run srun, sbatch, sq, sinfo.
For instance, if you have the Jupyter Notebook plugins,
they will just run those commands.
Or if you want to run a client that uses PySlurm, for instance,
or any library really, anything that uses lipslurm underneath
will automatically read the config files, right?
So, of course, you can write your own client,
handwritten from scratch, that just interacts with the Slurm REST to do stuff.
Yes, but you cannot leverage all the existing user client tools,
and the lipslurm, PySlurm, etc.
So if you want to create a Python tool, for instance, that leverages PySlurm,
this would be, I think, a good solution.
I think Slurm does have, like, a REST API, but it's considered very insecure.
So even the documentation tells you, like, don't use this.
I just didn't understand, like, for a long time now,
why everyone needs the config file, right?
I mean, why doesn't it need to be in sync?
Like, couldn't they just exchange the information over the protocols now
and just say, like, this is your Slurm server?
Yeah, that's a configless feature. That's a configless feature, essentially.
Yeah, but the configless feature just downloads the config.
Yes.
Next, like, config less OK.
Yes.
I download the config. I don't need the config beforehand.
It's like serverless. There's always a server somewhere.
Yes. Yeah, exactly.
So that's just how Slurm works.
Yeah.
So I'm still a little confused about the Slurm client container.
So the container is an application on the actual Slurm client,
because you have to document in the SlurmConf,
you have to sort of say what your clients are
so that the scheduler can intelligently decide how to schedule jobs, right?
I'm missing something.
No, you don't really need to declare all the clients for Slurm.
You just need to declare the worker nodes that are part of it.
But you can have any...
I mean, it depends on how you've configured it.
You can limit it. You can limit in Slurm which clients are allowed to connect,
but you don't have to.
So you could just...
But even if you do, you will need this,
because you will...
Even if you authorize a host name to connect as a client,
it will need to have the munch key and the SlurmConf files, et cetera.
Does this answer your question?
Well, no, so when you...
In the Slurm.conf, you sort of detail what your positions are,
and you have to kind of tell it what the capabilities are of your clients,
of your Slurm clients, right?
So that Slurm can decide how to schedule jobs.
I'm missing something.
Well, I think you're thinking about the compute nodes.
Yeah, I am.
Yeah, the node names part of the SlurmConf.
So the containers run on the compute nodes?
No, the containers would be...
Let me go back to one of the slides where...
So you're thinking maybe about the compute nodes,
each of which runs a Slurm DDemon,
and those you have to declare.
Yes, I think in 2023, by the way,
you will be able to dynamically spawn compute nodes,
but that's the future.
What I'm talking about is all the users and client tools
that connect to the controller to run SQ as info,
like when you use Slurm and you...
Hello.
So if you had some tooling that you automated
to gather metrics from Slurm or, yeah,
a Jupyter notebook service, for instance,
that connects to your cluster that wants to launch jobs,
that wants to run as batch SQ, whatever,
that's in that domain.
Yeah, I mean, the newest werewolf runs containers
on my back for the stream.
I mean, I think the newest version of werewolf
is set up to run containers on the Slurm clients, right?
It's sort of, you're actually launching containers
as applications, so that was kind of...
That's on the compute nodes.
On the compute nodes, yeah.
Yeah, yeah, that's the compute nodes.
Thank you for your talk.
So I have a question.
You are telling that you can pull the configuration
with your tool, but there are many...
Fine, you can't pull with configless.
For example, all the spank plugins,
or I think topology, you can pull it,
but various, like I said, spank plugins and so on.
So how do you manage this kind of config file
that are not ended by default by Slurm?
Right, that's correct.
So when you use the configless feature,
it will download the, you know, the Slurm Conf,
the C Group Conf, a lot of config files,
but it will not download your plugins, your plugin files.
But I think those are usually not needed
if you're running a client,
because those are usually just needed
for the Slurm D demons, right?
Even for the worker nodes.
Like the epilogue, the prologue,
you mean all of those plugin scripts, right?
The authentication plugins.
Those are usually needed by the Slurm D demon,
but if you're just writing a client,
but say you're automating something
with PySlurm to interact with it,
you don't need those files.
And Slurm will happily...
You can happily run...
all of those commands without those files.
Yeah, okay, so if I just summarize,
the idea is just to create some frontend nodes,
but not really work nodes.
That's right?
So you...
So if you want to use configless to set up a frontend node,
you might need those files from somewhere else.
But if you're just creating a container
to just interact with Slurm and send Slurm commands,
you don't need them, basically.
Because the plugin files are usually the...
Yeah, the epilogue prologue for the Slurm D
or the Slurm CTLD.
And that's not what these Slurm client containers are about.
So short answer, you usually don't need them.
Hello, thank you for the talk.
I'm wondering, in huge institutions,
like in CERN or EPFL,
would you run your own forked or patched Slurm
so you could fix maybe the authentication privileges?
Or is it just not done because it's...
I've never carried any Slurm patches, to be honest.
I've always, both at Slurm and at EPFL,
we just use Slurm out of the box.
It works well enough for our use cases.
It is true that you could, for instance, do a patch
to enable finer granularity for the permissions.
For instance, you could enable any user to pull the config file.
That would be a nice patch.
We don't do it.
Okay, thank you.
We have time for one short question.
Hi, thanks.
We actually are very interested in this
because we are applying...
We have a Jupyter Hub frontend
that actually talks to a Slurm cluster through SSH
because we don't want to install all that stuff,
like the munch and the full Slurm deployment
into the Jupyter Hub host.
And I'm wondering, how does it talk actually to Slurm control?
So is the Slurm control always listening to any...
any of the hosts that will talk to it?
Yes.
Or is there any restrictions to who is connecting
to the Slurm control demo?
So there's an alloc nodes setting in the SlurmConf, I believe,
which will allow you to restrict from which nodes
you can allocate resources.
Okay.
So you can limit it.
However, if you don't have that,
the Slurm will happily accept anything
because if you have the shared secret,
it's considered good enough.
Okay.
Or a valid JSON web token.
Okay.
Yeah.
Thank you.
Thank you very much, Pablo.
Thanks.
Thank you very much.