Okay, the next talk is a perfect fit after a previous talk.
So Olivier and Axel are going to talk about Troika, a system to easily manage, submit
your jobs to any HPC system.
Yeah, thanks for inviting me and thanks for letting me talk.
So yeah, Troika, as Kenneth said, is a system so that we can interact with job submission
systems with one given interface.
So just before I start, a bit of context where I work.
So I work for the European Center for Medium Range Weather Forecasts, which is a European-based
international organization.
And we run an operational weather forecasting service four times a day that we send out
to national meteorological services and private customers.
So we also operate quite a variety of services, like we have our own in-house research to
improve the models, to do climate analysis, reforcasts.
We operate services linked to climate change, for instance, as part of the EU Copernicus
Service, and we've just started a new project called Destination Earth.
So I'll talk a bit more about that because it's a nice entry to what I will present.
So it's a EU program for weather and climate.
It's a large collaboration that we drive with ESA, the European Space Agency and UMEDSAT,
the European Meteorological Satellite Organization.
And the goal is to run simulations of the Earth at one kilometer resolution.
So for those who are wondering, that's about 256 million points per vertical level.
So this project is quite big, and it will run on multiple HPC systems across Europe.
So for instance, I think Barcelona with BSC and Lumi in Finland, just to name two.
And that means we will require some level of flexibility to run our workflows.
So you notice I didn't say job because in weather forecasting and also for these projects,
we have lots of different tasks that we run together.
So here you can see an overview of what we run operationally.
But in practice, that's a few thousand tasks that run every time we want to run one of
these pipelines.
And we have multiple types of workflows in-house.
So the main one is the operational one, of course.
But then researchers have their own workflows.
We have support workflows like CICD, deploying software, or just fetching data and analyzing
data and things like that.
And that amounts to about half a million tasks per day on our HPC cluster.
And so sometimes we run parallel jobs, but most of those tasks are just small, like one
CPU or a few CPU tasks, just to do some processing.
So for that, we use a workflow manager that we developed called ECFlow, which basically
manages a task graph as a tree with additional dependencies.
So you can have dependencies on dates, loops, and things like that.
And that runs a script for every task.
So a task being one leaf in the tree I show here.
It stores variables for pre-processing if needed, keeps track of the task status, fetches
log files on demand.
What it doesn't do to keep it simple is connect to remote systems and talk to specific queuing
systems.
So ECFlow just runs commands on the server host, which is usually VM, and provides three
entry points, which are submit, monitor, and kill for every task.
And so if you want to run an actual job on an HPC system, that means you have to have
some kind of interface.
So first you can start by just saying, oh, yeah, the command is SSH to my cluster and
submit a job, and that's it, which works.
But when you change cluster, or even like there is an option to put, you're in trouble
because you have to change that variable, and it can be a bit painful, especially if
you have thousands of tasks, or you don't want to regenerate the whole workflow.
So next possible thing, you write a shell script.
So you could do multiple actions in your script.
You have a bit more flexibility, but I don't know if you tried handling configuration in
a shell script.
Usually it ends up quite easily into a nightmare.
It's very hard to maintain, and if you deal with several people, everyone has their own.
So we tried to have something a bit cleaner, and so we want to delegate it to a submit
interface that can be made generic, gives you lots of flexibility, and you can also
maintain it as a proper piece of software that means versioning, testing, and some level
at least of reproducibility.
So we call our software Torica because it runs mainly those three actions, submit, monitor,
and kill.
It's able to handle remote connection to a remote system, mostly using SSH.
It's also able to prepare the job script for submission, interact with a queuing system,
and optionally you can run hooks at diverse points.
So it's written in Python.
We put a strong emphasis in making it configurable so everything can be driven by configuration.
I'll show how this works afterwards.
And we want it to be extensible, so you can add new connection methods if running locally
on your server node or running over SSH isn't enough.
You could just add a plug-in if you want to support another queuing system, same.
And if you want to add some hooks, for instance, to create directories before your job runs
or copy files over before or after submitting a job, et cetera, you can also do it.
So as an example, that's how you would run Torica.
So it has quite a simple command line interface where you can control most of the flags you
will need in your day-to-day life.
So you choose the action you want to do, submit, monitor, or kill.
You give it a machine name which is defined in configuration.
Some options like the user, you tell it where to write the output file because that will
stay on the server.
And it serves as a reference if you want to copy some other files, they would be put
alongside this one.
And so here you can see the log below that shows the commands that would be actually
executed when doing that.
So as I said, everything that is configurable.
So each site has a name to identify it on the command line.
And then you define the connection type, local, SSH, whatever you want to add, a type.
So for now we support direct execution, slum, and PBS.
And then you can add some hooks, for instance, oh, yeah, before I start doing anything, check
the connection just to see whether it will actually work, or, oh, yeah, before submitting
the script, just make sure the directory containing the output file exists, or once
the job is submitted, copy the log file to the server so that we can see everything in
the same place rather than having files scattered around every system.
And so that's all good, but just having an alias to SBatch that does it remotely is not
really helpful.
So we need also to modify the job script to add some options that are understandable by
the submission system.
So for that we decided to have a new language, because obviously the directives are not interoperable
across submission systems.
And so we need some kind of translation.
We input some generic directives, and we can add some in the configuration as well.
And then we translate them, so either for things very simple, like, oh, yeah, the output
file in PBS is minus O in slum, it's minus, minus output.
So this kind of translation could have also plugins that compute resources, like if someone
gives you the number of nodes and the number of tasks per node, and you need the total
number of tasks, things like that, so you could add plugins, or if you have some specific
resource management in your HPC, you can add that as well.
And then on the output side, we have a generator that's site-specific, again, because we need
to adapt the directives to the system.
It can make the last few translations, for instance, the actual syntax of some options,
like mail options, most submission systems allow you to specify an email address to which
send an email for some of your tasks.
Only the syntax is slightly different for everyone, so it does that translation, and
it's able to add code, if you need, for instance, to define environment variables in your software.
So the main components that are extensible in Troika are, as I said, the interaction
with the queuing system.
So you have a parser that reads the native directives so that you can use them if you
need them for your processing, generates the job script, it runs the appropriate commands,
so either using QSOB, SBATCH, or whatever, it could use APIs if you have another system.
And it can also keep track of the submission, so most of the time, just keeping a job ID
so that if you want to monitor the task, you just say, oh, yeah, the script was this, and
Troika will know, oh, yeah, put the job ID in that file next to the script.
I don't need you to tell me where it is.
And so you can choose how you want to interact and define new interfaces if you want.
Same for the connection.
So the connection mostly does the running of commands on the remote system.
It's able to copy files over, if needed, both ways.
And you can have some hooks at various points at start-up just before submitting, just after
killing a job, for instance, if you want to tell a workflow manager that, oh, this task
doesn't exist anymore, I just killed it, or at exit if you want to move your log files
around, for instance.
And that allows you to perform extractions.
And then the last thing you can customize is the translation.
So if you want to generate more directives than the user provided, you can also do it.
And basically, you just pass a function that takes the input set of directives and updates
that set to whatever you need.
So as a bit of a success story for us, so we've just switched to a new HPC with a new
set of EC flow server VMs, new location, new everything.
So it's much simpler to actually be able to just change a config file rather than rewrite
a whole shell script that does all the submission for us.
And also, since we have lots of different users, they have different needs, they have
different ways of working.
And what we managed to do with Troika is that we managed to bring them all together to use
a single tool, which runs the operational workflows where they need to have tight control
over what they actually submit and all the options.
Research workflows, which need to be very flexible because every researcher might have
their own specific needs, but in the end, they run mostly the same kind of code.
So we need to have an interface that allows that.
And then we run also general purpose servers.
If someone has a data processing pipeline, for instance, they can just spawn a server
and do their work.
And that needs to have an easy to use interface because we don't want to teach people, oh,
yeah, you also need to know that to run your job.
So now what we do is we provide them with VMs where Troika is pre-installed, and many
of them just don't even notice that it's there.
And as a summary, so I said at the beginning that we handle about half a million jobs per
day, and most of them now pass through Troika, and it hasn't failed yet, so hopefully it
works well enough.
What it will help us with also going forward is supporting our software development.
So it's not necessarily tied to a workflow manager.
We want to control our CI CD pipeline also using that because some of the elements of
the pipeline have to run on our HPC system.
So basically what we could do is from a GitHub runner, we could use Troika to connect to
our HPC run jobs there to do testing, deployment, and everything.
We, as I said, run our in-house workflows, and we will continue to do that for the foreseeable
future.
It will help us to adapt to new HPC systems because every time we make a tender, any provider
could answer, and we don't control which submission system we will end up with, and even which
site-specific variants there will be in the set of options.
And then for destination Earth, as I mentioned before, we want to support multiple HPC with
minimal changes to the code.
And so just to tell a bit more, where do you want, we want to go from here.
So we want to support more queuing systems because, I mean, we support two, and one of
them quite well because if we use it, the other one a bit less maybe.
We want also to add functionality to inquire about the submission systems of, for instance,
which are the queues available, the petitions, things like that, so that the user doesn't
need to go to the server, check before running, like you could just run a command that fetches
all the information in a useful way and gives it to you without, we're abstracting basically
the specifics.
We also want to add some generic resource computation routines.
So we have some in-house, but they are very tied to the way we function, and so there
will be some work to make it more generic and then integrate it in the main source code
rather than in a plugin.
And for improvements to the code, we want to improve script generation.
For now it's a bit clunky, but it works.
We want to widen the coverage because you never test enough and provide packages to
install it on Debian-based machines, for instance, or RPMs for Red Hat systems, et cetera.
And if you want to contribute, feel free to talk to me or go to our GitHub page and I'll
stop for now and take questions.
Hello, thanks for the presentation.
So basically I've done something quite similar for my employer, sadly it cannot be open sourced,
but the problem that we have is we have legacy clusters with legacy job submission systems.
How did you manage to get the traction to migrate to Troika and to convince the user
to port their jobs, their developments to this new system?
So what we did first is that we made it as seamless as possible.
So if you want to interact with your job submission system without using our directives, you can.
They will just pass through, but you lose on the generosity.
And then what helped us is that we changed our HPC system, and that means we did basically
start afresh and everyone had to make changes, so we just pushed that onto them.
And I must say many of them have been happy because that meant we can do that for them
rather than them having to figure out the details of how do they submit jobs on that
new system and everything.
We can just tell them, oh yeah, it's reinstalled, it works.
And so yeah, that has been really helpful.
I actually have a follow-up question to that.
So one thing we have been doing, we switched recently, well, four or five years ago from
Torq to Slurm, and we didn't want to let all our users retrain themselves and learn
the Slurm commands, because in our experience, Slurm is a bit less user-friendly than Torq
is.
So what we did is we rolled a wrapper that people can still use QSub, but they're actually
submitting to Slurm, and it just, it translates the script in the background.
Troika doesn't do that now, right?
You have to use the Troika command, but it knows about the Slurm header.
Yeah, so you could technically do it.
We didn't want to encourage that, but technically you could, like, I think you could write a
script in three lines, a plugin that just takes the directives.
You would probably need to support all the directives you need, but we have a built-in
parser that is able to read, like, Slurm commands, for instance.
And so you just need to tell Troika, oh yeah, use those on top of whatever is specified
in configuration.
Is that something you would take pull requests on?
Yeah, if you want to.
Okay, we had another question.
Yeah.
Passed them away.
Hi, thank you for the presentation, very interesting.
So I'm an early programmer myself, and so my question for you is, how does it fail?
Like, have you studied or provoked, you know, intentional failures of the system, and have
you encountered funny behaviors, like, or plain hilarious faults of the system?
Yeah, we had, I mean, getting a new system has its lot of failures, so I don't know if
Axel, you want to take over for that, because you probably have handled some of the failures.
In the example of the command line provided, you can see that we redirect the output for
each submission, and this is a chance to analyze the submission and to decide what's the best
approach to deal with erroneous submission, meaning that some of them have to be reflected
the hard way to make it clearly visible, this is a problem.
And some others can be handled in a hidden way, or not so visible way, in a still deterministic
way, and so it may be hidden and still automatically handle the problems when they occur.
And this is what we expect with so many jobs to submit, to focus on the critical essential
for the human side, and to have a chance to teach the machines through the hook system
to manage with the specificities we have identified as problematic, but we want to keep ignored
or manage automatically until a fix is coming from the curing system, for example, if it
is related to a curing system problem or identified issues that may come with the next release.
So this is a way to deal with the failures that can occur at job submission.
Thank you.
Did I understand correctly that when you're monitoring a job, the reference is the script?
Yes.
Correct.
So that means everyone has to make sure their scripts are uniquely named each time, otherwise,
or is it the sort of script and where it is in the file system?
It's where it is in the file system.
So you are correct.
If someone deletes or renames their script, then it can cause a problem.
Submits with the same script.
So it's not a problem for us because our workflow manager basically does some pre-processing,
meaning that the script has some additional things like, oh, yeah, it's your second try
at that submission, so I will add.job2 at the end.
And so that's how we circumvent this issue, but you are definitely correct, and that's
something we will need to improve at some point.
But we didn't want to have to link to a database or something so that we can keep it simple.
Thanks.
You could just copy the script on submission, no?
We could copy it.
It's just that, yeah, if you have half a million scripts, we need to think, per day, we need
to think about cleanup.
Yeah.
Other questions?
Hello.
Users like things to be as simple as possible.
In order to do that, they would probably be nice to have some sort of central location
where recipes of various clusters would be sort of combined accessible for people to
be able to get access to.
Is that in your plan or?
What do you mean under configuration side?
So I could imagine a user turning up going, oh, I'm going to download Troika, and I'm
going to talk to this cluster that I have access to.
How do I get the configuration?
Oh, OK.
I see.
So we don't have that, but if Troika gets attraction, I think we could come up with a
website where you can host your configuration files or have some kind of index where you
can list them.
I think we would have all that's needed to do that pretty easily.
I think, hopefully, the configuration is easy enough so that you don't need to do much
on top of what's actually provided as examples.
But yeah, you are correct.
We could, if it gets popular, just provide configuration files for several systems or,
I mean, HPC system providers could also just give a configuration file with the system so
we can have it where Troika is installed and then the user doesn't even need to bother
about it.
Very small second one.
Given you've just done all this stuff, have you heard of a project called DRMAA, Distributed
Resource Manager Application API, it might make the insides of this slightly nicer for
your EC flow stuff, maybe it might take some inspiration for that.
Thank you.
A question, but also an observation.
A long time ago, there was a standard called DRMAA, it was an API.
Just mentioned.
It seems not to be used, maybe I'm wrong, but very quickly, your system, if you had
cloud-based resources on AWS, you've got an SSH connector.
Could you have, in the future, maybe run up some machines on AWS?
Yeah, that could be an option.
As long as you can write Python code to spawn up an image, a container somewhere.
Yeah, sure.
I think the API is for that, that just needs to be a plug-in that does the connection,
and that's it.
Cool.
Okay, we're out of time.
Just a comment.
I don't think you have any people using Troika outside of ECMEF.
No, that's the first time we actually presented outside.
All right.
Good.
So you're trying to start, or trying to get people to start using it?
Yes.
You're building a community, you're getting yourself into trouble.
We're going to get public requests and bug reports, but okay.
Thank you very much.
Thank you.
Very nice.
Thank you.
I'll just switch.
Okay.