All right, we'll get started with the next talk.
Julia Anon from Paratools is going to talk about testing and validation.
Hello, everyone.
Thanks again for the introduction.
So, yes, I'm Julia Anon.
I'm going to talk to you about testing.
As you may notice, I'm a bit happy to have a mic today.
So, please let me know if I'm becoming unreadable.
So, first of all, a bit of background, back in some years ago, we were actually, we still
had a team developing an MPI runtime, and while developing this runtime, we had the major
stake to develop a validation system to assess our software quality, but also to be able
to compare our implementation to others.
So, like everybody in HPC field at this time, we started to build our own ultra-specific
shell scripts to validate our implementation, because we were considering that our implementation
were too specific to be able to use some mainstream tools.
So, we started with self-scripts, great idea.
The fact is that with the team growing, people working in a separate place, working on multiple
heterogeneous machines, we had a huge issue to make people continue to validate, to use
this validation system, because it was slow, not really efficient, and hard to make it
grow.
Especially about maintenance, when we wanted to add anything from to the validation process,
it was just a nightmare, it was really costly in term of time, but also when our software
was evolving, was growing, just adding, just a little test into this non-requestion based
was a nightmare.
So, we started to consider why not creating a validation system able to take care of
HPC environment, and what it implies, like validating on multiple architecture, multiple
machine by multiple users, with multiple benchmarks, and we just thought about having
a generic tool able to handle all of that at what point.
So, what we want in that scenario, we are here in the case of validating an MPI implementation,
which means having a standard API, okay, so there is a lot of well-known MPI benchmarks
already existing, we don't have to rewrite a whole non-requestion, we have benchmarks,
you have proxy applications, and so on, so how to scale these benchmarks to any runtime
or any project using them to build a proper validation process.
So, what people want, I mean, what users of this validation system want, it's a simple
tool.
You don't want to have really complex Q, GTK, any kind of complex architecture to deploy
to test.
So, what we want is just a command line interface, basically, and really, really, really few
setup, really, really, really few configuration to deploy to have such test working, because
a lot, maybe it's not a generality here, but a lot of people hate tests, basically, so
we have to convince them that testing is good for software quality, something really simple,
but also able to handle any really complex scenario we may find in HPC, running an MPI
application is not that hard, but it may become really, really complex, and we don't want
to have to rewrite our, I use this tool every two years, because a new technology, a new
approach, a new paradigm is implemented, and the validation process should have to be rewriting.
So, from that state, we decided to write a tool to fit our needs, but to be able to be
used by people meeting the same requirements.
Before going further, I would like to tell you that testing is not a brand new field,
and some other projects are tackling such kind of issue until now, and so please take
a look at them, if you think that PCVS does not completely fit your needs, they are really,
really powerful tools in the field.
So today, I'm here to talk to you about PCVS.
It's a tool maintained by the CEA, it's retained in Python, yes, everything is in Python now,
so we are based on Python, it's a simple command line interface, and with few configuration
files, obviously in EML, it's a trend, the design of this framework, the testing framework
is to be, to bring simplicity when writing test logic to users, so we want tests to be
simple to write, easy to port, okay, and these benchmarks, written by the user, we want to
be able to be run in multiple environments, so we don't want to bound a test suite to
a given application, let's consider MPI run time, we don't want to have our Lulesh or
our IMB benchmark bound to a specific MPI implementation to be run on any kind of architecture.
So this is what we call in PCVS, the retargeting approach, the other approach we want to focus
on is the fact that we have heterogeneous test environments, we have benchmarks, how
to scale, automatically scale this benchmark to the actual test environment, consider,
for example, having users wanting to validate the IMB, but they are working on their work
session, we don't want to launch up to, I don't know, 100 MPI processes on their work
session, they're going to be not happy with you squashing their machine, but once this
validation system is run on a real HPC cluster, you want the test suite to automatically scale
to this supercomputer resources without having to rewrite 100 of configuration files, or
even one file is already too much, okay.
So this work is maintained by the CEA, which is a French research administration, and we
are collaborating with us, with them, I'm sorry, to make this tool more generic or not
generic and attempt to make as many users as possible, as many users as possible.
So at Eglance, some feature specificities provide, the idea is to split the test effort
into specific fields, the first one is the specification of test, what a benchmark need
to expose to build the test, so this kind of information is carried by the benchmark,
obviously, how to build, what is the program, what are the parameters, and so on.
And the environment, it's a testing environment, here is carried by the people deploying or
providing the testing resources, so most of the time it's just a team policy to schedule
jobs, or our system admins, right, all of this to pursue two goals, still the retargeting
of tests automatically when the user is calling this benchmark, depending on the compiler and
run time you want to target, and auto scaling test to test environment.
Obviously, as PCVS is a kind of test framework, we add to add some functionality around testing,
like in-place reporting, because most users are running their tests on HPC cluster where
the set of functionality can be restrained, so they don't have access to all their GitLab,
GitHub, Gira and so on stuff, so we add some basic tools to answer these needs, and beyond
a single execution validation run, we added a way to build the history of the validation
for a given application by storing results over time, and allowing the user to run simple
analysis, still in Python, to produce statistics over time.
So quickly, the architecture of PCVS, it's based on files, so Bison CLI, so Bison file
system, it's parsing some user inputs, and it's run through a dedicated connector to
mainly SLURM currently, mainly SLURM, but we are focusing on supporting as a batch manager.
So the idea is that the benchmark express job descriptions and resource requirements and
the environment will provide resources.
Let's consider, for example, a basic component called number of MPI tasks the job is taking,
the job will say, okay, my job is only running up to two processes, or it's only taking,
I don't know, cubic value of MPI processes, people are aware of users of the Lulech proxy
application will know what I mean by cubic, so it will describe constraints, and then
we will have environment configuration files called profile in PCVS, where the admin will
say, okay, in that context, I will have up to 100 nodes, so you will be able to spawn
up to 100 MPI processes to run your application.
Based on that, PCVS will cross this information and will say, okay, I have MPI jobs, I can
run up to 1000 MPI processes based on this specification, why not running the user benchmark
100 times once for each combination.
PCVS is an as an opt out approach, so it will consider that every combination provided by
the environment will be used to scale your tests, okay, so if your test is not able to
run up to 1000 jobs, it's up to you to specify that you can't reach this limit.
So here is a quick example, I don't know if you can see up there, but we have a really,
really basic environment configuration where you can see that there is what we call an
operator, this is a variadic component, and it can take up to 4 values for the NMPI attributes
and when describing a simple job, we're just saying, okay, my job just consists in running
a program, PCVS will enroll up to 4 common lines to run 4 independent tests to execute.
So PCVS will automatically build your test scenarios based on your specification.
So how to basically write a test, but more specifically a compilation test, there is
a lot of things to customize or you can build your tests, so it looks complicated, but as
you may see on the previous slide, all the keys in that TML are not mandatory, at least
the files one, obviously you have to specify which kind of application you want to compile,
so the framework will try to auto detect your language to select the proper compiler, obviously
you have a manually designed approach also.
If you are not based on compiling source files but already using a well-known build system,
we have also an interface to invoke directly build system to build your framework.
I'm thinking about, for example, Lulesh, which is using a Mac file, DIMB using a Mac file,
or even the Mpitch test suite using a Configure Mac-Macon style approach.
So you have many options, all of them are optional.
What I would like to highlight is a variant.
What a variant, it's a capability from PCVS to expose to job, to benchmarks, a specificity
or requirement this job has to be run.
So in that case, the OpenMP keyword probably means something to everyone here, but it just
you know a token saying that to run this job, the variant, the component OpenMP is required
to be scheduled and in the environment, the user will say, okay, what is my variant OpenMP
in case of GCC, GCC like compiler, it will be dash f OpenMP if it's Intel, it's not
the same option.
You see the ID?
I will add, PCVS will add Flavor depending on what you have specified in your environment.
How to write a run test.
So this is where the component, the iterative component takes place, we didn't port it yet
on CompilationModel because we have issues with the race condition reaching the same
file on the file system, but we are planning to containerize, to encapsulate, to isolate
such model to be able to support also CompilationTest, I'm sorry.
So what as a user have to do to integrate such test in your workflow, just write a PCVS.yml
file, you put it anywhere you want in your pass, I mean your benchmark pass, and it looks
like just a run node and everything below is totally optional.
Here we can see that we restrict, we restrict our add.out program to specific MPI values
because as a tester, I know that my test has this constraint.
You can even create this variadic component for your own application if you want to programmatically
generate a list of scenario that PCVS should integrate to its own process to build multiple
scenarios.
So in that case, with this, I'm sorry, why am I moving in the program node, you'll see
that with this three simple lines, it will be able to build three times the number of
scenarios that you were expected to have initially, right?
And what a test without having a way to express or validate a test, obviously.
So for any test, not only for run, you have a YAML description to say, okay, so I want
my job to exit with this particular return code, having an execution time within this
range matching or not matching this kind of pattern.
Even here is my script, give him the regular output of my test and I will tell you if it's
okay or not.
Okay, so I just write my test, how to run them now.
It's just celibate, so you just have to call PCVS run, but before running my benchmark,
I just have to create a profile to express the resources my environment has.
Obviously, in case of MPI, we provided some templates, some basic templates to initiate
the testing process.
So here, we are just creating a profile named my MPI based just on MPI.
So by quickly running that, I will have a full profile based on MPI running tests for
one, two, three, and four MPI processes, but from that, you can then expand the profile
to fit your needs.
The whole build of PCVS relies on a single directory.
And in that directory, you will find anything required to analyze the results and even rerun
in the same condition the tests for reproducibility.
You can see on the right, repository we provided alongside with PCVS, which is called PCVS benchmarks,
and we attempt to put in that repository many well-known MPI benchmarks, PCVS enabled, right?
So here is a fancy view of PCVS.
Obviously, there is many options when running a validation.
You can have an interactive approach, non-interactive approach, slur enabled, I mean, batch manager
enabled, running inside an allocation, outside an allocation, and once the whole configuration
has been generated, we have commands, especially a PCVS exec, to interact independently, uniquely
with your benchmarks, so for instance, what people are used to do is to run their validation
and after maybe 10 seconds, some failures appear, and they would like, without interrupting
the non-requestion system.
But rerun in an isolated environment, their tests to see why it failed.
So we have some extra commands to rerun this special pattern, okay?
Obviously, I'm going to just pick it up, obviously, everything is not always perfect, and the
static approach of the ML file is not what you need, you would like something more dynamic
because you have some stuff to interpret, to read, to process before knowing what you
want to test, even within a benchmark.
So we have a dynamic approach.
Instead of providing a static ML file, you will provide an executable script, an executable
file, whatever it is, and it will produce by itself the actual ML file.
This way, you can programmatically generate your benchmark suite without having to know
it in advance, without knowing in which environment your non-requestion base will be run.
Let's consider the NAS framework, where within the name of the binary bit, you have the number
of MPI processes, cool.
You know, we have run our tests, but we would like to see what it looks like, okay?
We have a test framework, not just a job runner.
So obviously, we had some tools to report results to the user, spoiler, cannot be compared
with real tools for that, okay?
And the idea is just to offer a way to users to grab their results directly on their machine,
in place, okay?
And essentially, it's just a way to look at tests at a glance to be able to rerun if
necessary in the process.
So as I said, we can isolate and rerun independently jobs, which is pretty convenient when some
failure want to be explored right away.
And in FINE, we are using a web server to report in a web browser directly, offering
more interactivity for your results.
So what it looks like, for example, here, gathered by Label, you can see that there
is some red, so some failures.
Let's dive into it.
You can see that some trouble with MPIO, what a surprise.
And when clicking a job, you'll see the complete log of this trend, so the command line and
the actual, so I truncated, I'm sorry, I truncated the actual error, and you can directly dive
into the error without leaving your actual SSH terminal.
So a quick overview of how to configure your site, so the test environment configuration
part.
This is also AML, you will define directly compilers, compiler and run times, and the
special variadic component here.
It's split in five different modules, why?
Because this whole profile can be split up to five blocks, independent blocks, we can
be distributed over a cluster because it's not always the same teams who are responsible
of this particular block.
Let's consider, for example, the variadic component, it's in charge of the team to build
this list, while for the compiler and run time, it may be in charge of the C side means
of the test environment machine.
And after running a single job, what I would like to see is to have a trend over multiple
run, all my test suite behave, and in PCVS we integrated a way of using to stack multiple
runs over time, and then run analysis on them, to build trends, to have more things than
just a test result, but a test result over time.
So here is an example of what you can do afterwards, running analysis directly on this
storized pass, and this is enabled thanks to, I would like to call that a DSL, but actually
just a Python API to interact with that, and you can build such beautiful graphics to see
over time the rates of success inside your test benchmark.
So finally, just a quick glance at the SPAC plus PCVS.
We are not in SPAC, but we are supporting SPAC, especially to do such things by specifying
a simple package, a simple spec package, we will be able to check any combination of building
this package to see if there is some curiosity into your package recipe.
For future work, we have many things scheduled, and most of the most interesting in the capture
in metrics, the capacity to PCVS to capture directly some metadata to be able to then
run analysis on them, and many other things, but I think I'm running out of time.
Thanks for your attention, I have two questions.
From your configuration file, I assume you already have control of the cluster, at least
you have allocated some nodes or something.
Do you have some step that then allocates and deallocates these resources on the fly
for each one of the tests?
So actually, currently, most of our test scenario has run through an MPI run with slurm enabled
or srun commands directly, so they are taking care of the resource allocations.
Some other users are just running the whole PCVS inside a given allocation, like resource
allocation, just a saloc, for example, and then any test doing srun does not pay the
cost of waiting the actual speed.
If some tests need some type of CPU, the other tests need other type of CPU, then you need
to, and if one of them is unavailable because an other user is using, you have to wait instead
of fail.
Yes, this is something we still haven't had a solution for, would be to be able to put
a job aside while we have the allocation.
Yes, this is something we are currently investigating, absolutely.
Do you have any questions?
Yeah, so one thing that I wanted to ask was kind of for your future work you had mentioned
building out a graphical front end using textualize, I was kind of wondering how much assessment
have you done into that, because I've done some work like trying to build GUIs with textualize
and while I do think that it's very interesting framework and it's great for making textual
GUIs, I think that it still has a bit of a way to come before it can really make a standalone
or comprehensive seal or a textual interface, so I was just wondering what your thoughts
were on that.
I'm not sure, I understand the whole question, but you mean, why did we choose textualize?
Absolutely, we discovered just recently because we were using Rich to highlight the output
of PCVS within the console and we are looking for a solution to present the things graphically
in a terminal and we still are looking for the ideal framework and as Rich is already
as textualize, I'm sorry, it's based on that, we are considering textualize, but if you
have any other offer to propose, I would be happy to discuss with you about that.
Thank you.