Okay, now we come to it.
Alright.
So last talk.
Alright, testing, testing.
Okay, so I started here.
My talk is about developing effective testing pipelines
for HPC applications.
Just to give a short introduction about who I am,
relatively new here.
So, you know, I first started in the HPC industry while I was in college.
I joined my university's HPC Institute as a software consultant,
so largely in that role.
I was kind of more focused on support.
So in that case, I was working on, like, singularity deployments,
you know, helping debug Fortran.
Not very good at writing Fortran, you know,
helping build custom kernels for Jupyter, all that good stuff.
And then I left for a bit to go work at another university
to do some adversarial machine learning workflow orchestration framework.
And then, eventually, I came back to my university's HPC site
then as an engineer, instead of a kind of support role.
So more in that case, what I worked on was, like,
building, you know, a suite of singularity containers
that clients could consume,
and then also working on, like, debugging tools
that interfaced with, like, Moab and Torque and PBS and all that.
And then after I graduated university, I joined Canonical.
So that was probably, like, nine months ago now.
So, you know, kind of the start then, you know,
what sent me down this path of wanting to develop
effective testing pipelines for HPC applications?
Well, first, when I started at Canonical, to be frank,
I wasn't very good at Debian packaging.
I had mostly deployed most of my software using bash scripts,
compiling from source.
And so part of that is I had to write provisioning scripts,
so I mostly work in cloud orchestration,
like, nodes and whatnot, onto clouds.
And so I had to, you know, develop dangerous code, you know.
So it's installing packages, making configurations,
kind of setting pre-seeds in, what is it, Debian, Ubuntu.
And so basically, I kind of had this, you know, dilemma of where,
okay, I want an easy way to test these provisioning scripts
without kind of having to go through all this manual effort.
So originally, you know, I just, you know,
typed in vagrant virtual box or any real kind of virtual machine
or a supervisor, bring it up, install the script,
and then once I was done with it, blow it out.
And then kind of also have this issue of where I had a desire
for reproducible tests, because even though, like, you know,
I know how to bring up, like, a virtual machine on my system,
you know, I might be working with someone who's off of Windows
or Mac, and so they don't have the same setup that I do,
or they're using a different distribution, in fact.
So this gave me the idea of, you know, kind of,
if I'm on my personal workstation here, so my laptop,
unfortunately, my current laptop's X server isn't working,
so the HDMI cable gets all screwed up,
so thank you, EasyBuild folks, for lending me your computer.
But basically, I had the idea that what if I take a test
that's written on my system and then have the ability to run it
using any hypervisor I want on any operating system
that I need to test that code on without any extra hassle
of having to really go through supporting
or teaching someone else how to bring up that instance.
And basically the case of, like, oh, I have to write code,
or run or anything else that needs to run,
like Ubuntu, CentOS, Alma, or Rocky.
And then kind of after working for that a bit,
you know, I had kind of an initial prototype
that would bring up, you know, basic operating system image,
and then kind of as I, you know, got more into my HBC work at Canonical,
I kind of realized, like, oh, okay, you know,
maybe don't want to rack up the cloud bill trying to test
and deploy to AWS or, you know, Azure,
or, you know, even our own private internal public cloud
or private cloud using OpenStack
for, you know, just trying to provision HBC nodes.
So, for example, have, like, identity management,
open LDAP, shared file system using NFS,
you know, have other options.
And then also working in the case of just setting up
and configuring a Slurm cluster, all kind of headless
without any kind of manual user intervention.
And so I kind of had this revelation where I'm like,
why waste precious compute time on your HBC cluster
because the fact is it's expensive, you know,
you're kind of paying for that tenancy.
And what if you could test your simulations,
jobs or applications and whatnot on a mini HBC cluster
that's kind of similar to the target platform
that you're deploying, but it's more of a mock
so that you can kind of get the general feel for the platform
before moving on to the more expensive resources.
So in this case, you know,
I started working on a custom testing framework
called CleanTest, which is basically a fancy Python testing library
that allows you to bring up many HBC clusters
on your local system and, you know,
just kind of in general usage for developers
who are in a hurry.
So kind of then breaking into a question, you know,
what exactly is a CleanTest?
Because, you know, originally I named it SimpleTest
because I just wanted it to be really dead simple,
but for some reason PyPI wouldn't let me register that name.
So CleanTest it is.
So basically it comes in three parts.
There's kind of a different breakdown.
First, you have the bootstrap and configuration stage.
So for that, it's kind of where you can register hooks and whatnot
for configuring the instances that you're bringing up
or do some more advanced bootstrapping.
So the example I had shown in the previous slide
of where you're able to bring up NFS,
open LDAP, slurm services and whatnot,
and then be able to inject scriptlets into that.
And so kind of that's when we get to the second part here,
which is a Tesla.
Kind of interesting little word that I came up with,
joking with some of my colleagues,
but basically a Tesla is an entire Python program
that's wrapped into a regular function,
and kind of the idea is that they contain the full body of the test
that you want to be able to run inside this virtual machine container
or, you know, test mini-cluster.
And then kind of the last part here
is the evaluation reporting aspect.
And then kind of with that, you know, I took a test, you know,
subtest framework, you know, agnostic approach
where I know that everyone kind of has their own taste that they like.
Some people like pie tests, some like unit tests, you know.
I don't want to sit here and make opinions for you,
so, you know, I want you to be able to write tests
in the format that you're most comfortable with.
Oh, that came out small, but that's okay.
So kind of the first part here
is the going more advanced into, like,
what exactly is the bootstrap in configuration
is that you're able to bring up example nodes,
you're able to provision them.
What is it? You can also register hooks,
so it's kind of similar to anyone who's done any Debian packaging.
Listening to Todd, I don't think he's still here.
Yeah, you seem to have some experience with Debian developers.
So, but...
How do you, like, circuit 2,000?
Yeah, circuit 2,000, yeah, it's cool.
But, yeah, so if anyone's ever, like, worked with P-Builder before,
kind of one of the features that I really like about P-Builder
for building Debian packages is that you can have different hooks
that you can ingest, or you can create that run at certain parts
of the package build.
So I kind of wanted to replicate that same functionality.
And then kind of the general process
is instantiating a configure instance,
so that's, like, a Python singleton class object
that basically takes in all that info, shares it across,
you know, the whole test suite.
They can bring up nodes that you need,
and then you define your hooks,
so that's kind of, like, what do I want to run
when I start the environment and then register them
so that, you know, when you run the test,
the program knows about it.
And then kind of the next part here is the testlets.
So that part kind of uses some little bit more tricky programming.
So in this case, I use Python decorators and metaprogramming,
so kind of reading class descriptors and whatnot,
assessing the current state of the program.
And what the decorators do in this case with the metaprogramming
is that rather than having that function run locally,
instead it kind of takes out the body of that function
that you defined.
And then it injects it inside of the container instance
and runs it there, so the idea is that, like,
oh, you could be working off of a new bunch of machine,
but you're developing for a cluster that's running Rocky Linux.
So in that case, you can just easily bring it up,
inject it in there, get the results back.
And then finally, kind of the last part,
evaluation slash reporting.
It's like each testlet kind of returns this results object,
so that kind of contains an exit code, standard out, standard error.
And from there, you can evaluate the results locally
instead of having to kind of instead do it inside of the container,
so you could say, like, oh, in this case,
if you're, like, doing a spread test,
so kind of if you have, like, oh, you know,
I need to test this on Ubuntu 22.04 or 20.04 and 18.04,
it returns a generator object of a name and a result.
So, you know, let's say that your code works on 22.04 and 20.04,
but the version of Python on 18.04 is too old for what you're trying to do,
so report back is an error.
So kind of then, you know, breaking into how does it exactly work.
So the idea is that you kind of start on your local host,
so that's kind of your computer there.
So you have the host operating system,
and then you have the clean test package installed
as, you know, I'm part of your Python interpreter,
which is a regular Python package.
The idea is that then, as you see that dotted line in the middle there,
it then makes the request of the hypervisor of your choice
and tells it, like, hey, so, you know, the user who wrote this test
told me that I need to bring up a certain instance,
you know, says that they need, like, a centOS image,
so bring up a centOS image for me,
and then once that's done, you know, what clean test does
is that it takes that test body function
and it kind of creates a simple JSON packet,
which is a checksum to verify the authenticity of the testlet,
the data, which is basically the testlet encoded,
and then, or any data necessary for the testlet,
and then the injectable, which is basically, like, hey,
you know, when you get this data packet,
here's what you need to do with it.
And then once that happens then, you know, clean test,
what it does is that it copies itself onto the container image
and then from there it ingests that data packet,
does the evaluation that you requested,
and then it returns that result object back to the clean test
that's on local host.
So kind of two different ways that it works.
Then for, like, how do you control the hypervisor?
The first way is kind of Archon,
which is a fancy word for director, you know.
I kind of wanted to have a buzzword in there somewhere.
But the idea is that what the Archon does
is that it's kind of more declarative approach
to doing clean tests, so rather than, you know,
saying, like, oh, you know, automatically do this, wrap it,
you know, you can kind of direct the deployment of said
mini-HBC cluster, and then what Harness does
is that instead of, you know, having to explicitly declare,
like, this is the infrastructure that I want for my deployment,
it just brings up an instance based on the function
that it's been wrapped around.
So this is a short demo video.
Let's see if I can choose better quality here.
Oh, don't tell me.
Came as a PDF, but let's see here.
All right, YouTube, sweet.
They go full-spring. There we go.
Oh, settings, playback.
180p. There we go.
It's a little interesting.
But yeah, so basically what happens here is that
just using, like, simple talks, in that case,
I use talks as kind of the test administrator.
I started a test, which is called LSD Archon,
which basically says bring up a test environment instance.
So first it starts with LDAP,
so it's provisioning an LDAP node on top of LSD hypervisor.
That's what I'm using here.
So first it starts with LDAP, and then after a few minutes,
for it to boot, crappy hotel Wi-Fi was when I was doing this.
And then see there?
Now you have the NFS image.
That starts provisioning, and then somewhere in here.
Now you have the Slurm CTLD node that comes up.
And now you have the Slurm,
three Slurm compute nodes that come up,
and the idea is that then what the framework does
is that it injects a testlet inside of Slurm CTLD,
and then from there it uses SBatch
to submit the job off to the test cluster.
Takes a bit.
Ooh.
A little too far ahead.
Give it a few seconds.
Yep, and then it cleans up the cluster
so that it doesn't linger on afterwards.
Oh, goodbye.
Oh, God.
What happened here?
All right.
Okay.
That's not fun.
Okay.
Yeah, I want to go back to the video.
I don't know why I jumped into the other video.
I just had an auto play moment.
Wow.
I feel like a school teacher.
This is right for the end.
Okay.
All right, thank you.
Yeah.
All right, okay.
Ooh, no auto play.
All right, so basically what happens here,
I'm going to full screen it so it's a little bit bigger.
What?
Come on.
I'm an engineer, not a YouTube video player
on a projector kind of guy, but what is it?
Yeah.
So what I've seen here is that the test starts,
brings up the nodes that you need to use.
So in that case, it was just LDAP for basic identity management,
manifest for shared file system,
and then just like Slurm for kind of resource management services.
And then from there, it just like injects like a little test script
to run.
And then, yeah, what's if the job succeeds,
it kind of copies back to the results.
And then, yeah, and then it says like, okay,
we get the result that we expect, so in the case of the test
that I wrote, it just prints out basically like,
I love doing research, and then it says like,
I love doing research in the log file for standard out.
So, yeah, pretty low fidelity right now is mostly a lot of work,
went into just getting it to work and all that.
Okay, now I want to go back to the slides.
There we go.
Hey, so now you saw that video, you know,
kind of overgoing some of the current limitations.
The first is that I'm kind of bad at playing YouTube videos
and presentations, but the next part here is that kind of,
right now, big issue is that there's kind of a lack of
robust multi-distribution support.
So currently, I mostly developed it to work on Ubuntu
and work with Ubuntu, I wonder why.
But you can't launch Alma, Rocky, CentOS, Arch instances, et cetera,
but kind of the macros, hooks, and like utilities that I built
into the framework aren't really fully there yet
for supporting it.
And then public documentation is behind because, you know,
usually I write code before I write documentation.
Unfortunately, it just seems to be how it always goes.
Yeah, so I need to update that.
And then kind of big issue right now
is lack of package manager integration.
So a lot of the support has been added ad hoc.
So currently, I support like charm libraries,
which is something that Canonically uses,
and then snap packages, which are kind of controversial,
depending on your opinions.
And then also just pips because I do a lot of Python development.
But in the future, I hope to add support
for like Debian packages, RPMs, you know, Arch installs,
it's back in EasyBuild.
So, yeah, and then lastly, I'm the only developer currently.
So, you know, code developed in isolation isn't reviewed
as thoroughly as it could be.
So, you know, yeah, I make design choices
based on what I think is appropriate.
So, yeah, so last thing too is that, you know,
testing framework I think is a lot cooler
than the video that I kind of struggled to play here.
So if you want to scan the QR code,
if you're interested, feel free.
Slides are also online as well.
So if you're not available right now, you can check it out.
And then lastly, you know, this is kind of a, you know,
call for involvement.
So, you know, really, at Canonical,
we're trying to start getting Ubuntu kind of geared better
for HPC, you know, we kind of know
that we're a little bit behind Red Hat
in the case of like network driver support and whatnot.
So, you know, just if you're interested
in using Ubuntu for your workflows and whatnot,
we have a public Mattermost channel
so you can scan that QR code or you can, you know,
check it out later.
But, yeah, we have a public channel for HPC online.
So, yeah, if there's something that's missing
or, you know, there's kind of some reason
why you're being held back on using Ubuntu for HPC,
we'd really love to hear that feedback.
So, yeah, that's it.
Thank you.
Any questions for Jason?
Last chance for today.
So, you're just doing, this is all Python code
that does this system for you.
Yes, yes.
So, how are you spinning up the LXD containers?
So, the idea is that, in the case of LXD,
it has a public API socket that uses,
I think it's like open API standard or something.
But, yeah, so you can make, like,
HTTP requests out to that API
that basically say, like, oh, you know,
I need this instance or tear this down
or set this configuration so I can install, like,
AppTanner or some other container hypervisor
inside of LXD and whatnot.
So, yeah, so I'm just using the API.
Any other questions?
So, yeah, I'm interested in this.
So, the, like, you had an NFS server
and some other stuff.
Are you, how are you, are those preassembled images
that you're just using?
Are you building up with some kind of configuration management
to use Ansible to build them or how do you do that?
Yeah, so currently what I have is,
like, the way that I provision,
I'm using, like, base Ubuntu images
for configuration right now,
but you do have the ability to register your own custom instances
and pull them in as well.
But basically, I have a little mechanism
built into the framework where you can write,
like, provisioning scripts using, like,
the clean test utilities.
So, there's some stuff for, like, installing apps,
running commands on, like, sub-processes on the unit
and then also, like, you can reach out to the network,
download anything you need.
So, yeah, I'm just using custom Python scripts,
the provision that are...
Yes, yes.
Anyone else?
Last chance there?
All right.
It's for the stream and the recording.
We've been doing well all day.
Let's keep it up.
Thank you very much.
Would you mind please to move to the previous slide
so I can scan my QR code correctly?
Okay, thank you very much.
You're welcome.
That was a good last question.
The other one?
Which one are you looking for?
This one or...?
Yeah, this one.
Okay.
That's all good.
All right.
If there's no more questions, we can wrap up here.
Thanks a lot, everyone.
That was a wrap for today.
The 9th HPC Dev Room at FOSDEM,
that means if FOSDEM likes us,
we can have a 10th one next year.
That would be really nice.
Some practical stuff.
If you're leaving the room, if you see any trash,
please take it with you and dump it
in an appropriate place outside.
And the FOSDEM team has asked us to ask you
to leave the building as soon as possible
so they can lock up the whole building.
There's another talk going on in Janssen,
in the really big room.
I would recommend going there.
It's a really nice way to wrap up FOSDEM,
and there will be places there to get one
or maybe two or three more beers
and have some more chats with people.
Thanks a lot, and hopefully see you next year.
Thank you.