I'll get one more minute.
Sure.
You got some naked nathalus.
You got sick just kind of out of the blue and then like, didn't have any.
I thought it was.
It was so sick like, nothing.
It's sick sick or sick as it is.
Yeah.
I've never known.
Yeah, I've got the beer induced one.
I'm the last speaker and it's our hero that fills in for, you know, the.
Yeah, I want to thank one or two missing.
So yeah, take it away from.
Yeah, thanks, Bjorn for letting me fill in.
I heard I had wanted like a 15 minute session just to kind of advertise this framework because
I'd like to encourage people to contribute to it.
I ended up with a 30 minute or however long minutes the session is because of the cancellation.
I'll have an hour.
Don't worry.
No, no, no, no, there's lunch and I can do it four times maybe.
Anyway, so quickly just about who's standing in front of you talking.
I work for Bentana.
I work on Linux kernel, also KVM, open SPI and KEMU.
And I'm trying to build, you know, the software system that we need for risk five.
So I'm also participating in these RVI working groups and rise that we'd heard about earlier today as well.
Prior to that, I worked on air 64 before risk five air 64 red hat also virtualization.
So the Linux and the KVM bits KEMU as well.
I've carried over into the risk five world as part of the vert stuff that I did previously.
I got involved with Katie Munitess, which existed before my time because it's quite old.
But I started, I wanted to use it for air 64 specifically.
And so I did some ports.
I'll support it to power PC and then kind of left that for others to maintain.
I don't think it's getting a lot of action, but it's there.
And I'm bringing it to risk five.
And that's what this talk is about is the fact that we now have this tool available to us.
So the outline is just Katie Munitess.
First, I'll give a quick overview of the framework generally.
And then it regarding risk five, the use cases I see that we could apply it to right away and also as the framework evolves.
And then the the and you part is my kind of appeal for contribution.
So, so as I said, Katie Munitess is actually quite old.
It's as old as KVM.
Avi created it shortly after his first couple of KVM commits in order to start testing.
So to make sure it actually works.
And then over that time, though, we've we've been expanding its targets.
So now we can actually test not just with QMU as the user space is originally,
but with KVM tool or you could probably put in Rust VMM or whatever you want in there.
Cross VMM.
I mean, with some efforts, probably it doesn't just drop in at the moment.
But you can already test other hypervisors.
People do that.
And we can even test it on hardware now because we've added at least x86 and air 64 at this point,
the ability to boot over some sort of a if you capable boot loader.
So then what is this test actually that I keep talking about these KVM tests?
And so they're actually like a little tiny guest kernel because that's what Avi needed for testing KVM, right?
He needed to have a guest, a guest OS that would have to boot and maybe exercise some stuff that the hypervisor needed to provide for it.
So that's what they are.
These little guest kernels and originally, you know, kind of booted in maybe hacky ways or whatever.
But over the time, we've actually tried to build the framework in a way that is easy to port and easy to maintain.
And so we even have DT support in there, some limited ACPI support for this booting.
Like I mentioned, we can boot with CFI protocol, which helps us to be able to do the booting over hardware directly rather than through hypervisor.
And then for air 64 ARM and RISC-5, I've also taken my notes from the Linux kernels boot requirements.
So, you know, particular registers need to be set in a particular way when you first jump into the kernel code.
And so we follow that protocol and then it makes, you know, everything just kind of work for bootloaders that already know how to do that.
Any bootloader that can boot Linux in this direct way can boot these unit tests.
And so, yeah, you're in privilege mode because it's like a little kernel in kernel mode.
So you can do all the things that you would do, manipulate the page tables, set up your own exception handlers,
generate exceptions and make sure they do what you expected them to do, things like that.
You know, you're privileged, so go nuts.
So, despite the fact that we're actually writing kernel code, we don't have to make it complicated.
We don't have to make it something that's hard to do or at least feel hard to do at first look.
So the framework tries to allow the unit tests to be written in a C-app type of way.
So you kind of look and feel that way.
You've got your main function, which is actually the entry point for the test.
And then we have a bunch of libc, api, ported over, not a bunch, but enough for most tests.
And we are, you know, of course welcome to add as necessary, whatever kind of looks like it's needed.
So all your expected ones, assert is there, which is, you know, of course one of the most important ones for a test framework.
Also, with the scripting wrapped around these tests, when you execute them, at least over QMU,
then when you do, when you get an assert or any sort of an unhandled exception,
you actually get a back trace for all the ports in a way that support stack walking.
So we have that, and then this is just a little snippet of code to show you that, you know, don't be afraid.
It's just the, and very simple. See, it's just main, even environment variables can be provided to the unit tests.
For that, we do a little trick where we take a text file of environment variables.
So, you know, your usual key equals val, just a whole list of those.
And we put them into an NDRD, so they're in RAM disk.
And we can just read them out of there, and we can find it through the DDE, all that stuff, just like we're supposed to.
And then we can load those environment variables into memory, and you can use them like a normal C program.
So that can also be nice for passing in your expected values and whatnot for unit tests.
You can also pass in expected values for the command line, of course, which is a little bit easier to do.
But it's, you know, if you have too many of them, it gets kind of ugly.
So, of course, you can also, for at least people who want to test on hardware, they're free to manipulate their device tree in any way they want.
So they could create a special node for test cases, sure, why not.
And then the unit tests would just, you know, parse that node and get all their input.
However, however you want to do it.
So how do you run the test?
So originally, it was, you know, from the command line just for running KVM Guests.
So that still, of course, works.
You can just pass the test, you know, as a kernel.
That's the kernel parameter to continue.
Depending on which KVM user space you're using, you'll do it in some similar way.
There's also some bash wrapped around all of that stuff.
It allows you to run all the tests for you automatically so it can be built into CI very easily.
And we do have it built into many different CI already.
So we run just a single group.
And then the reason is bash.
I mean, some people wonder why, because it gets kind of awkward to add more advanced functionality to the test harness having to write it in bash.
It was historically in bash, is probably the main reason.
But then we actually had a discussion a couple of times, like should we use Python or whatever, go whatever the latest thing is these days.
It's a little bit easier for the harness.
And we had some pushback from people who have been using this framework quite a lot.
And they like to have a very lightweight framework that they can put on an embedded, you know, busy box type thing.
There's nothing there except for bash.
And they didn't want to bring in libraries and everything else for something else.
So bash is not that painful.
We don't have that much functionality.
So I don't really have a problem with it.
Another thing we can do, we can build standalone tests with it.
So nothing changes except make space standalone.
And it'll actually wrap a lot of that bash around the binary after it converts the binary with base 64 to be embedded all into one nice text file,
each text file depending on how big your test is.
And you can actually just email that or send it to people.
So if you build a quick and dirty test, and I'll get to talking about quick and dirty tests a little later in the talk,
if you do that, like, you know, a few lines is to like prove your point that this is broken.
Then maybe you just want to package it up with this make standalone thing and mail it to somebody.
They can run it and see for themselves.
I don't think that's used a lot.
That was one of the things I invented that I thought would be useful,
but not too many people have been mailing these tests or whatever.
So now we know what the framework is, and this is a risk five talk, so we finally get to risk five.
So we already have a use case for it.
The tech PRS working group has more or less committed to using it for the SBI verification framework.
So the SBI for those of you that don't know, I guess most people in this room do,
is this interface between either supervisor mode and in mode, machine mode,
or also between a virtual supervisor mode and hypervisor.
And so we, you know, we either respect community or trying to keep that interfaces from going nuts
in all sorts of different directions.
We have a standard for it, the SBI spec.
And so we write when we want new functionality that we need,
the supervisor needs to ask for some service or some information from in mode
or we want to emulate that in mode for the guest.
Then we need to provide this interface, right, this SBI.
And so as we add these functions to the spec, we explain how in the spec it's supposed to work,
the parameters, etc., like usual.
Then it would be nice to be able to have a verification framework for that
so you also say, okay, you've written a nice, you know, addition to our spec,
a new extension, SBI extension, please show us, you know, how it's supposed to work.
And you could do that, and we do do that with Linux proof of concept codes.
We always submit patches for Linux and also for open SBI or Rust SBI that show that, you know, it works, right?
We prove our extensions.
But with the verification framework, we can actually avoid having to any,
focus on any specific projects or people having to involve an entire Linux kernel for the test.
They can just do this quick and, this quick small thing here.
And so that's the idea is to try to build all those function tests in there
and have regression tests for that as well for everybody's SBI implementations.
So we can test already, right now you can start writing tests for open SBI.
It's quite easy to run over QMU, you don't need hardware for that.
You can actually, with QMU, you can swap out open SBI and drop in Rust SBI.
That also works over QMU.
Probably other SBI implementations can be run from QMU.
Of course KVM is a SBI implementation because it emulates, so you can already test that as well.
That's one use case already, which could be started now.
So, we have a CPU validation as people actually have CPUs to validate.
And when we get the EFI support merged.
So I haven't done that yet.
I'll come to that too, like with current status.
But as we, when we get that done, then you'll be able to just put these tests directly,
boot them from U-boot or CBEFI and you'll be able to do some validation tests.
So ARM does that, I'm quite aware, because they've been involved with KVM unit tests for a long time now.
They're doing their memory model, litmus testing, they use KVM unit tests using the EFI support to go straight on hardware and run that.
So microbenchmarks are another great use case for KVM unit tests because while you can always find a way to create like some sort of a privilege level test where you write a kernel module in Linux
and then you like put it, I used to do that a lot, just like in the init of the module, I would have my whole test case and then I'd, you know, I just mod probe it and now it runs my test, right, that privilege.
But, which is kind of awkward to begin with, it's not a real test framework.
But it also requires Linux to be booted up and working and everything.
And it's not very good for a microbenchmark because you've got Linux doing whatever Linux wants to do.
And so you're not really isolating your instruction sequence.
But with KVM unit tests, you know, the world is yours.
The unit test is running there and nothing else.
So it's actually quite good for that.
When you get your timing numbers from that, they're pretty reasonable to trust.
Question.
Yeah.
So in this diagram, what does the test say?
Ah, yeah.
So the test is either this guest kernel or actually the host kernel.
It's one of those two.
So if it's bare metal, if you just launch it from the boot loader, you'll be the host.
That support isn't in the RISC-5 port yet.
But you can already do the guest kernel version.
Okay.
So, yeah, the tests are easier to write as we already talked about.
And the quick and dirty ones are even easier.
I do, I do the, so, so I do this a lot.
I actually, because I'm familiar with the test suite, I use it for a tool while I'm working on something else.
Like something for Linux or whatever.
I use it just for my own testing purposes.
And then it's kind of ugly and it doesn't really look like people would be maybe interested anyway.
It's too, like, one-off.
And so I just kind of toss it.
Or maybe I keep it for myself to look at later, but it's not shared,
which isn't really a very good open source approach.
So I've actually been thinking about that, that for these types of tests that don't really necessarily fit what we consider the main test suite,
maybe we should have a separate branch though for them.
So we still collect the code.
And I kind of did that already.
I recently wanted to test TCG.
So I kind of forgot to mention that for CPU validation, we already can, of course, test our, you know, emulators and our other models to see if they're correct.
So TCG is, you know, QME's emulation framework.
So I wanted to make sure that the MMU model that it had was able to handle the access to dirty bits correctly,
because there's actually a couple different ways to do it in spec.
And QME had picked one by default.
And then a couple extensions came along that actually allow you to decide which one you're going to use.
And a new bit was added, which is actually going to require another SPI call.
So we'll go back to the SPI verification for that.
Anyway, kind of, you know, balloons as we know.
And I wanted to make sure it was actually working the way it's supposed to right now.
So I wrote a test case in KVMunit tests.
And then I wasn't sure, okay, this is maybe not the one that we're going to merge because it's just for this one-off test.
But I've already decided maybe at least goes to a branch that we should keep track of these things.
And then, you know, and the other reason why posting them, even if they don't get merged in the end,
or at least not to the main branch, but to the side branch,
is because when people do post-tests, sometimes they reinvent something they need inside the test case to get the job done.
And that looks like something, oh, we should probably pull that into the common code, right?
We can let the framework evolve better the more people who contribute.
And there's no one and done.
Usually I write something, some quick and dirty test, and then like three weeks later,
I'm like, oh, yeah, I actually need that again because something similar is broken or whatever.
Yeah. I think I talked about everything on this slide.
Those are some links.
And, yeah.
So one thing I was going to do, because I have way more time than I need,
but I was just going to show that test that I just got done talking about.
So it's a little bit more complicated than that little snip that I shoved in the slide.
So you can see that it's still not that complicated, right?
Oh, yeah, sorry, everyone can try to brighten the screen somehow, maybe.
Yeah.
I don't know if I can turn off the light. Just smash it with a hammer.
Yeah, it's probably, you know what, maybe I can go to a black background and just cut the file.
It might be better.
Is this better than before?
Yeah, because black background is better.
Don't touch that.
That sounds like fire hazard there.
Anyway, so I'll just, you know, just kind of slowly scroll through it, I don't know.
Just to show you that really you can build these tests with like 100 lines of code,
and they achieve a pretty reasonably good goal, like making sure that an NMU
behaves correctly in like three different modes.
So, yeah, so I don't know if there's any particular lines here I want to point out,
so I wanted you to get a feel for what a test would look like if you guys decided to go sit down and write one.
You don't have to like, you know, you don't have to learn a whole big framework with some bizarre looking APIs.
The APIs that we have are minimal to begin with, so you're going to write your own functions.
But when you do need them, you know, they're pretty self-explanatory and C,
so you just, you know, you can grab for anything you need to know.
And yeah, that's the bottom of the file already. It's only like three page downs.
So, um...
So, does the actual return value get used? I mean, I noticed you're carefully returning a report summary.
Yeah.
But does anything actually look at the return value of this, May?
Yeah, so CIs will do that. So, like, this will dump a summary to the screen.
So, if you're just running it yourself, which I guess I might as well go ahead and...
So...
Yeah, you know, I'm feeling brave.
But, um, so, yeah, you can just run it.
And then it'll dump...
Yeah.
It'll dump stuff like this out.
And then CIs will, they know how to parse that, right?
So, they'll be looking...
And we have the...
We have those, you know, reports and report, pass type API to try to make sure you get a nice, uh, consistent format so that it's parsable.
You know, we don't use a TAP. Maybe we should.
We've done that in a different test suite that I'm involved in as well,
KVM self-test that's in the kernel.
We're starting to...
We're not there yet, but we're starting to migrate the TAP for that one.
Uh, yeah. This one we have our kind of our own thing going. We've had it a long time now.
Um, anyway, so that's like one and then there's like this...
Yeah, there was another test. You probably saw it said skip.
And it's skipping because I didn't give it an environment variable.
Uh, let's see. Yeah, that's the file.
So, this is that text file I mentioned before. You can create just, uh, you know, plain old text with all your environment variables.
And then when you want to pass it to the thing...
Um,
Oops.
It passes...
Like this.
And we'll just run that one group of tests this time.
We're seeing about live demo is I have to type in front of people.
Um, and, uh, so now we, now we're not skipping anymore.
Now we're passing because I gave it, I gave it the inventor ID, which is zero for KMU.
Um, and, uh, it matched.
Working demos, working passing, passing SPI test.
Uh, yeah.
You showed the failing test also.
Oh, yeah.
Yeah.
I want to see that it's true.
Yeah.
Good challenge.
Uh, yeah.
Forgive what I called this one. There we go.
So, yeah, this is that other one was the, uh, was the, uh, in a new testing.
Um, oh, yeah.
And so now it's here.
It is failing.
It's skipping.
Um, that's failing, but skipping.
And that's because, uh, this, uh, CPU, the default CPU is missing the, um, the extensions needed.
So we can, we can fix that, of course.
Um, something like this, um, we can actually add, um, we can add the extensions.
So, uh, still spot.
It was not there because I don't know why.
Oh, no, because that requires an extra, uh, extra step of adding an SPI implementation that allows you to turn on, uh, the, uh, AD bits, um, uh, the hardware AD bits, um, where you don't have that yet.
That's actually, we need to add an SPI extension.
I think we're going to call it FWFT, allowing us to tell SPI to flip, uh, bits and registers the machine, uh, environment, config, enable bits.
Because if you want to turn on this particular feature, uh, you need them, you need, uh, to be at the machine, uh, uh, mode level to be able to do that.
So I can't do it from, uh, the s mode level.
And so I actually hacked OpenSBI to let me do it and to test this out.
And I, I'm not going to look for that in a live demo file.
But, um, yeah, I have that.
It does work.
Yeah.
Um, yeah, I think, uh,
So what's in the run test.sh?
So you wrote the C file, right?
Uh, again?
So then you had this run test.sh?
Yeah.
So, did you write it as well or is it, so, or is that the test?
Okay, run test is just the, the, the test suite that kind of pulls everything together.
So, um, if we look at, uh, this one, for example, this on the screen, uh, the log here shows at the very top.
Which is at the bottom of the screen.
Uh, this, this time out, et cetera, et cetera, et cetera.
So that's actually the command line, the run test, figured out how to compose, uh, based on some configuration files and stuff.
And then this is the output of that.
There's, uh, this, um, uh, configuration file that you can, uh, provide, uh, for your groups of tests or for individual tests.
Uh, allowing you to, um, um, to tell run tests what to do to pull it all together.
I mean, of course, you can also just manually do the command line.
And I do do the manual QMU command line, uh, when I want to also, like, do something with GDB or, you know, make sure I get the, the address is dumped out and I can find them as obj dump or something.
So, um, yeah, I don't always do everything through run tests.
Actually, very rarely. That's more for the CIs after you've got the thing working.
Which one?
No, that's already there. That's, that's a static.
Yeah, it's committed to the repo.
Yeah. Yeah.
Um, yeah, nothing for scripts is automatically generated except for when you do the make standalone.
And then you get, uh, might as well show that because we're in demo mode now.
Um, so then you get this guy, which is generated.
So this batch script was automatically generated.
All this junk is the, uh, base 64 of the actual test code that was written in C.
Um, yeah.
And then, you know, this, some of this stuff is just kind of extracted directly from, uh, other scripts that are used by run tests and they're just chucked in there.
And now you can, now it's all one unit.
Yeah, you could put anything in there. I mean, don't trust someone to send you a reproducer.
Yeah, it could be like, sure. Yeah.
This is for developers passing things among trusted friends.
Yeah. Yeah.
Yeah. Make them sign it or yeah, just, yeah.
Sure. Yeah. Yeah. I mean, yeah, anything could, absolutely anything could be in there.
Right.
Like enter your password. Please. Thank you.
Uh, those tests are very similar to what is a case of test does and, uh, those tests integrated to case of test and if no, do we have such plans?
Yeah. So the question is more or less, how does this relate to his, uh, case of test?
Yeah. So there's, there's definitely overlap in what is tested.
Um, the frameworks are quite different and how they work.
Uh, there's, there's more overlap between this particular one and KVM self test, which are in case of test.
That's one of the many sub directors in there.
Uh, KVM self tests, KVM case self tests, uh, has started to be probably, uh,
be probably the, the main place we add new tests for KVM.
So actually, you may have noticed I did an entire presentation on KVM unit tests and I think I said KVM like only when I said the name of the framework,
but I never actually talked about testing KVM.
Uh, we do that still.
Uh, we have CI that's specifically testing KVM using this framework, but, um, now we usually use KVM case self tests for the new ones and even some of these reporting to that framework.
Um, and I'm seeing that this one's more going towards the testing of hardware or other hypervisors are still using it and stuff like that.
But, um, yeah, KVM wise, uh, and, and, and actually I talked to Paulo about that yesterday, uh, on my third beer or whatever.
But, um, I was like, you know, KVM self, the case self test is, is the way for the future for KVM testing and I'm not going to really talk about it too much tomorrow.
When I talk about KVM unit tests and say, ah, but KVM unit tests are still easier to write and he's right.
Like you can write a test case, uh, quicker, faster here.
So if you're doing KVM testing and you want to do those quick and dirty tests I was talking about, uh, you might jump to this one first because the other framework, uh, uh, well, it's growing like support quite fast,
but you have a little more boilerplate code and everything you have to do because you're actually, when you write that you're writing both the user space code and the guest code simultaneously for a test.
And here you only do the guest code.
So they initially, uh, here we can just simply read a test and if it's worse then we can move it to a case self test with bigger overhead.
Yeah, yeah.
And for your question on the other case self test stuff, like risk, there's a risk five directory there too, right?
Where we test some instructions.
That stuff is good.
We need that too, but it's user space only, right?
Yeah.
So this, this is down in the kernel level.
Okay. Thank you.
S mode.
Any other questions?
No, let me, let me appropriately go to the last slide.
There.
Thank you.
All right. That's it. See you next year.