All right, we'll get started.
We have another talk on MPI, but I think a very different one, running MPI applications
on the Toro Unicolon.
Exactly.
Yeah.
So, hello, everyone.
I'm Matthias.
Here, I'm going to talk about running MPI applications on Toro Unicolonial.
Usually speaking, a Unicolonial is a way to deploy a user application in a way that is
closer to the hardware by trying to reuse the operating system interference.
So, in the overall, it should perform better than just deploying a user application by
using a, during our proposed operating system.
First, I would like to introduce myself while I am passionate about operating system development
and mutualization technologies.
I had worked at Citrix, to take Huawei, and I'm currently a Baptist, and here I have my
email and my GitHub profile, if you want to know more about why I'm doing it.
So, I'm going to start to present what is exactly a Unicolonial, and then I'm going
to go to the details of what makes Toro special, and then I will show current implementation
of the MPI standard on top of Toro, and I will finish with a benchmark that I'm trying
to do to see if the current implementation is working as expected, or if there are things
that could be improved.
So, maybe you are already familiar with this picture.
This is more or less how a user application is deployed, either using a built-in machine
or bare metal.
So, what we have is the operating system, the user application, and the two different
modes, the RIN 3.0, which is the different modes in the X86 processor.
So, in general, what we have is that when a user application requires one to open a
file, send a packet, or whatever, it's going to trigger a Cisco, and then it's going to
be a switch in which the processor is running from, which is user space to kernel space,
so it's going to be processed here, kernel space, and come back, right?
In general, when we see what we have inside the kernel is, well, we have different components,
right, that, for example, have the scheduler, the file system, different drivers, and so
on.
So, in particular, when we have a scheduler, a scheduler is going to choose what is the
next process that's going to be executed.
One of these processes, or several of them, is going to be your MPI application, for example.
So, if you deploy your MPI application in a, by using a general proposal, but as a system,
your application is going to compete with other processes in the system for sure.
And also, what you have in the scheduler is some policy, which is going to decide which
is the next process to be deployed.
Also, we have components like the file system, and since we have a general proposal processing
system, we're going to have several drivers for different file systems, and different
drivers, and so on.
So, what some people observed was that there were too much generality in using a general
proposal processing system for a single proposed application, like, can be an MPI.
So, some people come up with a new architecture, they propose what they call Unicolonial.
You have some projects there, like OSV, Mirage OS, Unicrash, or Nano VMs.
What they do is just take the user application and compile it within the kernel itself.
So, at the end, what you have is a single binary that is going to be deployed, either
by using a virtual machine or a bare metal, right?
So, instead of, for example, having syscalls that we have in the case that we have a general
proposal processing system and different modes of execution, in the case of a Unicolonial,
we have simply calls, which are cheaper than using syscalls, for example.
In general, the projects, the projects that I presented before all come forward to the
epoxy standard, so it means that if you have any application written in C that come forward
epoxy, you can theoretically compile with the Unicolonial without any modification of
the user application.
In reality, this does not happen, and most of the time, the epoxy that the Unicolonial
implement is not complete, so you have to do some work to just, you cannot just take
your application and compile it and generate something, it doesn't work out of the box
in most of the cases, right?
So, in this context, what is total is also a Unicolonial, it's an application-oriented
Unicolonial, and the idea of total is to offer an API which is dedicated, I mean, to efficiently
deploy parallel application.
In the case of total, it is, it's not a epoxy complaint, it means that even if the nail of
the functions like this opens in close and so on, it's more or less the same nail, the
semantic of this function is slightly different, so I will not say that it's a epoxy complaint
in that sense, and I will explain that later.
So, let's say that the three building blocks of the total Unicolonial are the memory per
core, cooperative scheduler, and core-to-core communication based on built IEA.
Here I'm talking about the architect of the Unicolonial, I'm not talking about yet the
application of how we're going to build an application to compile tutorial, right?
And I'm going to explain these three points.
So, first, what happened in the total Unicolonial is that we have memory dedicated per core,
so at the beginning what we do is just allocate memory, I mean, to split the whole memory in
rations and we assign these rations per core, and for the moment the size of these rations
is just proportional to the number of cores that we have.
That makes that, for example, the memory allocator is quite simple, it doesn't require any communication
because we have chunks of data, I mean, yeah, the allocator is, we have one allocator per
core which means that we don't require any synchronization in the kernel to allocate
for one core.
It's quite, we call it per CPU data, let's say, yeah.
So, for example, if you have a thread in a core one and we want to locate memory, we're
going always to get it from the same rations and that happens also if you're on the core
two, we're going to use the rations, from rations two.
And the idea is that by doing this, we can then leverage technologies like hypertransport
or interquit path interconnect in which we can say, well, this core is going to access
this rations of memory and if you access all the rations, it's going to get a penalty
to do it, right?
So, talking about the scheduler, what happened in total is that we only have thread, so we
don't have process means that we, all threads share the view of the memory and we have mainly
one API to create thread, it's called a begin thread and it's the parameter that have to
say in which core each thread is going to run.
The scheduler scoperity, which means that it is the thread that's going to call the
scheduler to then choose another thread and this is by relying on the API call assist
thread switch and most of the time, this is just in bucket because we are going to be
idle for a while, so we just call the scheduler or we, for example, we're going to do some
IO.
So the scheduler is also very simple, we have, again, first CPU data, so we have one
cube per core and the scheduler is simple going to choose the next thread that is ready
and then the scheduler and this means that also we don't require any synchronization
at the level of the kernel to schedule a thread, so it's like each core run independently
one for another.
Fine I am going to talk a bit about how we communicate cores and basically what we have
is one dedicated reception cube per core for any other core in the system, so we have one
to one communication.
This basically relies on two primitives which is send and resist front and it's just by
using the destination core and from where we want to get a packet, for example.
These two primitives are the ingridients to then build more complicated APIs like MPI
gutter, MPI because and MPI scatter, so these are the building blocks for those API, for
example.
So to implement this core-to-core communication, I was using butaio, so I was just following
the specification, I will talk a little bit about this, I don't want to go too much into
detail so as to understand roughly how communication between core is done.
As I said before, we have one reception cube for each core, for any other core in the system,
so means that, for example, if core one want to get packets from core two, we have a reception
cube and also if core one want to send a packet to core two, it's going to have a transmission
cube and the number of queues is going to be for sure different if you have three cores,
for example, because the build queues are dedicated.
So basically how a build queue works is made of three RINs buffers.
So the first RIN buffer is the buffer which only contains descriptors to chunks of memory.
The second buffer is the aviable RIN and the third buffer is the user RIN.
Basically what happens is the aviable RIN is the buffers that the core one are exposing
to core two.
So if the core two want to send a packet to core one, it's going to get a buffer from
aviable RIN, put the data and then put it again in the user RIN.
This is basically how build.io works, just that if you are familiar with build.io, in
this case, for example, the consumer of aviable RIN is the core two, but if, for example,
if you are in a hypervisor and you're implementing some build.io device, the consumer is not
going to be the core two, but it's going to be the device model, QMU, for example, and
if you are familiar with that.
But it's the same scheme.
This allows that, for example, since we have one producer and one consumer, we can access
to the build queue without any synchronization, I mean, at least if we have only one consumer.
So you don't require any luck, for example, to access to the build queue.
So yeah, I already talked too much, I don't know how much time I have left, but I wanted
to show some examples about the implementation, maybe it's more fun that all the flyers should
show.
So what happened, how we, how we deploy an application by using Toiletware.
We have the MPI application, this is a C application for the moment, and you compile it with a
unit that's going just to link the application with some functions that are the implementation
of the MPI, the MPI, for example, MPI Bicass, Gatter, and so on, it's implemented in this
level in the MPI interface.
And this unit is going to use some MPI from the unique kernel.
So at the end, what you're going to get is an ELF and binary that could be used to deploy
your application, either as a built-in machine or a bare metal.
So you don't have any operating system intermediate there, you have only your application, the
threads and so on, but you don't have nothing else.
So if you want to see how it is deployed, if you get the MPI application that doesn't
see what is going to happen, we're going to get the main and then instantiate it to one
for every core in the system, as a thread.
So to benchmark the current implementation, I'm not very familiar with the MPI where I
was just coming from another domain, so I am not really familiar with how I had to benchmark
such implementation, and so I choose the also microbenchmarks, maybe you know them, maybe
not, and I just pick up one of them, like for example, MPI barrier, and I try to implement,
which is quite simple, the benchmark itself is quite simple, so I decided to implement
it.
I could not take the benchmark as it is, I have to do some rework to make it work, and
then my idea was to see how this behave when I was deploying this as a single VM, which
many cores.
The hardware that I use is this one, since I'm not familiar with the high performance
computing work, I'm not really sure if this is a hardware that you often use, it's quite
a new Intel, you can get it in Equinex, the price is four euros per hour.
So I launched the test and I tried to measure things, so I was just measuring the latency
of this, and I was taking into account the max latency through, I mean, over four, eight,
sixteen, or thirty-two cores.
I am getting values in the order of the microseconds, and then I found this paper, which was also
using this benchmark to measure their platform, and I was saying, well, in this paper, they
were reporting around 20 nanoseconds and 13 nanoseconds, sorry, this is nanoseconds,
this is microseconds, not nanoseconds, sorry, in this platform.
In any case, I will be very cautious about this graph, because I was getting a lot of
variation in the numbers, most of the time, for example, I was trying in a machine with
thirty-two cores, and the VM has already thirty-two BCPUs, so you should not test in
that sort of machine, because one of the threads is going to compete with the others,
with the main one of the hosts, so you should always test with less cores, less BCPU cores,
physical cores.
And, yeah, the idea is to continue doing this, I mean, improving the way I am measuring
this, and also try maybe in different hardware, and at the same time, I found a lot of packs
in the unicroner by doing this, so for example, at the beginning, I only support more or less
four cores, so I went from four to thirty-two, well, it was a number in a constant, but anyway,
I found many packs when I was doing this, so it is all, this is just a proof-of-concept
and a work in progress, so you don't take it too serious, I am trying to say, I don't
want to jump into any conclusion from this, and, yeah, it was fun to do, anyway.
So that's all, I don't know if you have any questions.
So you said this runs on bare metal.
Sorry?
The unicroner runs on bare metal.
Yeah, they are some.
How do you even install it?
I mean, operating systems are kind of complicated, right?
Sorry?
How do you even install it on bare metal?
Can you say that again?
How do you install it on bare metal?
How do you install it?
Yeah, like if I had this, how would I install it on bare metal?
Is there an installer or...?
Installer, you mean?
Yeah.
Now, you can just use some device to boot from, for example.
So it's bootable?
Yeah, that's it.
Yeah.
Okay.
Yeah.
Well, yeah.
There are many ways to do that, for example, you don't have to install it, for example.
You can use from a device that is removable, for example, you don't need to install it.
Any questions?
Thanks.
Thank you.
Thank you very much.