We continue with our next speaker, which is going to be a
V extension.
We continue with our next speaker, which is going to be
kind of a follow-up of the previous one, because it's
approximately the same topic, but this time about wrist drive
and arm.
So please welcome Remi.
Hi, good afternoon, everyone.
I hope you are done with the digestion.
So, yeah, this pretty much follows up our
compliments Karen's previous speech.
But before I go into the details, obviously, I work for big
companies, so I have to put this disclaimer.
And then if I speak too fast or if I don't articulate
properly, please stop.
Please stop me.
With that said, who am I?
I don't think it matters much, but this is my 16th time
in first day, and it's only my first presentation,
so bear with me.
Having said that, I don't work in this field at all,
so just a fancy thing for me to do.
So some history.
So has anybody ever seen this outside the computer museum?
Right.
Yeah, so that's the Cray one.
It's the first, indeed, it's the first vector processor.
It's from the late or second half of the 70s.
I wasn't even born back then.
But in point being, it's the first vector processor,
and we all now, finally, after almost 50 years,
are coming back to this kind of, maybe coming back to this
approach to calculations in computers.
But for people in my generation, this is more what we associate
with SIND before multimedia.
So this is POD, the first video game that actually used
the MMX, which MMX being the first SIND extensions
in the consumer business, in the consumer space, let's say.
So as I said, the MMX came in 1997, and that was 64-bit
vectors.
So you could compute over 64-bit at a time.
Minded, back then, computers was pretty much only 32-bits.
And two years later came SSE, and many, many, many versions
of SSE.
SSE2, which is more popular in multimedia use cases, 2000.
I'm not going to go through all the details of SSE
because there's like a billion, million versions.
And AVX1 came in 2008.
AVX2, which Karen mentioned, came in 2011.
That was the first to have 256-bits vectors.
Then AVX512, which was the topic of the previous presentation,
officially came in 2013.
But as Karen mentioned, the only real, real, proper CPUs
were only out in 2017.
On ARM side, the first SIND was actually 32-bit,
only on ARM V6, 2002.
That doesn't really seem to make sense, but that's because
it's basically calculating as a 4 times 8-bits or 2 times
16-bit at a time.
Then 128-bits came.
There was no 64-bit SIMD on ARM.
28-bit came with ARM V7, so Cortex-A8, usually called Neon,
officially called Advanced SIMD in 2005.
And on ARM V8, it's pretty much the same.
Now, it's not actually compatible on X86 or 64-bit,
but it came with basically ARM V8 in 2012.
It's also officially called Advanced SIMD,
and it's also colloquially known as Neon.
As for RISC-V, RISC-V is much more recent.
There is no SIMD.
The problem, and I've only summarized,
this is only a short summary, there's way more extension,
especially on the X86 side, is that every damn time
you have to rewrite your assembly, and as the questions
and answers in the previous talks and even the previous
previous talk covered, this is kind of damn consuming.
So, with that said, this was all fixed size SIMD,
so what about viable length SIMD,
which is what we will be talking about today.
So, how would you go about doing it?
Well, the simple way to do it is to just ask the CPU
what is your vector size, and if you do RISC-V,
this is how you do it.
So, control-register read operation,
the vector is called VL and B for vector lengths in bytes,
and it will store in this case, T0, whatever,
it's one register, the number of bytes in a vector,
and with that you could then iterate.
So, if you want to know the number of elements,
well, you have to do a left shift to compute
the number of elements, so if you want to have
32-bit elements, you divide by 4, shift by 2 bits.
You could do it like that, and then you would write your main,
you would take your C loop, you would convert it into assembler
to operate on however many elements at a time,
then you would probably unroll to, like,
if you have space in your vector bank,
you'd probably unroll to eliminate,
try to hit up the latency a little bit
because usually between instructions,
if you operate only on one dataset,
you will have inter-instruction latencies
which are going to hurt your performance,
so you typically, in multimedia, unroll twice,
so you will do, work over two sets of vectors
at the same time in parallel,
and when you have done all of that,
you will be working on however many, like, say,
32-bit, 32 items, 32 elements at a time,
so you have to deal with ages because you might not
have a multiple of 32 elements that you are dealing with.
And that's fine, and that's one way to do it.
In fact, last I checked, that's how Clang, LLVM,
does the three vectorization on risk five
if you enable it, even so you have,
it literally starts by reading the vector lengths
and then deal with ages and unrolls twice,
and if it manages to implement, I mean,
if you have enabled three vectorization
and you have enabled the risk five vectors,
but that's not really how you want to do it.
But before we go on how to actually do it,
what vector lengths are we dealing with here?
So, obviously, well, as mentioned earlier,
common values are 128 and 2,512 bits.
So, both arm and risk five guarantee that even if you have
a viable vector length, it's going to be at least 128 bits,
and it's also going to be a power of two bits,
which is kind of convenient for the calculations.
So, as far as I've seen, there are announcements
for real hardware which would have 256 and 312 bits
that you should be able to buy at some point in the near future.
More crazy stuff.
I've seen actually, like, designs also being announced
with 1,000 bits.
I don't know if they're going to store all those bits
in the physical register bank,
but it would be interesting if it happens.
And I haven't seen theoretical designs at 4,000 bits,
and I mean theoretical to the point that there is a schema,
theoretical in this case, I mean that there are actual
schematics of how you could write a chip
and they have even simulation of the performance
that the chip would get in certain algorithms
as to whether it's actually practically implementable
in an existing industrial process.
I don't know.
I'm not a specialist in electronics,
but that sounds a little bit questionable.
So, in theory, on the syntactic level,
so in the instruction and coding level,
you can have up to two power 16 bits,
at least on S5.
I'm not sure about that, actually.
So, how you actually do vector lengths,
how you're supposed to do a viable vector length,
a SIMD or vector processing, as it's often called,
also, practically vector and SIMD are synonyms.
Well, at first you have to use predication,
which is very highly prevalent in viable vector length scenarios.
Now, it's not a completely new concept.
Kieran mentioned the K-mask in AVX,
so AVX also has predication,
but in viable vector lengths, it's really essential
because this is basically the programming model
on viable vector lengths and or loops
is essentially built on predication.
And that's true both for ARM and RISC-5.
So, a predicate is a vector of Boolean.
So, like the K-mask in X86,
it's called the p-vector in ARM,
and it's the mask vector in RISC-5.
And as Kieran said, kind of repeating,
but it just specifies which of the elements in your vector,
it specifies which ones you will be loading
or modifying or storing out of a given instruction.
So, if it's a load instruction,
which values you load for memory and overwrite into the register,
if it's a stored instruction,
it's going to be the other way,
which values in memory are going to overwrite
versus which ones are going to live as they are.
And if it's a calculation instruction,
vector to vector, then it's going to affect which ones,
which results are actually stored into the register
versus which ones are just discarded.
So, on ARM-V9 or SVE,
one way you would typically do now your SVE loop
instead of, say, your NEON loop,
is you would start by counting down,
you would initialize, say, extend to a zero,
and then you would...
So, you have to imagine here
that you have your actual NEON or SVE loop,
so you will check...
You have this funny instruction,
which is actually called YLT or YLLO,
and you initialize the p-zero vector in this case,
which is one of the predicate registers
to say that, essentially,
you want to count how many elements you still have
in your input data.
So, here, we have...
We imagine that X0 is the number of elements
we have been given to this function.
X10 is the count of how far we've been,
so it's our iterator.
And we'll say...
Essentially, what we'll say is,
as long as X10 is larger...
As long as the number of elements we still have...
So, as long as X0 is larger than the size
of the vector that the CPU can handle,
we'll just set the predicate to handle to be clear,
so we'll use the full size of the vector for our programming.
And once the number of elements is more than zero,
but strictly less than the vector size
than the CPU can handle,
then we'll start basically just ignoring the values
at the end of the vector,
so we'll have a bunch of ones,
and then at the end, a bunch of zeros.
And this is how you abstract away and hide away
the complexity of dealing with the edge in your loop.
So, essentially, by doing this,
you don't care what is the actual capacity of...
You don't actually need, at any point, to know
how many elements you're dealing with
in any iteration of your loop,
because it's all hidden away by the...
Essentially, the size of the vector
and the size of the predicate are matched,
so you don't actually care.
And you also don't need to deal with edges,
because, well, even if there's one or two or three
or four elements left over at the end,
you can just deal with them in the last iteration,
which, of course, will be a little bit less efficient
than using the full size of the vector,
but it's still much faster than having a separate edge
if only because you will not be stressing
the instruction cache of the CPU.
So that's predication.
Now, unrolling.
If you thought about it,
all that I just said with predication,
it doesn't really work with unrolling,
because now you've counted down...
You've set your predicate vector to count down
how many elements you have still
in your total count of elements.
You can't unroll, because now, like,
you've said, oh, I have 10 elements left,
I'm going to use 10 elements in my vector,
but if you have...
It just doesn't work, like,
because if you had, like, one and a half vector left,
you would want to have one predicate with all the bit set
and another predicate with half of the bit set.
This doesn't really work very well.
And, yes, now, it's a bit of a hot tech.
Maybe I will never be ever again allowed
to write a session-peck code after this,
but just don't unroll if you use variable vector lengths.
Now, there may be cases where you can unroll
because, naturally, you have some kind of parallel
in your design aspect in your algorithm,
but the idea of vector processing
is that we have higher latency and larger vectors,
which, in the end, result in higher throughput,
and we shouldn't need to unroll.
I'm sure you will find actual designs real hardware,
real processors, where it will be faster if you do unroll,
and how much you need to unroll will depend on the design.
And, of course, if you are trying to squeeze the very last
MIPS out of a given specific piece of hardware,
then maybe you should unroll,
but, I think, generally speaking,
at least you shouldn't start by unrolling.
And another interesting thing to keep in mind,
which kind of already mentioned in the previous slide,
is that you don't have alignment issues.
The one common problem with CMD instruction set
is that the load and store instructions
require overaligned data,
typically aligned on the side of the vector,
which is very inconvenient when you're operating
from C or C++ code, because it's usually C or C++ allocator
will only allocate align on whatever the ABI specifies,
which on RBA, it would be 16 bytes for the stack
and 8 bytes for the heap.
So, usually, while at least both SV and RIC5 vectors,
the alignment needed is only the alignment of the element,
and it's not the alignment, it's not the side of the vector.
So, if you are operating on, say, 4 bytes pieces of data elements,
then you only need your vectors to be aligned on 4 bytes,
which is a very nice property for dealing,
especially on the edge cases,
and also you don't have to deal with,
like, if you have one input that is perfectly aligned
and the output is not perfectly aligned,
like, you end up having this weird mismatch
and you end up having to deal with different edge cases,
it's really a mess.
With vector processing, you don't do that,
so you don't actually have to worry about it.
So, with that, we've covered generality,
so how is it looking on ARM side,
and then we'll see RIC5 side, because it's a bit weird
if I would...
I thought, like, to put everything together,
but then it becomes a huge mess.
So, it's going to be a bit repetitive,
because, of course, there's a lot of similarities,
so SVE came about, like, five years ago,
a little bit more than five years ago,
I think it was announced late 2016,
if I recall correctly.
It was pretty much less on multimedia.
It was explicitly meant for other things,
like, well, scientific applications,
or engineering modeling and this kind of stuff,
well, HPC, and so nobody used it.
At least nobody in this room used it.
This was fixed with SVE2,
which is sometimes called ARMv9,
because it kind of comes with ARMv9,
but it's really called SVE2.
Fixed that issue, the realisation that, actually,
this is a good idea.
This pattern programming model is also interesting
for multimedia and crypto,
which was also missing from SVE1.
And so what they did is they just took,
so which neomonics are missing,
and added those,
and it's pretty much the same mnemonics
you just add the predicate register.
That's why this is, of course, a little bit more complicated,
but as I mentioned, you just use a while instruction,
which will then provision your predicate,
and you have to pick the element size
so that, of course, this adds up correctly,
and then you have a new set of branch conditions,
so first element, last element, and so on and so forth.
So the remaining elements will be determined
by the predicate register,
and the predicate register will set the condition flag,
and the while instruction will also subtract.
There is a count, the number of processed elements
from your output register.
And yeah, at this point, stop pretending that I'm at risk.
How do you detect this stuff?
So there's a processor macro,
otherwise, as usual, on ARMv8,
you have a bunch of privileged registers
for the OS to look at,
and then you have also Linux,
you have a bunch of flags in the auxiliary vector bit,
so it's all classic.
Another OS that you're out of luck.
Availability, so as we said, 2016,
but it didn't really work for us.
SV2 was specified in 2019,
but so the real hardware came earlier last year,
so Cortex-AX2 and all the other things
from dynamic IQ 110.
So Samsung actually knows 2,200,
and so Cortex-AX2 and all the other things,
they do have SVE,
unfortunately, it's only 128-bit vectors,
and it's pretty damn expensive,
but if you want to do it, you can find the hardware.
So RIS5, it's a different model.
Can I add?
Yeah.
There's also the Alibaba one, the Yi-Tian.
Yeah, maybe.
It's possible, yes.
It's only available in China, but it's available.
So on RIS5, the predication is a little bit different,
so they have separation between element count
and the actual predicate.
And so in practice in multimedia, maybe not in David,
but usually you don't use the predicate at all,
so we will instead just count the elements.
This is the instruction you always find
at the beginning of the loop, which considers the vectors.
So in this case, what we say is that we have
a certain number of input elements.
We want to get the number of output parameters
and the number of elements the CPU will deal with
in the iteration.
We then have to say the size of the element in bits,
in this case, for instance, 16 bits.
The group size, which is kind of free unrolling,
it will automatically, if you set it to 2,
it will use all the, and you say you want to use
vector 8, it will use vector 8 and vector 9
at the same time, for instance.
And tail mode, we always set it to agnostic
because we don't really care about tail mode
and mask mode, we also always set it to agnostic.
There might be use cases where you need to do something else,
which might be a little bit slower, but usually you don't.
This is about how to deal with the stuff that is masked
or with the element that are at the end of the vector
which we don't care about.
Usually you don't care about them,
so you just tell the CPU you don't care about them.
One cool thing about RISC-V,
the floating point registers are separate from the vectors
and like on ARM, so you have more registers available
if you have hybrid calculations between scalar and vector side.
But do mind the floating point convention,
calling convention when this happens,
otherwise you will screw up your register state
and confuse your CPU.
The interesting stuff also about RISC-V,
they have segmented load and store,
which is similar to structured load and store in ARM,
but they can do it up to 8 structures,
whereas ARM is only up to 4.
What is much more interesting perhaps is
strided loads and store where you can say,
I have this register X which contains a value
and that's going to be my stride.
So for instance with that you can put
the width of your video inside one register
and you can load all the pixels in a column in an instruction
without having to do weird shuffling and whatever.
Does that actually perform a practice?
I think that's going to depend on the design,
but normally it should be in the data cache
which should be okay.
So I'll come to that.
Yes, on the downside you don't have transposer
or zipping instructions, which should be annoying,
which is kind of the same,
so you have to replace it with strides.
So it's fine if you want to take every second element
from one vector and so on.
Feature detection, they have very, very detailed
pre-processor feature flags.
I mean you can download the slides if you're interested.
On the other hand, on runtime detection it's pretty crappy.
You have to trust the device tree node.
So you have to trust the boot loader to actually tell it
to your OS correctly in the device tree data structure
and otherwise there is a flag in there.
So the V, the Vth bit, so the 21,
because V is the 22nd later in the alphabet
is a vector flag in the auxiliary vector
for hardware capabilities on Linux.
Availability, unfortunately at this time there is no hardware.
Ali Baba, sorry, T-Head has made hardware available
but it's implementing version 0.71
from about 18 months before the standardised specification
which is implemented by Clang and GCC.
So you can kind of work with that
and it gives you some idea of the performance
but you're going to have to rewrite stuff
because it's not completely bit compatible
so it's kind of annoying.
I don't know when the stuff is going to happen.
I'm pretty sure it's going to happen
but I would guess by the end of this year
we are going to see hardware available.
Also I think one kind of not answering
or dodging the previous question
but because we have so many different vendors on RISC 5
and I think there's more than I did.
I only listed three here but I think there's other.
There might be big difference
between the performance characteristics
of the different vendors.
These are our references.
Yes, I have just a few questions.
Have you heard of the SVP64 project from Lever SoC yet
which is a kind of similar vector approach for PowerPC?
No, I haven't looked at PowerPC at all.
Another question that I had with my own CIDD programming workers
we often have applications that are inherently horizontal.
For example, let's say you are writing a vectorized string search operation
or you're doing something like decoding JPEGs
where you have these 8.8 blocks
where you want to do some sort of close-in transform on them
and they have this fixed size
and depending on the vector size
you want to break them up
or you maybe have to process multiple of them at the same time.
Is there an intelligent way to solve this?
I've had this case.
The question is when you have a naturally fixed size
input kind of block that you want to process at the time
how do you do this?
Because then you actually want to have a fixed size vector
in effect, paraphrasing the question.
I've had this case with the SVP64 a couple of times.
One way is to just check that the vector size of the CPU
is big enough and just do one at a time.
If you can, try to do it at a time
because it's always going to be a power of 2
so you should be able relatively easily to parallelize.
Obviously the ideal situation is to parallelize.
What you will have a problem is if your dataset is larger than the vector
then it's going to become complicated for you.
On RISC-5 you can deal with this with the group multiplier
which will allow you to use multiple vectors as a single vector.
And the last question I have is how do you realistically test
vectorized triangles?
When the hardware you have only supports one vector length at most
so you have to probably use some sort of relation to set up for this?
Most of the loops will not depend.
So the question is how do you test a different vector size
for validation I guess.
Most of the loops don't really care about the vector size
because if you have a simple case where you follow the simple pattern
it doesn't really care what the vector size is
except for benchmarking of course and you have a problem.
Otherwise QMU and Spark at least for RISC-5 support any vector size
to give that as long as it's a valid one from specification plan point.
Do you realistically really test for that?
Or do you just say it's simply not going to be a problem?
I mean personally when I've had the situation where I had a fixed size input
and I had to test with different vector size and I tested with different vector size
in most cases you don't actually care.
I mean then it's a matter of choice of you do your testing
and no strict you want to be with the validation I think.
Thank you.
We have no one on question now?
Firstly disclaimer, I'm related to the Leversock project with SB64.
It's similar to RISC-5 vectors but not exactly the same
but they share a lot of the common ideas.
You mentioned a very good point that CMD is not vector processing.
In order we had to try to report some code from Neon to SV2
and it was less than suboptimal let's say.
We had to revert back to the original C algorithm.