We'll talk about AVX-312 in FFNPEG.
He's also the co-organiser of this dev room.
Please welcome Kirano.
So yes, I'm going to be talking about AVX-512 in FFNPEG.
What is AVX-512?
AVX stands for Advanced Vector Extensions.
There will be a lot of acronyms and jargon, unfortunately,
in this one, but I will try and explain all of them.
So AVX-512 is a relatively new single instruction
multiple data instruction set for Intel CPUs from about 2017
and more recently in the last six months or so with AMD CPUs.
In particular, it has a larger 512-bit register size.
Many new instructions, which we'll talk about in a minute.
Comparisons, which are quite new, and also lots of other things
that are not so interesting in multimedia.
Cryptography, neural networks, and I'm sure there are other people
at Fastem who could talk a lot more about these kind of things.
As I mentioned, lots of fancy words, but the thing to bear in mind
is in FFNPEG, high schoolers have gone and written assembly.
This is heavily jargon-centric.
It sounds complicated, but actually quite a big reasonable chunk
of assembly in FFNPEG has been written by people who are in high school.
Why is this relevant now?
I've mentioned AVX-512 has been around since 2017, so why is it 2023?
Well, Skylake was the first CPU generation from Intel to have AVX-512 support,
but it had very large performance throttling when you used them,
so your effective CPU capability speed went down quite dramatically.
And so this was fine if you were doing high-performance computing in academia,
for example, like fluid dynamics, where you were using these instructions
100% of the time, that was fine.
But in multimedia is a mixture of assembly and C code,
where you're not necessarily always using these instructions.
So this relative main is sort of unused for the last couple of years.
You could still use these new instructions, though, with the smaller register sizes,
and I'll show an example of this later.
But the first Intel CPUs not to have throttling were the Islake 10th and 11th gen Intel CPUs.
They were the first to have no throttling, and this meant these ZMM-based instructions
could be first-class citizens.
How to get started, one of the tricky things as well in the last few years
has been actually getting access to devices that have this,
and unfortunately Intel have not made it easy.
From their 12th generation, CPUs have actually removed support in consumer equipment.
It's still available on AMD as in four CPUs, though.
And if using the cloud is your kind of thing,
available also from many cloud providers in the server CPU range,
such as AWS or others.
Personally, I think the easiest way is to buy an 11th generation Intel NUC.
That's what I did for FMpeg.
I bought two of them for the projects and host them.
The easiest way, it's only a few hundred euros.
It's quiet, it fits under your desk.
And that's the easiest way to get started, you get a full AVX512 stack.
So let's look at some of the existing work in multimedia that's using AVX512.
And probably most importantly, we had the sort of introduction from JB earlier today,
the David project, which is an AV1 decoder.
This added AVX512 support, I think a year or two ago.
It's particularly beneficial in AV1 because AV1 has large block sizes,
sort of in comparison to more traditional standards,
traditional codecs like H264 and others, which are smaller.
So AVX512 in David gave, I think, 10 to 20% overall.
So not just the functions themselves,
the overall decode performance was improved.
And it's actually been a running topic, which is quite interesting over today,
in FMPEG that we use, and David, and also we use this classic FMPEG H264 approach to assembly,
which is no intrinsics, no inline assembly, no special SIMD sort of libraries to make life easier.
It's raw assembly language, and I'll show some examples of that.
And also we don't also compile them in and force you to have a particular CPU generation.
And I know this is quite controversial. I think it's MongoDB, for example.
They forced one-year a particular CPU generation, and this was super controversial because not everybody had that.
So what we do in FMPEG is we detect CPU capabilities, and I'll show you the function in a minute.
And then we use function pointers, so we set them once at the beginning,
and therefore the overhead of doing that measurement is checked once,
and then there's function pointers that are executed after that.
And unfortunately, on Intel, there's a very messy Venn diagram of capabilities.
But in practice, we really, so far, and they may change their mind,
but care about these kind of two things.
So these are the CPU flags you get in FMPEG.
There are others, but the AVX-512-related ones are broadly speaking legacy Skylake,
and the newer ICL are put in bold for Ice Lake.
But you can see there are actually a lot of different subcategories in there.
But in practice, it's at the moment one or the other,
but as I mentioned, Intel are very keen on adding and removing features
and possibly even charging your subscription for certain features is one of their new ideas.
So it could be that newer additions to this are subscription-based,
or you buy and pay for it later, or something much more complicated.
So who knows?
So I guess, unfortunately, there's some sort of dependency
in explaining a few of the topics and some of the benefits
without explaining some of the backstory.
So historically, in old AVX, you had all the 256-bit registers,
and these were split in practice into lanes.
So in practice, you've got 228-bit lanes,
and instructions, broadly speaking, operated in these lanes.
So if you ran a instruction, it worked on data,
and it was actually quite difficult.
It was possible, but difficult to move data between these lanes.
And it's one of the historical limitations on existing AVX and AVX2 code that we have
in FMNPEG is lane crossing and all sorts of trickery
that essentially cost CPU cycles to take up this time,
that takes time to compensate for the lanes.
I have to talk a bit about KMAS registers as well.
So AVX512 has these new set of registers called KMASks,
K0 to K7, and this allows a destination register to remain unchanged.
So, for example, underneath, you could have an addition,
but actually it's a simple case,
and obviously you could just add zero, and it's unchanged,
but you could actually use the KMAS to say,
actually, I don't want addition to be applied to these elements.
I want this to be a pure pass-through,
or you could even force some of the elements to zero if you wanted to.
There's a specific, I think it's a flag that lets you do that.
And there's a whole set of new instructions to go and manipulate these KMAS registers,
and certainly David, in particular, uses, makes good use of KMASks.
So now that I've sort of explained some of the back story,
I think it's fair to say one of the most important instructions,
if not the most important instruction, is our shuffles in multimedia.
Also known as permutes, and there might be a technical difference
between a shuffle and a permute.
Someone might be able to correct me.
There might be some mathematical difference,
but these are the most important,
or one of the most important, instructions in multimedia.
And as you can see on the right, basically it lets you,
shuffles let you have various bits of data
and rearrange them in any way that you want.
Duplicate them, as you can see, or even set individual elements to zero.
And this is, for example, famously one use case of this
is in the zigzag scan of FFMPEG,
which groups larger coefficients in a block together.
But the way that that's done is via a zigzag scan.
The thing about vpermb, which is the new AVX-512 instruction,
is it lets you cross a lane.
This wasn't something that was possible in before.
And as I'll show you later, this makes things substantially faster in many cases.
pshuffb, probably one of the most commonly used instructions
in all of open source multimedia.
You do get grep, pshuffb, there'll be a huge, you know,
that your screen will be full of pshuffb.
They're used everywhere in open source multimedia.
pshuffb had a kind of useful benefit
that if you set the index to minus one,
you had to automatically do the zeroing out.
With vpermb, this isn't the case.
You have to actually use kmasks to do that.
So that just makes things slightly more complicated.
There's all sorts of other interesting permutes that AVX-512 offers.
I think David also, again, makes good use of this vperm2b,
so you can actually not just have one set of data,
you can actually permute from two different registers.
So you could have ijk, et cetera, et cetera in a different register,
and your output could be a mixture of both of those.
So that's kind of interesting.
Variable shifts.
You have now variable right shifts.
So I've given the example of a vpsrlvw logical right shift
and vpslvw variable left shift logical.
Big letter soup, quite confusing.
In fact, when writing this slide, I misspelt the word shift.
You can have a think about how that may have been spelt.
Thankfully, that's the good, thankfully, the rehearsals,
and we'll pick this up.
But this word soup is exceptionally confusing,
both when writing slides and writing code, it seems.
So historically, to do variable shifts,
so if you want to take, obviously, just to step back,
take an element and shift each element by a different amount,
this was quite complicated.
There's various bits of trickery, various idioms that people use
to try and emulate that, but they had limitations.
I think, for example, you were not shifting by zero,
possibly wasn't allowed in one of the various bits of trickery.
And so if you needed a zero shift,
you had to do it a different way, et cetera, et cetera.
But now you have this variable shift, and it's all usable.
Equally on the left shift, the naive way of doing an emulated
left shift is just to multiply, but these instructions
are actually faster than the multiply,
so there's still some benefit.
VP Turnlog D, this is, I think, no presentation
about AVX 512 could not fail to mention VP Turnlog D.
This instruction is literally a kitchen sink.
It's quite remarkable in what it can actually do.
You can literally program a truth table
within an individual instruction itself,
and, in theory, could replace up to eight different instructions.
So you could do a whole presentation on VP Turnlog D.
So I thought it would be best to try and pick one of the simplest ones,
which is a ternary operation.
So this is a bitwise equivalent to the C ternary operation.
So in each register, each bit is iterated through.
And you can see, for example, one, the ternary operation.
So if that bit set choose this or versus this,
and you can see the output of that is that.
And so, essentially, it's a bitwise operation of ZMM
is equal to ZMM0, a question mark, ZMM1, ZMM2,
but on a bitwise level.
And there's all sorts of other interesting things you can do,
and this article is very good.
It shows all sorts of interesting things you can do,
bit selects, all sorts of various different operations
that you can do on multiple XORs, for example.
So, yeah, also very interesting.
So let's look at a real-world example.
I don't know how well you can see that.
I was hoping the dark mode would actually make life easier,
but maybe it's made things worse.
But I'll talk about some of the mouse.
Is it the mouse?
Because the mouse on the Mac is dark.
But anyway, this is v2.10enc.
It's probably one of the most simplest assembly functions
in fmpeg, but what it does is it takes
three 8-bit samples from different memory locations.
It sort of, as part of its work, extends to 10 bits
and then packs those three 10-bit words into 32 bits.
So what's interesting in this function is
we're already starting to do lane crossing
that wasn't possible before.
So we load the y-samples, so the luma samples,
into the lower 256 bits.
We do the u-section of the chroma into the third,
or the second, if zero-indexed, portion of the register,
and then equally the same for v.
And then we do one, excuse me,
and then one single v per mb
can rearrange all of that in one go.
This was a lot more complicated back in the olden days.
P mad sub sw is some trickery
that unfortunately there's not going to be enough time
to explain, but eventually is a multiply and add,
and we use that to emulate a shift.
And then for the second element,
in the three elements, we need to do a d-word shift
because it actually spans the middle.
So therefore then we have sort of conflicting bits
in each register.
So how do we do a bit selection?
And this was quite a, I think it's a two or three,
even up around two through three different instructions
in the previous code.
And this can now be done in a single vpternlogd,
so essentially c ternary b or a.
So if bit c is set, choose the bit from b
or choose it from a otherwise.
And you'll see in a second that actually provides quite a big,
well certainly a measurable speed improvement.
So these are the benchmarks.
So this is, so I wanted to show a bit about how you can
get benefits from AVX 512 even on the older hardware
with the shorter existing registers.
These are not scientifically benchmarked,
I just ran them yesterday.
When you do benchmarking you should run them
10 or 100 of times, average them,
do standard deviations, et cetera.
But just for the simple case,
you can see that the c code versus the AVX 2 code
is around 10 times faster.
And you can see just by replacing,
I think it's a set of two or three different pans
or various boolean functions,
you can get a measurable increase just with one instruction
replacing three, even on the older YMM registers.
But where the big gains come are on Ice Lake,
you can see the c code versus the AVX 512 ICL,
there's a huge difference.
So by using vperm b and the ZMM,
you can already make the legacy AVX 512 twice as fast.
And if something was 10 times faster,
that now becomes 20 times faster.
And I often have to say that's not a multiply,
that's a times.
So it's massive improvement.
This was code that could, if you have a large resolution file,
take up an entire CPU core,
and now it takes essentially 5% of a core.
It's really tiny.
What AVX 512 code is next?
Anything really that's line-based or frame-based,
such as filtering or scaling,
I think the next thing we're working on is deinterlacing.
Anything involving comparisons,
I haven't really talked about comparisons,
but there are bits of code that often need to do comparisons.
That's going to be an obvious place for AVX 512.
Lots of places that do triple booleans,
multiple XORs or multiple XORs on ands,
and I think it's almost always possible
to replace that with a VP10 log D.
Likewise in the code base,
there's various different idioms and trickery
to try and emulate a variable left shift and right shift,
or multiplies for the left shifts
and trickery for the right shifts.
This could be used with the letter soup instructions
to try and produce that.
Intel provides an official manual to all of this.
It's very verbose, which is great in many cases
because it provides really precise detail
of how the instructions work,
but unfortunately is not at all approachable.
There's a few websites that try and simplify things.
I think this website on officedaytime.com
is some kind of Japanese website,
English that explains,
tries to group all the instructions
in some kind of logical ordering,
and that makes it a lot simpler to understand.
Any questions?
Hopefully I'll be able to answer them,
but thankfully at FosterM there's always somebody
with more knowledge than you in the room.
I can't see where they are, but I did see them at one point.
Thanks.
Thank you.
Any questions in the room?
Regarding the direct assembly writing of AVX-5.0,
there's about 7,000 instructions of AVX-5.0.
Why?
If you choose the direct assembly,
then you essentially might miss out
on potential instruction scheduling
between different architectures.
Compilers might schedule better
if you want to get a performance benefit in the future.
But then you have to ship a binary for each version.
Sorry, repeat the question.
You have to write in 3.6, that's what I'm saying.
In order to compile...
The question is the classic question,
can the compiler do a better job than a human question?
In David, certainly the register allocation
has not been very good in compilers historically.
David has shown this quite dramatically
because it has its own custom ABI internally,
and you wouldn't be able to do that with the compiler
like come up with your own internal ABI between functions.
So there's certainly 10% plus on the individual function,
speed gains versus doing it in intrinsics.
Some bits of some instructions are not available in intrinsics
like always.
It's a compromise.
Overall, it's been the way in FM Big X264
for the last 10 years, and I think all intrinsics
and in line assemblies banned,
and there's only one or two bits left,
and there's a very good reason why it needs to be there.
I have mixed experience about this.
I agree on the...
Ideally, assembly is better,
but we had some code in 3.6,
we compiled it with the latest Clang, 15,
and we saw a 15 to 20% speed increase.
But did you try writing it to begin with in...
Yes, it was in 3.6.
Write it in...
Write it originally in assembly and compare,
but it's...
So for example, some of this...
Sorry, you've gone to...
Some of the bit-twizzling in there,
for example, a compiler would never really have the understanding to do...
In fact, I did try chatGPT,
and chatGPT at least sort of understood a few of the concepts.
It's interesting because not quite out of a day job,
but I did ask chatGPT to write this function, actually,
just sort of to see what...
And it did have some vague idea what was going on.
It didn't need to sort of be helped, which is quite interesting.
Yep.
Is there any collaboration between the multimedia,
the people who write the codex,
and the guys writing the compiler who tell them,
look, perhaps you could target certain patterns?
Martin is a collaboration between people writing the compilers
and multimedia community.
Yes, in ARM in particular, I think,
is Martin here?
No, Martin is not here, but Martin spends a lot of time
talking to the compiler community and the linker community
on mostly miscompilations is more his thing.
And I think, yeah,
and I think there is also some sharing of mostly around the C code,
if the C code is badly miscompiled
or thought of the wrong approach,
because you can see, actually,
and in some versions of the compiler will really do a bad job
on the C and the assembly can be 40 times faster,
and that's...
Don't know if that's something you can really trust
if one day you change compiler version
and a function that you thought was immeasurable
is now 40 times slower than it is.
And then the question from the internet is,
did you have the occasion to look at
RVA-SVE vector instructions for FAMPEG?
Wow, that's a surprise for this person,
because the next speaker is going to be talking about this entire topic.
Where is the next speaker?
He's over there, and the next speaker here, Remy,
will be talking about this entire topic.
Another question?
Yeah, I was wondering.
So, obviously, the runtime CPU capability detection
and dispatching of the right functions is desirable,
but I don't think it's necessarily contradictory
to having some amount of abstraction.
Like, have you, for instance, looked into the highway library
that is being used in some places
that is trying to provide some kind of abstraction
while still allowing to do runtime dispatch?
So, the question was, have you looked into some of the abstraction libraries
like highway that's trying to do a sort of compromise
between runtime dispatch and abstraction?
I think this question was already answered,
I think, two presentations ago.
Not with highway, but I think with a different SIMD library,
but there have been various approaches,
LibOil, is it SIMD easy?
Various different approaches.
And again, the result from certain FAMPEG-264,
it has been righted by hand.
It's written once, and you know almost certainly
that it's going to be usable for a long time.
I didn't really talk about it, but the abstraction,
there is a lightweight abstraction layer in X-264 and FAMPEG
to try and basically to handle 32-bit, 64-bit,
and to handle other things like the different ABI cores.
The abstraction layer kind of handles
some of the future-proofing in that respect,
but there's a blog post online from Ronald,
if he's here, but he's not here.
He explains some of this.
It's another presentation in itself, unfortunately.
For your benchmark, do you know which optimization
the C-code was compiled with?
The question was, for the benchmark,
what optimizations were the C-code compiled with?
The GCC-03, varying versions of GCC.
In FAMPEG test suite, there's all sorts.
I think from GCC, there's a whole range,
depending on the build OS, but from 4 to 12, I think,
and maybe some people test nightly.
I think Martin certainly tests nightly for ARM.
I don't know if anyone tests nightly on X-86.
Some are LVM as well.
But again, I would be very surprised
if a compiler would be able to come up with something,
because what a human wrote,
because this is involving bit properties of the actual packing,
and actually the trick with PMAD SW is a kind of trick
to try and do a multiply and a zeroing at the same time,
and it probably doesn't have the level of thinking
to understand the bit patterns internally.
Something like chatGPT might one day,
which would be quite interesting, but I don't think the compiler does.
The last question.
I'm just going to follow up on what you said.
If you have a small algorithm, a small function like 10,
100 clients, maybe,
writing in the assembly might be easy,
but if you have a huge function,
like a filter, a variance filter, or something, a VCT,
writing it directly in the assembly might take a long time.
That's why originally we write it in C,
and then we try to write it in intrinsics.
So the question is,
a longer function might take a longer time to write in assembly
compared to C or intrinsics.
Yes, but there are DCTs and FMPEG,
but they're macroed, right?
Steps have macros to try and help that.
Again, the abstraction layer also adds, I think, macros
on top of what the normal assembler does in terms of macros,
so the blog post explains,
but swap is kind of interesting.
It lets you swap registers,
but then continue with them,
and the layer just handles all of that internally.
There's also just macros for, like, clipping.
I think it was on the example,
but clip is an example.
So clipUB is a macro,
and on the right target set,
it will go and use the right clipping functions
if they're available, for example,
and there's a bunch of these, I think,
that's how to fly. There's a few others like that.
Thank you, Kieran.