So, we continue with our next talk, again, about Codex, next time about VBC, with two
projects about encoding and decoding VBC, please welcome Adam.
Hi, everyone, so today I want to introduce you to VBank and VBdeck, those are open source
implementations for VBC.
Now, to pick everyone up about VBC, so VBC is this new codec that was finalized just
over two years ago, and if you want to know one thing about VBC, you basically, VBC allows
you to have the same quality as HEBC at half the bitrate, and on top of that, it was developed
by the Dravet, which is the joint video experts group, and it's called a versatile, because
it's applicable in versatile scenarios, right?
So we have support for screen content, HDR, as we heard in the previous talk, immersive
8K, and we can do some fancy stuff like doing adaptive streaming with OpenGob.
All right, so now let me talk you through a little bit of the background of our projects,
VBdeck and VBank.
So of course, those are both, you know, those are team efforts, right?
They're developed by a whole team of researchers at the front of our HHI, mostly in the video
coding systems group.
Now front of our HHI, if you don't know it, like modern video coding probably wouldn't
be what it is if it wasn't for HHI, and HHI is part of a biggest European research organization,
the Front of our Society, which is a big German non-profit.
And then about me, so I'm Adam Iskovsky, I've been at HHI since 2016, I've been leading
the project since 2019, and since about a year I'm also the co-head of the video coding
systems group.
So why did we even start the software project?
So basically, you know, there was HVC for which the test model was HM, and HVC uses
square blocks.
So they had this method of indexing blocks within, like within the frames using this
set index method, which is really amazing for square blocks.
And then, you know, they were exploring VVC based on the exploration model, which was
still based on HM, except VVC supports rectangular blocks, right?
So it's more than only square blocks, and there were really a lot of code for working
around this set index thing.
And at HHI, we wanted to do even more than that.
So we started work on our own partitioner, and we just decided that this is not going
to work.
And what we had to do, we basically had to write our own software to deal with it, which
we very creatively named the next software, which later became the VTM 1.0.
And basically, the biggest difference is we had one big map that was mapping the position
within the frame to like an object that was describing the current coding block.
So the next software, it became the VTM, which is the reference software for the VVC standard.
And you can see there in the graph how the VTM was developing over time with regards
to the gains over HM with the encoding time, decoding time.
And here, you can also see how we started our implementation projects from VTM.
So from the VTM 3.0, super early, we already started work on the VVDec.
And then from VTM 6.0, we started the work on VVNG.
Then in the early 2020, Benjamin Pross became my boss.
We basically started the VCS group, and he brought up the idea maybe we can do the project's
open source, which we did initially end of 2020 under a little bit shaky license.
But with some back and forth with the headquarters, we were able to change it to like a modified
BSD3 license.
And after some more back and forth, we actually have an unmodified, like a standard open source
license, the clear BSD3.
All right, so let's talk some more about the projects, you know, some hard facts.
So as I already mentioned, they are based off VTM, they are both written fully in C++,
but we do have like a pure C interface.
So you can integrate it into frameworks or, you know, just use it from a pure C code.
Those are C++ projects, so they are object-based, but it's kept very simple.
So we try to, you know, not hide anything, no getters, no setters, and like have all
the control over what is happening in the memory.
Contrary to some other, you know, some other projects, we do not do assembler at all.
We do only vectorization using intrinsics, which of course has the advantage that we
get stuff like ARM support for free through this amazing SIMD everywhere library.
Also support for Vasem cross compilation, so WebAssembly, which we also did.
And what we try to do, we try to make those projects as simple as possible.
So basically, we only expose options that are use case relevant, but like the coding
options, everything that's like connected to efficiency, we try to define it for the
user to just have the simplest experience possible.
Yeah, they're both available on GitHub under the PSD3 close clear license, as I already
mentioned.
All right, so how do we do the development?
The development is done internally without HHIs.
We have our own main Git repo that we basically, from which we push squashed updates to the
GitHub repo.
Why is it internal?
I know many people find it a little bit, you know, there might be issues with that, but
you know, here on the right, you can see, or maybe you cannot see, a typical magic
quest that I would, you know, do on the internal stuff, on the internal repo, and I think this
is just not something that is ready to be released to the public, you know.
So I would rather, you know, hide those kind of hiccups, and yeah, it just takes too much
time to make stuff nice for, you know, for being public.
And I also think not everyone at HCI might be comfortable, you know, being like a public
developer.
But yeah, all of the stuff that we have internally, it eventually goes to the public repo, either
for new releases, to fix bugs or issues, you know, we develop a big new feature, we would
push it, and you know, if someone was to make a large contribution, of course, to rebase
the code.
All right, so VBDec, the decoder project.
So the highlights, it's fully compliant with the main 10 profile of EBC, of course, give
or take a few bugs, so actually it was supposed to be fully compliant since the version 1.0.
But you know, stuff happens, we find bugs, we fix the bugs.
But basically there is no feature that is really missing.
It's multi-platform, so it works on all the major, all the major OSs, and it also works
on different architectures, thanks to Zimdi everywhere, as mentioned.
So we have, like, x86 support, ERM, I also saw people, you know, doing builds for RISC
V.
So this is very nice.
Also one thing I'm kind of proud of also is that from the first version we have like
a unified thread pool.
So also something that was already talked about, basically everything is, like, all the tasks
are collected within this one thread pool that just balances itself, right?
There is, like, no frame threads, slice threads, which also means we are multi-threading is
independent of bitstream features, right?
We don't need tiles, slices to be able to parallelize, so I think this is a really nice
feature.
All right, let's look a little bit into the development history.
So this on the left is, like, the performance graph for, you know, different resolutions,
different coding conditions, so like random access or all intram, and, you know, major
milestones mentioned 1.0, full compliance with the standard in the version 1.3.
We did a three times memory reduction, and, you know, I was also asking myself, how could
we ever release any other, how could we have ever released any other version?
But yeah, at least we managed to get it better.
1.4 we got a major performance boost based of external contributions in GitHub, so this
was really nice to see.
And in between, we really had a lot of, you know, small improvements, and as you can see
in the graphs, it does add up.
All right, about the VVank, so VVank, you know, it's an encoder project.
Of course, it's way more complex, way more interesting.
It has way more degrees of freedom, right?
Like the decoder just does this one thing and has zero degree of freedom, right?
Basically the standard tells us exactly what to do, and the encoder has all of those choices
that it has to do.
Anyway, what I like to say is, you know, it's basically the best open source encoder out
there.
As you can see here on the right, it's runtime versus efficiency.
So, you know, for a given runtime, we can have the best efficiency, and for any of our
working points, we can get this efficiency the fastest.
Of course, you know, it's not the best encoder if you want to encode UHD live.
We're not quite there yet.
But, you know, at those slower working points, it really doesn't get better than that, at
least not that I am aware of.
All right, so we have five presets on the encoder, from faster to slower.
And you can see, this is a single-threaded graph, you can see more or less how those
presets compare to XO65 in orange.
With very efficient multi-treading, at least like between, you know, up to eight threads
or with up to 16 threads with some additional options.
I'm going to talk about this a little bit in two slides.
And we have a very good optimization for human visual system based on the XPS and our metric,
which I'm also going to mention a little bit later.
We have really excellent rate control, so, you know, with bit rate deviations, rarely
more than 2%, and like almost never more than 5%.
As I mentioned, simple interface, and it's, you know, this thing can be used for academic,
amateur, commercial uses, the license really allows it all.
All right, so we have those five presets.
How do we derive those presets?
Like from an academic point of view, this is actually done in a very interesting way.
So we take all not use case-related options of the encoder, and we do, like, large-scale
optimization.
So we start with a very simple, very fast working point, and then we try to derive which
option should be changed as the next one.
And this is always the option which basically gives incrementally the next best working
point.
Right?
And this is a huge optimization.
It takes really a lot of compute time, so we don't do this so often, like every two
or three versions, or if we know one tool got implemented way better, we can, like,
try to only optimize for this tool within the option space from the last optimization.
We target HD and UHD, natural content, but we do sanity checks for resolution and, like,
screen content or, you know, HDR.
And the one issue that we still have is, you know, at the beginning here, you can see that
our curve gets a little bit steeper, right?
So we cannot go too fast, because at some point, for, like, every two times speed up,
we're just losing too much efficiency, this is because, you know, we started from the
reference software which was designed for the ability, and the efficiency is still work
in progress.
All right, about the multi-treading.
So our multi-treading is also, I would say, done differently than in many other encoders,
so we do the multi-treading over CTUs, so, like, the modern macro blocks and CTU lines,
like we simulate a wavefront parallel processing without using the syntax, and we parallelize
independent frames, right?
Like two frames are independent of each other and the references are already done, we can
do those in parallel, which, of course, means that, you know, how much we can parallelize
depends on the number of CTUs that are available, which are always more in high resolution or
with smaller CTU sizes.
That's why the faster and fast presets, which have smaller CTUs, they parallelize a little
bit better, which you can see on the top right there in the full HD parallelization efficiency
plot, right?
You can see, like, after eight, it kind of, well, it doesn't saturate, but, like, after
eight threads, there might be better ways to utilize the resources than to just enable
more threads.
Exactly, and how can we improve the scaling?
So we can improve the scaling by enabling normative features of EBC, so either doing
tiles that is independent regions within the picture, or enabling the normative wavefront,
which allows us to kill one dependency within the encoding, and this is on the bottom right,
so you can see, you know, if we enable those additional features, the multi-treading scaling
actually gets much better, but it costs between three to five percent of bitrate overhead.
But still, even with those other features, the encoder is not ready yet for more than,
let's say, 32 threads, so, like, you know, if you have a really, really big server, the
encoder will not be able to utilize all the cores.
Yeah, and about our optimization for the human visual system, it's based on the XPSNR, which
is, you know, this new metric that a colleague at HHI developed.
It has a really high correlation with most based on some public data sets.
There are publications.
You can look up.
They're mentioned on the bottom left, and it has been contributed to FFMPEG as filter,
and I think it is somewhere in the backlog of FFMPEG waiting to be looked at.
You can see on the right here a lot of graphs, so basically, in the last JVET meeting, no,
the JVET meeting one before last, there was a verification test with actual human subjects,
where we, you know, where VTM was tested with, like, the new compression technology, and
we submitted VVNG to be tested alongside of it in the slow preset, right?
So the slower preset has around the efficiency of VTM.
Slow preset is objectively five percent behind, and as you can see in the graphs, VVNG in
orange matches or outperforms VTM, which means that our visual optimization is well able
to at least close this five percent gap, if not even, you know, add more in terms of visual,
like, subject to visual quality.
So yeah, that's really nice.
All right.
VVNG in practice, you know, everyone asks in the end, so what kind of FPS can you achieve?
So we did some encodes onto mobile workstation kind of computers, encodes using all defaults,
which means eight threads, and, you know, for HD versus live, it's like, for faster,
it's around times four, like, live times four, so around 15 FPS, medium around lifetime,
times 30, right, so around two FPS, I'm talking about HD 60 FPS as live.
For UHD, the faster can do, like, 15 times live, fast 30, and medium, well, let's just
say medium would only be of interest for, like, large-scale VOD encodings, right?
But you know, FPS also depends on many other factors, like bit rate, content, you know,
your actual CPU and stuff like that, so this is more like ballpark numbers.
Excuse me, is this HDR content?
It is not HDR content, but it's 10-bit.
So it should be roughly the same for HDR, like we only do, we only test with 10-bit usually.
It works for 8-bit, but it's kind of 10-bit native.
All right, also some version history for the VVNC thing, for the VVNC project.
So our first major milestone was the 0.3, where we added frame threading, and you can
see on this multi-threaded efficiency versus speed graph that it was, you know, a huge
leap for us.
In the 1.0, we added the pure C interface, allowing, you know, the integrations into
pure C frameworks, 1.4, scene cut detection, 1.5, we added, like, arbitrary intra-period,
so it doesn't have to be aligned to anything.
We added a fast decode preset, and in the newest version, the thing that I really like
about it is that we added the ARM support, and, you know, every version, we had improved
ray control, things also to, you know, great community feedback, and, you know, from one
version to another, your encode might be 10% faster, but if you had a ray control problem
and it got fixed, it's going to be, you know, like, way, way better.
So this is, like, this hard-to-quantify improvements are actually one of the most important ones.
And you know, you can see how the curve behaves, extrapolating, I'm sure we're going to get
even faster and better in the future.
All right, about the ecosystem and the community, of course, you know, this is raw video encoding
and decoding, this is really only of academic interest, right, like, it doesn't really bring
you anything.
That's why we have been looking into FFMPEG support for a long time.
There was an open access paper over one and a half years ago that described how to do
it for the decoder.
There are some patches in the pipeline with FFMPEG.
We also put in our wiki how to apply them manually, if you, you know, if you want to
build it, you know, I've talked about this a lot, but the thing is, you know, if it's
not in FFMPEG, it doesn't exist.
That's why, you know, we put up a, like, how-to for you on how to do it.
All right, about playback, once you get this FFMPEG with VBC included, you can just link
whatever player uses FFMPEG as its back-end and it's going to work.
As far as I know, for VLC, you might force it to use the FFMPEG as the demuxer.
Not sure about it.
I didn't test it myself.
It comes, like, from community feedback.
I know there are some exo-player integrations going around, which allow you to, you know,
to use it in Android apps more easily.
We have a toy web browser example, but it's nothing like the VLC.js.
It's really a toy example.
And for maxing and demaxing, you can just use GPEG, sorry, for maxing, dashing, you
can just use GPEG since the version 2.0.
And I think it also needs to be linked to an FFMPEG with VBC support.
All right.
For the community, we do our open to external contributions and we wish to get some.
That's also why I'm talking to you, to get some interest.
So we try to, you know, make this more easy by, you know, stating that the authors of
VVNC are also retained at copyright.
We don't have, like, a contributors agreement, so this is, like, the only way we can make
it happen.
We are interested in all kinds of contributions, you know, efficiency, speed-up, and, you know,
we're going to, throughout the review this, test, generate, result, whatever is needed,
give proper feedback, march, so, yeah.
Please do if you're interested.
To conclude, you know, if you just entered the room, I talked to you about our open source
implementations of the VVC standard.
And I'm looking forward to, you know, contributions, also results, tests, if you want to try it
out, and general feedback.
Thanks a lot.
And a question in the room, yeah.
What confused me, because I know a little bit about the backstory, so H265, right, was
the super proprietary and the royalties, and the Alliance for Media Launch, the AV-1
company.
Yes.
And now your colleague is open source and free, right?
So to recap.
Why is it H266, is the free?
To recap the question, the question was H265 was a proprietary codec with licensing, AV-1
is a non-properary codec, and then I'm talking about VVC, which is the successor of HEVC,
so H265.
And there is a small confusion, so I'm not talking about the codec as being open source,
it's about the implementations of it, right, so you have to differentiate between implementations
and the technology itself.
There will be licensing for the technology itself, but, you know, there is still open
source implementations also of HEVC, so that's kind of the way I would like to see it, you
know.
So it won't be like with MP3, but also developed, it will be completely free to use, forever?
No, it won't.
I mean, the software is free to use, but, you know, if you build your streaming service
based on the technology, independent of which software you use, you have to pay royalties
for the technology.
Because you have patents, but patent issues?
Yes, I mean, there is, you know, this technology cost stuff to develop, and people want to
get their investment back, you know.
Okay, disclaimer, I'm doing, yes, in the optimizations for video codec in general.
My focus is on, and my comment to you is using a library like Cindy Everywhere, it works.
It's going to give you the initial results that you want, but you will never get the
optimal performance out of your hardware.
We had some really, really good examples of code, particular algorithms like more instructions
in forensics, a move mask type of in forensics, that are very common in popular in Intel.
If you try to port them with Cindy Everywhere or some kind of abstraction layer to ARM,
you're going to have to emulate this behavior, so ideally, you have to provide some way to
provide optimized functions for ARM or any other future architecture.
Otherwise, you're just going to have your Intel layer transferred, translated to ARM.
It will work, but you will never optimize your software.
So to recap, to recap the comment, there was a comment that Cindy Everywhere works, gives
a nice initial, you know, initial deliverable, but will never match hand optimized assembler.
We are...
How do you know assembler?
See, see the corresponding ARM intrinsics.
I mean, when I say assembler, I mean, intrinsics, so, you know, we are aware of it, and we
are looking a little bit of it, you know, there are different kinds of intrinsic kernels
that are implemented there.
You know, sometimes you have like two pieces of memory that needs to be added up and stored
somewhere else.
There, Cindy Everywhere works really nice, but when it's like lookup tables, shuffles,
it works worse.
We are aware.
We are looking into, you know, identifying the kernels where there's the biggest potential
for improvement and doing those manually.
We are looking at HDR, implementation of a ARM of VBX, okay, so I'm doing the ARM and
VMT optimizations for that, and the way to do the transpose, for example, and that is
completely different to the way it is, especially with, because for AVX2, you have 256 bit-wide
registers.
You don't have that with VM, but you have other instructions to help you try to tackle
the outcome.
Yeah.
So my point is that if you are open to contributions with specific optimizations that you might
find people to help with that, but if you restrict yourself to only using the library
and only use Intel intrinsics and then translate that one using the library, you might lose
some performance.
A lot.
We are not restricting ourselves, you know, to only using this.
It's just because, you know, we only have so many resources, this was the fastest way
to do it.
We can play it out on, you know, like actually this thing can play out VBC videos with our
software, but yeah, totally.
I think there is, there would need to be some changes to the build process, maybe to that
like structure of the project to enable this, but yeah, this is something really very interesting
and I think there will be a lot of research going on in that direction.
Thanks.
One last question, yeah?
How does it compare, what are the advantages of AV1 in terms of compression or computation?
Well, yeah, so the question is, what are the advantages of VBC or VVNG?
VBC over AV1.
All right.
So what are the advantages of VBC over AV1?
So VBC is a successor to HVBC, right?
It was done by people who were really knowledgeable on how to make a standard work.
So you know, it's the one thing, it just provides the additional bitrate savings, right?
So here you still see like 20% additional bitrate savings over like the best case of
AV1 and there isn't so many of these initial hiccups, right?
So the HDR support just works, you know, immersive stuff just works a little, like a lot of
those things, they just work.
And then we can do stuff on top of that, like doing open-gop adaptive streaming, which allows
you to reduce the bitrate by like another 10% on top of that, right?
Like with all the other standards, I think the adaptive streaming can only be done with
close-cops or with a prediction break.
I would say more mature, but you know, I know there are different views of this, more compression
efficiency and really versatile mature usability.
Thank you, Adam.
Thanks.
Thank you.