Hi, how are you doing? Welcome to FOSDOM. Congratulations on managing to get inside a room. This is the
largest one I've ever seen. Usually it's just looking at the doors of ones that are full.
So yeah, my name's Daniel Stone. I'm here to just give a relatively high level overview
of the graphic stack. My hope with this, like I said, it's fairly high level, is to give you a
decent understanding of all the different components that go into the modern graphic stack,
how they fit together. So if you're trying to work with it anyway, you won't be trying to
debug it because it's already perfect, but just being able to give you a good understanding
of how everything does fit together. And now we have graphics output working, so that's a good
start for this talk because that wasn't looking likely five minutes ago. Right, so the graphic
stack looks like this. Any questions? That's the simplified version as well. More sensibly,
if we try to build it up incrementally, just try and work through all of the different pieces
and different components in essentially the order of near to far, which is, you know,
in networking you think of upstream and downstream, usually in the graphics for the lot of what we
think of is what's close to your eye and what's far from your eye. So in our case, the display
is closest to your eyes, and this one's incredibly bright. In between, just underneath the display,
controlling the display and giving you determining what should be shown, we have the window system
layer, so that's your Wayland. It can be X11, but we don't talk about that. And then at the very
back end, at the upstream side, you've got the clients, which are actually presenting the thing
that you want to show. But then it turns out that your window system also uses the GPU to render,
so it's not just OpenGL games that use accelerated graphics. It's the window system, so the nice
diagram already gets a bit muddied because we're breaking the layers. And then maybe the window
system uses some media output because you want to stream stuff onto it or, you know, to stream a
conference talk, hello. And maybe one of your clients is also a window system because it turns
out that even Chrome is a Wayland server these days, so our lovely little, we have three classes
of three main components of our graphic stack. This illusion's already disappeared. But, you know,
let's pretend that everything is fine and let's just try to build it up. So for us, DOM and KMS,
the acronyms you mostly see, the direct rendering manager is anything to do with graphics or display
inside the kernel. It's a weird legacy name. And those are all of the GPU and display drivers.
And KMS is very specifically the part of DRM that actually controls the display. So when you're
talking about HDMI output or something like that, then it's going to be KMS. And KMS is that very
last step in the pipeline, the one that's closest to your eye. Its job is to turn pixels into light.
Some people will tell you that there's a thing called FB Dev as well, but that's not right. FB Dev
doesn't exist. And, yeah, in the division of responsibility as we go one step further back
from your eye, the Windows system's job is to fundamentally to take a bunch of images from
clients, combine them into a single image or multiple images if you have multiple displays,
get them out to the eye and bring input events back. So, you know, Wayland is a protocol and
nothing else. There's a very, there's a very small C layer in Wayland, which is really just IPC.
And apart from that, it's just protocols and conventions. So, you know, MATA, the GNOME users
is a Wayland server. Other popular ones would be KWIN, Western, WL routes. That's where all the
implementation actually lies. And, yeah, like I say, they just combine window images together,
get them out to the output device in the reverse direction they're bringing input back.
X11 doesn't exist either. So, that's, we'll move on. Yeah. So, OpenGL and Vulkan, in a way they fit
in. Their APIs, as we know for accelerated 3D, so you provide them a mesh and some textures and
some shaders. Run this thing, make it fast. Great. But they only handle rendering. So,
GL and Vulkan themselves have no concept of I want to be able to display to Wayland. That comes in
with EGL and what we call the Vulkan WSI for window system integration layer. Their job is to
bridge the two worlds. So, with OpenGL, you have EGL on the side that's the bridge between GL and
say Wayland. With Vulkan, you have Core Vulkan and then the WSI on the side is that bridge
bringing all the content across to the window system. And then there's GBM as well, which is
maybe the most ill-fitting part of what we have. GBM is kind of a side channel to bridge EGL to
KMS. So, right now, I mean, this is all happening through GNOME shell and MOTA. It's using GL to
render my image with the next slide as a bonus preview and this one that you can see. MOTA,
yeah, it uses GL to render and it uses EGL plus GBM to be able to pull images out to
kernel mode setting. And GBM is a really, really strange and idiosyncratic bridge. Some people
will tell you that GBM stands for the generic buffer manager. That's definitely not true.
Yeah, we had an idea that GBM would be the thing that let people kind of peek under the hood of
what EGL does as an implementation and be able to generically allocate buffers. We got as far as
making it work for kernel mode setting and then realized how terrible the whole problem space was.
So, we just pretended that it was never an acronym, that it's not generic and moved on with our
words. So, at the end of all that, before we get into something more meaty, we've got clients
rendering the content, maybe with the GPU, maybe just on the CPU, maybe it's just doing mem copy.
It will pass a handle to that content over to the Wayland compositor with some metadata,
some context. The compositor is going to pull it all together, choose how it's going to display it,
apply any kind of policy or what have you. And then it's going to just push that final image
out to KMS, which is going to turn it into electrons. So, we've got the diagram that's back
to making sense. So, if we're looking at how KMS is actually put together, every single discrete
device in your system is its own. I just have an Intel up top here. I have one DRM device,
which is the entire Intel GPU and display complex. If you're on ARM systems usually,
you're going to have two devices. The display and GPU are separate IP blocks from separate
vendors who aren't really on speaking terms. So, you'll have one DRM device for your display
controller and another DRM device for your GPU and they're completely separate. So, yeah,
four KMS devices. We've got connectors representing real displays. So, we've got an embedded
display port connector here and various display ports and HDMI connectors from my external outputs.
CRTCs, that does stand for CRT controller because that's how long ago it was when we
designed all this. CRTCs are the thing immediately upstream from connectors. They generate a pixel
stream for the displays. So, any kind of scaling, cropping, compositing is done in the CRTC space.
And CRTCs are just a combination of planes. So, planes, they take frame buffers. They
can scale. They can be positioned within the CRTC. They can be stacked. And then the CRTC
is the one that combines them. So, in quite a poor diagram, because for a graphics person,
I can't actually draw very well, more of a text person, to be honest. Yeah, it's the
frame buffer is just the client content. The plane is the one that's going to do any format
conversion or scaling or what have you. Then the CRTC combines them all together, pushes
them out to the connector. Then I think the important thing to bear in mind if you're
trying to reason about graphics pipelines is that timing flows backwards. Timing never
flows forwards. Because when you've got a physical display, it's going to refresh at
a certain point in time. Unless it's VRR, no one asked about VRR. We don't quite know
how that works yet. But timing flows backwards because this HDMI output is ticking at 60
hertz. That's happening at a very, very fixed point in time. And so that's the beginning
of our reference. When we know that we want to present stuff to HDMI, we know exactly
when the next refresh cycle is going to start, the next one after that, so on and so forth.
So timing is always flowing backwards. This goes right the whole way from the connector
back to the CRTC, back to the Windows system, and then back to the clients. It's always
starting from that fixed hardware source. So yeah, you want to use DRM and KMS. Good
for you. I'd recommend it. It's just a set of objects, like everything that turns out
in computer science. It's objects with properties, and that's it. So you open your KMS device,
you enumerate a list of objects, your CRTCs, your connectors, your planes, you look into
their properties. So this connector type is DisplayPort, this one's HDMI, whatever. And
then any time you want to actually affect something, so display new content, change
resolution, whatever, that's all done through what we call Atomic Mode Setting, which is
about 10 years old now, and it's a very low-level property-based interface. I wouldn't really
recommend trying to drive it yourself, but it is possible. So Atomic is just a list of
properties. So you've got all of your different objects and their different types. You know
how you want to put them together. You know that I want this plane to go to this CRTC,
to this connector, and so you take all of those objects, you do a massive property set,
and then you do an atomic check before you commit just to see if the configuration is
going to be accepted. One of the things about display hardware is that it's weird. It's
really, really weird. There are infinite constraints on what you can actually do with the display
hardware. So you might have three or four planes that you can use to composite content
without using the GPU, but you can only use a couple of them at a time, or only one of
them can have compressed content, or only two of them can be scaled. So because we don't
have a good generic way of expressing these constraints and of constraint solving within
the kernel, we do the dumbest possible thing. It's brute force. We just try every possible
configuration that will get us to where we want to and see which one's going to stick.
Then yeah, once you've gone through all that, you've done your atomic commit, you've got
a frame on screen, it lives there until you change it. Because DRM is, it's a frame by
frame API. It's not a producer-consumer where you connect a camera to an output and magic
things occur and you get a video stream. You know, that's the domain of high-level frameworks
like say PipeWire and Gstreamer have that pipeline concept. DRM is quite dumb. It just
does what you tell it to, and it doesn't do anything else until you tell it to do something
else. So yeah, we've essentially summing up, you know, we've enumerated all of our devices,
we've used the DRM to do that, all of the objects. And again, as with timing, we're
working backwards from the starting point of a connector. So we know that HDMI1 is the
thing that we want to light up, so you always work backwards from that when you're building
up your object tree. And then, you know, you are going to need a way to allocate some memory
to display. It's not just a malloc pointer. So we have Gem, the graphics execution manager.
It doesn't manage execution of any graphics jobs, it's just a memory allocator. This was
about the point where we stopped actually naming acronyms because we've got almost all
of them wrong. So Gem, you see a lot of, because that's the base of our kernel allocator for
all graphics and display memory. And BO is something you see a lot of as well. So really,
I told you about it, acronyms. So Gem BO is just, like a malloc pointer, it's untyped,
it's a raw bucket of bytes. It can be pixel buffers, it can be shaders, it can be geometry
meshes, whatever you want it to be. It doesn't have any properties or metadata, just a length
and some content. But you can't allocate them generically because hardware is really that
weird. We gave up on that a long time ago. So you're going to need some kind of hardware
specific API to come up with a Gem BO. And you might be quite disappointed about that,
which is reasonable. So we came up with dumb buffers as a specific class of Gem BOs designed
specifically for CPU rendering when you're displaying KMS. So if you have something like
Plymouth for your early start splash screen, that's not going to be using the GPU. It's
just going to be doing CPU rendering, no device dependent code. And dumb buffers are the path
to that there. I just wanted to get something up on the screen. I don't care if it's amazingly
fast or efficient, I just need it to work and work everywhere. So this is actually a
generic API inside KMS dumb buffers. Gives you a Gem BO, you can map it, you can fill
it up with some nice pixels. And then wrap that in a KMS frame buffer is what annotates
the BO with stuff like format and width and height and stuff that people think might be
important. So yeah, like I said, you can use it for splash screens. Please don't try to
use it for other stuff. It's not a generic memory allocation API either. It's just the
thing that works. So yeah, with all that being said, that's a reasonable end-to-end picture
of how to use KMS. You've allocated all the buffers you need or imported them from other
clients. You've attached those frame buffers to planes. You've stuck them on a CRTC to
get them in a kind of logical space and stacked against each other. You've set your CRTC
and connector up for the output path. Commit everything. Hopefully that works. Then the
kernel tells you that it's complete. You know when the next frame is going to be and you
just keep on going. You can't click these links if you're sitting in this room, but
they are clickable on the PDF. There's a bunch of pretty decent documentation examples and
formats because I'm not trying to show you the entire thing. Just give you a good idea
and some pointers. If you're bored of KMS or you just don't find display that exciting,
you might want to move on to the Windows system world. There's a super quick one through Wayland.
Again, it's the same thing. It's clients giving you images and you're giving clients pointer
and keyboard and top screen events in return. I think the main thing about Wayland that
people take a while to grasp is that it's descriptive rather than prescriptive. What
I mean by that is in X11, when you have a pop-up, you tell X as a client, put this window
exactly here on the screen. Give me all of the input events until I tell you otherwise
because you're dictating specific outcomes. Wayland is exactly the other direction from
that. The client tells the compositor, this is a pop-up. The compositor does the right
thing for pop-ups, including capturing input and making it always be on top, but still
letting your screensaver work, which is nice. It's just about the client annotating everything
it has with a bunch of descriptive information and properties and then relying on the server
to actually implement the right semantics. There's a fair bit of trust, but it gives
us much, much more flexibility because by the end after how many years of X11, we were
kind of painted into a corner really because clients were just dictating so much.
We tried to make sure that there were no pods in Wayland that required the compositor to
do a huge amount of work because it's such a critical part of the stack that you can't
have it burning loads and loads of time. Like I said at the start, your compositor could
be GNOME, K-Win, could be Western, Sway or something like that. They're all designed
for different things and different use cases like window managers in X11 were. I think
Western is the best one because I work on it. It's basically designed for everything
that isn't a desktop, literally planes, trains and automobiles, digital signage, that kind
of thing. It's really, really efficient and predictable and reliable, but I do use a desktop
so I have GNOME on this one. There are absolutely a pile of them to choose from, but they all
use the same protocol so they all look alike to the client.
It's just a large collection of essentially all extension interfaces. WLBuffer is much
like a frame buffer to handle to some pixels somewhere, no other information just width
and height. A WL surface is a window, can be a pop-up, can be an application window,
can be a subsurface. It takes the buffer, it just crops it and optionally it takes input
back. XDG surface is the main one you'd interact with really because that's what adds all the
desktop-like things of being able to resize and move windows and all that kind of thing.
WLC is where the input comes from because we're still bad at naming, it turns out. That
one was my fault actually. We did design Wailand fundamentally to be really, really easy
to extend so there are quite a pile of extensions that you need to sort through and deal with.
The nice thing is with it having been designed with KMS in mind, it's pretty similar. You've
got your compositor doing the final output at the end and that's composed of a bunch
of windows and surfaces which have got buffers attached to them. The compositor is the ultimate
source of the timing and it flows that timing back to the clients as feedback. If you take
that, it looks exactly the same as the KMS diagram we had earlier which is not really
any coincidence and using that is exactly the same flow as KMS. This slide was almost
copy and paste. Again, I'm not trying to give you a complete guide to how to write every
Wailand client in the world. Please do use a tool kit. They will make your lives much
easier so GTK, QT, STL, IMGUI, whatever. Use a compositor tool kit as well if you like.
Libwestern in particular and WL routes are tool kits you can use to build compositors
on top of good code bases. There's some links in here as well to Wailand info is a good
tool to inspect. WL hacks is a debugging tool. Western debug is another debugging tool. There's
some sample clients as well. The simple SHM and simple EGL are our kind of references
of how do I actually start using this and start approaching it. Now we've got all that
out of the way. I'm not going to try and explain GL to you because we'd be here forever. Like
I said, it's GL as a model for accelerated 3D is clients providing the vertex data so
you're kind of wire frame geometry, your input textures, material images, and your shader
programs as well to run to generate the final output. No shaders can deform the geometry
so you can do cool stuff. You can also do things like lighting per pixel and do that
in a nice reflective way that's all computational. I guess the main thing to recognize about GPU
is they're enormously parallel so thousands of threads, really. There's not much in the
way of synchronization or shared memory. They really, GPUs can't do branching like CPUs.
They want to have everything set up for them a long time in advance and just do straight
line things from there. It's a long, deep pipeline essentially and you want to make
that roughly as static as you can. The cost of being enormously fast and really, really
powerful, it turns out, is that they're really power hungry. That's why we have composition
in the display hardware as well because it turns out that just spinning up your GPU once
per frame to produce the final display output. I worked on a device where the video runtime
went from five hours if we didn't use the GPU to four hours if we did. It's a really
measurable cost to get a GPU involved. You only want to do it if you've got the right
reasons for it or if you actually need it. Like I said, it's just a pure 3D only API
when you talk about GL and GLES because it came out of SGI where you told it to draw
and it was drawing because there's only one screen and obviously it's going to come out
at the right place in the screen as a simpler time. Then SGI realized that they needed some
more nuance. They brought in GLX, which was the first go at integrating OpenGL with the
Windows system. Originally it had the X server processing all the commands. That was terrible.
We came up with the DRI for direct rendering infrastructure, not let the clients directly
access the GPU. It relied on central memory allocation. We came up with DRI2 where the
main innovation was that clients would manage their own memory in cooperation with the kernel
and also execute all of their own commands. That was so good that any time you see DRI
it just means accelerated rendering, so roughly describing the last 20 years. Any time you
see DRI2 it doesn't mean actual DRI2 in X11. It just means this kind of looks like a modern
Windows system by which I mean about the last 15 years. That can be confusing because those
two terms are massively ambiguous, but if you ever see DRI2 it probably means that you're
somewhere good. Then yeah, EGL is an abstraction of GLX. Rather than just plugging GL into
X11 it lets you do Wayland, Android, whatever. All it really does is give you Windows that
you can share with the Windows system, gives you some vague notion of timing, but it doesn't
have any kind of events, so the only way you can get a consistent frame timing is if you
block a lot in EGL. It just tries to hide everything and make it implicit, which again
is where GBM comes in because that's what lets us steal buffers away from EGL, push
them into KMS for display, handle our own timing and do it properly this time. EGL has
that shape and then not coincidentally Vulkan has a fairly similar shape. Vulkan is the
rendering API and that's it. Vulkan WSI is the EGL equivalent which provides that Windows
system integration of creating Windows, posting content to them and so on. The main difference
with Vulkan is that it's really, really explicit and clear about what it's doing. The downside
is that because it's so explicit and clear you end up typing a hell of a lot of code.
So it's more effort to use, but there's no magic hidden under Vulkan. You know exactly
what's going on for better or worse. It's really good on the desktop that on mobile
SOCs the hardware isn't necessarily entirely there yet. If you're doing high performance
things or you just like seeing what's going on under the hood, I'd recommend Vulkan.
And yeah, I think about the last bit that we'd end up having time for is I keep on going
on about how we, you know, just saying that EGL will get things from GL to Wayland. The
way we do that is DMA buff. It's a kernel concept about sharing memory regions between
different subsystems, different processes, different contexts, whatever. So, you know,
we've already got in the graphics side of things. We've got the gem buffer objects,
but they're local to one particular device and to one particular user context. So, you
know, when you want to export a buffer to your Wayland server or share it between, you
know, V4L for your video capture and, excuse me, sorry, V4L for your video capture and
your GPU to do some analysis on it. That's DMA buff, which just gives you a file descriptor
you can use as a handle to that memory area and import it into different contexts or subsystems
or places. And that's completely consistent throughout the stack, like all of Wayland,
EGL, KMS, Vulkan, everything I've discussed has DMA buff integration because that's our
lowest common denominator. So, yeah, we put it all together. I mean, because they're all
built on the same building blocks, it's largely how you think it is. Well, hopefully if I've
done a decent job of this talk, you know, the client's connecting to the compositor.
It's creating a window declaring some very simple annotations about that. It wants to
use the GPU, so it creates an EGL context pointing to the Wayland server. I'd like to
render over here. The Wayland server has some DMA buff protocols, which tells it what it
can and can't accept. The client uses GLES to render into that. That's wrapped in a
DMA buff and passed over to the compositor. The compositor is deciding how to place and
configure everything. It's importing that DMA buff that it's got from the client to
generate one final image. It's then waiting until the next deadline, you know, that sort
of 60 hertz cadence that we have. It's waiting until the next deadline to present that out
going into KMS. That might be KMS doing its own composition directly in the display hardware
or through the GPU itself. It's tough because the display hardware can do that final image
composition of taking your sort of four or five images, mashing them all together and
coming up with one. It is, like I said, a really measurable win on things like power and memory
bandwidth, memory usage as well, but it's kind of complicated in that, you know, it's hard
to know, be predictable about when you can and can't use it. It's a bit fiddly. It's
one of the reasons I recommend using compositor frameworks like LibWestern, which do do all
of this heavy lifting for you. You know, I've spent 10 years of my life trying to solve
this problem and wouldn't recommend anyone else does it. It's not even really that interesting.
Internally, Western has, like I said, that kind of brute force loop of just trying every
possible configuration that could work, seeing what happens and throwing it at KMS to check
if that will work. Currently, that's the most advanced one, but yeah, others are catching
up. I think really to sum up what I was trying to say about GPUs and efficiency is one of
the things that gets collaborate a lot is that no one realizes that every problem on
mobile comes down to memory bandwidth. And so you can solve every problem by just copying
buffers around more. But when you've got 4K buffers and you've got a low-end device,
it turns out that this is always where your performance problem is. It's down in things
like copies and naive memory usage. So yeah, that's just one thing to really be aware of
is try and go for a zero-copy pipeline because when you have 4K and 144Hz, you really don't
have much time and you don't want to spend it all just waiting for slow memory. Yeah,
with that, I think we're pretty much coming up on time. So yeah, there's the quick whirlwind
tour of how all that fits together. Anyone has any questions or wants to talk about how
Wayland's amazing? Please feel free. If you have any questions, please raise your hand.
When we launch a game in full screen, for example, does it go straight from GPU to screen
or does it go all the way through KMS on that? It will go through the Windows system. So
yeah, the question being, if you have a full screen game, will it go straight from the
GPU to the display or will the Windows system still be involved? It will still be there,
but ideally doing nothing. So it will just take the client buffer, give it directly to
KMS and ask KMS to display it in the happy case. But it's always involved as the mediator,
so when a notification pops up, it already has control, so it can show it.
Hello. I can't. Is it working? Yeah. Okay. So forget the super new big question. When
you say the frame buffer is tied to a plane, a plane is not a desktop, a plane is just
a window. When you tie a frame buffer to a plane, the plane goes in the compositor.
So the plane is a window, it's not the entire desktop. Yeah, exactly. So the CRTC is your
final output as one flat image and planes are windows within that CRTC. Thank you.
More questions? All right. Hello. Is it working? Hello. You mentioned that kernel mode setting
is used turning the pixels into... Sorry, could you please... Sorry. Yeah. You mentioned
that KMS kernel mode setting is used to turn the data into pixels on the screen. Is this
where graphics card drivers are involved, another vendor-specific software, or is that
earlier or later in the pipeline? Sorry, which parameters? So basically, where did graphics
card drivers come in? Because I know there's like vendor-specific hardware that requires
its own drivers somewhere in kernel space, I believe, so what does this fit in the pipeline?
So all of the properties and parameters are defined in kernel space, and we try to standardize
them as much as possible. So in the generic world, we do stick pretty religiously to a
standard set of parameters that have common behavior across everyone. If you go to things
like Android where you have hardware composer and vendor-based tells, it's completely different.
And they're all... That's more of a negotiation between kernel and user space, which are
both vendor-specific. That answers your question.
Do you know if there's any toolkit libraries for writing compositors that are not desktop-specific?
Any compositor libraries that are... Libraries for writing compositors that are not desktop-specific.
So it's like LibWestern is good for writing desktops, types, things, but for highly embedded
use cases, I've found any things that make it easy to write a compositor like that.
Yeah, so LibWestern's the one for those kind of embedded or single-purpose use cases. MOTA,
which is the basis of GNOME shell, can be used by anyone else, but it's really GPU reliant.
And WROOTS is, I guess, kind of in the middle. It's not as friendly and desktop-y as GNOME,
but it's not as sort of insanely efficient as Western, and that's the halfway house,
I guess.
Is there any tool you would recommend for profiling?
Sorry, could you speak up?
Is there any tool that you would recommend for profiling, the graphics tech? Is there
a tool for profiling the graphics tech?
Profiling, are there any tools for profiling the graphics tech? Kind of. So Mesa has integration
with a tool called Profetto, which is the basis of Android GPU Inspector. There's some
support in there for Western, specifically, to interpose its timeline on top of Profetto,
but it's pretty patchy, to be honest. We've been working on that basically to try and
make it easier so we can stop getting paid for debugging and profiling stuff, to be honest.
But yeah, it's a slow process. Profetto is the best one there.
I have a question. So why can't we do screen recording or screen sharing in a Bayland?
You can. Screen sharing in Bayland is done through the XDG screencast portal and we did
that because once, if you try to put it in Bayland itself as like a core protocol for
clients to use, it was really going against the grain because everything was designed
with this idea of the timing coming from the display and flowing back to the clients.
And then once you put it in the other way that the client's receiving content, it really
just is a terrible fit with pretty much every interface we had. So it's easier for us to
and also working for like sandboxing and containers to go with the XDG portal solution. And yeah,
it works every way basically. Okay. I think, yeah.
Okay. Thank you, Daniel. Thanks very much. Thank you.