Hello. Hi. Everyone, please take a seat. We're about to, well, we are starting. This session
is OpenJDK Project Weight Field, the Whalen desktop for JDK on Linux. It's going to be
a bit of a show here because we have three of us, so we're going to swap over. There's
only two mics. We'll do our best. Very quick intro and then I'll turn it over. So I'm
Phil Race. I work at Oracle. I'm the lead for the client libraries on OpenJDK. Next to
me is Alexei Ushakov. He actually used to work in the Java 2D group at Sun a long time
ago, but these days he works at JetBrains. And there's Nils DeGraf, who's a product manager
or product owner, product, whatever, at Red Hat in the desktop group. And so he is going
to do our first session. I'll hand it over to Nils.
Okay. That's good. Maybe I should.
You should. I should.
This is going to be very interesting to do.
Yes. I did not sign up for some dancing, but it's fine. The first, quickly about structure,
we're first going to quickly explain what Whalen is and then explain how OpenJDK, then
it tries to work on it and the whole weight field project. And then finally we have an
actual demo and some explanation by Alexei. So quickly for Whalen, because we don't have
that much time. So a bit closer. Okay. That should be better. First of all, what is Whalen?
What is X11, for example, that it tries to replace? So it's about displaying. So rendering
things into a, let's say, a matrix of pixels, something called, which sometimes called a
frame buffer. And you usually try to get that on a stream, but maybe also try to stream
that somewhere over the internet. And why do we need something fancy around that? It's
because once you have multiple applications trying to render something, you want to have
something like some kind of decisions around, okay, do we put them next to each other on
top of each other or that kind of thing? So basically window management or tiling or whatever
you want. And that's where a display server comes in. And you talk a display protocol
between the apps and a display server. And it's usually also very related to input management,
because, you know, if you have typing something, you want to go to your browser, for example,
and you don't want to type your browser in some key logger application or something.
So quickly about X11, which is, let's say, the old thing. So X11 started from the white.
So we have the X11 server, which you start up normally using something like startX, which
is going to listen to a socket, usually based on the display name, usually something like
zero. And then each of your applications, I mean, imagine those two top applications
being your browser or your file manager. And then they will actually say they will also
connect to that socket. Sometimes if you have to do a display equals variable for the environment,
and that's when which kind of socket it will try to listen to. So it will define the server.
Now you're going to say something like, OK, now please X11 server, can you create me a
new window? So X create window or something with this window, with this width and height.
And then you can do your fancy things that you normally would expect to be able to do
from a window manager. Now, that whole logic of should we be doing tiling or should we
be doing overlapping windows and so on. That's usually what another X11 client, which is
then usually the window manager comes in. And that actually helps in setting all the
logic, let's say. So that's how the usual thing in X11 goes. Now X11, a very oversimplified.
X11 is old. It's from the 80s. Now being old is not necessarily a problem. But it is older
than, for example, Java and Linux. But one of the things is that it made a lot of assumptions
that don't necessarily hold anymore and that are baked into the core protocol. For example,
it talks about a root window, which we, once you have multiple monitors, you can still try
to have like one big frame buffer that spans all of those monitors. But if you have multiple
TPI, you can do trouble. Once you have GPUs get complexer and complexer, there's overlay
planes and you want to do fancy stuff with that for performance reasons and battery reasons.
There's security. X11 allows you to screen share anything and do input sniffing and snooping
and spoofing without any kind of consent of notification. There's a dev room about Linux
on mobile. I do not want a device that actually could do all of that with my private data.
And there's also new UX paradigms like HDR, tiling window monitors and so on that actually
make it be a bit harder, especially HDR. It's very hard to do, for example, in X11. So at some
point people got together to create a new display protocol, which is that Wayland. So it's very
minimal, but really, really minimal. It really tries to make sure it does not fall into the trap
of making assumptions again. It goes to just say, okay, we have clients that want to send
something rendered to a server, a compositor, let's say, and then we can then extend things over.
So it doesn't even have a concept of top level windows, for example. You actually need an extension
for that. It's called the XZG shell. If you ever want to look it up, it's very fancy. And some
things you just don't want to have that in the display protocol anymore. For example, screen
sharing is also related to videos. So we said, okay, let's try to put that out of the protocol,
the core protocol and something with portals. But I will explain what portals are later.
So what does a typical Wayland session look like? It's again, we start from the right. So we have
the Wayland compositor, which you start. That's going to be, for example, with GNOME. It's going
to be GNOME shell with KDE Billy Quinn. This way it will be something else. And then you will start
a Wayland socket, which you can again listen to. You will talk to Wayland protocol. And a Wayland
client will say, okay, please create me a surface. And then using that extension, for example,
the protocol extension using XZG shell, you will then have something where you can say,
I want to create a top level window and I want to do it this size. You can do positioning
Wayland. Always fun. There's a lot of reasons for that. And for example, another Wayland client
can be X Wayland, which is its own X11 server. So actually, inside your Wayland session, let's say,
you can actually also then create multiple X11 clients, which will talk the whole X11 protocol
and X Wayland to the translation to Wayland itself in the best way possible. And I did lie a little
bit that this is not, or I did say that there's not everything yet. So there are some things that
we don't want to put into a display protocol anymore. We want to do that in the portals. So we
did something that that's new with Flatpak and snaps and all these containerization fancy methods.
We want to make sure that there's some kind of permission thing, let's say that allows you,
for example, I want to do screen sharing, let the user choose if that's okay or not. And then
the deepest interface within a Flatpak, you can access that. And then, for example, go to Pipe
Wire and other components which do not necessarily need to live in the compositor. You will go to
those and then we can go to the next step, which is how portals can be maybe implemented, or,
you know, how this can be used from within Wakefield. And I think that's going to be the part where
Philip is going to come in. Okay.
So Neil's described what Wayland is. And today, what we now have is a project to be able to support
that Wayland compositing desktop from openJDK. And what's it all about really? Well,
JDK is clearly a platform. It's not just an application and it abstracts away that
underlying platform. So we're not going to be exposing anything about Wayland. Today, on Linux,
it happens to be an X11 client. It's basically an X application at a, you know, crude level.
But to be able to support that Wayland desktop, we need to make some changes in JDK.
Some of the policies that Wayland has that Neil's touched on around security
and things like that mean that the things that just are supposed to work in JDK won't work,
even if we're trying to be that X going by that X Wayland
product client that Neil showed on his diagram, right, which is what we call X compatibility
mode. So that's the Wayland's compatibility mode for X11 applications. And although we don't even
work in that today, even if we make it, even if we start to work in that, is that really the long
term solution that we want, what we really would like is to maybe, you know, be a full on Wayland
native client. So Open JDK project Wakefield, there's a URL there, has started a couple of years ago.
And there are people from Red Hat, JetBrains, and Oracle who are working on this. We have a repo
at the standard kind of in the standard format for Open JDK project repos. And what are our goals?
Well, first off, we're going to try to support that compatibility mode properly. So we'll have a
fully concomit, conformant JDK, and everything works as well as it can do, as well as it should do
on the Wayland, when you're running on a Wayland desktop, but we'll be talking the X11 protocol.
You know, we don't talk about it here, but, you know, most people will see that these days, if you
log into a, you know, Linux desktop, they, and if you pay enough attention, there's an opportunity
to switch between pure X.org and the Wayland desktop, which supports that compatibility mode.
And right now, JDK only supports the X.org. So the longer term goal, as I just touched on, is that
we want to be able to support native Wayland. So the X11 compatibility mode is some touch
things that I'm going to touch on, but the much bigger thing is that native Wayland port.
And there's a list of, there's a list here I won't read out of the different kinds of things
that we need to deal with in making all of this work. And some of what we need to do
for the native Wayland port is really just starting to emerge in the latest versions of GNOME.
So it's not, this doesn't, this work is not intended for some, you know, older versions of Linux.
This is something that you'll want to, or have to use on upcoming versions of Linux.
And yeah, the policies of security that Wayland enforces, I think is the right word,
are going to be some of the drivers for the things that we need to change.
For example, the issues include, one of the most important things is we have an API that lets you
capture screen pixels. And capturing the screen is, you know, Neil's touch on is something that
Wayland very early on, I think, and is very clear about, doesn't like people to be able to do for
privacy reasons. But AWT has expected to be able to do that forever. We expect to be able to do
things for, like move the mouse, grab the pointer. We want to put our windows where we want to put
our windows. Wayland will say, no, that's kind of our job. And you can't actually find out where
windows are on the screen. Also in the X-Wayland compositing, in the X-Wayland mode, it's, there's
the high DPI support is not complete. So in some of those things that I described above,
Soundlight, they're things that you'd only need maybe for a test framework, but these are actually
part of our standard APIs. And these, not many applications use them, but we have to be able
to support it. And there's a bunch of bug level fixes that we've found we need to do as well.
And we actually, you know, as the project went on, we actually found some bugs that were on the
Wayland side as well. And there's a whole bunch of, all of these things that are described in
detail at that URL, which I obviously, I'm not going to read out for you. So where are we now?
JDK 21 pretty much did all of the work. We got the spec changes in that we needed. And
there is a complete new implementation of the robot screen capture that was done almost entirely,
well actually entirely by Alexander Zaginsev. It's using the screencast portal and pipe wire.
And it, so the first time somebody tries to do a screen capture, there's a manual process of saying,
yes, that's fine. And then after that, it's okay forever in theory. And there are some
follow-up fixes going into JDK 22. We basically, if you have a desktop with GNOME 42, we should be good.
And that will probably mean that we'll be able to, vendors will be able to officially support
running on the Wayland desktop. In this compatibility mode with 22, which should ship in a month.
And that's when we shift real focus to the pure Wayland toolkit. But there's, you know,
what's involved a bit more about that. Complete new toolkit spans all of AWT in the window
management and the rendering side. So all of these things here, creating windows, configuring windows,
the management of everything, integration of the desktop, how you render to the surface. We can't
use X11, open GL, sorry, XGL really. Or X render, which we, an X11 and X render is the default way
we do everything on Linux today. Desktop integration, all of these different things I'm listening here
need to be redone. When I was trying to describe it to somebody who's sort of more of a VM person,
it's like, well, we need a new JIT and we need a new GC. And so that's the kind of scope of the work.
So how would you do this? Well, I mean, a lot of GTK4 makes it fairly easy for a lot of applications
to just, you know, port over because it deals with all of that, hides it from you. And then,
you know, you would, Wayland, one of the things, it doesn't have a window manager, so
it's client side decoration. It's all client side. And GTK would do that for you. And everything,
I think, there, it sounds like it'd be easy to get a lot of things up and running, but it brings in a
lot of baggage. If you do an LEDD or something on a running GTK4 process, you'll be paging through
it for a while. And, but really, the, one of the problems was it's just really hard to get the
level of management of when you render to the surface in the way that we need to with GTK4.
There's more work with using the low level LibWayland, which is basically the equivalent of LibX11.
But, you know, we've generally, when we've tried to do something with, in the JDK, like the most,
the last example was we were doing a pipeline for Mac OS for their new
rendering engine. And they have, like, a high level toolkit that's intended for applications,
but we needed to use the lower level one. And it just sort of works out that way,
seems, every time. But anyway, there's some new challenges that Wayland
brings that, native Wayland brings that aren't necessarily there, that aren't there in the
X compatibility mode. We need a new library Lib, EI, or A, however people pronounce it.
That's just prototyping in GNOME 45. Well, it's the API, I believe, is final, but, you know,
I think it's the first time it's been out there. That inability to layout windows that I touched on,
it has some oddities like splash screens come up in the top left-hand corner of the screen.
And, you know, that's not great from my perspective. So there is a, so there is a,
already a toolkit in process and Alexi is actually going to be showing you that right now.
And it's called WLToolkit. So, hand over to Alexi. Okay. Okay. Yeah. We use a separate thread for
event handling in our prototype called WLToolkit. And actual handlers of Wayland events are
dispatched on event dispatch thread. So, on Wayland rendering happens on client side. So, the client
need to notify the compositor about the region that should be updated. So, we need to have a complete
frame content and then submit it to the Wayland. That's required some changes in AWT and swing code
to make sure that all the content is ready for submitting.
Also, we use software rendering for our toolkit. Actually, software rendering was the only option
in early ages of Java platform for swing applications. But since then, the situation has changed and now
Java platform provides hardware acceleration for all major desktop platforms. Surprisingly,
in current WLToolkit implementation, we have sometimes better performance than in X2 kit
in X11 compatibility mode. So, for example, if you know a swing mark benchmark, it shows about
40% better performance than in X2 kit. So, it's quite enough for desktop applications. So, we can use it
now. However, there are some areas where we still have lower performance. For example,
in current implementation, we have about three times worse performance comparing with the
hardware-executed X2 kit. So, yeah, of course, modern desktop applications
need rich text capabilities, including our IDs. So, we're going to work on this and
improve the performance of front rendering. Our current plan is to use Vulkan as a hardware
acceleration with our WLToolkit. And let's see our demos. So, let's try to run them. It's quite
unexpected resolution here. Here, you can see a special aw.toolkit.name property, a standard
property that is set to WLToolkit to enable this toolkit for us. And that's run, as swing said.
Oh, yeah. It looks like it fit the screen. Okay. You may notice that here we have
unusual controls, actually, if you'll mention that. It's because Wayland, in this core part,
doesn't provide client-side decorations. So, these controls were rendered by ours,
so in WLToolkit. Okay. And let's see how it works, actually. So, here is frames. So, buttons.
So, we have some animated curve here. Some combo boxes, dialogues. Okay.
And some checkboxes working. Some more dialogues here. So, progress bar demo.
Scrollable image here in scroll pane. And sliders. Yeah, here we have split pane.
Tabot pane with some animated images here. Tables. Yeah, they work quite well. And tooltips.
Yeah. And tree with expanded nodes. So, as you can see, all the controls are properly rendered here.
And then I would like to show one more standard demo, at least demos bundled with
platform for many years to show. Swing UI from work capabilities. It's Java 2D demo. It shows some
advanced graphics. So, we can see here curve rendering. Actually, it's not full speed, so we can
reduce the delay and see how it works actually quite well. Yeah. So, we have many different things
here. Some clipping, color conversions, compositing rules, font rendering, image ops. So, it's some
conversions for images. Some stroking capabilities. Here's a mix it. Yeah, demo with different
primitives, paint modes. We have also here path rendering and transformations. So,
as you can see, performance is quite acceptable. Now, we'll try because of the resolution to
launch real-world application. Community version of IntelliJ IDEA. Yeah. Probably, it would be...
Wow. Yes, it works. Yeah.
And here we also see that we use the same property at little kid.name,
that in special property file that we use for IDs. Yeah, here we can see actual
well and implementation here. So, it's the constructor. It's quite difficult to see something.
And here is the separate thread that I mentioned that handles well and events.
Okay, that's it. Thanks for listening. Any questions if we have time? We have a minute.
So, any questions?
So, what is missing for... I'm repeating the question. You showed IntelliJ working, so what is
missing? It was in details. We have Piki users and there are some stuff that in some corner cases
that it's not working well, but it's generally workable. So, we have some users who
gave us feedback about the quality. So, yeah. But we still need to polish it.
Yeah, that... If it wasn't completely clear, right, you can try this for yourself. The project
that I actually... Didn't you have a slide showing the branch? So, you just can check out that branch
from the Wakefield repository and build it yourself and try it. Yep. Yes, over there.
It does it work with JavaFX? No, this is at this point, this is implementing within JD
K. JavaFX is like a separate toolkit entirely and we have to repeat this exercise for that.
Unfortunately. Yes, over there.
Sorry, feedback.
Did we fix bugs in Whalen? No, we reported them to...
We had friends at Red Hat. So, we've had some calls and some of the developers, a couple of the
developers who kind of work on the desktop and even on Whalen directly there and will say,
yeah, I file a bug here or yeah, I think it is. So, we've reported bugs and they've been fixed. Yes.
Yes. Any plans to support fractional scaling? Any plans to support fractional scaling? My
recollection is that Whalen itself fundamentally decided not to support fractional scaling.
There's an extension? There's a... Of course, there's an extension.
So, yeah, there's a protocol extension to do fractional scaling and if the
WR2 gets wants to implement that, it can do that normally. But it should work. It should
definitely already with the native Whalen mode, it should already be better with
then the whole blurriness sometimes gets an excapable. Yeah. The one thing about that though
is just generally, I mean, with fractional scaling, we don't have to deal with fractional
scaling on Mac because that's always multiple. With the Windows look and feel, we are still
sorting out bugs trying to make that work. So, we would undoubtedly find a whole bunch of new bugs
with the GTK look and feel when we started doing that. So, it's not just a simple matter of saying,
oh, yes, you know, there'll be a mess to be sorted out that's separate from the Whalen project.
I think we're probably out of time. Yeah.