It's 10.30 on the spot.
So this is the start of the first call.
Perfect.
Thank you.
Thank you, everyone.
All right, so just to give a little introduction,
I need to mute myself real quick.
Just to give a little introduction about myself,
my name is Alfred Neumeyer.
I've been involved with the UBeport's project
on Ubuntu Touch for about four, close to five years now.
And I have started out with porting devices over to,
Ubuntu Touch over to devices that used to run Android kernels
and Android drivers, not mainline.
I used and reused the high bits that we made up.
And for that matter, I gained a lot of experience
with the desktop later on because I started out with the phone
and I really wanted to have this tablet experience
with a touchpad and a keyboard.
I couldn't get it quite right until some later hardware
offered the ability to do so.
But in the meantime, I was very interested in the desktop talk
and what can we accomplish with the desktop convergence
as the keyword.
And I'm going to mention that one real quick.
Let's see if the convergence thing actually
does what it's supposed to do, right?
Okay, black screen.
Does it stay black?
There you go.
Okay, fingerprint reader doesn't want to work,
but this one does.
So, yeah, you can actually, with Limeria at least,
you can use the phone as the virtual touchpad
and virtual keyboard just to give it a little bit more
presentation feel.
We need to, where's the full screen?
Is there a full screen?
Do we have full screen?
Otherwise, that one is sufficient, I hope, right?
Now it's very obvious where it's running.
Sorry, broke it to you.
So, the Limeria on the desktop,
less of the pocket-sized desktop.
As we can see, it is totally possible to do so,
considering that we're still at the starting phase
of where this really can go.
So, I'm going to talk about a little bit of the history
then about the evil convergence word,
the word that we all like to use,
but so many people are using so many different senses,
so many different ways,
then how we migrate it over a few technology pieces
like the system layer and the windowing protocol,
the UI toolkit I'm going to introduce to you
and notable components that should you be interested
in either working with us or taking our stuff
and working on your own stuff,
bring it over to another distro,
you're totally free to do so,
and I will mention a few components
that you might be interested in.
So, what do we have?
The history.
2013, the year of Ubuntu Phone.
As we hoped, as some of us hoped,
as it did not drop until it really dropped,
and that was until 2016.
In 2013, they introduced the Lumiri Shell,
what's now called Lumiri,
they called it Unity 8 back then
as a continuation of their desktop efforts,
and they introduced MIR, the MIR server library
that allows developers of compositors
to easily create a new,
these Weyland-based compositor,
back then they used their own protocol,
which caused a lot of problems
for some people in the community,
but they turned around,
they implemented Weyland on top of their own
display stack, display server,
which is Weyland compatible nowadays,
so that's something.
2016, five devices have been released
with Ubuntu Touch pre-installed,
which is quite interesting for something
that was so niche.
Yay, that was so niche.
That gives me a hint that I should speed up talking.
So, five devices released,
one tablet, four smartphones,
and they all ran, like, Android adaptations
that they ported over using libhybers.
That's how they quickly made it to market,
very easily.
I know for some people that's not interesting at all,
but if you want to go to market really quickly,
that's sadly what they had to do,
and others before them also had to do it.
Even Karsten Monk, who started the libhybers project,
even himself, he said,
well, if you want to do it quickly,
you need to do it that way.
2017, Canonical gives the Ubuntu Touch project
to UbiPorts.
We took it over, Canonical couldn't finance it anymore.
They had no interest in keeping up with the project,
overall, and gave it over to us,
and we are since then trying to make the best out of it,
and we have released 24 OTAs,
and we're in the process of releasing our first final images
of focal-based Ubuntu Touch 20.04.
We already have a beta version,
which you can download and install in a select few of your devices,
but, yeah, and Unity 8 gets renamed to Lumiri.
Why? Because it was very important for others to package it up
without asking Canonical for every single license,
oh, copyright-related license question.
So we named everything from Canonical and Ubuntu to Lumiri
so that it's generic, others can make use of it,
and they can package it for their distributions.
So, convergence.
What do we mean by convergence? Goddamn.
Screen real estate, first of all.
Let's go back to the basics.
The screen real estate, how many things can I actually put on the phone,
or on a tablet, or on a desktop?
Various, very much, right?
What's the next property? Is it multi-monitor capable?
Is there a possibility that you can plug in an external monitor
into your phone and have it work, like, easily,
without much configuration?
And can you do so wisely, too? Wired, wisely?
Both of them can work.
And which input methods are available to your device
at the moment that you want to use it?
We support various types of input devices,
from touch, keyboard, everything that you expect nowadays
from a personal computing device,
which accounts phones into the mix as well.
And we are pretty much ready to deliver
input events to applications whenever they need to be delivered.
And they deliver everything that the application expects.
If there's a new protocol coming up,
we are pretty much future-proof due to the MIR team working on their stuff.
The way the MIR handles the input and passes it on to the compositor
is very much making it possible to increase the abilities
of input methods overall.
And that gives me a hint to shut up again.
So, convergence.
What is actually convergence?
In our mind, and that is the ability to adapt
to different usage scenarios.
It's not which device does it run on.
It doesn't mean what output device,
it does not only mean what output and input device do you plug in.
It also means how is it adapting.
So, for example, this phone, when you use the wireless display capability,
it's still a phone.
It's just got a wireless screen attached to it,
which displays everything that a normal monitor would do,
but it's still a phone.
It's not wired.
It's only determined whether something is in phone mode
or in desktop mode or in tablet mode,
and that's what the shell usually does.
I will get to the components and each of their responsibilities later.
But for now, the basics are convergence.
Adapt to a usage scenario that you see fit as the shell
or as a user, basically,
because we cater towards what the user might want
at a very specific moment in time.
Both the shell and the applications need to be able to adapt
without the applications adapting.
You have a great shell and you can't use anything with it, right?
So, we need the applications to follow the same rules,
to follow the same standards,
and make it easy so with our custom UI toolkit
that we have built based on Qt and QML.
It actually existed before Qt front trolls.
One was a thing.
So, canonical had to get something out of the sleeves real quickly
and they made up the nowadays called Lumiere UI toolkit.
Back then it was just Ubuntu toolkit.
And another thing that goes into the whole convergence thing
is pausing the application as you expect nowadays from a phone
where the UI thread of an application is paused.
It doesn't do anything.
How can we provide the same functionality
with a typical GNU slash Linux way of handling things?
Like, we don't have this side gold Android process
that spawns other services in the background
and it does everything quote unquote hidden away from the user.
We as application developers need to think differently about
how to approach a modern day usage scenario
with a smartphone being able to power save
as well as being feasible and doing stuff in the background
when it's supposed to do it.
Not to run in the background every time, all the time
because that's wasting battery.
But we need to make sure that the application lifecycle
is something that works on a phone
and it might work differently on a tablet
and it might work differently on a desktop.
It definitely does differently on a desktop, right?
You don't want an application to pause underneath your feed
or something, right?
Pulling the rug away.
So, we do this for preservation of battery life
and for that, we need different policies
for every usage scenario.
And we do that by using Lumiri App Launch
which is a little library that handles everything
related to launching the applications.
It puts everything in a C group as you expect nowadays.
Specifically, it uses system to user sessions for that
and it also helps with application confinement and security
because on open to touch at least,
the main goal is to provide a store
that other people can download applications from
without worrying too much whether their data
can be accessed or not.
Of course, we can trust every application
but then we would need to review them.
They're a typical traditional open source way of,
a Debian way, whatever you might want to call it,
of importing data, software, code and auditing it
and then releasing it to others.
That's not what we want to do.
We want to give developer easy access
to release their stuff to users that they care about.
And for that, we need application confinement
and a few security practices that are part
of the whole convergence story
of the application lifecycle model that we have.
And the result or at least the dream for Canonical was this.
You have a phone, a tablet, a laptop and even a TV
which can run the same shell.
And that's the last missing property of convergence.
You have one code base running everywhere.
You have one code that is sufficiently adaptive
to each situation that it works as you expect.
It looks similar enough.
The input methods, of course, differ
but in reality, they provide exactly what the user
wants at a specific moment in time.
Swipe on the phone, it works like a phone.
Use your remote control on the TV.
It works like a TV.
It looks similar.
It feels similar.
The user experience part, right?
So, next one.
Migrations to new technologies.
It took us some time.
It took us some time to move over to system D
but the concept is very much applied
because upstart back then already used to start stuff
as individual session services.
And our Lumiri app launch, it reacts to apps appearing
and disappearing for us.
It gives us the events.
Hey, application XYZ is gone.
Remove it from the launcher because it's gone
and remove all the resources that you have allocated to RAM.
Right?
Migration to new mirror and Wayland.
That's a more interesting one because that one is pain.
Mirror 2.0 drop support for the old protocol
that everyone complained about, right?
Current Lumiri on Ubuntu Touch does a split
like Charles Kloss and them.
And in that sense, it runs both mirror client and Wayland.
And through Wayland, it also supports X Wayland.
So, the Wayland support on mirror on Lumiri is good enough
that you can actually spawn a GIMP application
running through X Wayland on a libhybers device
or on a Wayland device depending on which X Wayland version
you install.
And it works.
The problem is we want to go Wayland everywhere.
And for Wayland everywhere, we need to adapt a few new concepts,
a few new things that are right at this moment in development
on the mirror side.
So, Miral, an abstraction layer which allows basic windowing
to be done to really receive data about a window being placed
somewhere and stuff like that.
Miral takes it another step further which gives us the ability
to get those trust prompts that we use on Ubuntu Touch.
Those dialogues that pop up underneath an application
and overlay themselves on top of an application
where you have to say, okay, I give this application access
to my GPS.
I give this application access to my microphone.
It doesn't do it by default.
It doesn't allow it.
You, as a user, need to allow it.
And for that, we need a few more components,
like trust prompt integration into Miral.
And screen casting is also going to be tough
because screen casting is just different enough
on mainline devices versus hybrid devices
that buffer passing might work.
But then how are we going to tell which buffer
looks like an Android buffer?
And I think I need to figure out a few things on my side
with regards to Wayland and how WL buffers
are supposed to be passed around.
But if anyone here has a lot of Wayland knowledge
and wants to help us out here and wants to have
some screen casting for themselves too,
please get in touch with me and we can figure out
something that works for both sides, I guess.
So the UI toolkit, Lumiri UI toolkit,
this is a beautiful thing if you ask me.
I really like this.
The way that our browser, for example,
that you can see the Chrome of on the top,
that's the Lumiri UI toolkit rendering the tabs.
And depending on whether you resize the window,
it actually turns into something that you have to swipe up
from the bottom edge of your phone.
Because this responsive design thing is one way to achieve
convergence in a sense.
So the UI toolkit, it follows our human interface guidelines,
allows for convergent use cases,
and it is used by the Lumiri shell throughout the code base.
So we have something that can be used by both apps
and the shell so that both look the same
compared to others who write one toolkit for the shell
and one toolkit for everything else.
And in this sense, we have something that works
on both sides of the equation.
It's abstracted by Qt, and Qt does everything for us.
But what it does not do for us is handle the wayland connections
for that we need Mir, and we need Qt Mir as an integration point
that takes Mir's window contents, the buffers,
and displays them on the screen,
or rather the scene graph of Qt.
So when you put things on the scene graph,
it's like this tree type of structure,
you put things on the screen, they get rendered,
and then they get displayed on the screen as you expect
using OpenGL.
And Qt Mir also already does basic window management,
but it doesn't do everything that Lumiri needs to do.
It already does multiple monitors and also supports
multiple input types, whatever you expect nowadays
from a display that can be driven with input devices, right?
Lumiri itself, the shell, it uses Qt Mir
for the application window presentation.
So everything that Qt Mir gets from Mir,
it gives further down to Lumiri, Lumiri displays it.
We can even put some shader effects onto those tiny windows
and make it a little colorful, make them explode
as you used to from compass times,
and have them render quite interesting stuff
just because of the way that Qt does animation in QML,
which is quite interesting.
And it also does more sophisticated window management,
so window snapping, multiple workspaces,
the whole thing that you expect, right?
And Ethercast, that's another one.
This one is not for the mainline folks yet,
but take this as an invitation for mainline folks
to help us get it working on the Pinephone Pro, for example,
which someone in the community has actually started
to port on, right?
And they got Ubuntu Touch running on the Pinephone Pro
with Lumiri, with everything that you expect,
even the trust prompts, because they required
a few patches to our side.
And Ethercast in this scenario wouldn't work
because it doesn't know how to take those DRM and GBM
allocated buffers and pass it on to an encoder.
We just need the screencasting and the video encoding
to be done quickly.
Then we can achieve sort of like a 20% CPU usage
on one core type of deal for wireless display support.
So 20% of one CPU core being in use,
for me that counts as nothing, almost, right?
It's definitely usable, and I regularly do it
with my Pixel 3a, and it works flawlessly, right?
And one last notable component, or two of them,
is the off-screen keyboard, which also plays into the whole
Lumiri story, so to say.
So, Malit. We use Malit as the off-screen keyboard framework,
and Lumiri keyboard is a plug into it,
which connects over Wayland or over Miracleint to the shell.
And says, hey, I'm here. Take me as your input device.
And the shell says, yes, I like you.
In that sense, the Lumiri keyboard, it uses the Lumiri UI
toolkit as well as you would expect.
And it also has some nice little features like you swipe up.
When you have the off-screen keyboard sliding up,
and you swipe up from the bottom edge,
you get to do text selections very, very nicely,
and you can also do the same with the spacebar,
with the space key.
That's a totally awesome thing to play around with.
And with that being done, I think...
We have five minutes left in total.
In total?
Yes.
Then we're going to skip the demo because you saw everything already.
Well, that's pretty much it.
If you have any questions, I will play around with the device.
We can interactively together use it, right?
OK, there's a question.
So if I want to use my phone as an input device on Gento,
do I install Malik or something?
As an input device on what?
Gento.
Are you planning to bring the shell over the Lumiri experience
to Gento, for that matter?
I want, for example, I have Gento, I have Wayland,
and I want to use my phone as a touchpad.
No, no, that's because it is integrated.
This is the virtual touchpad of the phone,
and it's because it's integrated with the shell,
with the Wayland desktop that we are running.
This is on screen.
This is Lumiri, yeah.
This is all integrated into Lumiri.
So Lumiri knows this is a virtual touchpad.
Let me do all the virtual touchpad stuff,
and it does so by...
But is it available as a separate package that I'm creating?
No, no, no, no.
It's integrated into the system's desktop.
We're not playing Lego here.
I'm sorry, I didn't want to sound mean,
but it's like those things need it to be integrated
because they want it to release very soon,
very quickly, very on time.
But they need to have a separate library, for example,
using different... It uses U-inputs,
the typical user space injects events into the shell.
That's all it does. U-inputs, U-input specifically.
Sorry for my intellectual environment.
No.
Is there a plan to have a separate package
that I can use in a different distribution
where I use a Wayland, for example?
This is Wayland. This is totally Wayland.
Where you use Wayland. This is a Wayland desktop.
This is the desktop, but it also does the virtual screen,
the virtual input method for you.
You could, in theory, create a new Android,
selfish, all of them, Mobian application
that just does that and transfers it over the wire
to a desktop, but this is not it.
This is all integrated.
You want something like synergy, but synergy is the tool
which allows you to, like, have a separate machine running
and then move your mouse over to the open screen border.
They want something like synergy, but then accept that
you use the mouse over yourself.
Right.
But synergy doesn't work at a Wayland,
but work is on the way to the way synergy works.
Oh, multiple workspaces. I haven't shown that you're that.
Multiple workspaces.
At a workspace. Hooray.
It's just connected and it's done.
We do detect certain properties of the screen,
whether it is a TV screen, projector, monitor,
integrated thing.
We have a whole enum full of different types of monitors
that can be connected.
And we also do a check by screen size,
by, I think, by scaling of the device.
So if you put it on an 11-inch device,
and depending on the pixel density,
it would behave differently, right?
But yeah.
That's pretty much how it looks like.
This is a terminal that I'm not going to input my password in.
Yeah?
This is the Fairphone 4.
All right.
The last questions.
Are you public?
OK.
Public can already start setting up stuff for the mix.
So that ends the demo part, I guess.
Yes.
Sorry for that.
But you still have some time for question answers while public.
All right.
Sure.
Thank you.